├── LICENSE ├── README.md ├── active_learning_gpml.m ├── auroc.m ├── covmin.m ├── demo.m ├── initWorkspace.m ├── min_kernel.m ├── queryFunctions ├── select_sample_gpImpact.m ├── select_sample_gpMean.m ├── select_sample_gpUncertainty.m ├── select_sample_gpVariance.m ├── select_sample_gpWeight.m └── select_sample_random.m └── roc-analysis ├── auroc.m └── roc.m /LICENSE: -------------------------------------------------------------------------------- 1 | GNU LESSER GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | 9 | This version of the GNU Lesser General Public License incorporates 10 | the terms and conditions of version 3 of the GNU General Public 11 | License, supplemented by the additional permissions listed below. 12 | 13 | 0. Additional Definitions. 14 | 15 | As used herein, "this License" refers to version 3 of the GNU Lesser 16 | General Public License, and the "GNU GPL" refers to version 3 of the GNU 17 | General Public License. 18 | 19 | "The Library" refers to a covered work governed by this License, 20 | other than an Application or a Combined Work as defined below. 21 | 22 | An "Application" is any work that makes use of an interface provided 23 | by the Library, but which is not otherwise based on the Library. 24 | Defining a subclass of a class defined by the Library is deemed a mode 25 | of using an interface provided by the Library. 26 | 27 | A "Combined Work" is a work produced by combining or linking an 28 | Application with the Library. The particular version of the Library 29 | with which the Combined Work was made is also called the "Linked 30 | Version". 31 | 32 | The "Minimal Corresponding Source" for a Combined Work means the 33 | Corresponding Source for the Combined Work, excluding any source code 34 | for portions of the Combined Work that, considered in isolation, are 35 | based on the Application, and not on the Linked Version. 36 | 37 | The "Corresponding Application Code" for a Combined Work means the 38 | object code and/or source code for the Application, including any data 39 | and utility programs needed for reproducing the Combined Work from the 40 | Application, but excluding the System Libraries of the Combined Work. 41 | 42 | 1. Exception to Section 3 of the GNU GPL. 43 | 44 | You may convey a covered work under sections 3 and 4 of this License 45 | without being bound by section 3 of the GNU GPL. 46 | 47 | 2. Conveying Modified Versions. 48 | 49 | If you modify a copy of the Library, and, in your modifications, a 50 | facility refers to a function or data to be supplied by an Application 51 | that uses the facility (other than as an argument passed when the 52 | facility is invoked), then you may convey a copy of the modified 53 | version: 54 | 55 | a) under this License, provided that you make a good faith effort to 56 | ensure that, in the event an Application does not supply the 57 | function or data, the facility still operates, and performs 58 | whatever part of its purpose remains meaningful, or 59 | 60 | b) under the GNU GPL, with none of the additional permissions of 61 | this License applicable to that copy. 62 | 63 | 3. Object Code Incorporating Material from Library Header Files. 64 | 65 | The object code form of an Application may incorporate material from 66 | a header file that is part of the Library. You may convey such object 67 | code under terms of your choice, provided that, if the incorporated 68 | material is not limited to numerical parameters, data structure 69 | layouts and accessors, or small macros, inline functions and templates 70 | (ten or fewer lines in length), you do both of the following: 71 | 72 | a) Give prominent notice with each copy of the object code that the 73 | Library is used in it and that the Library and its use are 74 | covered by this License. 75 | 76 | b) Accompany the object code with a copy of the GNU GPL and this license 77 | document. 78 | 79 | 4. Combined Works. 80 | 81 | You may convey a Combined Work under terms of your choice that, 82 | taken together, effectively do not restrict modification of the 83 | portions of the Library contained in the Combined Work and reverse 84 | engineering for debugging such modifications, if you also do each of 85 | the following: 86 | 87 | a) Give prominent notice with each copy of the Combined Work that 88 | the Library is used in it and that the Library and its use are 89 | covered by this License. 90 | 91 | b) Accompany the Combined Work with a copy of the GNU GPL and this license 92 | document. 93 | 94 | c) For a Combined Work that displays copyright notices during 95 | execution, include the copyright notice for the Library among 96 | these notices, as well as a reference directing the user to the 97 | copies of the GNU GPL and this license document. 98 | 99 | d) Do one of the following: 100 | 101 | 0) Convey the Minimal Corresponding Source under the terms of this 102 | License, and the Corresponding Application Code in a form 103 | suitable for, and under terms that permit, the user to 104 | recombine or relink the Application with a modified version of 105 | the Linked Version to produce a modified Combined Work, in the 106 | manner specified by section 6 of the GNU GPL for conveying 107 | Corresponding Source. 108 | 109 | 1) Use a suitable shared library mechanism for linking with the 110 | Library. A suitable mechanism is one that (a) uses at run time 111 | a copy of the Library already present on the user's computer 112 | system, and (b) will operate properly with a modified version 113 | of the Library that is interface-compatible with the Linked 114 | Version. 115 | 116 | e) Provide Installation Information, but only if you would otherwise 117 | be required to provide such information under section 6 of the 118 | GNU GPL, and only to the extent that such information is 119 | necessary to install and execute a modified version of the 120 | Combined Work produced by recombining or relinking the 121 | Application with a modified version of the Linked Version. (If 122 | you use option 4d0, the Installation Information must accompany 123 | the Minimal Corresponding Source and Corresponding Application 124 | Code. If you use option 4d1, you must provide the Installation 125 | Information in the manner specified by section 6 of the GNU GPL 126 | for conveying Corresponding Source.) 127 | 128 | 5. Combined Libraries. 129 | 130 | You may place library facilities that are a work based on the 131 | Library side by side in a single library together with other library 132 | facilities that are not Applications and are not covered by this 133 | License, and convey such a combined library under terms of your 134 | choice, if you do both of the following: 135 | 136 | a) Accompany the combined library with a copy of the same work based 137 | on the Library, uncombined with any other library facilities, 138 | conveyed under the terms of this License. 139 | 140 | b) Give prominent notice with the combined library that part of it 141 | is a work based on the Library, and explaining where to find the 142 | accompanying uncombined form of the same work. 143 | 144 | 6. Revised Versions of the GNU Lesser General Public License. 145 | 146 | The Free Software Foundation may publish revised and/or new versions 147 | of the GNU Lesser General Public License from time to time. Such new 148 | versions will be similar in spirit to the present version, but may 149 | differ in detail to address new problems or concerns. 150 | 151 | Each version is given a distinguishing version number. If the 152 | Library as you received it specifies that a certain numbered version 153 | of the GNU Lesser General Public License "or any later version" 154 | applies to it, you have the option of following the terms and 155 | conditions either of that published version or of any later version 156 | published by the Free Software Foundation. If the Library as you 157 | received it does not specify a version number of the GNU Lesser 158 | General Public License, you may choose any version of the GNU Lesser 159 | General Public License ever published by the Free Software Foundation. 160 | 161 | If the Library as you received it specifies that a proxy can decide 162 | whether future versions of the GNU Lesser General Public License shall 163 | apply, that proxy's public statement of acceptance of any version is 164 | permanent authorization for you to choose that version for the 165 | Library. 166 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | COPYRIGHT 2 | ========= 3 | 4 | This package contains Matlab source code for active learning experiments with Gaussian process regression models as described in: 5 | 6 | Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler: 7 | "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 8 | Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 9 | 10 | Please cite that paper if you are using this code! 11 | 12 | (LGPL) copyright by Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler 13 | 14 | 15 | CONTENT 16 | ======= 17 | 18 | initWorkspace.m 19 | demo.m 20 | active_learning_gpml.m 21 | queryFunctions/select_sample_random.m 22 | queryFunctions/select_sample_gpMean.m 23 | queryFunctions/select_sample_gpVariance.m 24 | queryFunctions/select_sample_gpUncertainty.m 25 | queryFunctions/select_sample_gpWeight.m 26 | queryFunctions/select_sample_gpImpact.m 27 | roc-analysis/roc.m 28 | roc-analysis/auroc.m 29 | README.txt 30 | License.txt 31 | 32 | 33 | USAGE 34 | ===== 35 | 36 | First download the GPML (see below) and change the gpml path in initWorkspace.m 37 | 38 | - run initWorkspace 39 | 40 | - getting to know: 41 | (1) run demo.m, which performs an AL experiment on synthetically created 5D-histograms. Change the settings 42 | in the demo file to figure out how everything works. Don't forget to set up the gpml directory accordingly! 43 | 44 | - run an AL experiment: 45 | (1) Use the function "active_learning_gpml" to run a comparison of different AL-strategies for a given scenario 46 | (2) Please refer to the documentation in active_learning_gpml.m for explanations of input and output variables. 47 | 48 | 49 | 50 | NOTE 51 | ==== 52 | 53 | To keep things simple, GP models are computed using the GPML toolbox: 54 | 55 | http://mloss.org/revision/download/1206/ 56 | 57 | @MISC{Rasmussen10:GPML, 58 | author = {C. E. {Rasmussen} and H. {Nickisch}}, 59 | title = {GPML Gaussian Processes for Machine Learning Toolbox}, 60 | year = {2010}, 61 | note = {\url{http://mloss.org/software/view/263/}}, 62 | } 63 | 64 | For computing kernel values needed within the experiments, you can rely on the GPML toolbox as well. 65 | 66 | -------------------------------------------------------------------------------- /active_learning_gpml.m: -------------------------------------------------------------------------------- 1 | % Main method for an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function [perf_values times chosenIndices optimizedParameters]= active_learning_gpml(labeled_X, labeled_y, unlabeled_X, unlabeled_y, test_X, test_y, positive_class, numQueries, settings, queryStrategies, optimizationStep, verbose) 11 | % 12 | % BRIEF: 13 | % Perform an active-learning experiment, i.e., initially train a GP model 14 | % from labeled training examples, query new samples from a pool of 15 | % unlabeled examples, and evaluate the performance after every query on a 16 | % disjoint test set 17 | % 18 | % INPUT: 19 | % labeled_X -- (n_t x d) matrix with n_t training samples (rows) consisting of d dimensions (columns) 20 | % labeled_y -- (n_t x 1) column vector containing (multi-class) labels of n_t training samples 21 | % unlabeled_X -- (n_u x d) matrix with n_u samples (rows) consisting of d dimensions (columns) 22 | % unlabeled_y -- (n_u x 1) column vector containing (multi-class) 23 | % labels of n_u samples used as unlabeled pool within an AL experiment 24 | % test_X -- (n_t x d) matrix with n_t test samples (rows) consisting of d dimensions (columns) 25 | % test_y -- (n_t x 1) column vector containing (multi-class) labels of n_t test samples 26 | % positive_class -- (1 x 1) scalar indicating the multi-class label 27 | % of the class to be used as positive one 28 | % numQueries -- (1 x 1) scalar, how often do we want to query unlabeled examples in a single run? 29 | % settings -- (n x 1) column vector containing (multi-class) labels of n training samples 30 | % queryStrategies -- (1 x k) cell array, specify k query strategies 31 | % that shall be compared 32 | % Right now, the following strategies are 33 | % supported: 34 | % - random 35 | % - gp-mean (minimum absolute predictive mean) 36 | % - gp-var (maximum predictive variance) 37 | % - gp-unc (minimum "uncertainty") 38 | % - gp-weight (largest weight after update) 39 | % - gp-impact (largest model change after update) 40 | % optimizationStep -- (1 x 1) scalar, after how many AL-iterations do we want to perform 41 | % a re-optimization of current hyperparameters? 42 | % verbose -- (1 x 1) scalar, give some interesting output? 43 | % 44 | % OUTPUT: 45 | % perf_values -- the actual accuracies after every query step 46 | % (length(queryStrategies) x numQueries+1) 47 | % times -- the computation times needed for every run 48 | % (including training, computation of kernel values, and so on) 49 | % ( length(queryStrategies) x 1 ) 50 | % chosenIndices -- the queried indices for every run 51 | % (numQueries x length(queryStrategies) ) 52 | % optimizedParameters -- optional, if given, optimization is performed 53 | % every optimizationStep steps 54 | % if optimizationStep == 0: ( length(hyp.cov) + length(hyp.lik)) 55 | % else: ( length(hyp.cov)+length(hyp.lik) x floor(numQueries/optimizationStep)+1 x length(queryStrategies)) 56 | % 57 | % NOTE: 58 | % This program uses the GPML-Toolbox provided by 59 | % 60 | % 61 | % (C) copyright by Alexander Freytag and Erik Rodner and Paul Bodesheim and Joachim Denzler 62 | % 63 | 64 | function [perf_values times chosenIndices optimizedParameters]= active_learning_gpml(labeled_X, labeled_y, unlabeled_X, unlabeled_y, test_X, test_y, positive_class, numQueries, settings, queryStrategies, optimizationStep, verbose) 65 | % function [perf_values times chosenIndices optimizedParameters] = active_learning_gpml(labeled_X, labeled_y, unlabeled_X, unlabeled_y, test_X, test_y, positive_class, numQueries, settings, queryStrategies, optimizationStep, verbose) 66 | 67 | 68 | %% are we running octave? 69 | b_isOctave = exist('OCTAVE_VERSION') ~= 0; 70 | 71 | %% 72 | 73 | % relevant settings 74 | gpnoise = settings.gpnoise; 75 | %convert to gpml-scale 76 | gpnoise = 0.5*log(gpnoise); 77 | cov = settings.cov; 78 | loghyper.cov = settings.loghyper; 79 | lik = @likGauss; %standard assumption of Gaussian distribution y~f 80 | inf = @infExact; 81 | Ncg = 50; % number of conjugate gradient steps, same as in GPML toolbox 82 | loghyper.lik = gpnoise; 83 | mean = @meanZero; % zero-mean assumption 84 | loghyper.mean=[]; 85 | 86 | 87 | if nargin < 12 88 | verbose = false; 89 | end 90 | if nargin < 11 91 | optimizationStep = 0; 92 | end 93 | 94 | %convert multi-class labels to binary labels for the specified task 95 | current_labels = 2*(labeled_y==positive_class)-1; 96 | 97 | %optimization? 98 | %right now, we always perform an initial optimization! 99 | if Ncg==0 100 | hyp = loghyper; 101 | else 102 | hyp = minimize(loghyper,'gp', -Ncg, inf, mean, cov, lik, labeled_X, current_labels); % opt hypers 103 | end 104 | 105 | if ( optimizationStep == 0) 106 | optParams = [hyp.cov; hyp.lik]; 107 | else 108 | %pre-allocate memory 109 | optParams = zeros(length(hyp.cov)+length(hyp.lik), floor(numQueries/optimizationStep)+1,length(queryStrategies)); 110 | %store initially optimized parameters 111 | optParams(:,1,1:length(queryStrategies)) = repmat([hyp.cov; hyp.lik],1,length(queryStrategies) ); 112 | end 113 | 114 | 115 | %training 116 | %[~, ~, post] = gp(hyp, inf, mean, cov, lik, labeled_X, current_labels); 117 | % for octave compatibility 118 | [dummy1, dummy2, post] = gp(hyp, inf, mean, cov, lik, labeled_X, current_labels); 119 | alpha=post.alpha; 120 | L = post.L; 121 | sW = post.sW; 122 | sn2 = exp(2*hyp.lik); % noise variance of likGauss 123 | 124 | % init output variable 125 | perf_values = zeros( length(queryStrategies), numQueries+1 ); 126 | times = zeros ( length(queryStrategies),1 ); 127 | 128 | %copute similarities to test samples 129 | Ks = feval(cov{:},hyp.cov, labeled_X, test_X); 130 | 131 | %initial test evaluation 132 | classification_mu = Ks'*alpha; 133 | [tp fp] = roc( (test_y == positive_class), classification_mu); 134 | perf_values(1:length(queryStrategies),1) = auroc(tp,fp); 135 | 136 | if (verbose) 137 | sprintf('score (AuROC): %d',perf_values(1,1)) 138 | disp('first model evaluation') 139 | end 140 | 141 | initialClassifier.L = L; 142 | initialClassifier.alpha = alpha; 143 | initialClassifier.sn2 = sn2; 144 | initialClassifier.hyp = hyp; 145 | initialClassifier.Ks = Ks; 146 | 147 | 148 | chosenIndices = zeros( numQueries,length(queryStrategies) ); 149 | %store the indices of the unlabeled samples, that have been already 150 | %queried 151 | ind_labeled = []; 152 | ind_unlabeled = 1:length(unlabeled_y); 153 | %indices for test sets are not needed, since the test set is not affected 154 | %during active learning experiments 155 | 156 | initialClassifier.ind_labeled = ind_labeled; 157 | initialClassifier.ind_unlabeled = ind_unlabeled; 158 | 159 | for stratIndex=1:length(queryStrategies) 160 | 161 | if (verbose) 162 | disp(sprintf('Method %d of %d: %s', stratIndex, length(queryStrategies), queryStrategies{stratIndex})); 163 | end 164 | 165 | %select the strategy we want to use for actively querying new samples 166 | switch queryStrategies{stratIndex} 167 | case 'random' 168 | activeSelectionMethod = @select_sample_random; 169 | case 'gp-mean' 170 | activeSelectionMethod = @select_sample_gpMean; 171 | case 'gp-var' 172 | activeSelectionMethod = @select_sample_gpVariance; 173 | case 'gp-unc' 174 | activeSelectionMethod = @select_sample_gpUncertainty; 175 | case 'gp-weight' 176 | activeSelectionMethod = @select_sample_gpWeight; 177 | case 'gp-impact' 178 | activeSelectionMethod = @select_sample_gpImpact; 179 | otherwise 180 | error('wrong mode or not yet implemented'); 181 | end 182 | 183 | %reset the classifier to the initial version - thereby identical 184 | %initial conditions for different AL strategies are guaranteed 185 | L = initialClassifier.L; 186 | alpha = initialClassifier.alpha; 187 | sn2 = initialClassifier.sn2; 188 | hyp = initialClassifier.hyp; 189 | Ks = initialClassifier.Ks; 190 | 191 | ind_labeled = initialClassifier.ind_labeled; 192 | ind_unlabeled = initialClassifier.ind_unlabeled; 193 | 194 | 195 | if ( b_isOctave ) 196 | timeStamp=tic(); 197 | else 198 | timeStamp=tic; 199 | end 200 | 201 | %start actively querying new samples 202 | if ( numQueries > 0) 203 | 204 | Ks_unlabeled = feval(cov{:}, hyp.cov, [labeled_X;unlabeled_X(ind_labeled,:)], unlabeled_X(ind_unlabeled,:) ); 205 | Kss_unlabeled = feval(cov{:}, hyp.cov, unlabeled_X(ind_unlabeled,:), 'diag'); 206 | 207 | for i=2:numQueries+1 208 | 209 | % select next sample based on specified strategy 210 | 211 | %note: selected_sample is only the reletive index between 1 and 212 | %the current size of unlabeled samples -> convert it to the 213 | %original index afterwards 214 | selected_sample = activeSelectionMethod(Ks_unlabeled, Kss_unlabeled, L, alpha, hyp.lik); 215 | 216 | if (verbose) 217 | disp(sprintf('Query %d: sample of class %d (positive class = %d)',i-1,unlabeled_y(selected_sample),positive_class)); 218 | end 219 | % update sets of labeled and unlabeled samples 220 | originalIndexSelectedSample = ind_unlabeled(selected_sample); 221 | ind_labeled = [ind_labeled; originalIndexSelectedSample]; 222 | ind_unlabeled(selected_sample) = []; 223 | 224 | current_labels = 2*([labeled_y; unlabeled_y(ind_labeled)]==positive_class)-1; 225 | 226 | %%% re-run optimization or just a simple update? 227 | if ( (Ncg~=0) && (mod(i, optimizationStep) == 0) ) 228 | %re-run optimization 229 | hyp = minimize(loghyper,'gp', -Ncg, inf, mean, cov, lik, [labeled_X;unlabeled_X(ind_labeled,:)], current_labels); % opt hypers 230 | 231 | optParams(:,floor(i/optimizationStep)+1,stratIndex) = [hyp.cov; hyp.lik]; 232 | 233 | %re-train with newly optimized hyperparameters 234 | %[~, ~, post] = gp(hyp, inf, mean, cov, lik, [labeled_X;unlabeled_X(ind_labeled,:)], current_labels); 235 | % for octave compatibility 236 | [dummy1, dummy2, post] = gp(hyp, inf, mean, cov, lik, [labeled_X;unlabeled_X(ind_labeled,:)], current_labels); 237 | alpha=post.alpha; 238 | L = post.L; 239 | sn2 = exp(2*hyp.lik); % noise variance of likGauss 240 | 241 | %do we have to re-compute the kernel values? 242 | if (size(hyp.cov) == 0) 243 | %just update kernel values 244 | 245 | % update similarities of unlabeled samples 246 | Ks_unlabeled(:,selected_sample)=[]; 247 | Ks_unlabeled = [Ks_unlabeled; feval(cov{:},hyp.cov, unlabeled_X(ind_labeled(length(ind_labeled)),:), unlabeled_X(ind_unlabeled,:) )]; 248 | 249 | Kss_unlabeled(selected_sample) = []; 250 | 251 | Ks = [Ks; feval(cov{:},hyp.cov, unlabeled_X(ind_labeled(length(ind_labeled)),:), test_X)]; 252 | else 253 | %re-compute kernel values using the new hyperparameters 254 | 255 | %copute similarities to test samples 256 | Ks = feval(cov{:},hyp.cov, [labeled_X;unlabeled_X(ind_labeled,:)], test_X); 257 | %copute similarities to unlabeled samples 258 | Ks_unlabeled = feval(cov{:}, hyp.cov, [labeled_X;unlabeled_X(ind_labeled,:)], unlabeled_X(ind_unlabeled,:) ); 259 | Kss_unlabeled = feval(cov{:}, hyp.cov, unlabeled_X(ind_unlabeled,:), 'diag'); 260 | end 261 | 262 | else % no optimization 263 | % Cholesky update to include the selected sample 264 | l = (L'\ ( Ks_unlabeled(:,selected_sample)))/sn2; 265 | kss = ( feval(cov{:},hyp.cov,unlabeled_X(length(ind_labeled),:)) )/sn2; 266 | lss = sqrt(kss+1-l'*l); 267 | L = [L, l; zeros(1,length(l)), lss ]; 268 | 269 | % update similarities of unlabeled samples 270 | Ks_unlabeled(:,selected_sample) = []; 271 | Ks_unlabeled = [Ks_unlabeled; feval(cov{:},hyp.cov, unlabeled_X(ind_labeled(length(ind_labeled)),:), unlabeled_X(ind_unlabeled,:))]; 272 | 273 | Kss_unlabeled(selected_sample) = []; 274 | 275 | Ks = [Ks; feval(cov{:},hyp.cov, unlabeled_X(ind_labeled(length(ind_labeled)),:), test_X)]; 276 | end 277 | 278 | 279 | % determine new alpha and update Ks for testing 280 | alpha = solve_chol(L,current_labels)/sn2; 281 | 282 | 283 | % evaluate current model 284 | classification_mu = Ks'*alpha; 285 | 286 | [tp, fp] = roc( (test_y==positive_class), classification_mu ); 287 | perf_values(stratIndex,i) = auroc(tp,fp); 288 | 289 | end 290 | else 291 | % disp('No queries to take, therefor we are done here.') 292 | end 293 | 294 | if ( b_isOctave ) 295 | % OCTAVE compatibility 296 | times(stratIndex)=toc(); 297 | else 298 | times(stratIndex)=toc(timeStamp); 299 | end 300 | 301 | chosenIndices(:,stratIndex) = ind_labeled; 302 | 303 | end %for-loop over query strategies 304 | 305 | if (nargin > 3) 306 | optimizedParameters = optParams; 307 | end 308 | 309 | end 310 | -------------------------------------------------------------------------------- /auroc.m: -------------------------------------------------------------------------------- 1 | function aucScore = auroc(tp, fp) 2 | n = size(tp, 1); 3 | aucScore = sum((fp(2:n) - fp(1:n-1)).*(tp(2:n)+tp(1:n-1)))/2; 4 | end -------------------------------------------------------------------------------- /covmin.m: -------------------------------------------------------------------------------- 1 | function K = covmin(hyp, x, z) 2 | % See also COVFUNCTIONS.M. 3 | 4 | if nargin<2, K = '0'; return; end % report number of parameters 5 | if nargin<3, z = []; end % make sure, z exists 6 | xeqz = numel(z)==0; dg = strcmp(z,'diag') && numel(x)>0; % determine mode 7 | 8 | ell = exp(0); 9 | % ell = exp(hyp(1)); % exponent for histogram entries 10 | sf2 = exp(0); 11 | % sf2 = exp(hyp(2)); % factor for histogram entries 12 | 13 | % precompute min kernel 14 | if dg % vector kxx 15 | K = sum(x,2); 16 | % K = ones(size(x,1),1); 17 | else 18 | if xeqz % symmetric matrix Kxx 19 | K = sf2*min_kernel(x'.^ell); 20 | else % cross covariances Kxz 21 | K = sf2*min_kernel(x'.^ell,z'.^ell); 22 | end 23 | end 24 | 25 | if nargin<4 % covariances 26 | 27 | else % derivatives 28 | error('not yet implemented') 29 | end 30 | -------------------------------------------------------------------------------- /demo.m: -------------------------------------------------------------------------------- 1 | %% set up workspace and matlab path 2 | initWorkspace 3 | 4 | %% are we running octave? 5 | b_isOctave = exist('OCTAVE_VERSION') ~= 0; 6 | 7 | %% sample some random histograms 8 | 9 | dim = 5; 10 | m = 3.0; 11 | var = 1.0; 12 | 13 | offsetSpecific = 2.0; 14 | varSpecific = 2.0; 15 | 16 | 17 | %sample some training data 18 | nTrain = 2; 19 | % we simulate histograms since we use HIK as similarity measure in many 20 | % computer vision applications 21 | labeled_X = abs ( m + var.*randn(2*nTrain,dim) ); 22 | labeled_X(1:nTrain,1) = abs ( labeled_X(1:nTrain,1) + offsetSpecific + varSpecific.*randn(nTrain,1) ); 23 | labeled_X(nTrain+1:2*nTrain,3) = abs ( labeled_X(nTrain+1:2*nTrain,3) + offsetSpecific + varSpecific.*randn(nTrain,1) ); 24 | labeled_X = bsxfun(@times, labeled_X, 1./(sum(labeled_X, 2))); 25 | 26 | labeled_y = [ ones(nTrain,1); 2 * ones(nTrain,1) ]; 27 | 28 | %sample some data to query from 29 | nUnlabeled = 100; 30 | unlabeled_X = abs ( m + var.*randn(2*nUnlabeled,dim) ); 31 | unlabeled_X(1:nUnlabeled,1) = abs ( unlabeled_X(1:nUnlabeled,1) + offsetSpecific + varSpecific.*randn(nUnlabeled,1) ); 32 | unlabeled_X(nUnlabeled+1:2*nUnlabeled,3) = abs ( unlabeled_X(nUnlabeled+1:2*nUnlabeled,3) + offsetSpecific + varSpecific.*randn(nUnlabeled,1) ); 33 | unlabeled_X = bsxfun(@times, unlabeled_X, 1./(sum(unlabeled_X, 2))); 34 | 35 | %simulate your user response for this experiment 36 | unlabeled_y = [ ones(nUnlabeled,1); 2 * ones(nUnlabeled,1) ]; 37 | 38 | %sample some data to test on 39 | nTest = 100; 40 | test_X = abs ( m + var.*randn(2*nTest,dim) ); 41 | test_X(1:nTest,1) = abs ( test_X(1:nTest,1) + offsetSpecific + varSpecific.*randn(nTest,1) ); 42 | test_X(nTest+1:2*nTest,3) = abs ( test_X(nTest+1:2*nTest,3) + offsetSpecific + varSpecific.*randn(nTest,1) ); 43 | test_X = bsxfun(@times, test_X, 1./(sum(test_X, 2))); 44 | 45 | test_y = [ ones(nTest,1); 2 * ones(nTest,1) ]; 46 | 47 | %% set up variables and stuff 48 | 49 | % index of positive class 50 | positive_class = 1; 51 | 52 | % how many images do we want to query 53 | numQueries = 50; 54 | 55 | % further settings related to the classifier used 56 | settings.gpnoise = 0.1; 57 | settings.gp_numRegressors = 0.25; 58 | settings.binarySVM_outlierratio = 0.1; 59 | settings.cov = {'covmin'}; 60 | settings.loghyper = []; 61 | settings.balanced_classification = false; 62 | 63 | % which strategies do we want to compare? 64 | %chose a subset of the currently supported ones: {'random','gp-mean', 'gp-var', 'gp-unc', 'gp-weight', 'gp-impact'}; 65 | queryStrategies = {'gp-mean', 'gp-impact'}; 66 | 67 | % after how many steps shall we perform an optimization of hyperparameters? 68 | % note that usually hyperparameters vary less heavily given new input data 69 | % than your current model might do 70 | optimizationStep = 5; 71 | 72 | % print additional debug information? 73 | verbose = false; 74 | 75 | 76 | %% run the active learning experiment with your settings and data 77 | [perf_values alTimes chosenIndices optimizedParameters]= active_learning_gpml ... 78 | (labeled_X, labeled_y, unlabeled_X, unlabeled_y, test_X, test_y, positive_class, numQueries, settings, queryStrategies, optimizationStep, verbose); 79 | 80 | 81 | 82 | %% plot the results 83 | %specific color coding for the GCPR-AL paper 84 | c={'k-', 'b-', 'c-', 'b--', 'g-', 'm-'}; 85 | linewidth=3; 86 | 87 | clf; 88 | perfPlot=figure(1); 89 | hold on; 90 | for i=1:size(perf_values,1) 91 | plot( perf_values(i,:), c{i}, 'LineWidth', 2*linewidth, 'MarkerSize',8 ); 92 | end 93 | if b_isOctave 94 | legend(queryStrategies); 95 | else 96 | legend(queryStrategies, 'Location', 'SouthEast','fontSize', 16,'LineWidth', 3); 97 | end 98 | text_h=findobj(gca,'type','text'); 99 | set(text_h,'FontSize',14); 100 | set(gca, 'FontSize', 14); 101 | set(get(gca,'YLabel'), 'FontSize', 14); 102 | set(get(gca,'XLabel'), 'FontSize', 14); 103 | title('Performance comparison'); 104 | xlabel('Number of queries'); 105 | ylabel('AUC [%]'); 106 | hold off; 107 | 108 | timePlot=figure(2); 109 | bar( alTimes ); 110 | 111 | text_h=findobj(gca,'type','text'); 112 | set(text_h,'FontSize',14); 113 | set(gca, 'FontSize', 14); 114 | set(get(gca,'YLabel'), 'FontSize', 14); 115 | set(get(gca,'XLabel'), 'FontSize', 14); 116 | title('Time needed'); 117 | xlabel('Method'); 118 | ylabel('Time for AL Exp [s]'); 119 | hold off; 120 | 121 | 122 | paramPlot=figure(3); 123 | hold on; 124 | for i=1:size(optimizedParameters,3) 125 | plot( optimizedParameters(1,:,i), c{i}, 'LineWidth', 2*linewidth, 'MarkerSize',8 ); 126 | end 127 | if b_isOctave 128 | legend(queryStrategies); 129 | else 130 | legend(queryStrategies, 'Location', 'SouthEast','fontSize', 16,'LineWidth', 3); 131 | end 132 | text_h=findobj(gca,'type','text'); 133 | set(text_h,'FontSize',14); 134 | set(gca, 'FontSize', 14); 135 | set(get(gca,'YLabel'), 'FontSize', 14); 136 | set(get(gca,'XLabel'), 'FontSize', 14); 137 | title('Parameter development'); 138 | xlabel('Number of queries'); 139 | ylabel('Noise Parameter'); 140 | hold off; 141 | 142 | pause; 143 | close( perfPlot ); 144 | close( timePlot ); 145 | close( paramPlot ); 146 | -------------------------------------------------------------------------------- /initWorkspace.m: -------------------------------------------------------------------------------- 1 | %NOTE don't miss to set up your matlab path to the gpml toolbox 2 | GPMLDIR = '~/code/matlab/gpml'; 3 | addpath(genpath(GPMLDIR)); 4 | 5 | addpath('queryFunctions'); 6 | addpath('roc-analysis'); -------------------------------------------------------------------------------- /min_kernel.m: -------------------------------------------------------------------------------- 1 | function C = min_kernel(a, b) 2 | 3 | if nargin<1 || nargin>2 || nargout>1, error('Wrong number of arguments.'); end 4 | bsx = exist('bsxfun','builtin'); % since Matlab R2007a 7.4.0 and Octave 3.0 5 | if ~bsx, bsx = exist('bsxfun'); end % bsxfun is not yes "builtin" in Octave 6 | [D, n] = size(a); 7 | 8 | if nargin==1 % subtract mean 9 | b = a; m = n; 10 | else 11 | [d, m] = size(b); 12 | if d ~= D, error('Error: column lengths must agree.'); end 13 | end 14 | 15 | if bsx % compute squared distances 16 | C = zeros(n,m); 17 | % This code is working but still inefficient :( 18 | if ( n > m ) 19 | for i=1:m 20 | C(:,i)=sum(bsxfun(@min,a,b(:,i)),1)'; 21 | end 22 | else 23 | for j=1:n 24 | C(j,:)=sum(bsxfun(@min,a(:,j),b),1); 25 | end 26 | end 27 | 28 | else 29 | error('not yet implemented...'); 30 | end 31 | C = max(C,0); % numerical noise can cause C to negative i.e. C > -1e-14 32 | -------------------------------------------------------------------------------- /queryFunctions/select_sample_gpImpact.m: -------------------------------------------------------------------------------- 1 | % Query a new sample (with maximum absolute change on the updated weight vector) within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_gpImpact(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 11 | % 12 | % BRIEF: 13 | % Query a new sample from a pool of unlabeled ones such that the queried sample 14 | % results in the largest absolute change of the updated alpha vector of the GP regression model. 15 | % 16 | % INPUT: 17 | % Ks_unlabeled -- (n_t x n_u1) matrix of self similarities between n_t training samples and n_u unlabeled samples 18 | % Kss_unlabeled -- (n_u x 1) vector of self similarities of n_u unlabeled samples 19 | % L -- (n_t x n_t) cholesky matrix (upper triangle) computed from the 20 | % regularized kernel matrix of all training samples 21 | % alpha -- (n_t x 1) column vector, weight vector of GP 22 | % regression model 23 | % gpnoise -- (1 x 1) scalar, indicating the noise for 24 | % used for model regularization 25 | % 26 | % OUTPUT: 27 | % selected_sample -- scalar, index of chosen sample 28 | % 29 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 30 | % 31 | 32 | function selected_sample = select_sample_gpImpact(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 33 | mu = Ks_unlabeled'*alpha; 34 | 35 | sn2 = exp(2*gpnoise); 36 | V = L'\Ks_unlabeled; 37 | var = max(0,Kss_unlabeled-sum(V.*V./(sn2))'); 38 | 39 | % --- compute the first term --- 40 | % which downweights the impact of a new sample by the amount of 41 | % predicted uncertainty 42 | % note: actually, the first term is the invers of this number, but we 43 | % call the rdivide method lateron for easier computation 44 | firstTerm = sqrt(var.^2+sn2^2); 45 | 46 | % --- compute the second term --- 47 | % solve (K+\sigmaI)^-1 \cdot k_unlabeled for every unlabeled example 48 | % size of Ks_unlabeled: #Training \times #Unlabeled 49 | simVec=(L\V)/sn2; 50 | 51 | % append a-priori weight of new sample 52 | simVec=[simVec;-ones(1,size(mu,1))]; 53 | 54 | % the actual second term is the norm of the un-normalized alpha changes 55 | % as norm we simply take the L1 norm 56 | secondTerm=sum(abs(simVec),1); 57 | 58 | % --- compute the third term --- 59 | % this is the difference between predicted label and GT label 60 | % since we have not GT info, we take the sign of the most plausible class 61 | % -> in the binary setting, this is simply the sign of the mean value 62 | thirdTerm=mu-sign(mu); 63 | 64 | % we are interested in overall changes, so we take the abs 65 | thirdTermAbs=abs(thirdTerm); 66 | 67 | % --- combine (multiply) all three terms --- 68 | % use the built-in function for element-wise division 69 | % where the vector is automatically extended in the second dimension 70 | % towards the right size 71 | % 72 | impact=(thirdTermAbs./firstTerm).*secondTerm'; 73 | 74 | % take the one with the largest impact 75 | % that is, we chose the point which changes the model (weight vector 76 | % alpha) most heavily 77 | %[~, selected_sample]=max(impact); 78 | % OCTAVE compatbility 79 | [dummy1, selected_sample]=max(impact); 80 | end 81 | -------------------------------------------------------------------------------- /queryFunctions/select_sample_gpMean.m: -------------------------------------------------------------------------------- 1 | % Query a new sample (with minimum absolute predictive mean) within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_gpMean(Ks_unlabeled, ~, ~, alpha, ~) 11 | % 12 | % BRIEF: 13 | % Query a new sample from a pool of unlabeled ones such that the queried sample 14 | % has the smallest absolute predictive mean among all possible samples. 15 | % This query strategy was introduced by Kapoor et al. in "Gaussian Processes 16 | % for Object Categorization" (2010, IJCV) 17 | % 18 | % INPUT: 19 | % Ks_unlabeled -- (n_t x n_u1) matrix of self similarities between n_t training samples and n_u unlabeled samples 20 | % alpha -- (n_t x 1) column vector, weight vector of GP 21 | % regression model 22 | % 23 | % OUTPUT: 24 | % selected_sample -- scalar, index of chosen sample 25 | % 26 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 27 | % 28 | 29 | % function selected_sample = select_sample_gpMean(Ks_unlabeled, ~, ~, alpha, ~) 30 | % octave compatibility 31 | function selected_sample = select_sample_gpMean(Ks_unlabeled, dummy1, dummy2, alpha, dummy3) 32 | mu = Ks_unlabeled'*alpha; 33 | % [~, selected_sample] = min(abs(mu)); 34 | % octave compatibility 35 | [dummy4, selected_sample] = min(abs(mu)); 36 | end 37 | -------------------------------------------------------------------------------- /queryFunctions/select_sample_gpUncertainty.m: -------------------------------------------------------------------------------- 1 | % Query a new sample (with smallest "uncertainty", i.e., predictive mean / predictive variance) within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_gpUncertainty(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 11 | % 12 | % BRIEF: 13 | % Query a new sample from a pool of unlabeled ones such that the queried sample 14 | % results in the smallest "uncertainty", which is predictive mean divided by predictive variance. 15 | % This query strategy was introduced by Kapoor et al. in "Gaussian Processes 16 | % for Object Categorization" (2010, IJCV) 17 | % 18 | % INPUT: 19 | % Ks_unlabeled -- (n_t x n_u1) matrix of self similarities between n_t training samples and n_u unlabeled samples 20 | % Kss_unlabeled -- (n_u x 1) vector of self similarities of n_u unlabeled samples 21 | % L -- (n_t x n_t) cholesky matrix (upper triangle) computed from the 22 | % regularized kernel matrix of all training samples 23 | % alpha -- (n_t x 1) column vector, weight vector of GP 24 | % regression model 25 | % gpnoise -- (1 x 1) scalar, indicating the noise for 26 | % used for model regularization 27 | % 28 | % OUTPUT: 29 | % selected_sample -- scalar, index of chosen sample 30 | % 31 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 32 | % 33 | 34 | function selected_sample = select_sample_gpUncertainty(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 35 | mu = Ks_unlabeled'*alpha; 36 | 37 | sn2 = exp(2*gpnoise); 38 | V = L'\Ks_unlabeled; 39 | var = max(0,Kss_unlabeled-sum(V.*V./(sn2))'); 40 | 41 | heur = abs(mu)./sqrt(var+sn2); 42 | [~, selected_sample] = min(heur); 43 | end -------------------------------------------------------------------------------- /queryFunctions/select_sample_gpVariance.m: -------------------------------------------------------------------------------- 1 | % Query a new sample (with largest predictive variancess) within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_gpVariance(Ks_unlabeled, Kss_unlabeled, L, ~, gpnoise) 11 | % 12 | % BRIEF: 13 | % Query a new sample from a pool of unlabeled ones such that the queried sample 14 | % has the largest predictive variance among all possible samples. 15 | % This query strategy was introduced by Kapoor et al. in "Gaussian Processes 16 | % for Object Categorization" (2010, IJCV) 17 | % 18 | % INPUT: 19 | % Ks_unlabeled -- (n_t x n_u1) matrix of self similarities between n_t training samples and n_u unlabeled samples 20 | % Kss_unlabeled -- (n_u x 1) vector of self similarities of n_u unlabeled samples 21 | % L -- (n_t x n_t) cholesky matrix (upper triangle) computed from the 22 | % regularized kernel matrix of all training samples 23 | % gpnoise -- (1 x 1) scalar, indicating the noise for 24 | % used for model regularization 25 | % 26 | % OUTPUT: 27 | % selected_sample -- scalar, index of chosen sample 28 | % 29 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 30 | % 31 | 32 | function selected_sample = select_sample_gpVariance(Ks_unlabeled, Kss_unlabeled, L, ~, gpnoise) 33 | 34 | sn2 = exp(2*gpnoise); 35 | 36 | V = L'\Ks_unlabeled; 37 | var = max(0,Kss_unlabeled-sum(V.*V./(sn2))'); 38 | 39 | [~, selected_sample] = max(var); 40 | end -------------------------------------------------------------------------------- /queryFunctions/select_sample_gpWeight.m: -------------------------------------------------------------------------------- 1 | % Query a new sample (with maximum weight in the updated weight vector) within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_gpWeight(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 11 | % 12 | % BRIEF: 13 | % Query a new sample from a pool of unlabeled ones such that the queried sample 14 | % has the largest absolute weight after updating the alpha vector of the GP regression model. 15 | % 16 | % INPUT: 17 | % Ks_unlabeled -- (n_t x n_u1) matrix of self similarities between n_t training samples and n_u unlabeled samples 18 | % Kss_unlabeled -- (n_u x 1) vector of self similarities of n_u unlabeled samples 19 | % L -- (n_t x n_t) cholesky matrix (upper triangle) computed from the 20 | % regularized kernel matrix of all training samples 21 | % alpha -- (n_t x 1) column vector, weight vector of GP 22 | % regression model 23 | % gpnoise -- (1 x 1) scalar, indicating the noise for 24 | % used for model regularization 25 | % 26 | % OUTPUT: 27 | % selected_sample -- scalar, index of chosen sample 28 | % 29 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 30 | % 31 | 32 | function selected_sample = select_sample_gpWeight(Ks_unlabeled, Kss_unlabeled, L, alpha, gpnoise) 33 | mu = Ks_unlabeled'*alpha; 34 | 35 | sn2 = exp(2*gpnoise); 36 | V = L'\Ks_unlabeled; 37 | var = max(0,Kss_unlabeled-sum(V.*V./(sn2))'); 38 | 39 | % --- compute the first term --- 40 | % which downweights the impact of a new sample by the amount of 41 | % predicted uncertainty 42 | % note: actually, the first term is the invers of this number, but we 43 | % call the rdivide method lateron for easier computation 44 | firstTerm = sqrt(var.^2+sn2^2); 45 | 46 | % --- compute the second term --- 47 | % actually, for this query strategy the second term is constant to 1 48 | % therefore, we skip it since multiplying by one 49 | % doesn't change anything at all 50 | 51 | % --- compute the third term 52 | % this is the difference between predicted label and GT label 53 | % since we have not GT info, we take the sign of the most plausible class 54 | % -> in the binary setting, this is simply the sign of the mean value 55 | thirdTerm=mu-sign(mu); 56 | 57 | % we are interested in overall changes, so we take the abs 58 | thirdTerm=abs(thirdTerm); 59 | 60 | % --- combine (multiply) all two (implicitely three) terms --- 61 | % use the built-in function for element-wise division 62 | % where the vector is automatically extended in the second dimension 63 | % towards the right size 64 | % 65 | weight=bsxfun(@rdivide,thirdTerm,firstTerm); 66 | 67 | % take the one with the largest resulting weight 68 | % that is, we chose the point which has the highest influence for its most 69 | % plausible class 70 | [~, selected_sample]=max(weight); 71 | end -------------------------------------------------------------------------------- /queryFunctions/select_sample_random.m: -------------------------------------------------------------------------------- 1 | % Randomly query a new sample within an active learning experiment according to the work: 2 | % 3 | % Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler: 4 | % "Labeling examples that matter: Relevance-Based Active Learning with Gaussian Processes". 5 | % Proceedings of the German Conference on Pattern Recognition (GCPR), 2013. 6 | % 7 | % Please cite that paper if you are using this code! 8 | % 9 | % 10 | % function selected_sample = select_sample_random(~, Kss_unlabeled, ~, ~, ~) 11 | % 12 | % BRIEF: 13 | % Randomly query a new sample from a pool of unlabeled ones. 14 | % 15 | % INPUT: 16 | % Kss_unlabeled -- (n_u x 1) vector of self similarities of n_u unlabeled samples 17 | % 18 | % OUTPUT: 19 | % selected_sample -- scalar, index of chosen sample 20 | % 21 | % (C) copyright by Alexander Freytag and Paul Bodesheim and Erik Rodner and Joachim Denzler 22 | % 23 | 24 | function selected_sample = select_sample_random(~, Kss_unlabeled, ~, ~, ~) 25 | numberOfUnlabeled = length(Kss_unlabeled); 26 | selected_sample = randi(numberOfUnlabeled); 27 | end -------------------------------------------------------------------------------- /roc-analysis/auroc.m: -------------------------------------------------------------------------------- 1 | function aucScore = auroc(tp, fp) 2 | n = size(tp, 1); 3 | aucScore = sum((fp(2:n) - fp(1:n-1)).*(tp(2:n)+tp(1:n-1)))/2; 4 | end -------------------------------------------------------------------------------- /roc-analysis/roc.m: -------------------------------------------------------------------------------- 1 | function [tp, fp] = roc(gtLabels, scores) 2 | 3 | 4 | noInstances = size(scores,1); 5 | 6 | % sort by classifier output 7 | try % compatibility with old Matlab versions 8 | [scores,idx] = sort(scores, 'descend'); 9 | gtLabels = gtLabels(idx) > 0; 10 | catch 11 | [scores,idx] = sort(full(scores), 'descend'); 12 | gtLabels = gtLabels(idx) > 0; 13 | end 14 | 15 | % generate ROC 16 | 17 | noPositives = sum( gtLabels ); 18 | noNegatives = noInstances - noPositives; 19 | 20 | if ( noPositives <= 0 ) 21 | error('roc.m: no positive examples given!'); 22 | end 23 | 24 | if ( noNegatives <= 0 ) 25 | error('roc.m: no negative examples given!'); 26 | end 27 | 28 | %init 29 | fp = zeros( noInstances+2,1 ); 30 | tp = zeros( noInstances+2,1 ); 31 | FP = 0; 32 | TP = 0; 33 | n = 1; 34 | yprev = -realmax; 35 | 36 | % loop over scores 37 | for i=1:noInstances 38 | 39 | %did the score change? 40 | if ( scores(i) ~= yprev ) 41 | tp(n) = TP/noPositives; 42 | fp(n) = FP/noNegatives; 43 | yprev = scores(i); 44 | n = n + 1; 45 | end 46 | 47 | % update TP and FP rate for the current score 48 | if ( gtLabels(i) == 1 ) 49 | TP = TP + 1; 50 | else 51 | FP = FP + 1; 52 | end 53 | 54 | end 55 | 56 | %that's it :) 57 | tp(n) = 1; 58 | fp(n) = 1; 59 | fp = fp(1:n); 60 | tp = tp(1:n); 61 | 62 | end 63 | --------------------------------------------------------------------------------