├── .gitignore ├── LICENSE ├── README.md ├── __init__.py ├── decisionboundaryplot.py ├── demo.py ├── img ├── digits_logreg.png ├── digits_logreg_C10_good.png ├── digits_logreg_C1e-9_bad.png ├── dr_model_overview.png ├── wine_knn2_bad.png ├── wine_knn_svc.png ├── wine_rf50_good.png └── wine_svc1_underfit.png ├── requirements.txt ├── uci_loader.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | .project 3 | .pydevproject 4 | __pycache__/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Tamas Madl 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Plotting high-dimensional decision boundaries 2 | 3 | [![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/) 4 | [![MIT license](https://img.shields.io/badge/License-MIT-blue.svg)](https://lbesson.mit-license.org/) 5 | ![stability-experimental](https://img.shields.io/badge/stability-experimental-orange.svg) 6 | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/python/black) 7 | 8 | An experimental, scikit-learn compatible approach to **plot high-dimensional decision boundaries**. This facilitates **intuitive understanding**, and helps improve models by allowing the **visual inspection** of misclassified regions, model complexity, and the amount of overfitting/underfitting (augmenting more reliable but less intuitive tools such as training/test error curves and ROC curves); as well as of individual data points and their contribution to the decision surface. Finally, it shows which regions are likely to be misclassified, which are uncertain, and (in applications were active querying is possible) the proximity of which instances should be queried. 9 | 10 | The usual approach of classifying each vertex of a 2D grid to visualize decision boundaries (see e.g. the [Iris SVM example](http://scikit-learn.org/stable/auto_examples/svm/plot_iris.html)) in two dimensions breaks down when the dimensionality is higher. Although it would in principle be possible to construct a high-dimensional grid and project it down to 2D, this is intractable in practice, since the number of grid vertices to be classified grows exponentially with the number of dimensions. 11 | 12 | Instead, the present method samples from regions in which the given classifier is maximally uncertain (close to 0.5 prediction probability), which are near the decision surface by definition. Subsequently, it projects the decision surface samples (keypoints) into 2D for visualization, and generates and classifies some test points around the estimated decision boundary for validation (see Usage section below). 13 | 14 | **Example use case**: visualizing the **effect of regularization parameters**. Left: weakly regularized logistic regression (`C=1e-9`). Higher decision boundary complexity (noisy positions of cyan keypoints), more testing error (red markers). Right: regularized logistic regression (`C=10`). Lower complexity, no testing error. Data: sklearn toy digits (64 dimensions). 15 | ![Digits data - decision boundary comparison](img/digits_logreg.png) 16 | *(Green and blue markers: data points, round markers: training data, square markers: testing data; red highlights: misclassified data points; cyan pentagons: decision boundary keypoints [at which the classifier predicts 0.5 probability])* 17 | 18 | **Example use case**: visualizing **overfitting** and **underfitting**. Left: k-nearest neighbor with k=2 neighbors. Overfits: data points misclassified (highlighted in red) due to useless 'squiggles'. Right: over-regularized support vector machine (`C=1`, `gamma=0.07`). Misses the data point on the left (marked in red) due to underfitting. Data: UCI wine dataset (13 dimensions). 19 | ![Wine data - decision boundary comparison](img/wine_knn_svc.png) 20 | 21 | ## Usage 22 | 23 | 24 | The project requires [scikit-learn](http://scikit-learn.org/stable/install.html), [matplotlib](http://matplotlib.org/users/installing.html) and [NLopt](http://ab-initio.mit.edu/wiki/index.php/NLopt_Installation) to run. 25 | 26 | Usage example: 27 | 28 | ```python 29 | digits = load_digits(n_class = 2) # data 30 | model = KNeighborsClassifier(n_neighbors=10) # classifier able to produce probabilities 31 | 32 | db = DBPlot(model, PCA(n_components=2)) 33 | db.fit(digits.data, digits.target) 34 | db.plot().show() # use db.plot(generate_testpoints=False).show() to speed up plotting by skipping testpoint generation 35 | # decision_boundary_points_2d, testpoints_2d, y_testpoints, background = db.generate_plot() # for custom plotting and analysis 36 | ``` 37 | 38 | ## Functionality & Reliability 39 | 40 | 41 | Note: **this is an early stage research project, and work in progress** (it is by no means efficient or well tested)! 42 | 43 | The core idea is using **black-box optimization to find keypoints on the decision hypersurface** (those points in high-dimensional space for which prediction probability is very close to 0.5) which lie between the two classes in the 2D plot, and **projecting them to 2D** to estimate the location of the decision boundary. 44 | ![Four steps of plotting a high-dimensional decision boundary](img/dr_model_overview.png) 45 | 46 | The `DBPlot` object takes a **classifier** (the decision surface of which is to be estimated) and, optionally, a **dimensionality reduction method** (which defaults to `PCA`) as inputs. Most classifiers exposing `predict_proba` and `predict` methods should work, as long as it is possible for them to reach close to 0.5 prediction probability (note that this is not the case for e.g. 1-NN or a random forests with 3 trees! In these cases, the optimization will not converge). 47 | 48 | In terms of dimensionality reduction methods, the current version supports all [matrix decomposition](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) variants (including [PCA](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA), [Kernel PCA](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.KernelPCA.html#sklearn.decomposition.KernelPCA), [NMF](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html#sklearn.decomposition.NMF) etc.), as well as [Isomap](http://scikit-learn.org/stable/modules/generated/sklearn.manifold.Isomap.html#sklearn.manifold.Isomap) embeddings for non-linear dimensionality reduction preserving global topology, and any other method that has an implemented and exposed `transform(X)` function. This can include supervised dimensionality reduction, such as [LDA](http://scikit-learn.org/0.16/modules/generated/sklearn.lda.LDA.html) (with `solver='eigen'`), which projects to the most discriminative axes. 49 | 50 | Choosing a **dimensionality reduction method resulting in acceptabe class separation** is crucial for achieving interpretable results. 51 | 52 | When inspecting graphs and improving your classifier, you can trust 53 | - The actual data points (large green and blue points) 54 | - Misclassification feedback (red circles around the data points) 55 | - Generated test data points colored according to your classifier predictions (tiny, faint green and blue points) 56 | 57 | Everything else is a rough estimate intended for facilitating intuition, rather than precision; and needs to be traded off against runtime (set `generate_testpoints=False`, or decrease `n_decision_boundary_keypoints`, to decrease runtime). 58 | - The decision boundary keypoints (large cyan squares) are guaranteed to lie very close to the decision boundary (depending on the `acceptance_threshold` parameter setting). With very small tolerance, these are fairly reliable, but do NOT provide the full picture (a complete, reliable decision boundary could only be plotted with an infinite number of keypoints). To increase reliability, decrease `acceptance_threshold` or increase the number of decision boundary keypoints 59 | - The background shading reflects rough probability scores around the decision boundary, estimated from the generated test data points (its accuracy will depend on the number and coverage of these generated data points). As above, it is NOT a full picture (the generated data points do not provide full coverage, and only cover the space between the two classes, not beyond). To increase reliability, increase `n_generated_testpoints_per_keypoint` (or tweak the internal SVC approximating them in order to render the shading, e.g. by enabling `tune_background_model=True` for the plot) 60 | 61 | All plotted information is subject to the limitations inherent in forcing high-dimensional data into a low-dimensional plot. 62 | 63 | ## Installation 64 | Clone the repo, navigate to the repo directory and run the follwing two commands: 65 | ``` 66 | pip install -r requirements 67 | ``` 68 | ``` 69 | conda install -c conda-forge nlopt 70 | ``` 71 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/__init__.py -------------------------------------------------------------------------------- /decisionboundaryplot.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib.pyplot as mplt 3 | import os 4 | from sklearn.base import BaseEstimator 5 | from sklearn.model_selection import train_test_split 6 | from sklearn.decomposition import PCA 7 | from sklearn.neighbors import NearestNeighbors 8 | from sklearn.neighbors import KNeighborsClassifier 9 | import nlopt 10 | import random 11 | from scipy.spatial.distance import euclidean, squareform, pdist 12 | from utils import minimum_spanning_tree, polar_to_cartesian 13 | from sklearn.model_selection import GridSearchCV 14 | from sklearn.svm import SVC 15 | from sklearn.metrics import accuracy_score, f1_score 16 | 17 | 18 | class DBPlot(BaseEstimator): 19 | """ 20 | Heuristic approach to estimate and visualize high-dimensional decision 21 | boundaries for trained binary classifiers by using black-box optimization 22 | to find regions in which the classifier is maximally uncertain (0.5 prediction 23 | probability). The total number of keypoints representing the decision boundary 24 | will depend on n_connecting_keypoints and n_interpolated_keypoints. Reduce 25 | either or both to reduce runtime. 26 | 27 | Parameters 28 | ---------- 29 | estimator : BaseEstimator instance, optional (default=`KNeighborsClassifier(n_neighbors=10)`). 30 | Classifier for which the decision boundary should be plotted. Can be trained 31 | or untrained (in which case the fit method will train it). Must have 32 | probability estimates enabled (i.e. `estimator.predict_proba` must work). 33 | Make sure it is possible for probability estimates to get close to 0.5 34 | (more specifically, as close as specified by acceptance_threshold) - this usally 35 | requires setting an even number of neighbors, estimators etc. 36 | 37 | dimensionality_reduction : BaseEstimator instance, optional (default=`PCA(n_components=2)`). 38 | Dimensionality reduction method to help plot the decision boundary in 2D. Can be trained 39 | or untrained (in which case the fit method will train it). Must have n_components=2. 40 | Must be able to project new points into the 2D space after fitting 41 | (i.e. `dimensionality_reduction.transform` must work). 42 | 43 | acceptance_threshold : float, optional (default=0.03) 44 | Maximum allowed deviation from decision boundary (defined as the region 45 | with 0.5 prediction probability) when accepting decision boundary keypoints 46 | 47 | n_decision_boundary_keypoints : int, optional (default=60) 48 | Total number of decision boundary keypoints added, including both connecting 49 | and interpolated keypoints. 50 | 51 | n_connecting_keypoints : int, optional (default=None) 52 | Number of decision boundary keypoints estimated along lines connecting 53 | instances from two different classes (each such line must cross the 54 | decision boundary at least once). If None (default), it is set to 1/3 55 | of n_decision_boundary_keypoints 56 | 57 | n_interpolated_keypoints : int, optional (default=None) 58 | Number of decision boundary keypoints interpolated between connecting 59 | keypoints to increase keypoint density. If None (default), it is set to 60 | 2/3 of n_decision_boundary_keypoints 61 | 62 | n_generated_testpoints_per_keypoint : int, optional (default=15) 63 | Number of demo points generated around decision boundary keypoints, and 64 | labeled according to the specified classifier, in order to enrich and 65 | validate the decision boundary plot 66 | 67 | linear_iteration_budget : int, optional (default=100) 68 | Maximum number of iterations the optimizer is allowed to run for each 69 | keypoint estimation while looking along linear trajectories 70 | 71 | hypersphere_iteration_budget : int, optional (default=300) 72 | Maximum number of iterations the optimizer is allowed to run for each 73 | keypoint estimation while looking along hypersphere surfaces 74 | 75 | verbose: bool, optional (default=True) 76 | Verbose output 77 | """ 78 | 79 | def __init__( 80 | self, 81 | estimator=KNeighborsClassifier(n_neighbors=10), 82 | dimensionality_reduction=PCA(n_components=2), 83 | acceptance_threshold=0.03, 84 | n_decision_boundary_keypoints=60, 85 | n_connecting_keypoints=None, 86 | n_interpolated_keypoints=None, 87 | n_generated_testpoints_per_keypoint=15, 88 | linear_iteration_budget=100, 89 | hypersphere_iteration_budget=300, 90 | verbose=True, 91 | ): 92 | if acceptance_threshold == 0: 93 | raise Warning( 94 | "A nonzero acceptance threshold is strongly recommended so the optimizer can finish in finite time" 95 | ) 96 | if linear_iteration_budget < 2 or hypersphere_iteration_budget < 2: 97 | raise Exception("Invalid iteration budget") 98 | 99 | self.classifier = estimator 100 | self.dimensionality_reduction = dimensionality_reduction 101 | self.acceptance_threshold = acceptance_threshold 102 | 103 | if ( 104 | n_decision_boundary_keypoints 105 | and n_connecting_keypoints 106 | and n_interpolated_keypoints 107 | and n_connecting_keypoints + n_interpolated_keypoints 108 | != n_decision_boundary_keypoints 109 | ): 110 | raise Exception( 111 | "n_connecting_keypoints and n_interpolated_keypoints must sum to n_decision_boundary_keypoints (set them to None to use calculated suggestions)" 112 | ) 113 | 114 | self.n_connecting_keypoints = ( 115 | n_connecting_keypoints 116 | if n_connecting_keypoints != None 117 | else n_decision_boundary_keypoints / 3 118 | ) 119 | self.n_interpolated_keypoints = ( 120 | n_interpolated_keypoints 121 | if n_interpolated_keypoints != None 122 | else n_decision_boundary_keypoints * 2 / 3 123 | ) 124 | 125 | self.linear_iteration_budget = linear_iteration_budget 126 | self.n_generated_testpoints_per_keypoint = n_generated_testpoints_per_keypoint 127 | self.hypersphere_iteration_budget = hypersphere_iteration_budget 128 | self.verbose = verbose 129 | 130 | self.decision_boundary_points = [] 131 | self.decision_boundary_points_2d = [] 132 | self.X_testpoints = [] 133 | self.y_testpoints = [] 134 | self.background = [] 135 | self.steps = 3 136 | 137 | self.hypersphere_max_retry_budget = 20 138 | self.penalties_enabled = True 139 | self.random_gap_selection = False 140 | 141 | def setclassifier(self, estimator=KNeighborsClassifier(n_neighbors=10)): 142 | """Assign classifier for which decision boundary should be plotted. 143 | 144 | Parameters 145 | ---------- 146 | estimator : BaseEstimator instance, optional (default=KNeighborsClassifier(n_neighbors=10)). 147 | Classifier for which the decision boundary should be plotted. Must have 148 | probability estimates enabled (i.e. estimator.predict_proba must work). 149 | Make sure it is possible for probability estimates to get close to 0.5 150 | (more specifically, as close as specified by acceptance_threshold). 151 | """ 152 | self.classifier = estimator 153 | 154 | def fit(self, X, y, training_indices=None): 155 | """Specify data to be plotted, and fit classifier only if required (the 156 | specified clasifier is only trained if it has not been trained yet). 157 | 158 | All the input data is provided in the matrix X, and corresponding 159 | binary labels (values taking 0 or 1) in the vector y 160 | 161 | Parameters 162 | ---------- 163 | X : array-like, shape = [n_samples, n_features] 164 | A {n_samples by n_samples} size matrix containing data 165 | 166 | y : array-like, shape = [n_samples] 167 | Labels 168 | 169 | training_indices : array-like or float, optional (default=None) 170 | Indices on which the classifier has been trained / should be trained. 171 | If float, it is converted to a random sample with the specified proportion 172 | of the full dataset. 173 | 174 | Returns 175 | ------- 176 | self : returns an instance of self. 177 | """ 178 | if set(np.array(y, dtype=int).tolist()) != set([0, 1]): 179 | raise Exception( 180 | "Currently only implemented for binary classification. Make sure you pass in two classes (0 and 1)" 181 | ) 182 | 183 | if training_indices == None: 184 | train_idx = range(len(y)) 185 | elif type(training_indices) == float: 186 | train_idx, test_idx = train_test_split(range(len(y)), test_size=0.2) 187 | else: 188 | train_idx = training_indices 189 | 190 | self.X = X 191 | self.y = y 192 | self.train_idx = train_idx 193 | # self.test_idx = np.setdiff1d(np.arange(len(y)), self.train_idx, assume_unique=False) 194 | self.test_idx = list(set(range(len(y))).difference(set(self.train_idx))) 195 | 196 | # fit classifier if necessary 197 | try: 198 | self.classifier.predict([X[0]]) 199 | except: 200 | self.classifier.fit(X[train_idx, :], y[train_idx]) 201 | 202 | self.y_pred = self.classifier.predict(self.X) 203 | 204 | # fit DR method if necessary 205 | try: 206 | self.dimensionality_reduction.transform([X[0]]) 207 | except: 208 | self.dimensionality_reduction.fit(X, y) 209 | 210 | try: 211 | self.dimensionality_reduction.transform([X[0]]) 212 | except: 213 | raise Exception( 214 | "Please make sure your dimensionality reduction method has an exposed transform() method! If in doubt, use PCA or Isomap" 215 | ) 216 | 217 | # transform data 218 | self.X2d = self.dimensionality_reduction.transform(self.X) 219 | self.mean_2d_dist = np.mean(pdist(self.X2d)) 220 | self.X2d_xmin, self.X2d_xmax = np.min(self.X2d[:, 0]), np.max(self.X2d[:, 0]) 221 | self.X2d_ymin, self.X2d_ymax = np.min(self.X2d[:, 1]), np.max(self.X2d[:, 1]) 222 | 223 | self.majorityclass = 0 if list(y).count(0) > list(y).count(1) else 1 224 | self.minorityclass = 1 - self.majorityclass 225 | minority_idx, majority_idx = ( 226 | np.where(y == self.minorityclass)[0], 227 | np.where(y == self.majorityclass)[0], 228 | ) 229 | self.Xminor, self.Xmajor = X[minority_idx], X[majority_idx] 230 | self.Xminor2d, self.Xmajor2d = self.X2d[minority_idx], self.X2d[majority_idx] 231 | 232 | # set up efficient nearest neighbor models for later use 233 | self.nn_model_2d_majorityclass = NearestNeighbors(n_neighbors=2) 234 | self.nn_model_2d_majorityclass.fit(self.X2d[majority_idx, :]) 235 | 236 | self.nn_model_2d_minorityclass = NearestNeighbors(n_neighbors=2) 237 | self.nn_model_2d_minorityclass.fit(self.X2d[minority_idx, :]) 238 | 239 | # step 1. look for decision boundary points between corners of majority & 240 | # minority class distribution 241 | minority_corner_idx, majority_corner_idx = [], [] 242 | for extremum1 in [np.min, np.max]: 243 | for extremum2 in [np.min, np.max]: 244 | _, idx = self.nn_model_2d_minorityclass.kneighbors( 245 | [[extremum1(self.Xminor2d[:, 0]), extremum2(self.Xminor2d[:, 1])]] 246 | ) 247 | minority_corner_idx.append(idx[0][0]) 248 | _, idx = self.nn_model_2d_majorityclass.kneighbors( 249 | [[extremum1(self.Xmajor2d[:, 0]), extremum2(self.Xmajor2d[:, 1])]] 250 | ) 251 | majority_corner_idx.append(idx[0][0]) 252 | 253 | # optimize to find new db keypoints between corners 254 | self._linear_decision_boundary_optimization( 255 | minority_corner_idx, majority_corner_idx, all_combinations=True, step=1 256 | ) 257 | 258 | # step 2. look for decision boundary points on lines connecting randomly 259 | # sampled points of majority & minority class 260 | n_samples = int(self.n_connecting_keypoints) 261 | from_idx = list(random.sample(list(np.arange(len(self.Xminor))), n_samples)) 262 | to_idx = list(random.sample(list(np.arange(len(self.Xmajor))), n_samples)) 263 | 264 | # optimize to find new db keypoints between minority and majority class 265 | self._linear_decision_boundary_optimization( 266 | from_idx, to_idx, all_combinations=False, step=2 267 | ) 268 | 269 | if len(self.decision_boundary_points_2d) < 2: 270 | print( 271 | "Failed to find initial decision boundary. Retrying... If this keeps happening, increasing the acceptance threshold might help. Also, make sure the classifier is able to find a point with 0.5 prediction probability (usually requires an even number of estimators/neighbors/etc)." 272 | ) 273 | return self.fit(X, y, training_indices) 274 | 275 | # step 3. look for decision boundary points between already known db 276 | # points that are too distant (search on connecting line first, then on 277 | # surrounding hypersphere surfaces) 278 | edges, gap_distances, gap_probability_scores = ( 279 | self._get_sorted_db_keypoint_distances() 280 | ) # find gaps 281 | self.nn_model_decision_boundary_points = NearestNeighbors(n_neighbors=2) 282 | self.nn_model_decision_boundary_points.fit(self.decision_boundary_points) 283 | 284 | i = 0 285 | retries = 0 286 | while i < self.n_interpolated_keypoints: 287 | if self.verbose: 288 | print( 289 | "Step 3/{}:{}/".format(self.steps, i, self.n_interpolated_keypoints) 290 | ) 291 | if self.random_gap_selection: 292 | # randomly sample from sorted DB keypoint gaps? 293 | gap_idx = np.random.choice( 294 | len(gap_probability_scores), 1, p=gap_probability_scores 295 | )[0] 296 | else: 297 | # get largest gap 298 | gap_idx = 0 299 | from_point = self.decision_boundary_points[edges[gap_idx][0]] 300 | to_point = self.decision_boundary_points[edges[gap_idx][1]] 301 | 302 | # optimize to find new db keypoint along line connecting two db keypoints 303 | # with large gap 304 | db_point = self._find_decision_boundary_along_line( 305 | from_point, to_point, penalize_tangent_distance=self.penalties_enabled 306 | ) 307 | 308 | if self.decision_boundary_distance(db_point) > self.acceptance_threshold: 309 | if self.verbose: 310 | print( 311 | "No good solution along straight line - trying to find decision boundary on hypersphere surface around known decision boundary point" 312 | ) 313 | 314 | # hypersphere radius half the distance between from and to db keypoints 315 | R = euclidean(from_point, to_point) / 2.0 316 | # search around either source or target keypoint, with 0.5 probability, 317 | # hoping to find decision boundary in between 318 | if random.random() > 0.5: 319 | from_point = to_point 320 | 321 | # optimize to find new db keypoint on hypersphere surphase around known keypoint 322 | db_point = self._find_decision_boundary_on_hypersphere(from_point, R) 323 | if ( 324 | self.decision_boundary_distance(db_point) 325 | <= self.acceptance_threshold 326 | ): 327 | db_point2d = self.dimensionality_reduction.transform([db_point])[0] 328 | self.decision_boundary_points.append(db_point) 329 | self.decision_boundary_points_2d.append(db_point2d) 330 | i += 1 331 | retries = 0 332 | else: 333 | retries += 1 334 | if retries > self.hypersphere_max_retry_budget: 335 | i += 1 336 | dist = self.decision_boundary_distance(db_point) 337 | msg = "Found point is too distant from decision boundary ({}), but retry budget exceeded ({})" 338 | print(msg.format(dist, self.hypersphere_max_retry_budget)) 339 | elif self.verbose: 340 | dist = self.decision_boundary_distance(db_point) 341 | print( 342 | "Found point is too distant from decision boundary ({}) retrying...".format( 343 | dist 344 | ) 345 | ) 346 | 347 | else: 348 | db_point2d = self.dimensionality_reduction.transform([db_point])[0] 349 | self.decision_boundary_points.append(db_point) 350 | self.decision_boundary_points_2d.append(db_point2d) 351 | i += 1 352 | retries = 0 353 | 354 | edges, gap_distances, gap_probability_scores = ( 355 | self._get_sorted_db_keypoint_distances() 356 | ) # reload gaps 357 | 358 | self.decision_boundary_points = np.array(self.decision_boundary_points) 359 | self.decision_boundary_points_2d = np.array(self.decision_boundary_points_2d) 360 | 361 | if self.verbose: 362 | print( 363 | "Done fitting! Found {} decision boundary keypoints.".format( 364 | len(self.decision_boundary_points) 365 | ) 366 | ) 367 | 368 | return self 369 | 370 | def plot( 371 | self, 372 | plt=None, 373 | generate_testpoints=True, 374 | generate_background=True, 375 | tune_background_model=False, 376 | background_resolution=100, 377 | scatter_size_scale=1.0, 378 | legend=True, 379 | ): 380 | 381 | """Plots the dataset and the identified decision boundary in 2D. 382 | (If you wish to create custom plots, get the data using generate_plot() and plot it manually) 383 | 384 | Parameters 385 | ---------- 386 | plt : matplotlib.pyplot or axis object (default=matplotlib.pyplot) 387 | Object to be plotted on 388 | 389 | generate_testpoints : boolean, optional (default=True) 390 | Whether to generate demo points around the estimated decision boundary 391 | as a sanity check 392 | 393 | generate_background : boolean, optional (default=True) 394 | Whether to generate faint background plot (using prediction probabilities 395 | of a fitted suppor vector machine, trained on generated demo points) 396 | to aid visualization 397 | 398 | tune_background_model : boolean, optional (default=False) 399 | Whether to tune the parameters of the support vector machine generating 400 | the background 401 | 402 | background_resolution : int, optional (default=100) 403 | Desired resolution (height and width) of background to be generated 404 | 405 | scatter_size_scale : float, optional (default=1.0) 406 | Scaling factor for scatter plot marker size 407 | 408 | legend : boolean, optional (default=False) 409 | Whether to display a legend 410 | 411 | Returns 412 | ------- 413 | plt : The matplotlib.pyplot or axis object which has been passed in, after 414 | plotting the data and decision boundary on it. (plt.show() is NOT called 415 | and will be required) 416 | """ 417 | if plt == None: 418 | plt = mplt 419 | 420 | if len(self.X_testpoints) == 0: 421 | self.generate_plot( 422 | generate_testpoints=generate_testpoints, 423 | generate_background=generate_background, 424 | tune_background_model=tune_background_model, 425 | background_resolution=background_resolution, 426 | ) 427 | 428 | if generate_background and generate_testpoints: 429 | try: 430 | plt.imshow( 431 | np.flipud(self.background), 432 | extent=[self.X2d_xmin, self.X2d_xmax, self.X2d_ymin, self.X2d_ymax], 433 | cmap="GnBu", 434 | alpha=0.33, 435 | ) 436 | except (Exception, ex): 437 | print("Failed to render image background") 438 | 439 | # decision boundary 440 | plt.scatter( 441 | self.decision_boundary_points_2d[:, 0], 442 | self.decision_boundary_points_2d[:, 1], 443 | 600 * scatter_size_scale, 444 | c="c", 445 | marker="p", 446 | ) 447 | # generated demo points 448 | if generate_testpoints: 449 | plt.scatter( 450 | self.X_testpoints_2d[:, 0], 451 | self.X_testpoints_2d[:, 1], 452 | 20 * scatter_size_scale, 453 | c=["g" if i else "b" for i in self.y_testpoints], 454 | alpha=0.6, 455 | ) 456 | 457 | # training data 458 | plt.scatter( 459 | self.X2d[self.train_idx, 0], 460 | self.X2d[self.train_idx, 1], 461 | 40 * scatter_size_scale, 462 | facecolor=["g" if i else "b" for i in self.y[self.train_idx]], 463 | edgecolor=[ 464 | "g" 465 | if self.y_pred[self.train_idx[i]] == self.y[self.train_idx[i]] == 1 466 | else ( 467 | "b" 468 | if self.y_pred[self.train_idx[i]] == self.y[self.train_idx[i]] == 0 469 | else "r" 470 | ) 471 | for i in range(len(self.train_idx)) 472 | ], 473 | linewidths=5 * scatter_size_scale, 474 | ) 475 | # testing data 476 | plt.scatter( 477 | self.X2d[self.test_idx, 0], 478 | self.X2d[self.test_idx, 1], 479 | 150 * scatter_size_scale, 480 | facecolor=["g" if i else "b" for i in self.y[self.test_idx]], 481 | edgecolor=[ 482 | "g" 483 | if self.y_pred[self.test_idx[i]] == self.y[self.test_idx[i]] == 1 484 | else ( 485 | "b" 486 | if self.y_pred[self.test_idx[i]] == self.y[self.test_idx[i]] == 0 487 | else "r" 488 | ) 489 | for i in range(len(self.test_idx)) 490 | ], 491 | linewidths=5 * scatter_size_scale, 492 | marker="s", 493 | ) 494 | 495 | # label data points with their indices 496 | for i in range(len(self.X2d)): 497 | plt.text( 498 | self.X2d[i, 0] + (self.X2d_xmax - self.X2d_xmin) * 0.5e-2, 499 | self.X2d[i, 1] + (self.X2d_ymax - self.X2d_ymin) * 0.5e-2, 500 | str(i), 501 | size=8, 502 | ) 503 | 504 | if legend: 505 | plt.legend( 506 | [ 507 | "Estimated decision boundary keypoints", 508 | "Generated demo data around decision boundary", 509 | "Actual data (training set)", 510 | "Actual data (demo set)", 511 | ], 512 | loc="lower right", 513 | prop={"size": 9}, 514 | ) 515 | 516 | # decision boundary keypoints, in case not visible in background 517 | plt.scatter( 518 | self.decision_boundary_points_2d[:, 0], 519 | self.decision_boundary_points_2d[:, 1], 520 | 600 * scatter_size_scale, 521 | c="c", 522 | marker="p", 523 | alpha=0.1, 524 | ) 525 | plt.scatter( 526 | self.decision_boundary_points_2d[:, 0], 527 | self.decision_boundary_points_2d[:, 1], 528 | 30 * scatter_size_scale, 529 | c="c", 530 | marker="p", 531 | edgecolor="c", 532 | alpha=0.8, 533 | ) 534 | 535 | # minimum spanning tree through decision boundary keypoints 536 | D = pdist(self.decision_boundary_points_2d) 537 | edges = minimum_spanning_tree(squareform(D)) 538 | for e in edges: 539 | plt.plot( 540 | [ 541 | self.decision_boundary_points_2d[e[0], 0], 542 | self.decision_boundary_points_2d[e[1], 0], 543 | ], 544 | [ 545 | self.decision_boundary_points_2d[e[0], 1], 546 | self.decision_boundary_points_2d[e[1], 1], 547 | ], 548 | "--c", 549 | linewidth=4 * scatter_size_scale, 550 | ) 551 | plt.plot( 552 | [ 553 | self.decision_boundary_points_2d[e[0], 0], 554 | self.decision_boundary_points_2d[e[1], 0], 555 | ], 556 | [ 557 | self.decision_boundary_points_2d[e[0], 1], 558 | self.decision_boundary_points_2d[e[1], 1], 559 | ], 560 | "--k", 561 | linewidth=1, 562 | ) 563 | 564 | if len(self.test_idx) == 0: 565 | print("No demo performance calculated, as no testing data was specified") 566 | else: 567 | freq = np.array( 568 | np.unique(self.y[self.test_idx], return_counts=True) 569 | ).T.astype(float) 570 | imbalance = np.round( 571 | np.max((freq[0, 1], freq[1, 1])) / len(self.test_idx), 3 572 | ) 573 | acc_score = np.round( 574 | accuracy_score(self.y[self.test_idx], self.y_pred[self.test_idx]), 3 575 | ) 576 | f1 = np.round( 577 | f1_score(self.y[self.test_idx], self.y_pred[self.test_idx]), 3 578 | ) 579 | plt.title( 580 | "Test accuracy: " 581 | + str(acc_score) 582 | + ", F1 score: " 583 | + str(f1) 584 | + ". Imbalance (max chance accuracy): " 585 | + str(imbalance) 586 | ) 587 | 588 | if self.verbose: 589 | print( 590 | "Plot successfully generated! Don't forget to call the show() method to display it" 591 | ) 592 | 593 | return plt 594 | 595 | def generate_plot( 596 | self, 597 | generate_testpoints=True, 598 | generate_background=True, 599 | tune_background_model=False, 600 | background_resolution=100, 601 | ): 602 | """Generates and returns arrays for visualizing the dataset and the 603 | identified decision boundary in 2D. 604 | 605 | Parameters 606 | ---------- 607 | generate_testpoints : boolean, optional (default=True) 608 | Whether to generate demo points around the estimated decision boundary 609 | as a sanity check 610 | 611 | generate_background : boolean, optional (default=True) 612 | Whether to generate faint background plot (using prediction probabilities 613 | of a fitted suppor vector machine, trained on generated demo points) 614 | to aid visualization 615 | 616 | tune_background_model : boolean, optional (default=False) 617 | Whether to tune the parameters of the support vector machine generating 618 | the background 619 | 620 | background_resolution : int, optional (default=100) 621 | Desired resolution (height and width) of background to be generated 622 | 623 | Returns 624 | ------- 625 | decision_boundary_points_2d : array 626 | Array containing points in the dimensionality-reduced 2D space which 627 | are very close to the true decision boundary 628 | 629 | X_testpoints_2d : array 630 | Array containing generated demo points in the dimensionality-reduced 631 | 2D space which surround the decision boundary and can be used for 632 | visual feedback to estimate which area would be assigned which class 633 | 634 | y_testpoints : array 635 | Classifier predictions for each of the generated demo points 636 | 637 | background: array 638 | Generated background image showing prediction probabilities of the 639 | classifier in each region (only returned if generate_background is set 640 | to True!) 641 | 642 | """ 643 | if len(self.decision_boundary_points) == 0: 644 | raise Exception("Please call the fit method first!") 645 | 646 | if not generate_testpoints and generate_background: 647 | print("Warning: cannot generate a background without testpoints") 648 | 649 | if len(self.X_testpoints) == 0 and generate_testpoints: 650 | if self.verbose: 651 | print("Generating demo points around decision boundary...") 652 | self._generate_testpoints() 653 | 654 | if generate_background and generate_testpoints: 655 | if tune_background_model: 656 | params = { 657 | "C": np.power(10, np.linspace(0, 2, 2)), 658 | "gamma": np.power(10, np.linspace(-2, 0, 2)), 659 | } 660 | grid = GridSearchCV( 661 | SVC(), params, n_jobs=-1 if os.name != "nt" else 1 662 | ) 663 | grid.fit( 664 | np.vstack((self.X2d[self.train_idx], self.X_testpoints_2d)), 665 | np.hstack((self.y[self.train_idx], self.y_testpoints)), 666 | ) 667 | bestparams = grid.best_params_ 668 | else: 669 | bestparams = {"C": 1, "gamma": 1} 670 | 671 | self.background_model = SVC( 672 | probability=True, C=bestparams["C"], gamma=bestparams["gamma"] 673 | ).fit( 674 | np.vstack((self.X2d[self.train_idx], self.X_testpoints_2d)), 675 | np.hstack((self.y[self.train_idx], self.y_testpoints)), 676 | ) 677 | xx, yy = np.meshgrid( 678 | np.linspace(self.X2d_xmin, self.X2d_xmax, background_resolution), 679 | np.linspace(self.X2d_ymin, self.X2d_ymax, background_resolution), 680 | ) 681 | Z = self.background_model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[ 682 | :, 0 683 | ] 684 | Z = Z.reshape((background_resolution, background_resolution)) 685 | self.background = Z 686 | 687 | if generate_background and generate_testpoints: 688 | return ( 689 | self.decision_boundary_points_2d, 690 | self.X_testpoints_2d, 691 | self.y_testpoints, 692 | Z, 693 | ) 694 | elif generate_testpoints: 695 | return ( 696 | self.decision_boundary_points_2d, 697 | self.X_testpoints_2d, 698 | self.y_testpoints, 699 | ) 700 | else: 701 | return self.decision_boundary_points_2d 702 | 703 | def _generate_testpoints(self, tries=100): 704 | """Generate random demo points around decision boundary keypoints 705 | """ 706 | nn_model = NearestNeighbors(n_neighbors=3) 707 | nn_model.fit(self.decision_boundary_points) 708 | 709 | nn_model_2d = NearestNeighbors(n_neighbors=2) 710 | nn_model_2d.fit(self.decision_boundary_points_2d) 711 | # max_radius = 2*np.max([nn_model_2d.kneighbors([self.decision_boundary_points_2d[i]])[0][0][1] for i in range(len(self.decision_boundary_points_2d))]) 712 | 713 | self.X_testpoints = np.zeros((0, self.X.shape[1])) 714 | self.y_testpoints = [] 715 | for i in range(len(self.decision_boundary_points)): 716 | if self.verbose: 717 | msg = "Generating testpoint for plotting {}/{}" 718 | print(msg.format(i, len(self.decision_boundary_points))) 719 | testpoints = np.zeros((0, self.X.shape[1])) 720 | # generate Np points in Gaussian around decision_boundary_points[i] with 721 | # radius depending on the distance to the next point 722 | d, idx = nn_model.kneighbors([self.decision_boundary_points[i]]) 723 | radius = d[0][1] if d[0][1] != 0 else d[0][2] 724 | if radius == 0: 725 | radius = np.mean(pdist(self.decision_boundary_points_2d)) 726 | max_radius = radius * 2 727 | radius /= 5.0 728 | 729 | # add demo points, keeping some balance 730 | max_imbalance = 5.0 731 | y_testpoints = [] 732 | for j in range(self.n_generated_testpoints_per_keypoint - 2): 733 | c_radius = radius 734 | freq = np.array(np.unique(y_testpoints, return_counts=True)).T.astype( 735 | float 736 | ) 737 | imbalanced = freq.shape[0] != 0 738 | if freq.shape[0] == 2 and ( 739 | freq[0, 1] / freq[1, 1] < 1.0 / max_imbalance 740 | or freq[0, 1] / freq[1, 1] > max_imbalance 741 | ): 742 | imbalanced = True 743 | 744 | for try_i in range(tries): 745 | testpoint = np.random.normal( 746 | self.decision_boundary_points[i], radius, (1, self.X.shape[1]) 747 | ) 748 | try: 749 | testpoint2d = self.dimensionality_reduction.transform( 750 | testpoint 751 | )[0] 752 | except: # DR can fail e.g. if NMF gets negative values 753 | testpoint = [] 754 | continue 755 | # demo point needs to be close to current key point 756 | if ( 757 | euclidean(testpoint2d, self.decision_boundary_points_2d[i]) 758 | <= max_radius 759 | ): 760 | if not imbalanced: # needs to be not imbalanced 761 | break 762 | y_pred = self.classifier.predict(testpoint)[0] 763 | # imbalanced but this would actually improve things 764 | if freq.shape[0] == 2 and freq[y_pred, 1] < freq[1 - y_pred, 1]: 765 | break 766 | c_radius /= 2.0 767 | if len(testpoint) != 0: 768 | testpoints = np.vstack((testpoints, testpoint)) 769 | y_testpoints.append(self.classifier.predict(testpoint)[0]) 770 | 771 | self.X_testpoints = np.vstack((self.X_testpoints, testpoints)) 772 | self.y_testpoints = np.hstack((self.y_testpoints, y_testpoints)) 773 | self.X_testpoints_2d = self.dimensionality_reduction.transform( 774 | self.X_testpoints 775 | ) 776 | 777 | idx_within_bounds = np.where( 778 | (self.X_testpoints_2d[:, 0] >= self.X2d_xmin) 779 | & (self.X_testpoints_2d[:, 0] <= self.X2d_xmax) 780 | & (self.X_testpoints_2d[:, 1] >= self.X2d_ymin) 781 | & (self.X_testpoints_2d[:, 1] <= self.X2d_ymax) 782 | )[0] 783 | self.X_testpoints = self.X_testpoints[idx_within_bounds] 784 | self.y_testpoints = self.y_testpoints[idx_within_bounds] 785 | self.X_testpoints_2d = self.X_testpoints_2d[idx_within_bounds] 786 | 787 | def decision_boundary_distance(self, x, grad=0): 788 | """Returns the distance of the given point from the decision boundary, 789 | i.e. the distance from the region with maximal uncertainty (0.5 790 | prediction probability)""" 791 | return np.abs(0.5 - self.classifier.predict_proba([x])[0][1]) 792 | 793 | def get_decision_boundary_keypoints(self): 794 | """Returns the arrays of located decision boundary keypoints (both in the 795 | original feature space, and in the dimensionality-reduced 2D space) 796 | 797 | Returns 798 | ------- 799 | decision_boundary_points : array 800 | Array containing points in the original feature space which are very 801 | close to the true decision boundary (closer than acceptance_threshold) 802 | 803 | decision_boundary_points_2d : array 804 | Array containing points in the dimensionality-reduced 2D space which 805 | are very close to the true decision boundary 806 | """ 807 | if len(self.decision_boundary_points) == 0: 808 | raise Exception("Please call the fit method first!") 809 | return self.decision_boundary_points, self.decision_boundary_points_2d 810 | 811 | def _get_sorted_db_keypoint_distances(self, N=None): 812 | """Use a minimum spanning tree heuristic to find the N largest gaps in the 813 | line constituted by the current decision boundary keypoints. 814 | """ 815 | if N == None: 816 | N = self.n_interpolated_keypoints 817 | edges = minimum_spanning_tree( 818 | squareform(pdist(self.decision_boundary_points_2d)) 819 | ) 820 | edged = np.array( 821 | [ 822 | euclidean( 823 | self.decision_boundary_points_2d[u], 824 | self.decision_boundary_points_2d[v], 825 | ) 826 | for u, v in edges 827 | ] 828 | ) 829 | gap_edge_idx = np.argsort(edged)[::-1][: int(N)] 830 | edges = edges[gap_edge_idx] 831 | gap_distances = np.square(edged[gap_edge_idx]) 832 | gap_probability_scores = gap_distances / np.sum(gap_distances) 833 | return edges, gap_distances, gap_probability_scores 834 | 835 | def _linear_decision_boundary_optimization( 836 | self, 837 | from_idx, 838 | to_idx, 839 | all_combinations=True, 840 | retry_neighbor_if_failed=True, 841 | step=None, 842 | suppress_output=False, 843 | ): 844 | """Use global optimization to locate the decision boundary along lines 845 | defined by instances from_idx and to_idx in the dataset (from_idx and to_idx 846 | have to contain indices from distinct classes to guarantee the existence of 847 | a decision boundary between them!) 848 | """ 849 | step_str = ( 850 | ("Step " + str(step) + "/" + str(self.steps) + ":") if step != None else "" 851 | ) 852 | 853 | retries = 4 if retry_neighbor_if_failed else 1 854 | for i in range(len(from_idx)): 855 | n = len(to_idx) if all_combinations else 1 856 | for j in range(n): 857 | from_i = from_idx[i] 858 | to_i = to_idx[j] if all_combinations else to_idx[i] 859 | for k in range(retries): 860 | if k == 0: 861 | from_point = self.Xminor[from_i] 862 | to_point = self.Xmajor[to_i] 863 | else: 864 | # first attempt failed, try nearest neighbors of source and destination 865 | # point instead 866 | _, idx = self.nn_model_2d_minorityclass.kneighbors( 867 | [self.Xminor2d[from_i]] 868 | ) 869 | from_point = self.Xminor[idx[0][k // 2]] 870 | _, idx = self.nn_model_2d_minorityclass.kneighbors( 871 | [self.Xmajor2d[to_i]] 872 | ) 873 | to_point = self.Xmajor[idx[0][k % 2]] 874 | 875 | if euclidean(from_point, to_point) == 0: 876 | break # no decision boundary between equivalent points 877 | 878 | db_point = self._find_decision_boundary_along_line( 879 | from_point, 880 | to_point, 881 | penalize_tangent_distance=self.penalties_enabled, 882 | penalize_extremes=self.penalties_enabled, 883 | ) 884 | 885 | if ( 886 | self.decision_boundary_distance(db_point) 887 | <= self.acceptance_threshold 888 | ): 889 | db_point2d = self.dimensionality_reduction.transform( 890 | [db_point] 891 | )[0] 892 | if ( 893 | db_point2d[0] >= self.X2d_xmin 894 | and db_point2d[0] <= self.X2d_xmax 895 | and db_point2d[1] >= self.X2d_ymin 896 | and db_point2d[1] <= self.X2d_ymax 897 | ): 898 | self.decision_boundary_points.append(db_point) 899 | self.decision_boundary_points_2d.append(db_point2d) 900 | if self.verbose and not suppress_output: 901 | # , ": New decision boundary keypoint found using linear optimization!" 902 | print( 903 | "{} {}/{}".format( 904 | step_str, i * n + j, len(from_idx) * n 905 | ) 906 | ) 907 | break 908 | else: 909 | if self.verbose and not suppress_output: 910 | msg = "{} {}/{}: Rejected decision boundary keypoint (outside of plot area)" 911 | print( 912 | msg.format(step_str, i * n + j, len(from_idx) * n) 913 | ) 914 | 915 | def _find_decision_boundary_along_line( 916 | self, 917 | from_point, 918 | to_point, 919 | penalize_extremes=False, 920 | penalize_tangent_distance=False, 921 | ): 922 | def objective(l, grad=0): 923 | # interpolate between source and destionation; calculate distance from 924 | # decision boundary 925 | X = from_point + l[0] * (to_point - from_point) 926 | error = self.decision_boundary_distance(X) 927 | 928 | if penalize_tangent_distance: 929 | # distance from tangent between class1 and class0 point in 2d space 930 | x0, y0 = self.dimensionality_reduction.transform([X])[0] 931 | x1, y1 = self.dimensionality_reduction.transform([from_point])[0] 932 | x2, y2 = self.dimensionality_reduction.transform([to_point])[0] 933 | error += ( 934 | 1e-12 935 | * np.abs((y2 - y1) * x0 - (x2 - x1) * y0 + x2 * y1 - y2 * x1) 936 | / np.sqrt((y2 - y1) ** 2 + (x2 - x1) ** 2) 937 | ) 938 | 939 | if penalize_extremes: 940 | error += 1e-8 * np.abs(0.5 - l[0]) 941 | 942 | return error 943 | 944 | optimizer = self._get_optimizer() 945 | optimizer.set_min_objective(objective) 946 | cl = optimizer.optimize([random.random()]) 947 | db_point = from_point + cl[0] * (to_point - from_point) 948 | return db_point 949 | 950 | def _find_decision_boundary_on_hypersphere(self, centroid, R, penalize_known=False): 951 | def objective(phi, grad=0): 952 | # search on hypersphere surface in polar coordinates - map back to cartesian 953 | cx = centroid + polar_to_cartesian(phi, R) 954 | try: 955 | cx2d = self.dimensionality_reduction.transform([cx])[0] 956 | error = self.decision_boundary_distance(cx) 957 | if penalize_known: 958 | # slight penalty for being too close to already known decision boundary 959 | # keypoints 960 | db_distances = [ 961 | euclidean(cx2d, self.decision_boundary_points_2d[k]) 962 | for k in range(len(self.decision_boundary_points_2d)) 963 | ] 964 | error += ( 965 | 1e-8 966 | * ( 967 | (self.mean_2d_dist - np.min(db_distances)) 968 | / self.mean_2d_dist 969 | ) 970 | ** 2 971 | ) 972 | return error 973 | except (Exception, ex): 974 | print("Error in objective function:", ex) 975 | return np.infty 976 | 977 | optimizer = self._get_optimizer( 978 | D=self.X.shape[1] - 1, 979 | upper_bound=2 * np.pi, 980 | iteration_budget=self.hypersphere_iteration_budget, 981 | ) 982 | optimizer.set_min_objective(objective) 983 | db_phi = optimizer.optimize( 984 | [random.random() * 2 * np.pi for k in range(self.X.shape[1] - 1)] 985 | ) 986 | db_point = centroid + polar_to_cartesian(db_phi, R) 987 | return db_point 988 | 989 | def _get_optimizer(self, D=1, upper_bound=1, iteration_budget=None): 990 | """Utility function creating an NLOPT optimizer with default 991 | parameters depending on this objects parameters 992 | """ 993 | if iteration_budget == None: 994 | iteration_budget = self.linear_iteration_budget 995 | 996 | opt = nlopt.opt(nlopt.GN_DIRECT_L_RAND, D) 997 | # opt.set_stopval(self.acceptance_threshold/10.0) 998 | opt.set_ftol_rel(1e-5) 999 | opt.set_maxeval(iteration_budget) 1000 | opt.set_lower_bounds(0) 1001 | opt.set_upper_bounds(upper_bound) 1002 | 1003 | return opt 1004 | -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | import uci_loader 2 | import numpy as np 3 | from decisionboundaryplot import DBPlot 4 | from sklearn.ensemble import RandomForestClassifier 5 | from sklearn.linear_model import LogisticRegression 6 | from sklearn.datasets import load_digits 7 | import matplotlib.pyplot as plt 8 | from sklearn.model_selection import learning_curve 9 | from numpy.random.mtrand import permutation 10 | from sklearn.svm import SVC 11 | from sklearn.neighbors import KNeighborsClassifier 12 | 13 | if __name__ == "__main__": 14 | # load data 15 | X, y = uci_loader.getdataset("iris") 16 | # data = load_digits(n_class = 2) 17 | # X,y = data.data, data.target 18 | y[y != 0] = 1 19 | 20 | # shuffle data 21 | random_idx = permutation(np.arange(len(y))) 22 | X = X[random_idx] 23 | y = y[random_idx] 24 | 25 | # create model 26 | model = LogisticRegression(C=1) 27 | # model = RandomForestClassifier(n_estimators=10) 28 | 29 | # plot high-dimensional decision boundary 30 | db = DBPlot(model) 31 | db.fit(X, y, training_indices=0.5) 32 | db.plot( 33 | plt, generate_testpoints=True 34 | ) # set generate_testpoints=False to speed up plotting 35 | 36 | plt.show() 37 | 38 | # plot learning curves for comparison 39 | N = 10 40 | train_sizes, train_scores, test_scores = learning_curve(model, X, y, cv=5) 41 | plt.errorbar( 42 | train_sizes, 43 | np.mean(train_scores, axis=1), 44 | np.std(train_scores, axis=1) / np.sqrt(N), 45 | ) 46 | plt.errorbar( 47 | train_sizes, 48 | np.mean(test_scores, axis=1), 49 | np.std(test_scores, axis=1) / np.sqrt(N), 50 | c="r", 51 | ) 52 | 53 | plt.legend(["Accuracies on training set", "Accuracies on test set"]) 54 | plt.xlabel("Number of data points") 55 | plt.title(str(model)) 56 | plt.show() 57 | -------------------------------------------------------------------------------- /img/digits_logreg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/digits_logreg.png -------------------------------------------------------------------------------- /img/digits_logreg_C10_good.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/digits_logreg_C10_good.png -------------------------------------------------------------------------------- /img/digits_logreg_C1e-9_bad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/digits_logreg_C1e-9_bad.png -------------------------------------------------------------------------------- /img/dr_model_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/dr_model_overview.png -------------------------------------------------------------------------------- /img/wine_knn2_bad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/wine_knn2_bad.png -------------------------------------------------------------------------------- /img/wine_knn_svc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/wine_knn_svc.png -------------------------------------------------------------------------------- /img/wine_rf50_good.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/wine_rf50_good.png -------------------------------------------------------------------------------- /img/wine_svc1_underfit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmadl/highdimensional-decision-boundary-plot/a01ed1d57ec611ad916e4bcb40e402e0fda2a40b/img/wine_svc1_underfit.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | matplotlib==3.0.2 2 | scipy==1.1.0 3 | numpy==1.15.4 4 | nlopt==2.6.1 5 | scikit_learn==0.21.2 6 | -------------------------------------------------------------------------------- /uci_loader.py: -------------------------------------------------------------------------------- 1 | from sklearn.datasets import fetch_openml 2 | from sklearn.preprocessing import OneHotEncoder 3 | import numpy as np 4 | 5 | 6 | def dshape(X): 7 | if len(X.shape) == 1: 8 | return X.reshape(-1, 1) 9 | else: 10 | return X if X.shape[0] > X.shape[1] else X.T 11 | 12 | 13 | def unpack(t): 14 | while type(t) == list or type(t) == np.ndarray: 15 | t = t[0] 16 | return t 17 | 18 | 19 | def tonumeric(lst): 20 | lbls = {} 21 | for t in lst.flatten(): 22 | if unpack(t) not in lbls: 23 | lbls[unpack(t)] = len(lbls.keys()) 24 | return np.array([lbls[unpack(t)] for t in lst.flatten()]) 25 | 26 | 27 | def getdataset(datasetname, onehot_encode_strings=True): 28 | # load 29 | dataset = fetch_openml(datasetname) 30 | # get X and y 31 | X = dshape(dataset.data) 32 | try: 33 | target = dshape(dataset.target) 34 | except: 35 | print("WARNING: No target found. Taking last column of data matrix as target") 36 | target = X[:, -1] 37 | X = X[:, :-1] 38 | if ( 39 | len(target.shape) > 1 and target.shape[1] > X.shape[1] 40 | ): # some mldata sets are mixed up... 41 | X = target 42 | target = dshape(dataset.data) 43 | if len(X.shape) == 1 or X.shape[1] <= 1: 44 | for k in dataset.keys(): 45 | if k != "data" and k != "target" and len(dataset[k]) == X.shape[1]: 46 | X = np.hstack((X, dshape(dataset[k]))) 47 | # one-hot for categorical values 48 | if onehot_encode_strings: 49 | cat_ft = [ 50 | i 51 | for i in range(X.shape[1]) 52 | if "str" in str(type(unpack(X[0, i]))) 53 | or "unicode" in str(type(unpack(X[0, i]))) 54 | ] 55 | if len(cat_ft): 56 | for i in cat_ft: 57 | X[:, i] = tonumeric(X[:, i]) 58 | X = OneHotEncoder(categorical_features=cat_ft).fit_transform(X) 59 | # if sparse, make dense 60 | try: 61 | X = X.toarray() 62 | except: 63 | pass 64 | # convert y to monotonically increasing ints 65 | y = tonumeric(target).astype(int) 66 | return np.nan_to_num(X.astype(float)), y 67 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | def minimum_spanning_tree(X, copy_X=True): 5 | """X are edge weights of fully connected graph""" 6 | if copy_X: 7 | X = X.copy() 8 | 9 | if X.shape[0] != X.shape[1]: 10 | raise ValueError("X needs to be square matrix of edge weights") 11 | n_vertices = X.shape[0] 12 | spanning_edges = [] 13 | 14 | # initialize with node 0: 15 | visited_vertices = [0] 16 | num_visited = 1 17 | # exclude self connections: 18 | diag_indices = np.arange(n_vertices) 19 | X[diag_indices, diag_indices] = np.inf 20 | 21 | while num_visited != n_vertices: 22 | new_edge = np.argmin(X[visited_vertices], axis=None) 23 | # 2d encoding of new_edge from flat, get correct indices 24 | new_edge = divmod(new_edge, n_vertices) 25 | new_edge = [visited_vertices[new_edge[0]], new_edge[1]] 26 | # add edge to tree 27 | spanning_edges.append(new_edge) 28 | visited_vertices.append(new_edge[1]) 29 | # remove all edges inside current tree 30 | X[visited_vertices, new_edge[1]] = np.inf 31 | X[new_edge[1], visited_vertices] = np.inf 32 | num_visited += 1 33 | return np.vstack(spanning_edges) 34 | 35 | 36 | def polar_to_cartesian(arr, r): 37 | a = np.concatenate((np.array([2 * np.pi]), arr)) 38 | si = np.sin(a) 39 | si[0] = 1 40 | si = np.cumprod(si) 41 | co = np.cos(a) 42 | co = np.roll(co, -1) 43 | return si * co * r 44 | --------------------------------------------------------------------------------