├── .gitignore ├── plots ├── html │ ├── .DS_Store │ ├── PCA_analysis.html │ ├── roc.html │ ├── roc_pr.html │ ├── roc_pr_curve.html │ ├── roc_pr_plot.html │ └── roc_cvfold │ │ └── bokeh-1.0.4.min.css └── png │ ├── Normalized_corr_matrix.png │ ├── Z_normalized_corr_matrix.png │ └── Z_normalized_corr_matrix_Abs.png ├── pip_requirements.txt ├── code ├── __pycache__ │ └── accstats.cpython-36.pyc ├── accstats.py ├── pca_feature_correlation.py └── roc_pr_curve.py ├── README.md └── LICENSE /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ 2 | __pycache__/* 3 | .ipynb_checkpoints 4 | .ipynb_checkpoints/* 5 | .DS_Store 6 | -------------------------------------------------------------------------------- /plots/html/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaurav-kaushik/Data-Visualizations-Medium/HEAD/plots/html/.DS_Store -------------------------------------------------------------------------------- /pip_requirements.txt: -------------------------------------------------------------------------------- 1 | seaborn>=0.9.0 2 | bokeh>=0.12.0 3 | numpy>=1.14.0 4 | pandas>=0.22.0 5 | matplotlib>=2.1.0 6 | scikit-learn>=0.19.0 7 | -------------------------------------------------------------------------------- /plots/png/Normalized_corr_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaurav-kaushik/Data-Visualizations-Medium/HEAD/plots/png/Normalized_corr_matrix.png -------------------------------------------------------------------------------- /plots/png/Z_normalized_corr_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaurav-kaushik/Data-Visualizations-Medium/HEAD/plots/png/Z_normalized_corr_matrix.png -------------------------------------------------------------------------------- /code/__pycache__/accstats.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaurav-kaushik/Data-Visualizations-Medium/HEAD/code/__pycache__/accstats.cpython-36.pyc -------------------------------------------------------------------------------- /plots/png/Z_normalized_corr_matrix_Abs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gaurav-kaushik/Data-Visualizations-Medium/HEAD/plots/png/Z_normalized_corr_matrix_Abs.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Understanding Data and Machine Learning Models with Visualizations 2 | 3 | ### Part 1 - PCA and Feature Correlation 4 | 5 | * code/Interactive PCA and Feature Correlation.ipynb 6 | * code/pca\_feature\_correlation.py 7 | * Post on [Cascade.Bio blog](https://medium.com/cascade-bio-blog/creating-visualizations-to-better-understand-your-data-and-models-part-1-a51e7e5af9c0) 8 | 9 | ### Part 2 - Machine Learning Decision Boundary Visualization 10 | 11 | * code/Interactive\_Model\_Predictions\_and\_Decision\_Boundaries.ipynb 12 | * Post on [Cascade.Bio blog](https://medium.com/cascade-bio-blog/creating-visualizations-to-better-understand-your-data-and-models-part-2-28d5c46e956) 13 | 14 | ### Part 3 - ROC Curves 15 | 16 | * code/Interactive\_ROC\_analysis.ipynb 17 | * code/accstats.py 18 | * code/roc\_pr\_curve.py 19 | * Post on [HackerNoon](https://medium.com/hackernoon/making-sense-of-real-world-data-roc-curves-and-when-to-use-them-90a17e6d1db) 20 | 21 | 22 | --- 23 | 24 | Note: code was developed in Python 3.6 and likely not backwards compatible because of liberal use of f-strings (sorry not sorry) -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019 Gaurav Kaushik 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /code/accstats.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import numpy as np 5 | 6 | 7 | def confusion_matrix(acc:float=0.995, subpop:float=1e-4, population:float=8.5e6) -> dict: 8 | """ 9 | Generates confusion matrix and derived variables 10 | based on accuracy of detecting a fraction (subpop) of a population 11 | 12 | Args: 13 | acc (float): accuracy of detecting subpop (0-1) 14 | subpop (float): fraction of population in subpopulation (0-1) 15 | population (float): Total population size (absolute number) 16 | 17 | Returns: 18 | dict: derived variables as a dictionary 19 | """ 20 | 21 | # Inputs 22 | population = int(population) 23 | print(f"\nInputs\n------------") 24 | print(f"Accuracy (%): {100*acc}") 25 | print(f"Subpopulation (%): {100*subpop}") 26 | print(f"Population size: {population}") 27 | print(f"Predicted subpopulation size: {int(subpop*population)}") 28 | 29 | # Check variables 30 | if acc > 1 or acc < 0: 31 | print("\nERROR: give valid accuracy (0 to 1).") 32 | return 33 | if subpop > 1 or subpop < 0: 34 | print("\nERROR: give valid subpop percent (0 to 1).") 35 | return 36 | if population < 1: 37 | print("\nERROR: cannot have zero or negative populations.") 38 | return 39 | 40 | # confusion matrix 41 | tp = np.rint(population*subpop*acc).astype(int) 42 | fp = np.rint(population*(1-acc)).astype(int) 43 | tn = np.rint(population*(1-subpop)*acc).astype(int) 44 | fn = np.rint(population*subpop).astype(int) - tp 45 | print(f"\nResults\n------------") 46 | print(f"True Positives (Power): {tp}") 47 | print(f"False Positives (Type I): {fp}") 48 | print(f"True Negatives: {tn}") 49 | print(f"False Negatives (Type II): {fn}") 50 | 51 | # derivations 52 | round_var = 4 # round vars to this place 53 | tpr = np.round((tp)/(tp+fn), round_var) 54 | fpr = np.round((fp)/(fp+tn), round_var) 55 | precision = np.round((tp)/(tp+fp), round_var) 56 | specificity = np.round((tn)/(tn+fp), round_var) 57 | fdr = np.round((fp)/(fp+tp), round_var) 58 | fscore = np.round(2*tp/(2*tp+fp+fn), round_var) 59 | print(f"\nDerivations\n------------") 60 | print(f"True Positive Rate (Recall): {tpr}") 61 | print(f"False Positive Rate: {fpr}") 62 | print(f"Precision: {precision}") 63 | print(f"Specificity: {specificity}") 64 | print(f"False Discovery Rate: {fdr}") 65 | print(f"F-Score: {fscore}") 66 | 67 | # output a dictionary of derived variables 68 | output = { 69 | 'True_Positives':tp, 70 | 'False_Positives':fp, 71 | 'True_Negatives':tn, 72 | 'False_Negatives':fn, 73 | 'TPR':tpr, 74 | 'FPR':fpr, 75 | 'Precision':precision, 76 | 'Specificity':specificity, 77 | 'FDR':fdr, 78 | 'FScore':fscore 79 | } 80 | 81 | return output 82 | 83 | 84 | if __name__ == '__main__': 85 | parser = argparse.ArgumentParser() 86 | parser.add_argument("-a", "--acc", type=float, default=0.995, help="Accuracy (0 -> 1)") 87 | parser.add_argument("-s", "--sub", type=float, default=1e-4, help="Subpop fraction (0 -> 1)") 88 | parser.add_argument("-p", "--pop", type=float, default=8.5e6, help="Pop size") 89 | args = parser.parse_args() 90 | accuracy = args.acc 91 | subpopulation = args.sub 92 | population = args.pop 93 | 94 | # main 95 | confusion_matrix(accuracy, subpopulation, population) 96 | -------------------------------------------------------------------------------- /code/pca_feature_correlation.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import os 5 | import pandas as pd 6 | import numpy as np 7 | import seaborn as sns 8 | from sklearn import datasets 9 | from sklearn.decomposition import PCA 10 | from sklearn.preprocessing import StandardScaler 11 | import matplotlib.pyplot as plt 12 | from matplotlib.colors import cnames 13 | from itertools import cycle 14 | from bokeh.plotting import output_file, figure, show, ColumnDataSource 15 | from bokeh.models import HoverTool 16 | import warnings 17 | warnings.filterwarnings(action='ignore') 18 | 19 | 20 | def get_float_list(range_max:int, div:int=100) -> list: 21 | """ To get 0 -> 1, range_max must be same order of mag as div """ 22 | return [float(x)/div for x in range(int(range_max))] 23 | 24 | 25 | def get_colorcycle(colordict:dict): 26 | """ Subset cnames with a string match and get a color cycle for plotting """ 27 | return cycle(list(colordict.keys())) 28 | 29 | 30 | def get_colordict(filter_:str='dark') -> dict: 31 | """ return dictionary of colornames by filter """ 32 | return dict((k, v) for k, v in cnames.items() if filter_ in k) 33 | 34 | 35 | def pca_report_interactive(X, scale_X:bool=True, save_plot:bool=False): 36 | """ 37 | X: input data matrix 38 | scale_X: determine whether to rescale X (StandardScaler) [default: True, X is not prescaled 39 | save_plot: save plot to file (html) and not show 40 | """ 41 | 42 | # calculate mean and var 43 | X_mean, X_var = X.mean(), X.var() 44 | print('\n*--- PCA Report ---*\n') 45 | print(f'X mean:\t\t{X_mean:.3f}\nX variance:\t{X_var:.3f}') 46 | 47 | if scale_X: 48 | # rescale and run PCA 49 | print("\n...Rescaling data...\n") 50 | scaler = StandardScaler() 51 | X_scaled = scaler.fit_transform(X) 52 | X_s_mean, X_s_var = X_scaled.mean(), X_scaled.var() 53 | print(f'X_scaled mean:\t\t{np.round(X_s_mean):.3f}') 54 | print(f'X_scaled variance:\t{np.round(X_s_var):.3f}\n') 55 | pca_ = PCA().fit(X_scaled) 56 | X_pca = PCA().fit_transform(X) 57 | else: 58 | # run PCA directly 59 | print("...Assuming data is properly scaled...") 60 | pca_ = PCA().fit(X) 61 | X_pca = PCA().fit_transform(X) 62 | 63 | # Get cumulative explained variance for each dimension 64 | pca_evr = pca_.explained_variance_ratio_ 65 | cumsum_ = np.cumsum(pca_evr) 66 | 67 | # Get dimensions where var >= 95% and values for variance at 2D, 3D 68 | dim_95 = np.argmax(cumsum_ >= 0.95) + 1 69 | twoD = np.round(cumsum_[1], decimals=3)*100 70 | threeD = np.round(cumsum_[2], decimals=3)*100 71 | instances_, dims_ = X.shape 72 | 73 | # check shape of X 74 | if dims_ > instances_: 75 | print("WARNING: number of features greater than number of instances.") 76 | dimensions = list(range(1, instances_+1)) 77 | else: 78 | dimensions = list(range(1, dims_+1)) 79 | 80 | # Print report 81 | print("\n -- Summary --") 82 | print(f"You can reduce from {dims_} to {dim_95} dimensions while retaining 95% of variance.") 83 | print(f"2 principal components explain {twoD:.2f}% of variance.") 84 | print(f"3 principal components explain {threeD:.2f}% of variance.") 85 | 86 | """ - Plotting - """ 87 | # Create custom HoverTool -- we'll name each ROC curve 'ROC' so we only see info on hover there 88 | hover_ = HoverTool(names=['PCA'], tooltips=[("dimensions", "@x_dim"), 89 | ("cumulative variance", "@y_cumvar"), 90 | ("explained variance", "@y_var")]) 91 | p_tools = [hover_, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 92 | 93 | # insert 0 at beginning for cleaner plotting 94 | cumsum_plot = np.insert(cumsum_, 0, 0) 95 | pca_evr_plot = np.insert(pca_evr, 0, 0) 96 | dimensions_plot = np.insert(dimensions, 0, 0) 97 | 98 | """ 99 | ColumnDataSource 100 | - a special type in Bokeh that allows you to store data for plotting 101 | - store data as dict (key:list) 102 | - to plot two keys against one another, make sure they're the same length! 103 | - below: 104 | x_dim # of dimensions (length = # of dimensions) 105 | y_cumvar # cumulative variance (length = # of dimensions) 106 | var_95 # y = 0.95 (length = # of dimensions) 107 | zero_one # list of 0 to 1 108 | twoD # x = 2 109 | threeD # x = 3 110 | """ 111 | 112 | # get sources 113 | source_PCA = ColumnDataSource(data=dict(x_dim = dimensions_plot,y_cumvar = cumsum_plot, y_var = pca_evr_plot)) 114 | source_var95 = ColumnDataSource(data=dict(var95_x = [dim_95]*96, var95_y = get_float_list(96))) 115 | source_twoD = ColumnDataSource(data=dict(twoD_x = [2]*(int(twoD)+1), twoD_y = get_float_list(twoD+1))) 116 | source_threeD = ColumnDataSource(data=dict(threeD_x = [3]*(int(threeD)+1), threeD_y = get_float_list(threeD+1))) 117 | 118 | """ PLOT """ 119 | # set up figure and add axis labels 120 | p = figure(title='PCA Analysis', tools=p_tools) 121 | p.xaxis.axis_label = f'N of {dims_} Principal Components' 122 | p.yaxis.axis_label = 'Variance Explained (per PC & Cumulative)' 123 | 124 | # add reference lines: y=0.95, x=2, x=3 125 | p.line('twoD_x', 'twoD_y', line_width=0.5, line_dash='dotted', color='#435363', source=source_twoD) # x=2 126 | p.line('threeD_x', 'threeD_y', line_width=0.5, line_dash='dotted', color='#435363', source=source_threeD) # x=3 127 | p.line('var95_x', 'var95_y', line_width=2, line_dash='dotted', color='#435363', source=source_var95) # var = 0.95 128 | 129 | # add bar plot for variance per dimension 130 | p.vbar(x='x_dim', top='y_var', width=.5, bottom=0, color='#D9F2EF', source=source_PCA, name='PCA') 131 | 132 | # add cumulative variance (scatter + line) 133 | p.line('x_dim', 'y_cumvar', line_width=1, color='#F79737', source=source_PCA) 134 | p.circle('x_dim', 'y_cumvar', size=7, color='#FF4C00', source=source_PCA, name='PCA') 135 | 136 | # change gridlines 137 | p.ygrid.grid_line_alpha = 0.25 138 | p.xgrid.grid_line_alpha = 0.25 139 | 140 | # change axis bounds and grid 141 | p.xaxis.bounds = (0, dims_) 142 | p.yaxis.bounds = (0, 1) 143 | p.grid.bounds = (0, dims_) 144 | 145 | # save and show p 146 | if save_plot: 147 | output_file('PCA_analysis.html') 148 | show(p) 149 | 150 | # output PCA info as a dataframe 151 | df_PCA = pd.DataFrame({'dimension': dimensions, 'variance_cumulative': cumsum_, 'variance': pca_evr}).set_index(['dimension']) 152 | 153 | return df_PCA, X_pca, pca_evr 154 | 155 | 156 | def pca_feature_correlation(X, X_pca, explained_var, features:list=None, fig_dpi:int=150, save_plot:bool=False): 157 | """ 158 | 1. Get dot product of X and X_pca 159 | 2. Run normalizations of X*X_pca 160 | 3. Retrieve df/matrices 161 | 162 | X: data (numpy matrix) 163 | X_pca: PCA 164 | explained_var: explained variance matrix 165 | features: list of feature names 166 | fig_dpi: dpi to use for heatmaps 167 | save_plot: save plot to file (html) and not show 168 | """ 169 | 170 | # Add zeroes for data where features > instances 171 | outer_diff = X.T.shape[0] - X_pca.shape[1] 172 | if outer_diff > 0: # outer dims must match to get sq matrix 173 | Z = np.zeros([X_pca.shape[0], outer_diff]) 174 | X_pca = np.c_[X_pca, Z] 175 | explained_var = np.append(explained_var, np.zeros(outer_diff)) 176 | 177 | # Get correlation between original features (X) and PCs (X_pca) 178 | dot_matrix = np.dot(X.T, X_pca) 179 | print(f"X*X_pca: {X.T.shape} * {X_pca.shape} = {dot_matrix.shape}") 180 | 181 | # Correlation matrix -> df 182 | df_dotproduct = pd.DataFrame(dot_matrix) 183 | df_dotproduct.columns = [''.join(['PC', f'{i+1}']) for i in range(dot_matrix.shape[0])] 184 | if any(features): df_dotproduct.index = features 185 | 186 | # Normalize & Sort 187 | df_n, df_na, df_nabv = normalize_dataframe(df_dotproduct, explained_var, plot_opt=True, save_plot=save_plot) 188 | 189 | return df_dotproduct, df_n, df_na, df_nabv 190 | 191 | 192 | def normalize_dataframe(df, explained_var=None, fig_dpi:int=150, plot_opt:bool=True, save_plot:bool=False): 193 | """ 194 | 1. Get z-normalized df (normalized to µ=0, σ=1) 195 | 2. Get absolute value of z-normalized df 196 | 3. If explained_variance matrix provided, dot it w/ (2) 197 | """ 198 | # Normalize, Reindex, & Sort 199 | df_norm = (df.copy()-df.mean())/df.std() 200 | df_norm = df_norm.sort_values(list(df_norm.columns), ascending=False) 201 | 202 | # Absolute value of normalized (& sort) 203 | df_abs = df_norm.copy().abs().set_index(df_norm.index) 204 | df_abs = df_abs.sort_values(by=list(df_abs.columns), ascending=False) 205 | 206 | # Plot 207 | if plot_opt: 208 | # Z-normalized corr matrix 209 | plt.figure(dpi=fig_dpi) 210 | ax_normal = sns.heatmap(df_norm, cmap="RdBu") 211 | ax_normal.set_title("Z-Normalized Data") 212 | if save_plot: 213 | plt.savefig('Z_normalized_corr_matrix.png') 214 | else: 215 | plt.show() 216 | 217 | # |Z-normalized corr matrix| 218 | plt.figure(dpi=fig_dpi) 219 | ax_abs = sns.heatmap(df_abs, cmap="Purples") 220 | ax_abs.set_title("|Z-Normalized|") 221 | if save_plot: 222 | plt.savefig('Z_normalized_corr_matrix_Abs.png') 223 | else: 224 | plt.show() 225 | 226 | # Re-normalize by explained var (& sort) 227 | if explained_var.any(): 228 | df_byvar = df_abs.copy()*explained_var 229 | df_byvar = df_byvar.sort_values(by=list(df_norm.columns), ascending=False) 230 | if plot_opt: 231 | plt.figure(dpi=fig_dpi) 232 | ax_relative = sns.heatmap(df_byvar, cmap="Purples") 233 | ax_relative.set_title("|Z-Normalized|*Explained_Variance") 234 | if save_plot: 235 | plt.savefig('Normalized_corr_matrix.png') 236 | else: 237 | plt.show() 238 | else: 239 | df_byvar = None 240 | return df_norm, df_abs, df_byvar 241 | 242 | 243 | def pca_rank_features(df_nabv, verbose:bool=True): 244 | """ 245 | Given a dataframe df_nabv with dimensions [f, p], where: 246 | f = features (sorted) 247 | p = principal components 248 | df_nabv.values are |Z-normalized X|*pca_.explained_variance_ratio_ 249 | 250 | 1. Create column of sum of each row, sort by it 'score_' 251 | 3. Set index as 'rank' 252 | """ 253 | df_rank = df_nabv.copy().assign(score_ = df_nabv.sum(axis=1)).sort_values('score_', ascending=False) 254 | df_rank['feature_'] = df_rank.index 255 | df_rank.index = range(1, len(df_rank)+1) 256 | df_rank.drop(df_nabv.columns, axis=1, inplace=True) 257 | df_rank.index.rename('rank', inplace=True) 258 | if verbose: print(df_rank) 259 | return df_rank 260 | 261 | 262 | def pca_full_report(X, features_:list=None, fig_dpi:int=150, save_plot:bool=False): 263 | """ 264 | Run complete PCA workflow: 265 | 1. pca_report_interactive() 266 | 2. pca_feature_correlation() 267 | 3. pca_rank_features() 268 | 269 | X: data (numpy array) 270 | features_: list of feature names 271 | fig_dpi: image resolution 272 | 273 | """ 274 | # Retrieve the interactive report 275 | df_pca, X_pca, pca_evr = pca_report_interactive(X, save_plot=save_plot) 276 | # Get feature-PC correlation matrices 277 | df_corr, df_n, df_na, df_nabv = pca_feature_correlation(X, X_pca, pca_evr, features_, fig_dpi, save_plot) 278 | # Get rank for each feature 279 | df_rank = pca_rank_features(df_nabv) 280 | return (df_pca, X_pca, pca_evr, df_corr, df_n, df_na, df_nabv, df_rank) 281 | 282 | 283 | if __name__ == '__main__': 284 | """ IRIS """ 285 | data = datasets.load_iris() 286 | outputs = pca_full_report(X=data.data, features_=data.feature_names, save_plot=True) 287 | 288 | -------------------------------------------------------------------------------- /plots/html/PCA_analysis.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Bokeh Plot 7 | 8 | 9 | 10 | 11 | 14 | 15 | 16 | 17 |
18 |
19 |
20 | 21 | 24 | 59 | 60 | -------------------------------------------------------------------------------- /plots/html/roc.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Bokeh Plot 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 |
34 | 35 | 36 | 37 | 38 | 39 | 42 | 77 | 78 | 79 | 80 | -------------------------------------------------------------------------------- /plots/html/roc_pr.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | Bokeh Plot 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 |
39 | 40 | 41 | 42 | 43 | 44 | 47 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /plots/html/roc_pr_curve.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | Bokeh Plot 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 |
39 | 40 | 41 | 42 | 43 | 44 | 47 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /plots/html/roc_pr_plot.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | Bokeh Plot 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 |
39 | 40 | 41 | 42 | 43 | 44 | 47 | 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /code/roc_pr_curve.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import pandas as pd 4 | import numpy as np 5 | from itertools import cycle 6 | from sklearn import datasets 7 | from sklearn.datasets import make_classification 8 | from sklearn.decomposition import PCA 9 | from sklearn.preprocessing import StandardScaler 10 | from sklearn.metrics import roc_curve, auc, precision_recall_curve, average_precision_score, accuracy_score 11 | from sklearn.model_selection import train_test_split, StratifiedKFold 12 | from sklearn.ensemble import RandomForestClassifier 13 | from sklearn.linear_model import LogisticRegression 14 | from sklearn.naive_bayes import GaussianNB 15 | from bokeh.plotting import output_file, figure, show, ColumnDataSource 16 | from bokeh.models import HoverTool 17 | from bokeh.layouts import row 18 | import matplotlib.pyplot as plt 19 | from matplotlib.colors import cnames 20 | cnames = dict((k, v) for k, v in cnames.items() if 'dark' in k) 21 | 22 | 23 | def PCA_2D_labeled(X, y, cnames:list, target_names:list): 24 | """ 25 | Get a quick 2D rescaled PCA of a labeled dataset 26 | 27 | Args: 28 | X (numpy.ndarray): data 29 | y (numpy.ndarray): labels 30 | cnames: a list of color names (str) 31 | target_names: a list of target names (str) 32 | 33 | Returns: 34 | matplotlib plot object 35 | """ 36 | 37 | # rescaled, 2D PCA 38 | X_2D = PCA(2).fit_transform(StandardScaler().fit_transform(X)) 39 | 40 | # plot 41 | plt.figure(dpi=150) 42 | for c, i, t in zip(['red', 'green'], set(y), target_names): 43 | # plot each column with a color pertaining to the labels 44 | plt.scatter(X_2D[y==i, 0], X_2D[y==i, 1], color=c, alpha=.2, lw=1, label=t) 45 | plt.legend(loc='best') 46 | plt.xticks([]) 47 | plt.yticks([]) 48 | plt.xlabel('PC1') 49 | plt.ylabel('PC2') 50 | plt.tight_layout() 51 | plt.show() 52 | return plt 53 | 54 | 55 | """ Function to return all plots for a classifer """ 56 | 57 | def classifier_plots(clf_trained, X_test, y_test, target_names:list, minority_idx:int=0, ylog:bool=False): 58 | """ 59 | Get summary plots for a trained classifier 60 | 61 | Args: 62 | clf_trained: trained sklearn clf 63 | X_test (np.ndarray): withheld test data 64 | y_test (np.ndarray): withheld test data labels 65 | target_names (list): list of target labels/names 66 | minority_idx: (int): index for the minority class (e.g. 0, 1) 67 | ylog (str): toggle log-scaling on yaxis 68 | 69 | Returns: 70 | None 71 | """ 72 | 73 | """ Probabilty Dist """ 74 | # get the probability distribution 75 | probas = clf_trained.predict_proba(X_test) 76 | 77 | # PLOT - count 78 | plt.figure(dpi=150) 79 | plt.hist(probas, bins=20) 80 | plt.title('Classification Probabilities') 81 | plt.xlabel('Probability') 82 | plt.ylabel('# of Instances') 83 | plt.xlim([0.5, 1.0]) 84 | if ylog: plt.yscale('log') 85 | plt.legend(target_names) 86 | plt.show() 87 | 88 | 89 | # PLOT - density 90 | plt.figure(dpi=150) 91 | plt.hist(probas[:, minority_idx], bins=20, density=True) 92 | plt.title('Classification Density (Minority)') 93 | plt.xlabel('Probability') 94 | plt.ylabel('% of Total') 95 | if ylog: plt.yscale('log') 96 | plt.xlim([0, 1.0]) 97 | plt.legend(target_names) 98 | plt.show() 99 | 100 | """ ROC curve """ 101 | 102 | # get false and true positive rates 103 | fpr, tpr, _ = roc_curve(y_test, probas[:,0], pos_label=0) 104 | 105 | # get area under the curve 106 | clf_auc = auc(fpr, tpr) 107 | 108 | # PLOT ROC curve 109 | plt.figure(dpi=150) 110 | plt.plot(fpr, tpr, lw=1, color='green', label=f'AUC = {clf_auc:.3f}') 111 | plt.plot([0,1], [0,1], '--k', lw=0.5, label='Random') 112 | plt.title('ROC') 113 | plt.xlabel('False Positive Rate') 114 | plt.ylabel('True Positive Rate (Recall)') 115 | plt.xlim([-0.05, 1.05]) 116 | plt.ylim([-0.05, 1.05]) 117 | plt.legend() 118 | plt.show() 119 | 120 | """ Precision Recall Curve """ 121 | 122 | # get precision and recall values 123 | precision, recall, _ = precision_recall_curve(y_test, probas[:,0], pos_label=0) 124 | 125 | # average precision score 126 | avg_precision = average_precision_score(y_test, probas[:,0]) 127 | 128 | # precision auc 129 | pr_auc = auc(recall, precision) 130 | 131 | # plot 132 | plt.figure(dpi=150) 133 | plt.plot(recall, precision, lw=1, color='blue', label=f'AP={avg_precision:.3f}; AUC={pr_auc:.3f}') 134 | plt.fill_between(recall, precision, -1, facecolor='lightblue', alpha=0.5) 135 | 136 | plt.title('PR Curve') 137 | plt.xlabel('Recall (TPR)') 138 | plt.ylabel('Precision') 139 | plt.xlim([-0.05, 1.05]) 140 | plt.ylim([-0.05, 1.05]) 141 | plt.legend() 142 | plt.show() 143 | 144 | 145 | """ 146 | Interactive Plots and Utility Functions 147 | """ 148 | 149 | def get_clf_name(clf): 150 | """ 151 | get_clf_name takes a classifer (trained or untrained) and returns its name as a string 152 | clf.__str__() will return a string of the classifiers name and params 153 | e.g. "LogisticRegresion(...)" or "(("Logistic Regression(...)")) 154 | We then split on "(", use filter to drop empty strings, convert to list, and return first item 155 | clf: sklearn classifier (e.g. rf = RandomForestClassifier()) 156 | """ 157 | return list(filter(None, clf.__str__().split("(")))[0] 158 | 159 | 160 | def get_ROC_data(data, clf, pos_label_=None, verbose=False): 161 | 162 | """ 163 | source_ROC, df_ROC, clf = get_ROC_data(data, clf, verbose=False) 164 | 165 | get_ROC_data will return ColumnDataSource and dataframes with TPR and FPR 166 | for a particular dataset and an untrained classifier. The CSD can be used 167 | to plot a Bokeh plot while the dataframe can be used for additional 168 | exploration and plotting with other libs. Note that the dataframes 169 | are returned with metadata (e.g. AUC and the clf used). 170 | 171 | data: tuple of our data (X_train, X_test, y_train, y_test) 172 | where each item in the tuple is a numpy ndarray 173 | clf: an untrained classifier (e.g. rf = RandomForestClassifier()) 174 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 175 | verbose: print warnings [default: False] 176 | """ 177 | 178 | # split data into training, testing 179 | (X_train, X_test, y_train, y_test) = data 180 | 181 | # train and retrieve probabilities of class per feature for the test data 182 | probas_ = clf.fit(X_train, y_train).predict_proba(X_test) 183 | 184 | # get false and true positive rates for positive labels 185 | # (and thresholds, which is not used but shown here for fyi) 186 | if not pos_label_: 187 | pos_label_ = np.max(y_train) 188 | if verbose: 189 | print(f"Warning: Maximum target value of '{pos_label_}' used as positive.") 190 | print("You can use 'pos_label_' to indicate your own.") 191 | 192 | # get values for roc curve 193 | fpr, tpr, thresholds = roc_curve(y_test, probas_[:,1], pos_label=pos_label_) 194 | thresholds[0] = np.nan 195 | 196 | # get area under the curve (AUC) 197 | roc_auc = auc(fpr, tpr) 198 | 199 | # create legend variables - we'll create an array with len(tpr) 200 | auc_ = [f"AUC: {roc_auc:.3f}"]*len(tpr) 201 | clf_name = get_clf_name(clf) 202 | clf_ = [f"{clf_name}, AUC: {roc_auc:.3f}"]*len(tpr) 203 | 204 | # create bokeh column source for plotting new ROC 205 | source_ROC = ColumnDataSource(data=dict(x_fpr=fpr, 206 | y_tpr=tpr, 207 | thresh=thresholds, 208 | auc_legend=auc_, 209 | clf_legend=clf_)) 210 | 211 | # create output dataframe with TPR and FPR, and metadata 212 | df_ROC = pd.DataFrame({'TPR':tpr, 'FPR':fpr, 'Thresholds':thresholds}) 213 | df_ROC.auc = roc_auc 214 | df_ROC.clf = get_clf_name(clf) 215 | df_ROC.score = clf.score(X_test, y_test) 216 | 217 | return source_ROC, df_ROC, clf 218 | 219 | 220 | def interpolate_mean_tpr(FPRs=None, TPRs=None, df_list=None): 221 | """ 222 | mean_fpr, mean_tpr = interpolate_mean_tpr(FPRs=None, TPRs=None, df_list=None) 223 | 224 | FPRs: False positive rates (list of n arrays) 225 | TPRs: True positive rates (list of n arrays) 226 | df_list: DataFrames with TPR, FPR columns (list of n DataFrames) 227 | """ 228 | 229 | # seed empty linspace 230 | mean_tpr, mean_fpr = 0, np.linspace(0, 1, 101) 231 | 232 | if TPRs and FPRs: 233 | for idx, PRs in enumerate(zip(FPRs, TPRs)): 234 | mean_tpr += np.interp(mean_fpr, PRs[0], PRs[1]) 235 | 236 | elif df_list: 237 | for idx, df_ in enumerate(df_list): 238 | mean_tpr += np.interp(mean_fpr, df_.FPR, df_.TPR) 239 | 240 | else: 241 | print("Please give valid inputs.") 242 | return None, None 243 | 244 | # normalize by length of inputs (# indices looped over) 245 | mean_tpr /= (idx+1) 246 | 247 | # add origin point 248 | mean_fpr = np.insert(mean_fpr, 0, 0) 249 | mean_tpr = np.insert(mean_tpr, 0, 0) 250 | 251 | return mean_fpr, mean_tpr 252 | 253 | 254 | def plot_ROC(clf, X, y, test_size_:float=0.5, pos_label_:str=None, filename:str=None, verbose:bool=False): 255 | """ 256 | clf, classifiers, df_ROCs = plot_ROC(clf, X, y, pos_label_=None, verbose=False) 257 | 258 | Plot an interactive ROC curve for a binary classifier. 259 | It returns the original clf, a classifier for each cv, a list of dataframes for each cv. 260 | 261 | clf: untrained classifier object (e.g. rf_clf = RandomForestClassifer()) 262 | X: training + testing data 263 | y: targets (numeric/integers) 264 | test_size: fraction of data to be reserved for testing [default: 0.5] 265 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 266 | filename: if provided, save to html [default: None] 267 | verbose: print warnings [default: False] 268 | """ 269 | 270 | """ Split and get ROC curve data """ 271 | data = train_test_split(X, y, test_size=test_size_) 272 | source_ROC, df_ROC, clf = get_ROC_data(data, clf, pos_label_, verbose) 273 | 274 | """ Set up initial PLOT """ 275 | # Create custom HoverTool -- we'll name each ROC curve 'ROC' so we only see info on hover there 276 | hover_ = HoverTool(names=['ROC'], tooltips=[("TPR", "@y_tpr"), ("FPR", "@x_fpr"), ("Thresh", "@thresh")]) 277 | 278 | # Create your toolbox 279 | p_tools = [hover_, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 280 | 281 | # Create figure and labels 282 | clf_name = get_clf_name(clf) 283 | p = figure(title=f'{clf_name} ROC curve', tools=p_tools) 284 | p.xaxis.axis_label = 'False Positive Rate' 285 | p.yaxis.axis_label = 'True Positive Rate' 286 | 287 | """ PLOT ROC """ 288 | p.line('x_fpr', 'y_tpr', line_width=1, color="blue", source=source_ROC) 289 | p.circle('x_fpr', 'y_tpr', size=3, color="orange", legend='auc_legend', source=source_ROC, name='ROC') 290 | 291 | """ Plot Threshold==0.5 """ 292 | # get value closest to threshold == 0.5 293 | df_half = df_ROC.dropna().iloc[(df_ROC['Thresholds'].dropna()-0.5).abs().argsort()[:2]] 294 | df_half['Legend'] = 'Thresh~0.5' 295 | source_half = ColumnDataSource(data=dict(x_fpr=df_half.FPR, 296 | y_tpr=df_half.TPR, 297 | thresh=df_half.Thresholds, 298 | legend_=df_half.Legend)) 299 | p.circle('x_fpr', 'y_tpr', size=5, color="blue", source=source_half, legend="legend_", name='ROC') 300 | 301 | """ PLOT chance line """ 302 | # Plot chance (tpr = fpr) 303 | p.line([0, 1], [0, 1], line_dash='dashed', line_width=0.5, color='black', name='Chance') 304 | 305 | # Finishing touches 306 | p.legend.location = "bottom_right" 307 | 308 | """ save and show """ 309 | if filename: 310 | output_file(filename) 311 | show(p) 312 | 313 | return clf, df_ROC 314 | 315 | 316 | def plot_ROC_CV(clf, X, y, cv_fold=3, pos_label_=None, verbose=False): 317 | 318 | """ 319 | clf, classifiers, df_ROCs = plot_ROC_CV(clf, X, y, cv_fold=3, pos_label_=None, verbose=False) 320 | 321 | Plot an interactive ROC curve for a binary classifier with n='cv-fold' cross-validations. 322 | It returns the original clf, a classifier for each cv, a list of dataframes for each cv, 323 | and precision_info, which is a tuple of (precision, recall, avg_precision) of types (array, array, float). 324 | 325 | clf: untrained classifier object (e.g. rf_clf = RandomForestClassifer()) 326 | X: training + testing data 327 | y: targets (numeric/integers) 328 | cv_fold: cross-validations to run [default: 3] 329 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 330 | verbose: print warnings [default: False] 331 | """ 332 | 333 | """ Check cross-validations to run and get stratification """ 334 | # Check cross-validation > 1 and get stratified data 335 | if cv_fold > 1: 336 | skf = StratifiedKFold(cv_fold) 337 | else: 338 | print(f"cv_fold must be greater than 1. You have input {cv_fold}") 339 | return clf 340 | 341 | """ Get source data for each ROC curve """ 342 | # Loop over each split in the data and get source data, df, and clf 343 | source_ROCs, df_ROCs, classifiers = [], [], [] 344 | for idx, val in enumerate(skf.split(X, y)): 345 | (train, test) = val 346 | data = (X[train], X[test], y[train], y[test]) # not that skf returns indices, not values 347 | source_, df_, clf_ = get_ROC_data(data, clf, pos_label_, verbose) 348 | source_ROCs.append(source_) 349 | df_ROCs.append(df_) 350 | classifiers.append(clf) 351 | 352 | """ Set up initial PLOT """ 353 | # Create custom HoverTool -- we'll name each ROC curve 'ROC' so we only see info on hover there 354 | hover_ = HoverTool(names=['ROC'], tooltips=[("TPR", "@y_tpr"), ("FPR", "@x_fpr"), ("Threshold", "@thresh")]) 355 | 356 | # Create your toolbox 357 | p_tools = [hover_, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 358 | 359 | # Create figure and labels 360 | clf_name = get_clf_name(clf) 361 | p = figure(title=f'{clf_name} ROC curve with {cv_fold}-fold cross-validation', tools=p_tools) 362 | p.xaxis.axis_label = 'False Positive Rate' 363 | p.yaxis.axis_label = 'True Positive Rate' 364 | 365 | """ Get ROC CURVE for each iteration """ 366 | # Set the matplotlib colorwheel as a cycle 367 | colors_ = cycle(list(cnames.keys())) 368 | 369 | # plot each ROC curve - loop over source_ROCs, colors_ 370 | for _, val in enumerate(zip(source_ROCs, colors_)): 371 | (ROC, color_) = val 372 | p.line('x_fpr', 'y_tpr', line_width=1, color=color_, source=ROC) 373 | p.circle('x_fpr', 'y_tpr', size=10, color=color_, legend='auc_legend', source=ROC, name='ROC') 374 | 375 | """ Mean ROC and AUC for all curves and plot """ 376 | # process inputs 377 | mean_fpr, mean_tpr = interpolate_mean_tpr(df_list=df_ROCs) 378 | mean_auc = auc(mean_fpr, mean_tpr) 379 | mean_legend = [f'Mean, AUC: {mean_auc:.3f}']*len(mean_tpr) 380 | 381 | # Create ColumnDataSource 382 | source_ROC_mean = ColumnDataSource(data=dict(x_fpr=mean_fpr, 383 | y_tpr=mean_tpr, 384 | auc_legend=mean_legend)) 385 | 386 | # Plot mean ROC 387 | p.line('x_fpr', 'y_tpr', legend='auc_legend', color='black', 388 | line_width=3.33, line_alpha=0.33, line_dash='dashdot', source=source_ROC_mean, name='ROC') 389 | 390 | # Plot chance (tpr = fpr) 391 | p.line([0, 1], [0, 1], line_dash='dashed', line_width=0.5, color='black', name='Chance') 392 | 393 | # Finishing touches 394 | p.legend.location = "bottom_right" 395 | show(p) 396 | 397 | return clf, classifiers, df_ROCs 398 | 399 | 400 | def plot_ROC_clfs(classifiers, X, y, test_size=0.33, pos_label_=None, verbose=False): 401 | 402 | """ 403 | clf, classifiers, df_ROCs, precision_info = plot_ROC_clfs(clf, X, y,pos_label_=None, verbose=False) 404 | 405 | Plot an interactive ROC curve for a binary classifier with n='cv-fold' cross-validations. 406 | It returns the original clf, a classifier for each cv, and a list of dataframes for each cv. 407 | precision_info is a tuple of (precision, recall, avg_precision) of types (array, array, float). 408 | 409 | classifiers: list of untrained classifiers 410 | X: training + testing data 411 | y: targets (numeric/integers) 412 | test_size: test size for train_test_split (0 < x < 1) 413 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 414 | verbose: print warnings [default: False] 415 | """ 416 | 417 | """ Get source data for each ROC curve """ 418 | 419 | # Get training and test data 420 | (data_) = train_test_split(X, y, test_size=test_size) 421 | 422 | # Loop over each CLASSIFIER now -- note that we don't redefine our classifiers 423 | source_ROCs, df_ROCs = [], [] 424 | for _, clf_ in enumerate(classifiers): 425 | source_, df_, clf_ = get_ROC_data(data_, clf_, pos_label_, verbose) 426 | source_ROCs.append(source_) 427 | df_ROCs.append(df_) 428 | 429 | """ Set up initial PLOT """ 430 | 431 | # Create custom HoverTool -- we'll name each ROC curve 'ROC' so we only see info on hover there 432 | hover_ = HoverTool(names=['ROC'], tooltips=[("TPR", "@y_tpr"), ("FPR", "@x_fpr"), ("Threshold", "@thresh")]) 433 | 434 | # Create your toolbox 435 | p_tools = [hover_, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 436 | 437 | # Create figure and labels 438 | p = figure(title=f'Benchmarking {len(classifiers)} classifiers', tools=p_tools) 439 | p.xaxis.axis_label = 'False Positive Rate' 440 | p.yaxis.axis_label = 'True Positive Rate' 441 | 442 | """ Get ROC CURVE for each iteration """ 443 | 444 | # Set the matplotlib colorwheel as a cycle 445 | colors_ = cycle(list(cnames.keys())) 446 | 447 | # loop over source, color and plot each ROC curve 448 | for _, val in enumerate(zip(source_ROCs, colors_)): 449 | (ROC, color_) = val 450 | p.line('x_fpr', 'y_tpr', line_width=1, color=color_, source=ROC) 451 | p.circle('x_fpr', 'y_tpr', size=5, color=color_, legend='clf_legend', source=ROC, name='ROC') 452 | 453 | """ Mean ROC and AUC for all curves and plot """ 454 | 455 | # process mean values, legend, ColumnDataSource 456 | mean_fpr, mean_tpr = interpolate_mean_tpr(df_list=df_ROCs) 457 | mean_auc = auc(mean_fpr, mean_tpr) 458 | mean_legend = [f'Mean, AUC: {mean_auc:.3f}']*len(mean_tpr) 459 | source_ROC_mean = ColumnDataSource(data=dict(x_fpr = mean_fpr, y_tpr = mean_tpr, roc_legend=mean_legend)) 460 | 461 | # PLOT mean ROC 462 | p.line('x_fpr', 'y_tpr', legend='roc_legend', color='black', 463 | line_width=5, line_alpha=0.3, line_dash='dashed', source=source_ROC_mean, name='ROC') 464 | 465 | # PLOT chance (tpr = fpr) 466 | p.line([0, 1], [0, 1], line_dash='dashed', line_width=0.2, color='black', name='Chance') 467 | 468 | # Finishing touches 469 | p.legend.location = "bottom_right" 470 | show(p) 471 | 472 | # Print scores 473 | print("Scores:") 474 | # Get scores for each classifier: 475 | for i, df_ in enumerate(df_ROCs): 476 | print(df_.clf, np.round(df_.score, decimals=3)) 477 | 478 | return classifiers, df_ROCs 479 | 480 | 481 | def get_ROC_PR_data(data, clf, pos_label_=None, verbose=False): 482 | 483 | """ 484 | source, df, clf = get_ROC_PR_data(data, clf, verbose=False) 485 | 486 | get_ROC_data will return ColumnDataSource and dataframes with TPR and FPR 487 | for a particular dataset and an untrained classifier. The CSD can be used 488 | to plot a Bokeh plot while the dataframe can be used for additional 489 | exploration and plotting with other libs. Note that the dataframes 490 | are returned with metadata (e.g. AUC and the clf used). 491 | 492 | data: tuple of our data (X_train, X_test, y_train, y_test) 493 | where each item in the tuple is a numpy ndarray 494 | clf: an untrained classifier (e.g. rf = RandomForestClassifier()) 495 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 496 | verbose: print warnings [default: False] 497 | """ 498 | 499 | # split data into training, testing 500 | (X_train, X_test, y_train, y_test) = data 501 | 502 | # train and retrieve probabilities of class per feature for the test data 503 | probas = clf.fit(X_train, y_train).predict_proba(X_test) 504 | 505 | # get false and true positive rates for positive labels 506 | # (and thresholds, which is not used but shown here for fyi) 507 | if not pos_label_: 508 | pos_label_ = np.max(y_train) 509 | if verbose: 510 | print(f"Warning: Maximum target value of '{pos_label_}' used as positive.") 511 | print("You can use 'pos_label_' to indicate your own.") 512 | 513 | """ ROC """ 514 | fpr, tpr, roc_thresholds = roc_curve(y_test, probas[:,1], pos_label=pos_label_) 515 | roc_thresholds[0] = np.nan 516 | 517 | # get area under the curve (AUC) 518 | roc_auc = auc(fpr, tpr) 519 | 520 | """ PR """ 521 | # get precision and recall values 522 | precision, recall, pr_thresholds = precision_recall_curve(y_test, probas[:,1], pos_label=pos_label_) 523 | pr_thresholds = np.insert(pr_thresholds, 0, 0) # do this to correct lengths 524 | 525 | # average precision score 526 | avg_precision = average_precision_score(y_test, probas[:,1]) 527 | 528 | # precision auc 529 | pr_auc = auc(recall, precision) 530 | 531 | 532 | """ Create Sources """ 533 | # create legend variables - we'll create an array with len(tpr) 534 | roc_auc_ = [f"AUC: {roc_auc:.3f}"]*len(tpr) 535 | pr_auc_ = [f"AUC: {pr_auc:.3f}"]*len(precision) 536 | clf_name = get_clf_name(clf) 537 | clf_roc = [f"{clf_name}, AUC: {roc_auc:.3f}"]*len(tpr) 538 | clf_pr = [f"{clf_name}, AUC: {pr_auc:.3f}"]*len(precision) 539 | 540 | # create bokeh column source for plotting new ROC 541 | source_ROC = ColumnDataSource(data=dict(x_fpr=fpr, 542 | y_tpr=tpr, 543 | thresh_roc=roc_thresholds, 544 | auc_legend=roc_auc_, 545 | clf_legend=clf_roc)) 546 | 547 | source_PR = ColumnDataSource(data=dict(x_rec=recall, 548 | y_prec=precision, 549 | thresh_pr=pr_thresholds, 550 | auc_legend=pr_auc_, 551 | clf_legend=clf_pr)) 552 | 553 | """ Dataframes """ 554 | # create output dataframe with TPR and FPR, and metadata 555 | df_ROC = pd.DataFrame({'TPR':tpr, 'FPR':fpr, 'Thresholds':roc_thresholds}) 556 | df_ROC.auc = roc_auc 557 | df_ROC.clf = get_clf_name(clf) 558 | df_ROC.score = clf.score(X_test, y_test) 559 | 560 | # create output dataframe with TPR and FPR, and metadata 561 | df_PR = pd.DataFrame({'Recall':recall, 'Precision':precision, 'Thresholds':pr_thresholds}) 562 | df_PR.auc = pr_auc 563 | df_PR.clf = get_clf_name(clf) 564 | df_PR.score = clf.score(X_test, y_test) 565 | 566 | return source_ROC, source_PR, df_ROC, df_PR, clf 567 | 568 | 569 | def plot_ROC_PR(clf, X, y, test_size_:float=0.5, pos_label_:str=None, filename:str=None, verbose:bool=False): 570 | """ 571 | clf, classifiers, df_ROCs = plot_ROC(clf, X, y, pos_label_=None, verbose=False) 572 | 573 | Plot an interactive ROC curve for a binary classifier. 574 | It returns the original clf, a classifier for each cv, a list of dataframes for each cv. 575 | 576 | clf: untrained classifier object (e.g. rf_clf = RandomForestClassifer()) 577 | X: training + testing data 578 | y: targets (numeric/integers) 579 | test_size: fraction of data to be reserved for testing [default: 0.5] 580 | pos_label_: if targets are not binary (0, 1) then indicate integer for "positive" [default: None] 581 | filename: if provided, save to html [default: None] 582 | verbose: print warnings [default: False] 583 | """ 584 | 585 | """ Split and get ROC curve data """ 586 | data = train_test_split(X, y, test_size=test_size_) 587 | source_ROC, source_PR, df_ROC, df_PR, clf = get_ROC_PR_data(data, clf, pos_label_, verbose) 588 | 589 | 590 | """ PLOT ROC """ 591 | 592 | # Create custom HoverTool -- we'll make one for each curve 593 | hover_ROC = HoverTool(names=['ROC'], tooltips=[("TPR", "@y_tpr"), 594 | ("FPR", "@x_fpr"), 595 | ("Thresh", "@thresh_roc"), 596 | ]) 597 | 598 | # Create your toolbox 599 | p_tools_ROC = [hover_ROC, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 600 | 601 | clf_name = get_clf_name(clf) 602 | p1 = figure(title=f'{clf_name} ROC curve', tools=p_tools_ROC) 603 | p1.xaxis.axis_label = 'False Positive Rate' 604 | p1.yaxis.axis_label = 'True Positive Rate' 605 | 606 | # plot curve and datapts 607 | p1.line('x_fpr', 'y_tpr', line_width=1, color="blue", source=source_ROC) 608 | p1.circle('x_fpr', 'y_tpr', size=3, color="orange", legend='auc_legend', source=source_ROC, name='ROC') 609 | 610 | # highlight values closest to threshold == 0.5 611 | df_half = df_ROC.dropna().iloc[(df_ROC['Thresholds'].dropna()-0.5).abs().argsort()[:2]] 612 | df_half['Legend'] = 'Thresh~0.5' 613 | source_half = ColumnDataSource(data=dict(x_fpr=df_half.FPR, 614 | y_tpr=df_half.TPR, 615 | thresh_roc=df_half.Thresholds, 616 | legend_=df_half.Legend)) 617 | p1.circle('x_fpr', 'y_tpr', size=5, color="blue", source=source_half, legend="legend_", name='ROC') 618 | 619 | # Plot chance (tpr = fpr) 620 | p1.line([0, 1], [0, 1], line_dash='dashed', line_width=0.5, color='black', name='Chance') 621 | 622 | # Finishing touches 623 | p1.legend.location = "bottom_right" 624 | 625 | """ PLOT PR """ 626 | 627 | # Create custom HoverTool -- we'll make one for each curve 628 | hover_PR = HoverTool(names=['PR'], tooltips=[("Precision", "@y_prec"), 629 | ("Recall", "@x_rec"), 630 | ("Thresh", "@thresh_pr") 631 | ]) 632 | 633 | # Create your toolbox 634 | p_tools_PR = [hover_PR, 'crosshair', 'zoom_in', 'zoom_out', 'save', 'reset', 'tap', 'box_zoom'] 635 | 636 | p2 = figure(title=f'{clf_name} PR curve', tools=p_tools_PR) 637 | p2.xaxis.axis_label = 'Recall' 638 | p2.yaxis.axis_label = 'Precision' 639 | 640 | p2.line('x_rec', 'y_prec', line_width=1, color="blue", source=source_PR) 641 | p2.circle('x_rec', 'y_prec', size=3, color="orange", legend='auc_legend', source=source_PR, name='PR') 642 | 643 | # highlight values closest to threshold == 0.5 644 | df_half = df_PR.dropna().iloc[(df_PR['Thresholds'].dropna()-0.5).abs().argsort()[:2]] 645 | df_half['Legend'] = 'Thresh~0.5' 646 | source_half = ColumnDataSource(data=dict(x_rec=df_half.Recall, 647 | y_prec=df_half.Precision, 648 | thresh_pr=df_half.Thresholds, 649 | legend_=df_half.Legend)) 650 | p2.circle('x_rec', 'y_prec', size=5, color="blue", source=source_half, legend="legend_", name='PR') 651 | 652 | # Plot chance (prec = rec) 653 | p2.line([0, 1], [1, 0], line_dash='dashed', line_width=0.5, color='black', name='Chance') 654 | 655 | # Finishing touches 656 | p2.legend.location = "bottom_left" 657 | 658 | """ save and show """ 659 | if filename: 660 | output_file(filename) 661 | show(row(p1, p2)) 662 | 663 | return clf, df_ROC 664 | 665 | 666 | if __name__ == '__main__': 667 | 668 | # get UCI Breast Cancer Data 669 | data = datasets.load_breast_cancer() 670 | X = data.data 671 | y = data.target 672 | target_names = list(data.target_names) 673 | 674 | # Classifier plots 675 | X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5) 676 | rf_clf = RandomForestClassifier(n_estimators=100) 677 | rf_clf.fit(X_train, y_train) 678 | classifier_plots(rf_clf, X_test, y_test, target_names, ylog=False) 679 | 680 | # ROC curve for synthetic data 681 | rf_clf = RandomForestClassifier(n_estimators=100) 682 | rf_clf, df_rf_ROC = plot_ROC(rf_clf, X, y, test_size_=0.5) 683 | 684 | # Interactive ROC curve with cross-validation 685 | rf_clf = RandomForestClassifier(n_estimators=100) 686 | rf_clf, rf_cv_clfs, df_rf_ROCs = plot_ROC_CV(rf_clf, X, y) 687 | 688 | # Benchmark classifiers 689 | rf_bench = RandomForestClassifier(random_state=42, n_estimators=100) 690 | lr_bench = LogisticRegression(random_state=42, solver='saga') 691 | gnb_bench = GaussianNB(priors=None, var_smoothing=1e-06) 692 | clfs_benchmark, dfs_bench = plot_ROC_clfs(classifiers=[rf_bench, lr_bench, gnb_bench], X=X, y=y) 693 | 694 | # Interactive ROC and PR curves 695 | rf_clf = RandomForestClassifier(n_estimators=100) 696 | rf_clf, df_rf_ROC_PR = plot_ROC_PR(rf_clf, X, y, test_size_=0.33) 697 | -------------------------------------------------------------------------------- /plots/html/roc_cvfold/bokeh-1.0.4.min.css: -------------------------------------------------------------------------------- 1 | .bk-root{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:10pt;position:relative;width:auto;height:auto}.bk-root .bk-shading{position:absolute;display:block;border:1px dashed green}.bk-root .bk-tile-attribution a{color:black}.bk-root .bk-tool-icon-box-select{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg0kduFrowAAAIdJREFUWMPtVtEKwCAI9KL//4e9DPZ3+wP3KgOjNZouFYI4C8q7s7DtB1lGIeMoRMRinCLXg/ML3EcFqpjjloOyZxRntxpwQ8HsgHYARKFAtSFrCg3TCdMFCE1BuuALEXJLjC4qENsFVXCESZw38/kWLOkC/K4PcOc/Hj03WkoDT3EaWW9egQul6CUbq90JTwAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-box-zoom{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg82t254aQAAAkBJREFUWMPN11+E1FEUB/DPTFn2qaeIpcSwr5NlUyJiKWVXWUqvlUh/iE3RY9mUekkPPURtLKNRrFJEeuphGfUUaVliiX1aVjGs6aG7+XX9ZnZ+d2fTl2vmnHvPPfeee/79Sk+may2/UQq/q7Qu+bAJoxjHIKqB/wlfUMcMVqI9bLZ+DGIKwzlzQ2GcxCx2xwvKOUKlaHTiX8bHNspjDONHkOmJBW5jIof/FvPh/06MZOb6cRc7cGn1AKUE5cdzlM/gAr5F/O24H3xkFRfxAbVygvK+cIsspjGWo1zgjeFpxL+BvnLw7laBA4xjIFJwrgu52DoVjKdY4HBEX8dSF3JLYe1fe6UcYCii3xWQjdfuSTnAtoheKCC7GNED5Zx4L4qt61jbTLHA94geKSC7P7ZeShQ0Inoi1IJuEOeORooFXkV0FZNdZs5qvFfKAeqYy7nZ6yg//HG0MBfffh71lFrQDCW2EvEP4mt4okZUDftz9rmGZkotmMxJRtlisy+MTniAWrty3AlXw0hFM2TD89l+oNsoOJXjbIs4EpqNtTCLXbiZ0g+M4mFObj8U3vsNjoZCVcmk60ZwthpepLZkB/AsivWfOJZxtpUQHfWib7KWDwzjeegBZJSdKFiE2qJTFFTwElsi/unQ/awXrU4WGMD7nOJxBY/1EO2iYConq93CHT1GOwucjdqnRyFz+VcHmMNefMY9nNkA3SWUOoXhQviSWQ4huLIRFlirFixnQq/XaKXUgg2xQNGv4V7x/RcW+AXPB3h7H1PaiQAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-zoom-in{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEgsUBmL8iQAAA2JJREFUWMO9l12IlFUYx3//MzPrLpSjkm5oN4FFIWVEl66IQlFYwtLOzozsjHdGRSCRF0sfBEXRVV0FQuQiLm5CZNBFgRRaRLVFhbJ2EdiN5gbK7toObTPn6eYdPTvNzPvOBz5Xh/ec5/n/n89zXtEHmZqeSXSuXBz/3zfdKvBWJHQrwZuRcP0El+QkbQXeBX6WZEgm6TtJk5lM5o4Lc+cV6qpf4Ga20Tm338zeATItVK9Ker6yvPzp4NDQ3+XieGsCU9MzTYumGbhz7m4ze9/MHgvBgItACrgfGAj2jgAvAYs3wlEujjc13kii8YyZrXXOfWhmo9GnFUlvOOemarVapVqtkslksmb2KjARqL62ecuWN9NxbRInzrldAXhV0uFSIfdew7G/gNLU9MwS8CwSmE3Oz88fcXG5blfpqVRq0Ix8VIAAX0XgrVL7HDCHGcCaWrV60LUBN8Dae58aQIxEqcA592I9M610JL0cpG/U9TIHJNKY3RV5z0R+7Nd4HZ0P1g/2RMBuegLAsRMnb4vT8d5vqKfMzOgtAlADrkmqGywmiMBTwfr3dC9j1Xv/r6Tvg/5/5ejxE6cO7M9faVbQZrYNOFSPmqQvVo9FKexvi5uWX58943aM7DwAfBDY+FbSCxP5sdkGx55GeguzrUEXPaSo2pFkAbiSZQCAzZJOmdkjwd6SpB/M7KykQTPbA2wDhoIzRzcNDx9MJwGNIXdJ0mEzmwbujL7dbma7gd03A7lKfnTOvf74nl0r6bonTUbujRSUCrm2d4L3/kvn3JPe+8+BDW2i9o+kT7z3kxP5sYsA6W47oE64TsR7P9tQL4vA2mh9WdIscKxUyJ0M7aR7acOGzikD65EQLEjaa2ZXzMwDFeB6qZBbbLTRE4EGeSaozNOZgYFf8qP7lmIvs354n0qlHpB0T7B9Ogl4IgJJrmjv/SiQjbrkD+BMUkfSbYATPdckrTOzkciWAXOlQu5cYgLdPEIapud9wMOR9zVJH3ViKx333mtHMJvNuoWFhZ3A+ojMcja77njXBEKwJJfTcqUyCIQ34Mf7nnh0paMnXacFuGoC1mr3AtuDfLzd8Zuyl+rfuGn4HLAD+Az4qZQf+61TAj0Noj8vX6oC35SL43u7teG6rf5+iXppwW7/JUL5D03qaFRvvUe+AAAAAElFTkSuQmCC")}.bk-root .bk-tool-icon-zoom-out{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEgsHgty9VwAAA0FJREFUWMO9l09oXFUUxn/fmXlpItppi22k7UJBRSlVkCytSAuKUloIdjKT0El3FXVXdVFKRVAQV7qQohsNwdA0UFvBhYtqUVyIVlRaogtFQVq7qSTVjA3z3nHzBq/jvPmTN/Ss7rv3nvN99/y794kByMzcfE/7picn/jenmwWeRUI3E7wdCRskuCSTdDfwBvCtJEdySV9KOhpF0e0/LF5SqKtBgbv7ZjObcvfXgShD9Zqk5+orKx8Oj4z8NT05kU1gZm6+bdK0Azezu9z9hLs/HoIBvwAF4H5gKFh7B3gBWFY3460kWve4+3oze9fdx9OpVUmvmNlMHMf1RqNBFEUldz8OHAxUX9q6bduryut+Sfvc/Wz62ZD0fK1afjND9y3gGSRwv1GMojstTxUUCoVhdyopEYDzKXjWwZ4FFnEHWBc3Goet00m7lZlZYQixKw0FZnakGZksHUnHgvCN5/KARBH37enpOVg58H13HV0Kxg/kIuD/ngSA2ZMLt3bTSZJkUzNk7k4+D0AM/CGpaXCyBw/sC8Y/qZd2GpZiuL9YLN4Sx/HpoP5/c/exQ1OVq+1yyt13SLoArEsJnMjlgfOffvK3u58Kprab2QezJxfG2iTzUzI70wRPG9jbmpmb95SNB9mpzp7/j2yVdNbdx4K565K+cvfPJQ27+x5gBzAS7Hlvy+jo4WIvoC3kWpcvS3rR3eeAO9K529x9N7C7zX6AC2b28hN7Hl1Vt44niVq13LUjmtlYkiQfA5s6eO+GpDNJkhw9NFX5ueNt2ARodyF1IHIN2JiOl4H16fiKpK+B2Vq1vBAqFAf4IJkGNiIhWJK0192vunsC1IE/a9XycquNXARa5OnApeeioaHvKuP7r3dTGsiLqFAo7JR0T7B8rhfwXARa2us4UEqr5Ffgs151i/08oTNKdIO770ptObBYq5Yv5ibQq/sl3Qc8lJ4+lnSqH1vFfp9koZRKJVtaWnqkWXqSVkqlDe+vmUDWpZMlK/X6MBDegKf3P/nYaj8ErN9fqZBYEsf3Ag8G8Xit33BaniTcvGX0IvAw8BHwTa1y4Md+CeRqRL9fudwAvpienNi7Vhu21uwflOT+L+i1X2TJP57iUvUFtHWsAAAAAElFTkSuQmCC")}.bk-root .bk-tool-icon-help{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAAXNSR0IArs4c6QAAAAlwSFlzAAALEwAACxMBAJqcGAAABltpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIgogICAgICAgICAgICB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIKICAgICAgICAgICAgeG1sbnM6c3RSZWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiCiAgICAgICAgICAgIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIKICAgICAgICAgICAgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIj4KICAgICAgICAgPHRpZmY6UmVzb2x1dGlvblVuaXQ+MjwvdGlmZjpSZXNvbHV0aW9uVW5pdD4KICAgICAgICAgPHRpZmY6Q29tcHJlc3Npb24+NTwvdGlmZjpDb21wcmVzc2lvbj4KICAgICAgICAgPHRpZmY6WFJlc29sdXRpb24+NzI8L3RpZmY6WFJlc29sdXRpb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOllSZXNvbHV0aW9uPjcyPC90aWZmOllSZXNvbHV0aW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFlEaW1lbnNpb24+MzI8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpQaXhlbFhEaW1lbnNpb24+MzI8L2V4aWY6UGl4ZWxYRGltZW5zaW9uPgogICAgICAgICA8ZXhpZjpDb2xvclNwYWNlPjE8L2V4aWY6Q29sb3JTcGFjZT4KICAgICAgICAgPHhtcE1NOkluc3RhbmNlSUQ+eG1wLmlpZDpBODVDNDBDMzIwQjMxMUU0ODREQUYzNzM5QTM2MjBCRTwveG1wTU06SW5zdGFuY2VJRD4KICAgICAgICAgPHhtcE1NOkRvY3VtZW50SUQ+eG1wLmRpZDpBODVDNDBDNDIwQjMxMUU0ODREQUYzNzM5QTM2MjBCRTwveG1wTU06RG9jdW1lbnRJRD4KICAgICAgICAgPHhtcE1NOkRlcml2ZWRGcm9tIHJkZjpwYXJzZVR5cGU9IlJlc291cmNlIj4KICAgICAgICAgICAgPHN0UmVmOmluc3RhbmNlSUQ+eG1wLmlpZDpBODVDNDBDMTIwQjMxMUU0ODREQUYzNzM5QTM2MjBCRTwvc3RSZWY6aW5zdGFuY2VJRD4KICAgICAgICAgICAgPHN0UmVmOmRvY3VtZW50SUQ+eG1wLmRpZDpBODVDNDBDMjIwQjMxMUU0ODREQUYzNzM5QTM2MjBCRTwvc3RSZWY6ZG9jdW1lbnRJRD4KICAgICAgICAgPC94bXBNTTpEZXJpdmVkRnJvbT4KICAgICAgICAgPGRjOnN1YmplY3Q+CiAgICAgICAgICAgIDxyZGY6U2VxLz4KICAgICAgICAgPC9kYzpzdWJqZWN0PgogICAgICAgICA8eG1wOk1vZGlmeURhdGU+MjAxNjoxMToyOCAxMToxMTo4MjwveG1wOk1vZGlmeURhdGU+CiAgICAgICAgIDx4bXA6Q3JlYXRvclRvb2w+UGl4ZWxtYXRvciAzLjY8L3htcDpDcmVhdG9yVG9vbD4KICAgICAgPC9yZGY6RGVzY3JpcHRpb24+CiAgIDwvcmRmOlJERj4KPC94OnhtcG1ldGE+Cphjt2AAAAT7SURBVFgJxRdbaFxFdGb2bhui227BWrsVKYgf2kJUbP9EUPuzEB803WTXJjH61Q/7Ya1+CMYKEVTsh4J/EpvY7BoabUiNiA8s1p+4KIhpoUUEselHqyS76TbZ3HuP58ydc3d2u4+IkQxczpz3mZkzZ86VYpXjvenpjZsLhUcliE4AuUuASAgptmt1EFdwPiclzIIUUwubNn17OJlcXo1p2UpodHRiux9xB1Eug1+slbzhFxGOKc851tu7/0oznYYBDA8Pt0U2tL8KQryIq2tvZqQhD0QJHRz3yqWhgYGBpXpydQMwqz6NCnurleCSADkJEfgKfOePqL80R/wV1ZaQyr1LenKfkPCkEPKeaj0xg7vxVL3duCmA0Vyuw/fl52hgBxsBED+h4Cv9z3R/zbRm8MTJTx7HQN7GQB6w5C4L4SX7M5lfLBpurjXMyvNIShiyi0l1pL8n9b7EDGPR8fHxzSsQ6XDB3618/xqo6Pk25V5MpVJllgHM1BO58RdQ612kOYZ+GXdij70TYQB05mpj+1kU5G2fB+l3PZtOf8NGx6ambnMXb3yAxg8wjSEG6OKKR9oicBQD+ZvpH2Wzj0lQpxCPG9qMv1x6hHNCsSAlHM7ZOa682vlI9tRDbvHGbD3nZAPpDoD/3JIrLpAs26UFkC3EMUA99hpfGtEBfJjNJnS2Gwnadnvl+Xw+iuc3DAJuNyIaSCHpilVldyDjjUxj3WDZIAhxhHHyRcdNuA7AAfUaXzVKODpzFiZ4/uLvh5G+m2no+C/pyIf7MqlEJB7bpqR6nXkEUfbeawuLaZsW2ISfNQ2vtaktQlGFQyIVGT0o2+2EC4iQNGwjBIN9qdQ5Qg4mk4X4rW3vCClLtowE2FOFUxKDfNmiZci3ovKKRFPh4FK9q4Zbdr+lKKJiA13TcHR2dmLBgdmQ0GAS2MZaEowY+XbAk09IvgtYZGp16SyvFhaHcIUh645t8T9DBCcnz5zZ4hZLu3DzK2QlL1QQa0Y+pHiJKPSuOGj3PmZTheM5w2TwqBxnvBZOTk7G5gvXJ5Aelms8wnJURL+olSWcfEhf6gDoUXPMq6ZlqbzWU2pE+3hi4s6F68tfIj9cBMlikr7Z0/P0b/X0yIcUXsDCF1WhtL4OROHaXk+xlkbV0Cu732Nmhc4peaWSg73pA8dq5RkvO37ldUTfXCKZv2q45MkhvG87WQEzpCCUSvV1d9GONBy3lMvgKSwrZig8gjAietWY0QriylO2jIo4yVbOSb7KB/qmI9BPKjHpSSXYauRyn92Nq9/Kcrj13x3s3v8D481glQ/0raiNYgX9njPSBOImbrHZePl+tfFmc9sH+Xaoh8NjOKSVdDMhjjYzQLy+dFceH5+IJQf9VYXX4tROg4ZFU8m31M3mfPEqUoJqCGJfvWpo2xnNfdrhC28n06SCeSzNZxlvBINGRXCtKS7EY1uV6V7HWAm38y1cXaXsMcOCvr9ySPj+af7A1U2HJXHzVNvUXVLIGyPf+jV0pf8GHoN+TLAyPkidTCi2RpPApmnR0Bd1zGRaB/B8Oj2HSw7LLbVR1MmskW8RdEWVXSJf3JbpAMgRtc4IZoxTh9qotQjCasm46M0YX9pV1VmbpvRH5OwwgdRtSg2vKaAz/1dNKVtb17Y8DCL4HVufHxMOYl1/zTgIgiYvBnFKfaNp3YjTdPz3n9Na8//X7/k/O1tdwopcZlcAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-hover{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4oVHp0SwAAAQJJREFUWMPtlsENgzAMRb8RQ5VJItFDOgaZAMaAA0iZpN3KPZSoEEHSQBCViI/G8pfNt/KAFFcPshPdoAGgZkYVVYjQAFCyFLN8tlAbXRwAxp61nc9XCkGERpZCxRDvBl0zoxp7K98GAACxxH29srNNmPsK2l7zHoHHXZDr+/9vwDfB3kgeSB5IHkgeOH0DmesJjSXi6pUvkYt5u9teVy6aWREDM0D0BRvmGRV5N6DsQkMzI64FidtI5t3AOKWaFhuioY8dlYf9TO1PREUh/9HVeAqzIThHgWZ6MuNmC1jiL1mK4pAzlKUojEmNsxcmL0J60tazWjLZFpClPbd9BMJfL95145YajN5RHQAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-crosshair{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADEUlEQVRYR81XXVIaQRCeHqug8CXmBNETaE4gniDwIgpVspxAbxC9ATkBkCpQ8gKeQDiB5AQxNyAvUlrldr7eHxyGXZi1rMJ5opbp7m++7un+htSGF204vsoMoNXrlzSpfWa1oxQfhAegCZGaEtPorHo8znIoJwCt6+td8uk7ApUQCIHTF4BNAWzImq8ap6cP68CsBdDp9i9ZqXM7ML79g/EnCWD+jgMKENKqWT+tXK0CkQqgNRjs0OxpQIqKhoMxaG6/6JeRnK7T6yO2UvVqhYSlLX+ryORfgKn9ORDFIy7ky41yGcwsr0QAQfDH5zucOswx819fs4egI9OFCcD8DjBF7VNbEX0JzdWEt3NHSSASAcCxBDqMgt/623kvyTgNgNjJIfTjk4D4FqaJR1715MjmYAmA5Bx3AwUXQL+t105KaTlcBSC26XRvhjEIoLiq1yqXpr8FAGG16/ug4IT27fxBWu7EiQuAiImJpEMKE6nYM30uAIDDttSUOPfJP7JzbjPhAiBIh9QE67vIvoOi9WJfCwDavf40ulpjbCqmUf+W753ezURuh7Dg1SqflwAEHU6pgfyBq9Y4qx0LG++2fnZ/eUzcstmdM2AWH+jfc+liWdBJfSENf8Lifi3GVwC9mybOfi5dzatWVrbbLIHNva8p5h/16gkaFiLGGxbufkoE6XguwePiXLF3XmMfCUCUAqtKXU7sumd1CowOuJEi3Pg1FBpjitIGhyvVSfvmjci6ZR+rFQfDiPVE2jFYeICQ+PoewwjC5h7CZld6DBdyu6nDSKgzOyIMhmhK5TTqXYbRorZYM46TmpKAAOrGWwSJJekSB1yqJNOzp1Gs7YJ0EDeySDIMtJbQHh6Kf/uFfNFZkolJICRmz0P8DKWZuIG2g1hpok+Mk0Qphs0h9lzMtWRoNvYLuVImUWrmPJDlBKeRBDfATGOpHkhw670QSHWGLLckmF1PTsMlYqMJpyUbiO0weiMMceqLVTcotnMCYAYJJbcuQrVgZFP0NOOJYpr62pf3AmrHfWUG4O7abefGAfwH7EXSMJafOlYAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-lasso-select{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEgwlGP1qdAAABMBJREFUWMO9V1uIVVUY/r61z57ZMx4DnbzgkbQXL5iCJphlWdpIGY4jpFBkEiU9ZNaDRRcITcIwMwgxoQtU2IMXdAZfMjFvpERXYiSbysyBEXFmyuHMnLP32uvrwT2xnY5nxvHQ93Jg7fWv71/r//7L4a59TRgqJk+Z6v3a+sv0OI5nk5wu6VaSVZImAThHsgjgrKTvM5nMUWvtmf5n8HodCIKgOgzDhc65pSTrJQWDsSNpJX1ljHnDOfdT37oZLLHv+8OMMasKhcIJ59xHAJYMlhwAJGUAzJfUTHLFuFzOG5QDU6dNMyQfs9Yedc5tBpAD4IYYNQGoBrDtQnt7/b0LFrJsCHzfn2itfQfAnZLiazytA3AaQAuAiwDaEgeNpGkkswAWSBqRONB38b88z5uTKePt6iiKXkk8jq+iJC5LOmiMaTLGHLPWhmWeHr7vV0dRtATAapAzIVmSo51zyzIlbm2stesFPA6pKk0r6Ryg93y/ek8YFvPOOTg3cDSiKCoC2OP7/rEoirYm4rUkF12lAWNM1lr7lqQn0+QA8gI2jBg5cj6Aj8OwmB+KAKIoukhyp6SRJAUgl0ndPLDWPi9pJQCbuviXvu+/GIZhW1dnJ24UJFuTjCCA2ADA8sYGWmsXS3qmL94kDYAtkh4Nw7ANlQJ5U6INT1KrAYC9zQdykl7nFSj5fXp5Y8NWVBhy7mUAjqShMYdMXV2dJ2klyRwAJ8lIeuGWCRMP7N7frEqSG2OmAFhKshNAp5wrmO7u7jEAngPQm1S2z2pqapr+OPt7XEly0oxwzq2RdFmSD2AMgKKJouhhAL4kA+Cs53l7e3t7uytJHgRBreTWkXwkKVJnJD0B4GAGwIJE9R6AFufc6UqSZ7PZbD6ff5dkA4CQZEHSqwAOISmXtwGIE+F1SeqqIP8d+Xz+C0mLJYWSAODteXffczjdDQNJ0BWMCoLg5gqIbRTJNwHsljQhUb0luWPM2LE7Thw/9m/5NCT/TByxAOYWi8X6/gdWV1dnfN8fNRBxJpMZTXKdc+6IpFVJWAEgkvSJpA0X2tvtVTaSjgOYBCAEEADYSHK87/sfhmEYA9gShuEDkgzJHyWtB/B1irQ2juP7ADxkrX0wOUOpzmdpzEY590HJ7Ni1r2kSyZOSiv2+hSRjSTXp/QAukzySNJOJkmalyNIl10hqMcasdc61XDNcQRD8BnITgNp+36r6kfcNFMMlLQGwTNLMEuQGQBfJl2bdPru+HDkAZAqFQux53jZHEsC6aw0eg2gylNRBcqcx5v04ji999+03AwsWAOI4Lsy9a94WkisAnE5a5WCJYwCfA1g7LJudI2lTHMeXBm1faiQzxkyRtF3S5CTupeAB+KG2tnZFT0/P30NO2VKLzrmfAbwGMipjG5Oc0dPTc0Md05SZ5U4Q2FxChErtEYD7jTGNQ3UgM8Asv90Yc9I5LSKRlXSI5CxJa0jWSALJjKRnAewfkniT+vwf7N7fXHK9rq7O7+jo+BTA/NRrdBpjnnLOnUrvXd7YMPQXSBunneno6IhIHgYwW1JtkgmBpBkATlVMAwOk3nFJ+VSoqgCMr6gIy2FcLtdKspAedyQN/98caDt/3kpyabUmf8WvG/8A1vODTBVE/0MAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-pan{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4lKssI9gAAAOtJREFUWMPVll0KwyAMgNPgoc0JzDX2Mtgp3csKErSamGabIEUo/T6bHz0ezxdsjPJ5kvUDaROem7VJAp3gufkbtwtI+JYEOsHNEugIN0mgM1wtsVoF1MnyKtZHZBW4DVxoMh6jaAW0MTfnBAbALyUwCD6UwEB4VyJN4FXx4aqUAACgFLjzrsRP9AECAP4Cm88QtJeJrGivdeNdPpko+j1H7XzUB+6WYHmo4eDk4wj41XFMEfBZGXpK0F/eB+QhVcXslVo7i6eANjF5NYSojCN7wi05MJNgbfKiMaPZA75TBVKCrWWbnGrb3DPePZ9Bcbe/QecAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-xpan{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4X4hxZdgAAAMpJREFUWMPtlsEKwjAMhr/pwOOedINJe/PobWXCfAIvgo/nA4heOiilZQqN2yE5lpD/I38SWt3uD9aMHSuHAiiAAmwaYCqoM/0KMABtQYDW11wEaHyiEei28bWb8LGOkk5C4iEEgE11YBQWDyHGuAMD0CeS30IQPfACbC3o+Vd2bOIOWMCtoO1mC+ap3CfmoCokFs/SZd6E0ILjnzrhvFbyEJ2FIZzXyB6iZ3AkjITn8WOdSbbAoaD4NSW+tIZdQYBOPyQKoAAKkIsPv0se4A/1UC0AAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-ypan{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4anK0lywAAAMVJREFUWMPtlzEKwzAMRX/S7rlpIMXeOnaLaME36FLo8XqCdNFghGljyc4kgQi2Q/SUj0F/eL7eMMTKz6j9wNlYPGRrFcSoLH4XxQPvdQeYuPOlcLbw2dRTgqvoXEaolWM0aP4LYm0NkHYWzyFSSwlmzjw2sR6OvAXNwgEcwAEcwAEcwAEcoGYk20SiMCHlmVoCzACoojEqjHBmCeJOCOo1lgPA7Q8E8TvdjMmHuzsV3NFD4w+1t+Ai/gTx3qHuOFqdMQB8ASMwJX0IEHOeAAAAAElFTkSuQmCC")}.bk-root .bk-tool-icon-range{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAAXNSR0IArs4c6QAAAAlwSFlzAAALEwAACxMBAJqcGAAABCJpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IlhNUCBDb3JlIDUuNC4wIj4KICAgPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICAgICAgPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIKICAgICAgICAgICAgeG1sbnM6dGlmZj0iaHR0cDovL25zLmFkb2JlLmNvbS90aWZmLzEuMC8iCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIgogICAgICAgICAgICB4bWxuczpkYz0iaHR0cDovL3B1cmwub3JnL2RjL2VsZW1lbnRzLzEuMS8iCiAgICAgICAgICAgIHhtbG5zOnhtcD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLyI+CiAgICAgICAgIDx0aWZmOlJlc29sdXRpb25Vbml0PjI8L3RpZmY6UmVzb2x1dGlvblVuaXQ+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjU8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOlhSZXNvbHV0aW9uPjcyPC90aWZmOlhSZXNvbHV0aW9uPgogICAgICAgICA8dGlmZjpPcmllbnRhdGlvbj4xPC90aWZmOk9yaWVudGF0aW9uPgogICAgICAgICA8dGlmZjpZUmVzb2x1dGlvbj43MjwvdGlmZjpZUmVzb2x1dGlvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjMyPC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6Q29sb3JTcGFjZT4xPC9leGlmOkNvbG9yU3BhY2U+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4zMjwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgICAgIDxkYzpzdWJqZWN0PgogICAgICAgICAgICA8cmRmOkJhZy8+CiAgICAgICAgIDwvZGM6c3ViamVjdD4KICAgICAgICAgPHhtcDpNb2RpZnlEYXRlPjIwMTgtMDQtMjhUMTQ6MDQ6NDk8L3htcDpNb2RpZnlEYXRlPgogICAgICAgICA8eG1wOkNyZWF0b3JUb29sPlBpeGVsbWF0b3IgMy43PC94bXA6Q3JlYXRvclRvb2w+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgrsrWBhAAAD60lEQVRYCcVWv2scRxSemZ097SHbSeWkcYwwclDhzr1Q5T6QE1LghP6BGNIYJGRWNlaZItiFK1mr+JAu4HQu0kjpU8sgF3ITAsaFg0hOvt2Zyfvmdsa7a610Unx44Zgf773vvfneezPHNzrbhn3CT3xC3wPXYOC8LDzqdi8YY/gwh4BeknS/2th6dr2kf94AOp3OFyWgMyziOPbMDxV9FTtJnl1ut795Xd0/YQ0/vtYQwMT1KXWCfr2IjOWwtNehwN4xL9ykTrm6Pzl58yLn3J+mKh9mXbT3uRjGEDph+O8/TjfP5dBp7Ha7AX7O3o5nZeD/0E/OGyXntDgzA0X6qmCnrVutVlrUWV9f/3xo+pwhGDhvEPHOjoxnZjJggXmMHzBQ7NGNp9vxk61fr0HR7e/u7pZzCGHlc7qwBYYTT7tJYSx1AQzppyFPft5apta9w7SKcn0b7P7+/jCsDQ5mbc0dCmIJGDN0ehdcjsmkm6A6KUeKFOTE11PLxrC7Ukqh3ylL2fT0NAP9q6ur6rRCJJYsbKB0JsbCKMuy+xREePDyxQPCz+Crlw062QcA5wBOOt1l6vIl2WiI9F1fN6Q+BBqit6hEC4Hk08GQJMn4myjSP7RavVxgdaVUh/3U6HCMsPr9pYnJKRziHtWQ+un58+hGs6nsjQSjpuTyKGN3CX+FBwHXSiEVgjP+O8X6N12kIePES+GzTKAkGbNp8yJsGUMVzz8jPKReiyAQRimy5/cjye5RpF8utFp/+nwmT7d/NMzcFkS7yjJNGDaPURQxIQThEQy0SyF4l5WJYYhBa816vZ6dU7A6CAhbZVow/pDe0O9hVOoCi13r4BgBAvJHqMSQL2vE/iH6IAXEwgrRVUmBoRRwnwJQT98xEeVeSUyB4dJ5nwJBKdCFFGRmUCcu7rwIYypCTblaChuNBhWODrman5ub+4v0rMNBt8z6Ezh7GksJQpCbm79cMQE7QBFm/X6f0rjWnv8WRYg/QdbUpwDAEBy8vPyA8rNGzg3a8MiElwiM7dAtRqNoNptjGPM1laVxP9umWEMGLOKhKUOJDtBwDmzsw9fC/CzHr9SGuCTi2LbbKvVtmqXpCjMihBFa79Wrt5fGx9PDzc3fmu32Lf8qFliwU9emKhBSp+kRKn/hu9k1COEDbFdt/BoKWOAkuEbdVYyoIXv8+I/QK9dMHEb1Knb7MHOv8LFFOsjzCVHWOD7Ltn+MXCRF4729vWMDK+p8rLkvwjLg4N4v741m5YuwCI9CvHp1Ha8gFdBoPnQAkGsYYGxxcfEI7QQlFCTGUXwjAz4tWF+EpymOWu7fglE7qsOvrYE6g4+9/x/vhRbMdLOCFgAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-polygon-select{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEjc1OfiVKAAAAe1JREFUWMPt1r9rU1EUB/DPK0XbqphFHETo4OCiFhwF0V1KHbRSROLqon+AUMVRRFBwEbRFMBiV+mMW/wIxi5OD1kERRVKRJHUwLvfBTZrU5OWBGXLgQu7Jfe98z/ec7z0vKa88b2q1BDtRHdAPBaylm1NzsxsOjPnPNt6WSWprbft+/c3I3zOAjhT1Y4+fvcjEQJIXnVECSa+AhqIHqlHH5lWCZoe+Gk4GRgDG86j9SAUdlDBSQaZhlOkuHyoVdJmsw98D1S5fM4NYM1LCpqM+Lwa240oLgmZzpVZvzKT75VLZcqksSZKWlQeAy/iORVwIvh31xvotvK7VG3Px4aWHj3Jl4C2uYSvq+Bn8v6LLbaVWb9zsBiKLCvbiNG7gLm7jAYqbPHMJMziZ9lsKoh8GtqCEVVzHftwJn+TFHp4/hg8BSCYVfMOZoPEv2NZGdy9WCGUr9toDR3E2/H4V6nwRe/BmgN65H1ZhvMuB3XiKIyFoGefwO6ysVkUlrNUNsyAK/jli533Q+Y8cJFvAeXyMS1CI/jiMr/gUtD2LQwMGr4R3p7bY3oQHQ5b38CT4D2AXXg6YcQXHpyYnlqKsi5iOAVSwL9zd7zJ09r+Cpwq72omFMazjT9Dnibym0dTkRDUKrrgwH7MwXVyYB38BstaGDfLUTsgAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-redo{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4itK+dVQAAAaFJREFUWMPt1L1rFFEUBfDfJDaBBSslIFjbaSFp1FJQFMVCHkzhKIqdUYOCoBgErVz8rCwiTDMwBCIKipDWyip/gxAIWAmBgBC0eYFh2Gx2l9lFcA5M8e59782Zc84dWrT435Hs1siLchqn43MS0zgW22vYxjesYjVLw3YjBPKinMUTBOwf8J5fKLGYpWFjJAJ5Uc7gIW6jM6Kim3iNZ1katgYmEL/6I+YasvY7Lg6iRpIX5VF8wuEe/XV8wGf8jN6LWTiAc7iEQ7ucPZ+lYW0vAtfwvlbfwCKW9gpXDOv1mJvZHiSO91MiyYsyiQSuxtpXXM7SsDmM5nlRdrCMMz3sOJWl4Xevc/vwBzdwAl+yNNwZxfRI+GxelK9ikHcwh8d4NNR/YFRES1ZwoTYdR7I0rNf3TzVNIGbmSvR/Bx08mIgCFSVu4l2ltIWD9WxNGR+W8KOynqnZ0rwCeVG+wa0hjrxtWoF5dAfc28V8Mib/n+Nev5dnabg/zgw87aNEN/bHOwVRiRe4Wym9zNKwMKkpgIWKEt24njxiJlq0aPFv4i9ZWXMSPPhE/QAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-reset{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4gWqH8eQAABLdJREFUWMPtlktsVGUUx3/nfvfOlLQaY2IiRRMQIRpI0PjamJhoVASDvNpCpYw1vJQYSVwZwIVQF6wwRHmkAUof9ElrI6VqDAXcID4TF0IiYQMkSlTokNCZ+b7jove2t+NMH7rQBWd3v+989/zP+Z8X3Jb/WGQySvUNTQBJESkNguAVYIWqzhaRhwBU9WcR+QXoymazn6jqzUQiMQSQzWZRVdal1vwzAI2tHQBPOuc2AbWTdOyQ53n7nHNfRwee51GzqoIQMCLDpr3x/tLQ0oZzrk5Vj0/BOEBt+KYuOlBVGlrahr0Wob27t3gEjnZ2AyQzmUwHsDgP6J/AYRE553neDwDOuUdU9QngNeCumK4TkRMhZUORcYC1qysLA6iuSQHIwkWLD6lqapQsuSmwTVV3h99I7EcAR462A2xR2Ilq6ehTaejvO1774kuLNALR33eclsaGsQDe3fYegHl43vyNwEeqGl1963mm2jl7YZRTQ82qlWP4HM6ZToC5ztkW4LHQoALru7s6Di5dvlIj/e6ujrEAWoZDn8hmMjXATMACGaAVuBjXTVVXFc/AxhaA+4zvn1DV+eHxVWPMAmvtb5GeMWZyZVhI2rt7qVy2pOh9U1snwIPW2vMi4oWJuBPYHkVAVScPoKmtkzVVK6cEMsyJraHhiCqJqJUwj/JRz7TW1iSSyR2rVyylqa0Ta+24Ic8vXaAEmDFc/l5Z2A/80OibuVyuz/f9ElUdHCmvw82t5HK5h6y1PYhsz2YyGw43t2KtBZHIGwB6+j4rCkBVUdV7gXrggnPuu8h4eP+xMeZS2D0rJYZ6AdAMzAt1b4nI26p6IFZOY8pugijcKSIHVLUK0LyST4vnrVfnWr3mjmP4QTATaERkXkypRFX3isjmuHdRJEK6Ckqquopp06bdKCkp2Sgi7XnGLcg7gzeutwNIiPYc8HixqIrIOlU9ONVIhHPEd851icgSVXUiskVV94gIqoonIt0i8gfQCfwae38e6BWRXuBZz5jZ8VbaOE4EIqlZVUEQBLlkMplS1QER2RwkEnsSyaREDUzyeNsvIhvCMqkH1kdIJ2o+k8iJB1LVVRfjZ6nqqlEAIbdVQGto8Lrv+/dbawcjAL7vc+6bs+zetetfLSHxniIFGofGGsU2oC7eOCbDfZ7nQawBOSAX74SF9oEPImOq+r7nmVmxb5raukZa8UReGmNmhbMkAwwBH467EYVZe49z7kdgenj8k7V2oTHm8kgdWcvrNdVFjR8cHkYzjDH9wLjDaEwEzpwa4MypgWvAjtjxfGNMj4jMiT+M+kFsZI/Q6Pv+HGNMT8w4wI7TAyevxXVPD5z8+zD64tRXAMHVK1eaVLUyVvuDqroV2BOnJF4ZIedviUidqt4Re9s+vbx8zZXLl7PR2+nl5Tz/zNOFp2FzxzGAklw22wUsLLaSKXwf8vhosZUM6PeDYEUum70VHfpBwKsVyyfeikOP6oBNwN1TrLbfgX3A1kKLzKeff8nLLzw38T5wZDgxn1LnNk5lLRfP26/OnR2hwfNYW2Atn9RCsrf+EECyrKysDFimqhXhyjY3VLkAXBKRDqA7nU6nS0tLhyIj6XSaN9bVclv+l/IXAmkwvZc+jNUAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-save{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4UexUIzAAAAIRJREFUWMNjXLhs5X+GAQRMDAMMWJDYjGhyf7CoIQf8x2H+f0KGM9M7BBio5FNcITo408CoA0YdQM1cwEhtB/ylgqMkCJmFLwrOQguj/xTg50hmkeyARAYGhlNUCIXjDAwM0eREwTUGBgbz0Ww46oBRB4w6YNQBow4YdcCIahP+H5EhAAAH2R8hH3Rg0QAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-tap-select{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAA2hpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMC1jMDYxIDY0LjE0MDk0OSwgMjAxMC8xMi8wNy0xMDo1NzowMSAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0UmVmPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VSZWYjIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iIHhtcE1NOk9yaWdpbmFsRG9jdW1lbnRJRD0ieG1wLmRpZDo3NzIwRUFGMDYyMjE2ODExOTdBNUNBNjVEQTY5OTRDRSIgeG1wTU06RG9jdW1lbnRJRD0ieG1wLmRpZDpCOTJBQzE0RDQ0RDUxMUU0QTE0ODk2NTE1M0M0MkZENCIgeG1wTU06SW5zdGFuY2VJRD0ieG1wLmlpZDpCOTJBQzE0QzQ0RDUxMUU0QTE0ODk2NTE1M0M0MkZENCIgeG1wOkNyZWF0b3JUb29sPSJBZG9iZSBQaG90b3Nob3AgQ1M1LjEgTWFjaW50b3NoIj4gPHhtcE1NOkRlcml2ZWRGcm9tIHN0UmVmOmluc3RhbmNlSUQ9InhtcC5paWQ6OTQ0QzIwMUM1RjIxNjgxMUE3QkFFMzhGRjc2NTI3MjgiIHN0UmVmOmRvY3VtZW50SUQ9InhtcC5kaWQ6NzcyMEVBRjA2MjIxNjgxMTk3QTVDQTY1REE2OTk0Q0UiLz4gPC9yZGY6RGVzY3JpcHRpb24+IDwvcmRmOlJERj4gPC94OnhtcG1ldGE+IDw/eHBhY2tldCBlbmQ9InIiPz6eYZ88AAADLklEQVR42rSXf2TUYRzHv7tuGcfE6Vwb5zLSSjEj7Y9KWqfEmFZJP+yPMdKKmUrrn0iUfjhWlLFi6YfNrF+StBoTo39iYkTGco4xxxG59P7k/T2PT8/37nu3bx9ezvPj+zyf5/PreS78bGLS8SmrwE6yje3NHJsDBTALpknBz6JhH3NiYAB0gHqPOVv52wJ6QQ48BzdAttTioRJjdeA8mAHHS2xuk3p+M8M16ipVQE49Ds6CiFO9RLjGONf05QLx6wPQaBlbBlPgJVgkP0ETiIJ2sB/E1XfimjfgBOOlKDUqCGOcqBcQnw6BYW5YTo4wbvQhMmCfGRemC2rBiGXzWUb+kM/NRZ6CHWBM9ce5R61NgX6ayhSJ5EPlItlDRNkz4JbFHf06BkSzHjXxM+gDv1S/mPUo2AXWgt9UUHL/IVhS8yUV1/EbV3o4N+NaoE9Fu/i827K5pNYHnqAVJECShWmAaddpscYFFXwR7vnXBRGlnUN/L6kqKJlxnRUuDbaDBiL+vst5d4gpcpBrqk/2jIgCKVUolhntplzivHmwh4stGOPfwBWwl/2dpp8p7xjQZqFLiQJtauKkivYm+kzccpK57yXfOUe+P23JqAnVbhMFmlXntCWnxbT31am9ZJ4BJifsUmNTqt0cYhA5ypympPg7VkEKunPbVb8cIG+0kyHLJZNR7fUMooUKFHAPkfQo58VLK+RzwRDd4FdWG9mjpaAXzqkJa1R7kQttqEABWXMjOOxxVRfnhRm5URX1prk/0pQHwNcKlchZ+jdpC+hFdVqO0my9Hj5dkYgCn1Rfh/KdlNDHrJhPqlDih+IfBd6qwpOgEqYMsorJ2HtWxtagLJDn/W3KRfPOZhoeBJfZPgVeGKeKrkQBh5dLXl25Ny3pc4/1fkTdbvFqFQgbxWeYD0hXulhQ0pYiM1jG547fcbMQpVnHTZEn9W3ljsCzwHxCdVteNHIZvQa7/7cC7nV6zHIfyFP9EXjFa7YxKAVqPP4bxhhoLWW+z9JyCb6M/MREg59/RlmmXbmneIybB+YC/ay+yrffqEddDzwGvKxxDmzhc0tc80XVgblqFfgjwAAPubcGjAOl1wAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-undo{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4em8Dh0gAAAatJREFUWMPt1rFrFFEQBvDfGhACASshkL/ALpWVrSAKEQV5sIULWlgZNSgIFkGIVQ412gkBt1lYLERREFJqJRaW1oHAoZUQsDqwecWy7N3tbe6C4H2wxc682Zn3zTfvLXPM8b8j6RqYF+UCzsfnHBawGt3fMcAX7GEvS8NgKgXkRbmMxwg41TLsN0psZmnodyogL8pFPMIdLHUk7hA7eJKl4U/rAuKu3+HslFr/FZezNPSTFslX8QErDe4DvMVH/Iq9F7VwGpdwZUjsPtaSFjv/1vCBPjaxO0xcNbHejLpZrrlvJCMCT+JzA+2fcC1Lw+GE4l3CG1yIptfjCtiKoqtiJ0vD3aM0Py/K57iIMxgkQxat4EdN7e9xdRzlk+LEEPvDWvIDXJ928sYxjL36icWK+VaWhlezOIqbGFirJd/H7szugrwoX+D2BDEvszSsT5OBdfRaru/F9dPXQF6U27g/KnmWhgctxqyzBrZGMNGL/rHI0nDkKXiKexXTsywNGx0OnFbFNk3BRoWJXnw//j+ivCi32/S8CxPVNiWOAdUiJtXITIqYY45/Cn8B2D97FYW2H+IAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-wheel-pan{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEgswOmEYWAAABddJREFUWMO9l09oXNcVxn/n3vc0fzRjj2RHyIZ6ERuy6CarxJtS0pQSCsXNpqGFWK5tTHAwyqIGN7VdEts1LV04BEoxdlJnUbfNogtDCYWQRZOSxtAUCoFiJY0pWJVUjeTKM9LMe+9+Xcyb8ZMychuofeHCffeee7/vnXvOuefYlV/+mv932//tb91z/Y2rvxmMHQ+4FcEfOIGN4A+UwDDwoQScc7vM7AIwB8yZ2QXn3K77Ab6OgJnVgeOSbkqaBiaACUnTkm4Cx3OZzwf+qzcRQup1zNZ9RwDe+0YI4YKZTUn6zCGSMLOfAF/03r+QZdnyfwO+ePEiI6N1nPMgMDMkETLRbd2mXG8gCbd9YiIKIUxLKoLfBN7I+80+CUlTIYTp7RMT0b3Af37p8kh5y9gZcy4Fzt+5szqSaxkzUR7dwtrKMmaGW242d0t6vrD/He/90865o865o977p4F3Ctp4frnZ3L0Z+OryUrVSrZ0z8ZxhHjhcq1XPrS43q/0flDlK9XpPA2ma7gMeyvfPx3H8TJZlH4YQWiGEVpZlH8Zx/Awwn8s8lKbpvmq1ahvB641SXNk6dhLskNA2MIBtwKHK1vGTW8bKMRbAMgyPqWeETxUM8VSSJAv52JmZA0iSZMHMThWwnipXKp8hsLLcSaIR92oU8xjSayCQXotiHotG3Ku3m+0EOQwPQCDggMf7BzQajSs5eAk4B5zLx4O1vD2eJMmAQKliscgASJMw21pansFs1swQ/DNLmUmTMNuXX+taXHTDaj5OW612R1JZ0nFJJ/J+XFJ5aWmpA6S5bHV8fHsPHFU6q3pJCjtFxtrKMuXRLUUXXxdrRLazFOtUolZlsGhmACsgnHPTwJnCnjP5HMBKLotzxsTE9rgDL0t6LoriKsDIaB31ZEK+JxQJRHFUBR2NqLw8OTkZR0OC0ntm9k1JWU7OA4vD/mZ+YfElsANmNEKi75vztzB5M8uAr+bx48me88g757PQ1U5zNg52YH7hX8l6f+4Fi3c3BqHNmkI4YQOV2MGCNu9qHPYCewfzbrC+XSGcWEcgTRKA3wFfyzdDz5d+D3x9CIcfA4eBbQS9LscskgfLnHNPAnslvS/pbZDHLLPADpx9N9fqpSIBH8cxWZY9m6bpb4Ev5fN/iKLo2TRNgdx/eo8Wk5O7Ts/N/SOSdMjHdj4kmgkIEJLJzPZKetvMTkIvFLsR25Ml2gfuF5M7vnA66sdooJYkCSGERe/9VAjhzRxoKk3Tvg3U8nulVqvx8cyNpER2umM+SdOkbc5B8JhpqBdIgTRR24h+lpKen731aRIN7thscH9Zlv0d2F8YD2TIX7F2uw3A7ZWV1a0TYz9ca8cJZHRbuRuaDfUCw9/qJHamPOKToAwHtHN6lMvlSkH2o7wDMDo6WuGuQbbn5+YAKNcb3J5fSvrhtTY+vsOPuD1IOyRhMOkj9kSx29HfXB5RUnS964NT2+3vbGbxG9auO2cDNuV6A8NTb5TitBuOpQkfYD2vwOxgmvBB2g3Hto5X42EJyVsFlztbKpXGNgqVSqUxSWcLU2+tdToa9hasLjfPYlwGa+bTi8Dl1dvNsyvNtQQL9MO2w+HM7BqwlAtPdrvdq9773WAVsIr3fne3270KTOYyS2Z2bbXdHhogKmPj7YWF+VOSXs/v/9KdO+0fVBrjbRkgB/KIDBnYu9f/7D+ZmfmRxPd6qwB8YmZXcq1MAQ/nJhTM+OnDe/a8+PGNG9lm19V/D1Qw7HXZlcRa69+U6w38l5/4ipxzf5X0CPBILjcGPJH34pVcc8692FxcXLlXRnTwwH7+9P4f8aWe3fY59LIqo1NMyQBCCHNmdgx4BegUWefjDvCKmR0LIcz9L8nokSNH+PRvH4HC3YQ098pSbevg24qlmZmNmtmjkg4D3+j/tZldkvQXSa3PW5ptlpL3ZaIN99OS9F7+IgKUgSyEkNyv2nHT7DZX0dr9rpjua2l2r4rogRAYVqZvnPsPqVnpEXjEaB4AAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-wheel-zoom{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEgskILvMJQAABTtJREFUWMPdl1+MXVUVxn/fPvf2zrSFmUKnoBCUdjRoVaIxEpO2JhilMYBCtBQS2hejpg1Uo2NUrIFAoyGmtiE+GHwQGtvQJhqDmKYRBv+URFsFDNCSptH60DJTO3dKnX/33rM/H7rvsDu9M20fDMaVnGTvtb69z7fWXmvtc/TEzqd4OyXwNsv/FwFJQVI/sA14SZKRLOlPkr5TrVYXHz70quYkEEK4TtI2YAgYkrQthHDdhV5uuw+43/ZrwCbgRttgY/tjtrc0m83X3/f+D6ydnJhYcB4BSZcBA7aP2d4ELAGW2N5k+xgwkDB0IH19CGGH7R8B1aQeAf4KvAw0ku4K2zu7uru3ApdPEyiKohd4TNKjtjt5h6RHgccSNrddbvuHtm9Jqoak7xVF8WFgdavV+pSk5cCObNmXgK++85prCj3z28HKqZMnH7D9YAY4BvwujT8BvCuL1INX9vVt+dfwcCvNb7f9q2RuSfrGvWu/sL2Nf3LX7pzvj4ENSGBPVarVd4fRkZFltjdmoMGiKO4IIWwIIWwoiuIOYDDzeOPoyMiyFLkum7WJCMDztrcrTTrIRuAQZ6NcK1utL4dWq/VZoC8BhqvV6l1lWb4YYxyLMY6VZflitVq9CxhOmL60hhCKeYiV7WMKIXw9jT1HpXw3c+bOAKzOjJubzebJrKQCQLPZPClpc7bP6rMYKtjXth2OMf7tIkr11Wz8oQDc1Fb09vY+kQw1YAuwJY2nbUluAnCWpKkaFl6IQIzxivaR2SYA89sJVK/Xp2x32R6w/a30DNjuqtfrU0ArYecDCEqgLqm94T0dEm9mBG7PxkdDlkBnkhebgIezNQ8nHcCZPL9ijE1Jf/bZZoPtzbavmqNZLbf9tSxq+yoduuJ+SZ+zXSZyBXCqU+d8fvC5yRUrV+0G2j3g2hDCLyXd/+Su3QdnvP/zCuH72LWsgf2k0oHlH2c2odlkxcpVEdgr6aDtjyb8x20/J+mA7T9I6rL9SWA5dne2/GdXLl58qNJh398An85yTMA+4DOz8Dgu6Zu2dwJXJ91ltm8Gbp7Fgb+EEB4aHhpq5CEtACqVyr3AC0AlPS8k3TSmQ2YPhhBuS/1/LpmS9JTtNTHGfwBU2uUALARotVqniqJYH2Pck85pfavVaufAwnQvnHc0McaDKVptebN94QAnJB0EdtjekydyZXqjs/0ZgLIs/w6sy8bnYGYJ63pgERKC05JutT1kOwITwL9tvzlzUQUYB+Zjs2DBgu6xsbGJZHstByZbezregcBXeCsEz1bnzXt5anLyzLq71zDLxTRdVgemdx0fv2e2w5thO5DbiqL4oKT3ZKpnpyYnz+SY2ZpTAPZmJfdIrVZbNBNUq9UW2X4kU+2dcf53Aj1pj2PA7y/6m1DS00A9za9uNBq7iqJYBuoGdRdFsazRaOzKSqye1rTbaa/tlbYrqXQP2X4FIA9/J1l39xrC0v7+w5IeB8XkwS1lWe6TGJAYKMty31tfO4qSHl/a3384I3CDpI+kzC4lnRfrue6GytEjR8oQwlY73gC0L4qlth/q0M1/LYWtR48cKQF6enrC6dOnVwGLEpnxnp7en4+O1i/tszzGOCTpPmB7ahb57QUwBWyXdF+McWg6MScmuoA8OX8xOlpvXGz422XYTsB/SnpA0h7bX5R0WzI9HUL4qe2XbI+dk3xl+V7gxoztD5jRI+YK/zkEEokx2/uB/RdzIfUtueqVN04cXwF8G3iHY3z9Urw/j8ClyhsnjrcS2Vv/J/8NLxT+/zqBTkcxU/cfEkyEAu3kmjAAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-box-edit{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEg4QfHjM1QAAAGRJREFUWMNjXLhsJcNAAiaGAQYsDAwM/+lsJ+OgCwGsLqMB+D8o08CoA0YdMOqAUQewDFQdMBoFIyoN/B/U7YFRB7DQIc7xyo9GwbBMA4xDqhxgISH1klXbDYk0QOseEeOgDgEAIS0JQleje6IAAAAASUVORK5CYII=")}.bk-root .bk-tool-icon-freehand-draw{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAADTElEQVRYCeWWTWwMYRjH/88721X1lZJIGxJxcEE4OOiBgzjXWh8TJKR76kWacOBGxdEJIdk4VChZI/phidRBHMRRIr7DSUiaSCRFRM3u88gz+o7Z6bBTdjmYZPf9eJ55fv/5zzvvDPC/H9QsA66Olo9Ga+/MdR+Ljm2/KQIULsz9FqItGdOfJKLhApLgVkiSCGODjWit7QpKWy+TNrFeXvzKVUT8NiTVaIgDcbiCFJ7GiT8WkARXAdYBK0Lbhi/CenArRNskuM7/tgNp4ArQ42dwjf3WY5gWTqC7O/NbNn2Xkfw/YwdSw/We14HP2IEZwX+y9cZ9SH0LmgFP7UCz4KkENBNeV0Cz4b8U8DfgKiDxMWwUXETqLvJpCQpXZfawbzS7t9v5pL19cHBwfja7YA0y/lyCM0+E5hv5+piZXwKYcF23as+37bTXsQVqgkL0p/34fHR7DcBtbetFsBmGDwMOJCggYG55yw7dMlk6DuC1Bdu2RsCU9TYWQq2IoGbsreZ5NzvEqfSBsIsIy8OTbcdgiRHeh4o8AFAEwDakbY2AaCCpH7V9aGhoUUUy3UyVbkPYFuYLDlUZH8XBpwxkK0Dbgxg5HcVi0ent7a0RULMIozaHBSMfF9b2SzdutFcFB2FkwMIJOG6qfteXOa1nHZ48tyefuwyfT9s6wtzZ3t7eZse2DR2I228TtHXzuWCx9g8MtK5cuHCZTH4tiHEOa4xFngvTyS8f35d6enomiCi4/foEXBkZaQuukChL4FYA2Whd7YcC4gEdW3CpdL3LtGAVCVYJywEyTpAuJKeMOKXZs/Bw947C50KhUFOG4cwz35cjWNBlHGeD53n3xsfHP/T19U1qciggar8Fa4I3PHobIotBWBtc2hSiChyZxVzM53Pv7FVH6Tp3uVy+g0r1ImD2GjIrQGYIxjnfuXTZGICS5k/bBwJoubwEFX4TLah9EXomJGMA3za+f9913Yl4TnzsDQ+vE6YTZOjHh4ngibstt1pzQwd04F0bPStEBpXqRoBeQ/AKghfBnOEKgS+Q7z91Xfdz/HGKg8Ox7z8iYD9z6wqTkZFgnvhMGP9VZ2or1XVkPM9z0mytSfVsHa1RLBZbLoyNzUnK+ydz3wC6I9x+lwbngwAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-poly-draw{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEjglo9eZgwAAAc5JREFUWMPt1zFrU1EUB/DfS4OmVTGDIChCP4BgnQXRxVHqIJUupp9AB8VBQcRBQUXIB9DWQoMRiXZzcnQSA34A7aAuHSJKkgo2LvfBrU3aJnlYkBy4vHcP557zP/9z3r33JdXa647N0kHSZd5Nn0rSxc8G3cXp85sMcnZZ8vge3osZ+l3vB8CWFA0iL14t79h210swAjACMAIwAjACkB90D/8/GchI9ve4nPwTBh5E9ws7OepzGWb9EddSn51Op9ZstadSg4VK1UKlKkmSDSMLALewiuNh/hVJq71Wxttmqz0dG88vPc+MgWP4grvYG3SLOBrZFFFrttqPe4HIDxh4GSei+98iSlusuYopXEAjBtEPA3tQwUpwluAbDm4TPJUz+BTW9l2Ce6G7L0X/Bw8D3T/7SKKIDzHg7QCcxjvcQAEtXAnrrg/RP0/DKPbqgcN4iVOR7gcO4dcQgRuoh7HSqwlP4n20m63jJu5n8MkWMYfP3UowhzdR8FU8w9iQwevBdyq3/27CMRzAE5yLuvsRLg+ZcR1nJ8YL81HWJUzGAPaFZwe/Q5MdyYDyNHgjzO90YyGHtVDncuiJchaHw8R4oREFV5qdiVmYLM3OgD9k5209/atmIAAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-point-draw{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gEMEiERGWPELgAAA4RJREFUWMO1lr1uG1cQhb9ztdRSP7AF1QxgwKlcuZSqRC9gWUUUINWqTh5AnaFOnVPEteQmRuhCURqWsSqqc9IolREXdEvQBElxtdw7KURSFEVKu4w8wAKLxdw9Z+bMnRmZGXfZ29//II8th4WwGVNyIoQLYB5vxA9Caq04iUd9A+7ZlsNC2I7TdSd2hZXMJKlnTqp9jtl/GBaqoyQ0noFKpUIzBicYYc+DEFpxkglc4oVJa5gvDn8v1xV2irG3FM4NSVwjUKlUaMcpJhCGmSEJQ6QGD8M5WnHCd8+f3QCXpPLx8WNwv0j6Bm9FMK7FJ3WBE+R/2t7c/GBmFvSBrzRTCsyTDjXrxUgEMtpxynJYmJoBJ4VAybwVARgvL7Oik0okCodnKpVKX7P0leiVMb0VvbJT+upznK4vh0GIeQwwQStJkHQD3MwsCALTJRG7Qrdrj5m/djgYaIa0hlkRdJk26XEgC9txurccBtVW3IudBImmZuACUP+ZlIDBt9FKcubYNTcAH/X0RYM1E7utJPlqe+uZzPxUcEkiSS4sTT95n15Mud0xWC0o2PAWOCdK3KYZlFxfM+tHOcnMzNr1es18ug+cgsVjP4yBU/Ppfrter1m/+l0+zYygML1xRVHU7TSb1cSzBzoBzszsH+AMdJJ49jrNZjWKou6wBnwOzcyndBpNbuueURR1Dw8Pq35p9cc5p/Dy9Dypt7jXrtdGwQECS9NPhr6Gq6txUzNigE6zydLK6lTw12/KT4FGFEUfJX2YJNONq5tVs4ODA7sD/DnwJ/BoADZuE3tHFs12dna6d4C/BI6AlbyzI8ii2TTw12/KK33gb2cdXsNZoAntbZC2SeO4c9592k/5eNQbiwvFd1kJuFGwLJr1wSPg/SwpvyFBHufOeXcFeAlE97U/uCxOY+P3b+Bn4B3Q+L8EdJfD4a+/AbC4UBzPxiPg3wlHZquB28Cn2IuR9x3gr3uV4DbwfvSDOvi4uFA8BDZmIRHkjHpS9Ht9iRqd8+5G3g05mAGcQbsdiX5QJ428G7Kygo8XYdb1/K4NWVmjzkNge2sz84bs+ELmpDDLtqWsNZBXgvmw8CTtpWVMT7x5YWBjLARnwZfKQNYN2U2LPvrh+5nBt7c2M2/It9bArCTKR8eZN+SJ13AScPnoODeRdqNenH+wul5w2gUr2WUjMFAt8bZ/0axX/wNnv4H8vTFb1QAAAABJRU5ErkJggg==")}.bk-root .bk-tool-icon-poly-edit{background-image:url("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4gELFi46qJmxxAAABV9JREFUWMOdl19vFFUYxn9n9u9sCyylUIzWUoMQBAWCMdEEIt6xIRQSLIEKtvHe6AcA4yeQb7CAUNJy0daLeomJN8SEULAC2kBBapBKoLvbmdl/c14vdmY7u91tF95kknPOnHmf95znPc97Ro2OTeBbdjFDT3c32ZxVHUOE9kSMB0/m6ExuoJn1H+ur6Y+OTfD50SMN5168OgrAlyf7CfuD+z7+iDs3p8hkLUQ0iFQ/yFl5Nm/qonfHVva+s32Zw9GxCYILsZ08tpNfBhbs+1YN4OH9+7huGdECSBVfqUosbsllfmauBqiR+cCNwOr7AEo8pPHJnymXykhg5fUWjoQpl0vVvhZhbSzGoUOHqgBlt6B6uruj2Zy1E9jo0fhfeyL2x4Mnc8VErK0KUEOB64JSyptfG4RSytsJjUJVxw2lsFy3urL9nx1Qd25ObctkrVMi+jQivd7U2ZyV/3Hzpq7h3h1b/7p9Y0o8v8rwAbTWrGpSocN/FGDlbAI0Rl23PCBan0Ok158H9Ipwzi25A/Mzc9Gl/BYx/E4kYqC1NKRARNAaDCNUM27Z+Zr+ouXs0q4+LSLBHPYCFkTkC6uU39kwCdsS7WRKmaYUiAhdnZ3MPX2K4+QjQI+C94A93rMzm8ltMwyDeDzWjMZeEb2pYQDdW3vITU2jtUZ5QThOPgm8C7wP7J15OPsBsB3oWpGnVWisCeDS1VHj4vBI92+/3tgB7Ab2AruAXiDBK5oIOkhtkEYRNRuJhObrd8Dl9ewf4D5wG7hVLpen29vb5wzD+BrkbBMaL3d1dk5nsrnlFDTTFWAWmAZueWD3gCemGde2k2fw1Al1YXhEvjozoO49eczdqekrWmsc2zlrmvEKOGoW1GUjFLqSk2KpJrCLwyMCPAP+BO54QL8DM6YZX/ClsP9YnwKkXnIBP4jdIpJRpdJTCYdMwwi98KU0Hjc/dDILNyUcwTCWdOSMJ0TRmBktGRhLugu0xyLk7CIqVNm+0bGJptl1YXikD0grpY4Rjc4a8Fbgdab/6OGbAJeCUuyJnnHmZH9pbSyGuBXV8NUwlUpR1EWyixmSyTWEwqGlJ2Swbo2JXbAAfgDGgGQA9I1A9t1tlq0AxrXxn0ilUpw4fhQqYkH/sT41OTnJJwf2s6FjI5mshdYa7bqVR2uezr9MJmJt14FvGrh/O9D+e6UkM/xyCuCqEKCYnJyUTKFQrZDHjxzGshwWLQcRsOz8Hi85P23id0ug/XilAMLBmm4tPGdoaKjSH5+oAGrhwvBI9SjZTn4QSK9yenoD7dlrExPoJlXW8G8ytpNHxRKk02lGxsdRKFwXLNvx5yY94HQLGhGk4LFCYQSqaE0AwWM1eOoEbR0dKBSW7bC4mKuffxs4D/wCLKwQQPAUzIkslfp6cVomROWSolh0GjldAM4nzDi2k9/i5UAzC9aKfwNJ3zgJg9YEvN6+C7SHgKm69+sD7RfNnKTTaZRPQfAut4oFV//IS7gkcB34VlVo8kGzphlfB+DU+TfNGBpZtRastvrvARJmfMF28ge9sc2B9/PNnCilMIDwK6y8/ow/Ai4kvILTljAXvDvEvrqKSUs60KolzPjBxspavQD2tKqCAGF/Ba+xE/Wbilu54wZV8NEKF5fXzQHl/bh4hUsE0WAXSlDMYcQSrQXgCmsTseXHsJkNnjqBFGwKJaHsKlxtUHYVhbLCzr1kaOA4bcn1y1Swmb+iLpJKpVrfgdpfsiVVCYcgluwgnU7jEgJ4s5UkLFtWYyHyEg0/N1q1tmQH+YXnAMFr97Nmv3p+0QsHQRsF8qpBOE5+rb9Nkaj50tVQKjqh4OU3GNL/1/So3vuUgbAAAAAASUVORK5CYII=")}.bk-root .bk-grid-row,.bk-root .bk-grid-column{display:flex;display:-webkit-flex;flex-wrap:nowrap;-webkit-flex-wrap:nowrap}.bk-root .bk-grid-row>*,.bk-root .bk-grid-column>*{flex-shrink:0;-webkit-flex-shrink:0}.bk-root .bk-grid-row{flex-direction:row;-webkit-flex-direction:row}.bk-root .bk-grid-column{flex-direction:column;-webkit-flex-direction:column}.bk-root .bk-canvas-wrapper{position:relative;font-size:12pt}.bk-root .bk-canvas,.bk-root .bk-canvas-overlays,.bk-root .bk-canvas-events{position:absolute;top:0;left:0;width:100%;height:100%}.bk-root .bk-canvas-map{position:absolute;border:0}.bk-root .bk-logo{margin:5px;position:relative;display:block;background-repeat:no-repeat}.bk-root .bk-logo.bk-grey{filter:url("data:image/svg+xml;utf8,#grayscale");filter:gray;-webkit-filter:grayscale(100%)}.bk-root .bk-logo-notebook{display:inline-block;vertical-align:middle;margin-right:5px}.bk-root .bk-logo-small{width:20px;height:20px;background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAABx0RVh0U29mdHdhcmUAQWRvYmUgRmlyZXdvcmtzIENTNui8sowAAAOkSURBVDiNjZRtaJVlGMd/1/08zzln5zjP1LWcU9N0NkN8m2CYjpgQYQXqSs0I84OLIC0hkEKoPtiH3gmKoiJDU7QpLgoLjLIQCpEsNJ1vqUOdO7ppbuec5+V+rj4ctwzd8IIbbi6u+8f1539dt3A78eXC7QizUF7gyV1fD1Yqg4JWz84yffhm0qkFqBogB9rM8tZdtwVsPUhWhGcFJngGeWrPzHm5oaMmkfEg1usvLFyc8jLRqDOMru7AyC8saQr7GG7f5fvDeH7Ej8CM66nIF+8yngt6HWaKh7k49Soy9nXurCi1o3qUbS3zWfrYeQDTB/Qj6kX6Ybhw4B+bOYoLKCC9H3Nu/leUTZ1JdRWkkn2ldcCamzrcf47KKXdAJllSlxAOkRgyHsGC/zRday5Qld9DyoM4/q/rUoy/CXh3jzOu3bHUVZeU+DEn8FInkPBFlu3+nW3Nw0mk6vCDiWg8CeJaxEwuHS3+z5RgY+YBR6V1Z1nxSOfoaPa4LASWxxdNp+VWTk7+4vzaou8v8PN+xo+KY2xsw6une2frhw05CTYOmQvsEhjhWjn0bmXPjpE1+kplmmkP3suftwTubK9Vq22qKmrBhpY4jvd5afdRA3wGjFAgcnTK2s4hY0/GPNIb0nErGMCRxWOOX64Z8RAC4oCXdklmEvcL8o0BfkNK4lUg9HTl+oPlQxdNo3Mg4Nv175e/1LDGzZen30MEjRUtmXSfiTVu1kK8W4txyV6BMKlbgk3lMwYCiusNy9fVfvvwMxv8Ynl6vxoByANLTWplvuj/nF9m2+PDtt1eiHPBr1oIfhCChQMBw6Aw0UulqTKZdfVvfG7VcfIqLG9bcldL/+pdWTLxLUy8Qq38heUIjh4XlzZxzQm19lLFlr8vdQ97rjZVOLf8nclzckbcD4wxXMidpX30sFd37Fv/GtwwhzhxGVAprjbg0gCAEeIgwCZyTV2Z1REEW8O4py0wsjeloKoMr6iCY6dP92H6Vw/oTyICIthibxjm/DfN9lVz8IqtqKYLUXfoKVMVQVVJOElGjrnnUt9T9wbgp8AyYKaGlqingHZU/uG2NTZSVqwHQTWkx9hxjkpWDaCg6Ckj5qebgBVbT3V3NNXMSiWSDdGV3hrtzla7J+duwPOToIg42ChPQOQjspnSlp1V+Gjdged7+8UN5CRAV7a5EdFNwCjEaBR27b3W890TE7g24NAP/mMDXRWrGoFPQI9ls/MWO2dWFAar/xcOIImbbpA3zgAAAABJRU5ErkJggg==)}.bk-root .bk-toolbar,.bk-root .bk-toolbar *{box-sizing:border-box;margin:0;padding:0}.bk-root .bk-toolbar-hidden{visibility:hidden;opacity:0;transition:visibility .3s linear,opacity .3s linear}.bk-root .bk-toolbar,.bk-root .bk-button-bar{display:flex;display:-webkit-flex;flex-wrap:nowrap;-webkit-flex-wrap:nowrap;align-items:center;-webkit-align-items:center;user-select:none;-moz-user-select:none;-webkit-user-select:none;-ms-user-select:none}.bk-root .bk-toolbar .bk-logo{flex-shrink:0;-webkit-flex-shrink:0}.bk-root .bk-toolbar-above,.bk-root .bk-toolbar-below{flex-direction:row;-webkit-flex-direction:row;justify-content:flex-end;-webkit-justify-content:flex-end}.bk-root .bk-toolbar-above .bk-button-bar,.bk-root .bk-toolbar-below .bk-button-bar{display:flex;display:-webkit-flex;flex-direction:row;-webkit-flex-direction:row}.bk-root .bk-toolbar-above .bk-logo,.bk-root .bk-toolbar-below .bk-logo{order:1;-webkit-order:1;margin-left:5px}.bk-root .bk-toolbar-left,.bk-root .bk-toolbar-right{flex-direction:column;-webkit-flex-direction:column;justify-content:flex-start;-webkit-justify-content:flex-start}.bk-root .bk-toolbar-left .bk-button-bar,.bk-root .bk-toolbar-right .bk-button-bar{display:flex;display:-webkit-flex;flex-direction:column;-webkit-flex-direction:column}.bk-root .bk-toolbar-left .bk-logo,.bk-root .bk-toolbar-right .bk-logo{order:0;-webkit-order:0;margin-bottom:5px}.bk-root .bk-toolbar-button{width:30px;height:30px;background-size:60%;background-color:transparent;background-repeat:no-repeat;background-position:center center}.bk-root .bk-toolbar-button:hover{background-color:#f9f9f9}.bk-root .bk-toolbar-button:focus{outline:0}.bk-root .bk-toolbar-button::-moz-focus-inner{border:0}.bk-root .bk-toolbar-above .bk-toolbar-button{border-bottom:2px solid transparent}.bk-root .bk-toolbar-above .bk-toolbar-button.bk-active{border-bottom-color:#26aae1}.bk-root .bk-toolbar-below .bk-toolbar-button{border-top:2px solid transparent}.bk-root .bk-toolbar-below .bk-toolbar-button.bk-active{border-top-color:#26aae1}.bk-root .bk-toolbar-right .bk-toolbar-button{border-left:2px solid transparent}.bk-root .bk-toolbar-right .bk-toolbar-button.bk-active{border-left-color:#26aae1}.bk-root .bk-toolbar-left .bk-toolbar-button{border-right:2px solid transparent}.bk-root .bk-toolbar-left .bk-toolbar-button.bk-active{border-right-color:#26aae1}.bk-root .bk-button-bar+.bk-button-bar:before{content:" ";display:inline-block;background-color:lightgray}.bk-root .bk-toolbar-above .bk-button-bar+.bk-button-bar:before,.bk-root .bk-toolbar-below .bk-button-bar+.bk-button-bar:before{height:10px;width:1px}.bk-root .bk-toolbar-left .bk-button-bar+.bk-button-bar:before,.bk-root .bk-toolbar-right .bk-button-bar+.bk-button-bar:before{height:1px;width:10px}.bk-root .bk-tooltip{font-family:"HelveticaNeue-Light","Helvetica Neue Light","Helvetica Neue",Helvetica,Arial,"Lucida Grande",sans-serif;font-weight:300;font-size:12px;position:absolute;padding:5px;border:1px solid #e5e5e5;color:#2f2f2f;background-color:white;pointer-events:none;opacity:.95}.bk-root .bk-tooltip>div:not(:first-child){margin-top:5px;border-top:#e5e5e5 1px dashed}.bk-root .bk-tooltip.bk-left.bk-tooltip-arrow::before{position:absolute;margin:-7px 0 0 0;top:50%;width:0;height:0;border-style:solid;border-width:7px 0 7px 0;border-color:transparent;content:" ";display:block;left:-10px;border-right-width:10px;border-right-color:#909599}.bk-root .bk-tooltip.bk-left::before{left:-10px;border-right-width:10px;border-right-color:#909599}.bk-root .bk-tooltip.bk-right.bk-tooltip-arrow::after{position:absolute;margin:-7px 0 0 0;top:50%;width:0;height:0;border-style:solid;border-width:7px 0 7px 0;border-color:transparent;content:" ";display:block;right:-10px;border-left-width:10px;border-left-color:#909599}.bk-root .bk-tooltip.bk-right::after{right:-10px;border-left-width:10px;border-left-color:#909599}.bk-root .bk-tooltip.bk-above::before{position:absolute;margin:0 0 0 -7px;left:50%;width:0;height:0;border-style:solid;border-width:0 7px 0 7px;border-color:transparent;content:" ";display:block;top:-10px;border-bottom-width:10px;border-bottom-color:#909599}.bk-root .bk-tooltip.bk-below::after{position:absolute;margin:0 0 0 -7px;left:50%;width:0;height:0;border-style:solid;border-width:0 7px 0 7px;border-color:transparent;content:" ";display:block;bottom:-10px;border-top-width:10px;border-top-color:#909599}.bk-root .bk-tooltip-row-label{text-align:right;color:#26aae1}.bk-root .bk-tooltip-row-value{color:default}.bk-root .bk-tooltip-color-block{width:12px;height:12px;margin-left:5px;margin-right:5px;outline:#ddd solid 1px;display:inline-block}.rendered_html .bk-root .bk-tooltip table,.rendered_html .bk-root .bk-tooltip tr,.rendered_html .bk-root .bk-tooltip th,.rendered_html .bk-root .bk-tooltip td{border:0;padding:1px} --------------------------------------------------------------------------------