├── .gitignore ├── images └── selection_vs_extraction.png ├── 01 Feature Selection.ipynb ├── 02 Credit Card Fraud Detection.ipynb └── 03 Sequential Feature Selection Algorithm.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | 3 | # Python 4 | *.pyc 5 | 6 | # Log 7 | *.log 8 | -------------------------------------------------------------------------------- /images/selection_vs_extraction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AndersonJo/feature-selection/master/images/selection_vs_extraction.png -------------------------------------------------------------------------------- /01 Feature Selection.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 289, 6 | "metadata": { 7 | "collapsed": false, 8 | "deletable": true, 9 | "editable": true 10 | }, 11 | "outputs": [ 12 | { 13 | "name": "stdout", 14 | "output_type": "stream", 15 | "text": [ 16 | "Populating the interactive namespace from numpy and matplotlib\n" 17 | ] 18 | } 19 | ], 20 | "source": [ 21 | "%pylab inline\n", 22 | "import numpy as np\n", 23 | "import pandas as pd\n", 24 | "\n", 25 | "from sklearn.datasets import load_iris\n", 26 | "from sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer\n", 27 | "from sklearn.feature_selection import VarianceThreshold, chi2\n", 28 | "from scipy.stats import chisquare\n", 29 | "\n", 30 | "from IPython.display import display" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": { 36 | "deletable": true, 37 | "editable": true 38 | }, 39 | "source": [ 40 | "# Installing Scikit-Feature \n", 41 | "\n", 42 | "http://featureselection.asu.edu/index.php\n", 43 | "\n", 44 | "```\n", 45 | "unzip scikit-feature-1.0.0.zip\n", 46 | "cd scikit-feature-1.0.0\n", 47 | "sudo python3.6 setup.py install\n", 48 | "```" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "# Prerequisite\n", 56 | "\n", 57 | "### Population and Sample\n", 58 | "\n", 59 | "| 용어 | 한국말 | 의미 | 예제 |\n", 60 | "|:----|:-----|:-----|:----|\n", 61 | "| population | 모집단 | 전체 집단을 의미. | 30대 전체 한국 남성 |\n", 62 | "| parameter | 모수 | 전체 population에 대한 summary number를 의미 | 30대 전체 한국 남성의 평균 몸무게
mean $ \\mu $, variance $ \\sigma^2 $, std $ \\sigma $ | \n", 63 | "| sample | 표본 | population으로부터 추출된 population을 대표하는 부분집합 | 30대 전체 한국 남성중 랜덤으로 선택된 100명 |\n", 64 | "| statistic | 통계치 | sample에 대한 summary number | mean $ \\bar{x} $, variance $ s^2 $, std $ s $|\n", 65 | "\n" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "### Confidency Interval" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": { 78 | "deletable": true, 79 | "editable": true 80 | }, 81 | "source": [ 82 | "# Low Variance Filter\n", 83 | "\n", 84 | "가장 쉽게 접근할수 있으며, 모델에 영향력을 갖고 있지 않는 데이터들을 빠르게 제거할 수 있는 방법입니다.
\n", 85 | "VarianceThreshold는 지속적으로 zero-variance features 그리고 지속적으로 동일한 값을 내놓은 features들을 제거합니다.\n", 86 | "\n", 87 | "### Example) Low variance for Bernoulli Distribution\n", 88 | "\n", 89 | "boolean 데이터에 대해서 feature selection을 하려고 합니다.
\n", 90 | "Success(True) 또는 Failure(False)같이 2개로만 나눠지는 경우 **Bernoulli Distribution**을 따릅니다.
\n", 91 | "\n", 92 | "성공(True)값은 $ p $ 로 나타내며, 실패(False)는 $ 1 - p $ 로 정의를 합니다.
\n", 93 | "Bernoulli Random Variable의 Variance구하는 공식은 다음과 같습니다.\n", 94 | "\n", 95 | "$$ Var[X] = p(1-p) $$\n", 96 | "\n", 97 | "아래의 예제에서는 80%이상이 0 또는 1이 지속되는 데이터를 삭제를 합니다." 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 170, 103 | "metadata": { 104 | "collapsed": false, 105 | "deletable": true, 106 | "editable": true 107 | }, 108 | "outputs": [ 109 | { 110 | "name": "stdout", 111 | "output_type": "stream", 112 | "text": [ 113 | "[0]:0.14 \n", 114 | "[1]:0.22 \n", 115 | "[2]:0.25\n" 116 | ] 117 | }, 118 | { 119 | "data": { 120 | "text/plain": [ 121 | "array([[0, 1],\n", 122 | " [1, 0],\n", 123 | " [0, 0],\n", 124 | " [1, 1],\n", 125 | " [1, 0],\n", 126 | " [1, 1]])" 127 | ] 128 | }, 129 | "execution_count": 170, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "def var(data):\n", 136 | " return round(np.var(data), 2)\n", 137 | "\n", 138 | "# List of Variances\n", 139 | "data = np.array([[0, 0, 1], \n", 140 | " [0, 1, 0], \n", 141 | " [1, 0, 0], \n", 142 | " [0, 1, 1], \n", 143 | " [0, 1, 0], \n", 144 | " [0, 1, 1]])\n", 145 | "\n", 146 | "print('[0]:{0} \\n[1]:{1} \\n[2]:{2}'.format(var(data[:, 0]), var(data[:, 1]), var(data[:, 2])))\n", 147 | "\n", 148 | "valsel = VarianceThreshold(threshold=0.8 * (1-0.8)) # 0.8 * (1-0.8) = 0.15999\n", 149 | "valsel.fit_transform(data)" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": { 155 | "deletable": true, 156 | "editable": true 157 | }, 158 | "source": [ 159 | "### Example) Low variance for different variance range\n", 160 | "\n", 161 | "Data1 ~ Data3 는 모두 normal distribution을 따르지만, Data1의 경우 0에 매우 가까우며, Data2의 경우 값 자체가 높습니다.
\n", 162 | "Data1은 Variance Threshold 기준으로 variance값 자체가 작기때문에 feature selection에서 탈락하게 됩니다.\n", 163 | "\n", 164 | "고루고루 값이 분포되어 있지만 variance값이 작아서 탈락이 된다면 Standardization 또는 MinMaxScaling을 해줍니다.
\n", 165 | "[Normalization before low variance](http://www.kdnuggets.com/2015/05/7-methods-data-dimensionality-reduction.html) 를 참고 합니다.\n", 166 | "\n", 167 | "> variance is range dependent; therefore normalization is required before applying this technique\n", 168 | "\n", 169 | "**아래에서는 각 데이터의 variance값의 평균을 구한뒤 20% 보다 적은 variance값을 갖은 데이터를 제거합니다.**" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": 234, 175 | "metadata": { 176 | "collapsed": false, 177 | "deletable": true, 178 | "editable": true 179 | }, 180 | "outputs": [ 181 | { 182 | "name": "stdout", 183 | "output_type": "stream", 184 | "text": [ 185 | "[Before Scaling]\n", 186 | "Data1 Variance: 0.0001\n", 187 | "Data2 Variance: 403.1223\n", 188 | "Data3 Variance: 0.996\n", 189 | "Data4 Variance: 0.002\n", 190 | "\n", 191 | "[After Scaling]\n", 192 | "Data1 Variance: 0.01315\n", 193 | "Data2 Variance: 0.01556\n", 194 | "Data3 Variance: 0.01388\n", 195 | "Data4 Variance: 0.002\n", 196 | "\n", 197 | "Thresold: 0.00222891870716\n" 198 | ] 199 | }, 200 | { 201 | "data": { 202 | "text/plain": [ 203 | "array([[ 0.43043046, 0.3634528 , 0.40977524],\n", 204 | " [ 0.5702359 , 0.4049609 , 0.42229292],\n", 205 | " [ 0.57210535, 0.53095295, 0.566753 ],\n", 206 | " ..., \n", 207 | " [ 0.51042491, 0.298668 , 0.53970593],\n", 208 | " [ 0.56976632, 0.60402107, 0.43984602],\n", 209 | " [ 0.41341096, 0.67872648, 0.39401205]])" 210 | ] 211 | }, 212 | "execution_count": 234, 213 | "metadata": {}, 214 | "output_type": "execute_result" 215 | } 216 | ], 217 | "source": [ 218 | "x = np.random.normal(0, size=(50000, 4))\n", 219 | "x[:, 0] /= 100\n", 220 | "x[:, 1] += 10\n", 221 | "x[:, 1] **= 2\n", 222 | "x[:, 3] = 0\n", 223 | "x[0:100, 3] = 1\n", 224 | "\n", 225 | "print('[Before Scaling]')\n", 226 | "print('Data1 Variance:', round(np.var(x[:, 0]), 5))\n", 227 | "print('Data2 Variance:', round(np.var(x[:, 1]), 4))\n", 228 | "print('Data3 Variance:', round(np.var(x[:, 2]), 4))\n", 229 | "print('Data4 Variance:', round(np.var(x[:, 3]), 4))\n", 230 | "\n", 231 | "x = MinMaxScaler().fit_transform(x)\n", 232 | "\n", 233 | "print('\\n[After Scaling]')\n", 234 | "print('Data1 Variance:', round(np.var(x[:, 0]), 5))\n", 235 | "print('Data2 Variance:', round(np.var(x[:, 1]), 5))\n", 236 | "print('Data3 Variance:', round(np.var(x[:, 2]), 5))\n", 237 | "print('Data4 Variance:', round(np.var(x[:, 3]), 5))\n", 238 | "\n", 239 | "threshold = np.mean([np.var(x[:, i]) for i in range(4)]) *0.2\n", 240 | "\n", 241 | "print('\\nThresold:', threshold)\n", 242 | "low_variance_feature_selection(x, threshold=threshold)" 243 | ] 244 | }, 245 | { 246 | "cell_type": "markdown", 247 | "metadata": { 248 | "deletable": true, 249 | "editable": true 250 | }, 251 | "source": [ 252 | "# Chi Sqaure Test\n", 253 | "\n", 254 | "Chi sqaure test는 일반적으로 두가지 상황에서 사용이 될 수 있습니다.
\n", 255 | "첫번째로는 Goodness-of-fit test(적합도 검정)로서 관측된 데이터가 예측한 분포를 따르는지 검정하는 방법입니다.
\n", 256 | "\n", 257 | "\n", 258 | "$$ \\chi^2_c = \\sum \\frac{ (O_i - E_i)^2 }{E_i} $$\n", 259 | "\n", 260 | "* subscript $ c $ : **the degree of freedom** \n", 261 | "* $ O $ : observed value (관측값) \n", 262 | "* $ E $ : expected value \n", 263 | "\n", 264 | "Chi square는 observed value와 expected value사이가 얼마나 다른지를 타나내며, **categorical variables** 에서 사용됩니다.
\n", 265 | "Chi square는 몇가지 variations들이 존재하며, 데이터 그리고 hypothesis에 따라서 적용이 달라집니다.\n", 266 | "\n", 267 | "계산된 결과값이 낮으면 2개의 데이터셋 사이에는 높은 연관성이 존재하며,
\n", 268 | "observed value와 expected value가 서로 완전히 동일하다면 (no difference) chi-square의 값은 0이 됩니다. " 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": 356, 274 | "metadata": { 275 | "collapsed": false, 276 | "deletable": true, 277 | "editable": true 278 | }, 279 | "outputs": [ 280 | { 281 | "data": { 282 | "text/html": [ 283 | "
\n", 284 | "\n", 297 | "\n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | "
blackred
196
2105
31214
41115
5811
6109
\n", 338 | "
" 339 | ], 340 | "text/plain": [ 341 | " black red\n", 342 | "1 9 6\n", 343 | "2 10 5\n", 344 | "3 12 14\n", 345 | "4 11 15\n", 346 | "5 8 11\n", 347 | "6 10 9" 348 | ] 349 | }, 350 | "metadata": {}, 351 | "output_type": "display_data" 352 | }, 353 | { 354 | "name": "stdout", 355 | "output_type": "stream", 356 | "text": [ 357 | "[Black]\tX-squared: 0.0 \tp-value: 1.0\n", 358 | "[Red]\tX-squared: 8.4 \tp-value: 0.1355\n", 359 | "6.51287878788\n" 360 | ] 361 | } 362 | ], 363 | "source": [ 364 | "# Data\n", 365 | "data = pd.DataFrame({'black': [9, 10, 12, 11, 8, 10], \n", 366 | " 'red': [6, 5, 14, 15, 11, 9]}, \n", 367 | " index=range(1, 7))\n", 368 | "display(data)\n", 369 | "black = data.black.values\n", 370 | "red = data.red.values\n", 371 | "\n", 372 | "def my_chisquare(obs, exp):\n", 373 | " return np.sum((obs - exp)**2/exp)\n", 374 | "\n", 375 | "# Calculate Chi-Square\n", 376 | "chi_value, p_value = chisquare(black, black)\n", 377 | "print('[Black]\\tX-squared:', chi_value, '\\tp-value:', round(p_value, 4))\n", 378 | "\n", 379 | "chi_value, p_value = chisquare(red)\n", 380 | "print('[Red]\\tX-squared:', chi_value, '\\tp-value:', round(p_value, 4))\n", 381 | "\n", 382 | "print( my_chisquare(red, black))" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": 341, 388 | "metadata": { 389 | "collapsed": false, 390 | "deletable": true, 391 | "editable": true 392 | }, 393 | "outputs": [ 394 | { 395 | "data": { 396 | "text/plain": [ 397 | "(array([ 1.]), array([ 0.96256577]))" 398 | ] 399 | }, 400 | "execution_count": 341, 401 | "metadata": {}, 402 | "output_type": "execute_result" 403 | } 404 | ], 405 | "source": [ 406 | "black = black\n", 407 | "red = red\n", 408 | "\n", 409 | "chi2(black.reshape(-1, 1), red)" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "metadata": { 415 | "deletable": true, 416 | "editable": true 417 | }, 418 | "source": [ 419 | "# T-Test" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": null, 425 | "metadata": { 426 | "collapsed": true, 427 | "deletable": true, 428 | "editable": true 429 | }, 430 | "outputs": [], 431 | "source": [] 432 | } 433 | ], 434 | "metadata": { 435 | "kernelspec": { 436 | "display_name": "Python 3", 437 | "language": "python", 438 | "name": "python3" 439 | }, 440 | "language_info": { 441 | "codemirror_mode": { 442 | "name": "ipython", 443 | "version": 3 444 | }, 445 | "file_extension": ".py", 446 | "mimetype": "text/x-python", 447 | "name": "python", 448 | "nbconvert_exporter": "python", 449 | "pygments_lexer": "ipython3", 450 | "version": "3.6.1" 451 | } 452 | }, 453 | "nbformat": 4, 454 | "nbformat_minor": 2 455 | } 456 | -------------------------------------------------------------------------------- /02 Credit Card Fraud Detection.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Credit Card Fraud Detection " 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "Populating the interactive namespace from numpy and matplotlib\n" 20 | ] 21 | }, 22 | { 23 | "name": "stderr", 24 | "output_type": "stream", 25 | "text": [ 26 | "Using TensorFlow backend.\n", 27 | "/usr/local/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n", 28 | " \"This module will be removed in 0.20.\", DeprecationWarning)\n" 29 | ] 30 | } 31 | ], 32 | "source": [ 33 | "%pylab inline\n", 34 | "from keras.models import Sequential\n", 35 | "from keras.layers import Dense, Dropout, Activation, Embedding, LSTM\n", 36 | "\n", 37 | "from sklearn.cross_validation import train_test_split\n", 38 | "from sklearn.linear_model import LogisticRegression\n", 39 | "from sklearn.tree import DecisionTreeClassifier\n", 40 | "from sklearn.metrics import accuracy_score\n", 41 | "from sklearn.svm import SVC\n", 42 | "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n", 43 | "from sklearn.utils import resample\n", 44 | "from sklearn.model_selection import cross_val_score\n", 45 | "\n", 46 | "from keras.models import Sequential\n", 47 | "from keras.layers import Dense, Activation, Dropout, BatchNormalization\n", 48 | "from keras.layers.advanced_activations import LeakyReLU\n", 49 | "from keras.optimizers import SGD, RMSprop\n", 50 | "from keras.regularizers import l2\n", 51 | "from keras.backend.tensorflow_backend import set_session\n", 52 | "from keras.callbacks import EarlyStopping\n", 53 | "from keras.utils import np_utils\n", 54 | "import keras\n", 55 | "\n", 56 | "import tensorflow as tf\n", 57 | "import pandas as pd\n", 58 | "import numpy as np\n", 59 | "\n", 60 | "\n", 61 | "from IPython.display import SVG, Image\n", 62 | "from keras.utils.visualize_util import model_to_dot\n", 63 | "\n", 64 | "np.random.seed(0)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "## Data\n", 72 | "\n", 73 | "[Credit Card Fraud Detection - Kaggle](https://www.kaggle.com/dalpozz/creditcardfraud)에서 다운받을수 있습니다.\n", 74 | "\n", 75 | "데이터는 2013년 유럽 카드회사의 이틀동안 일어난 transactions에 관한 것이며,
\n", 76 | "492건의 frauds 가 284,807건의 transactions중에 일어 났습니다.\n", 77 | "\n", 78 | "Class에서 1은 fraud를 뜻하며, 0은 아닌것을 말합니다.\n", 79 | "\n", 80 | "Time데이터는 첫번재 Column으로부터 몇초 이후에 발생한 transaction이라는 뜻입니다.
\n", 81 | "나머지 데이터들은 PCA의 규제에 의해서 어떤 데이터인지 밝히지 않습니다." 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 2, 87 | "metadata": { 88 | "collapsed": true 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "data = pd.read_csv('/dataset/credit-card-fraud-detection/creditcard.csv')\n", 93 | "\n", 94 | "# Preprocessing Amount\n", 95 | "amt_scale = MinMaxScaler()\n", 96 | "data['NormAmount'] = amt_scale.fit_transform(data['Amount'].values.reshape(-1, 1))\n", 97 | "\n", 98 | "# Split Train and Test Data\n", 99 | "X = data.drop(['Time', 'Amount', 'Class'], axis=1).as_matrix()\n", 100 | "Y = data['Class'].as_matrix()\n", 101 | "\n", 102 | "# Standardization\n", 103 | "scale_x = MinMaxScaler()\n", 104 | "X = scale_x.fit_transform(X)\n", 105 | "\n", 106 | "train_x, test_x, train_y, test_y = train_test_split(X, Y, test_size=0.25, random_state=1)\n", 107 | "\n", 108 | "fraud_test_y = test_y == 1\n", 109 | "fraud_test_x = test_x[fraud_test_y]\n", 110 | "fraud_test_y = test_y[fraud_test_y]\n", 111 | "\n", 112 | "train_category_y = np_utils.to_categorical(train_y)\n", 113 | "test_category_y = np_utils.to_categorical(test_y)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "#### Checking the number of fraud transactions in training and test data" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": 3, 126 | "metadata": {}, 127 | "outputs": [ 128 | { 129 | "name": "stdout", 130 | "output_type": "stream", 131 | "text": [ 132 | "The number of Fraud transactions in Training Data: 381\n", 133 | "The number of Fraud transactions in Test Data: 111\n" 134 | ] 135 | } 136 | ], 137 | "source": [ 138 | "print('The number of Fraud transactions in Training Data:', train_y[train_y == 1].shape[0])\n", 139 | "print('The number of Fraud transactions in Test Data:', test_y[test_y == 1].shape[0])" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "#### Checking the target classes\n", 147 | "\n", 148 | "fraud transactions이 492개밖에 되지 않기 때문에, 일반적인 classification algorithm으로 돌리면 물론 정확도는 매우 높게 나오지만.. \n", 149 | "실상은 1에 해당하는 fraud transactions에서는 대부분 틀릴 가능성이 매우 높습니다. \n" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 4, 155 | "metadata": {}, 156 | "outputs": [ 157 | { 158 | "data": { 159 | "text/plain": [ 160 | "0 284315\n", 161 | "1 492\n", 162 | "Name: Class, dtype: int64" 163 | ] 164 | }, 165 | "execution_count": 4, 166 | "metadata": {}, 167 | "output_type": "execute_result" 168 | } 169 | ], 170 | "source": [ 171 | "pd.value_counts(data['Class'], sort=True)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "## Resampling\n", 179 | "\n", 180 | "resampling에는 여러가지 방법이 있습니다. \n", 181 | "\n", 182 | "1. Over Sampling: SMOTE (Synthetic Minority Over-Sampling Technique)\n", 183 | "2. Under Sampling\n", 184 | "\n", 185 | "아래의 resample function에서는 5:5의 비율로 under sampling을 해줍니다.
\n", 186 | "resample을 하면서 시간관계가 어차피 깨지기 때문에 (사실 각각의 transactions들 사이에 상관관계가 있는지도 모르겠음)
\n", 187 | "shuffle을 통해서 train되는 데이터를 augment해줍니다." 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 5, 193 | "metadata": {}, 194 | "outputs": [ 195 | { 196 | "name": "stdout", 197 | "output_type": "stream", 198 | "text": [ 199 | "resampled_train_x: (762, 29)\n", 200 | "resampled_train_y: (762,)\n" 201 | ] 202 | } 203 | ], 204 | "source": [ 205 | "def resample(X, Y, ratio=1.):\n", 206 | " index = np.arange(Y.shape[0])\n", 207 | " fraud_indices = index[Y == 1]\n", 208 | " normal_indices = index[Y == 0]\n", 209 | " normal_n = int(len(fraud_indices) * ratio)\n", 210 | " \n", 211 | " random_normal_indices = np.random.permutation(normal_indices)[:normal_n]\n", 212 | " \n", 213 | " sample_indices = np.concatenate([fraud_indices, random_normal_indices])\n", 214 | " np.random.shuffle(sample_indices)\n", 215 | " sample_indices = np.array(sample_indices)\n", 216 | " \n", 217 | " sample_x = X[sample_indices]\n", 218 | " sample_y = Y[sample_indices]\n", 219 | " return sample_x, sample_y\n", 220 | "\n", 221 | "resampled_train_x, resampled_train_y = resample(train_x, train_y)\n", 222 | "\n", 223 | "print('resampled_train_x:', resampled_train_x.shape)\n", 224 | "print('resampled_train_y:', resampled_train_y.shape)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "# Logistic Regression\n", 232 | "\n", 233 | "전체적으로 0.99% accuracy를 보이지만, 실제 fraud data만 test를 했을때는 0.57%로.. 실질적으로 못맞추는 수준입니다.
\n", 234 | "사실 일반적인 알고리즘으로 학습시키기 위해서는 over sampling (SMOTE 같은) 또는 under sampling이 필요합니다.
\n", 235 | "sampling을 통해서 skewed data를 보정하는 것입니다.\n", 236 | "\n", 237 | "#### resample 없이 데이터 학습뒤 예측하면.." 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 6, 243 | "metadata": {}, 244 | "outputs": [ 245 | { 246 | "data": { 247 | "text/plain": [ 248 | "0.99908710429482317" 249 | ] 250 | }, 251 | "execution_count": 6, 252 | "metadata": {}, 253 | "output_type": "execute_result" 254 | } 255 | ], 256 | "source": [ 257 | "lg = LogisticRegression()\n", 258 | "lg.fit(train_x, train_y)\n", 259 | "predicted_y = lg.predict(test_x)\n", 260 | "accuracy_score(test_y, predicted_y)" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": 7, 266 | "metadata": {}, 267 | "outputs": [ 268 | { 269 | "data": { 270 | "text/plain": [ 271 | "0.51351351351351349" 272 | ] 273 | }, 274 | "execution_count": 7, 275 | "metadata": {}, 276 | "output_type": "execute_result" 277 | } 278 | ], 279 | "source": [ 280 | "predicted_y = lg.predict(fraud_test_x)\n", 281 | "accuracy_score(fraud_test_y, predicted_y)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": {}, 287 | "source": [ 288 | "#### resampled data로 학습뒤 예측하면..." 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 8, 294 | "metadata": {}, 295 | "outputs": [ 296 | { 297 | "data": { 298 | "text/plain": [ 299 | "0.9970506446448133" 300 | ] 301 | }, 302 | "execution_count": 8, 303 | "metadata": {}, 304 | "output_type": "execute_result" 305 | } 306 | ], 307 | "source": [ 308 | "lg = LogisticRegression()\n", 309 | "lg.fit(*resample(train_x, train_y))\n", 310 | "\n", 311 | "predicted_y = lg.predict(test_x)\n", 312 | "accuracy_score(test_y, predicted_y)" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": 9, 318 | "metadata": {}, 319 | "outputs": [ 320 | { 321 | "data": { 322 | "text/plain": [ 323 | "0.77477477477477474" 324 | ] 325 | }, 326 | "execution_count": 9, 327 | "metadata": {}, 328 | "output_type": "execute_result" 329 | } 330 | ], 331 | "source": [ 332 | "predicted_y = lg.predict(fraud_test_x)\n", 333 | "accuracy_score(fraud_test_y, predicted_y)" 334 | ] 335 | }, 336 | { 337 | "cell_type": "markdown", 338 | "metadata": {}, 339 | "source": [ 340 | "# Decision Tree\n", 341 | "\n", 342 | "#### resample 없이 데이터 학습뒤 예측하면.." 343 | ] 344 | }, 345 | { 346 | "cell_type": "code", 347 | "execution_count": 10, 348 | "metadata": {}, 349 | "outputs": [ 350 | { 351 | "data": { 352 | "text/plain": [ 353 | "0.99925563888654811" 354 | ] 355 | }, 356 | "execution_count": 10, 357 | "metadata": {}, 358 | "output_type": "execute_result" 359 | } 360 | ], 361 | "source": [ 362 | "dtc = DecisionTreeClassifier(max_depth=10, criterion='entropy')\n", 363 | "dtc.fit(train_x, train_y)\n", 364 | "predicted_y = dtc.predict(test_x)\n", 365 | "accuracy_score(test_y, predicted_y)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 11, 371 | "metadata": {}, 372 | "outputs": [ 373 | { 374 | "data": { 375 | "text/plain": [ 376 | "0.72972972972972971" 377 | ] 378 | }, 379 | "execution_count": 11, 380 | "metadata": {}, 381 | "output_type": "execute_result" 382 | } 383 | ], 384 | "source": [ 385 | "predicted_y = dtc.predict(fraud_test_x)\n", 386 | "accuracy_score(fraud_test_y, predicted_y)" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "#### resampled data로 학습뒤 예측하면..." 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": 12, 399 | "metadata": {}, 400 | "outputs": [ 401 | { 402 | "name": "stdout", 403 | "output_type": "stream", 404 | "text": [ 405 | "0.899974719811\n" 406 | ] 407 | } 408 | ], 409 | "source": [ 410 | "dtc = DecisionTreeClassifier(max_depth=10, criterion='entropy')\n", 411 | "dtc.fit(*resample(train_x, train_y))\n", 412 | "predicted_y = dtc.predict(test_x)\n", 413 | "print(accuracy_score(test_y, predicted_y))" 414 | ] 415 | }, 416 | { 417 | "cell_type": "code", 418 | "execution_count": 13, 419 | "metadata": {}, 420 | "outputs": [ 421 | { 422 | "data": { 423 | "text/plain": [ 424 | "0.95495495495495497" 425 | ] 426 | }, 427 | "execution_count": 13, 428 | "metadata": {}, 429 | "output_type": "execute_result" 430 | } 431 | ], 432 | "source": [ 433 | "predicted_y = dtc.predict(fraud_test_x)\n", 434 | "accuracy_score(fraud_test_y, predicted_y)" 435 | ] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "metadata": {}, 440 | "source": [ 441 | "# Deep Learning with Keras\n", 442 | "\n", 443 | "하.. 드디어 딥러닝으로.. 해보면 어떤 결과가 나올 것인가.. Sampling VS UnSampling!" 444 | ] 445 | }, 446 | { 447 | "cell_type": "markdown", 448 | "metadata": {}, 449 | "source": [ 450 | "## Model" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 14, 456 | "metadata": { 457 | "collapsed": true 458 | }, 459 | "outputs": [], 460 | "source": [ 461 | "config = tf.ConfigProto()\n", 462 | "config.gpu_options.per_process_gpu_memory_fraction = 0.1\n", 463 | "set_session(tf.InteractiveSession(config=config))" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 15, 469 | "metadata": { 470 | "scrolled": false 471 | }, 472 | "outputs": [ 473 | { 474 | "name": "stdout", 475 | "output_type": "stream", 476 | "text": [ 477 | "_________________________________________________________________\n", 478 | "Layer (type) Output Shape Param # \n", 479 | "=================================================================\n", 480 | "dense_1 (Dense) (None, 256) 7680 \n", 481 | "_________________________________________________________________\n", 482 | "batch_normalization_1 (Batch (None, 256) 1024 \n", 483 | "_________________________________________________________________\n", 484 | "activation_1 (Activation) (None, 256) 0 \n", 485 | "_________________________________________________________________\n", 486 | "dropout_1 (Dropout) (None, 256) 0 \n", 487 | "_________________________________________________________________\n", 488 | "dense_2 (Dense) (None, 160) 41120 \n", 489 | "_________________________________________________________________\n", 490 | "batch_normalization_2 (Batch (None, 160) 640 \n", 491 | "_________________________________________________________________\n", 492 | "activation_2 (Activation) (None, 160) 0 \n", 493 | "_________________________________________________________________\n", 494 | "dropout_2 (Dropout) (None, 160) 0 \n", 495 | "_________________________________________________________________\n", 496 | "dense_3 (Dense) (None, 128) 20608 \n", 497 | "_________________________________________________________________\n", 498 | "batch_normalization_3 (Batch (None, 128) 512 \n", 499 | "_________________________________________________________________\n", 500 | "activation_3 (Activation) (None, 128) 0 \n", 501 | "_________________________________________________________________\n", 502 | "dropout_3 (Dropout) (None, 128) 0 \n", 503 | "_________________________________________________________________\n", 504 | "dense_4 (Dense) (None, 96) 12384 \n", 505 | "_________________________________________________________________\n", 506 | "batch_normalization_4 (Batch (None, 96) 384 \n", 507 | "_________________________________________________________________\n", 508 | "activation_4 (Activation) (None, 96) 0 \n", 509 | "_________________________________________________________________\n", 510 | "dropout_4 (Dropout) (None, 96) 0 \n", 511 | "_________________________________________________________________\n", 512 | "dense_5 (Dense) (None, 1) 97 \n", 513 | "_________________________________________________________________\n", 514 | "activation_5 (Activation) (None, 1) 0 \n", 515 | "=================================================================\n", 516 | "Total params: 84,449\n", 517 | "Trainable params: 83,169\n", 518 | "Non-trainable params: 1,280\n", 519 | "_________________________________________________________________\n" 520 | ] 521 | } 522 | ], 523 | "source": [ 524 | "def generate_model():\n", 525 | " np.random.seed(0)\n", 526 | " model = Sequential()\n", 527 | " model.add(Dense(256, input_dim=29))\n", 528 | " model.add(BatchNormalization())\n", 529 | " model.add(Activation('relu'))\n", 530 | " model.add(Dropout(0.5))\n", 531 | " \n", 532 | " model.add(Dense(160))\n", 533 | " model.add(BatchNormalization())\n", 534 | " model.add(Activation('relu'))\n", 535 | " model.add(Dropout(0.4))\n", 536 | " \n", 537 | " model.add(Dense(128))\n", 538 | " model.add(BatchNormalization())\n", 539 | " model.add(Activation('relu'))\n", 540 | " model.add(Dropout(0.3))\n", 541 | " \n", 542 | " model.add(Dense(96))\n", 543 | " model.add(BatchNormalization())\n", 544 | " model.add(Activation('relu'))\n", 545 | " model.add(Dropout(0.2))\n", 546 | "\n", 547 | " model.add(Dense(1))\n", 548 | " model.add(Activation('sigmoid'))\n", 549 | "\n", 550 | " model.compile(loss='binary_crossentropy', \n", 551 | " optimizer='adam', \n", 552 | " metrics=['accuracy'])\n", 553 | " return model\n", 554 | "\n", 555 | "\n", 556 | "# # Visualization\n", 557 | "model = generate_model()\n", 558 | "model.summary()\n", 559 | "# SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg'))\n" 560 | ] 561 | }, 562 | { 563 | "cell_type": "markdown", 564 | "metadata": {}, 565 | "source": [ 566 | "#### resample 없이 데이터 학습뒤 예측하면.." 567 | ] 568 | }, 569 | { 570 | "cell_type": "code", 571 | "execution_count": 16, 572 | "metadata": { 573 | "scrolled": false 574 | }, 575 | "outputs": [ 576 | { 577 | "name": "stdout", 578 | "output_type": "stream", 579 | "text": [ 580 | "Epoch 1/10\n", 581 | "27s - loss: 0.0100 - acc: 0.9980\n", 582 | "Epoch 2/10\n", 583 | "27s - loss: 0.0041 - acc: 0.9993\n", 584 | "Epoch 3/10\n", 585 | "27s - loss: 0.0038 - acc: 0.9993\n", 586 | "Epoch 4/10\n", 587 | "27s - loss: 0.0036 - acc: 0.9994\n", 588 | "Epoch 5/10\n", 589 | "27s - loss: 0.0034 - acc: 0.9993\n", 590 | "Epoch 6/10\n", 591 | "27s - loss: 0.0035 - acc: 0.9993\n", 592 | "Epoch 7/10\n", 593 | "27s - loss: 0.0034 - acc: 0.9994\n", 594 | "Epoch 8/10\n", 595 | "27s - loss: 0.0034 - acc: 0.9994\n", 596 | "Epoch 9/10\n", 597 | "27s - loss: 0.0034 - acc: 0.9994\n", 598 | "Epoch 10/10\n", 599 | "27s - loss: 0.0033 - acc: 0.9994\n" 600 | ] 601 | }, 602 | { 603 | "data": { 604 | "text/plain": [ 605 | "" 606 | ] 607 | }, 608 | "execution_count": 16, 609 | "metadata": {}, 610 | "output_type": "execute_result" 611 | } 612 | ], 613 | "source": [ 614 | "model = generate_model()\n", 615 | "model.fit(train_x, train_y, verbose=2)" 616 | ] 617 | }, 618 | { 619 | "cell_type": "code", 620 | "execution_count": 17, 621 | "metadata": {}, 622 | "outputs": [ 623 | { 624 | "name": "stdout", 625 | "output_type": "stream", 626 | "text": [ 627 | "0.999311817084\n" 628 | ] 629 | } 630 | ], 631 | "source": [ 632 | "predicted_y = model.predict(test_x)\n", 633 | "predicted_y = predicted_y.reshape(predicted_y.shape[0])\n", 634 | "predicted_y = np.where(predicted_y >= 0.5, 1, 0)\n", 635 | "print(accuracy_score(test_y, predicted_y))" 636 | ] 637 | }, 638 | { 639 | "cell_type": "code", 640 | "execution_count": 18, 641 | "metadata": {}, 642 | "outputs": [ 643 | { 644 | "data": { 645 | "text/plain": [ 646 | "0.74774774774774777" 647 | ] 648 | }, 649 | "execution_count": 18, 650 | "metadata": {}, 651 | "output_type": "execute_result" 652 | } 653 | ], 654 | "source": [ 655 | "predicted_y = model.predict(fraud_test_x)\n", 656 | "predicted_y = predicted_y.reshape(predicted_y.shape[0])\n", 657 | "predicted_y = np.where(predicted_y >= 0.5, 1, 0)\n", 658 | "accuracy_score(fraud_test_y, predicted_y)" 659 | ] 660 | }, 661 | { 662 | "cell_type": "markdown", 663 | "metadata": {}, 664 | "source": [ 665 | "#### resampled data로 학습뒤 예측하면..." 666 | ] 667 | }, 668 | { 669 | "cell_type": "code", 670 | "execution_count": 19, 671 | "metadata": {}, 672 | "outputs": [ 673 | { 674 | "name": "stdout", 675 | "output_type": "stream", 676 | "text": [ 677 | "[ 1] epoch:50 loss:0.1747 acc:0.9351 \n", 678 | "[ 2] epoch:50 loss:0.1154 acc:0.957 \n", 679 | "[ 3] epoch:50 loss:0.09408 acc:0.9642 \n", 680 | "[ 4] epoch:50 loss:0.09055 acc:0.9646 \n", 681 | "[ 5] epoch:50 loss:0.06973 acc:0.9732 \n", 682 | "[ 6] epoch:50 loss:0.06866 acc:0.9734 \n", 683 | "[ 7] epoch:50 loss:0.05985 acc:0.9764 \n", 684 | "[ 8] epoch:50 loss:0.05556 acc:0.9785 \n", 685 | "[ 9] epoch:50 loss:0.05241 acc:0.9789 \n", 686 | "[10] epoch:50 loss:0.04598 acc:0.9827 \n", 687 | "[11] epoch:50 loss:0.04265 acc:0.9832 \n", 688 | "[12] epoch:50 loss:0.03687 acc:0.9853 \n", 689 | "[13] epoch:50 loss:0.04159 acc:0.9841 \n", 690 | "[14] epoch:50 loss:0.03852 acc:0.9851 \n", 691 | "[15] epoch:50 loss:0.03585 acc:0.9861 \n", 692 | "[16] epoch:50 loss:0.04974 acc:0.9817 \n", 693 | "[17] epoch:50 loss:0.03046 acc:0.9885 \n", 694 | "[18] epoch:50 loss:0.03561 acc:0.9875 \n", 695 | "[19] epoch:50 loss:0.03836 acc:0.986 \n", 696 | "[20] epoch:50 loss:0.02893 acc:0.9888 \n", 697 | "[21] epoch:50 loss:0.0328 acc:0.9882 \n", 698 | "[22] epoch:50 loss:0.02712 acc:0.9898 \n", 699 | "[23] epoch:50 loss:0.03774 acc:0.9865 \n", 700 | "[24] epoch:50 loss:0.02368 acc:0.991 \n", 701 | "[25] epoch:50 loss:0.02782 acc:0.9902 \n", 702 | "[26] epoch:50 loss:0.02484 acc:0.9912 \n", 703 | "[27] epoch:50 loss:0.02646 acc:0.9899 \n", 704 | "[28] epoch:50 loss:0.03993 acc:0.9902 \n", 705 | "[29] epoch:50 loss:0.03121 acc:0.9904 \n", 706 | "[30] epoch:50 loss:0.02874 acc:0.9895 \n", 707 | "[31] epoch:50 loss:0.02508 acc:0.9908 \n", 708 | "[32] epoch:50 loss:0.02123 acc:0.9922 \n", 709 | "[33] epoch:50 loss:0.02777 acc:0.9896 \n", 710 | "[34] epoch:50 loss:0.02843 acc:0.9895 \n", 711 | "[35] epoch:50 loss:0.0226 acc:0.9919 \n", 712 | "[36] epoch:50 loss:0.0198 acc:0.9927 \n", 713 | "[37] epoch:50 loss:0.01881 acc:0.9933 \n", 714 | "[38] epoch:50 loss:0.02115 acc:0.9926 \n", 715 | "[39] epoch:50 loss:0.02356 acc:0.9915 \n", 716 | "[40] epoch:50 loss:0.0213 acc:0.993 \n", 717 | "[41] epoch:50 loss:0.0213 acc:0.9927 \n", 718 | "[42] epoch:50 loss:0.01553 acc:0.9946 \n", 719 | "[43] epoch:50 loss:0.02371 acc:0.9913 \n", 720 | "[44] epoch:50 loss:0.01749 acc:0.9938 \n", 721 | "[45] epoch:50 loss:0.02064 acc:0.9931 \n", 722 | "[46] epoch:50 loss:0.0221 acc:0.9923 \n", 723 | "[47] epoch:50 loss:0.02131 acc:0.9923 \n", 724 | "[48] epoch:50 loss:0.02364 acc:0.9917 \n", 725 | "[49] epoch:50 loss:0.0204 acc:0.9928 \n", 726 | "[50] epoch:50 loss:0.02468 acc:0.9928 \n", 727 | "[51] epoch:50 loss:0.02787 acc:0.9907 \n", 728 | "[52] epoch:50 loss:0.02036 acc:0.9931 \n", 729 | "[53] epoch:50 loss:0.015 acc:0.9948 \n", 730 | "[54] epoch:50 loss:0.01952 acc:0.993 \n", 731 | "[55] epoch:50 loss:0.02051 acc:0.9928 \n", 732 | "[56] epoch:50 loss:0.01304 acc:0.9953 \n", 733 | "[57] epoch:50 loss:0.02302 acc:0.9912 \n", 734 | "[58] epoch:50 loss:0.02222 acc:0.9923 \n", 735 | "[59] epoch:50 loss:0.02169 acc:0.9924 \n", 736 | "[60] epoch:50 loss:0.01611 acc:0.9939 \n", 737 | "[61] epoch:50 loss:0.01526 acc:0.9949 \n", 738 | "[62] epoch:50 loss:0.02123 acc:0.9926 \n", 739 | "[63] epoch:50 loss:0.02727 acc:0.9909 \n", 740 | "[64] epoch:50 loss:0.01674 acc:0.9944 \n", 741 | "[65] epoch:50 loss:0.02663 acc:0.9933 \n", 742 | "[66] epoch:50 loss:0.01323 acc:0.9957 \n", 743 | "[67] epoch:50 loss:0.01539 acc:0.9948 \n", 744 | "[68] epoch:50 loss:0.01403 acc:0.9949 \n", 745 | "[69] epoch:50 loss:0.01818 acc:0.994 \n", 746 | "[70] epoch:50 loss:0.02583 acc:0.992 \n", 747 | "[71] epoch:50 loss:0.0177 acc:0.9939 \n", 748 | "[72] epoch:50 loss:0.02266 acc:0.9922 \n", 749 | "[73] epoch:50 loss:0.01834 acc:0.9937 \n", 750 | "[74] epoch:50 loss:0.02084 acc:0.9924 \n", 751 | "[75] epoch:50 loss:0.0177 acc:0.9936 \n", 752 | "[76] epoch:50 loss:0.01589 acc:0.9946 \n", 753 | "[77] epoch:50 loss:0.0155 acc:0.9946 \n", 754 | "[78] epoch:50 loss:0.01654 acc:0.9937 \n", 755 | "[79] epoch:50 loss:0.01753 acc:0.9938 \n", 756 | "[80] epoch:50 loss:0.02277 acc:0.9924 \n", 757 | "[81] epoch:50 loss:0.0181 acc:0.9938 \n", 758 | "[82] epoch:50 loss:0.01394 acc:0.9949 \n", 759 | "[83] epoch:50 loss:0.01233 acc:0.9963 \n", 760 | "[84] epoch:50 loss:0.02108 acc:0.9949 \n", 761 | "[85] epoch:50 loss:0.01739 acc:0.9942 \n", 762 | "[86] epoch:50 loss:0.01714 acc:0.9941 \n", 763 | "[87] epoch:50 loss:0.01889 acc:0.994 \n", 764 | "[88] epoch:50 loss:0.01492 acc:0.9947 \n", 765 | "[89] epoch:50 loss:0.01542 acc:0.9948 \n", 766 | "[90] epoch:50 loss:0.01788 acc:0.994 \n", 767 | "[91] epoch:50 loss:0.02462 acc:0.9915 \n", 768 | "[92] epoch:50 loss:0.01342 acc:0.9953 \n", 769 | "[93] epoch:50 loss:0.01917 acc:0.9935 \n", 770 | "[94] epoch:50 loss:0.01176 acc:0.9957 \n", 771 | "[95] epoch:50 loss:0.01745 acc:0.9946 \n", 772 | "[96] epoch:50 loss:0.01573 acc:0.9945 \n", 773 | "[97] epoch:50 loss:0.0196 acc:0.9933 \n", 774 | "[98] epoch:50 loss:0.01235 acc:0.9955 \n", 775 | "[99] epoch:50 loss:0.01158 acc:0.9957 \n", 776 | "[100] epoch:50 loss:0.01595 acc:0.9941 \n" 777 | ] 778 | } 779 | ], 780 | "source": [ 781 | "# # Visualization\n", 782 | "model = generate_model()\n", 783 | "early_stopping = EarlyStopping(monitor='loss', patience=10)\n", 784 | "\n", 785 | "for i in range(100):\n", 786 | " _train_data = resample(train_x, train_y, ratio=1)\n", 787 | " \n", 788 | " history = model.fit(*_train_data,\n", 789 | " verbose=0, \n", 790 | " epochs=50,)\n", 791 | "# callbacks=[early_stopping])\n", 792 | " loss = np.mean(history.history.get('loss'))\n", 793 | " acc = np.mean(history.history.get('acc'))\n", 794 | " epoch = len(history.epoch)\n", 795 | " print(f'[{i+1:2}] epoch:{epoch:<2} loss:{loss:<8.4} acc:{acc:<8.4}')" 796 | ] 797 | }, 798 | { 799 | "cell_type": "code", 800 | "execution_count": 20, 801 | "metadata": {}, 802 | "outputs": [ 803 | { 804 | "name": "stdout", 805 | "output_type": "stream", 806 | "text": [ 807 | "0.997457936575\n" 808 | ] 809 | } 810 | ], 811 | "source": [ 812 | "predicted_y = model.predict(test_x)\n", 813 | "predicted_y = predicted_y.reshape(predicted_y.shape[0])\n", 814 | "predicted_y = np.where(predicted_y >= 0.5, 1, 0)\n", 815 | "print(accuracy_score(test_y, predicted_y))" 816 | ] 817 | }, 818 | { 819 | "cell_type": "code", 820 | "execution_count": 21, 821 | "metadata": {}, 822 | "outputs": [ 823 | { 824 | "data": { 825 | "text/plain": [ 826 | "0.8288288288288288" 827 | ] 828 | }, 829 | "execution_count": 21, 830 | "metadata": {}, 831 | "output_type": "execute_result" 832 | } 833 | ], 834 | "source": [ 835 | "predicted_y = model.predict(fraud_test_x)\n", 836 | "predicted_y = predicted_y.reshape(predicted_y.shape[0])\n", 837 | "predicted_y = np.where(predicted_y >= 0.5, 1, 0)\n", 838 | "accuracy_score(fraud_test_y, predicted_y)" 839 | ] 840 | } 841 | ], 842 | "metadata": { 843 | "kernelspec": { 844 | "display_name": "Python 3", 845 | "language": "python", 846 | "name": "python3" 847 | }, 848 | "language_info": { 849 | "codemirror_mode": { 850 | "name": "ipython", 851 | "version": 3 852 | }, 853 | "file_extension": ".py", 854 | "mimetype": "text/x-python", 855 | "name": "python", 856 | "nbconvert_exporter": "python", 857 | "pygments_lexer": "ipython3", 858 | "version": "3.6.1" 859 | } 860 | }, 861 | "nbformat": 4, 862 | "nbformat_minor": 2 863 | } 864 | -------------------------------------------------------------------------------- /03 Sequential Feature Selection Algorithm.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Introduction" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Feature Slection vs Feature Extraction \n", 15 | "\n", 16 | "Dimensionality Reduction에는 2가지 종류의 스킬이 있습니다. \n", 17 | "\n", 18 | "1. Feature Selection: original features의 일부분 (subset)을 선택합니다. \n", 19 | "2. Feature Extraction: original features에서 중요한 정보를 꺼내서 **새로운** feature subspace를 만듭니다.\n", 20 | "\n", 21 | "\n", 22 | "\n", 23 | "기본적으로 Feature Selection은 4가지가 있습니다. \n", 24 | "\n", 25 | "1. **Sequential Foward Selection (SFS)**\n", 26 | "2. **Sequential Backward Selection (SBS)**\n", 27 | "3. **Sequential Floating Forward Selection (SFFS)**\n", 28 | "4. **Sequential Floating Backward Selection (SFBS)**" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Sequential Feature Selection Algorithm\n", 36 | "\n", 37 | "Sequential Feature Selection Algorithm은 greedy search algorithm으로서
\n", 38 | "d-dimensional feature space를 k-dimensional feature subspace로 바꿔줍니다.
\n", 39 | "이때 k < d 라는 조건을 갖습니다.\n", 40 | "\n", 41 | "특히 regularization을 지원하지 않는 알고리즘에 사용이 될 수 있는 장점이 있습니다. \n", 42 | "\n", 43 | "특히 Model이 overfitting을 겪을 경우, Sequential Backward Selection (SBS)같은 알고리즘이..
\n", 44 | "간혹 prediction수치 자체를 높이는 경우도 있습니다.\n" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### Feature Selection Definition\n", 52 | "\n", 53 | "$ X = \\{ x_i\\; |\\; i=1...N \\} $ 이 주어졌을때 subset $ Y_M = \\{ x_{i_1}, x_{i_2}, ..., x_{i_M} \\} $ 을 찾습니다.
\n", 54 | "이때 $ M < N $ 이며 $ x_i \\in X $ 입니다.\n", 55 | "\n", 56 | "$$ \\begin{bmatrix} x_1 \\\\ x_2 \\\\ . \\\\ . \\\\ . \\\\ x_N \\end{bmatrix} \\rightarrow feature\\ selection \\rightarrow \\begin{bmatrix} x_{i_1} \\\\ x_{i_2} \\\\ \\\\ x_{i_M} \\end{bmatrix} $$" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "# Sequential Backward Selection (SBS)\n", 64 | "\n", 65 | "SBS알고리즘은 지정한 갯수의 features들을 뽑을때까지 계속 feature를 지워나가는 형태입니다.
\n", 66 | "이를 위해서 **Criterion function J**를 정의해야 합니다. \n", 67 | "\n", 68 | "\n", 69 | "**Input:**
\n", 70 | "$ X_d $ 에서 d값은 전체 the dimensionality of the full feature space를 나타냅니다.\n", 71 | "\n", 72 | "**Output:**
\n", 73 | "$ X_k $ 에서 k값은 subset feature의 갯수 이며, $ k < d $ 입니다.\n", 74 | "\n", 75 | "**Initialization**
\n", 76 | "$ k = d $ 로 시작을 합니다.\n", 77 | "\n", 78 | "**Execution**
\n", 79 | "특정 feature $ x^- $ 지정하고 criterion function J통해서 가장 적은 performance(accurace)를 보이는 feature를 삭제 합니다.
\n", 80 | "즉 하나하나씩 각각의 feature들을 다 돌아보며.. 가장 적게 accuracy를 보이는 녀석을 1개씩 삭제시켜나가는 것입니다. \n", 81 | "\n", 82 | "$$ x^- = argmax\\ J(X_k -x) $$\n", 83 | "\n", 84 | "criterion function J를 maximize" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 64, 90 | "metadata": { 91 | "collapsed": false 92 | }, 93 | "outputs": [ 94 | { 95 | "name": "stdout", 96 | "output_type": "stream", 97 | "text": [ 98 | "Populating the interactive namespace from numpy and matplotlib\n" 99 | ] 100 | } 101 | ], 102 | "source": [ 103 | "%pylab inline\n", 104 | "import numpy as np \n", 105 | "import pandas as pd\n", 106 | "from sklearn.preprocessing import StandardScaler\n", 107 | "from sklearn.cross_validation import train_test_split\n", 108 | "from sklearn.neighbors import KNeighborsClassifier\n", 109 | "from sklearn.metrics import accuracy_score\n", 110 | "from itertools import combinations\n", 111 | "\n", 112 | "# Set Random Seed\n", 113 | "np.random.seed(1)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "## Wine Data\n", 121 | "\n", 122 | "* [https://archive.ics.uci.edu/ml/datasets/Wine](https://archive.ics.uci.edu/ml/datasets/Wine)\n", 123 | "\n", 124 | "\n", 125 | "1. Alcohol \n", 126 | "2. Malic acid \n", 127 | "3. Ash \n", 128 | "4. Alcalinity of ash \n", 129 | "5. Magnesium \n", 130 | "6. Total phenols \n", 131 | "7. Flavanoids \n", 132 | "8. Nonflavanoid phenols \n", 133 | "9. Proanthocyanins \n", 134 | "10. Color intensity \n", 135 | "11. Hue \n", 136 | "12. OD280/OD315 of diluted wines \n", 137 | "13. Proline \n", 138 | "\n", 139 | "\n", 140 | "**Classes**\n", 141 | "\n", 142 | "1. class 1: 59\n", 143 | "2. class 2: 71\n", 144 | "3. class 3: 48\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 194, 150 | "metadata": { 151 | "collapsed": false 152 | }, 153 | "outputs": [ 154 | { 155 | "name": "stdout", 156 | "output_type": "stream", 157 | "text": [ 158 | "Train X: (133, 13)\n", 159 | "Train Y: (133,)\n", 160 | "Test X: (45, 13)\n", 161 | "Test Y: (45,)\n" 162 | ] 163 | } 164 | ], 165 | "source": [ 166 | "data = pd.read_csv('../../data/wine/wine_data.csv')\n", 167 | "\n", 168 | "COLUMNS = data.columns\n", 169 | "X = data.iloc[:, 1:]\n", 170 | "Y = data.iloc[:, :1]\n", 171 | "\n", 172 | "# Standardization \n", 173 | "scaler = StandardScaler()\n", 174 | "X = scaler.fit_transform(X)\n", 175 | "\n", 176 | "# Split Train and Test Data\n", 177 | "train_X, test_X, train_Y, test_Y = train_test_split(X, Y, test_size=0.25, random_state=1)\n", 178 | "train_Y = train_Y.get_values().reshape(-1)\n", 179 | "test_Y = test_Y.get_values().reshape(-1)\n", 180 | "\n", 181 | "print('Train X:', train_X.shape)\n", 182 | "print('Train Y:', train_Y.shape)\n", 183 | "print('Test X:', test_X.shape)\n", 184 | "print('Test Y:', test_Y.shape)" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "### Sequential Backward Selection" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 158, 197 | "metadata": { 198 | "collapsed": false 199 | }, 200 | "outputs": [], 201 | "source": [ 202 | "class SequentialBackwardSelection(object):\n", 203 | " def __init__(self, model, k):\n", 204 | " \"\"\"\n", 205 | " @param model: a model to optimize\n", 206 | " @param k : the number of output features to be reduced. \n", 207 | " \"\"\"\n", 208 | " self.model = model\n", 209 | " self.k = k\n", 210 | " \n", 211 | " self.dims = []\n", 212 | " self.scores = []\n", 213 | " self.subsets = []\n", 214 | " \n", 215 | " def fit(self, dataset):\n", 216 | " assert 4 == len(dataset)\n", 217 | " \n", 218 | " dim = X.shape[1]\n", 219 | " indices = tuple(range(dim))\n", 220 | " \n", 221 | " score = self.calculate_score(dataset, indices)\n", 222 | " \n", 223 | " self.dims.append(dim)\n", 224 | " self.scores.append(score)\n", 225 | " self.subsets.append(indices)\n", 226 | " \n", 227 | " while dim > self.k:\n", 228 | " scores = []\n", 229 | " subsets = []\n", 230 | " for p in combinations(indices, r=dim-1):\n", 231 | " score = self.calculate_score(dataset, p)\n", 232 | " scores.append(score)\n", 233 | " subsets.append(p)\n", 234 | "\n", 235 | " best = np.argmax(scores)\n", 236 | " indices = subsets[best]\n", 237 | " dim -= 1\n", 238 | " \n", 239 | " self.dims.append(dim)\n", 240 | " self.scores.append(scores[best])\n", 241 | " self.subsets.append(indices)\n", 242 | " \n", 243 | " def calculate_score(self, dataset, indices):\n", 244 | " _X, _Y, test_X, test_Y = dataset\n", 245 | " knn.fit(_X[:, indices], _Y)\n", 246 | " y_preds = knn.predict(test_X[:, indices])\n", 247 | " score = accuracy_score(test_Y, y_preds)\n", 248 | " return score\n", 249 | " \n", 250 | "\n", 251 | "knn = KNeighborsClassifier(n_neighbors=2)\n", 252 | "sbs = SequentialBackwardSelection(knn, k=1)\n", 253 | "sbs.fit([train_X, train_Y, test_X, test_Y])" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "### K-feature값에 따른 Accuracy의 변화량\n", 261 | "\n", 262 | "아래의 결과에 따르면 K-feature값이 5 Ehsms 6일때 full features일때보다 더 높은 Accuracy를 보였습니다. \n" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 155, 268 | "metadata": { 269 | "collapsed": false 270 | }, 271 | "outputs": [ 272 | { 273 | "data": { 274 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh4AAAGHCAYAAAD/QltcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3XmYHGW59/HvzSqRwyZIQN6AgGwKSEAhLsgiYdGMLEIU\nOErCIpCoBElU4BAFPRBEUAg7gQAehgwCIyAaQBHIAUQnyHIkoJEQQIREEIQJQsj9/vFUYk+nZ6a7\nprvr7uT3ua6+dGqqq781aTJPqp6qNndHREREpBlWKDpARERElh8aeIiIiEjTaOAhIiIiTaOBh4iI\niDSNBh4iIiLSNBp4iIiISNNo4CEiIiJNo4GHiIiINI0GHiIiItI0GniILOPMbGMzW2RmXyq6pR4q\n7Y+ZfcfMFhXZ1QrM7D/M7EozeyH7GZ5ddJMsfzTwkHDM7PjsL8UHim6RluHAMj3wMLPfmNmjFZbv\naWbdZvZ7M1urn838F3AYcAFwOHBdA1IxszFm9p+N2La0Pg08JKJDgaeBj5rZpkXHSEs4AxhUdESD\nLfXBWma2B3AL8EdgT3f/Rz/b2B34X3f/b3e/zt3/0IBOgLGABh5SkQYeEoqZvR/4GHAiMJ/0r7OQ\nzGxZ/0XXMtx9kbu/VXRHM5nZp0iDjlnAXu7+ahVPey/Q3+AkLDN7V9ENMnAaeEg0hwEvAz8Hfkov\nAw9Lvm5mj5rZAjN7ycx+YWZDy9Y73Mx+a2ZvmNnLZnaPme1V8v1FZnZahe3PMbMrS77+crburmZ2\nkZm9CDybfW9ItmxWdsh7vpl1mNnGFba7ppmdZ2ZPm9mbZvasmV1tZuuY2bvN7HUzO6/C895nZgvN\n7Jt9/fCy7U81s3+Y2StmdhVQ8fC7mW1pZj81s79nP8PfmdmIsnUW7/cnzezSbN9ezZqX2q6Z7Wtm\n92b78ZqZ3WZm25StM9XM/mlmG5pZZ/b/XzKzH5iZ5dmfSnM8su7zzexzZvZY9vN+3Mz2rvD83bJT\nFQvM7E9mdkwt80bM7ODs+d1mNs/MrjWzDfPudxWv90ngNuAp0qDjlX7W3zPbl42A/bOfzTuLG81s\nVTM73cz+nP2cnjGzM81s5bLtHGlmvzKzF7Of1eNmdnTZOs8CWwCfzl5nkZndkX3ve2b2doW+o7L1\nNixZ9pyZ3WRm+2Q/2zeB0SXf/3LJz/zvZvY/5T9ziWmlogNEyhwK3OjuC82sHTjWzHZ0966y9a4E\nvkwaoFxOei9/EtgFmAlgZhOBicD/ks5tvwXsTDrcfGc/HUsd1s5cBLwEfBd4d7bsI9nrtgPPAZsA\nxwN3m9k27v5m1vNuYAawJTAFeBhYF2gDNnL3R83sZmCkmZ3o7qUNh2b/+5N+um8hHTG6mPQv4QOA\nq8v3x8w+mLU8B5wJvAEcAnSa2YHu/rOy7U4GXiH9PLfM9m8I6We5eJv/CUwFfglMIJ36OA64z8x2\ncPe52apO+kfPdOBB4BvAp0lHuf4MXFrr/mRfV/oz+yRwIOnP7Z/A14CfmtmQxb+szWwH4BfAX0nv\nk5Wy/53fyzZ7MLMjSO/H3wLfAtYHTgA+lu33azn2u6/X+xjpfT8b+LS7v1zF0x4jzem4IHvej7Ll\nL2eDnp8DHwUuIQ1mts/6NiO9LxY7jvS+/RmwEPgccKmZ4e6XZ+uMJf28/056bxnwQsnPoNLPtNJy\nBz5Ies9fQvr5PJH9DCYCp5HmqFxOOpLzddLp2R3c/fUqfiZSFHfXQ48QD2BH0gTB3UuWzQXOLVtv\n92y9c/vY1makvxhv6Oc1FwGnVVj+NHBlyddfztb9DWBl665a4fkfzdY/rGTZd4F3gLY+evbK1hle\ntvwPwK/72ZfPZa95YskyA+7JtvmlkuV3kX6BrFS2jRnArAr7/VtgxZLlJ2Xb/Gz29btJR6ouLtve\neqQByyUly67Knnty2bpdwEM592ci8E6FP9sFwCYly7bNlh9fsuwW0qBk/ZJlm5IGqj22WeFnvhLw\nt+zPZ5WS5ftlrzOx1v3u47XuJg2GXgUeBdbJ8d/Ys8BNZcuOAN4GPlq2/Pisd6d+3ut3Ak+ULXsC\nuKPCumcAb1VYfmT2WhuWtb4D7Fa27qak/7a/UbZ822w/Tqr156JHcx861SKRHEb6S/w3JcumAV8o\nOxR9EOkv9dP72NYBpF9Sfa1TKwcu9+xvuSUL3f+1+P+b2Upmtg7wF9K59NJTPwcCj7j7LX28xl2k\nfx0uOcVkZh8CtgOu7advX9JfvJeUtDnpX7lLfn5mtjZp8HYDsKaZvWfxA7gD+ICZbVC27cvc/Z2S\nry8m/VLYL/t6OLAmcH3Z9pw0aNmdpZX/C/8+0i+VmvanH3e6+5yS5z8GvLb4dcxsBWBPoNPdXyxZ\n7y+koyD92Yn0r+2LvGSOibvfTjpC85kKz+lvv/vybmB14EXSYKkePk86IjK77M/ubtLPecmfXdl7\nfY1svXuALcxstTr1lPqTu/+mbNlBpPfVjWW9L5D+u6v0XpNAdKpFQsh+AYwk/WW3ack44yHSId89\nSb+UIf0l/Vfvewb/pqTByRN1Tp1TvsDShLeTSf9yfB///qXopF/Gi21GmrfSK3d3M/sf0immd3k6\nTXMY6V/ufT4X2Bh4wd27y5Y/Wfb15lnjGcD3KmWQfpmWHh7/c1nnG2b2Aum0Uuk27+5le6+VLXvT\n3f9etuwVYO0c+9OXZyssK32d9wKrUbZ/mUrLym1M2r+nKnxvFvDxsmXV7Hdf/gRcA5wNtJvZweUD\n4WxguUrJom5372uQ8gHSn9+8Ct9b/F5YvO1Pko7cfZSeVxEtfq8vqHI/qvV0hWWbAyuSBhnlKr3X\nJBgNPCSKPYANgC8AXyz7npN++d5V/qQGWrGX5ZX+Yp1MOiVxHunc/auk5mnkm8B9DTAe2B+4nvTz\nuLWfXx61WNx0Dmm+QSXV/NIt36aT5hG8WOH7C8u+fqfCOo3Q2+vUNJmzjga83+5+jpmtS5pHczlw\nVNkqt/DvAY+T5hMd08cmVyCdKjqJyj+XuQBm9gHSaZXHgXGkQd1bpDlKX6W693pvc2Zq+e9tBdL7\naZ9enlOv/06kQTTwkCgW/8I6nqX/8jsIOMDMjs0O9c4GhpvZWn0c9ZhN+gtqG9L58N68QtlVEtlM\n/vJTDX05CJjq7hNKtrFq+Xazpg/1tzF3/z8zexg4zMyeJ03iHFNFxzPAHmY2qOwowVZl6y3+l+Lb\n7v7rKrZrpH8V37NkQZoouwFpUiKkfTNgXpXbrEa1+zMQLwFvkv4VXe4DVTz/GdJ+b0nPU4Rky54Z\nSFxv3P1b2Sm9o8zsFXcfX/Ltr9Pzvfd8P5ubDWzp7pWOVpVqA1YGPlN6WsoqXCVE7wOMV4AVK/yZ\nbtLPa5f3rgj8pfQ0mrQOzfGQwmWnKg4g/av+Zne/qfRBOqKwBukvPoAbSe/diX1stpP0l99p/Vyq\nOBvYtWzZV+j9X2CVvMPS/y19rcI2bgS2N7PPVbHNa4G9SVdHzCddKdKf20m/GI5bvCA7hfVVSn4R\nuPs80i/Jr5jZ4PKNZP+aLneMmZX+Q+V40v7dnn09nXSI++Sy9fraZn+q2p+BcPdFpCNp+5f+LMxs\nc3r/F3Wp35MGL8eWXnpqZvsCW5MueW2Ur5BOv33DzE5evNDdZ7r7r0se/Z2a6gA2NrNR5d8ws9VK\n5m4sPlqzQsn31wYq3Yr/DSpfxr14gLrkvzkzW53abjZ2I+nPv+J//9mATALTEQ+J4HPAf5AOEVfy\nIOn882Gkq1R+Y2bXAl8zsy1Iv5RXIF06+Wt3v8jdZ5vZ94FTSZdz3gT8i3Tp6/Pufkq27SuAS8zs\np6TDyNuTJkpWOt/d2wDmNuA/zew10h0kh5HmpMwvW+8HpIl8N1i6H0UX8B5gBPCVbOLjYteRzuPv\nT5q4WM0h+ltJlw6fZelGbH8kTWj9jwrrjiFNanzMzC4nHQVZP2t/H7BD2fqrAL8ysw7SEYfjgPvc\n/TYAd/+nmR1HOk0008yuJ/0Mh5AmWM4gDcZqUcv+DMR3SH/m95vZxaS/F8eQJlx+uK8nerrs+5uk\ny2nvzS4BH0za17/w78tW6y6bD3QYaW7F97IjHxfn2NRU4GDgcjP7NHA/6WewdbZ8d9JRw+nAJOD2\n7D2zBnA0aS7Qe8u22QUcmQ2IZgN/c/d7SBN2nwemmtk52bqjs21UdQ8Od/9Tdjnt6Wa2GenvjddJ\n87oOIE0+Pr/Gn4E0U9GX1eihB+meAK8D7+pjnStJh8TXzr420v0P/o90HvhvpAHAh8ue92XSv0q7\nSQOBXwN7lHzfgP/m31cJ/Bx4P+mXxpSy7bwDDK3QtgZpAPMiaX7Hz0mH6XtsI1t3LeDHpPPmC0iH\n4qcs3q+ydW/LXnPnGn6Wa5F+kbxCurz1KtIVMT0uP83W3ST7/vPZz3Zu9mdxQIX9/gTpSpbFl3Ne\nDaxV4fV3JR2peJn0r96nsv3boWSdq4BXKzx3IrAwz/708tx3gB9XeJ1Kfy67Ze+TBVnzKNJA8Y0q\nf+6fL3mfzct+PhuUrVP1fvfyGneTrooqXz6INEB7G/hCP9uYS7pPTvnyFUlzRh7LfgbzSVcjnQy8\nu2S9EcAj2X7+mXRE7iiWvhR2cPb+fTX73h0l3xsKPJC9zl9Ig7xKl9NWbC35/oHAvaQjba+R/i74\nEbBptf+96FHMw7I/QBEJJjtK8yF336LAhi+TBn0fcfeZRXUUIbuZ2zbuvmXRLSLLkhBzPCzdjvkW\nM3ve0m1z26p4zm5m1mXp9r5PZX9Bln5/ov37dr2LH39s3F6I1E92H43PkE5dSINZ2WeAZFdw7Efl\ny4NFZACizPF4N+lyrinATf2tbGabkA7jXUS6lfSngSvM7K/uXnor7MdJ59oXn5svv6RPJJTsvf0J\n0uHrt4DLiuzJFHXpaTP9xcymkg79bwIcSzr99IMCm0SWSSEGHu7+S7JZ+/1cgbDYcaRLqRZfvvik\nmX2CdG156cBjoacZ/CKt4lOkuQBzSHMYXio2B6jTFSTB/YJ0D5nBpEnI95NubT670CqRZVCIgUcO\nu7D0zaSmk27gVOoD2X0Q3iRNZvq2u1e6k6FICO5+NWliYgjRehrF3Y8sukFkeRFijkcOg1n67ogv\nAmtkN26CdAnmEaR7IRxLulLh3uzGRyIiIlKAVj3i0S93L70V9ONm9hDp0sVDSIeye8g+ZGhv0iHu\nN5vRKCIisox4F2l+1HRf+vOIemjVgcffSDc7KrU+8JqXfHpiKXd/1cyeovKtkSENOv6nfokiIiLL\nncNIN0DsVasOPB4gfWR2qeHZ8oqy2/JuRu+XJ84B+MlPfsLWW29dh8T6GjduHOedVz6FJQa11S5q\nF6gtr6htUbtAbXlFbHviiSc4/PDDocIneJcLMfDI5l0s/lhtSB+Lvj3wsrs/a2Znku5ot/heHZcA\nY8xsEunmRnuS7hy4X8k2f0C65fIzpFtAf5d0OW17LxlvAmy99dYMHTq0nrtXF2uuuWbILlBbHlG7\nQG15RW2L2gVqyytyG1VMVQgx8AB2It2ox7PHD7PlV5Pu4z8Y+H+LV3b3OWb2GdJVLF8DngOOdPfS\nK102Ih3ueQ/pFsYzgF36O/cU1d/+9reiE3qlttpF7QK15RW1LWoXqC2vyG3VCDHw8PThQb1eYePu\nS31qorvfC+zYx3O+WJ+6GJ5/vr9Pti6O2moXtQvUllfUtqhdoLa8IrdVo1Uvp13u7Lhjr2Oswqmt\ndlG7QG15RW2L2gVqyytyWzU08GgRX/xi3AM4aqtd1C5QW15R26J2gdryitxWDX06bcbMhgJdXV1d\nkSftiIiIhDNz5szFR2J27O+TrHXEQ0RERJpGA48WMWrUUvNrw1Bb7aJ2gdryitoWtQvUllfktmpo\n4NEihg8fXnRCr9RWu6hdoLa8orZF7QK15RW5rRqa45HRHA8REZF8NMdDREREQtLAQ0RERJpGA48W\nMWPGjKITeqW22kXtArXlFbUtaheoLa/IbdXQwKNFnH322UUn9EpttYvaBWrLK2pb1C5QW16R26qh\nyaWZ6JNLu7u7GTRoUNEZFamtdlG7QG15RW2L2gVqyytimyaXLoOivclKqa12UbtAbXlFbYvaBWrL\nK3JbNTTwEBERkabRwENERESaRgOPFjF+/PiiE3qlttpF7QK15RW1LWoXqC2vyG3V0MCjRQwZMqTo\nhF6prXZRu0BteUVti9oFassrcls1dFVLJvpVLSIiIlHpqhYREREJSQMPERERaRoNPFrErFmzik7o\nldpqF7UL1JZX1LaoXaC2vCK3VUMDjxYxYcKEohN6pbbaRe0CteUVtS1qF6gtr8ht1dDk0kz0yaVz\n584NO5NZbbWL2gVqyytqW9QuUFteEdtqmVyqgUcm+sBDREQkKl3VIiIiIiFp4CEiIiJNo4FHi5g0\naVLRCb1SW+2idoHa8oraFrUL1JZX5LZqaODRIrq7u4tO6JXaahe1C9SWV9S2qF2gtrwit1VDk0sz\nmlwqIiKSjyaXioiISEgaeIiIiEjTaODRIubPn190Qq/UVruoXaC2vKK2Re0CteUVua0aGni0iNGj\nRxed0Cu11S5qF6gtr6htUbtAbXlFbquGJpdmok8unTlzZsguUFseUbtAbXlFbYvaBWrLK2Kbbpme\nQ/SBh4iISFS6qkVERERC0sBDREREmkYDjxYxZcqUohN6pbbaRe0CteUVtS1qF6gtr8ht1dDAo0XM\nnNnnKbNCqa12UbtAbXlFbYvaBWrLK3JbNTS5NKPJpSIiIvlocqmIiIiEFGLgYWafNLNbzOx5M1tk\nZm1VPGc3M+syszfN7Ckz+3KFdcaY2dNmtsDMHjSzjzRmD0RERKQaIQYewLuBPwDHA/2e+zGzTYDb\ngF8B2wM/Bq4ws71K1hkJ/BCYCOwAPAJMN7N169wuIiIiVQox8HD3X7r7ae7+M8CqeMpxwF/cfYK7\nP+nuFwI/BcaVrDMOuNTdr3H3WcCxQDfQkveabWvr9yBQYdRWu6hdoLa8orZF7QK15RW5rRohBh45\n7ALcVbZsOjAMwMxWBnYkHREBwNMs2rsWr9Nqxo4dW3RCr9RWu6hdoLa8orZF7QK15RW5rRrhrmox\ns0XA/u5+Sx/rPAlc6e6TSpbtSzr9MghYB3geGObuvy1ZZxKwq7svNfjQVS3SNL//Pdx8MyxaVHRJ\nZTvtBAceCFbNwUcRkdquasHdQz2ARUBbP+s8CXyzbNm+wDvAqsAG2XZ2LltnEvBAL9scCvj666/v\nI0aM6PHYZZdd/Oabb/ZS06dP9xEjRni5448/3q+44ooey7q6unzEiBE+b968HstPO+00P+uss3os\ne+aZZ3zEiBH+xBNP9Fh+/vnn+0knndRj2RtvvOEjRozw++67r8fy6667zo844oil2g455BDtR5H7\n8dJL7kcd5W7mvv76fsi73+03Dx7svtlmSx7TN9jARwwa1GOZb7aZH7/GGn7Feuv1WNa10UY+YtAg\nn7fJJj2Wn7b22n7WOuv0WPbMkCE+YtAgf+L//b8ey89fd10/ac01/71sk038DfAR73mP3zd16rL9\n56H90H5oP3Ltx3XXXbfkd+Pi35m77rqrk+ZoDvV+fs+36hGPe4Audz+xZNkRwHnuvnZ2qqUbOKh0\nO2Y2FVjT3Q+osE0d8ZDGWLgQLrkE/uu/0tff/z4ccwystFKxXb254w742tfgz3+GMWPgu9+FtdYq\nukpEAlse7uPxALBn2bLh2XLc/W2gq3QdM7Ps6/ub1FhXnZ2dRSf0Sm19uO8+2HHH9Iv84IPhqafg\n+OPpvO22Yrv60NndDY8+CmedBVdeCVtskf43wKmhwv88+xC1LWoXqC2vyG3VCDHwMLN3m9n2Zvbh\nbNGm2df/L/v+mWZ2dclTLsnWmWRmW5rZ8cDngXNL1jkXONrMvmRmW2XPGQRMbfgONUB7e3vRCb1S\nWwV//Sscdhjsuiustho89BBcdhmst16xXVVob2+HVVaBk06CJ5+E4cPhyCNh2DD43e+KbwsqalvU\nLlBbXpHbqhHiVIuZfQq4m6Xv4XG1u482s6uAjd19j5Ln7AqcB2wDPAec7u7Xlm33eGACsD7pPiFf\ndfff99KgUy0ycG+9BT/+MZx+ehpwTJoEX/4yrBBijJ/ffffB2LHw2GNpEPLf/71kECUiUsuplhAD\njwg08JABK50bMXYsfOc7y9bciIUL4dJL4dRT09ff+x585Stx56qISNMsD3M8ROKYMyddfrr33jB4\nMDz8MPzoR8vWoAPSAGPMmDRP5eCD4atfTfNX7ruv6DIRaSEaeIjktWBBuuJj663THI7rr4e774Zt\nty26rLHWWy/NV/ntb+Fd70rzWA47LM1rERHphwYeLWLUqFFFJ/RquWtzh5/9DLbZJl0ae8IJMGsW\njBxZ9U23lomf2Uc+Ag88kK54ufNO2HJL+MEP0jyXotsKELUtaheoLa/IbdXQwKNFDB8+vOiEXi1X\nbU8+CfvuC/vvD1ttBY8/DmeeCauvXmxXHdXUtsIKMGpUOv0yejR8+9uw3XZpvkvRbU0WtS1qF6gt\nr8ht1dDk0owml0qf/vnPNJnyvPNgo43SHI4RI3Rb8XKPPZbmftxzDxxwAJx7LmyySdFVItJgmlwq\nUi/ucN116ejG+eenu4/+3/9BW5sGHZVsu22a59LenuaAbL11mgezYEHRZSIShAYeIr159FHYbbc0\ncXKXXeCJJ9LAY7XVii6LzQy+8IV0WuqEE9I8mG22SfNidIRVZLmngUeLmDFjRtEJvVrm2l55Jd2P\nY4cd4KWX0nyFG2+s6ymDZe5nVsnqq6f5L48/no4Y7b9/mh/z5JPFtzVA1LaoXaC2vCK3VUMDjxZx\n9tlnF53Qq2WmbdEimDIlfTbJVVelu44+8gjstVexXU1W97YttoDbb09HPJ56Kp2O+da34PXXi2+r\no6htUbtAbXlFbquGJpdmok8u7e7uZtCgQUVnVLRMtD30ULrb6O9+B4cfngYdG25YfFcBGtq2YEG6\n5PbMM2GddeCcc9JpmSrnyyy3P7cBiNoFassrYpsmly6Dor3JSrV027x5cNRRsPPO6f4T994L117b\n0EFHVV0FamjbaqvBaael+TK77AKHHprm0Tz6aPFtAxS1LWoXqC2vyG3V0MBDlk8LF8Lkyek0wI03\nwoUXwu9/D5/8ZNFly4dNNkk/9+nT4cUX03yar30N/vGPostEpME08JDlz733wtCh6RfdwQeneQfH\nH68POyvC8OHpaMekSWlezRZbpDuhLlpUdJmINIgGHi1i/PjxRSf0qmXann8+XRr7qU/BoEFpXsdl\nlxXy8e4t8zNrhlVWgZNOSle7DB8ORx4Jw4al+TZFt9UgalvULlBbXpHbqqGBR4sYMmRI0Qm9Ct/2\n1ltw9tnps0TuvDP9i/r++2GnnYrtCqqwtg03hJ/8JB2RevPNNO/m6KPTPJyi26oQtS1qF6gtr8ht\n1dBVLZnoV7VITtOnp1Mqs2enq1a+851l7+Pql0ULF8Kll8Kpp6avzzgDjj1Wp8NEgqrlqhb9VyzL\npqefhhNPhM7OdGrlpz9d9j+uflmy0kowZgwccgicckoaPF5+eboCRkTieemlqlfVwEOWLQsWpNMq\nZ52V7hPR3l7Tx9VLMOutl+bhHH10Ovrx618XXSQildTweUwaeLSIWbNmsdVWWxWdUVGINvd0Z8xx\n49Ik0m98A045hVnPPcdWAQcdIX5mvQjZ9pGPwPTpMdsyUduidoHa8grZNnMmpFMt/dLk0hYxYcKE\nohN6VXjbk0+mzwA54ID0mSCPP57ujLn66sW39SJqF6gtr6htUbtAbXlFbquGJpdmok8unTt3btiZ\nzIW1/fOf8L3vwXnnwUYbwY9+BCNG9DitEvXnFrUL1JZX1LaoXaC2vCK21TK5VAOPTPSBh5RwT3M3\nxo+Hl1+Gk09O94HQx9WLiBRCn9Uiy65HH01XNhx2WPqsjyeegP/6Lw06RERahAYe0hpeeSVdUrnD\nDumyrTvuSJ/1sckmRZeJiEgNNPBoEZMmTSo6oVcNbVu0CKZMSZ/hcdVV6TM9HnkE9tqr+LYBiNoF\nassralvULlBbXpHbqqGBR4vo7u4uOqFXDWt76KF0OuWoo2CffdLVKyedlD7bo+i2AYraBWrLK2pb\n1C5QW16R26qhyaUZTS4NZN48+Pa305GO7beHCy7Qx9WLiASmW6ZLa1q4EC65JE0WBbjwQjjmGH0+\nh4jIMkSnWiSGe++FoUPTBNKDD4annoLjj9egQ0RkGaOBR4uYP39+0Qm9GlDb88+nS2M/9SkYNCjN\n67jssvQZHUW3NVDULlBbXlHbonaB2vKK3FYNDTxaxOjRo4tO6FWutrfeSh/mtuWWcOedcOWVcP/9\nsNNOxbc1QdQuUFteUduidoHa8orcVhV31yNNsB0KeFdXl0cUtcs9R9v06e5bbum+4oruX/+6+yuv\nNCbM4/7cona5qy2vqG1Ru9zVllfEtq6uLgccGOr9/L7VVS0ZXdXSBHPmwIknws03p1MrF1wA225b\ndJWIiAyQbpkusSxYAN/9Lmy9dZrDcf31cPfdGnSIiCyHdMmANI473HILnHBCmkT6jW/AKafA6qsX\nXSYiIgXREY8WMWXKlKITelWx7cknYd99Yf/9Yaut4PHH4cwzmz7oiPpzi9oFassralvULlBbXpHb\nqqGBR4uYObPPU2aF6tH2+uvwrW+l0yhPPQU/+xncfnv6rJWi2wKJ2gVqyytqW9QuUFtekduqocml\nGU0uHSD3NHfjpJPg5Zfh5JPT/9fH1YuILPM0uVSa69FHYbfd4NBD04e6PfFEuu25Bh0iIlJGAw8Z\nmFNPTbc6f+kluOMOuPFG2GSToqtERCSoMAMPMxtjZk+b2QIze9DMPtLHuiuZ2Wlm9uds/YfNbO+y\ndSaa2aKyxx8bvyfLkTlz4PvfT1erPPII7LVX0UUiIhJciIGHmY0EfghMBHYAHgGmm9m6vTzl+8DR\nwBhga+BS4GYz275svceB9YHB2eMT9a9vjra2tqITlnbDDbDaarQ99hisskrRNRWF/LkRtwvUllfU\ntqhdoLZ9CfQkAAAgAElEQVS8IrdVI8TAAxgHXOru17j7LOBYoBvo7Yb0hwPfd/fp7j7H3S8Bbge+\nUbbeQnef5+4vZY+XG7YHDTZ27NiiE5Y2bRp85jOMPeGEokt6FfLnRtwuUFteUduidoHa8orcVo3C\nr2oxs5VJg4yD3P2WkuVTgTXd/YAKz5kPjHf3q0qWXQt83N03zb6eCJwEvAa8CTwAfNvdn+2lQ1e1\n1GL2bNh8c+joSB9jLyIiy61Wu6plXWBF4MWy5S+STo9UMh040cw2t2Qv4EBgg5J1HgSOAPYmHUF5\nP3Cvmb27ju3Lr46O9DH2n/lM0SUiItJCIgw88vg68CdgFvAv4HzgSmDR4hWy0zA3uvvj7n4nsB+w\nNnBIAb3Lno4OGDEiDT5ERESqFGHgMR94hzQJtNT6wN8qPcHd57v7gcAgYGN33xp4A/hLby/i7q8C\nTwGb9xWz33770dbW1uMxbNgwOjs7e6x3xx13VJzgM2bMmKVuZztz5kza2tqYP39+j+UTJ05k0qRJ\nPZbNnTuXtrY2Zs2a1WP50Ucfzfjx43ss6+7upq2tjRkzZvRY3t7ezqhRo5ZqGzlyZH3248EH4Q9/\ngJEjl2y32v244IILmrof++67b0P+PAa6H52dnfX786jzfkybNq2Y91UV+3H55ZeHeF9V2o8f/vCH\nhb+vKu1HZ2dniPdVpf3o7OwM8b6qtB+dnZ0h3leV9qOzs7PQ91V7e/uS342DBw+mra2NcePGLfWc\nXrl74Q/SaZEfl3xtwLOkeRzVPH9l0hGQM/pYZ3Xg78DYXr4/FPCuri6P6JBDDik64d/OOMN99dXd\nu7vdPVhbmahtUbvc1ZZX1LaoXe5qyytiW1dXlwMODPV+fmcXPrkUwMwOAaaS5mI8RLrK5fPAVu4+\nz8yuAZ5z95Oz9T8KvA/4A7AR6TLcTUg7/Fq2zg+AW4FnsnW/C2wHbOPuf6/QoMml1dpuu/RZLP/z\nP0WXiIhIALVMLl2pOUl9c/eO7J4dp5NOsfwB2Nvd52WrbAQsLHnKu4DvkSaMvg78HDh88aCj5DnX\nAe8B5gEzgF0qDTqkBk88AY89Bt/7XtElIiLSgkIMPADc/SLgol6+t0fZ1/cCH+xne1+sX50s0dEB\na6wBe+/d/7oiIiJlIkwulVbS0QH77w+rrlp0iYiItCANPFpEpVnGTff44/DHP8IhPa9IDtHWi6ht\nUbtAbXlFbYvaBWrLK3JbNTTwaBHDhw8vOiEd7VhrraU+DC5EWy+itkXtArXlFbUtaheoLa/IbdUI\ncVVLBLqqpR/usNVW8PGPw5VXFl0jIiKBtNot06UVPPooPPXUUqdZREREaqGBh1SnowPWWQf23LPo\nEhERaWEaeLSI8tvaNpU7TJsGBx4IK6+81LcLbetH1LaoXaC2vKK2Re0CteUVua0aGni0iLPPPru4\nF3/4YZg9u9fTLIW29SNqW9QuUFteUduidoHa8orcVg1NLs1En1za3d3NoKI+Cfab30wTSl94AVZa\n+p5zhbb1I2pb1C5QW15R26J2gdryitimyaXLoMLeZO5pfsdBB1UcdECBbVWI2ha1C9SWV9S2qF2g\ntrwit1VDAw/p2+9/D3PmwMiRRZeIiMgyQAMP6du0afDe98KuuxZdIiIiywANPFrE+PHjm/+ii0+z\nfP7zsOKKva5WSFuVorZF7QK15RW1LWoXqC2vyG3V0MCjRQwZMqT5L/rgg/Dss/2eZimkrUpR26J2\ngdryitoWtQvUllfktmroqpZM9KtaCjFuXDrV8uyzfR7xEBGR5ZuuapGBW7QIbrgBDj5Ygw4REakb\nDTyksvvvh+ef12eziIhIXdU88DCzTRsRIn2bNWtWc1+wowPe9z4YNqzfVZveVoOobVG7QG15RW2L\n2gVqyytyWzXyHPH4s5ndbWaHm9m76l4kFU2YMKF5L/bOO+k0yyGHwAr9v0Wa2lajqG1Ru0BteUVt\ni9oFassrcls1ap5camYfBkYBXwRWAaYBU9z9ofrnNU/0yaVz585t3kzme+6B3XaDBx6AXXbpd/Wm\nttUoalvULlBbXlHbonaB2vKK2FbL5NLcV7WY2UpAG3AEsA/wFHAlcK27z8u10QJFH3g01fHHw89/\nnu5YalZ0jYiIBNeUq1rcfaG73wQcDHwT2Bw4B3jWzK4xsw3yblsKtHAh3HhjOs2iQYeIiNRZ7oGH\nme1kZhcBLwAnkgYdmwF7ARsCP6tLoTTXvffCSy/pahYREWmIPFe1nGhmjwH3kwYYXwI2dvdT3f1p\nd7+PdPplOT9fUV+TJk1qzgtNmwbvfz/stFPVT2laWw5R26J2gdryitoWtQvUllfktmpU/pzzvh1H\nmssx1d1f6GWdl4Ajc1fJUrq7uxv/IotPsxx1VE2nWZrSllPUtqhdoLa8orZF7QK15RW5rRq6ZXpG\nk0uBO+6AvfeGmTNhhx2KrhERkRbR0MmlZjbKzA6usPxgM/tyrduTQDo6YPPN4cMfLrpERESWUXkm\nl34bmF9h+UvAyQPLkcK8/TbcdJOuZhERkYbKM/AYAjxdYfkz2fekAebPrzTWq6O77oJXXoGRI2t+\nasPbBiBqW9QuUFteUduidoHa8orcVo08A4+XgO0qLN8e+PvAcqQ3o0ePbuwLdHTAllvCttvW/NSG\ntw1A1LaoXaC2vKK2Re0CteUVua0q7l7TA5gEzAF2B1bMHntky86pdXtRHqTLf72rq8sjamjXm2+6\nr7mm+2mn5Xp61J+Ze9y2qF3uassralvULne15RWxraurywEHhno/v2/zfFbLKsC1pDuWLswWrwBc\nAxzr7m8NaCRUkOX6qpbbboMRI+Dxx+GDHyy6RkREWkwtV7XUfB+PbGAx0sz+i3R6ZQHwmLs/kydW\nAujogG220aBDREQaLs8NxABw96dIHwwnrezNN6GzE046qegSERFZDuT6rBYz28jMjjezs8zs3NJH\nvQMlmTJlSmM2PH06/POfA/psloa11UHUtqhdoLa8orZF7QK15RW5rRp5biC2J/Ak6dbp3yBNMh0F\njAZ056kGmTmzz1Nm+U2bBtttB1ttlXsTDWurg6htUbtAbXlFbYvaBWrLK3JbNfJMLn0I+IW7TzSz\nf5LmebwE/A/wS3e/uP6ZjbdcTi5dsADWWw++/W045ZSia0REpEU19JbpwNakK1ggXdWymru/DpwG\nfDPH9qQov/gFvPHGgE6ziIiI1CLPwOMNYJXs/78AbFbyvXUHXCTNM21a+jC4D3yg6BIREVlO5Bl4\nPAh8Ivv/twM/NLNTgCuz70kreOONdP8OHe0QEZEmyjPwOBH4bfb/JwK/AkaS7lx6ZH2ypFxbW1t9\nN/jzn0N3d10GHnVvq6OobVG7QG15RW2L2gVqyytyWzVqGniY2YrARsBcAHd/w92Pdfft3P2ggdxE\nzMzGmNnTZrbAzB40s4/0se5KZnaamf05W/9hM9t7INuMbuzYsfXdYEcH7LQTbLrpgDdV97Y6itoW\ntQvUllfUtqhdoLa8IrdVI89VLW8CW7t7pU+ozRdhNhK4GjgGeAgYR7ol+xbuvtTH8JnZJOBQ4CjS\npb37AOcCw9z9kZzbXH6uann99XQ1y+mnw/jxRdeIiEiLa/RVLY8DA/9nck/jgEvd/Rp3nwUcC3ST\n7g1SyeHA9919urvPcfdLSPNNvjGAbS4/br013bFU8ztERKTJ8gw8TgXOMbPPmtkGZrZG6aPWjZnZ\nysCOpLkiAHg6DHMXMKyXp60K/Kts2QKySa85t7n86OiAnXeGjTcuukRERJYzeQYet5NuGnYL8Bzw\nSvb4R/a/tVoXWBF4sWz5i8DgXp4zHTjRzDa3ZC/gQGCDAWwztM7Ozvps6LXX0v07Ro6sz/aoY1sD\nRG2L2gVqyytqW9QuUFtekduqkWfgsXvJY4+Sx+Kvm+HrwJ+AWaQjH+eTLuddNNAN77fffrS1tfV4\nDBs2bKk/6DvuuKPizOIxY8YsdR/9mTNn0tbWxvz5PaeWTJw4kUmTJvVYNnfuXNra2pg1a1aP5aef\nfjrjy+ZjdHd309bWxowZM3osb29vZ9SoUUu1jRw5ks7TToN//Qs+//m67ccpp5xS9X5ccMEF9dmP\nKv88TjrppIb8eQx0P9rb22vaj0a9ryrtx7XXXtuwP4+B7scVV1wR4n1VaT8uuuiiwt9Xlfajvb09\nxPuq0n60t7eHeF9V2o/29vYQ76tK+9He3l7o+6q9vX3J78bBgwfT1tbGuHHjlnpOb2qeXFpv2WmR\nbuAgd7+lZPlUYE13P6CP564CvMfdXzCzs4DPuPu2eba53EwubWuDv/8d/vd/iy4REZFlRC2TS1eq\ndeNmtmtf33f3e2vZnru/bWZdwJ6k0zeYmWVfn9/Pc98CXsgGGgcB1w90m8u0f/wDfvlLOOecoktE\nRGQ5VfPAA/hNhWWlh01WzLHNc4Gp2WBh8aWvg4CpAGZ2DfCcu5+cff1R4H3AH0j3FZkIGPCDare5\nXPrZz2DhwiWnWURERJotz8Bj7bKvVwZ2AM4Acn3Eqbt3mNm6wOnA+qQBxd7uPi9bZSPSB9It9i7g\ne8D7gdeBnwOHu/trNWxz+TNtGnzyk7DhhkWXiIjIcqrmyaXu/mrZY76730n6ZNqz84a4+0Xuvom7\nr+buw9z99yXf28PdR5d8fa+7f9DdB7n7e919lLv/rZZttppKk31q8vLLcOedDbl3x4DbGihqW9Qu\nUFteUduidoHa8orcVo08V7X05kVgyzpuT0oMHz58YBvo7IRFi+Cgg+oTVGLAbQ0UtS1qF6gtr6ht\nUbtAbXlFbqtGnlumb1e+iHT/jG8BK7n7J5Z+VnzL/FUte+8Nb78Nv/510SUiIrKMaehVLaS5Ek4a\ncJR6EN2OPKb58+FXv4LJk4suERGR5Vyegcf7y75eBMxz9zfr0CONcNNN4N6Q0ywiIiK1yDO59Jmy\nx7MadDRe+d3latLRAXvskT6RtgEG1NZgUduidoHa8oraFrUL1JZX5LZq1DzwMLPzzexrFZaPNbMf\n1SdLyp19ds4Lhl56Ce6+u6GfRJu7rQmitkXtArXlFbUtaheoLa/IbdXIM7n0eaDN3bvKlg8FbnH3\njerY1zTRJ5d2d3czaNCg2p948cXw1a/Ciy/Ce95T/zAG0NYEUduidoHa8oraFrUL1JZXxLZaJpfm\nuZz2PcCrFZa/RvpUWGmA3G+yjg749KcbNuiAAbQ1QdS2qF2gtryitkXtArXlFbmtGnkGHn8G9qmw\nfF/gLwPLkbp64QW45x4YObLoEhERESDfVS3nApPNbD1g8U0h9gS+AZxQrzCpgxtvhJVWgv33L7pE\nREQEyHdVy5WkQcaRwN3Z43DgOHe/vL55stj48eNrf1JHBwwfDmuXf7xOfeVqa5KobVG7QG15RW2L\n2gVqyytyWzXyHPHA3S8GLs6Oeixw99frmyXlhgwZUtsTnn8eZsyAqVMb0lOq5rYmitoWtQvUllfU\ntqhdoLa8IrdVI89VLe8n3Rr9T2XLPwC87e5z6pfXPNGvaqnZj38MEyaky2nXXLPoGhERWYY1+qqW\nqcDHKizfOfueRDBtGuyzjwYdIiISSp6Bxw7A/1ZY/iDw4YHlSF3MnQsPPNDQm4aJiIjkkWfg4cB/\nVFi+JrDiwHKkN7Nmzap+5Z/+FFZdFdraGhdUoqa2JovaFrUL1JZX1LaoXaC2vCK3VSPPwONe4Ntm\ntmSQkf3/bwOtfQP5wCZMmFD9ytOmwX77wX9UGh/WX01tTRa1LWoXqC2vqG1Ru0BteUVuq0aeyaXb\nkAYf/wDuyxZ/knTEY3d3f7yuhU0SfXLp3Llzq5vJPGcOvP/90N4OX/hCw7ughrYCRG2L2gVqyytq\nW9QuUFteEdsaOrnU3f8IbAd0AO8lnXa5Btii9lSpVtVvso4OWG01+OxnGxtUItp/AKWitkXtArXl\nFbUtaheoLa/IbdXIex+PvwInA5jZGsAXgF8CO6F5HsXq6IDPfAZWX73oEhERkaXkmeMBgJntamZX\nA38FTiLdwXSXeoVJDrNnQ1eXPptFRETCqmngYWaDzexbZvYn4AbSJ9KuCuzv7t9y9981IlJg0qRJ\n/a/U0QGDBqWJpU1UVVtBorZF7QK15RW1LWoXqC2vyG3VqHrgYWa3Ak+S5necAGzo7l9tVJj01N3d\n3f9KHR0wYkQafDRRVW0FidoWtQvUllfUtqhdoLa8IrdVo+qrWsxsIXA+cHHp7dLN7G1g+2zSacuK\nflVLv556CrbcEm66CQ44oOgaERFZjjTqqpZPkK5g6TKz35rZWDNbdwCdUk8dHWlC6T77FF0iIiLS\nq6oHHu7+oLsfDWwAXEq6kuWv2Tb2MrPm3K1KKuvogM99Ll1KKyIiElSe+3i84e5XuvsngG2BHwLf\nAl4ys1vqHSjJ/Pnze//mE0/AY48V9tksfbYVLGpb1C5QW15R26J2gdryitxWjdyX0wK4+5PuPgHY\nCPhifZKkktGjR/f+zY4OWGMN2Hvv5gWV6LOtYFHbonaB2vKK2ha1C9SWV+S2qri7HmmC7VDAu7q6\nPKJeuxYtct96a/cvfam5QSWi/szc47ZF7XJXW15R26J2uastr4htXV1dTvoQ2aHez+/bmj+rZVnV\nsle1PP44bLst3HZbumOpiIhIkzX0s1okmI4OWGst2GuvoktERET6pYFHK3OHadPSfTtWWaXoGhER\nkX5p4NEipkyZsvTCRx9NNw4r6GqWxSq2BRG1LWoXqC2vqG1Ru0BteUVuq4YGHi1i5swKp8ymTYN1\n1oE992x+UImKbUFEbYvaBWrLK2pb1C5QW16R26qhyaWZlptc6g4f+ADsvjtcfnnRNSIishzT5NLl\nwcMPw+zZMHJk0SUiIiJV08CjVU2bBuuuC7vtVnSJiIhI1TTwaEXu6TLagw6ClVYqukZERKRqGni0\niLa2tn9/8bvfwZw5YU6z9GgLJmpb1C5QW15R26J2gdryitxWDQ08WsTYsWP//UVHB6y/Puy6a3FB\nJXq0BRO1LWoXqC2vqG1Ru0BteUVuq4auasm0zFUt7rDxxtDWBpMnF10jIiLSmle1mNkYM3vazBaY\n2YNm9pF+1j/BzGaZWbeZzTWzc81s1ZLvTzSzRWWPPzZ+TxrswQfh2WcLv2mYiIhIHiFmJprZSOCH\nwDHAQ8A4YLqZbeHu8yusfyhwJnAE8ACwBXA1sAg4qWTVx4E9Acu+XtigXWiejg7YYAP4+MeLLhER\nEalZlCMe44BL3f0ad58FHAt0A6N7WX8YMMPdp7n7XHe/C2gHPlq23kJ3n+fuL2WPlxu2Bw3W2dkJ\nixbBDTfAwQfDiisWnbREZ2dn0Qm9itoWtQvUllfUtqhdoLa8IrdVo/CBh5mtDOwI/GrxMk8TT+4i\nDTAquR/YcfHpGDPbFNgP+HnZeh8ws+fNbLaZ/cTM/l/dd6BJ2tvb4f774fnnw51maW9vLzqhV1Hb\nonaB2vKK2ha1C9SWV+S2ahQ+udTMNgCeB4a5+29Llk8CdnX3ioMPM/sqcA7pNMqKwCXuPqbk+3sD\nqwNPAhsA3wE2BD7k7m9U2F78yaVf+xrcfDM88wysUPiYUUREBGjRyaW1MLPdgJNJp2R2AA4EPmtm\npy5ex92nu/uN7v64u99JOiKyNtDn4YL99tuPtra2Ho9hw4YtdWjrjjvuqHgt9ZgxY5b65MCZM2fS\n1tbG/Pk9p6tMnDiRSZMm9Vg2d+5c2tramDVrVo/lF/z4x4yfMiWdZskGHd3d3bS1tTFjxowe67a3\ntzNq1Kil2kaOHFn8flxwAePHj++xTPuh/dB+aD+0H62zH+3t7Ut+Nw4ePJi2tjbGjRu31HN6E+GI\nx8qk+RwHufstJcunAmu6+wEVnnMv8IC7f7Nk2WGkeSKr9/FaDwF3uvspFb4X+4jHPfek26M/8ADs\nskvRNSIiIku01BEPd38b6CJdfQKAmVn29f29PG0Q6QqWUotKnrsUM1sd2Ax4YYDJxZg2DYYMgZ13\nLrpEREQkt8IHHplzgaPN7EtmthVwCWlwMRXAzK4xs/8uWf9W4DgzG2lmm5jZXsDpwC3ZxFTM7Adm\ntquZbWxmHwNuJl1O23qzchYuZNRVV6VJpZXHVYWqdFguiqhtUbtAbXlFbYvaBWrLK3JbNULcx8Pd\nO8xsXdLgYX3gD8De7j4vW2Ujet6D4wzSEY4zgPcB84BbgFNL1tkIuA54T/b9GcAu7v73Bu5KY9x7\nL8PffDPMZ7OUGz58eNEJvYraFrUL1JZX1LaoXaC2vCK3VaPwOR5RhJ7jccwxcNddMHt2yCMeIiKy\nfGupOR7Sj1dfheuugy99SYMOERFpeRp4RHfNNfCvf6WjHiIiIi1OA4/IFi1Kn0B70EHM+Mtfiq7p\nVfl14JFEbYvaBWrLK2pb1C5QW16R26qhgUdkv/oVPPUUjB3L2WefXXRNr9RWu6hdoLa8orZF7QK1\n5RW5rRqaXJoJObn0c59Lt0d/+GG6Fyxg0KBBRRdV1N3drbYaRe0CteUVtS1qF6gtr4httUwuDXE5\nrVQwZw7ceitcdhmYhXuTlVJb7aJ2gdryitoWtQvUllfktmroVEtUF18Ma64Jhx5adImIiEjdaOAR\n0YIFcMUVcOSR0OIjWxERkVIaeER0/fXwyitw3HFLFpV/smAkaqtd1C5QW15R26J2gdryitxWDQ08\nonGHCy6A/faDzTZbsnjIkCEFRvVNbbWL2gVqyytqW9QuUFtekduqoataMmGuanngAfjYx+AXv4B9\n9imuQ0REpEq6ZXormzwZNt8cWvxDgERERCrRwCOSv/0NbrgBxoyBFfRHIyIiyx79dovk8sth5ZXh\niCOW+tasWbOa31MltdUuaheoLa+obVG7QG15RW6rhgYeUbz9NlxyCfznf8Jaay317QkTJhQQVR21\n1S5qF6gtr6htUbtAbXlFbquGJpdmCp9cesMNcMgh8OijsO22S3177ty5YWcyq612UbtAbXlFbYva\nBWrLK2JbLZNLNfDIFD7w+NSnwAx+85vmv7aIiMgA6LNaWs2jj8K996ajHiIiIsswzfGI4MIL4X3v\nS59GKyIisgzTwKNor7wCP/kJHHtsuqKlF5MmTWpiVG3UVruoXaC2vKK2Re0CteUVua0aGngUberU\ndEXL0Uf3uVp3d3dzenJQW+2idoHa8oraFrUL1JZX5LZqaHJpppDJpYsWwRZbwC67pKMeIiIiLUiT\nS1vF9Okwe7YGHSIistzQqZYiTZ4MO+4IO+9cdImIiEhTaOBRlD//OX0C7dix6f4d/Zg/f34TovJR\nW+2idoHa8oraFrUL1JZX5LZqaOBRlIsvhnXWgZEjq1p99OjRDQ7KT221i9oFassralvULlBbXpHb\nquLueqQJtkMB7+rq8oZ7/XX3tdZy/+Y3q35KU7pyUlvtona5qy2vqG1Ru9zVllfEtq6uLgccGOr9\n/L7VVS2Zpl7Vcvnl6b4ds2fDJps09rVEREQarJarWnSqpdnc06TSESM06BARkeWOBh7NNmNG+myW\nsWOLLhEREWk6DTyabfJk2HJL2HPPmp42ZcqUBgUNnNpqF7UL1JZX1LaoXaC2vCK3VUMDj2Z6/nm4\n6aaqL6EtNXNmn6fMCqW22kXtArXlFbUtaheoLa/IbdXQ5NJMUyaXTpwI556bBiBrrNGY1xAREWky\nTS6N6K234NJL4ctf1qBDRESWWxp4NMuNN8KLL8KYMUWXiIiIFEYDj2aZPDlNKN1666JLRERECqOB\nRzPMnAn33z+gS2jb2trqGFRfaqtd1C5QW15R26J2gdryitxWDQ08muHCC2HIEPjsZ3NvYmzg+36o\nrXZRu0BteUVti9oFassrcls1dFVLpmFXtfz977DRRumKlm99q37bFRERCUJXtURy5ZXpNulHHll0\niYiISOHCDDzMbIyZPW1mC8zsQTP7SD/rn2Bms8ys28zmmtm5ZrbqQLZZd++8AxddBF/4Aqy3XlNf\nWkREJKIQAw8zGwn8EJgI7AA8Akw3s3V7Wf9Q4Mxs/a2A0cBI4Pt5t9kQt98Oc+bU5XNZOjs7B97T\nIGqrXdQuUFteUduidoHa8orcVo0QAw9gHHCpu1/j7rOAY4Fu0oCikmHADHef5u5z3f0uoB346AC2\nWX+TJ8POO8NOOw14U+3t7XUIagy11S5qF6gtr6htUbtAbXlFbqtG4ZNLzWxl0oDgIHe/pWT5VGBN\ndz+gwnO+CFwI7O3uvzOzTYHbgKvdfVLObdZ3cumTT8JWW8G118Lhhw98eyIiIkHVMrl0peYk9Wld\nYEXgxbLlLwJbVnqCu7dnp0xmmJllz7/E3Sfl3WbdXXRRmtdx8MFNeTkREZFWEOVUS03MbDfgZNLp\nkx2AA4HPmtmpRXYt8c9/wtSpcMwxsOqq/a4uIiKyvIgw8JgPvAOsX7Z8feBvvTzndOAad7/K3f/P\n3X9GGogsvlFGnm0CsN9++9HW1tbjMWzYsKUm89xxxx0V7x43ZswYphx3HLzxBnzlK0A6BNXW1sb8\n+fN7rDtx4kQmTZrUY9ncuXNpa2tj1qxZPZZfcMEFjB8/vsey7u5u2tramDFjRo/l7e3tjBo1aqm2\nkSNH1rYfU6b0WKb90H5oP7Qf2g/tR3t7+5LfjYMHD6atrY1x48Yt9ZxeuXvhD+BB4MclXxvwLDC+\nl/V/D5xZtuyLwOv8e95KrdscCnhXV5cPyKJF7tts437QQQPbTpkjjjiirturJ7XVLmqXu9ryitoW\ntctdbXlFbOvq6nLAgaHez+/8CHM8AM4FpppZF/AQ6YqUQcBUADO7BnjO3U/O1r8VGGdmfwB+C3yA\ndBTkFvcls2X73GbD/OY38Mc/ptuk19Hw4cPrur16UlvtonaB2vKK2ha1C9SWV+S2ahR+VctiZnY8\nMIF0OuQPwFfd/ffZ934NzHH30dnXKwCnAP8JvA+YB9wCnOrur1WzzQqvX5+rWg46KF3R8thjYJZ/\nO3c0o4sAABNOSURBVCIiIi2i1a5qAcDdLwIu6uV7e5R9vQg4I3vk2mZDzJ0LnZ3paIcGHSIiIkuJ\nMLl02XHppbD66rpvh4iISC808KiXN9+Eyy6DUaPS4KPOymceR6K22kXtArXlFbUtaheoLa/IbdXQ\nwKNebrgB5s+H449vyObPPvvshmy3HtRWu6hdoLa8orZF7QK15RW5rRphJpcWbcCTS3feGdZeG375\ny7q3QbrmetCgQQ3Z9kCprXZRu0BteUVti9oFassrYltLTi5taQ89lB633tqwl4j2JiulttpF7QK1\n5RW1LWoXqC2vyG3V0KmWerjwQnj/+2HffYsuERERCU0Dj4GaNw+uvz7N7VhxxaJrREREQtPAY6Cu\nuAJWWAFGj27oy5TfZz8StdUuaheoLa+obVG7QG15RW6rhgYeA7FwIVx8MRx2GKyzTkNfasiQIQ3d\n/kCorXZRu0BteUVti9oFassrcls1dFVLJtdVLTffDAceCDNnwg47NLRPREQkqlquatERj4GYPBk+\n/nENOkRERKqky2nz+uMf4de/hvb2oktERERaho545HXhhTB4cDrV0gSzZs1qyuvkobbaRe0CteUV\ntS1qF6gtr8ht1dDAI49XX4Wrr4avfAVWWaUpLzlhwoSmvE4eaqtd1C5QW15R26J2gdryitxWDU0u\nzdQ0ufSCC+DEE+GZZ2DDDZvSN3fu3LAzmdVWu6hdoLa8orZF7QK15RWxrZbJpRp4ZKoeeCxaBFtv\nnSaUXn990/pERESi0me1NNKvfgVPPQVTphRdIiIi0nI0x6NWkyfD9tuny2hFRESkJhp41GLOnPQJ\ntGPHgllTX3rSpElNfb1aqK12UbtAbXlFbYvaBWrLK3JbNTTwqMXFF8Oaa8Khhzb9pbu7u5v+mtVS\nW+2idoHa8oraFrUL1JZX5LZqaHJppt/JpQsWwEYbwahRcM45Te8TERGJSrdMb4Trr4dXXoHjjiu6\nREREpGVp4FEN93Tvjn33hc02K7pGRESkZWngUY0HH4SHH06TSgsyf/78wl67P2qrXdQuUFteUdui\ndoHa8orcVg0NPKoxeXI60rH33oUljB49urDX7o/aahe1C9SWV9S2qF2gtrwit1XF3fVIE2yHAt7V\n1eU9vPCC+8oru597rhdpqa5A1Fa7qF3uassralvULne15RWxraurywEHhno/v291VUum16tazjgD\nzjoLnnsO1l67sD4REZGodFVLvbz9NlxyCRx+uAYdIiIidaCBR186O+Gvf4UxY4ouERERWSZo4NGX\nyZNh111hu+2KLmFK4A+lU1vtonaB2vKK2ha1C9SWV+S2amjg0ZtHH4V77y30EtpSM2f2ecqsUGqr\nXdQuUFteUduidoHa8orcVg1NLs0sNbn0K1+B225LHwy38spF54mIiISlyaUD9cor8JOfwLHHatAh\nIiJSRxp4VDJ1arqi5eijiy4RERFZpmjgUW7RIrjwQjj4YBg8uOgaERGRZYoGHuUeeABmzw4zqXSx\ntra2ohN6pbbaRe0CteUVtS1qF6gtr8ht1dDAo9y0aTB0KOyyS9ElPYwNNhAqpbbaRe0CteUVtS1q\nF6gtr8ht1dBVLZklV7UAQ6+8EkaNKjpJRESkJeiqloFYYw34wheKrhAREVkmaeBRbv/9YbXViq4Q\nERFZJmngUe7gg4suqKizs7PohF6prXZRu0BteUVti9oFassrcls1wgw8zGyMmT1tZgvM7EEz+0gf\n695tZosqPG4tWeeqCt+/vd+QDTes0x7V16RJk4pO6JXaahe1C9SWV9S2qF2gtrwit1VjpaIDAMxs\nJPBD4BjgIWAcMN3MtnD3+RWecgCwSsnX6wKPAB1l6/0COAKw7Ot/1TG7qdZbb72iE3qlttpF7QK1\n5RW1LWoXqC2vyG3ViHLEYxxwqbtf4+6zgGOBbmB0pZXd/R/u/tLiBzAceAP4admq/3L3eSXrvtrI\nnRAREZG+FT7wMLOVgR2BXy1e5uka37uAYVVuZjTQ7u4LypbvZmYvmtksM7vIzNapS7SIiIjkUvjA\ng3SaZEXgxbLlLwL93rPczD4KfBC4ouxbvwC+BOwBTAA+BdxuZoaIiIgUIsQcjwE6EnjM3btKF7p7\n6XyP/zOzx4DZwG7A3RW28y6Avfbaiw996EM9vvHyyy9zxBFHsPvuuy9Z9sADD9DR0cF5553XY92z\nzjqLrbbaiv3333/JsieeeILLLruM00477f+3d/9BV5Z1HsffHxMkpZTJNjVlw7SsWBTHKae0Zw0F\no9WGEa1VlyWHxWob1HGG+GFIOItGE2KG40xt+SPQzTU3nV0BGaSUX2cV0zGk/IHKzyDZkF8lwnf/\nuK4HDg/neZ7DD891kM9r5gw8932fw+d+zs25v+e6r+u+6NGjx87ld955J926dWPo0KE7l61evZpJ\nkyYxYsQIevXqtXP53LlzGTJkCNdee+3OZVu3bmXMmDEMGTKEvn377lw+Y8YMFi5cyPjx43fLNmrU\nKAYMGHDA92POnDmMGDGirv24//77WbNmTcP2Y/bs2YwbN+6Avx/7ux+VSoX+/fu/I+/H/u7HokWL\naGlpKX5c1dqPBQsW0NLSUvy4qrUf8+bNo6WlpehxVWs/KpUKU6dOLX5c1dqPSqXCxIkTix9Xtfaj\nUqkwcuTI4sdVrf2oVCoMHz682HE1Y8YMZs6cyfr161m1ahW9e/dm48aNrZt2oxPF71yaL7VsAS6J\niIerlt8FHB0Rgzp47pHAKuCGiPhRHf/WWmBsRPy4xrrLgWl7vwdmZmaWXRER0zvaoHiLR0Rsk/Q0\n0A94GCBfDukH/LCTp19GGt3SacEg6UTgA8DqdjaZCVwBvAr8pZ7sZmZmBqSWjo+QzqUdKt7iASDp\nMuAu0miW1uG0g4HTImKdpHuAFRExps3zngCWR8TlbZYfBdwIPAisAU4BvgccBfSJiG3v7B6ZmZlZ\nLcVbPCD1x5B0LDAB+BDwW2BARKzLm5wIvF39HEkfAz4LXFDjJbcDfUidS48hXY6ZCYxz0WFmZlZO\nU7R4mJmZ2aGhGYbTmpmZ2SHChYeZmZk1zCFfeEg6V9LDklbmieQuLp2plaTRkiqS3sx3YH0o920p\nnevrkp6VtCE/5ku6sHSuWiSNyu/r5CbIcmONiQuXlM7VStIJku6V9CdJW/J7fGbhTMvamRDy9pK5\ncrbDJN0k6ZX8+3pJ0g2lc7WS1F3SFEmv5nxPSjqrQI5OP2MlTZC0Kud8TNIppXNJGiRpZv7/sENS\nn3c6Uz3ZJB0u6XuSnpO0KW9zt6TjG5Vvfx3yhQdppMtvgW8Czdbh5VzgduAzwPlAF2CWpPcWTQXL\ngW8DZ5Judz8H+JWkTxRN1Uae4Xg4aQLBZvE8qQP1cflxTtk4iaRjgHmkiRQHAJ8Argf+r2Qu4Cx2\n/a6OI3UmD/acELKEUcDVpM+O00h3SB4p6VtFU+3y76TbElwB9AYeA2YXOEF1+Bkr6dvAt0j/Vz9N\nmndrpqSubbdtZK68/gnS+9roc0NH2Y4EzgC+C/QlTZr6ceBXjQy4XyLCj/wAdgAXl87RQb5jc8Zz\nSmepke0N4Gulc1Tl6Q78nnTL/MeByU2Q6UZgcekc7WS7Bfh16Rx15JwC/KF0jpzlEeDHbZb9J3BP\nE2TrBmwDLmyz/ClgQsFce3zGkkYdXlf18/uBrcBlJXNVrfvbvL5Ps/zOamxzFmk054ml3tu9ebjF\n4+ByDKn6XV86SKvc3PxVUhW+oHSeKlOBRyJiTukgbZyam0ZflvRzSSeVDpRdBDwl6Rf5st5iScNK\nh6qW73J8BembfDOYD/STdCqApNOBzwH/UzRVcjhpDqy/tlm+lSZpZQOQ1IvUklU9SeibwCLqnyTU\ndp0b/lw6SD2a4j4e1rl8N9cpwJMRUbxfgKTepEKjG7ARGBQRS8umSnIhdAbpW0AzWQgMJbXEHA+M\nB34jqXdEbC6YC+Bk4BvAD4B/IzV5/1DSXyPi3qLJdhkEHA3cXTpIdgvp2/lSSdtJl67HRsT9ZWNB\nRGyStAD4jqSlpEk3LyedzF8sGm53x5FOmPs0SaiBpCNIx+L0iNhUOk89XHgcPO4APkn6RtUMlgKn\nk04Eg4F7JH2+dPGRb40/BTg/muxmcRFRfSvh5yVVgNdIt/7/WZlUOx0GVCLiO/nnZ3Nx+XWgWQqP\nq4BHI2JN6SDZV0gn868CS0jF7m2SVjVJsXYl8FNgJekGjIuB6aR+WfYuIOlw4AFS8fbNwnHq5kst\nBwFJPwIGAn8fEe3NNdNQEfF2RLwSEc9ExFhSB85rSucifah+EFgsaZukbUALcI2kt3LLUVOIiA3A\nH0i39C9tNfBCm2UvAD0LZNmDpJ6kDtZ7TPBY0CTg5oh4ICJ+FxHTgFuB0YVzARARyyLiPFJHxZMi\n4mzS3FavlE22mzWASB2uq30or7N2VBUdJwH9D5bWDnDh0fRy0fFl4LyIeL10ng4cBhxROgQwG/g7\n0rfP0/PjKeDnwOmRe2I1A0ndgY/S/sSFjTSP1DO+2sdJLTLN4CpS83sz9J9odSR7jjjYQZN9rkbE\n1oj4o6QepBFL/1U6U6uIWEYqMPq1LpP0ftJIvvmlctXQNJ8bsFvRcTLQLyJKjz7bK4f8pRalCeVO\nIVXdACfnTmLrI2J5uWQg6Q7gH4GLgc2SWr8VbIiIYjPoSpoIPAq8DryP1OGvBehfKlOr3Fditz4w\nkjYDb0RE22/0DSXp+6SREK8BHyYNh3sbuK9kruxWYJ6k0aShqp8BhgH/UjQVO/s3DQXuiogdheNU\newQYK2k58DvS8PLrgJ8UTZVJ6k/6XPs9cCqphWYJaULORubo7DN2CnCDpJdIs4PfBKzgHR4e2lmu\nXKj1JP1fFXBaPhbXRETbPikNy0b6ovIg6cvVPwBdqs4N65vtEnNNpYfVlH6QTpg7SEORqh8/bYJs\ntXJtB4YUzvUTUnPtVtK3lVnAF0r/vjrIO4fmGE57H+kDdSupaJsO9CqdqyrfQOA5YAvpRHpV6Uw5\n1wX5uD+ldJY2uY4CJgPLSPeeeJFUTB5eOlvOdynwUj7eVgK3Ae8rkKPTz1hSR+tV+dib2Yj3urNc\nwD+3s35cyWzsGt5bvbz158+XPu7qeXiSODMzM2uYproWaWZmZu9uLjzMzMysYVx4mJmZWcO48DAz\nM7OGceFhZmZmDePCw8zMzBrGhYeZmZk1jAsPMzMzaxgXHmZ2wEk6StJDkjZI2i7pyNKZzKw5uPAw\nsw5J+pmkX7ZZNljSVknXtfO0rwGfJs35cnxEbDlAWZ6QNOlAvJaZlXHITxJnZntH0jDgduDqiLin\nnc0+CiyJiKWNS1Y/SV3iYJhMy+xdyC0eZlY3SSNJk419pb2iQ9ITwDVAP0k7JM3Ky4+QNFnSSkmb\nJM2XdG7V846VdJ+kFZI2S3pW0qVV6+8FPgdcn193u6QTJA2TtK5Nhkskbav6+SZJ/ytpuKRlwMa8\nXJLGSnpF0hZJiyUNqnpeD0nTJa3N65dKuvIA/CrNDllu8TCzuki6BfgG8KWImNvBphcBPyC1elwK\nvJWX3wmcDAwmzWo8GJgh6VMR8SrwXmARMBHYRJrye5qklyLiGeBfSdO7PwVMAIiItZICqDXbZdtl\np+VsXybN5gkwLucYBrwMnAdMl/SFiFgA3EyannwA8Eb++xEd7LuZdcKFh5nVYyDphN2vk6KDiPiz\npC3AWxGxDkDSR4ArgRNalwHflzQQGAqMj4jlwJSql7o9r78UeCYi3sytGFsiYu0+7MN7gCsjYkPO\n1A0YSZpK/Om8zV2SWoCrgQXASfnffiavf30f/l0zq+LCw8zq8SxwLDBB0hcjYjOApCHA1LxNABdE\nxKIaz+9DOvG/LElVy7sCK/JrvQe4AbgE+HBe15XU0nAgLGstOrKPkVpZHm+TqQtQyX+/A3hA0lnA\nY8BD7eyfmdXJhYeZ1WMl6ZLEXNLlkQtz8fFL4Mmq7Va08/zupEsuZ9RYtyn/OZp0KecaYAmwmVTU\ndO0k2w5AbZZ1qbHd5hqZIF1G+WObdX8BiIj/ltQT+BJwPqlImRIRYzrJZGbtcOFhZnWJiOX5MsTj\nwExJAyJiE7sKh44sJhUDH+ygxeCzpBaF/wCQdBipT8fiqm3eIrWcVFsHHCOpa0S09ifpW0em5/Pr\n9cz9OWqKiD8BdwN3S1pA6l/iwsNsH7nwMLO6RcSKXHzMBWbllo+NdTxvqaRfkDqLXk+6dPM3QD/g\n6YiYBbwIXCTpbOBN4HrS5Z1qrwJn51aIzRHxBrCQ1EJxs6SppAKm05Enuc/IrcBtkroA84GjgXOA\n9RExTdJNpMsuS0iXZQbmv5vZPvJwWjPbKxGxCmgBPkC67NK9k6e0+idgGjAZWAo8CJwJLM/rJwDP\nAbOA2cBrwMNtXmMS6bLKC8BaSSfkFokhpBErz5H6iHy3zn0ZTRq5MoZUUDwKXAgsy5tsA24hFUqP\nkwocD6c12w+KqDUKzczMzOzAc4uHmZmZNYwLDzMzM2sYFx5mZmbWMC48zMzMrGFceJiZmVnDuPAw\nMzOzhnHhYWZmZg3jwsPMzMwaxoWHmZmZNYwLDzMzM2sYFx5mZmbWMC48zMzMrGH+H8NohRVNtloG\nAAAAAElFTkSuQmCC\n", 275 | "text/plain": [ 276 | "" 277 | ] 278 | }, 279 | "metadata": {}, 280 | "output_type": "display_data" 281 | } 282 | ], 283 | "source": [ 284 | "title('Accuracy depending on K-feature')\n", 285 | "xticks(range(13))\n", 286 | "xlabel('K-features')\n", 287 | "ylabel('Accuracy')\n", 288 | "plot(sbs.dims, sbs.scores, color='red')\n", 289 | "grid()" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "### 가장 높은 ACCURACY를 보이는 features값들\n", 297 | "\n", 298 | "더 높아지는 이유는 accuracy에 noise를 주거나 상관없는 값들을 없애면서 생기는 현상입니다. " 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": 218, 304 | "metadata": { 305 | "collapsed": false 306 | }, 307 | "outputs": [ 308 | { 309 | "name": "stdout", 310 | "output_type": "stream", 311 | "text": [ 312 | "Best Accuracy: 1.0\n", 313 | "['Alcohol', 'Ash', 'Alcalinity of ash', 'Flavanoids', 'Proanthocyanins', 'Hue']\n" 314 | ] 315 | } 316 | ], 317 | "source": [ 318 | "max_index = np.argmax(sbs.scores)\n", 319 | "max_subset = sbs.subsets[max_index]\n", 320 | "max_subset = list(max_subset)\n", 321 | "best_subset = COLUMNS[1:][max_subset].tolist()\n", 322 | "\n", 323 | "print('Best Accuracy:', sbs.scores[max_index])\n", 324 | "print(best_subset)" 325 | ] 326 | } 327 | ], 328 | "metadata": { 329 | "kernelspec": { 330 | "display_name": "Python 3", 331 | "language": "python", 332 | "name": "python3" 333 | }, 334 | "language_info": { 335 | "codemirror_mode": { 336 | "name": "ipython", 337 | "version": 3 338 | }, 339 | "file_extension": ".py", 340 | "mimetype": "text/x-python", 341 | "name": "python", 342 | "nbconvert_exporter": "python", 343 | "pygments_lexer": "ipython3", 344 | "version": "3.6.0" 345 | } 346 | }, 347 | "nbformat": 4, 348 | "nbformat_minor": 2 349 | } 350 | --------------------------------------------------------------------------------