├── .DS_Store ├── CreditCardFraud_Detection ├── .DS_Store ├── LogisticRegression_Part1.ipynb ├── LogisticRegression_Part2.ipynb └── LogisticRegression_PartIII.ipynb ├── DeepHedging ├── .DS_Store ├── DeepHedging_Part1.ipynb ├── rnn_050.zip └── rnn_099.zip ├── ExoticsDerviatePricingWithTensorFlow ├── .DS_Store ├── BarriersTensorFlow.ipynb └── BermudanTensorFlow.ipynb ├── Fraud_Anomaly_Detection ├── .DS_Store ├── .gitignore ├── BehaviorNet.ipynb ├── SampleData.ipynb └── version1 │ ├── BehaviorNet.ipynb │ └── SampleData.ipynb └── README.md /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/.DS_Store -------------------------------------------------------------------------------- /CreditCardFraud_Detection/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/CreditCardFraud_Detection/.DS_Store -------------------------------------------------------------------------------- /CreditCardFraud_Detection/LogisticRegression_Part1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "metadata": { 5 | "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", 6 | "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", 7 | "trusted": false, 8 | "collapsed": true 9 | }, 10 | "cell_type": "code", 11 | "source": "# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np\nimport pandas as pd\nimport tensorflow as tf\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import roc_curve, roc_auc_score, classification_report, accuracy_score, confusion_matrix \nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.", 12 | "execution_count": null, 13 | "outputs": [] 14 | }, 15 | { 16 | "metadata": { 17 | "_uuid": "6d3b41843580ae037846b7079f2fd79d7add5147", 18 | "_cell_guid": "41650b3a-dff8-4d14-9595-6b3fb4ccdea7" 19 | }, 20 | "cell_type": "markdown", 21 | "source": "## Load data and visualize the data" 22 | }, 23 | { 24 | "metadata": { 25 | "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a", 26 | "collapsed": true, 27 | "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0", 28 | "trusted": false 29 | }, 30 | "cell_type": "code", 31 | "source": "credit_card = pd.read_csv('../input/creditcard.csv')", 32 | "execution_count": null, 33 | "outputs": [] 34 | }, 35 | { 36 | "metadata": { 37 | "_uuid": "c4e55ed51d60613c4f19457af0131b4fb973fad9", 38 | "_cell_guid": "fd1b18c4-d0bf-4d81-b784-3fcec2578c12", 39 | "trusted": false, 40 | "collapsed": true 41 | }, 42 | "cell_type": "code", 43 | "source": "f, ax = plt.subplots(figsize=(7, 5))\nsns.countplot(x='Class', data=credit_card)\n_ = plt.title('# Fraud vs NonFraud')\n_ = plt.xlabel('Class (1==Fraud)')", 44 | "execution_count": null, 45 | "outputs": [] 46 | }, 47 | { 48 | "metadata": { 49 | "_uuid": "5b132aff2b6d28b35e2c0ff256a3f97ff74f3cb3", 50 | "_cell_guid": "19175b74-8157-418c-b6b0-6c2a1b719814" 51 | }, 52 | "cell_type": "markdown", 53 | "source": "As we can see we have mostly non-fraudulent transactions. Such a problem is also called inbalanced class problem.\n\n99.8% of all transactions are non-fraudulent. The easiest classifier would always predict no fraud and would be in almost all cases correct. Such classifier would have a very high accuracy but is quite useless." 54 | }, 55 | { 56 | "metadata": { 57 | "_uuid": "5ee8b7fcd59def3b33733b5bbfddf39709dfd077", 58 | "_cell_guid": "0c90460f-5a34-46a0-bd6f-88e699b27d2f", 59 | "trusted": false, 60 | "collapsed": true 61 | }, 62 | "cell_type": "code", 63 | "source": "base_line_accuracy = 1-np.sum(credit_card.Class)/credit_card.shape[0]\nbase_line_accuracy", 64 | "execution_count": null, 65 | "outputs": [] 66 | }, 67 | { 68 | "metadata": { 69 | "_uuid": "8ecd2b9433bc7759f08a3c5c08627a57f7a2a2e3", 70 | "_cell_guid": "0f4090aa-ccfc-442f-ada5-03466fccbb59" 71 | }, 72 | "cell_type": "markdown", 73 | "source": "For such an inbalanced class problem we could use over or undersampling methods to try to balance the classes (see inbalance-learn for example: https://github.com/scikit-learn-contrib/imbalanced-learn), but this out of the scope of todays post. We will come back to this in a later post.\n\nAs accuracy is not very informative in this case the AUC (Aera under the curve) a better metric to assess the model quality. The AUC in a two class classification class is equal to the probability that our classifier will detect a fraudulent transaction given one fraudulent and genuiune transaction to choice from. Guessing would have a probability of 50%." 74 | }, 75 | { 76 | "metadata": { 77 | "_uuid": "16728c540fc206821df972fcd071646f48b385e5", 78 | "collapsed": true, 79 | "_cell_guid": "3de8033f-5ddb-4a12-9230-c1cd01ab68e2", 80 | "trusted": false 81 | }, 82 | "cell_type": "code", 83 | "source": "X = credit_card.drop(columns='Class', axis=1)\ny = credit_card.Class.values", 84 | "execution_count": null, 85 | "outputs": [] 86 | }, 87 | { 88 | "metadata": { 89 | "_uuid": "7fd243cb53c6126ef7001ac3a8c92649b6effd94", 90 | "_cell_guid": "cacba1af-23cd-4a35-9603-c904387956b4" 91 | }, 92 | "cell_type": "markdown", 93 | "source": "Due to the construction of the dataset (PCA transformed features, which minimizes the correlation between factors), we dont have any highly correlated features. Multicolinearity could cause problems in a logisitc regression.\n\nTo test for multicolinearity one could look into the correlation matrix (works only for non categorical features) or run partial regressions and compare the standard errors or use pseudo-R^2 values and calculate Variance-Inflation-Factors.\n\n" 94 | }, 95 | { 96 | "metadata": { 97 | "_uuid": "5c3a98163d12601711b33425f479edfb37a92fca", 98 | "_cell_guid": "ff310572-91fe-40d7-8370-89088320d9f0", 99 | "trusted": false, 100 | "collapsed": true 101 | }, 102 | "cell_type": "code", 103 | "source": "corr = X.corr()\n\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\ncmap = sns.diverging_palette(220, 10, as_cmap=True)\n\n# Draw the heatmap with the mask and correct aspect ratio\nf, ax = plt.subplots(figsize=(11, 9))\nsns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n square=True, linewidths=.5, cbar_kws={\"shrink\": .5})", 104 | "execution_count": null, 105 | "outputs": [] 106 | }, 107 | { 108 | "metadata": { 109 | "_uuid": "be916ae070cd9df31f67306364b5faa00de99b79", 110 | "_cell_guid": "65c81b41-418d-47fa-a484-98e643aaf208" 111 | }, 112 | "cell_type": "markdown", 113 | "source": "## Logisitc Regression with Sklearn\n\nShort reminder of Logistic Regression:\n\nIn Logisitic Regression the logits (logs of the odds) are assumed to be a linear function of the features\n\n$$L=\\log(\\frac{P(Y=1)}{1-P(Y=1)}) = \\beta_0 + \\sum_{i=1}^n \\beta_i X_i. $$\n\nSolving this equatation for $p=P(Y=1)$ yields to\n\n$$ p = \\frac{\\exp(L)}{1-\\exp(L)}.$$\n\nThe parameters $\\beta_i$ can be derived by Maximum Likelihood Estimation (MLE). The likelihood for a given $m$ observation $Y_j$ is\n\n$$ lkl = \\prod_{j=1}^m p^{Y_j}(1-p)^{1-Y_j}.$$\n\nTo find the maximum of the likelihood is equivalent to the minimize the negative logarithm of the likelihood (loglikelihood).\n\n$$ -llkh = -\\sum_{j=1}^m Y_j \\log(p) + (1-Y_j) \\log(1-p),$$\n\nwhich is numerical more stable. The log-likelihood function has the same form as the cross-entropy error function for a discrete case.\n\nSo finding the maximum likelihood estimator is the same problem as minimizing the average cross entropy error function.\n\nIn SciKit-Learn uses by default a coordinate descent algorithm to find the minimum of L2 regularized version of the loss function (see. http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression).\n\nThe main difference between L1 (Lasso) and L2 (Ridge) regulaziation is, that the L1 prefer a sparse solution (the higher the regulazation parameter the more parameter will be zero) while L2 enforce small parameter values." 114 | }, 115 | { 116 | "metadata": { 117 | "_uuid": "67c42133bf74c0f3aaa84e49456fe04a2f5e9d96", 118 | "_cell_guid": "763a278e-91af-4fe6-a255-ddaea0bbd790" 119 | }, 120 | "cell_type": "markdown", 121 | "source": "## Train the model\n\n### Training and test set\n\nFirst we split our data set into a train and a validation set by using the function train_test_split. The model performace " 122 | }, 123 | { 124 | "metadata": { 125 | "_uuid": "d19b826169ff464f82eba4e0b349bacc4361c8a0", 126 | "collapsed": true, 127 | "_cell_guid": "d54b91c2-fc78-4b24-a7c8-5b1a974cdf36", 128 | "trusted": false 129 | }, 130 | "cell_type": "code", 131 | "source": "np.random.seed(42)\nX_train, X_test, y_train, y_test = train_test_split(X, y)", 132 | "execution_count": null, 133 | "outputs": [] 134 | }, 135 | { 136 | "metadata": { 137 | "_uuid": "d1ea1658ffca81f4f9c83e9759b90f1311ce7a71", 138 | "_cell_guid": "88a09d8f-4238-4560-a180-17d048ae5dd7" 139 | }, 140 | "cell_type": "markdown", 141 | "source": "### Model definition\n\nAs preperation we standardize our features to have zero mean and a unit standard deviation. The convergence of gradient descent algorithm are better. We use the class `StandardScaler`. The class *StandardScaler* has the method `fit_transform()` which learn the mean $\\mu_i$ and standard deviation $\\sigma_i$ of each feature $i$ and return a standardized version $\\frac{x_i - \\mu_i}{\\sigma}$. We learn the mean and sd on the training data. We can apply the same standardization on the test set with the function *transform()*.\n\n\nThe logistic regression is implemented in the class `LogisticRegression`, we will use for now the default parameterization. The model can be fit using the function `fit()`. After fitting the model can be used to make predicitons `predict()` or return the estimated the class probabilities `predict_proba()`.\n\nWe combine both steps into a Pipeline. The pipline performs both steps automatically. When we call the method `fit()` of the pipeline, it will invoke the method `fit_and_transform()` for all but the last step and the method `fit()` of the last step, which is equivalent to:\n\n```python\nlr.fit(scaler.fit_transform(X_train), y_train)\n```\n\nor visualized as a dataflow:\n\n```X_train => scaler.fit_transform(.) => lr.fit(., y_train)```\n\nIf we invoke the method `predict()` of the pipeline its equvivalent to\n\n\n```python\nlr.predict(scaler.transform(X_train))\n```\n\n\n\n" 142 | }, 143 | { 144 | "metadata": { 145 | "_uuid": "86ecda4d1700405da9ed0d8bc6f7d8eabc0f1ddb", 146 | "collapsed": true, 147 | "_cell_guid": "fb45b18e-9357-4a98-9d98-078f96c3a591", 148 | "trusted": false 149 | }, 150 | "cell_type": "code", 151 | "source": "scaler = StandardScaler()\nlr = LogisticRegression()\nmodel1 = Pipeline([('standardize', scaler),\n ('log_reg', lr)])", 152 | "execution_count": null, 153 | "outputs": [] 154 | }, 155 | { 156 | "metadata": { 157 | "_uuid": "7c6abdd6d0db2ffede02f01d2d3f1ff916de0eee", 158 | "_cell_guid": "804537d0-b635-4e90-a88c-56fac8749e2e" 159 | }, 160 | "cell_type": "markdown", 161 | "source": "In the next step we fit our model to the training data" 162 | }, 163 | { 164 | "metadata": { 165 | "_uuid": "bc769713962566efac8b2310f45d7ebe79dcb173", 166 | "_cell_guid": "b8f46ca5-f69a-4755-abbf-9cc76723c133", 167 | "trusted": false, 168 | "collapsed": true 169 | }, 170 | "cell_type": "code", 171 | "source": "model1.fit(X_train, y_train)", 172 | "execution_count": null, 173 | "outputs": [] 174 | }, 175 | { 176 | "metadata": { 177 | "_uuid": "dc612edede19c594a8f28e3c396f0797a98888ba", 178 | "_cell_guid": "4c3a4e42-36ee-4814-b8b2-0513ed7404cb" 179 | }, 180 | "cell_type": "markdown", 181 | "source": "### Training score and Test score\n\n`confusion_matrix()` returns the confusion matrix, C where $C_{0,0}$ are the true negatives (TN) and $C_{0,1}$ the false positives (FP) and vice-versa for the positives in the 2nd row. We use the function `accurary_score()` to calculate the accuracy our models on the train and test data. We see that the accuracy is quite high (99,9%) which is expected in such an unbalanced class problem. With the method `roc_auc_score()`can we get the area under the receiver-operator-curve (AUC) for our simple model." 182 | }, 183 | { 184 | "metadata": { 185 | "_uuid": "e4f22259d9018f233874d9e4afdb42ec882bd967", 186 | "_cell_guid": "f708ecc2-9114-4b5e-86fe-bd8423aa622c", 187 | "trusted": false, 188 | "collapsed": true 189 | }, 190 | "cell_type": "code", 191 | "source": "y_train_hat = model1.predict(X_train)\ny_train_hat_probs = model1.predict_proba(X_train)[:,1]\ntrain_accuracy = accuracy_score(y_train, y_train_hat)*100\ntrain_auc_roc = roc_auc_score(y_train, y_train_hat_probs)*100\nprint('Confusion matrix:\\n', confusion_matrix(y_train, y_train_hat))\nprint('Training accuracy: %.4f %%' % train_accuracy)\nprint('Training AUC: %.4f %%' % train_auc_roc)", 192 | "execution_count": null, 193 | "outputs": [] 194 | }, 195 | { 196 | "metadata": { 197 | "_uuid": "6c97c5b3180d8cc2cd7019137aae19d45f53906d", 198 | "_cell_guid": "94f52985-d9ca-4fa3-ba67-84b1ab4f27c6" 199 | }, 200 | "cell_type": "markdown", 201 | "source": "![](http://)Our model is able to detect 68 fraudulent transactions out of 113 (recall of 60%) and produce 12 false alarms (<0.02%) on the test data." 202 | }, 203 | { 204 | "metadata": { 205 | "_uuid": "a83a2229893813ad2986910d20f562b369b76eb4", 206 | "_cell_guid": "ca356e72-46ac-4766-8640-11d644c4534f", 207 | "trusted": false, 208 | "collapsed": true 209 | }, 210 | "cell_type": "code", 211 | "source": "y_test_hat = model1.predict(X_test)\ny_test_hat_probs = model1.predict_proba(X_test)[:,1]\ntest_accuracy = accuracy_score(y_test, y_test_hat)*100\ntest_auc_roc = roc_auc_score(y_test, y_test_hat_probs)*100\nprint('Confusion matrix:\\n', confusion_matrix(y_test, y_test_hat))\nprint('Training accuracy: %.4f %%' % test_accuracy)\nprint('Training AUC: %.4f %%' % test_auc_roc)", 212 | "execution_count": null, 213 | "outputs": [] 214 | }, 215 | { 216 | "metadata": { 217 | "_uuid": "c22dc7db8f1a4c9d133c74ec502cfe9614667be8", 218 | "_cell_guid": "fddb2dfa-7826-40ce-b49d-6177fc659aa2" 219 | }, 220 | "cell_type": "markdown", 221 | "source": "With the function `classification_report()` we print get the precision, recall per each class." 222 | }, 223 | { 224 | "metadata": { 225 | "_uuid": "258dda7a672ec39dd33f095dbc59cc42832f50b0", 226 | "_cell_guid": "26394d30-80ec-4db0-929c-0a81585d4d66", 227 | "trusted": false, 228 | "collapsed": true 229 | }, 230 | "cell_type": "code", 231 | "source": "print(classification_report(y_test, y_test_hat, digits=6))", 232 | "execution_count": null, 233 | "outputs": [] 234 | }, 235 | { 236 | "metadata": { 237 | "_uuid": "e0fed511c8db4728079d5210a3a6a97a5eadfb72", 238 | "_cell_guid": "c715932b-cac8-4582-a866-dcc913a5b6d0" 239 | }, 240 | "cell_type": "markdown", 241 | "source": "To visualize the Receiver-Operator-Curve we use the function `roc_curve`. The method returns the true positive rate (recall) and the false positive rate (probability for a false alarm) for a bunch of different thresholds. This curve shows the trade-off between recall (detect fraud) and false alarm probability.\n\nIf we classifiy all transaction as fraud, we would have a recall of 100% but also the highest false alarm rate possible (100%). The naive way to minimize the false alarm probability is to classify all transaction as legitime. **" 242 | }, 243 | { 244 | "metadata": { 245 | "_uuid": "9317259c2c241aa6cb00346b5b91bd485d7ec448", 246 | "_cell_guid": "d5db8aad-8866-4639-9a1c-cd32e33e11a1", 247 | "trusted": false, 248 | "collapsed": true 249 | }, 250 | "cell_type": "code", 251 | "source": "fpr, tpr, thresholds = roc_curve(y_test, y_test_hat_probs, drop_intermediate=True)\n\nf, ax = plt.subplots(figsize=(9, 6))\n_ = plt.plot(fpr, tpr, [0,1], [0, 1])\n_ = plt.title('AUC ROC')\n_ = plt.xlabel('False positive rate')\n_ = plt.ylabel('True positive rate')\nplt.style.use('seaborn')\n\nplt.savefig('auc_roc.png', dpi=600)", 252 | "execution_count": null, 253 | "outputs": [] 254 | }, 255 | { 256 | "metadata": { 257 | "_uuid": "9e1692cf8dd385592b20e30f229ff192a30ce5c7", 258 | "collapsed": true, 259 | "_cell_guid": "9d834f9c-5817-4df8-b4a4-e31103236daf" 260 | }, 261 | "cell_type": "markdown", 262 | "source": "Our model classify all transaction with a fraud probability => 50% as fraud. If we choose the threshold higher, we could reach a lower false positive rate but we would also miss more fraudulent transactions. If we choose the thredhold lower we can catch more fraud but need to investigate more false positives.\n\nDepending on the costs for each error, it make sense to select another threshold.\n\nIf we set the threshold to 90% the recall decrease from 60% to 45%. while the false positve rate is the same. We can see that our model assign some non-fraudulent a very high probability to be fraud" 263 | }, 264 | { 265 | "metadata": { 266 | "_uuid": "dc371656eeeced96d8343a37312d92bf116eede0", 267 | "_cell_guid": "a5a4d2a5-c86d-4e27-8245-08325644794d", 268 | "trusted": false, 269 | "collapsed": true 270 | }, 271 | "cell_type": "code", 272 | "source": "y_hat_90 = (y_test_hat_probs > 0.90 )*1\nprint('Confusion matrix:\\n', confusion_matrix(y_test, y_hat_90))\nprint(classification_report(y_test, y_hat_90, digits=6))\n", 273 | "execution_count": null, 274 | "outputs": [] 275 | }, 276 | { 277 | "metadata": { 278 | "_uuid": "9f177e8213acc10e0dc7ab4f49c33a06989dce9d", 279 | "_cell_guid": "d8009ffc-1caf-4cf5-808b-3347ad39eb49" 280 | }, 281 | "cell_type": "markdown", 282 | "source": "If we set the threshold down to 10%, we can detect around 75% of all fraud case but almost double our false positive rate (now 25 false alarms)" 283 | }, 284 | { 285 | "metadata": { 286 | "_uuid": "69fbb96b4062e67e8df8beddab7cd48598b6e377", 287 | "_cell_guid": "df14952e-6d10-4964-b977-fd732a878347", 288 | "trusted": false, 289 | "collapsed": true 290 | }, 291 | "cell_type": "code", 292 | "source": "y_hat_10 = (y_test_hat_probs > 0.10)*1\nprint('Confusion matrix:\\n', confusion_matrix(y_test, y_hat_10))\nprint(classification_report(y_test, y_hat_10, digits=4))", 293 | "execution_count": null, 294 | "outputs": [] 295 | }, 296 | { 297 | "metadata": { 298 | "_uuid": "500e582bb9055575cf61d0e7fccd465afbab1c5b", 299 | "_cell_guid": "9e9a216b-0533-46d8-bec0-a82b7bccc667" 300 | }, 301 | "cell_type": "markdown", 302 | "source": "Where to go from here?\n-------------------\n\nWe just scratched the surface of sklearn and logistic regression. For example we could spent much more time with the \n\n- feature selection / engineering (which is a bit hard without any background information about the features),\n- we could try techniques to counter the data inbalance and \n- we could use cross-validation to fine tune the hyperparameters (regulaziation constant C) or\n- try a different regulization (Lasso/Elastic Net) or \n- optimizer (stochastic gradient desent instead of coordinate descnet)\n- adjust class weights to move the decision boundary (make missed frauds more expansive in the loss function)\n!- and finally we could try different classifer models in sklearn like decision trees, random forrests, knn, naive bayes or support vector machines. \n\nBut for now we will stop here and will implement a logisitc regression model with stochastic gradient descent in TensorFlow and then extend it to a neural net.\n\n" 303 | }, 304 | { 305 | "metadata": { 306 | "_uuid": "524a67dd28ec3821aa96eeff657fdabefdc31d73", 307 | "collapsed": true, 308 | "_cell_guid": "8c14b78c-f44c-4806-bac2-406eccb3112d", 309 | "trusted": false 310 | }, 311 | "cell_type": "code", 312 | "source": "", 313 | "execution_count": null, 314 | "outputs": [] 315 | }, 316 | { 317 | "metadata": { 318 | "_uuid": "9d30d8caf08443532574ea35023cf4e7166ed384", 319 | "collapsed": true, 320 | "_cell_guid": "120534db-b6c5-4790-bd74-0050d2e1c1dd", 321 | "trusted": false 322 | }, 323 | "cell_type": "code", 324 | "source": "", 325 | "execution_count": null, 326 | "outputs": [] 327 | } 328 | ], 329 | "metadata": { 330 | "language_info": { 331 | "file_extension": ".py", 332 | "codemirror_mode": { 333 | "name": "ipython", 334 | "version": 3 335 | }, 336 | "nbconvert_exporter": "python", 337 | "pygments_lexer": "ipython3", 338 | "name": "python", 339 | "version": "3.6.5", 340 | "mimetype": "text/x-python" 341 | }, 342 | "kernelspec": { 343 | "display_name": "Python 3", 344 | "language": "python", 345 | "name": "python3" 346 | } 347 | }, 348 | "nbformat": 4, 349 | "nbformat_minor": 1 350 | } -------------------------------------------------------------------------------- /CreditCardFraud_Detection/LogisticRegression_Part2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "metadata": { 5 | "_cell_guid": "e213423f-e835-4674-ac02-30b212ebfaa3", 6 | "_uuid": "611fb07afbef4c67abd933990ba3cdc671b2853e" 7 | }, 8 | "cell_type": "markdown", 9 | "source": "# From Logistic Regression in SciKit-Learn to Deep Learning with TensorFlow – A fraud detection case study – Part II\n\nWe will continue to build our credit card fraud detection model. In the previous post we used scikit-learn to detect fraudulent transactions with a logistic regression model. This time we will build a logistic regression in TensorFlow from scratch. We will start with some TensorFlow basics and then see how to minimize a loss function with (stochastic) gradient descent. \n\nWe will fit our model to our training set by minimizing the cross entropy and minimizing the cross entropy is aquivalent to maximizing the likelihood (see previous part for details). In the next part we will extend this model with some hidden layer and build a deep neural network to detect fraud. And we will see how to use the High-Level API to build the same model much easier and quicker.\n\nAs usual you will find the corresponding notebook on GitHub.\n\nFirst we will load the data, split it into a training and test set and normalize the features as in our previous model.\n " 10 | }, 11 | { 12 | "metadata": { 13 | "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", 14 | "_kg_hide-input": true, 15 | "_kg_hide-output": true, 16 | "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5", 17 | "collapsed": true, 18 | "trusted": false 19 | }, 20 | "cell_type": "code", 21 | "source": "import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport keras\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import roc_curve, roc_auc_score, classification_report, accuracy_score\nimport seaborn as sns\nimport matplotlib.pyplot as plt", 22 | "execution_count": 12, 23 | "outputs": [] 24 | }, 25 | { 26 | "metadata": { 27 | "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0", 28 | "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a", 29 | "collapsed": true, 30 | "trusted": false 31 | }, 32 | "cell_type": "code", 33 | "source": "credit_card = pd.read_csv('../input/creditcard.csv')", 34 | "execution_count": 13, 35 | "outputs": [] 36 | }, 37 | { 38 | "metadata": { 39 | "_cell_guid": "8f0b146a-362a-4a96-af7c-694fa40fb63e", 40 | "_uuid": "e81516a64b2983fea7c09e0cd562f0eef27caf24", 41 | "collapsed": true, 42 | "trusted": false 43 | }, 44 | "cell_type": "code", 45 | "source": "X = credit_card.drop(columns='Class', axis=1)\ny = credit_card.Class.values", 46 | "execution_count": 14, 47 | "outputs": [] 48 | }, 49 | { 50 | "metadata": { 51 | "_cell_guid": "d66eb884-7fe7-4a08-a644-88aa965b9a27", 52 | "_uuid": "b0f365b6bc854e6558fb080368766910d4a98c14", 53 | "collapsed": true, 54 | "trusted": false 55 | }, 56 | "cell_type": "code", 57 | "source": "np.random.seed(42)\nX_train, X_test, y_train, y_test = train_test_split(X, y)\nscaler = StandardScaler()\nX_train = scaler.fit_transform(X_train)\nX_test = scaler.transform(X_test)", 58 | "execution_count": 15, 59 | "outputs": [] 60 | }, 61 | { 62 | "metadata": { 63 | "_cell_guid": "8045b5b7-650e-4fc9-aafd-24370ce7eb5c", 64 | "_uuid": "f46cbae4db147cc30d81d78d6421b802d140485c" 65 | }, 66 | "cell_type": "markdown", 67 | "source": "## Short TensorFlow introduction \n\n\nTensorFlow uses a dataflow graph to represent the computation in terms of the dependencies between individual operations. \n\nWe first define the dataflow graph, then we create a TensorFlow session to run or calculate it.\n\nStart with a very simple graph. Lets say we have a two dimensional vector \n\n$ x =(x_1,x_2) $\n\nand want to compute $a x^2 = (ax_1^2, ax_2^2)$, where $a$ is a constant (e.g.5).\n\nFirst we define the input, a `tf.placeholder` object. We will use `tf.placeholder` to feed our data to our models (like our features and the value we try to predict). We can also use `tf.placerholder` to feed hyperparameter in our model (e.g. a learning_rate). We have to define the type of the input (in this case it's a floating point number) and the shape of the input (a two dimensional vector). \n\nAnother class is the `tf.constant`, as the name indicates it represent a constant in our computational graph." 68 | }, 69 | { 70 | "metadata": { 71 | "_cell_guid": "c8c9ebc7-f9b9-4837-a131-caeea4f8377d", 72 | "_uuid": "683013951b7175c8651a52fd51e0ae1954088e81", 73 | "collapsed": true, 74 | "trusted": false 75 | }, 76 | "cell_type": "code", 77 | "source": "input_x = tf.placeholder(tf.float32)\na = tf.constant(5., tf.float32, name='a', shape=(2,))", 78 | "execution_count": 16, 79 | "outputs": [] 80 | }, 81 | { 82 | "metadata": { 83 | "_cell_guid": "fcfbbd32-a232-4621-aba3-648d9886fad3", 84 | "_uuid": "1dec22092bcc1377ede5bf6a267febcb8cb10cb0" 85 | }, 86 | "cell_type": "markdown", 87 | "source": " Now we define our operator y." 88 | }, 89 | { 90 | "metadata": { 91 | "_cell_guid": "a02b8e1b-2f43-41b4-bb0c-16399f78924a", 92 | "_uuid": "56d9aede3e51c809c3b188c60427a3a26771d858", 93 | "collapsed": true, 94 | "trusted": false 95 | }, 96 | "cell_type": "code", 97 | "source": "y = a * tf.pow(input_x,2)", 98 | "execution_count": 17, 99 | "outputs": [] 100 | }, 101 | { 102 | "metadata": { 103 | "_cell_guid": "3fa8114f-e5da-42ea-8448-713c52f4759d", 104 | "_uuid": "98205774cfb40ae86baf6466f796f40a7c6e6f8c" 105 | }, 106 | "cell_type": "markdown", 107 | "source": "We can create a TensorFlow session and run our operator with the ´run()´ method of the session. We have to feed our input vector `x` as a dictionary into the graph. Lets say $x=(1,2)$" 108 | }, 109 | { 110 | "metadata": { 111 | "_cell_guid": "e4d2be23-4d0f-44dd-9241-a62e78387103", 112 | "_uuid": "84315e9597fced012667131707366cfc76b00a2d", 113 | "trusted": false 114 | }, 115 | "cell_type": "code", 116 | "source": "x = np.array([1,2])\nwith tf.Session() as sess:\n result = sess.run(y, {input_x: x})\n print(result)", 117 | "execution_count": 18, 118 | "outputs": [ 119 | { 120 | "name": "stdout", 121 | "output_type": "stream", 122 | "text": "[ 5. 20.]\n" 123 | } 124 | ] 125 | }, 126 | { 127 | "metadata": { 128 | "_cell_guid": "c6abf39c-d9f9-4e01-9551-9d3d88c56567", 129 | "_uuid": "7f6801a2390afae050df4f0fdeb20b6592b05563" 130 | }, 131 | "cell_type": "markdown", 132 | "source": "### Automatic differentiation in TensorFlow\n\nMost machine learning and deep learning problems are at the end minimization problems. We either have to minimize a loss function or we have to maximize a reward function in the case of reinforced learning. A bread-and-butter method to solve these kind of problems is the gradient descent (ascent) method.\n\nThe power of TensorFlow is the automatic or algorithmic differentiation of our computational graph. We can get the anayltical derviates (or gradients) for almost 'free'. We dont need to derive the formula for the gradient by ourself or implement it.\n\nThe gradient $\\nabla f(x)$ is the multidimensional generalization of the derivate of a function. Its the vector of the partial derivates of the function and points into the direction with the strongest increase. If we have have real valued function $f(x)$ with $x$ being a n-dimensional vector, then $f$ is decreasing the fastest when we go from point $x$ into the direction of the negative gradient. \n\nTo get the gradient we use the `tf.gradient`class. Lets say we want to derive y with respect to our input x. The function call looks like that:" 133 | }, 134 | { 135 | "metadata": { 136 | "_cell_guid": "72ab7a38-2922-4fbd-bb42-5151c2a3fca9", 137 | "_uuid": "5d8f40c51837dfc66ab94397ac9975fdee3c959c", 138 | "trusted": false 139 | }, 140 | "cell_type": "code", 141 | "source": "g = tf.gradients(y, [input_x])\ngrad_y_x = 0\nwith tf.Session() as sess:\n grad_y_x = sess.run(g,{input_x: np.array([1,2])})\n print(grad_y_x)\n", 142 | "execution_count": 19, 143 | "outputs": [ 144 | { 145 | "name": "stdout", 146 | "output_type": "stream", 147 | "text": "[array([10., 20.], dtype=float32)]\n" 148 | } 149 | ] 150 | }, 151 | { 152 | "metadata": { 153 | "_cell_guid": "e7ba75b5-fa67-4afb-bfe0-202fed2b229d", 154 | "_uuid": "845a2fa997a9d82cec0f682297deb976fc7abfbf" 155 | }, 156 | "cell_type": "markdown", 157 | "source": "### Chain Rule in TensorFlow\n\nLets check a simple example for the chain rule. We come later back to the chain rule when we talk about back propagation in the coming part. Back propagation is one of the key concepts in deep learning. Lets recall the chain rule, which we all learned at high school:\n\n$$\\frac{d}{dx}f(g(x)) = f'(g(x))g'(x)$$\n\nFor our example we use our function y and chain it to a new function z=log(y).\n\nThe 'inner' partial derivate of $g$ with respect to $x_i$ is $\\frac{\\partial y}{\\partial x_i} = 10x_i $ and the outer one with respect to $y$ is $\\frac{\\partial z}{\\partial y_i} =\\frac{1}{5x_i^2} $. The partial derivate is $\\frac{2x_i }{x_i^2}$.\n\nWith TensorFlow, we can calulate the outer and inner derivate seperatly or in one step.\nIn example we will calculate two gradients one with respect to $y$ and one with respect to $x$. Multiplying elementwise the gradient with respect to $y$ with the gradient of $y$ with respect to $x$ (inner derivative) yield to the gradient of $z$ with respect to $x$.\n" 158 | }, 159 | { 160 | "metadata": { 161 | "_cell_guid": "9b750fba-21ec-466e-be05-03e2f015374f", 162 | "_uuid": "0c33805554e7dfd3ff11d372cf055fada34565b7", 163 | "trusted": false 164 | }, 165 | "cell_type": "code", 166 | "source": "z = tf.log(y)\nwith tf.Session() as sess:\n result_z = sess.run(z, {input_x: np.array([1,2])})\n print('z =', result_z)\n delta_z = tf.gradients(z, [y, input_x])\n grad_z_y, grad_z_x = sess.run(delta_z, {input_x: np.array([1,2])})\n print('Gradient with respect to y', grad_z_y)\n print('Gradient with respect to x', grad_z_x)\n print('Manual chain rule', grad_z_y * grad_y_x)", 167 | "execution_count": 20, 168 | "outputs": [ 169 | { 170 | "name": "stdout", 171 | "output_type": "stream", 172 | "text": "z = [1.609438 2.9957323]\nGradient with respect to y [0.2 0.05]\nGradient with respect to x [2. 1.]\nManual chain rule [[2. 1.]]\n" 173 | } 174 | ] 175 | }, 176 | { 177 | "metadata": { 178 | "_cell_guid": "78828a2a-ab7e-4aa3-a24f-f02dfab340c2", 179 | "_uuid": "1aa41fc42860411e437d3f1a2617a2bd0588d921", 180 | "collapsed": true 181 | }, 182 | "cell_type": "markdown", 183 | "source": "### Gradient descent method \n\nAs mentioned before the gradient is very useful if we need to minimize a loss function. \n\nIt's like hiking down a hill, we walk step by step into the direction of the steepest descent and finally we reach the valley. The gradient provides us the information in which direct we need to walk.\n\nSo if we want to minimize the function \n$f$ (e.g. root mean squared error, negative likelihood, ...) we can apply an iterative algorithm\n\n$$x_n = x_{n-1} - \\gamma \\nabla f(x_{n-1}),$$\n\nwith a starting point $x_0$. These kind of methods are called gradient descent methods.\n\nUnder particular circumstances we can be sure that we reach the global minimum but in general this is not true. Sometimes it can happen that we reach a local minima or a plateau.\nTo aviod stucking in local minima there are plenty extensions to the plain vanilla gradient descent (e.g. simulated annealing). In Machine Learning literature the dradient descent method is often called Batch Gradient method, because you will use all data points to calculate the gradients. \n\nWe will usually multiply the gradient with a factor before we subtract it from our previous value, the so called learning rate. If the learning rate is too large, we will make large steps into the direction but it can happen that we step over the minimum and miss it. If the learning rate is too small the algorithm takes longer to converge. There are extensions which adept the learning rate to the parameters (e.g ADAM, RMSProp or AdaGrad) to achive faster and better convergence (see for example http://ruder.io/optimizing-gradient-descent/index.html).\n\n### Example Batch Gradient descent\n\nLets see how to use it on a linear regression problem. We generate 1000 random observations $y = 2 x_1 + 3x_2 * \\epsilon$, with $\\epsilon$ normal distributed with mean zero and standard deviation of 0.2.\n" 184 | }, 185 | { 186 | "metadata": { 187 | "_cell_guid": "ad532d22-451c-4e1a-8d64-ba3970afe33a", 188 | "_uuid": "873113c5d0756cedae119d374d35533bdfeff0e5", 189 | "collapsed": true, 190 | "trusted": false 191 | }, 192 | "cell_type": "code", 193 | "source": "#Generate data\nnp.random.seed(42)\neps = 0.2 * np.random.randn(1000)\nx = np.random.randn(2,1000)\ny = 2 * x[0,:] + 3 * x[1,:] + eps", 194 | "execution_count": 21, 195 | "outputs": [] 196 | }, 197 | { 198 | "metadata": { 199 | "_cell_guid": "996d2365-5439-4103-9bfb-16b716bcf291", 200 | "_uuid": "7580291f51e4c15c1c6af85170d8b4370d3f4ac0" 201 | }, 202 | "cell_type": "markdown", 203 | "source": "We use a simple linear model to predict y. Our model is $$\\hat{y_i} = w_1 x_{i,1} + w_2 x_{i,2}$$ for an observation $x_i$ and we want to minimize the mean squared error of our predictions $$\\frac{1}{1000} \\sum (y_i-\\hat{y_i})^2.$$\n\nClearly we could use the well known least square estimators for the weights, but we want to minimize the error with a gradient descent method in TensorFlow. \n\nWe use the `tf.Variable`class to store the parameters $w$ which we want to learn (estimate) from the data. We specify the shape of the tensor, through the intial values. The inital values are the starting point of our minimization. \n\nSince we have a linear model, we can represent our model with an single matrix multiplication of our observation matrix (row obs, columns features) with our weight (parameter) matrix w.\n\n" 204 | }, 205 | { 206 | "metadata": { 207 | "_cell_guid": "5b2e6a74-d998-4c7c-9a43-c96039ef06c6", 208 | "_uuid": "c5f7882fc202be77e67ffa3057add9b0087b1eb0", 209 | "collapsed": true, 210 | "trusted": false 211 | }, 212 | "cell_type": "code", 213 | "source": "# Setup the computational graph with loss function\ninput_x = tf.placeholder(tf.float32, shape=(2,None))\ny_true = tf.placeholder(tf.float32, shape=(None,))\nw = tf.Variable(initial_value=np.ones((1,2)), dtype=tf.float32)\ny_hat = tf.matmul(w, input_x)\nloss = tf.reduce_mean(tf.square(y_hat - y_true))", 214 | "execution_count": 22, 215 | "outputs": [] 216 | }, 217 | { 218 | "metadata": { 219 | "_cell_guid": "2c673a46-cd45-496a-8e73-ef2166d36b2f", 220 | "_uuid": "f56b302a845aa3dc89d3ccf680f1af560abca1ae" 221 | }, 222 | "cell_type": "markdown", 223 | "source": "In the next step we are going to apply our batch gradient descent algorithm. We define gradient of the loss with respect to our weights $w$ `grad_loss_w`. We also need to initialize our weights with the inital value (starting point our optimization). TensorFlow has a operator for this `tf.global_variables_initializer()`. In our session we have to run this operator first. And then we can apply our algorithm.\n\nWe calculate the gradient and apply it to our weights with the function `assign()`. " 224 | }, 225 | { 226 | "metadata": { 227 | "_cell_guid": "170b1423-54d6-4f11-bf1b-ce3c4e090319", 228 | "_uuid": "2802b24eee65b886b80478e97fffc48e03523807", 229 | "trusted": false 230 | }, 231 | "cell_type": "code", 232 | "source": "grad_loss_w = tf.gradients(loss, [w])\ninit = tf.global_variables_initializer()\nlosses = np.zeros(10)\nwith tf.Session() as sess:\n # Initialize the variables\n sess.run(init)\n # Gradient descent\n for i in range(0,10):\n # Calculate gradient\n dloss_dw = sess.run(grad_loss_w, {input_x:x,\n y_true:y})\n # Apply gradient to weights with learning rate\n sess.run(w.assign(w - 0.1 * dloss_dw[0]))\n # Output the loss\n losses[i] = sess.run(loss, {input_x:x,\n y_true:y})\n print(i+1, 'th Step, current loss: ', losses[i])\n print('Found minimum', sess.run(w))\nplt.plot(range(10), losses)\nplt.title('Loss')\nplt.xlabel('Iteration')\n_ = plt.ylabel('RMSE')", 233 | "execution_count": 24, 234 | "outputs": [ 235 | { 236 | "name": "stdout", 237 | "output_type": "stream", 238 | "text": "1 th Step, current loss: 3.180712938308716\n2 th Step, current loss: 2.085484027862549\n3 th Step, current loss: 1.3719860315322876\n4 th Step, current loss: 0.9071645140647888\n5 th Step, current loss: 0.60434490442276\n6 th Step, current loss: 0.4070635139942169\n7 th Step, current loss: 0.2785366475582123\n8 th Step, current loss: 0.19480183720588684\n9 th Step, current loss: 0.14024850726127625\n10 th Step, current loss: 0.10470643639564514\nFound minimum [[1.8796383 2.7672703]]\n" 239 | }, 240 | { 241 | "data": { 242 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8lfXd//HXJ3uRMBJWAoQliigg\nkeFiVK21g95urdhhRdy2eveuvXv3d7e977vLWmtxFKt2aLFWrdrWaoeAkxGmIKBhGlYCAcJKyPj8\n/sghhpCF5JzrJOf9fDzOw3POdZ0rb85DeOd7ra+5OyIiIgBxQQcQEZHooVIQEZF6KgUREamnUhAR\nkXoqBRERqadSEBGReioFERGpp1IQaYaZbTSz84POIRJJKgUREamnUhA5TmZ2g5kVmVmZmb1kZn1D\n75uZ/czMSsxsr5mtMLMRoWUXm9l7ZrbPzLaY2d3B/ilEmqZSEDkOZjYF+AFwBdAH2AQ8HVp8IXAe\ncBLQFbgS2BVa9hhwo7t3AUYAr0UwtkibJQQdQKSD+QLwuLsvATCze4DdZpYPVAFdgJOBhe6+usHn\nqoDhZrbc3XcDuyOaWqSNNFIQOT59qRsdAODu+6kbDeS6+2vATOBBYIeZzTKzzNCqlwIXA5vMbJ6Z\nTYhwbpE2USmIHJ+twIAjL8wsHegBbAFw9wfcfQxwKnW7kf499P4id58K9AReAJ6JcG6RNlEpiLQs\n0cxSjjyo+8f8y2Y2ysySgf8DFrj7RjM708zGmVkicACoAGrMLMnMvmBmWe5eBZQDNYH9iURaoFIQ\nadnLwKEGj3OB/wKeA7YBg4GrQutmAo9Sd7xgE3W7le4NLZsGbDSzcmAGcG2E8oscF9MkOyIicoRG\nCiIiUk+lICIi9VQKIiJST6UgIiL1OtwVzdnZ2Z6fnx90DBGRDmXx4sU73T2ntfU6XCnk5+dTWFgY\ndAwRkQ7FzDa1vpZ2H4mISAMqBRERqadSEBGReioFERGpp1IQEZF6KgUREamnUhARkXoxUwobdh7g\nu39eRVVNbdBRRESiVgyVwn6eeGsjLyzdEnQUEZGoFTOlMHlYT4b3yeThueuoqdUcEiIiTYmZUjAz\nbpk8hPU7D/C3lduCjiMiEpViphQALhrRm0E56Tw4Zx2acU5E5FgxVQrxccbNk4awels5r60pCTqO\niEjUialSAJg6qi953VKZOadIowURkUZirhQS4+O4ceJglm7ewzvrdgUdR0QkqsRcKQBcPiaPnl2S\nmTmnKOgoIiJRJSZLISUxnhvOHcTb63axZPPuoOOIiESNsJWCmaWY2UIzW25mq8zsu02sk2xmfzCz\nIjNbYGb54crT2DXj+tM1LZEHX9NoQUTkiHCOFCqBKe4+EhgFXGRm4xutcz2w292HAD8DfhTGPEdJ\nT07gK2cP5F9rSnhva3mkfqyISFQLWyl4nf2hl4mhR+PTfaYCvwk9fxb4hJlZuDI19sUJ+WQkJ/Dg\nXI0WREQgzMcUzCzezJYBJcA/3H1Bo1VygQ8B3L0a2Av0aGI7082s0MwKS0tL2y1fVloi0yYM4OV3\nt7GudH/rHxAR6eTCWgruXuPuo4A8YKyZjWi0SlOjgmMuHnD3We5e4O4FOTk57Zrx+nMGkpwQx8Nz\n17XrdkVEOqKInH3k7nuAucBFjRYVA/0AzCwByALKIpHpiOyMZK46sz8vLN1C8e6DkfzRIiJRJ5xn\nH+WYWdfQ81TgfGBNo9VeAr4Yen4Z8JoHcJnxjRMHYQa/nLc+0j9aRCSqhHOk0AeYY2YrgEXUHVP4\ni5l9z8w+F1rnMaCHmRUBXwe+GcY8zQfNSuXSM/L4Q+GHlJRXBBFBRCQqJIRrw+6+AhjdxPvfafC8\nArg8XBmOx4yJg3mm8EN+9eYGvnXxKUHHEREJRExe0dyU/Ox0PjuyL0/O38TuA4eDjiMiEgiVQgM3\nTxrCwcM1PPH2xqCjiIgEQqXQwLDeXbhweC9+/dYG9lVUBR1HRCTiVAqN3DplCOUV1Tw5f3PQUURE\nIk6l0MjpeV05d2g2j725noqqmqDjiIhElEqhCbdOHsLO/Yd5eqFGCyISW1QKTRg3qAdn5nfjl6+v\n53B1bdBxREQiRqXQjFsmD2Hb3gr+tLQ46CgiIhGjUmjGxJNyOC03i4fnrqO6RqMFEYkNKoVmmBm3\nTB7Mxl0H+eu724KOIyISESqFFlw4vDdDe2bw0Jx11NZG/D59IiIRp1JoQVyccfPkwazdsY9/rt4R\ndBwRkbBTKbTis6f3pX/3NB6cU0QAd/UWEYkolUIrEuLjmDFxMMuL9/Jm0c6g44iIhJVKoQ0uHZNL\n78wUZr5WFHQUEZGwUim0QXJCPDecN4gFG8oo3BjR2UJFRCJKpdBGV4/tR/f0JGbO0WhBRDovlUIb\npSUlcP05A5m7tpSVW/YGHUdEJCxUCsdh2oQBdElJ4EGNFkSkk1IpHIfMlES+OCGfV1Ztp6hkX9Bx\nRETanUrhOH3lnIGkJMTz0Jx1QUcREWl3KoXj1D09iWvG9efF5VvZvOtg0HFERNqVSuFjmH7eIOLN\neOR1jRZEpHMJWymYWT8zm2Nmq81slZnd0cQ6k8xsr5ktCz2+E6487alXZgqXFeTxbGEx2/dWBB1H\nRKTdhHOkUA3c5e6nAOOBW8xseBPrveHuo0KP74UxT7u6aeJgatx59I31QUcREWk3YSsFd9/m7ktC\nz/cBq4HccP28SOvXPY2pI/vy+wWbKTtwOOg4IiLtIiLHFMwsHxgNLGhi8QQzW25mfzOzU5v5/HQz\nKzSzwtLS0jAmPT43Tx5MRXUNj7+5IegoIiLtIuylYGYZwHPAne5e3mjxEmCAu48EfgG80NQ23H2W\nuxe4e0FOTk54Ax+HIT27cNGpvfnNOxspr6gKOo6IyAkLaymYWSJ1hfCUuz/feLm7l7v7/tDzl4FE\nM8sOZ6b2dsvkIeyrqOZ372wKOoqIyAkL59lHBjwGrHb3+5pZp3doPcxsbCjPrnBlCocRuVlMGpbD\nY29u4ODh6qDjiIickHCOFM4GpgFTGpxyerGZzTCzGaF1LgNWmtly4AHgKu+A05vdOnkIZQcOM3vh\nh0FHERE5IQnh2rC7vwlYK+vMBGaGK0OkFOR3Z9zA7sx6fR3Xju9PckJ80JFERD4WXdHcTm6dMoQd\n5ZU8t3hL0FFERD42lUI7OWdINiPzsnhk3jqqa2qDjiMi8rGoFNqJmXHL5CFsLjvIn1dsDTqOiMjH\nolJoR+ef0othvbrw0Jx11NZ2uOPlIiIqhfYUF2fcPHkwH5Ts5+/vbQ86jojIcVMptLPPnN6X/B5p\nzJxTRAc8u1ZEYpxKoZ3Fxxk3TRrMyi3lzHs/eu7TJCLSFiqFMPi30Xn0zUrhwTlFQUcRETkuKoUw\nSEqIY/p5g1i0cTcL1neou3aISIxTKYTJVWP7k52RxEyNFkSkA1EphElKYjzXnzOINz7YyfIP9wQd\nR0SkTVQKYXTt+P5kpiTo2IKIdBgqhTDqkpLIl84eyN/f28Ha7fuCjiMi0iqVQph9+ax80pLieWiu\nRgsiEv1UCmHWLT2Ja8cP4M/Lt7Jx54Gg44iItEilEAFfPWcgCfFxPDJvXdBRRERapFKIgJ6ZKVxZ\n0I/nlhSzdc+hoOOIiDRLpRAhN04chDvMen190FFERJqlUoiQvG5pfH50Lk8v2szO/ZVBxxERaZJK\nIYJumjSYyupaHntzQ9BRRESapFKIoME5GVx8Wh9++/ZGNu3SmUgiEn1UChF2z6dOJj7OuG32Ug5X\nay5nEYkuKoUIy+uWxo8vO50VxXv58Strgo4jInKUsJWCmfUzszlmttrMVpnZHU2sY2b2gJkVmdkK\nMzsjXHmiyUUj+nDdhAH86s0NvLZmR9BxRETqhXOkUA3c5e6nAOOBW8xseKN1PgUMDT2mAw+HMU9U\n+dbFp3BKn0zuemY52/bq2gURiQ5hKwV33+buS0LP9wGrgdxGq00Ffut15gNdzaxPuDJFk5TEeGZe\nM5rK6lrueHoZNbWaz1lEgheRYwpmlg+MBhY0WpQLfNjgdTHHFgdmNt3MCs2ssLS088x7PDgng+9P\nHcHCDWU88K8Pgo4jIhL+UjCzDOA54E53L2+8uImPHPMrs7vPcvcCdy/IyckJR8zAXDomj0vOyOUX\nr33AO+s0daeIBCuspWBmidQVwlPu/nwTqxQD/Rq8zgO2hjNTNPr+1BHkZ6dzx9NL2aWrnUUkQOE8\n+8iAx4DV7n5fM6u9BFwXOgtpPLDX3beFK1O0Sk9O4BdXj2bPoSru+uNyanV8QUQCEs6RwtnANGCK\nmS0LPS42sxlmNiO0zsvAeqAIeBS4OYx5otqpfbP49qdPYe7aUt0GQ0QCkxCuDbv7mzR9zKDhOg7c\nEq4MHc208QN4q2gnP3plDWcO7M6ofl2DjiQiMUZXNEcRM+PHl46kV2YKt81eQnlFVdCRRCTGqBSi\nTFZaIg9cPZqteyq457l3qRtMiYhEhkohCo0Z0I27LxzGX9/dxu8Xbg46jojEEJVClLrxvEGcOzSb\n7/35PdZsb3x5h4hIeKgUolRcnHHfFaPITE3k1t8v5eDh6qAjiUgMUClEsZwuydx/5SjWle7nv19a\nFXQcEYkBLZaCmU1p8Hxgo2WXhCuUfOTsIdncMmkIzxQW8+KyLUHHEZFOrrWRwr0Nnj/XaNm32zmL\nNOPO84dyZn43vvX8u2zYqWk8RSR8WisFa+Z5U68lTBLi4/j5VaNJiI/jttlLqKyuCTqSiHRSrZWC\nN/O8qdcSRn27pnLv5SNZuaWcH/5N03iKSHi0dpuLQWb2EnWjgiPPCb0e2PzHJBwuGN6LL52VzxNv\nbeSswdlcMLxX0JFEpJOxlq6YNbOJLX3Y3ee1e6JWFBQUeGFhYaR/bNSorK7h0offpnj3IV6+/Vz6\ndk0NOpKIdABmttjdC1pbr8XdR+4+r+EDeBsop+522BEvBIHkhHh+cfUZVFXXcvvspVTX1AYdSUQ6\nkdZOSX3EzE4NPc8ClgO/BZaa2dURyCdNGJidzv9dchqFm3bzc03jKSLtqLUDzee6+5Grpr4MvO/u\npwFjgG+ENZm0aOqoXC4fk8fMOUW8VbQz6Dgi0km0VgqHGzy/AHgBwN23hy2RtNl3p57K4JwM7vzD\nMkr3aRpPETlxrZXCHjP7jJmNpm4mtVcAzCwB0BHOgKUlJTDzmtGUH6ri688s0zSeInLCWiuFG4Fb\ngSeAOxuMED4B/DWcwaRtTu6dyXc+O5w3PtjJL19fH3QcEengWrxOwd3fBy5q4v1XgVfDFUqOzzVj\n+/N20S7u/ftaxg7szpgB3YKOJCIdVIulYGYPtLTc3W9v3zjycZgZ/3fJaSwv3sPts5fy8u3nkpWW\nGHQsEemAWtt9NAM4B9gKFAKLGz0kSmSlJjLzmjPYUV7Bfzy3QtN4isjH0lop9AFmAZ8EpgGJwEvu\n/ht3/024w8nxGdWvK9+4aBivrNrOk/M3BR1HRDqg1q5o3uXuj7j7ZOBLQFdglZlNi0Q4OX5fPWcQ\nk4bl8P2/rmbV1r1BxxGRDqZNM6+Z2RnAncC1wN9ow64jM3vczErMbGUzyyeZ2V4zWxZ6fOd4gkvT\n4uKMn14+kq6pidw2eykHKjWNp4i0XWu3ufiumS0Gvg7MAwrc/Xp3f68N2/41TZy51Mgb7j4q9Phe\nmxJLq3pkJHP/VaPYsPMA33lR03iKSNu1NlL4LyALGAn8AFhiZivM7F0zW9HSB939daCsfWLK8Tpr\ncDa3TRnKc0uKeX5JcdBxRKSDaG0+hXDPmTDBzJZTd3bT3Q3us3QUM5sOTAfo379/mCN1HrdPGcL8\n9bv49gsrGdmvK4NzMoKOJCJRrrUDzZuaegDF1J2qeiKWAAPcfSTwC0L3VWomxyx3L3D3gpycnBP8\nsbEjIT6OB64aTXJCHLf+fikVVZrGU0Ra1toxhUwzu8fMZprZhVbnNmA9cMWJ/GB3L3f3/aHnLwOJ\nZpZ9ItuUY/XOSuGnV4xk9bZyfvDy6qDjiEiUa+2Ywu+AYcC7wFeBvwOXAVPdfeqJ/GAz621mFno+\nNpRl14lsU5o25eReXH/OQH7zziZeWakb3IpI81qdozk0fwJm9itgJ9Df3fe1tmEzmw1MArLNrBj4\nf9Rd/Ia7P0JdudxkZtXAIeAq12W4YfMfF53Moo1lfOPZ5YzIzSSvW1rQkUQkCrU2R/MSdz+juddB\niPU5mk/Epl0H+PQDb3JSrwz+cOMEEuPbdJmKiHQC7TJHMzDSzMpDj33A6Ueem1l5+0SVSBnQI50f\nXHIaSzbv4b5/vB90HBGJQq3dOjs+UkEkMj47si9vr9vJw3PXMWFQD847SWdzichHtP8gBn3nM6dy\nUq8Mbn5qCW98UBp0HBGJIiqFGJSaFM9vvzKOvG6pfPmJRbriWUTqqRRiVO+sFJ6ZMYFxg7rz9WeW\n8+CcIs3BICIqhViWmZLIE18ay7+NzuUnr67lP19YSXVNbdCxRCRArV2nIJ1cUkIc910xkj5ZKTw0\ndx079lbwi2tGk5ak/zVEYpFGCoKZ8Y2LTub7nx/BnLUlXD1rPjv3VwYdS0QCoFKQetPGD+CX0wpY\nu2Mflzz0Nht2Hgg6kohEmEpBjnLB8F7MvmE8+yurueSht1iyeXfQkUQkglQKcozR/bvx/E1nkZma\nyDWPzufvq3QTPZFYoVKQJuVnp/P8TWcxrHcmM55czO/e2Rh0JBGJAJWCNKtHRjKzbxjHlJN78l8v\nruKHf1tDba2uZRDpzFQK0qK0pAQeuXYMXxjXn0fmreNrzyyjslozuIl0VjoZXVqVEB/H/3x+BLnd\nUvnxK2spKa/kkWljyEpNDDqaiLQzjRSkTcyMmycN4WdXjqRwUxlXPPIOW/ccCjqWiLQzlYIcl38b\nncevvzyWrXsOcclDb7Nmu6bVEOlMVApy3M4eks0zMybgOJc//A5vF+0MOpKItBOVgnwsp/TJ5E83\nn02fril88YmFvLhsS9CRRKQdqBTkY+vbNZU/zjiLMQO6ccfTy3h47jrdflukg1MpyAnJSk3kN18Z\ny2dH9uVHr6zhOy+uokbXMoh0WDolVU5YckI8P79yFH27pvDLeevZXl7BA1eNJjVJU3yLdDQaKUi7\niIsz7vnUKXz3c6fyz9U7uPrR+ezS7bdFOpywlYKZPW5mJWa2spnlZmYPmFmRma0wszPClUUi54tn\n5fPwF8awels5lz78Npt26fbbIh1JOEcKvwYuamH5p4Chocd04OEwZpEIumhEb35/wzj2HKrikofe\nZtmHe4KOJCJtFLZScPfXgbIWVpkK/NbrzAe6mlmfcOWRyBozoDvP3XQWacnxXD1rPv9avSPoSCLS\nBkEeU8gFPmzwujj03jHMbLqZFZpZYWlpaUTCyYkbnJPB8zedzZCeGdzw20KeWrAp6Egi0oogS8Ga\neK/JcxndfZa7F7h7QU5OTphjSXvK6ZLM09PHM/GkHP7zTyu599W1upZBJIoFWQrFQL8Gr/OArQFl\nkTBKT07g0esKuOrMfsycU8Rdf1zO4eraoGOJSBOCLIWXgOtCZyGNB/a6+7YA80gYJcTH8YNLTuPr\nF5zE80u28JVfL2JfRVXQsUSkkXCekjobeAcYZmbFZna9mc0wsxmhVV4G1gNFwKPAzeHKItHBzLj9\nE0P5yWWnM3/9Li5/5B22760IOpaINGAdbf9uQUGBFxYWBh1DTtC890u5+cnFZKUmMuu6AkbkZgUd\nSaRTM7PF7l7Q2nq6olkCMfGkHP5w4wSqa53PzXyT/3phJWUHDgcdSyTmqRQkMCNys3j1zvOYNn4A\nv1+4mUk/mcPjb26gqkYHoUWColKQQHVLT+K7U0fwtzvOZWS/rnzvL+9x0f2vM3dtSdDRRGKSSkGi\nwkm9uvDbr4zl0esKqKl1vvTEIr78xELWle4POppITFEpSNQwMy4Y3otXv3Ye37r4ZAo37uaTP3ud\n7//lPfYe0umrIpGgUpCok5wQz/TzBvPa3ZO4bEwej7+1gcn3zuXJ+Zs0gY9ImKkUJGrldEnmh5ee\nzp9vPYchPTP49gsr+fQDb/D2up1BRxPptFQKEvVG5Gbxh+njeegLZ7CvopprHl3AjN8tZvOug0FH\nE+l0VArSIZgZF5/Wh3/dNZG7LzyJ1z8o5fz75vGjV9awv7I66HginYZKQTqUlMR4bp0ylNfumsRn\nTu/Dw3PXMfneufyx8ENqdbxB5ISpFKRD6p2Vwn1XjuJPN59FXrdU/v3ZFXz+obco3NjSvE4i0hqV\ngnRoo/t347kZZ3H/laMoKa/kskfe4fbZS9m651DQ0UQ6JJWCdHhxccbnR+fy2t0TuX3KEF5dtZ0p\nP53L/f98n0OHa4KOJ9KhqBSk00hLSuDrFw7jX3dN5PxTenH/Pz/gEz+dy4vLtmi2N5E2UilIp5PX\nLY2Z15zBMzdOoHtGEnc8vYzLHnmHFcV7go4mEvVUCtJpjR3YnRdvOYcfX3o6m3Yd4HMz3+LuPy6n\npFwT+4g0R6UgnVp8nHHFmf2Yc/ckbpw4iBeXbWHyvXN5cE4RFVU63iDSmEpBYkKXlETu+dQp/ONr\nEzlrSDY/eXUtF/xsHq+s3K7jDSINqBQkpuRnp/PodQU89dVxpCUmMOPJxVzz6AJWbysPOppIVNAc\nzRKzqmtqmb1wM/f94332Hqpiysm9uGxMHlNO7klSgn5fks6lrXM0J0QijEg0SoiPY9qEfD43MpdH\nXl/Hs4uL+efqHXRPT2LqqL5cNiaPU/tmBR1TJKI0UhAJqa6p5Y0PdvLs4mL+8d4ODtfUcnLvLlw2\nJo/Pj84lOyM56IgiH1tbRwoqBZEm7Dl4mD8v38qzi4tZXryXhDhj0rCe2r0kHVZUlIKZXQT8HIgH\nfuXuP2y0/EvAT4AtobdmuvuvWtqmSkEi7YMd+3h2STHPL9lC6b5KuqUlMnVUbmj3UiZmFnREkVYF\nXgpmFg+8D1wAFAOLgKvd/b0G63wJKHD3W9u6XZWCBKW6ppY3ikK7l1Zp95J0LNFwoHksUOTu60OB\nngamAu+1+CmRKJUQH8fkYT2ZPKxn3e6lFdt4dnEx//PX1fzwb2u0e0k6hXCWQi7wYYPXxcC4Jta7\n1MzOo25U8TV3/7CJdUSiSte0JKaNH8C08QPqdy/9ackW/rl6h3YvSYcWzt1HlwOfdPevhl5PA8a6\n+20N1ukB7Hf3SjObAVzh7lOa2NZ0YDpA//79x2zatCksmUVOhHYvSTSLhmMKE4D/dvdPhl7fA+Du\nP2hm/XigzN1bPDFcxxSkI2i4e2n5h3t09pIELhqOKSwChprZQOrOLroKuKbhCmbWx923hV5+Dlgd\nxjwiEaPdS9JRhfuU1IuB+6k7JfVxd/9fM/seUOjuL5nZD6grg2qgDLjJ3de0tE2NFKSj0u4lCVLg\nu4/CRaUgncHeg1X8eUXdxXHL6ncv5XDpGXmcPTSbzJTEoCNKJ6NSEOkgikr28eziLTy/pJiSfZXE\nGQzvm8nY/B6MG9Sdsfnd6ZaeFHRM6eBUCiIdTHVNLYs27mb++l0s3FDGks27qayuBWBYry6MHdi9\nriQGdqdnl5SA00pHo1IQ6eAqq2t4t3gvCzaUsWBDGYs3lnHgcN1scYOy0+sLYuzAHuR2TQ04rUQ7\nlYJIJ1NdU8vKreUs3LCLBevLWLixjH0V1QDkdUtl7MDujB/Yg7EDuzOgR5rOapKjqBREOrmaWmfN\n9nIWbiirL4myA4cB6JWZzNiBPRg3sDvjBnZnSM8MlUSMUymIxBh3p6hkf/3upgXrd1GyrxKAHulJ\nnJn/0TGJU3pnEhenkogl0XDxmohEkJkxtFcXhvbqwrXjB+DubNp1kIUbypi/oe7g9SurtgOQmZLQ\noCR6MKJvJgnxuspaVAoinZaZkZ+dTn52Olec2Q+ALXsOfXRMYkMZ/1pTAkB6UjxnDOjG+EF1xyRO\nz8siOSE+yPgSEO0+EolhJeUVLNxYVl8Sa3fsAyAx3sjvkc6QnhkMzslgSM+6x6CcdNKS9LtkR6Rj\nCiJy3MoOHGbRxjKWbt5DUcl+1pXuZ9OuA9Q2+Gcit2sqg3tmMCRUFoNz6sqjh27TEdV0TEFEjlv3\n9CQ+eWpvPnlq7/r3Kqtr2LTrIEUl++uLoqhkPws37KKiqrZ+vW5piUeNLI4UR27XVB3U7kBUCiLS\nouSEeE7q1YWTenU56v3aWmfLnkP1JbGudD/rSg7w9/d28PSij+bKSkmMY1D2R7ugjpRGfnaajltE\nIZWCiHwscXFGv+5p9OuexqRhPY9aVnbg8FGjiqKS/SzetJuXlm+tXyc+zujfPY3BOelH747qmaEb\nAgZIpSAi7a57elLoFhzdj3r/0OGauhFFg7JYV7qfee+XUlXz0YGLnl2SQ6OJdHpnptArM5lemSn0\nykyhd2YKXdMSdTFemKgURCRiUpPiGZGbxYjcoydYrK6pZXPZQdaVHqgvi6LS/byycnv9VdoNJSXE\n0Sszmd6ZKfQMFUXj4uiVmUJqknZPHS+VgogELiE+jkE5GQzKyeCC4b2OWlZZXUNJeSUl+yrYvreS\nHeUV7CivYHvov+9tLee11SUcqqo5ZruZKQn0zkqpL4tjiySF7IwkXbjXgEpBRKJackJ8/bGL5rg7\n+yqrKSn/qDi2l1fUvS6vYEd5JUUlOynZV0lN7dGn4ccZZGck0zsrhZ5dUuiddWxx9MhIIis1kcQY\nKA+Vgoh0eGZGZkoimSmJDOnZpdn1amqdXQcqKSmvZPveY4ujePdBFm8qY/fBqiY/n54UT1ZqIpmp\niXRNSyQrte7RNS2p/v2s1ES6pjZclkiXlETiO8hpuSoFEYkZ8XFGzy51I4LGxzUaqqiqoXRfJdvL\nK9i+t4LdBw+z52AVew81eBysYuPOg+w9VMWeQ4ePumajKV1SEupLIiu14SPpqAI5allaIhlJCRG9\nzkOlICLSSEpi67usGqusrqkvi4blcUyZhB47yvez52AV5YeqOFzTfKHEGfUjkGvHDeCG8wa1xx+x\nWSoFEZF2kJwQT88u8cc9Vaq7U1FVy55Dh48qlT2H6gqjYbHkdAn/rURUCiIiATIzUpPiSU1KpU9W\n8NOqdv5D6SIi0mZhLQUzu8jYlNlKAAAFnUlEQVTM1ppZkZl9s4nlyWb2h9DyBWaWH848IiLSsrCV\ngpnFAw8CnwKGA1eb2fBGq10P7Hb3IcDPgB+FK4+IiLQunCOFsUCRu69398PA08DURutMBX4Tev4s\n8AnTDU1ERAITzlLIBT5s8Lo49F6T67h7NbAX6BHGTCIi0oJwlkJTv/E3nuatLetgZtPNrNDMCktL\nS9slnIiIHCucpVAM9GvwOg/Y2tw6ZpYAZAFljTfk7rPcvcDdC3JycsIUV0REwlkKi4ChZjbQzJKA\nq4CXGq3zEvDF0PPLgNe8o00aLSLSiVg4/w02s4uB+4F44HF3/18z+x5Q6O4vmVkK8DtgNHUjhKvc\nfX0r2ywFNn3MSNnAzo/52c5I38fR9H18RN/F0TrD9zHA3Vvd1RLWUog2Zlbo7gVB54gW+j6Opu/j\nI/oujhZL34euaBYRkXoqBRERqRdrpTAr6ABRRt/H0fR9fETfxdFi5vuIqWMKIiLSslgbKYiISAtU\nCiIiUi9mSqG123jHEjPrZ2ZzzGy1ma0yszuCzhQ0M4s3s6Vm9pegswTNzLqa2bNmtib0/8iEoDMF\nxcy+Fvo7stLMZoeurerUYqIU2ngb71hSDdzl7qcA44FbYvz7ALgDWB10iCjxc+AVdz8ZGEmMfi9m\nlgvcDhS4+wjqLsK9KthU4RcTpUDbbuMdM9x9m7svCT3fR91f+sZ3sI0ZZpYHfBr4VdBZgmZmmcB5\nwGMA7n7Y3fcEmypQCUBq6N5saRx7/7ZOJ1ZKoS238Y5JodnuRgMLgk0SqPuBbwC1QQeJAoOAUuCJ\n0O60X5lZetChguDuW4B7gc3ANmCvu/892FThFyul0KZbdMcaM8sAngPudPfyoPMEwcw+A5S4++Kg\ns0SJBOAM4GF3Hw0cAGLyGJyZdaNuj8JAoC+QbmbXBpsq/GKlFNpyG++YYmaJ1BXCU+7+fNB5AnQ2\n8Dkz20jdbsUpZvZksJECVQwUu/uRkeOz1JVELDof2ODupe5eBTwPnBVwprCLlVJoy228Y0ZoytPH\ngNXufl/QeYLk7ve4e56751P3/8Vr7t7pfxtsjrtvBz40s2Ghtz4BvBdgpCBtBsabWVro78wniIGD\n7glBB4gEd682s1uBV/noNt6rAo4VpLOBacC7ZrYs9N633P3lADNJ9LgNeCr0C9R64MsB5wmEuy8w\ns2eBJdSdsbeUGLjdhW5zISIi9WJl95GIiLSBSkFEROqpFEREpJ5KQURE6qkURESknkpBYpaZ7Q/9\nN9/MrmnnbX+r0eu323P7IuGiUhCBfOC4SiF0592WHFUK7t7pr4SVzkGlIAI/BM41s2Wh++fHm9lP\nzGyRma0wsxsBzGxSaB6K3wPvht57wcwWh+65Pz303g+pu7PmMjN7KvTekVGJhba90szeNbMrG2x7\nboN5DJ4KXUUrElExcUWzSCu+Cdzt7p8BCP3jvtfdzzSzZOAtMztyd8yxwAh33xB6/RV3LzOzVGCR\nmT3n7t80s1vdfVQTP+sSYBR18xRkhz7zemjZaOBU6u7L9RZ1V56/2f5/XJHmaaQgcqwLgetCtwBZ\nAPQAhoaWLWxQCAC3m9lyYD51N10cSsvOAWa7e4277wDmAWc22Haxu9cCy6jbrSUSURopiBzLgNvc\n/dWj3jSbRN2tpBu+Ph+Y4O4HzWwu0Np0jS3tEqps8LwG/f2UAGikIAL7gC4NXr8K3BS6vThmdlIz\nE81kAbtDhXAydVObHlF15PONvA5cGTpukUPdLGcL2+VPIdIO9JuICKwAqkO7gX5N3RzF+cCS0MHe\nUuDzTXzuFWCGma0A1lK3C+mIWcAKM1vi7l9o8P6fgAnAcuomevqGu28PlYpI4HSXVBERqafdRyIi\nUk+lICIi9VQKIiJST6UgIiL1VAoiIlJPpSAiIvVUCiIiUu//AwK126YTuwu8AAAAAElFTkSuQmCC\n", 243 | "text/plain": "" 244 | }, 245 | "metadata": {}, 246 | "output_type": "display_data" 247 | } 248 | ] 249 | }, 250 | { 251 | "metadata": { 252 | "_cell_guid": "9d67392f-497c-4e83-b332-d9ff7d97f08a", 253 | "_uuid": "5a6e7656ef20bbb4cf2c45a01d64d4adaa4c1515", 254 | "collapsed": true 255 | }, 256 | "cell_type": "markdown", 257 | "source": "Luckily we don't need to program everytime the same algorithm by ourself. TensorFlow provide many of gradient descent algorithms, e.g. \n`tf.train.GradientDescentOptimizer`, `tf.train.AdagradDAOptimizer` or `tf.train.RMSPropOptimizer` just to mention a few. They compute the gradient and apply it to the weights automatically. In the case of the `GradientDescentOptimizer`we only need to specify the learning rate and tell the optimizer which loss function we want to minimize. We call the method `minimize` which returns our training or optimization operator. In our loop we just need to run the operator." 258 | }, 259 | { 260 | "metadata": { 261 | "_cell_guid": "23488a4e-4d2b-4431-8c91-adebfe056758", 262 | "_uuid": "9f6cbda094745f973eaa84a488f96d91418be1d2", 263 | "trusted": false 264 | }, 265 | "cell_type": "code", 266 | "source": "tf.train.RMSPropOptimizer\noptimizer = tf.train.GradientDescentOptimizer(0.1)\ntrain = optimizer.minimize(loss)\ninit = tf.global_variables_initializer()\nlosses = np.zeros(10)\nwith tf.Session() as sess:\n # Initialize the variables\n sess.run(init)\n # Gradient descent\n for i in range(0,10):\n _, losses[i] = sess.run([train, loss], {input_x:x,\n y_true:y})\n print('Found minimum', sess.run(w))\nplt.plot(range(10), losses)\nplt.title('Loss')\nplt.xlabel('Iteration')\n_ = plt.ylabel('RMSE')", 267 | "execution_count": 25, 268 | "outputs": [ 269 | { 270 | "name": "stdout", 271 | "output_type": "stream", 272 | "text": "Found minimum [[1.8796383 2.7672703]]\n" 273 | }, 274 | { 275 | "data": { 276 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEWCAYAAABliCz2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl4VfW59vHvk3kgAyEBhAAhBEVB\nEIhMggJ6rLVWT9U6t9WjxQHUng6e2rfntD2TPafWWl8VpWq1tWKtaKu1DvUVHFCRMKM4EAgzJExJ\nGBIyPO8f2UBQSAJkZSXZ9+e6crHndWdf5F5r//Zav2XujoiIdH4xYQcQEZG2ocIXEYkSKnwRkSih\nwhcRiRIqfBGRKKHCFxGJEip8EZEoocKXqGRmJWZ2Ttg5RNqSCl9EJEqo8EUaMbNvm9lKM9tuZi+Y\nWa/I7WZmvzKzUjMrN7OlZjYkct/5ZvaRmVWa2QYz+364v4XI4anwRSLMbDJwF3AZcAKwBng6cve5\nwJnAiUAmcDmwLXLfo8CN7p4GDAHeaMPYIi0WF3YAkXbkauAxd18IYGZ3AjvMLA+oAdKAQcAH7r6i\n0fNqgFPMbIm77wB2tGlqkRbSFr7IQb1o2KoHwN130bAV39vd3wDuBx4AtpjZDDNLjzz0EuB8YI2Z\nvWlmY9s4t0iLqPBFDtoI9Nt/xcxSgW7ABgB3v8/dRwKDaRja+UHk9vnufhHQHfgz8Ewb5xZpERW+\nRLN4M0va/0NDUV9nZqeZWSLw38A8dy8xs9PNbLSZxQO7gSqgzswSzOxqM8tw9xqgAqgL7TcSaYIK\nX6LZ34C9jX4mAP8KzAI2AQOAKyKPTQd+Q8P4/Boahnrujtz3DaDEzCqAm4Br2ii/yFExnQBFRCQ6\naAtfRCRKqPBFRKKECl9EJEqo8EVEokS7OtI2Ozvb8/Lywo4hItJhLFiwYKu757TksYEWvpmVAJU0\n7Jdc6+6FTT0+Ly+PoqKiICOJiHQqZram+Uc1aIst/EnuvrUNliMiIk3QGL6ISJQIuvAdeM3MFpjZ\nlMM9wMymmFmRmRWVlZUFHEdEJHoFXfhnuPsI4MvAVDM78/MPcPcZ7l7o7oU5OS363kFERI5BoIXv\n7hsj/5YCzwOjglyeiIgcWWCFb2apZpa2/zINZwxaHtTyRESkaUHupdMDeN7M9i/nKXd/JcDliYhI\nEwIrfHdfBQwL6vX3q6qp43fvlTA0N5Mx+d2CXpyISIfV4XfLNIPH3inhV3//NOwoIiLtWocv/MS4\nWL59Zj7zVm+nqGR72HFERNqtDl/4AFeO6kNWagL3z14ZdhQRkXarUxR+SkIc14/vz5xPyli+oTzs\nOCIi7VKnKHyAb4ztR1pSHA9oK19E5LA6TeGnJ8XzrbF5vPLhZlaWVoYdR0Sk3ek0hQ/wT+P7kxQX\ny4Ozi8OOIiLS7nSqws9KTeCq0X35y5KNrN22J+w4IiLtSqcqfIApZ+YTa8ZDb2krX0SksU5X+D3S\nk7i0MJdni9azubwq7DgiIu1Gpyt8gJvPGkCdO795e1XYUURE2o1OWfh9slK4aFgvnpq3lu2794Ud\nR0SkXeiUhQ9wy6QBVNXW8dg7q8OOIiLSLnTawi/onsZ5g3vyxHslVFTVhB1HRCR0nbbwAaZOKqCy\nqpbfv7cm7CgiIqHr1IU/pHcGE0/K4dF3VrNnX23YcUREQtWpCx9g2qQCtu/ex8wP1oUdRUQkVJ2+\n8AvzshjdP4sZbxVTXVsXdhwRkdB0+sIHmDa5gC0V1cxasCHsKCIioYmKwh9fkM2w3AweerOY2rr6\nsOOIiIQiKgrfzJg6qYC12/fw4tKNYccREQlFVBQ+wDkn9+CkHmk8OLuY+noPO46ISJuLmsKPiTFu\nmTSAz0p38dpHm8OOIyLS5qKm8AEuGNqLvG4p3D97Je7ayheR6BJVhR8bY9w8cQDLN1Tw5qdlYccR\nEWlTUVX4AF8bnkuvjCSd7FxEok7UFX5CXAxTzsxnfskO5q3aFnYcEZE2E3WFD3DFqL5kd0ngfm3l\ni0gUicrCT4qP5frx+bz92VaWrNsZdhwRkTYRlYUPcM2YvqQnxWksX0SiRtQWflpSPNee0Z/XPtrC\nJ5srw44jIhK4qC18gOvG5ZGSEMuDc7SVLyKdX1QXftfUBK4Z048Xl2ykZOvusOOIiAQq8MI3s1gz\nW2Rmfw16WcfihvH9iYuN4aE3i8OOIiISqLbYwr8dWNEGyzkm3dOTuLywD7MWrmfjzr1hxxERCUyg\nhW9mucBXgEeCXM7xuvGsfNxhxlurwo4iIhKYoLfw7wXuAI541hEzm2JmRWZWVFYWzvw2uV1T+Mfh\nvXl6/lq27qoOJYOISNACK3wzuwAodfcFTT3O3We4e6G7F+bk5AQVp1k3TxxAdW09j76zOrQMIiJB\nCnIL/wzgQjMrAZ4GJpvZkwEu77gMyOnC+aeewO/fW0P5npqw44iItLrACt/d73T3XHfPA64A3nD3\na4JaXmuYOrGAXdW1PPFeSdhRRERaXVTvh/95p/RK5+xB3Xls7mp2V9eGHUdEpFW1SeG7+xx3v6At\nlnW8pk4uYOeeGp6atzbsKCIirUpb+J8zom9Xxg3oxoy3V1FVUxd2HBGRVqPCP4xpkwooq6zmTwvW\nhx1FRKTVqPAPY+yAbgzvm8lDc4qpqTviIQQiIh2KCv8wzIxpkwrYsHMvf1m8Mew4IiKtQoV/BJMH\ndWdQzzQenLOSunoPO46IyHFT4R+BmTF1UgGrynbzyvLNYccRETluKvwmnH/qCeRnp3L/7JW4aytf\nRDo2FX4TYmOMmyYOYMWmCmZ/Uhp2HBGR46LCb8bXhvemd2Yy97+hrXwR6dhU+M2Ij43hxrPyWbh2\nJ++t2hZ2HBGRY6bCb4HLCvuQ3SWRB2brZOci0nGp8FsgKT6Wb0/oz9yV21i0dkfYcUREjokKv4Wu\nHtOPjOR4beWLSIelwm+hLolxXHdGHq+vKGXFpoqw44iIHDUV/lG4dlweqQmx2soXkQ5JhX8UMlMS\nuGZsP15atolVZbvCjiMiclRU+EfphvH5JMTGMH1OcdhRRESOigr/KOWkJXLF6X14ftEG1u/YE3Yc\nEZEWU+EfgylnDQBgxlurQk4iItJyKvxj0DszmYtH9Obp+esorawKO46ISIuo8I/RzRMLqK2r59G3\nV4cdRUSkRVT4x6h/dipfGdqLJ99fw849+8KOIyLSLBX+cZg6aQC799Xx27klYUcREWmWCv84DOqZ\nzjkn9+Dxd0vYVV0bdhwRkSap8I/TtMkFlO+t4cn314QdRUSkSSr843Ran0zGF2TzyNurqaqpCzuO\niMgRqfBbwdRJBWzdVc0f568LO4qIyBGp8FvBmPwsRvbrysNvFrOvtj7sOCIih6XCbwVmxrRJBWws\nr+K3c7Vfvoi0Tyr8VjLxpBy+NLgHv3j1Exav2xl2HBGRL1DhtxIz438vGUaP9CRunbmQiqqasCOJ\niBxChd+KMlLiue/K09i4s4o7Zy3D3cOOJCJygAq/lY3sl8X3zj2Rl5Zt4qkP1oYdR0TkgMAK38yS\nzOwDM1tiZh+a2c+CWlZ7c9OZA5gwMJt/f/EjPt6s89+KSPsQ5BZ+NTDZ3YcBpwHnmdmYAJfXbsTE\nGPdcdhppSfFMe2oRe/Zp2gURCV9ghe8N9p/4NT7yEzWD2jlpidx7+WkUl+3ipy98GHYcEZFgx/DN\nLNbMFgOlwN/dfd5hHjPFzIrMrKisrCzIOG1u/MBsbpk4gGeK1vOXxRvCjiMiUS7Qwnf3Onc/DcgF\nRpnZkMM8Zoa7F7p7YU5OTpBxQvHP55xIYb+u/Oi5ZazeujvsOCISxdpkLx133wnMAc5ri+W1J3Gx\nMfz6yuHExcZw68yFVNdqgjURCUeQe+nkmFlm5HIycA7wcVDLa896Zybzi0uHsnxDBT9/OSrfAhFp\nB4Lcwj8BmG1mS4H5NIzh/zXA5bVr5w7uybXj8vjt3BL+/tGWsOOISBQKci+dpe4+3N2HuvsQd//3\noJbVUdx5/iAG90rnB88uYePOvWHHEZEooyNt21BiXCz3XzWCmtp6bpu5iNo6TaUsIm1Hhd/G+men\n8l9fO5WiNTv49f/7LOw4IhJFVPgh+Mfhvfn6yFzun72SuSu3hh1HRKKECj8kP7toMPnZqXznj4sp\nq6wOO46IRIEmC9/MJje63P9z910cVKhokJIQx/1XjaB8bw3ffWYx9fVRM+uEiISkuS38uxtdnvW5\n+37cylmizsknpPNvF5zC259t5eG3VoUdR0Q6ueYK345w+XDX5RhcPbov55/ak7tf+4QFa3aEHUdE\nOrHmCt+PcPlw1+UYmBl3XTyUEzKSuG3mIsr36NSIIhKM5go/38xeMLMXG13ef71/M8+VFspIjuf/\nXjmcLRVV/MuspTo1oogEIq6Z+y9qdPnuz933+etyHIb37coPvnQSd738MU++v4ZvjM0LO5KIdDJN\nFr67v9n4upnFA0OADe5eGmSwaPTtCfm8t2ob//HSCkb068rgXhlhRxKRTqS53TIfMrPBkcsZwBLg\nd8AiM7uyDfJFlZgY45dfH0Zmcjy3zlzE7mqdGlFEWk9zY/gT3H3/+fmuAz5191OBkcAdgSaLUt26\nJHLvFaexeutu/u0vOjWiiLSe5gp/X6PL/wD8GcDdNweWSBg3IJtbJxUwa+F6nlu4Puw4ItJJNFf4\nO83sAjMbDpwBvAJgZnFActDhotltZw9kVF4WP/7zcorLdjX/BBGRZjRX+DcC04DfAt9ptGV/NvBS\nkMGiXcOpEU8jMS6GaU8toqpGp0YUkePTZOG7+6fufp67n+bujze6/VV3/17g6aLcCRnJ3P31YazY\nVMFdf1sRdhwR6eCa3C3TzO5r6n53v61148jnnX1yD64f359H31nN2AHZnDekZ9iRRKSDau7Aq5uA\n5cAzwEY0f04o7jjvJD5YvZ07nl3CkN7p5HZNCTuSiHRAzY3hnwDMAL4EfAOIB15w9yfc/Ymgw0mD\nhlMjDqfe4baZi6jRqRFF5Bg0N4a/zd0fcvdJwLVAJvChmX2jLcLJQf26pfLfF5/KwrU7uefvn4Yd\nR0Q6oBad8crMRgDfAa4BXgYWBBlKDu/CYb244vQ+TJ9TzFufloUdR0Q6mOamVviZmS0Avgu8CRS6\n+/Xu/lGbpJMv+MlXBzOwexe++8xiSiurwo4jIh1Ic1v4/wpkAMOAu4CFZrbUzJaZ2dLA08kXJCfE\ncv9VI6isquW7f1yiUyOKSIs1t5eO5rxvh07qmcZPLxzMnc8tY/qbxUydVBB2JBHpAJqbHnnN4W43\ns1jgCuCw90vwrji9D3NXbuWev3/K6P5ZFOZlhR1JRNq55sbw083sTjO738zOtQa3AquAy9omohxO\nw6kRT6V3ZjK3zVzEzj37mn+SiES15sbwfw+cBCwDbgBeAy4FLnL3i5p6ogQvLSme+68aTtmuau54\nVqdGFJGmNXtOW3e/1t0fBq4ECoEL3H1x8NGkJYbmZvIv5w3itY+28MS7JWHHEZF2rLnCr9l/wd3r\ngNXuXhlsJDla14/vz+RB3fnvv33M8g3lYccRkXaqucIfZmYVkZ9KYOj+y2ZW0RYBpXlmxt1fH0ZW\nagLTnlrILp0aUUQOo7mpFWLdPT3yk+bucY0up7dVSGleVmoCv77iNNZu38OPn1+m8XwR+YIWTa0g\nHcPo/G7cfvaJ/HnxRn7ywofU6aAsEWmkuQOvjpmZ9QF+B/QE6oEZ7v7roJYnDW6dXMDufbXMeGsV\nm8uruO/K4STFx4YdS0TagSC38GuB77n7ycAYYKqZnRLg8gSIiTF+dP7J/OSrp/D3FVu46jfvs323\n9tEXkQAL3903ufvCyOVKYAXQO6jlyaGuO6M/068ewYcbK7hk+rus3bYn7EgiErI2GcM3szxgODDv\nMPdNMbMiMysqK9OUv63pvCEn8IcbRrNjzz4unj6XJet2hh1JREIUeOGbWRdgFvAdd//CrpzuPsPd\nC929MCcnJ+g4UacwL4tnbxpHUnwsV8x4nzc+3hJ2JBEJSaCFb2bxNJT9H9z9uSCXJUdW0L0Lz90y\njgHdU7nhiSJmfrA27EgiEoLACt/MDHgUWOHu9wS1HGmZ7mlJ/HHKWCYMzOHO55Zxz2ufaF99kSgT\n5Bb+GTSc+HyymS2O/Jwf4PKkGamJcTzyrUIuL+zDfW+s5Pt/WqoTootEkcD2w3f3dwAL6vXl2MTH\nxvDzS06lV2Yyv3r9U0orq3jw6hGkJcWHHU1EAqYjbaOQmXH7OQP530uH8m7xNi5/+H22VOj8uCKd\nnQo/il1W2IfHrj2dkm27ufjBd/lsiyZCFenMVPhR7qwTc3jmxrFU19ZzyfR3+WD19rAjiUhAVPjC\nkN4ZPH/LOLLTErnmkXm8tHRT2JFEJAAqfAGgT1YKz908jqG5GUx9aiGPvL0q7Egi0spU+HJAZkoC\nT94wmi8P6cl/vrSCn72oKZZFOhMVvhwiKT6WB64awT+d0Z/fzi1h2lMLqaqpCzuWiLSCwPbDl44r\nJsb4t6+eQq/MJP7zpRWUVc7jkW8VkpmSEHY0ETkO2sKXI7phQj73XzWcpevLuWT6u6zbrimWRToy\nFb406YKhvfj99aMoq6zm4unvsnxDediRROQYqfClWaPzuzHr5nEkxMZw2cPvMeeT0rAjicgxUOFL\niwzskcZzt4wjr1sq1z9RxDPz14UdSUSOkgpfWqxHehLP3DSWcQO6ccespdz7+qeaYlmkA1Hhy1Hp\nkhjHY9eezqUjc7n39c/44axlmmJZpIPQbply1OJjY/jFpUPplZHEfW+sZEtlFQ9cNYLURP13EmnP\ntIUvx8TM+O65J3HXxafy9mdbuXzGe5RWaoplkfZMhS/H5cpRffnNN0dSXNowxXJx2a6wI4nIEajw\n5bhNHtSDP944hqqaOi6Z/i5FJZpiWaQ9UuFLqxiam8lzN59B15QErnpkHi8v0xTLIu2NCl9aTd9u\nKcy6eRxDeqVzS2SK5XrNtinSbqjwpVVlpSbw1LfHcO4pPfjPl1bwtQfnsmCNhnhE2gMVvrS6pPhY\npl89kl9dPozNFVVcMv09bn96ERt37g07mkhUU+FLIGJijK8Nz+WN703k1skFvLJ8M5N/OYd7X/+U\nvfs0v75IGFT4EqjUxDi+d+5JvP7dszh7UA/uff0zzv7lHF5YslHTMoi0MRW+tIk+WSk8cPUI/jhl\nDF1TE7ht5iK+/tB7LFuv6ZZF2ooKX9rU6PxuvDBtPP9zyamUbNvNhQ+8ww/+tERH6Yq0ARW+tLnY\nGOPy0/sy+/sTmTIhnz8v3sCkX8xh+pxiqms1vi8SFBW+hCYtKZ47zz+Z1/75LMYOyOZ/XvmYf7jn\nLV5Zvlnj+yIBUOFL6Ppnp/LItwr5/fWjSIqP4aYnF3DVb+axYlNF2NFEOhUVvrQbEwbm8LfbJvDv\nFw1mxeYKvnLf2/yf55exbVd12NFEOgUVvrQrcbExfHNsHnO+P5Fvjs3j6fnrmHj3HB59Z7VOtCJy\nnFT40i5lpiTw0wsH88rtExjetyv/8deP+NK9bzH7Y51AXeRYqfClXRvYI40nrjudx64tBIfrHp/P\ntx77gJWllWFHE+lwAit8M3vMzErNbHlQy5DoYGZMHtSDV75zJj/+ysksXLOD8+59m5+9+CHle2rC\njifSYQS5hf84cF6Ary9RJiEuhhsm5DP7BxP5emEfHn+3hIl3z+b375VQq/F9kWYFVvju/hageXGl\n1WV3SeSui0/lpVsncFLPNP71Lx/ylfveYe7KrWFHE2nXQh/DN7MpZlZkZkVlZWVhx5EO5JRe6cz8\n9himXz2C3ftqufqReUz5XRFrtu0OO5pIu2RBHtFoZnnAX919SEseX1hY6EVFRYHlkc6rqqaOR99Z\nzQOzV1Jb51w3Po9pkwpIS4oPO5pIoMxsgbsXtuSxoW/hi7SGpPhYpk4qYPb3J/LVYb14+M1VTLr7\nTZ6Zv06nWRSJUOFLp9IjPYlfXjaMv0w9g75Zydwxayn/8Ks3efjNYkorNCOnRLfAhnTMbCYwEcgG\ntgA/cfdHm3qOhnSkNbk7Ly7dxONzV7Nw7U5iY4yzTszh0pG5nH1ydxLjYsOOKHLcjmZIJ9Ax/KOl\nwpegFJftYtaC9Ty3cAObK6rISI7nwmG9uHRkLkNzMzCzsCOKHBMVvsgR1NU7c1du5dkF63n1w81U\n19YzsHsXLh2Zy9eG96Z7elLYEUWOigpfpAUqqmp4aekmnl2wngVrdhBjRIZ8+nD2yd1JiteQj7R/\nKnyRo1RctovnFjYM+Wwq15CPdBwqfJFjVFfvvFvcMOTzynIN+Uj7p8IXaQUa8pGOQIUv0spWle1i\nloZ8pB1S4YsEREM+0t6o8EXagIZ8pD1Q4Yu0MQ35SFhU+CIhqat33ivexrML1vFyoyGfS0bm8uUh\nPemblaLyl1alwhdpByqqavhbZMinaM0OAHqmJzGqfxaj87MY3T+LATldtAKQ46LCF2lnSrbu5p2V\nW5m3ejvzVm2jtLIagG6pCQ0rgP5ZjOrfjUE904iJ0QpAWu5oCj8u6DAiAnnZqeRlp3LNmH64O2u2\n7eGD1dt5f/U25q3azsvLNwOQnhQXWQF0Y1T/LAb3SicuVrOYS+tQ4Yu0MTM7sAK47PQ+AKzf0bAC\n+GD1duat3s7rK0oBSE2IZWRewyeA0f2zGJqbSUKcVgBybFT4Iu1AbtcUcrumcPGIXABKK6qYd2AF\nsI1fvPoJAEnxMQzv05XR+VmM6p/FiL5dtfuntJjG8EU6gO279zX6BLCNjzZV4A4JsTEM65NxYBho\nZL+upCZqOy6a6EtbkU6ufG8NC9Zsj3wJvJ1lG8qpq3diY4whvTMODAEV5mWRkawTuXdmKnyRKLO7\nupaFa3c0fAJYtZ3F63ayr64eMzi5Zzqj+mcxJj+L0/Oy6NYlMey40opU+CJRrqqmjsXrdh4YAlqw\nZgdVNfUAnJCRxICcLhR078KA7l0YkJNKQfcu5HRJ1DEBHZB2yxSJcknxsYzJ78aY/G7AQPbV1rNs\nQznzS7bz6ZZKikt38aeidezeV3fgOelJcQ0rgcjKYP9PbtcUYnVsQKegwheJAglxMYzs15WR/boe\nuM3d2VxRRXHpblaWVrKybBcrS3cx+5My/rRg/SHPzc9OjXwaiKwIcrqQn5OqPYQ6GBW+SJQyM07I\nSOaEjGTGD8w+5L7yPTWsLNtFcekuiiMrguUbynl52Sbqff/zIbdrMgX7h4cafTLITEkI4TeS5qjw\nReQLMlLiv/CJABq+GyjZtpuVpbsO/BSX7ebd4m1U19YfeFx2lwTyG30a2P99Qa+MJH1PECIVvoi0\nWFJ8LIN6pjOoZ/oht9fVOxt27GVlWWVkiGgXK8t28dLSTZTvrTnwuJSEWAZEhoN6ZSbTIy2RnhlJ\ndE9Pomd6EjlpicRrKonAqPBF5LjFxhh9u6XQt1sKkwcdvN3d2bZ73+c+EeyiqGQHWyo2UVt/6F6C\nZtAtNZGeGYn0SEuiR0YSPdKS6JmReGCl0CM9ia4p8fqkcAxU+CISGDMju0si2V0SI3sMHVRf72zf\ns48tFVVsqahic3n1gctbKqrYWF7F4nU72bZ73xdeNyEuhh7pX1wp9IisEHpEVg7JCfpSuTEVvoiE\nIibm4MpgcK+MIz6uuraOssr9K4NqNpdXsaWyii3lDddXbKxgdkUpexrtYrpfWlIcPdOTGoaNDrNS\n6J6WSGZKPMnxsVHxiUGFLyLtWmJc7IHJ5ZpSWVXDlorqyKeFQ1cKDbufbqW0svoLw0gA8bFGRnIC\nGclxZCTHk5mSQEZyPBnJ8aQnx5MZuZyRHE9GysHr6cnxHWrXVBW+iHQKaUnxpCXFU9C9yxEfU1/f\n8J3C/mGj0spqyvfWUL63hp17aqiIXC6trOKz0krK99RQWV1LUxMSJMXHHFwZJMdHVhzxkRVH/KH3\nfe56W39BrcIXkagRE2PkpCWSk5bIkN5HHkZqrK7eqayqOWTFsP/ygZ/IbTv37mPDzr2s2FTBzj37\nDjmS+XBSE2LJSI6nd9dk/nTTuNb4FZukwhcRaUJsjJGZknBMB5PV1NUf+NSwM/JvxWFWGnFtNHWF\nCl9EJCDxsTF065LYbmYo1REOIiJRItDCN7PzzOwTM1tpZj8MclkiItK0wArfzGKBB4AvA6cAV5rZ\nKUEtT0REmhbkFv4oYKW7r3L3fcDTwEUBLk9ERJoQZOH3BtY1ur4+ctshzGyKmRWZWVFZWVmAcURE\noluQhX+4/Yy+cPiCu89w90J3L8zJyQkwjohIdAuy8NcDfRpdzwU2Brg8ERFpQpCFPx8YaGb9zSwB\nuAJ4IcDliYhIE8ybmiTieF/c7HzgXiAWeMzd/6uZx5cBa45xcdnA1mN8bmej9+JQej8OpffjoM7w\nXvRz9xaNhwda+G3JzIrcvTDsHO2B3otD6f04lN6Pg6LtvdCRtiIiUUKFLyISJTpT4c8IO0A7ovfi\nUHo/DqX346Coei86zRi+iIg0rTNt4YuISBNU+CIiUaLDF76mYD7IzPqY2WwzW2FmH5rZ7WFnCpuZ\nxZrZIjP7a9hZwmZmmWb2rJl9HPk/MjbsTGEys3+O/J0sN7OZZpYUdqagdejC1xTMX1ALfM/dTwbG\nAFOj/P0AuB1YEXaIduLXwCvuPggYRhS/L2bWG7gNKHT3ITQcHHpFuKmC16ELH03BfAh33+TuCyOX\nK2n4g/7CDKXRwsxyga8Aj4SdJWxmlg6cCTwK4O773H1nuKlCFwckm1kckEIUzPXV0Qu/RVMwRyMz\nywOGA/PCTRKqe4E7gPqwg7QD+UAZ8NvIENcjZpYadqiwuPsG4G5gLbAJKHf318JNFbyOXvgtmoI5\n2phZF2AW8B13rwg7TxjM7AKg1N0XhJ2lnYgDRgDT3X04sBuI2u+8zKwrDaMB/YFeQKqZXRNuquB1\n9MLXFMyfY2bxNJT9H9z9ubDzhOgM4EIzK6FhqG+ymT0ZbqRQrQfWu/v+T3zP0rACiFbnAKvdvczd\na4DngHEhZwpcRy98TcHciJkZDWO0K9z9nrDzhMnd73T3XHfPo+H/xRvu3um34I7E3TcD68zspMhN\nZwMfhRgpbGuBMWaWEvm7OZvt5YakAAACaElEQVQo+BI7LuwAx8Pda81sGvAqB6dg/jDkWGE6A/gG\nsMzMFkdu+5G7/y3ETNJ+3Ar8IbJxtAq4LuQ8oXH3eWb2LLCQhr3bFhEF0yxoagURkSjR0Yd0RESk\nhVT4IiJRQoUvIhIlVPgiIlFChS8iEiVU+NIpmdmuyL95ZnZVK7/2jz53/d3WfH2RoKjwpbPLA46q\n8COzsDblkMJ3905/hKZ0Dip86ex+Dkwws8WR+c9jzewXZjbfzJaa2Y0AZjYxci6Bp4Blkdv+bGYL\nInOmT4nc9nMaZlhcbGZ/iNy2/9OERV57uZktM7PLG732nEZz0f8hcnSnSJvq0EfairTAD4Hvu/sF\nAJHiLnf3080sEZhrZvtnSRwFDHH31ZHr/+Tu280sGZhvZrPc/YdmNs3dTzvMsi4GTqNhrvnsyHPe\nitw3HBhMw1xPc2k4Kvqd1v91RY5MW/gSbc4FvhmZemIe0A0YGLnvg0ZlD3CbmS0B3qdhkr6BNG08\nMNPd69x9C/AmcHqj117v7vXAYhqGmkTalLbwJdoYcKu7v3rIjWYTaZgyuPH1c4Cx7r7HzOYAzZ0C\nr6lhmupGl+vQ356EQFv40tlVAmmNrr8K3ByZRhozO/EIJwLJAHZEyn4QDaeM3K9m//M/5y3g8sj3\nBDk0nGHqg1b5LURagbYypLNbCtRGhmYep+G8rnnAwsgXp2XAPx7mea8AN5nZUuATGoZ19psBLDWz\nhe5+daPbnwfGAktoOBHPHe6+ObLCEAmdZssUEYkSGtIREYkSKnwRkSihwhcRiRIqfBGRKKHCFxGJ\nEip8EZEoocIXEYkS/x/Qprah+Ph1UAAAAABJRU5ErkJggg==\n", 277 | "text/plain": "" 278 | }, 279 | "metadata": {}, 280 | "output_type": "display_data" 281 | } 282 | ] 283 | }, 284 | { 285 | "metadata": { 286 | "_cell_guid": "0cccca52-090a-4684-a068-fd3a25c156c5", 287 | "_uuid": "2ba42cb3a0fd3ca3297c22a406d35d54c3d85a44", 288 | "collapsed": true 289 | }, 290 | "cell_type": "markdown", 291 | "source": "### Stochastic Gradient Descent and Mini-Batch Gradient\n\nOne extension to batch gradient descent is the stochastic gradient descent. Instead of calculate the gradient for all observation we just randomly pick one observation (without replacement) an evaluate the gradient at this point. We repeat this until we used all data points, we call this an epoch. We repeat that process for several epochs.\n\nAnother variant use more than one random data point per gradient. Its the so called mini-batch gradient. Please feel free to play with the batch_size and the learning rate to see the effect of the optimization. One advantage is that we don't need to keep all data in memory for optimization, especially if we talking about big data. We just need to load small batches at once to calculate the gradient" 292 | }, 293 | { 294 | "metadata": { 295 | "_cell_guid": "8418ed34-56a3-436a-bf3d-cbddc7bb149e", 296 | "_kg_hide-input": false, 297 | "_kg_hide-output": false, 298 | "_uuid": "94250b7b8821da34736a6bf574f7295bfb0b6eec", 299 | "trusted": false 300 | }, 301 | "cell_type": "code", 302 | "source": "np.random.seed(42)\noptimizer = tf.train.GradientDescentOptimizer(0.1)\ntrain = optimizer.minimize(loss)\ninit = tf.global_variables_initializer()\nn_epochs = 10\nbatch_size = 25\nlosses = np.zeros(n_epochs)\nwith tf.Session() as sess:\n # Initialize the variables\n sess.run(init)\n # Gradient descent\n indices = np.arange(x.shape[1])\n for epoch in range(0,n_epochs):\n np.random.shuffle(indices)\n for i in range(int(np.ceil(x.shape[1]/batch_size))):\n idx = indices[i*batch_size:(i+1)*batch_size]\n x_i = x[:,idx]\n x_i = x_i.reshape(2,batch_size)\n y_i = y[idx]\n sess.run(train, {input_x: x_i, \n y_true:y_i})\n \n if epoch%1==0: \n loss_i = sess.run(loss, {input_x: x, \n y_true:y})\n print(epoch, 'th Epoch Loss: ', loss_i)\n loss_i = sess.run(loss, {input_x: x, \n y_true:y})\n losses[epoch]=loss_i\n print('Found minimum', sess.run(w))\nplt.plot(range(n_epochs), losses)\nplt.title('Loss')\nplt.xlabel('Iteration')\n_ = plt.ylabel('RMSE')", 303 | "execution_count": 28, 304 | "outputs": [ 305 | { 306 | "name": "stdout", 307 | "output_type": "stream", 308 | "text": "0 th Epoch Loss: 0.038630348\n1 th Epoch Loss: 0.038789663\n2 th Epoch Loss: 0.038370226\n3 th Epoch Loss: 0.038309097\n4 th Epoch Loss: 0.03850872\n5 th Epoch Loss: 0.038366485\n6 th Epoch Loss: 0.038474593\n7 th Epoch Loss: 0.038679037\n8 th Epoch Loss: 0.03856797\n9 th Epoch Loss: 0.038507216\nFound minimum [[1.9929324 2.9882016]]\n" 309 | }, 310 | { 311 | "data": { 312 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEWCAYAAABbgYH9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8lfXZ+PHPlT1IDhvJYW/IQYYR\nkIJaQcVJrVYBtbbVqnWPDu3Q1ufp0/qrrXXVamtbq4AoLqy7WhVX2CvsGcJMBELIIOv6/XHuYEwT\nksC5z31OzvV+vfLy5J7XHZAr33F/L1FVjDHGmFCL8zoAY4wxbZMlGGOMMa6wBGOMMcYVlmCMMca4\nwhKMMcYYV1iCMcYY4wpLMMYYY1xhCcaYMBCRrSIy2es4jAknSzDGGGNcYQnGGA+JyPdFZKOI7BOR\neSKS5WwXEXlQRPaKSLGIrBCRgLPvXBFZLSIlIrJDRH7o7VMY0zhLMMZ4RETOAH4DXAp0B7YBzzm7\nzwJOBQYB7YHLgC+cfU8B16lqBhAA3g9j2Ma0WILXARgTwy4H/qaqSwBE5G5gv4j0AaqADGAIsEBV\n19Q7rwoYJiLLVXU/sD+sURvTQtaCMcY7WQRbLQCo6iGCrRS/qr4PPAo8BuwRkSdFJNM59GLgXGCb\niHwoIqeEOW5jWsQSjDHe2Qn0rvtGRNKBTsAOAFV9WFVPArIJdpX9yNm+UFWnAl2BV4Dnwxy3MS1i\nCcaY8EkUkZS6L4KJ4bsiMlJEkoH/A3JVdauInCwiY0UkESgFKoAaEUkSkctFxKeqVcBBoMazJzLm\nKCzBGBM+bwDl9b4mAr8AXgR2Af2Bac6xmcBfCI6vbCPYdfaAs+9KYKuIHASuB64IU/zGtIpYwTFj\njDFusBaMMcYYV1iCMcYY4wpLMMYYY1xhCcYYY4wrYvpN/s6dO2ufPn28DsMYY6LK4sWLi1S1S3PH\nxXSC6dOnD4sWLfI6DGOMiSoisq35o6yLzBhjjEtcTTAiMkVE1jnLkd/VyP5kEZnj7M91FvlDRMaI\nyDLna7mIXFTvnNtFJE9EVonIbOeNaERkkogscc75WEQGuPlsxhhjjs61BCMi8QQX6jsHGAZMF5Fh\nDQ67GtivqgOAB4H7ne2rgBxVHQlMAZ4QkQQR8QO3OPsCQDxfvvn8OHC5c84s4OduPZsxxpjmudmC\nGQNsVNXNqlpJsM7F1AbHTAWedj7PBSaJiKhqmapWO9tTgPrLDSQAqSKSAKQRXDAQ55i61WZ99bYb\nY4zxgJuD/H5ge73vC4CxTR2jqtUiUkxwNdkiERkL/I3garNXOglnh4g8AOQTXMvpHVV9x7nWNcAb\nIlJOcAHAcY0FJSLXAtcC9OrV67gf0hhjTOPcbMFII9saLnzW5DGqmquq2cDJwN3OCrQdCLZ6+hKs\npZEuInUL/d0OnKuqPYC/A39oLChVfVJVc1Q1p0uXZmfZGWOMOUZuJpgCoGe973vw391WR45xurx8\nwL76BziV/EoJloadDGxR1UJnqfKXgPEi0gUYoaq5zmlzgPGhfRxjjDGt4WaCWQgMFJG+IpJEcDB+\nXoNj5gFXOZ8vAd5XVXXOSQAQkd7AYGArwa6xcSKSJiICTALWEFzS3Ccig5xrnelsb9PeW7OHjXtL\nvA7DGGMa5doYjDOmchPwNsHZXn9T1TwRuQ9YpKrzgKeAZ0RkI8GWS92MsAnAXSJSBdQCN6hqEcGx\nmbnAEqAaWAo86dzr+8CLIlJLMOF8z61niwRbi0q59pnFnDm0G3++8iSvwzHGmP8S0/VgcnJyNFrf\n5L9jzjJeWrqDHh1S+fgnZ3gdjjEmhojIYlXNae44e5M/Cm3YU8LLy3bQKT2Jgv3lHCir9DokY4z5\nL5ZgotAf/72BtMR47rkg+N5q3s6DHkdkjDH/zRJMlMnbWczrK3dx9YS+nDowOM161Y5ij6Myxpj/\nFtOrKUejP7yznsyUBK6e2A9faiL+9qmstARjjIlA1oKJIkvy9/Pe2r1cd1p/fKmJAAT8mdZFZoyJ\nSJZgosgf3llPp/QkvjO+z5FtgSwfW4pKKamo8i4wY4xphCWYKPHZpi/4eGMRPzi9P+nJX/ZsBvw+\nAFZbK8YYE2EswUQBVeUP766jW2YyV4zr/ZV92f7gAtKrLMEYYyKMJZgo8NGGIhZu3c9NZwwkJTH+\nK/u6ZqTQLTPZZpIZYyKOJZgIp6r8/p11+NuncllOz0aPCWT5LMEYYyKOJZgI9+7qPawoKObWyQNJ\nSmj8jyvb72NT4SHKKqsb3W+MMV6wBBPBamuVP7y7nr6d0/nmKH+TxwWyMqlVWLPLVlY2xkQOSzAR\n7F8rd7F2dwm3TR5IQnzTf1R1M8nydlo3mTEmcliCiVDVNbX88d31DO6WwQUnZh312O6+FDqlJ9k4\njDEmoliCiVAvL93B5qJSbj9zEHFxjVWW/pKIkO33sXKHTVU2xkQOSzARqLK6lofe28Bwv4+zs7u1\n6JxAViYb9pRQUVXjcnTGGNMylmAi0POLtlOwv5w7zxpEsDJ08wJ+H9W1yvo9NtBvjIkMlmAiTEVV\nDY+8v4Gc3h04bVCXFp8XyAoO9K+ybjJjTISwBBNhZubms+fgYe48a3CLWy8APTumkpmSwCqbSWaM\niRCWYCJI6eFqHv9gI18b0IlT+ndq1bkiQsBvb/QbYyKHJZgI8o9Pt1J0qJI7zhx8TOcH/D7W7iqh\nqqY2xJEZY0zrWYKJEMXlVTzx4SbOGNKVk3p3OKZrZGdlUllTy4Y9h0IcnTHGtJ4lmAjx1MdbOFhR\nzR1nDjrma9S90W/jMMaYSGAJJgLsK63kbx9v4ZzACUeSxLHo2ymd9KR48mwcxhgTASzBRIAnPtpE\naWU1tx9H6wUgLk7IzvJZ8TFjTESwBOOxvSUVPP3pVr4x0s+gbhnHfb1sfyardx6kplZDEJ0xxhw7\nSzAe+9N/NlFVo9w6aWBIrhfI8lFeVcPmQhvoN8Z4yxKMh3YeKGdWbj7fOqkHfTqnh+SaNtBvjIkU\nlmA89Mj7GwC4OUStF4D+XdJJSYyzJWOMMZ6zBOORrUWlPL+ogOljeuJvnxqy6ybExzG0e6a90W+M\n8ZwlGI88/N4GEuKEG78+IOTXDmT5WL3zILU20G+M8ZAlGA9s2FPCy8t2cNX4PnTNTAn59QP+TEoO\nV7NtX1nIr22MMS3laoIRkSkisk5ENorIXY3sTxaROc7+XBHp42wfIyLLnK/lInJRvXNuF5E8EVkl\nIrNFJMXZPr/eOTtF5BU3n+14/PHfG0hLjOe6U/u5cv3sI0v3WzeZMcY7riUYEYkHHgPOAYYB00Vk\nWIPDrgb2q+oA4EHgfmf7KiBHVUcCU4AnRCRBRPzALc6+ABAPTANQ1YmqOtI55zPgJbee7Xjk7Szm\n9ZW7+N6EvnRql+zKPQZ1yyAxXmwmmTHGU262YMYAG1V1s6pWAs8BUxscMxV42vk8F5gkIqKqZapa\n7WxPAeoPJiQAqSKSAKQBO+tfUEQygDOAiGzBPPjuejJTErhmojutF4CkhDgGn5BBns0kM8Z4yM0E\n4we21/u+wNnW6DFOQikGOgGIyFgRyQNWAterarWq7gAeAPKBXUCxqr7T4JoXAe+paqP/uorItSKy\nSEQWFRYWHtcDttbS/P38e81erjutP77URFfvNdzvY9XOYlRtoN8Y4w03E0xj5Rgb/mvX5DGqmquq\n2cDJwN0ikiIiHQi2evoCWUC6iFzR4PzpwOymglLVJ1U1R1VzunRpeUniUPjDu+vpmJ7Ed8b3cf1e\n2Vk+DpRVUbC/3PV7GWNMY9xMMAVAz3rf96BBd1b9Y5wuLx+wr/4BqroGKAUCwGRgi6oWqmoVwXGW\n8XXHikgngl1zr4f0SULg881fMH9DETec3p/05ATX71f3Rn+ejcMYYzziZoJZCAwUkb4ikkRwMH5e\ng2PmAVc5ny8B3ldVdc5JABCR3sBgYCvBrrFxIpImwYL1k4A19a73LeBfqlrh1kMdC1Xl9++so2tG\nMleM6x2Wew45IYP4OLE3+o0xnnHtV2lVrRaRm4C3Cc72+puq5onIfcAiVZ0HPAU8IyIbCbZcpjmn\nTwDuEpEqoBa4QVWLgCIRmQssAaqBpcCT9W47DfitW890rD7aUMTCrfv5n6nZpCTGh+WeKYnxDOza\nzmaSGWM842pfjaq+AbzRYNs99T5XEGx1NDzvGeCZJq55L3BvE/tOP45wXVHXevG3T+XSk3s2f0II\nBfw+Pli3F1Ul2OAzxpjwsTf5Xfbu6j2sKCjm1kkDSU4IT+ulTiArk6JDlewtORzW+xpjDFiCcVVt\nrfKHd9fTt3M63xzdcIa2++oG+lcWWDeZMSb8LMG46PWVu1i7u4TbJg8kIT78P+qh3TMRsdowJrYd\nrq7hwkc/5rH/bPQ6lJjj/nzZGFVdU8uD/17P4G4ZXHBilicxpCcn0K9zus0kMzHt7bxgN/XKHcUM\n9/s4dVB433+LZdaCccnLS3ewubCU288cRFycdwPsAb/P3oUxMW1W7jZ6dEhlQJd23PH8MvaWRNRb\nDG2aJRgXVFbX8tB7Gwj4Mzk7u5unsQz3+9hVXEHRIRvoN7FnU+EhPt+8j+ljevHY5aM5dLia255b\nRo3VSgoLSzAueH7Rdgr2l3PnWYM9nx5sS/ebWDY7N5+EOOFbOT0Y1C2DX16QzaebvuBPNh4TFpZg\nQqyiqoZH3t/ASb07cHoE9PUOy8oEIG+njcOY2FJRVcPcJQWcld2NrhnBwn6XndyTC0dk8eC/17Ng\ny75mrmCOlyWYEJuZm8+eg4e586xBnrdeAHypifTulGYtGBNz3lq1mwNlVcwY8+XyTCLCry8K0LNj\nGrfMXsq+0koPI2z7LMGEUOnhah7/YCPj+3difP/OXodzRCDLZ1OVTcyZlZtP705pjO/f6SvbM1IS\neWzGaPaVVvKjF5ZbSQsXWYIJoac/20rRoUruPGuQ16F8RcDvY/u+corLqrwOxZiw2LCnhAVbg4P7\njc3iDPh93H3uEN5bu5enPt7iQYSxwRJMiBysqOKJDzfz9cFdOKl3R6/D+YqAv24cxloxJjbMWpBP\nYrxwyUk9mjzmO+P7cOawbtz/1lqWbz8QxuhihyWYEHlq/haKy6u486zBXofyX+pmkq20cRgTAyqq\nanhxcQFnZ59A53bJTR4nIvzukhPpmpHCTbOXcLDCWvihZgkmBPaXVvLUx1s4J3DCkfW/IknH9CT8\n7VNZZTPJTAx4fcUuDlZUM2Nsr2aPbZ+WxMPTR7LzQAV3v7jSxmNCzBJMCPz5o02UVlZz+5mRNfZS\nX3ZWJnnWgjExYNaCfPp1TueUfp2aPxg4qXdH7jxrEK+v3MWsBfkuRxdbLMEcp70lFTz96Vamjshi\nULcMr8NpUsDvY3NRKSXWDWDasHW7S1i8bT/Tx/Rq1WsC15/an4kDO/Or11azZpe19EPFEsxx+tN/\nNlFVo9w6OXJbLxBcMgZgza4SjyMxxj2zcreRFB/HxUcZ3G9MXJzw4GUj8aUmctOsJZQernYpwthi\nCeY47DxQzqzcfC4Z3YO+ndO9Dueosp2ZZPbCpWmryitreGnpDs4ZfgId05NafX7ndsk8dNlINheV\ncs+reS5EGHsswRyHR97fiKLcPGmA16E0q2tGCl0zki3BmDbrXyt2UlJRzYwxzQ/uN2X8gM7c/PUB\nvLikgBcXF4QwuthkCeYYbfuilBcWbWf6mF706JDmdTgtEvDbG/2m7Zq1IJ8BXdsxpu/xvYd2y6SB\njOnbkV+8uopNhYdCFF1ssgRzjB56bwPxccKNX4/81kudQFYmG/ceoryyxutQjAmpNbsOsjT/QKsH\n9xuTEB/Hw9NGkZwQx40zl1BRZf+/HCtLMMdg494SXlm6g6vG96FbZorX4bRYwO+jVmHNbpslY9qW\nWbn5JCXEcfFof0iud4Ivhd9fOoK1u0v439dXh+SascgSzDF48N0NpCbGc92p/bwOpVXqXgK192FM\nW1JWWc0rS3dw3vDutE9r/eB+U84Y0o3vT+zLs5/n88bKXSG7bixJ8DqAaHTO8BMY168jnY6yDEUk\n6u5LoWN6ki0ZY9qU15bvpORwy97cb60fnT2EBVv385O5Kxju99GzY3SMt0YKa8Ecg/NPzOLKU/p4\nHUariQjZWZms2mFdZKbtmJWbz8Cu7cjp3SHk105KiOPR6aNA4KbZS6msrg35PdoySzAxJuD3sX5P\nCYerbeDSRL9VO4pZXlDMjLHHP7jflJ4d07j/4hNZvv0Av3t7rSv3aKsswcSY4X4f1bXK+t02/dJE\nv1kL8klOiOObo1r35n5rnTu8O1eM68Vf5m/h/bV7XL1XW2IJJsYEnKX77X0YE+0OHa7m1aU7OP/E\nLHxpia7f7+fnDWPICRnc+fxydhWXu36/tsASTIzp2TGVjJQEe6PfRL15y3ZSWlnjyuB+Y1IS43ns\n8tEcrq7l1tnLqK6x8ZjmWIKJMSJCIMtnCcZEvVkLtjHkhAxG92oftnv279KO//1GgAVb9/HwexvC\ndt9oZQkmBgX8mazZXUKV/QZmotSKggOs2nHQ1cH9pnxzdA8uHt2DR/6zkU83FoX13tHGEkwMCvh9\nVFbXsnGvDfSb6DQrN5/UxHi+MSo0b+631n1Ts+nXOZ1b5yyjsOSwJzFEA1cTjIhMEZF1IrJRRO5q\nZH+yiMxx9ueKSB9n+xgRWeZ8LReRi+qdc7uI5InIKhGZLSIpznYRkV+LyHoRWSMit7j5bNGs7o1+\n6yYz0aikoop5y3dywYjuZKa4P7jfmPTkBB6dMZri8irueH4ZtbVWarkxriUYEYkHHgPOAYYB00Vk\nWIPDrgb2q+oA4EHgfmf7KiBHVUcCU4AnRCRBRPzALc6+ABAPTHPO+Q7QExiiqkOB59x6tmjXt1M6\n6Unx5O20Fy5N9Hll2U7KKmuYMba3p3EM7Z7JvRcMY/6GIv780SZPY4lUbrZgxgAbVXWzqlYS/Ad/\naoNjpgJPO5/nApNERFS1TFXrSsqlAPV/PUgAUkUkAUgDdjrbfwDcp6q1AKq6N+RP1EbExQnDsjKt\nBWOijqoyKzefYd0zGdHD53U4zBjTi/OGd+f376xn8bZ9XocTcdxMMH5ge73vC5xtjR7jJJRioBOA\niIwVkTxgJXC9qlar6g7gASAf2AUUq+o7zrX6A5eJyCIReVNEBjYWlIhc6xyzqLCwMCQPGo2ys3zk\n7TxIjTXtTRRZtv0Aa3Z5M7jfGBHhNxcPJ6t9CjfPWsqBskqvQ4oobiaYxv70G/5r1uQxqpqrqtnA\nycDdIpIiIh0Itnr6AllAuohc4ZyXDFSoag7wF+BvjQWlqk+qao6q5nTp0qXVD9VWBPw+yqtq2FJk\nA/0meszKzSctKZ6pI7O8DuWIzJREHp0+msJDh/nR3BWo2i9tddxMMAUEx0Tq9ODL7qz/Osbp8vIB\nX2lnquoaoBQIAJOBLapaqKpVwEvA+HrXetH5/DJwYsiepA0afmSg38ZhTHQoLq/itRU7mToyiwyP\nBvebMqJne34yZQjvrt7DPz7d6nU4EcPNBLMQGCgifUUkieBg/LwGx8wDrnI+XwK8r6rqnJMAICK9\ngcHAVoJdY+NEJE2C7eNJwBrn/FeAM5zPpwHr3XmstqF/l3SSE+JsHMZEjVeX7aCiqpYZY7wd3G/K\n1RP6MmlIV37zxlpWFtj/V+BignHGVG4C3iaYBJ5X1TwRuU9ELnQOewroJCIbgTuAuqnME4DlIrKM\nYGvkBlUtUtVcgpMBlhAcm4kDnnTO+S1wsYisBH4DXOPWs7UFCfFxDO2eaWuSmahQN7g/3O9jeAQM\n7jdGRHjgWyPo1C6Jm2YvoaSiyuuQPCex3F+Yk5OjixYt8joMz/z8lZW8unQny+89i7g47wdMjWnK\n4m37ufjxT/m/i4aHbe2xY7Vgyz6mPfkZ552YxcPTRkbEZIRQE5HFznj3Udmb/DEskOWj5HA1+fvK\nvA7FmKOalZtPelI8F0bQ4H5TxvTtyB1nDuK15TuZs3B78ye0YZZgYtiRN/qtm8xEsOKyKv61YidT\nR/lplxwdVd5/cPoAvjagE798LY91u0u8DsczlmBi2KBuGSTGi80kMxHtpaUFHK6uZcaYyO4aqy8+\nTnjwspG0S07gpllLKK+MzQqylmBiWFJCHINPyCDPWjAmQtUN7o/o4TvS4o4WXTNSePCykWwsPMQv\n5+V5HY4njppgROSMep/7Ntj3TbeCMuFTVxsmlid7mMi1aNt+Nuw9FPED+02ZOLALN5zenzmLtvPq\nsh1ehxN2zbVgHqj3+cUG+34e4liMB7L9PvaXVbHjgJWANZFnVm4+GckJXDAi8gf3m3L75EHk9O7A\nT19ayZaiUq/DCavmEow08bmx700UCmRlAvZGv4k8+0sreX3lLr4xyk9aUnQM7jcmIT6Oh6ePIjEh\njhtnLqGiKnbGY5pLMNrE58a+N1FoaPdM4uPExmFMxHlxSQGV1bVR2z1WX1b7VB64ZASrdx3kkj9/\nGjMtmeYSTD8RmScir9X7XPd932bONVEgJTGegV3b2ZIxJqKoKrMW5DOqV3uGds/0OpyQmDysG3/9\ndg4F+8s5/+H5MTEm01y7s379lgca7Gv4vYlS2Vk+PtoQu6ULTOTJ3bKPzYWl/O6StrVm7eRh3Xjj\nloncMnsptz63jE82FvHLC7OjugvwaI7aglHVD+t/AZ8CB4E1zvemDQj4MyksOcyegxVeh2IM4Azu\npyRw/onRO7jflKz2qTx37Thu+voAXlhcwIWPftJmX8Zsbpryn0Uk2/nsA5YD/wSWisj0MMRnwuDI\nG/3WTWYiwL7SSt5atZuLR/cgNSne63BckRAfxw/PHswz3xvLgbIqLnz0Y2YvyG9zrws0NwYzUVXr\n3hD6LrBeVYcDJwE/djUyEzbDumciYjPJTGSYu3g7lTVtY3C/ORMGdubNWycypm9H7n5pJTfPXsrB\nNrQKc3MJpn79zzMJ1lxBVXe7FpEJu/TkBPp1Trc1yYznVJXZC7aT07sDg7pleB1OWHTJSObp747h\nR2cP5s1Vuzn/4Y9ZUXDA67BCorkEc0BEzheRUcDXgLfgSPXJVLeDM+ET8PvIsy4y47HPNn3BlqLS\nmGi91BcXJ9z49QHMuXYc1TW1XPz4p/x1/uao7zJrLsFcR7Bo2N+B2+q1XCYBr7sZmAmvQJaPncUV\nfHHosNehmBg2c0E+vtREzh3e3etQPJHTpyNv3DqR0wZ15X9fX8M1Ty9if2ll8ydGqOZmka1X1Smq\nOlJV/1Fv+9uqeqfr0ZmwyfY7b/TvtHEY442iQ4d5Jy84uJ+S2DYH91uifVoSf/n2Sdx7wTDmbyji\n3Ifns2DLPq/DOiZHnXwtIg8fbb+q3hLacIxXsrO+nEl22qAuHkdjYtELiwqoqlFmjO3pdSieExG+\n+7W+nNynIzfNWsK0Jz/j9smDuOHrA4iPouqzzXWRXQ9MAHYCi4DFDb5MG+FLTaR3pzRbMsZ4orZW\neW5hPmP6dGRA19gY3G+JgN/HazdP4PwTs/j9u+u58qlc9kbR+2rNJZjuwJPA2cCVQCIwT1WfVtWn\n3Q7OhFdw6X7rIjPh9+mmL9j2RVnMDe63REZKIg9NG8n9Fw9nSf5+zn14Ph+tj46VN5obg/lCVf+s\nql8HvgO0B/JE5MpwBGfCK9ufSf6+MorL2s48fBMdZi3YRoe0RKYETvA6lIgkIlx2ci/m3TSBjulJ\nfPtvC7j/rbVU1dR6HdpRtaiipYiMBm4DrgDexLrH2qSAMw6Tt8u6yUz47C2p4J28PTE/uN8Sg7pl\n8OqNE5g+piePf7CJy574jIL9ZV6H1aTmlor5lYgsBu4APgRyVPVqVV0dluhMWGUfqQ1jCcaEzwuL\nCqiuVaZb91iLpCbF85tvnsjD00exfs8hzn1oPm/nRea77821YH4B+IARwG+AJSKyQkRWisgK16Mz\nYdWpXTJZvhQbhzFhUze4P65fR/p3aed1OFHlwhFZ/OvmCfTulM51zyzm3ldXRVwxs+bWiLaaLzEm\n4PfZkjEmbOZvLGL7vnJ+dPYQr0OJSn06pzP3B6dw/5vr+NsnW1i0bT+PTB9FvwhJ1s0N8m9r7Aso\nIDh92bQxAb+PLUWlHDpc7XUoJgbMyt1Gx/Qkzs7u5nUoUSs5IZ57LhjGX7+dw44D5VzwyMe8sjQy\nipk1NwaTKSJ3i8ijInKWBN0MbAYuDU+IJpwC/kxUYc0u6yYz7tpzsIJ/r9nLt07qQXKCDe4fr7pi\nZsOyMrltzjJ+9MJyyiq9/UWxuTGYZ4DBwErgGuAd4BJgqqpOPdqJJjrVzSRbWWDdZMZdzy/cTk2t\nMn2MDe6HSlb7VGZ/fxw3nzGAuUuCxczW7vbul8XmEkw/Vf2Oqj4BTAdygPNVdZn7oRkvdM1MoUtG\nso3DGFfV1CrPLdzO1wZ0ok/ndK/DaVMS4uO486wvi5lNffQTZuV6U8ysuQRz5I07Va0Btqhq26zt\naY4Y7veRZzPJjIs+Wl/IjgPlzBjT2+tQ2qz6xcx++vJKbvKgmFlzCWaEiBx0vkqAE+s+i4j9C9RG\nBbIy2bC3hPLKyJryaNqOmbn5dG6XxJnDbHDfTXXFzH48ZTBvOcXMlm8PXzGz5maRxatqpvOVoaoJ\n9T5nhitIE17Zfh+1iqd9t9HqnbzdTPnjR+yL4hoebttVXM77a/fwrZyeJCW0aDERcxzi4oQbTv+y\nmNklfw5fMTNX/3RFZIqIrBORjSJyVyP7k0VkjrM/V0T6ONvHiMgy52u5iFxU75zbRSRPRFaJyGwR\nSXG2/0NEttQ7b6Sbz9aWBfzO0v1WG6bVHvtgE2t3l/Dgu+u9DiVizVm4nVqF6Sfb4H441RUzO31w\nsJjZsjC0ZFxLMCISDzwGnAMMA6aLyLAGh10N7FfVAcCDwP3O9lUEl6UZCUwBnhCRBBHxA7c4+wJA\nPDCt3vV+5BRHG2kTEY5dli+FDmmJrLKZZK2ysqCY5dsP4G+fyszcbdYCbER1TS1zFm5n4sDO9OqU\n5nU4Mad9WhJPXnkSL1x/CqN6dXD9fm62YMYAG1V1s6pWAs8BDac2TwXqlv2fC0wSEVHVMlWtm8Cd\nAtRvyyUAqSKSAKQRrFVjQkhUk4NhAAAbAUlEQVRE7I3+Y/Ds59tITYxn9vfHkZmayK/mrY76muqh\n9sG6QnYVV3C5rTvmGRHh5D4dw3IvNxOMH9he7/sCZ1ujxzgJpRjoBCAiY0Ukj+A7ONerarWq7gAe\nAPKBXUCxqr5T73q/dtZKe1BEkhsLSkSuFZFFIrKosDA6aip4IeD3sX5PCYerbaC/JYrLq3h1+Q6m\njsyiV6c07jhzEJ9t/oK38/Z4HVpEmbUgny4ZyUwaaoP7scDNBNNYXc+Gv841eYyq5qpqNnAycLeI\npIhIB4Ktnr5AFpAuIlc4590NDHGO7wj8pLGgVPVJVc1R1ZwuXaw0cFMCWT6qapQNew55HUpUeHFx\nARVVtVwxLjjtdsaYXgzq1o5fv7E64hYg9MqOA+V8sG4vl+X0JDHeBvdjgZt/ygVA/eLaPfjv7qwj\nxzhdXj5gX/0DVHUNUAoEgMkE38UpVNUq4CVgvHPcLg06DPydYBedOUYBvy3d31KqyrO52xjZs/2R\nCRIJ8XHce0E22/eV89THWzyOMDLMWZCPAped3LPZY03b4GaCWQgMFJG+IpJEcDB+XoNj5gFXOZ8v\nAd5XVXXOSQAQkd4El6vZSrBrbJyIpImIAJOANc5x3Z3/CvANghMFzDHq1TGNjJQEVlqCadZnm75g\nc2HpkdZLna8N6MxZw7rx2H82sieK6qi7obqmljmLtnPqwC707GiD+7HCtQTjjKncBLxNMAk8r6p5\nInKfiFzoHPYU0ElENhIsalY3lXkCsFxElgEvAzeoapGq5hKcDLCE4NhMHPCkc85MEVnpbO8M/K9b\nzxYLRITsrEybqtwCz+Zuo31aIuef2P2/9v3svKFU1yj3v7XWg8gix/tr97Ln4GFm2OB+TGmuHsxx\nUdU3gDcabLun3ucK4FuNnPcMwYU2G7vmvcC9jWw/43jjNV813O/j6c+2UVVTa33mTdhzsIK38/bw\nva/1abTcb+9O6Vw9sS+Pf7CJK8f1DsvU0Eg0a0E+3TKTmTSkq9ehmDCyfzVMkwJ+H5XVtWwqtIH+\npjy3ILgi8OVjm15T68avD6BLRjK/em01tbWxN215+74yPlxfyGU5PUmwX1Riiv1pmyZlO0v3Wwnl\nxlXX1DJ7QT4TB3Y+6orA7ZIT+MmUISzbfoBXlkVGIahwmrNwOwJcZsvyxxxLMKZJfTunk5YUbzPJ\nmvDvNXvZfbCCK8c1vyLwN0f5GdHDx2/fXEtpDFULrXIG908f3BV/+1SvwzFhZgnGNCk+ThjWPdMS\nTBNm5m6juy+FM1owrhAXJ9xzQTZ7Sw7zpw82hiG6yPDemj0UlhxmhrVeYpIlGHNUAb+P1bsOUhOD\nYwdHs6WolPkbipg+pleLxxVO6t2Bi0b5+cv8LWzfV+ZyhN6rqVX+On8L3X0pnD7YXmqORZZgzFEF\n/D7KKmvYUlTqdSgRZebn20iIE6a18qXBn0wZQrwI//fGGpciixyPvL+BRdv2c9vkgTa4H6PsT90c\nVd0b/Xm28OURFVU1vLC4gLOzT6BrZkqrzj3Bl8KNX+/Pm6t28+mmIpci9N7HG4p46L0NXDTKz6U5\n9uZ+rLIEY45qQJd2JCfE2ThMPa8t30lxedV/vbnfUtdM7EePDqnc99pqqmtqQxyd93YXV3Drc0sZ\n0KUdv74oQHBxDROLLMGYo0qIj2NI90ybqlzPs7n5DOjajnH9jm3J85TEeH527lDW7i7huYXbmz8h\nilTV1HLz7CWUVdbw+BWjSUty9V1uE+EswZhmBbIyWbWz2Gqb8GVRsSvG9jqu38ynBE5gXL+O/P6d\ndRSXVYUwQm898M46Fm7dz2++OZwBXTO8Dsd4zBKMadZwv4+SimryY2DmU3Pqiop986Qex3UdEeGe\n87MpLq/ij++1jfLK767ewxMfbmbG2F58Y1TD0k8mFlmCMc2qW4I+1rvJisu+LCqWmZJ43NcblpXJ\n9DG9+Odn29iwpyQEEXpn+74y7nx+GQF/Jvec37AyuolVlmBMswZ2a0divMR8CeUXl3y1qFgo3HHm\nINKT4rnvX9FbXvlwdQ03zlqCAn+acVKji36a2GQJxjQrOSGeQd0yYnomWWNFxUKhU7tkbps8iPkb\ninh/7d6QXTecfv36GlYUFPO7S0bQq5PVejFfsgRjWiSQ5WPVjtgd6K8rKtaSdcda68pTetO/Szr/\n86/VVFZH17Tl15bv5J+fbeOaCX2ZEjjB63BMhLEEY1ok0MPH/rIqdhbHZmXGZz4PFhU7r5GiYscr\nMT6Oey7IZusXZfzj0+gpr7yp8BB3vbiC0b3a85NzhngdjolAlmBMiwSygm/0x2I32Z6DFbyzeg+X\n5vR0bXzhtEFdmDSkKw+/t5HCksOu3COUyitruOHZJSQlxPHojNFWkM40yv5WmBYZ2j2T+DghLwYT\nTF1RMbdXBP7ZeUM5XF3DA2+vc/U+ofCLV1exfm8Jf5w2iixbht80wRKMaZGUxHgGdGnHqp2xNVW5\nrqjYqYO6HLWoWCj069KO736tL88v3s7KgshN5M8v2s7cxQXc/PUBnDbIVkk2TbMEY1os2x97tWHq\niopdMTY89UxuOmMAndKT+NVreRE5oWLNroP84pVVjO/fiVsnD/I6HBPhLMGYFgtk+dhbcpi9B2Nn\noP/Zz1teVCwUMlMS+dHZg1m0bT+vrdgVlnu2VElFFTfMXIIvNZGHpo0iPs4WsTRHZwnGtNjwHs4b\n/THywuXmwkN8vLGIGa0oKhYKl5zUk4A/k9+8sYbyypqw3fdoVJW7XlxJ/r4yHpk+ii4ZyV6HZKKA\nJRjTYkO7ZyISO0vGzMzNJyFOuGxMeOuZxMcJ916Qza7iCv784aaw3rsp//xsG6+v3MUPzxrM2H6d\nvA7HRAlLMKbF2iUn0LdzekyMw1RU1TB3cQFnB06ga0brioqFwsl9OnLBiCz+/OEmdhwoD/v961u2\n/QD/+/pqJg3pynWn9vM0FhNdLMGYVglk+ciLgZlkR4qKjQ39m/stddc5QxCB33hYXvlAWSU3zlxC\n14wUfn/pCOJs3MW0giUY0yoBfyY7DpSzr7TS61Bc9ezn246rqFgo+Nunct2p/fnXil0s2LIv7Pev\nrVXufH45e0sqeOzy0bRPSwp7DCa6WYIxrfLl0v1tt5tsRcEBlhcUH3dRsVC4/rT+dPel8KvX8qip\nDe+05Sc+2sx7a/fys3OHMrJn+7De27QNlmBMq2Rntf2ZZKEqKhYKqUnx3H3uUPJ2HuSFReErr5y7\n+QseeGcd5w3vzlXj+4TtvqZtsQRjWsWXmkivjmnktdGZZMVlVcxbvpNvjApNUbFQuODE7uT07sDv\n3l7HwQr3yysXlhzm5tlL6dUxjd9ePNzzVpyJXpZgTKsF/JlttgUz1ykqdrmHg/sNiQSnLe8rq+SR\n9za4eq+aWuXW55ZSXF7Fny4fTUaEJFkTnSzBmFbLzvKx7Ysyisvd/206nFSVmbnbGNUrtEXFQmF4\nDx+XntSTv3+ylU2Fh1y7z0P/Xs+nm77gf6YGGNo907X7mNhgCca0Wt0/vnltrBVTV1TMy6nJR/PD\nsweTkhjPr193Z9ryh+sLeeQ/G7nkpB5cenJ4Xy41bZOrCUZEpojIOhHZKCJ3NbI/WUTmOPtzRaSP\ns32MiCxzvpaLyEX1zrldRPJEZJWIzBaRlAbXfERE3PsVzxypDdPWxmHcLCoWCl0ykrll0gDeX7uX\n/6wLbXnlXcXl3D5nGYO6ZvA/UwMhvbaJXa4lGBGJBx4DzgGGAdNFZFiDw64G9qvqAOBB4H5n+yog\nR1VHAlOAJ0QkQUT8wC3OvgAQD0yrd88cwOZTuqxTu2SyfCltahwmHEXFQuE74/vSt3OwvHJVTWjK\nK1fV1HLTrKUcrqrhT1eMJjUpcp/fRBc3WzBjgI2qullVK4HngKkNjpkKPO18ngtMEhFR1TJVrXa2\npwD1XwBIAFJFJAFIA3bCkYT2O+DHrjyN+Ypsv69NvQsze0E+NbXK5WFalv9YJSXE8fPzhrK5sJR/\nfrYtJNf8f2+tZfG2/fz24hPp36VdSK5pDLibYPxA/Yn7Bc62Ro9xEkox0AlARMaKSB6wErheVatV\ndQfwAJAP7AKKVfUd51o3AfNUNbLWOG+jAlk+NheVUnq4uvmDI1xVvaJivTu5W1QsFM4Y0pVTB3Xh\nj/9ezxeHjq+88tt5u/nL/C18+5TeXDAiK0QRGhPkZoJpbPJ8w1eRmzxGVXNVNRs4GbhbRFJEpAPB\nVk9fIAtIF5ErRCQL+BbwSLNBiVwrIotEZFFhYWErHsfUF/BnohosQBXt3luzhz0HD4etqNjxEhHu\nOX8oZZU1/P7d9cd8nfwvyvjhC8s5sYePn503NIQRGhPkZoIpAOpPRemB053V2DFOl5cP+MqiS6q6\nBigFAsBkYIuqFqpqFfASMB4YBQwANorIViBNRDY2FpSqPqmqOaqa06WLlXs9VsOdmWQr20A32bOf\n55MVxqJioTCgawbfPqU3sxfkH9NsvoqqGm6YtRgBHpsxmuQEG3cxoedmglkIDBSRviKSRHAwfl6D\nY+YBVzmfLwHeV1V1zkkAEJHewGBgK8GusXEikibB14snAWtU9XVVPUFV+6hqH6DMmThgXNI1M4Uu\nGclRXxumrqjY9DAXFQuF2yYNon1qIve9trrV5ZX/51+rWbXjIL+/dCQ9O6a5FKGJda79H+WMqdwE\nvA2sAZ5X1TwRuU9ELnQOewro5LQ27gDqpjJPAJaLyDLgZeAGVS1S1VyCkwGWEBybiQOedOsZzNEF\nsjKj/l0Yr4qKhYIvLZE7zxpM7pZ9vLlqd4vPe3XZDmbm5nPdqf04c1g3FyM0sS7BzYur6hvAGw22\n3VPvcwXBsZOG5z0DPNPENe8F7m3mvjYVJgwCfh8fbSiioqomoqf2NqW80tuiYqEwfUwvnv18G79+\nfQ1nDOna7J/Dxr0l3P3SSk7u04Efnj04TFGaWBVdfQImomRn+aipVdbuLvE6lGPy2opgUbErx0Xm\nm/stER8n3HPBMHYcKOev8zcf9diyymp+8OwSUhPjeWT6aBKjrEvQRB/7G2aO2fAe0T3QP/PzbQzs\n2o6xfb0rKhYK4/t35pzACTz2n03sLq5o9BhV5ecvr2Jj4SEemjaKE3zR2WIz0cUSjDlmWb4UOqQl\nkheFCaauqNjlEVBULBR+eu5QalS5/621je6fs3A7Ly3dwa2TBjJhYOcwR2dilSUYc8xEhIDfF5VL\nxkRSUbFQ6NkxjWsn9uPlpTtYvG3/V/bl7Szmnnl5TBzYmZvPGOhRhCYWWYIxxyU7y8e63SVUVodm\nXaxwiMSiYqHwg9P70y0zmftey6PWKa98sKKKG2YuoWNaEn+8bCTxcdHfWjPRwxKMOS4BfyZVNcr6\nPdEz0F9XVOyKKB7cb0x6cgJ3nTOE5QXFvLR0B6rKT+auoGB/OY/MGEWndsleh2hijCUYc1wCWcGB\n/qc/3RoVrRhVZebnwaJi2VmRVVQsFKaO8DOqV3vuf2stj/1nI2+u2s1Ppgzm5D7RPZHBRCdLMOa4\n9O6UxtUT+vLC4gK+9edP2b6vzOuQjurTTV+wuag0qqcmH01cXLC8cmHJYR54Zz1nDuvG9yf28zos\nE6MswZjjIiL84vxhPH75aDYXlXLuw/N5c2XkLmj97Ofb6JCWyLnDI7OoWCiM7Nmeq07pzcCu7Xjg\nkhFtYpaciU6uvslvYsc5w7sT8Pu4adYSfjBzCd8+pTc/PXdoRL3hv7s4WFTs6gl9IyouN/zywmxU\ngy0aY7xiLRgTMj07pvHC9eO5ZkJf/vnZNi5+/FO2FJV6HdYRzy2MjqJioSAillyM5yzBmJBKSojj\n5+cP46/fzmHHgXLOf3g+ry7b4XVYUVdUzJi2wBKMccXkYd1445aJDO2eya3PLeOuF1dQXlnjWTx1\nRcXa6uC+MZHIEoxxTVb7VGZfO44bTu/Pcwu3M/Wxj9ng0fsy0VhUzJhoZwnGuCoxPo4fTxnC098b\nwxeHKrnw0U94YdH2sMZQV1Rsxthe9ia7MWFkCcaExWmDuvDGrRMZ0dPHj+au4I45yyg9XB2We9cV\nFbv05OgrKmZMNLMEY8KmW2YKM68Zx22TB/Lysh1c8OjHrNnlbsnl8soaXli0nSlRXFTMmGhlCcaE\nVXyccNvkQcy8ZiwlFdVMfewTZuZua3VN+ZZ6bcVODlZUt7l1x4yJBpZgjCfG9+/Mm7dOZGzfjvzs\n5VXcPHspJRVVIb9PWykqZkw0sgRjPNO5XTJPf3cMP54ymDdX7eb8Rz5mZUHoasvUFRW7YlxvWy7F\nGA9YgjGeiosTbjh9AM9dO47K6loufvxT/vHJlpB0mdUVFbtotD8EkRpjWssSjIkIJ/fpyBu3TGTi\nwM788rXVXPfMYorLjr3L7MuiYv42VVTMmGhiCcZEjA7pSfz1qhx+ft5Q3l+7l3Mfns+S/P3Nn9iI\nL4uKtf11x4yJVJZgTEQREa6Z2I+5PxiPCFz658948qNNR0oAt0RdUbHRbbSomDHRwhKMiUgje7bn\n9VsmMnloN/7vjbVc/fRC9pVWtujcuqJiNjXZGG9ZgjERy5eayONXjOa+qdl8svELzn1oPgu27Gv2\nvGc+a/tFxYyJBpZgTEQTEb59Sh9eumE8KYlxTHvyMx59f0OTXWa7iyt4d80eLs3p2eaLihkT6SzB\nmKgQ8Pt47eYJnHdiFg+8s56r/r6AwpLD/3VcXVGxGTFQVMyYSGcJxkSNjJREHp42kt9+czgLtuzj\nnIfm88nGoiP764qKnWZFxYyJCJZgTFQREaaN6cWrN30NX2oCVzyVyx/eXU9NrR4pKmaD+8ZEhgSv\nAzDmWAw5IZPXbp7AL17J4+H3NpC7+Quqamrxt0+1omLGRAhLMCZqpSUl8PtLR3BK/0784pVVlFfV\n8MOzBllRMWMihKtdZCIyRUTWichGEbmrkf3JIjLH2Z8rIn2c7WNEZJnztVxELqp3zu0ikiciq0Rk\ntoikONufco5dISJzRaSdm89mIsclJ/XgtZsncPWEvlw5ro/X4RhjHOJWHQ4RiQfWA2cCBcBCYLqq\nrq53zA3Aiap6vYhMAy5S1ctEJA2oVNVqEekOLAeygG7Ax8AwVS0XkeeBN1T1HyKSqaoHnev+Adir\nqr89Wow5OTm6aNGikD+7Mca0ZSKyWFVzmjvOzRbMGGCjqm5W1UrgOWBqg2OmAk87n+cCk0REVLVM\nVevq6aYA9bNgApAqIglAGrAToF5yESC1wTnGGGPCzM0E4we21/u+wNnW6DFOQikGOgGIyFgRyQNW\nAterarWq7gAeAPKBXUCxqr5TdzER+TuwGxgCPOLGQxljjGkZNxNMYyOtDVsVTR6jqrmqmg2cDNwt\nIiki0oFgq6cvwS6zdBG54siJqt91tq8BLms0KJFrRWSRiCwqLCxs7TMZY4xpITcTTAHQs973PXC6\nsxo7xuny8gFfWWxKVdcApUAAmAxsUdVCVa0CXgLGNzi+BpgDXNxYUKr6pKrmqGpOly5djvHRjDHG\nNMfNBLMQGCgifUUkCZgGzGtwzDzgKufzJcD7qqrOOQkAItIbGAxsJdg1Nk5E0pyxlknAGgka4Bwv\nwAXAWhefzRhjTDNcew/GmQF2E/A2EA/8TVXzROQ+YJGqzgOeAp4RkY0EWy7TnNMnAHeJSBVQC9yg\nqkVAkYjMBZYA1cBS4EmCXW1Pi0im83k58AO3ns0YY0zzXJumHA1smrIxxrReJExTNsYYE8NiugUj\nIoXAtmM8vTNQ1OxRscN+Hl+yn8VX2c/jq9rCz6O3qjY7SyqmE8zxEJFFLWkixgr7eXzJfhZfZT+P\nr4qln4d1kRljjHGFJRhjjDGusARz7J70OoAIYz+PL9nP4qvs5/FVMfPzsDEYY4wxrrAWjDHGGFdY\ngjHGGOMKSzDHoLlKnbFCRHqKyH9EZI1TZfRWr2OKBCISLyJLReRfXsfiNRFp71SYXev8PTnF65i8\n0lQ13rbMEkwrOZU6HwPOAYYB00VkmLdReaYauFNVhwLjgBtj+GdR360ES0YYeAh4S1WHACOI0Z+L\niPiBW4AcVQ0QXJ9x2tHPin6WYFqvJZU6Y4Kq7lLVJc7nEoL/eDQsKhdTRKQHcB7wV69j8Zqz+Oyp\nBBe1RVUrVfWAt1F5qtFqvG2ZJZjWa0mlzpgjIn2AUUCut5F47o/AjwmuAh7r+gGFwN+dLsO/iki6\n10F5oblqvG2VJZjWa0mlzpgiIu2AF4HbVPWg1/F4RUTOB/aq6mKvY4kQCcBo4HFVHUWwcGBMjlk2\nV423rbIE03otqdQZM0QkkWBymamqL3kdj8e+BlwoIlsJdp2eISLPehuSpwqAAlWta9XOJZhwYlGz\n1XjbIkswrdeSSp0xwake+hSwRlX/4HU8XlPVu1W1h6r2Ifj34n1VbfO/pTZFVXcD20VksLNpErDa\nw5C81Gg1Xo9jcp1rFS3bqqYqdXoclle+BlwJrBSRZc62n6rqGx7GZCLLzcBM55exzcB3PY7HE6qa\n20Q13jbNlooxxhjjCusiM8YY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY4wxrrAEY0wIiMgh5799\nRGRGiK/90wbffxrK6xvjFkswxoRWH6BVCcZZoftovpJgVLXNvwFu2gZLMMaE1m+BiSKyzKn/ES8i\nvxORhSKyQkSuAxCR051aOrOAlc62V0RksVMz5Fpn228JrsC7TERmOtvqWkviXHuViKwUkcvqXfuD\nenVYZjpvjxsTVvYmvzGhdRfwQ1U9H8BJFMWqerKIJAOfiEjdKrpjgICqbnG+/56q7hORVGChiLyo\nqneJyE2qOrKRe30TGEmwzkpn55yPnH2jgGyC6+R9QnDVhY9D/7jGNM1aMMa46yzg285SOrlAJ2Cg\ns29BveQCcIuILAc+J7ig6kCObgIwW1VrVHUP8CFwcr1rF6hqLbCMYNedMWFlLRhj3CXAzar69lc2\nipxOcPn6+t9PBk5R1TIR+QBorqTu0bq9Dtf7XIP9v248YC0YY0KrBMio9/3bwA+csgaIyKAmim75\ngP1OchlCsAR1naq68xv4CLjMGefpQrB65IKQPIUxIWC/1RgTWiuAaqer6x8Ea9L3AZY4A+2FwDca\nOe8t4HoRWQGsI9hNVudJYIWILFHVy+ttfxk4BVhOsOjdj1V1t5OgjPGcraZsjDHGFdZFZowxxhWW\nYIwxxrjCEowxxhhXWIIxxhjjCkswxhhjXGEJxhhjjCsswRhjjHHF/wcqeIcuFEpKBgAAAABJRU5E\nrkJggg==\n", 313 | "text/plain": "" 314 | }, 315 | "metadata": {}, 316 | "output_type": "display_data" 317 | } 318 | ] 319 | }, 320 | { 321 | "metadata": { 322 | "_cell_guid": "b45c744b-b029-45d2-ba7f-a95443f7670d", 323 | "_uuid": "3975743d7015bea464894b6f0c0af75d2076a3ed", 324 | "collapsed": true 325 | }, 326 | "cell_type": "markdown", 327 | "source": "## Logistic Regression in TensorFlow\n\nNow we have all tools to build our Logistic Regression model in TensorFlow. \nIts quite similar to our previous toy example. Logisitc regression is also a kind of linear model, it belong to the class of generalized linear models with with the logit as a link function. As we have seen in the previous part we assume in logistic regression that the logits (logarithm of the odds) are linear in the parameters/weights.\n\nOur data set has 30 features, so we adjust the placeholders and the weights accordingly. We have seen that the minimizing the cross entropy is aquivalent to maximizing the likelihood function.. TensorFlow provides us with the loss function `sigmod_cross_entropy`, so we don't need to implement the loss function by ourself (let us use this little shortcut, the cross entropy or negative log likelihood is quite easy to implement). The loss function takes the logits and the true lables (response) as inputs. It computes the entropy elementwise, so we have to take the mean or sum of the output of the loss function." 328 | }, 329 | { 330 | "metadata": { 331 | "_cell_guid": "ed48f23a-d4c5-4ec3-a7e1-a3a3a7566758", 332 | "_uuid": "dad569099b3e4d573e5f0e62bfb0a55b9202af72", 333 | "collapsed": true, 334 | "trusted": false 335 | }, 336 | "cell_type": "code", 337 | "source": "# Setup the computational graph with loss function\ninput_x = tf.placeholder(tf.float32, shape=(None, 30))\ny_true = tf.placeholder(tf.float32, shape=(None,1))\nw = tf.Variable(initial_value=tf.random_normal((30,1), 0, 0.1, seed=42), dtype=tf.float32)\nlogit = tf.matmul(input_x, w)\nloss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y_true, logits=logit))", 338 | "execution_count": 30, 339 | "outputs": [] 340 | }, 341 | { 342 | "metadata": { 343 | "_cell_guid": "ba95512d-d387-46e1-82cc-9f1947192e36", 344 | "_uuid": "64ebcd39217eec69a8861be461a0a4f54d62a0e4", 345 | "collapsed": true 346 | }, 347 | "cell_type": "markdown", 348 | "source": "To get the prediction we have to use the sigmoid function on the logits." 349 | }, 350 | { 351 | "metadata": { 352 | "collapsed": true, 353 | "trusted": false, 354 | "_uuid": "4a2ef4e336bef3a2a3b4a2fe3e1109442e3b6e6d" 355 | }, 356 | "cell_type": "code", 357 | "source": "y_prob = tf.sigmoid(logit)", 358 | "execution_count": 31, 359 | "outputs": [] 360 | }, 361 | { 362 | "metadata": { 363 | "_uuid": "bd3bc7c90df1dd836a75b82f8acae81dc46ad83a" 364 | }, 365 | "cell_type": "markdown", 366 | "source": "For the training we can almost reuse the code of the gradient descent example. We just need to adjust the number of iterations (100, feel free to play with this parameter). and the function call. In each iteration we call our training operator, calculate the current loss and the current probabilities and store the information to visualize the training.\n\nEvery ten epoch we print the current loss and AUC score." 367 | }, 368 | { 369 | "metadata": { 370 | "_cell_guid": "23bf5788-b68d-4d21-bfc1-7d0d39fe7f62", 371 | "_uuid": "90f3a11bf9d202214a8e0d5434eccb73d9a76c2d", 372 | "trusted": false 373 | }, 374 | "cell_type": "code", 375 | "source": "optimizer = tf.train.GradientDescentOptimizer(0.5)\ntrain = optimizer.minimize(loss)\ninit = tf.global_variables_initializer()\nn_epochs = 100\nlosses = np.zeros(n_epochs)\naucs = np.zeros(n_epochs)\nwith tf.Session() as sess:\n # Initialize the variables\n sess.run(init)\n # Gradient descent\n for i in range(0,n_epochs):\n _, iloss, y_hat = sess.run([train, loss, y_prob], {input_x: X_train,\n y_true: y_train.reshape(y_train.shape[0],1)})\n losses[i] = iloss\n aucs[i] = roc_auc_score(y_train, y_hat)\n if i%10==0:\n print('%i th Epoch Train AUC: %.4f Loss: %.4f' % (i, aucs[i], losses[i]))\n \n # Calculate test auc\n y_test_hat = sess.run(y_prob, {input_x: X_test,\n y_true: y_test.reshape(y_test.shape[0],1)})\n weights = sess.run(w)", 376 | "execution_count": 36, 377 | "outputs": [ 378 | { 379 | "name": "stdout", 380 | "output_type": "stream", 381 | "text": "0 th Epoch Train AUC: 0.1518 Loss: 0.7446\n10 th Epoch Train AUC: 0.8105 Loss: 0.6960\n20 th Epoch Train AUC: 0.8659 Loss: 0.6906\n30 th Epoch Train AUC: 0.9640 Loss: 0.6893\n40 th Epoch Train AUC: 0.9798 Loss: 0.6884\n50 th Epoch Train AUC: 0.9816 Loss: 0.6876\n60 th Epoch Train AUC: 0.9818 Loss: 0.6868\n70 th Epoch Train AUC: 0.9818 Loss: 0.6861\n80 th Epoch Train AUC: 0.9819 Loss: 0.6853\n90 th Epoch Train AUC: 0.9820 Loss: 0.6845\n" 382 | } 383 | ] 384 | }, 385 | { 386 | "metadata": { 387 | "_uuid": "b610a085c613c9bc0cfce212ca25e816ebb4aad4" 388 | }, 389 | "cell_type": "markdown", 390 | "source": "" 391 | }, 392 | { 393 | "metadata": { 394 | "trusted": false, 395 | "_uuid": "a4e1d5a7b686347305dc590a5749a59e8c677aba" 396 | }, 397 | "cell_type": "code", 398 | "source": "plt.figure(figsize=(11,6))\nplt.subplot(2,1,1)\nplt.plot(range(n_epochs), losses)\nplt.title('Loss')\nplt.xlabel('Iteration')\nplt.ylabel('Loss')\nplt.subplot(2,1,2)\nplt.plot(range(n_epochs), aucs)\nplt.title('AUC')\nplt.xlabel('Epoch')\nplt.ylabel('AUC Score')\nplt.tight_layout()", 399 | "execution_count": 35, 400 | "outputs": [ 401 | { 402 | "data": { 403 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxAAAAGoCAYAAADW/wPMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XmYXNV95//3t6t619JaQbsECJld\nQLPbMV5iYxuDtxiwibfExLvjiTM/kp8zSZx4xk4yXpI4nuB9wWCwsQcvMV6xsc2iFiCMAIEQILUk\n0C61tl7P/FElqdRqSaXurr69vF/PU0/de+651d+Gekr16XPOvZFSQpIkSZLKUZV1AZIkSZJGDgOE\nJEmSpLIZICRJkiSVzQAhSZIkqWwGCEmSJEllM0BIkiRJKpsBQpIkSVLZDBCSpH6JiKcj4qVZ1yFJ\nGloGCEmSJEllM0BIkgZVRLwzIlZGxJaIuD0iZhbbIyI+FREbImJ7RDwUEacXj70yIh6JiLaIWBsR\nH872t5AkHY4BQpI0aCLixcD/At4IzACeAW4uHn4Z8AfAyUATcBWwuXjsi8CfpZTGA6cDvxjCsiVJ\nxyCfdQGSpFHlzcCXUkr3A0TEXwFbI2I+0AmMB54H3JdSerTkvE7g1IhYllLaCmwd0qolSWVzBEKS\nNJhmUhh1ACCltJPCKMOslNIvgH8HPgs8FxE3RMSEYtfXA68EnomIX0XERUNctySpTAYISdJgWgfM\n27cTEY3AFGAtQErpX1NK5wKnUZjK9JfF9iUppSuB6cD3gFuGuG5JUpkMEJKkgaiOiLp9Dwpf/N8e\nEYsjohb4n8C9KaWnI+K8iLggIqqBXcBeoDsiaiLizRExMaXUCewAujP7jSRJR2SAkCQNxI+APSWP\nFwB/A3wHWA+cCFxd7DsB+DyF9Q3PUJja9C/FY38MPB0RO4B3AdcOUf2SpGMUKaWsa5AkSZI0QjgC\nIUmSJKlsBghJkiRJZTNASJIkSSqbAUKSJElS2UbNnainTp2a5s+fn3UZkiRJ0oi0dOnSTSmlaUfr\nN2oCxPz582lpacm6DEmSJGlEiohnyunnFCZJkiRJZTNASJIkSSqbAUKSJElS2QwQkiRJkspmgBgk\nT23axRfuWkVKKetSJEmSpIoxQAySe1Zt5h9/+ChPbNiZdSmSJElSxRggBsmliwqXzL1zxYaMK5Ek\nSZIqxwAxSGZMrGfRceO5c8XGrEuRJEmSKsYAMYguXTSNJU9vYWd7V9alSJIkSRVhgBhEL1w0jc7u\nxO9Wbsq6FEmSJKkiDBCDqHneZBprctz5uNOYJEmSNDoZIAZRTb6KS06ayq9WbPRyrpIkSRqVDBCD\n7NJF01m7bQ8rvZyrJEmSRqGKBoiIuCwiVkTEyoi4vo/jn4qIB4uPxyNiW6/jEyJibUT8eyXrHEwH\nLufqNCZJkiSNPhULEBGRAz4LvAI4FbgmIk4t7ZNS+lBKaXFKaTHwb8BtvV7mH4BfVarGSpjZVM/J\nx43jzse9H4QkSZJGn0qOQJwPrEwprUopdQA3A1ceof81wE37diLiXOA44CcVrLEiLl00nSVPbWWX\nl3OVJEnSKFPJADELWFOy31psO0REzAMWAL8o7lcB/xv4yyP9gIi4LiJaIqJl48bhM2Xo0kXT6Oju\n4XdPbs66FEmSJGlQVTJARB9th7s00dXAt1NK3cX99wA/SimtOUz/wouldENKqTml1Dxt2rQBlDq4\n9l/OdYXTmCRJkjS65Cv42q3AnJL92cC6w/S9Gnhvyf5FwAsi4j3AOKAmInamlA5ZiD0c7buc653F\ny7lG9JWlJEmSpJGnkiMQS4CFEbEgImoohITbe3eKiEXAJODufW0ppTenlOamlOYDHwa+NlLCwz5e\nzlWSJEmjUcUCREqpC3gfcAfwKHBLSml5RHw0Iq4o6XoNcHMaZXde83KukiRJGo1itHxvb25uTi0t\nLVmXcZCXfepXTBtfy41/emHWpUiSJElHFBFLU0rNR+vnnagryMu5SpIkabQxQFTQpSd7OVdJkiSN\nLgaICmqe7+VcJUmSNLoYICqoJl/FxSWXc5UkSZJGOgNEhV26aBprt+3hyY1ezlWSJEkjnwGiwi5d\nNB3wcq6SJEkaHQwQFTarqZ6TjxvHzx91HYQkSZJGPgPEELjstOO596nNbGxrz7oUSZIkaUAMEEPg\n8rNm0pPgvx5en3UpkiRJ0oAYIIbAyceNZ9Fx4/nBMgOEJEmSRjYDxBC5/MwZ3Pf0FtZv35N1KZIk\nSVK/GSCGyOVnzQTghw85CiFJkqSRywAxRBZMbeT0WRP4vgFCkiRJI1hFA0REXBYRKyJiZURc38fx\nT0XEg8XH4xGxrdg+LyKWFtuXR8S7KlnnULn8zJksW7ONNVt2Z12KJEmS1C8VCxARkQM+C7wCOBW4\nJiJOLe2TUvpQSmlxSmkx8G/AbcVD64GLi+0XANdHxMxK1TpUXnXGDAB+4CiEJEmSRqhKjkCcD6xM\nKa1KKXUANwNXHqH/NcBNACmljpTSvpsm1Fa4ziEzZ3IDZ89t4vvL1mVdiiRJktQvlfxiPgtYU7Lf\nWmw7RETMAxYAvyhpmxMRDxVf4xMppVHxrfvyM2fyyPodPLlxZ9alSJIkSceskgEi+mhLh+l7NfDt\nlFL3/o4prUkpnQmcBLw1Io475AdEXBcRLRHRsnHjxkEputJedcYMIvCeEJIkSRqRKhkgWoE5Jfuz\ngcONIlxNcfpSb8WRh+XAC/o4dkNKqTml1Dxt2rQBljs0jp9Yx3nzJ/P9h9aR0uHylCRJkjQ8VTJA\nLAEWRsSCiKihEBJu790pIhYBk4C7S9pmR0R9cXsScAmwooK1DqlXnzmDlRt2suK5tqxLkSRJko5J\nxQJESqkLeB9wB/AocEtKaXlEfDQirijpeg1wczr4z/GnAPdGxDLgV8C/pJR+X6lah9plp8+gymlM\nkiRJGoFitEyjaW5uTi0tLVmXUbZrv3AvrVt388sPX0pEX8tFJEmSpKETEUtTSs1H6zcqLo86El1+\n5gye3rybh9fuyLoUSZIkqWwGiIxcdvrx5KuCHzw0Kq5OK0mSpDHCAJGRpoYaXrBwKj94aL1XY5Ik\nSdKIYYDI0OVnzmTttj3cv3pr1qVIkiRJZTFAZOhlpx1HfXWOby1Zc/TOkiRJ0jBggMjQ+LpqXnP2\nTG5fto7tuzuzLkeSJEk6KgNExq69cB57O3v49v2tWZciSZIkHZUBImOnzZzI2XObuPGeZ1xMLUmS\npGHPADEM/PGF81i1aRe/e3Jz1qVIkiRJR2SAGAZeecYMJjVU8/W7n8m6FEmSJOmIDBDDQF11jjee\nN4efPvocz27fm3U5kiRJ0mEZIIaJN58/j56UuOm+1VmXIkmSJB2WAWKYmDulgReePI2b7ltNZ3dP\n1uVIkiRJfTJADCN/fOE8NrS189NHnsu6FEmSJKlPFQ0QEXFZRKyIiJURcX0fxz8VEQ8WH49HxLZi\n++KIuDsilkfEQxFxVSXrHC4uXTSdWU31fOMeF1NLkiRpeKpYgIiIHPBZ4BXAqcA1EXFqaZ+U0odS\nSotTSouBfwNuKx7aDbwlpXQacBnw6YhoqlStw0WuKnjTBXP53ZObWbmhLetyJEmSpENUcgTifGBl\nSmlVSqkDuBm48gj9rwFuAkgpPZ5SeqK4vQ7YAEyrYK3DxlXnzaE6F3zjHhdTS5IkafipZICYBawp\n2W8tth0iIuYBC4Bf9HHsfKAGeLICNQ47U8fV8sozZvCd+1vZ3dGVdTmSJEnSQSoZIKKPtnSYvlcD\n304pdR/0AhEzgK8Db08pHXJpooi4LiJaIqJl48aNAy54uLj2wnm07e3i9gfXZV2KJEmSdJBKBohW\nYE7J/mzgcN+Ir6Y4fWmfiJgA/BD4SErpnr5OSindkFJqTik1T5s2emY4Nc+bxPOOH8/X73mGlA6X\nuSRJkqShV8kAsQRYGBELIqKGQki4vXeniFgETALuLmmrAb4LfC2ldGsFaxyWIoK3Xjyf5et2cNcT\nm7IuR5IkSdqvYgEipdQFvA+4A3gUuCWltDwiPhoRV5R0vQa4OR38p/Y3An8AvK3kMq+LK1XrcPT6\nc2Yzq6meT/3scUchJEmSNGzEaPly2tzcnFpaWrIuY1B9897V/PV3f8+X334eL1o0PetyJEmSNIpF\nxNKUUvPR+pU1AhERJ0ZEbXH70oj4wFi4L0PW3nDubGZPqufTP3UUQpIkScNDuVOYvgN0R8RJwBcp\nXHL1mxWrSgDU5Kt4/4tPYlnrdn7x2Iasy5EkSZLKDhA9xTUNrwU+nVL6EDCjcmVpn9edM5u5kxv4\n9M+ecBRCkiRJmSs3QHRGxDXAW4EfFNuqK1OSSlXnqnjfi0/i92u387NHHYWQJElStsoNEG8HLgI+\nllJ6KiIWAN+oXFkq9bqzZzFvSgOfci2EJEmSMlZWgEgpPZJS+kBK6aaImASMTyl9vMK1qSifq+ID\nL17II+t3cMfy57IuR5IkSWNYuVdhujMiJkTEZGAZ8OWI+GRlS1OpKxfPZMHURj79s8fp6XEUQpIk\nSdkodwrTxJTSDuB1wJdTSucCL61cWeotn6vigy9ZyGPPtvHj5c9mXY4kSZLGqHIDRD4iZlC4Q/QP\njtZZlfHqs2Zy4rRGPvOzJxyFkCRJUibKDRAfBe4AnkwpLYmIE4AnKleW+pKrCj7wkoWseK6NH/5+\nfdblSJIkaQwqdxH1rSmlM1NK7y7ur0opvb6ypakvl585k5OPG8c/3fEYezq6sy5HkiRJY0y5i6hn\nR8R3I2JDRDwXEd+JiNmVLk6HylUF/3Dl6azZsod//YWDQJIkSRpa5U5h+jJwOzATmAV8v9imDFxw\nwhTe2Dybz/96FY89uyPrciRJkjSGlBsgpqWUvpxS6io+vgJMq2BdOoq/esUpTKiv5q9v+70LqiVJ\nkjRkyg0QmyLi2ojIFR/XApuPdlJEXBYRKyJiZURc38fxT0XEg8XH4xGxreTYjyNiW0R41ac+TGqs\n4f9/5Sncv3obNy1ZnXU5kiRJGiPKDRDvoHAJ12eB9cAbgLcf6YSIyAGfBV4BnApcExGnlvZJKX0o\npbQ4pbQY+DfgtpLD/wz8cZn1jUmvO2cWF50whY//12NsaNubdTmSJEkaA8q9CtPqlNIVKaVpKaXp\nKaXXULip3JGcD6wsXrGpA7gZuPII/a8Bbir5mT8H2sqpb6yKCD722tNp7+zhH37waNblSJIkaQwo\ndwSiL//tKMdnAWtK9luLbYeIiHnAAuAXx1JARFwXES0R0bJx48ZjOXXUOGHaON77opP4/rJ13Lli\nQ9blSJIkaZQbSICIfhw/3Grfq4Fvp5SO6cYGKaUbUkrNKaXmadPG7prud116AidMa+Rv/u/D3htC\nkiRJFTWQAHG0S/+0AnNK9mcD6w7T92pKpi/p2NTmc/zP157hvSEkSZJUcUcMEBHRFhE7+ni0Ubgn\nxJEsARZGxIKIqKEQEm7v42csAiYBd/fzdxBw4QlT+KNzC/eGWL5ue9blSJIkaZQ6YoBIKY1PKU3o\n4zE+pZQ/yrldwPuAO4BHgVtSSssj4qMRcUVJ12uAm1NKB41oRMRdwK3ASyKiNSJe3p9fcCz561ee\nwpRxNbz7G/ezfXdn1uVIkiRpFIpe39tHrObm5tTS0pJ1GZlb+swWrvrPe3jhydP4/Fuaqao62lIV\nSZIkCSJiaUqp+Wj9BrIGQsPQufMm85FXncLPH9vAf9y5MutyJEmSNMoYIEaht148nyvOmsn//unj\n/PrxsXl5W0mSJFWGAWIUigg+/vozOHn6eD548wO0bt2ddUmSJEkaJQwQo1RDTZ7PXXsOXd2J99x4\nP3s7vT+EJEmSBs4AMYqdMG0c//xHZ/FQ63b+/vuPZF2OJEmSRgEDxCh32enH864XnshN963mlpY1\nWZcjSZKkEc4AMQZ8+GUnc/GJU/jIdx/mzhUbsi5HkiRJI5gBYgzI56r4jzefw0nTx3Hd15fy25Wb\nsi5JkiRJI5QBYoxoaqjhG396AQumNPInX13CPas2Z12SJEmSRiADxBgyubGGG995AbMnNfCOryyh\n5ektWZckSZKkEcYAMcZMHVfLN//0Ao6bUMfbvryEB9dsy7okSZIkjSAGiDFo+oQ6vvnOC5jcWMMf\nf/FeHl67PeuSJEmSNEIYIMaoGRPr+eY7L2BCXTXXGiIkSZJUJgPEGDZ7UgM3vfNCGqpzvPE/7+a/\nfr8+65IkSZI0zFU0QETEZRGxIiJWRsT1fRz/VEQ8WHw8HhHbSo69NSKeKD7eWsk6x7K5Uxr43nsv\nYdHx43n3jffzyZ+soKcnZV2WJEmShqlIqTJfFiMiBzwO/CHQCiwBrkkpPXKY/u8Hzk4pvSMiJgMt\nQDOQgKXAuSmlrYf7ec3NzamlpWWQf4uxo72rm7/53sPc0tLKS085jk9ddRbj66qzLkuSJElDJCKW\nppSaj9avkiMQ5wMrU0qrUkodwM3AlUfofw1wU3H75cBPU0pbiqHhp8BlFax1zKvN5/jE68/k7684\njV+u2MBr/+N3PLVpV9ZlSZIkaZipZICYBawp2W8tth0iIuYBC4BfHMu5EXFdRLRERMvGjRsHpeix\nLCJ468Xz+fo7zmfzznau/Pff8KvH/e8qSZKkAyoZIKKPtsPNl7oa+HZKqftYzk0p3ZBSak4pNU+b\nNq2fZaq3i0+ayu3vez4zm+p5+5fv42M/fITdHV1ZlyVJkqRhoJIBohWYU7I/G1h3mL5Xc2D60rGe\nqwqYM7mB295zMVedN5fP3/UUL//0r/nNE5uyLkuSJEkZq2SAWAIsjIgFEVFDISTc3rtTRCwCJgF3\nlzTfAbwsIiZFxCTgZcU2DaGGmjz/63VncPN1F5KvquLaL97LX966jG27O7IuTZIkSRmpWIBIKXUB\n76Pwxf9R4JaU0vKI+GhEXFHS9Rrg5lRyOaiU0hbgHyiEkCXAR4ttysCFJ0zhvz74At596Ync9sBa\nXvrJX/GDh9ZRqSt4SZIkafiq2GVch5qXcR0ay9dt5//7zkM8vHYHL1o0jQ+/fBGnzZyYdVmSJEka\noOFwGVeNQqfNnMj33nMJf/3K59HyzFZe9a+/4d3fWMqKZ9uyLk2SJElDwBEI9dv2PZ188TdP8aXf\nPMWuji5edcYM/vylJ3PS9HFZlyZJkqRjVO4IhAFCA7Ztdwefv2sVX/7t0+zt7ObKxbN41wtPZNHx\n47MuTZIkSWUyQGjIbd7Zzg2/XsVX736avZ09nDtvEm86fy6vOnMGddW5rMuTJEnSERgglJktuzr4\nztJWvnnfap7atIuJ9dW8/pzZvOmCuU5vkiRJGqYMEMpcSom7n9zMjfet5ifLn6WzO3H+/Mm8+qwZ\nvOy04zluQl3WJUqSJKnIAKFhZdPOdm5taeXWpWtYtXEXAOfMbeIVp8/g5acdz9wpDRlXKEmSNLYZ\nIDQspZRYuWEnP374WX68/FmWr9sBwKkzJvDSU4/jkhOnsHhuE7V510xIkiQNJQOERoTVm3dzx/JC\nmLh/9VZSgrrqKs6bP5mLTpzCxSdO5fSZE8jnvGWJJElSJRkgNOJs393JvU9t5ndPbubuJzez4rnC\nzenG1+ZZPLeJM2dP5MzZTZw1u4njJ7p+QpIkaTAZIDTibWxr555Vm7l71WYeXL2NFc+10d1TeL9O\nH1/LmbMLoWLR8eM5+bjxzJ3cQK4qMq5akiRpZCo3QOSHohipP6aNr+XVZ83k1WfNBGBvZzfL1+3g\nodZtPNS6nWWt2/jZo8/t71+br+Kk6eM4+bhCoDhxWiPzpzYyd3KD96GQJEkaJAYIjRh11TnOnTeJ\nc+dN2t+2q72LJzbs5PHn2njiuTZWPLeTe1Zt5rsPrD3o3OMn1DF3SgPzpzQwb0ojs5rqmdlUz8ym\nOo6bUEe1aywkSZLKYoDQiNZYm2fxnCYWz2k6qH37nk6e3rSLZ7bs5plNu3h6825Wb9nFL1dsZGNb\n60F9qwKmj69jZlMdM5rqmT6+lunj6wrPEw5sNzVUE+EUKUmSNLZVNEBExGXAZ4Ac8IWU0sf76PNG\n4O+ABCxLKb2p2P4J4FXFbv+QUvpWJWvV6DKxvpqz5jRxVq9gAbC7o4t12/aybtuewmP7ge1H1u3g\nzh172dXRfch5+apgcmMNkxtrmDqutmS7hqaGGpoaqpnUUMPE+ur92w01OUOHJEkaVSoWICIiB3wW\n+EOgFVgSEbenlB4p6bMQ+CvgkpTS1oiYXmx/FXAOsBioBX4VEf+VUtpRqXo1djTU5Dlp+jhOmj7u\nsH12tXexoa2djW3tbGjby4Yd7Wza2c6WXR1s2tnBll3trNm6my07O2hr7zrs61Tnggl11Uyor2ZC\nXb7wXF9daKvLM74uz7jaPOPqqhlfl2d8bZ5xdXkaawvtjbV5GqpzVLk4XJIkDROVHIE4H1iZUloF\nEBE3A1cCj5T0eSfw2ZTSVoCU0oZi+6nAr1JKXUBXRCwDLgNuqWC90n6NtXkW1OZZMLXxqH3bu7rZ\nvruTrbs72ba7g217is+7O9m2p5MdezrZsbeL7cXttVv3sH1PJ23tXXR09ZRXT01uf6hoqM3RUJOn\nsabw3FA8Vl+To6E6V3guthe2C4+66hz1+45X56mrqaImV+UIiSRJOiaVDBCzgDUl+63ABb36nAwQ\nEb+lMM3p71JKPwaWAX8bEZ8EGoAXcXDwoHjedcB1AHPnzh3s+qWy1OZzTJ+QY/qEY783RXtXNzv3\ndrGzvYu2kudd7YXtXe37trsLzx1d7OkobG/e1cHqLbvZ3dHNzvZCe1fPsV2WuSrYHyzqqnPUVVdR\nX5OjLl8IGrX5Ylvp8eoctdU5avNVxbbS7ar955S273uuyVU5miJJ0ghXyQDR17eE3t9u8sBC4FJg\nNnBXRJyeUvpJRJwH/A7YCNwNHDJPJKV0A3ADFO4DMXilS0OjNp+jdlyOKeNqB+X1Ort72N3RzZ6O\nbnZ3dBW2Owv7ezq72dvZvf/4vva9ncXtzm7aO3v2t+9s72LTzg72Fs/b129vZ3mjJodTk6uithg0\navOF7bp8rth2IHCUHt+/nc9Rk6/qo71qf6ipzVcV+xy6X5Ov8l4hkiQNUCUDRCswp2R/NrCujz73\npJQ6gaciYgWFQLEkpfQx4GMAEfFN4IkK1iqNCtW5KibWVzGxvrpiPyOlRHtXT+FRDBR7uwrhY/9z\nZ/dBbXs7e+joKrQXzi20tRf77Gtr7+ph664O2ot9O/b9nOLxzu6B/50gXxX7A8eBMHNw6NgfUkoC\nS03vPsVzSwNRaVtN78BjkJEkjRKVDBBLgIURsQBYC1wNvKlXn+8B1wBfiYipFKY0rSouwG5KKW2O\niDOBM4GfVLBWSWWKiP1Tl6hgUOlLd0/aH0Q6unuKAWNfGDkQQjpKAk5Hd8/+kNJxSJ9ioOnsKfTr\nKoSWne1dB9pKX6P4MwdDripKgsaBgFIIITlqc73bDgST3uHkSK/T+1htyfF9r1OdC9fCSJLKVrEA\nkVLqioj3AXdQWN/wpZTS8oj4KNCSUrq9eOxlEfEI0A38ZTE01FGYzgSwA7i2uKBa0hiWqwrqi4vD\ns5JSKoaNQpjoKAks+0JJR1chbJSGkPZe/Tq6uw89t7t0vzANbdueA69zyM/rHpwwAxTCRF9BJH9g\ndOWgtj4DTq7XeQcHnOre5/QKPKXbBhpJGr4ipdGxdKC5uTm1tLRkXYYkDZl9YaZ3EOnYH166Dwo6\npcc7jhB4SoNK6chMaXDpOORn9dB9jIv4j6R3oOhru/Ywx6pzh4609O5z4HjuCMcOnJevcpRG0ugX\nEUtTSs1H6+edqCVphIqI4pqM7EZkSu2bYnbQVLFeYaN3gDlkv/foTnGkprM7layJOTDVrPdrdHYf\nHJQG629kERwcMA4TTGr2rYU5TIDZH24OF5B6j+70Dk0lgcd1NJKyYoCQJA2Kg6eYDe36mL6klAqh\n5qCpYX2PxOwLKn0eK9k/JBT12t++p7O43d1nv8G4EMA+petoSqeF9Z4qVu6oTO/Xqu411ay6V7++\nfpbraaSxwQAhSRqVIoJ8Lsjnqmioybqagp5ioOnsYzTmaOGk/ZBj3Qf1Kx256ew+cO7u3V10dKch\nCTVw5Oln1X2sn6nuFUp6n7cvpNT20Xa486pL273/jDToDBCSJA2Rqqqgrqp4FbNhoneo2TddbN8a\nmv37pVPEDgoihUssH276We8RnML9agoXCOg9Pa005AzikhryVXFI8OhzBKX3Av9cFdX5OGitzL5Q\nUp2LwpS1Pi4M4GiNRjsDhCRJY9hwDDUAXd09h6yPOex6lzLX2+wbmWkv3S/p19doTWmAGswrnwFH\nnUJ2tNGaQog5cDGA6lwcNsSUM3WtOucFA1QeA4QkSRp28rmqYTX9DArravaNtvQ51ewo62o6DzMt\nrbOP8NPXaM2+QNPeVTpi1EPXIA7X7L9gQB+XXy6dTlYYmTlw8YD94eUwU80OG2z2j+rk9r9mdR/9\nDTbDiwFCkiSpDBFBTb7wV39qs67mgO6edNCITO+RlSOFmNLws/9Yd2lgKe23LzwVbuC5Y0/XEQLU\n4F7aufRKaEe9zPMR1sv0FYxqe7XtG9mp7XX+QRcjKLblxmiwMUBIkiSNYLmqIDcMp6GVG2xK23tP\nTevrnN5tpefsbO86eESnj4AzmOtrel/iuXdg6SvAHG4qWkNNnve+6KTBK66CDBCSJEkadMM52BwU\nLnpNJTtc4Olr5KavK6qVXpSgnAsH7JsWV5evMkBIkiRJw83B96wZPgZzylelVWVdgCRJkjTWjaS7\nyxsgJEmSJJXNACFJkiSpbAYISZIkSWUzQEiSJEkqW6Q0clZ8H0lEbASeybiMqcCmjGvQ8OJ7Qr35\nnlBvvifUm+8J9TZU74l5KaVpR+s0agLEcBARLSml5qzr0PDhe0K9+Z5Qb74n1JvvCfU23N4TTmGS\nJEmSVDYDhCRJkqSyGSAG1w1ZF6Bhx/eEevM9od58T6g33xPqbVi9J1wDIUmSJKlsjkBIkiRJKpsB\nQpIkSVLZDBCDJCIui4gVEbEyIq7Puh4NvYiYExG/jIhHI2J5RHyw2D45In4aEU8UnydlXauGTkTk\nIuKBiPhBcX9BRNxbfD98KyIEDF9yAAAgAElEQVRqsq5RQycimiLi2xHxWPGz4iI/I8a2iPhQ8d+M\nhyPipoio83Ni7ImIL0XEhoh4uKStz8+GKPjX4nfOhyLinKGu1wAxCCIiB3wWeAVwKnBNRJyabVXK\nQBfwFymlU4ALgfcW3wfXAz9PKS0Efl7c19jxQeDRkv1PAJ8qvh+2An+SSVXKymeAH6eUngecReG9\n4WfEGBURs4APAM0ppdOBHHA1fk6MRV8BLuvVdrjPhlcAC4uP64DPDVGN+xkgBsf5wMqU0qqUUgdw\nM3BlxjVpiKWU1qeU7i9ut1H4YjCLwnvhq8VuXwVek02FGmoRMRt4FfCF4n4ALwa+Xezi+2EMiYgJ\nwB8AXwRIKXWklLbhZ8RYlwfqIyIPNADr8XNizEkp/RrY0qv5cJ8NVwJfSwX3AE0RMWNoKi0wQAyO\nWcCakv3WYpvGqIiYD5wN3Ascl1JaD4WQAUzPrjINsU8D/x3oKe5PAballLqK+35WjC0nABuBLxen\ntX0hIhrxM2LMSimtBf4FWE0hOGwHluLnhAoO99mQ+fdOA8TgiD7avD7uGBUR44DvAH+eUtqRdT3K\nRkRcDmxIKS0tbe6jq58VY0ceOAf4XErpbGAXTlca04pz2q8EFgAzgUYK01N683NCpTL/t8QAMTha\ngTkl+7OBdRnVogxFRDWF8HBjSum2YvNz+4YWi88bsqpPQ+oS4IqIeJrCtMYXUxiRaCpOVQA/K8aa\nVqA1pXRvcf/bFAKFnxFj10uBp1JKG1NKncBtwMX4OaGCw302ZP690wAxOJYAC4tXTaihsADq9oxr\n0hArzm//IvBoSumTJYduB95a3H4r8H+HujYNvZTSX6WUZqeU5lP4TPhFSunNwC+BNxS7+X4YQ1JK\nzwJrImJRseklwCP4GTGWrQYujIiG4r8h+94Tfk4IDv/ZcDvwluLVmC4Etu+b6jRUvBP1IImIV1L4\n62IO+FJK6WMZl6QhFhHPB+4Cfs+BOe9/TWEdxC3AXAr/WPxRSqn3QimNYhFxKfDhlNLlEXEChRGJ\nycADwLUppfYs69PQiYjFFBbV1wCrgLdT+GOenxFjVET8PXAVhSv5PQD8KYX57H5OjCERcRNwKTAV\neA74W+B79PHZUAyb/07hqk27gbenlFqGtF4DhCRJkqRyOYVJkiRJUtkMEJIkSZLKZoCQJEmSVDYD\nhCRJkqSyGSAkSZIklc0AIUnqU0TsLD7Pj4g3DfJr/3Wv/d8N5utLkirHACFJOpr5wDEFiIjIHaXL\nQQEipXTxMdYkScqIAUKSdDQfB14QEQ9GxIciIhcR/xwRSyLioYj4MyjcMC8ifhkR36RwQ0Ui4nsR\nsTQilkfEdcW2jwP1xde7sdi2b7Qjiq/9cET8PiKuKnntOyPi2xHxWETcWLyZkiRpiOWzLkCSNOxd\nT/FO2gDFILA9pXReRNQCv42InxT7ng+cnlJ6qrj/juKdU+uBJRHxnZTS9RHxvpTS4j5+1uuAxcBZ\nFO7IuiQifl08djZwGrAO+C1wCfCbwf91JUlH4giEJOlYvQx4S0Q8CNwLTAEWFo/dVxIeAD4QEcuA\ne4A5Jf0O5/nATSml7pTSc8CvgPNKXrs1pdQDPEhhapUkaYg5AiFJOlYBvD+ldMdBjRGXArt67b8U\nuCiltDsi7gTqynjtw2kv2e7Gf8MkKROOQEiSjqYNGF+yfwfw7oioBoiIkyOisY/zJgJbi+HhecCF\nJcc6953fy6+Bq4rrLKYBfwDcNyi/hSRpUPjXG0nS0TwEdBWnIn0F+AyF6UP3FxcybwRe08d5Pwbe\nFREPASsoTGPa5wbgoYi4P6X05pL27wIXAcuABPz3lNKzxQAiSRoGIqWUdQ2SJEmSRginMEmSJEkq\nmwFCkiRJUtkMEJIkSZLKZoCQJEmSVDYDhCRJkqSyGSAkSZIklc0AIUmSJKlsBghJkiRJZTNASJIk\nSSqbAUKSJElS2QwQkiRJkspmgJAkSZJUNgOEJEmSpLIZICRJAxYRd0bE1oio7dX2p736XRoRrSX7\nEREfiIiHI2JXRLRGxK0RccZQ1i9JKp8BQpI0IBExH3gBkIArjvH0zwAfBD4ATAZOBr4HvGrwKpQk\nDaZ81gVIkka8twD3APcCbwVuLeekiFgIvBe4KKV0X8mhGwe9QknSoDFASJIG6i3AJykEiHsi4riU\n0nNlnPcSoLVXeJAkDXNOYZIk9VtEPB+YB9ySUloKPAm8qczTpwDrK1WbJKkyDBCSpIF4K/CTlNKm\n4v43i20AXUB1r/7VQGdxezMwo+IVSpIGlVOYJEn9EhH1wBuBXEQ8W2yuBZoi4ixgNTC/12kLgGeK\n2z8HPhsRzSmlliEoWZI0CByBkCT112uAbuBUYHHxcQpwF4V1Ed8C3h4R5xcv13oy8CHgZoCU0hPA\nfwA3FS/vWhMRdRFxdURcn8HvI0kqQ6SUsq5BkjQCRcSPgeUppb/o1f5G4F+B2RSCxF8Ac4ANwBeA\nf0op9RT7BoVLuF5HYXRiK/Ab4KMppeVD9KtIko6BAUKSJElS2ZzCJEmSJKlsBghJkiRJZTNASJIk\nSSqbAUKSJElS2UbNfSCmTp2a5s+fn3UZkiRJ0oi0dOnSTSmlaUfrl0mAiIgvAZcDG1JKp/dxPIDP\nAK8EdgNvSyndf6TXnD9/Pi0t3odIkiRJ6o+IeObovbKbwvQV4LIjHH8FsLD4uA743BDUJEmSJOko\nMgkQKaVfA1uO0OVK4Gup4B6gKSJmDE11kiRJkg5nuC6ingWsKdlvLbYdJCKui4iWiGjZuHHjkBUn\nSZIkjVXDNUBEH22H3DI7pXRDSqk5pdQ8bdpR13tIkiRJGqDhehWmVmBOyf5sYF1GtUgaI1JKdPUk\n2rt66Ojqob2rm86uRGdPD13dic7uHrp6El3dPXR2J7p7El09PXT3pP2PrpLt7pTo6Un0JEq2C8dS\ngp5UONZT0q8nJVJpeyrUlYCenuJzKpy/v33ffvF3SKl3GyQKO/v77G8/sF/4j1Dom9K+/ya99ott\nB/Y46Ni+/44H75eee8jfgujdVFLNQcfK7XfI6x/zgQz09WezsWaQ/3+kQX7BI73HDl9Df35O/+ru\n388amp/T3x82dL9T9u8VgNp8Fbe955JBraVShmuAuB14X0TcDFwAbE8prc+4JkkZ6u5J7GzvYlfx\nsaezmz0d3ezp7GZvZ3dxv2f/fntnN3u7ethb3N/beeDY7o4D5+57bu/qpqOrh55h8KUyAqoiqAqI\nCILC/r72KPaJQ9r27UNQ2N7Xzv5zDj0WAHHgO2yU/IzCodi/XXqc0j4lfQ/eP7hj9HWspMb9+73+\nexxo7/VNO/rcPKRL9HUw+ni9DAz2l5cRa9D/f0Tf/98H8or9eL3+/E6DXfeRf1Y/6uv3z+rHOf36\nOUP3Ox2+hmM/pyY/XCcGHSqry7jeBFwKTI2IVuBvgWqAlNL/AX5E4RKuKylcxvXtWdQpafDt6ehm\n0852Nu/qYHPxefvuTrbt6WD7nk627e5k+57Co21vF217DwSGY1VXXUVddY66fG7/dn1NjoaaHJMa\nqqmvyVNfXUV9dY7a6hy1+SpqclXUVu97zlGdq6I6F+Srqsjnotd2FbmqIBdBrirI54J8VVBV3C99\nrqqCXOzb3td+cDDIFbf784+fJElDJZMAkVK65ijHE/DeISpH0iDZ2d7F2q17WLttN2u37qF12x7W\nbdvLum172NjWzqad7ezu6DsI5KqCifXVNNVXM6G+msmNNcyb0si42jzjanM01uaL23kaa/M01OSo\nr85RV3yur84VwkIxKNTmq/wiLklSBQzXKUyShrH2rm6e3LCLx57dwWPPtvHo+sLzxrb2g/rV5KqY\n0VTHzIn1nDO3iSnjapkyroapjYXnKeNqmdJYQ1NDNeNq837hlyRpBDBASDqqHXs7ufvJzfx25Sbu\ne2oLKzfspKu4WKAmX8XJx43j0pOnceL0ccxqqmfWpHpmN9UzdVwtVVWGAkmSRhMDhKRDdHT18MDq\nrfx25SbuWrmJZWu20ZOgvjpH8/xJvOh50zllxgROOX48C6Y2ks+NnIVfkiRpYAwQkoDCJULve3oL\n371/LT/6/Xra2ruoCjhzdhPvfdFJXHLSVM6e20RtPpd1qZIkKUMGCGmMW7mhjdvuX8v/fXAda7ft\noaEmx2WnH8/LTzueC0+YwsT66qxLlCRJw4gBQhqD9nZ2c9v9a7npvtX8fu12qgJesHAa//2yRfzh\nqcfRUONHgyRJ6pvfEqQxZMfeTm68ZzVf/M1TbNrZzqkzJvA3l5/Kq8+awfTxdVmXJ0mSRgADhDQG\nbNixly/99mluvOcZ2tq7eMHCqbz7hYu56MQpXjpVkiQdEwOENIo99uwOvvq7Z/jO0la6enp45Rkz\neNcLT+T0WROzLk2SJI1QBghplNnT0c0PHlrHN+9bzQOrt1GTr+KPmmdz3R+cwLwpjVmXJ0mSRjgD\nhDRKPLp+Bzfdt5rvPrCWtr1dnDitkY+86hRef85sJjXWZF2eJEkaJQwQ0gi1u6OLJU9v5XdPbuI3\nT2xi+bod1OSreNUZM7jm/LmcN3+S6xskSdKgM0BII8Sejm4eat3G757czO+e3MSDa7bR2Z2ozgVn\nz5nkaIMkSRoSBghpmEkp8eyOvTy6fgePrm/jkfU7eHT9Dp7etIueBFUBZ8yayJ88/wQuPnEKzfMn\ned8GSZI0ZPzWIWWgpyexoa2dZzbv4pktu1mzZTfPbN7NM1t288zmXWzb3bm/75zJ9Zxy/ARefeZM\nTp81kfMXTPbu0JIkKTMGCGmQdHX3sH1PJ1t3d7B1dydbd3WwZVcHG9ra2dC2l+d2tLOhrZ2NO/ay\ncWc7nd1p/7m5qmBmUx3zJjfyyjNm8Lzjx3PKjAksOn48E+oMC5IkafgwQGhMSynR2Z3Y09nNno5u\ndnd0sbujm53tXexq72JXR3fhub2Ltr2Fx469nbTt7SzZ7mLrrg527O067M+Z1FDN9PF1TJ9Qy4nT\npjB9fB2zmuqYO6WReZMbmDWpnupc1RD+5pIkSf1jgNCw1tHVw7bdHbS1d7G7vZtdHV3s7uhiV3v3\n/uf2rh7au4rPnYXtvYc8F7b3dnazt9i+p6ObPZ3ddPekoxdSNK42z/i6wmNCXTXTxtVywtRxTGqo\nZlJjDZMaamhqqGZSQw2TG2uY1FjDtHG11OQNB5IkaXQwQChTKSWe3LiTXz++iSc2tLF5Zwebi1N/\nNu9sP+Jf9XuryVVRm6+itrqK2nyuuJ2jrrqKunyOqePy1OYL+/U1eeqrc9TXVNFQk6euOkdDTY76\n6hyNtXkaa4rPtYXnhpo842rz5Kq8LKokSRrbDBAaclt2dfDblZu464mN3PXEJtZv3wvA1HE1TB1X\ny+TGGk6bOWH/9uTGGsbX5WmsydNQm6Ox5sAX+/rqHHXVOWpyVVT55V6SJKniDBAaMr95YhP/fMdj\nPLR2OynBhLo8z184lQ8snMbzT5rKnMkNWZcoSZKkozBAqOL2dnbziR8/xpd/+zQLpjbyoZeezAsW\nTuXM2U1OCZIkSRphDBCqqEfW7eDPv/UAjz+3k7deNI/rX3EK9TW5rMuSJElSPxkgVBHdPYkv3LWK\nf/nJCpoaavjqO87nhSdPy7osSZIkDVAmASIiLgM+A+SAL6SUPt7r+Fzgq0BTsc/1KaUfDXmh6pfW\nrbv5i1uWce9TW7jstOP5n687g8mNNVmXJUmSpEEw5AEiInLAZ4E/BFqBJRFxe0rpkZJuHwFuSSl9\nLiJOBX4EzB/qWnXsnniujTf+5910dPXwz284kzecO5sI1zlIkiSNFlmMQJwPrEwprQKIiJuBK4HS\nAJGACcXticC6Ia1Q/dK6dTd//MX7yOeq+M67L+aEaeOyLkmSJEmDLIvb484C1pTstxbbSv0dcG1E\ntFIYfXh/Xy8UEddFREtEtGzcuLEStapMm3a285Yv3seuji6+9o7zDQ+SJEmjVBYBoq/5LKnX/jXA\nV1JKs4FXAl+PiENqTSndkFJqTik1T5vmAt2stO3t5G1fvo912/fwpbedxykzJhz9JEmSJI1IWQSI\nVmBOyf5sDp2i9CfALQAppbuBOmDqkFSnY7K3s5t3fq2Fx9a38bk3n8t58ydnXZIkSZIqKIsAsQRY\nGBELIqIGuBq4vVef1cBLACLiFAoBwjlKw0xXdw8fuOkB7lm1hX/5o7N40fOmZ12SJEmSKmzIA0RK\nqQt4H3AH8CiFqy0tj4iPRsQVxW5/AbwzIpYBNwFvSyn1nuakDKWU+Kvbfs9PHnmOv331qbzm7N7L\nWCRJkjQaZXIfiOI9HX7Uq+1/lGw/Alwy1HWpfP/7J49z69JWPvCShbz9kgVZlyNJkqQhksUUJo1w\nP3xoPf/+y5Vcfd4cPvTShVmXI0mSpCFkgNAxeezZHXz41mWcM7eJv7/yNG8SJ0mSNMYYIFS2bbs7\nuO5rSxlfl+f/XHsutflc1iVJkiRpiGWyBkIjT3dP4v03PcCz2/dy859dyPQJdVmXJEmSpAwYIFSW\nf7rjMe56YhMff90ZnDN3UtblSJIkKSNOYdJRfX/ZOv7zV6t48wVzufr8uVmXI0mSpAwZIHREj6zb\nwV9+exnN8ybxt68+LetyJEmSlDEDhA5r664Orvt6CxPrq/mPa8+hJu/bRZIkaaxzDYT61NXdw/tv\neoANO9r51p9dyPTxLpqWJEnSAEcgIqIhIv4mIj5f3F8YEZcPTmnK0sf/6zF+s3IT//ja0znbRdOS\nJEkqGuiclC8D7cBFxf1W4B8H+JrK2HcfaOULv3mKt140jzc2z8m6HEmSJA0jAw0QJ6aU/gnoBEgp\n7QG8NfEI9vvW7Vz/nd9z/oLJfOTyU7MuR5IkScPMQANER0TUAwkgIk6kMCKhEWjTznb+7OstTGms\n4T/efA7VORdNS5Ik6WADXUT9t8CPgTkRcSNwCfC2gRalodfZ3cN7bryfzbs6+Pa7LmbquNqsS5Ik\nSdIw1O8AEREBPAa8DriQwtSlD6aUNg1SbRpC//iDR7jvqS186qqzOGP2xKzLkSRJ0jDV7wCRUkoR\n8b2U0rnADwexJg2xW1rW8NW7n+FPn7+A1549O+tyJEmSNIwNdJL7PRFx3qBUokys2riTv/new1xy\n0hSuf8Xzsi5HkiRJw9xA10C8CPiziHgG2EVhGlNKKZ054MpUcd09iQ/fuoy66hyffONi8i6aliRJ\n0lEMNEC8YlCqUCa+9JunuH/1Nj511VkcN8E7TUuSJOnoBvQn55TSM0AT8Orio6nYpmFu5Yad/PNP\nVvCHpx7HaxbPyrocSZIkjRADChAR8UHgRmB68fGNiHj/YBSmytk3damhJsfHXns6hQtqSZIkSUc3\n0ClMfwJckFLaBRARnwDuBv5toIWpcj5/1yoeXLONz1y9mOnjnbokSZKk8g101WwA3SX73cU2DVNP\nPNfGJ3/6OC8/7TiuOGtm1uVIkiRphBnoCMSXgXsj4rvF/dcAXxzga6pCurp7+PCty2isyfGPrznD\nqUuSJEk6ZgMKECmlT0bEncDzKYw8vD2l9MBgFKbB95+/XsWy1u38+5vOZtr42qzLkSRJ0gg00EXU\nFwJPpJT+NaX0GWBlRFxQxnmXRcSKiFgZEdcfps8bI+KRiFgeEd8cSJ2Cx59r4zM/e4JXnnE8l5/p\n1CVJkiT1z0DXQHwO2Fmyv6vYdlgRkQM+S+EeEqcC10TEqb36LAT+CrgkpXQa8OcDrHPM+8zPn6Cu\nuoqPXnl61qVIkiRpBBvwIuqUUtq3k1Lq4ejTos4HVqaUVqWUOoCbgSt79Xkn8NmU0tbi624YYJ1j\n2pZdHfxk+bO84dw5TB3n1CVJkiT130ADxKqI+EBEVBcfHwRWHeWcWcCakv3WYlupk4GTI+K3EXFP\nRFzW1wtFxHUR0RIRLRs3buz3LzHa3XZ/K53diavOm5N1KZIkSRrhBhog3gVcDKwtPi4ArjvKOX1d\n+if12s8DC4FLgWuAL0RE0yEnpXRDSqk5pdQ8bdq0Yyx9bEgpcfOSNZw9t4lFx4/PuhxJkiSNcAO9\nCtMG4OpjPK0VKP1T+GxgXR997kkpdQJPRcQKCoFiSX9rHavuX72VlRt28onXn5F1KZIkSRoF+jUC\nERHvLC50Jgq+FBHbI+KhiDjnKKcvARZGxIKIqKEQQG7v1ed7wIuKrz+VwpSmo02NUh9uvm8NjTU5\nr7wkSZKkQdHfKUwfBJ4ubl8DnAWcAPw34DNHOjGl1AW8D7gDeBS4JaW0PCI+GhFXFLvdAWyOiEeA\nXwJ/mVLa3M9ax6y2vZ384KH1vPqsmTTWDvSegZIkSVL/pzB1FacXAVwOfK34Bf9nEfFPRzs5pfQj\n4Ee92v5HyXaiEEb+Wz/rE/D9ZevZ09nt4mlJkiQNmv6OQPRExIyIqANeAvys5Fj9wMvSYPjWktUs\nOm48i+ccsv5ckiRJ6pf+Boj/AbRQmMZ0e0ppOUBEvBDXKgwLj6zbwbLW7Vx13hwi+rrwlSRJknTs\n+jWFKaX0g4iYB4zfd7O3ohbgqkGpTANyS8saanJVvPbs3rfYkCRJkvqv3ytri4uht/Zq2zXgijRg\nezu7ue3+Vl5++vFMaqzJuhxJkiSNIgO9kZyGoTuWP8uOvV1c7eJpSZIkDTIDxCh0831rmDO5notO\nmJJ1KZIkSRpl+nsjuZdHxBv6aH9zRPzhwMtSfz29aRd3r9rMVc1zqKpy8bQkSZIGV39HIP4e+FUf\n7T8HPtr/cjRQt7SsoSrgDec6fUmSJEmDr78BoiGltLF3Y0rpWaBxYCWpv7q6e7h1aSsvWjSd4yfW\nZV2OJEmSRqH+Boi6iDjkCk4RUY03ksvMnSs2srGtnTe6eFqSJEkV0t8AcRvw+YjYP9pQ3P4/xWPK\nwK1L1zB1XA0vft70rEuRJEnSKNXfAPER4DngmYhYGhH3U7gr9cbiMQ2xTTvb+fmjG3jN4llU57y4\nliRJkiqjv3ei7gKuj4i/B04qNq9MKe0ZtMp0TL73wFq6ehJ/1Oz0JUmSJFVOvwJERLyuV1MCmiLi\nwZRS28DL0rFIKfHtpa2cNXsii44fn3U5kiRJGsX6FSCAV/fRNhk4MyL+JKX0iwHUpGP08NodPPZs\nG//wmtOzLkWSJEmjXH+nML29r/aImAfcAlwwkKJ0bG5pWUNtvoorzpqZdSmSJEka5QZ1tW36f+3d\neZCV1ZnH8e9jIwhiVASVTUDFBXdgXFMZg2bEPXEZRTNjHGescowSJjPGJBVT4yRVibGiY+JYMYmJ\nVllxA6MxLuMwGp0yLtgQ3JVBhEaUdkEj+/LMH/fVadsGrs3lvhfu91PV1fc993D5NZw69z79nvc9\nma8BW9byNbVuy1au5q4Z8zlmn53Ztrf/9JIkSdq4alpARMSewPJavqbW7cHn3+T9Zas4feyQsqNI\nkiSpCXT3IurfUblwuqN+wEDgbzY0lKp3+9NtDNp2Kw7frX/ZUSRJktQEunsR9ZWdjhN4G3glM1ds\nWCRV6/VFS3n0lXYu+vzutGwRZceRJElSE+juRdR/6Ko9Io6IiLMy88INi6VqTGltIxNOG+PeD5Ik\nSaqP7p6B+EhEHAicBfw18CowZUNfU+v34d4Ph4zoxy479Ck7jiRJkppEd6+B2AM4E5hAZenSrUBk\n5udrmE3r8NScd5nz9hIuGjey7CiSJElqIt09A/Ei8ChwYmbOAoiISTVLpfW6fdo8+vbqwbH77Vx2\nFEmSJDWR7t7G9VTgDeChiPh5RBwFVH0Vb0SMj4iXImJWRFy6jn6nRURGxNhu5twsLV6+it8/s4Dj\n9xtIn54bvApNkiRJqlq3CojMvDMzzwD2Ah4GJgE7RcR1EfFX6/qzEdECXAscC4wCJkTEqC76bQNc\nDDzRnYybs98/s4AlK1a794MkSZLqboM2ksvMxZl5c2aeAAwBZgBrPaNQOBiYlZmzi1u+3gKc3EW/\nfwOuAJZtSMbN0R3T2ti1/9aMGbZ92VEkSZLUZGq2E3VmvpOZP8vMcevpOhiY1+G4rWj7SEQcBAzN\nzHvW9UIRcX5ETIuIae3t7d3Kval57e3FPDnnHU4dM4QI936QJElSfdWsgPgUuvrU+9Gu1hGxBXAV\n8PX1vVBmXp+ZYzNz7IABA2oYsXFNbp1PBJwyevD6O0uSJEk1VkYB0QZ03PlsCPB6h+NtgH2BhyNi\nDnAocLcXUsOaNcmU1jY+u3t/Bm7bu+w4kiRJakJlFBBPASMjYkRE9KSyn8TdHz6Zme9lZv/MHJ6Z\nw4HHgZMyc1oJWRvKk3Peoe3dpZw62ounJUmSVI66FxCZuQr4KvAA8AJwW2Y+FxGXR8RJ9c6zKZn8\ndBt9e/XgmH3c+0GSJEnlKGUTgcy8F7i3U9tla+l7ZD0yNbolK1Zx7zMLOH7/gfTu2VJ2HEmSJDWp\nMpYwqRvuf/YNFq9YzWljhq6/syRJkrSRWEBsIia3trFLvz78xXD3fpAkSVJ5LCA2AfMXLeWx/32b\nU0YPdu8HSZIklcoCYhNwZ2sbmXj3JUmSJJXOAqLBZSaTW+dzyIh+DO3Xp+w4kiRJanIWEA2ude4i\nXn1rMaeO8eyDJEmSymcB0eAmt7bRe8sWjttvYNlRJEmSJAuIRrZs5Wp+96fXGb/vzvTtVcqWHZIk\nSdLHWEA0sAeff5M/L1vFaS5fkiRJUoOwgGhgk1vbGLTtVhy26w5lR5EkSZIAC4iGtfD9ZTzycjtf\nGj2YLbZw7wdJkiQ1BguIBnXrU/NYk3CKez9IkiSpgVhANKB3F6/g+kdmc/TeO7HbgL5lx5EkSZI+\nYgHRgP7j4VksXrGKS8bvWXYUSZIk6WMsIBrM/EVLufGx1zh19BD22GmbsuNIkiRJH2MB0WCuevBl\nCJj0hT3KjiJJkiR9ggVEA3nxjfeZ3NrGVw4fzqDtepcdR5IkSfoEC4gG8qP7X6Jvrx7845G7lR1F\nkiRJ6pIFRIN48tV3mKMr+ewAAAkwSURBVPriQi44cje269Oz7DiSJElSlywgGkBm8oP7XmCnz/Ti\n3MNHlB1HkiRJWisLiAbw4PNv0jp3EV87eg9692wpO44kSZK0VhYQJVu1eg1XPPASuw7YmtPHuOu0\nJEmSGpsFRMmmtM5n1sIPuOSYPenR4n+HJEmSGpufWEv03tKVXPVfL3Pg0O04Zp+dy44jSZIkrZcF\nREkWL1/FV371JG99sJzvnDCKiCg7kiRJkrRepRQQETE+Il6KiFkRcWkXz/9TRDwfETMjYmpEDCsj\n58aybOVq/uGmacxse4+fTBjNmGHblx1JkiRJqkrdC4iIaAGuBY4FRgETImJUp27TgbGZuT9wB3BF\nfVNuPCtXr+HCm1v54+y3ufL0/Rm/r0uXJEmStOko4wzEwcCszJydmSuAW4CTO3bIzIcyc0lx+Diw\nWdyeaPWaZNKtM5j64kK+98V9+dJBm8WPJUmSpCZSRgExGJjX4bitaFub84D7unoiIs6PiGkRMa29\nvb2GEWtvzZrkm1Nmcs/MBXzruL04+5DNalWWJEmSmkQZBURXVwtnlx0jvgyMBX7U1fOZeX1mjs3M\nsQMGDKhhxNrKTC6/53lum9bGxeN25/zP7VZ2JEmSJKlbepTwd7YBQzscDwFe79wpIo4Gvg38ZWYu\nr1O2mlu5eg0/uO9Ffv3YHP7uiBFM+sIeZUeSJEmSuq2MAuIpYGREjADmA2cCZ3XsEBEHAT8Dxmfm\nwvpHrI25by9h4q3TmT53EeccNozvnLC3t2uVJEnSJq3uBURmroqIrwIPAC3ADZn5XERcDkzLzLup\nLFnqC9xefOCem5kn1Tvrhrhrxny+feezRMBPJhzEiQcMKjuSJEmStMHKOANBZt4L3Nup7bIOj4+u\ne6ga+WD5Ki777bNMmT6fMcO25+ozDmRovz5lx5IkSZJqopQCYnM1Y94iJt4ynXnvLGHiUSO5aNzu\n9Ghxs29JkiRtPiwgamTy0218Y/JMdtymF7ecfxgHj+hXdiRJkiSp5iwgauSAodty0gGD+O6J+7Bt\nny3LjiNJkiRtFBYQNbL7jtvw4zMOLDuGJEmStFG5QF+SJElS1SwgJEmSJFXNAkKSJElS1SwgJEmS\nJFXNAkKSJElS1SIzy85QExHRDrxWcoz+wFslZ1BjcUyoM8eEOnNMqDPHhDqr15gYlpkD1tdpsykg\nGkFETMvMsWXnUONwTKgzx4Q6c0yoM8eEOmu0MeESJkmSJElVs4CQJEmSVDULiNq6vuwAajiOCXXm\nmFBnjgl15phQZw01JrwGQpIkSVLVPAMhSZIkqWoWEJIkSZKqZgFRIxExPiJeiohZEXFp2XlUfxEx\nNCIeiogXIuK5iJhYtPeLiAcj4pXi+/ZlZ1X9RERLREyPiHuK4xER8UQxHm6NiJ5lZ1T9RMR2EXFH\nRLxYzBWHOUc0t4iYVLxnPBsRv4mIrZwnmk9E3BARCyPi2Q5tXc4NUXFN8ZlzZkSMrndeC4gaiIgW\n4FrgWGAUMCEiRpWbSiVYBXw9M/cGDgUuLMbBpcDUzBwJTC2O1TwmAi90OP4hcFUxHt4Fzisllcry\n78D9mbkXcACVseEc0aQiYjBwMTA2M/cFWoAzcZ5oRr8GxndqW9vccCwwsvg6H7iuThk/YgFRGwcD\nszJzdmauAG4BTi45k+osMxdkZmvx+M9UPhgMpjIWbiy63Qh8sZyEqreIGAIcD/yiOA5gHHBH0cXx\n0EQi4jPA54BfAmTmisxchHNEs+sB9I6IHkAfYAHOE00nMx8B3unUvLa54WTgpqx4HNguIgbWJ2mF\nBURtDAbmdThuK9rUpCJiOHAQ8ASwU2YugEqRAexYXjLV2dXAJcCa4ngHYFFmriqOnSuay65AO/Cr\nYlnbLyJia5wjmlZmzgeuBOZSKRzeA57GeUIVa5sbSv/caQFRG9FFm/fHbVIR0ReYDHwtM98vO4/K\nEREnAAsz8+mOzV10da5oHj2A0cB1mXkQsBiXKzW1Yk37ycAIYBCwNZXlKZ05T6ij0t9LLCBqow0Y\n2uF4CPB6SVlUoojYkkrxcHNmTima3/zw1GLxfWFZ+VRXRwAnRcQcKssax1E5I7FdsVQBnCuaTRvQ\nlplPFMd3UCkonCOa19HAq5nZnpkrgSnA4ThPqGJtc0PpnzstIGrjKWBkcdeEnlQugLq75Eyqs2J9\n+y+BFzLzxx2euhs4p3h8DnBXvbOp/jLzm5k5JDOHU5kT/jszzwYeAk4rujkemkhmvgHMi4g9i6aj\ngOdxjmhmc4FDI6JP8R7y4ZhwnhCsfW64G/jb4m5MhwLvfbjUqV7cibpGIuI4Kr9dbAFuyMzvlxxJ\ndRYRnwUeBZ7h/9e8f4vKdRC3AbtQebM4PTM7XyilzVhEHAn8c2aeEBG7Ujkj0Q+YDnw5M5eXmU/1\nExEHUrmovicwGziXyi/znCOaVET8K3AGlTv5TQf+nsp6dueJJhIRvwGOBPoDbwLfBX5LF3NDUWz+\nlMpdm5YA52bmtLrmtYCQJEmSVC2XMEmSJEmqmgWEJEmSpKpZQEiSJEmqmgWEJEmSpKpZQEiSJEmq\nmgWEJOlTiYjVETGjw1fNdlOOiOER8WytXk+SVHs91t9FkqSPWZqZB5YdQpJUDs9ASJJqIiLmRMQP\nI+LJ4mv3on1YREyNiJnF912K9p0i4s6I+FPxdXjxUi0R8fOIeC4i/jMiepf2Q0mSPsECQpL0afXu\ntITpjA7PvZ+ZB1PZJfXqou2nwE2ZuT9wM3BN0X4N8IfMPAAYDTxXtI8Ers3MfYBFwKkb+eeRJH0K\n7kQtSfpUIuKDzOzbRfscYFxmzo6ILYE3MnOHiHgLGJiZK4v2BZnZPyLagSGZubzDawwHHszMkcXx\nN4AtM/N7G/8nkyRVwzMQkqRayrU8Xlufrizv8Hg1Xq8nSQ3FAkKSVEtndPj+x+LxY8CZxeOzgf8p\nHk8FLgCIiJaI+Ey9QkqSus/f6kiSPq3eETGjw/H9mfnhrVx7RcQTVH5BNaFouxi4ISL+BWgHzi3a\nJwLXR8R5VM40XAAs2OjpJUkbxGsgJEk1UVwDMTYz3yo7iyRp43EJkyRJkqSqeQZCkiRJUtU8AyFJ\nkiSpahYQkiRJkqpmASFJkiSpahYQkiRJkqpmASFJkiSpav8HoAfXQ8djG5QAAAAASUVORK5CYII=\n", 404 | "text/plain": "" 405 | }, 406 | "metadata": {}, 407 | "output_type": "display_data" 408 | } 409 | ] 410 | }, 411 | { 412 | "metadata": { 413 | "_uuid": "9466db27a70b25e1c5ccd0e9c2943ffd45adf77a" 414 | }, 415 | "cell_type": "markdown", 416 | "source": "The AUC score of the test date is around 98%." 417 | }, 418 | { 419 | "metadata": { 420 | "_cell_guid": "b00c7f03-0997-4efe-8fe6-9924d49907af", 421 | "_uuid": "6b60fa0146635fd4214e1f8d66afc4b2cd103295", 422 | "trusted": false 423 | }, 424 | "cell_type": "code", 425 | "source": "roc_auc_score(y_test, y_test_hat) * 100", 426 | "execution_count": 40, 427 | "outputs": [ 428 | { 429 | "data": { 430 | "text/plain": "98.14928488618965" 431 | }, 432 | "execution_count": 40, 433 | "metadata": {}, 434 | "output_type": "execute_result" 435 | } 436 | ] 437 | }, 438 | { 439 | "metadata": { 440 | "_uuid": "473049735f97986c79854b6159d483f18c543355" 441 | }, 442 | "cell_type": "markdown", 443 | "source": "Since we have only 30 features we can easily visualize the influence of each feature in our model." 444 | }, 445 | { 446 | "metadata": { 447 | "_cell_guid": "be0d878a-1908-4080-9c81-e59d818b6025", 448 | "_kg_hide-input": false, 449 | "_kg_hide-output": false, 450 | "_uuid": "12652fe2ad6be38b5104e604781e89a3ff50a16f", 451 | "trusted": false 452 | }, 453 | "cell_type": "code", 454 | "source": "def get_color(c):\n if -0.01 < c and c < 0.0075:\n return 'orange'\n elif c>=0.0075:\n return 'green'\n else:\n return 'red'\n\nplt.figure(figsize=(10,8))\ncolors = [get_color(c) for c in np.sort(weights[:,0])]\nplt.bar(np.arange(30), np.sort(weights[:,0]), width=0.3, color=colors)\nfeature_names = X.columns[np.argsort(weights[:,0])]\nplt.title('Feature influence')\nplt.ylabel('weight')\n_ = plt.xticks(np.arange(30), feature_names, rotation=60, ha='right')", 455 | "execution_count": 42, 456 | "outputs": [ 457 | { 458 | "data": { 459 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAAH/CAYAAAAFTeprAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmYZGV59/HvPTOALAbZZAAZRgUV\nUBxhAFciIGERGCSiiAsgCJoQjcYFCcqiyIgr7kFEEURQUBZBjeAefdUBFcUlLIIgBBBFERMVud8/\nntNSNN1dp7qqu+qZ+X6uq66uOlV113PW/tVzlorMRJIkSaNtzrAbIEmSpO4MbZIkSRUwtEmSJFXA\n0CZJklQBQ5skSVIFDG2SJEkVMLRJUhcR8aGIeEPL164aERdFxO8i4tMRcVBEfHOm2yhp+Tdv2A2Q\ntPyKiOuB9YG/dgx+VGbe3EfNpwNnZubD+mtde5n50h5e/mzKOK+TmfdExEEz0ypJKxp72iTNtL0y\nc42O27QD2yBExEx/Wd0E+O/MvGeGP0fSCsbQJmkoIuKJEfGtiLgzIn7Y9KCNPXdwRPw0Iu6KiOsi\n4vBm+OrA54ENI+IPzW3DiPhYRLy54/1Pj4ibOh5fHxGvi4grgbsjYl7zvvMi4vaI+EVEvHyKtv6t\n/ljtiPi3iLgtIm6JiIOb544D3gg8t2nbIePqLIyI7AyOEfHViDi04/GLm3H/bUR8MSI26XguI+Kl\nEXF18/z7IyI6nn9Jx3T7SURs3QxvPa6SRpehTdKsi4iNgIuBNwNrA68GzouI9ZqX3AbsCfwdcDDw\nrojYOjPvBnYHbp5Gz93zgGcCDwHuBS4CfghsBOwM/GtE7Nqy1nxgzea9hwDvj4i1MvMY4C3AOU3b\nPtKyHgARsQ9wFLAvsB7wDeCT4162J7At8HjgOcCuzXv3A44FXkSZbnsDd0TEnD7HVdKIMLRJmmnn\nN71pd0bE+c2wFwCXZOYlmXlvZn4JWAbsAZCZF2fmtVl8DfhP4Gl9tuM9mXljZv4vJfSsl5nHZ+af\nM/M64MPA/i1r/QU4PjP/kpmXAH8AHt1n+wAOB07MzJ82u1ffAizq7G0DlmbmnZn5S+ArwKJm+KHA\nSZn5vWa6XZOZN9D/uEoaEZ6IIGmm7ZOZl44btgmwX0Ts1TFsJUoIISJ2B44BHkX5crka8KM+23Hj\nuM/fMCLu7Bg2l9Kz1cYd445Z+yOwRp/tG2vXyRHxjo5hQekhu6F5/D+TfO7GwLWT1OxnXCWNCEOb\npGG4ETgjM18y/omIWAU4j7Kb74LM/EvTQzd27FZOUO9uSrAbM3+C13S+70bgF5m52XQa34e7m7+r\nAb9v7ne29UbghMz8xDRq3wg8cpLhwxhXSQPm7lFJw3AmsFdE7BoRcyPiQc0B/g8DVgZWAW4H7ml6\n3f6h4723AutExJodw34A7BERa0fEfOBfu3z+d4HfNycnrNq04bERse3AxnACmXk78CvgBc1nvpj7\nB60PAa+PiC0BImLN5li1Nk4FXh0R20SxabNbdSjjKmnwDG2SZl1m3ggsoRx0fzulN+g1wJzMvAt4\nOfAp4LfAAcCFHe/9GeXg/Oua4+Q2BM6gHGh/PeX4t3O6fP5fgb0ox4P9Avg1JfSsOdX7BuQllHG9\nA9gS+FZHuz4LvBU4OyJ+D/yYcuJFV5n5aeAE4CzgLuB8YO0hj6ukAYrMifY0SJIkaZTY0yZJklQB\nQ5skSVIFDG2SJEkVMLRJkiRVwNAmSZJUgeXu4rrrrrtuLly4cNjNkCRJ6uryyy//dWau1/2Vy2Fo\nW7hwIcuWLRt2MyRJkrqKiBu6v6pw96gkSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJ\nFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmSJFXA0CZJklQBQ5skSVIFDG2SJEkVMLRJkiRV\nwNAmSZJUAUObJElSBQxtkiRJFZg37AZIkiQNWxwXDxiWx+QQWjI5e9okSZIqYGiTJEmqgKFNkiSp\nAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQK\nGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQLzht0ASZKkfsRx8YBh\neUwOoSUza6g9bRGxW0T8PCKuiYgjJ3h+h4i4IiLuiYhnD6ONkiRJo2BooS0i5gLvB3YHtgCeFxFb\njHvZL4GDgLNmt3WSJEmjZZi7R7cDrsnM6wAi4mxgCfCTsRdk5vXNc/cOo4GSJEmjYpi7RzcCbux4\nfFMzTJIkSeMMs6ftgUcNwrSOGoyIw4DDABYsWNBPmyRJ0ixZUU4gGJRh9rTdBGzc8fhhwM3TKZSZ\np2Tm4sxcvN566w2kcZIkSaNkmKHte8BmEfHwiFgZ2B+4cIjtkSRJGllDC22ZeQ9wBPBF4KfApzLz\nqog4PiL2BoiIbSPiJmA/4D8i4qphtVeSJGmYhnpx3cy8BLhk3LA3dtz/HmW3qSRJ0grNn7GSJEmq\ngKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkC\nhjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY\n2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBo\nkyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgKFN\nkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkiowb9gNkCRJ9Ynj4gHD8pgc\nQktWHPa0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRXwRARJklYwnkRQJ3vaJEmSKmBokyRJqoChTZIk\nqQKGNkmSpAp4IoIkSZWY6AQC8CSCFYU9bZIkSRUwtEmSJFXA0CZJklQBQ5skSVIFDG2SJEkVMLRJ\nkiRVwNAmSZJUAUObJElSBQxtkiRJFfAXESRJmgUT/ZqBv2SgXtjTJkmSVAFDmyRJUgUMbZIkSRUw\ntEmSJFXA0CZJklQBQ5skSVIFvOSHJEldeLkOjQJ72iRJkipgaJMkSaqAu0clScutiXZrgrs2VSd7\n2iRJkipgaJMkSaqAoU2SJKkCHtMmSRq8syY4luyAHo8jG0QNaTky1NAWEbsBJwNzgVMzc+m451cB\nPg5sA9wBPDczr5/tdkrSCmOioASGJWkEDG33aETMBd4P7A5sATwvIrYY97JDgN9m5qbAu4C3zm4r\nJUmSRsMwe9q2A67JzOsAIuJsYAnwk47XLAGObe6fC7wvIiIz/conaXSMyq5AdydKy7VhnoiwEXBj\nx+ObmmETviYz7wF+B6wzK62TJEkaITGsTquI2A/YNTMPbR6/ENguM/+l4zVXNa+5qXl8bfOaO8bV\nOgw4DGDBggXb3HDDDbMxAg8c1uu0nKkao9QWx2fi4aPSlmHUgNHpVbJnStKQRcTlmbm4zWuH2dN2\nE7Bxx+OHATdP9pqImAesCfxmfKHMPCUzF2fm4vXWW2+GmitJkjQ8wzym7XvAZhHxcOBXwP7AAeNe\ncyFwIPBt4NnAlz2eTdLA2KsmqSJDC22ZeU9EHAF8kXLJj9My86qIOB5YlpkXAh8BzoiIayg9bPsP\nq72SRoyBS9IKZqjXacvMS4BLxg17Y8f9/wP2m+12SZphBi5J6pk/YyVJklQBQ5skSVIFDG2SJEkV\nMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmSJFXA\n0CZJklQBQ5skSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmSVAFD\nmyRJUgUMbZIkSRUwtEmSJFXA0CZJklQBQ5skSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElSBQxt\nkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgXmDbsBkiqSOewWSNIKy542SZKkChjaJEmSKmBokyRJqoCh\nTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2\nSZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNok\nSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkirQKrRF\nxCvaDJMkSdLMaNvTduAEww4aYDskSZI0hXlTPRkRzwMOAB4eERd2PPVg4I6ZbJgkSZLuM2VoA74F\n3AKsC7yjY/hdwJUz1ShJMyBz2C2QJPVhytCWmTcANwBPmp3mSJIkaSJtT0TYNyKujojfRcTvI+Ku\niPj9TDdOkiRJRbfdo2NOAvbKzJ/OZGMkSZI0sbZnj95qYJMkSRqebmeP7tvcXRYR5wDnA38aez4z\nPzODbZMkSVKj2+7RvTru/xH4h47HCRjaJEmSZkG3s0cPnq2GSJIkaXKtTkSIiPdMMPh3wLLMvGCw\nTZIkSdJ4bU9EeBCwCLi6uW0FrA0cEhHvnqG2SZIkqdH2kh+bAjtl5j0AEfFB4D+BXYAf9fqhEbE2\ncA6wELgeeE5m/naC130BeCLwzczcs9fPkSRJWl607WnbCFi94/HqwIaZ+Vc6zibtwZHAZZm5GXBZ\n83gibwNeOI360vIl84E3SdIKpW1oOwn4QUR8NCI+BnwfeHtErA5cOo3PXQKc3tw/Hdhnohdl5mWU\n3zmVJElaobXaPZqZH4mIS4DtgACOysybm6dfM43PXT8zb2lq3xIRD51GDWn02SMmSRqQbhfXfUxm\n/iwitm4G3dj8nR8R8zPziineeykwf4Kn/n16TZ2ynYcBhwEsWLBg0OUlSZKGrltP26soYegdEzyX\nwE6TvTEznzHZcxFxa0Rs0PSybQDc1qaxU3zWKcApAIsXL7ZrQ5IkLXe6XVz3sObvjgP+3AuBA4Gl\nzV+v9SZJkjSFViciRMRqEXF0RJzSPN4sIvq5BMdSYJeIuJpy2ZClTd3FEXFqx+d+A/g0sHNE3BQR\nu/bxmZIkSdVqe522jwKXA09uHt9ECVOfm86HZuYdwM4TDF8GHNrx+GnTqS9JkrS8aXvJj0dm5knA\nXwAy838pZ5FKkiRpFrQNbX+OiFUpJx8QEY9kehfVlSRJ0jS03T16DPAFYOOI+ATwFOCgmWqUJEmS\n7q9taHsRcDFwLnAd8IrM/PWMtUqSJEn308uJCE+lnOn5CMpPWn09M0+esZZJkiTpb9r+jNWXI+Jr\nwLbAjsBLgS0BQ5skSdIsaBXaIuIyYHXg28A3gG0zs69fMZAkSVJ7bc8evRL4M/BYYCvgsc3ZpJIk\nSZoFbXePvhIgItYADqYc4zYfWGXmmiZJkqQxbXePHgE8DdgGuAE4jbKbVJIkSbOg7dmjqwLvBC7P\nzHtmsD2SJEmaQNvdo2+b6YZIIydz2C2QJOlv2p6IIEmSpCEytEmSJFXA0CZJklQBQ5skSVIFDG2S\nJEkVMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmS\nJFXA0CZJklQBQ5skSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmS\nVAFDmyRJUgUMbZIkSRUwtEmSJFXA0CZJklQBQ5skSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElS\nBQxtkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmSJFXA0CZJklQBQ5skSVIFDG2SJEkV\nMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmSJFVg\n3rAbIM2IzGG3QJKkgbKnTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKeMkPjRYv\n1SFJ0oTsaZMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgGePanA881OSpBljT5skSVIFDG2S\nJEkVMLRJkiRVYCihLSLWjogvRcTVzd+1JnjNooj4dkRcFRFXRsRzh9FWSZKkUTCsnrYjgcsyczPg\nsubxeH8EXpSZWwK7Ae+OiIfMYhslSZJGxrBC2xLg9Ob+6cA+41+Qmf+dmVc3928GbgPWm7UWSpIk\njZBhhbb1M/MWgObvQ6d6cURsB6wMXDvJ84dFxLKIWHb77bcPvLGSJEnDNmPXaYuIS4H5Ezz17z3W\n2QA4AzgwM++d6DWZeQpwCsDixYu9WJgkSVruzFhoy8xnTPZcRNwaERtk5i1NKLttktf9HXAxcHRm\n/r8ZaqokSdLIG9bu0QuBA5v7BwIXjH9BRKwMfBb4eGZ+ehbbJkmSNHKGFdqWArtExNXALs1jImJx\nRJzavOY5wA7AQRHxg+a2aDjNlSRJGq6h/PZoZt4B7DzB8GXAoc39M4EzZ7lpkiRJI8kfjFfhj71L\nkjTS/BkrSZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJ\nkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpArMG3YD\n1KfMYbdAkiTNAnvaJEmSKmBokyRJqoChTZIkqQKGNkmSpAp4IsIweRKBJElqyZ42SZKkChjaJEmS\nKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmq\ngKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkC\nhjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY\n2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkChjaJEmSKmBo\nkyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJkqQKGNokSZIqYGiTJEmqgKFN\nkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkiowlNAWEWtHxJci4urm71oT\nvGaTiLg8In4QEVdFxEuH0VZJkqRRMKyetiOByzJzM+Cy5vF4twBPzsxFwPbAkRGx4Sy2UZIkaWQM\nK7QtAU5v7p8O7DP+BZn558z8U/NwFdyVK0mSVmDDCkLrZ+YtAM3fh070oojYOCKuBG4E3pqZN89i\nGyVJkkbGvJkqHBGXAvMneOrf29bIzBuBrZrdoudHxLmZeesEn3UYcBjAggULptliSZKk0TVjoS0z\nnzHZcxFxa0RskJm3RMQGwG1dat0cEVcBTwPOneD5U4BTABYvXpz9tVySJGn0DGv36IXAgc39A4EL\nxr8gIh4WEas299cCngL8fNZaKEmSNEKGFdqWArtExNXALs1jImJxRJzavGZz4DsR8UPga8DbM/NH\nQ2mtJEnSkM3Y7tGpZOYdwM4TDF8GHNrc/xKw1Sw3TZIkaSR5GQ1JkqQKGNokSZIqYGiTJEmqgKFN\nkiSpAoY2SZKkChjaJEmSKmBokyRJqoChTZIkqQKGNkmSpAoY2iRJkipgaJMkSaqAoU2SJKkChjZJ\nkqQKGNokSZIqYGiTJEmqwLxhN6BamcNugSRJWoHY0yZJklQBQ5skSVIFDG2SJEkVMLRJkiRVwNAm\nSZJUAUObJElSBQxtkiRJFTC0SZIkVcDQJkmSVAFDmyRJUgUMbZIkSRUwtEmSJFXA0CZJklQBQ5sk\nSVIFDG2SJEkVMLRJkiRVwNAmSZJUAUObJElSBQxtkiRJFYjMHHYbBioibgdumMWPXBf49YjUGZUa\no9QWx2e02zIqNUapLY7PaLfF8RnttoxKjV5skpnrtXnhchfaZltELMvMxaNQZ1RqjFJbHJ/Rbsuo\n1Biltjg+o90Wx2e02zIqNWaKu0clSZIqYGiTJEmqgKGtf6eMUJ1RqTGoOqNSY1B1RqXGoOosTzUG\nVWdUagyqzqjUGFSdUakxqDqjUmNQdZanGjPCY9okSZIqYE+bJElSBQxtU4iIGHYblldOW/XC5eU+\nTouZ47SdOU7bwTC0TSH72Hc8toBGxLx+2jAqC/r4dvTbrn6m7Sgalfm0vJrO8jJK6+Agl49+153l\nbVkd5LZplLZLHctvz+MzyGkyqOUlMzMizBx9cgJOICIeERGnRMSGzePpTKfVATLznoiY08fCOqdp\nw/rTfD8RsXJEPKq5P3eaZeY2758Pf1sBo9cVOiI2iogDImK1ccNb14mI7SLi0IjYvJfPHldjtYjY\nICK2mW6Npk7AQP6RDnxdnObGfm6/bYmIdSJiQZ81xpb7J0XEyyLiiGmUGdQ6OK+ZLgun+f6B1BjE\nutMYxDZlEPN4IOsgA9g2DWibP2gPioi1p7ltGZsm68O0p8lzI+Io4JkTPNdLnRdFxD4RMS8z722G\njcL0rZITbmJvBZ4K7AgwtqD16OyIuCIitsnMezPz3mkGpuMi4gzgmIjYD6a1kT4Z+FhErJKZf51G\nGwBOioiPA2+JiDMiYlE2emzPacC6mflHuC9Etv0W1mzgPwQ8BfiviFjUDF+px/E5BVhKmb5PHfcZ\nvawXx0XE3/f42fcTEXP63ZhFxJYR8ZyI2KK5xTQ39h8ADpxoWe1hPp8OTDhNWs7jOc36shA4FbgV\neE9E7NDy88cMah18O2WZOy4ith3f1lms0de602EQ25S+5nFjUOvgILZNfW/zI+LxHff76dl6SnP3\nLcADvqy0rH1CRLwHODoiPhIRm49Nk5Zt2BY4ClgTODEito+INSNiI2j/JTUiNgDeB7wQWBoROzVP\nrdE832o+R8TmTfjbtM3rJ6kxkPkzdJnpreMG7AZcBjwD+BVwDLBSjzVWBd4LXEP5KYwLmuE7A6sA\nq7SsswT4JvB04FDgPGDDHtvyeOAHwAXAp4GHN8Pn9FBjL2AZsBWwNXAkcCVwErBaD3W2A77T8fjF\nzXT6OLB2yxqfBQ5t7r+BsuH/EvBaYOOWNfYGvg7Mb6brCcDhlH8ga/Y4bS8H1mgez2tureZv857H\nAp8BnjVueC/zZzvge5TAdQZwIfAy4JG91AIWAVd1jM+GwObAE3poyx7A1zsePwV4PvBSYPUel91T\ngBcAi4EvNMO2aIZNOU4DXAf3BL7RzOsTm+XsacCLxqbTLNXoe91p3jeIbUrf83iA62Df2yYGs83f\nD7gXOBPYtGN46/W4ef3KwKuA/wFuGhsHYOUeauwD/Bdl27II+CrwM+D4Hpb7U4EXNfdf1ywnX2um\n69Nb1hi7OsVRwDub+ftB4NimXttlZRvgu8C5wI+B1/QyTQc5f0bhZk/bA50AvDUzL6VsWBYCT+yl\nQGb+b1PnbZR/fL+MiLso4WLVzPxTy1IvB07IzK9m5qmUfz7PG3sy2u0ePBY4KTOXAL+hrNBkb98k\nNwQuy8wrgSszcymwL/Bg4LAe6vwfcAtARLyM8s/3AuAvwNeab2WTioidKRudU5tB/wR8i7KRXQQc\n3/Kb2yLgY5n5P8BDKBv+BNYBPhERa7Ycn6XA0sz8Q0TsSNmgLQNeObabpYV3U4LesyLiwxGxNdw3\nfyJi5RY1DgVOzsx/Ao6mBJat6X1eb0cJN3+IiL0oG+6lwMsi4uCWNV5BmSdExOHAGyn/1B8PvKnH\nHpQrgOuAd1DWJygb3yd1G6cBroNHACdm5g+B3wEHUXYXPYnSs9Om524QNfpadzoMYpsyiHk8qHVw\nENumN9PnNh/YlTJtrwe+FBEnRMSDO9bj7dsUycw/Z+Y7gbOB3wKfj4i9MvPPUXavnxQRq3Qp81jg\nI5n548z8AfAJ4KPAWpSwPqWIeAIl4DwpIjYBXk35MvjPwF3AIS23S49u5uPFwEMpXxY+SvnisCXw\n7IhYq0Wdo4EPZeazKcv9HhHR6nc6Owxk/oyEYafGUbkBQVnR/6V5PJfyjfy1wNXALj3UmgusRFlA\nD26GXQr8N/B7YLcWNValLGRbcd83lt0pGygo3c3v7FLjOcCXOx4/DbgWeA899HoAmwJfAZ7bMWwl\nyj/5zwALe6i1FHgJ8Epgccfw9wI7dnnvUcDhzf01gReMm38XAuu3aMPuwA2UQHsXTU8S5Vvux4Dt\nW9TYkbJhuwjYGPg88I/ALpSN5Gl07w16MuXb49aUDe3rm3pvaZ5/NPC8LjUe13zW8TQ9FM3jVzRt\nekPL+fJwYAdKT+Z8yj+NHSj/FPen7A6bsleoWV9eB3ykac/V3Nfb91jgUz0uK8+ghLZvdkyvK9vU\noAThsfnZ8zrYLE+rjU3/ZtwuovmWTvmnczZNz/UUdVYF9u+nRkd73trHuhNNW15B2abM6VgXetmm\nPKhZTk8D3jTdeUwJrTcAxzHNdbB5/aMo26b9O4a12jY10+TvgFc0j6e1zW+WtZ2B+R1tuoDSa70X\nJaSf06LOnI77qzd/nw/8BPhkM13ObFFnT0o4WdLM8683w/Zqlrcpt/2UL8HPp4TZTwKXjnv+68AW\nLdqxFDikuX90U3ML4BeUQP1BuvTiU3pB/5Pyv3ns/+BZwOua+4uBnVrMn51o/jdMd/6Mym3oDRjV\n27gV6CXNwrt5j+97OKX35SDgB82w/botqOPqzeu4v0qz0m0OfJvS4zDVe/8DeG1nuyjfIk8Fdu9l\nOlA2sj+jdGs/uuP5HwFbth0Pysb0CuAO4Phm2OrAD4HtpmoH5Z/Kj2i67cc9fyjw+R6m677NRuQN\nndOxadvWLd7/KuBASki5lyZYjI1rs1F4aJcarwde3PF4LUqwfjvlH84fged3qXF08/rjmvl9GvDV\n5rkNKceTdN0F0CynL2rG6wLRyCd+AAAMkklEQVRKj9TaHc9/E9im5bRdROlBPHbc8MunWoeAuc3f\ndYEFzXKxGWXXyNcouzaOaDuPm1qP6Hcd7Kj1sAnGp+s/r0HVaNad7/e67kxQZ9rblHHz+ORe5/G4\n1+5L6TmZ1jrY8fo9Kdumc5nGtmn88tfcb73N73jPSuMe79KMy73AU7u8Nzrun0jpJd6T8sVhLiVs\n/zPtdym+kPIF5TzKnpax4V8HNpjifXMo27XvA/9CCbZvaubR44FD6OgI6FLn+ZTdmftTwuOxlEN1\nxkJym/+nS2k6UjqGbQ+c3dz/NvCPMz1/Ruk29AaMyo37h6054/6uTvkG+bIeaoyFlFdSvknuP822\nzB3XltcCdwJv61ajWVmupCPkNBuAw4HbgF3btqNjOhzbvPcsSrBo883vAaGBEg6upfQSnAm8r+W0\neSalu/3vO8ZnPvBzYKtptGMvyjeuT1LC7Bkt2jAXOJiyK3QHyrfAzTqeP5TmGKwu8+fAZv4c2DF8\npebvmcCXWtR4AfBTyu6LfSm9q2M9OUcAF7eZP5SN/BWU45L2Bb5D2cA/h9LjO+WGepJpu2bH/TdT\ndtl0a8uqlH+2p1HC44HN8K49qM3rNqeEz87jVl41jXVwc0rQ23SS57uOz0Tr8TRqjI1P5/L16l7X\nnYnGh/u2UW23KZs3y31nW9aaxvgcNK7G3tNYBzdv1p/NOpab4+lh2zTJNBnb3rbd5o+1437Tlfu2\n128BPt1ifDq375dQelU/TAlwO9Li+LxJ1sHO+bMU+FTL5X8PynZ2e2ATSkC/lNLr1UsQfibwRcpu\n2ZcB5/bw3rHg97f/YZQQuTJlL9Y7gLPaTpOOeTu31/kzSrehN2AUbrQ4GLxZiSf9ljNFjTWAZwyg\nLWMb2LHdclP24nS8734hp2P4AZ0bzrbtaJ57cPP+LbptTLrUWaMZn/WBB7UcnzmUIHwhTa8F5Vv/\nfn204xHNRml/4ME9zKu9m43Ydh1tW5eya+Xx050/zUbl57TsbaB8YzwPeFpHOzaiHFS9qIfxeSZw\nPmWD/UjKP6MLKKFnwvDSZZkd252xCeUYqK7LLGUj/a5mGduFsqv1XO4Lb5P2GvLAA5bHdqGs3uM6\nOOWBz8BjKIF90iA5xTSZ00ONznZcBRw1nXVnsvHpaEvXbcoEbXn9uOcXdpvHE9R43XTWwQnqHNnx\nXKttU7d53LHcTLXN77acrEY5MWLdlsvdmpT1eOzkg60oPVxnAYd1ee9ky9tYUJlP6XVv25a53Led\nfWizvK3X9v2dy3tT51Pcv+d+Xg81xraRO3QM+0SzzE4aIFusgz3Nn1G5Db0Bo3CjfIO4kHIm1ocZ\n98+yYyWKPmq0PYNvyjrNa1oFtrHPZVzImWY7thn3fHT+7WO6zJ2qDmUX897A3uOGv4ayy+wRg5qu\n01huxqbtRdwXIDcdv5GYzvyhZehrXjuPEqw+N67GE3scn7lNnc/QwxmFLZfZSc8I7NiIrkHpOTyk\noz0bUnbJvLlFOz5Ls7uZEuS/Aqw3jfnatQ5dzthsOU3WmkY7HhDyWqyDbcZnyp7MljUeMo0arbdl\ng6zTbXy470vyVNv8NtOkl7M+l1BODPkA5WSZseF70GVXYsvlbdJwz+Tb2ddRtrNdj7vsUuf1TZ1H\nt6kz7r0TfVF/LM3xzX1Ok57OEh6F29AbMOwbgzkYfLIaJ1K6cx/TrUbLtmxBx8H3k9ToFnIm7V3r\noR2bD2h8utZpxuVsyjetayi9MB+k7Kb5LPCBAU3XA1rU6TZtHzmAGo+ajRpd6oxtqNscc9Jt2j6G\nLsflddQ6jXKpguuAZ3HfiRVdL6PC5AcsH9nc73rAcpc6Y71229Kl124Q06RlOwYxPtvR5YD7AU2T\nbjW2B3Ye0Ph0OzB9EAe3t6nRZnzmjnu8BDinWX6m3Hsw4OVtqu3s+cD7W7al2/a6za78btu3RzeP\npwrUbaZJ123+KN685Ee5QOQlmXlFZv6YcgHMk4CVI+IzlAMyu02nyWqsROnuvqJFjTZtWUY5LX4q\nj6PsHjg8Iq6JiHdFxAcp3+B+Qzmgtd92XD6g8elaJzMvpGxwnknZdfY9yvQ8gHKpgCcNoB3LKOG6\nm27T9lUDqPHyWaoxVZ3bmjr/3KJGt2l7xVRvHrvcRUQ8mRLSnkI58PlllIum7kS5lme3S3Q8Hbgo\nM+/KZqtM2d22qLn/XsqJHt1MVucJzf33UHZjTaWvadJDOwYxPidTzqDsty3dpkm3Gu+mrM/ddKtz\nMt2nS7cabZaVNjW6jk82FzuPiPMiYt/MvIDSs/x54PnNxYK7XeKj7+Wty3Z2TUoI6qrF9vopU7x9\nzGz8D7uCdtv80TPs1DjMG4M7GLyvGoOs07x2bJfj9pSF/yWUswG/Any/tvGZ4jPWoftlKAbajn6m\n7ajVGJVlpXntx+k4AJ3Sg3EC8GW6HzfZ9wHLg6oziGkyguMz9Bqj1JZBjc/Y+5q/h1Mupvtl7juR\naDO694KOxHZ20HUYke3SKN6G3oBRuDGYg8H7rjHIOpPU7mWlGdnxYRpXsZ7J6drrtB31GrO1rNBc\n6oJy7a9XNq8/nY7LVwDrTGMe93TA8kzUGfA2ZZTGZ6g1RqktgxqfcTWXUi6k/AEmOON4Jpe3CWoO\n5NcCBlWnqTXr/8NG7Tb0BozCjcEcDN53jUHWGV9zGNNkpsZnWPN4UNN2VGvM5rJCuQr/55qN6CbN\nsIWU685dSPnZm1aX+ZiqHbQ4YHkm6szUNmWUxmcYNUapLQOqsQvlOLYNO4Y9jxLcpjx+edDL2yjf\nZmu7VMNt6A0Y0gJQw8HgPdUZhWmyPI6Pt5mZts3rdqAcG3Qv5XigsbNIn0y5+vuT+2xH1wOWB1Vn\nlrYpozQ+s1JjlNoyqPHpeN9cykWxz6Gc/POkZviOwKv7nCYr3PZtRZkmY/vTVygRsTdlP/mDKWeH\nXkTZRfNdykz/VWYeMdM1BlmnX46P2up32kbEUZSfKVrSPN6I8uPw21GuRfbhAbXjlsz8p4iInGJD\nN4g6s7RNGaXxmZUao9SWQY3PBHUfTjnpZx3KRaCXAAdl5lf6mCYr3PZtRZkmK2Rog3LWWmb+tfmh\n2EdSLqT4HMrlBR6SmU+YssCAagyyTr8cH7U13WkbEetTfnbpfMrxJj/MzF82z+1EOctrDuXXLf44\nU+2YiTqjUmOU2uL4TPr+bSjHXG0N/Fdmvi0itqZcBPevmfnF2Rif5c2KME1W2NA2mYhYB/hTZv5h\nmDUGWadfjo/a6jZtI+I0yvE6yyiXR7iNclmAb2Tm3c1rdpyql2EQ7ZjNOqNSY5TasqKPT0R8mXLx\n1+9RLhX0MMrPrF013TZMty0rguVpmhjaGhExJzPvHXaNQdYZlXYsb+OjB2ozbSNiIXBOZm7fPN6J\nclzbQ4Drge9m5rdmuh2zVWdUaoxSWxwfiIinA8dn5g4dw44E7s7M985mW5Z3y+M08eK6jUHM2EEt\nHKOykDk+aqvNtM3M64HdOx5/mXJNq29SdgsdHBFbznQ7ZqvOqNQYpbY4PkC5qPhNEfG4jmE/AvYc\nQluWa8vjNJk37AZIWnFk5m/G7jcHbN8FnBsRyygnJ/S9e0gaZZl5V0R8G/hqRPwH5YKxr6P8PqY0\nJXePShqaXs+0k2rTcXD8xpTLeaxJ+R3Oe4HjgTuBmzLz3UNspiphaJMkaYZFxBeAPwBXAdsA3wA+\nmJm/H2rDVBWPaZMkaQZERDR/16WcaPNs4ETgTZTrsn0xInYbYhNVGY9pkyRpBnTs+l8CbBcRewIX\nZ+Z3IuIa4KuUE3GkVtw9KknSgI1dbiIi/g3YFbibcrHX84DzM/PWoTZQVXL3qCRJA9YEttWBxcC+\nmfksym/s7gacFhG7DLWBqpKhTZKkmbEL8A/AsU3P28VNePsK5ZdBpJ64e1SSpAGJiDnA3Mz8S0Ss\nBOwD7EX58fIzMvOioTZQVTO0SZI0ABHxUOA7wOeAdYGzKNdhW5VyfbaDgZ8Dr83MvwyrnaqXZ49K\nkjQYDwJup+z6/AxwEHAtcADwXWBt4K8GNk2Xx7RJkjQAmflLYCnwXOBPmfmPlM6R7wJnAx8D3jC0\nBqp69rRJkjQgmfmZiPgtsE9E3A48E9gjM6+NiJXsZVM/7GmTJGlAml9B+C5wF3Ap8O0msM0xsKlf\n9rRJkjQgza8g3A0cHRF3Ao+LiAXAjcNtmZYH9rRJkjQz3kc5OWH39FINGgAv+SFJ0gxprtW2embe\nOey2qH6GNkmSpAq4e1SSJKkChjZJkqQKGNokSZIqYGiTJEmqgKFNkiSpAoY2SZKkCvx/WCG9Am7G\nn54AAAAASUVORK5CYII=\n", 460 | "text/plain": "" 461 | }, 462 | "metadata": {}, 463 | "output_type": "display_data" 464 | } 465 | ] 466 | }, 467 | { 468 | "metadata": { 469 | "_cell_guid": "3bcda3b1-602f-47d5-aaa5-e188975a9fc3", 470 | "_uuid": "900de6c28b610bf5eabef61d73b9eb2eb7b0b7f3", 471 | "collapsed": true 472 | }, 473 | "cell_type": "markdown", 474 | "source": "So thats it for today. I hoped you enjoyed reading the post. Please download or fork the notebook on GitHub and play with the code and change the parameters.\n\nIn the next post we will add a hidden layer to our model and build a neural network. And we will see how to use the High-Level API to build the same model much easier. If you want to learn more about gradient descent and optimization have a look into the following links\n\n* Some lecture note of the Unversity of Toronto: https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf\n* And of course Wikipedia.\n\nSo long.\n\n\n" 475 | }, 476 | { 477 | "metadata": { 478 | "collapsed": true, 479 | "trusted": false, 480 | "_uuid": "515e75c3bebb06319edae6403d75d08cd768eeb0" 481 | }, 482 | "cell_type": "code", 483 | "source": "", 484 | "execution_count": null, 485 | "outputs": [] 486 | } 487 | ], 488 | "metadata": { 489 | "kernelspec": { 490 | "display_name": "Python 3", 491 | "language": "python", 492 | "name": "python3" 493 | }, 494 | "language_info": { 495 | "name": "python", 496 | "version": "3.6.5", 497 | "mimetype": "text/x-python", 498 | "codemirror_mode": { 499 | "name": "ipython", 500 | "version": 3 501 | }, 502 | "pygments_lexer": "ipython3", 503 | "nbconvert_exporter": "python", 504 | "file_extension": ".py" 505 | } 506 | }, 507 | "nbformat": 4, 508 | "nbformat_minor": 1 509 | } -------------------------------------------------------------------------------- /DeepHedging/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/DeepHedging/.DS_Store -------------------------------------------------------------------------------- /DeepHedging/rnn_050.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/DeepHedging/rnn_050.zip -------------------------------------------------------------------------------- /DeepHedging/rnn_099.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/DeepHedging/rnn_099.zip -------------------------------------------------------------------------------- /ExoticsDerviatePricingWithTensorFlow/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/ExoticsDerviatePricingWithTensorFlow/.DS_Store -------------------------------------------------------------------------------- /ExoticsDerviatePricingWithTensorFlow/BermudanTensorFlow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Bermudan Options in TensorFlow\n", 8 | "\n", 9 | "By Matthias Groncki, 2018\n", 10 | "\n", 11 | "This code was written to demonstrate how to implement the Longstaff-Schwartz method in TensorFlow\n" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 180, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "import pandas as pd\n", 22 | "import tensorflow as tf\n", 23 | "import datetime as dt\n", 24 | "import scipy.stats as stats\n", 25 | "from sklearn.linear_model import LinearRegression\n", 26 | "from sklearn.metrics import mean_squared_error\n", 27 | "import matplotlib.pyplot as plt" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "### Helper function" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 184, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "def get_continuation_function():\n", 44 | " \"\"\"\n", 45 | " Create the graph to solve the longstaff schwartz regression \n", 46 | " to calculate the continuation value given a state via a\n", 47 | " stochastic gradient optimization\n", 48 | " \"\"\"\n", 49 | " X = tf.placeholder(tf.float32, (None, 3))\n", 50 | " y = tf.placeholder(tf.float32, (None, 1))\n", 51 | " w = tf.Variable(initial_value=tf.random_normal((3,1))*0.1)\n", 52 | " b = tf.Variable(initial_value=tf.ones(1)*1)\n", 53 | " y_hat = tf.add(tf.matmul(X,w), b)\n", 54 | " pre_error = tf.pow(y-y_hat,2)\n", 55 | " error = tf.reduce_mean(pre_error)\n", 56 | " train = tf.train.AdamOptimizer(0.1).minimize(error)\n", 57 | " return(X, y, train, w, b, y_hat)\n", 58 | "\n", 59 | "def feature_matrix_from_current_state(state):\n", 60 | " \"\"\"\n", 61 | " calculate the first 4 chebyshev polynominals of the current state\n", 62 | " and returns a (n x 4) feature maxtix with n numbers of simulated paths\n", 63 | " \n", 64 | " input: state : n-dim vector with the state of the process at exercise time t\n", 65 | " \n", 66 | " output: feature matrix for the longstaff schwartz regression\n", 67 | " \"\"\"\n", 68 | " feature_0 = tf.pow(state,0)\n", 69 | "\n", 70 | " feature_1 = tf.pow(state,1)\n", 71 | " feature_1_mean = tf.reduce_mean(feature_1)\n", 72 | " feature_1_std = tf.sqrt(tf.reduce_sum(tf.square(feature_1 - feature_1_mean))/(N_samples_pricing+1))\n", 73 | " feature_1_norm = (feature_1 - feature_1_mean) / feature_1_std\n", 74 | "\n", 75 | "\n", 76 | " feature_2 = 2*tf.pow(state,2)-1\n", 77 | " feature_2_mean = tf.reduce_mean(feature_2)\n", 78 | " feature_2_std = tf.sqrt(tf.reduce_sum(tf.square(feature_2 - feature_2_mean))/(N_samples_pricing+1))\n", 79 | " feature_2_norm = (feature_2 - feature_2_mean) / feature_2_std\n", 80 | "\n", 81 | " feature_3 = 4*tf.pow(state,3)-3*feature_1\n", 82 | " feature_3_mean = tf.reduce_mean(feature_3)\n", 83 | " feature_3_std = tf.sqrt(tf.reduce_sum(tf.square(feature_3 - feature_3_mean))/(N_samples_pricing+1))\n", 84 | " feature_3_norm = (feature_3 - feature_3_mean) / feature_3_std\n", 85 | "\n", 86 | " features = tf.concat([feature_1_norm, feature_2_norm, feature_3_norm], axis=0)\n", 87 | " features = tf.reshape(features, shape=(3, N_samples_pricing))\n", 88 | " features = tf.transpose(features)\n", 89 | " return features\n" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 185, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "def pricing_function(number_call_dates):\n", 99 | " \"\"\"\n", 100 | " Create the computational graph for the pricing and generates for each call date a \n", 101 | " linear model including the training operator\n", 102 | " \"\"\"\n", 103 | " S = tf.placeholder(tf.float32)\n", 104 | " # First excerise date\n", 105 | " dts = tf.placeholder(tf.float32)\n", 106 | " # 2nd exersice date\n", 107 | " K = tf.placeholder(tf.float32)\n", 108 | " r = tf.placeholder(tf.float32)\n", 109 | " sigma = tf.placeholder(tf.float32)\n", 110 | " dW = tf.placeholder(tf.float32)\n", 111 | " \n", 112 | " S_t = S * tf.cumprod(tf.exp((r-sigma**2/2)*dts + sigma*tf.sqrt(dts)*dW), axis=1)\n", 113 | " \n", 114 | " E_t = tf.exp(-r*tf.cumsum(dts))*tf.maximum(S_t-K, 0)\n", 115 | "\n", 116 | " \n", 117 | " continuationValues = []\n", 118 | " training_functions = []\n", 119 | " exercises = []\n", 120 | " \n", 121 | " previous_exersies = 0\n", 122 | " npv = 0\n", 123 | " for i in range(number_call_dates-1):\n", 124 | " (input_x, input_y, train, w, b, y_hat) = get_continuation_function()\n", 125 | " training_functions.append((input_x, input_y, train, w, b, y_hat))\n", 126 | " X = feature_matrix_from_current_state(S_t[:, i])\n", 127 | " contValue = tf.add(tf.matmul(X, w),b)\n", 128 | " continuationValues.append(contValue)\n", 129 | " inMoney = tf.cast(tf.greater(E_t[:,i], 0.), tf.float32)\n", 130 | " exercise = tf.cast(tf.greater(E_t[:,i], contValue[:,0]), tf.float32) * inMoney \n", 131 | " exercises.append(exercise)\n", 132 | " exercise = exercise * (1-previous_exersies)\n", 133 | " previous_exersies += exercise\n", 134 | " npv += exercise*E_t[:,i]\n", 135 | " \n", 136 | " # Last exercise date\n", 137 | " inMoney = tf.cast(tf.greater(E_t[:,-1], 0.), tf.float32)\n", 138 | " exercise = inMoney * (1-previous_exersies)\n", 139 | " npv += exercise*E_t[:,-1]\n", 140 | " npv = tf.reduce_mean(npv)\n", 141 | " greeks = tf.gradients(npv, [S, r, sigma])\n", 142 | " return([S, dts, K, r, sigma, dW, S_t, E_t, npv, greeks, training_functions])" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "### Pricing function" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "def bermudanMC_tensorFlow(S_0, strike, exTimes, impliedvol, riskfree_r, random_train, random_pricing):\n", 159 | " \"\"\"\n", 160 | " Calculate the npv of a bermudan call option\n", 161 | " \n", 162 | " input:\n", 163 | " \n", 164 | " S_0 : Spot at time 0\n", 165 | " strike: Strike of the Call\n", 166 | " exTimes: Exercise times as list\n", 167 | " impliedvol: implied vol of the BlackScholes process\n", 168 | " riskfree_r: risk free interest rate\n", 169 | " random_train: n1 x len(exTimes) N(0,1) distributed random numbers to generate n1 training paths\n", 170 | " random_train: n2 x len(exTimes) N(0,1) distributed random numbers to generate n2 pricing paths\n", 171 | " \n", 172 | " returns:\n", 173 | " \n", 174 | " npv, (delta, rho, vega)\n", 175 | " \"\"\"\n", 176 | " n_excerises = len(exTimes)\n", 177 | " with tf.Session() as sess:\n", 178 | " \n", 179 | " S, dts, K, r, sigma, dW, S_t, E_t, npv, greeks, training_functions = pricing_function(n_excerises)\n", 180 | " sess.run(tf.global_variables_initializer())\n", 181 | " paths, exercise_values = sess.run([S_t, E_t], {\n", 182 | " S: S_0,\n", 183 | " dts : exTimes,\n", 184 | " K : strike,\n", 185 | " r : riskfree_r,\n", 186 | " sigma: impliedvol,\n", 187 | " dW : random_train\n", 188 | " }) \n", 189 | " \n", 190 | " # Backward iteration to learn the continuation value approximation for each call date\n", 191 | " for i in range(n_excerises-1)[::-1]:\n", 192 | " (input_x, input_y, train, w, b, y_hat) = training_functions[i]\n", 193 | " y = exercise_values[:, i+1:i+2]\n", 194 | " X = paths[:, i]\n", 195 | " X = np.c_[X**1, 2*X**2-1, 4*X**3 - 3 * X]\n", 196 | " X = (X - np.mean(X, axis=0)) / np.std(X, axis=0)\n", 197 | " for epoch in range(80):\n", 198 | " _ = sess.run(train, {input_x:X[exercise_values[:,i]>0], \n", 199 | " input_y:y[exercise_values[:,i]>0]})\n", 200 | " cont_value = sess.run(y_hat, {input_x:X, \n", 201 | " input_y:y})\n", 202 | " exercise_values[:, i:i+1] = np.maximum(exercise_values[:, i:i+1], cont_value)\n", 203 | " plt.figure(figsize=(10,10))\n", 204 | " plt.scatter(paths[:,i], y)\n", 205 | " plt.scatter(paths[:,i], cont_value, color='red')\n", 206 | " plt.title('Continuation Value approx')\n", 207 | " plt.ylabel('NPV t_%i'%i)\n", 208 | " plt.xlabel('S_%i'%i)\n", 209 | " \n", 210 | " \n", 211 | " # Forward simulation and applying the learned approximation\n", 212 | " npv, greeks = sess.run([npv, greeks], { S: S_0,\n", 213 | " dts : exTimes,\n", 214 | " K : strike,\n", 215 | " r : riskfree_r,\n", 216 | " sigma: impliedvol,\n", 217 | " dW : random_pricing\n", 218 | " })\n", 219 | " return(npv, greeks)\n", 220 | " \n" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "### Example Pricing" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 187, 233 | "metadata": {}, 234 | "outputs": [], 235 | "source": [ 236 | "N_samples_learn = 10000\n", 237 | "N_samples_pricing = 100000\n", 238 | "calldates = 4\n", 239 | "np.random.seed(42)\n", 240 | "# Training paths\n", 241 | "N = np.random.randn(N_samples_learn,calldates)\n", 242 | "# Pricing paths\n", 243 | "N_pricing = np.random.randn(N_samples_pricing,calldates)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 188, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "npv, greeks = bermudanMC_tensorFlow(100., 125., [1., 1., 1., 1.], 0.2, 0.03, N, N_pricing)" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": 189, 258 | "metadata": {}, 259 | "outputs": [ 260 | { 261 | "data": { 262 | "text/plain": [ 263 | "11.111191" 264 | ] 265 | }, 266 | "execution_count": 189, 267 | "metadata": {}, 268 | "output_type": "execute_result" 269 | } 270 | ], 271 | "source": [ 272 | "npv" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": 190, 278 | "metadata": {}, 279 | "outputs": [ 280 | { 281 | "data": { 282 | "text/plain": [ 283 | "[0.48537952, 129.15599, 79.27229]" 284 | ] 285 | }, 286 | "execution_count": 190, 287 | "metadata": {}, 288 | "output_type": "execute_result" 289 | } 290 | ], 291 | "source": [ 292 | "greeks" 293 | ] 294 | }, 295 | { 296 | "cell_type": "markdown", 297 | "metadata": {}, 298 | "source": [ 299 | "### Runtime vs number of exercise dates" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 206, 305 | "metadata": {}, 306 | "outputs": [], 307 | "source": [ 308 | "runtime = {}\n", 309 | "for t in [[1.,1.], [1.,1.,1.], [1.,1.,1., 1.], [1.,1.,1., 1., 1,], [1.,1.,1., 1., 1, 1,]]:\n", 310 | " calldates = len(t)\n", 311 | " np.random.seed(42)\n", 312 | " # Training paths\n", 313 | " N = np.random.randn(N_samples_learn,calldates)\n", 314 | " # Pricing paths\n", 315 | " N_pricing = np.random.randn(N_samples_pricing,calldates)\n", 316 | " start = dt.datetime.now()\n", 317 | " bermudanMC_tensorFlow(100., 125., t, 0.2, 0.03, N, N_pricing)\n", 318 | " runtime[len(t)] = \"%s\" % (dt.datetime.now()-start)" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 207, 324 | "metadata": {}, 325 | "outputs": [ 326 | { 327 | "data": { 328 | "text/html": [ 329 | "
\n", 330 | "\n", 343 | "\n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | "
RunTime
20:00:06.474073
30:00:07.971672
40:00:10.828472
50:00:11.803448
60:00:14.999749
\n", 373 | "
" 374 | ], 375 | "text/plain": [ 376 | " RunTime\n", 377 | "2 0:00:06.474073\n", 378 | "3 0:00:07.971672\n", 379 | "4 0:00:10.828472\n", 380 | "5 0:00:11.803448\n", 381 | "6 0:00:14.999749" 382 | ] 383 | }, 384 | "execution_count": 207, 385 | "metadata": {}, 386 | "output_type": "execute_result" 387 | } 388 | ], 389 | "source": [ 390 | "pd.DataFrame(list(runtime.values()), list(runtime.keys()), columns=['RunTime'])" 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": null, 396 | "metadata": {}, 397 | "outputs": [], 398 | "source": [] 399 | } 400 | ], 401 | "metadata": { 402 | "kernelspec": { 403 | "display_name": "Python 3", 404 | "language": "python", 405 | "name": "python3" 406 | }, 407 | "language_info": { 408 | "codemirror_mode": { 409 | "name": "ipython", 410 | "version": 3 411 | }, 412 | "file_extension": ".py", 413 | "mimetype": "text/x-python", 414 | "name": "python", 415 | "nbconvert_exporter": "python", 416 | "pygments_lexer": "ipython3", 417 | "version": "3.6.4" 418 | } 419 | }, 420 | "nbformat": 4, 421 | "nbformat_minor": 2 422 | } 423 | -------------------------------------------------------------------------------- /Fraud_Anomaly_Detection/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mgroncki/DataScienceNotebooks/c031fbbb1a297d3b3407863b13f0cd461a82bb3f/Fraud_Anomaly_Detection/.DS_Store -------------------------------------------------------------------------------- /Fraud_Anomaly_Detection/.gitignore: -------------------------------------------------------------------------------- 1 | Data 2 | models 3 | *.png 4 | *.npy 5 | .ipynb_checkpoints 6 | -------------------------------------------------------------------------------- /Fraud_Anomaly_Detection/SampleData.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Sample Data Generation\n", 8 | "\n", 9 | "This script generates artificial log files by simulating user activities \n", 10 | "\n", 11 | "The data is used to train and test different models (like an hidden markov model, recurrent neural networks and generative adversial networks to detect outliers / fraud in user transactional behavior. For example the employee usage of a CRM system to detect data theft or customer behavior in a online shop to sport fraudulent orders.\n", 12 | "\n", 13 | "Simulation is done via Markov Chain Monte Carlo Simulation.\n", 14 | "\n", 15 | "We define a set of possible activities like (start, end, action_1, ..., action_n) and a set of users. Each user belong to one of n classes (eg. normal behavior and fraudulent behavior). For each class we have a \n", 16 | "transition matrix giving the probabilities that a user perform a specific action given his previous action and his state.\n" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 2, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import os\n", 26 | "import numpy as np\n", 27 | "import pandas as pd" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 42, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "# Configuration\n", 37 | "\n", 38 | "# first two actions need to be start and end\n", 39 | "actions = ['start', 'end', 'action_1', 'action_2', 'action_3']\n", 40 | "\n", 41 | "n_users = 200\n", 42 | "\n", 43 | "n_states = 1\n", 44 | "\n", 45 | "# Probability that a user is one of these states\n", 46 | "p_fraudulent_user = 0.02\n", 47 | "p_commit_fraud = 0.25\n", 48 | "\n", 49 | "p_states = [1]\n", 50 | "\n", 51 | "n_activities_per_user_per_day = 100\n", 52 | "\n", 53 | "n_days = 1\n", 54 | "\n", 55 | "transitions = [\n", 56 | " # Normal behavior\n", 57 | " np.array([\n", 58 | " [0.00, 0.00, 1/3, 1/3, 1/3],\n", 59 | " [1.00, 0.00, 0.00, 0.00, 0.00],\n", 60 | " [0.00, 0.01, 0.09, 0.45, 0.45],\n", 61 | " [0.00, 0.88, 0.01, 0.10, 0.01],\n", 62 | " [0.00, 0.68, 0.01, 0.30, 0.01],\n", 63 | " ]),\n", 64 | "]\n", 65 | "\n", 66 | "fraud_transitions = [\n", 67 | " # Fraudulent Behavior\n", 68 | " np.array([\n", 69 | " [0.00, 0.00, 1.00, 0.00, 0.00],\n", 70 | " [1.00, 0.00, 0.00, 0.00, 0.00],\n", 71 | " [0.00, 0.19, 0.80, 0.005, 0.005],\n", 72 | " [0.00, 0.88, 0.01, 0.10, 0.01],\n", 73 | " [0.00, 0.68, 0.01, 0.30, 0.01],\n", 74 | " ])\n", 75 | "]\n", 76 | "\n", 77 | "assert len(p_states)==n_states, 'Inconsitent number of states and state probs'\n", 78 | "assert np.sum(p_states)==1, 'State probs doesnt sum up to one'\n", 79 | "assert len(transitions)==n_states, 'Inconsitent number of transition matrixes and state'\n", 80 | "for i in range(n_states):\n", 81 | " assert np.allclose(transitions[i].sum(), len(actions))\n", 82 | " assert np.allclose(transitions[i].cumsum(axis=1)[:,-1],1)\n", 83 | "assert np.allclose(fraud_transitions[0].sum(), len(actions))\n", 84 | "assert np.allclose(fraud_transitions[0].cumsum(axis=1)[:,-1],1)" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 43, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "# Simulation\n", 101 | "def simulate_markov_chain(transition_matrix, actions):\n", 102 | " '''\n", 103 | " Simulate a user session using the corresponding transition matrix\n", 104 | " given the current user state\n", 105 | " \n", 106 | " Parameter:\n", 107 | " \n", 108 | " transitions : list of transition matrixes\n", 109 | " state_id: int of the current user state\n", 110 | " actions: list of all available actions\n", 111 | " \n", 112 | " Returns:\n", 113 | " \n", 114 | " list of artificial activities in one session\n", 115 | " '''\n", 116 | " activities = [actions[0]]\n", 117 | " while activities[-1] != 'end':\n", 118 | " prev_action = actions.index(activities[-1])\n", 119 | " transition_probs = transition_matrix[prev_action]\n", 120 | " next_action = np.random.choice(actions, size=1, p=transition_probs)[0]\n", 121 | " activities.append(next_action)\n", 122 | " return activities\n", 123 | "\n", 124 | "\n", 125 | "np.random.seed(42)\n", 126 | "log_data = []\n", 127 | "for user in range(n_users):\n", 128 | " user_class = np.random.choice(np.arange(0, n_states), size=1, p=p_states)[0]\n", 129 | " user_pot_fraud = np.random.binomial(1, p_fraudulent_user)\n", 130 | " if user_pot_fraud == 0:\n", 131 | " transitions_matrices = [transitions[user_class]]\n", 132 | " probs = [1]\n", 133 | " else:\n", 134 | " transitions_matrices = [transitions[user_class], fraud_transitions[0]]\n", 135 | " probs = [1-p_commit_fraud, p_commit_fraud]\n", 136 | " for day in range(n_days):\n", 137 | " commit_fraud = np.random.choice(np.arange(len(transitions_matrices)), size=n_activities_per_user_per_day, p=probs)\n", 138 | " for fraud_flat in commit_fraud:\n", 139 | " activities = simulate_markov_chain(transitions_matrices[fraud_flat], actions)\n", 140 | " log_data.append((user, day, i, activities, user_pot_fraud, fraud_flat))" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 44, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "logfile = pd.DataFrame(log_data, columns=('UserID', 'Day', 'UserSessionId', 'SessionActivity', 'PotentialFraudster', 'FraudulentActivity'))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 45, 155 | "metadata": {}, 156 | "outputs": [ 157 | { 158 | "data": { 159 | "text/html": [ 160 | "
\n", 161 | "\n", 174 | "\n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | " \n", 204 | " \n", 205 | " \n", 206 | " \n", 207 | " \n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | " \n", 222 | " \n", 223 | " \n", 224 | " \n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | " \n", 381 | " \n", 382 | " \n", 383 | " \n", 384 | " \n", 385 | " \n", 386 | " \n", 387 | " \n", 388 | " \n", 389 | " \n", 390 | " \n", 391 | " \n", 392 | " \n", 393 | " \n", 394 | " \n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 415 | " \n", 416 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | " \n", 448 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 452 | " \n", 453 | " \n", 454 | " \n", 455 | " \n", 456 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 460 | " \n", 461 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 497 | " \n", 498 | " \n", 499 | " \n", 500 | " \n", 501 | " \n", 502 | " \n", 503 | " \n", 504 | " \n", 505 | " \n", 506 | " \n", 507 | " \n", 508 | " \n", 509 | " \n", 510 | " \n", 511 | " \n", 512 | " \n", 513 | " \n", 514 | " \n", 515 | " \n", 516 | " \n", 517 | " \n", 518 | " \n", 519 | " \n", 520 | " \n", 521 | " \n", 522 | " \n", 523 | " \n", 524 | " \n", 525 | " \n", 526 | " \n", 527 | " \n", 528 | " \n", 529 | " \n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | " \n", 534 | " \n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 542 | " \n", 543 | " \n", 544 | " \n", 545 | " \n", 546 | " \n", 547 | " \n", 548 | " \n", 549 | " \n", 550 | " \n", 551 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | " \n", 717 | " \n", 718 | " \n", 719 | " \n", 720 | " \n", 721 | " \n", 722 | " \n", 723 | " \n", 724 | " \n", 725 | " \n", 726 | " \n", 727 | " \n", 728 | " \n", 729 | " \n", 730 | " \n", 731 | " \n", 732 | " \n", 733 | " \n", 734 | " \n", 735 | " \n", 736 | " \n", 737 | "
UserIDDayUserSessionIdSessionActivityPotentialFraudsterFraudulentActivity
95009500[start, action_3, end]10
95019500[start, action_3, end]10
95029500[start, action_1, action_3, end]10
95039500[start, action_1, action_3, action_2, end]10
95049500[start, action_3, action_2, end]10
95059500[start, action_1, end]11
95069500[start, action_1, action_1, end]11
95079500[start, action_2, end]10
95089500[start, action_3, end]10
95099500[start, action_3, end]10
95109500[start, action_1, action_2, end]10
95119500[start, action_3, end]10
95129500[start, action_2, end]10
95139500[start, action_1, action_1, end]11
95149500[start, action_1, action_2, end]10
95159500[start, action_3, end]10
95169500[start, action_1, action_1, action_1, end]11
95179500[start, action_1, action_2, end]10
95189500[start, action_3, action_2, end]10
95199500[start, action_1, action_3, action_2, action_2...10
95209500[start, action_2, end]10
95219500[start, action_3, end]10
95229500[start, action_1, action_1, end]11
95239500[start, action_2, end]10
95249500[start, action_1, action_1, action_1, action_1...11
95259500[start, action_1, action_2, end]10
95269500[start, action_1, action_1, action_1, end]11
95279500[start, action_1, action_1, action_1, action_1...11
95289500[start, action_2, end]10
95299500[start, action_1, action_1, action_1, action_1...11
.....................
1717017100[start, action_3, end]10
1717117100[start, action_3, action_2, end]10
1717217100[start, action_1, action_2, end]10
1717317100[start, action_1, end]11
1717417100[start, action_1, action_1, action_3, end]10
1717517100[start, action_1, action_1, action_2, end]11
1717617100[start, action_1, action_3, end]10
1717717100[start, action_2, action_2, end]10
1717817100[start, action_1, action_1, action_1, end]11
1717917100[start, action_3, end]10
1718017100[start, action_1, action_1, end]11
1718117100[start, action_1, action_3, action_2, end]10
1718217100[start, action_1, action_3, end]10
1718317100[start, action_3, end]10
1718417100[start, action_2, end]10
1718517100[start, action_2, end]10
1718617100[start, action_1, end]11
1718717100[start, action_1, action_2, end]10
1718817100[start, action_1, action_3, end]10
1718917100[start, action_1, action_1, action_1, end]11
1719017100[start, action_1, action_3, action_2, action_2...10
1719117100[start, action_2, end]10
1719217100[start, action_1, action_1, action_1, end]11
1719317100[start, action_1, end]11
1719417100[start, action_1, action_1, action_1, action_1...11
1719517100[start, action_1, action_2, end]10
1719617100[start, action_2, end]10
1719717100[start, action_1, action_2, action_2, end]10
1719817100[start, action_1, end]11
1719917100[start, action_1, action_1, action_1, action_1...11
\n", 738 | "

300 rows × 6 columns

\n", 739 | "
" 740 | ], 741 | "text/plain": [ 742 | " UserID Day UserSessionId \\\n", 743 | "9500 95 0 0 \n", 744 | "9501 95 0 0 \n", 745 | "9502 95 0 0 \n", 746 | "9503 95 0 0 \n", 747 | "9504 95 0 0 \n", 748 | "9505 95 0 0 \n", 749 | "9506 95 0 0 \n", 750 | "9507 95 0 0 \n", 751 | "9508 95 0 0 \n", 752 | "9509 95 0 0 \n", 753 | "9510 95 0 0 \n", 754 | "9511 95 0 0 \n", 755 | "9512 95 0 0 \n", 756 | "9513 95 0 0 \n", 757 | "9514 95 0 0 \n", 758 | "9515 95 0 0 \n", 759 | "9516 95 0 0 \n", 760 | "9517 95 0 0 \n", 761 | "9518 95 0 0 \n", 762 | "9519 95 0 0 \n", 763 | "9520 95 0 0 \n", 764 | "9521 95 0 0 \n", 765 | "9522 95 0 0 \n", 766 | "9523 95 0 0 \n", 767 | "9524 95 0 0 \n", 768 | "9525 95 0 0 \n", 769 | "9526 95 0 0 \n", 770 | "9527 95 0 0 \n", 771 | "9528 95 0 0 \n", 772 | "9529 95 0 0 \n", 773 | "... ... ... ... \n", 774 | "17170 171 0 0 \n", 775 | "17171 171 0 0 \n", 776 | "17172 171 0 0 \n", 777 | "17173 171 0 0 \n", 778 | "17174 171 0 0 \n", 779 | "17175 171 0 0 \n", 780 | "17176 171 0 0 \n", 781 | "17177 171 0 0 \n", 782 | "17178 171 0 0 \n", 783 | "17179 171 0 0 \n", 784 | "17180 171 0 0 \n", 785 | "17181 171 0 0 \n", 786 | "17182 171 0 0 \n", 787 | "17183 171 0 0 \n", 788 | "17184 171 0 0 \n", 789 | "17185 171 0 0 \n", 790 | "17186 171 0 0 \n", 791 | "17187 171 0 0 \n", 792 | "17188 171 0 0 \n", 793 | "17189 171 0 0 \n", 794 | "17190 171 0 0 \n", 795 | "17191 171 0 0 \n", 796 | "17192 171 0 0 \n", 797 | "17193 171 0 0 \n", 798 | "17194 171 0 0 \n", 799 | "17195 171 0 0 \n", 800 | "17196 171 0 0 \n", 801 | "17197 171 0 0 \n", 802 | "17198 171 0 0 \n", 803 | "17199 171 0 0 \n", 804 | "\n", 805 | " SessionActivity PotentialFraudster \\\n", 806 | "9500 [start, action_3, end] 1 \n", 807 | "9501 [start, action_3, end] 1 \n", 808 | "9502 [start, action_1, action_3, end] 1 \n", 809 | "9503 [start, action_1, action_3, action_2, end] 1 \n", 810 | "9504 [start, action_3, action_2, end] 1 \n", 811 | "9505 [start, action_1, end] 1 \n", 812 | "9506 [start, action_1, action_1, end] 1 \n", 813 | "9507 [start, action_2, end] 1 \n", 814 | "9508 [start, action_3, end] 1 \n", 815 | "9509 [start, action_3, end] 1 \n", 816 | "9510 [start, action_1, action_2, end] 1 \n", 817 | "9511 [start, action_3, end] 1 \n", 818 | "9512 [start, action_2, end] 1 \n", 819 | "9513 [start, action_1, action_1, end] 1 \n", 820 | "9514 [start, action_1, action_2, end] 1 \n", 821 | "9515 [start, action_3, end] 1 \n", 822 | "9516 [start, action_1, action_1, action_1, end] 1 \n", 823 | "9517 [start, action_1, action_2, end] 1 \n", 824 | "9518 [start, action_3, action_2, end] 1 \n", 825 | "9519 [start, action_1, action_3, action_2, action_2... 1 \n", 826 | "9520 [start, action_2, end] 1 \n", 827 | "9521 [start, action_3, end] 1 \n", 828 | "9522 [start, action_1, action_1, end] 1 \n", 829 | "9523 [start, action_2, end] 1 \n", 830 | "9524 [start, action_1, action_1, action_1, action_1... 1 \n", 831 | "9525 [start, action_1, action_2, end] 1 \n", 832 | "9526 [start, action_1, action_1, action_1, end] 1 \n", 833 | "9527 [start, action_1, action_1, action_1, action_1... 1 \n", 834 | "9528 [start, action_2, end] 1 \n", 835 | "9529 [start, action_1, action_1, action_1, action_1... 1 \n", 836 | "... ... ... \n", 837 | "17170 [start, action_3, end] 1 \n", 838 | "17171 [start, action_3, action_2, end] 1 \n", 839 | "17172 [start, action_1, action_2, end] 1 \n", 840 | "17173 [start, action_1, end] 1 \n", 841 | "17174 [start, action_1, action_1, action_3, end] 1 \n", 842 | "17175 [start, action_1, action_1, action_2, end] 1 \n", 843 | "17176 [start, action_1, action_3, end] 1 \n", 844 | "17177 [start, action_2, action_2, end] 1 \n", 845 | "17178 [start, action_1, action_1, action_1, end] 1 \n", 846 | "17179 [start, action_3, end] 1 \n", 847 | "17180 [start, action_1, action_1, end] 1 \n", 848 | "17181 [start, action_1, action_3, action_2, end] 1 \n", 849 | "17182 [start, action_1, action_3, end] 1 \n", 850 | "17183 [start, action_3, end] 1 \n", 851 | "17184 [start, action_2, end] 1 \n", 852 | "17185 [start, action_2, end] 1 \n", 853 | "17186 [start, action_1, end] 1 \n", 854 | "17187 [start, action_1, action_2, end] 1 \n", 855 | "17188 [start, action_1, action_3, end] 1 \n", 856 | "17189 [start, action_1, action_1, action_1, end] 1 \n", 857 | "17190 [start, action_1, action_3, action_2, action_2... 1 \n", 858 | "17191 [start, action_2, end] 1 \n", 859 | "17192 [start, action_1, action_1, action_1, end] 1 \n", 860 | "17193 [start, action_1, end] 1 \n", 861 | "17194 [start, action_1, action_1, action_1, action_1... 1 \n", 862 | "17195 [start, action_1, action_2, end] 1 \n", 863 | "17196 [start, action_2, end] 1 \n", 864 | "17197 [start, action_1, action_2, action_2, end] 1 \n", 865 | "17198 [start, action_1, end] 1 \n", 866 | "17199 [start, action_1, action_1, action_1, action_1... 1 \n", 867 | "\n", 868 | " FraudulentActivity \n", 869 | "9500 0 \n", 870 | "9501 0 \n", 871 | "9502 0 \n", 872 | "9503 0 \n", 873 | "9504 0 \n", 874 | "9505 1 \n", 875 | "9506 1 \n", 876 | "9507 0 \n", 877 | "9508 0 \n", 878 | "9509 0 \n", 879 | "9510 0 \n", 880 | "9511 0 \n", 881 | "9512 0 \n", 882 | "9513 1 \n", 883 | "9514 0 \n", 884 | "9515 0 \n", 885 | "9516 1 \n", 886 | "9517 0 \n", 887 | "9518 0 \n", 888 | "9519 0 \n", 889 | "9520 0 \n", 890 | "9521 0 \n", 891 | "9522 1 \n", 892 | "9523 0 \n", 893 | "9524 1 \n", 894 | "9525 0 \n", 895 | "9526 1 \n", 896 | "9527 1 \n", 897 | "9528 0 \n", 898 | "9529 1 \n", 899 | "... ... \n", 900 | "17170 0 \n", 901 | "17171 0 \n", 902 | "17172 0 \n", 903 | "17173 1 \n", 904 | "17174 0 \n", 905 | "17175 1 \n", 906 | "17176 0 \n", 907 | "17177 0 \n", 908 | "17178 1 \n", 909 | "17179 0 \n", 910 | "17180 1 \n", 911 | "17181 0 \n", 912 | "17182 0 \n", 913 | "17183 0 \n", 914 | "17184 0 \n", 915 | "17185 0 \n", 916 | "17186 1 \n", 917 | "17187 0 \n", 918 | "17188 0 \n", 919 | "17189 1 \n", 920 | "17190 0 \n", 921 | "17191 0 \n", 922 | "17192 1 \n", 923 | "17193 1 \n", 924 | "17194 1 \n", 925 | "17195 0 \n", 926 | "17196 0 \n", 927 | "17197 0 \n", 928 | "17198 1 \n", 929 | "17199 1 \n", 930 | "\n", 931 | "[300 rows x 6 columns]" 932 | ] 933 | }, 934 | "execution_count": 45, 935 | "metadata": {}, 936 | "output_type": "execute_result" 937 | } 938 | ], 939 | "source": [ 940 | "logfile[logfile.PotentialFraudster==1]" 941 | ] 942 | }, 943 | { 944 | "cell_type": "code", 945 | "execution_count": 6, 946 | "metadata": {}, 947 | "outputs": [], 948 | "source": [ 949 | "%%bash \n", 950 | "mkdir Data" 951 | ] 952 | }, 953 | { 954 | "cell_type": "code", 955 | "execution_count": 46, 956 | "metadata": {}, 957 | "outputs": [], 958 | "source": [ 959 | "logfile.to_pickle('./Data/logfile_fraudulent_activities.pkl')" 960 | ] 961 | }, 962 | { 963 | "cell_type": "code", 964 | "execution_count": 47, 965 | "metadata": {}, 966 | "outputs": [], 967 | "source": [ 968 | "id2actions = np.array(['']+actions)\n", 969 | "np.save('./Data/id2action', id2actions)" 970 | ] 971 | }, 972 | { 973 | "cell_type": "code", 974 | "execution_count": 48, 975 | "metadata": {}, 976 | "outputs": [ 977 | { 978 | "name": "stdout", 979 | "output_type": "stream", 980 | "text": [ 981 | "total 12752\n", 982 | "8604106216 0 drwxr-xr-x 5 matthiasgroncki staff 160 Nov 9 07:10 .\n", 983 | "8604100559 0 drwxr-xr-x 13 matthiasgroncki staff 416 Nov 9 07:10 ..\n", 984 | "8604106313 8 -rw-r--r-- 1 matthiasgroncki staff 320 Nov 9 07:10 id2action.npy\n", 985 | "8604126099 8544 -rw-r--r-- 1 matthiasgroncki staff 4373842 Nov 9 07:10 logfile_fraudulent_activities.pkl\n", 986 | "8604106218 4200 -rw-r--r-- 1 matthiasgroncki staff 2147215 Nov 8 16:10 logfile_v1.pkl\n" 987 | ] 988 | } 989 | ], 990 | "source": [ 991 | "%%bash\n", 992 | "\n", 993 | "cd Data\n", 994 | "ls -lisa" 995 | ] 996 | }, 997 | { 998 | "cell_type": "code", 999 | "execution_count": null, 1000 | "metadata": {}, 1001 | "outputs": [], 1002 | "source": [] 1003 | }, 1004 | { 1005 | "cell_type": "code", 1006 | "execution_count": null, 1007 | "metadata": {}, 1008 | "outputs": [], 1009 | "source": [] 1010 | } 1011 | ], 1012 | "metadata": { 1013 | "celltoolbar": "Slideshow", 1014 | "kernelspec": { 1015 | "display_name": "Python [conda env:anaconda]", 1016 | "language": "python", 1017 | "name": "conda-env-anaconda-py" 1018 | }, 1019 | "language_info": { 1020 | "codemirror_mode": { 1021 | "name": "ipython", 1022 | "version": 3 1023 | }, 1024 | "file_extension": ".py", 1025 | "mimetype": "text/x-python", 1026 | "name": "python", 1027 | "nbconvert_exporter": "python", 1028 | "pygments_lexer": "ipython3", 1029 | "version": "3.6.4" 1030 | } 1031 | }, 1032 | "nbformat": 4, 1033 | "nbformat_minor": 2 1034 | } 1035 | -------------------------------------------------------------------------------- /Fraud_Anomaly_Detection/version1/SampleData.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Sample Data Generation\n", 8 | "\n", 9 | "This script generates artificial log files by simulating user activities \n", 10 | "\n", 11 | "The data is used to train and test different models (like an hidden markov model, recurrent neural networks and generative adversial networks to detect outliers / fraud in user transactional behavior. For example the employee usage of a CRM system to detect data theft or customer behavior in a online shop to sport fraudulent orders.\n", 12 | "\n", 13 | "Simulation is done via Markov Chain Monte Carlo Simulation.\n", 14 | "\n", 15 | "We define a set of possible activities like (start, end, action_1, ..., action_n) and a set of users. Each user belong to one of n classes (eg. normal behavior and fraudulent behavior). For each class we have a \n", 16 | "transition matrix giving the probabilities that a user perform a specific action given his previous action and his state.\n" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 2, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import os\n", 26 | "import numpy as np\n", 27 | "import pandas as pd" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 3, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "# Configuration\n", 37 | "\n", 38 | "# first two actions need to be start and end\n", 39 | "actions = ['start', 'end', 'action_1', 'action_2', 'action_3']\n", 40 | "\n", 41 | "n_users = 100\n", 42 | "\n", 43 | "n_states = 2\n", 44 | "\n", 45 | "# Probability that a user is one of these states\n", 46 | "p_states = [0.98, 0.02]\n", 47 | "\n", 48 | "n_activities_per_user_per_day = 100\n", 49 | "\n", 50 | "n_days = 1\n", 51 | "\n", 52 | "transitions = [\n", 53 | " # Normal behavior\n", 54 | " np.array([\n", 55 | " [0.00, 0.00, 1/3, 1/3, 1/3],\n", 56 | " [1.00, 0.00, 0.00, 0.00, 0.00],\n", 57 | " [0.00, 0.01, 0.09, 0.45, 0.45],\n", 58 | " [0.00, 0.88, 0.01, 0.10, 0.01],\n", 59 | " [0.00, 0.68, 0.01, 0.30, 0.01],\n", 60 | " ]),\n", 61 | " # Fraudulent Behavior\n", 62 | " np.array([\n", 63 | " [0.00, 0.00, 0.8, 0.1, 0.1],\n", 64 | " [1.00, 0.00, 0.00, 0.00, 0.00],\n", 65 | " [0.00, 0.2, 0.70, 0.05, 0.05],\n", 66 | " [0.00, 0.88, 0.01, 0.10, 0.01],\n", 67 | " [0.00, 0.68, 0.01, 0.30, 0.01],\n", 68 | " ])\n", 69 | "]\n", 70 | "\n", 71 | "assert len(p_states)==n_states, 'Inconsitent number of states and state probs'\n", 72 | "assert np.sum(p_states)==1, 'State probs doesnt sum up to one'\n", 73 | "assert len(transitions)==n_states, 'Inconsitent number of transition matrixes and state'\n", 74 | "for i in range(n_states):\n", 75 | " assert np.allclose(transitions[i].sum(), len(actions))\n", 76 | " assert np.allclose(transitions[i].cumsum(axis=1)[:,-1],1)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 4, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# Simulation\n", 86 | "def simulate_markov_chain(transitions, state_id, actions):\n", 87 | " '''\n", 88 | " Simulate a user session using the corresponding transition matrix\n", 89 | " given the current user state\n", 90 | " \n", 91 | " Parameter:\n", 92 | " \n", 93 | " transitions : list of transition matrixes\n", 94 | " state_id: int of the current user state\n", 95 | " actions: list of all available actions\n", 96 | " \n", 97 | " Returns:\n", 98 | " \n", 99 | " list of artificial activities in one session\n", 100 | " '''\n", 101 | " activities = [actions[0]]\n", 102 | " while activities[-1] != 'end':\n", 103 | " prev_action = actions.index(activities[-1])\n", 104 | " transition_probs = transitions[state_id][prev_action]\n", 105 | " next_action = np.random.choice(actions, size=1, p=transition_probs)[0]\n", 106 | " activities.append(next_action)\n", 107 | " return activities\n", 108 | "\n", 109 | "\n", 110 | "np.random.seed(42)\n", 111 | "log_data = []\n", 112 | "for user in range(n_users):\n", 113 | " for day in range(n_days):\n", 114 | " current_state = np.random.choice(np.arange(0, n_states), size=1, p=p_states)[0]\n", 115 | " for i in range(n_activities_per_user_per_day):\n", 116 | " activities = simulate_markov_chain(transitions, current_state, actions)\n", 117 | " log_data.append((user, day, i, activities, current_state))" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 5, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "logfile = pd.DataFrame(log_data, columns=('UserID', 'Day', 'UserSessionId', 'SessionActivity', 'State'))" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": 6, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "%%bash \n", 136 | "mkdir Data" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 7, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "logfile.to_pickle('./Data/logfile_v1.pkl')" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 13, 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "id2actions = np.array(['']+actions)\n", 155 | "np.save('./Data/id2action', id2actions)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 14, 161 | "metadata": {}, 162 | "outputs": [ 163 | { 164 | "name": "stdout", 165 | "output_type": "stream", 166 | "text": [ 167 | "total 4208\n", 168 | "8604106216 0 drwxr-xr-x 4 matthiasgroncki staff 128 Nov 8 16:11 .\n", 169 | "8604100559 0 drwxr-xr-x 8 matthiasgroncki staff 256 Nov 8 16:11 ..\n", 170 | "8604106313 8 -rw-r--r-- 1 matthiasgroncki staff 320 Nov 8 16:11 id2action.npy\n", 171 | "8604106218 4200 -rw-r--r-- 1 matthiasgroncki staff 2147215 Nov 8 16:10 logfile_v1.pkl\n" 172 | ] 173 | } 174 | ], 175 | "source": [ 176 | "%%bash\n", 177 | "\n", 178 | "cd Data\n", 179 | "ls -lisa" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "metadata": {}, 193 | "outputs": [], 194 | "source": [] 195 | } 196 | ], 197 | "metadata": { 198 | "celltoolbar": "Slideshow", 199 | "kernelspec": { 200 | "display_name": "Python [conda env:anaconda]", 201 | "language": "python", 202 | "name": "conda-env-anaconda-py" 203 | }, 204 | "language_info": { 205 | "codemirror_mode": { 206 | "name": "ipython", 207 | "version": 3 208 | }, 209 | "file_extension": ".py", 210 | "mimetype": "text/x-python", 211 | "name": "python", 212 | "nbconvert_exporter": "python", 213 | "pygments_lexer": "ipython3", 214 | "version": "3.6.4" 215 | } 216 | }, 217 | "nbformat": 4, 218 | "nbformat_minor": 2 219 | } 220 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Notebooks related to my blog 2 | 3 | Here you find the corresponding source code to my blog https://ipythonquant.wordpress.com 4 | 5 | - CreditCardFraudDetection: 6 | 7 | Tutorial to develop a use deep learning to detect fraudulent credit card transactions from scratch using TensorFlow Low Level API. A step-by-step introduction starting to build a Logistic Regression in TensorFlow to a Deep neural network 8 | 9 | - Fraud_Anonmaly_Detection: 10 | 11 | Tutorial series to use recurrent neural networks with PyTorch in Python to detect anomalies (potential fraudulent) in transactional data. In the first version we use a LSTM network with an embedding layer to learn a simple user profiles on synthetic transctional data (e.g. a logfile). 12 | 13 | - ExoticDerivatePricingWithTensorFlow: 14 | 15 | Tutorial to use TensorFlow for Monte-Carlo derivate pricing. Pricing Bermudan and Barrier Options 16 | 17 | - DeepHedging: 18 | 19 | Use a deep neural networks to derive a model free option hedging strategy in TensorFlow. 20 | 21 | 22 | --------------------------------------------------------------------------------