├── LICENSE
├── Notebook
    ├── Section 1 - Introduction to PyTorch-v2-soln.ipynb
    ├── Section 1 - Introduction to PyTorch-v2.ipynb
    ├── Section 2 - Feedforward Neural Network_v2c.ipynb
    ├── Section 3 - Practical Aspects of Building Deep Learning Models-v3.ipynb
    ├── Section 3 - Practical Aspects of Building Deep Learning Models-v3_soln.ipynb
    ├── Section 4 - CNN Example.ipynb
    ├── Section 5 - RNN and NLP - Car Example v3b.ipynb
    ├── Section 5 - RNN and NLP v2-experiment2.ipynb
    ├── Section 6 - RNN for TSA Example - v2.ipynb
    └── data
    │   ├── section5
    │       ├── cache
    │       │   ├── testing.csv
    │       │   └── training.csv
    │       └── car_review.csv
    │   └── section6
    │       ├── series1_Xtrain.csv
    │       └── series1_ytrain.csv
└── README.md


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2019 Packt
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/Notebook/Section 3 - Practical Aspects of Building Deep Learning Models-v3.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "<div class=\"alert alert-block alert-info\">\n",
   8 |     "<font size=\"6\"><b><center> Section 3</font></center>\n",
   9 |     "<br>\n",
  10 |     "<font size=\"6\"><b><center> Practical Aspects of Building Deep Learning Models </font></center>\n",
  11 |     "</div>"
  12 |    ]
  13 |   },
  14 |   {
  15 |    "cell_type": "markdown",
  16 |    "metadata": {},
  17 |    "source": [
  18 |     "**In this example, we will illustrate the comparison between training and validation losses and regularization techniques using dropout and L2-norm regularization.**"
  19 |    ]
  20 |   },
  21 |   {
  22 |    "cell_type": "markdown",
  23 |    "metadata": {},
  24 |    "source": [
  25 |     "<div class=\"alert alert-block alert-info\">\n",
  26 |     "</div>"
  27 |    ]
  28 |   },
  29 |   {
  30 |    "cell_type": "markdown",
  31 |    "metadata": {},
  32 |    "source": [
  33 |     "# Notations for Regularization"
  34 |    ]
  35 |   },
  36 |   {
  37 |    "cell_type": "markdown",
  38 |    "metadata": {
  39 |     "ExecuteTime": {
  40 |      "end_time": "2019-04-06T15:09:37.958331Z",
  41 |      "start_time": "2019-04-06T15:09:37.950241Z"
  42 |     }
  43 |    },
  44 |    "source": [
  45 |     "Regularized objective function:\n",
  46 |     "\n",
  47 |     "$ C = C_0 + \\text{Penalty Term} $\n",
  48 |     "\n",
  49 |     "where $C_0$ is the unregularized objective function\n",
  50 |     "\n",
  51 |     "$L^2$ norm penalty term typically take the form of $ \\frac{1}{2} ||w||^2_2 $"
  52 |    ]
  53 |   },
  54 |   {
  55 |    "cell_type": "markdown",
  56 |    "metadata": {},
  57 |    "source": [
  58 |     "### Equation by equation"
  59 |    ]
  60 |   },
  61 |   {
  62 |    "cell_type": "markdown",
  63 |    "metadata": {},
  64 |    "source": [
  65 |     "**Inputs to hidden layer 1**\n",
  66 |     "\n",
  67 |     "$z_1(\\mathbf{x}; \\mathbf{w_1},b_1) = \\sum_{j=1}^4 w_{1,j}^{(1)} x_j + b_1^{(1)}$\n",
  68 |     "\n",
  69 |     "\n",
  70 |     "$z_2(\\mathbf{x}; \\mathbf{w_2},b_2) = \\sum_{j=1}^4 w_{2,j}^{(1)} x_j + b_2^{(1)}$\n",
  71 |     "\n",
  72 |     "$\\vdots$\n",
  73 |     "\n",
  74 |     "$z_6(\\mathbf{x}; \\mathbf{w_6},b_6) = \\sum_{j=1}^4 w_{6,j}^{(1)} x_j + b_6^{(1)}$\n",
  75 |     "\n",
  76 |     "where $j=1,2,3,4$ (in this example) is the index for the inputs\n",
  77 |     "\n",
  78 |     "$h_k(\\mathbf{x}; \\mathbf{w_k},b_k) = g_1(z_k)$ where $k=1,2,\\dots,6$ is the index for the hidden units\n",
  79 |     "\n",
  80 |     "\n",
  81 |     "**Hidden layer 1 to output**\n",
  82 |     "\n",
  83 |     "$y_1 = g_2 \\left( \\sum_{k=1}^6 w_{1,k}^{(2)} + b_1^{(2)} \\right)$\n",
  84 |     "\n",
  85 |     "$y_2 = g_2 \\left( \\sum_{k=1}^6 w_{2,k}^{(2)} + b_2^{(2)} \\right)$"
  86 |    ]
  87 |   },
  88 |   {
  89 |    "cell_type": "markdown",
  90 |    "metadata": {},
  91 |    "source": [
  92 |     "### Sequential layerwise architecture"
  93 |    ]
  94 |   },
  95 |   {
  96 |    "cell_type": "markdown",
  97 |    "metadata": {},
  98 |    "source": [
  99 |     "First layer:\n",
 100 |     "$$ \\mathbf{h}^{(1)} = g^{(1)} \\left(\\mathbf{W}^{(1)} \\mathbf{x} + \\mathbf{b}^{(1)}  \\right) $$\n",
 101 |     "\n",
 102 |     "Second layer:\n",
 103 |     "$$ \\mathbf{h}^{(2)} = g^{(2)} \\left(\\mathbf{W}^{(2)T} \\mathbf{h}^{(1)} + \\mathbf{b}^{(2)}  \\right) $$\n",
 104 |     "\n",
 105 |     "\n",
 106 |     "\n",
 107 |     "$\\vdots$\n",
 108 |     "\n",
 109 |     "\n",
 110 |     "$l^{th}$ layer:\n",
 111 |     "$$ \\mathbf{h}^{(l)} = g^{(l)} \\left(\\mathbf{W}^{(l)T} \\mathbf{h}^{(l-1)} + \\mathbf{b}^{(l)}  \\right) $$\n",
 112 |     "\n"
 113 |    ]
 114 |   },
 115 |   {
 116 |    "cell_type": "markdown",
 117 |    "metadata": {},
 118 |    "source": [
 119 |     "<div class=\"alert alert-block alert-info\">\n",
 120 |     "</div>"
 121 |    ]
 122 |   },
 123 |   {
 124 |    "cell_type": "markdown",
 125 |    "metadata": {},
 126 |    "source": [
 127 |     "# Example 1: Feedforward Network Without Regularization"
 128 |    ]
 129 |   },
 130 |   {
 131 |    "cell_type": "markdown",
 132 |    "metadata": {},
 133 |    "source": [
 134 |     "## Simulate and Visualize Data"
 135 |    ]
 136 |   },
 137 |   {
 138 |    "cell_type": "markdown",
 139 |    "metadata": {},
 140 |    "source": [
 141 |     "The output is related to the input using the following function\n",
 142 |     "\n",
 143 |     "$$y_i = 3x_{i,1} + x^2 exp(x_{i,1}) + \\epsilon_i$$\n",
 144 |     "\n",
 145 |     "where $\\epsilon$ is an independently and identically distributed (i.i.d.) random variable and $i = 1,2,\\dots,n$ is an index of examples (or observations)"
 146 |    ]
 147 |   },
 148 |   {
 149 |    "cell_type": "code",
 150 |    "execution_count": null,
 151 |    "metadata": {
 152 |     "ExecuteTime": {
 153 |      "end_time": "2019-04-11T03:10:45.835205Z",
 154 |      "start_time": "2019-04-11T03:10:43.675905Z"
 155 |     }
 156 |    },
 157 |    "outputs": [],
 158 |    "source": [
 159 |     "import numpy as np\n",
 160 |     "import pandas as pd\n",
 161 |     "import matplotlib.pyplot as plt\n",
 162 |     "import seaborn as sns\n",
 163 |     "%matplotlib inline\n",
 164 |     "\n",
 165 |     "n = 200 # number of examples (or observations)\n",
 166 |     "\n",
 167 |     "# Generate a set of n random numbers from a standard normal distribution\n",
 168 |     "epsilon = np.random.randn(n)\n",
 169 |     "\n",
 170 |     "# Generate a set of n random numbers from a uniform[0,1] distribution; name it x1\n",
 171 |     "# and create another variable, which we name x2, from x1\n",
 172 |     "x1 = np.random.uniform(0,1,n)\n",
 173 |     "x2 = 1. / (1. + np.exp(-np.power(x1,3))) + 0.5*epsilon\n",
 174 |     "x3 = np.random.randn(n)\n",
 175 |     "\n",
 176 |     "X = pd.DataFrame({'x1':x1, 'x2':x2})\n",
 177 |     "\n",
 178 |     "# Create the data generating mechanism\n",
 179 |     "y = 3*x1 + np.power(x1,2)*np.exp(x1) + 0.8*x2 + 0.5*x3 + epsilon"
 180 |    ]
 181 |   },
 182 |   {
 183 |    "cell_type": "markdown",
 184 |    "metadata": {},
 185 |    "source": [
 186 |     "## Split the data into a Training and Validation Set"
 187 |    ]
 188 |   },
 189 |   {
 190 |    "cell_type": "code",
 191 |    "execution_count": null,
 192 |    "metadata": {
 193 |     "ExecuteTime": {
 194 |      "end_time": "2019-04-11T03:10:45.897007Z",
 195 |      "start_time": "2019-04-11T03:10:45.837374Z"
 196 |     }
 197 |    },
 198 |    "outputs": [],
 199 |    "source": [
 200 |     "from sklearn.model_selection import train_test_split\n",
 201 |     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.5)"
 202 |    ]
 203 |   },
 204 |   {
 205 |    "cell_type": "markdown",
 206 |    "metadata": {},
 207 |    "source": [
 208 |     "### Basic Descriptive Analysis"
 209 |    ]
 210 |   },
 211 |   {
 212 |    "cell_type": "code",
 213 |    "execution_count": null,
 214 |    "metadata": {
 215 |     "ExecuteTime": {
 216 |      "end_time": "2019-04-11T03:10:45.938411Z",
 217 |      "start_time": "2019-04-11T03:10:45.899566Z"
 218 |     }
 219 |    },
 220 |    "outputs": [],
 221 |    "source": [
 222 |     "print(X_train.describe().T.round(4), \"\\n\")\n",
 223 |     "print(pd.DataFrame({'y':y_train}).describe().T.round(4))"
 224 |    ]
 225 |   },
 226 |   {
 227 |    "cell_type": "code",
 228 |    "execution_count": null,
 229 |    "metadata": {
 230 |     "ExecuteTime": {
 231 |      "end_time": "2019-04-11T03:10:45.954432Z",
 232 |      "start_time": "2019-04-11T03:10:45.940933Z"
 233 |     }
 234 |    },
 235 |    "outputs": [],
 236 |    "source": [
 237 |     "# Correlation between x1 and x2 in the training set\n",
 238 |     "np.corrcoef(X_train.iloc[:,0], X_train.iloc[:,1])[0][1].round(2)"
 239 |    ]
 240 |   },
 241 |   {
 242 |    "cell_type": "code",
 243 |    "execution_count": null,
 244 |    "metadata": {
 245 |     "ExecuteTime": {
 246 |      "end_time": "2019-04-11T03:10:46.989883Z",
 247 |      "start_time": "2019-04-11T03:10:45.957582Z"
 248 |     }
 249 |    },
 250 |    "outputs": [],
 251 |    "source": [
 252 |     "fig = plt.figure(figsize=(15,10))\n",
 253 |     "\n",
 254 |     "plt.subplot(3, 3, 1)\n",
 255 |     "plt.hist(y_train)\n",
 256 |     "plt.title(\"y (Training Set)\")\n",
 257 |     "\n",
 258 |     "plt.subplot(3, 3, 2)\n",
 259 |     "plt.scatter(X_train.iloc[:,0], y_train)\n",
 260 |     "plt.title(\"y vs x1 (Training Set)\")\n",
 261 |     "\n",
 262 |     "plt.subplot(3, 3, 3)\n",
 263 |     "plt.scatter(X_train.iloc[:,1], y_train)\n",
 264 |     "plt.title(\"y vs x2 (Training Set)\")\n",
 265 |     "\n",
 266 |     "plt.subplot(3, 3, 5)\n",
 267 |     "plt.hist(X_train.iloc[:,0])\n",
 268 |     "plt.title(\"x1 (Training Set)\")\n",
 269 |     "\n",
 270 |     "plt.subplot(3, 3, 6)\n",
 271 |     "plt.scatter(X_train.iloc[:,0], X_train.iloc[:,1])\n",
 272 |     "plt.title(\"x2 vs x1 (Training Set)\")\n",
 273 |     "\n",
 274 |     "plt.subplot(3, 3, 9)\n",
 275 |     "plt.hist(X_train.iloc[:,1])\n",
 276 |     "plt.title(\"x2 (Training Set)\")\n",
 277 |     "\n",
 278 |     "plt.tight_layout()"
 279 |    ]
 280 |   },
 281 |   {
 282 |    "cell_type": "markdown",
 283 |    "metadata": {},
 284 |    "source": [
 285 |     "<div class=\"alert alert-block alert-warning\">\n",
 286 |     "Note: Before training, `numpy array` and `pandas dataframe` need to be converted to `PyTorch's tensors`\n",
 287 |     "</div>"
 288 |    ]
 289 |   },
 290 |   {
 291 |    "cell_type": "code",
 292 |    "execution_count": null,
 293 |    "metadata": {
 294 |     "ExecuteTime": {
 295 |      "end_time": "2019-04-11T03:10:47.223917Z",
 296 |      "start_time": "2019-04-11T03:10:46.992792Z"
 297 |     }
 298 |    },
 299 |    "outputs": [],
 300 |    "source": [
 301 |     "# convert numpy array to tensor in shape of input size\n",
 302 |     "import torch \n",
 303 |     "\n",
 304 |     "X_train_ts = torch.from_numpy(X_train.values.reshape(-1,2)).float()\n",
 305 |     "y_train_ts = torch.from_numpy(y_train.reshape(-1,1)).float()\n",
 306 |     "\n",
 307 |     "X_test_ts = torch.from_numpy(X_test.values.reshape(-1,2)).float()\n",
 308 |     "y_test_ts = torch.from_numpy(y_test.reshape(-1,1)).float() # y_test is a numpy array"
 309 |    ]
 310 |   },
 311 |   {
 312 |    "cell_type": "code",
 313 |    "execution_count": null,
 314 |    "metadata": {
 315 |     "ExecuteTime": {
 316 |      "end_time": "2019-04-11T03:10:47.234580Z",
 317 |      "start_time": "2019-04-11T03:10:47.226455Z"
 318 |     }
 319 |    },
 320 |    "outputs": [],
 321 |    "source": [
 322 |     "print(\"X_train_ts is of %s type\" %type(X_train_ts))\n",
 323 |     "print(\"y_train_ts is of %s type\" %type(y_train_ts))\n",
 324 |     "print(\"X_test_ts is of %s type\" %type(X_test_ts))\n",
 325 |     "print(\"y_test_ts is of %s type\" %type(y_test_ts))"
 326 |    ]
 327 |   },
 328 |   {
 329 |    "cell_type": "code",
 330 |    "execution_count": null,
 331 |    "metadata": {
 332 |     "ExecuteTime": {
 333 |      "end_time": "2019-04-11T03:10:47.250339Z",
 334 |      "start_time": "2019-04-11T03:10:47.237666Z"
 335 |     }
 336 |    },
 337 |    "outputs": [],
 338 |    "source": [
 339 |     "print(X_train_ts.shape)\n",
 340 |     "print(y_train_ts.shape)\n",
 341 |     "print(X_test_ts.shape)\n",
 342 |     "print(y_test_ts.shape)"
 343 |    ]
 344 |   },
 345 |   {
 346 |    "cell_type": "code",
 347 |    "execution_count": null,
 348 |    "metadata": {
 349 |     "ExecuteTime": {
 350 |      "end_time": "2019-04-11T03:10:47.265520Z",
 351 |      "start_time": "2019-04-11T03:10:47.254969Z"
 352 |     }
 353 |    },
 354 |    "outputs": [],
 355 |    "source": [
 356 |     "# First 5 rows of tensor X\n",
 357 |     "print(\"First 5 rows of tensor X_train_ts\", \"\\n\",  X_train_ts[:5], \"\\n\")\n",
 358 |     "print(\"First 5 rows of tensor y_train_ts\", \"\\n\",  y_train_ts[:5], \"\\n\")"
 359 |    ]
 360 |   },
 361 |   {
 362 |    "cell_type": "markdown",
 363 |    "metadata": {},
 364 |    "source": [
 365 |     "## Define a Feed-forward network with 1 hidden layers"
 366 |    ]
 367 |   },
 368 |   {
 369 |    "cell_type": "code",
 370 |    "execution_count": null,
 371 |    "metadata": {
 372 |     "ExecuteTime": {
 373 |      "end_time": "2019-04-11T03:10:47.277111Z",
 374 |      "start_time": "2019-04-11T03:10:47.268471Z"
 375 |     }
 376 |    },
 377 |    "outputs": [],
 378 |    "source": [
 379 |     "# Let's confirm the dimensions of the inputs and outpu\n",
 380 |     "print(\"train_size: \", len(X_train_ts), \"\\n\")\n",
 381 |     "print(\"X shape:\", X_train_ts.shape, \"\\n\")\n",
 382 |     "print(\"y shape:\", y_train_ts.shape, \"\\n\")\n",
 383 |     "print(X_train_ts.shape[1])"
 384 |    ]
 385 |   },
 386 |   {
 387 |    "cell_type": "code",
 388 |    "execution_count": null,
 389 |    "metadata": {
 390 |     "ExecuteTime": {
 391 |      "end_time": "2019-04-11T03:10:47.290824Z",
 392 |      "start_time": "2019-04-11T03:10:47.279901Z"
 393 |     }
 394 |    },
 395 |    "outputs": [],
 396 |    "source": [
 397 |     "from __future__ import print_function\n",
 398 |     "\n",
 399 |     "import torch\n",
 400 |     "import torch.nn as nn\n",
 401 |     "import torch.nn.functional as F\n",
 402 |     "from torch.autograd import Variable\n",
 403 |     "\n",
 404 |     "class ffNet(nn.Module):\n",
 405 |     "    \"\"\"\n",
 406 |     "    D_in: input dimension\n",
 407 |     "    D_h1: dimension of hidden layer 1\n",
 408 |     "    D_out: output dimension\n",
 409 |     "    \"\"\"\n",
 410 |     "    \n",
 411 |     "    def __init__(self, D_in, D_h1, D_h2, D_out):\n",
 412 |     "        super(ffNet, self).__init__()\n",
 413 |     "        self.linear1 = torch.nn.Linear(D_in, D_h1)\n",
 414 |     "        self.linear2 = torch.nn.Linear(D_h1, D_out)\n",
 415 |     "\n",
 416 |     "    def forward(self, x):\n",
 417 |     "        h1 = F.relu(self.linear1(x))\n",
 418 |     "        y_pred = self.linear2(h1)\n",
 419 |     "        return y_pred"
 420 |    ]
 421 |   },
 422 |   {
 423 |    "cell_type": "markdown",
 424 |    "metadata": {},
 425 |    "source": [
 426 |     "## Construct the model by instantiating the class defined above"
 427 |    ]
 428 |   },
 429 |   {
 430 |    "cell_type": "code",
 431 |    "execution_count": null,
 432 |    "metadata": {
 433 |     "ExecuteTime": {
 434 |      "end_time": "2019-04-11T03:10:47.299700Z",
 435 |      "start_time": "2019-04-11T03:10:47.293785Z"
 436 |     }
 437 |    },
 438 |    "outputs": [],
 439 |    "source": [
 440 |     "X_train.shape[1]"
 441 |    ]
 442 |   },
 443 |   {
 444 |    "cell_type": "code",
 445 |    "execution_count": null,
 446 |    "metadata": {
 447 |     "ExecuteTime": {
 448 |      "end_time": "2019-04-11T03:10:47.310647Z",
 449 |      "start_time": "2019-04-11T03:10:47.303388Z"
 450 |     }
 451 |    },
 452 |    "outputs": [],
 453 |    "source": [
 454 |     "# Construct the model by instantiating the class defined above\n",
 455 |     "D_in = X_train.shape[1]\n",
 456 |     "D_h1, D_h2 = 8,4\n",
 457 |     "D_out = 1\n",
 458 |     "\n",
 459 |     "ffnet = ffNet(D_in, D_h1, D_h2, D_out)\n",
 460 |     "print(ffnet)"
 461 |    ]
 462 |   },
 463 |   {
 464 |    "cell_type": "markdown",
 465 |    "metadata": {},
 466 |    "source": [
 467 |     "## Define loss function and optimization algorithm"
 468 |    ]
 469 |   },
 470 |   {
 471 |    "cell_type": "code",
 472 |    "execution_count": null,
 473 |    "metadata": {
 474 |     "ExecuteTime": {
 475 |      "end_time": "2019-04-11T03:10:47.318512Z",
 476 |      "start_time": "2019-04-11T03:10:47.313929Z"
 477 |     }
 478 |    },
 479 |    "outputs": [],
 480 |    "source": [
 481 |     "# Define Optimizer and Loss Function\n",
 482 |     "\n",
 483 |     "optimizer = torch.optim.SGD(ffnet.parameters(), lr=0.01)\n",
 484 |     "\n",
 485 |     "loss_func = torch.nn.MSELoss()"
 486 |    ]
 487 |   },
 488 |   {
 489 |    "cell_type": "markdown",
 490 |    "metadata": {},
 491 |    "source": [
 492 |     "## Model Training"
 493 |    ]
 494 |   },
 495 |   {
 496 |    "cell_type": "code",
 497 |    "execution_count": null,
 498 |    "metadata": {
 499 |     "ExecuteTime": {
 500 |      "end_time": "2019-04-11T03:10:49.535855Z",
 501 |      "start_time": "2019-04-11T03:10:47.321226Z"
 502 |     },
 503 |     "scrolled": false
 504 |    },
 505 |    "outputs": [],
 506 |    "source": [
 507 |     "X_data = X_train_ts\n",
 508 |     "y_data = y_train_ts\n",
 509 |     "\n",
 510 |     "X_data_val = X_test_ts\n",
 511 |     "y_data_val = y_test_ts\n",
 512 |     "\n",
 513 |     "n_epoch = 500\n",
 514 |     "\n",
 515 |     "train_loss, val_loss = [],[]\n",
 516 |     "\n",
 517 |     "for epoch in range(1, n_epoch + 1):\n",
 518 |     "    y_pred = ffnet(X_data)    \n",
 519 |     "    epoch_loss_train = loss_func(y_pred, y_data) \n",
 520 |     "    \n",
 521 |     "    y_pred_val = ffnet(X_data_val)\n",
 522 |     "    epoch_loss_val = loss_func(y_pred_val, y_data_val)     \n",
 523 |     "    \n",
 524 |     "    optimizer.zero_grad()\n",
 525 |     "    epoch_loss_train.backward()        \n",
 526 |     "    optimizer.step()       \n",
 527 |     "    \n",
 528 |     "    train_loss.append(epoch_loss_train)\n",
 529 |     "    val_loss.append(epoch_loss_val)\n",
 530 |     "    \n",
 531 |     "    #if epoch <= 5 or epoch % 100 == 0:\n",
 532 |     "    if epoch % 100 == 0:\n",
 533 |     "        print('Epoch {}, Training loss {}, Validation loss {}'.format(epoch, round(float(epoch_loss_train),4), round(float(epoch_loss_val),4)))\n",
 534 |     "        \n",
 535 |     "        #plt.cla()\n",
 536 |     "        \n",
 537 |     "        fig = plt.figure(figsize=(16,6))\n",
 538 |     "        plt.subplot(1, 2, 1)\n",
 539 |     "        plt.scatter(X_train.iloc[:,0], y_train, label=\"actual\")\n",
 540 |     "        plt.scatter(X_train.iloc[:,0], y_pred.detach().numpy(), label=\"prediction\")\n",
 541 |     "        plt.title(\"Training\")\n",
 542 |     "        plt.legend()\n",
 543 |     "\n",
 544 |     "        plt.subplot(1, 2, 2)\n",
 545 |     "        plt.scatter(X_test.iloc[:,0], y_test, label=\"actual\")\n",
 546 |     "        plt.scatter(X_test.iloc[:,0], y_pred_val.detach().numpy(),label=\"prediction\")\n",
 547 |     "        plt.title(\"Validation\")\n",
 548 |     "        plt.legend()\n",
 549 |     "        \n",
 550 |     "        #plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 10, 'color':  'red'})\n",
 551 |     "        #plt.pause(0.1)\n",
 552 |     "\n",
 553 |     "        plt.show()"
 554 |    ]
 555 |   },
 556 |   {
 557 |    "cell_type": "markdown",
 558 |    "metadata": {},
 559 |    "source": [
 560 |     "## Plot loss curves"
 561 |    ]
 562 |   },
 563 |   {
 564 |    "cell_type": "code",
 565 |    "execution_count": null,
 566 |    "metadata": {
 567 |     "ExecuteTime": {
 568 |      "end_time": "2019-04-11T03:10:49.847213Z",
 569 |      "start_time": "2019-04-11T03:10:49.538950Z"
 570 |     }
 571 |    },
 572 |    "outputs": [],
 573 |    "source": [
 574 |     "fig = plt.figure(figsize=(12,8))\n",
 575 |     "plt.plot(range(1,len(train_loss)+1),train_loss,'b',label = 'training loss')\n",
 576 |     "plt.plot(range(1,len(val_loss)+1),val_loss,'g',label = 'validation loss')\n",
 577 |     "plt.legend()\n",
 578 |     "plt.title(\"Training and Validation Loss Curves\")"
 579 |    ]
 580 |   },
 581 |   {
 582 |    "cell_type": "markdown",
 583 |    "metadata": {},
 584 |    "source": [
 585 |     "# Example 2: Feedforward Network With a Dropout Layer"
 586 |    ]
 587 |   },
 588 |   {
 589 |    "cell_type": "code",
 590 |    "execution_count": null,
 591 |    "metadata": {
 592 |     "ExecuteTime": {
 593 |      "end_time": "2019-04-11T03:10:49.862640Z",
 594 |      "start_time": "2019-04-11T03:10:49.850775Z"
 595 |     }
 596 |    },
 597 |    "outputs": [],
 598 |    "source": [
 599 |     "from __future__ import print_function\n",
 600 |     "\n",
 601 |     "import torch\n",
 602 |     "import torch.nn as nn\n",
 603 |     "import torch.nn.functional as F\n",
 604 |     "from torch.autograd import Variable\n",
 605 |     "\n",
 606 |     "class ffNet(nn.Module):\n",
 607 |     "    \"\"\"\n",
 608 |     "    D_in: input dimension\n",
 609 |     "    D_h1: dimension of hidden layer 1\n",
 610 |     "    D_out: output dimension\n",
 611 |     "    \"\"\"\n",
 612 |     "    \n",
 613 |     "    def __init__(self, D_in, D_h1, D_h2, D_out):\n",
 614 |     "        super(ffNet, self).__init__()\n",
 615 |     "        self.linear1 = torch.nn.Linear(D_in, D_h1)\n",
 616 |     "        self.linear1_drop = torch.nn.Dropout(p=0.8)\n",
 617 |     "        self.linear2 = torch.nn.Linear(D_h1, D_out)\n",
 618 |     "\n",
 619 |     "    def forward(self, x):\n",
 620 |     "        h1 = F.relu(self.linear1(x))\n",
 621 |     "        y_pred = self.linear2(h1)\n",
 622 |     "        return y_pred"
 623 |    ]
 624 |   },
 625 |   {
 626 |    "cell_type": "markdown",
 627 |    "metadata": {},
 628 |    "source": [
 629 |     "## Construct the model by instantiating the class defined above"
 630 |    ]
 631 |   },
 632 |   {
 633 |    "cell_type": "code",
 634 |    "execution_count": null,
 635 |    "metadata": {
 636 |     "ExecuteTime": {
 637 |      "end_time": "2019-04-11T03:10:49.875428Z",
 638 |      "start_time": "2019-04-11T03:10:49.866520Z"
 639 |     }
 640 |    },
 641 |    "outputs": [],
 642 |    "source": [
 643 |     "# Construct the model by instantiating the class defined above\n",
 644 |     "D_in = X_train.shape[1]\n",
 645 |     "D_h1, D_h2 = 8,4\n",
 646 |     "D_out = 1\n",
 647 |     "\n",
 648 |     "ffnet = ffNet(D_in, D_h1, D_h2, D_out)\n",
 649 |     "print(ffnet)"
 650 |    ]
 651 |   },
 652 |   {
 653 |    "cell_type": "markdown",
 654 |    "metadata": {},
 655 |    "source": [
 656 |     "## Define loss function and optimization algorithm"
 657 |    ]
 658 |   },
 659 |   {
 660 |    "cell_type": "code",
 661 |    "execution_count": null,
 662 |    "metadata": {
 663 |     "ExecuteTime": {
 664 |      "end_time": "2019-04-11T03:10:49.884274Z",
 665 |      "start_time": "2019-04-11T03:10:49.878488Z"
 666 |     }
 667 |    },
 668 |    "outputs": [],
 669 |    "source": [
 670 |     "# Define Optimizer and Loss Function\n",
 671 |     "\n",
 672 |     "#optimizer = torch.optim.SGD(ffnet.parameters(), lr=0.01, weight_decay=1e-4)\n",
 673 |     "optimizer = torch.optim.SGD(ffnet.parameters(), lr=0.01)\n",
 674 |     "\n",
 675 |     "loss_func = torch.nn.MSELoss()"
 676 |    ]
 677 |   },
 678 |   {
 679 |    "cell_type": "markdown",
 680 |    "metadata": {},
 681 |    "source": [
 682 |     "## Model Training"
 683 |    ]
 684 |   },
 685 |   {
 686 |    "cell_type": "code",
 687 |    "execution_count": null,
 688 |    "metadata": {
 689 |     "ExecuteTime": {
 690 |      "end_time": "2019-04-11T03:10:52.557954Z",
 691 |      "start_time": "2019-04-11T03:10:49.889285Z"
 692 |     },
 693 |     "scrolled": false
 694 |    },
 695 |    "outputs": [],
 696 |    "source": [
 697 |     "X_data = X_train_ts\n",
 698 |     "y_data = y_train_ts\n",
 699 |     "\n",
 700 |     "X_data_val = X_test_ts\n",
 701 |     "y_data_val = y_test_ts\n",
 702 |     "\n",
 703 |     "n_epoch = 500\n",
 704 |     "\n",
 705 |     "train_loss, val_loss = [],[]\n",
 706 |     "\n",
 707 |     "for epoch in range(1, n_epoch + 1):\n",
 708 |     "    y_pred = ffnet(X_data)    \n",
 709 |     "    epoch_loss_train = loss_func(y_pred, y_data) \n",
 710 |     "    \n",
 711 |     "    y_pred_val = ffnet(X_data_val)\n",
 712 |     "    epoch_loss_val = loss_func(y_pred_val, y_data_val)     \n",
 713 |     "    \n",
 714 |     "    optimizer.zero_grad()\n",
 715 |     "    epoch_loss_train.backward()        \n",
 716 |     "    optimizer.step()       \n",
 717 |     "    \n",
 718 |     "    train_loss.append(epoch_loss_train)\n",
 719 |     "    val_loss.append(epoch_loss_val)\n",
 720 |     "    \n",
 721 |     "    #if epoch <= 5 or epoch % 100 == 0:\n",
 722 |     "    if epoch % 100 == 0:\n",
 723 |     "        print('Epoch {}, Training loss {}, Validation loss {}'.format(epoch, round(float(epoch_loss_train),4), round(float(epoch_loss_val),4)))\n",
 724 |     "        \n",
 725 |     "        #plt.cla()\n",
 726 |     "        \n",
 727 |     "        fig = plt.figure(figsize=(16,6))\n",
 728 |     "        plt.subplot(1, 2, 1)\n",
 729 |     "        plt.scatter(X_train.iloc[:,0], y_train, label=\"actual\")\n",
 730 |     "        plt.scatter(X_train.iloc[:,0], y_pred.detach().numpy(), label=\"prediction\")\n",
 731 |     "        plt.title(\"Training\")\n",
 732 |     "        plt.legend()\n",
 733 |     "\n",
 734 |     "        plt.subplot(1, 2, 2)\n",
 735 |     "        plt.scatter(X_test.iloc[:,0], y_test, label=\"actual\")\n",
 736 |     "        plt.scatter(X_test.iloc[:,0], y_pred_val.detach().numpy(),label=\"prediction\")\n",
 737 |     "        plt.title(\"Validation\")\n",
 738 |     "        plt.legend()\n",
 739 |     "        \n",
 740 |     "        #plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 10, 'color':  'red'})\n",
 741 |     "        #plt.pause(0.1)\n",
 742 |     "\n",
 743 |     "        plt.show()"
 744 |    ]
 745 |   },
 746 |   {
 747 |    "cell_type": "markdown",
 748 |    "metadata": {},
 749 |    "source": [
 750 |     "## Plot loss curves"
 751 |    ]
 752 |   },
 753 |   {
 754 |    "cell_type": "code",
 755 |    "execution_count": null,
 756 |    "metadata": {
 757 |     "ExecuteTime": {
 758 |      "end_time": "2019-04-11T03:10:52.848996Z",
 759 |      "start_time": "2019-04-11T03:10:52.560684Z"
 760 |     },
 761 |     "scrolled": false
 762 |    },
 763 |    "outputs": [],
 764 |    "source": [
 765 |     "fig = plt.figure(figsize=(12,8))\n",
 766 |     "plt.plot(range(1,len(train_loss)+1),train_loss,'b',label = 'training loss')\n",
 767 |     "plt.plot(range(1,len(val_loss)+1),val_loss,'g',label = 'validation loss')\n",
 768 |     "plt.legend()\n",
 769 |     "plt.title(\"Training and Validation Loss Curves\")"
 770 |    ]
 771 |   },
 772 |   {
 773 |    "cell_type": "markdown",
 774 |    "metadata": {},
 775 |    "source": [
 776 |     "# Lab 3"
 777 |    ]
 778 |   },
 779 |   {
 780 |    "cell_type": "markdown",
 781 |    "metadata": {},
 782 |    "source": [
 783 |     "In this lab, you will replicate the above exercise with a different simulated dataset. \n",
 784 |     "\n",
 785 |     "  - You will simply run the codes to simulate the data and conduct some descriptive analysis.\n",
 786 |     "\n",
 787 |     "  - Then, you will design a network, instantiate it, run the training loop, and evaluate the model by plotting computing the loss of the training and validation sets."
 788 |    ]
 789 |   },
 790 |   {
 791 |    "cell_type": "markdown",
 792 |    "metadata": {},
 793 |    "source": [
 794 |     "## Simulate and Visualize Data"
 795 |    ]
 796 |   },
 797 |   {
 798 |    "cell_type": "code",
 799 |    "execution_count": null,
 800 |    "metadata": {
 801 |     "ExecuteTime": {
 802 |      "end_time": "2019-04-11T03:16:46.617514Z",
 803 |      "start_time": "2019-04-11T03:16:46.602265Z"
 804 |     }
 805 |    },
 806 |    "outputs": [],
 807 |    "source": [
 808 |     "import numpy as np\n",
 809 |     "import pandas as pd\n",
 810 |     "import matplotlib.pyplot as plt\n",
 811 |     "import seaborn as sns\n",
 812 |     "%matplotlib inline\n",
 813 |     "\n",
 814 |     "n = 200 # number of examples (or observations)\n",
 815 |     "\n",
 816 |     "# Generate a set of n random numbers from a standard normal distribution\n",
 817 |     "epsilon = np.random.randn(n)\n",
 818 |     "\n",
 819 |     "# Generate a set of n random numbers from a uniform[0,1] distribution; name it x1\n",
 820 |     "# and create another variable, which we name x2, from x1\n",
 821 |     "x1 = np.random.uniform(0,1,n)\n",
 822 |     "\n",
 823 |     "X = pd.DataFrame({'x1':x1})\n",
 824 |     "\n",
 825 |     "# Create the data generating mechanism\n",
 826 |     "y = 1.5*x1 + np.power(x1,3)*np.exp(x1) + epsilon"
 827 |    ]
 828 |   },
 829 |   {
 830 |    "cell_type": "markdown",
 831 |    "metadata": {},
 832 |    "source": [
 833 |     "## Split the data into a Training and Validation Set"
 834 |    ]
 835 |   },
 836 |   {
 837 |    "cell_type": "code",
 838 |    "execution_count": null,
 839 |    "metadata": {
 840 |     "ExecuteTime": {
 841 |      "end_time": "2019-04-11T03:16:54.276737Z",
 842 |      "start_time": "2019-04-11T03:16:54.271811Z"
 843 |     }
 844 |    },
 845 |    "outputs": [],
 846 |    "source": [
 847 |     "from sklearn.model_selection import train_test_split\n",
 848 |     "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.5)"
 849 |    ]
 850 |   },
 851 |   {
 852 |    "cell_type": "markdown",
 853 |    "metadata": {},
 854 |    "source": [
 855 |     "### Basic Descriptive Analysis"
 856 |    ]
 857 |   },
 858 |   {
 859 |    "cell_type": "code",
 860 |    "execution_count": null,
 861 |    "metadata": {
 862 |     "ExecuteTime": {
 863 |      "end_time": "2019-04-11T03:16:55.938611Z",
 864 |      "start_time": "2019-04-11T03:16:55.912778Z"
 865 |     }
 866 |    },
 867 |    "outputs": [],
 868 |    "source": [
 869 |     "print(X_train.describe().T.round(4), \"\\n\")\n",
 870 |     "print(pd.DataFrame({'y':y_train}).describe().T.round(4))"
 871 |    ]
 872 |   },
 873 |   {
 874 |    "cell_type": "code",
 875 |    "execution_count": null,
 876 |    "metadata": {
 877 |     "ExecuteTime": {
 878 |      "end_time": "2019-04-11T03:18:03.534505Z",
 879 |      "start_time": "2019-04-11T03:18:02.739133Z"
 880 |     }
 881 |    },
 882 |    "outputs": [],
 883 |    "source": [
 884 |     "fig = plt.figure(figsize=(15,10))\n",
 885 |     "\n",
 886 |     "plt.subplot(2, 2, 1)\n",
 887 |     "plt.hist(y_train)\n",
 888 |     "plt.title(\"y (Training Set)\")\n",
 889 |     "\n",
 890 |     "plt.subplot(2, 2, 2)\n",
 891 |     "plt.scatter(X_train.iloc[:,0], y_train)\n",
 892 |     "plt.title(\"y vs x1 (Training Set)\")\n",
 893 |     "\n",
 894 |     "plt.subplot(2, 2, 4)\n",
 895 |     "plt.hist(X_train.iloc[:,0])\n",
 896 |     "plt.title(\"x1 (Training Set)\")\n",
 897 |     "\n",
 898 |     "plt.tight_layout()"
 899 |    ]
 900 |   },
 901 |   {
 902 |    "cell_type": "markdown",
 903 |    "metadata": {},
 904 |    "source": [
 905 |     "<div class=\"alert alert-block alert-warning\">\n",
 906 |     "Note: Before training, `numpy array` and `pandas dataframe` need to be converted to `PyTorch's tensors`\n",
 907 |     "</div>"
 908 |    ]
 909 |   },
 910 |   {
 911 |    "cell_type": "code",
 912 |    "execution_count": null,
 913 |    "metadata": {
 914 |     "ExecuteTime": {
 915 |      "end_time": "2019-04-11T03:19:11.141327Z",
 916 |      "start_time": "2019-04-11T03:19:11.134477Z"
 917 |     }
 918 |    },
 919 |    "outputs": [],
 920 |    "source": [
 921 |     "# convert numpy array to tensor in shape of input size\n",
 922 |     "import torch \n",
 923 |     "\n",
 924 |     "X_train_ts = torch.from_numpy(X_train.values.reshape(-1,1)).float()\n",
 925 |     "y_train_ts = torch.from_numpy(y_train.reshape(-1,1)).float()\n",
 926 |     "\n",
 927 |     "X_test_ts = torch.from_numpy(X_test.values.reshape(-1,1)).float()\n",
 928 |     "y_test_ts = torch.from_numpy(y_test.reshape(-1,1)).float() # y_test is a numpy array"
 929 |    ]
 930 |   },
 931 |   {
 932 |    "cell_type": "code",
 933 |    "execution_count": null,
 934 |    "metadata": {
 935 |     "ExecuteTime": {
 936 |      "end_time": "2019-04-11T03:19:11.587381Z",
 937 |      "start_time": "2019-04-11T03:19:11.579680Z"
 938 |     }
 939 |    },
 940 |    "outputs": [],
 941 |    "source": [
 942 |     "print(\"X_train_ts is of %s type\" %type(X_train_ts))\n",
 943 |     "print(\"y_train_ts is of %s type\" %type(y_train_ts))\n",
 944 |     "print(\"X_test_ts is of %s type\" %type(X_test_ts))\n",
 945 |     "print(\"y_test_ts is of %s type\" %type(y_test_ts))"
 946 |    ]
 947 |   },
 948 |   {
 949 |    "cell_type": "markdown",
 950 |    "metadata": {},
 951 |    "source": [
 952 |     "**Question: Do the shape of these tensors make sense to you?**"
 953 |    ]
 954 |   },
 955 |   {
 956 |    "cell_type": "code",
 957 |    "execution_count": null,
 958 |    "metadata": {
 959 |     "ExecuteTime": {
 960 |      "end_time": "2019-04-11T03:19:12.467957Z",
 961 |      "start_time": "2019-04-11T03:19:12.461054Z"
 962 |     }
 963 |    },
 964 |    "outputs": [],
 965 |    "source": [
 966 |     "print(X_train_ts.shape)\n",
 967 |     "print(y_train_ts.shape)\n",
 968 |     "print(X_test_ts.shape)\n",
 969 |     "print(y_test_ts.shape)"
 970 |    ]
 971 |   },
 972 |   {
 973 |    "cell_type": "markdown",
 974 |    "metadata": {},
 975 |    "source": [
 976 |     "## Define a Feed-forward network with 1 hidden layers"
 977 |    ]
 978 |   },
 979 |   {
 980 |    "cell_type": "markdown",
 981 |    "metadata": {},
 982 |    "source": [
 983 |     "**Question: Do the shape in the following tensors confirm your expectation?**"
 984 |    ]
 985 |   },
 986 |   {
 987 |    "cell_type": "code",
 988 |    "execution_count": null,
 989 |    "metadata": {
 990 |     "ExecuteTime": {
 991 |      "end_time": "2019-04-11T03:22:37.613546Z",
 992 |      "start_time": "2019-04-11T03:22:37.605072Z"
 993 |     }
 994 |    },
 995 |    "outputs": [],
 996 |    "source": [
 997 |     "# Let's confirm the dimensions of the inputs and outpu\n",
 998 |     "print(\"train_size: \", len(X_train_ts), \"\\n\")\n",
 999 |     "print(\"X shape:\", X_train_ts.shape, \"\\n\")\n",
1000 |     "print(\"y shape:\", y_train_ts.shape, \"\\n\")\n",
1001 |     "print(X_train_ts.shape[1])"
1002 |    ]
1003 |   },
1004 |   {
1005 |    "cell_type": "markdown",
1006 |    "metadata": {},
1007 |    "source": [
1008 |     "**Run the following code to design a one hidden-layer feedforward network.**"
1009 |    ]
1010 |   },
1011 |   {
1012 |    "cell_type": "code",
1013 |    "execution_count": null,
1014 |    "metadata": {
1015 |     "ExecuteTime": {
1016 |      "end_time": "2019-04-11T03:22:51.808779Z",
1017 |      "start_time": "2019-04-11T03:22:51.797935Z"
1018 |     }
1019 |    },
1020 |    "outputs": [],
1021 |    "source": [
1022 |     "from __future__ import print_function\n",
1023 |     "\n",
1024 |     "import torch\n",
1025 |     "import torch.nn as nn\n",
1026 |     "import torch.nn.functional as F\n",
1027 |     "from torch.autograd import Variable\n",
1028 |     "\n",
1029 |     "class ffNet(nn.Module):\n",
1030 |     "    \"\"\"\n",
1031 |     "    D_in: input dimension\n",
1032 |     "    D_h1: dimension of hidden layer 1\n",
1033 |     "    D_out: output dimension\n",
1034 |     "    \"\"\"\n",
1035 |     "    \n",
1036 |     "    def __init__(self, D_in, D_h1, D_out):\n",
1037 |     "        super(ffNet, self).__init__()\n",
1038 |     "        self.linear1 = torch.nn.Linear(D_in, D_h1)\n",
1039 |     "        self.linear2 = torch.nn.Linear(D_h1, D_out)\n",
1040 |     "\n",
1041 |     "    def forward(self, x):\n",
1042 |     "        h1 = F.relu(self.linear1(x))\n",
1043 |     "        y_pred = self.linear2(h1)\n",
1044 |     "        return y_pred"
1045 |    ]
1046 |   },
1047 |   {
1048 |    "cell_type": "markdown",
1049 |    "metadata": {},
1050 |    "source": [
1051 |     "## Construct the model by instantiating the class defined above"
1052 |    ]
1053 |   },
1054 |   {
1055 |    "cell_type": "code",
1056 |    "execution_count": null,
1057 |    "metadata": {
1058 |     "ExecuteTime": {
1059 |      "end_time": "2019-04-11T03:22:57.818421Z",
1060 |      "start_time": "2019-04-11T03:22:57.812728Z"
1061 |     }
1062 |    },
1063 |    "outputs": [],
1064 |    "source": [
1065 |     "X_train.shape[1]"
1066 |    ]
1067 |   },
1068 |   {
1069 |    "cell_type": "markdown",
1070 |    "metadata": {},
1071 |    "source": [
1072 |     "**Choose your parameters**"
1073 |    ]
1074 |   },
1075 |   {
1076 |    "cell_type": "code",
1077 |    "execution_count": null,
1078 |    "metadata": {
1079 |     "ExecuteTime": {
1080 |      "end_time": "2019-04-11T03:23:14.005390Z",
1081 |      "start_time": "2019-04-11T03:23:13.998497Z"
1082 |     }
1083 |    },
1084 |    "outputs": [],
1085 |    "source": [
1086 |     "# Construct the model by instantiating the class defined above\n",
1087 |     "D_in  = 0 # Replace 0 with your code here\n",
1088 |     "D_h1  = 0 # Replace 0 with an integer\n",
1089 |     "D_out = 0 # Replace 0 with the dimension of the output\n",
1090 |     "\n",
1091 |     "ffnet = ffNet(D_in, D_h1, D_out)\n",
1092 |     "print(ffnet)"
1093 |    ]
1094 |   },
1095 |   {
1096 |    "cell_type": "markdown",
1097 |    "metadata": {},
1098 |    "source": [
1099 |     "## Define loss function and optimization algorithm"
1100 |    ]
1101 |   },
1102 |   {
1103 |    "cell_type": "markdown",
1104 |    "metadata": {},
1105 |    "source": [
1106 |     "**You can specify the learning rate if you want.**"
1107 |    ]
1108 |   },
1109 |   {
1110 |    "cell_type": "code",
1111 |    "execution_count": null,
1112 |    "metadata": {
1113 |     "ExecuteTime": {
1114 |      "end_time": "2019-04-11T03:23:18.073980Z",
1115 |      "start_time": "2019-04-11T03:23:18.067792Z"
1116 |     }
1117 |    },
1118 |    "outputs": [],
1119 |    "source": [
1120 |     "# Define Optimizer and Loss Function\n",
1121 |     "\n",
1122 |     "learning_rate = 0.01 #You can specify the learning rate\n",
1123 |     "\n",
1124 |     "optimizer = torch.optim.SGD(ffnet.parameters(), lr=learning_rate)\n",
1125 |     "\n",
1126 |     "loss_func = torch.nn.MSELoss()"
1127 |    ]
1128 |   },
1129 |   {
1130 |    "cell_type": "markdown",
1131 |    "metadata": {},
1132 |    "source": [
1133 |     "## Model Training"
1134 |    ]
1135 |   },
1136 |   {
1137 |    "cell_type": "markdown",
1138 |    "metadata": {},
1139 |    "source": [
1140 |     "**1. Choose the number of epochs for your training**\n",
1141 |     "**2. Fill in some details in the code below.**"
1142 |    ]
1143 |   },
1144 |   {
1145 |    "cell_type": "code",
1146 |    "execution_count": null,
1147 |    "metadata": {
1148 |     "ExecuteTime": {
1149 |      "end_time": "2019-04-11T03:23:49.190594Z",
1150 |      "start_time": "2019-04-11T03:23:43.707774Z"
1151 |     },
1152 |     "scrolled": false
1153 |    },
1154 |    "outputs": [],
1155 |    "source": [
1156 |     "X_data = X_train_ts\n",
1157 |     "y_data = y_train_ts\n",
1158 |     "\n",
1159 |     "X_data_val = X_test_ts\n",
1160 |     "y_data_val = y_test_ts\n",
1161 |     "\n",
1162 |     "n_epoch = 0 # Choose the number of epochs for your training\n",
1163 |     "\n",
1164 |     "train_loss, val_loss = [],[]\n",
1165 |     "\n",
1166 |     "for epoch in range(1, n_epoch + 1):\n",
1167 |     "    y_pred = ffnet(A) # Fill in the blank by replacing A   \n",
1168 |     "    epoch_loss_train = loss_func(y_pred, y_data) \n",
1169 |     "    \n",
1170 |     "    y_pred_val = ffnet(B) # Fill in the blank by replacing B\n",
1171 |     "    epoch_loss_val = loss_func(C, D) # Fill in the blank by replacing C and D\n",
1172 |     "    \n",
1173 |     "    optimizer.zero_grad()\n",
1174 |     "    epoch_loss_train.backward()        \n",
1175 |     "    optimizer.step()       \n",
1176 |     "    \n",
1177 |     "    train_loss.append(epoch_loss_train)\n",
1178 |     "    val_loss.append(epoch_loss_val)\n",
1179 |     "    \n",
1180 |     "    #if epoch <= 5 or epoch % 100 == 0:\n",
1181 |     "    if epoch % 100 == 0:\n",
1182 |     "        print('Epoch {}, Training loss {}, Validation loss {}'.format(epoch, round(float(epoch_loss_train),4), round(float(epoch_loss_val),4)))\n",
1183 |     "                \n",
1184 |     "        fig = plt.figure(figsize=(16,6))\n",
1185 |     "        plt.subplot(1, 2, 1)\n",
1186 |     "        plt.scatter(X_train.iloc[:,0], y_train, label=\"actual\")\n",
1187 |     "        plt.scatter(X_train.iloc[:,0], y_pred.detach().numpy(), label=\"prediction\")\n",
1188 |     "        plt.title(\"Training\")\n",
1189 |     "        plt.legend()\n",
1190 |     "\n",
1191 |     "        plt.subplot(1, 2, 2)\n",
1192 |     "        plt.scatter(X_test.iloc[:,0], y_test, label=\"actual\")\n",
1193 |     "        plt.scatter(X_test.iloc[:,0], y_pred_val.detach().numpy(),label=\"prediction\")\n",
1194 |     "        plt.title(\"Validation\")\n",
1195 |     "        plt.legend()\n",
1196 |     "        \n",
1197 |     "        plt.show()"
1198 |    ]
1199 |   },
1200 |   {
1201 |    "cell_type": "markdown",
1202 |    "metadata": {},
1203 |    "source": [
1204 |     "## Plot loss curves"
1205 |    ]
1206 |   },
1207 |   {
1208 |    "cell_type": "code",
1209 |    "execution_count": null,
1210 |    "metadata": {
1211 |     "ExecuteTime": {
1212 |      "end_time": "2019-04-11T03:23:56.504952Z",
1213 |      "start_time": "2019-04-11T03:23:56.146179Z"
1214 |     }
1215 |    },
1216 |    "outputs": [],
1217 |    "source": [
1218 |     "fig = plt.figure(figsize=(12,8))\n",
1219 |     "plt.plot(range(1,len(train_loss)+1),train_loss,'b',label = 'training loss')\n",
1220 |     "plt.plot(range(1,len(val_loss)+1),val_loss,'g',label = 'validation loss')\n",
1221 |     "plt.legend()\n",
1222 |     "plt.title(\"Training and Validation Loss Curves\")"
1223 |    ]
1224 |   },
1225 |   {
1226 |    "cell_type": "markdown",
1227 |    "metadata": {},
1228 |    "source": [
1229 |     "# Feedforward Network With a Dropout Layer"
1230 |    ]
1231 |   },
1232 |   {
1233 |    "cell_type": "markdown",
1234 |    "metadata": {},
1235 |    "source": [
1236 |     "**Specify the dropout layer.**"
1237 |    ]
1238 |   },
1239 |   {
1240 |    "cell_type": "code",
1241 |    "execution_count": null,
1242 |    "metadata": {
1243 |     "ExecuteTime": {
1244 |      "end_time": "2019-04-11T03:24:32.979044Z",
1245 |      "start_time": "2019-04-11T03:24:32.961831Z"
1246 |     }
1247 |    },
1248 |    "outputs": [],
1249 |    "source": [
1250 |     "from __future__ import print_function\n",
1251 |     "\n",
1252 |     "import torch\n",
1253 |     "import torch.nn as nn\n",
1254 |     "import torch.nn.functional as F\n",
1255 |     "from torch.autograd import Variable\n",
1256 |     "\n",
1257 |     "class ffNet(nn.Module):\n",
1258 |     "    \"\"\"\n",
1259 |     "    D_in: input dimension\n",
1260 |     "    D_h1: dimension of hidden layer 1\n",
1261 |     "    D_out: output dimension\n",
1262 |     "    \"\"\"\n",
1263 |     "    \n",
1264 |     "    def __init__(self, D_in, D_h1, D_out):\n",
1265 |     "        super(ffNet, self).__init__()\n",
1266 |     "        self.linear1 = torch.nn.Linear(D_in, D_h1)\n",
1267 |     "        self.linear1_drop = DROPOUT_LAYER # Specify the dropout layer by replacing DROPOUT_LAYER\n",
1268 |     "        self.linear2 = torch.nn.Linear(D_h1, D_out)\n",
1269 |     "\n",
1270 |     "    def forward(self, x):\n",
1271 |     "        h1 = F.relu(self.linear1(x))\n",
1272 |     "        y_pred = self.linear2(h1)\n",
1273 |     "        return y_pred"
1274 |    ]
1275 |   },
1276 |   {
1277 |    "cell_type": "markdown",
1278 |    "metadata": {},
1279 |    "source": [
1280 |     "## Construct the model by instantiating the class defined above"
1281 |    ]
1282 |   },
1283 |   {
1284 |    "cell_type": "markdown",
1285 |    "metadata": {},
1286 |    "source": [
1287 |     "**Specify the dimensions.**"
1288 |    ]
1289 |   },
1290 |   {
1291 |    "cell_type": "code",
1292 |    "execution_count": null,
1293 |    "metadata": {
1294 |     "ExecuteTime": {
1295 |      "end_time": "2019-04-11T03:24:46.707041Z",
1296 |      "start_time": "2019-04-11T03:24:46.700576Z"
1297 |     }
1298 |    },
1299 |    "outputs": [],
1300 |    "source": [
1301 |     "# Construct the model by instantiating the class defined above\n",
1302 |     "D_in  = 0 # Specify D_in\n",
1303 |     "D_h1  = 0 # Specify D_h1\n",
1304 |     "D_out = 0 # Specify D_out\n",
1305 |     "\n",
1306 |     "ffnet = ffNet(D_in, D_h1, D_out)\n",
1307 |     "print(ffnet)"
1308 |    ]
1309 |   },
1310 |   {
1311 |    "cell_type": "markdown",
1312 |    "metadata": {},
1313 |    "source": [
1314 |     "## Define loss function and optimization algorithm"
1315 |    ]
1316 |   },
1317 |   {
1318 |    "cell_type": "markdown",
1319 |    "metadata": {},
1320 |    "source": [
1321 |     "**Change the learning rate.**"
1322 |    ]
1323 |   },
1324 |   {
1325 |    "cell_type": "code",
1326 |    "execution_count": null,
1327 |    "metadata": {
1328 |     "ExecuteTime": {
1329 |      "end_time": "2019-04-11T03:24:57.014872Z",
1330 |      "start_time": "2019-04-11T03:24:57.009627Z"
1331 |     }
1332 |    },
1333 |    "outputs": [],
1334 |    "source": [
1335 |     "# Define Optimizer and Loss Function\n",
1336 |     "\n",
1337 |     "\n",
1338 |     "optimizer = torch.optim.SGD(ffnet.parameters(), lr=1e-6) # Change the learning rate \n",
1339 |     "\n",
1340 |     "loss_func = torch.nn.MSELoss()"
1341 |    ]
1342 |   },
1343 |   {
1344 |    "cell_type": "markdown",
1345 |    "metadata": {},
1346 |    "source": [
1347 |     "## Model Training"
1348 |    ]
1349 |   },
1350 |   {
1351 |    "cell_type": "code",
1352 |    "execution_count": null,
1353 |    "metadata": {
1354 |     "ExecuteTime": {
1355 |      "end_time": "2019-04-11T03:25:11.970793Z",
1356 |      "start_time": "2019-04-11T03:25:06.273365Z"
1357 |     },
1358 |     "scrolled": false
1359 |    },
1360 |    "outputs": [],
1361 |    "source": [
1362 |     "X_data = X_train_ts\n",
1363 |     "y_data = y_train_ts\n",
1364 |     "\n",
1365 |     "X_data_val = X_test_ts\n",
1366 |     "y_data_val = y_test_ts\n",
1367 |     "\n",
1368 |     "n_epoch = 1000\n",
1369 |     "\n",
1370 |     "train_loss, val_loss = [],[]\n",
1371 |     "\n",
1372 |     "for epoch in range(1, n_epoch + 1):\n",
1373 |     "    y_pred = ffnet(X_data)    \n",
1374 |     "    epoch_loss_train = loss_func(y_pred, y_data) \n",
1375 |     "    \n",
1376 |     "    y_pred_val = ffnet(X_data_val)\n",
1377 |     "    epoch_loss_val = loss_func(y_pred_val, y_data_val)     \n",
1378 |     "    \n",
1379 |     "    optimizer.zero_grad()\n",
1380 |     "    epoch_loss_train.backward()        \n",
1381 |     "    optimizer.step()       \n",
1382 |     "    \n",
1383 |     "    train_loss.append(epoch_loss_train)\n",
1384 |     "    val_loss.append(epoch_loss_val)\n",
1385 |     "    \n",
1386 |     "    #if epoch <= 5 or epoch % 100 == 0:\n",
1387 |     "    if epoch % 100 == 0:\n",
1388 |     "        print('Epoch {}, Training loss {}, Validation loss {}'.format(epoch, round(float(epoch_loss_train),4), round(float(epoch_loss_val),4)))\n",
1389 |     "        \n",
1390 |     "        #plt.cla()\n",
1391 |     "        \n",
1392 |     "        fig = plt.figure(figsize=(16,6))\n",
1393 |     "        plt.subplot(1, 2, 1)\n",
1394 |     "        plt.scatter(X_train.iloc[:,0], y_train, label=\"actual\")\n",
1395 |     "        plt.scatter(X_train.iloc[:,0], y_pred.detach().numpy(), label=\"prediction\")\n",
1396 |     "        plt.title(\"Training\")\n",
1397 |     "        plt.legend()\n",
1398 |     "\n",
1399 |     "        plt.subplot(1, 2, 2)\n",
1400 |     "        plt.scatter(X_test.iloc[:,0], y_test, label=\"actual\")\n",
1401 |     "        plt.scatter(X_test.iloc[:,0], y_pred_val.detach().numpy(),label=\"prediction\")\n",
1402 |     "        plt.title(\"Validation\")\n",
1403 |     "        plt.legend()\n",
1404 |     "        \n",
1405 |     "        #plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 10, 'color':  'red'})\n",
1406 |     "        #plt.pause(0.1)\n",
1407 |     "\n",
1408 |     "        plt.show()"
1409 |    ]
1410 |   },
1411 |   {
1412 |    "cell_type": "markdown",
1413 |    "metadata": {},
1414 |    "source": [
1415 |     "## Plot loss curves"
1416 |    ]
1417 |   },
1418 |   {
1419 |    "cell_type": "code",
1420 |    "execution_count": null,
1421 |    "metadata": {
1422 |     "ExecuteTime": {
1423 |      "end_time": "2019-04-11T03:25:23.339872Z",
1424 |      "start_time": "2019-04-11T03:25:22.998287Z"
1425 |     },
1426 |     "scrolled": false
1427 |    },
1428 |    "outputs": [],
1429 |    "source": [
1430 |     "fig = plt.figure(figsize=(12,8))\n",
1431 |     "plt.plot(range(1,len(train_loss)+1),train_loss,'b',label = 'training loss')\n",
1432 |     "plt.plot(range(1,len(val_loss)+1),val_loss,'g',label = 'validation loss')\n",
1433 |     "plt.legend()\n",
1434 |     "plt.title(\"Training and Validation Loss Curves\")"
1435 |    ]
1436 |   },
1437 |   {
1438 |    "cell_type": "code",
1439 |    "execution_count": null,
1440 |    "metadata": {},
1441 |    "outputs": [],
1442 |    "source": []
1443 |   }
1444 |  ],
1445 |  "metadata": {
1446 |   "kernelspec": {
1447 |    "display_name": "Python [conda env:dl]",
1448 |    "language": "python",
1449 |    "name": "conda-env-dl-py"
1450 |   },
1451 |   "language_info": {
1452 |    "codemirror_mode": {
1453 |     "name": "ipython",
1454 |     "version": 3
1455 |    },
1456 |    "file_extension": ".py",
1457 |    "mimetype": "text/x-python",
1458 |    "name": "python",
1459 |    "nbconvert_exporter": "python",
1460 |    "pygments_lexer": "ipython3",
1461 |    "version": "3.6.7"
1462 |   },
1463 |   "toc": {
1464 |    "base_numbering": 1,
1465 |    "nav_menu": {},
1466 |    "number_sections": true,
1467 |    "sideBar": true,
1468 |    "skip_h1_title": false,
1469 |    "title_cell": "Table of Contents",
1470 |    "title_sidebar": "Contents",
1471 |    "toc_cell": false,
1472 |    "toc_position": {},
1473 |    "toc_section_display": true,
1474 |    "toc_window_display": false
1475 |   },
1476 |   "varInspector": {
1477 |    "cols": {
1478 |     "lenName": 16,
1479 |     "lenType": 16,
1480 |     "lenVar": 40
1481 |    },
1482 |    "kernels_config": {
1483 |     "python": {
1484 |      "delete_cmd_postfix": "",
1485 |      "delete_cmd_prefix": "del ",
1486 |      "library": "var_list.py",
1487 |      "varRefreshCmd": "print(var_dic_list())"
1488 |     },
1489 |     "r": {
1490 |      "delete_cmd_postfix": ") ",
1491 |      "delete_cmd_prefix": "rm(",
1492 |      "library": "var_list.r",
1493 |      "varRefreshCmd": "cat(var_dic_list()) "
1494 |     }
1495 |    },
1496 |    "types_to_exclude": [
1497 |     "module",
1498 |     "function",
1499 |     "builtin_function_or_method",
1500 |     "instance",
1501 |     "_Feature"
1502 |    ],
1503 |    "window_display": false
1504 |   }
1505 |  },
1506 |  "nbformat": 4,
1507 |  "nbformat_minor": 2
1508 | }
1509 | 


--------------------------------------------------------------------------------
/Notebook/Section 4 - CNN Example.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "<div class=\"alert alert-block alert-info\">\n",
   8 |     "<font size=\"5\"><b><center> Section 4</font></center>\n",
   9 |     "<br>\n",
  10 |     "<font size=\"5\"><b><center> Image classification Using a Convolutional Neural Retwork in PyTorch</font></center>\n",
  11 |     "</div>"
  12 |    ]
  13 |   },
  14 |   {
  15 |    "cell_type": "markdown",
  16 |    "metadata": {},
  17 |    "source": [
  18 |     "Credit: This example is obtained from the following book:\n",
  19 |     "\n",
  20 |     "Subramanian, Vishnu. 2018. \"*Deep Learning with PyTorch: A Practical Approach to Building Neural Network Models Using PyTorch.*\" Birmingham, U.K., Packt Publishing."
  21 |    ]
  22 |   },
  23 |   {
  24 |    "cell_type": "markdown",
  25 |    "metadata": {},
  26 |    "source": [
  27 |     "# Example 1: Classifying Hand-written Digits"
  28 |    ]
  29 |   },
  30 |   {
  31 |    "cell_type": "markdown",
  32 |    "metadata": {},
  33 |    "source": [
  34 |     "[The MNIST Database of Handwritten Digits](http://yann.lecun.com/exdb/mnist/)\n",
  35 |     "\n",
  36 |     "    * A training set of 60,000 examples\n",
  37 |     "    * A test set of 10,000 examples\n",
  38 |     "    * The digits have been size-normalized and centered in a fixed-size image"
  39 |    ]
  40 |   },
  41 |   {
  42 |    "cell_type": "code",
  43 |    "execution_count": 1,
  44 |    "metadata": {
  45 |     "ExecuteTime": {
  46 |      "end_time": "2019-04-07T22:51:29.077218Z",
  47 |      "start_time": "2019-04-07T22:51:27.273347Z"
  48 |     }
  49 |    },
  50 |    "outputs": [],
  51 |    "source": [
  52 |     "import torch\n",
  53 |     "import torch.nn as nn\n",
  54 |     "import torch.nn.functional as F\n",
  55 |     "import torch.optim as optim\n",
  56 |     "\n",
  57 |     "from torchvision import datasets,transforms\n",
  58 |     "from torch.autograd import Variable\n",
  59 |     "\n",
  60 |     "import matplotlib.pyplot as plt\n",
  61 |     "%matplotlib inline"
  62 |    ]
  63 |   },
  64 |   {
  65 |    "cell_type": "markdown",
  66 |    "metadata": {},
  67 |    "source": [
  68 |     "## Check if a CUDA compatible devise is available"
  69 |    ]
  70 |   },
  71 |   {
  72 |    "cell_type": "code",
  73 |    "execution_count": 2,
  74 |    "metadata": {
  75 |     "ExecuteTime": {
  76 |      "end_time": "2019-04-07T22:51:29.086417Z",
  77 |      "start_time": "2019-04-07T22:51:29.081068Z"
  78 |     }
  79 |    },
  80 |    "outputs": [],
  81 |    "source": [
  82 |     "is_cuda=False\n",
  83 |     "if torch.cuda.is_available():\n",
  84 |     "    is_cuda = True"
  85 |    ]
  86 |   },
  87 |   {
  88 |    "cell_type": "markdown",
  89 |    "metadata": {},
  90 |    "source": [
  91 |     "## Load, Transform, and Feed the Transformed Data into `DataLoader`"
  92 |    ]
  93 |   },
  94 |   {
  95 |    "cell_type": "code",
  96 |    "execution_count": 3,
  97 |    "metadata": {
  98 |     "ExecuteTime": {
  99 |      "end_time": "2019-04-07T22:51:32.557711Z",
 100 |      "start_time": "2019-04-07T22:51:29.090554Z"
 101 |     }
 102 |    },
 103 |    "outputs": [
 104 |     {
 105 |      "name": "stdout",
 106 |      "output_type": "stream",
 107 |      "text": [
 108 |       "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
 109 |       "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
 110 |       "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
 111 |       "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
 112 |       "Processing...\n",
 113 |       "Done!\n"
 114 |      ]
 115 |     }
 116 |    ],
 117 |    "source": [
 118 |     "transformation = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307,), (0.3081,))])\n",
 119 |     "\n",
 120 |     "train_dataset = datasets.MNIST('data/section4/',train=True,transform=transformation,download=True)\n",
 121 |     "test_dataset = datasets.MNIST('data/section4/',train=False,transform=transformation,download=True)\n",
 122 |     "\n",
 123 |     "train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=32,shuffle=True)\n",
 124 |     "test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=32,shuffle=True)"
 125 |    ]
 126 |   },
 127 |   {
 128 |    "cell_type": "markdown",
 129 |    "metadata": {},
 130 |    "source": [
 131 |     "## Examine the a batch of examples from the original data"
 132 |    ]
 133 |   },
 134 |   {
 135 |    "cell_type": "code",
 136 |    "execution_count": 4,
 137 |    "metadata": {
 138 |     "ExecuteTime": {
 139 |      "end_time": "2019-04-07T22:51:32.594456Z",
 140 |      "start_time": "2019-04-07T22:51:32.569677Z"
 141 |     }
 142 |    },
 143 |    "outputs": [],
 144 |    "source": [
 145 |     "sample_data = next(iter(train_loader))"
 146 |    ]
 147 |   },
 148 |   {
 149 |    "cell_type": "code",
 150 |    "execution_count": 5,
 151 |    "metadata": {
 152 |     "ExecuteTime": {
 153 |      "end_time": "2019-04-07T22:51:32.611083Z",
 154 |      "start_time": "2019-04-07T22:51:32.601326Z"
 155 |     }
 156 |    },
 157 |    "outputs": [
 158 |     {
 159 |      "data": {
 160 |       "text/plain": [
 161 |        "list"
 162 |       ]
 163 |      },
 164 |      "execution_count": 5,
 165 |      "metadata": {},
 166 |      "output_type": "execute_result"
 167 |     }
 168 |    ],
 169 |    "source": [
 170 |     "type(sample_data)"
 171 |    ]
 172 |   },
 173 |   {
 174 |    "cell_type": "code",
 175 |    "execution_count": 6,
 176 |    "metadata": {
 177 |     "ExecuteTime": {
 178 |      "end_time": "2019-04-07T22:51:32.623369Z",
 179 |      "start_time": "2019-04-07T22:51:32.614194Z"
 180 |     }
 181 |    },
 182 |    "outputs": [
 183 |     {
 184 |      "data": {
 185 |       "text/plain": [
 186 |        "2"
 187 |       ]
 188 |      },
 189 |      "execution_count": 6,
 190 |      "metadata": {},
 191 |      "output_type": "execute_result"
 192 |     }
 193 |    ],
 194 |    "source": [
 195 |     "len(sample_data)"
 196 |    ]
 197 |   },
 198 |   {
 199 |    "cell_type": "code",
 200 |    "execution_count": 7,
 201 |    "metadata": {
 202 |     "ExecuteTime": {
 203 |      "end_time": "2019-04-07T22:51:32.638445Z",
 204 |      "start_time": "2019-04-07T22:51:32.628056Z"
 205 |     }
 206 |    },
 207 |    "outputs": [
 208 |     {
 209 |      "name": "stdout",
 210 |      "output_type": "stream",
 211 |      "text": [
 212 |       "torch.Size([32, 1, 28, 28])\n",
 213 |       "torch.Size([32])\n"
 214 |      ]
 215 |     }
 216 |    ],
 217 |    "source": [
 218 |     "print(sample_data[0].size())\n",
 219 |     "print(sample_data[1].size())"
 220 |    ]
 221 |   },
 222 |   {
 223 |    "cell_type": "code",
 224 |    "execution_count": 8,
 225 |    "metadata": {
 226 |     "ExecuteTime": {
 227 |      "end_time": "2019-04-07T22:51:32.652934Z",
 228 |      "start_time": "2019-04-07T22:51:32.645245Z"
 229 |     }
 230 |    },
 231 |    "outputs": [
 232 |     {
 233 |      "data": {
 234 |       "text/plain": [
 235 |        "tensor([3, 0, 0, 3, 5, 2, 1, 7, 5, 4, 1, 0, 7, 3, 6, 9, 2, 2, 5, 2, 2, 6, 8, 1,\n",
 236 |        "        8, 4, 1, 8, 2, 7, 0, 8])"
 237 |       ]
 238 |      },
 239 |      "execution_count": 8,
 240 |      "metadata": {},
 241 |      "output_type": "execute_result"
 242 |     }
 243 |    ],
 244 |    "source": [
 245 |     "# The labels of the 32 images\n",
 246 |     "sample_data[1]"
 247 |    ]
 248 |   },
 249 |   {
 250 |    "cell_type": "code",
 251 |    "execution_count": 9,
 252 |    "metadata": {
 253 |     "ExecuteTime": {
 254 |      "end_time": "2019-04-07T22:51:32.677995Z",
 255 |      "start_time": "2019-04-07T22:51:32.655952Z"
 256 |     }
 257 |    },
 258 |    "outputs": [
 259 |     {
 260 |      "data": {
 261 |       "text/plain": [
 262 |        "tensor([[[-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 263 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 264 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 265 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 266 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 267 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 268 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 269 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 270 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 271 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 272 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 273 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 274 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 275 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 276 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 277 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 278 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 279 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 280 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 281 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 282 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 283 |        "          -0.4242, -0.4242,  0.1486,  1.1668,  1.6632,  2.8088,  2.8088,\n",
 284 |        "           2.8088,  1.6123,  0.0595, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 285 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 286 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 287 |        "           0.9759,  1.7141,  2.7578,  2.7960,  2.7960,  2.7960,  2.7960,\n",
 288 |        "           2.7960,  2.7960,  1.5996,  0.0722, -0.4242, -0.4242, -0.4242,\n",
 289 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 290 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  0.1995,\n",
 291 |        "           2.6687,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,\n",
 292 |        "           2.7960,  2.7960,  2.7960,  0.6450, -0.4242, -0.4242, -0.4242,\n",
 293 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 294 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.3097,\n",
 295 |        "           2.5797,  2.7960,  2.7960,  1.6632,  0.1104, -0.1824,  0.2504,\n",
 296 |        "           2.4651,  2.7960,  2.7960,  1.3196, -0.4242, -0.4242, -0.4242,\n",
 297 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 298 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 299 |        "          -0.1187, -0.1060, -0.1060, -0.3733, -0.4242, -0.4242, -0.4242,\n",
 300 |        "           0.4922,  2.7960,  2.7960,  2.2996, -0.4242, -0.4242, -0.4242,\n",
 301 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 302 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 303 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 304 |        "           0.3268,  2.7960,  2.7960,  2.2996, -0.4242, -0.4242, -0.4242,\n",
 305 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 306 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 307 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 308 |        "           0.3268,  2.7960,  2.7960,  2.2996, -0.4242, -0.4242, -0.4242,\n",
 309 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 310 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 311 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 312 |        "           1.6887,  2.7960,  2.7960,  1.7905, -0.4242, -0.4242, -0.4242,\n",
 313 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 314 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 315 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,  1.5614,\n",
 316 |        "           2.6306,  2.7960,  2.6306,  0.3395, -0.4242, -0.4242, -0.4242,\n",
 317 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 318 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 319 |        "          -0.0806,  0.4159,  0.4159,  1.8160,  1.5487,  1.7269,  2.6306,\n",
 320 |        "           2.7960,  2.7960,  0.9377, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 321 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 322 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 323 |        "           2.1469,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,  2.7960,\n",
 324 |        "           2.7960,  2.7960,  2.2360,  1.7269,  0.2377, -0.4242, -0.4242,\n",
 325 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 326 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 327 |        "           2.5669,  2.7960,  2.7960,  2.7960,  2.7960,  2.2614,  2.6433,\n",
 328 |        "           2.7960,  2.7960,  2.7960,  2.7960,  2.6815,  0.4668, -0.4242,\n",
 329 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 330 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 331 |        "           0.4922,  0.5686,  0.5686,  0.5686,  0.5686, -0.2206,  0.3395,\n",
 332 |        "           0.5686,  0.9123,  2.6178,  2.7960,  2.7960,  2.0196, -0.1824,\n",
 333 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 334 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 335 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 336 |        "          -0.4242, -0.4242,  0.6195,  2.6306,  2.7960,  2.7960,  0.8995,\n",
 337 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 338 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 339 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 340 |        "          -0.4242, -0.4242, -0.3478,  1.8669,  2.7960,  2.7960,  1.7396,\n",
 341 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 342 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 343 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 344 |        "          -0.4242, -0.3478,  1.2814,  2.7960,  2.7960,  2.7960,  0.8995,\n",
 345 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 346 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 347 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.2206,\n",
 348 |        "           0.2122,  1.7523,  2.7960,  2.7960,  2.7960,  2.4778,  0.1231,\n",
 349 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 350 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 351 |        "          -0.4242, -0.4242, -0.4242, -0.2715,  0.6068,  0.2249,  2.4778,\n",
 352 |        "           2.7960,  2.7960,  2.7960,  2.7706,  1.5614,  0.1104, -0.4242,\n",
 353 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 354 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 355 |        "          -0.4242, -0.4242, -0.4242,  2.2742,  2.7960,  2.7960,  2.7324,\n",
 356 |        "           2.7197,  2.7197,  2.4142,  0.8359, -0.4242, -0.4242, -0.4242,\n",
 357 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 358 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 359 |        "          -0.4242, -0.4242, -0.4242,  0.1104,  1.6505,  1.4723, -0.0678,\n",
 360 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 361 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 362 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 363 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 364 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 365 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 366 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 367 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 368 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 369 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242],\n",
 370 |        "         [-0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 371 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 372 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242,\n",
 373 |        "          -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242, -0.4242]]])"
 374 |       ]
 375 |      },
 376 |      "execution_count": 9,
 377 |      "metadata": {},
 378 |      "output_type": "execute_result"
 379 |     }
 380 |    ],
 381 |    "source": [
 382 |     "# the pixels representing in a a 28 x 28 matrix corresponding to the first image\n",
 383 |     "sample_data[0][0]"
 384 |    ]
 385 |   },
 386 |   {
 387 |    "cell_type": "code",
 388 |    "execution_count": 10,
 389 |    "metadata": {
 390 |     "ExecuteTime": {
 391 |      "end_time": "2019-04-07T22:51:32.698511Z",
 392 |      "start_time": "2019-04-07T22:51:32.691147Z"
 393 |     }
 394 |    },
 395 |    "outputs": [
 396 |     {
 397 |      "data": {
 398 |       "text/plain": [
 399 |        "tensor(3)"
 400 |       ]
 401 |      },
 402 |      "execution_count": 10,
 403 |      "metadata": {},
 404 |      "output_type": "execute_result"
 405 |     }
 406 |    ],
 407 |    "source": [
 408 |     "# the label corresponding to the first image\n",
 409 |     "sample_data[1][0]"
 410 |    ]
 411 |   },
 412 |   {
 413 |    "cell_type": "code",
 414 |    "execution_count": 11,
 415 |    "metadata": {
 416 |     "ExecuteTime": {
 417 |      "end_time": "2019-04-07T22:51:32.709136Z",
 418 |      "start_time": "2019-04-07T22:51:32.701929Z"
 419 |     }
 420 |    },
 421 |    "outputs": [],
 422 |    "source": [
 423 |     "def plot_img(image):\n",
 424 |     "    image = image.numpy()[0]\n",
 425 |     "    mean = 0.1307\n",
 426 |     "    std = 0.3081\n",
 427 |     "    image = ((mean * image) + std)\n",
 428 |     "    plt.imshow(image,cmap='gray')"
 429 |    ]
 430 |   },
 431 |   {
 432 |    "cell_type": "markdown",
 433 |    "metadata": {},
 434 |    "source": [
 435 |     "### Visualize the Labels"
 436 |    ]
 437 |   },
 438 |   {
 439 |    "cell_type": "code",
 440 |    "execution_count": 12,
 441 |    "metadata": {
 442 |     "ExecuteTime": {
 443 |      "end_time": "2019-04-07T22:51:33.256195Z",
 444 |      "start_time": "2019-04-07T22:51:32.712021Z"
 445 |     }
 446 |    },
 447 |    "outputs": [
 448 |     {
 449 |      "data": {
 450 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADX5JREFUeJzt3W+IVfedx/HPd219oobELTsZpm40\nJdko88BuBlmILC1djZGCUYg0eWJp6RRSyQr7oMn0wSaExBLWlj5qmKI4lta2xAyRIq2tlGpgKU5C\nNnFMNNli05EZ/8RCpxAwxm8f3DPtJM79nTv3nHvPGb/vFwxz7/nee86Xw3zm3HN/99yfubsAxPMP\nVTcAoBqEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUJ/o5sbMjI8TAh3m7tbK4wod+c1sk5md\nMbN3zOzxIusC0F3W7mf7zWyRpLOSNkiakHRS0sPufjrxHI78QId148i/TtI77v57d78q6SeSthRY\nH4AuKhL+Pkl/nHV/Ilv2EWY2aGZjZjZWYFsAStbxN/zcfVjSsMTLfqBOihz5z0taMev+p7NlABaA\nIuE/KekuM1tlZoslfUnS4XLaAtBpbb/sd/drZrZT0i8lLZK0z93HS+sMQEe1PdTX1sY45wc6risf\n8gGwcBF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQf\nCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVNtTdEuSmZ2T\nNC3pQ0nX3H2gjKai6e/vT9a3bt2arN9zzz1Na4888khbPbXqwIEDyfozzzzTtHb27Nmy28E8FAp/\n5vPufrmE9QDoIl72A0EVDb9LOmpmr5jZYBkNAeiOoi/717v7eTP7J0m/MrO33P347Adk/xT4xwDU\nTKEjv7ufz35flDQqad0cjxl29wHeDATqpe3wm9kSM1s2c1vSRkmnymoMQGcVednfI2nUzGbW82N3\n/0UpXQHoOHP37m3MrHsb66LHHnssWb/vvvuS9W3btiXrixYtmndPdZH6HMDu3buTzz1z5kzZ7YTg\n7tbK4xjqA4Ii/EBQhB8IivADQRF+ICjCDwTFUF+L1q5d27R24sSJ5HOXLFlSdjs3he3btyfrL7zw\nQpc6ubkw1AcgifADQRF+ICjCDwRF+IGgCD8QFOEHgirj23tDSF1eevXq1eRzi47zHzx4MFlPfT32\nhQsXks9dtWpVsj46Opqs9/X1Jesp69bd8MVPH8E4f2dx5AeCIvxAUIQfCIrwA0ERfiAowg8ERfiB\noLievwS33357sp7NbdC2S5cuJevXrl0rtP6UTZs2JetHjhxpe915X829evXqttcdGdfzA0gi/EBQ\nhB8IivADQRF+ICjCDwRF+IGgcq/nN7N9kr4o6aK792fLlkv6qaSVks5J2u7uf+pcm/U2NTVVdQsd\nMz09XXUL6JBWjvz7JX38kx6PSzrm7ndJOpbdB7CA5Ibf3Y9LuvKxxVskjWS3RyQ9WHJfADqs3XP+\nHnefzG5PSeopqR8AXVL4O/zc3VOf2TezQUmDRbcDoFztHvkvmFmvJGW/LzZ7oLsPu/uAuw+0uS0A\nHdBu+A9L2pHd3iHppXLaAdAtueE3s4OS/lfSv5jZhJl9VdK3JW0ws7cl/Ud2H8ACknvO7+4PNyl9\noeReAHQRn/ADgiL8QFCEHwiK8ANBEX4gKMIPBMUU3UjauXNnx9b91ltvdWzdyMeRHwiK8ANBEX4g\nKMIPBEX4gaAIPxAU4QeCYpw/uL179ybrmzdvLrT+1Fd/79mzp9C6UQxHfiAowg8ERfiBoAg/EBTh\nB4Ii/EBQhB8IinH+EvT19SXrd999d6H179q1K1m/dOlS09ott9ySfO7999+frC9btixZzzM0NNS0\n9vLLLxdaN4rhyA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQZm7px9gtk/SFyVddPf+bNmTkr4maWaA\necjdj+RuzCy9sQXq5MmTyfq9997bpU7q59SpU01red/b/9xzzyXrY2NjbfV0s3N3a+VxrRz590va\nNMfy77r72uwnN/gA6iU3/O5+XNKVLvQCoIuKnPPvNLPXzWyfmd1WWkcAuqLd8H9f0mckrZU0Kanp\nl7GZ2aCZjZkZJ2hAjbQVfne/4O4fuvt1ST+QtC7x2GF3H3D3gXabBFC+tsJvZr2z7m6V1PwtXQC1\nlHtJr5kdlPQ5SZ8yswlJ/y3pc2a2VpJLOifp6x3sEUAH5I7zl7qxm3Sc//Lly8n68uXLu9TJjcbH\nx5P1O+64I1lfunRpme3My/vvv5+sb926NVk/evRome0sGGWO8wO4CRF+ICjCDwRF+IGgCD8QFOEH\ngmKorwSbNs110ePfLV68uEud3Oj48ePJen9/f7KeN0yZ97Xlu3fvblrL+1rxPCdOnEjWt23b1rT2\n3nvvFdp2nTHUByCJ8ANBEX4gKMIPBEX4gaAIPxAU4QeCYpwfHZW67Hb//v3J5xadHvzRRx9tWnv+\n+ecLrbvOGOcHkET4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzo+OSn2XwejoaPK5DzzwQKFtHzp0qGnt\noYceKrTuOmOcH0AS4QeCIvxAUIQfCIrwA0ERfiAowg8E9Ym8B5jZCkkHJPVIcknD7v49M1su6aeS\nVko6J2m7u/+pc62ijvLmJHjqqaea1oqO4+c5duxYR9e/0LVy5L8m6b/cfY2kf5P0DTNbI+lxScfc\n/S5Jx7L7ABaI3PC7+6S7v5rdnpb0pqQ+SVskjWQPG5H0YKeaBFC+eZ3zm9lKSZ+V9DtJPe4+mZWm\n1DgtALBA5J7zzzCzpZIOSdrl7n82+/vHh93dm31u38wGJQ0WbRRAuVo68pvZJ9UI/o/c/cVs8QUz\n683qvZIuzvVcdx929wF3HyijYQDlyA2/NQ7xeyW96e7fmVU6LGlHdnuHpJfKbw9Ap+Re0mtm6yWd\nkPSGpOvZ4iE1zvt/JumfJf1BjaG+Kznr4pLemrn11luT9TvvvDNZHxoaStZT02QXNTk5maxv3Lix\naW18fLzsdmqj1Ut6c8/53f1lSc1W9oX5NAWgPviEHxAU4QeCIvxAUIQfCIrwA0ERfiColj/ei+os\nXbo0Wd+wYUPT2rvvvpt87uHDh5P13t7eZL2I69evJ+sjIyPJ+p49e5L106dPz7unSDjyA0ERfiAo\nwg8ERfiBoAg/EBThB4Ii/EBQTNG9AORde7569eq21/3BBx8k60X/PqampprWnn766eRz9+7dW2jb\nUTFFN4Akwg8ERfiBoAg/EBThB4Ii/EBQhB8IinH+BWDNmjXJ+hNPPNG0Nj09nXzus88+m6xPTEwk\n66gfxvkBJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFC54/xmtkLSAUk9klzSsLt/z8yelPQ1SZeyhw65\n+5GcdTHOD3RYq+P8rYS/V1Kvu79qZsskvSLpQUnbJf3F3f+n1aYIP9B5rYY/d8Yed5+UNJndnjaz\nNyX1FWsPQNXmdc5vZislfVbS77JFO83sdTPbZ2a3NXnOoJmNmdlYoU4BlKrlz/ab2VJJv5X0jLu/\naGY9ki6r8T7A02qcGnwlZx287Ac6rLRzfkkys09K+rmkX7r7d+aor5T0c3fvz1kP4Qc6rLQLe8zM\nJO2V9Obs4GdvBM7YKunUfJsEUJ1W3u1fL+mEpDckzcypPCTpYUlr1XjZf07S17M3B1Pr4sgPdFip\nL/vLQviBzuN6fgBJhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4\ngaByv8CzZJcl/WHW/U9ly+qorr3VtS+J3tpVZm93tPrArl7Pf8PGzcbcfaCyBhLq2ltd+5LorV1V\n9cbLfiAowg8EVXX4hyvefkpde6trXxK9tauS3io95wdQnaqP/AAqUkn4zWyTmZ0xs3fM7PEqemjG\nzM6Z2Rtm9lrVU4xl06BdNLNTs5YtN7Nfmdnb2e85p0mrqLcnzex8tu9eM7PNFfW2wsx+Y2anzWzc\nzP4zW17pvkv0Vcl+6/rLfjNbJOmspA2SJiSdlPSwu5/uaiNNmNk5SQPuXvmYsJn9u6S/SDowMxuS\nmT0n6Yq7fzv7x3mbu3+zJr09qXnO3Nyh3prNLP1lVbjvypzxugxVHPnXSXrH3X/v7lcl/UTSlgr6\nqD13Py7pyscWb5E0kt0eUeOPp+ua9FYL7j7p7q9mt6clzcwsXem+S/RViSrC3yfpj7PuT6heU367\npKNm9oqZDVbdzBx6Zs2MNCWpp8pm5pA7c3M3fWxm6drsu3ZmvC4bb/jdaL27/6ukByR9I3t5W0ve\nOGer03DN9yV9Ro1p3CYl7amymWxm6UOSdrn7n2fXqtx3c/RVyX6rIvznJa2Ydf/T2bJacPfz2e+L\nkkbVOE2pkwszk6Rmvy9W3M/fuPsFd//Q3a9L+oEq3HfZzNKHJP3I3V/MFle+7+bqq6r9VkX4T0q6\ny8xWmdliSV+SdLiCPm5gZkuyN2JkZkskbVT9Zh8+LGlHdnuHpJcq7OUj6jJzc7OZpVXxvqvdjNfu\n3vUfSZvVeMf//yV9q4oemvR1p6T/y37Gq+5N0kE1XgZ+oMZ7I1+V9I+Sjkl6W9KvJS2vUW8/VGM2\n59fVCFpvRb2tV+Ml/euSXst+Nle97xJ9VbLf+IQfEBRv+AFBEX4gKMIPBEX4gaAIPxAU4QeCIvxA\nUIQfCOqvpr9u0/bCLQUAAAAASUVORK5CYII=\n",
 451 |       "text/plain": [
 452 |        "<Figure size 432x288 with 1 Axes>"
 453 |       ]
 454 |      },
 455 |      "metadata": {},
 456 |      "output_type": "display_data"
 457 |     }
 458 |    ],
 459 |    "source": [
 460 |     "# Plot the first image\n",
 461 |     "plot_img(sample_data[0][0])"
 462 |    ]
 463 |   },
 464 |   {
 465 |    "cell_type": "code",
 466 |    "execution_count": 13,
 467 |    "metadata": {
 468 |     "ExecuteTime": {
 469 |      "end_time": "2019-04-07T22:51:33.547187Z",
 470 |      "start_time": "2019-04-07T22:51:33.271961Z"
 471 |     },
 472 |     "scrolled": true
 473 |    },
 474 |    "outputs": [
 475 |     {
 476 |      "data": {
 477 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADhxJREFUeJzt3XGMVeWZx/HfIxQTbTVq42QUlFp1\nTfUPMKOpCamtXQlLmiCJmqJRlhLHmJpszRpLWEdIpIHULWpCUkMjQjeVtioqIc1SJOvSlVIdDSso\ntroNpENGkCB0+EcW5ukfc6adwtz33rnn3HvO8Hw/yWTunOfe9zy99ce5577n3tfcXQDiOavsBgCU\ng/ADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwhqYjt3ZmZcTgi0mLtbI/fLdeQ3s1lm9nsz+8jM\nFuUZC0B7WbPX9pvZBEl/kHSrpD5Jb0ma5+7vJx7DkR9osXYc+W+U9JG7/9Hdj0v6uaQ5OcYD0EZ5\nwn+ppD+N+Lsv2/Z3zKzbzHrNrDfHvgAUrOVv+Ln7akmrJV72A1WS58i/X9KUEX9PzrYBGAfyhP8t\nSVeZ2ZfMbJKkb0vaWExbAFqt6Zf97n7CzB6UtFnSBElr3P29wjoD0FJNT/U1tTPO+YGWa8tFPgDG\nL8IPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+\nICjCDwRF+IGgCD8QFOEHgmrrEt0Yf6ZOnZqsP/TQQ8n6ggULataWLVvWTEt/9dRTTyXrx48fzzX+\nmY4jPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ElWuVXjPbK2lA0klJJ9y9q879WaW3YlasWJGsP/DA\nA8n6eeedV2Q7Y3L48OFk/frrr69Z27dvX9HtVEajq/QWcZHPN9z9UAHjAGgjXvYDQeUNv0v6tZm9\nbWbdRTQEoD3yvuyf4e77zexiSVvM7AN33zbyDtk/CvzDAFRMriO/u+/Pfh+U9LKkG0e5z2p376r3\nZiCA9mo6/GZ2rpl9Yfi2pJmSdhfVGIDWyvOyv0PSy2Y2PM7z7v6fhXQFoOVyzfOPeWfM87fd8uXL\nk/WHH344WT9y5EiyvmPHjjH3NOyss9IvPGfPnt302JL0yiuv1Kw98cQTycdu3749177L1Og8P1N9\nQFCEHwiK8ANBEX4gKMIPBEX4gaCY6hsHrrjiimT9nnvuqVlbsmRJ8rFbt25N1ut9Nffu3c1f11Vv\nqq9e748++mjT4+/cuTP52FtuuSVZ//TTT5P1MjHVByCJ8ANBEX4gKMIPBEX4gaAIPxAU4QeCYp6/\nAq699tpkfdOmTcl6ahnto0ePJh97++23J+uvvfZasl6mPB9Xnjgx/VUWqY8DS9K9996brA8MDCTr\nrcQ8P4Akwg8ERfiBoAg/EBThB4Ii/EBQhB8Iinn+Nqj3efx6n6lPzePXc/PNNyfr27ZtS9bHsz17\n9tSsXXPNNbnGXrBgQbK+du3aXOPnwTw/gCTCDwRF+IGgCD8QFOEHgiL8QFCEHwgq/aFmSWa2RtK3\nJB109+uybRdK+oWkqZL2SrrT3av7ReYlu/jii5P1PPP4Uvo6gTfeeCPX2ONZT09PzdoLL7zQxk6q\nqZEj/1pJs07ZtkjSVne/StLW7G8A40jd8Lv7NkmHT9k8R9K67PY6SbcV3BeAFmv2nL/D3fuz2x9L\n6iioHwBtUvecvx5399Q1+2bWLak7734AFKvZI/8BM+uUpOz3wVp3dPfV7t7l7l1N7gtACzQb/o2S\n5me350t6tZh2ALRL3fCb2XpJv5X0D2bWZ2YLJa2QdKuZfSjpH7O/AYwjdc/53X1ejdI3C+7ljLVi\nRWv/bXz99ddr1k6ePNnSfVfZhg0batZ6e3uTj+3qOvPPUrnCDwiK8ANBEX4gKMIPBEX4gaAIPxBU\n7st7IT3yyCPJ+k033ZRr/GPHjiXrTz/9dK7xz1SDg4NN1aLgyA8ERfiBoAg/EBThB4Ii/EBQhB8I\nivADQTHPX4B6yzVPmjQp1/grV65M1gcGBnKNH9GTTz6ZrK9fvz5Zv/rqq4tspxQc+YGgCD8QFOEH\ngiL8QFCEHwiK8ANBEX4gKOb5GzRr1qkLFf/NZZddlmvs48ePJ+u7du3KNT5O19/fX/9OCffdd1+y\nvnjx4lzjtwNHfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8Iqu48v5mtkfQtSQfd/bps21JJ90n6JLvb\nYnf/VauarILOzs6atXPOOSfX2H19fcn6iy++mGt8nO7888/P9fijR48W1El5Gjnyr5U02hUuT7r7\ntOznjA4+cCaqG3533ybpcBt6AdBGec75HzSzd81sjZldUFhHANqi2fD/WNKXJU2T1C/pR7XuaGbd\nZtZrZr1N7gtACzQVfnc/4O4n3X1Q0k8k3Zi472p373L3rmabBFC8psJvZiPf+p4raXcx7QBol0am\n+tZL+rqkL5pZn6Qlkr5uZtMkuaS9ku5vYY8AWqBu+N193iibn21BL5U2cWLrvvrgxIkTLRs7sgkT\nJtSs9fT05Bp72bJluR5fBVzhBwRF+IGgCD8QFOEHgiL8QFCEHwiKr+5u0PLly1s29qJFi1o2dmSr\nVq2qWevq4oJTjvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBTz/Bi3Fi5cmKzfcccdTY995MiRZP2D\nDz5oeuyq4MgPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0GZu7dvZ2bt21nBDh06VLN20UUX5Rq7tze9\nktkNN9yQa/yqmjZtWrJ+//3p5SDq1c2sZi31/6ck3XXXXcn6li1bkvUyuXvt/+EjcOQHgiL8QFCE\nHwiK8ANBEX4gKMIPBEX4gaDqzvOb2RRJP5XUIcklrXb3p83sQkm/kDRV0l5Jd7r7p3XGGrfz/EuW\nLKlZW7p0aa6xP/vss6b3LUk7duzItf88uru7k/Urr7yyZu3yyy9PPrajo6Opnoa9+eabNWuPPfZY\n8rGbN2/Ote8yFTnPf0LSv7r7VyR9VdJ3zewrkhZJ2uruV0namv0NYJyoG35373f3d7LbA5L2SLpU\n0hxJ67K7rZN0W6uaBFC8MZ3zm9lUSdMl/U5Sh7v3Z6WPNXRaAGCcaPg7/Mzs85JekvQ9d//zyOum\n3d1rnc+bWbek9IkhgLZr6MhvZp/TUPB/5u4bss0HzKwzq3dKOjjaY919tbt3uTsrIwIVUjf8NnSI\nf1bSHndfOaK0UdL87PZ8Sa8W3x6AVmlkqm+GpN9I2iVpMNu8WEPn/b+UdJmkfRqa6jtcZ6xxO9U3\nZcqUmrXt27cnHzt58uSi2wmh3sduP/nkk2R95syZNWt9fX1N9TQeNDrVV/ec393/R1Ktwb45lqYA\nVAdX+AFBEX4gKMIPBEX4gaAIPxAU4QeC4qu7CzB37txkvaenJ1mfPn16ke1UyuDgYM3ac889l3zs\nqlWrkvWdO3c21dOZjq/uBpBE+IGgCD8QFOEHgiL8QFCEHwiK8ANBMc/fBpdcckmyfvfddyfrZ599\ndrL++OOPj7mnYfXmyp9//vmmx5akgYGBmrVnnnkm19gYHfP8AJIIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAo5vmBMwzz/ACSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqLrhN7MpZvZfZva+mb1nZv+SbV9qZvvN\nbGf2M7v17QIoSt2LfMysU1Knu79jZl+Q9Lak2yTdKemYu/97wzvjIh+g5Rq9yGdiAwP1S+rPbg+Y\n2R5Jl+ZrD0DZxnTOb2ZTJU2X9Lts04Nm9q6ZrTGzC2o8ptvMes2sN1enAArV8LX9ZvZ5Sf8t6Qfu\nvsHMOiQdkuSSHtfQqcF36ozBy36gxRp92d9Q+M3sc5I2Sdrs7itHqU+VtMndr6szDuEHWqywD/aY\nmUl6VtKekcHP3ggcNlfS7rE2CaA8jbzbP0PSbyTtkjS83vJiSfMkTdPQy/69ku7P3hxMjcWRH2ix\nQl/2F4XwA63H5/kBJBF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU\n4QeCqvsFngU7JGnfiL+/mG2roqr2VtW+JHprVpG9Xd7oHdv6ef7Tdm7W6+5dpTWQUNXeqtqXRG/N\nKqs3XvYDQRF+IKiyw7+65P2nVLW3qvYl0VuzSumt1HN+AOUp+8gPoCSlhN/MZpnZ783sIzNbVEYP\ntZjZXjPbla08XOoSY9kyaAfNbPeIbRea2RYz+zD7PeoyaSX1VomVmxMrS5f63FVtxeu2v+w3swmS\n/iDpVkl9kt6SNM/d329rIzWY2V5JXe5e+pywmX1N0jFJPx1eDcnMfijpsLuvyP7hvMDdv1+R3pZq\njCs3t6i3WitL/7NKfO6KXPG6CGUc+W+U9JG7/9Hdj0v6uaQ5JfRRee6+TdLhUzbPkbQuu71OQ//x\ntF2N3irB3fvd/Z3s9oCk4ZWlS33uEn2VoozwXyrpTyP+7lO1lvx2Sb82s7fNrLvsZkbRMWJlpI8l\ndZTZzCjqrtzcTqesLF2Z566ZFa+Lxht+p5vh7tdL+idJ381e3laSD52zVWm65seSvqyhZdz6Jf2o\nzGaylaVfkvQ9d//zyFqZz90ofZXyvJUR/v2Spoz4e3K2rRLcfX/2+6CklzV0mlIlB4YXSc1+Hyy5\nn79y9wPuftLdByX9RCU+d9nK0i9J+pm7b8g2l/7cjdZXWc9bGeF/S9JVZvYlM5sk6duSNpbQx2nM\n7NzsjRiZ2bmSZqp6qw9vlDQ/uz1f0qsl9vJ3qrJyc62VpVXyc1e5Fa/dve0/kmZr6B3//5P0b2X0\nUKOvKyT9b/bzXtm9SVqvoZeB/6+h90YWSrpI0lZJH0p6TdKFFertPzS0mvO7GgpaZ0m9zdDQS/p3\nJe3MfmaX/dwl+irleeMKPyAo3vADgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxDUXwDRbpm85l3W\ntwAAAABJRU5ErkJggg==\n",
 478 |       "text/plain": [
 479 |        "<Figure size 432x288 with 1 Axes>"
 480 |       ]
 481 |      },
 482 |      "metadata": {},
 483 |      "output_type": "display_data"
 484 |     }
 485 |    ],
 486 |    "source": [
 487 |     "# Plot the second image\n",
 488 |     "plot_img(sample_data[0][1])"
 489 |    ]
 490 |   },
 491 |   {
 492 |    "cell_type": "code",
 493 |    "execution_count": 14,
 494 |    "metadata": {
 495 |     "ExecuteTime": {
 496 |      "end_time": "2019-04-07T22:51:33.888404Z",
 497 |      "start_time": "2019-04-07T22:51:33.559125Z"
 498 |     }
 499 |    },
 500 |    "outputs": [
 501 |     {
 502 |      "data": {
 503 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADkZJREFUeJzt3X+MVfWZx/HPA0JMhD9QZsfBIQXx\nZyGRbkaziePKSqmsaYKYiOWPdTZrOk0sUczGLGGja9KYmLqt4a8m00gYtQVWBCFNaUvR+CNuGkBY\nxFGKU2k64whVagompjI8/WMO2xHnfO9w77n33OF5v5LJ3Huee855cjKfOefec8/5mrsLQDyTym4A\nQDkIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoC5q5MrMjK8TAnXm7jae19W05zezpWZ22Mze\nM7M1tSwLQGNZtd/tN7PJkn4raYmkAUl7JK10977EPOz5gTprxJ7/Jknvufvv3P0vkjZJWlbD8gA0\nUC3hv0LSH0Y9H8imfYGZdZvZXjPbW8O6ABSs7h/4uXuPpB6Jw36gmdSy5x+UNHvU8/ZsGoAJoJbw\n75F0tZnNNbOpkr4laUcxbQGot6oP+939tJmtkvRLSZMlrXf3twvrDEBdVX2qr6qV8Z4fqLuGfMkH\nwMRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAo\nwg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFBVD9EtSWZ2VNJJ\nScOSTrt7RxFNoXFaWlqS9Xnz5iXrkyal9x8333xzbu3aa69Nzrt48eJk/ZNPPknW9+3bl1t77bXX\nkvP29vYm6xeCmsKf+Sd3/6iA5QBoIA77gaBqDb9L+pWZ7TOz7iIaAtAYtR72d7r7oJn9naRdZvau\nu786+gXZPwX+MQBNpqY9v7sPZr+PS9om6aYxXtPj7h18GAg0l6rDb2aXmNn0s48lfUPSoaIaA1Bf\ntRz2t0raZmZnl/NTd/9FIV0BqDtz98atzKxxKwvkuuuuy609+uijyXlvueWWZL29vb2qnppdf39/\nsn7VVVc1qJPiubuN53Wc6gOCIvxAUIQfCIrwA0ERfiAowg8EVcRVfaizBx98MFlfs2ZNbu3yyy8v\nup0Lwssvv1x2C6Vjzw8ERfiBoAg/EBThB4Ii/EBQhB8IivADQXFJbxN46KGHkvUnn3wyWZ88eXJu\n7dSpU8l5n3/++WT92LFjyXot7r333mR91qxZNS1/165dubUVK1Yk5610W/BmxiW9AJIIPxAU4QeC\nIvxAUIQfCIrwA0ERfiAorudvgKlTpybr99xzT7KeOo9fSWqIbEk6ePBg1cuu1f79+5P1zZs317T8\nTz/9NLc2kc/jF4U9PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EVfE8v5mtl/RNScfdfUE27VJJmyXN\nkXRU0gp3/1P92pzYTp8+nazXes75/fffz60dOXKkpmXX09atW5P1l156KVm/7bbbimwnnPHs+TdI\nWnrOtDWSdrv71ZJ2Z88BTCAVw+/ur0o6cc7kZZJ6s8e9ku4suC8AdVbte/5Wdx/KHn8oqbWgfgA0\nSM3f7Xd3T92bz8y6JXXXuh4Axap2z3/MzNokKft9PO+F7t7j7h3u3lHlugDUQbXh3yGpK3vcJWl7\nMe0AaJSK4TezjZL+V9K1ZjZgZvdJekLSEjM7Iunr2XMAE0jF9/zuvjKntLjgXi5YZ86cSda3bduW\nrN9+++3J+ty5c3Nry5YtS867adOmZL2eHn744WS91vP4qe8/gG/4AWERfiAowg8ERfiBoAg/EBTh\nB4JiiO4mMGXKlGT90KFDyfo111yTW6s0xPbSpedesPlFBw4cSNbN0qNB33rrrbm1Spf0zpgxI1nf\nvj393bK77747t/b5558n553IGKIbQBLhB4Ii/EBQhB8IivADQRF+ICjCDwTFef4JYP78+cn6rl27\ncmttbW3JeV9//fVkPXWeXpKWL1+erG/ZsiVZT3nllVeS9fvvvz9Z7+vrq3rdExnn+QEkEX4gKMIP\nBEX4gaAIPxAU4QeCIvxAUJznvwA88MADubWnnnoqOe+kSen//8PDw8l6pev5U8sfGBhIzrto0aJk\nvb+/P1mPivP8AJIIPxAU4QeCIvxAUIQfCIrwA0ERfiCoiuf5zWy9pG9KOu7uC7Jpj0n6tqQ/Zi9b\n6+4/r7gyzvM33IYNG5L1rq6uuq4/9ffV2dmZnPeNN94oup0QijzPv0HSWCM7POXuC7OfisEH0Fwq\nht/dX5V0ogG9AGigWt7zrzKzg2a23szS4yoBaDrVhv9HkuZJWihpSNIP8l5oZt1mttfM9la5LgB1\nUFX43f2Yuw+7+xlJP5Z0U+K1Pe7e4e4d1TYJoHhVhd/MRt8Sdrmk9DCyAJrORZVeYGYbJS2SNNPM\nBiT9l6RFZrZQkks6Kuk7dewRQB1UDL+7rxxj8tN16AVVamlpya1dfPHFDezkyx5//PHcGufxy8U3\n/ICgCD8QFOEHgiL8QFCEHwiK8ANBVTzVh/LNmjUrWX/xxRdzazfeeGNN6z516lSyPm3atGR94cKF\nNa0f9cOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCYojuJpC6JFeSDh8+nKzPmFH9LRSfe+65ZH3d\nunXJ+p49e5L1jz/+OLd2ww03JOcdHBxM1jE2hugGkET4gaAIPxAU4QeCIvxAUIQfCIrwA0FxPX8D\ntLe3J+v79+9P1iudx//ss89yaytXjnXn9b/ZuXNnsr5gwYJkvZLLLrsstzZ9+vSalo3asOcHgiL8\nQFCEHwiK8ANBEX4gKMIPBEX4gaAqnuc3s9mSnpHUKskl9bj7OjO7VNJmSXMkHZW0wt3/VL9WJ65H\nHnkkWZ85c2ZNy1+1alVuLXVP/0Y4ceJEbu3kyZMN7ATnGs+e/7Skf3f3r0r6B0nfNbOvSlojabe7\nXy1pd/YcwARRMfzuPuTub2aPT0p6R9IVkpZJ6s1e1ivpzno1CaB45/We38zmSPqapN9IanX3oaz0\noUbeFgCYIMb93X4zmybpBUmr3f3PZn+7TZi7e979+cysW1J3rY0CKNa49vxmNkUjwf+Ju2/NJh8z\ns7as3ibp+FjzunuPu3e4e0cRDQMoRsXw28gu/mlJ77j7D0eVdkjqyh53SdpefHsA6mU8h/03S/oX\nSW+Z2YFs2lpJT0j6HzO7T9LvJa2oT4vNr9IQ2nfddVdNy1+7dm2y/uyzz9a0/Hrq6+vLrXFr7nJV\nDL+7vy4p7z7gi4ttB0Cj8A0/ICjCDwRF+IGgCD8QFOEHgiL8QFDcursAH3zwQbJe6dbcS5YsSdYX\nLVqUrG/YsCG3NjQ0lFuTpCuvvDJZX716dbJeSX9/f03zo37Y8wNBEX4gKMIPBEX4gaAIPxAU4QeC\nIvxAUOY+5t236rOynFt9XehaWlqS9d27dyfr119/fbJ+5syZ3Nrw8HBy3osuSn/VY8qUKVWvW5Lm\nz5+fW3v33XeT86I67p53Cf4XsOcHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaA4zz8BdHZ2JusbN27M\nrbW3t9e07vXr1yfrW7ZsSdZ37txZ0/px/jjPDyCJ8ANBEX4gKMIPBEX4gaAIPxAU4QeCqnie38xm\nS3pGUqskl9Tj7uvM7DFJ35b0x+yla9395xWWxXl+oM7Ge55/POFvk9Tm7m+a2XRJ+yTdKWmFpFPu\n/t/jbYrwA/U33vBXHLHH3YckDWWPT5rZO5KuqK09AGU7r/f8ZjZH0tck/SabtMrMDprZejObkTNP\nt5ntNbO9NXUKoFDj/m6/mU2T9Iqkx919q5m1SvpII58DfE8jbw3+rcIyOOwH6qyw9/ySZGZTJP1M\n0i/d/Ydj1OdI+pm7L6iwHMIP1FlhF/aYmUl6WtI7o4OffRB41nJJh863SQDlGc+n/Z2SXpP0lqSz\n92leK2mlpIUaOew/Kuk72YeDqWWx5wfqrNDD/qIQfqD+uJ4fQBLhB4Ii/EBQhB8IivADQRF+ICjC\nDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIo38CzYR5J+P+r5zGxaM2rW3pq1L4neqlVkb18Z\n7wsbej3/l1ZuttfdO0prIKFZe2vWviR6q1ZZvXHYDwRF+IGgyg5/T8nrT2nW3pq1L4neqlVKb6W+\n5wdQnrL3/ABKUkr4zWypmR02s/fMbE0ZPeQxs6Nm9paZHSh7iLFsGLTjZnZo1LRLzWyXmR3Jfo85\nTFpJvT1mZoPZtjtgZneU1NtsM3vZzPrM7G0zezCbXuq2S/RVynZr+GG/mU2W9FtJSyQNSNojaaW7\n9zW0kRxmdlRSh7uXfk7YzP5R0ilJz5wdDcnMvi/phLs/kf3jnOHu/9EkvT2m8xy5uU695Y0s/a8q\ncdsVOeJ1EcrY898k6T13/527/0XSJknLSuij6bn7q5JOnDN5maTe7HGvRv54Gi6nt6bg7kPu/mb2\n+KSksyNLl7rtEn2VoozwXyHpD6OeD6i5hvx2Sb8ys31m1l12M2NoHTUy0oeSWstsZgwVR25upHNG\nlm6abVfNiNdF4wO/L+t097+X9M+Svpsd3jYlH3nP1kyna34kaZ5GhnEbkvSDMpvJRpZ+QdJqd//z\n6FqZ226MvkrZbmWEf1DS7FHP27NpTcHdB7PfxyVt08jblGZy7Owgqdnv4yX38//c/Zi7D7v7GUk/\nVonbLhtZ+gVJP3H3rdnk0rfdWH2Vtd3KCP8eSVeb2VwzmyrpW5J2lNDHl5jZJdkHMTKzSyR9Q803\n+vAOSV3Z4y5J20vs5QuaZeTmvJGlVfK2a7oRr9294T+S7tDIJ/79kv6zjB5y+rpS0v9lP2+X3Zuk\njRo5DPxcI5+N3CfpMkm7JR2R9GtJlzZRb89qZDTngxoJWltJvXVq5JD+oKQD2c8dZW+7RF+lbDe+\n4QcExQd+QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeC+ivcD5dhEV6CmQAAAABJRU5ErkJggg==\n",
 504 |       "text/plain": [
 505 |        "<Figure size 432x288 with 1 Axes>"
 506 |       ]
 507 |      },
 508 |      "metadata": {},
 509 |      "output_type": "display_data"
 510 |     }
 511 |    ],
 512 |    "source": [
 513 |     "# Plot the 32nd image\n",
 514 |     "plot_img(sample_data[0][31])"
 515 |    ]
 516 |   },
 517 |   {
 518 |    "cell_type": "markdown",
 519 |    "metadata": {},
 520 |    "source": [
 521 |     "## Specify a CNN"
 522 |    ]
 523 |   },
 524 |   {
 525 |    "cell_type": "code",
 526 |    "execution_count": 15,
 527 |    "metadata": {
 528 |     "ExecuteTime": {
 529 |      "end_time": "2019-04-07T22:51:33.919660Z",
 530 |      "start_time": "2019-04-07T22:51:33.894165Z"
 531 |     }
 532 |    },
 533 |    "outputs": [],
 534 |    "source": [
 535 |     "class Net(nn.Module):\n",
 536 |     "    def __init__(self):\n",
 537 |     "        super().__init__()\n",
 538 |     "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
 539 |     "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
 540 |     "        self.conv2_drop = nn.Dropout2d()\n",
 541 |     "        self.fc1 = nn.Linear(320, 50)\n",
 542 |     "        self.fc2 = nn.Linear(50, 10)\n",
 543 |     "\n",
 544 |     "    def forward(self, x):\n",
 545 |     "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
 546 |     "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
 547 |     "        x = x.view(-1, 320)\n",
 548 |     "        x = F.relu(self.fc1(x))\n",
 549 |     "        #x = F.dropout(x,p=0.1, training=self.training)\n",
 550 |     "        x = self.fc2(x)\n",
 551 |     "        return F.log_softmax(x,dim=1)"
 552 |    ]
 553 |   },
 554 |   {
 555 |    "cell_type": "code",
 556 |    "execution_count": 16,
 557 |    "metadata": {
 558 |     "ExecuteTime": {
 559 |      "end_time": "2019-04-07T22:51:33.934889Z",
 560 |      "start_time": "2019-04-07T22:51:33.925826Z"
 561 |     }
 562 |    },
 563 |    "outputs": [],
 564 |    "source": [
 565 |     "model = Net()\n",
 566 |     "if is_cuda:\n",
 567 |     "    model.cuda()"
 568 |    ]
 569 |   },
 570 |   {
 571 |    "cell_type": "code",
 572 |    "execution_count": 17,
 573 |    "metadata": {
 574 |     "ExecuteTime": {
 575 |      "end_time": "2019-04-07T22:51:33.951738Z",
 576 |      "start_time": "2019-04-07T22:51:33.942183Z"
 577 |     }
 578 |    },
 579 |    "outputs": [
 580 |     {
 581 |      "data": {
 582 |       "text/plain": [
 583 |        "False"
 584 |       ]
 585 |      },
 586 |      "execution_count": 17,
 587 |      "metadata": {},
 588 |      "output_type": "execute_result"
 589 |     }
 590 |    ],
 591 |    "source": [
 592 |     "is_cuda"
 593 |    ]
 594 |   },
 595 |   {
 596 |    "cell_type": "markdown",
 597 |    "metadata": {},
 598 |    "source": [
 599 |     "## Define an Optimization Algorithm"
 600 |    ]
 601 |   },
 602 |   {
 603 |    "cell_type": "code",
 604 |    "execution_count": 18,
 605 |    "metadata": {
 606 |     "ExecuteTime": {
 607 |      "end_time": "2019-04-07T22:51:33.969283Z",
 608 |      "start_time": "2019-04-07T22:51:33.954646Z"
 609 |     }
 610 |    },
 611 |    "outputs": [],
 612 |    "source": [
 613 |     "optimizer = optim.SGD(model.parameters(),lr=0.01)"
 614 |    ]
 615 |   },
 616 |   {
 617 |    "cell_type": "code",
 618 |    "execution_count": 19,
 619 |    "metadata": {
 620 |     "ExecuteTime": {
 621 |      "end_time": "2019-04-07T22:51:34.013607Z",
 622 |      "start_time": "2019-04-07T22:51:33.980168Z"
 623 |     }
 624 |    },
 625 |    "outputs": [],
 626 |    "source": [
 627 |     "data , target = next(iter(train_loader))"
 628 |    ]
 629 |   },
 630 |   {
 631 |    "cell_type": "code",
 632 |    "execution_count": 20,
 633 |    "metadata": {
 634 |     "ExecuteTime": {
 635 |      "end_time": "2019-04-07T22:51:34.062296Z",
 636 |      "start_time": "2019-04-07T22:51:34.019115Z"
 637 |     }
 638 |    },
 639 |    "outputs": [
 640 |     {
 641 |      "name": "stdout",
 642 |      "output_type": "stream",
 643 |      "text": [
 644 |       "<class 'torch.Tensor'>\n",
 645 |       "<class 'torch.Tensor'>\n"
 646 |      ]
 647 |     }
 648 |    ],
 649 |    "source": [
 650 |     "print(type(data))\n",
 651 |     "print(type(target))"
 652 |    ]
 653 |   },
 654 |   {
 655 |    "cell_type": "code",
 656 |    "execution_count": 21,
 657 |    "metadata": {
 658 |     "ExecuteTime": {
 659 |      "end_time": "2019-04-07T22:51:34.088605Z",
 660 |      "start_time": "2019-04-07T22:51:34.073281Z"
 661 |     }
 662 |    },
 663 |    "outputs": [
 664 |     {
 665 |      "name": "stdout",
 666 |      "output_type": "stream",
 667 |      "text": [
 668 |       "torch.Size([32, 1, 28, 28])\n",
 669 |       "torch.Size([32])\n"
 670 |      ]
 671 |     }
 672 |    ],
 673 |    "source": [
 674 |     "print(data.shape)\n",
 675 |     "print(target.shape)"
 676 |    ]
 677 |   },
 678 |   {
 679 |    "cell_type": "code",
 680 |    "execution_count": 22,
 681 |    "metadata": {
 682 |     "ExecuteTime": {
 683 |      "end_time": "2019-04-07T22:51:34.254267Z",
 684 |      "start_time": "2019-04-07T22:51:34.097000Z"
 685 |     }
 686 |    },
 687 |    "outputs": [],
 688 |    "source": [
 689 |     "#output = model(Variable(data.cuda()))\n",
 690 |     "output = model(Variable(data))"
 691 |    ]
 692 |   },
 693 |   {
 694 |    "cell_type": "code",
 695 |    "execution_count": 23,
 696 |    "metadata": {
 697 |     "ExecuteTime": {
 698 |      "end_time": "2019-04-07T22:51:34.264290Z",
 699 |      "start_time": "2019-04-07T22:51:34.257896Z"
 700 |     }
 701 |    },
 702 |    "outputs": [
 703 |     {
 704 |      "data": {
 705 |       "text/plain": [
 706 |        "torch.Size([32, 10])"
 707 |       ]
 708 |      },
 709 |      "execution_count": 23,
 710 |      "metadata": {},
 711 |      "output_type": "execute_result"
 712 |     }
 713 |    ],
 714 |    "source": [
 715 |     "output.size()"
 716 |    ]
 717 |   },
 718 |   {
 719 |    "cell_type": "code",
 720 |    "execution_count": 24,
 721 |    "metadata": {
 722 |     "ExecuteTime": {
 723 |      "end_time": "2019-04-07T22:51:34.274650Z",
 724 |      "start_time": "2019-04-07T22:51:34.268396Z"
 725 |     }
 726 |    },
 727 |    "outputs": [
 728 |     {
 729 |      "data": {
 730 |       "text/plain": [
 731 |        "torch.Size([32])"
 732 |       ]
 733 |      },
 734 |      "execution_count": 24,
 735 |      "metadata": {},
 736 |      "output_type": "execute_result"
 737 |     }
 738 |    ],
 739 |    "source": [
 740 |     "target.size()"
 741 |    ]
 742 |   },
 743 |   {
 744 |    "cell_type": "markdown",
 745 |    "metadata": {},
 746 |    "source": [
 747 |     "## Train the model"
 748 |    ]
 749 |   },
 750 |   {
 751 |    "cell_type": "code",
 752 |    "execution_count": 25,
 753 |    "metadata": {
 754 |     "ExecuteTime": {
 755 |      "end_time": "2019-04-07T22:51:34.312342Z",
 756 |      "start_time": "2019-04-07T22:51:34.279843Z"
 757 |     }
 758 |    },
 759 |    "outputs": [],
 760 |    "source": [
 761 |     "def fit(epoch,model,data_loader,phase='training',volatile=False):\n",
 762 |     "    if phase == 'training':\n",
 763 |     "        model.train()\n",
 764 |     "    if phase == 'validation':\n",
 765 |     "        model.eval()\n",
 766 |     "        volatile=True\n",
 767 |     "    running_loss = 0.0\n",
 768 |     "    running_correct = 0\n",
 769 |     "    for batch_idx , (data,target) in enumerate(data_loader):\n",
 770 |     "        if is_cuda:\n",
 771 |     "            data,target = data.cuda(),target.cuda()\n",
 772 |     "        data , target = Variable(data,volatile),Variable(target)\n",
 773 |     "        if phase == 'training':\n",
 774 |     "            optimizer.zero_grad()\n",
 775 |     "        output = model(data)\n",
 776 |     "        loss = F.nll_loss(output,target)\n",
 777 |     "        \n",
 778 |     "        running_loss += F.nll_loss(output,target,size_average=False).data[0]\n",
 779 |     "        preds = output.data.max(dim=1,keepdim=True)[1]\n",
 780 |     "        running_correct += preds.eq(target.data.view_as(preds)).cpu().sum()\n",
 781 |     "        if phase == 'training':\n",
 782 |     "            loss.backward()\n",
 783 |     "            optimizer.step()\n",
 784 |     "    \n",
 785 |     "    loss = running_loss/len(data_loader.dataset)\n",
 786 |     "    accuracy = 100. * running_correct/len(data_loader.dataset)\n",
 787 |     "    #print(\"Loss: %2.2f, Accuracy: %2.2f\" % (loss, accuracy))\n",
 788 |     "    #print(f'{phase} loss is {loss:{5}.{2}} and {phase} accuracy is {running_correct}/{len(data_loader.dataset)}{accuracy:{10}.{4}}')\n",
 789 |     "    return loss,accuracy"
 790 |    ]
 791 |   },
 792 |   {
 793 |    "cell_type": "code",
 794 |    "execution_count": 26,
 795 |    "metadata": {
 796 |     "ExecuteTime": {
 797 |      "end_time": "2019-04-07T22:51:34.318393Z",
 798 |      "start_time": "2019-04-07T22:51:34.315302Z"
 799 |     }
 800 |    },
 801 |    "outputs": [],
 802 |    "source": [
 803 |     "#loss, acc = fit(epoch,model,train_loader, phase='training')"
 804 |    ]
 805 |   },
 806 |   {
 807 |    "cell_type": "code",
 808 |    "execution_count": 27,
 809 |    "metadata": {
 810 |     "ExecuteTime": {
 811 |      "end_time": "2019-04-07T23:45:58.691953Z",
 812 |      "start_time": "2019-04-07T22:51:34.321783Z"
 813 |     },
 814 |     "scrolled": false
 815 |    },
 816 |    "outputs": [
 817 |     {
 818 |      "name": "stderr",
 819 |      "output_type": "stream",
 820 |      "text": [
 821 |       "/Users/jeffrey/anaconda2/envs/dl/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
 822 |       "  warnings.warn(warning.format(ret))\n",
 823 |       "/Users/jeffrey/anaconda2/envs/dl/lib/python3.6/site-packages/ipykernel/__main__.py:18: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number\n"
 824 |      ]
 825 |     },
 826 |     {
 827 |      "name": "stdout",
 828 |      "output_type": "stream",
 829 |      "text": [
 830 |       "Execution Time:  54.41\n"
 831 |      ]
 832 |     }
 833 |    ],
 834 |    "source": [
 835 |     "import time\n",
 836 |     "start = time.time()\n",
 837 |     "\n",
 838 |     "n_epoch = 50\n",
 839 |     "\n",
 840 |     "train_losses , train_accuracy = [],[]\n",
 841 |     "val_losses , val_accuracy = [],[]\n",
 842 |     "for epoch in range(1, n_epoch):\n",
 843 |     "    epoch_loss, epoch_accuracy = fit(epoch,model,train_loader,phase='training')\n",
 844 |     "    val_epoch_loss , val_epoch_accuracy = fit(epoch,model,test_loader,phase='validation')\n",
 845 |     "    train_losses.append(epoch_loss)\n",
 846 |     "    train_accuracy.append(epoch_accuracy)\n",
 847 |     "    val_losses.append(val_epoch_loss)\n",
 848 |     "    val_accuracy.append(val_epoch_accuracy)\n",
 849 |     "    \n",
 850 |     "end = time.time()\n",
 851 |     "print(\"Execution Time: \", round(((end-start)/60.0),2))"
 852 |    ]
 853 |   },
 854 |   {
 855 |    "cell_type": "code",
 856 |    "execution_count": 28,
 857 |    "metadata": {
 858 |     "ExecuteTime": {
 859 |      "end_time": "2019-04-07T23:45:58.723283Z",
 860 |      "start_time": "2019-04-07T23:45:58.710559Z"
 861 |     }
 862 |    },
 863 |    "outputs": [
 864 |     {
 865 |      "name": "stdout",
 866 |      "output_type": "stream",
 867 |      "text": [
 868 |       "54.41\n"
 869 |      ]
 870 |     }
 871 |    ],
 872 |    "source": [
 873 |     "print(round(((end-start)/60.0),2))"
 874 |    ]
 875 |   },
 876 |   {
 877 |    "cell_type": "markdown",
 878 |    "metadata": {},
 879 |    "source": [
 880 |     "## Plot the training and validation loss and accuracy curves"
 881 |    ]
 882 |   },
 883 |   {
 884 |    "cell_type": "code",
 885 |    "execution_count": 29,
 886 |    "metadata": {
 887 |     "ExecuteTime": {
 888 |      "end_time": "2019-04-07T23:45:59.124046Z",
 889 |      "start_time": "2019-04-07T23:45:58.727111Z"
 890 |     },
 891 |     "scrolled": false
 892 |    },
 893 |    "outputs": [
 894 |     {
 895 |      "data": {
 896 |       "text/plain": [
 897 |        "<matplotlib.legend.Legend at 0x11b00e908>"
 898 |       ]
 899 |      },
 900 |      "execution_count": 29,
 901 |      "metadata": {},
 902 |      "output_type": "execute_result"
 903 |     },
 904 |     {
 905 |      "data": {
 906 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X90VPWd//HnmyQQws8IwSq/gltU\nCD8lpbSsBSt6QCyW1h+wcCo9Wk49uuqx9Surrm7poT896tpl7WJX1wVapVRbVFyqLa76/fqDoIi/\nKwJCQCUgBDAECHl//7gzYQgzk8lkkknuvB7n3DNz73zm3s+dTF733s+987nm7oiISLh0ynYFREQk\n8xTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiISQwl1EJITys7Xgvn37emlpabYW\nLyLSIa1fv363u5c0VS5r4V5aWkpFRUW2Fi8i0iGZ2UeplFOzjIhICCncRURCSOEuIhJCWWtzF5G2\nd/ToUSorK6mtrc12VaQJhYWFDBgwgIKCgrTer3AXySGVlZX06NGD0tJSzCzb1ZEE3J09e/ZQWVnJ\nkCFD0ppHh2qWWb4cSkuhU6fgcfnybNdIpGOpra2lT58+CvZ2zszo06dPi46wOsye+/LlMH8+1NQE\n4x99FIwDzJmTvXqJdDQK9o6hpX+nDrPnftttx4M9qqYmmC4iIifqMOG+bVvzpotI+7Nv3z7+/d//\nPa33XnTRRezbty9pmTvuuINnn302rfk3Vlpayu7duzMyr2zoMOE+aFDzpotIy2X6PFeycK+rq0v6\n3tWrV9O7d++kZRYuXMiUKVPSrl+YpBTuZjbVzN43s01mtiBBmcvN7B0ze9vMfpvZasKiRVBUdOK0\noqJguohkXvQ810cfgfvx81wtCfgFCxbw4YcfMmbMGG6++Waee+45zj33XGbMmMHw4cMB+OY3v8m4\nceMoKytjyZIlDe+N7klv3bqVYcOG8b3vfY+ysjIuvPBCDh06BMC8efNYuXJlQ/k777yTc845h5Ej\nR/Lee+8BUFVVxQUXXEBZWRlXX301gwcPbnIP/e6772bEiBGMGDGCe++9F4DPP/+c6dOnM3r0aEaM\nGMGjjz7asI7Dhw9n1KhR/PCHP0z/w2opd086AHnAh8AZQGfgDWB4ozJDgdeB4sh4v6bmO27cOG+u\nZcvcBw92Nwsely1r9ixEcto777yTctnBg92DWD9xGDw4/eVv2bLFy8rKGsbXrl3rRUVFvnnz5oZp\ne/bscXf3mpoaLysr8927d0fqM9irqqp8y5YtnpeX56+//rq7u1922WW+dOlSd3e/8sor/fe//31D\n+fvuu8/d3RcvXuxXXXWVu7tfe+21/pOf/MTd3Z9++mkHvKqqKs76B8urqKjwESNG+MGDB/3AgQM+\nfPhwf+2113zlypV+9dVXN5Tft2+f7969288880yvr693d/e9e/em/2F5/L8XUOFN5Ku7p7TnPh7Y\n5O6b3f0I8AhwSaMy3wMWu/veyAZjVwu3OXHNmQNbt0J9ffCoq2REWk9bnecaP378Cddy33fffYwe\nPZoJEyawfft2Pvjgg5PeM2TIEMaMGQPAuHHj2Lp1a9x5f+tb3zqpzIsvvsisWbMAmDp1KsXFxUnr\n9+KLLzJz5ky6detG9+7d+da3vsULL7zAyJEjeeaZZ7jlllt44YUX6NWrF7169aKwsJCrrrqKxx57\njKLGzQ1tKJVw7w9sjxmvjEyLdSZwppn9XzN72cymZqqCIpIdbXWeq1u3bg3Pn3vuOZ599lleeukl\n3njjDcaOHRv3Wu8uXbo0PM/Ly0vYXh8tl6xMus4880xee+01Ro4cye23387ChQvJz8/n1Vdf5dJL\nL+XJJ59k6tTsRWGmTqjmEzTNTAZmAw+Y2UlnPsxsvplVmFlFVVVVhhYtIq2hNc5z9ejRgwMHDiR8\nvbq6muLiYoqKinjvvfd4+eWX019YAhMnTmTFihUA/PnPf2bv3r1Jy5977rn88Y9/pKamhs8//5zH\nH3+cc889l507d1JUVMTcuXO5+eabee211zh48CDV1dVcdNFF3HPPPbzxxhsZr3+qUvkR0w5gYMz4\ngMi0WJXAK+5+FNhiZn8jCPt1sYXcfQmwBKC8vNzTrbSItL5os+dttwVNMYMGBcHekubQPn36MHHi\nREaMGMG0adOYPn36Ca9PnTqVX//61wwbNoyzzjqLCRMmtGAN4rvzzjuZPXs2S5cu5Stf+Qpf+MIX\n6NGjR8Ly55xzDvPmzWP8+PEAXH311YwdO5Y1a9Zw880306lTJwoKCrj//vs5cOAAl1xyCbW1tbg7\nd999d8brnyoL2ueTFDDLB/4GnE8Q6uuAf3D3t2PKTAVmu/uVZtaX4OTqGHffk2i+5eXlrpt1iLSt\nd999l2HDhmW7Gll1+PBh8vLyyM/P56WXXuKaa65hw4YN2a5WXPH+Xma23t3Lm3pvk3vu7l5nZtcB\nawiunHnQ3d82s4UEZ21XRV670MzeAY4BNycLdhGRbNm2bRuXX3459fX1dO7cmQceeCDbVWoVKfUt\n4+6rgdWNpt0R89yBmyKDiEi7NXToUF5//fVsV6PVdZhfqIqISOoU7iIiIaRwFxEJIYW7iEgIKdxF\npF3r3r07ADt37uTSSy+NW2by5Mk0dWn1vffeS03MTSFS6UI4Ff/yL//CXXfd1eL5ZJrCXUQ6hNNP\nP72hx8d0NA73VLoQ7sgU7iLSZhYsWMDixYsbxqN7vQcPHuT8889v6J73T3/600nv3bp1KyNGjADg\n0KFDzJo1i2HDhjFz5syGLn8BrrnmGsrLyykrK+POO+8Egs7Idu7cyXnnncd5550HnHgzjnhd+ibr\nWjiRDRs2MGHCBEaNGsXMmTMbuja47777GroBjnZa9r//+7+MGTOGMWPGMHbs2KTdMqSjw9xDVUQy\n7MYbIdO/zBwzBiLhGM8VV1zBjTfeyLXXXgvAihUrWLNmDYWFhTz++OP07NmT3bt3M2HCBGbMmJHw\nPqL3338/RUVFvPvuu2zcuJFzzjmn4bVFixZxyimncOzYMc4//3w2btzI9ddfz913383atWvp27fv\nCfNav349Dz30EK+88gruzpe//GUmTZpEcXExH3zwAb/73e944IEHuPzyy/nDH/7A3LlzE67fd77z\nHX71q18xadIk7rjjDn70ox9x77338rOf/YwtW7bQpUuXhqagu+66i8WLFzNx4kQOHjxIYWFhyh9z\nKrTnLiJtZuzYsezatYudO3fyxhtvUFxczMCBA3F3br31VkaNGsWUKVPYsWMHn376acL5PP/88w0h\nO2rUKEaNGtXw2ooVKzjnnHMYO3Ysb7/9Nu+8807SOiXq0hdS71oYgk7P9u3bx6RJkwC48soref75\n5xvqOGfOHJYtW0Z+frBPPXHiRG666Sbuu+8+9u3b1zA9U7TnLpKrkuxht6bLLruMlStX8sknn3DF\nFVcAsHz5cqqqqli/fj0FBQWUlpbG7eq3KVu2bOGuu+5i3bp1FBcXM2/evLTmE9W4a+GmmmUSeeqp\np3j++ed54oknWLRoEW+++SYLFixg+vTprF69mokTJ7JmzRrOPvvstOvamPbcRaRNXXHFFTzyyCOs\nXLmSyy67DAj2evv160dBQQFr167lo48+SjqPr33ta/z2t8HdPN966y02btwIwP79++nWrRu9evXi\n008/5emnn254T6LuhhN16dtcvXr1ori4uGGvf+nSpUyaNIn6+nq2b9/Oeeedx89//nOqq6s5ePAg\nH374ISNHjuSWW27hS1/6UsNtADNFe+4i0qbKyso4cOAA/fv357TTTgNgzpw5fOMb32DkyJGUl5c3\nuQd7zTXX8N3vfpdhw4YxbNgwxo0bB8Do0aMZO3YsZ599NgMHDmTixIkN75k/fz5Tp07l9NNPZ+3a\ntQ3TE3Xpm6wJJpGHH36Y73//+9TU1HDGGWfw0EMPcezYMebOnUt1dTXuzvXXX0/v3r3553/+Z9au\nXUunTp0oKytj2rRpzV5eMk12+dta1OWvSNtTl78dS0u6/FWzjIhICCncRURCSOEukmOy1RQrzdPS\nv5PCXSSHFBYWsmfPHgV8O+fu7Nmzp0U/bNLVMiI5ZMCAAVRWVlJVVZXtqkgTCgsLGTBgQNrvV7iL\n5JCCggKGDBmS7WpIG1CzjIhICCncRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhFIKdzObambvm9km\nM1sQ5/V5ZlZlZhsiw9WZr6qIiKSqyevczSwPWAxcAFQC68xslbs3vr3Jo+5+XSvUUUREmimVPffx\nwCZ33+zuR4BHgEtat1oiItISqYR7f2B7zHhlZFpj3zazjWa20swGxpuRmc03swozq9DPn0VEWk+m\nTqg+AZS6+yjgGeDheIXcfYm7l7t7eUlJSYYWLSIijaUS7juA2D3xAZFpDdx9j7sfjoz+BhiXmeqJ\niEg6Ugn3dcBQMxtiZp2BWcCq2AJmdlrM6Azg3cxVUUREmqvJq2Xcvc7MrgPWAHnAg+7+tpktBCrc\nfRVwvZnNAOqAz4B5rVhnERFpgm6QLSLSgegG2SIiOUzhLiISQgp3EZEQUriLiISQwl1EJIQU7iIi\nIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCnc\nRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAilFO5mNtXM3jezTWa2IEm5\nb5uZm1l55qooIiLN1WS4m1kesBiYBgwHZpvZ8DjlegA3AK9kupIiItI8qey5jwc2uftmdz8CPAJc\nEqfcj4GfA7UZrJ+IiKQhlXDvD2yPGa+MTGtgZucAA939qQzWTURE0tTiE6pm1gm4G/hBCmXnm1mF\nmVVUVVW1dNEiIpJAKuG+AxgYMz4gMi2qBzACeM7MtgITgFXxTqq6+xJ3L3f38pKSkvRrLSIiSaUS\n7uuAoWY2xMw6A7OAVdEX3b3a3fu6e6m7lwIvAzPcvaJVaiwiIk1qMtzdvQ64DlgDvAuscPe3zWyh\nmc1o7QqKiEjz5adSyN1XA6sbTbsjQdnJLa+WiIi0hH6hKiISQgp3EZEQUriLiISQwl1EJIQU7iIi\nIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCnc\nRURCSOEuIhJCCncRkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQiilcDezqWb2vplt\nMrMFcV7/vpm9aWYbzOxFMxue+aqKiEiqmgx3M8sDFgPTgOHA7Djh/Vt3H+nuY4BfAHdnvKYiIpKy\nVPbcxwOb3H2zux8BHgEuiS3g7vtjRrsBnrkqiohIc+WnUKY/sD1mvBL4cuNCZnYtcBPQGfh6Rmon\nIiJpydgJVXdf7O5/B9wC3B6vjJnNN7MKM6uoqqrK1KJFRKSRVMJ9BzAwZnxAZFoijwDfjPeCuy9x\n93J3Ly8pKUm9liIi0iyphPs6YKiZDTGzzsAsYFVsATMbGjM6Hfggc1UUEZHmarLN3d3rzOw6YA2Q\nBzzo7m+b2UKgwt1XAdeZ2RTgKLAXuLI1Ky0iIsmlckIVd18NrG407Y6Y5zdkuF4iItIC+oWqiEgI\nKdxFREJI4S4iEkIKdxGREFK4i4iEkMJdRCSEFO4iIiGkcBcRCSGFu4hICCncRURCSOEuIhJCCncR\nkRBSuIuIhJDCXUQkhBTuIiIhpHAXEQkhhbuISAgp3EVEQkjhLiISQgp3EZEQUriLiISQwl1EJIQU\n7iIiIaRwFxEJIYW7iEgIpRTuZjbVzN43s01mtiDO6zeZ2TtmttHM/mJmgzNfVRERSVWT4W5mecBi\nYBowHJhtZsMbFXsdKHf3UcBK4BeZrmhTli+H0lLo1Cl4XL68rWsgItJ+pLLnPh7Y5O6b3f0I8Ahw\nSWwBd1/r7jWR0ZeBAZmtZnLLl8P8+fDRR+AePM6fr4AXkdyVSrj3B7bHjFdGpiVyFfB0vBfMbL6Z\nVZhZRVVVVeq1bMJtt0FNzYnTamqC6SIiuSijJ1TNbC5QDvwy3uvuvsTdy929vKSkJGPL3batedNF\nRMIulXDfAQyMGR8QmXYCM5sC3AbMcPfDmaleagYNat50EZGwSyXc1wFDzWyImXUGZgGrYguY2Vjg\nPwiCfVfmq5ncokVQVHTitKKiYLqISC5qMtzdvQ64DlgDvAuscPe3zWyhmc2IFPsl0B34vZltMLNV\nCWbXKubMgSVLYPBgMAselywJpouI5CJz96wsuLy83CsqKrKybBGRjsrM1rt7eVPl9AtVEZEQUriL\niISQwl1EJIQU7iIiIaRwFxEJoY4X7k88Ad/+dtCJjIiIxNXxwv2zz+Cxx+D117NdExGRdqvjhfu0\nacEvlZ54Its1ERFptzpeuPfrBxMmKNxFRJLoeOEO8I1vwPr1sHNntmsiItIudcxwv/ji4PGpp5os\nqjs0iUgu6pjhPmJE0DtYE00zukOTiOSqjhnuZkHTzLPPwqFDCYvpDk0ikqs6ZrhD0DRz6BD89a8J\ni+gOTSKSqzpuuE+eDN27J22a0R2aRCRXddxw79IFLrwQnnwy4a9VdYcmEclVHTfcIWia2bEDNmyI\n+7Lu0CQiuapjh/v06U3+WnXOHNi6Ferrg8dosOsSSREJs44d7v36wZe/3Oxfq+oSSREJu44d7hBc\nEllRAR9/nPJbdImkiIRdxw/3ZvxaNUqXSIpI2HX8cB85Mri2sRlNM7pEUkTCruOHe4q/Vo3V1CWS\nOtkqIh1dxw93CJpmampg7dqUiie7RFInW0UkDFIKdzObambvm9kmM1sQ5/WvmdlrZlZnZpdmvppN\nmDwZunVrVtNMokskk51s1R69iHQUTYa7meUBi4FpwHBgtpkNb1RsGzAP+G2mK5iSwsImf62aqkQn\nVaN78NqjF5GOIJU99/HAJnff7O5HgEeAS2ILuPtWd98I1LdCHVNz8cVQWZnw16qpSnRSNS9Pl0+K\nSMeRSrj3B7bHjFdGprUvF18c9Ddzzz0tmk2ik63HjsUvr8snRaQ9atMTqmY238wqzKyiqqoqszPv\n1w9uugmWLoVXX017NolOtg4eHL/8oEFqixeR9ieVcN8BDIwZHxCZ1mzuvsTdy929vKSkJJ1ZJPdP\n/wSnngo33tiitvd4J1sT7dFfdJHa4kWk/Ukl3NcBQ81siJl1BmYBq1q3Wmnq0QN+8hN46SV49NGM\nzjrRHv3q1cnb4rVXLyLZYJ7CHq6ZXQTcC+QBD7r7IjNbCFS4+yoz+xLwOFAM1AKfuHtZsnmWl5d7\nRUVFi1fgJMeOwZe+BLt3w/vvQ9eumV9GjE6d4h8kmAUtRPPnnxj+RUXBRgGCDcC2bUHTzqJF6opY\nRJpmZuvdvbzJcqmEe2totXAHeP55mDQJfvxjuP321llGRGlp0BTTWLSNPt5rffoEP6ZV6ItIc6Ua\n7uH4hWpjX/saXHop/PSnwc08WlGyrgwSXUmzZ0/8ppwbbkjefq8mHhFJVTjDHeAXv4C6Orj11lZd\nTLKuDJrbEVmi0I/+OlYnbkUkVeEN9yFDgksj//u/Yd26Vl1Uoq4MEu3V9+nTvPlv26ZuEUSkecIb\n7pCxSyPTlWiv/l//tXmhP2hQet0iKPRFcpi7Z2UYN26ct4nf/MYd3B96qG2Wl6Jly9wHD3Y3Cx6X\nLQuGoqKgutGhqOh42djp0SEvL/70Pn0SzyvR8pNNF5H2geAqxSYzNvzhXlfnXl4erOqUKe5r17rX\n17fNstOQLHTjhXW8YE82ROcZb17XXJN4g6DQF2kfFO6x9u93/8Uv3E89NVjlr37V/amn2nXIxxMv\nYBPt0Scaou/N5lFAsg2FNiIiySnc46mpcf+3f3MfNChY9bFj3Zcudd+8ucMFfVSivfA+fRLvuZs1\nb4OQyaOApo4OdOQgkpzCPZkjR9wffND9zDOPp0jPnu7nnut+3XXuDzzgvm6d+7Fj2atjM7R2+30m\njwISTR88OPG8kh056ChAco3CPRV1de4vv+z+H/8R7FJ+9avu3bsfT5CRI91XruwwId9Yc9vvE+1V\nt8VRgFnz59VU6Dd3g5DJDYU2LNJaFO7pOnbM/YMPgj37s84KPqLRo90fe6zDNt3E05yAy+RRQDp7\n7s0d0jkKyGRzUbLyyT77TP4dJbwU7plQVxe0yQ8dGnxUY8a4//GPoQr5VGXqKCCdEE105JDJo4BM\nNhclO9JJ97xCczci6RyFaEPRMSjcM+noUfeHH3b/u78LPrIBA9xnznRftMj9f/7Hvaoq2zXMqta+\nWiadk8aZOgpIZ0ORbF7pnldozvqnM690L4PN5N8+U9+vsFO4t4ZoyM+efXxvPjZRvv1t91/+0v3F\nF4MrcyRj0tlzbU4gtlVzUXM3FG3VXJXOZbCZPGpr7t84nXklm97c7142Kdzbwt697n/9a3AN/RVX\nuA8Zcvyblp8f/HjqH/8x+DasX+++Z0/yJp3qavdXXw02IHfd5b5mjfvu3W23Ph1Qpo4CMtlclCwQ\n0/ldQjZPWmdyQ5FsA5rJjXEmj06yfWI+HoV7tnzyifuf/uS+YIH75MknfzN69nQfNcp9xgz3G25w\nv/Za9/PPd+/fP/F/0pAh7pdd5v6zn7k/84z7m2+6f/ih+8cfBxuEo0ezvdYdSms3F6UTFuk0MaXT\n5p/Jy2Dborkqk/NK5+ikrU7MN0eq4R7Om3W0J3V18M478OGHsGVL0G3k1q3B8y1bIC8Pzj4bhg0L\nHqND376wcSOsXw8VFcGweXPi5XTuHNx1qksXKCw88bF796DXstLSoLfM6HD66UGvYkeOwOefxx8O\nHjxx/Ngx+OIXYfhwGDo0WG5j9fXB+r75ZrAOAF//OkyYEL98B7Z8efNvrhLvPZD8rl3NuaNXOvO6\n8kp4+OGTp3ftGnRF3Viym9Hk5QVfk1SnDx4c1L85UZTJeSWS6XpB4hv7bN2a+jJy+05MHYV70F1k\nqj77DDZsCB5raoKwrak5cTh8GGprT3zcvz/49uzceeI3NT8/GI/3bUxFfv7xoD/rLKiqCsL8rbeO\np0SnSMej9fXQrVtwh6wLLoApU6CsrOn137//+IZw8+Ygaerrg3rX1x9/bga9ekFxMZxyyomPxcXQ\nuzcUFDS9TnV1wefWrVvqf5sjR4LPsIW3dEy2oWjuRiSdeWVqo5NoQ5Fo+pIlwXKbc9eydOaVKHgT\nMQs+h3jzaq7oVyle3JoFX+PU56Vwl8YOHw7+c6NhGf3Wdu8ehFnjId50gL/9LTgaiR02bQpCdPRo\nGDUKRo4MHocPD8LvuefgmWfg2WeD90MQvr17B6FYWHh86NIFdu0Kwvyzz05ch7y8YIMRHcyCx/r6\nkzu8b6xbt2B5sWF/4ECwAYkO0Xnk5wdHTyUlJw6HDwd1q6oKhl27oLo6eE/fvsFuWOwwYECwPp06\nBXWPDp06BUlz5MiJQ3SjHD1qih45HTwYTO/fPzjqOuOMYCgtPf53aY76+qD+lZUnD4cPB+sfGT7Y\nms/L6/L59GA3DhYP5LwrBzFp7kAYNIjla/py2+128oZimXPnbXVUbqun/6A8Fi7KY85cO3EDMtD5\n6Y+OMHtmLX9YXsutPzgMtYfowQF6sp+Szvu5YV41BbX7eW7Vfvbvc7oUd+XCGV0ZP6krL67vysMr\nurK9qpDifgV8d34BF16Uz9PPFvCjnxRwsDaPU/iMU/mUAQW7mDJqFx+/8Smn1O3CcPbRm8/ze3Ow\noJidh3qzl2Kq6UU1vdhPT3qc3pPrb+/J937Qk32HukQ+OKd713p6dj1K9Wd1FHCUrgR17sEBenc6\nQFF98LwzR6imF3spputpxVR3KuatHb3ZT0/g+I6D9tylfaurC0Irlb3dbduCkH/llSC4amuD3bPa\n2uND375BeEWbkKLPi4sTL+PIEdi3L9gg7N0bPH72WTAtOuzde/z50aPQs2cw9Ohx/HlhYVAuGuDR\nYffuIKj79QuCPvpYUhKE9bZtwQYz+tjUxiYVRUXBRrZ792BjVFkZfGax+vULXj92LBjq6o4/Ro9u\nGh/tRF+LVVAQNNUVFQWvR4ejR4PHAweC4I9VWBi8x/3Ev9/hwyfP3+z4xg1OnldrM6O2R1+2HSqh\n9mgeffP2UlKwj4Lag02+9QgFGE4BdS2uxjE6cQ338wDzG442mnOfZIW7SDa5B01IO3YEG536+uPh\nG32elxdsLDp3PnHo0iXY2BQVHW/Wip3v7t3BUU1sc9WhQ8GeduzRQX7+8SOG6BFOdMjLgy98ITiy\niA79+p28vHjL3rYNtm8PHrdtC5r78vNPPNcTfR49Qmk8wInlYp9HN7I9ewZNbdGNb6dOwXo2Hmpr\nj2+Ajh49Phw7FuwM9OsX3LSnT5+gno3V1fH731Tzqx/v49DOvXzx1AN8/x/2M2ns/uCobP/+YMNm\nFmwAI0c0r71ZwJP/k8/Hewvp3KcHM7/Tg8nf6MHTL/bgl7/uwdadnTn7tGpu+u5eppQHOxbr/7KX\n//fkXpbtn8Gng8endI6mMYW7iEgIpRru4b7NnohIjlK4i4iEUErhbmZTzex9M9tkZgvivN7FzB6N\nvP6KmZVmuqIiIpK6JsPdzPKAxcA0YDgw28yGNyp2FbDX3b8I3AP8PNMVFRGR1KWy5z4e2OTum939\nCPAIcEmjMpcAD0eerwTON2vOr3NERCSTUgn3/sD2mPHKyLS4Zdy9DqgG+mSigiIi0nxtekLVzOab\nWYWZVVRVVbXlokVEckoq4b4DGBgzPiAyLW4ZM8sHegEndTfk7kvcvdzdy0tKStKrsYiINCnOz7VO\nsg4YamZDCEJ8FvAPjcqsAq4EXgIuBf7qTfw6av369bvNrKkuefoCu1OoYxjl8rpDbq9/Lq875Pb6\np7Lug1OZUZPh7u51ZnYdsAbIAx5097fNbCFBv8KrgP8ElprZJuAzgg1AU/NtctfdzCpS+SVWGOXy\nukNur38urzvk9vpnct1T2XPH3VcDqxtNuyPmeS1wWSYqJCIiLadfqIqIhFB7D/cl2a5AFuXyukNu\nr38urzvk9vpnbN2z1iukiIhDJo9+AAADC0lEQVS0nva+5y4iImlol+HeVEdlYWNmD5rZLjN7K2ba\nKWb2jJl9EHkszmYdW4uZDTSztWb2jpm9bWY3RKbnyvoXmtmrZvZGZP1/FJk+JNIJ36ZIp3zhurN4\nDDPLM7PXzezJyHhOrLuZbTWzN81sg5lVRKZl7Hvf7sI9xY7Kwua/gKmNpi0A/uLuQ4G/RMbDqA74\ngbsPByYA10b+3rmy/oeBr7v7aGAMMNXMJhB0vndPpDO+vQSd84XVDcC7MeO5tO7nufuYmMsfM/a9\nb3fhTmodlYWKuz9P8PuAWLGdsT0MfLNNK9VG3P1jd38t8vwAwT95f3Jn/d3dozfxLIgMDnydoBM+\nCPH6m9kAYDrwm8i4kSPrnkDGvvftMdxT6agsF5zq7h9Hnn8CnJrNyrSFyH0AxgKvkEPrH2mW2ADs\nAp4BPgT2RTrhg3D/D9wL/B8gejftPuTOujvwZzNbb2bzI9My9r1P6UdMkl3u7mYW6suazKw78Afg\nRnffH9tjdNjX392PAWPMrDfwOHB2lqvUJszsYmCXu683s8nZrk8W/L277zCzfsAzZvZe7Ist/d63\nxz33VDoqywWfmtlpAJHHXVmuT6sxswKCYF/u7o9FJufM+ke5+z5gLfAVoHekEz4I7//ARGCGmW0l\naH79OvCv5Ma64+47Io+7CDbq48ng9749hntDR2WRs+SzCDomyzXRztiIPP4pi3VpNZE21v8E3nX3\nu2NeypX1L4nssWNmXYELCM47rCXohA9Cuv7u/k/uPsDdSwn+z//q7nPIgXU3s25m1iP6HLgQeIsM\nfu/b5Y+YzOwigra4aEdli7JcpVZlZr8DJhP0CPcpcCfwR2AFMAj4CLjc3RufdO3wzOzvgReANzne\n7norQbt7Lqz/KIITZ3kEO1sr3H2hmZ1BsDd7CvA6MNfdD2evpq0r0izzQ3e/OBfWPbKOj0dG84Hf\nuvsiM+tDhr737TLcRUSkZdpjs4yIiLSQwl1EJIQU7iIiIaRwFxEJIYW7iEgIKdxFREJI4S4iEkIK\ndxGREPr/NTY7L1mzB9YAAAAASUVORK5CYII=\n",
 907 |       "text/plain": [
 908 |        "<Figure size 432x288 with 1 Axes>"
 909 |       ]
 910 |      },
 911 |      "metadata": {},
 912 |      "output_type": "display_data"
 913 |     }
 914 |    ],
 915 |    "source": [
 916 |     "plt.plot(range(1,len(train_losses)+1),train_losses,'bo',label = 'training loss')\n",
 917 |     "plt.plot(range(1,len(val_losses)+1),val_losses,'r',label = 'validation loss')\n",
 918 |     "plt.legend()"
 919 |    ]
 920 |   },
 921 |   {
 922 |    "cell_type": "code",
 923 |    "execution_count": 30,
 924 |    "metadata": {
 925 |     "ExecuteTime": {
 926 |      "end_time": "2019-04-07T23:45:59.474304Z",
 927 |      "start_time": "2019-04-07T23:45:59.127737Z"
 928 |     }
 929 |    },
 930 |    "outputs": [
 931 |     {
 932 |      "data": {
 933 |       "text/plain": [
 934 |        "<matplotlib.legend.Legend at 0x108086438>"
 935 |       ]
 936 |      },
 937 |      "execution_count": 30,
 938 |      "metadata": {},
 939 |      "output_type": "execute_result"
 940 |     },
 941 |     {
 942 |      "data": {
 943 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VdW99/HPjwTFIGiIURkTWgfm\nwQREUYtSLFqnalO0OEALFAeUWotjxWp5rl5t9XGgNVWLFhy4IGpvVQbF4r0OGBQURcXHICYiRAHB\ngkjg9/yxNzHAOclJcpKTZH/fr9d5nZx91lpnbYnf7LP23muZuyMiItHRItUdEBGRhqXgFxGJGAW/\niEjEKPhFRCJGwS8iEjEKfhGRiFHwi4hEjIJfRCRiFPwiIhGTnuoOxHLQQQd5bm5uqrshItJkLFmy\n5At3z06kbKMM/tzcXIqKilLdDRGRJsPMPkm0rIZ6REQiRsEvIhIxCn4RkYhR8IuIRIyCX0QkYhT8\nIiIRo+AXEYmYRnkdvzQjn34Kr78OP/1p4nUWLgweIlGz//4waVK9f4yCX+rPjh1wzjnwxhvw7LNw\nyinV11m5Mii3bRuY1X8fRRqTQw5R8EsT98ADQei3bQuXXQbLl8N++8Uv7w4TJsA++0BxMbRv33B9\nFYkQjfFL/Vi3Dq69FoYMgSefhI8/httuq7rO7Nkwdy784Q8KfZF6pOCX+nH11bB5M0ydCkOHwnnn\nwa23BkM5sWzeDBMnQr9+cMklDdtXkYhR8Evy/c//wLRpcNVV0L17sO2PfwyGcCZMCIZ09nTzzVBa\nCn/+M6RrBFKkPin4Jbm2b4eLL4YuXeCGG77b3r59MIQzd24wpFPZ8uVw550wdiwMGtSw/RWJIAW/\nJNc99wRBfvfd0Lr17u9dckkwlDNxYjC0A8HR/8UXw4EHwn/8R8P3VySCFPySPCUlMHkynHYanHHG\n3u+npwdDOaWl8PvfB9seeSQYGvrP/4SsrIbtr0hEKfgjasYMyM2FFi2C5xkzqt6eSJ2Zna/km3+X\n89RJd4NZ7PKDBsHYsey88y5+dugi1o36LUv2PYYZLUfVW7/Ultpqqm3VG3dvdI+8vDxPua1b3UtL\nm+Vj9j2l/r1Wpd6e7x7fa1Xq11wYe/vse6qv8xNmu4Nfzy2ekeF+8cXuGRnuwVhO8MjIcJ8+3X3m\nn7/wMrJ8Gy29nBbeh6VV1qmqrenTa1ZHbamtptLW9Ok1iyygyBPM2HoN8No+Uh7869e75+bu/i+h\nR7WPDzjc9+EbB/e0tNjFcnKCx2gedAf/ExMr3otXp7q2alJHbamtptJWTk7NYqsmwW9B+cYlPz/f\nU7rm7qWXwl/+ArffHsyd0cz86ldQk3/1XRMnVFfnn/yYz+hYdVthY+7OEF7iFY7lW/atQW9itVWr\n6mpLbTXqtsxg586alLcl7p6fSNmELpg2syuAsQQZ8Fd3v8vMngCODIscCGx0934x6q4CNgM7gPJE\nO5YyRUXBCcgJE+DKK1Pdm3ox9//AJzGWZU5LC6bX2VNOTvBckzrxtnfpsqst4yVOTFJbyeyX2lJb\njaOtXXXqRXVfCYBewHIgg+APxQLgsD3K/BG4MU79VcBBiX4FcU/hUE95uXt+vvuhh7pv3JiaPjSA\n2ow1NpdxU7WltppKWykd4wcKgAcrvf4dMKnSawM+BQ6PU7/pBP/UqcF/kkcfTc3nN6Dp04MxRLPg\nedcvWbzttamjttSW2qpbWzVRk+CvdozfzLoDTwPHAFuBF8IPmBC+fwLwJ48zhGNmxcAGgiHi+929\nsLpvISkZ41+7Fo48EvLyYMECTQksIk1KUsf43X2Fmd0GzAP+DSwlGK/f5TzgsSqaOM7dS83sYGC+\nmb3v7otidHocMA6gS70ObsUxaRJs2QL33afQF5FmLaEbuNz9QXfPc/cTCI7ePwQws3TgbOCJKuqW\nhs/rgDnAwDjlCt09393zs7Oza7YXdfWvfwV3kE6aBN26Nexni4g0sISCPzxax8y6EAT9o+FbPwTe\nd/eSOPVam1mbXT8DJxOcKG48tm8P5pDJzYXrrqtVE8m+U68h7hQUkQhL5EQA8DLwHrAMGFpp+zRg\n/B5lOwDPhj9/L6yzDHgXuD6Rz2vQk7u33Rac0P3HP2pVPd6Z+tqexW+Iqwhqc+JIRBo3dANXglav\nDuaLHzYMnnqqVk3k5sa+Nrc2174n83r5qtrKyYFVq/beLiJNV9Jv4Gq2Jk4MDoTvuqvWTaxeXbPt\nEDuoa1unNm1V9Z6INH/RnZ3zn/+EOXPgxhuDw/ZaincBUpcu8d9LS0tendq0lYqLpkSk8Yhm8G/d\nGkzJ0K1bnadlmDIFMjJ235aREWyP9964ccmrU5u2pkyp2T6KSDOT6MmAhnzU+8nd3/0uONP54otJ\naS7Zd+o1xJ2CItK8oJO7VfjwQ+jdGwoKYPr0+vkMEZEGVpOTu9Ea6nEPplxu1QruuCPVvRERSYlo\nXdXzX/8VzMNzzz1w6KGp7o2ISEpE54h/06bg8s2jjoKLL66yaKNZF1NEpB5E54j/ppvg88+DG7Xi\nXQNJEObjxgXztUFwA9S4cfC//wsPP7z3doCRI+u36yIiyRSNk7vLlgXTLY8dG6yuVYV4d+JWdYes\n7oIVkVTTyd3Kdu4MJmHLzEzoAvZ4d7XW5g5ZEZHGqPkH/7Rp8MorwVU87dpVW7w2d9uKiDQlzTv4\nv/wymGP/+OPhwgsTqlKbu21FRJqS5h38114LGzfC1KkJr6o1ciQUFgZj92bBc2Fh0ESs7TqxKyJN\nTfM9ufvaa3DMMXDVVXD77cnpmIhII6WTu+XlwbX6HTvC5Mmp7o2ISKPSPK/jnzoVli6FWbNg//1T\n3RsRkUYl0TV3rzCz5Wb2rplNDLfdZGalZrY0fJwap+5wM/vAzD4ys2uS2fmY1qyBG26A4cPh7LPr\n/eNERJqaaoPfzHoBY4GBQF/gNDM7LHz7TnfvFz6ejVE3DbgPOAXoAZxnZj2S1vtYfvMb+PbbYD6e\nKk7oavoFEYmqRI74uwOvu/sWdy8H/gUkeig9EPjI3T9292+Bx4Eza9fVBLzwAjz2WHA1z2GHxS22\na1qGTz4JJuzcNf2Cwl9EoiCR4F8OHG9mWWaWAZwKdA7fu8zM3jazh8wsM0bdjsCnlV6XhNuSb9u2\nYMrl738frr66yqLXX//dnDu7bNkSbBcRae6qDX53XwHcBswDngeWAjuAPwPfB/oBa4A/1qUjZjbO\nzIrMrKisrKzmDZSXwymnwL33BvPtV6E2C6SLiDQXCZ3cdfcH3T3P3U8ANgAfuvtad9/h7juBvxIM\n6+yplO++HQB0CrfF+oxCd8939/zs7Oya7QVA69Zw553BSd1qaBFyEYmyRK/qOTh87kIwvv+ombWv\nVOQnBENCe3oDONzMuprZPsC5wDN163LdaRFyEYmyRK/jn21mWcB24FJ332hm95hZP8CBVcCvAMys\nA/CAu5/q7uVmdhkwF0gDHnL3d5O+FzW0a5qF668Phne6dAlCX9MviEgUNN8pG0REIkRTNoiISFwK\nfhGRiGn2wa87dEVEdtc8J2kLxVs4HXQiV0Siq1kf8esOXRGRvTXr4NcduiIie2vWwa87dEVE9tas\ng1936IqI7K1ZB3+8hdN1YldEoqxZX9UDQcgr6EVEvtOsj/hFRGRvCn4RkYhR8IuIRIyCX0QkYhT8\nIiIRo+AXEYkYBb+ISMQkuubuFWa23MzeNbOJ4bbbzex9M3vbzOaY2YFx6q4ys3fMbKmZaVktEZEU\nqzb4zawXMBYYCPQFTjOzw4D5QC937wN8CFxbRTMnunu/RJcFExGR+pPIEX934HV33+Lu5cC/gLPd\nfV74GuA1oFN9dVJERJInkeBfDhxvZllmlgGcCnTeo8wvgOfi1HdgnpktMbNxte+qiIgkQ7Vz9bj7\nCjO7DZgH/BtYCuzY9b6ZXQ+UA/EWNTzO3UvN7GBgvpm97+6L9iwU/lEYB9BF8yaLiNSbhE7uuvuD\n7p7n7icAGwjG9DGzUcBpwEh39zh1S8PndcAcgnMFscoVunu+u+dnZ2fXeEdERCQxiV7Vc3D43AU4\nG3jUzIYDk4Az3H1LnHqtzazNrp+BkwmGjkREJEUSnZZ5tpllAduBS919o5ndC+xLMHwD8Jq7jzez\nDsAD7n4qcAgwJ3w/HXjU3Z9P+l6IiEjCEgp+dz8+xrbD4pT9jOAEMO7+McEloCIi0kjozl0RkYhR\n8IuIRIyCX0QkYhT8IiIRo+AXEYkYBb+ISMQo+EVEIkbBLyISMQp+EZGIUfCLiESMgl9EJGIU/CIi\nEaPgFxGJGAW/iEjEKPhFRCJGwS8iEjEKfhGRiEl0zd0rzGy5mb1rZhPDbe3MbL6ZrQyfM+PUvSgs\ns9LMLkpm50VEpOaqDX4z6wWMBQYSLKN4mpkdBlwDvODuhwMvhK/3rNsOmAwcHdafHO8PhIiINIxE\njvi7A6+7+xZ3Lwf+BZwNnAk8HJZ5GDgrRt0fAfPdfb27bwDmA8Pr3m0REamtRIJ/OXC8mWWZWQbB\nQuqdgUPcfU1Y5nPgkBh1OwKfVnpdEm4TEZEUSa+ugLuvMLPbgHnAv4GlwI49yriZeV06YmbjgHEA\nXbp0qUtTIiJShYRO7rr7g+6e5+4nABuAD4G1ZtYeIHxeF6NqKcG3g106hdtifUahu+e7e352dnZN\n9kFERGog0at6Dg6fuxCM7z8KPAPsukrnIuDpGFXnAiebWWZ4UvfkcJuIiKRItUM9odlmlgVsBy51\n941mdisw08x+CXwC/AzAzPKB8e4+xt3Xm9ktwBthOze7+/ok74OIiNSAuddpaL5e5Ofne1FRUaq7\nISLSZJjZEnfPT6Ss7twVEYkYBb+ISMQo+EVEIkbBLyISMQp+EZGIUfCLiESMgl9EJGIU/CIiEaPg\nFxGJGAW/iEjEKPhFRCJGwS8iEjEKfhGRiFHwi4hEjIJfRCRiFPwiIhGj4BcRiZiEll40s18DYwAH\n3gFGA/OBNmGRg4HF7n5WjLo7wjoAq939jLp2WkREaq/a4DezjsDlQA9332pmM4Fz3f34SmVmE3ux\ndYCt7t4vKb0VEZE6S3SoJx3Yz8zSgQzgs11vmFlb4CTgqeR3T0REkq3a4Hf3UuAOYDWwBvjK3edV\nKnIW8IK7b4rTRCszKzKz18xsr6EgERFpWNUGv5llAmcCXYEOQGszO79SkfOAx6poIidc+f3nwF1m\n9v04nzMu/ANRVFZWlvAOiIhIzSQy1PNDoNjdy9x9O/AkcCyAmR0EDAT+Ga9y+I0Bd/8YeAnoH6dc\nobvnu3t+dnZ2jXZCREQSl0jwrwYGmVmGmRkwFFgRvvdT4L/d/ZtYFc0s08z2DX8+CBgMvFf3bouI\nSG0lMsb/OjALeJPgsswWQGH49rnsMcxjZvlm9kD4sjtQZGbLgIXAre6u4BcRSSFz91T3YS/5+fle\nVFSU6m6IiDQZZrYkPJ9aLd25KyISMQp+EZGIUfCLiESMgl9EJGIU/CIiEaPgFxGJGAW/iEjEKPhF\nRCJGwS8iEjEKfhGRiFHwi4hEjIJfRCRiFPwiIhGj4BcRiRgFv4hIxCj4RUQiRsEvIhIxCQW/mf3a\nzN41s+Vm9piZtTKzaWZWbGZLw0e/OHUvMrOV4eOi5HZfRERqKr26AmbWEbgc6OHuW81sJsFauwC/\ndfdZVdRtB0wG8gEHlpjZM+6+oe5dFxGR2kh0qCcd2M/M0oEM4LME6/0ImO/u68Ownw8Mr3k3RUQk\nWaoNfncvBe4AVgNrgK/cfV749hQze9vM7jSzfWNU7wh8Wul1SbhNRERSpNrgN7NM4EygK9ABaG1m\n5wPXAt2AAUA74Oq6dMTMxplZkZkVlZWV1aUpERGpQiJDPT8Eit29zN23A08Cx7r7Gg9sA/4GDIxR\ntxToXOl1p3DbXty90N3z3T0/Ozu7ZnshIiIJSyT4VwODzCzDzAwYCqwws/YA4bazgOUx6s4FTjaz\nzPCbw8nhNhERSZFqr+px99fNbBbwJlAOvAUUAs+ZWTZgwFJgPICZ5QPj3X2Mu683s1uAN8Lmbnb3\n9fWwHyIikiBz91T3YS/5+fleVFSU6m6IiDQZZrbE3fMTKas7d0VEIkbBLyISMQp+EZGIUfCLiESM\ngl9EJGIU/CIiEaPgFxGJGAW/iEjEKPhFRCJGwS8iEjEKfhGRiFHwi4hEjIJfRCRiFPwiIhGj4BcR\niRgFv4hIxCj4RUQiJqHgN7Nfm9m7ZrbczB4zs1ZmNsPMPgi3PWRmLePU3WFmS8PHM8ntvoiI1FS1\nwW9mHYHLgXx37wWkAecCM4BuQG9gP2BMnCa2unu/8HFGcrotIiK1Ve1i65XK7Wdm24EM4DN3n7fr\nTTNbDHSqh/6JSAPZvn07JSUlfPPNN6nuilShVatWdOrUiZYtYw6yJKTa4Hf3UjO7A1gNbAXm7RH6\nLYELgCvi9dPMioBy4FZ3f6rWvRWRelNSUkKbNm3Izc3FzFLdHYnB3fnyyy8pKSmha9eutW4nkaGe\nTOBMoCvQAWhtZudXKjIVWOTuL8dpIidc+f3nwF1m9v04nzPOzIrMrKisrKxGOyEidffNN9+QlZWl\n0G/EzIysrKw6fytL5OTuD4Fidy9z9+3Ak8CxYScmA9nAlfEqu3tp+Pwx8BLQP065QnfPd/f87Ozs\nGu2EiCSHQr/xS8a/USLBvxoYZGYZFnziUGCFmY0BfgSc5+4743Qw08z2DX8+CBgMvFfnXotIs7Nx\n40amTp1aq7qnnnoqGzduTHKPmq9qg9/dXwdmAW8C74R1CoG/AIcAr4aXat4IYGb5ZvZAWL07UGRm\ny4CFBGP8Cn6RZmDGDMjNhRYtgucZM+rWXlXBX15eXmXdZ599lgMPPLBuHagH7s7OnTGPi1PL3Rvd\nIy8vz0WkYb333nsJl50+3T0jwx2+e2RkBNtra8SIEd6qVSvv27evX3XVVb5w4UI/7rjj/PTTT/fD\nDz/c3d3PPPNMP+qoo7xHjx5+//33V9TNycnxsrIyLy4u9m7duvmYMWO8R48ePmzYMN+yZcten/XM\nM8/4wIEDvV+/fj506FD//PPP3d198+bNPmrUKO/Vq5f37t3bZ82a5e7uzz33nPfv39/79OnjJ510\nkru7T5482W+//faKNnv27OnFxcVeXFzsRxxxhF9wwQXeo0cPX7VqlY8fP97z8vK8R48efuONN1bU\nWbx4sR9zzDHep08fHzBggG/atMmPP/54f+uttyrKDB482JcuXbpb/2P9WwFFnmDGpjzkYz0U/CIN\nrybBn5Oze+jveuTk1P7zi4uLvWfPnhWvFy5c6BkZGf7xxx9XbPvyyy/d3X3Lli3es2dP/+KLL8L+\nfBf8aWlpFcFZUFDgf//73/f6rPXr1/vOnTvd3f2vf/2rX3nlle7uPmnSJL/iiit2K7du3Trv1KlT\nRT929aGq4Dczf/XVV/fqd3l5uf/gBz/wZcuW+bZt27xr166+ePFid3f/6quvfPv27T5t2rSKPnzw\nwQceKw/rGvyJXscvIlJh9eqaba+tgQMH7nbZ4t13382cOXMA+PTTT1m5ciVZWVm71enatSv9+vUD\nIC8vj1WrVu3VbklJCSNGjGDNmjV8++23FZ+xYMECHn/88YpymZmZ/OMf/+CEE06oKNOuXbtq+52T\nk8OgQYMqXs+cOZPCwkLKy8tZs2YN7733HmZG+/btGTBgAABt27YFoKCggFtuuYXbb7+dhx56iFGj\nRlX7eTWluXpEpMa6dKnZ9tpq3bp1xc8vvfQSCxYs4NVXX2XZsmX0798/5mWN++67b8XPaWlpMc8P\nTJgwgcsuu4x33nmH+++/v1aXR6anp+82fl+5jcr9Li4u5o477uCFF17g7bff5sc//nGVn5eRkcGw\nYcN4+umnmTlzJiNHjqxx36qj4BeRGpsyBTIydt+WkRFsr602bdqwefPmuO9/9dVXZGZmkpGRwfvv\nv89rr71W68/66quv6NixIwAPP/xwxfZhw4Zx3333VbzesGEDgwYNYtGiRRQXFwOwfv16AHJzc3nz\nzTcBePPNNyve39OmTZto3bo1BxxwAGvXruW5554D4Mgjj2TNmjW88cYbAGzevLnij9SYMWO4/PLL\nGTBgAJmZmbXez3gU/CJSYyNHQmEh5OSAWfBcWBhsr62srCwGDx5Mr169+O1vf7vX+8OHD6e8vJzu\n3btzzTXX7DaUUlM33XQTBQUF5OXlcdBBB1Vsv+GGG9iwYQO9evWib9++LFy4kOzsbAoLCzn77LPp\n27cvI0aMAOCcc85h/fr19OzZk3vvvZcjjjgi5mf17duX/v37061bN37+858zePBgAPbZZx+eeOIJ\nJkyYQN++fRk2bFjFN4G8vDzatm3L6NGja72PVbHgnEDjkp+f70VFRanuhkikrFixgu7du6e6GwJ8\n9tlnDBkyhPfff58WLfY+Po/1b2VmSzyYJaFaOuIXEWlEHnnkEY4++mimTJkSM/STQVf1iIg0Ihde\neCEXXnhhvX6GjvhFRCJGwS8iEjEKfhGRiFHwi4hEjIJfRJqs/fffP9VdaJIU/CIitVTddNGNlYJf\nRBqFa665ZrfpEm666SbuuOMOvv76a4YOHcpRRx1F7969efrpp6tt66yzziIvL4+ePXtSWFhYsf35\n55/nqKOOom/fvgwdOhSAr7/+mtGjR9O7d2/69OnD7Nmzgd2/TcyaNatisrRRo0Yxfvx4jj76aCZN\nmsTixYs55phj6N+/P8ceeywffPABADt27OCqq66iV69e9OnTh3vuuYcXX3yRs846q6Ld+fPn85Of\n/KT2/9FqSdfxi8jeJk6EpUuT22a/fnDXXXHfHjFiBBMnTuTSSy8Fghkt586dS6tWrZgzZw5t27bl\niy++YNCgQZxxxhlVLkH40EMP0a5dO7Zu3cqAAQM455xz2LlzJ2PHjmXRokV07dq1Ys6dW265hQMO\nOIB33nkHCObnqU5JSQmvvPIKaWlpbNq0iZdffpn09HQWLFjAddddx+zZsyksLGTVqlUsXbqU9PR0\n1q9fT2ZmJpdccgllZWVkZ2fzt7/9jV/84hc1+a+YFAp+EWkU+vfvz7p16/jss88oKysjMzOTzp07\ns337dq677joWLVpEixYtKC0tZe3atRx66KFx24o1fXNZWVnM6ZVjTcVcnYKCAtLS0oBgwreLLrqI\nlStXYmZs3769ot3x48eTnp6+2+ddcMEFTJ8+ndGjR/Pqq6/yyCOP1PQ/VZ0lFPxm9mtgDOAEyy+O\nBtoDjwNZwBLgAnf/Nkbda4FfAjuAy919bnK6LiL1pooj8/pUUFDArFmz+PzzzysmQ5sxYwZlZWUs\nWbKEli1bkpubW+W0xpWnb87IyGDIkCG1mna58jeKPetXnnb5d7/7HSeeeCJz5sxh1apVDBkypMp2\nR48ezemnn06rVq0oKCio+MPQkKod4zezjsDlQL679wLSgHOB24A73f0wYANBuO9Zt0dYticwHJhq\nZmnJ6/53kr3+p4g0vBEjRvD4448za9YsCgoKgOCI+uCDD6Zly5YsXLiQTz75pMo24k3fHG965VhT\nMQMccsghrFixgp07d1Z8e4j3ebumeJ42bVrF9mHDhnH//fdXnADe9XkdOnSgQ4cO/OEPf6i32Ter\nk+jJ3XRgPzNLBzKANcBJBIuwAzwMnBWj3pnA4+6+zd2LgY+AgXXr8t5mzIBx4+CTT4IF4D75JHit\n8BdpWnr27MnmzZvp2LEj7du3B2DkyJEUFRXRu3dvHnnkEbp161ZlG/Gmb443vXKsqZgBbr31Vk47\n7TSOPfbYir7EMmnSJK699lr69++/21U+Y8aMoUuXLvTp04e+ffvy6KOPVrw3cuRIOnfunLLZUBOa\nltnMrgCmAFuBecAVwGvh0T5m1hl4LvxGULnevWG56eHrB8Nys6hCTadlzs0Nwn5POTkQY9U1EYlB\n0zI3nMsuu4z+/fvzy1/uNVCSkHqfltnMMgmO3LsCHYDWBMM2SWVm48ysyMyKysrKalS3odb/FBGp\nq7y8PN5++23OP//8lPUhkbMKPwSK3b0MwMyeBAYDB5pZuruXA52A0hh1S4HOlV7HK4e7FwKFEBzx\nJ7wHBOt8xjriT/b6nyIidbVkyZJUdyGhMf7VwCAzy7DgNPdQ4D1gIfDTsMxFQKy7Kp4BzjWzfc2s\nK3A4sLju3d5dfaz/KSLSXFUb/O7+OsFJ3DcJLuVsQXBkfjVwpZl9RHBJ54MAZnaGmd0c1n0XmEnw\nh+J54FJ335HsnaiP9T9FoqgxLsUqu0vGv5HW3BURAIqLi2nTpg1ZWVlV3hUrqePufPnll2zevLni\nRrRdanJyV3fuiggAnTp1oqSkhJpeXCENq1WrVnTq1KlObSj4RQSAli1b7nUUKc2TZucUEYkYBb+I\nSMQo+EVEIqZRXtVjZmVAVTMxHQR80UDdaYyivP9R3neI9v5r36uW4+7ZiTTWKIO/OmZWlOhlS81R\nlPc/yvsO0d5/7Xvy9l1DPSIiEaPgFxGJmKYa/IXVF2nWorz/Ud53iPb+a9+TpEmO8YuISO011SN+\nERGppSYX/GY23Mw+MLOPzOyaVPenvpnZQ2a2zsyWV9rWzszmm9nK8DkzlX2sL2bW2cwWmtl7ZvZu\nuBJcJPbfzFqZ2WIzWxbu++/D7V3N7PXw9/8JM9sn1X2tL2aWZmZvmdl/h6+jtO+rzOwdM1tqZkXh\ntqT93jep4A8Xar8POAXoAZwXLujenE1j7xXPrgFecPfDgRfC181ROfAbd+8BDAIuDf+9o7D/24CT\n3L0v0A8YbmaDgNuAO8NlTzcAtVu7r2m4AlhR6XWU9h3gRHfvV+kyzqT93jep4CdYqP0jd//Y3b8F\nHidYFrLZcvdFwPo9Np9JsMA9xF/ovslz9zXu/mb482aCEOhIBPbfA1+HL1uGDwdOIlgfA5rpvgOY\nWSfgx8AD4WsjIvtehaT93je14O8IfFrpdUm4LWoOcfc14c+fA4eksjMNwcxygf7A60Rk/8OhjqXA\nOmA+8P+AjeFyp9C8f//vAiYBO8PXWURn3yH4Iz/PzJaY2bhwW9J+7zUtcxPn7m5mzfrSLDPbH5gN\nTHT3TZUXCWnO+x+uVtfPzA4AcAbvAAABkklEQVQE5gDdUtylBmFmpwHr3H2JmQ1JdX9S5Dh3LzWz\ng4H5ZvZ+5Tfr+nvf1I74E168vZlba2btAcLndSnuT70xs5YEoT/D3Z8MN0dm/wHcfSPBGtfHAAea\n2a4Dtub6+z8YOMPMVhEM554E/F+ise8AuHtp+LyO4I/+QJL4e9/Ugv8N4PDw7P4+wLkEC7pHzTME\nC9xD/IXum7xwXPdBYIW7/6nSW81+/80sOzzSx8z2A4YRnONYCPw0LNYs993dr3X3Tu6eS/D/+Ivu\nPpII7DuAmbU2sza7fgZOBpaTxN/7JncDl5mdSjD+lwY85O5TUtylemVmjwFDCGbnWwtMBp4iWMS+\nC8Espj9z9z1PADd5ZnYc8DLwDt+N9V5HMM7frPffzPoQnMBLIzhAm+nuN5vZ9wiOgtsBbwHnu/u2\n1PW0foVDPVe5+2lR2fdwP+eEL9OBR919ipllkaTf+yYX/CIiUjdNbahHRETqSMEvIhIxCn4RkYhR\n8IuIRIyCX0QkYhT8IiIRo+AXEYkYBb+ISMT8fxNY3PJNSFd9AAAAAElFTkSuQmCC\n",
 944 |       "text/plain": [
 945 |        "<Figure size 432x288 with 1 Axes>"
 946 |       ]
 947 |      },
 948 |      "metadata": {},
 949 |      "output_type": "display_data"
 950 |     }
 951 |    ],
 952 |    "source": [
 953 |     "plt.plot(range(1,len(train_accuracy)+1),train_accuracy,'bo',label = 'train accuracy')\n",
 954 |     "plt.plot(range(1,len(val_accuracy)+1),val_accuracy,'r',label = 'val accuracy')\n",
 955 |     "plt.legend()"
 956 |    ]
 957 |   },
 958 |   {
 959 |    "cell_type": "markdown",
 960 |    "metadata": {},
 961 |    "source": [
 962 |     "# Exercise"
 963 |    ]
 964 |   },
 965 |   {
 966 |    "cell_type": "markdown",
 967 |    "metadata": {},
 968 |    "source": [
 969 |     "Re-define the model under Section 1.4 and see if you can achieve similar results using a simpler model."
 970 |    ]
 971 |   },
 972 |   {
 973 |    "cell_type": "code",
 974 |    "execution_count": null,
 975 |    "metadata": {},
 976 |    "outputs": [],
 977 |    "source": []
 978 |   }
 979 |  ],
 980 |  "metadata": {
 981 |   "kernelspec": {
 982 |    "display_name": "Python [conda env:dl]",
 983 |    "language": "python",
 984 |    "name": "conda-env-dl-py"
 985 |   },
 986 |   "language_info": {
 987 |    "codemirror_mode": {
 988 |     "name": "ipython",
 989 |     "version": 3
 990 |    },
 991 |    "file_extension": ".py",
 992 |    "mimetype": "text/x-python",
 993 |    "name": "python",
 994 |    "nbconvert_exporter": "python",
 995 |    "pygments_lexer": "ipython3",
 996 |    "version": "3.6.7"
 997 |   },
 998 |   "toc": {
 999 |    "base_numbering": 1,
1000 |    "nav_menu": {},
1001 |    "number_sections": true,
1002 |    "sideBar": true,
1003 |    "skip_h1_title": false,
1004 |    "title_cell": "Table of Contents",
1005 |    "title_sidebar": "Contents",
1006 |    "toc_cell": false,
1007 |    "toc_position": {},
1008 |    "toc_section_display": true,
1009 |    "toc_window_display": false
1010 |   },
1011 |   "varInspector": {
1012 |    "cols": {
1013 |     "lenName": 16,
1014 |     "lenType": 16,
1015 |     "lenVar": 40
1016 |    },
1017 |    "kernels_config": {
1018 |     "python": {
1019 |      "delete_cmd_postfix": "",
1020 |      "delete_cmd_prefix": "del ",
1021 |      "library": "var_list.py",
1022 |      "varRefreshCmd": "print(var_dic_list())"
1023 |     },
1024 |     "r": {
1025 |      "delete_cmd_postfix": ") ",
1026 |      "delete_cmd_prefix": "rm(",
1027 |      "library": "var_list.r",
1028 |      "varRefreshCmd": "cat(var_dic_list()) "
1029 |     }
1030 |    },
1031 |    "types_to_exclude": [
1032 |     "module",
1033 |     "function",
1034 |     "builtin_function_or_method",
1035 |     "instance",
1036 |     "_Feature"
1037 |    ],
1038 |    "window_display": false
1039 |   }
1040 |  },
1041 |  "nbformat": 4,
1042 |  "nbformat_minor": 2
1043 | }
1044 | 


--------------------------------------------------------------------------------
/Notebook/Section 5 - RNN and NLP v2-experiment2.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "<div class=\"alert alert-block alert-info\">\n",
  8 |     "<font size=\"5\"><b><center> Section 5</font></center>\n",
  9 |     "<br>\n",
 10 |     "<font size=\"5\"><b><center>Recurrent Neural Network in PyTorch with an Introduction to Natural Language Processing</font></center>\n",
 11 |     "</div>"
 12 |    ]
 13 |   },
 14 |   {
 15 |    "cell_type": "markdown",
 16 |    "metadata": {},
 17 |    "source": [
 18 |     "Credit: This example is obtained from the following book:\n",
 19 |     "\n",
 20 |     "Subramanian, Vishnu. 2018. \"*Deep Learning with PyTorch: A Practical Approach to Building Neural Network Models Using PyTorch.*\" Birmingham, U.K., Packt Publishing."
 21 |    ]
 22 |   },
 23 |   {
 24 |    "cell_type": "markdown",
 25 |    "metadata": {},
 26 |    "source": [
 27 |     "# Simple Text Processing"
 28 |    ]
 29 |   },
 30 |   {
 31 |    "cell_type": "markdown",
 32 |    "metadata": {},
 33 |    "source": [
 34 |     "## Typically Data Preprocessing Steps before Modeling Training for NLP Applications\n",
 35 |     "\n",
 36 |     "* Read the data from disk\n",
 37 |     "* Tokenize the text\n",
 38 |     "* Create a mapping from word to a unique integer\n",
 39 |     "* Convert the text into lists of integers\n",
 40 |     "* Load the data in whatever format your deep learning framework requires\n",
 41 |     "* Pad the text so that all the sequences are the same length, so you can process them in batch"
 42 |    ]
 43 |   },
 44 |   {
 45 |    "cell_type": "markdown",
 46 |    "metadata": {},
 47 |    "source": [
 48 |     "## Word Embedding"
 49 |    ]
 50 |   },
 51 |   {
 52 |    "cell_type": "markdown",
 53 |    "metadata": {},
 54 |    "source": [
 55 |     "Word embedding is a very popular way of representing text data in problems that are solved by deep learning algorithms\n",
 56 |     "\n",
 57 |     "Word embedding provides a dense representation of a word filled with floating numbers. \n",
 58 |     "\n",
 59 |     "It drastically reduces the dimension of the dictionary\n",
 60 |     "\n"
 61 |    ]
 62 |   },
 63 |   {
 64 |    "cell_type": "markdown",
 65 |    "metadata": {
 66 |     "heading_collapsed": true
 67 |    },
 68 |    "source": [
 69 |     "###  `Torchtext` and Training word embedding by building a sentiment classifier"
 70 |    ]
 71 |   },
 72 |   {
 73 |    "cell_type": "markdown",
 74 |    "metadata": {
 75 |     "hidden": true
 76 |    },
 77 |    "source": [
 78 |     "Torchtext takes a declarative approach to loading its data: \n",
 79 |     "\n",
 80 |     "  * you tell torchtext how you want the data to look like, and torchtext handles it for you\n",
 81 |     "  \n",
 82 |     "  * Declaring a Field: The Field specifies how you want a certain field to be processed\n",
 83 |     "  \n",
 84 |     "The `Field` class is a fundamental component of torchtext and is what makes preprocessing very easy\n",
 85 |     "\n"
 86 |    ]
 87 |   },
 88 |   {
 89 |    "cell_type": "markdown",
 90 |    "metadata": {},
 91 |    "source": [
 92 |     "### Load `torchtext.datasets`"
 93 |    ]
 94 |   },
 95 |   {
 96 |    "cell_type": "markdown",
 97 |    "metadata": {},
 98 |    "source": [
 99 |     "# Use LSTM for Sentiment Classification"
100 |    ]
101 |   },
102 |   {
103 |    "cell_type": "markdown",
104 |    "metadata": {},
105 |    "source": [
106 |     "1. Preparing the data\n",
107 |     "2. Creating the batches \n",
108 |     "3. Creating the network \n",
109 |     "4. Training the model"
110 |    ]
111 |   },
112 |   {
113 |    "cell_type": "code",
114 |    "execution_count": 1,
115 |    "metadata": {
116 |     "ExecuteTime": {
117 |      "end_time": "2019-04-08T22:42:54.266651Z",
118 |      "start_time": "2019-04-08T22:42:15.332306Z"
119 |     }
120 |    },
121 |    "outputs": [],
122 |    "source": [
123 |     "from torchtext import data, datasets\n",
124 |     "from torchtext.vocab import GloVe,FastText,CharNGram\n",
125 |     "\n",
126 |     "TEXT = data.Field(lower=True, fix_length=100,batch_first=False)\n",
127 |     "LABEL = data.Field(sequential=False,)\n",
128 |     "\n",
129 |     "train, test = datasets.imdb.IMDB.splits(TEXT, LABEL)\n",
130 |     "\n",
131 |     "TEXT.build_vocab(train, vectors=GloVe(name='6B', dim=300),max_size=10000,min_freq=10)\n",
132 |     "LABEL.build_vocab(train,)"
133 |    ]
134 |   },
135 |   {
136 |    "cell_type": "code",
137 |    "execution_count": 2,
138 |    "metadata": {
139 |     "ExecuteTime": {
140 |      "end_time": "2019-04-08T22:44:03.691481Z",
141 |      "start_time": "2019-04-08T22:44:03.675373Z"
142 |     }
143 |    },
144 |    "outputs": [
145 |     {
146 |      "data": {
147 |       "text/plain": [
148 |        "10002"
149 |       ]
150 |      },
151 |      "execution_count": 2,
152 |      "metadata": {},
153 |      "output_type": "execute_result"
154 |     }
155 |    ],
156 |    "source": [
157 |     "len(TEXT.vocab.vectors)"
158 |    ]
159 |   },
160 |   {
161 |    "cell_type": "code",
162 |    "execution_count": 3,
163 |    "metadata": {
164 |     "ExecuteTime": {
165 |      "end_time": "2019-04-08T22:44:05.098565Z",
166 |      "start_time": "2019-04-08T22:44:05.088897Z"
167 |     }
168 |    },
169 |    "outputs": [
170 |     {
171 |      "name": "stderr",
172 |      "output_type": "stream",
173 |      "text": [
174 |       "The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.\n",
175 |       "The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.\n"
176 |      ]
177 |     }
178 |    ],
179 |    "source": [
180 |     "train_iter, test_iter = data.BucketIterator.splits((train, test), batch_size=32, device=-1)\n",
181 |     "train_iter.repeat = False\n",
182 |     "test_iter.repeat = False"
183 |    ]
184 |   },
185 |   {
186 |    "cell_type": "code",
187 |    "execution_count": 4,
188 |    "metadata": {
189 |     "ExecuteTime": {
190 |      "end_time": "2019-04-08T22:44:06.275137Z",
191 |      "start_time": "2019-04-08T22:44:06.270580Z"
192 |     }
193 |    },
194 |    "outputs": [],
195 |    "source": [
196 |     "import torch.nn as nn\n",
197 |     "import torch.optim as optim\n",
198 |     "import torch.nn.functional as F\n",
199 |     "from torch.autograd import Variable"
200 |    ]
201 |   },
202 |   {
203 |    "cell_type": "code",
204 |    "execution_count": 5,
205 |    "metadata": {
206 |     "ExecuteTime": {
207 |      "end_time": "2019-04-08T22:44:07.313294Z",
208 |      "start_time": "2019-04-08T22:44:07.298139Z"
209 |     }
210 |    },
211 |    "outputs": [],
212 |    "source": [
213 |     "class IMDBRnn(nn.Module):\n",
214 |     "    \n",
215 |     "    def __init__(self,vocab,hidden_size,n_cat,bs=1,nl=2):\n",
216 |     "        super().__init__()\n",
217 |     "        self.hidden_size = hidden_size\n",
218 |     "        self.bs = bs\n",
219 |     "        self.nl = nl\n",
220 |     "        self.e = nn.Embedding(n_vocab,hidden_size)\n",
221 |     "        self.rnn = nn.LSTM(hidden_size,hidden_size,nl)\n",
222 |     "        self.fc2 = nn.Linear(hidden_size,n_cat)\n",
223 |     "        self.softmax = nn.LogSoftmax(dim=-1)\n",
224 |     "\n",
225 |     "    def forward(self,inp):\n",
226 |     "        bs = inp.size()[1]\n",
227 |     "        if bs != self.bs:\n",
228 |     "            self.bs = bs\n",
229 |     "        e_out = self.e(inp)\n",
230 |     "        h0 = c0 = Variable(e_out.data.new(*(self.nl,self.bs,self.hidden_size)).zero_())\n",
231 |     "        rnn_o,_ = self.rnn(e_out,(h0,c0))\n",
232 |     "        rnn_o = rnn_o[-1]\n",
233 |     "        fc = F.dropout(self.fc2(rnn_o),p=0.8)\n",
234 |     "        return self.softmax(fc)"
235 |    ]
236 |   },
237 |   {
238 |    "cell_type": "code",
239 |    "execution_count": 6,
240 |    "metadata": {
241 |     "ExecuteTime": {
242 |      "end_time": "2019-04-08T22:44:08.565456Z",
243 |      "start_time": "2019-04-08T22:44:08.560302Z"
244 |     }
245 |    },
246 |    "outputs": [],
247 |    "source": [
248 |     "n_vocab = len(TEXT.vocab)\n",
249 |     "n_hidden = 100"
250 |    ]
251 |   },
252 |   {
253 |    "cell_type": "code",
254 |    "execution_count": 7,
255 |    "metadata": {
256 |     "ExecuteTime": {
257 |      "end_time": "2019-04-08T22:44:12.310520Z",
258 |      "start_time": "2019-04-08T22:44:12.271214Z"
259 |     },
260 |     "code_folding": []
261 |    },
262 |    "outputs": [],
263 |    "source": [
264 |     "model = IMDBRnn(n_vocab,n_hidden,n_cat=3,bs=32)\n",
265 |     "#model = model.cuda()\n",
266 |     "\n",
267 |     "optimizer = optim.Adam(model.parameters(),lr=1e-3)\n",
268 |     "\n",
269 |     "def fit(epoch,model,data_loader,phase='training',volatile=False):\n",
270 |     "    if phase == 'training':\n",
271 |     "        model.train()\n",
272 |     "    if phase == 'validation':\n",
273 |     "        model.eval()\n",
274 |     "        volatile=True\n",
275 |     "    running_loss = 0.0\n",
276 |     "    running_correct = 0\n",
277 |     "    for batch_idx , batch in enumerate(data_loader):\n",
278 |     "        text , target = batch.text , batch.label\n",
279 |     "#         if is_cuda:\n",
280 |     "#             text,target = text.cuda(),target.cuda()\n",
281 |     "        \n",
282 |     "        if phase == 'training':\n",
283 |     "            optimizer.zero_grad()\n",
284 |     "        output = model(text)\n",
285 |     "        loss = F.nll_loss(output,target)\n",
286 |     "        \n",
287 |     "        #running_loss += F.nll_loss(output,target,size_average=False).data[0]\n",
288 |     "        running_loss += F.nll_loss(output,target,size_average=False).data\n",
289 |     "        preds = output.data.max(dim=1,keepdim=True)[1]\n",
290 |     "        running_correct += preds.eq(target.data.view_as(preds)).cpu().sum()\n",
291 |     "        if phase == 'training':\n",
292 |     "            loss.backward()\n",
293 |     "            optimizer.step()\n",
294 |     "    \n",
295 |     "    loss = running_loss/len(data_loader.dataset)\n",
296 |     "    accuracy = 100. * running_correct/len(data_loader.dataset)\n",
297 |     "    print(\"epoch: \", epoch, \"loss: \", loss, \"accuracy: \", accuracy)\n",
298 |     "    #print(f'{phase} loss is {loss:{5}.{2}} and {phase} accuracy is {running_correct}/{len(data_loader.dataset)}{accuracy:{10}.{4}}')\n",
299 |     "    return loss,accuracy"
300 |    ]
301 |   },
302 |   {
303 |    "cell_type": "code",
304 |    "execution_count": 8,
305 |    "metadata": {
306 |     "ExecuteTime": {
307 |      "end_time": "2019-04-08T23:54:32.198000Z",
308 |      "start_time": "2019-04-08T22:45:14.817732Z"
309 |     }
310 |    },
311 |    "outputs": [
312 |     {
313 |      "name": "stderr",
314 |      "output_type": "stream",
315 |      "text": [
316 |       "/Users/jeffrey/anaconda2/envs/dl/lib/python3.6/site-packages/torch/nn/_reduction.py:49: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
317 |       "  warnings.warn(warning.format(ret))\n"
318 |      ]
319 |     },
320 |     {
321 |      "name": "stdout",
322 |      "output_type": "stream",
323 |      "text": [
324 |       "epoch:  1 loss:  tensor(1.0033) accuracy:  tensor(49)\n",
325 |       "epoch:  1 loss:  tensor(1.0032) accuracy:  tensor(49)\n",
326 |       "epoch:  2 loss:  tensor(0.9953) accuracy:  tensor(50)\n",
327 |       "epoch:  2 loss:  tensor(1.0013) accuracy:  tensor(50)\n",
328 |       "epoch:  3 loss:  tensor(1.0006) accuracy:  tensor(50)\n",
329 |       "epoch:  3 loss:  tensor(1.0029) accuracy:  tensor(49)\n",
330 |       "epoch:  4 loss:  tensor(1.0017) accuracy:  tensor(49)\n",
331 |       "epoch:  4 loss:  tensor(1.0013) accuracy:  tensor(50)\n",
332 |       "epoch:  5 loss:  tensor(0.9996) accuracy:  tensor(49)\n",
333 |       "epoch:  5 loss:  tensor(1.0014) accuracy:  tensor(49)\n",
334 |       "epoch:  6 loss:  tensor(1.0021) accuracy:  tensor(49)\n",
335 |       "epoch:  6 loss:  tensor(0.9967) accuracy:  tensor(50)\n",
336 |       "epoch:  7 loss:  tensor(0.9996) accuracy:  tensor(50)\n",
337 |       "epoch:  7 loss:  tensor(1.0007) accuracy:  tensor(50)\n",
338 |       "epoch:  8 loss:  tensor(0.9939) accuracy:  tensor(51)\n",
339 |       "epoch:  8 loss:  tensor(0.9715) accuracy:  tensor(54)\n",
340 |       "epoch:  9 loss:  tensor(0.9452) accuracy:  tensor(57)\n",
341 |       "epoch:  9 loss:  tensor(0.9408) accuracy:  tensor(57)\n",
342 |       "epoch:  10 loss:  tensor(0.9110) accuracy:  tensor(59)\n",
343 |       "epoch:  10 loss:  tensor(0.9207) accuracy:  tensor(58)\n",
344 |       "epoch:  11 loss:  tensor(0.8968) accuracy:  tensor(61)\n",
345 |       "epoch:  11 loss:  tensor(0.9228) accuracy:  tensor(59)\n",
346 |       "epoch:  12 loss:  tensor(0.8757) accuracy:  tensor(62)\n",
347 |       "epoch:  12 loss:  tensor(0.9147) accuracy:  tensor(60)\n",
348 |       "epoch:  13 loss:  tensor(0.8700) accuracy:  tensor(62)\n",
349 |       "epoch:  13 loss:  tensor(0.9154) accuracy:  tensor(59)\n",
350 |       "epoch:  14 loss:  tensor(0.8537) accuracy:  tensor(63)\n",
351 |       "epoch:  14 loss:  tensor(0.9107) accuracy:  tensor(60)\n",
352 |       "epoch:  15 loss:  tensor(0.8455) accuracy:  tensor(63)\n",
353 |       "epoch:  15 loss:  tensor(0.9230) accuracy:  tensor(60)\n",
354 |       "epoch:  16 loss:  tensor(0.8319) accuracy:  tensor(64)\n",
355 |       "epoch:  16 loss:  tensor(0.9202) accuracy:  tensor(60)\n",
356 |       "epoch:  17 loss:  tensor(0.8274) accuracy:  tensor(64)\n",
357 |       "epoch:  17 loss:  tensor(0.9392) accuracy:  tensor(60)\n",
358 |       "epoch:  18 loss:  tensor(0.8199) accuracy:  tensor(65)\n",
359 |       "epoch:  18 loss:  tensor(0.9498) accuracy:  tensor(60)\n",
360 |       "epoch:  19 loss:  tensor(0.8122) accuracy:  tensor(65)\n",
361 |       "epoch:  19 loss:  tensor(0.9666) accuracy:  tensor(60)\n",
362 |       "69.28943564891816\n",
363 |       "Execution Time:  69.3 minutes\n"
364 |      ]
365 |     }
366 |    ],
367 |    "source": [
368 |     "import time\n",
369 |     "start = time.time()\n",
370 |     "\n",
371 |     "train_losses , train_accuracy = [],[]\n",
372 |     "val_losses , val_accuracy = [],[]\n",
373 |     "\n",
374 |     "for epoch in range(1,20):\n",
375 |     "\n",
376 |     "    epoch_loss, epoch_accuracy = fit(epoch,model,train_iter,phase='training')\n",
377 |     "    val_epoch_loss , val_epoch_accuracy = fit(epoch,model,test_iter,phase='validation')\n",
378 |     "    train_losses.append(epoch_loss)\n",
379 |     "    train_accuracy.append(epoch_accuracy)\n",
380 |     "    val_losses.append(val_epoch_loss)\n",
381 |     "    val_accuracy.append(val_epoch_accuracy)\n",
382 |     "    \n",
383 |     "end = time.time()\n",
384 |     "print((end-start)/60)\n",
385 |     "print(\"Execution Time: \", round(((end-start)/60),1), \"minutes\")"
386 |    ]
387 |   },
388 |   {
389 |    "cell_type": "code",
390 |    "execution_count": 9,
391 |    "metadata": {
392 |     "ExecuteTime": {
393 |      "end_time": "2019-04-09T00:34:59.546310Z",
394 |      "start_time": "2019-04-09T00:34:58.418822Z"
395 |     }
396 |    },
397 |    "outputs": [
398 |     {
399 |      "data": {
400 |       "text/plain": [
401 |        "<matplotlib.legend.Legend at 0x149e1f5c0>"
402 |       ]
403 |      },
404 |      "execution_count": 9,
405 |      "metadata": {},
406 |      "output_type": "execute_result"
407 |     },
408 |     {
409 |      "data": {
410 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VOX1wPHvSQAxiKxRIoEEFSUJ\nWyAilgpaEQMuKGqVxgqKpe60LgVFQFFcfooilaLYumGUUtxwQdxAxaolrLIqsgYQArIakO38/nhv\nwhCyTJJJ7kxyPs9zn5l573vvnJlM5sy9911EVTHGGGOi/A7AGGNMeLCEYIwxBrCEYIwxxmMJwRhj\nDGAJwRhjjMcSgjHGGMASgjHGGI8lBGOMMYAlBGOMMZ4afgdQGo0bN9bExES/wzDGmIgyZ86cLaoa\nW1K9iEoIiYmJZGVl+R2GMcZEFBFZE0w9O2VkjDEGsIRgjDHGYwnBGGMMEGHXEIwxlW///v1kZ2ez\nd+9ev0MxJahduzbx8fHUrFmzTNtbQjDGFCs7O5u6deuSmJiIiPgdjimCqrJ161ays7Np0aJFmfZR\n5U8ZZWZCYiJERbnbzEy/IzImsuzdu5dGjRpZMghzIkKjRo3KdSRXpY8QMjNh4EDIzXWP16xxjwEy\nMvyLy5hIY8kgMpT371SljxCGDnXJoAuzOJ+PaMVSJHc3Q4f6FNChQ1COKUvD5WgnXOIwxoRWUEcI\nIvICcBGwWVVbF7JegKeBXkAu0F9V53rr+gH3eVUfUtWXvfKOwEvAscAHwCAN8QTPa9e622E8yAV8\nlF++bU19aNsMmhWxxMdD7drBP9GePbBxI/z0k7sNvO/d5q76iVrbN7OfmmyJbkLtk+OIbRMHcUUs\nsbHuG9cTLkc74RKHqT62b9/Oa6+9xs0331zqbXv16sVrr71G/fr1i6wzfPhwunbtSvfu3csTJnC4\n82zjxo3LvS8/SDDfwSLSFdgNvFJEQugF3IZLCGcCT6vqmSLSEMgC0gAF5gAdVXWbiPwPuB34FpcQ\nxqrqtOLiSEtL09L0VE5MdF9YTcmmBatoxjqasY7kuuvod+46WOctW7YcvXFs7NGJAgr9smfnzqO3\nj4qCE0+EuDjWH2zCx4viWH/wRGqxjzg2Eh+1kdQmG6mXuxG2bz96++jo/O2Ji+O1mXGs2N2EjcTx\nK8cgKILSuKHy6CN6+OgjcCmkbG7WId7/MIoZ21JZ17wL9z9cq1Rf5HnvaUEJCbB6dXD7yMx0R29r\n10Lz5jBqlCWTcLZ06VKSkpKCrh/qv+/q1au56KKLWLRo0VHrDhw4QI0a4XPmOxwSQmF/LxGZo6pp\nJW6sqkEtQCKwqIh1zwF9Ax4vB+KAvsBzBet565YFlB9Rr6ilY8eOWhqvvqoaE3PkN2JMjCs/Qm6u\n6vffq376qepLL6k++KDqwIGqPXuqtm6tWq/e4R3UqaN66qmqZ5+teuWVqrffrvrww6ovvqg6bZrq\n/PmqP/2keuBA/u4TEgp+U7slISHg+VeuVP3qK9UpU1SfeUZ16FDV6693MbRvrxs5UQ8ihe+ojMtO\njtN3oy/Rb68br7pqVVDvqRQRgkhwf5NJz+/US475UB9kqP6bK7UJGwr/m5iwsWTJkqDrBv0/VwpX\nXXWV1q5dW9u1a6d33XWXzpgxQ3/729/qxRdfrC1btlRV1d69e2uHDh00OTlZn3vuufxtExISNCcn\nR1etWqWtWrXSG264QZOTk/X888/X3NxcVVXt16+f/uc//8mvP3z4cE1NTdXWrVvr0qVLVVV18+bN\n2r17d01OTtYBAwZo8+bNNScn56hY855PVXX06NGakpKiKSkp+tRTT6mq6u7du7VXr17atm1bTUlJ\n0UmTJqmq6uDBgzUpKUnbtGmjd955Z9nfLC387wVkaTDf88FU0pITwnvAbwMef4o7KrgLuC+gfJhX\nlgZ8ElB+NvBeSTGUNiGoug9iQoL7wkpIKMcHc8cO1Z07y7Rpeb9EVV3s0ezXJmzQ5qzWZqzReNZq\np6bZquvXq27YoLpxo0tGmzap5uSobtmiunWr6s8/q27bpinNduhx7NSGbNGLeUfHcZOuJPFwQKef\nrjpokEts3j9LYXEUm9wK2rRJ9Y03VP/yF9UOHfQAUaqg+4nW/UTri/Qrfnvju9IkhFJ/PoKwatUq\nTUlJyX88Y8YMjYmJ0ZUrV+aXbd26VVVVc3NzNSUlRbds2eLFczghREdH67x581RV9corr9SJEyeq\n6tEJYezYsaqqOm7cOB0wYICqqt5yyy368MMPq6rqtGnTFCg2IWRlZWnr1q119+7dumvXLk1OTta5\nc+fqlClT9IYbbsivv337dt2yZYuedtppeujQIVVV3bZtW9nfLC1fQgj7i8oiMlBEskQkKycnp9Tb\nZ2S4UxmHDrnbMh+6Hn881K1bpk2bNy9deWFGjYJjYmrwE3GsJYF1NOfnmGbc/lhTOOkkd2qpSRN3\nmumEE6BxY2jUCBo2hAYNoH59lmQfz27q8jONeJdLuIV/cDIracUyGDMGWrSA556Dnj3ddj17wtNP\nw/Ll+RfDR42CmJgjY4uJceWoujd54kR3YaFVKxfP5ZfDs8/C8cfzMEM5n4+oz3ZGcyf9eZkOzMm/\n3mMiW1F/x1D/fTt16nREW/uxY8fSrl07OnfuzLp16/jhhx+O2qZFixa0b98egI4dO7K6iHOcffr0\nOarOrFmzuPrqqwFIT0+nQYMGxcY3a9YsLrvsMurUqcNxxx1Hnz59+PLLL2nTpg0ff/wxgwcP5ssv\nv6RevXrUq1eP2rVrM2DAAN58801iCv6DVaJQJYT1QLOAx/FeWXHl8YWUH0VVJ6hqmqqmxcaWOHpr\nWCr2SzRIGRkwYYI7Vy/ibidMKF2CKzwBCXsTTodBg2DaNPj5Z/jwQ/jzn2HVKvjLX9wX+8knw003\nkVF3Ki88vYuEBIjiEN3jFjPjqmfJ+CDDPUGLFnDttTB5Mpx6Kjz6KHz1lbtOMmMG/0oYySeczy8c\nx8PcSw6NGc2dNG8W0vYExieh+PETjDp16uTfnzlzJp988glff/01CxYsIDU1tdC2+Mccc0z+/ejo\naA4cOFDovvPqFVenrE477TTmzp1LmzZtuO+++xg5ciQ1atTgf//7H1dccQXvvfce6enpIX3O0ghV\nQpgKXCtOZ2CHqm4EpgM9RKSBiDQAegDTvXU7RaSz10LpWuCdEMUScuVtZhmKL/O8/ZTnaCeoxHTs\nsXDBBe6IYdkyWLkSxo+Hdu3g1Vehd2+uurkRq088k4MNY/l4Y2s6vXgTzJgBv/kN/P3vMH8+bN0K\n770Hgwe7cu+fLDCGndRjBA9wDp/z8uVTS/diTFgKxY+fgurWrcuuXbuKXL9jxw4aNGhATEwMy5Yt\n45tvvin7kxWhS5cuTJ48GYCPPvqIbdu2FVv/7LPP5u233yY3N5dffvmFt956i7PPPpsNGzYQExPD\nNddcw913383cuXPZvXs3O3bsoFevXjz11FMsWLAg5PEHK9hmp68D5wCNRSQbGAHUBFDVZ3GthHoB\nK3DNTq/z1v0sIg8Cs71djVTVn737N3O42ek0bwk7oWpmmZHhf0uavOcvVQuQFi3gxhvdsm+f+7X/\n4Yfw3/9C797QtSucfbY7ggiiU0zBGKY3G8iOA8/Q7b274dGeUKtWUK/FWiqFpzJ9xkrQqFEjunTp\nQuvWrenZsycXXnjhEevT09N59tlnSUpK4vTTT6dz587leAWFGzFiBH379mXixImcddZZNGnShLrF\nnELu0KED/fv3p1OnTgDccMMNpKamMn36dO6++26ioqKoWbMm48ePZ9euXfTu3Zu9e/eiqjz55JMh\njz9owVxoCJelLBeVy6siLpKZAt5/372pY8YEVb0iWrKYopXmonJVtXfvXt2/f7+qqv73v//Vdu3a\n+RxR0ar0RWW/VdZFsmqtZ0/o3h0eeMBdwyhBXg/0QLm5+NcD3VR5a9eu5YwzzqBdu3bcfvvtPP/8\n836HVCHCp0dHmGrevPCOWKG+SFaticDo0dC+PTz0EJRwyGxJ2lS2li1bMm/ePL/DqHB2hFCCirhI\nZgrRti0MGADPPAMrVhRbtbJashhT3VhCKEGoWgiZIDz4oLuoPHhwsdUsSRtTMSwhBCFkndtM8Zo0\ngSFD4M034YsviqxmSdqYimEJwYSXO+5wo83ecYfLwEWwJG1M6FlCMOElJgYefhjmzIHXXvM7GhOh\njjvuOAA2bNjAFVdcUWidc845h5JGTx4zZgy5AU3aevXqxfbCRicupfvvv58nnnii3PsJNUsIJvxk\nZEDHjnDPPUe3LzWmFE466SSmTJlS5u0LJoQPPvig2LkVIp0lBBN+oqJc09Ps7BKboJqqb8iQIYwb\nNy7/cd6v6927d3PeeefRoUMH2rRpwzvvHD36zerVq2nd2k3hsmfPHq6++mqSkpK47LLL2LNnT369\nm266ibS0NFJSUhgxYgTgBszbsGED5557Lueeey7g5jvY4s2f8uSTT9K6dWtat27NmDFj8p8vKSmJ\nP/3pT6SkpNCjR48jnqcw8+fPp3PnzrRt25bLLrssf1iMsWPHkpycTNu2bfMH1vv8889p37497du3\nJzU1tdghPcokmN5r4bL40VPZ+Oiyy9z8Exs3+h1JtXZEz9dBg1S7dQvtMmhQsc8/d+5c7dq1a/7j\npKQkXbt2re7fv1937Nihqqo5OTl6yimn5A8hXadOHVU9cujs0aNH63XXXaeqqgsWLNDo6GidPXu2\nqh4ePvvAgQParVs3XbBggaoeOb9B4OOihrcubpjtQCNGjNDHH39cVVXbtGmjM2fOVFXVYcOG6SDv\n/YiLi9O9e/eq6uEhsS+66CKdNWuWqqru2rUrv/d0IOupbKqmxx5z4ycNG+Z3JMZHqampbN68mQ0b\nNrBgwQIaNGhAs2bNUFXuvfde2rZtS/fu3Vm/fj2bNm0qcj9ffPEF11xzDQBt27albdu2+esmT55M\nhw4dSE1NZfHixSxZsqTYmIoa3hqCH2Yb3MB827dvp1u3bgD069ePL7wWdm3btiUjI4NXX301f1a4\nLl26cMcddzB27Fi2b98e8tnirKeyCV8tW8Ktt7qRV2+7zXVeM/7yTo1UtiuvvJIpU6bw008/cdVV\nVwGQmZlJTk4Oc+bMoWbNmiQmJhY67HVJVq1axRNPPMHs2bNp0KAB/fv3L9N+8hQcZrukU0ZFef/9\n9/niiy949913GTVqFN999x1Dhgzhwgsv5IMPPqBLly5Mnz6dVq1alTnWguwIwYS3YcPcBD933pk/\nSY+pfq666iomTZrElClTuPLKKwH36/qEE06gZs2azJgxgzWFjTEToGvXrrzmtVxbtGgRCxcuBGDn\nzp3UqVOHevXqsWnTJqZNOzzwclFDbxc1vHVp1atXjwYNGuQfXUycOJFu3bpx6NAh1q1bx7nnnstj\njz3Gjh072L17Nz/++CNt2rRh8ODBnHHGGSxbtqzUz1kcO0Iw4a1BAxg+3E3UM20a9Orld0TGBykp\nKezatYumTZsSFxcHQEZGBhdffDFt2rQhLS2txF/KN910E9dddx1JSUkkJSXRsWNHANq1a0dqaiqt\nWrWiWbNmdOnSJX+bgQMHkp6ezkknncSMGTPyy4sa3rq400NFefnll7nxxhvJzc3l5JNP5sUXX+Tg\nwYNcc8017NixA1Xl9ttvp379+gwbNowZM2YQFRVFSkoKPXv2LPXzFUc0gn51paWlaUnthk0VtG8f\ntG4NNWrAwoXu1lSapUuXkpSU5HcYJkiF/b1EZI6qppW0rZ0yMuGvVi14/HFYuhSq6LDDxoQDSwgm\nMlxyCXTr5k4f7djhdzTGVElBJQQRSReR5SKyQkSGFLI+QUQ+FZGFIjJTROK98nNFZH7AsldELvXW\nvSQiqwLWtQ/tSzNViojrpLZ1qxvawlSqSDq1XJ2V9+9UYkIQkWhgHNATSAb6ikhygWpPAK+oaltg\nJPCIF9wMVW2vqu2B3+HmW/4oYLu789ar6vxyvRJT9XXoANde65o+rlrldzTVRu3atdm6daslhTCn\nqmzdupXatWuXeR/BXJ3rBKxQ1ZUAIjIJ6A0E9txIBu7w7s8A3i5kP1cA01TVBqcxZTdqFEye7MY5\nmjTJ72iqhfj4eLKzs8nJyfE7FFOC2rVrEx8fX+btg0kITYF1AY+zgTML1FkA9AGeBi4D6opII1Xd\nGlDnaqDgwDSjRGQ48CkwRFV/LU3wphpq2hTuvhtGjoRBg+Css/yOqMqrWbMmLVq08DsMUwlCdVH5\nLqCbiMwDugHrgYN5K0UkDmgDTA/Y5h6gFXAG0BAodJosERkoIlkikmW/UAzgEkJcnJszwU5jGBMy\nwSSE9UCzgMfxXlk+Vd2gqn1UNRUY6pUFDhr+e+AtVd0fsE3eiGW/Ai/iTk0dRVUnqGqaqqbFxsYG\n9aJMFXfccfDQQ/DNN+70kTEmJIJJCLOBliLSQkRq4U79TA2sICKNRSRvX/cALxTYR1/g9QLbxHm3\nAlwKLCp9+Kba6tcP2rVz8y+XY9wZY8xhJSYEVT0A3Io73bMUmKyqi0VkpIhc4lU7B1guIt8DJwL5\n052LSCLuCOPzArvOFJHvgO+AxsBD5XolpnqJjobRo2HNGhg/3u9ojKkSbOgKE9l69IC5c2HlSjj+\neL+jMSYs2dAVpnp4+GHXWW30aL8jMSbiWUIwkS0tDa680iWEzZv9jsaYiGYJwUS+Bx90F5ZHjSq5\nrjGRZvNmN/z7rxXfTcsSgol8p58O11/vLi6XYTx6Y8LW1q3QvTtMmACLF1f401lCMFXD8OGu5dGI\nEX5HYkxobNsG558PP/wA777rxvKqYJYQTNUQH+/mXZ44Eb77zu9ojCmfHTvgggvcUcFbb8F551XK\n01pCMFXHkCGu6enQoX5HYkzZ7drlpoqdNw+mTIH09Ep7aksIpupo2ND1XH73XfjqK7+jMab0cnPh\n4ovh22/daL4XX1ypT28JwVQtt98OTZq4o4UI6nRpDHv2QO/e8OWX7tTn5ZdXegiWEEzVUqeOu8A8\naxZMm+Z3NMYE59dfoU8f+PRTeOEF6NvXlzAsIZiq54Yb4JRT3CQ6hw75HY0xxdu3D37/e/jwQ9e8\ntF8/30KxhGCqnpo1XWe1hQttVjUT3g4cgD/8AaZOhWeecT9mfGQJwVRNV13lhsceNsz9AjMm3Bw8\n6OYIf+MNePJJuOUWvyOyhGCqqKgoeOQRNwrqP//pdzTGHOnQIRgwAF5/HR59FP76V78jAiwhmKos\nPR26dnXzL+/e7Xc0xjiHDsGNN8LLL8MDD7im0mHCEoKpukTcUcKmTfD0035HY4xrCn377fD883Dv\nve6UZhixhGCqtt/8Bi65BP7v/9xAYcb4RRXuvBPGjXO3Dz3kfrSEkaASgoiki8hyEVkhIkMKWZ8g\nIp+KyEIRmSki8QHrDorIfG+ZGlDeQkS+9fb5b2++ZmNCb9QoNxzAo4/6HYmprlTdEcFTT7kxtx5/\nPOySAQSREEQkGhgH9ASSgb4iklyg2hPAK6raFhgJPBKwbo+qtveWSwLKHwOeUtVTgW3AgHK8DmOK\n1rq1a83x979Ddrbf0Zjq6IEH3A+SP//Znb4Mw2QAwR0hdAJWqOpKVd0HTAJ6F6iTDHzm3Z9RyPoj\niIgAvwOmeEUvA5cGG7QxpXb//e5X2gMP+B2JqW4efth97q67Dv7xj7BNBhBcQmgKrAt4nO2VBVoA\n9PHuXwbUFZFG3uPaIpIlIt+ISN6XfiNgu6oeKGafAIjIQG/7rJycnCDCNaYQiYlw001uWIBly/yO\nxlQHW7bANde40XczMtyF5KjwvmwbqujuArqJyDygG7AeOOitS1DVNOAPwBgROaU0O1bVCaqapqpp\nsbGxIQrXVEv33gsxMWHXssNUMaquf0FSEkye7MbWeuklN4FTmAsmIawHmgU8jvfK8qnqBlXto6qp\nwFCvbLt3u967XQnMBFKBrUB9EalR1D6NCbkTTnCtO6ZMgdmzycx0Bw5RUe42M9PvAE3Ey852rdr+\n8Ado0QLmzHGni2rUKHnbMBBMQpgNtPRaBdUCrgamBlYQkcYikreve4AXvPIGInJMXh2gC7BEVRV3\nreEKb5t+wDvlfTHGlOiOO6BxYzZedw8DB8KaNe4H3Zo1MHCgJQVTRocOuYHpUlLciKWjR8PXX0Ob\nNn5HViolJgTvPP+twHRgKTBZVReLyEgRyWs1dA6wXES+B04ERnnlSUCWiCzAJYBHVXWJt24wcIeI\nrMBdU/hXiF6TMUXzZlSLW/wpZ+V+csSq3FybbM2UwYoVborLP/8Z0tLcFK533BERp4gKEo2gSUTS\n0tI0KyvL7zBMpNu7lzXHnk4OsZzBbOBwqw8RGzHbBOnAAdevYPhwOOYYd1Rw/fVh2YpIROZ413KL\nFd6XvI2pCLVr8/dGD5DGHC7njSNWNW/uU0wmsixcCGedBX/7G1xwASxZ4garC8NkUBqWEEy11OGp\nP7JEkhnFUKJxrZ9jYlynZmOK9Ouv7oigY0d34enf/4a33oKTTvI7spCwhGCqpT/8MZqcQaM4ne+5\njpdISHDXBDMy/I7MhK2vv4bUVDf5Ut++sHSpm+kswo8KAtk1BFN9qbrB79auhcWLoX59vyMy4Wj3\nbrjvPhg7Fpo1g+eec0OrRxC7hmBMSURgzBjIyXG/+A4eLHkbU718/LFrOvr0025Gs0WLIi4ZlIYl\nBFO9nXmmm8v2ww9dT2Zj8rz+OvTo4VoQffmlGxyxbl2/o6pQkdF9zpiKNHAgzJ/v5kxo1871MjXV\n26xZ0L+/m3Hvww/h2GP9jqhS2BGCMeBOHXXt6poOzpnjdzTGTz/8AL17u/FM3nqr2iQDsIRgjFOr\nFvznP268o0svddNumupn61bo1csNcPXBB9Cwod8RVSpLCMbkOeEEeOcd96Vw+eWuzbmpPvbudT8G\n1q1zn4NTSjUwc5VgCcGYQO3bu6GKv/oKbr3VNU01VZ+qG3Zi1ix45RXXHLkasovKxhT0+9/DggVu\npqvUVLj5Zr8jMhVt+HDXquiRR9zfv5qyIwRjCvPgg3DRRTBoEMyc6Xc0piK9+CI89JBrUDB4sN/R\n+MoSgjGFiYqCV1+FU0+FK6+E1av9jshUhM8+c82Ozz8fxo+vUsNQlIUlBGOKUq+eu7i4f7+72PjL\nL35HZEJpyRLo0wdOP921MKtZ0++IfGcJwZjinHYaTJrkJj257jq7yFxVbNoEF17o+hi8/75L/sYS\ngjElSk+HRx91vyIfecTvaEx55ea6eY83b4Z334WEBL8jChtBJQQRSReR5SKyQkSGFLI+QUQ+FZGF\nIjJTROK98vYi8rWILPbWXRWwzUsiskpE5ntL+9C9LGNC7K673JAW993nvkRMZDp0CP74R5g9G157\nzU15afKVmBBEJBoYB/QEkoG+IpJcoNoTwCuq2hYYCeT9jMoFrlXVFCAdGCMigWMM362q7b1lfjlf\nizEVRwT++U/XDDUjw42FbyLP3/4Gb74JTz7phqcwRwjmCKETsEJVV6rqPmASUPCdTAY+8+7PyFuv\nqt+r6g/e/Q3AZiA2FIEbU+mOPRbeftvd9u4N27b5HZEpjfHj3bzHt97qmhObowSTEJoC6wIeZ3tl\ngRYAfbz7lwF1RaRRYAUR6QTUAn4MKB7lnUp6SkSOKezJRWSgiGSJSFZOTk4Q4RpTgZo1gzfecM1Q\nbQ6FyDFtmksEF13kBjKs5s1LixKqi8p3Ad1EZB7QDVgP5P+niEgcMBG4TlUPecX3AK2AM4CGQKE9\nQlR1gqqmqWpabKwdXJgw8NvfwrhxMH063HNP+fen6i5wbt9e/n1VFV9+CZ9+Cj/9VP6WXQsWuN7H\n7dq53sjR0aGJsQoKZuiK9UCzgMfxXlk+73RQHwAROQ64XFW3e4+PB94HhqrqNwHbbPTu/ioiL+KS\nijGR4U9/cnMoPP64+6IpaTLmHTtg1arCl9WrXcuXY45xnaQGD4amBQ/Cq4ncXLjtNnjhhcNlDRpA\nSgokJ7vbvPtNmpT8S3/9ete8tH59eO89OO64io0/wpU4p7KI1AC+B87DJYLZwB9UdXFAncbAz6p6\nSERGAQdVdbiI1AKmAe+q6pgC+41T1Y0iIsBTwF5VPaoFUyCbU9mElf37XQ/Xb791Uy02alT0l37B\nX/9160KLFkcuCxa4gdWiouCGG2DIEHeKqrpYssT9kl+yxB15/e537v7ixYeXwOs2DRocThKBt3Fx\nLlHs2uXmuFixwg1a166df6/NZ8HOqVxiQvB21gsYA0QDL6jqKBEZCWSp6lQRuQLXskiBL4BbVPVX\nEbkGeBFYHLC7/qo6X0Q+w11gFmA+cKOq7i4uDksIJuzk5MAZZ8CaNUeW167tJljJ+7IPvN+ihfsy\nK+zX7apVrq/Diy+69QMGuMRQ1dvKv/yyG0SwTh03ZEiPHkfXUXUdygKTRN79n38+XK9+fZcccnNh\n4UJ3ZFCF50EORkgTQriwhGDC0sqVrinjSScd/vI/8UT3S7+s1qxxneH+9S/3uH9/96u5RYtQRBw+\nfvnFTV7/8svQrZvrG3DSSaXbR941mIJJYt06GDbMDWtdzVlCMKYqWLcOHnsMnn/edaq69lq4996q\nMXnL4sVu4MBly1yHv+HDoYaNyF8Rgk0INnSFMeGsWTN45hl3FHLzze4X9OmnuyOGH37wO7qyUXUX\njc84w53q+egjGDnSkkEYsIRgTCRo2hSeftolhttvh8mToVUrNwzD8uV+Rxe83buhXz93beSss1xL\nre7d/Y7KeOyUkTGRaNMmeOIJ+Mc/YM8euPpqd9olucCoMqpuruBffnEXWX/5peglb318PFxwQehb\nOH33nWtFtHw5jBjh4rU+AZXCriEYUx1s3uzG5XnmGfeFnpR0OAHkfckfOlTyfgqTkuJa5/Ts6Trj\nHVPoYAIlU3UXx2+7zbUAeu01OPfcsu3LlIklBGOqky1b3CmlRYtc082YGHcbzBJY99hj3UXeDz90\nwz18+SXs2+fW/e53LkGkp8PJJwcX165dcOONLgl07+6alJ54YsW+F+YolhCMqQSZmTB0KKxdC82b\nw6hRJXdajii7d8OMGYcTxKoE5m/+AAAQD0lEQVRVrvy00w4nh3POcYmkoLwhI1asgAcecM1m7RSR\nLywhGFPBMjPdSBO5uYfLYmJgwoQqlhTyqLqWTXnJYeZMd3qqdm3XhyDv9NJpp7k3YdAgaNjQjR/U\nrZvf0VdrlhCMqWCJiUd3UAbXqXj16sqOxgd79sDnn7sE8eGHh1s7xca6Htw9esDEiXDCCf7GaYJO\nCNbw15gyWru2dOVVzrHHHj5tBK5J7PTp8Nln0Lkz/PWv5eutbSqdJQRjyqh588KPEJo3r/xYwsLJ\nJ8NNN7nFRCRL38aU0ahR7ppBoJgYV25MJLKEYEwZZWS4a6cJCW5g0oSEKnxB2VQLdsrImHLIyLAE\nYKoOO0IwxhgDWEIwxhjjCSohiEi6iCwXkRUictQ0lyKSICKfishCEZkpIvEB6/qJyA/e0i+gvKOI\nfOftc6w3laYxxhiflJgQRCQaGAf0BJKBviJSYEhFngBeUdW2wEjcdJqISENgBHAm0AkYISINvG3G\nA38CWnpL9Z7jzhhjfBbMEUInYIWqrlTVfcAkoHeBOsnAZ979GQHrLwA+VtWfVXUb8DGQLiJxwPGq\n+o26rtKvAJeW87UYY4wph2ASQlNgXcDjbK8s0AKgj3f/MqCuiDQqZtum3v3i9mmMMaYSheqi8l1A\nNxGZB3QD1gMHQ7FjERkoIlkikpWTkxOKXRoTVjIz3bhIUVHuNjPT74hMdRVMQlgPBE6dFO+V5VPV\nDaraR1VTgaFe2fZitl3v3S9ynwH7nqCqaaqaFhsbG0S4xkSOvBFT16xxg4muWeMeW1IwfggmIcwG\nWopICxGpBVwNTA2sICKNRSRvX/cAL3j3pwM9RKSBdzG5BzBdVTcCO0Wks9e66FrgnRC8HmMiytCh\nRw6fDe7x0KH+xGOqtxITgqoeAG7FfbkvBSar6mIRGSkil3jVzgGWi8j3wInAKG/bn4EHcUllNjDS\nKwO4GfgnsAL4EZgWqhdlTKSo9iOmmrBi8yEY46NqP6eCqRTBzodgPZWN8ZGNmGrCiSUEY3xkI6aa\ncGKjnRrjMxsx1YQLO0IwxhgDWEIwxhjjsYRgjDEGsIRgjDHGYwnBGGMMYAnBGGOMxxKCMcYYwBKC\nMcYYjyUEY4wxgCUEY4wxHksIxhhjAEsIxhhjPJYQjDHGAJYQjDHGeIJKCCKSLiLLRWSFiAwpZH1z\nEZkhIvNEZKGI9PLKM0RkfsBySETae+tmevvMW3dCaF+aMdVHZqabfS0qyt1mZvodkYlEJc6HICLR\nwDjgfCAbmC0iU1V1SUC1+3BzLY8XkWTgAyBRVTOBTG8/bYC3VXV+wHYZqmpzYhpTDpmZMHAg5Oa6\nx2vWuMdg8yyY0gnmCKETsEJVV6rqPmAS0LtAHQWO9+7XAzYUsp++3rbGmBAaOvRwMsiTm+vKjSmN\nYBJCU2BdwONsryzQ/cA1IpKNOzq4rZD9XAW8XqDsRe900TARkeBCNsYEWru2dOXGFCVUF5X7Ai+p\najzQC5goIvn7FpEzgVxVXRSwTYaqtgHO9pY/FrZjERkoIlkikpWTkxOicI2pOpo3L125MUUJJiGs\nB5oFPI73ygINACYDqOrXQG2gccD6qylwdKCq673bXcBruFNTR1HVCaqapqppsbGxQYRrTPUyahTE\nxBxZFhPjyo0pjWASwmygpYi0EJFauC/3qQXqrAXOAxCRJFxCyPEeRwG/J+D6gYjUEJHG3v2awEXA\nIowxpZaRARMmQEICiLjbCRNKf0HZWiqZElsZqeoBEbkVmA5EAy+o6mIRGQlkqepU4E7geRH5K+4C\nc39VVW8XXYF1qroyYLfHANO9ZBANfAI8H7JXZUw1k5FRvhZF1lLJAMjh7+3wl5aWpllZ1krVmFBL\nTHRJoKCEBFi9urKjMaEmInNUNa2ketZT2RhjLZUMYAnBGIO1VDKOJQRjjLVUMoAlBGMMoWupZCJb\nia2MjDHVQ3lbKpnIZ0cIxhhjAEsIxhhjPJYQjDHGAJYQjDHGeCwhGGOMASwhGGOM8VhCMMYYA1hC\nMMYY47GEYIwxBrCEYIwJIZtkJ7LZ0BXGmJCwSXYinx0hGGNCYujQw8kgT26uKzeRIaiEICLpIrJc\nRFaIyJBC1jcXkRkiMk9EFopIL688UUT2iMh8b3k2YJuOIvKdt8+xIiKhe1nGmMpmk+xEvhITgohE\nA+OAnkAy0FdEkgtUuw+YrKqpwNXAPwLW/aiq7b3lxoDy8cCfgJbekl72l2GM8ZtNshP5gjlC6ASs\nUNWVqroPmAT0LlBHgeO9+/WADcXtUETigONV9Rt1kzq/AlxaqsiNMWHFJtmJfMEkhKbAuoDH2V5Z\noPuBa0QkG/gAuC1gXQvvVNLnInJ2wD6zS9inMSaC2CQ7kS9UrYz6Ai+p6mgROQuYKCKtgY1Ac1Xd\nKiIdgbdFJKU0OxaRgcBAgOZ27GlMWLNJdiJbMEcI64FmAY/jvbJAA4DJAKr6NVAbaKyqv6rqVq98\nDvAjcJq3fXwJ+8TbboKqpqlqWmxsbBDhGmOMKYtgEsJsoKWItBCRWriLxlML1FkLnAcgIkm4hJAj\nIrHeRWlE5GTcxeOVqroR2Ckinb3WRdcC74TkFRljIpp1bvNPiaeMVPWAiNwKTAeigRdUdbGIjASy\nVHUqcCfwvIj8FXeBub+qqoh0BUaKyH7gEHCjqv7s7fpm4CXgWGCatxhjqjHr3OYvcY18IkNaWppm\nZWX5HYYxpoIkJrokUFBCAqxeXdnRVB0iMkdV00qqZz2VjTFhwzq3+csSgjEmbFjnNn9ZQjDGhA3r\n3OYvSwjGmLBhndv8ZcNfG2PCinVu848dIRhjjAEsIRhjjPFYQjDGGANYQjDGGOOxhGCMMQawhGCM\nMcZjCcEYU+XYiKllY/0QjDFVio2YWnZ2hGCMqVKGDj2cDPLk5rpyUzxLCMaYKsVGTC07SwjGmCol\nVCOmVsfrEJYQjDFVSihGTM27DrFmDagevg5R1ZNCUAlBRNJFZLmIrBCRIYWsby4iM0RknogsFJFe\nXvn5IjJHRL7zbn8XsM1Mb5/zveWE0L0sY0x1FYoRU6vrdYgSp9AUkWjge+B8IBuYDfRV1SUBdSYA\n81R1vIgkAx+oaqKIpAKbVHWDiLQGpqtqU2+bmcBdqhr0nJg2haYxpjJERbkjg4JE4NChyo+nvEI5\nhWYnYIWqrlTVfcAkoHeBOgoc792vB2wAUNV5qrrBK18MHCsixwTzAowxxi/Vdea2YBJCU2BdwONs\nryzQ/cA1IpINfADcVsh+LgfmquqvAWUveqeLhomIBB+2McZUnOo6c1uoLir3BV5S1XigFzBRRPL3\nLSIpwGPAnwO2yVDVNsDZ3vLHwnYsIgNFJEtEsnJyckIUrjHGFK26ztwWTEJYDzQLeBzvlQUaAEwG\nUNWvgdpAYwARiQfeAq5V1R/zNlDV9d7tLuA13Kmpo6jqBFVNU9W02NjYYF6TMcaUW0YGrF7trhms\nXl31kwEElxBmAy1FpIWI1AKuBqYWqLMWOA9ARJJwCSFHROoD7wNDVPWrvMoiUkNE8hJGTeAiYFF5\nX4wxxoSLSOzHUOJYRqp6QERuBaYD0cALqrpYREYCWao6FbgTeF5E/oq7wNxfVdXb7lRguIgM93bZ\nA/gFmO4lg2jgE+D5UL84Y4zxQ6SOp1Ris9NwYs1OjTGRIDHRJYGCEhLc6afKFspmp8YYY0ohUsdT\nsoRgjDEhFqn9GCwhGGNMiEVqPwZLCMYYE2KR2o/BZkwzxpgKkJER/gmgIDtCMMaYMFXZfRnsCMEY\nY8KQH30Z7AjBGGPCkB9zMlhCMMaYMORHXwZLCMYYE4b86MtgCcEYY8KQH30ZLCEYY0wY8qMvg7Uy\nMsaYMFXZfRnsCMEYYwxgCcEYY4zHEoIxxhjAEoIxxhiPJQRjjDFAhE2hKSI5QCET04WNxsAWv4MI\nUqTEanGGVqTECZETayTEmaCqsSVViqiEEO5EJCuYeUvDQaTEanGGVqTECZETa6TEGQw7ZWSMMQaw\nhGCMMcZjCSG0JvgdQClESqwWZ2hFSpwQObFGSpwlsmsIxhhjADtCMMYY47GEUEoi0kxEZojIEhFZ\nLCKDCqlzjojsEJH53jLcj1i9WFaLyHdeHFmFrBcRGSsiK0RkoYh08CHG0wPeq/kislNE/lKgji/v\nqYi8ICKbRWRRQFlDEflYRH7wbhsUsW0/r84PItLPhzgfF5Fl3t/1LRGpX8S2xX5GKinW+0VkfcDf\nt1cR26aLyHLv8zrEhzj/HRDjahGZX8S2lfqehoyq2lKKBYgDOnj36wLfA8kF6pwDvOd3rF4sq4HG\nxazvBUwDBOgMfOtzvNHAT7h2076/p0BXoAOwKKDs/4Ah3v0hwGOFbNcQWOndNvDuN6jkOHsANbz7\njxUWZzCfkUqK9X7griA+Gz8CJwO1gAUF//cqOs4C60cDw8PhPQ3VYkcIpaSqG1V1rnd/F7AUaOpv\nVOXSG3hFnW+A+iIS52M85wE/qmpYdEBU1S+AnwsU9wZe9u6/DFxayKYXAB+r6s+qug34GEivzDhV\n9SNVPeA9/AaIr6jnL40i3tNgdAJWqOpKVd0HTML9LSpEcXGKiAC/B16vqOf3gyWEchCRRCAV+LaQ\n1WeJyAIRmSYiKZUa2JEU+EhE5ojIwELWNwXWBTzOxt8EdzVF/5OFy3t6oqpu9O7/BJxYSJ1we1+v\nxx0JFqakz0hludU7vfVCEafhwuk9PRvYpKo/FLE+XN7TUrGEUEYichzwBvAXVd1ZYPVc3CmPdsDf\ngbcrO74Av1XVDkBP4BYR6epjLMUSkVrAJcB/ClkdTu9pPnXnB8K6qZ6IDAUOAJlFVAmHz8h44BSg\nPbARdzomnPWl+KODcHhPS80SQhmISE1cMshU1TcLrlfVnaq627v/AVBTRBpXcph5saz3bjcDb+EO\nuwOtB5oFPI73yvzQE5irqpsKrgin9xTYlHdazbvdXEidsHhfRaQ/cBGQ4SWvowTxGalwqrpJVQ+q\n6iHg+SJiCJf3tAbQB/h3UXXC4T0tC0sIpeSdO/wXsFRVnyyiThOvHiLSCfc+b628KPPjqCMidfPu\n4y4yLipQbSpwrdfaqDOwI+B0SGUr8ldXuLynnqlAXquhfsA7hdSZDvQQkQbe6Y8eXlmlEZF04G/A\nJaqaW0SdYD4jFa7AdavLiohhNtBSRFp4R5NX4/4Wla07sExVswtbGS7vaZn4fVU70hbgt7hTBAuB\n+d7SC7gRuNGrcyuwGNcK4hvgNz7FerIXwwIvnqFeeWCsAozDtd74DkjzKdY6uC/4egFlvr+nuAS1\nEdiPO2c9AGgEfAr8AHwCNPTqpgH/DNj2emCFt1znQ5wrcOfc8z6nz3p1TwI+KO4z4kOsE73P30Lc\nl3xcwVi9x71wLft+rOhYC4vTK38p73MZUNfX9zRUi/VUNsYYA9gpI2OMMR5LCMYYYwBLCMYYYzyW\nEIwxxgCWEIwxxngsIRhjjAEsIRhjjPFYQjDGGAPA/wP+VwdCaAclTAAAAABJRU5ErkJggg==\n",
411 |       "text/plain": [
412 |        "<Figure size 432x288 with 1 Axes>"
413 |       ]
414 |      },
415 |      "metadata": {},
416 |      "output_type": "display_data"
417 |     }
418 |    ],
419 |    "source": [
420 |     "import matplotlib.pyplot as plt\n",
421 |     "%matplotlib inline\n",
422 |     "\n",
423 |     "plt.plot(range(1,len(train_losses)+1),train_losses,'bo',label = 'training loss')\n",
424 |     "plt.plot(range(1,len(val_losses)+1),val_losses,'r',label = 'validation loss')\n",
425 |     "plt.legend()"
426 |    ]
427 |   },
428 |   {
429 |    "cell_type": "code",
430 |    "execution_count": 10,
431 |    "metadata": {
432 |     "ExecuteTime": {
433 |      "end_time": "2019-04-09T00:34:59.840977Z",
434 |      "start_time": "2019-04-09T00:34:59.638075Z"
435 |     }
436 |    },
437 |    "outputs": [
438 |     {
439 |      "data": {
440 |       "text/plain": [
441 |        "<matplotlib.legend.Legend at 0x149e91208>"
442 |       ]
443 |      },
444 |      "execution_count": 10,
445 |      "metadata": {},
446 |      "output_type": "execute_result"
447 |     },
448 |     {
449 |      "data": {
450 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl4VdW5x/Hvy2QMIpOoKEKwVlSQ\nMERFRcQiXovzgNbGiniVxwn1tmpxam2tncDWoeo1eh0osUFRqrZOjMVaHAKCWEBRCRJkCKAMBWTI\ne/9YJzFAQk6SM+f3eZ7znLP32XudNzvJm5W112DujoiIpL8myQ5ARERiQwldRCRDKKGLiGQIJXQR\nkQyhhC4ikiGU0EVEMoQSuohIhlBCFxHJEEroIiIZolkiP2y//fbznJycRH6kiEjamzVr1mp371Db\ncQlN6Dk5ORQXFyfyI0VE0p6ZLYnmODW5iIhkCCV0EZEMoYQuIpIhEtqGXp1t27ZRWlrKli1bkh2K\n1CIrK4tOnTrRvHnzZIciItVIekIvLS2lVatW5OTkYGbJDkdq4O6sWbOG0tJSunbtmuxwRKQaSW9y\n2bJlC+3bt1cyT3FmRvv27fWflDQqhYWQkwNNmoTnwsLklBGtpNfQASXzNKHvkzQmhYUwYgRs2hS2\nlywJ2wD5+Ykroy6SXkMXEUlFd9zxbSKusGlT2J/IMuqi0Sf0r7/+mkceeaRe5w4ZMoSvv/46xhGJ\nSCr44ou67Y9XGXWRdgk91u1Re0ro27dv3+O5r776Km3atGlYAHHg7pSXlyc7DJG01rlz3fbHq4y6\nSKuEXtEetWQJuH/bHtWQpD5q1Cg+++wzevXqxS233ML06dM56aSTOPvssznqqKMAOPfcc+nbty/d\nu3enoKCg8tycnBxWr15NSUkJRx55JFdddRXdu3fntNNOY/Pmzbt91iuvvMJxxx1H7969OfXUU1m5\nciUAGzduZPjw4Rx99NH07NmTF154AYDXX3+dPn36kJuby6BBgwC4++67GTNmTGWZPXr0oKSkhJKS\nErp168Zll11Gjx49WLp0Kddccw15eXl0796dn//855XnvP/++5xwwgnk5uZy7LHHsmHDBgYMGMCc\nOXMqj+nfvz9z586t/4UVSXP33gvZ2Tvvy84O+xNZRp24e8Ieffv29V3Nnz9/t3016dLFPaTynR9d\nukRdxG4WL17s3bt3r9yeNm2aZ2dn++eff165b82aNe7uvmnTJu/evbuvXr06Ek8XLysr88WLF3vT\npk39gw8+cHf3oUOH+p///OfdPmvt2rVeXl7u7u6PP/64//jHP3Z391tvvdVvvPHGnY5btWqVd+rU\nqTKOihh+/vOf++jRoyuP7d69uy9evNgXL17sZuYzZ87cLe7t27f7ySef7HPnzvVvvvnGu3bt6u+9\n9567u69bt863bdvmTz/9dGUMH3/8sVf3vXKv2/dLJN2NGxfyi1l4HjcuOWUAxR5Fjk2JXi7RSlR7\n1LHHHrtTX+sHH3yQiRMnArB06VIWLVpE+/btdzqna9eu9OrVC4C+fftSUlKyW7mlpaVcfPHFLF++\nnK1bt1Z+xuTJkykqKqo8rm3btrzyyisMGDCg8ph27drVGneXLl3o169f5fZzzz1HQUEB27dvZ/ny\n5cyfPx8zo2PHjhxzzDEA7LvvvgAMHTqUe+65h9GjR/Pkk09y+eWX1/p5IpkuP7/hvVFiUUa00qrJ\nJVHtUS1btqx8PX36dCZPnszMmTOZO3cuvXv3rrYv9l577VX5umnTptW2v48cOZLrr7+eefPm8dhj\nj9WrT3ezZs12ah+vWkbVuBcvXsyYMWOYMmUKH374IWecccYePy87O5vBgwfz0ksv8dxzz5GfqJ9A\nEYmZtEro8WiPatWqFRs2bKjx/XXr1tG2bVuys7NZuHAh77zzTr0/a926dRx88MEAPPPMM5X7Bw8e\nzMMPP1y5/dVXX9GvXz9mzJjB4sWLAVi7di0Q2u1nz54NwOzZsyvf39X69etp2bIlrVu3ZuXKlbz2\n2msAdOvWjeXLl/P+++8DsGHDhso/PldeeSU33HADxxxzDG3btq331ykiyZFWCT0/HwoKoEsXMAvP\nBQUN+3emffv2nHjiifTo0YNbbrllt/dPP/10tm/fzpFHHsmoUaN2atKoq7vvvpuhQ4fSt29f9ttv\nv8r9d955J1999RU9evQgNzeXadOm0aFDBwoKCjj//PPJzc3l4osvBuCCCy5g7dq1dO/enT/96U8c\nfvjh1X5Wbm4uvXv35ogjjuCHP/whJ554IgAtWrRg/PjxjBw5ktzcXAYPHlxZc+/bty/77rsvw4cP\nr/fXKCLJY6G9PTHy8vJ81wUuFixYwJFHHpmwGKRmX375JQMHDmThwoU0aVL933p9v0QSz8xmuXte\nbcdFVUM3szZmNsHMFprZAjM7vsp7PzEzN7P99lSGpLaxY8dy3HHHce+999aYzEUktUX7m/sA8Lq7\nHwHkAgsAzOwQ4DQgTuOeJFEuu+wyli5dytChQ5MdikjaTYqVKmrttmhmrYEBwOUA7r4V2Bp5+4/A\nrcBLcYpPRBqZdJwUK1VEU0PvCpQBT5nZB2b2hJm1NLNzgGXuruGEIhIz6TgpVqqIJqE3A/oAj7p7\nb+A/wN3A7cDPajvZzEaYWbGZFZeVlTUkVhFpBNJxUqxUEU1CLwVK3f3dyPYEQoLvCsw1sxKgEzDb\nzA7c9WR3L3D3PHfP69ChQ4zCFpFMlY6TYqWKWhO6u68AlppZt8iuQcBsd9/f3XPcPYeQ9PtEjs14\n++yzT7JDEMlYaTkpVoqItpfLSKDQzD4EegG/jl9IUpvapvUVSWexGEAYj0GI6SCqhO7ucyLNJj3d\n/Vx3/2qX93PcfXV8QoyvUaNG7TTsvmJ62o0bNzJo0CD69OnD0UcfzUsv1d6Rp6ZpdqubBremKXOr\n1v4nTJhQOUnW5ZdfztVXX81xxx3Hrbfeynvvvcfxxx9P7969OeGEE/j4448B2LFjBzfffDM9evSg\nZ8+ePPTQQ0ydOpVzzz23stxJkyZx3nnn1f+iicRZfj6UlEB5eXiuTyKORRnpJrVmW7zpJqgyJ3dM\n9OoF999f49sXX3wxN910E9dddx0QZih84403yMrKYuLEiey7776sXr2afv36cfbZZ+9xXc0nn3yS\ndu3asXnzZo455hguuOACysvLueqqq5gxYwZdu3atnJPlnnvuoXXr1sybNw8I87fUprS0lH/96180\nbdqU9evX89Zbb9GsWTMmT57M7bffzgsvvEBBQQElJSXMmTOHZs2asXbtWtq2bcu1115LWVkZHTp0\n4KmnnuKKK66oy1UUkTSQWgk9CXr37s2qVav48ssvKSsro23bthxyyCFs27aN22+/nRkzZtCkSROW\nLVvGypUrOfDA3e77Vqpumt2ysrJqp8Gtbsrc2gwdOpSmTZsCYaKvYcOGsWjRIsyMbdu2VZZ79dVX\n06xZs50+70c/+hHjxo1j+PDhzJw5k7Fjx9b1UolIikuthL6HmnQ8DR06lAkTJrBixYrKSbAKCwsp\nKytj1qxZNG/enJycnD1OP1t1mt3s7GwGDhxYr+lxq/4HsOv5VafHveuuuzjllFOYOHEiJSUlDBw4\ncI/lDh8+nLPOOousrCyGDh1amfBFJHNo0g5Cs0tRURETJkyoHPq+bt069t9/f5o3b860adNYsmTJ\nHsuoaZrdmqbBrW7KXIADDjiABQsWUF5eXlnbr+nzKqbiffrppyv3Dx48mMcee6zyxmnF5x100EEc\ndNBB/OpXv9JsiiIZSgkd6N69Oxs2bODggw+mY8eOAOTn51NcXMzRRx/N2LFjOeKII/ZYRk3T7NY0\nDW51U+YC/Pa3v+XMM8/khBNOqIylOrfeeiu33XYbvXv33qnXy5VXXknnzp3p2bMnubm5PPvss5Xv\n5efnc8ghh2i2RKmR5lBJc9GsUxerR0PXFJWGue666/yJJ55oUBn6fmWucePcs7N3Xq83O7tua2DG\nogzZHVGuKaoaeiPRt29fPvzwQy699NJkhyIpSnOopD/dGWskZs2alewQJMVpDpX0lxI1dE/gqklS\nf/o+ZTbNoZL+kp7Qs7KyWLNmjZJFinN31qxZQ1ZWVrJDkTjRHCrpL+lNLp06daK0tBRNrZv6srKy\n6NSpU7LDkDipGBp/xx2hiaRz55CI6zqHSkPLkPpL+iLRIiKyZzFdJFpERFKfErqISIZQQhcRyRBK\n6CIiGSKqhG5mbcxsgpktNLMFZna8mY2ObH9oZhPNrE28gxXJVJpDRWIh2hr6A8Dr7n4EkAssACYB\nPdy9J/AJcFt8QhTJbIWFMGIELFkSZj9ZsiRs1yUhx6IMSX+1dls0s9bAHOBQr+FgMzsPuNDd99jb\nVN0WRXaXkxMS8K66dAlLpyWqDEldsey22BUoA54ysw/M7Akza7nLMVcAr9UQyAgzKzazYg0eEtmd\n5lCRWIkmoTcD+gCPuntv4D/AqIo3zewOYDtQ7T937l7gYYHpvA4dOsQgZJHMojlUJFaiSeilQKm7\nvxvZnkBI8JjZ5cCZQH5NzTEismeaQ0VipdaE7u4rgKVm1i2yaxAw38xOB24Fznb3TTUWICJ7lJ8P\nBQWhvdssPBcU1H0OlYaWIekvqrlczKwX8ATQAvgcGA68D+wFrIkc9o67X72ncnRTVESk7qK9KRrV\nbIvuPgfYtbDD6hOYiIjEh0aKiohkCCV0EZEMoYQuIpIhlNBFRDKEErpIA2lSLEkVSV9TVCSdVUyK\ntSkyEqNiUixQH3BJPNXQRRrgjju+TeYVNm0K+0USTQldpAE0KZakEiV0kQbQpFiSSpTQRRpAk2JJ\nKlFCF2kATYolqUS9XEQaKD9fCVxSg2roIiIZQgldRCRDKKGLiGSIqBK6mbUxswlmttDMFpjZ8WbW\nzswmmdmiyHPbeAcrIiI1i7aG/gDwursfAeQCCwgLRU9x9+8CU6iycLSIiCRerQndzFoDA4D/A3D3\nre7+NXAO8EzksGeAc+MVpIiI1C6abotdgTLgKTPLBWYBNwIHuPvyyDErgAPiE6JII+AOjz4KH30U\nRiW1TVIL5jffwC9/Ce++m5zPz2S/+Q0cc0xcPyKahN4M6AOMdPd3zewBdmlecXc3s2pXmzazEcAI\ngM4aDy2yu9Wr4Yor4JVXwvbf/w5/+QuccEJi4/jkE/jBD+CDD+DYY6F588R+fqbbsSPuHxFNQi8F\nSt294k/2BEJCX2lmHd19uZl1BFZVd7K7FwAFAHl5edUmfZFGa/r0MCpp9Wp44AE47jj44Q9hwAD4\nxS9g1Cho2jS+MbjD2LFw3XWQlQUvvwxnnRXfz5S4qLUN3d1XAEvNrFtk1yBgPvAyMCyybxjwUlwi\nFMlE27fDz34G3/setGwJ77wDN9wQEvrs2XDRRXDnnTB4MHz5Zfzi2LABfvQjuPxyyMuDuXOVzNNY\ntL1cRgKFZvYh0Av4NfBbYLCZLQJOjWyLSG2++AJOOQXuuQcuuywk8N69v32/deuwcsaTT4a27Nzc\n0AwTa8XF4XP/8pfQbj5lChx8cOw/RxImqoTu7nPcPc/de7r7ue7+lbuvcfdB7v5ddz/V3dfGO1iR\ntDdxIvTqBXPmwLhx8PTTsM8+ux9nBsOHw6xZcNBBcOaZ8D//E25aNlR5Odx3X2ij37oV/vEPuOuu\n+DftSNxppKhIImzeHNqozz8fDj003HiMZkavI44ItfSRI+H++0MS/uST+sexahWccQbcfHN4njMH\n+vevf3mSUpTQReJt/vzQNv7II/CTn8C//gWHHRb9+VlZ8OCD8Ne/QkkJ9OkTbmLW1eTJoflm2jR4\n+GF48UVo167u5UjKUkIXiRd3eOKJcLNxxQp49VUYMwZatKhfeeecE25a9u0Lw4aFm5kbNtR+3rZt\ncNttcNppoX/7e+/BtdeGZh3JKEroIvHw9dehT/dVV4Vmkrlz4fvfb3i5nTrB1KmhS+Ozz4ba+qxZ\nNR9fUhK6QP72t/Df/w3vvw89ezY8DklJSujSqBUWQk4ONGkSngsLY1DoO++E3iMvvBBGB775JnTs\nGIOCI5o2DV0ep02DLVvg+OPhD38INzurev75cAN2/nwoKoLHHw9dJCVjKaFLo1VYCCNGwJIloXVk\nyZKwXe+kXl4easIVNxn/+c8wMKhJnH7NBgwINzWHDAlt82edFW56btoUvpCLLgo3VefMgYsvjk8M\nklLMPXGDN/Py8ry4uDhhnyeyJzk5IYnvqkuX0FJRJytWhDbtyZNh6NCwsGibNjGIMgru395wbds2\nPBYuhJ/+NPQv1xD+tGdms9w9r7bjVEOXRuuLL+q2v0avvx7apd9+OyTy8eMTl8wh3Ny87rrQvbFN\nG1i7Ft54IzT3KJk3KlokWhqtzp2rr6FHPYfc1q1w++1hkE6PHmFelqOOimWIdZObC/PmhWkFsrKS\nF4ckjWro0mjdey9kZ++8Lzs77K/Vp5/CiSeGZH7NNaErYDKTeYVmzZTMGzEldGm08vNDC0mXLqHV\nokuXsF3rAM7CwtCL5dNPQ0+WRx6BvfdOSMwie6ImF2nU8vOjG4EPwMaNcP318MwzoXb+7LN1aJ8R\niT/V0EWi8cEHYYTm2LFhIqvp05XMJeUooYvsiXtYeKJfv1BDnzo1dAVspn9uJfXop1KkJqtXhyls\n//a3MH3tU0/BfvslOyqRGqmGLlKd6dNDN8A33ww19JdfVjKXlBdVQjezEjObZ2ZzzKw4sq+Xmb1T\nsc/Mjo1vqCIJUHVpuH32+XZpOM1MKGmgLk0up7j76irbvwd+4e6vmdmQyPbAWAYnklBffBEWaH77\n7bDG5kMPVb+akEiKakgbugP7Rl63BuK4kq1InL34YphedseOsDRc1H0ZRVJHtAndgTfNzIHH3L0A\nuAl4w8zGEJpuTohTjCLxs3lzmNTq0UfDQhRFRfCd7yQ7KpF6iTah93f3ZWa2PzDJzBYCFwL/4+4v\nmNlFwP8Bp+56opmNAEYAdFa/XUkl8+eHRSjmzQtJ/de/rv9qQiIpoM7T55rZ3cBG4C6gjbu7mRmw\nzt333dO5mj5XUkLF0nA33hjayMeOhdNPT3ZUIjWK2fS5ZtbSzFpVvAZOAz4itJmfHDnse8Ci+ocr\nkiBffx0WexgxIgzfnztXyVwyRjRNLgcAE0MlnGbAs+7+upltBB4ws2bAFiLNKiIpa+ZMuOQSWLYs\nrCx0yy3xW01IJAlqTeju/jmQW83+fwJ94xGUSEyVl8PvfhfmYDnkEHjrrTCUXyTDaOi/ZLbly8PS\ncFOmhDU2H3sssasJiSSQErpkrtdeg2HDwqRajz8e+plrxKdkMDUgSubZujV0QxwyBA48EIqL4cor\nlcwl46mGLpnl009D3/JZs+Daa2HMGK0mJI2GauiStgoLIScndFTJyYG3rxkXlob7/PMwlP/hh5XM\npVFRDV3SUmFh6Eq+aRO0ZCO/WHIdJ/7vWFYd3p/9JxVqNSFplFRDl7R0xx0hmR/EMmbTh0sZxy/4\nGcdvmaZkLo2WauiSlr74Ijzfxm/IoYRBTOEfDMSWJjcukWRSDV3SUufO0J7VXMGTjONS/hGZil+V\nc2nMlNAlLd17L9zY/BGy2cx9/ASA7OywX6SxUkKXtJR//mZu2ftPTN17CAusO126QEGB1qWQxk1t\n6JKexo4la30Z35t2C+UDkx2MSGpQDV3Sz44dcN99YYWhk0+u/XiRRkI1dEk/L78MixbB+PEazi9S\nhWrokn5Gjw5DQ88/P9mRiKQU1dAlvbz9dlio4sEHoZl+fEWqiqqGbmYlZjbPzOaYWXGV/SPNbKGZ\n/dvMfh+/MEUixoyBdu3giiuSHYlIyqlLFecUd19dsWFmpwDnALnu/o2Z7R/z6ESq+uQTeOmlMO6/\nZctkRyOSchrShn4N8Ft3/wbA3VfFJiSRGtx3H7RoAddfn+xIRFJStAndgTfNbJaZVSwGfThwkpm9\na2b/MLNjqjvRzEaYWbGZFZeVlcUiZmmMVq6EZ56Byy6DAw5IdjQiKSnaJpf+7r4s0qwyycwWRs5t\nB/QDjgGeM7ND3d2rnujuBUABQF5eniNSH3/607crEYlItaKqobv7ssjzKmAicCxQCrzowXtAObBf\nvAKVRuw//4FHHoGzz4Zu3ZIdjUjKqjWhm1lLM2tV8Ro4DfgI+CtwSmT/4UALYHVN5YjU21NPwdq1\ncMstyY5EJKVF0+RyADDRwoi8ZsCz7v66mbUAnjSzj4CtwLBdm1tEGmz7dvjDH+D44+HEE5MdjUhK\nqzWhu/vnQG41+7cCl8YjKJFKL74IixeH/uciskca+i+pyz0M8z/sMDjnnGRHI5LyNHZaUteMGVBc\nDI8+Ck2bJjsakZSnGrqkrtGjoUMHGDYs2ZGIpAUldElN8+fD3/8eRoXuvXeyoxFJC0rokprGjAmJ\n/Nprkx2JSNpQQpfU8+WXMG4cDB8O+2msmki0lNAl9Tz0UFhm7sc/TnYkImlFCV1Sy4YNoVfL+efD\nd76T7GhE0ooSuqSWJ56Adevg5puTHYlI2lFCl9SxbRv88Y9w0klw3HHJjkYk7WhgkaSO556DpUvh\n4YeTHYlIWlINXVJDxTD/I46AM85IdjQiaUk1dEkNU6bA3LmhDb2J6hki9aHfHEkNo0eHpeXy85Md\niUjaUkKX5Js7F958E264AbKykh2NSNqKKqGbWYmZzTOzOWZWvMt7PzEzNzMN6ZP6GTMGWraEa65J\ndiQiaa0ubeinuPtOS8yZ2SGEJem+iGlU0ngsXQpFRXDdddC2bbKjEUlrDW1y+SNwK6Cl56R+Hngg\n9HC56aZkRyKS9qJN6A68aWazzGwEgJmdAyxz97lxi04y27p1UFAAQ4dCTk6yoxFJe9E2ufR392Vm\ntj8wycwWArcTmlv2KPIHYARA586d6x2oZKAJE8LcLaqdi8REVDV0d18WeV4FTAROBroCc82sBOgE\nzDazA6s5t8Dd89w9r0OHDjELXDJAUVGYgOvYY5MdiUhGqDWhm1lLM2tV8ZpQK3/f3fd39xx3zwFK\ngT7uviKu0UrmWLUKpk6Fiy8Gs2RHI5IRomlyOQCYaOGXrhnwrLu/HteoJPNNmADl5fCDHyQ7EpGM\nUWtCd/fPgdxajsmJVUDSSIwfD0cdBT16JDsSkYyhkaKSeKWl8NZbam4RiTEldEm8558Pfc8vvjjZ\nkYhkFCV0Sbzx46FXL+jWLdmRiGQUJXRJrMWL4d13dTNUJA6U0CWxnnsuPKu5RSTmlNAlsYqKwnqh\nGuovEnNK6JI4H38Mc+aouUUkTpTQJXHGjw/dFIcOTXYkIhlJCV0Swz00t5x0Ehx8cLKjEclISuiS\nGB99BAsWqLlFJI6U0CUxioqgSRO44IJkRyKSsZTQJf7cQ/v5oEGw//7JjkYkYymhS/zNmgWffaa+\n5yJxpoQu8VdUBM2bw3nnJTsSkYymhC7xVV4eRof+139Bu3bJjkYkoymhS3zNnAlLl6q5RSQBolok\nOrJu6AZgB7Dd3fPMbDRwFrAV+AwY7u5fxytQSVPjx0NWFpx9drIjEcl4damhn+Luvdw9L7I9Cejh\n7j2BT4DbYh6dpLcdO0Jzy5AhsO++yY5GJOPVu8nF3d909+2RzXeATrEJSTLGP/4BK1dqMJFIgkSb\n0B1408xmmdmIat6/AngtdmFJRhg/Hlq2hDPOSHYkIo1CVG3oQH93X2Zm+wOTzGyhu88AMLM7gO1A\nYXUnRv4AjADo3LlzDEKWtLBtG0yYENrOs7OTHY1IoxBVDd3dl0WeVwETgWMBzOxy4Ewg3929hnML\n3D3P3fM6dOgQk6AlDUyZAmvXqrlFJIFqTehm1tLMWlW8Bk4DPjKz04FbgbPdfVN8w5S0U1QErVuH\n/ucikhDRNLkcAEw0s4rjn3X3183sU2AvQhMMwDvufnXcIpX0sWULTJwI558Pe+2V7GhEGo1aE7q7\nfw7kVrP/sLhEJOnvjTdg/Xo1t4gkmEaKSuwVFUH79vC97yU7EpFGRQldYus//4GXX4YLLwwTcolI\nwiihS2z9/e+waZPmbhFJAiV0ia3x4+HAA2HAgGRHItLoKKFL7KxfH2roF10ETZvWenhhIeTkhJXp\ncnLCtojUX7QjRUVq99JL8M03UTW3FBbCiBGhdQZgyZKwDZCfH8cYRTKYaugSO+PHQ+fO0K9frYfe\ncce3ybzCpk1hv4jUjxK6xMbataH/+UUXhTaUWnzxRd32i0jtlNAlNiZOhO3box5MVNM8bZq/TaT+\nlNAlNoqK4LDDoE+fqA6/997dJ2HMzg77RaR+lNCl4VauhKlTw83QMK9PrfLzoaAAunQJp3TpErZ1\nQ1Sk/tTLRRruhRegvLzOc7fk5yuBi8SSaujScEVFcNRR0KNHsiMRadSU0KVhSkvhn//UzIoiKUAJ\nXRrm+efBXXO3iKQAJXRpmKIi6N0bDj882ZGINHpRJXQzKzGzeWY2x8yKI/vamdkkM1sUeW4b31Al\n5SxeDO+9p+YWkRRRlxr6Ke7ey93zItujgCnu/l1gSmQ7JcViEihNJLWzwkL4XZ/xAPR/8KJGfz1E\nUkFDui2eAwyMvH4GmA78tIHxxFzFJFC+aROt+YYlS9rWeRKomE4ktWZNGEGz9951PDHGSkth9ep6\nnfrqq/Cne+CxLc8yk368vSyHDzSxlkjyuXutD2AxMBuYBYyI7Pu6yvtWdbumR9++fT3RunRxP5G3\nfAmH+Fe09gt5ziHsr0sZ4c7fzo+6lOHu7oWF7q1auefkuM+cWceTY2T7dvd77nFv0qT6L6qOj5E8\nUP/rISJRAYo9ilwdbQ29v7svM7P9gUlmtnCXPwpuZl7diWY2AhgB0DnRE3Xs2MGPlvyGu/k5JeTw\nCYfzPBfxGCP48ZI/Atm1FgExmEhq40YYORKefjrMRLh8OZx0EvzqV3DLLVFNZhUTy5bBpZfC9Olw\nySVhIq16OO88cGArLZjMqZX7NbGWSJJFk/WrPoC7gZuBj4GOkX0dgY9rOzehNfTSUveBA93Bx/FD\nb8U6b8ZW/w0/dQdf2Ly7+7x5URXVoBr6Bx+4d+vmbuZ+113u27a5f/WV+4UXhkJOPdV9+fKGfKXR\neeUV9/bt3bOz3Z96yr28vN7AsJjnAAAI3klEQVRFxew/FhGJClHW0KNJ4C2BVlVe/ws4HRgNjIrs\nHwX8vrayEpbQqySvf414yrP3Lt8p8Zy515u+qfUB7llZ7o8+WmtyGzcu5MGqZWRnh/01Ki93f+AB\n9xYt3A86yH3q1N3ff+yxEEOHDu6vvdbwr7s6W7a433hjCLpXL/eFCxtcZL2uh4jUWywT+qHA3Mjj\n38Adkf3tCb1bFgGTgXa1lRX3hL5li/sNN+yWvMaNC7VHs/A8bpy7r1jhftpp4dgLLnBfu3aPRVdb\nRk3KytzPOivy1+PMsF2Tjz5y79EjHHvzze7ffFO3r3lPPv7YvXfvUPYNN7hv3hyzout0PUSkQWKW\n0GP5iGtCX7gwJPGK5LVlS+3n7Njh/vvfuzdr5t65s/s//9nwOKZNcz/44FAzv//+6Jo2Nm1yv/rq\nEPsxx7h/+mnDYigvd3/6afeWLcN/Ki+/3LDyRCSpGk9Cj0Xyevdd90MPdW/aNPQA2b697mVs2xba\nyM3cv/td99mz617GhAnubdqEnjCFhXU/3919/Xr3/PzwrT355HAvQUTSWuNI6OvWxS55rVvnfskl\noayBA+tW1hdfuPfvH84dNsx9w4b6x1FS4n7iiaGsyy+vW1nvv+/+ne+ELom/+EX9/jCJSMrJ/IRe\nNXn98pexSV7l5e5PPhnu8LVvH26u1ubFF93btnXfZ5/YNSRv2+Z+552htt+tW+gpsyc7driPGePe\nvLn7IYe4z5gRmzhEJCVkbkLfNXm99VbDy9zVggXuubnh8tx4Y/Xt8Zs2uV97bTimb1/3RYtiH8fU\nqaGHTIsWocdMde3xK1e6n356iOPcc93XrIl9HCKSVBmT0Kv2pujTaaUv6xlJXuedF9/ktXmz+8iR\n4bN69w49RirMn+9+9NHhvZ/8JOqeKfXqGVJWFnrKgPtZZ/nzj5ZVlnHJ/pN9U+sD3ffay/3hh6Pu\nW64eKiLpJSMSetX+zoOY5F9yoG9mL3/38kcaNDCmTl56yb1du3DT9emn3R9/3H3vvUPf8VdfjbqY\nBvXdLi93v/9+396shZfawT6ISX4vt/kOzOfbkf63X89NTBwikhQZkdArRiTezq98B+b/5kjvwYeJ\nH5G4dKn7gAHfZsBBg9y//LJORcRidOWQjrP9Y75beXIBV3o2G5MzL42IJEy0Cd3CsYmRl5fnxcXF\nUR/fpElINxcxnkFM4SbuZzPZmIU1iRNqxw64/35o2hRuuKHO869UfC27qsvX0qQJZPtG7uRXFJPH\nC1xYrzIaGoeIJJaZzfJvpy6v+bhUTug5OWGq2l116QIlJTELKyFi8bWkShkikljRJvSUXoLu3nvD\n1OFVZWeH/ekmFl9LqpQhIikqmnaZWD0a2ssl3XtkxOJrSZUyRCRxyIQ2dBERyZAmFxERiZ4SuohI\nhlBCFxHJEEroIiIZQgldRCRDJLSXi5mVAdUMa0kp+wGrkx1EFBRnbKVLnJA+sSrO2Oni7h1qOyih\nCT0dmFlxNN2Dkk1xxla6xAnpE6viTDw1uYiIZAgldBGRDKGEvruCZAcQJcUZW+kSJ6RPrIozwdSG\nLiKSIVRDFxHJEI0uoZvZIWY2zczmm9m/zezGao4ZaGbrzGxO5PGzZMQaiaXEzOZF4thtZjMLHjSz\nT83sQzPrk4QYu1W5VnPMbL2Z3bTLMUm5pmb2pJmtMrOPquxrZ2aTzGxR5LltDecOixyzyMyGJSnW\n0Wa2MPK9nWhmbWo4d48/JwmI824zW1bl+zukhnNPN7OPIz+vo5IQ5/gqMZaY2Zwazk3Y9YypaKZk\nzKQH0BHoE3ndCvgEOGqXYwYCf0t2rJFYSoD99vD+EOA1wIB+wLtJjrcpsILQbzbp1xQYAPQBPqqy\n7/fAqMjrUcDvqjmvHfB55Llt5HXbJMR6GtAs8vp31cUazc9JAuK8G7g5ip+Nz4BDgRbA3F1/9+Id\n5y7v3wf8LNnXM5aPRldDd/fl7j478noDsAA4OLlRNcg5wFgP3gHamFnHJMYzCPjM3VNiAJm7zwDW\n7rL7HOCZyOtngHOrOfW/gEnuvtbdvwImAafHLVCqj9Xd33T37ZHNd4BO8YwhGjVc02gcC3zq7p+7\n+1agiPC9iIs9xWlmBlwE/CVen58MjS6hV2VmOUBv4N1q3j7ezOaa2Wtm1j2hge3MgTfNbJaZjajm\n/YOBpVW2S0nuH6gfUPMvSapc0wPcfXnk9QrggGqOSbXrCnAF4b+x6tT2c5II10eahp6soRkrla7p\nScBKd19Uw/upcD3rrNEmdDPbB3gBuMnd1+/y9mxCk0Eu8BDw10THV0V/d+8DfB+4zswGJDGWPTKz\nFsDZwPPVvJ1K17SSh/+vU76rl5ndAWwHCms4JNk/J48C3wF6AcsJzRmp7BL2XDtP9vWsl0aZ0M2s\nOSGZF7r7i7u+7+7r3X1j5PWrQHMz2y/BYVbEsizyvAqYSPi3taplwCFVtjtF9iXD94HZ7r5y1zdS\n6ZoCKyuapSLPq6o5JmWuq5ldDpwJ5Ef+AO0mip+TuHL3le6+w93Lgcdr+PyUuKZm1gw4Hxhf0zHJ\nvp711egSeqTt7P+ABe7+hxqOOTByHGZ2LOE6rUlclJVxtDSzVhWvCTfIPtrlsJeByyK9XfoB66o0\nJyRajbWeVLmmES8DFb1WhgEvVXPMG8BpZtY20nxwWmRfQpnZ6cCtwNnuvqmGY6L5OYmrXe7bnFfD\n578PfNfMukb+m/sB4XuRaKcCC929tLo3U+F61luy78om+gH0J/yL/SEwJ/IYAlwNXB055nrg34S7\n8O8AJyQp1kMjMcyNxHNHZH/VWA14mNB7YB6Ql6RYWxISdOsq+5J+TQl/YJYD2whttv8NtAemAIuA\nyUC7yLF5wBNVzr0C+DTyGJ6kWD8ltDtX/Kz+b+TYg4BX9/RzkuA4/xz5+fuQkKQ77hpnZHsIoWfZ\nZ8mIM7L/6YqfyyrHJu16xvKhkaIiIhmi0TW5iIhkKiV0EZEMoYQuIpIhlNBFRDKEErqISIZQQhcR\nyRBK6CIiGUIJXUQkQ/w/BxiZeKzWLXYAAAAASUVORK5CYII=\n",
451 |       "text/plain": [
452 |        "<Figure size 432x288 with 1 Axes>"
453 |       ]
454 |      },
455 |      "metadata": {},
456 |      "output_type": "display_data"
457 |     }
458 |    ],
459 |    "source": [
460 |     "plt.plot(range(1,len(train_accuracy)+1),train_accuracy,'bo',label = 'train accuracy')\n",
461 |     "plt.plot(range(1,len(val_accuracy)+1),val_accuracy,'r',label = 'val accuracy')\n",
462 |     "plt.legend()"
463 |    ]
464 |   },
465 |   {
466 |    "cell_type": "code",
467 |    "execution_count": null,
468 |    "metadata": {},
469 |    "outputs": [],
470 |    "source": []
471 |   }
472 |  ],
473 |  "metadata": {
474 |   "kernelspec": {
475 |    "display_name": "Python [conda env:dl]",
476 |    "language": "python",
477 |    "name": "conda-env-dl-py"
478 |   },
479 |   "language_info": {
480 |    "codemirror_mode": {
481 |     "name": "ipython",
482 |     "version": 3
483 |    },
484 |    "file_extension": ".py",
485 |    "mimetype": "text/x-python",
486 |    "name": "python",
487 |    "nbconvert_exporter": "python",
488 |    "pygments_lexer": "ipython3",
489 |    "version": "3.6.7"
490 |   },
491 |   "toc": {
492 |    "base_numbering": 1,
493 |    "nav_menu": {},
494 |    "number_sections": true,
495 |    "sideBar": true,
496 |    "skip_h1_title": false,
497 |    "title_cell": "Table of Contents",
498 |    "title_sidebar": "Contents",
499 |    "toc_cell": false,
500 |    "toc_position": {},
501 |    "toc_section_display": true,
502 |    "toc_window_display": false
503 |   },
504 |   "varInspector": {
505 |    "cols": {
506 |     "lenName": 16,
507 |     "lenType": 16,
508 |     "lenVar": 40
509 |    },
510 |    "kernels_config": {
511 |     "python": {
512 |      "delete_cmd_postfix": "",
513 |      "delete_cmd_prefix": "del ",
514 |      "library": "var_list.py",
515 |      "varRefreshCmd": "print(var_dic_list())"
516 |     },
517 |     "r": {
518 |      "delete_cmd_postfix": ") ",
519 |      "delete_cmd_prefix": "rm(",
520 |      "library": "var_list.r",
521 |      "varRefreshCmd": "cat(var_dic_list()) "
522 |     }
523 |    },
524 |    "types_to_exclude": [
525 |     "module",
526 |     "function",
527 |     "builtin_function_or_method",
528 |     "instance",
529 |     "_Feature"
530 |    ],
531 |    "window_display": false
532 |   }
533 |  },
534 |  "nbformat": 4,
535 |  "nbformat_minor": 2
536 | }
537 | 


--------------------------------------------------------------------------------
/Notebook/data/section6/series1_Xtrain.csv:
--------------------------------------------------------------------------------
 1 | -2.450477613392171294e-02,-1.755566312546923691e-01,-7.838639448072381999e-02,2.175259428451395693e-01,-3.244331469657286970e-02,2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02
 2 | -1.755566312546923691e-01,-7.838639448072381999e-02,2.175259428451395693e-01,-3.244331469657286970e-02,2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01
 3 | -7.838639448072381999e-02,2.175259428451395693e-01,-3.244331469657286970e-02,2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01
 4 | 2.175259428451395693e-01,-3.244331469657286970e-02,2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02
 5 | -3.244331469657286970e-02,2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01
 6 | 2.846883176895338985e-01,-1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01
 7 | -1.820889845938324647e-01,9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01
 8 | 9.098424833831947334e-02,4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01
 9 | 4.654354052957652144e-01,9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02
10 | 9.888777767065648483e-02,-1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01
11 | -1.269294512989796730e-01,-1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02
12 | -1.208791575705959187e-01,-2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02
13 | -2.740670233924963339e-03,3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02
14 | 3.890662283697150781e-02,9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01
15 | 9.446635557782918913e-02,-2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01
16 | -2.373757951107593467e-01,-5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02
17 | -5.287728890671301168e-02,-1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02
18 | -1.237119568764060984e-01,-2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02
19 | -2.264681954506769512e-01,9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02
20 | 9.869829553160965052e-02,1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01
21 | 1.820782193763630374e-01,-1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01
22 | -1.686233766861378047e-01,5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01
23 | 5.431798219834728381e-02,1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02
24 | 1.890295703933203841e-01,1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02
25 | 1.286949950230161233e-01,1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01
26 | 1.556939009220553727e-01,-1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02
27 | -1.320145200312870526e-01,4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02
28 | 4.477180612332511134e-02,1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02
29 | 1.634939125073499588e-01,7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02
30 | 7.712144500935499969e-02,-7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02
31 | -7.400578863873377711e-02,2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02
32 | 2.266199820141617804e-02,-1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02
33 | -1.915772783543262248e-01,-1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02
34 | -1.856746267109581905e-01,2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01
35 | 2.210499907194347913e-02,-5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01
36 | -5.950034412144390022e-02,4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03
37 | 4.223250814338921216e-02,-7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02
38 | -7.726015650015510827e-02,-1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02
39 | -1.009016078948293260e-01,1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02
40 | 1.432909838879019016e-01,1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02
41 | 1.308926158980145971e-01,-8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01
42 | -8.947956278991478496e-02,9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01
43 | 9.205657966235250567e-02,1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02
44 | 1.421588936348644472e-01,8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02
45 | 8.966223224907717004e-02,9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02
46 | 9.675867426389285486e-02,1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02
47 | 1.145770484988862387e-02,-6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02
48 | -6.619252614385424949e-02,-4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02
49 | -4.880814523516521092e-02,-8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02
50 | -8.721007271931062454e-02,-3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02
51 | -3.460915048891051449e-02,9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01
52 | 9.025319433950558523e-02,-1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02
53 | -1.632052081599277837e-01,-1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02
54 | -1.181326122866943329e-01,-8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02
55 | -8.050129637396957749e-03,-1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04
56 | -1.973019804463441509e-02,2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02
57 | 2.193244276327200107e-02,1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02
58 | 1.781190620484248213e-02,3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02
59 | 3.103514255095075891e-02,1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02
60 | 1.330544384146788439e-01,1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02
61 | 1.115094287646074822e-01,-1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02
62 | -1.494407311730083961e-02,8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02
63 | 8.059363916605562228e-02,2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02
64 | 2.313535278373939422e-02,-5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02
65 | -5.599486132336146949e-02,5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02
66 | 5.136282737600912640e-02,4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01
67 | 4.974270823347717863e-02,-6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02
68 | -6.248895197686359787e-02,-8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02
69 | -8.139705309052219784e-02,-1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02
70 | -1.334271127830742587e-01,-4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02
71 | -4.745897853771704339e-02,1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02
72 | 1.731873150439666373e-02,-7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02
73 | -7.255800503234495491e-02,5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03
74 | 5.486038020152614325e-04,5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02
75 | 5.829412102344513208e-02,1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02
76 | 1.738615755022613479e-02,2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02,7.577017637904974456e-02
77 | 2.112576546993064769e-02,7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02,7.577017637904974456e-02,2.691879535049822997e-02
78 | 7.306140679844927432e-02,2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02,7.577017637904974456e-02,2.691879535049822997e-02,6.374986197453935688e-02
79 | 2.167369253259386619e-02,6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02,7.577017637904974456e-02,2.691879535049822997e-02,6.374986197453935688e-02,9.156146234057413458e-02
80 | 6.310540503079302421e-02,6.519694160081293544e-02,2.842745776743858116e-02,7.375395612694646275e-02,-4.075139811957750879e-02,-1.144381064408687837e-01,-2.653298352688192654e-02,-1.567233953349612963e-02,-5.673691585334017518e-02,-2.517066424745018072e-02,-4.134389301899149549e-02,-2.669970920745882645e-02,-6.801703316738186983e-03,-1.930791981497653742e-02,3.550238996852180667e-02,7.577017637904974456e-02,2.691879535049822997e-02,6.374986197453935688e-02,9.156146234057413458e-02,-2.548518833198146379e-03
81 | 


--------------------------------------------------------------------------------
/Notebook/data/section6/series1_ytrain.csv:
--------------------------------------------------------------------------------
 1 | 1.820782193763630374e-01
 2 | -1.686233766861378047e-01
 3 | 5.431798219834728381e-02
 4 | 1.890295703933203841e-01
 5 | 1.286949950230161233e-01
 6 | 1.556939009220553727e-01
 7 | -1.320145200312870526e-01
 8 | 4.477180612332511134e-02
 9 | 1.634939125073499588e-01
10 | 7.712144500935499969e-02
11 | -7.400578863873377711e-02
12 | 2.266199820141617804e-02
13 | -1.915772783543262248e-01
14 | -1.856746267109581905e-01
15 | 2.210499907194347913e-02
16 | -5.950034412144390022e-02
17 | 4.223250814338921216e-02
18 | -7.726015650015510827e-02
19 | -1.009016078948293260e-01
20 | 1.432909838879019016e-01
21 | 1.308926158980145971e-01
22 | -8.947956278991478496e-02
23 | 9.205657966235250567e-02
24 | 1.421588936348644472e-01
25 | 8.966223224907717004e-02
26 | 9.675867426389285486e-02
27 | 1.145770484988862387e-02
28 | -6.619252614385424949e-02
29 | -4.880814523516521092e-02
30 | -8.721007271931062454e-02
31 | -3.460915048891051449e-02
32 | 9.025319433950558523e-02
33 | -1.632052081599277837e-01
34 | -1.181326122866943329e-01
35 | -8.050129637396957749e-03
36 | -1.973019804463441509e-02
37 | 2.193244276327200107e-02
38 | 1.781190620484248213e-02
39 | 3.103514255095075891e-02
40 | 1.330544384146788439e-01
41 | 1.115094287646074822e-01
42 | -1.494407311730083961e-02
43 | 8.059363916605562228e-02
44 | 2.313535278373939422e-02
45 | -5.599486132336146949e-02
46 | 5.136282737600912640e-02
47 | 4.974270823347717863e-02
48 | -6.248895197686359787e-02
49 | -8.139705309052219784e-02
50 | -1.334271127830742587e-01
51 | -4.745897853771704339e-02
52 | 1.731873150439666373e-02
53 | -7.255800503234495491e-02
54 | 5.486038020152614325e-04
55 | 5.829412102344513208e-02
56 | 1.738615755022613479e-02
57 | 2.112576546993064769e-02
58 | 7.306140679844927432e-02
59 | 2.167369253259386619e-02
60 | 6.310540503079302421e-02
61 | 6.519694160081293544e-02
62 | 2.842745776743858116e-02
63 | 7.375395612694646275e-02
64 | -4.075139811957750879e-02
65 | -1.144381064408687837e-01
66 | -2.653298352688192654e-02
67 | -1.567233953349612963e-02
68 | -5.673691585334017518e-02
69 | -2.517066424745018072e-02
70 | -4.134389301899149549e-02
71 | -2.669970920745882645e-02
72 | -6.801703316738186983e-03
73 | -1.930791981497653742e-02
74 | 3.550238996852180667e-02
75 | 7.577017637904974456e-02
76 | 2.691879535049822997e-02
77 | 6.374986197453935688e-02
78 | 9.156146234057413458e-02
79 | -2.548518833198146379e-03
80 | -5.522199146434654538e-03
81 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | 
 2 | 
 3 | 
 4 | ## [Get this title for $10 on Packt's Spring Sale](https://www.packt.com/C13555?utm_source=github&utm_medium=packt-github-repo&utm_campaign=spring_10_dollar_2022)
 5 | -----
 6 | For a limited period, all eBooks and Videos are only $10. All the practical content you need \- by developers, for developers
 7 | 
 8 | # Practical-Deep-Learning-with-PyTorch
 9 | Live Training by Packt, with Jeffrey Yau.
10 | ### Download a free PDF
11 | 
12 |  <i>If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.<br>Simply click on the link to claim your free PDF.</i>
13 | <p align="center"> <a href="https://packt.link/free-ebook/9781838556839">https://packt.link/free-ebook/9781838556839 </a> </p>


--------------------------------------------------------------------------------