├── Course 1: Getting started with TensorFlow 2 ├── Capstone Project.ipynb ├── Week 2 Programming Assignment.ipynb ├── Week 3 Programming Assignment.ipynb └── Week 4 Programming Assignment.ipynb ├── Course 2: Customising your models with TensorFlow 2 ├── Capstone Project.ipynb ├── Week 1 Programming Assignment.ipynb ├── Week 2 Programming Assignment.ipynb ├── Week 3 Programming Assignment.ipynb └── Week 4 Programming Assignment.ipynb ├── Course 3: Probabilistic Deep Learning with TensorFlow 2 ├── Capstone Project.ipynb ├── Week 1 Programming Assignment.ipynb ├── Week 2 Programming Assignment.ipynb ├── Week 3 Programming Assignment.ipynb └── Week 4 Programming Assignment.ipynb └── README.md /Course 1: Getting started with TensorFlow 2/Week 2 Programming Assignment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "coursera": { 6 | "course_slug": "tensor-flow-2-1", 7 | "graded_item_id": "g0YqY", 8 | "launcher_item_id": "N6gmY" 9 | }, 10 | "kernelspec": { 11 | "display_name": "Python 3", 12 | "language": "python", 13 | "name": "python3" 14 | }, 15 | "language_info": { 16 | "codemirror_mode": { 17 | "name": "ipython", 18 | "version": 3 19 | }, 20 | "file_extension": ".py", 21 | "mimetype": "text/x-python", 22 | "name": "python", 23 | "nbconvert_exporter": "python", 24 | "pygments_lexer": "ipython3", 25 | "version": "3.7.1" 26 | }, 27 | "colab": { 28 | "name": "Week 2 Programming Assignment.ipynb", 29 | "provenance": [], 30 | "collapsed_sections": [] 31 | } 32 | }, 33 | "cells": [ 34 | { 35 | "cell_type": "markdown", 36 | "metadata": { 37 | "id": "O-21wiLf-gCD" 38 | }, 39 | "source": [ 40 | "# Programming Assignment" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": { 46 | "id": "fxkainBa-gCF" 47 | }, 48 | "source": [ 49 | "## CNN classifier for the MNIST dataset" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": { 55 | "id": "XQKECTiE-gCG" 56 | }, 57 | "source": [ 58 | "### Instructions\n", 59 | "\n", 60 | "In this notebook, you will write code to build, compile and fit a convolutional neural network (CNN) model to the MNIST dataset of images of handwritten digits.\n", 61 | "\n", 62 | "Some code cells are provided you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line: \n", 63 | "\n", 64 | "`#### GRADED CELL ####`\n", 65 | "\n", 66 | "Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly. Inside these graded cells, you can use any functions or classes that are imported below, but make sure you don't use any variables that are outside the scope of the function.\n", 67 | "\n", 68 | "### How to submit\n", 69 | "\n", 70 | "Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.\n", 71 | "\n", 72 | "### Let's get started!\n", 73 | "\n", 74 | "We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here." 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "metadata": { 80 | "id": "eR7qaZCl-gCJ" 81 | }, 82 | "source": [ 83 | "#### PACKAGE IMPORTS ####\n", 84 | "\n", 85 | "# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook\n", 86 | "\n", 87 | "import tensorflow as tf\n", 88 | "import pandas as pd\n", 89 | "import numpy as np\n", 90 | "import matplotlib.pyplot as plt\n", 91 | "%matplotlib inline\n", 92 | "\n", 93 | "# If you would like to make further imports from Tensorflow, add them here\n", 94 | "from tensorflow.keras.models import Sequential\n", 95 | "from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n" 96 | ], 97 | "execution_count": 2, 98 | "outputs": [] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": { 103 | "id": "VOQk31Sc-gCN" 104 | }, 105 | "source": [ 106 | "#### The MNIST dataset\n", 107 | "\n", 108 | "In this assignment, you will use the [MNIST dataset](http://yann.lecun.com/exdb/mnist/). It consists of a training set of 60,000 handwritten digits with corresponding labels, and a test set of 10,000 images. The images have been normalised and centred. The dataset is frequently used in machine learning research, and has become a standard benchmark for image classification models. \n", 109 | "\n", 110 | "- Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. \"Gradient-based learning applied to document recognition.\" Proceedings of the IEEE, 86(11):2278-2324, November 1998.\n", 111 | "\n", 112 | "Your goal is to construct a neural network that classifies images of handwritten digits into one of 10 classes." 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": { 118 | "id": "mOxMGi5e-gCP" 119 | }, 120 | "source": [ 121 | "#### Load and preprocess the data" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "metadata": { 127 | "id": "8zzRQzxA-gCQ", 128 | "outputId": "46e4916f-06e6-443b-d01c-e793d3286ff1", 129 | "colab": { 130 | "base_uri": "https://localhost:8080/", 131 | "height": 54 132 | } 133 | }, 134 | "source": [ 135 | "# Run this cell to load the MNIST data\n", 136 | "\n", 137 | "mnist_data = tf.keras.datasets.mnist\n", 138 | "(train_images, train_labels), (test_images, test_labels) = mnist_data.load_data()" 139 | ], 140 | "execution_count": 3, 141 | "outputs": [ 142 | { 143 | "output_type": "stream", 144 | "text": [ 145 | "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n", 146 | "11493376/11490434 [==============================] - 0s 0us/step\n" 147 | ], 148 | "name": "stdout" 149 | } 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": { 155 | "id": "MEeA_9-6-gCV" 156 | }, 157 | "source": [ 158 | "First, preprocess the data by scaling the training and test images so their values lie in the range from 0 to 1." 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "metadata": { 164 | "id": "8AW1YX_9-gCX" 165 | }, 166 | "source": [ 167 | "#### GRADED CELL ####\n", 168 | "\n", 169 | "# Complete the following function. \n", 170 | "# Make sure to not change the function name or arguments.\n", 171 | "\n", 172 | "def scale_mnist_data(train_images, test_images):\n", 173 | " \"\"\"\n", 174 | " This function takes in the training and test images as loaded in the cell above, and scales them\n", 175 | " so that they have minimum and maximum values equal to 0 and 1 respectively.\n", 176 | " Your function should return a tuple (train_images, test_images) of scaled training and test images.\n", 177 | " \"\"\"\n", 178 | " return (train_images/255, test_images/255)\n", 179 | " " 180 | ], 181 | "execution_count": 4, 182 | "outputs": [] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "metadata": { 187 | "id": "XgMBPB9d-gCa" 188 | }, 189 | "source": [ 190 | "# Run your function on the input data\n", 191 | "\n", 192 | "scaled_train_images, scaled_test_images = scale_mnist_data(train_images, test_images)" 193 | ], 194 | "execution_count": 5, 195 | "outputs": [] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "metadata": { 200 | "id": "g1r-ULOQv2o3" 201 | }, 202 | "source": [ 203 | "# Add a dummy channel dimension\n", 204 | "\n", 205 | "scaled_train_images = scaled_train_images[..., np.newaxis]\n", 206 | "scaled_test_images = scaled_test_images[..., np.newaxis]" 207 | ], 208 | "execution_count": 6, 209 | "outputs": [] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": { 214 | "id": "Cy--eSWq-gCc" 215 | }, 216 | "source": [ 217 | "#### Build the convolutional neural network model" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": { 223 | "id": "5rnippry-gCd" 224 | }, 225 | "source": [ 226 | "We are now ready to construct a model to fit to the data. Using the Sequential API, build your CNN model according to the following spec:\n", 227 | "\n", 228 | "* The model should use the `input_shape` in the function argument to set the input size in the first layer.\n", 229 | "* A 2D convolutional layer with a 3x3 kernel and 8 filters. Use 'SAME' zero padding and ReLU activation functions. Make sure to provide the `input_shape` keyword argument in this first layer.\n", 230 | "* A max pooling layer, with a 2x2 window, and default strides.\n", 231 | "* A flatten layer, which unrolls the input into a one-dimensional tensor.\n", 232 | "* Two dense hidden layers, each with 64 units and ReLU activation functions.\n", 233 | "* A dense output layer with 10 units and the softmax activation function.\n", 234 | "\n", 235 | "In particular, your neural network should have six layers." 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "metadata": { 241 | "id": "N-N7ArQ1-gCe" 242 | }, 243 | "source": [ 244 | "#### GRADED CELL ####\n", 245 | "\n", 246 | "# Complete the following function. \n", 247 | "# Make sure to not change the function name or arguments.\n", 248 | "\n", 249 | "def get_model(input_shape):\n", 250 | " \"\"\"\n", 251 | " This function should build a Sequential model according to the above specification. Ensure the \n", 252 | " weights are initialised by providing the input_shape argument in the first layer, given by the\n", 253 | " function argument.\n", 254 | " Your function should return the model.\n", 255 | " \"\"\"\n", 256 | " model = Sequential([\n", 257 | " Conv2D(filters = 8, kernel_size = 3, padding = 'same', activation = 'relu', input_shape = input_shape),\n", 258 | " MaxPooling2D((2,2)),\n", 259 | " Flatten(),\n", 260 | " Dense(64, activation = 'relu'),\n", 261 | " Dense(64, activation = 'relu'), \n", 262 | " Dense(10, activation = 'softmax') \n", 263 | " ])\n", 264 | " return model\n", 265 | " " 266 | ], 267 | "execution_count": 7, 268 | "outputs": [] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "metadata": { 273 | "id": "9L_2kj9A-gCi" 274 | }, 275 | "source": [ 276 | "# Run your function to get the model\n", 277 | "\n", 278 | "model = get_model(scaled_train_images[0].shape)" 279 | ], 280 | "execution_count": 8, 281 | "outputs": [] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "metadata": { 286 | "id": "uvrW1EA1-gCl" 287 | }, 288 | "source": [ 289 | "#### Compile the model\n", 290 | "\n", 291 | "You should now compile the model using the `compile` method. To do so, you need to specify an optimizer, a loss function and a metric to judge the performance of your model." 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "metadata": { 297 | "id": "_x9mU2Li-gCm" 298 | }, 299 | "source": [ 300 | "#### GRADED CELL ####\n", 301 | "\n", 302 | "# Complete the following function. \n", 303 | "# Make sure to not change the function name or arguments.\n", 304 | "\n", 305 | "def compile_model(model):\n", 306 | " \"\"\"\n", 307 | " This function takes in the model returned from your get_model function, and compiles it with an optimiser,\n", 308 | " loss function and metric.\n", 309 | " Compile the model using the Adam optimiser (with default settings), the cross-entropy loss function and\n", 310 | " accuracy as the only metric. \n", 311 | " Your function doesn't need to return anything; the model will be compiled in-place.\n", 312 | " \"\"\"\n", 313 | " opt = tf.keras.optimizers.Adam()\n", 314 | " acc = tf.keras.metrics.SparseCategoricalAccuracy()\n", 315 | " model.compile(optimizer = opt, loss = 'sparse_categorical_crossentropy', metrics = [acc])\n", 316 | " \n", 317 | " " 318 | ], 319 | "execution_count": 9, 320 | "outputs": [] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "metadata": { 325 | "id": "pY08R9yB-gCr" 326 | }, 327 | "source": [ 328 | "# Run your function to compile the model\n", 329 | "\n", 330 | "compile_model(model)" 331 | ], 332 | "execution_count": 10, 333 | "outputs": [] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": { 338 | "id": "pHUcXibk-gCv" 339 | }, 340 | "source": [ 341 | "#### Fit the model to the training data\n", 342 | "\n", 343 | "Now you should train the model on the MNIST dataset, using the model's `fit` method. Set the training to run for 5 epochs, and return the training history to be used for plotting the learning curves." 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "metadata": { 349 | "id": "cDnNXqN1-gCw" 350 | }, 351 | "source": [ 352 | "#### GRADED CELL ####\n", 353 | "\n", 354 | "# Complete the following function. \n", 355 | "# Make sure to not change the function name or arguments.\n", 356 | "\n", 357 | "def train_model(model, scaled_train_images, train_labels):\n", 358 | " \"\"\"\n", 359 | " This function should train the model for 5 epochs on the scaled_train_images and train_labels. \n", 360 | " Your function should return the training history, as returned by model.fit.\n", 361 | " \"\"\"\n", 362 | " history = model.fit(scaled_train_images, train_labels, epochs = 5, batch_size = 256)\n", 363 | " return history\n", 364 | " " 365 | ], 366 | "execution_count": 11, 367 | "outputs": [] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "metadata": { 372 | "id": "Y1n3wh49-gCz", 373 | "outputId": "d645d7f1-8b0f-413e-b6b3-ebc324a80ceb", 374 | "colab": { 375 | "base_uri": "https://localhost:8080/", 376 | "height": 201 377 | } 378 | }, 379 | "source": [ 380 | "# Run your function to train the model\n", 381 | "\n", 382 | "history = train_model(model, scaled_train_images, train_labels)" 383 | ], 384 | "execution_count": 12, 385 | "outputs": [ 386 | { 387 | "output_type": "stream", 388 | "text": [ 389 | "Epoch 1/5\n", 390 | "235/235 [==============================] - 15s 64ms/step - loss: 0.4271 - sparse_categorical_accuracy: 0.8816\n", 391 | "Epoch 2/5\n", 392 | "235/235 [==============================] - 15s 63ms/step - loss: 0.1358 - sparse_categorical_accuracy: 0.9597\n", 393 | "Epoch 3/5\n", 394 | "235/235 [==============================] - 15s 63ms/step - loss: 0.0890 - sparse_categorical_accuracy: 0.9732\n", 395 | "Epoch 4/5\n", 396 | "235/235 [==============================] - 15s 63ms/step - loss: 0.0671 - sparse_categorical_accuracy: 0.9800\n", 397 | "Epoch 5/5\n", 398 | "235/235 [==============================] - 16s 69ms/step - loss: 0.0550 - sparse_categorical_accuracy: 0.9832\n" 399 | ], 400 | "name": "stdout" 401 | } 402 | ] 403 | }, 404 | { 405 | "cell_type": "markdown", 406 | "metadata": { 407 | "id": "rhd3yK0i-gC3" 408 | }, 409 | "source": [ 410 | "#### Plot the learning curves\n", 411 | "\n", 412 | "We will now plot two graphs:\n", 413 | "* Epoch vs accuracy\n", 414 | "* Epoch vs loss\n", 415 | "\n", 416 | "We will load the model history into a pandas `DataFrame` and use the `plot` method to output the required graphs." 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "metadata": { 422 | "id": "y0t2Xjgq-gC4" 423 | }, 424 | "source": [ 425 | "# Run this cell to load the model history into a pandas DataFrame\n", 426 | "\n", 427 | "frame = pd.DataFrame(history.history)" 428 | ], 429 | "execution_count": 13, 430 | "outputs": [] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "metadata": { 435 | "id": "xQqYQiR4-gC7", 436 | "outputId": "5f055f55-3978-4c6f-b93d-0ee92a6cd9b1", 437 | "colab": { 438 | "base_uri": "https://localhost:8080/", 439 | "height": 313 440 | } 441 | }, 442 | "source": [ 443 | "# Run this cell to make the Accuracy vs Epochs plot\n", 444 | "\n", 445 | "acc_plot = frame.plot(y=\"sparse_categorical_accuracy\", title=\"Accuracy vs Epochs\", legend=False)\n", 446 | "acc_plot.set(xlabel=\"Epochs\", ylabel=\"Accuracy\")" 447 | ], 448 | "execution_count": 14, 449 | "outputs": [ 450 | { 451 | "output_type": "execute_result", 452 | "data": { 453 | "text/plain": [ 454 | "[Text(0, 0.5, 'Accuracy'), Text(0.5, 0, 'Epochs')]" 455 | ] 456 | }, 457 | "metadata": { 458 | "tags": [] 459 | }, 460 | "execution_count": 14 461 | }, 462 | { 463 | "output_type": "display_data", 464 | "data": { 465 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhcZZn38e+PTjpbd8hKQsgKJMEgCUIbFgUyIApIYBQdQRGZURh1VBx1Rpj3vVxwGOZVZ8ZRUAcRZZNFVC6CICAEmXFYEsAOayAgkKQTsoes3enu+/3jnEoqneru6qRPV3XX73NddXHWOncfUueu+3nqPEcRgZmZWVv7lToAMzMrT04QZmZWkBOEmZkV5ARhZmYFOUGYmVlBThBmZlaQE4SZtUtSSDq01HFYaThBWMlJeljSekkDSh1LOZP0mqRtkjbnva4qdVzWdzlBWElJmgycAARwVg8fu19PHq+bzI2ImrzX50odkPVdThBWahcAjwE/Bz6Rv0LSBEm/lrRa0tr8b8uSLpL0gqRNkp6XdFS6fLcmEUk/l/TP6fQcScskfVXSSuBnkoZLujs9xvp0enze/iMk/UxSQ7r+znT5s5Lm5m3XX9IaSe9o+wemcZ6ZN98vPd5RkgZKuin9+zZIWiBpTFdPoqQLJf1R0lWSNkp6UdIpeevHSbpL0jpJSyRdlLeuStI/SXolPZ9PSpqQ9/bvkfRyGt/VkpTud6ikP6THWyPptq7GbeXNCcJK7QLg5vT1vtzFUVIVcDfwOjAZOAi4NV33YeAb6b5DSSqPtUUebywwApgEXEzyGfhZOj8R2AbkN9vcCAwGDgcOAP4jXX4DcH7edmcAKyLi6QLHvAU4L2/+fcCaiHiKJCnuD0wARgKfTmPYG8cArwCjgK8Dv5Y0Il13K7AMGAd8CPgXSSen676UxncGyfn8G2Br3vueCbwTmAn8VRo/wLeA+4HhwHjgB3sZt5WriPDLr5K8gHcDO4BR6fyLwN+n08cBq4F+Bfa7D7iknfcM4NC8+Z8D/5xOzwGagIEdxHQksD6dPhBoBYYX2G4csAkYms7fAfxjO+95aLrt4HT+ZuBr6fTfAP8LzCzifL0GbAY25L0uStddCDQAytv+CeDjJMmnBajNW3cl8PN0ejFwdgfn891587cDl6bTNwDXAONL/W/Jr2xeriCslD4B3B8Ra9L5X7CrmWkC8HpENBfYbwLJN+W9sToitudmJA2W9F+SXpf0FvAIMCytYCYA6yJifds3iYgG4I/AOZKGAaeTXPj3EBFLgBeAuZIGk1Q8v0hX30iS8G5Nm7G+Lal/B/H/ZUQMy3v9JG/d8ojIH33zdZJENi79Oza1WXdQOt3Z+VyZN70VqEmn/xEQ8ISk5yT9TQfvYb1Qb+yksz5A0iCS5oqqtD8AYADJxXkWsBSYKKlfgSSxFDiknbfeStIklDOWpGklp+3wxV8GpgPHRMRKSUcCT5Nc+JYCIyQNi4gNBY51PfApks/RoxGxvP2/eGcz037A82nSICJ2AN8Evpl22N9D8o3+px28V3sOkqS8JDERuIukshghqTYvSUwEcvHmzuezXTlYRKwELgKQ9G7g95Ieyf1t1vu5grBS+UuSZo8ZJM06RwJvA/6bpG/hCWAF8K+ShqSdue9K970W+Iqko5U4VNKkdN2fgI+mHa+nASd1EkctSZv/hrS9/uu5FRGxArgX+GHamd1f0ol5+94JHAVcQtLc0pFbgfcCn2FX9YCkv5B0RFqxvEXS5NbayXu15wDgC2mcHyY5n/dExFKSZqwr0/M4E/gkcFO637XAtyRNTc/nTEkjOzuYpA/ndeivJ0m+exu7lSEnCCuVTwA/i4g3ImJl7kXSQfwxkm/wc0na798gqQI+AhARvwSuILnQbiK5UOc6Yy9J99uQvs+dncTxPWAQsIbk11S/a7P+4yQX7ReBVcAXcysiYhvwK2AK8OuODpImm0eB44H8X/uMJem/eIukGeoPJM1O7Zmn3e+D+E3euseBqenfcgXwoYjIdd6fR9LZ3wD8Bvh6RPw+XffvJH0L96dx/JTknHTmncDjkjaTVCqXRMSrRexnvYR2b7I0s66Q9DVgWkSc3+nG2cZxIfCpiHh3KeOwvsV9EGZ7KW2S+iRJlWHW57iJyWwvpDeaLQXujYhHSh2PWRbcxGRmZgW5gjAzs4L6TB/EqFGjYvLkyaUOw8ysV3nyySfXRMToQuv6TIKYPHkyCxcuLHUYZma9iqTX21vnJiYzMyvICcLMzApygjAzs4KcIMzMrCAnCDMzK8gJwszMCnKCMDOzgvrMfRBmZn1ZRLCpsZl1m5tYu6WJdVuaWLelkbVbmhg2qJqPHjOx24/pBGFmVgKtrcFb23fsvNiv3bzror9m53QuGTSyfssOmloKP4/pqInDnCDMzMpVS2uwYeuui3pywW/clQC2NLFu867p9VubaGktPFhqzYB+jBhSzYgh1YzbfyBvHzeUETXVjBxSzcghA3ZOj0jnB1VXZfI3OUGYmRWwo6WV9Vua2lzgG/O+1ec39SQX/PYGxx46sB8jawYwYkg1k0YO5qhJw9IEMGDnhX7EkGpG1lQzfHA1A/tnc8HvqkwTRPpM4P8EqoBrI+Jf26yfBFwHjAbWAedHxLJ03beB95N0pD9A8jhDj01uZnulsbmlTVPOruabQss3bttR8H0kGD5410V96gE16Tf5dFnNgLxv99UMH1JN/6re+XugzBJE+hD2q4FTSZ4nvEDSXRHxfN5m3wVuiIjrJZ0MXAl8XNLxwLuAmel2/0Py8PmHs4rXzHqXbU0trN3SWLD5Zt2W3b/pr9vcxKbG5oLvU7WfGD541wX+beOG7naBHzFkwM5v9yOGJN/wq/ZTD/+1pZFlBTEbWJJ7iLmkW4GzgfwEMQP4Ujo9n10PmA9gIFBN8vD6/sCbGcZqZmWgsbmFV1dvoWHDtt2ab3Lt+eu2NO3swN22o6Xge/Sv0m7NNxOGD951sa/ZddEfmU4PHdif/Srkgt9VWSaIg0geyZizDDimzTb1wAdJmqE+ANRKGhkRj0qaD6wgSRBXRcQLbQ8g6WLgYoCJE7u/B9/MstHaGryxbiuL39zE4pWbdv73z2u27NFxO6Dffjsv7iOGDODg0TU7m3dG1eR9w0+3qR3QD8kX/O5Q6k7qrwBXSboQeARYDrRIOhR4GzA+3e4BSSdExH/n7xwR1wDXANTV1bl/wqzMRASrNzXuSgRpMnj5zc27VQATRwxm2phaTjt8LNPG1jJxxOCdzTyDq6t8wS+RLBPEcmBC3vz4dNlOEdFAUkEgqQY4JyI2pA+EfywiNqfr7gWOA3ZLEGZWPjZt38FLb25i8crNLF75Fi+u3MRLb25i/dZdnb2jagYwfWwN582eyPSxNUwfO5SpB9QwZECpv6taIVn+X1kATJU0hSQxnAt8NH8DSaOAdRHRClxG8osmgDeAiyRdSdLEdBLwvQxjNbMiNTa38MqqLbz05qadSWDxyk0s37Bt5zZDqquYNraW094+lmljapk+tpbpY2oZWTOghJFbV2WWICKiWdLngPtIfuZ6XUQ8J+lyYGFE3AXMAa6UFCRNTH+X7n4HcDLwDEmH9e8iYl5WsZrZnorpJ+hfJQ4ZXcPRk4bz0WMmctjYWqaNqeWgYYPc8dsHqK/cWlBXVxd+JrVZ13W1n+CwsbVMG5v8d8qoIb32N/6WkPRkRNQVWueGP7MK4n4C6wr/Hzfrg7rST/C+w8fu7COYPtb9BLaLE4RZL7Y3/QS5ROB+AuuME4RZLxARrN7cuKuPIK0KXurkfoLDxtYyeeQQqvu5n8C6zgnCrMy07SfIVQW79xNUM31sLefOnsBhY2vdT2CZ8L8msxLZ236CaWNrGeV+AusBThBmGWttDZau38qLK91PYL2LE4RZBlpagyf+vI55ixq495kVuzUPTRgxiOljhrqfwMqeE4RZN4kInnpjA3cvauC3i1awalMjg/pX8Z4ZY3jXISOZnt5l7H4C6y38L9VsH0QEzzW8xbxFDdxdv4LlG7ZRXbUfc6aPZu6scZzytgMYXO2PmfVO/pdrtheWrNrEXfUruLu+gVfXbKFqP3HC1FH8/anTeO/hYxg6sH+pQzTbZ04QZkV6Y+1W5i1qYF59Ay+u3IQEx04ZyadOOJjT3j6WEUOqSx2iWbdygjDrwMqN27l7UQPzFq2gfukGAI6aOIyvz53BGUccyJihA0scoVl2nCDM2li7uZF7nl3JvPoGFry2jgg4fNxQLj39MN5/xIFMGDG41CGa9QgnCDNg47Yd3PdckhT+95W1tLQGh4wewhdPmcaZsw7kkNE1pQ7RrMc5QVjF2tLYzO9feJN59Sv4w0ur2NESTBgxiL898WDmzhrHYWNr/Sxkq2hOEFZRtu9o4eHFq5hXv4IHX3yT7TtaGTN0ABccN5m5s8Yxa/z+TgpmKScI6/N2tLTyPy+vYV59A/c//yabG5sZOaSaDx09nrkzx/HOySM8nIVZAU4Q1ie1tAaPv7o2Geri2ZVs2LqD2oH9OOOIscydNY7jDh5JPz8q06xDThDWZ7S2Bk8vXc+8+hX89pkVrN7UyODqKk6dMYa5M8dxwrRRDOhXVeowzXoNJwjr1XYOdVHfwN2L0qEu+u3HydMPYO6scZx82AEMqnZSMNsbThDWK7385ibm1Sc3sP15zRb6pUNdfPm90zh1xhhqPdSF2T5zgrBe4/W1W7h70Yrdhro47uCRXHziwZx2+FiGe6gLs27lBGFlbcXGbfw2TQr1yzYCcPSk4XwjHeriAA91YZYZJwgrO2s2N3LvMyuYV7+CJ15bB8ARB+3PZacfxvtnHsj44R7qwqwnOEFYWdi4NR3qYlEDf1yyhtaAqQfU8OVTp3HmrHFMGTWk1CGaVRwnCCuZzY3N/P75N5lX38AjL69mR0swaeRgPjvnUObOGsf0sbWlDtGsojlBWI/avqOF+S+uYt6iBh58YRWNza0cuP9ALjw+GeriiIM81IVZuXCCsMw1NbfyP0tWM69+Bfc/t5ItTS2MqqnmI++cwNxZ4zh64nAPdWFWhpwgLBMtrcFjr65lXn0y1MXGbTsYOrAfZ84cx9xZ4zj24BEe6sKszGWaICSdBvwnUAVcGxH/2mb9JOA6YDSwDjg/Ipal6yYC1wITgADOiIjXsozX9k1ra/DUG+uZV9/Ab59ZyZrNjQzJDXUxaxwnTB1NdT8nBbPeIrMEIakKuBo4FVgGLJB0V0Q8n7fZd4EbIuJ6SScDVwIfT9fdAFwREQ9IqgFas4rV9l5E8Ozyt5i3qIG76xto2LidAf324+TDkqEu/mK6h7ow662yrCBmA0si4lUASbcCZwP5CWIG8KV0ej5wZ7rtDKBfRDwAEBGbM4zT9sJLuaEu6ht4be1W+leJE6eO5h9Om8573uahLsz6giwTxEHA0rz5ZcAxbbapBz5I0gz1AaBW0khgGrBB0q+BKcDvgUsjoiV/Z0kXAxcDTJw4MYu/wdrY0dLK+dc+zuN/Xsd+guMPGcVn5hzC+w4fy7DBHurCrC8pdSf1V4CrJF0IPAIsB1pI4joBeAfwBnAbcCHw0/ydI+Ia4BqAurq66KmgK9mvnlzG439ex9+/ZxofPWYio2sHlDokM8tIlgliOUkHc874dNlOEdFAUkGQ9jOcExEbJC0D/pTXPHUncCxtEoT1rKbmVq6av4RZE4bxhVMO9f0KZn1clj8pWQBMlTRFUjVwLnBX/gaSRknKxXAZyS+acvsOkzQ6nT+Z3fsurAR+/dQylq3fxhffM9XJwawCZJYgIqIZ+BxwH/ACcHtEPCfpcklnpZvNARZLegkYA1yR7ttC0vz0oKRnAAE/ySpW61x+9TBn2ujOdzCzXi/TPoiIuAe4p82yr+VN3wHc0c6+DwAzs4zPiperHr71l2939WBWIXzXknXK1YNZZXKCsE6578GsMjlBWIeamlv5wUOuHswqkROEdehXTy1j+QZXD2aVyAnC2tXU3MpVDy3hSFcPZhXJCcLa5erBrLI5QVhB+dXDSa4ezCqSE4QV5OrBzJwgbA+uHswMnCCsgDuedPVgZk4Q1kZTcytXz3f1YGZOENaGqwczy3GCsJ1y1cM7Jrp6MDMnCMuzq3qY5urBzJwgLJFfPZw4dVSpwzGzMuAEYYCrBzPbkxOEuXows4KcIIxfPrnU1YOZ7cEJosI1Nbdy9UOuHsxsT04QFe6XTy6lYeN2Vw9mtgcniAqWqx6OcvVgZgU4QVQwVw9m1hEniAqVXz2c4OrBzApwgqhQrh7MrDNOEBXI1YOZFcMJogLdvtDVg5l1zgmiwjQ2t/DD+a4ezKxzThAV5pcLl7l6MLOidJogJM2V5ETSB+Sqh6MnDXf1YGadKubC/xHgZUnflnRY1gFZdnZVD35anJl1rtMEERHnA+8AXgF+LulRSRdLqu1sX0mnSVosaYmkSwusnyTpQUmLJD0saXyb9UMlLZN0VRf+Jisgv3p496GuHsysc0U1HUXEW8AdwK3AgcAHgKckfb69fSRVAVcDpwMzgPMkzWiz2XeBGyJiJnA5cGWb9d8CHikmRuuYqwcz66pi+iDOkvQb4GGgPzA7Ik4HZgFf7mDX2cCSiHg1IppIksvZbbaZATyUTs/PXy/paGAMcH9xf4q1p7G5hatdPZhZFxVTQZwD/EdEHBER34mIVQARsRX4ZAf7HQQszZtfli7LVw98MJ3+AFAraWTaKf5vwFc6Cixt6looaeHq1auL+FMq0+0Ll7HC1YOZdVExCeIbwBO5GUmDJE0GiIgH9/H4XwFOkvQ0cBKwHGgBPgvcExHLOto5Iq6JiLqIqBs9evQ+htI3ue/BzPZWvyK2+SVwfN58S7rsnZ3stxyYkDc/Pl22U0Q0kFYQkmqAcyJig6TjgBMkfRaoAaolbY6IPTq6rWO56uHbH5rp6sHMuqSYBNEv7UMAICKaJFUXsd8CYKqkKSSJ4Vzgo/kbSBoFrIuIVuAy4Lr0GB/L2+ZCoM7Joety1UOdqwcz2wvFNDGtlnRWbkbS2cCaznaKiGbgc8B9wAvA7RHxnKTL895vDrBY0kskHdJXdDF+68CuvgffNW1mXaeI6HgD6RDgZmAcIJKO5wsiYkn24RWvrq4uFi5cWOowykZjcwtzvvMwBw0bxC8/fZwThJkVJOnJiKgrtK7TJqaIeAU4Nu0jICI2d3N8loFc9fCdD81ycjCzvVJMHwSS3g8cDgzMXWwi4vIM47J9kN/38K5DR5Y6HDPrpYq5Ue7HJOMxfZ6kienDwKSM47J9cPuCpe57MLN9Vkwn9fERcQGwPiK+CRwHTMs2LNtbyV3Tr7h6MLN9VkyC2J7+d6ukccAOkvGYrAzdvmApK99y9WBm+66YPoh5koYB3wGeAgL4SaZR2V7JVQ/vnOzqwcz2XYcJIh0T6cGI2AD8StLdwMCI2Ngj0VmX5KqHf/sr/3LJzPZdh01M6R3OV+fNNzo5lKf86uH4Q1w9mNm+K6YP4kFJ58hfScvabe57MLNuVkyC+FuSwfkaJb0laZOktzKOy7pg+44Wfujqwcy6WTF3Unf6aFErrdsXuu/BzLpfpwlC0omFlkeEHwVaBlw9mFlWivmZ6z/kTQ8keZTok8DJmURkXeLqwcyyUkwT09z8eUkTgO9lFpEVLVc9zJ48wtWDmXW7Yjqp21oGvK27A7Guy1UPfta0mWWhmD6IH5DcPQ1JQjmS5I5qK6H86uE4Vw9mloFi+iDyn8LTDNwSEX/MKB4rUu6+h39334OZZaSYBHEHsD0iWgAkVUkaHBFbsw3N2rN9Rws/fHiJqwczy1RRd1IDg/LmBwG/zyYcK8ZtC5by5luN7nsws0wVkyAG5j9mNJ0enF1I1hFXD2bWU4pJEFskHZWbkXQ0sC27kKwjrh7MrKcU0wfxReCXkhpIHjk6luQRpNbDdlYPU1w9mFn2irlRboGkw4Dp6aLFEbEj27CskFz18B8fOdLVg5llrtMmJkl/BwyJiGcj4lmgRtJnsw/N8u1WPRzs6sHMsldMH8RF6RPlAIiI9cBF2YVkhdz6xBvuezCzHlVMgqjKf1iQpCqgOruQrK2kenjF1YOZ9ahiEsTvgNsknSLpFOAW4N5sw7J8tz7xBqs2uXows55VzK+YvgpcDHw6nV9E8ksm6wGuHsysVDqtICKiFXgceI3kWRAnAy9kG5bluHows1Jpt4KQNA04L32tAW4DiIi/6JnQLFc9HDNlBMcfMqrU4ZhZhemogniRpFo4MyLeHRE/AFq68uaSTpO0WNISSZcWWD9J0oOSFkl6WNL4dPmRkh6V9Fy6riJvzNtVPUwrdShmVoE6ShAfBFYA8yX9JO2gLrqNI/2109XA6cAM4DxJM9ps9l3ghoiYCVwOXJku3wpcEBGHA6cB35M0rNhj9wX51YPvmjazUmg3QUTEnRFxLnAYMJ9kyI0DJP1I0nuLeO/ZwJKIeDUimoBbgbPbbDMDeCidnp9bHxEvRcTL6XQDsAoYXfyf1fvd4urBzEqsmE7qLRHxi/TZ1OOBp0l+2dSZg4ClefPL0mX56kkqFYAPALWSdvu6LGk2yX0Xr7Q9gKSLJS2UtHD16tVFhNQ7bN/Rwo9cPZhZiXXpmdQRsT4iromIU7rp+F8BTpL0NHASsJy8fg5JBwI3An+d/pqqbTzXRERdRNSNHt13CgxXD2ZWDoq5D2JvLQcm5M2PT5ftlDYffRBAUg1wTm5YD0lDgd8C/yciHsswzrLi6sHMykWXKoguWgBMlTRFUjVwLnBX/gaSRknKxXAZcF26vBr4DUkH9h0Zxlh2XD2YWbnILEFERDPwOeA+khvrbo+I5yRdLumsdLM5wGJJLwFjgCvS5X8FnAhcKOlP6evIrGItF7nq4diDXT2YWell2cRERNwD3NNm2dfypu8A9qgQIuIm4KYsYytHuerhP899R6lDMTPLtInJuiB334OrBzMrF04QZeIXj7/B6k2NXHKK+x7MrDw4QZSB7Tta+NEfXD2YWXlxgigDrh7MrBw5QZSYqwczK1dOECWWqx5834OZlRsniBLKVQ/HHTySY/20ODMrM04QJbSz7+E9U0sdipnZHpwgSsTVg5mVOyeIErnZ1YOZlTkniBLYvqOFH7t6MLMy5wRRAq4ezKw3cILoYa4ezKy3cILoYTfvvO/B1YOZlTcniB6Uqx6OP2Qkx7h6MLMy5wTRg2567PV0zCVXD2ZW/pwgesi2phZ+/IdXXT2YWa/hBNFDbn78ddZsdvVgZr2HE0QPcPVgZr2RE0QPcPVgZr2RE0TGXD2YWW/lBJGxXPXg5z2YWW/jBJGhXPXwrkNHMnvKiFKHY2bWJU4QGdrV9+Dqwcx6HyeIjCTVwyuuHsys1+pX6gD6qqR6aOKHrh7MrJdyBZEBVw9m1he4gsiAqwcz6wtcQXQzVw9m1le4guhmuerhR77vwcx6uUwrCEmnSVosaYmkSwusnyTpQUmLJD0saXzeuk9Iejl9fSLLOLtLrnp496GjeOdkVw9m1rtlliAkVQFXA6cDM4DzJM1os9l3gRsiYiZwOXBluu8I4OvAMcBs4OuShmcVa3e56bGkevCzps2sL8iygpgNLImIVyOiCbgVOLvNNjOAh9Lp+Xnr3wc8EBHrImI98ABwWoax7rOtTc381yOuHsys78gyQRwELM2bX5Yuy1cPfDCd/gBQK2lkkfsi6WJJCyUtXL16dbcFvjdufuwNVw9m1qeU+ldMXwFOkvQ0cBKwHGgpdueIuCYi6iKibvTo0VnF2ClXD2bWF2X5K6blwIS8+fHpsp0iooG0gpBUA5wTERskLQfmtNn34Qxj3SeuHsysL8qyglgATJU0RVI1cC5wV/4GkkZJysVwGXBdOn0f8F5Jw9PO6femy8qOqwcz66sySxAR0Qx8juTC/gJwe0Q8J+lySWelm80BFkt6CRgDXJHuuw74FkmSWQBcni4rO7nq4YuuHsysj1FElDqGblFXVxcLFy7s0WNubWrmhP83nxnjhnLjJ4/p0WObmXUHSU9GRF2hdaXupO7VbnrsddZuafKzps2sT3KC2Etbm5r5rz+8yglTR1Hnvgcz64OcIPaSqwcz6+ucIPaCqwczqwROEHvB1YOZVQIniC5y9WBmlcIJooty1YPvezCzvs4Jogvyq4ejJ7l6MLO+zQmiC2581NWDmVUOJ4giJWMuuXows8rhBFGkGx99nXWuHsysgjhBFMHVg5lVIieIIrh6MLNK5ATRCVcPZlapnCA6ccPO6mFaqUMxM+tRThAd2NLYzDWPvMqJ00Zz9KThpQ7HzKxHOUF04MbHkurBYy6ZWSVygmiHqwczq3ROEO1w9WBmlc4JogBXD2ZmThAFuXowM3OC2IOrBzOzhBNEGzf4rmkzM8AJYjdJ9fAKJ00bzVETXT2YWWVzgshzw6Ovs37rDi5x9WBm5gSR4+rBzGx3ThApVw9mZrtzgsDVg5lZIU4QuHowMyuk4hOEqwczs8IyTRCSTpO0WNISSZcWWD9R0nxJT0taJOmMdHl/SddLekbSC5IuyyrGzY3NHHfISFcPZmZt9MvqjSVVAVcDpwLLgAWS7oqI5/M2+7/A7RHxI0kzgHuAycCHgQERcYSkwcDzkm6JiNe6O84xQwfyw48d3d1va2bW62VZQcwGlkTEqxHRBNwKnN1mmwCGptP7Aw15y4dI6gcMApqAtzKM1czM2sgyQRwELM2bX5Yuy/cN4HxJy0iqh8+ny+8AtgArgDeA70bEurYHkHSxpIWSFq5evbqbwzczq2yl7qQ+D/h5RIwHzgBulLQfSfXRAowDpgBflnRw250j4pqIqIuIutGjR/dk3GZmfV6WCWI5MCFvfny6LN8ngdsBIuJRYCAwCvgo8LuI2BERq4A/AnUZxmpmZm1kmSAWAFMlTZFUDZwL3NVmmzeAUwAkvY0kQaxOl5+cLh8CHAu8mGGsZmbWRmYJIiKagc8B9wEvkPxa6TlJl0s6K93sy8BFkuqBW4ALIyJIfv1UI+k5kkTzs4hYlFWsZma2JyXX496vrq4uFi5cWOowzMx6FUlPRkTBJvxSd1KbmVmZ6jMVhKTVwOv78BajgDXdFPYnIfAAAAW5SURBVE53clxd47i6xnF1TV+Ma1JEFPwZaJ9JEPtK0sL2yqxSclxd47i6xnF1TaXF5SYmMzMryAnCzMwKcoLY5ZpSB9AOx9U1jqtrHFfXVFRc7oMwM7OCXEGYmVlBThBmZlZQRSWIIp5wN0DSben6xyVNLpO4LpS0WtKf0teneiiu6yStkvRsO+sl6ftp3IskHVUmcc2RtDHvfH2th+KakD4h8XlJz0m6pMA2PX7Oioyrx8+ZpIGSnpBUn8b1zQLb9Phnssi4SvKZTI9dlT6F8+4C67r3fEVERbyAKuAV4GCgGqgHZrTZ5rPAj9Ppc4HbyiSuC4GrSnDOTgSOAp5tZ/0ZwL2ASAZUfLxM4poD3F2C83UgcFQ6XQu8VOD/ZY+fsyLj6vFzlp6DmnS6P/A4cGybbUrxmSwmrpJ8JtNjfwn4RaH/X919viqpgijmCXdnA9en03cAp0hSGcRVEhHxCLDHg5rynA3cEInHgGGSDiyDuEoiIlZExFPp9CaSQSrbPiSrx89ZkXH1uPQcbE5n+6evtr+a6fHPZJFxlYSk8cD7gWvb2aRbz1clJYhinnC3c5tIRqPdCIwsg7gAzkmbJO6QNKHA+lIoNvZSOC5tIrhX0uE9ffC0tH8HybfPfCU9Zx3EBSU4Z2lzyZ+AVcADEdHu+erBz2QxcUFpPpPfA/4RaG1nfbeer0pKEL3ZPGByRMwEHmDXNwQr7CmS8WVmAT8A7uzJg0uqAX4FfDEiyuZZ6p3EVZJzFhEtEXEkyQPFZkt6e08ctzNFxNXjn0lJZwKrIuLJrI+VU0kJopgn3O3cRlI/YH9gbanjioi1EdGYzl4LHJ1xTMUq5pz2uIh4K9dEEBH3AP0ljeqJY0vqT3IRvjkifl1gk5Kcs87iKuU5S4+5AZgPnNZmVSk+k53GVaLP5LuAsyS9RtIUfbKkm9ps063nq5ISRDFPuLsL+EQ6/SHgoUh7e0oZV5s26rNI2pDLwV3ABekvc44FNkbEilIHJWlsrt1V0mySf+eZX1TSY/4UeCEi/r2dzXr8nBUTVynOmaTRkoal04OAU9nzyZE9/pksJq5SfCYj4rKIGB8Rk0muEw9FxPltNuvW89Vvb3fsbSKiWVLuCXdVwHWRPuEOWBgRd5F8iG6UtISkE/TcMonrC0qewtecxnVh1nEBSLqF5NctoyQtA75O0mFHRPwYuIfkVzlLgK3AX5dJXB8CPiOpGdgGnNsDiR6Sb3gfB55J268B/gmYmBdbKc5ZMXGV4pwdCFwvqYokId0eEXeX+jNZZFwl+UwWkuX58lAbZmZWUCU1MZmZWRc4QZiZWUFOEGZmVpAThJmZFeQEYWZmBTlBmHVCUkveqJ1/UoERd/fhvSernVFpzUqtYu6DMNsH29JhF8wqiisIs70k6TVJ35b0TPr8gEPT5ZMlPZQO5PagpInp8jGSfpMOiFcv6fj0raok/UTJswfuT+/eRdIXlDzDYZGkW0v0Z1oFc4Iw69ygNk1MH8lbtzEijgCuIhlpE5LB7q5PB3K7Gfh+uvz7wB/SAfGOAp5Ll08Fro6Iw4ENwDnp8kuBd6Tv8+ms/jiz9vhOarNOSNocETUFlr8GnBwRr6aD4a2MiJGS1gAHRsSOdPmKiBglaTUwPm+Qt9zw2w9ExNR0/qtA/4j4Z0m/AzaTjKx6Z94zCsx6hCsIs30T7Ux3RWPedAu7+gbfD1xNUm0sSEfnNOsxThBm++Yjef99NJ3+X3YNkvYx4L/T6QeBz8DOB9Ls396bStoPmBAR84GvkgzbvEcVY5YlfyMx69ygvFFQAX4XEbmfug6XtIikCjgvXfZ54GeS/gFYza4RWy8BrpH0SZJK4TNAe0N9VwE3pUlEwPfTZxOY9Rj3QZjtpbQPoi4i1pQ6FrMsuInJzMwKcgVhZmYFuYIwM7OCnCDMzKwgJwgzMyvICcLMzApygjAzs4L+P3LBdPTpjRGSAAAAAElFTkSuQmCC\n", 466 | "text/plain": [ 467 | "
" 468 | ] 469 | }, 470 | "metadata": { 471 | "tags": [], 472 | "needs_background": "light" 473 | } 474 | } 475 | ] 476 | }, 477 | { 478 | "cell_type": "code", 479 | "metadata": { 480 | "id": "JGgTGfH4-gDA", 481 | "outputId": "64673367-8537-4adb-ab83-d535015b7a0b", 482 | "colab": { 483 | "base_uri": "https://localhost:8080/", 484 | "height": 313 485 | } 486 | }, 487 | "source": [ 488 | "# Run this cell to make the Loss vs Epochs plot\n", 489 | "\n", 490 | "acc_plot = frame.plot(y=\"loss\", title = \"Loss vs Epochs\",legend=False)\n", 491 | "acc_plot.set(xlabel=\"Epochs\", ylabel=\"Loss\")" 492 | ], 493 | "execution_count": 15, 494 | "outputs": [ 495 | { 496 | "output_type": "execute_result", 497 | "data": { 498 | "text/plain": [ 499 | "[Text(0, 0.5, 'Loss'), Text(0.5, 0, 'Epochs')]" 500 | ] 501 | }, 502 | "metadata": { 503 | "tags": [] 504 | }, 505 | "execution_count": 15 506 | }, 507 | { 508 | "output_type": "display_data", 509 | "data": { 510 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xU9Z3/8dcnd0gIEBJu4RoSrYCKGlHrpQpVQVtwq63Y2tatrdWqdB/2t1u7299u1939/XrZ7a9ecNW6ttpWqbWXpS1oXQHvF4IiN0VDuF9DkEsC5Pr5/TEHnMQJTCAzZ5J5Px+PeTBzzvec+czRmXe+53su5u6IiIh0lBF2ASIikpoUECIiEpMCQkREYlJAiIhITAoIERGJSQEhIiIxKSBE0oCZ3WBmL4Vdh/QsCgjpkcxsvZl9Muw6joeZXWxmbWZW3+FxXti1iUTLCrsAkTS11d1HhF2EyNGoByG9ipnlmtlPzGxr8PiJmeUG84rN7E9mtsfMdpvZi2aWEcz7tpltMbP9ZrbGzKbGWPc5ZrbdzDKjpv2VmS0Pnk82syoz22dmO8zsx8f5GRab2f81szeCdf23mRVFzZ9hZquCz7HYzE6JmjfSzH5nZrVmVmdm93VY97+b2Qdmts7MpkdNv8HMaoLPv87MvnA8tUvvooCQ3uYfgHOBScDpwGTgu8G8bwGbgRJgCPD3gJvZycBtwNnu3g+4HFjfccXu/jrQAEyJmvx54PHg+d3A3e5eCIwDnjyBz/El4CvAMKAFuAfAzE4CngD+Jvgc84E/mllOEFx/AjYAY4BSYG7UOs8B1gDFwA+B/7KI/GD904PP/3Fg2QnULr2EAkJ6my8Ad7n7TnevBf4Z+GIwr5nID+5od2929xc9cjGyViAXGG9m2e6+3t3XdrL+J4DrAMysH3BFMO3w+svNrNjd6939taPUOTzoAUQ/8qPm/8LdV7p7A/C/gc8FAXAt8Gd3f9bdm4F/B/oQ+VGfDAwH/tbdG9z9kLtHD0xvcPefunsr8GiwLYYE89qAiWbWx923ufuqo9QuaUIBIb3NcCJ/QR+2IZgG8COgGvhLsDvlTgB3rybyF/n3gJ1mNtfMhhPb48Bngt1WnwHedPfD73cjcBLwrpktMbNPHaXOre4+oMOjIWr+pg6fIZvIX/7tPp+7twVtS4GRREKgpZP33B613IHgaUHwvtcCNwPbzOzPZvaxo9QuaUIBIb3NVmB01OtRwTTcfb+7f8vdy4AZwB2Hxxrc/XF3vyBY1oEfxFq5u68m8gM9nfa7l3D39939OmBwsPxTHXoFXTGyw2doBnZ1/HxmZkHbLUSCYpSZdfngE3d/xt0vJdKreBf46XHWLb2IAkJ6smwzy4t6ZBHZ3fNdMysxs2LgH4FfApjZp8ysPPhR3Utk11KbmZ1sZlOCXsEh4CCRXS6deRz4JnAR8JvDE83sejMrCf6q3xNMPtp6juZ6MxtvZn2Bu4Cngl1DTwJXmtlUM8smMq7SCLwCvAFsA75vZvnBNjn/WG9kZkPMbGYQZo1A/QnULb2IAkJ6svlEfswPP74H/CtQBSwHVgBvBtMAKoD/IfID+Cpwv7svIjL+8H0if6FvJ9ID+M5R3vcJ4BPAQnffFTV9GrDKzOqJDFjPcveDnaxjeIzzIK6Omv8L4OdBPXnAbAB3XwNcD9wb1Ptp4NPu3hQEyKeBcmAjkQH5a4/yOQ7LAO4g0jvZHXy2W+JYTno50w2DRFKLmS0GfunuD4ddi6Q39SBERCQmBYSIiMSkXUwiIhKTehAiIhJTr7lYX3FxsY8ZMybsMkREepSlS5fucveSWPN6TUCMGTOGqqqqsMsQEelRzGxDZ/O0i0lERGJSQIiISEwKCBERiUkBISIiMSkgREQkJgWEiIjEpIAQEZGY0j4g9h5s5j/+soa1tfVhlyIiklLSPiCaW9v46Ys1zFlUHXYpIiIpJe0Dorggl+vPGc1/L9vK+l0Nx15ARCRNpH1AANx0URmZGcb9i9WLEBE5TAEBDC7M4/OTR/G7N7ewafeBsMsREUkJCojA1z9RRoYZ9y9eG3YpIiIpQQERGNa/D587ewRPLd3Elj2d3WdeRCR9KCCi3HJxOQAPqBchIqKAiFY6oA/XnDWCXy/ZxPa9h8IuR0QkVAqIDr5xcTmt7jz4gnoRIpLeFBAdjCzqy2fOKOXx1zeyc796ESKSvhQQMdx6SXnkDOsXasIuRUQkNAkNCDObZmZrzKzazO48SrurzczNrDJq2neC5daY2eWJrLOjMcX5zJxUyi9f28iu+sZkvrWISMpIWECYWSYwB5gOjAeuM7PxMdr1A74JvB41bTwwC5gATAPuD9aXNLdeUs6hllYefnFdMt9WRCRlJLIHMRmodvcad28C5gIzY7T7F+AHQPQO/5nAXHdvdPd1QHWwvqQpH1zAp04bzmOvrmd3Q1My31pEJCUkMiBKgU1RrzcH044wszOBke7+564umwy3TynnQFMrj7ykXoSIpJ/QBqnNLAP4MfCtE1jHTWZWZWZVtbW13Vdc4KQh/bji1KE8+sp69h5o7vb1i4ikskQGxBZgZNTrEcG0w/oBE4HFZrYeOBeYFwxUH2tZANz9IXevdPfKkpKSbi4/4rZLKtjf2MLPXlEvQkTSSyIDYglQYWZjzSyHyKDzvMMz3X2vuxe7+xh3HwO8Bsxw96qg3SwzyzWzsUAF8EYCa+3U+OGFXDZ+CI+8tI59h9SLEJH0kbCAcPcW4DbgGeAd4El3X2Vmd5nZjGMsuwp4ElgNPA3c6u6tiar1WG6fUsG+Qy089sr6sEoQEUk6c/ewa+gWlZWVXlVVlbD1f+XnS3hz4we89O0pFORmJex9RESSycyWuntlrHk6kzpOt08pZ8+BZn7x6oawSxERSQoFRJzOGDWQi04q4acv1nCgqSXsckREEk4B0QXfnFrO7oYmfvXaxrBLERFJOAVEF5w1uojzywfx4As1HGoObcxcRCQpFBBdNHtKBbvqG3niDfUiRKR3U0B00TllgzhnbBEPPL9WvQgR6dUUEMdh9tQKduxr5DdVm47dWESkh1JAHIePjxvEWaMHcv/itTS2qBchIr2TAuI4mBmzp1awbe8hfrv0I5eIEhHpFRQQx+miimJOHzmAOYuqaW5tC7scEZFup4A4TmbGN6eWs2XPQX7/pnoRItL7KCBOwCUnD2ZiaSFzFlfTol6EiPQyCogTYGbMnlLBhroDzHt7a9jliIh0KwXECbp0/BBOGVbIfQuraW3rHVfGFREBBcQJMzNun1JOza4G/rRcvQgR6T0UEN1g2oShVAwu4N6F1bSpFyEivYQCohtkZBi3T62gemc9C1ZuD7scEZFuoYDoJleeOoyyknzuXfi+ehEi0iskNCDMbJqZrTGzajO7M8b8m81shZktM7OXzGx8MH2MmR0Mpi8zswcSWWd3yMyIjEW8u30/f1m9I+xyREROWMICwswygTnAdGA8cN3hAIjyuLuf6u6TgB8CP46at9bdJwWPmxNVZ3f69GnDGTOoL/cufJ/ecq9vEUlfiexBTAaq3b3G3ZuAucDM6Abuvi/qZT7Qo39VszIzuPWSclZt3cfCd3eGXY6IyAlJZECUAtHXw94cTGvHzG41s7VEehCzo2aNNbO3zOx5M7sw1huY2U1mVmVmVbW1td1Z+3G76oxSRhb14Z7n1IsQkZ4t9EFqd5/j7uOAbwPfDSZvA0a5+xnAHcDjZlYYY9mH3L3S3StLSkqSV/RRZGdm8I2Ly3l7816efy81QktE5HgkMiC2ACOjXo8IpnVmLnAVgLs3untd8HwpsBY4KUF1drurzxzB8P553K1ehIj0YIkMiCVAhZmNNbMcYBYwL7qBmVVEvbwSeD+YXhIMcmNmZUAFUJPAWrtVTlYGt1xSzlsb9/BydV3Y5YiIHJeEBYS7twC3Ac8A7wBPuvsqM7vLzGYEzW4zs1VmtozIrqQvB9MvApYH058Cbnb33YmqNRE+VzmCoYV53P3ce+pFiEiPZL3lx6uystKrqqrCLqOdn7+8ju/9cTVPfO1czhs3KOxyREQ+wsyWuntlrHmhD1L3ZrMmj6KkXy73Lnw/7FJERLpMAZFAedmZfP2iMl5ZW8eS9T1qD5mIiAIi0b5wzmiKC3K45zn1IkSkZ1FAJFifnEy+dmEZL76/izc3fhB2OSIicVNAJMH1545mYN9s7lUvQkR6EAVEEuTnZvHVC8tYtKaW5Zv3hF2OiEhcFBBJ8qXzRlOYl8U9z1WHXYqISFwUEEnSLy+bGy8o43/e2cGqrXvDLkdE5JgUEEl0w/lj6JebxX0L1YsQkdSngEii/n2y+evzx7Bg5XbWbN8fdjkiIkelgEiyr1wwlvycTJ1dLSIpTwGRZAP65vDlj4/hzyu2Ub1TvQgRSV0KiBDceMFY8rIyNRYhIilNARGCQQW5fPG80cx7eys1tfVhlyMiEpMCIiRfu7CM7MwM5ixaG3YpIiIxKSBCUtIvly+cM5o/LNvCxroDYZcjIvIRCogQff0TZWRmGPcv1liEiKQeBUSIhhTmcd3ZI3lq6WY2f6BehIikloQGhJlNM7M1ZlZtZnfGmH+zma0ws2Vm9pKZjY+a951guTVmdnki6wzTzRePI8OM/1yssQgRSS0JCwgzywTmANOB8cB10QEQeNzdT3X3ScAPgR8Hy44HZgETgGnA/cH6ep1h/fvw2coRPFm1ia17DoZdjojIEYnsQUwGqt29xt2bgLnAzOgG7r4v6mU+4MHzmcBcd29093VAdbC+XumWi8fhDg8+r16EiKSORAZEKbAp6vXmYFo7Znarma0l0oOY3cVlbzKzKjOrqq2t7bbCk23EwL5cfeYInliyiR37DoVdjogIkAKD1O4+x93HAd8GvtvFZR9y90p3rywpKUlMgUnyjUvG0drmPPh8TdiliIgAiQ2ILcDIqNcjgmmdmQtcdZzL9nijB+Vz1aRSHn9jA7X7G8MuR0QkoQGxBKgws7FmlkNk0HledAMzq4h6eSVw+BKn84BZZpZrZmOBCuCNBNaaEm69ZBxNLW08/KJ6ESISvoQFhLu3ALcBzwDvAE+6+yozu8vMZgTNbjOzVWa2DLgD+HKw7CrgSWA18DRwq7u3JqrWVFFWUsCM04fz2KsbqKtXL0JEwmXufuxWPUBlZaVXVVWFXcYJq965n0v/3wvc8olx/N20j4Vdjoj0cma21N0rY80LfZBa2isf3I8rTx3Go6+sZ8+BprDLEZE0poBIQbdNKaehqZVHXloXdikiksYUECnoY0MLmTZhKD97eT17DzaHXY6IpCkFRIq6fWo5+xtb+PnL68MuRUTSlAIiRU0Y3p9PnjKER15ex/5D6kWISPIpIFLY7Knl7D3YzGOvbgi7FBFJQwqIFHbaiAFccnIJD79YQ0NjS9jliEiaUUCkuNunVvDBgWZ++Zp6ESKSXAqIFHfmqIFcWFHMQy/UcLCp159MLiIpRAHRA8yeWkFdQxO/el29CBFJHgVED3D2mCLOKxvEgy/UcKhZvQgRSQ4FRA8xe2oFtfsbmfvGxrBLEZE0oYDoIc4tK2LymCIeeL6Gxhb1IkQk8RQQPYSZMXtqBdv3HeI3VZvDLkdE0oACogc5v3wQZ44awH8uXktTS1vY5YhIL6eA6EEO9yK27DnI795UL0JEEksB0cN84qQSTh/RnzmLq2luVS9CRBInroAws3wzywien2RmM8wsO47lppnZGjOrNrM7Y8y/w8xWm9lyM3vOzEZHzWs1s2XBY17HZdOVmXH7lAo27T7IH97aEnY5ItKLxduDeAHIM7NS4C/AF4GfH20BM8sE5gDTgfHAdWY2vkOzt4BKdz8NeAr4YdS8g+4+KXjMQI6Yespgxg8rZM6ialrUixCRBIk3IMzdDwCfAe53988CE46xzGSg2t1r3L0JmAvMjG7g7ouC9QK8BoyIv/T0dXgsYn3dAf64fGvY5YhILxV3QJjZecAXgD8H0zKPsUwpsCnq9eZgWmduBBZEvc4zsyoze83MruqkqJuCNlW1tbXHKKd3uWz8ED42tB/3Laymtc3DLkdEeqF4A+JvgO8Av3f3VWZWBizqriLM7HqgEvhR1OTR7l4JfB74iZmN67icuz/k7pXuXllSUtJd5fQIGRmRsYi1tQ3MX7Et7HJEpBeKKyDc/Xl3n+HuPwgGq3e5++xjLLYFGBn1ekQwrR0z+yTwD8AMd2+Mes8twb81wGLgjHhqTSfTJw6lYnAB9y58nzb1IkSkm8V7FNPjZlZoZvnASmC1mf3tMRZbAlSY2VgzywFmAe2ORjKzM4AHiYTDzqjpA80sN3heDJwPrI73Q6WLjAzjtinlvLejnmdWbQ+7HBHpZeLdxTTe3fcBVxEZJxhL5EimTrl7C3Ab8AzwDvBksHvqLjM7fFTSj4AC4DcdDmc9Bagys7eJ7Mr6vrsrIGL41GnDKSvO5+7n1IsQke6VFWe77OC8h6uA+9y92cyO+Wvk7vOB+R2m/WPU8092stwrwKlx1pbWMjOMWy8p51u/eZv/eWcHl00YGnZJItJLxNuDeBBYD+QDLwQntO1LVFHSNTMnDWdUUV/uWfg+7upFiEj3iHeQ+h53L3X3KzxiA3BJgmuTOGVlZnDbJeWs3LKPRWt2HnsBEZE4xDtI3d/Mfnz4nAMz+w8ivQlJEX91ZimlA/pwz3PV6kWISLeIdxfTI8B+4HPBYx/ws0QVJV2XnZnBrZeUs2zTHl58f1fY5YhILxBvQIxz938KLptR4+7/DJQlsjDpuqvPKmV4/zzufk5jESJy4uINiINmdsHhF2Z2PnAwMSXJ8crNyuSWi8exdMMHvLq2LuxyRKSHizcgbgbmmNl6M1sP3Ad8PWFVyXH7bOVIhhTmcvdz74ddioj0cPEexfS2u58OnAac5u5nAFMSWpkcl7zsTL5+0TheX7eb12rUixCR49elO8q5+77gjGqAOxJQj3SD6yaPorggl3sXqhchIsfvRG45at1WhXSrPjmZfP2iMl6urmPpht1hlyMiPdSJBIQOk0lhXzh3FEX5OdzzXHXYpYhID3XUgDCz/Wa2L8ZjPzA8STXKceibk8XXLizj+fdqWbZpT9jliEgPdNSAcPd+7l4Y49HP3eO90J+E5IvnjWZA32zu1RFNInIcTmQXk6S4gtwsvnrBWJ57dycrt+wNuxwR6WEUEL3clz4+hsK8LO5RL0JEukgB0csV5mXz1+eP5S+rd7B6q67QLiLxU0Ckga+cP5aC3CzuW6RehIjETwGRBvr3zeaGj49h/ortvLdjf9jliEgPkdCAMLNpZrbGzKrN7M4Y8+8ws9VmttzMngvuVHd43pfN7P3g8eVE1pkObrxgLH1zMrlvoc6LEJH4JCwgzCwTmANMB8YD15nZ+A7N3gIq3f004Cngh8GyRcA/AecAk4F/MrOBiao1HQzMz+FL543hj8u3Ur2zPuxyRKQHSGQPYjJQHdw/ogmYC8yMbuDui9z9QPDyNWBE8Pxy4Fl33+3uHwDPAtMSWGta+OqFY8nLyuT+RepFiMixJTIgSoFNUa83B9M6cyOwoCvLmtlNh2+DWltbe4Ll9n7FBblcf+4o/rBsC+t3NYRdjoikuJQYpDaz64FK4EddWc7dH3L3SnevLCkpSUxxvczXLiojOzODOepFiMgxJDIgtgAjo16PCKa1Y2afBP4BmOHujV1ZVrpucL88Pn/OKH731hY27T5w7AVEJG0lMiCWABVmNtbMcoBZwLzoBmZ2BvAgkXDYGTXrGeAyMxsYDE5fFkyTbvD1i8aRacb9i9WLEJHOJSwg3L0FuI3ID/s7wJPuvsrM7jKzGUGzHwEFwG/MbJmZzQuW3Q38C5GQWQLcFUyTbjC0fx7Xnj2Sp5ZuZsse3VpcRGIz995xW4fKykqvqqoKu4weY8ueg1z8o0XMOnsU/3LVxLDLEZGQmNlSd6+MNS8lBqkl+UoH9OGas0by6yWb2L73UNjliEgKUkCksW9cPI42dx54fm3YpYhIClJApLGRRX35zJmlPPHGRnbuUy9CRNpTQKS5Wy8pp6XNeeiFmrBLEZEUo4BIc6MH5TNz0nB++foGdtU3HnsBEUkbCgjh1kvKaWxp46cvqhchIh9SQAjjSgr49GnD+cWrG9jd0BR2OSKSIhQQAsBtU8o52NzKIy+tC7sUEUkRCggB4KQh/bhi4jB+/sp69h5oDrscEUkBCgg54rYp5dQ3tvDIy+pFiIgCQqKcMqyQyycM4ZGX17HvkHoRIulOASHt3D6lgv2HWnj05fVhlyIiIVNASDsTS/vzyVMG8/BL66hvbAm7HBEJkQJCPuL2KRXsPdjMY6+uD7sUEQmRAkI+4vSRA/jESSU8/OI6DjSpFyGSrhQQEtPsqRXsbmjiV69tDLsUEQmJAkJiOmv0QC4oL+bBF2o42NQadjkiEoKEBoSZTTOzNWZWbWZ3xph/kZm9aWYtZnZNh3mtwW1Ij9yKVJJr9tQKdtU38sQb6kWIpKOEBYSZZQJzgOnAeOA6MxvfodlG4Abg8RirOOjuk4LHjBjzJcEmjy3i3LIiHnh+LYea1YsQSTeJ7EFMBqrdvcbdm4C5wMzoBu6+3t2XA20JrENOwOypFezc38iTVZvCLkVEkiyRAVEKRP+qbA6mxSvPzKrM7DUzuypWAzO7KWhTVVtbeyK1SifOKxvE2WMG8p+L19LYol6ESDpJ5UHq0e5eCXwe+ImZjevYwN0fcvdKd68sKSlJfoVpwMy4fUoF2/Ye4qmlm8MuR0SSKJEBsQUYGfV6RDAtLu6+Jfi3BlgMnNGdxUn8LqwoZtLIAdy/aC3NrdobKJIuEhkQS4AKMxtrZjnALCCuo5HMbKCZ5QbPi4HzgdUJq1SOysz45tQKtuw5yO/fjDvjRaSHS1hAuHsLcBvwDPAO8KS7rzKzu8xsBoCZnW1mm4HPAg+a2apg8VOAKjN7G1gEfN/dFRAhuvjkEk4t7c99i6ppUS9CJC2Yu4ddQ7eorKz0qqqqsMvo1Z5dvYOvPVbFf3z2dK4+a0TY5YhINzCzpcF470ek8iC1pJhPnjKYU4YVct+ialrbescfFiLSOQWExC0yFlHOul0N3Pnb5VSt302bgkKk18oKuwDpWS4bP5RrzhrBfy/bym+WbmZIYS7TJgxl+qnDOHtMEZkZFnaJItJNNAYhx2X/oWYWvruT+Su2sXhNLY0tbRQX5HDZhKFcMXEY55QVkZ2pDqpIqjvaGIQCQk5YQ2MLi9fUMn/lNha9u5MDTa0M6JvNZeOHMP3UYZw/rpicLIWFSCpSQEjSHGpu5fn3almwYhvPvbOT/Y0t9MvL4tJTImFxYUUxedmZYZcpIoGjBYTGIKRb5WVncvmEoVw+YSiNLa28XL2L+Su28+zqHfzurS3k52Qy5ZQhXDFxKBefPJg+OQoLkVSlgJCEyc3KZMrHhjDlY0Nobm3j1bV1LFi5jWdW7eCPb2+lT3YmF59cwvRThzHlY4MpyNX/jiKpRLuYJOlaWtt4Y/1uFqzYztOrtlO7v5GcrAwuqijhilOHMvWUIfTvkx12mSJpQWMQkrJa25w3N37A/BXbeHrldrbtPUR2pnFBeTHTJw7j0vFDGJifE3aZIr2WAkJ6hLY2Z9nmPTy9cjvzV2xj8wcHycwwPj5uENMmRsY1igtywy5TpFdRQEiP4+6s2rqP+Su2sWDldtbtaiDDIrdBnT5xGNMmDmVIYV7YZYr0eAoI6dHcnTU79jN/xXYWrNjG+zvrAagcPZBpEyNncZcO6BNylSI9kwJCepXqnftZsGI781du551t+wA4feQApk8cyvSJQxk9KD/kCkV6DgWE9FrrdzWwYOV2FqzcxvLNewGYMLwwEhanDmNcSUHIFYqkNgWEpIVNuw/wzKrIAPebG/cAcPKQfkybOJQrTh3GSUMKMNPFBEWiKSAk7Wzbe5CnV25nwcrtLFm/G3coK8nnimCAe8LwQoWFCAoISXM79x/imVU7eHrlNl5dW0ebw6iivkw/dSjTJw7j9BH9FRaStkILCDObBtwNZAIPu/v3O8y/CPgJcBowy92fipr3ZeC7wct/dfdHj/ZeCgiJR119I8+u3sGCldt5uXoXLW1O6YA+kaOhJg7lzFEDydA9LSSNhBIQZpYJvAdcCmwGlgDXufvqqDZjgELgfwHzDgeEmRUBVUAl4MBS4Cx3/6Cz91NASFftPdDMs+9EehYvvLeLptY23QBJ0k5YV3OdDFS7e01QxFxgJnAkINx9fTCvrcOylwPPuvvuYP6zwDTgiQTWK2mmf99srjlrBNecNaLdDZDmLtnEo69u0A2QJO0lMiBKgU1RrzcD55zAsqUdG5nZTcBNAKNGjTq+KkWAfnnZzJxUysxJpe1ugPSHt7bw+OsbdQMkSUs9+vrK7v4Q8BBEdjGFXI70Evm5WVx52jCuPG1YuxsgLVixnSerNusGSJI2EhkQW4CRUa9HBNPiXfbiDssu7paqRLpAN0CSdJbIgFgCVJjZWCI/+LOAz8e57DPA/zGzgcHry4DvdH+JIvHTDZAk3ST6MNcriBzGmgk84u7/ZmZ3AVXuPs/MzgZ+DwwEDgHb3X1CsOxXgL8PVvVv7v6zo72XjmKSsOgGSNKT6UQ5kSTp7AZIpwwrpKw4n7KSAspK8ikrLmBscb52SUnoFBAiIWhrc97evIenV21n9dZ91NQ2sGXPwXZtSgf0CQIjKjxKChhWmKcT9iQpwjoPQiStZWQYZ4wayBmjBh6ZdrCplXW7GqjZVU9NbQM1tfXU7Grgt29uob6x5Ui7vOwMxhZHAmNcVHiMLc6nX552V0lyKCBEkqhPTibjhxcyfnhhu+nuTu3+RtbWtg+PlVv2smDFNtqiOvqD++Ue6WmUFeczLgiPEQP76sxv6VYKCJEUYGYMLsxjcGEe540b1G5eY0srG+sOfCQ85q/Yxp4DzUfa5WRmMHpQ33bhUVZSwLiSfAb0zUn2R5JeQAEhkuJyszKpGNKPiiH9PjJvd0NTZDdVbQNrg/Co3lnPwnd30tz6YbejKD8nCIz24TGqqK/OCpdOKSBEerCi/ByK8ouoHFPUbnpLaxubPjh4JDxqdtWztoNoTogAAAp/SURBVLaBhe/W8mTV5iPtMjOMUUV9Y4ZHcUGOLoOe5hQQIr1QVmYGY4sjg9pTT2k/b+/B5shAeVR41NQ28FL1LhpbPrxuZr+8rMguqujwKMlnzKB8XV4kTSggRNJM/z7ZTBo5gEkjB7Sb3tbmbNlzkJoO4fFqTR2/e+vDq+SYHT489/Ag+YfhMbQwT72OXkQBISJA5LDckUV9GVnUl0+cVNJuXkNjS3B4bvvwqFq/mwNNrUfa9c3JZGxx9K6qyFFWY4vzydelR3oc/RcTkWPKz81iYml/Jpb2bzfd3dmxr5Ga2nrWRoXHsk0f8KflW4k+D3doYV6wqypyJvnh8Bg+oI8Oz01RCggROW5mxtD+eQztn8fHy4vbzTvU3MqGugNHTgZcG4THvGVb2Xfow5MCc7IyGDso/8iJgGUlBYwY2IdB+TkMKshlQJ9snVUeEgWEiCREXnYmJw/tx8lD2x+e6+7UNTS1O5O8praeNdv385fVO2hta3/5nwyDovzcIDAioTEoP+dIgAwqaP+8X26WxkG6iQJCRJLKzCguyKW4IJfJY9sfntvc2sbG3QfYvvcQdQ1N1NU3UlffRF3D4X+bWLF5D3UNTeyP6oVEy8nMoCieMMnPobggVxdMPAoFhIikjOzMDMaVFDCupOCYbRtbWtnd0HQkOD4Mk+B58G9NbT276hs51NwWcz19sjNjhklxQU4QNB+GSVF+TlqdWKiAEJEeKTcrk2H9+zCsf5+42h9oajl6mDQ0sWPfIVZv3UddQ2O7M9Gj9cvLOhIWscKkOPi3KD+HgX2zycrsuYGigBCRtNA3J4u+RVmMLOp7zLbuzv7GIFCO9Ebah0ldfSMb6g7w5sY97G5opC1GnpjBwL45R8KkONjNFStMigtyKMxLrQF5BYSISAdmRmFeNoV52Ywtzj9m+7Y2Z8/B5vZh0tBh/KS+iXe376OuoandRRajZWUYA2OESfHh3V9RYTKoIJf8nMyEDsgnNCDMbBpwN5Fbjj7s7t/vMD8XeAw4C6gDrnX39WY2BngHWBM0fc3db05krSIixysjw4LrYuVQEUf75tY2PjjQdCQ4PhImQQ9l06YD1NU3tbtXSLScrAyK83M4a0wR9153Rvd+KBIYEGaWCcwBLgU2A0vMbJ67r45qdiPwgbuXm9ks4AfAtcG8te4+KVH1iYiEJTszg8H98hjcLy+u9oeaowfkPxomg/vlJqTORPYgJgPV7l4DYGZzgZlAdEDMBL4XPH8KuM90ALOISDt52ZkMH9CH4QPiG5DvLokcXi8FNkW93hxMi9nG3VuAvcDhu6WMNbO3zOx5M7swgXWKiEgMqTpIvQ0Y5e51ZnYW8Aczm+Du+6IbmdlNwE0Ao0aNCqFMEZHeK5E9iC3AyKjXI4JpMduYWRbQH6hz90Z3rwNw96XAWuCkjm/g7g+5e6W7V5aUlHScLSIiJyCRAbEEqDCzsWaWA8wC5nVoMw/4cvD8GmChu7uZlQSD3JhZGVAB1CSwVhER6SBhu5jcvcXMbgOeIXKY6yPuvsrM7gKq3H0e8F/AL8ysGthNJEQALgLuMrNmoA242d13J6pWERH5KHOPfTp5T1NZWelVVVVhlyEi0qOY2VJ3r4w1r+deJERERBJKASEiIjH1ml1MZlYLbDiBVRQDu7qpnO6kurpGdXWN6uqa3ljXaHePeRhorwmIE2VmVZ3thwuT6uoa1dU1qqtr0q0u7WISEZGYFBAiIhKTAuJDD4VdQCdUV9eorq5RXV2TVnVpDEJERGJSD0JERGJSQIiISExpFRBmNs3M1phZtZndGWN+rpn9Opj/enDr01So6wYzqzWzZcHjq0mq6xEz22lmKzuZb2Z2T1D3cjM7M0XqutjM9kZtr39MUl0jzWyRma02s1Vm9s0YbZK+zeKsK+nbzMzyzOwNM3s7qOufY7RJ+ncyzrpC+U4G750Z3CvnTzHmde/2cve0eBC5YOBaoAzIAd4Gxndo8w3ggeD5LODXKVLXDcB9IWyzi4AzgZWdzL8CWAAYcC7weorUdTHwpxC21zDgzOB5P+C9GP8tk77N4qwr6dss2AYFwfNs4HXg3A5twvhOxlNXKN/J4L3vAB6P9d+ru7dXOvUgjtwC1d2bgMO3QI02E3g0eP4UMDUJt0CNp65QuPsLRK6y25mZwGMe8RowwMyGpUBdoXD3be7+ZvB8P/AOH72LYtK3WZx1JV2wDeqDl9nBo+NRM0n/TsZZVyjMbARwJfBwJ026dXulU0Cc6C1Qw6wL4Opgl8RTZjYyxvwwxFt7GM4LdhEsMLMJyX7zoGt/BpG/PqOFus2OUheEsM2C3SXLgJ3As+7e6fZK4ncynrognO/kT4C/I3IbhFi6dXulU0D0ZH8Exrj7acCzfPgXgsT2JpHry5wO3Av8IZlvbmYFwG+Bv/EOt8kN0zHqCmWbuXuru08icsfJyWY2MRnveyxx1JX076SZfQrY6ZG7bCZFOgXEcd8CNey63L3O3RuDlw8DZyW4pnjFs02Tzt33Hd5F4O7zgWwzK07Ge5tZNpEf4V+5++9iNAllmx2rrjC3WfCee4BFwLQOs8L4Th6zrpC+k+cDM8xsPZFd0VPM7Jcd2nTr9kqngDjuW6CGXVeHfdQziOxDTgXzgC8FR+acC+x1921hF2VmQw/vdzWzyUT+P0/4j0rwnv8FvOPuP+6kWdK3WTx1hbHNLHJr4QHB8z7ApcC7HZol/TsZT11hfCfd/TvuPsLdxxD5nVjo7td3aNat2ythtxxNNX5it0ANu67ZZjYDaAnquiHRdQGY2RNEjm4pNrPNwD8RGbDD3R8A5hM5KqcaOAD8dYrUdQ1wi5m1AAeBWUkIeoj8hfdFYEWw/xrg74FRUbWFsc3iqSuMbTYMeNQi95/PAJ509z+F/Z2Ms65QvpOxJHJ76VIbIiISUzrtYhIRkS5QQIiISEwKCBERiUkBISIiMSkgREQkJgWEyDGYWWvUVTuXWYwr7p7AusdYJ1elFQlb2pwHIXICDgaXXRBJK+pBiBwnM1tvZj80sxXB/QPKg+ljzGxhcCG358xsVDB9iJn9Prgg3ttm9vFgVZlm9lOL3HvgL8HZu5jZbIvcw2G5mc0N6WNKGlNAiBxbnw67mK6NmrfX3U8F7iNypU2IXOzu0eBCbr8C7gmm3wM8H1wQ70xgVTC9Apjj7hOAPcDVwfQ7gTOC9dycqA8n0hmdSS1yDGZW7+4FMaavB6a4e01wMbzt7j7IzHYBw9y9OZi+zd2LzawWGBF1kbfDl99+1t0rgtffBrLd/V/N7GmgnsiVVf8QdY8CkaRQD0LkxHgnz7uiMep5Kx+ODV4JzCHS21gSXJ1TJGkUECIn5tqof18Nnr/ChxdJ+wLwYvD8OeAWOHJDmv6drdTMMoCR7r4I+DaRyzZ/pBcjkkj6i0Tk2PpEXQUV4Gl3P3yo60AzW06kF3BdMO124Gdm9rdALR9esfWbwENmdiORnsItQGeX+s4EfhmEiAH3BPcmEEkajUGIHKdgDKLS3XeFXYtIImgXk4iIxKQehIiIxKQehIiIxKSAEBGRmBQQIiISkwJCRERiUkCIiEhM/x+WRVMY1G2/FgAAAABJRU5ErkJggg==\n", 511 | "text/plain": [ 512 | "
" 513 | ] 514 | }, 515 | "metadata": { 516 | "tags": [], 517 | "needs_background": "light" 518 | } 519 | } 520 | ] 521 | }, 522 | { 523 | "cell_type": "markdown", 524 | "metadata": { 525 | "id": "ziq-tFlU-gDD" 526 | }, 527 | "source": [ 528 | "#### Evaluate the model\n", 529 | "\n", 530 | "Finally, you should evaluate the performance of your model on the test set, by calling the model's `evaluate` method." 531 | ] 532 | }, 533 | { 534 | "cell_type": "code", 535 | "metadata": { 536 | "id": "CSqA8zUi-gDE" 537 | }, 538 | "source": [ 539 | "#### GRADED CELL ####\n", 540 | "\n", 541 | "# Complete the following function. \n", 542 | "# Make sure to not change the function name or arguments.\n", 543 | "\n", 544 | "def evaluate_model(model, scaled_test_images, test_labels):\n", 545 | " \"\"\"\n", 546 | " This function should evaluate the model on the scaled_test_images and test_labels. \n", 547 | " Your function should return a tuple (test_loss, test_accuracy).\n", 548 | " \"\"\"\n", 549 | " test_loss, test_acc = model.evaluate(scaled_test_images, test_labels)\n", 550 | " return (test_loss, test_acc)\n", 551 | " " 552 | ], 553 | "execution_count": 16, 554 | "outputs": [] 555 | }, 556 | { 557 | "cell_type": "code", 558 | "metadata": { 559 | "id": "SSNhInQD-gDG", 560 | "outputId": "f8892701-f150-4c83-f0db-c85ee95766b3", 561 | "colab": { 562 | "base_uri": "https://localhost:8080/", 563 | "height": 72 564 | } 565 | }, 566 | "source": [ 567 | "# Run your function to evaluate the model\n", 568 | "\n", 569 | "test_loss, test_accuracy = evaluate_model(model, scaled_test_images, test_labels)\n", 570 | "print(f\"Test loss: {test_loss}\")\n", 571 | "print(f\"Test accuracy: {test_accuracy}\")" 572 | ], 573 | "execution_count": 17, 574 | "outputs": [ 575 | { 576 | "output_type": "stream", 577 | "text": [ 578 | "313/313 [==============================] - 2s 7ms/step - loss: 0.0589 - sparse_categorical_accuracy: 0.9812\n", 579 | "Test loss: 0.05894160270690918\n", 580 | "Test accuracy: 0.9811999797821045\n" 581 | ], 582 | "name": "stdout" 583 | } 584 | ] 585 | }, 586 | { 587 | "cell_type": "markdown", 588 | "metadata": { 589 | "id": "SP09yVMK-gDK" 590 | }, 591 | "source": [ 592 | "#### Model predictions\n", 593 | "\n", 594 | "Let's see some model predictions! We will randomly select four images from the test data, and display the image and label for each. \n", 595 | "\n", 596 | "For each test image, model's prediction (the label with maximum probability) is shown, together with a plot showing the model's categorical distribution." 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "metadata": { 602 | "id": "ZrUM42t_-gDL", 603 | "outputId": "114ba37f-450c-4c88-af86-312cb0af6873", 604 | "colab": { 605 | "base_uri": "https://localhost:8080/", 606 | "height": 716 607 | } 608 | }, 609 | "source": [ 610 | "# Run this cell to get model predictions on randomly selected test images\n", 611 | "\n", 612 | "num_test_images = scaled_test_images.shape[0]\n", 613 | "\n", 614 | "random_inx = np.random.choice(num_test_images, 4)\n", 615 | "random_test_images = scaled_test_images[random_inx, ...]\n", 616 | "random_test_labels = test_labels[random_inx, ...]\n", 617 | "\n", 618 | "predictions = model.predict(random_test_images)\n", 619 | "\n", 620 | "fig, axes = plt.subplots(4, 2, figsize=(16, 12))\n", 621 | "fig.subplots_adjust(hspace=0.4, wspace=-0.2)\n", 622 | "\n", 623 | "for i, (prediction, image, label) in enumerate(zip(predictions, random_test_images, random_test_labels)):\n", 624 | " axes[i, 0].imshow(np.squeeze(image))\n", 625 | " axes[i, 0].get_xaxis().set_visible(False)\n", 626 | " axes[i, 0].get_yaxis().set_visible(False)\n", 627 | " axes[i, 0].text(10., -1.5, f'Digit {label}')\n", 628 | " axes[i, 1].bar(np.arange(len(prediction)), prediction)\n", 629 | " axes[i, 1].set_xticks(np.arange(len(prediction)))\n", 630 | " axes[i, 1].set_title(f\"Categorical distribution. Model prediction: {np.argmax(prediction)}\")\n", 631 | " \n", 632 | "plt.show()" 633 | ], 634 | "execution_count": 18, 635 | "outputs": [ 636 | { 637 | "output_type": "display_data", 638 | "data": { 639 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtUAAAK7CAYAAAA9TXNBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZxkVX3//9ebYRk2QZlxgRkclNGIS1xGlLgRQQOK4DcxBsS4M/EXUBM1Bo1RRGMw32iIX9GIgODGIi7fUUcgRnEDgQFBBURHBGcAZVgFXIbl8/vj3vFbXd1dXTPV09U9/XrOox5T595T53zq1u3uT5177r2pKiRJkiRtuM2GHYAkSZI005lUS5IkSQMyqZYkSZIGZFItSZIkDcikWpIkSRqQSbUkSZI0IJNqSRJJ7kzysAHbODnJe/qsuyhJJdm8LX81ycsH6b+j7WckuaqjfE2SfSej7ba9y5PsPVntTaUkeydZ3Wfdo5J8amPHNE7fI+Lc0G3evS9IG5NJtSRNkiQvSbKiTVBvaBPFp/f52kqy+8aOcTxVtV1VXT3E/vevqlMmqtfPdqqqb1fVIycjrrG+KFTVo6vq3Mlof4K+K8mN6754tMu2aJfNqptM9LvNu/ePydwX+uj7+CRXJbkvySumok9NLybVkjQJkrwROBZ4L/AgYFfgw8BBw4xrIp0J26ZgU3s/wK3A/h3l/dtlM0qSOcOOYQpcBvwtcMmwA9FwmFRL0oCS7AAcDRxeVZ+vqruq6u6q+lJV/UNbZ88k5ye5rR3F/lCSLdt132qbuqwd5f6rdvkBSS5tX3Neksd19PnEJN9PckeSzyY5vXNENclhSVYmuSXJsiQ7d6yrJIcn+Snw045lu7fPt07y/iTXJrk9yXeSbN2u+2ySX7bLv5Xk0X1uozlJ/j3JTUmuBp7ftf7cJK9pn++e5JttHzclOX287bRumkCSf0zyS+Dj40xxeHKSK5LcmuTjSea2bb4iyXe6Yqk2hqXAocBb2v6+1K7/w3SSJFslOTbJ9e3j2CRbtevWxfamdnT5hiSv7Gd7dfgk8LKO8suAT3TFu3P7Gd/SfuaHdazbuh1tvzXJFcCTx3jt55KsSfLzJK/vJ6iO9/a29jO6JsmhHetPTvKRJMuT3AX8aa+++oizc5vPafv9Wbv/X5xkYa/9o6OdR7X72m1pppQc2BXzcUm+0rZ7QZKH97M9AKrquKr6H+B3/b5Gm5ZZlVQnuTfNH6jLk1zW/qLbrF23JMkH+2jjvPb/RUle0qPeWe0P7Zcn7x1Imqb2AuYCX+hR517g74F5bf19aEa1qKpntnX+uJ2GcXqSJwAnAX8D7AR8FFjWJnFbtn2dDDwAOBX4X+s6SvJs4F+BFwMPAa4FTuuK54XAU4A9xoj134EnAX/Stv8W4L523VeBxcADaUbkPt3jPXc6DDgAeAKwBHhRj7rvBs4B7g8sAP4PjL2d2vKD2zgfCiwdp81DgT8DHg48Anj7RAFX1fE07+/f2v5eMEa1fwKeCjwe+GNgz662HwzsAOwCvBo4Lsn9J+q7wxeBZybZsX3dM4D/21XnNGA1sDPNdn1vuw8AvJPmPT+c5v3/Yd56+/fvSzQjrLvQ7JN/l+TP+oztwTT78y5tu8cn6Zxq8RLgX4DtgfMm6GvcOMfwRuAQ4HnA/YBXAb/psX+se79btDGcQ7P/vg74dFfMBwPvotn3Vrbxr3v9l5McOcE20Sw2q5Jq4LdV9fiqejTwHJrDaO8EqKoVVTXhN/Sq+pP26SKaXxjj+d/AXw8WrqQZYifgpqq6Z7wKVXVxVX2vqu6pqmtokuRn9WhzKfDRqrqgqu5t5xv/niaBeyqwOfDBdkT888CFHa89FDipqi6pqt8DbwX2SrKoo86/VtUtVfXbzk7bROtVwBuq6rq27/Padqiqk6rqjrZ8FPDHaUbqJ/Ji4NiqWlVVt9Ak/eO5myZB3rmqfldV3+lRF5qE/51V9fvu99PhQx19/wtNUjYZDgWOrqobq2oNTULW+bv/7nb93VW1HLgTWJ85vr+jSQT/qn0so2MkNMlC4GnAP7bb6lLgBP7f6PaLgX9pP+tVQOfg0ZOB+VV1dFWtbefUf4wmsezXP7fb/ZvAV9r+1vm/VfXdqroPeOwEffWKs9trgLdX1VXVuKyqbu4j1qcC2wHHtDF8HfgyI/eFL1TVhe3P8qdpviwBUFUHVNUxffSjWWq2JdV/UFU30vzROiKNvdeNKieZn+S/2xHtE9IcAp3XrruzbeIY4BntyPffj9H+/wB3TNHbkTRcNwPz0mM+b5JHtCNdv0zya5q51/N6tPlQ4E3tEa/bktwGLKQZjdwZuK6qOk9WW9XxfGea0WkAqurONsZdxqnfaR7NqPvPxngPc5Ic0x52/zVwTcdrJrJzV5/XjleRZmQ8wIXt7+FXTdD2mqqa6JB7d987j1dxPY3Y1mO0fXPXl63f0CR26+MTNEnyqKkfbV+3VFXn35tr+X+fda/t/lBg56597G005wT049aququr7c733tnvRH2tz/6xkDH2zz7sDKxqk/zOfjp/Ln7Z8XxDPivNYrM2qQZovynPoTkM1OmdwNfbEe0zaU446nYk8O125Ps/Nm6kkqa582lGkV/Yo85HgB8Di6vqfjQJRXrUX0Uzcrdjx2ObqjoVuAHYJUnn6xd2PL+eJokBIMm2NKPp13XUGe/qETfRjISONZf0JTQnXu5LM6Vh0boueryPdW7oinGs36tNYFW/rKrDqmpnmukvH07vK370cyWM7r6vb5/fBWyzbkWSB69n2yO2dVfbk+XbNNN4HgR0j9pfDzwgyfZdMaz7rHtt91XAz7v2se2r6nl9xnX/dt/qbLvzvXd/6evVV9/7R9tW33OdO1wPLFw37bOjn+vGqS+tl1mdVPfwdNr5h1V1FjPwTGtJU6eqbgfeQTNf9oVJtklz6bP9k/xbW2174NfAnUn+CPj/upr5FdB5neiPAa9N8pT2aNq2SZ7fJk/n08zRPiLJ5kkOopnLu86pwCuTPD7NSXPvBS5op51M9F7uo5nL/YE0J5bNSbJX2872NF8ebqZJRN/b/1biDOD1SRa0c4PHnZua5C+TLGiLt9IkZ+tGF7u3U78Ob/t+AM086HXzbS8DHt1uq7k0U1o6TdTfqcDb2yOc82j2g0m9tnN7ROIFwIFdRydop0qcB/xrkrlpTmZ9dUcMZwBvTXL/dpu+ruPlFwJ3pDnJc+v2s35MkhEnCU7gXUm2TPIMmjnznx2n3kR99Yqz2wnAu5Msbn82Hpdkp3Zdr8/rAprR57e0P59702zX7vMNNki7HebSfMncov08zLNmkVn9Yae50cG9wI3DjkXSzFZV76c5gertwBqa0bQjaE40A3gzzUjvHTQJ8+ldTRwFnNIeGn9xVa2gObnvQzSJ5UrgFW1fa4E/p0mebgNeSjM3dN28568B/wx8jmYE8OGs3zzZNwM/BC4CbgHeR/P34hM0h8uvA64AvrcebX4MOJsmib0E+HyPuk8GLmin2y2jmd+97hraR9Gxndaj/8/QnKB2Nc3UgfcAVNVPaK7c8jWaK6F0jwSfCOzR9vdFRnsPsAL4Ac02u2Rd2xNJcwWLr/ZTt6our6rLx1l9CM1Rg+tpTmB9Z7sPQDPH+1rg5zTv/5Mdbd5Lkwg/vl1/E03C2s8ceWimStza9vtp4LVV9eNx4p+or3HjHMMHaJLwc2i+qJ4IbN2uO4px9o/25+YFNOdT3URzycuXjRdztzTXnX9bjyrnAL+lOcH3+Pb5M3vU1yYmXV96N2lJ7qyq7drn82l+CZxfVe9sv7G+uaoOSHIc8Iuqel+S59L8IZhfVTetayPJk4APVNW4Jxp1trmx35uk2S3JBcB/VdXHhx2LNn3t37dPVdWCiepKs8VsG6neuj2x8HKaUYlzaL4dd3sX8NwkPwL+kubbePdJhz8A7k1zab5RJyom+TbNYbB90lzLs99LFEnShJI8K8mD2+kfLwceB5w17Lgkabba1O481VNVjXtHp2puf3puW7wd+LOquifJXsCTOy4ntV37/93As8doal17z5iksCVpLI+kOQS+Lc2UhhdV1Q3DDUmSZq9ZNf2jX0kW0/yx2gxYC/xtVV003KgkSZI0XZlUS5IkSQNar+kfW2armsu2E1fUJuN33MXa+n0/16DVepg3b14tWrRo2GFIkqT1cPHFF99UVfPHWrdeSfVctuUp2WdyotKMcEH9z7BD2CQtWrSIFStWDDsMSZK0HpKMe7fP2Xb1D0mSJGnSmVRLPSQ5KcmN7eUVx1qfJB9MsjLJD5I8capjlCRJw2dSLfV2MrBfj/X7A4vbx1LgI1MQkyRJmmZMqqUequpbNLdpHs9BwCeq8T1gxyQPmZroJEnSdDGrbv4ibQS7AKs6yqvbZaNuwpFkKc1oNrvuuuuUBDeTLDryK8MOYZRrjnn+sEOQJM0QjlRLU6Sqjq+qJVW1ZP78Ma/GI0mSZiiTamkw1wELO8oL2mWSJGkWMamWBrMMeFl7FZCnArdX1aipH5IkadPmnGqphySnAnsD85KsBt4JbAFQVf8FLAeeB6wEfgO8cjiRSpKkYTKplnqoqkMmWF/A4VMUjiRJmqac/iFJkiQNyKRakiRJGpBJtSRJkjQgk2pJkiRpQCbVkiRJ0oBMqiVJkqQBeUm9jWjOTg8YtWzLz28xovzFxWePqvOE9/7tiPIDP3Te5AYmSZKkSeVItSRJkjQgk2pJkiRpQCbVkiRJ0oCcU70R3fHMxaOWfX33D48o73PF/xpV58HfumVE+b7JDUuSJEmTzJFqSZIkaUAm1ZIkSdKATKolSZKkAZlUS5IkSQPyRMVJNOeRu48oH37MGaPq/OTutSPKm+/7i1F1PDFxekmyH/CfwBzghKo6pmv9rsApwI5tnSOravmUBypJkobGkWqphyRzgOOA/YE9gEOS7NFV7e3AGVX1BOBg4MNIkqRZxaRa6m1PYGVVXV1Va4HTgIO66hRwv/b5DsD1UxifJEmaBkyqpd52AVZ1lFe3yzodBbw0yWpgOfC6sRpKsjTJiiQr1qxZszFilSRJQ+Kc6kl07Z8/cET5L7e7eVSd3c8emW89ghUbNSZNiUOAk6vq/Un2Aj6Z5DFVNWJ6fFUdDxwPsGTJkhpCnJIkaSNxpFrq7TpgYUd5Qbus06uBMwCq6nxgLjBvSqKTJEnTgkm11NtFwOIkuyXZkuZExGVddX4B7AOQ5FE0SbXzOyRJmkVMqqUequoe4AjgbOBKmqt8XJ7k6CQHttXeBByW5DLgVOAVVeX0DkmSZhHnVEsTaK85vbxr2Ts6nl8BPG2q45IkSdOHSfUGmjN//qhljz3gxyPKP1x796g6jzrm1hHleyc3LEmSJA2B0z8kSZKkAZlUS5IkSQMyqZYkSZIG5JzqDfST/+y+qR5cuejEEeX/9dMXjqpz709+ttFikiRJ0nA4Ui1JkiQNyKRakiRJGpBJtSRJkjQgk2pJkiRpQJ6o2Kc5j37kiPIPnnn8qDof//WiEeX7XjN3Y4YkSZKkacKRakmSJGlAJtWSJEnSgEyqJUmSpAE5p7pPV711mxHlLTJnVJ1jLtpvRHnxyks2akySJEmaHhypliRJkgZkUi1NIMl+Sa5KsjLJkePUeXGSK5JcnuQzUx2jJEkaLqd/SD0kmQMcBzwHWA1clGRZVV3RUWcx8FbgaVV1a5IHDidaSZI0LI5US73tCaysqqurai1wGnBQV53DgOOq6laAqrpximOUJElD5kj1GObMnz9q2due9NUR5R+svXdUnUWf8jvKJmgXYFVHeTXwlK46jwBI8l1gDnBUVZ3V3VCSpcBSgF133XWjBCtJkobDLFAa3ObAYmBv4BDgY0l27K5UVcdX1ZKqWjJ/jC9ukiRp5jKplnq7DljYUV7QLuu0GlhWVXdX1c+Bn9Ak2ZIkaZYwqZZ6uwhYnGS3JFsCBwPLuup8kWaUmiTzaKaDXD2VQUqSpOFyTvUYVr1i9CDjK+539ojy7mf/7ag6jzhnxUaLScNRVfckOQI4m2a+9ElVdXmSo4EVVbWsXffcJFcA9wL/UFU3Dy9qSZI01UyqpQlU1XJgedeyd3Q8L+CN7UOSJM1CTv+QJEmSBmRSLUmSJA3IpFqSJEkakHOqx3Dka06fsM42P91yCiKRJEnSTOBItSRJkjQgk2pJkiRpQCbVkiRJ0oCcUz2GJ8/9xRhLtx5R2uZXNTXBSJIkadpzpFqSJEkakEm1JEmSNCCTakmSJGlAJtWSJEnSgDxREfj5e/caUV4w58JRdZ71wxeNKD/g49/bqDFJkiRp5nCkWpIkSRqQSbUkSZI0IJNqaQJJ9ktyVZKVSY7sUe8vklSSJVMZnyRJGj7nVAN7PO3qEeUtMmdUnTuXP3hEedu6elQdbXqSzAGOA54DrAYuSrKsqq7oqrc98AbggqmPUpIkDZsj1VJvewIrq+rqqloLnAYcNEa9dwPvA343lcFJkqTpwaRa6m0XYFVHeXW77A+SPBFYWFVf6dVQkqVJViRZsWbNmsmPVJIkDY1JtTSAJJsBHwDeNFHdqjq+qpZU1ZL58+dv/OAkSdKUMamWersOWNhRXtAuW2d74DHAuUmuAZ4KLPNkRUmSZpdZd6LinDFGCF/4oO+PKD/t0oNH1XnICZeOKN83uWFp+roIWJxkN5pk+mDgJetWVtXtwLx15STnAm+uqhVTHKckSRoiR6qlHqrqHuAI4GzgSuCMqro8ydFJDhxudJIkabqYdSPV0vqqquXA8q5l7xin7t5TEZMkSZpeHKmWJEmSBjTrRqpXvXLxqGWHbn/WiPIxF88bVef+v/npRotJkiRJM5sj1ZIkSdKATKolSZKkAZlUS5IkSQMyqZYkSZIGNOtOVLxrl9G3bdmMjChXRlWRJEmSxuVItSRJkjQgk2pJkiRpQCbVkiRJ0oBm3ZzqHRbdNmGd1BQEIkmSpE2GI9WSJEnSgEyqJUmSpAGZVEuSJEkDMqmWJEmSBjTrTlTcZqu1E9Z52nN/OGrZuds/dUR59zddNPqF9927wXFp+kqyH/CfwBzghKo6pmv9G4HXAPcAa4BXVdW1Ux6oJEkaGkeqpR6SzAGOA/YH9gAOSbJHV7XvA0uq6nHAmcC/TW2UkiRp2Eyqpd72BFZW1dVVtRY4DTios0JVfaOqftMWvwcsmOIYJUnSkJlUS73tAqzqKK9ul43n1cBXx1qRZGmSFUlWrFmzZhJDlCRJwzbr5lRv/8/bjlp21qnbjCj/18Jvjqrz6HMfO3KB86fVJclLgSXAs8ZaX1XHA8cDLFmyxFsMSZK0CZl1SbW0nq4DFnaUF7TLRkiyL/BPwLOq6vdTFJskSZomnP4h9XYRsDjJbkm2BA4GlnVWSPIE4KPAgVV14xBilCRJQ2ZSLfVQVfcARwBnA1cCZ1TV5UmOTnJgW+1/A9sBn01yaZJl4zQnSZI2UU7/kCZQVcuB5V3L3tHxfN8pD0qSJE0rsy6protG39jlg7v/0cjyGK/bjfM3UkSSJEma6Zz+IUmSJA3IpFqSJEkakEm1JEmSNCCTakmSJGlAJtWSJEnSgEyqJUmSpAGZVEuSJEkDMqmWJEmSBmRSLUmSJA3IpFqSJEkakEm1JEmSNCCTakmSJGlAJtWSJEnSgEyqJUmSpAGZVEsTSLJfkquSrExy5Bjrt0pyerv+giSLpj5KSZI0TCbVUg9J5gDHAfsDewCHJNmjq9qrgVuranfgP4D3TW2UkiRp2Eyqpd72BFZW1dVVtRY4DTioq85BwCnt8zOBfZJkCmOUJElDtvn6VL6DW2/6Wp157cYKRtPSQ4cdwJDtAqzqKK8GnjJenaq6J8ntwE7ATZ2VkiwFlrbFO5NctVEinjzz6HoPM8SkxZ2pPeYw67f3FDPuqWXcU8u4N55x86L1Sqqrav7gsUizU1UdDxw/7Dj6lWRFVS0Zdhzry7inlnFPLeOeWsY9tWZq3Os4/UPq7TpgYUd5QbtszDpJNgd2AG6ekugkSdK0YFIt9XYRsDjJbkm2BA4GlnXVWQa8vH3+IuDrVVVTGKMkSRqy9Zr+Ic027RzpI4CzgTnASVV1eZKjgRVVtQw4EfhkkpXALTSJ96ZgxkxV6WLcU8u4p5ZxTy3jnlozNW4A4oCaJEmSNBinf0iSJEkDMqmWJEmSBmRSLWmUiW7NPh0lOSnJjUl+NOxY1keShUm+keSKJJcnecOwY+pHkrlJLkxyWRv3u4Yd0/pIMifJ95N8edix9CvJNUl+mOTSJCuGHU+/kuyY5MwkP05yZZK9hh3TRJI8st3O6x6/TvJ3w46rH0n+vv2Z/FGSU5PMHXZM/Ujyhjbmy2fKtu7mnGpJI7S3Zv8J8Byam91cBBxSVVcMNbAJJHkmcCfwiap6zLDj6VeShwAPqapLkmwPXAy8cAZs7wDbVtWdSbYAvgO8oaq+N+TQ+pLkjcAS4H5VdcCw4+lHkmuAJVU13W+OMUKSU4BvV9UJ7VWUtqmq24YdV7/a34nXAU+pqml9A7wku9D8LO5RVb9NcgawvKpOHm5kvSV5DM0di/cE1gJnAa+tqpVDDWw9OVItqVs/t2afdqrqWzRXX5lRquqGqrqkfX4HcCXNXTqntWrc2Ra3aB8zYpQmyQLg+cAJw45lU5dkB+CZNFdJoqrWzqSEurUP8LPpnlB32BzYur1vwjbA9UOOpx+PAi6oqt9U1T3AN4E/H3JM682kWlK3sW7NPu2TvE1BkkXAE4ALhhtJf9opFJcCNwL/XVUzIm7gWOAtwH3DDmQ9FXBOkouTLB12MH3aDVgDfLydbnNCkm2HHdR6Ohg4ddhB9KOqrgP+HfgFcANwe1WdM9yo+vIj4BlJdkqyDfA8Rt54bUYwqZakaSDJdsDngL+rql8PO55+VNW9VfV4mjuN7tkewp3WkhwA3FhVFw87lg3w9Kp6IrA/cHg75Wm62xx4IvCRqnoCcBcwI87TAGinqxwIfHbYsfQjyf1pjizuBuwMbJvkpcONamJVdSXwPuAcmqkflwL3DjWoDWBSLalbP7dm1yRq5yR/Dvh0VX1+2PGsr/Zw/jeA/YYdSx+eBhzYzk8+DXh2kk8NN6T+tKOQVNWNwBdopmpNd6uB1R1HMc6kSbJniv2BS6rqV8MOpE/7Aj+vqjVVdTfweeBPhhxTX6rqxKp6UlU9E7iV5tyeGcWkWlK3fm7NrknSnvB3InBlVX1g2PH0K8n8JDu2z7emObH1x8ONamJV9daqWlBVi2j27a9X1bQfyUuybXsiK+30iefSHDKf1qrql8CqJI9sF+0DTOuTcLscwgyZ+tH6BfDUJNu0v1v2oTlPY9pL8sD2/11p5lN/ZrgRrT9vUy5phPFuzT7ksCaU5FRgb2BektXAO6vqxOFG1ZenAX8N/LCdnwzwtqpaPsSY+vEQ4JT2ygibAWdU1Yy5PN0M9CDgC02exObAZ6rqrOGG1LfXAZ9uv6RfDbxyyPH0pf3y8hzgb4YdS7+q6oIkZwKXAPcA32fm3Pr7c0l2Au4GDp+BJ7R6ST1JkiRpUE7/kCRJkgZkUi1JkiQNyKRakiRJGpBJtSRJkjQgk2pJkiRpQCbVkiRJ0oBMqiVJkqQBmVRLkiRJAzKpliRJkgZkUi1JkiQNyKRakiRJGpBJtSRJkjQgk2pJkiRpQCbVkiRJ0oBMqiVJkqQBmVRLkiRJAzKpliRJkgZkUi1JkiQNyKRakkSSO5M8bMA2Tk7ynj7rLkpSSTZvy19N8vJB+u9o+xlJruooX5Nk38lou23v8iR7T1Z7UynJ3klW91n3qCSf2tgxjdP3iDg3dJt37wvSxmRSLUmTJMlLkqxoE9Qb2kTx6X2+tpLsvrFjHE9VbVdVVw+x//2r6pSJ6vWznarq21X1yMmIa6wvClX16Ko6dzLan6DvSnLjui8e7bIt2mW1sfufTvrd5t37x2TuCxP0+4z2577zUUn+YmP3renDpFqSJkGSNwLHAu8FHgTsCnwYOGiYcU2kM2HbFGxq7we4Fdi/o7x/u2xGSTJn2DFsTG3yvt26B3AAcCdw1pBD0xSaVUl1knuTXNoeRrosyZuSbNauW5Lkg320cV77/6IkLxmnzp+2/ax7/C7JCyf33UiaLpLsABwNHF5Vn6+qu6rq7qr6UlX9Q1tnzyTnJ7mtHcX+UJIt23Xfapu6rB3h+qt2+QHt75DbkpyX5HEdfT4xyfeT3JHks0lO7xxRTXJYkpVJbkmyLMnOHesqyeFJfgr8tGPZ7u3zrZO8P8m1SW5P8p0kW7frPpvkl+3ybyV5dJ/baE6Sf09yU5Krged3rT83yWva57sn+Wbbx01JTh9vO62bJpDkH5P8Evj4OFMcnpzkiiS3Jvl4krltm69I8p2uWKqNYSlwKPCWtr8vtev/MJ0kyVZJjk1yffs4NslW7bp1sb2pHV2+Ickr+9leHT4JvKyj/DLgE13x7tx+xre0n/lhHeu2TjPafmuSK4Anj/HazyVZk+TnSV7fT1Ad7+1t7Wd0TZJDO9afnOQjSZYnuQv401599RFn5zaf0/b7s3b/vzjJwl77R0c7j2r3tdvS5AIHdsV8XJKvtO1ekOTh/WyPMbwcOLOq7trA12smqqpZ8wDu7Hj+QOBrwLs2sK29gS/3Ue8BwC3ANsN+/z58+Ng4D2A/4B5g8x51ngQ8FdgcWARcCfxdx/oCdu8oPwG4EXgKMIfmj/Q1wFbAlsC1wBuALYA/B9YC72lf+2zgJuCJbf3/A3yrq6//bn8/bd3dP3AccC6wS9v3nwBbteteBWzftnsscGlHuyevi2GM9/9a4MfAwrbfb7R9bt6uPxd4Tfv8VOCfaAZ+5gJP77Gd9m63/fvamLZul63uqHMN8KOOvr/bsa1eAXynK9bObTHqPbXt7ds+Pxr4Hs3flPnAecC7u2I7uv2cngf8Brh/n/tVAY8BfgXsCNy/ff4YoDrqfYvmqMhc4PHAGuDZ7bpjgG+373thux1Wt+s2Ay4G3kGzTz0MuBr4s3b9UcCnxolt3Xv7QLvdnwXcBTyyY7vdDjyt7WebCfoaN84xtvk/AD8EHgkE+GNgpx77x7r3uwWwEnhbG8OzgTu6YgV2+gQAACAASURBVL4Z2JPm5/TTwGkdbX0ZOLKPz23btt29h/27ycfUPmbVSHWnqroRWAockcbeSb4MkGR+kv9uv8We0I7WzGvX3dk2cQzwjDSjSH/fo6sXAV+tqt9szPcjaah2Am6qqnvGq1BVF1fV96rqnqq6BvgoTSIynqXAR6vqgqq6t5r5xr+nSczXJecfrGZE/PPAhR2vPRQ4qaouqarfA28F9kqyqKPOv1bVLVX1285O0xy9exXwhqq6ru37vLYdquqkqrqjLR8F/HGakfqJvBg4tqpWVdUtwL/2qHs38FBg56r6XVV9p0ddgPuAd1bV77vfT4cPdfT9L8AhfcTcj0OBo6vqxqpaA7wL+OuO9Xe36++uquU0UwLWZ47v74AvAX/VPpa1ywBIspAmcf3HdltdCpzA/xvdfjHwL+1nvQroPCL7ZGB+VR1dVWurmVP/MeDg9Yjvn9vt/k3gK21/6/zfqvpuVd0HPHaCvnrF2e01wNur6qpqXFZVN/cR61OB7YBj2hi+TpMod+4LX6iqC9uf5U/TfEkBoKoOqKpj+ujnz2m+1H6zj7rahMzapBqg/aGeQzPC0OmdwNer6tHAmTRzI7sdCXy7qh5fVf/Ro5uDaUZdJG26bgbmpcd83iSPSPLlNFMnfk0z93pejzYfCrypPUx9W5LbaEbwdm4f11VV58lqqzqe70wzkg1AVd3ZxrjLOPU7zaMZ8fzZGO9hTpJj2sPuv6YZPVz3mons3NXnteNVBN5CMwJ5YTu48aoJ2l5TVb+boE533zuPV3E9jdjWY7R9c9eXrd/QJHbr4xM0SfKoqR9tX7dU1R1dMezSsX687f5QYOeufextNOcE9OPWGjm9ofu9d/Y7UV/rs38sZIz9sw87A6vaJL+zn86fi192PN+Qzwqao0qf6Pr51Cwwq5PqHp4OnAZQVWexgSeFJHkIzbfzsycvNEnT0Pk0o8i9zp34CM30h8VVdT+ahCI96q+iGbnbseOxTVWdCtwA7JKk8/ULO55fT5PEAJBkW5rR9Os66oz3B/8mmpHQseaSvoTmxMt9gR1oprEwwftY54auGMcarGgCq/plVR1WVTsDfwN8OL2v+NFP8tLd9/Xt87topiYAkOTB69n2iG3d1fZk+TbwEJoEtHvU/nrgAUm274ph3Wfda7uvAn7etY9tX1XP6zOu+7f7Vmfbne+9+0tfr7763j/atjZkrvP1wML2aExnP9eNU3+9tUcO9mb0lx/NArM6qU5zTdZ7aeYtbgwvpjmUdPdGal/SNFBVt9PMFT0uyQuTbJPm0mf7J/m3ttr2wK+BO5P8EfD/dTXzK5p5put8DHhtkqe0U9S2TfL8Nnk6n+Z31xFJNk9yEM080HVOBV6Z5PFpTpp7L3BBO+1kovdyH3AS8IH2xLI5SfZq29me5svDzTSJ6Hv730qcAbw+yYIk96c52jemJH+ZZEFbvJUmOVs3uti9nfp1eNv3A2jma5/eLr8MeHS7rebSTGnpNFF/pwJvb6cNzqPZDyb12s7tiOcLgAO7Rz/bqRLnAf+aZG6ak1lf3RHDGcBbk9y/3aav63j5hcAdaU7y3Lr9rB+TZMRJghN4V5ItkzyD5ooXnx2n3kR99Yqz2wnAu5Msbn82Hpdkp3Zdr8/rAprR57e0P59702zX0/p+txP7a+C8qtqQkXTNcLM2qU4yH/gvmnl23SMR36WdF5bkuTQnh3S7g+YPTC+H4NQPaVaoqvcDbwTeTnOi2CrgCOCLbZU304z03kGTMJ/e1cRRwCntofEXV9UK4DDgQzSJ5Uqak+qoqrU08zZfDdwGvJRmbui6ec9fA/4Z+BzNCODDWb95sm+mORHsIpoTrd9H8/fiEzSHy68DrqA5Qa9fH6M5ancZcAnw+R51nwxc0J7Dsoxmfve6a2gfRcd2Wo/+PwOcQ3Ny3M+A9wBU1U9oTiT8Gs2VULpHgk8E9mj7+yKjvQdYAfyAZptdsq7tiaS5gsVX+6lbVZdX1eXjrD6E5qjB9cAXaOaXf61d9y6az+znNO//kx1t3kuTCD++XX8TTcLazxx5aKZK3Nr2+2ngtVX143Hin6ivceMcwwdokvBzaL6onkhzgir02D/an5sX0FyW8CaakztfNl7M3dJcd/5tE1R7GTDh9da1acpsmvKT5F6aX3pb0Jy1/EngA1V1X/uN9c1VdUCSB9Ikww+iGRE6AFhUVb9PcmdVbZdkC5o/EDsBJ3fPq25PCPousLBr/pYkTbokFwD/VVUfH3Ys2vS1fzM/VVULJqorzRazKqnuV3uY896quifJXsBHqurxE71OkqZKkmcBV9GMuB1Kc+TtYVV1w1AD06xgUi2NtqndeWqy7Aqc0Z7MsJbmEKwkTSePpDkEvi3NlIYXmVBL0vA4Ui1JkiQNaNaeqChJkiRNlvWa/rFltqq5bDtxRW0yfsddrK3f93MNWq2HefPm1aJFi4YdhiRJWg8XX3zxTVU1f6x165VUz2VbnpJ9JicqzQgX1P8MO4RN0qJFi1ixYsWww5AkSeshybh3+3T6hyRJkjQgk2qphyQnJbkxyY/GWZ8kH0yyMskPkjxxqmOUJEnDZ1It9XYysF+P9fsDi9vHUuAjUxCTJEmaZkyqpR6q6ls0t2kez0HAJ6rxPWDHJA+ZmugkSdJ04c1fpMHsAqzqKK9ul426CUeSpTSj2ey6665TEpwkzUaLjvzKsEMY5Zpjnj/sELSROVItTZGqOr6qllTVkvnzx7wajyRJmqFMqqXBXAcs7CgvaJdJkqRZxKRaGswy4GXtVUCeCtxeVaOmfkiSpE2bc6qlHpKcCuwNzEuyGngnsAVAVf0XsBx4HrAS+A3wyuFEKkmShsmkWuqhqg6ZYH0Bh09ROJIkaZpy+ockSZI0IJNqSZIkaUAm1ZIkSdKATKolSZKkAZlUS5IkSQMyqZYkSZIGZFItSZIkDcikWpIkSRqQSbUkSZI0IJNqSZIkaUAm1ZIkSdKATKolSZKkAZlUS5IkSQMyqZYkSZIGZFItSZIkDWjzYQcwiCd9/74R5Xc/8NJRdZ75wxeNKN/1lQePqrPT5b8fUZ7701+NqnPPqtUbEqI2AUn2A/4TmAOcUFXHdK3fFTgF2LGtc2RVLZ/yQCVJ0tA4Ui31kGQOcBywP7AHcEiSPbqqvR04o6qeABwMfHhqo5QkScNmUi31tiewsqqurqq1wGnAQV11Crhf+3wH4PopjE+SJE0DJtVSb7sAqzrKq9tlnY4CXppkNbAceN1YDSVZmmRFkhVr1qzZGLFKkqQhmdFzqrvdR41a9vXHnj6ivNljR3+PuI+Rc7O/fNdOo+oc+YVDR5Qf9o/nb0iI2jQdApxcVe9PshfwySSPqaoRO1ZVHQ8cD7BkyZLRO6skSZqxHKmWersOWNhRXtAu6/Rq4AyAqjofmAvMm5LoJEnStGBSLfV2EbA4yW5JtqQ5EXFZV51fAPsAJHkUTVLt/A5JkmYRk2qph6q6BzgCOBu4kuYqH5cnOTrJgW21NwGHJbkMOBV4RVU5vUOSpFlkk5pTLW0M7TWnl3cte0fH8yuAp011XJIkafqY0Un1l6999IjyC3e8eFSdJ2w5cjB+MzJGSyPrvHDb20bVOPClHxpRPv4Fi0bV+cqL9hpRvveKn4zRlyRJkjY1Tv+QJEmSBmRSLUmSJA3IpFqSJEka0IyeU/2QF145onzUo14yqs7aB28/onzNa0ZflOGUvU4cUd5zq9F1um8Qs3SHa0bVWXHSohHl6586qookSZI2QY5US5IkSQMyqZYkSZIGZFItSZIkDcikWpIkSRrQjD5Rsdu9V/501LI5I89l5OHfGP26o3niiPLmCxeMqnPFOx88orxy/+NH1Tl+4bkjygfwpHEilSRJ0qbEkWpJkiRpQCbVkiRJ0oBMqiVJkqQBbVJzqifLPatWj1r2iNeMXPbMs140qs7XH3v6iPLNr95rVJ2dTjx/wOgkSZI03ThSLUmSJA3IpFqaQJL9klyVZGWSI8ep8+IkVyS5PMlnpjpGSZI0XE7/kHpIMgc4DngOsBq4KMmyqrqio85i4K3A06rq1iQPHE60kiRpWByplnrbE1hZVVdX1VrgNOCgrjqHAcdV1a0AVXXjFMcoSZKGzJHqDfSSXS8atWwzv6NsinYBVnWUVwNP6arzCIAk3wXmAEdV1VndDSVZCiwF2HXXXTdKsJIkaTjMAqXBbQ4sBvYGDgE+lmTH7kpVdXxVLamqJfPnz5/iECVJ0sZkUi31dh2wsKO8oF3WaTWwrKrurqqfAz+hSbIlSdIsYVIt9XYRsDjJbkm2BA4GlnXV+SLNKDVJ5tFMB7l6KoOUJEnD5ZzqDbR0h2tGLbuP+0aUd/rhnVMUjTaWqronyRHA2TTzpU+qqsuTHA2sqKpl7brnJrkCuBf4h6q6eXhRS5KkqWZSLU2gqpYDy7uWvaPjeQFvbB+SJGkWcvqHJEmSNCCTakmSJGlAJtWSJEnSgJxT3affHrTniPJmXDJGLb+jSJIkzUZmgZIkSdKATKolSZKkAZlUS5IkSQNyTnWfbvmjkZvqPmpUne6bv9z82O1G1dnpwsmNS5IkScPnSLUkSZI0IJNqSZIkaUAm1ZIkSdKATKolSZKkAXmi4gbajIy5tNNOJ54/NcFIkiRpqBypliRJkgZkUi1NIMl+Sa5KsjLJkT3q/UWSSrJkKuOTJEnDZ1It9ZBkDnAcsD+wB3BIkj3GqLc98AbggqmNUJIkTQcm1RvoPmqMx8h/2iTsCaysqqurai1wGnDQGPXeDbwP+N1UBidJkqYHk2qpt12AVR3l1e2yP0jyRGBhVX1lKgOTJEnTh0m1NIAkmwEfAN7UR92lSVYkWbFmzZqNH5wkSZoyJtVSb9cBCzvKC9pl62wPPAY4N8k1wFOBZWOdrFhVx1fVkqpaMn/+/I0YsiRJmmom1VJvFwGLk+yWZEvgYGDZupVVdXtVzauqRVW1CPgecGBVrRhOuJIkaRhMqjfQZmSMx8h/mvmq6h7gCOBs4ErgjKq6PMnRSQ4cbnSSJGm68I6K0gSqajmwvGvZO8apu/dUxCRJkqYXh1MlSZKkAZlUS5IkSQNy+scGuo8aY5k3fJEkSZqNHKmWJEmSBmRSLUmSJA3IpFqSJEkakEm1JEmSNCBPVNxAm5Exl0qSJGn2MQuUJEmSBmRSLUmSJA3IpFqSJEkakHOqN5A3f5EkSdI6jlRLkiRJAzKpliRJkgZkUi1JkiQNyKRamkCS/ZJclWRlkiPHWP/GJFck+UGS/0ny0GHEKUmShsekul8Z+dgic0Y9Nuv6p5kvyRzgOGB/YA/gkCR7dFX7PrCkqh4HnAn829RGKUmShs3MT+ptT2BlVV1dVWuB04CDOitU1Teq6jdt8XvAgimOUZIkDZlJtdTbLsCqjvLqdtl4Xg18daNGJEmSph2vUy1NkiQvBZYAzxpn/VJgKcCuu+46hZFJkqSNzaS6Tztdfs+I8t1176g63vxlk3QdsLCjvKBdNkKSfYF/Ap5VVb8fq6GqOh44HmDJkiWj7x4kSZJmLKd/SL1dBCxOsluSLYGDgWWdFZI8AfgocGBV3TiEGCVJ0pCZVEs9VNU9wBHA2cCVwBlVdXmSo5Mc2Fb738B2wGeTXJpk2TjNSZKkTZTTP6QJVNVyYHnXsnd0PN93yoOSJEnTiiPVkiRJ0oAcqe7T3C9dOKK8xUfnjKpzd9epZ5svHH254ntWrZ7UuCRJkjR8jlRLkiRJAzKpliRJkgZkUi1JkiQNyDnVG+hhZ/7NqGVX/sWHRpTvfPzOo+rMdU61JEnSJseRakmSJGlAJtWSJEnSgEyqJUmSpAGZVEuSJEkD8kTFDbT49ReMWnbg6588ojyXC0fVkSRJ0qbHkWpJkiRpQCbVkiRJ0oBMqiVJkqQBmVRLkiRJAzKpliaQZL8kVyVZmeTIMdZvleT0dv0FSRZNfZSSJGmYTKqlHpLMAY4D9gf2AA5JskdXtVcDt1bV7sB/AO+b2iglSdKwmVRLve0JrKyqq6tqLXAacFBXnYOAU9rnZwL7JMkUxihJkobMpFrqbRdgVUd5dbtszDpVdQ9wO7DTlEQnSZKmhfW6+csd3HrT1+rMazdWMJqWHjrsADYVSZYCS9vinUmuGmY8fZgH3DTsIDaAcU8t455axj21Ji3uTO3EwFm/vTeicfOi9Uqqq2r+4LFIM8p1wMKO8oJ22Vh1VifZHNgBuLm7oao6Hjh+I8U56ZKsqKolw45jfRn31DLuqWXcU8u4p9ZMjXsdp39IvV0ELE6yW5ItgYOBZV11lgEvb5+/CPh6VdUUxihJkoZsvUaqpdmmqu5JcgRwNjAHOKmqLk9yNLCiqpYBJwKfTLISuIUm8ZYkSbOISbU0gapaDizvWvaOjue/A/5yquOaAjNmqkoX455axj21jHtqGffUmqlxAxCPUkuSJEmDcU61JEmSNCCTakmjTHRr9ukoyUlJbkzyo2HHsj6SLEzyjSRXJLk8yRuGHVM/ksxNcmGSy9q43zXsmNZHkjlJvp/ky8OOpV9JrknywySXJlkx7Hj6lWTHJGcm+XGSK5PsNeyYJpLkke12Xvf4dZK/G3Zc/Ujy9+3P5I+SnJpk7rBj6keSN7QxXz5TtnU3p39IGqG9NftPgOfQ3OzmIuCQqrpiqIFNIMkzgTuBT1TVY4YdT7+SPAR4SFVdkmR74GLghTNgewfYtqruTLIF8B3gDVX1vSGH1pckbwSWAPerqgOGHU8/klwDLKmq6X4d3xGSnAJ8u6pOaK+itE1V3TbsuPrV/k68DnhKVU3re3Uk2YXmZ3GPqvptkjOA5VV18nAj6y3JY2juWLwnsBY4C3htVa0camDryZFqSd36uTX7tFNV36K5+sqMUlU3VNUl7fM7gCsZfdfOaacad7bFLdrHjBilSbIAeD5wwrBj2dQl2QF4Js1VkqiqtTMpoW7tA/xsuifUHTYHtm7vm7ANcP2Q4+nHo4ALquo37Z2Jvwn8+ZBjWm8m1ZK69XNrdm0ESRYBTwAuGG4k/WmnUFwK3Aj8d1XNiLiBY4G3APcNO5D1VMA5SS5u79A6E+wGrAE+3k63OSHJtsMOaj0dDJw67CD6UVXXAf8O/AK4Abi9qs4ZblR9+RHwjCQ7JdkGeB4jb7w2I5hUS9I0kGQ74HPA31XVr4cdTz+q6t6qejzNnUb3bA/hTmtJDgBurKqLhx3LBnh6VT0R2B84vJ3yNN1tDjwR+EhVPQG4C5gR52kAtNNVDgQ+O+xY+pHk/jRHFncDdga2TfLS4UY1saq6EngfcA7N1I9LgXuHGtQGMKmW1K2fW7NrErVzkj8HfLqqPj/seNZXezj/G8B+w46lD08DDmznJ58GPDvJp4YbUn/aUUiq6kbgCzRTtaa71cDqjqMYZ9Ik2TPF/sAlVfWrYQfSp32Bn1fVmqq6G/g88CdDjqkvVXViVT2pqp4J3Epzbs+MYlItqVs/t2bXJGlP+DsRuLKqPjDsePqVZH6SHdvnW9Oc2Prj4UY1sap6a1UtqKpFNPv216tq2o/kJdm2PZGVdvrEc2kOmU9rVfVLYFWSR7aL9gGm9Um4XQ5hhkz9aP0CeGqSbdrfLfvQnKcx7SV5YPv/rjTzqT8z3IjWn3dUlDTCeLdmH3JYE0pyKrA3MC/JauCdVXXicKPqy9OAvwZ+2M5PBnhbeyfP6ewhwCntlRE2A86oqhlzeboZ6EHAF5o8ic2Bz1TVWcMNqW+vAz7dfkm/GnjlkOPpS/vl5TnA3ww7ln5V1QVJzgQuAe4Bvs/MuUvh55LsBNwNHD4DT2j1knqSJEnSoJz+IUmSJA3IpFqSJEkakEm1JEmSNCCTakmSJGlAJtWSJEnSgEyqJUmSpAGZVEuSJEkDMqmWJEmSBmRSLUmSJA3IpFqSJEkakEm1JEmSNCCTakmSJGlAJtWSJEnSgEyqJUmSpAGZVEuSJEkDMqmWJEmSBmRSLUmSJA3IpFqSJEkakEm1JIkkdyZ52IBtnJzkPX3WXZSkkmzelr+a5OWD9N/R9jOSXNVRvibJvpPRdtve5Un2nqz2plKSvZOs7rPuUUk+tbFjGqfvEXFu6Dbv3hekjcmkWpImSZKXJFnRJqg3tIni0/t8bSXZfWPHOJ6q2q6qrh5i//tX1SkT1etnO1XVt6vqkZMR11hfFKrq0VV17mS0P0HfleTGdV882mVbtMtqY/c/nfS7zbv3j8ncF/ro+wVJftT+/J+XZI+p6FfTh0m1JE2CJG8EjgXeCzwI2BX4MHDQMOOaSGfCtinY1N4PcCuwf0d5/3bZjJJkzrBj2JiSLAY+DbwW2BH4ErBsE9wf1YNJtSQNKMkOwNHA4VX1+aq6q6rurqovVdU/tHX2THJ+ktvaUewPJdmyXfettqnL2lGuv2qXH5Dk0vY15yV5XEefT0zy/SR3JPlsktM7R1STHJZkZZJbkixLsnPHukpyeJKfAj/tWLZ7+3zrJO9Pcm2S25N8J8nW7brPJvllu/xbSR7d5zaak+Tfk9yU5Grg+V3rz03ymvb57km+2fZxU5LTx9tO66YJJPnHJL8EPj7OFIcnJ7kiya1JPp5kbtvmK5J8pyuWamNYChwKvKXt70vt+j9MJ0myVZJjk1zfPo5NslW7bl1sb2pHl29I8sp+tleHTwIv6yi/DPhEV7w7t5/xLe1nfljHuq3b0fZbk1wBPHmM134uyZokP0/y+n6C6nhvb2s/o2uSHNqx/uQkH0myPMldwJ/26quPODu3+Zy23/+/vTuPt6su7z3++RLmQVASB5JgqEYrcq1DRJy54gAOYHutBbWONfUqztaL1gpiHfAqVStaERS0CgIOjYqC1gkrIgFFBcVGBglDCTOICgnP/WOt9O7hDPtkJ2efk/N5v17nlf1b67fXevbKOmc/+7ef31q/ac//85Isnuj86NjOg9pz7aY0JSUH9sR8TJKvtds9J8n9BjkewNOAs6rqB1W1FjgKWAg8ccDnazMwp5LqJOvSvEFdmOSC9g/dFu26ZUk+PMA2ftj+uyTJ8ybod1Sar4F+kfYNUtJm69HAtsCXJuizDng9ML/tvx/wSoCqekLb58/aMozPJ3kY8Engb4FdgY/TjHxtkyYZ/xJwAnAP4CTgz9fvKMmTgPcAzwXuA1wOnNwTz7OBRwFjfUX9fuARwGPa7b8ZuKtd93VgKXBP4Hya0blBvBx4JvAwYBnwnAn6vhM4E7g7sAj4Zxj7OLXte7dx3hdYPs42n0+T+NwPeADwtskCrqpjaV7f+9r9PWuMbn8P7AM8FPgzYO+ebd8b2JkmwXoZcEySu0+27w5fBp6QZJf2eY8H/q2nz8nAamA3muP67vYcADic5jXfj+b1/3fdevv+9xXggja+/YDXJXnagLHdm+Z8Xthu99gknaUWzwPeBewE/HCSfY0b5xjeABwCPB24G/BS4PYJzo/1r3erNoYzac7fVwOf7Yn5YOAdNOfeqjb+9c//apLDJogrPY8D7DVBf21m5lRSDfy+qh5aVQ8GnkLzNdrhAFW1sqom/YReVY9pHy6h+YPRJ8kzgIfT/JF9FPCmJHcbPnxJM9SuwHXtCNWYquq8qvpRVa2tqstokuSJRrGWAx+vqnOqal1bb/xHmgRuH2BL4MPtiPgXgR93PPf5wCer6vyq+iPwFuDRSZZ09HlPVd1QVb/v3GmbaL0UeG1VXdnu+4ftdqiqT1bVrW37CODP0ozUT+a5wAer6oqquoEm6R/PnTQJ8m5V9Yeq+sEEfaFJ+A+vqj/2vp4OH+nY97tokrKN4fnAkVV1bVWtoUnI/rpj/Z3t+jur6nTgNmAqNb5/oEkE/6r9WdEuAyDJYuCxwP9pj9VPgeP4/6PbzwXe1f5fXwF0Dh49ElhQVUdW1R1tTf0naBLLQf1De9y/B3yt3d96/1ZV/1FVdwH/Y5J9TRRnr78B3lZVF1fjgqq6foBY9wF2BN7bxvBt4Kt0nwtfqqoft7/Ln6V5Hwegqp5ZVe8dZ9vfAp7YjoxvDbwV2BrYfoC4tJmYa0n1f6uqa2netA5NY98kXwVIsiDJN9sR7ePSfAU6v113W7uJ9wKPb0e+X9+z+T2B77dvnr8DfgbsPz2vTNIIXA/MzwT1k0ke0I50XZPkFpra6/kTbPO+wBvbr6lvSnITsJhmNHI34Mqq6pysdkXH491oRqcBqKrb2hgXjtO/03yaUfffjPEa5iV5b/u1+y3AZR3PmcxuPfu8fLyONCPjAX7c/h1+6STbXlNVf5ikT+++dxuv4xR1Hesxtn19z4et22kSu6n4NE2S3Ff60e7rhqq6tSeGhR3rxzvu9wV26znH3kozJ2AQN7bvcZ3b7nztnfudbF9TOT8WM8b5OYDdgCvaJL9zP52/F9d0PB74/6qqfkUzuv4R4Gqa34mLaL5B0BwxZ5NqgPaT8jyar4E6HQ58ux3RPo1mwlGvw2jqpx5aVf/Us+4CYP8k27fJ+P+k+SMgafN0Ns0o8rMn6PMx4FfA0qq6G01CkQn6X0EzcrdLx8/2VXUSzZv2wiSdz+/8G3MVTRIDQJIdaEbTr+zoM97VI66jGQkdq5b0eTQTL59MU9KwZP0uJngd613dE+NYf1ebwKquqaqXV9VuNOUvH83EV/wY5EoYvfu+qn38OzpGE5Pce4rb7jrWPdveWM6iKeO5F9A7an8VcI8kO/XEsP7/eqLjfgVwac85tlNVPX3AuO7enlud2+587b0f+iba18DnR7utQWudO10FLF5f9tmxnyvH6T8lVXVaVe1VVbvS5BFLgHM3xrY1O8zppHoCj6OtP6yqbzDFmdZVdSZwOk0N2Uk0b7jrNnKMkmaIqroZeDtNveyz2w/UWyU5IMn72m47AbcAtyX5U+B/92zmv4DO60R/AnhFkke136btkOQZbfK0/m/KoUm2THIQTS3veicBL0ny0DST5t4NnNOW6iaUfwAAIABJREFUnUz2Wu6iqeU+Os3EsnlJHt1uZyeaDw/X0ySi7x78KHEK8Joki9ra4HFrU5P8ZZJFbfNGmuRs/ehi73Ea1Kvafd+Dpg56fb3tBcCD22O1LU1JS6fJ9ncS8Lb2G875NOfBRr22c/uNxLOAA3u+naAtlfgh8J4k26aZzPqyjhhOAd6S5O7tMX11x9N/DNyaZpLndu3/9V5JuiYJTuIdSbZO8niamvlTx+k32b4mirPXccA7kyxtfzcekmTXdt1E/1/n0Iw+v7n9/dyX5rj2zjfYIEke0b6uBcCxwIp2BFtzxJxOqtPc6GAdcO3G3nZVvasdxX4KzSjOrzf2PiTNHFX1AZoJVG8D1tCMph1KM9EM4E00I7230iTMn+/ZxBHAie1X48+tqpU0k/s+QpNYrgJe3O7rDuAvaJKnm4AX0NSGrq97/hbwD8AXaEYA78fU6mTfBPycZpTtBporGWxBU3pwOc3I3kXAj6awzU8AZ9AksecDX5yg7yOBc9pyuxU09d3rr6F9BB3HaQr7/xzNBLVLaEoH/hGgqn5Nc+WWb9FcCaV3JPh4YM92f1+m3z8CK2nK/H7evrZBb4Dz1iRfH6RvVV1YVReOs/oQmlHRq2gmsB7engPQ1HhfDlxK8/o/07HNdTSJ8EPb9dfRJKyD1MhDUypxY7vfzwKvGC+JHGBf48Y5hqNpkvAzaT6oHg9s1647gnHOj/b35lk086muo7nk5QsHTXzTXHf+rRN0+RDN7+PFNMfl5RP01WYoPR96N2tJbquqHdvHC2j+CJxdVYe3n1jfVFXPTHIM8NuqOirJU2neCBZU1XXrt5HkEcDRVdU30SjN9Th3qarr21GDzwEPrQkmMUnSMJKcA/xLVX1q1LFo89e+Z/5rVS2arK80V8y1i5Jvl+SnwFbAWppPwkeP0e8dwElJ/prma9ZraEaXOv0MWJfkAuCEnrrqrYCz2nLHW4AXmFBL2piSPJFmROw6mitQPAT4xkiDkqQ5bE4l1VU17h2dqrn96Xfb5s3A06pqbZJHA4+s/385qR3bf+8EnjTGpmhnoXt7Ukmb0gNpvgLfgaak4TlVdfVoQ5KkuWtOlX8MKs3tRk+hqSG8A3hlVTmDV5IkSWMyqZYkSZKGNKev/iFJkiRtDFOqqd4629S27DB5R202/sDvuKP+OMiNHTQF8+fPryVLlow6DEmSNAXnnXfedVW1YKx1U0qqt2UHHpX9Nk5UmhXOqX8fdQibpSVLlrBy5cpRhyFJkqYgyeXjrbP8Q5pAkk8muTbJL8ZZnyQfTrIqyc+SPHy6Y5QkSaNnUi1N7ARg/wnWHwAsbX+WAx+bhpgkSdIMY1ItTaCqvk9zm+bxHAR8uho/AnZJcp/piU6SJM0UJtXScBYCV3S0V7fLJEnSHDKn7qgojVKS5TQlIuy+++4jjkaSpI1jyWFfG3UIfS577zOmfZ+OVEvDuRJY3NFe1C7rU1XHVtWyqlq2YMGYV+ORJEmzlEm1NJwVwAvbq4DsA9xcVVePOihJkjS9LP+QJpDkJGBfYH6S1cDhwFYAVfUvwOnA04FVwO3AS0YTqSRJGiWTamkCVXXIJOsLeNU0hSNJkmYoyz8kSZKkIZlUS5IkSUMyqZYkSZKGZFItSZIkDcmkWpIkSRqSSbUkSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGZFItSZIkDcmkWpIkSRqSSbUkSZI0JJNqSZIkaUgm1dIkkuyf5OIkq5IcNsb63ZN8J8lPkvwsydNHEackSRodk2ppAknmAccABwB7Aock2bOn29uAU6rqYcDBwEenN0pJkjRqJtXSxPYGVlXVJVV1B3AycFBPnwLu1j7eGbhqGuOTJEkzgEm1NLGFwBUd7dXtsk5HAC9Isho4HXj1WBtKsjzJyiQr16xZsylilSRJI7LlqAOYCe7Y/5Fd7f/94VP6+jx3x5u72jeuu72vz8O/9rqu9p7v6h+wXHvF6g0JUTPbIcAJVfWBJI8GPpNkr6q6q7NTVR0LHAuwbNmyGkGckiRpE3GkWprYlcDijvaidlmnlwGnAFTV2cC2wPxpiU6SJM0IJtXSxM4FlibZI8nWNBMRV/T0+S2wH0CSB9Ek1dZ3SJI0h5hUSxOoqrXAocAZwC9prvJxYZIjkxzYdnsj8PIkFwAnAS+uKss7JEmaQ6ypliZRVafTTEDsXPb2jscXAY+d7rgkSdLMMeeS6utf/ui+ZW9/84ld7QO2v7Wvz2/X/r6rfcO6rfr6/PSZH+pqH/Rvr+nrs40TFSVJkjY7ln9IkiRJQzKpliRJkoZkUi1JkiQNabOvqb7xRd011N94+/v7+vzXuu7PFo/4wBv7+iw+9bdd7br5lr4+C7+5bkNClCRJ0iznSLUkSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIW1WExW3XLJ737K/PexLXe3H/OCVfX12//i8rvZ9vvvDvj5re9pXfWnPvj5H3LP7JjJH/Hibvj5OZZQkSdr8OFItSZIkDcmkWpIkSRqSSbUkSZI0pM2qpvpXr92tb9mtd23b1b7/oVf09Vl3/Q1T3tfCnW/uW/b8s1/e1d56+fZ9fW7fvbs6+0/f+Iu+PnfdfvuU45EkSdLoOFItSZIkDcmkWppEkv2TXJxkVZLDxunz3CQXJbkwyeemO0ZJkjRam1X5h7SxJZkHHAM8BVgNnJtkRVVd1NFnKfAW4LFVdWOSe44mWkmSNCqOVEsT2xtYVVWXVNUdwMnAQT19Xg4cU1U3AlTVtdMcoyRJGrFZPVK99kmP6Gqf/ZwP9PXZ70N/19VezK/6+mSrrbvadecdGxTPP+99Uve+n9g/4fC6db/var/0mL/p39Av+mPUyCwEOme3rgYe1dPnAQBJ/gOYBxxRVd/o3VCS5cBygN13779RkSRJmr0cqZaGtyWwFNgXOAT4RJJdejtV1bFVtayqli1YsGCaQ5QkSZuSSbU0sSuBxR3tRe2yTquBFVV1Z1VdCvyaJsmWJElzhEm1NLFzgaVJ9kiyNXAwsKKnz5dpRqlJMp+mHOSS6QxSkiSN1qyuqf7t/t210Of+cde+PotP/M+u9pUv/NO+Pgs/3V3DPNbNYG577j5d7Qdsf2Ffn39410u72t9/zX/09fmvP96tq32X9dMzWlWtTXIocAZNvfQnq+rCJEcCK6tqRbvuqUkuAtYBf1dV148uakmSNN1mdVItTYeqOh04vWfZ2zseF/CG9keSJM1Bln9IkiRJQzKpliRJkoZkUi1JkiQNabOqqT76sqf2LdtyzW+72rfsuaSvz61Hdy9bdO+d+/pctXptV3vnZ23b1+cea87uat/0yu37+nznnL262kv5UV8fSZIkzS6OVEuSJElDMqmWJEmShmRSLUmSJA1pVtdU73bWuq72h5/7+b4+Z/ziwV3tL+1yTF+fo294SFf7q0ft29fnT7/w0672uj/8YdAwJUmStJlzpFqSJEkakkm1JEmSNCSTakmSJGlIJtWSJEnSkGb1RMVtv/LjrvZrb31VX5/rH9x9k5bPsH9fn3ufcEFXe+ff9d+Q5a4B4tlyj/t2td9/n1P7+lz8bw/uWyZJkqTZzZFqSZIkaUgm1ZIkSdKQTKqlSSTZP8nFSVYlOWyCfv8rSSVZNp3xSZKk0ZvVNdW95n33/L5l9/zu5M8bpF56EJc+b2FXe6vM6+uzxZ0ba2+aDknmAccATwFWA+cmWVFVF/X02wl4LXDO9EcpSZJGzZFqaWJ7A6uq6pKqugM4GThojH7vBI4CvNWmJElzkEm1NLGFwBUd7dXtsv+W5OHA4qr62kQbSrI8ycokK9esWbPxI5UkSSNjUi0NIckWwNHAGyfrW1XHVtWyqlq2YMGCTR+cJEmaNibV0sSuBBZ3tBe1y9bbCdgL+G6Sy4B9gBVOVpQkaW7ZrCYqjtpWj7yxq338zbv39dnirJ9MVzjaOM4FlibZgyaZPhh43vqVVXUzMH99O8l3gTdV1cppjlOSJI2QI9XSBKpqLXAocAbwS+CUqrowyZFJDhxtdJIkaaZwpFqaRFWdDpzes+zt4/TddzpikiRJM4sj1ZIkSdKQHKneUFv039hl7/v8tqv9vm8+q6/PUn60yUKSJEnSaDhSLUmSJA3JpFqSJEkakkm1JEmSNCSTakmSJGlITlTcQHc++WF9yz666ONd7ad8Y8/pCkeSJEkj5Ei1JEmSNCSTakmSJGlIJtWSJEnSkKyp3kBrHrJ137Lb646u9la3r52ucCRJkjRCjlRLkiRJQzKpliRJkoZkUi1JkiQNyaRakiRJGpITFQe1xbyu5h+X3dbX5WWXPqv7Kd/7ySYNSdMjyf7Ah4B5wHFV9d6e9W8A/gZYC6wBXlpVl097oJIkaWQcqZYmkGQecAxwALAncEiS3ltl/gRYVlUPAU4D3je9UUqSpFEzqZYmtjewqqouqao7gJOBgzo7VNV3qur2tvkjYNE0xyhJkkbMpFqa2ELgio726nbZeF4GfH2sFUmWJ1mZZOWaNWs2YoiSJGnUrKke0BY7bN/VvvDxn+rr88BvLu9qL+W6TRqTZpYkLwCWAU8ca31VHQscC7Bs2bKaxtAkSdImZlItTexKYHFHe1G7rEuSJwN/Dzyxqv44TbFJkqQZwvIPaWLnAkuT7JFka+BgYEVnhyQPAz4OHFhV144gRkmSNGIm1dIEqmotcChwBvBL4JSqujDJkUkObLv9X2BH4NQkP02yYpzNSZKkzZTlH9Ikqup04PSeZW/vePzkaQ9KkiTNKCbVA1rzV3v1LPnuKMKQJEnSDGT5hyRJkjQkk2pJkiRpSCbVkiRJ0pCsqR7Qum0zaZ8df7bNNEQiSZKkmcaRakmSJGlIJtWSJEnSkEyqJUmSpCGZVEuSJElDcqLiBlq99vd9yxZ/cXVXe+10BSNJkqSRcqRakiRJGpJJtSRJkjQkk2pJkiRpSNZUD+ieH/lhV/sVH3ncGL1+Oz3BSJIkaUZxpFqSJEkakkm1NIkk+ye5OMmqJIeNsX6bJJ9v15+TZMn0RylJkkbJpFqaQJJ5wDHAAcCewCFJ9uzp9jLgxqq6P/BPwFHTG6UkSRo1k2ppYnsDq6rqkqq6AzgZOKinz0HAie3j04D9kmQaY5QkSSM2pYmKt3Ljdd+q0y7fVMFoRrrvqAMYsYXAFR3t1cCjxutTVWuT3AzsClzX2SnJcmB527wtycWbJOKNZz49r2GWMO7pZdzTy7inl3FPr40Wdzbdd8bj5kVTSqqrasHwsUhzU1UdCxw76jgGlWRlVS0bdRxTZdzTy7inl3FPL+OeXrM17vUs/5AmdiWwuKO9qF02Zp8kWwI7A9dPS3SSJGlGMKmWJnYusDTJHkm2Bg4GVvT0WQG8qH38HODbVVXTGKMkSRoxb/4iTaCtkT4UOAOYB3yyqi5MciSwsqpWAMcDn0myCriBJvHeHMyaUpUexj29jHt6Gff0Mu7pNVvjBiAOqEmSJEnDsfxDkiRJGpJJtSRJkjQkk2pJfSa7NftMlOSTSa5N8otRxzIVSRYn+U6Si5JcmOS1o45pEEm2TfLjJBe0cb9j1DFNRZJ5SX6S5KujjmVQSS5L8vMkP02yctTxDCrJLklOS/KrJL9M8uhRxzSZJA9sj/P6n1uSvG7UcQ0iyevb38lfJDkpybajjmkQSV7bxnzhbDnWvaypltSlvTX7r4Gn0Nzs5lzgkKq6aKSBTSLJE4DbgE9X1V6jjmdQSe4D3Keqzk+yE3Ae8OxZcLwD7FBVtyXZCvgB8Nqq+tGIQxtIkjcAy4C7VdUzRx3PIJJcBiyrqll1U48kJwJnVdVx7VWUtq+qm0Yd16Dav4lXAo+qqhl9A7wkC2l+F/esqt8nOQU4vapOGG1kE0uyF80di/cG7gC+AbyiqlaNNLApcqRaUq9Bbs0+41TV92muvjKrVNXVVXV++/hW4Jc0d+mc0apxW9vcqv2ZFaM0SRYBzwCOG3Usm7skOwNPoLlKElV1x2xKqFv7Ab+Z6Ql1hy2B7dr7JmwPXDXieAbxIOCcqrq9qtYC3wP+YsQxTZlJtaReY92afcYneZuDJEuAhwHnjDaSwbQlFD8FrgW+WVWzIm7gg8CbgbtGHcgUFXBmkvOSLB91MAPaA1gDfKottzkuyQ6jDmqKDgZOGnUQg6iqK4H3A78FrgZurqozRxvVQH4BPD7Jrkm2B55O943XZgWTakmaAZLsCHwBeF1V3TLqeAZRVeuq6qE0dxrdu/0Kd0ZL8kzg2qo6b9SxbIDHVdXDgQOAV7UlTzPdlsDDgY9V1cOA3wGzYp4GQFuuciBw6qhjGUSSu9N8s7gHsBuwQ5IXjDaqyVXVL4GjgDNpSj9+CqwbaVAbwKRaUq9Bbs2ujaitSf4C8Nmq+uKo45mq9uv87wD7jzqWATwWOLCtTz4ZeFKSfx1tSINpRyGpqmuBL9GUas10q4HVHd9inEaTZM8WBwDnV9V/jTqQAT0ZuLSq1lTVncAXgceMOKaBVNXxVfWIqnoCcCPN3J5ZxaRaUq9Bbs2ujaSd8Hc88MuqOnrU8QwqyYIku7SPt6OZ2Pqr0UY1uap6S1UtqqolNOf2t6tqxo/kJdmhnchKWz7xVJqvzGe0qroGuCLJA9tF+wEzehJuj0OYJaUfrd8C+yTZvv3bsh/NPI0ZL8k92393p6mn/txoI5o6b1Muqct4t2YfcViTSnISsC8wP8lq4PCqOn60UQ3kscBfAz9v65MB3lpVp48wpkHcBzixvTLCFsApVTVrLk83C90L+FKTJ7El8Lmq+sZoQxrYq4HPth/SLwFeMuJ4BtJ+eHkK8LejjmVQVXVOktOA84G1wE+YPbf+/kKSXYE7gVfNwgmtXlJPkiRJGpblH5IkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGZFItSZIkDcmkWpIkSRqSSbUkSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGZFItSZIkDcmkWpIkSRqSSbUkSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIZlUS5JIcluSPxlyGyck+ccB+y5JUkm2bNtfT/KiYfbfse3HJ7m4o31ZkidvjG2327swyb4ba3vTKcm+SVYP2PeIJP+6qWMaZ99dcW7oMe89F6RNyaRakjaSJM9LsrJNUK9uE8XHDfjcSnL/TR3jeKpqx6q6ZIT7P6CqTpys3yDHqarOqqoHboy4xvqgUFUPrqrvboztT7LvSnLt+g8e7bKt2mW1qfc/kwx6zHvPj415Lkyy3/lJ/iPJ9UluSnJ2ksdu6v1qZjGplqSNIMkbgA8C7wbuBewOfBQ4aJRxTaYzYdscbG6vB7gROKCjfUC7bFZJMm/UMWxitwEvBRYAdweOAr6yGZ6PmoBJtSQNKcnOwJHAq6rqi1X1u6q6s6q+UlV/1/bZux29uqkdxf5Ikq3bdd9vN3VBO8r9V+3yZyb5afucHyZ5SMc+H57kJ0luTXJqks93jqgmeXmSVUluSLIiyW4d6yrJq5L8J/CfHcvu3z7eLskHklye5OYkP0iyXbvu1CTXtMu/n+TBAx6jeUnen+S6JJcAz+hZ/90kf9M+vn+S77X7uC7J58c7TuvLBJL8nyTXAJ8ap8ThkUkuSnJjkk8l2bbd5ouT/KAnlmpjWA48H3hzu7+vtOv/u5wkyTZJPpjkqvbng0m2adetj+2N7ejy1UleMsjx6vAZ4IUd7RcCn+6Jd7f2//iG9v/85R3rtmtH229MchHwyDGe+4Uka5JcmuQ1gwTV8dre2v4fXZbk+R3rT0jysSSnJ/kd8D8n2tcAcXYe83ntfn/Tnv/nJVk80fnRsZ0HtefaTWlKSg7sifmYJF9rt3tOkvsNcjyq6g9VdXFV3QUEWEeTXN9jkOdr8zCnkuok69K8QV2Y5IL2D90W7bplST48wDZ+2P67JMnzxulz3yTnd+zrFRv3lUiaYR4NbAt8aYI+64DXA/Pb/vsBrwSoqie0ff6sLcP4fJKHAZ8E/hbYFfg4sKJN4rZu93UCzZv2ScCfr99RkicB7wGeC9wHuBw4uSeeZwOPAvYcI9b3A48AHtNu/83AXe26rwNLgXsC5wOfneA1d3o58EzgYcAy4DkT9H0ncCZNUrII+GcY+zi17Xu3cd4XWD7ONp8PPA24H/AA4G2TBVxVx9K8vve1+3vWGN3+HtgHeCjwZ8DePdu+N7AzsBB4GXBMkrtPtu8OXwaekGSX9nmPB/6tp8/JwGpgN5rj+u72HAA4nOY134/m9f933Xr7/vcV4II2vv2A1yV52oCx3ZvmfF7YbvfYJJ2lFs8D3gXsBPxwkn2NG+cY3gAcAjwduBvNCPHtE5wf61/vVm0MZ9Kcv68GPtsT88HAO2jOvVVt/Ouf/9Ukh010QJL8DPgDsAI4rqqunai/NjNVNWd+gNs6Ht8T+Bbwjg3c1r7AV8dZtzWwTft4R+AyYLdRv35//PFn0/zQJGzXTPE5rwO+1NEu4P4d7Y8B7+x5zsXAE4EnAFcC6Vj3A+Af28fH0ySC69ftCNwJLOnY15N6tl3A/WkGW35Pk5hM9hp2aZ+3c9s+YX0MY/T9NvCKjvZT2+du2ba/C/xN+/jTwLHAojG203uc9gXuALbtWba6o31Zz76fDvymffxi4Afj7WOs19Ru78nt498AT+9Y9zTgso44fr/+NbbLrgX2GfAcWf9/chzNh6tXAJ9ol1XbZzHNB7adOp73HuCE9vElwP4d65avPzY0H6p+27PPtwCfah8fAfzrOLHtC6wFduhYdgrwDx3H7dMd6ybb17hxjnHMLwYOmuiYjXUu0HwguQbYomP9ScARHTEf13Oe/Goqv9ft87alSfpfNNXn+jO7f+bUSHWnaj49LgcOTWPfJF8FSLIgyTfbUebj0nwFOr9dd1u7ifcCj29Ho1/fs+07quqPbXMb5tg3AtIcdD0wPxPUTyZ5QDvSdU2SW2hqr+dPsM37Am9sv6a+KclNNAnUbu3PlVXVOVntio7Hu9GMTgNQVbe1MS4cp3+n+TRJwW/GeA3zkry3/dr9FppEZ/1zJrNbzz4vH68jzch4gB+3f4dfOsm211TVHybp07vv3cbrOEVdx3qMbV9fVWs72rfTfMiZik/TlH30lX60+7qhqm7tiWFhx/rxjvt9gd16zrG30swJGMSNVfW7nm13vvbO/U62r6mcH4sZ4/wcwG7AFdWUaHTup/P34pqOxxvyf0U1pSAnAYcl+bMNiFOz1JxO9qqZ6T6PZtS60+HAt6vqwcBpNBOOeh0GnFVVD62qf+pd2dZ3/Yzmj8RRVXXVxo1e0gxyNvBHmpKK8XwM+BWwtKruRpNQZIL+VwDvqqpdOn62b9+srwYWJul8/uKOx1fRJDEAJNmBpoTkyo4+41094jqar6/HqiV9Hs3EyyfTlDQsWb+LCV7Helf3xDjW39UmsKprqurlVbUbzQjtRzPxFT8GuRJG777X/03+HbD9+hVJ7j3FbXcd655tbyxn0ZTx3IvmG4ne/d8jyU49Maz/v57ouF8BXNpzju1UVU8fMK67t+dW57Y7X3vvh76J9jXw+dFua6Ba5x5XAYvXl3127OfKcfoPaytgqMtUanaZ00n1BB5HW39YVd9gA2ZaV9UVVfUQmq/pXpRk0E/+kmaZqroZeDtNveyzk2yf5tJnByR5X9ttJ+AW4LYkfwr8757N/Bfdb8CfAF6R5FHtt2k7JHlGmzydTfOV/6FJtkxyEE0t73onAS9J8tA0k+beDZxTVZcN8FruoqnlPjrNxLJ5SR7dbmcnmg8P19Mkou8e/ChxCvCaJIva2uBxa1OT/GWSRW3zRprkbP3oYu9xGtSr2n3fg6YOen297QXAg9tjtS1NyUOnyfZ3EvC29hvO+TTnwUa9tnP7jcSzgAN7vp2gqq6gqVd+T5Jt00xmfVlHDKcAb0ly9/aYvrrj6T8Gbk0zyXO79v96ryRdkwQn8Y4kWyd5PE3N/Knj9JtsXxPF2es44J1Jlra/Gw9Jsmu7bqL/r3NoRp/f3P5+7ktzXHvnG0xZkn2SPK49Ftsl+T80H4LOGXbbmj3mdFKd5kYH62hq3DaJdoT6FzS1XJI2U1X1AZoJVG8D1tCMph1KM9EM4E00I7230iTMn+/ZxBHAie1X48+tqpU0k/s+QpNYrqKp/6Wq7gD+giZ5ugl4AfBVmoSXqvoW8A/AF2hGAO9HMwFrUG8Cfg6cC9xAc3mwLWhKDy6nGdm7CPjRFLb5CeAMmiT2fOCLE/R9JHBOW263Anht/f9raB9Bx3Gawv4/RzNB7RKa0oF/BKiqX9NcueVbNFdC6R0JPh7Ys93fl+n3j8BK4Gc0x+z89dueTJorWHx9kL5VdWFVXTjO6kNovjW4imYC6+HtOQDNpLvLgUtpXv9nOra5jiYRfmi7/jqahHXnQWKiKZW4sd3vZ2nq1n81TvyT7WvcOMdwNE0SfibNB9Xjge3adUcwzvnR/t48i+ayhNfRXPLyhePF3CvNdeffOs7qbYBjaD5wXklTj/0Mv6WeW9LzoXezluS2qtqxfbyA5o/A2VV1ePuJ9U1V9cwkx9BMqDgqyVNp3ggWVNV167eR5BHA0VX1xDH2s4imju737YjMOcD/qqqfT9NLlTTHJDkH+Jeq+tSoY9Hmr33P/NeqWjRZX2mumGsj1du1EwsvpBmVOJPm03GvdwBPTfIL4C9pPo3f2tPnZ8C6NJfme33PugfRjLJcAHwPeL8JtaSNKckTk9y7Lf94EfAQ4BujjkuS5qo5daefqhr3jk7V3P70u23zZuBpVbU2yaOBR66/msf6ke6quhN40hiboqq+SfMGJ0mbygNpvgLfgaak4TlVdfVoQ5KkuWtOlX8MKslSmjerLWiuf/rKqjp3tFFJkiRppjKpliRJkoY0pfKPrbNNbcsOk3cxsnNfAAARf0lEQVTUZuMP/I476o+DXINWUzB//vxasmTJqMOQJElTcN55511XVQvGWjelpHpbduBR2W/jRKVZ4Zz691GHsFlasmQJK1euHHUYkiRpCpKMe7fPuXb1D0mSJGmjM6mWJpDkk0mubS+vONb6JPlwklVJfpbk4dMdoyRJGj2TamliJwD7T7D+AGBp+7Mc+Ng0xCRJkmYYk2ppAlX1fZrbNI/nIODT1fgRsEuS+0xPdJIkaaaYUzd/kTaBhcAVHe3V7bK+m3AkWU4zms3uu+8+LcFJ41ly2NdGHUKfy977jFGHIEkbzJFqaZpU1bFVtayqli1YMObVeCRJ0ixlUi0N50pgcUd7UbtMkiTNISbV0nBWAC9srwKyD3BzVfWVfkiSpM2bNdXSBJKcBOwLzE+yGjgc2Aqgqv4FOB14OrAKuB14yWgilSRJo2RSLU2gqg6ZZH0Br5qmcCRJ0gxl+YckSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGNGsuqXfX4x/Wt2zJ+3/d1X7xgrP6+rz5sFd2tXc85UcbNzBJkiTNeY5US5IkSUMyqZYkSZKGZFItSZIkDWnW1FSvet5WfctOX/T9rvYWpK/PGUd/sKt94hFL+/r80+nP6Grf77Tb+/ps8fPfdLXv+t3vxg9WkiRJc4oj1ZIkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGNGsmKm63esNC3SbdExyX73xZX5/lhxzTveCQ/u28+7r/0dX+zDef0Nfnfm/yxjKboyT7Ax8C5gHHVdV7e9bvDpwI7NL2OayqTp/2QCVJ0sg4Ui1NIMk84BjgAGBP4JAke/Z0extwSlU9DDgY+Oj0RilJkkbNpFqa2N7Aqqq6pKruAE4GDurpU8Dd2sc7A1dNY3ySJGkGMKmWJrYQuKKjvbpd1ukI4AVJVgOnA68ea0NJlidZmWTlmjVrNkWskiRpRGZNTfXuR/24b9lTzn9FV3veG/+rr8+Rf/Llrvbe29QG7f9t83/R1X7Zc/vjecG/v76rvc3Xz92gfWnWOQQ4oao+kOTRwGeS7FVVd3V2qqpjgWMBli1btmEnoiRJmpEcqZYmdiWwuKO9qF3W6WXAKQBVdTawLTB/WqKTJEkzgkm1NLFzgaVJ9kiyNc1ExBU9fX4L7AeQ5EE0SbX1HZIkzSEm1dIEqmotcChwBvBLmqt8XJjkyCQHtt3eCLw8yQXAScCLq8ryDkmS5pBZU1MtjUp7zenTe5a9vePxRcBjpzsuSZI0c8yapLrWru1b1jcR8Ov9z3vX0ud0tX/9inv29TnrL9/f1Z4/b7tJ47nXGH3uet11k8YjSZKkzY/lH5IkSdKQTKolSZKkIZlUS5IkSUOaNTXVG2rdf17S1b7fGy/p6/PiNz6uq/3rj+7d1+fSZx/bvaD7vh4A/OFz9+5qb8elg4YpSZKkWcyRakmSJGlIJtWSJEnSkEyqJUmSpCGZVEuSJElD2uwnKg7i9j9/VFf7ZY/7Xl+fdT0TE//999v09Zm/4lfdz9kIsUmSJGnmc6RakiRJGpJJtSRJkjQkk2pJkiRpSHOupnrLP1nSt+zAd36rq/26u/96jGemq/W3Z76kr8cDbvzxMKFJkiRplnKkWpIkSRqSSbU0iST7J7k4yaokh43T57lJLkpyYZLPTXeMkiRptOZc+Yc0FUnmAccATwFWA+cmWVFVF3X0WQq8BXhsVd2Y5J6jiVaSJI2KI9XSxPYGVlXVJVV1B3AycFBPn5cDx1TVjQBVde00xyhJkkZszo1UX/vhrfuWjT0xcWJnPfPovmX/8phHd7XPf879+/qsW3XplPelkVoIXNHRXg08qqfPAwCS/AcwDziiqr7Ru6Eky4HlALvvvvsmCVaSJI2GI9XS8LYElgL7AocAn0iyS2+nqjq2qpZV1bIFCxZMc4iSJGlTMqmWJnYlsLijvahd1mk1sKKq7qyqS4Ff0yTZkiRpjjCpliZ2LrA0yR5JtgYOBlb09PkyzSg1SebTlINcMp1BSpKk0ZpzNdU7bH3HRtnOveZt17fs8AU/7Wp/4Wur+/occcEzu9r3fe7PN0o82jSqam2SQ4EzaOqlP1lVFyY5ElhZVSvadU9NchGwDvi7qrp+dFFLkqTpNueSammqqup04PSeZW/veFzAG9ofSZI0B1n+IUmSJA3JpFqSJEkakkm1JEmSNKQ5V1O9/ev6b/7yP/7q1V3tJx7wk74+L5z/H13tfbbJpPv6yx3756r95WNP7Go/6Vmv6Ouz7Vd+POm2JUmSNHM4Ui1JkiQNyaRakiRJGpJJtSRJkjSkOVdTve6iX/ctu+/h3csuO7z/ee/e48+72r96zX36+nz1z4/uat9/q20mjecp7/p+37KzVy7paq+9+ppJtyNJkqTRcaRakiRJGpJJtSRJkjQkk2pJkiRpSCbVkiRJ0pDm3ETFDbX20su72vd//eV9fV75jdd0tU897kN9fXbeYtuu9gE7/ayvzw933HNDQpQkSdKIOFItSZIkDcmkWpIkSRqSSbU0iST7J7k4yaokh03Q738lqSTLpjM+SZI0etZUb0Rbn7Gyq/3k9/1dX59zD/vnrvaxa57Y12fdf16ycQPTBksyDzgGeAqwGjg3yYqquqin307Aa4Fzpj9KSZI0ao5USxPbG1hVVZdU1R3AycBBY/R7J3AU8IfpDE6SJM0MJtXSxBYCV3S0V7fL/luShwOLq+prE20oyfIkK5OsXLNmzcaPVJIkjYxJtTSEJFsARwNvnKxvVR1bVcuqatmCBQs2fXCSJGnamFRLE7sSWNzRXtQuW28nYC/gu0kuA/YBVjhZUZKkucWJipvQvX9066R9vvPth/Yt24OzN0U42jDnAkuT7EGTTB8MPG/9yqq6GZi/vp3ku8CbqmolkiRpznCkWppAVa0FDgXOAH4JnFJVFyY5MsmBo41OkiTNFI5US5OoqtOB03uWvX2cvvtOR0ySJGlmcaRakiRJGpIj1RvRlgt362qv/vu1fX22IF3tPzn1lr4+tXHDkiRJ0ibmSLUkSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIW1WExW32GGH/mU77djVvuumm/v77HqP7gVJX59eNzxhcd+ytx15Qlf7qdv9rq/Ptet+39Wu8y6cdF+SJEma2RypliRJkoZkUi1JkiQNyaRakiRJGtJmVVO9zw9v7Fv2gl3O6Gr/83X79vV52z1P7WrvvMW2fX16b9py1wbeouUZ7/m7rvYCzt6g7UiSJGnmcKRakiRJGpJJtSRJkjQkk2pJkiRpSCbVkiRJ0pA2q4mKz9n5vL5lu2+5XVf7/977nDGe2T8xcTJ/rDv7ln3q5gd2tf/p3Cf39XnAJ8/vam/YdEdNpyT7Ax8C5gHHVdV7e9a/AfgbYC2wBnhpVV0+7YFKkqSRcaRamkCSecAxwAHAnsAhSfbs6fYTYFlVPQQ4DXjf9EYpSZJGzaRamtjewKqquqSq7gBOBg7q7FBV36mq29vmj4BF0xyjJEkaMZNqaWILgSs62qvbZeN5GfD1sVYkWZ5kZZKVa9as2YghSpKkUdusaqpf9arX9C276q//OOnzvveYj3a158/brq/Pg856cVd7wRf6++x4ane99lL6a7ytod58JXkBsAx44ljrq+pY4FiAZcuWeSpIkrQZ2aySamkTuBJY3NFe1C7rkuTJwN8DT6yqyT/JSZKkzYrlH9LEzgWWJtkjydbAwcCKzg5JHgZ8HDiwqq4dQYySJGnETKqlCVTVWuBQ4Azgl8ApVXVhkiOTHNh2+7/AjsCpSX6aZMU4m5MkSZspyz+kSVTV6cDpPcve3vG4/4LkkiRpTtmskuptvnZu37I9vjb5817M4ybtswc/25CQJEmSNAdY/iFJkiQNyaRakiRJGpJJtSRJkjQkk2pJkiRpSCbVkiRJ0pBMqiVJkqQhmVRLkiRJQzKpliRJkoZkUi1JkiQNyaRakiRJGpJJtSRJkjQkk2pJkiRpSCbVkiRJ0pBMqiVJkqQhmVRLk0iyf5KLk6xKctgY67dJ8vl2/TlJlkx/lJIkaZRMqqUJJJkHHAMcAOwJHJJkz55uLwNurKr7A/8EHDW9UUqSpFEzqZYmtjewqqouqao7gJOBg3r6HASc2D4+DdgvSaYxRkmSNGJbTqXzrdx43bfqtMs3VTCake476gBGbCFwRUd7NfCo8fpU1dokNwO7Atd1dkqyHFjeNm9LcvEmiXjjmU/Pa5gljHt6bbS4M73f8cz54z3NjHt6GfemM25eNKWkuqoWDB+LNDdV1bHAsaOOY1BJVlbVslHHMVXGPb2Me3oZ9/Qy7uk1W+Nez/IPaWJXAos72ovaZWP2SbIlsDNw/bREJ0mSZgSTamli5wJLk+yRZGvgYGBFT58VwIvax88Bvl1VNY0xSpKkEZtS+Yc017Q10ocCZwDzgE9W1YVJjgRWVtUK4HjgM0lWATfQJN6bg1lTqtLDuKeXcU8v455exj29ZmvcAMQBNUmSJGk4ln9IkiRJQzKpliRJkoZkUi2pz2S3Zp+JknwyybVJfjHqWKYiyeIk30lyUZILk7x21DENIsm2SX6c5II27neMOqapSDIvyU+SfHXUsQwqyWVJfp7kp0lWjjqeQSXZJclpSX6V5JdJHj3qmCaT5IHtcV7/c0uS1406rkEkeX37O/mLJCcl2XbUMQ0iyWvbmC+cLce6lzXVkrq0t2b/NfAUmpvdnAscUlUXjTSwSSR5AnAb8Omq2mvU8QwqyX2A+1TV+Ul2As4Dnj0LjneAHarqtiRbAT8AXltVPxpxaANJ8gZgGXC3qnrmqOMZRJLLgGVVNdNvjtElyYnAWVV1XHsVpe2r6qZRxzWo9m/ilcCjqmpG3wAvyUKa38U9q+r3SU4BTq+qE0Yb2cSS7EVzx+K9gTuAbwCvqKpVIw1sihypltRrkFuzzzhV9X2aq6/MKlV1dVWd3z6+FfglzV06Z7Rq3NY2t2p/ZsUoTZJFwDOA40Ydy+Yuyc7AE2iukkRV3TGbEurWfsBvZnpC3WFLYLv2vgnbA1eNOJ5BPAg4p6pur6q1wPeAvxhxTFNmUi2p11i3Zp/xSd7mIMkS4GHAOaONZDBtCcVPgWuBb1bVrIgb+CDwZuCuUQcyRQWcmeS8JMtHHcyA9gDWAJ9qy22OS7LDqIOaooOBk0YdxCCq6krg/cBvgauBm6vqzNFGNZBfAI9PsmuS7YGn033jtVnBpFqSZoAkOwJfAF5XVbeMOp5BVNW6qnoozZ1G926/wp3RkjwTuLaqzht1LBvgcVX1cOAA4FVtydNMtyXwcOBjVfUw4HfArJinAdCWqxwInDrqWAaR5O403yzuAewG7JDkBaONanJV9UvgKOBMmtKPnwLrRhrUBjCpltRrkFuzayNqa5K/AHy2qr446nimqv06/zvA/qOOZQCPBQ5s65NPBp6U5F9HG9Jg2lFIqupa4Es0pVoz3Wpgdce3GKfRJNmzxQHA+VX1X6MOZEBPBi6tqjVVdSfwReAxI45pIFV1fFU9oqqeANxIM7dnVjGpltRrkFuzayNpJ/wdD/yyqo4edTyDSrIgyS7t4+1oJrb+arRRTa6q3lJVi6pqCc25/e2qmvEjeUl2aCey0pZPPJXmK/MZraquAa5I8sB20X7AjJ6E2+MQZknpR+u3wD5Jtm//tuxHM09jxktyz/bf3WnqqT832oimztuUS+oy3q3ZRxzWpJKcBOwLzE+yGji8qo4fbVQDeSzw18DP2/pkgLdW1ekjjGkQ9wFObK+MsAVwSlXNmsvTzUL3Ar7U5ElsCXyuqr4x2pAG9mrgs+2H9EuAl4w4noG0H16eAvztqGMZVFWdk+Q04HxgLfATZs+tv7+QZFfgTuBVs3BCq5fUkyRJkoZl+YckSZI0JJNqSZIkaUgm1ZIkSdKQTKolSZKkIZlUS5IkSUMyqZYkSZKGZFItSZIkDen/Afer/+D2JAAIAAAAAElFTkSuQmCC\n", 640 | "text/plain": [ 641 | "
" 642 | ] 643 | }, 644 | "metadata": { 645 | "tags": [], 646 | "needs_background": "light" 647 | } 648 | } 649 | ] 650 | }, 651 | { 652 | "cell_type": "markdown", 653 | "metadata": { 654 | "id": "_y6mwJLs-gDP" 655 | }, 656 | "source": [ 657 | "Congratulations for completing this programming assignment! In the next week of the course we will take a look at including validation and regularisation in our model training, and introduce Keras callbacks." 658 | ] 659 | } 660 | ] 661 | } -------------------------------------------------------------------------------- /Course 1: Getting started with TensorFlow 2/Week 4 Programming Assignment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Programming Assignment\n", 8 | "\n", 9 | "## Saving and loading models, with application to the EuroSat dataset\n", 10 | "\n", 11 | "### Instructions\n", 12 | "\n", 13 | "In this notebook, you will create a neural network that classifies land uses and land covers from satellite imagery. You will save your model using Tensorflow's callbacks and reload it later. You will also load in a pre-trained neural network classifier and compare performance with it. \n", 14 | "\n", 15 | "Some code cells are provided for you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line: \n", 16 | "\n", 17 | "`#### GRADED CELL ####`\n", 18 | "\n", 19 | "Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly. Inside these graded cells, you can use any functions or classes that are imported below, but make sure you don't use any variables that are outside the scope of the function.\n", 20 | "\n", 21 | "### How to submit\n", 22 | "\n", 23 | "Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.\n", 24 | "\n", 25 | "### Let's get started!\n", 26 | "\n", 27 | "We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here." 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 1, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "#### PACKAGE IMPORTS ####\n", 37 | "\n", 38 | "# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook\n", 39 | "\n", 40 | "import tensorflow as tf\n", 41 | "from tensorflow.keras.preprocessing.image import load_img, img_to_array\n", 42 | "from tensorflow.keras.models import Sequential, load_model\n", 43 | "from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n", 44 | "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n", 45 | "import os\n", 46 | "import numpy as np\n", 47 | "import pandas as pd\n", 48 | "\n", 49 | "# If you would like to make further imports from tensorflow, add them here\n", 50 | "from tensorflow.keras import initializers, regularizers\n" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "![EuroSAT overview image](data/eurosat_overview.jpg)\n", 58 | "\n", 59 | "#### The EuroSAT dataset\n", 60 | "\n", 61 | "In this assignment, you will use the [EuroSAT dataset](https://github.com/phelber/EuroSAT). It consists of 27000 labelled Sentinel-2 satellite images of different land uses: residential, industrial, highway, river, forest, pasture, herbaceous vegetation, annual crop, permanent crop and sea/lake. For a reference, see the following papers:\n", 62 | "- Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.\n", 63 | "- Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. Patrick Helber, Benjamin Bischke, Andreas Dengel. 2018 IEEE International Geoscience and Remote Sensing Symposium, 2018.\n", 64 | "\n", 65 | "Your goal is to construct a neural network that classifies a satellite image into one of these 10 classes, as well as applying some of the saving and loading techniques you have learned in the previous sessions." 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "#### Import the data\n", 73 | "\n", 74 | "The dataset you will train your model on is a subset of the total data, with 4000 training images and 1000 testing images, with roughly equal numbers of each class. The code to import the data is provided below." 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 2, 80 | "metadata": {}, 81 | "outputs": [], 82 | "source": [ 83 | "# Run this cell to import the Eurosat data\n", 84 | "\n", 85 | "def load_eurosat_data():\n", 86 | " data_dir = 'data/'\n", 87 | " x_train = np.load(os.path.join(data_dir, 'x_train.npy'))\n", 88 | " y_train = np.load(os.path.join(data_dir, 'y_train.npy'))\n", 89 | " x_test = np.load(os.path.join(data_dir, 'x_test.npy'))\n", 90 | " y_test = np.load(os.path.join(data_dir, 'y_test.npy'))\n", 91 | " return (x_train, y_train), (x_test, y_test)\n", 92 | "\n", 93 | "(x_train, y_train), (x_test, y_test) = load_eurosat_data()\n", 94 | "x_train = x_train / 255.0\n", 95 | "x_test = x_test / 255.0" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "#### Build the neural network model" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "You can now construct a model to fit to the data. Using the Sequential API, build your model according to the following specifications:\n", 110 | "\n", 111 | "* The model should use the input_shape in the function argument to set the input size in the first layer.\n", 112 | "* The first layer should be a Conv2D layer with 16 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_1'.\n", 113 | "* The second layer should also be a Conv2D layer with 8 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_2'.\n", 114 | "* The third layer should be a MaxPooling2D layer with a pooling window size of 8x8. Name this layer 'pool_1'.\n", 115 | "* The fourth layer should be a Flatten layer, named 'flatten'.\n", 116 | "* The fifth layer should be a Dense layer with 32 units, a ReLU activation. Name this layer 'dense_1'.\n", 117 | "* The sixth and final layer should be a Dense layer with 10 units and softmax activation. Name this layer 'dense_2'.\n", 118 | "\n", 119 | "In total, the network should have 6 layers." 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 3, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "#### GRADED CELL ####\n", 129 | "\n", 130 | "# Complete the following function. \n", 131 | "# Make sure to not change the function name or arguments.\n", 132 | "\n", 133 | "def get_new_model(input_shape):\n", 134 | " \"\"\"\n", 135 | " This function should build a Sequential model according to the above specification. Ensure the \n", 136 | " weights are initialised by providing the input_shape argument in the first layer, given by the\n", 137 | " function argument.\n", 138 | " Your function should also compile the model with the Adam optimiser, sparse categorical cross\n", 139 | " entropy loss function, and a single accuracy metric.\n", 140 | " \"\"\"\n", 141 | " initializer = tf.keras.initializers.he_uniform()\n", 142 | " model = Sequential([\n", 143 | " Conv2D(filters=16, input_shape= input_shape, kernel_size=(3, 3), kernel_initializer=initializer , activation='relu', padding='same', name='conv_1'),\n", 144 | " Conv2D(filters=8, kernel_size=(3, 3),activation='relu', padding='same', name='conv_2'),\n", 145 | " MaxPooling2D(pool_size=(8, 8), name='pool_1'),\n", 146 | " Flatten(name='flatten'),\n", 147 | " Dense(units=32, activation='relu', name='dense_1'),\n", 148 | " Dense(units=10, activation='softmax', name='dense_2') \n", 149 | " ])\n", 150 | " \n", 151 | " model.compile(optimizer='adam',\n", 152 | " loss='sparse_categorical_crossentropy',\n", 153 | " metrics=['accuracy'])\n", 154 | " return model\n", 155 | " " 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "#### Compile and evaluate the model" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 4, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "# Run your function to create the model\n", 172 | "\n", 173 | "model = get_new_model(x_train[0].shape)" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 5, 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "# Run this cell to define a function to evaluate a model's test accuracy\n", 183 | "\n", 184 | "def get_test_accuracy(model, x_test, y_test):\n", 185 | " \"\"\"Test model classification accuracy\"\"\"\n", 186 | " test_loss, test_acc = model.evaluate(x=x_test, y=y_test, verbose=0)\n", 187 | " print('accuracy: {acc:0.3f}'.format(acc=test_acc))" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 6, 193 | "metadata": {}, 194 | "outputs": [ 195 | { 196 | "name": "stdout", 197 | "output_type": "stream", 198 | "text": [ 199 | "Model: \"sequential\"\n", 200 | "_________________________________________________________________\n", 201 | "Layer (type) Output Shape Param # \n", 202 | "=================================================================\n", 203 | "conv_1 (Conv2D) (None, 64, 64, 16) 448 \n", 204 | "_________________________________________________________________\n", 205 | "conv_2 (Conv2D) (None, 64, 64, 8) 1160 \n", 206 | "_________________________________________________________________\n", 207 | "pool_1 (MaxPooling2D) (None, 8, 8, 8) 0 \n", 208 | "_________________________________________________________________\n", 209 | "flatten (Flatten) (None, 512) 0 \n", 210 | "_________________________________________________________________\n", 211 | "dense_1 (Dense) (None, 32) 16416 \n", 212 | "_________________________________________________________________\n", 213 | "dense_2 (Dense) (None, 10) 330 \n", 214 | "=================================================================\n", 215 | "Total params: 18,354\n", 216 | "Trainable params: 18,354\n", 217 | "Non-trainable params: 0\n", 218 | "_________________________________________________________________\n", 219 | "accuracy: 0.105\n" 220 | ] 221 | } 222 | ], 223 | "source": [ 224 | "# Print the model summary and calculate its initialised test accuracy\n", 225 | "\n", 226 | "model.summary()\n", 227 | "get_test_accuracy(model, x_test, y_test)" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "#### Create checkpoints to save model during training, with a criterion\n", 235 | "\n", 236 | "You will now create three callbacks:\n", 237 | "- `checkpoint_every_epoch`: checkpoint that saves the model weights every epoch during training\n", 238 | "- `checkpoint_best_only`: checkpoint that saves only the weights with the highest validation accuracy. Use the testing data as the validation data.\n", 239 | "- `early_stopping`: early stopping object that ends training if the validation accuracy has not improved in 3 epochs." 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": 7, 245 | "metadata": {}, 246 | "outputs": [], 247 | "source": [ 248 | "#### GRADED CELL ####\n", 249 | "\n", 250 | "# Complete the following functions. \n", 251 | "# Make sure to not change the function names or arguments.\n", 252 | "\n", 253 | "def get_checkpoint_every_epoch():\n", 254 | " \"\"\"\n", 255 | " This function should return a ModelCheckpoint object that:\n", 256 | " - saves the weights only at the end of every epoch\n", 257 | " - saves into a directory called 'checkpoints_every_epoch' inside the current working directory\n", 258 | " - generates filenames in that directory like 'checkpoint_XXX' where\n", 259 | " XXX is the epoch number formatted to have three digits, e.g. 001, 002, 003, etc.\n", 260 | " \"\"\"\n", 261 | " checkpoint_path = 'checkpoints_every_epoch/checkpoint_{epoch:03d}'\n", 262 | " checkpoint = ModelCheckpoint(filepath = checkpoint_path, \n", 263 | " frequency = 'epoch', \n", 264 | " save_weights_only= True, \n", 265 | " verbose = 1)\n", 266 | " return checkpoint\n", 267 | " \n", 268 | " \n", 269 | "\n", 270 | "\n", 271 | "def get_checkpoint_best_only():\n", 272 | " \"\"\"\n", 273 | " This function should return a ModelCheckpoint object that:\n", 274 | " - saves only the weights that generate the highest validation (testing) accuracy\n", 275 | " - saves into a directory called 'checkpoints_best_only' inside the current working directory\n", 276 | " - generates a file called 'checkpoints_best_only/checkpoint' \n", 277 | " \"\"\"\n", 278 | " checkpoint_path = 'checkpoints_best_only/checkpoint'\n", 279 | " checkpoint = ModelCheckpoint(filepath = checkpoint_path, save_weights_only=True,\n", 280 | " save_freq= \"epoch\",\n", 281 | " monitor = \"val_accuracy\",\n", 282 | " save_best_only = True,\n", 283 | " verbose = 1)\n", 284 | " return checkpoint\n", 285 | " \n", 286 | " " 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": 8, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "#### GRADED CELL ####\n", 296 | "\n", 297 | "# Complete the following function. \n", 298 | "# Make sure to not change the function name or arguments.\n", 299 | "\n", 300 | "def get_early_stopping():\n", 301 | " \"\"\"\n", 302 | " This function should return an EarlyStopping callback that stops training when\n", 303 | " the validation (testing) accuracy has not improved in the last 3 epochs.\n", 304 | " HINT: use the EarlyStopping callback with the correct 'monitor' and 'patience'\n", 305 | " \"\"\"\n", 306 | " early_stopping = EarlyStopping(monitor='val_accuracy', patience = 3, mode='max')\n", 307 | " \n", 308 | " return early_stopping\n", 309 | " " 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 9, 315 | "metadata": {}, 316 | "outputs": [], 317 | "source": [ 318 | "# Run this cell to create the callbacks\n", 319 | "\n", 320 | "checkpoint_every_epoch = get_checkpoint_every_epoch()\n", 321 | "checkpoint_best_only = get_checkpoint_best_only()\n", 322 | "early_stopping = get_early_stopping()" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "#### Train model using the callbacks\n", 330 | "\n", 331 | "Now, you will train the model using the three callbacks you created. If you created the callbacks correctly, three things should happen:\n", 332 | "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_every_epoch`\n", 333 | "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_best_only` **only** if those weights lead to the highest test accuracy\n", 334 | "- Training stops when the testing accuracy has not improved in three epochs.\n", 335 | "\n", 336 | "You should then have two directories:\n", 337 | "- A directory called `checkpoints_every_epoch` containing filenames that include `checkpoint_001`, `checkpoint_002`, etc with the `001`, `002` corresponding to the epoch\n", 338 | "- A directory called `checkpoints_best_only` containing filenames that include `checkpoint`, which contain only the weights leading to the highest testing accuracy" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": 10, 344 | "metadata": {}, 345 | "outputs": [ 346 | { 347 | "name": "stdout", 348 | "output_type": "stream", 349 | "text": [ 350 | "Train on 4000 samples, validate on 1000 samples\n", 351 | "Epoch 1/50\n", 352 | "3968/4000 [============================>.] - ETA: 0s - loss: 1.9475 - accuracy: 0.2437\n", 353 | "Epoch 00001: saving model to checkpoints_every_epoch/checkpoint_001\n", 354 | "\n", 355 | "Epoch 00001: val_accuracy improved from -inf to 0.35100, saving model to checkpoints_best_only/checkpoint\n", 356 | "4000/4000 [==============================] - 79s 20ms/sample - loss: 1.9451 - accuracy: 0.2447 - val_loss: 1.6925 - val_accuracy: 0.3510\n", 357 | "Epoch 2/50\n", 358 | "3968/4000 [============================>.] - ETA: 0s - loss: 1.4209 - accuracy: 0.4688\n", 359 | "Epoch 00002: saving model to checkpoints_every_epoch/checkpoint_002\n", 360 | "\n", 361 | "Epoch 00002: val_accuracy improved from 0.35100 to 0.47400, saving model to checkpoints_best_only/checkpoint\n", 362 | "4000/4000 [==============================] - 77s 19ms/sample - loss: 1.4200 - accuracy: 0.4688 - val_loss: 1.3441 - val_accuracy: 0.4740\n", 363 | "Epoch 3/50\n", 364 | "3968/4000 [============================>.] - ETA: 0s - loss: 1.2414 - accuracy: 0.5454\n", 365 | "Epoch 00003: saving model to checkpoints_every_epoch/checkpoint_003\n", 366 | "\n", 367 | "Epoch 00003: val_accuracy improved from 0.47400 to 0.52900, saving model to checkpoints_best_only/checkpoint\n", 368 | "4000/4000 [==============================] - 77s 19ms/sample - loss: 1.2396 - accuracy: 0.5460 - val_loss: 1.2553 - val_accuracy: 0.5290\n", 369 | "Epoch 4/50\n", 370 | "3968/4000 [============================>.] - ETA: 0s - loss: 1.1699 - accuracy: 0.5680\n", 371 | "Epoch 00004: saving model to checkpoints_every_epoch/checkpoint_004\n", 372 | "\n", 373 | "Epoch 00004: val_accuracy improved from 0.52900 to 0.57100, saving model to checkpoints_best_only/checkpoint\n", 374 | "4000/4000 [==============================] - 74s 19ms/sample - loss: 1.1686 - accuracy: 0.5680 - val_loss: 1.1740 - val_accuracy: 0.5710\n", 375 | "Epoch 5/50\n", 376 | "3968/4000 [============================>.] - ETA: 0s - loss: 1.0576 - accuracy: 0.6187\n", 377 | "Epoch 00005: saving model to checkpoints_every_epoch/checkpoint_005\n", 378 | "\n", 379 | "Epoch 00005: val_accuracy improved from 0.57100 to 0.60800, saving model to checkpoints_best_only/checkpoint\n", 380 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 1.0606 - accuracy: 0.6180 - val_loss: 1.1063 - val_accuracy: 0.6080\n", 381 | "Epoch 6/50\n", 382 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.9814 - accuracy: 0.6507\n", 383 | "Epoch 00006: saving model to checkpoints_every_epoch/checkpoint_006\n", 384 | "\n", 385 | "Epoch 00006: val_accuracy improved from 0.60800 to 0.61100, saving model to checkpoints_best_only/checkpoint\n", 386 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.9795 - accuracy: 0.6515 - val_loss: 1.0476 - val_accuracy: 0.6110\n", 387 | "Epoch 7/50\n", 388 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.9328 - accuracy: 0.6600\n", 389 | "Epoch 00007: saving model to checkpoints_every_epoch/checkpoint_007\n", 390 | "\n", 391 | "Epoch 00007: val_accuracy improved from 0.61100 to 0.62400, saving model to checkpoints_best_only/checkpoint\n", 392 | "4000/4000 [==============================] - 74s 18ms/sample - loss: 0.9359 - accuracy: 0.6593 - val_loss: 1.0400 - val_accuracy: 0.6240\n", 393 | "Epoch 8/50\n", 394 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.8789 - accuracy: 0.6809\n", 395 | "Epoch 00008: saving model to checkpoints_every_epoch/checkpoint_008\n", 396 | "\n", 397 | "Epoch 00008: val_accuracy improved from 0.62400 to 0.64000, saving model to checkpoints_best_only/checkpoint\n", 398 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.8788 - accuracy: 0.6808 - val_loss: 0.9899 - val_accuracy: 0.6400\n", 399 | "Epoch 9/50\n", 400 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.8470 - accuracy: 0.7004\n", 401 | "Epoch 00009: saving model to checkpoints_every_epoch/checkpoint_009\n", 402 | "\n", 403 | "Epoch 00009: val_accuracy improved from 0.64000 to 0.65300, saving model to checkpoints_best_only/checkpoint\n", 404 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.8461 - accuracy: 0.7005 - val_loss: 0.9651 - val_accuracy: 0.6530\n", 405 | "Epoch 10/50\n", 406 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.8084 - accuracy: 0.7160\n", 407 | "Epoch 00010: saving model to checkpoints_every_epoch/checkpoint_010\n", 408 | "\n", 409 | "Epoch 00010: val_accuracy improved from 0.65300 to 0.67400, saving model to checkpoints_best_only/checkpoint\n", 410 | "4000/4000 [==============================] - 74s 19ms/sample - loss: 0.8086 - accuracy: 0.7150 - val_loss: 0.9469 - val_accuracy: 0.6740\n", 411 | "Epoch 11/50\n", 412 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.7830 - accuracy: 0.7243\n", 413 | "Epoch 00011: saving model to checkpoints_every_epoch/checkpoint_011\n", 414 | "\n", 415 | "Epoch 00011: val_accuracy did not improve from 0.67400\n", 416 | "4000/4000 [==============================] - 74s 19ms/sample - loss: 0.7828 - accuracy: 0.7237 - val_loss: 0.9371 - val_accuracy: 0.6450\n", 417 | "Epoch 12/50\n", 418 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.7553 - accuracy: 0.7271\n", 419 | "Epoch 00012: saving model to checkpoints_every_epoch/checkpoint_012\n", 420 | "\n", 421 | "Epoch 00012: val_accuracy did not improve from 0.67400\n", 422 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.7552 - accuracy: 0.7270 - val_loss: 0.9189 - val_accuracy: 0.6700\n", 423 | "Epoch 13/50\n", 424 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.7273 - accuracy: 0.7351\n", 425 | "Epoch 00013: saving model to checkpoints_every_epoch/checkpoint_013\n", 426 | "\n", 427 | "Epoch 00013: val_accuracy improved from 0.67400 to 0.69400, saving model to checkpoints_best_only/checkpoint\n", 428 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.7260 - accuracy: 0.7358 - val_loss: 0.8506 - val_accuracy: 0.6940\n", 429 | "Epoch 14/50\n", 430 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6882 - accuracy: 0.7573\n", 431 | "Epoch 00014: saving model to checkpoints_every_epoch/checkpoint_014\n", 432 | "\n", 433 | "Epoch 00014: val_accuracy did not improve from 0.69400\n", 434 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.6865 - accuracy: 0.7575 - val_loss: 0.8378 - val_accuracy: 0.6890\n", 435 | "Epoch 15/50\n", 436 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6728 - accuracy: 0.7553\n", 437 | "Epoch 00015: saving model to checkpoints_every_epoch/checkpoint_015\n", 438 | "\n", 439 | "Epoch 00015: val_accuracy did not improve from 0.69400\n", 440 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.6731 - accuracy: 0.7552 - val_loss: 0.8680 - val_accuracy: 0.6840\n", 441 | "Epoch 16/50\n", 442 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6485 - accuracy: 0.7686\n", 443 | "Epoch 00016: saving model to checkpoints_every_epoch/checkpoint_016\n", 444 | "\n", 445 | "Epoch 00016: val_accuracy improved from 0.69400 to 0.71900, saving model to checkpoints_best_only/checkpoint\n", 446 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.6481 - accuracy: 0.7682 - val_loss: 0.7914 - val_accuracy: 0.7190\n", 447 | "Epoch 17/50\n", 448 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6467 - accuracy: 0.7714\n", 449 | "Epoch 00017: saving model to checkpoints_every_epoch/checkpoint_017\n", 450 | "\n", 451 | "Epoch 00017: val_accuracy did not improve from 0.71900\n", 452 | "4000/4000 [==============================] - 74s 19ms/sample - loss: 0.6453 - accuracy: 0.7722 - val_loss: 0.7929 - val_accuracy: 0.7140\n", 453 | "Epoch 18/50\n", 454 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6251 - accuracy: 0.7732\n", 455 | "Epoch 00018: saving model to checkpoints_every_epoch/checkpoint_018\n", 456 | "\n", 457 | "Epoch 00018: val_accuracy did not improve from 0.71900\n", 458 | "4000/4000 [==============================] - 75s 19ms/sample - loss: 0.6263 - accuracy: 0.7728 - val_loss: 0.7847 - val_accuracy: 0.7160\n", 459 | "Epoch 19/50\n", 460 | "3968/4000 [============================>.] - ETA: 0s - loss: 0.6067 - accuracy: 0.7805\n", 461 | "Epoch 00019: saving model to checkpoints_every_epoch/checkpoint_019\n", 462 | "\n", 463 | "Epoch 00019: val_accuracy did not improve from 0.71900\n", 464 | "4000/4000 [==============================] - 73s 18ms/sample - loss: 0.6058 - accuracy: 0.7810 - val_loss: 0.9287 - val_accuracy: 0.6890\n" 465 | ] 466 | }, 467 | { 468 | "data": { 469 | "text/plain": [ 470 | "" 471 | ] 472 | }, 473 | "execution_count": 10, 474 | "metadata": {}, 475 | "output_type": "execute_result" 476 | } 477 | ], 478 | "source": [ 479 | "# Train model using the callbacks you just created\n", 480 | "\n", 481 | "callbacks = [checkpoint_every_epoch, checkpoint_best_only, early_stopping]\n", 482 | "model.fit(x_train, y_train, epochs=50, validation_data=(x_test, y_test), callbacks=callbacks)" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "metadata": {}, 488 | "source": [ 489 | "#### Create new instance of model and load on both sets of weights\n", 490 | "\n", 491 | "Now you will use the weights you just saved in a fresh model. You should create two functions, both of which take a freshly instantiated model instance:\n", 492 | "- `model_last_epoch` should contain the weights from the latest saved epoch\n", 493 | "- `model_best_epoch` should contain the weights from the saved epoch with the highest testing accuracy\n", 494 | "\n", 495 | "_Hint: use the_ `tf.train.latest_checkpoint` _function to get the filename of the latest saved checkpoint file. Check the docs_ [_here_](https://www.tensorflow.org/api_docs/python/tf/train/latest_checkpoint)." 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": 13, 501 | "metadata": {}, 502 | "outputs": [], 503 | "source": [ 504 | "#### GRADED CELL ####\n", 505 | "\n", 506 | "# Complete the following functions. \n", 507 | "# Make sure to not change the function name or arguments.\n", 508 | "\n", 509 | "def get_model_last_epoch(model):\n", 510 | " \"\"\"\n", 511 | " This function should create a new instance of the CNN you created earlier,\n", 512 | " load on the weights from the last training epoch, and return this model.\n", 513 | " \"\"\"\n", 514 | " model.load_weights(tf.train.latest_checkpoint(\"checkpoints_every_epoch\"))\n", 515 | " return model\n", 516 | " \n", 517 | " \n", 518 | "def get_model_best_epoch(model):\n", 519 | " \"\"\"\n", 520 | " This function should create a new instance of the CNN you created earlier, load \n", 521 | " on the weights leading to the highest validation accuracy, and return this model.\n", 522 | " \"\"\"\n", 523 | " model.load_weights(tf.train.latest_checkpoint(\"checkpoints_best_only\"))\n", 524 | " return model\n", 525 | " " 526 | ] 527 | }, 528 | { 529 | "cell_type": "code", 530 | "execution_count": 14, 531 | "metadata": {}, 532 | "outputs": [ 533 | { 534 | "name": "stdout", 535 | "output_type": "stream", 536 | "text": [ 537 | "Model with last epoch weights:\n", 538 | "accuracy: 0.689\n", 539 | "\n", 540 | "Model with best epoch weights:\n", 541 | "accuracy: 0.719\n" 542 | ] 543 | } 544 | ], 545 | "source": [ 546 | "# Run this cell to create two models: one with the weights from the last training\n", 547 | "# epoch, and one with the weights leading to the highest validation (testing) accuracy.\n", 548 | "# Verify that the second has a higher validation (testing) accuarcy.\n", 549 | "\n", 550 | "model_last_epoch = get_model_last_epoch(get_new_model(x_train[0].shape))\n", 551 | "model_best_epoch = get_model_best_epoch(get_new_model(x_train[0].shape))\n", 552 | "print('Model with last epoch weights:')\n", 553 | "get_test_accuracy(model_last_epoch, x_test, y_test)\n", 554 | "print('')\n", 555 | "print('Model with best epoch weights:')\n", 556 | "get_test_accuracy(model_best_epoch, x_test, y_test)" 557 | ] 558 | }, 559 | { 560 | "cell_type": "markdown", 561 | "metadata": {}, 562 | "source": [ 563 | "#### Load, from scratch, a model trained on the EuroSat dataset.\n", 564 | "\n", 565 | "In your workspace, you will find another model trained on the `EuroSAT` dataset in `.h5` format. This model is trained on a larger subset of the EuroSAT dataset and has a more complex architecture. The path to the model is `models/EuroSatNet.h5`. See how its testing accuracy compares to your model!" 566 | ] 567 | }, 568 | { 569 | "cell_type": "code", 570 | "execution_count": 15, 571 | "metadata": {}, 572 | "outputs": [], 573 | "source": [ 574 | "#### GRADED CELL ####\n", 575 | "\n", 576 | "# Complete the following functions. \n", 577 | "# Make sure to not change the function name or arguments.\n", 578 | "\n", 579 | "def get_model_eurosatnet():\n", 580 | " \"\"\"\n", 581 | " This function should return the pretrained EuroSatNet.h5 model.\n", 582 | " \"\"\"\n", 583 | " model = load_model(\"models/EuroSatNet.h5\")\n", 584 | " return model\n", 585 | " " 586 | ] 587 | }, 588 | { 589 | "cell_type": "code", 590 | "execution_count": 16, 591 | "metadata": {}, 592 | "outputs": [ 593 | { 594 | "name": "stdout", 595 | "output_type": "stream", 596 | "text": [ 597 | "Model: \"sequential_21\"\n", 598 | "_________________________________________________________________\n", 599 | "Layer (type) Output Shape Param # \n", 600 | "=================================================================\n", 601 | "conv_1 (Conv2D) (None, 64, 64, 16) 448 \n", 602 | "_________________________________________________________________\n", 603 | "conv_2 (Conv2D) (None, 64, 64, 16) 6416 \n", 604 | "_________________________________________________________________\n", 605 | "pool_1 (MaxPooling2D) (None, 32, 32, 16) 0 \n", 606 | "_________________________________________________________________\n", 607 | "conv_3 (Conv2D) (None, 32, 32, 16) 2320 \n", 608 | "_________________________________________________________________\n", 609 | "conv_4 (Conv2D) (None, 32, 32, 16) 6416 \n", 610 | "_________________________________________________________________\n", 611 | "pool_2 (MaxPooling2D) (None, 16, 16, 16) 0 \n", 612 | "_________________________________________________________________\n", 613 | "conv_5 (Conv2D) (None, 16, 16, 16) 2320 \n", 614 | "_________________________________________________________________\n", 615 | "conv_6 (Conv2D) (None, 16, 16, 16) 6416 \n", 616 | "_________________________________________________________________\n", 617 | "pool_3 (MaxPooling2D) (None, 8, 8, 16) 0 \n", 618 | "_________________________________________________________________\n", 619 | "conv_7 (Conv2D) (None, 8, 8, 16) 2320 \n", 620 | "_________________________________________________________________\n", 621 | "conv_8 (Conv2D) (None, 8, 8, 16) 6416 \n", 622 | "_________________________________________________________________\n", 623 | "pool_4 (MaxPooling2D) (None, 4, 4, 16) 0 \n", 624 | "_________________________________________________________________\n", 625 | "flatten (Flatten) (None, 256) 0 \n", 626 | "_________________________________________________________________\n", 627 | "dense_1 (Dense) (None, 32) 8224 \n", 628 | "_________________________________________________________________\n", 629 | "dense_2 (Dense) (None, 10) 330 \n", 630 | "=================================================================\n", 631 | "Total params: 41,626\n", 632 | "Trainable params: 41,626\n", 633 | "Non-trainable params: 0\n", 634 | "_________________________________________________________________\n", 635 | "accuracy: 0.810\n" 636 | ] 637 | } 638 | ], 639 | "source": [ 640 | "# Run this cell to print a summary of the EuroSatNet model, along with its validation accuracy.\n", 641 | "\n", 642 | "model_eurosatnet = get_model_eurosatnet()\n", 643 | "model_eurosatnet.summary()\n", 644 | "get_test_accuracy(model_eurosatnet, x_test, y_test)" 645 | ] 646 | }, 647 | { 648 | "cell_type": "markdown", 649 | "metadata": {}, 650 | "source": [ 651 | "Congratulations for completing this programming assignment! You're now ready to move on to the capstone project for this course." 652 | ] 653 | } 654 | ], 655 | "metadata": { 656 | "coursera": { 657 | "course_slug": "tensor-flow-2-1", 658 | "graded_item_id": "JaRY0", 659 | "launcher_item_id": "mJ8fg" 660 | }, 661 | "kernelspec": { 662 | "display_name": "Python 3", 663 | "language": "python", 664 | "name": "python3" 665 | }, 666 | "language_info": { 667 | "codemirror_mode": { 668 | "name": "ipython", 669 | "version": 3 670 | }, 671 | "file_extension": ".py", 672 | "mimetype": "text/x-python", 673 | "name": "python", 674 | "nbconvert_exporter": "python", 675 | "pygments_lexer": "ipython3", 676 | "version": "3.7.1" 677 | } 678 | }, 679 | "nbformat": 4, 680 | "nbformat_minor": 2 681 | } 682 | -------------------------------------------------------------------------------- /Course 2: Customising your models with TensorFlow 2/Week 3 Programming Assignment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Programming Assignment" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Language model for the Shakespeare dataset" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "### Instructions\n", 22 | "\n", 23 | "In this notebook, you will use the text preprocessing tools and RNN models to build a character-level language model. You will then train your model on the works of Shakespeare, and use the network to generate your own text..\n", 24 | "\n", 25 | "Some code cells are provided you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line: \n", 26 | "\n", 27 | "`#### GRADED CELL ####`\n", 28 | "\n", 29 | "Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly. Inside these graded cells, you can use any functions or classes that are imported below, but make sure you don't use any variables that are outside the scope of the function.\n", 30 | "\n", 31 | "### How to submit\n", 32 | "\n", 33 | "Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.\n", 34 | "\n", 35 | "### Let's get started!\n", 36 | "\n", 37 | "We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here." 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 1, 43 | "metadata": {}, 44 | "outputs": [], 45 | "source": [ 46 | "#### PACKAGE IMPORTS ####\n", 47 | "\n", 48 | "# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook\n", 49 | "\n", 50 | "import tensorflow as tf\n", 51 | "import numpy as np\n", 52 | "import json\n", 53 | "import matplotlib.pyplot as plt\n", 54 | "%matplotlib inline\n", 55 | "\n", 56 | "# If you would like to make further imports from tensorflow, add them here\n", 57 | "\n" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "![Shakespeare image](data/shakespeare.png)\n", 65 | "\n", 66 | "#### The Shakespeare dataset\n", 67 | "\n", 68 | "In this assignment, you will use a subset of the [Shakespeare dataset](http://shakespeare.mit.edu). It consists of a single text file with several excerpts concatenated together. The data is in raw text form, and so far has not yet had any preprocessing. \n", 69 | "\n", 70 | "Your goal is to construct an unsupervised character-level sequence model that can generate text according to a distribution learned from the dataset." 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "#### Load and inspect the dataset" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 2, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "# Load the text file into a string\n", 87 | "\n", 88 | "with open('data/Shakespeare.txt', 'r', encoding='utf-8') as file:\n", 89 | " text = file.read()" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 3, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "# Create a list of chunks of text\n", 99 | "\n", 100 | "text_chunks = text.split('.')" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "To give you a feel for what the text looks like, we will print a few chunks from the list." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 4, 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "name": "stdout", 117 | "output_type": "stream", 118 | "text": [ 119 | "\n", 120 | "\n", 121 | "KING RICHARD III:\n", 122 | "Shall we hear from thee, Tyrrel, ere we sleep?\n", 123 | "\n", 124 | "TYRREL:\n", 125 | "Ye shall, my Lord\n", 126 | "\n", 127 | "\n", 128 | "GREMIO:\n", 129 | "Let me entreat you\n", 130 | "\n", 131 | "\n", 132 | "POLIXENES:\n", 133 | "Wherefore, gentle maiden,\n", 134 | "Do you neglect them?\n", 135 | "\n", 136 | "PERDITA:\n", 137 | "For I have heard it said\n", 138 | "There is an art which in their piedness shares\n", 139 | "With great creating nature\n", 140 | "\n", 141 | "Most mighty prince, my Lord Northumberland,\n", 142 | "What says King Bolingbroke? will his majesty\n", 143 | "Give Richard leave to live till Richard die?\n", 144 | "You make a leg, and Bolingbroke says ay\n", 145 | "\n", 146 | "\n", 147 | "GLOUCESTER:\n", 148 | "Sirs, take up the corse\n" 149 | ] 150 | } 151 | ], 152 | "source": [ 153 | "# Display some randomly selected text samples\n", 154 | "\n", 155 | "num_samples = 5\n", 156 | "inx = np.random.choice(len(text_chunks), num_samples, replace=False)\n", 157 | "for chunk in np.array(text_chunks)[inx]:\n", 158 | " print(chunk)" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "#### Create a character-level tokenizer" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "You should now write a function that returns a `Tokenizer` object. The function takes a list of strings as an argument, and should create a `Tokenizer` according to the following specification:\n", 173 | "\n", 174 | "* The number of tokens should be unlimited (there should be as many as required by the dataset).\n", 175 | "* Tokens should be created at the character level (not at the word level, which is the default behaviour).\n", 176 | "* No characters should be filtered out or ignored.\n", 177 | "* The original capitalization should be retained (do not convert the text to lower case)\n", 178 | "\n", 179 | "The `Tokenizer` should be fit to the `list_of_strings` argument and returned by the function. \n", 180 | "\n", 181 | "**Hint:** you may need to refer to the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer) for the `Tokenizer`." 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 5, 187 | "metadata": {}, 188 | "outputs": [], 189 | "source": [ 190 | "#### GRADED CELL ####\n", 191 | "\n", 192 | "# Complete the following function.\n", 193 | "# Make sure not to change the function name or arguments.\n", 194 | "\n", 195 | "def create_character_tokenizer(list_of_strings):\n", 196 | " \"\"\"\n", 197 | " This function takes a list of strings as its argument. It should create \n", 198 | " and return a Tokenizer according to the above specifications. \n", 199 | " \"\"\"\n", 200 | " tokenizer = tf.keras.preprocessing.text.Tokenizer(char_level=True, filters=None, lower=False)\n", 201 | " tokenizer.fit_on_texts(list_of_strings)\n", 202 | " return tokenizer" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 6, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [ 211 | "# Get the tokenizer\n", 212 | "\n", 213 | "tokenizer = create_character_tokenizer(text_chunks)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "#### Tokenize the text\n", 221 | "\n", 222 | "You should now write a function to use the tokenizer to map each string in `text_chunks` to its corresponding encoded sequence. The following function takes a fitted `Tokenizer` object in the first argument (as returned by `create_character_tokenizer`) and a list of strings in the second argument. The function should return a list of lists, where each sublist is a sequence of integer tokens encoding the text sequences according to the mapping stored in the tokenizer.\n", 223 | "\n", 224 | "**Hint:** you may need to refer to the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer) for the `Tokenizer`." 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 7, 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "#### GRADED CELL ####\n", 234 | "\n", 235 | "# Complete the following function.\n", 236 | "# Make sure not to change the function name or arguments.\n", 237 | "\n", 238 | "def strings_to_sequences(tokenizer, list_of_strings):\n", 239 | " \"\"\"\n", 240 | " This function takes a tokenizer object and a list of strings as its arguments.\n", 241 | " It should use the tokenizer to map the text chunks to sequences of tokens and\n", 242 | " then return this list of encoded sequences.\n", 243 | " \"\"\"\n", 244 | " seq = tokenizer.texts_to_sequences(list_of_strings)\n", 245 | " return seq" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": 8, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "# Encode the text chunks into tokens\n", 255 | "\n", 256 | "seq_chunks = strings_to_sequences(tokenizer, text_chunks)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "#### Pad the encoded sequences and store them in a numpy array" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "Since not all of the text chunks are the same length, you will need to pad them in order to train on batches. You should now complete the following function, which takes the list of lists of tokens, and creates a single numpy array with the token sequences in the rows, according to the following specification:\n", 271 | "\n", 272 | "* The longest allowed sequence should be 500 tokens. Any sequence that is longer should be shortened by truncating the beginning of the sequence.\n", 273 | "* Use zeros for padding the sequences. The zero padding should be placed before the sequences as required.\n", 274 | "\n", 275 | "The function should then return the resulting numpy array.\n", 276 | "\n", 277 | "**Hint:** you may want to refer to the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/sequence/pad_sequences) for the `pad_sequences` function." 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 9, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "#### GRADED CELL ####\n", 287 | "\n", 288 | "# Complete the following function.\n", 289 | "# Make sure not to change the function name or arguments.\n", 290 | "\n", 291 | "def make_padded_dataset(sequence_chunks):\n", 292 | " \"\"\"\n", 293 | " This function takes a list of lists of tokenized sequences, and transforms\n", 294 | " them into a 2D numpy array, padding the sequences as necessary according to\n", 295 | " the above specification. The function should then return the numpy array.\n", 296 | " \"\"\"\n", 297 | " padded = tf.keras.preprocessing.sequence.pad_sequences(sequence_chunks, maxlen=500,\n", 298 | " padding='pre', truncating='pre',\n", 299 | " value=0)\n", 300 | " return padded" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": 10, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "# Pad the token sequence chunks and get the numpy array\n", 310 | "\n", 311 | "padded_sequences = make_padded_dataset(seq_chunks)" 312 | ] 313 | }, 314 | { 315 | "cell_type": "markdown", 316 | "metadata": {}, 317 | "source": [ 318 | "#### Create model inputs and targets\n", 319 | "\n", 320 | "Now you are ready to build your RNN model. The model will receive a sequence of characters and predict the next character in the sequence. At training time, the model can be passed an input sequence, with the target sequence is shifted by one.\n", 321 | "\n", 322 | "For example, the expression `To be or not to be` appears in Shakespeare's play 'Hamlet'. Given input `To be or not to b`, the correct prediction is `o be or not to be`. Notice that the prediction is the same length as the input!\n", 323 | "\n", 324 | "![sequence_prediction_example](data/rnn_example.png)\n", 325 | "\n", 326 | "You should now write the following function to create an input and target array from the current `padded_sequences` array. The function has a single argument that is a 2D numpy array of shape `(num_examples, max_seq_len)`. It should fulfil the following specification:\n", 327 | "\n", 328 | "* The function should return an input array and an output array, both of size `(num_examples, max_seq_len - 1)`.\n", 329 | "* The input array should contain the first `max_seq_len - 1` tokens of each sequence. \n", 330 | "* The output array should contain the last `max_seq_len - 1` tokens of each sequence. \n", 331 | "\n", 332 | "The function should then return the tuple `(input_array, output_array)`. Note that it is possible to complete this function using numpy indexing alone!" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 11, 338 | "metadata": {}, 339 | "outputs": [], 340 | "source": [ 341 | "#### GRADED CELL ####\n", 342 | "\n", 343 | "# Complete the following function.\n", 344 | "# Make sure not to change the function name or arguments.\n", 345 | "\n", 346 | "def create_inputs_and_targets(array_of_sequences):\n", 347 | " \"\"\"\n", 348 | " This function takes a 2D numpy array of token sequences, and returns a tuple of two\n", 349 | " elements: the first element is the input array and the second element is the output\n", 350 | " array, which are defined according to the above specification.\n", 351 | " \"\"\"\n", 352 | " num_examples, max_seq_len = array_of_sequences.shape\n", 353 | " x = array_of_sequences[:, :-1]\n", 354 | " y = array_of_sequences[:, 1:]\n", 355 | " return x, y" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": 12, 361 | "metadata": {}, 362 | "outputs": [], 363 | "source": [ 364 | "# Create the input and output arrays\n", 365 | "\n", 366 | "input_seq, target_seq = create_inputs_and_targets(padded_sequences)" 367 | ] 368 | }, 369 | { 370 | "cell_type": "markdown", 371 | "metadata": {}, 372 | "source": [ 373 | "#### Preprocess sequence array for stateful RNN\n", 374 | "\n", 375 | "We will build our RNN language model to be stateful, so that the internal state of the RNN will be maintained across batches. For this to be effective, we need to make sure that each element of every batch follows on from the corresponding element of the preceding batch (you may want to look back at the \"Stateful RNNs\" reading notebook earlier in the week).\n", 376 | "\n", 377 | "The following code processes the input and output sequence arrays so that they are ready to be split into batches for training a stateful RNN, by re-ordering the sequence examples (the rows) according to a specified batch size. " 378 | ] 379 | }, 380 | { 381 | "cell_type": "code", 382 | "execution_count": 13, 383 | "metadata": {}, 384 | "outputs": [], 385 | "source": [ 386 | "# Fix the batch size for training\n", 387 | "\n", 388 | "batch_size = 32" 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "execution_count": 14, 394 | "metadata": {}, 395 | "outputs": [], 396 | "source": [ 397 | "# Prepare input and output arrays for training the stateful RNN\n", 398 | "\n", 399 | "num_examples = input_seq.shape[0]\n", 400 | "\n", 401 | "num_processed_examples = num_examples - (num_examples % batch_size)\n", 402 | "\n", 403 | "input_seq = input_seq[:num_processed_examples]\n", 404 | "target_seq = target_seq[:num_processed_examples]\n", 405 | "\n", 406 | "steps = int(num_processed_examples / 32) # steps per epoch\n", 407 | "\n", 408 | "inx = np.empty((0,), dtype=np.int32)\n", 409 | "for i in range(steps):\n", 410 | " inx = np.concatenate((inx, i + np.arange(0, num_processed_examples, steps)))\n", 411 | "\n", 412 | "input_seq_stateful = input_seq[inx]\n", 413 | "target_seq_stateful = target_seq[inx]" 414 | ] 415 | }, 416 | { 417 | "cell_type": "markdown", 418 | "metadata": {}, 419 | "source": [ 420 | "#### Split the data into training and validation sets\n", 421 | "\n", 422 | "We will set aside approximately 20% of the data for validation." 423 | ] 424 | }, 425 | { 426 | "cell_type": "code", 427 | "execution_count": 15, 428 | "metadata": {}, 429 | "outputs": [], 430 | "source": [ 431 | "# Create the training and validation splits\n", 432 | "\n", 433 | "num_train_examples = int(batch_size * ((0.8 * num_processed_examples) // batch_size))\n", 434 | "\n", 435 | "input_train = input_seq_stateful[:num_train_examples]\n", 436 | "target_train = target_seq_stateful[:num_train_examples]\n", 437 | "\n", 438 | "input_valid = input_seq_stateful[num_train_examples:]\n", 439 | "target_valid = target_seq_stateful[num_train_examples:]" 440 | ] 441 | }, 442 | { 443 | "cell_type": "markdown", 444 | "metadata": {}, 445 | "source": [ 446 | "#### Create training and validation Dataset objects\n", 447 | "\n", 448 | "You should now write a function to take the training and validation input and target arrays, and create training and validation `tf.data.Dataset` objects. The function takes an input array and target array in the first two arguments, and the batch size in the third argument. Your function should do the following:\n", 449 | "\n", 450 | "* Create a `Dataset` using the `from_tensor_slices` static method, passing in a tuple of the input and output numpy arrays.\n", 451 | "* Batch the `Dataset` using the `batch_size` argument, setting `drop_remainder` to `True`. \n", 452 | "\n", 453 | "The function should then return the `Dataset` object." 454 | ] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "execution_count": 16, 459 | "metadata": {}, 460 | "outputs": [], 461 | "source": [ 462 | "#### GRADED CELL ####\n", 463 | "\n", 464 | "# Complete the following function.\n", 465 | "# Make sure not to change the function name or arguments.\n", 466 | "\n", 467 | "def make_Dataset(input_array, target_array, batch_size):\n", 468 | " \"\"\"\n", 469 | " This function takes two 2D numpy arrays in the first two arguments, and an integer\n", 470 | " batch_size in the third argument. It should create and return a Dataset object \n", 471 | " using the two numpy arrays and batch size according to the above specification.\n", 472 | " \"\"\"\n", 473 | " dataset = tf.data.Dataset.from_tensor_slices((input_array, target_array))\n", 474 | " dataset = dataset.batch(batch_size, drop_remainder=True)\n", 475 | " return dataset" 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": 17, 481 | "metadata": {}, 482 | "outputs": [], 483 | "source": [ 484 | "# Create the training and validation Datasets\n", 485 | "\n", 486 | "train_data = make_Dataset(input_train, target_train, batch_size)\n", 487 | "valid_data = make_Dataset(input_valid, target_valid, batch_size)" 488 | ] 489 | }, 490 | { 491 | "cell_type": "markdown", 492 | "metadata": {}, 493 | "source": [ 494 | "#### Build the recurrent neural network model" 495 | ] 496 | }, 497 | { 498 | "cell_type": "markdown", 499 | "metadata": {}, 500 | "source": [ 501 | "You are now ready to build your RNN character-level language model. You should write the following function to build the model; the function takes arguments for the batch size and vocabulary size (number of tokens). Using the Sequential API, your function should build your model according to the following specifications:\n", 502 | "\n", 503 | "* The first layer should be an Embedding layer with an embedding dimension of 256 and set the vocabulary size to `vocab_size` from the function argument.\n", 504 | "* The Embedding layer should also mask the zero padding in the input sequences.\n", 505 | "* The Embedding layer should also set the `batch_input_shape` to `(batch_size, None)` (a fixed batch size is required for stateful RNNs).\n", 506 | "* The next layer should be a (uni-directional) GRU layer with 1024 units, set to be a stateful RNN layer.\n", 507 | "* The GRU layer should return the full sequence, instead of just the output state at the final time step.\n", 508 | "* The final layer should be a Dense layer with `vocab_size` units and no activation function.\n", 509 | "\n", 510 | "In total, the network should have 3 layers." 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": 18, 516 | "metadata": {}, 517 | "outputs": [], 518 | "source": [ 519 | "#### GRADED CELL ####\n", 520 | "\n", 521 | "# Complete the following function.\n", 522 | "# Make sure not to change the function name or arguments.\n", 523 | "\n", 524 | "def get_model(vocab_size, batch_size):\n", 525 | " \"\"\"\n", 526 | " This function takes a vocabulary size and batch size, and builds and returns a \n", 527 | " Sequential model according to the above specification.\n", 528 | " \"\"\"\n", 529 | " model = tf.keras.Sequential([\n", 530 | " tf.keras.layers.Embedding(vocab_size, 256, batch_input_shape=(batch_size, None), mask_zero=True),\n", 531 | " tf.keras.layers.GRU(1024, return_sequences=True, stateful=True),\n", 532 | " tf.keras.layers.Dense(vocab_size)\n", 533 | " ])\n", 534 | " return model" 535 | ] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "execution_count": 19, 540 | "metadata": {}, 541 | "outputs": [ 542 | { 543 | "name": "stdout", 544 | "output_type": "stream", 545 | "text": [ 546 | "Model: \"sequential\"\n", 547 | "_________________________________________________________________\n", 548 | "Layer (type) Output Shape Param # \n", 549 | "=================================================================\n", 550 | "embedding (Embedding) (32, None, 256) 16640 \n", 551 | "_________________________________________________________________\n", 552 | "gru (GRU) (32, None, 1024) 3938304 \n", 553 | "_________________________________________________________________\n", 554 | "dense (Dense) (32, None, 65) 66625 \n", 555 | "=================================================================\n", 556 | "Total params: 4,021,569\n", 557 | "Trainable params: 4,021,569\n", 558 | "Non-trainable params: 0\n", 559 | "_________________________________________________________________\n" 560 | ] 561 | } 562 | ], 563 | "source": [ 564 | "# Build the model and print the model summary\n", 565 | "\n", 566 | "model = get_model(len(tokenizer.word_index) + 1, batch_size)\n", 567 | "model.summary()" 568 | ] 569 | }, 570 | { 571 | "cell_type": "markdown", 572 | "metadata": {}, 573 | "source": [ 574 | "#### Compile and train the model\n", 575 | "\n", 576 | "You are now ready to compile and train the model. For this model and dataset, the training time is very long. Therefore for this assignment it is not a requirement to train the model. We have pre-trained a model for you (using the code below) and saved the model weights, which can be loaded to get the model predictions. \n", 577 | "\n", 578 | "It is recommended to use accelerator hardware (e.g. using Colab) when training this model. It would also be beneficial to increase the size of the model, e.g. by stacking extra recurrent layers." 579 | ] 580 | }, 581 | { 582 | "cell_type": "code", 583 | "execution_count": 20, 584 | "metadata": {}, 585 | "outputs": [], 586 | "source": [ 587 | "# Choose whether to train a new model or load the pre-trained model\n", 588 | "\n", 589 | "skip_training = True" 590 | ] 591 | }, 592 | { 593 | "cell_type": "code", 594 | "execution_count": 21, 595 | "metadata": {}, 596 | "outputs": [], 597 | "source": [ 598 | "# Compile and train the model, or load pre-trained weights\n", 599 | "\n", 600 | "if not skip_training:\n", 601 | " checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(filepath='./models/ckpt',\n", 602 | " save_weights_only=True,\n", 603 | " save_best_only=True)\n", 604 | " model.compile(optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", 605 | " metrics=['sparse_categorical_accuracy'])\n", 606 | " history = model.fit(train_data, epochs=15, validation_data=valid_data, \n", 607 | " validation_steps=50, callbacks=[checkpoint_callback])" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": 22, 613 | "metadata": {}, 614 | "outputs": [], 615 | "source": [ 616 | "# Save model history as a json file, or load it if using pre-trained weights\n", 617 | "\n", 618 | "if not skip_training:\n", 619 | " history_dict = dict()\n", 620 | " for k, v in history.history.items():\n", 621 | " history_dict[k] = [float(val) for val in history.history[k]]\n", 622 | " with open('models/history.json', 'w+') as json_file:\n", 623 | " json.dump(history_dict, json_file, sort_keys=True, indent=4)\n", 624 | "else:\n", 625 | " with open('models/history.json', 'r') as json_file:\n", 626 | " history_dict = json.load(json_file)" 627 | ] 628 | }, 629 | { 630 | "cell_type": "markdown", 631 | "metadata": {}, 632 | "source": [ 633 | "#### Plot the learning curves" 634 | ] 635 | }, 636 | { 637 | "cell_type": "code", 638 | "execution_count": 23, 639 | "metadata": {}, 640 | "outputs": [ 641 | { 642 | "data": { 643 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3sAAAFNCAYAAAC5cXZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xl8FdXdx/HPLwtJgIRsQAIBEhbZwiKQICqICxWoK1Irat3ro4/Wbra11rba1tZWa7XW1rZWfWpVVLRqFVSsG7gRQPYdEiCQhECAJEDIdp4/ZgIXSEKAJDcJ3/frNa+7zJm5v3vFO/nec86MOecQERERERGRtiUk2AWIiIiIiIhI41PYExERERERaYMU9kRERERERNoghT0REREREZE2SGFPRERERESkDVLYExERERERaYMU9kSkyZlZqpk5MwsLdi0iIiJNwczuNbN/BbsOkUAKe9KqmdmHZrbTzCKCXYuIiEhrZmY5ZnZesOsQkcajsCetlpmlAmMBB1zUzK+tHioRERERadEU9qQ1uwb4HHgGuDZwhZlFmdnvzWyjme02s7lmFuWvO9PMPjWzXWa22cyu85//0MxuCtjHdWY2N+CxM7PbzGwtsNZ/7lF/H8VmtsDMxga0DzWzu81svZmV+Ot7mNnjZvb7w+r9j5l95/A3aGZPmNlDhz33upl9z7//IzPb4u9/tZmd25APzsy6mdkrZlZoZtlmdkfAunvNbIaZvejvd6GZDQtYP9D/rHaZ2XIzuyhgXZ2fu+8qM9tkZtvN7CcB22Wa2Xz/cywws4cb8j5ERKR5mNk3zWydmRWZ2Rtm1s1/3szsD2a2zf/eX2Jm6f66yWa2wj+WbDGzO2vZb4R/PEkPeK6zme0zsy5mlmhmb/ptisxsjpk16O9XM7vAzBb5235qZkMD1uWY2Y/9+naa2dNmFnm09+uvG2xms/11BWZ2d8DLtjOzf/rvebmZjQrY7riO2SInxDmnRUurXIB1wP8CI4EKoGvAuseBD4HuQChwOhAB9ARKgGlAOJAADPe3+RC4KWAf1wFzAx47YDYQD0T5z13t7yMM+D6QD0T6634ALAX6AwYM89tmAluBEL9dIrA3sP6A1xwHbAbMfxwH7AO6+fvdDHTz16UCfRrwuYUAC4CfAe2A3sAG4Hx//b3+5znV/4zuBLL9++H+5363v+05/ufZ/yife6r/+f0diPI/i/3AQH+7z4Bv+Pc7AqcF+9+XFi1atJxsC5ADnFfL8+cA24ER/nf6Y8DH/rrz/WNKrH+sGwgk++vygLH+/ThgRB2v+xRwf8Dj24C3/fu/AZ4IOAaNrTkmHuW9jAC2AaP949G1/vuLCHivy4AeeMf1T4BfNeD9Rvvv6/tApP94tL/uXqAMmOy/5m+Az/11x3XM1qLlRBf17EmrZGZnAr2Al5xzC4D1wJX+uhDgBuDbzrktzrkq59ynzrn9wFXAe865F5xzFc65Hc65Rcfw0r9xzhU55/YBOOf+5e+j0jn3e7yDQn+/7U3APc651c6z2G87D9gN1PyidwXwoXOuoJbXm4MXkmp6DKcCnznntgJV/usNMrNw51yOc259A95DBtDZOfcL51y5c24DXgi7IqDNAufcDOdcBfAw3gHtNH/pCDzgb/s+8CYw7Sife437nHP7nHOLgcV4oQ+8cNnXzBKdc6XOuc8b8D5ERKR5XAU85Zxb6H+n/xgYY950igq8wDMAL4StdM7l+dtV4B2jYpxzO51zC+vY//N4P8LWuNJ/rmYfyUAv/7g9xznnGlDzN4G/Oue+8I9H/4f3I+NpAW3+5Jzb7JwrAu4PqKG+93sBkO+c+71zrsw5V+Kc+yJgn3OdczOdc1XAsxw8zh3vMVvkhCjsSWt1LfCuc267//h5Dg7lTMQLJ7V9ifao4/mG2hz4wMy+b2Yr/aEru4BO/usf7bX+D69XEP/22doa+Qe06Rw8AF0JPOevWwd8B++XxG1mNj1wmEk9egHd/GEtu/y67wa61vY+nXPVQC5eb2I3YLP/XI2NeD159X3uNfID7u/FC44ANwKnAKvMLMvMLmjA+xARkebRDe+7HgDnXCmwA+ju/+j3J7yRHQVm9jczi/GbXobXy7XRzD4yszF17P99IMrMRptZL2A48G9/3YN4I0reNbMNZnZXA2vuBXz/sGNdD/+91Ag8pm8MWFfn++Xof0ccfpyLNLOwEzhmi5wQhT1pdfw5YJcDZ5lZvpnlA98Fhvlzy7bjDaPoU8vmm+t4HmAP0D7gcVItbQ78mmje/Lwf+bXEOedi8XrsrAGv9S/gYr/egcBrdbQDeAGY6h8ARwOvHCjGueedczW9nA74bT37qbEZyHbOxQYs0c65yQFtegS8zxAgBW/o6Vagx2HzJXoCW6j/c6+Xc26tc24a0MV/DzPMrMOx7kdERJrEVrzjDAD+93MC3nc/zrk/OudGAoPxfrj7gf98lnPuYrzv9teAl2rbuf8D4kt4P2xeCbzpnCvx15U4577vnOsNXAh8r4Fz3TbjDQ0NPNa1d869ENCmR8D9nv77PNr7re/YXq/jPGaLnBCFPWmNLsEbDjEI79e/4XiBaQ5wjX/QeAp42LwTkYSa2RjzLs/wHHCemV1uZmFmlmBmw/39LgKmmFl7M+uL19tUn2igEigEwszsZ0BMwPongV+aWT9/AvtQM0sAcM7lAll4PXqv1AwLrY1z7kv/NZ4E3nHO7QIws/5mdo7/vsrw5vJVHf3jYx5Q7E8Uj/I/n3QzywhoM9LMpph31tHv4A19+Rz4Ai8U/9DMws1sPN7Bd/pRPvd6mdnVZtbZ38cu/+mGvBcREWlc4WYWGbCE4Y2eud7Mhvvf6b8GvnDO5ZhZht8jF453fCgDqsysnZldZWad/CkBxdT/vf488HW8IZQ1QzhrTrLS18wsYB8NOT78HbjFr83MrIOZfdXMogPa3GZmKWYWjzfC5cWAWmp9v3hTF5LM7DvmnVwm2sxGH62YEzhmi5wQhT1pja4FnnbObXLO5dcseMNIrvIPTHfinRwlCyjC+/UsxDm3CW9Iyff95xdxcDz9H4ByoABvmOVzR6njHWAWsAZvuEcZhw4JeRjvl8p38Q5Q/8A7OUmN/wOGUMcQzsO8AJxHwAEQb+z/A3g9avl4v5zeDeAfYJfXtiN/HsGFeCE529/+SbwhqDVexzvo7gS+AUzx50qU413mYpK/3Z/xAvYqf7taP/cGvL+JwHIzKwUeBa5wzpU1YDsREWlcM/GCSM1yr3Puv8BP8UaW5OH1bNXM847BC1Y78Y6FO4Cas0h/A8gxs2LgFg5OXziCP+9tD94QylkBq/oB7wGleCfz+rNz7kMAM5tlh54JM3B/8/Hm7f3Jr20d3onXAj2Pd4ze4C+/8ret8/36PY4T8I6j+Xhn5z67rvcVoM5jtkhTqjnDn4g0MzMbhzecM/WwOXBBZWb3An2dc3UelEVERFozM8vBOwP3e8GuRaQpqWdPJAj84S7fBp5sSUFPRERERNoOhT2RZmZmA/HmpSUDjwS5HBERERFpozSMU0REREREpA1Sz56IiIiIiEgbpLAnIiIiIiLSBoUFu4BjlZiY6FJTU4NdhoiINIMFCxZsd851DnYdrYWOkSIiJ4eGHh9bXdhLTU1l/vz5wS5DRESagZltDHYNrYmOkSIiJ4eGHh81jFNERERERKQNUtgTERERERFpgxT2RERERERE2qBWN2dPRERERERanoqKCnJzcykrKwt2KW1GZGQkKSkphIeHH9f2CnsiIiIiInLCcnNziY6OJjU1FTMLdjmtnnOOHTt2kJubS1pa2nHtQ8M4RURERETkhJWVlZGQkKCg10jMjISEhBPqKVXYExERERGRRqGg17hO9PNU2BMRERERkVZvx44dDB8+nOHDh5OUlET37t0PPC4vL2/QPq6//npWr15db5vHH3+c5557rjFKbnKasyciIiIiIq1eQkICixYtAuDee++lY8eO3HnnnYe0cc7hnCMkpPY+r6effvqor3PbbbedeLHNRD17IiLSqNYXlvKPudk454JdihyjT9ZtZ9bSvGCXISLSqNatW0d6ejq33HILI0aMIC8vj5tvvplRo0YxePBgfvGLXxxoe+aZZ7Jo0SIqKyuJjY3lrrvuYtiwYYwZM4Zt27YBcM899/DII48caH/XXXeRmZlJ//79+fTTTwHYs2cPl112GcOGDWPatGmMGjXqQBBtTgp7IiJywjYUlvKn99cy8ZGPOff3H/HLN1ewvrA02GXJMXpqbjYPvlv/8CURkdZoxYoV3HjjjXz55Zd0796dBx54gPnz57N48WJmz57NihUrjthm9+7dnHXWWSxevJgxY8bw1FNP1bpv5xzz5s3jwQcfPBAcH3vsMZKSkli8eDF33XUXX375ZZO+v7poGKeIiByXDYWlzFyax1tL81mZVwzAqF5x/PzCQUxKTyapU2SQK5RjlZEWz39XbWN76X4SO0YEuxwRacXu+89yVmwtbtR9DuoWw88vHHxc2/bp04eMjIwDj1944QX+8Y9/UFlZydatW1mxYgWDBg06ZJuoqCgmTZoEwMiRI5kzZ06t+54yZcqBNjk5OQDMnTuXH/3oRwAMGzaMwYOPr+4TpbAnIiINlr19jxfwluSxwg94I3vF8bMLBjFpSBLJnaKCXKGciIzUeADm5xQxMT05yNWIiDSeDh06HLi/du1aHn30UebNm0dsbCxXX311rZc3aNeu3YH7oaGhVFZW1rrviIiII9q0lKkMCnsiIlKvnO17eKuWgPfTCwYxWQGvTRnSvROR4SF8ka2wJyIn5nh74JpDcXEx0dHRxMTEkJeXxzvvvMPEiRMb9TXOPPNMXnrpJcaOHcvSpUtrHSbaHJo07JnZROBRIBR40jn3wGHrfwBcFVDLQKCzc66oKesSEZH6bdxxMOAt94fhjOgZyz1fHcjkIcl0i1XAa4vahYUwvEcsWTk6DItI2zVixAgGDRpEeno6vXv35owzzmj01/jWt77FNddcw9ChQxkxYgTp6el06tSp0V/naKypuhjNLBRYA0wAcoEsYJpzrtZYa2YXAt91zp1T335HjRrl5s+f39jlioic9GoC3syleSzb4gW8U3vG8tUhyUELeGa2wDk3qtlfuJVqjGPkw++u5k8frGPxz79CdGR4I1UmIieDlStXMnDgwGCX0SJUVlZSWVlJZGQka9eu5Stf+Qpr164lLOzY+9pq+1wbenxsyp69TGCdc26DX9B04GKgrj7MacALTViPiIgcZtOOvQcC3tItuwEY3sPrwZs0JJnu6sE76WSkxVP9PizctIuzTukc7HJERFql0tJSzj33XCorK3HO8de//vW4gt6JaspX7A5sDnicC4yuraGZtQcmArc3YT0iIic95xw5O/byzvJ83lpyaMD7yeSBTBqSREpc+yBXKcE0omccoSFGVnaRwp6IyHGKjY1lwYIFwS6jScOe1fJcXWNGLwQ+qWuunpndDNwM0LNnz8apTkSkjSspq2B1fgmr8ktYlV/MqrwSVueXULLfO1PYsB6x3D15AJPSk+kRr4DXnFrynPYOEWGkd4thnubtiYi0ek0Z9nKBHgGPU4CtdbS9gnqGcDrn/gb8Dbz5CI1VoIhIW1BZVU3Ojj1eqMvzg11+Cbk79x1oEx0ZxsCkGC45tTsDkqMZ16+zAl6Q+HPaHydgTruZvRE4p9059yDwoN++Zk57s6WvjNR4/vn5RvZXVhERFtpcLysiIo2sKcNeFtDPzNKALXiB7srDG5lZJ+As4OomrEVEpE0oLNnv99YVszKvhNUFxawpKKW8shqA0BCjT+cOnNozjmmZPRmYHE3/pBi6dYrErLYBFxIELX5Oe0ZaPE/OzWZp7m5G+dfeExGR1qfJwp5zrtLMbgfewRum8pRzbrmZ3eKvf8JveinwrnNuT1PVIiLS2pRVVLFuWykr87xeupqAt720/ECbLtER9E+K5rrTUxmQFE3/pGj6dumonpiWr8XPaa+5uPq8nCKFPRGRVqxJTwnjnJsJzDzsuScOe/wM8ExT1iEi0pJVVzvWFZYyL7uI+TlFLN2ym+zte6j2B61HhIXQPymacwZ0oX9SDAP9YJfQMSK4hcvxarQ57dA089rjO7Sjb5eOzMsu4n/HN8ouRUSa3Pjx4/nxj3/M+eeff+C5Rx55hDVr1vDnP/+51m06duxIaWkpW7du5Y477mDGjBm17vehhx5i1Ki6r3TwyCOPcPPNN9O+vTdFYvLkyTz//PPExsae4Ls6Mc1//k8RkZNcRVU1y7cWk5VdxLwcL+Dt3FsBQOfoCIb38K5tNyA5hgFJ0fRK6EBoiIZgtiGNNqcdmm5ee0ZqPG8u3kpVtdO/PxFpFaZNm8b06dMPCXvTp0/nwQcfPOq23bp1qzXoNdQjjzzC1VdffSDszZw58yhbNA+FPRGRJravvIovN+1kXk4RWTlFLNy4i30VVQCkJrTnvIFdyUiLJzM1nl4J7TW3ru1rFXPaM9PieGHeJlblFzO4W6dglCAickymTp3KPffcw/79+4mIiCAnJ4etW7cyfPhwzj33XHbu3ElFRQW/+tWvuPjiiw/ZNicnhwsuuIBly5axb98+rr/+elasWMHAgQPZt+/gCc9uvfVWsrKy2LdvH1OnTuW+++7jj3/8I1u3buXss88mMTGRDz74gNTUVObPn09iYiIPP/wwTz31FAA33XQT3/nOd8jJyWHSpEmceeaZfPrpp3Tv3p3XX3+dqKjGvb6twp6ISCPbtbec+Tk7ycrxeu6W5u6mstphBgOSYrh8VMqBcNclJjLY5Uozay1z2mvm7WVlFynsiUirkJCQQGZmJm+//TYXX3wx06dP5+tf/zpRUVH8+9//JiYmhu3bt3Paaadx0UUX1fnj6l/+8hfat2/PkiVLWLJkCSNGjDiw7v777yc+Pp6qqirOPfdclixZwh133MHDDz/MBx98QGJi4iH7WrBgAU8//TRffPEFzjlGjx7NWWedRVxcHGvXruWFF17g73//O5dffjmvvPIKV1/duL/vKeyJiJygvN37mJft9dplZe9kdUEJAOGhxtCUWG4a25vRafGM6BVHp6jwIFcrLUFrmNOeEtee7rFRZOXs5Loz0oJVhoi0VrPugvyljbvPpCEw6YF6m9QM5awJe0899RTOOe6++24+/vhjQkJC2LJlCwUFBSQlJdW6j48//pg77rgDgKFDhzJ06NAD61566SX+9re/UVlZSV5eHitWrDhk/eHmzp3LpZdeSocOHQCYMmUKc+bM4aKLLiItLY3hw4cDMHLkSHJyco7l02gQhT0RkWPgnGPD9j0H5ttl5RSxucgb3tGhXSgjesVxwdBkMtLiGd4jlshwnRlTWq+M1Dg+Wb8D55yGF4tIq3DJJZfwve99j4ULF7Jv3z5GjBjBM888Q2FhIQsWLCA8PJzU1FTKysrq3U9t33nZ2dk89NBDZGVlERcXx3XXXXfU/ThX91TqiIiDJ1oLDQ09ZLhoY1HYExE5itL9lXy8ppDZKwqYs7bwwOUP4ju0IyM1jmvHpJKZFs+g5BjCQkOCXK1I48lIi+e1RVvZuGMvqYkdgl2OiLQmR+mBayodO3Zk/Pjx3HDDDUybNg2A3bt306VLF8LDw/nggw/YuHFjvfsYN24czz33HGeffTbLli1jyZIlABQXF9OhQwc6depEQUEBs2bNYvz48QBER0dTUlJyxDDOcePGcd1113HXXXfhnOPf//43zz77bOO/8Too7ImI1GJbcRmzVxYwe0UBn67bQXlVNbHtwznrlM6c1juBjNR4+nTuoN4OadMyA663p7AnIq3FtGnTmDJlCtOnTwfgqquu4sILL2TUqFEMHz6cAQMG1Lv9rbfeyvXXX8/QoUMZPnw4mZmZAAwbNoxTTz2VwYMH07t3b84444wD29x8881MmjSJ5ORkPvjggwPPjxgxguuuu+7APm666SZOPfXUJhmyWRurr2uxJRo1apSbP39+sMsQkTbGOce6baW8u8ILeIs27wKgR3wUEwYmMWFQVzJS49Rz18zMbIFzru4LG8khGvsY6ZxjxC9nc97Arjz4tWGNtl8RaZtWrlzJwIEDg11Gm1Pb59rQ46N69kTkpFVV7ViwcSezV+Qze0UBOTv2AjA0pRPfn3AKEwZ3pX/XaPXeyUnLzBiVGs+8nDqv6S4iIi2Ywp6InFT2lVfx8Vpv/t37q7ZRtKec8FBjTJ9EbhzbmwkDu5LUSZdDEKmRmRrP7BUFbCsu06VCRERaGYU9EWnztpfu57/+/Ls5a7ezv7Ka6MgwzhnQhQmDunLWKZ2JjtQlEURqk5F2cN7eBUO7BbkaERE5Fgp7ItImbSgsZbY//27Bpp04B91jo5iW2ZMJg7qSmRZPuObfiRzV4G4xRIWHkpWtsCciR6dLtTSuEz2/isKeiLQJzjm+3LyLd5cXMHtFPusL9wDeH6rfPrcfEwZ1ZVByjA5AIscoPDSEkb3imJezM9iliEgLFxkZyY4dO0hISNDxthE459ixYweRkcc/hF5hT0RatS279vHvhbnMWJBLzo69hIUYp/VO4JoxqZw3qCvdY6OCXaJIq5eRGs8j/13D7n0VdIrSkGcRqV1KSgq5ubkUFhYGu5Q2IzIykpSUlOPeXmFPRFqdfeVVvLsin5fn5/LJ+u04B2N6J3D7OV4Pnv4YFWlcGWlxOAcLN+7k7AFdgl2OiLRQ4eHhpKWlBbsMCaCwJyKtgnOOhZt2MmNBLm8uzqNkfyUpcVF8+9x+XDYihR7x7YNdokibdWqPOMJDjXk5RQp7IiKtiMKeiLRoebv38erCLbyyIJcN2/cQFR7K5CHJTB2Zwui0eEJCNCdApKlFtQslvXsnsrJ1vT0RkdZEYU9EWpyyiireXVHAjAW5zF1bSLWD0Wnx3Dq+D5OGJNMxQl9dIs0tMzWepz7Jpqyiisjw0GCXIyIiDaC/mESkRXDOsWjzLl5ekMt/Fm+lpKyS7rFR3H5OPy4b0Z1eCR2CXaLISS0jNZ6/fryBRZt3cVrvhGCXIyIiDaCwJyJBVVBcxqsLtzBjwWbWF+4hMjyEyeneMM3TeidomKZICzEqNQ6ArOwihT0RkVZCYU9Eml1ZRRXvrfSGaX68xhummZEax83jejN5SDLRkTqbpkhLE9u+Hf27RjMvR/P2RERaC4U9EWkWzjmW5O5mxoJc3li8ld37KkjuFMn/ju/LZSNTSEvUME2Rli4zLZ5XF+ZSWVVNWGhIsMsREZGjUNgTkSa1raSM177cwowFuawpKCUiLISJ6Ul8bWQPxvRJIFTDNEVajYy0eJ79fCMr80oYktIp2OWIiMhRKOyJSKPbX1nFf1duY8aCXD5aU0hVtWNkrzh+fekQLhiWTIyGaYq0Spmp8QDMyylS2BMRaQUU9kSkUTjnWLalmBkLNvP64q3s2ltBUkwk/zOuN5eNTKFP547BLlFETlBSp0h6xEeRlV3EjWemBbscERE5CoU9ETkhhSX7eX2RN0xzVX4J7cJCOH9wEl8bmcIZfRM1TFOkjclIjeej1YU45zDT/98iIi2Zwp6IHLPyymreX7WNGQs288Fqb5jm8B6x3H9pOhcM7UanKA3TFGmrMlPjeXXhFtYX7qFvF/XYi4i0ZAp7ItJgy7Z4Z9N8fdEWdu6toEt0BN8c25upI7vTt0t0sMsTkWaQkebN28vKKVLYExFp4RT2RKReO0r389qircxYkMvKvGLahYYwYXBXpo5MYWzfRJ1+XeQk0zuxA4kd25GVXcS0zJ7BLkdEROqhsCciR6ioquaDVd7ZNN9ftY3KasewHrH88pJ0LhyaTGz7dsEuUUSCxMwY1SteF1cXEWkFFPZE5ICVecW8PN8bprljTzmdoyO48cw0LhuZwildNUxTRDwZafG8vTyfvN37SO4UFexyRESkDgp7Iie5zUV7eXtZPq8t2sLyrd4wzfMGdWHqyBTG9eusYZoicoTR/ry9edlFXDy8e5CrERGRuijsiZyENhSWMmtZPrOW5bFsSzEAQ7p34r6LBnPRsG7EddAwTRGp28DkGDpGhJGVo7AnItKSKeyJnAScc6wuKGHW0nzeXpbP6oISAE7tGcvdkwcwcXAyPRPaB7lKEWktQkOMEb3iyMreGexSRESkHgp7Im2Uc45lW4qZtSyPWcvyyd6+BzPvgsg/v3AQE9OTNNdGRI5bZmocD727hl17y3XSJhGRFkphT6QNqa52fLl5F7OW5vH28nxyd+4jNMQY0zuBm8am8ZVBSXSOjgh2mSLSBmSk1lxvbycTBnUNcjUiIlIbhT2RVq6q2jEvu4i3l3kBr6B4P+1CQzizXyJ3nNuPCQO7ag6eiDS6YT1iaRcaQlZOkcKeiEgLpbAn0gpVVFXz2fodzFqWx7vLC9ixp5yIsBDG9+/MpPRkzhnYhZjI8GCXKSJtWGR4KENTOjEvW9fbExFpqRT2RFqJsooq5q7dzqxl+by3soDd+yro0C6Uswd0YfKQZMb370z7dvpfWkSaT0ZaPH//eAN7yyv1/SMi0gLpm1mkBdtXXsVHa7Yxc2k+76/aRun+SmIiwzhvUFcmpScztl8ikeGhwS5TRE5Smanx/OXD9SzatIvT+yYGuxwRETmMwp5IC7NnfyXvr9rGrGV5fLCqkH0VVcR3aMcFQ5OZmJ7E6X0SaRemC52LSPCNTI3DDOblFCnsiYi0QAp7Ii1AcVkF76/cxsyleXy0ppD9ldUkdozgspHdmZyeTGZaPGGhCngi0rLERIYzMCmGrBzN2xMRaYkU9kSCZNfecmavKGDWsnzmrt1OeVU1STGRTMvsyeQhyYzsFUdoiAW7TBGRemWmxfNi1mYqqqoJ149SIiItisKeSDMq2lPOu8vzmbksn0/Xbaey2tE9NoprT+/FxPRkTu0RS4gCnoi0Ihmp8TzzaQ7Ltuzm1J5xwS5HREQCKOyJNLHCkv28szyfWcvy+HxDEVXVjl4J7blpbG8mD0liSPdOmCngiUjrlJHmBbysnCKFPRGRFkZhT6QJ5O8u4+1lecxclk9WThHOQe/OHbj1rD5MGpLEoOQYBTwRaRO6REeSmtCeedk7uXlcsKsREZFACntyBhcvAAAgAElEQVQijWTLrn3MWprHrGX5LNi4E4D+XaP59rn9mDwkmX5dOirgiUiblJEaz+yVBVRXOw1FFxFpQRT2RE5A/u4yXl+0hZnL8lm8eRcAg7vF8IPz+zMxPYk+nTsGuUIRaYnMbCLwKBAKPOmce6CWNuOBR4BwYLtz7qxmLfIYZKTF8/KCXNYVlnJK1+hglyMiIj6FPZFjtK+8ineW5/PKwlw+WbedagfDUjpx16QBTEpPoldCh2CXKCItmJmFAo8DE4BcIMvM3nDOrQhoEwv8GZjonNtkZl2CU23DZKbGAzAvu0hhT0SkBVHYE2mA6mrHvJwiXl2Yy8yl+ZTuryQlLorbz+nHlFO7k5qogCciDZYJrHPObQAws+nAxcCKgDZXAq865zYBOOe2NXuVx6BXQnu6REeQlVPE1af1CnY5IiLia9Kw19aGqcjJJ2f7Hl5dmMurX24hd+c+OkaEMXlIElNGpJCZGq+5KSJyPLoDmwMe5wKjD2tzChBuZh8C0cCjzrl/1rYzM7sZuBmgZ8+ejV5sQ5gZGWnxZGXr4uoiIi1Jk4W9tjhMRU4Ou/dV8NaSPF5ZmMuCjTsxgzP7JvKD8/vzlUFJRLULDXaJItK61fYrkTvscRgwEjgXiAI+M7PPnXNrjtjQub8BfwMYNWrU4ftpNpmp8by1JI/cnXtJiWsfrDJERCRAU/bstblhKtJ2VVZVM2ftdmYszGX2igLKK6vp16Ujd00awCXDu5PUKTLYJYpI25EL9Ah4nAJsraXNdufcHmCPmX0MDAOOCHuNausiKMmH/hOPedOMgHl7CnsiIi1DU4a9Rh2mItIUVmwt5tWFuby2aCvbS/cT1z6cKzN7MmVEd13sXESaShbQz8zSgC3AFXg/fgZ6HfiTmYUB7fCOn39o8srevQe2r4XeiyH82H7k6p8UTXRkGFk5RUwZkdJEBYqIyLFoyrDXaMNUWsJ8BGk7Ckv28/qiLbyycAsr84oJDzXOGdCFy0akML5/F9qFhQS7RBFpw5xzlWZ2O/AO3pz2p5xzy83sFn/9E865lWb2NrAEqMab976syYsb9wP450Xw5bOQ+c1j2jQ0xBjVK455mrcnItJiNGXYa7RhKi1lPoK0XmUVVby3soBXF27hozWFVFU7hvWI5RcXD+bCod2I69Au2CWKyEnEOTcTmHnYc08c9vhB4MHmrIu0cZCSCZ88CiOuhbBj+27MSIvng9WF7CjdT0LHiCYqUkREGqopw17LHaYiJwXnHItzd/Ni1mbeXLKVkrJKkmIiuXlcby4b0Z2+XXQtKBGRQ5jBWT+E56bCkukw4ppj2rzmentZOTuZmJ7UFBWKiMgxaLKw16KHqUibVlZRxRuLt/LsZxtZumU3UeGhTEr3Lpcwpk8CobpcgohI3fqeB8nDYM7DMOxKCG34nwpDUjrRLiyErJwihT0RkRagSa+z12KHqUibtGnHXv71xUZemr+ZXXsr6NelI7+8eDCXjkihY0ST/lMXEWk7zLy5ey9eDctfhaGXN3jTiLBQTu0RS1aO5u2JiLQE+gtYWrXqasdHawr552c5fLimkBAzJg5O4htjejE6LV5n0xQROR79vwpdBsHHD0H6VAhp+ImrMtPi+fOH69mzv5IO+qFNRCSo9C0srdLOPeW8NH8z//piI5uL9tE5OoI7zunHlaN70jVG18QTETkhISEw9vvwyo2w8g0YfEmDN81Ijaeqeh0LN+1kbL/OTVikiIgcjcKetCpLcnfxz8828p/FW9lfWU1mWjw/mjiA8wcnER6qSyYcoaoS9hXBnkJvKS08eH9PIezZfvB+xV4IbectYREBtxHeGfkOWVfzXMBtaLsjnztkP+2gfTzEdIf2Cd5QMRFpuQZfCh/+xuvdG3Rxg/+fHdErjhDzLq6usCciElwKe9LilVVU8daSPP75+UYWb95F+3ahTB2ZwjfG9GJAUkxwi3MO9u2EqnKwUAjxlwP3w/z7jRREnYP9JUcGtboC3N4ijry8JV5dHTpDh0TvNqEPhLeHqgqo2g+V+733VHNbVuw9X1Vx5LrK/d66YxHaDqKTveAXkwwx3bz7gc91TDqmE0OISCMLCfV69167Fda8Df0nNWizjhFhDO7WSdfbExFpAfSXlLRYm4v28twXm3hp/maK9pTTu3MH7r1wEFNGphATGd68xVSUQdEG2LEWtq+FHev827VQtrth+zgQ/GpCYMhhgbCeoOicF9z2FNYdrCI7+QGuMyT2g16nQ4cuBwPdgSURImMbL4CCV9+BoFjuhcAD9/3byjLYuwOKt0LxFijJ8+5vWQgr3zzyfVkIdOzqB8BuAcGwe8Bz3SA8qvHeh4gcasjX/N69B+GUiQ3u3ctIjee5Lzayv7KKiLDQJi5SRETqorAnLUp1tWPOuu08+1kO/121DQMmDOrKNWNSOb1PQtOecMU5L4DUhLjt6w6Gu12bOKSHLLobJPaF9Msgoa8XOKqrvMVVQXVlwP3anq+up43fzlVBtd8OoGt67cGtYxdvWGRYEC9gbOYN3QxrB8dTRk0PafEWPwz6S4l/u2M9ZM+B/bUE66i4wwJgQG9htB8IIztp2KjI8QgNhzO/B29+B9a/D33PbdBmmWlxPPVJNsu27GZkr/gmLlJEROqisCctwu69Fby8YDP/+nwjOTv2ktixHbeN78uVo3vSLbaRe272l3o9czVLTbjbsR7KSw+2C2/vBbmUUTBsmtdbltDXWyI6Nm5NJzszbz5f+3hIGlJ3u/2lB3sED+8hLN4KeYthz7YjtwvvUMtw0W4BS3don9i4vZ0ibcXwK+Gj33lz9xoY9kb5F1efl71TYU9EJIgU9iSolm/dzbOfbeS1RVsoq6hmZK84vjvhFCamJ5340J/SQihYBoWrDu2tK9ka0Mggtgck9IOeY7wgl9jPexzTTb1BLU1ER4jo5/03qktlOZTmH9pDGNhLmDPXC4g1PaY1QsL9EFhPKOyY5PVeipxMwiLgjG/D2z+CnE8g9YyjbpLYMYLenTuQlVPErfRphiJFRKQ2CnsSFJuL9vKLN1cwe0UBkeEhXDK8O98Y04vB3Tod+86qKr0gl78MCpb6t8ugtOBgm4hO3rDLtHHebYIfGOJ7a85XWxPWDmJ7ektdqqu9+Y+H9wzW9BbmL4U173hnKD2EecNno5P8YaLJBwNidLeDz0fF6YcCaVtGXgtzHoKPfweprzdok9Fp8by1JI/qakdIiP5/EBEJBoU9aVb7yqv4y0freeKj9YSFGHd+5RS+cVoqndo38IQre4u8IFcT6AqWwbZVB0/uERIOnQdAn3O8OW5J6d6FgTt01h/fclBICER39Za6OAdlu6A477Aho1ugJB92b4HcLO+kM4cLjQgIhEn+3MHkw8Jhsn5okNYjPApO/xbM/hlszoIeGUfdJCM1nhfmbWZ1QQkDk4N85mQRkZOUwp40C+cc7yzP55dvrmTLrn1cNKwbd08eSFKnOi6AXl3lnf0yf+mh4a54y8E2HTp7gW70zdB1iBfsEk/xTiggcqLMvB66qDjoOqjudpX7vfBXkueHwTxvyGhJvnc/b0kdvYR4Z0U9omcw2Xv+iOsdtqv7fs2tftCQpjTqBpj7B6+H78oXj9o8w5+3l5VTpLAnIhIkCnvS5NZtK+W+/yxnztrtDEiKZvrNp3Fa74SDDcp2Q8HyQ4dhblsJlfu89RYKnftDrzOg62Av1HUdUn+vjEhzCYuAuF7eUhfnvH/nJfn+3MG8w8JhnvdvvrTAO1Pr8QoJPywAhh96cfvAC9+HR0JYVP234e0hLNLr1anrtua+gmbbFxENp90GH/zKOxlS8rB6m6fERZHcKZIvsou4Zkxq89QoIiKHUNiTJlNSVsFj76/jqbnZRLUL5d4LB3H1ab0ICzHY9DksfRnWzoZdGw9uFBXn9daNuv7gMMzOA4J7WQGRE2UGUbHe0mVA3e2qq6B0G+wv9i9Wf9hF7o+4mH1t68sDnqs49HqHNev3l3jXjqzcd+htXddwbIiwyCOD4NWvej2V0nZkfhM+/aN3Zs6vP1tvUzMjIzWezzfswDnXtJfOERGRWinsSaNzzvHaoi38euYqCkv28/VRPfjBxP4k7t0AH/wSls3wrlsXFuWdxnvktV6w65quM2DKyS0k1A9HQQpI1dVHBsBab/2lsqz+W/1I0/ZExcLo//Eusr5tJXQZWG/zjLR43li8lU1Fe+mV0KGZihQRkRoKe9Kolm/dzb1vLCcrZyfDUjrxzJRkBu+YDc/e6s25s1DoPR7O/gkM+Ko3LEhEWoaQEGjXwVtE6jL6VvjszzDn93DZk/U2zTxwvb0ihT0RkSBQ2JNGsWtvOb9/dw3PfbGRnlH7mTFqAyOL/4u9+KnXICUDJj0Igy+Bjl2CW6yIiBy/DgmQcQN89jiM/zEk1H0dvX5dOtIpKpysnCK+NqpHMxYpIiKgsCcnqKra8WLWZh57ezGjyr9gVpcvOaXkC2xZhXdmzLPvgSFTIT4t2KWKiEhjGfMtmPd3mPswXPx4nc1CQoyM1DiycnY2Y3EiIlJDYU+O24LsQt549TmG7prN+2ELiArfB9Xd4LRbYMjXIGmo5t+JiLRF0V1hxLUw/x8w7of1no02My2e91ZuY1tJGV2i67jcjoiINAmFPTk2zrFzzacsf+dJBux4j/usmPLIGMLTvwZDL4dep3snmRARkbbtjDtg/lPwyaNwwcN1Nqu53t78nJ1MHqKzs4qINCeFPWmYwtVULX6RPQumE7dvC6NcONkJ4+h49nVEDjxfZ90TETnZdEqBU6+CL5+FcT+o8zIb6d07ERUeyrzsIoU9EZFmprAndSveCktnwNKXIH8pEMKiqsGs6nwlEy67kYHdddAWETmpnfEdWPisd+29ib+ptUl4aAin9oxlXnZRMxcnIiIKe3KkijL46Lfe0BxXRU7EAJ6puIYvY8Zz+4Vn8s2BXXRxXBER8U6+NfRymP80nPk96Ni51mYZqfH88f21FJdVEBMZ3sxFioicvEKCXYC0MBs/gyfOgLkPs7LLZM6vepTz99xL3Nl38OL3LmHCoK4KeiIictDY70NlGXz2pzqbZKbF4xws2KizcoqINCeFPfHsL4G37oSnJ+Kqyvl7r4eYtHEaqaek8973zuLb5/UjMlwnXhERkcMk9oPBl0LWk7C39qGap/aMJSzEyNJQThGRZqWwJ7B2Njx+GmQ9iRt9Cz9L/jv3r+7GHef05YmrR9Ijvn2wKxQRkZZs3J1QXgpf/LXW1e3bhTG4eyeychT2RESak8LeyWzPDnj1ZnhuKrTrQPX17/DjvVfx7Jc7+NY5ffnuhFM0ZFNERI6u62AYcAF88RcoK661SWZqHIs376asoqqZixMROXkp7J2MnINlr8Djmd7tWT+i+uaP+cmC9kzP2sxtZ/fhewp6IiJyLMbdCWW7Ievvta7OTEugvKqaJbm7m7kwEZGTl8LeyaZ4K0y/EmbcALE94OaPcON/zE/fWssL8zZx6/g+3PmV/gp6IiJybLqdCn0nwGePQ/meI1aP6hUHoKGcIiLNSGHvZOEcLHgGHh8N6z+Ar/wKbnwP13UwP319Gc99sYn/Oas3PzxfQU9ERI7TuB/A3h3epRgOE9ehHad07ajr7YmINCOFvZPBjvXwfxfCf74NycPg1k/g9G/hQkL5+RvL+dfnm/ifcb25a+IABT0RETl+PUdD6ljvIusVZUeszkiNZ8HGnVRVuyAUJyJy8lHYa8uqKuHTx+AvZ0DeYrjwUbj2P5DQB+cc9/1nBf/8bCPfHJvGXZMU9EREpBGc9UMoLYAvnz1iVWZaPKX7K1mZV/tJXEREpHEdNeyZ2e1mFtccxUgjKlgO/5gA794Dfc6G276AkdeBGc45fvHmCp75NIcbz0zj7skDFfRERKRxpI6FHqNh7iNQWX7IqozUeAA+WlMYjMpERE46DenZSwKyzOwlM5toSgUtW+V+eP9++Os42LUJpj4NVzwPMd0AcM7xyzdX8vQnOVx/Rir3fFVBT0REGpGZN3evOBeWTD9kVbfYKMb2S+QvH64nb/e+IBUoInLyOGrYc87dA/QD/gFcB6w1s1+bWZ8mrk2O1eZ58MRY+Ph3kD4Vbs+C9CnegRcv6N3/1kqe+iSb605P5WcXDFLQExGRxtf3PEgeDnMe9qYUBLj/kiFUVlfz09eW4Zzm7omINKUGzdlz3rdxvr9UAnHADDP7XRPWJg21vxRm3QX/+ApU7IWrXoEpf4X28QeaOOf49cyVPDk3m2vH9OLnFyroiYhIE6np3duZ7V3PNUDPhPZ8f0J/3lu5jbeW5gWpQBGRk0ND5uzdYWYLgN8BnwBDnHO3AiOBy5q4Pjma9e/DX8bAF09A5jfhfz+Dfucd0sQ5xwOzVvH3OdlcM6YX9140WEFPRESaVv/J0GUQzHkIqqsPWXX9GakM6d6Je99Yzq695XXsQERETlRDevYSgSnOufOdcy875yoAnHPVwAVNWp3UbW8RvPa/8OylEBoBN7wNkx+EiOhDmjnn+O3bq/nrxxu4+rSe3KegJyIizSEkBMbdCdvXwMrXD1kVFhrCby8bys69Fdz/1sogFSgi0vY1JOzNBA5cAdXMos1sNIBzTt/QwbC3CJ48FxZPh7F3wi1zoedpRzRzzvHgO6t54qP1XDW6J7+4KF1BT0REms+gSyChH3z8EBw2P29Qtxj+Z1xvXl6Qy9y124NUoIhI29aQsPcXoDTg8R7/OQmGqkqYcQPszoXr3oRzfwrhkUc0c87x0Lur+fOH65mW2ZNfXpxOSIiCnoiINKOQUBj7fShYBmvePmL1Hef2Iy2xA3f/eyn7yquCUKCISNvWkLBnLuB0Wf7wzbCmK0nq9d97YcMH8NWHodfptTZxzvHw7DU8/sF6rsjowf2XKOiJiEiQDJkKsb3go98d0bsXGR7Kb6YMYVPRXv7w3pogFSgi0nY1JOxt8E/SEu4v3wY2NHVhUoslL8Onj0HGN2HEN+ps9sh7a3ns/XV8fVQPfn3pEAU9EZEWxr9u7WozW2dmd9WyfryZ7TazRf7ys2DU2ShCw+HM78LWhd5JxQ5zWu8EpmX24Mk5G1iauzsIBYqItF0NCXu3AKcDW4BcYDRwc1MWJbXYugjeuB16nQETf1Nns0feW8Oj/13L10am8JspCnoiIi2NmYUCjwOTgEHANDMbVEvTOc654f7yi2YtsrENvxJiuntz92px16SBJHaM4EevLKGiqrrWNiIicuwaclH1bc65K5xzXZxzXZ1zVzrntjVHceLbsx1evBraJ8LX/s/7lbQWj763lkfeW8vUkSn89rKhCnoiIi1TJrDOObfBOVcOTAcuDnJNTSssAs74Nmz6FHLmHrG6U1Q4v7g4nRV5xTw5JzsIBYqItE0Nuc5epJndZmZ/NrOnapbmKE6Aqgp46VrYUwhX/As6dq612WP/Xcsf3lvDlBHdFfRERJqRmfUxswj//nh/6kNsPZt0BzYHPM71nzvcGDNbbGazzGxwPa9/s5nNN7P5hYWFx/UemsWIa6BDF/j4wVpXT0xPYuLgJB55bw3Z2/c0c3EiIm1TQ4ZxPgskAecDHwEpQElTFiUB3vkJbJwLF/4Rup1aa5PHP1jH72evYcqp3Xlw6jBCFfRERJrTK0CVmfUF/gGkAc/X0762L2l32OOFQC/n3DDgMeC1unbmnPubc26Uc25U5861/yDYIoRHwenfgg0fwqYvam1y38WDaRcWwo9fXYJzh38kIiJyrBoS9vo6534K7HHO/R/wVWBI05YlAHz5L5j3VxhzOwz7eq1N/vzhOh58ZzWXDO/Gg19T0BMRCYJq51wlcCnwiHPuu0ByPe1zgR4Bj1OArYENnHPFzrlS//5MINzMEhu37CAYdQN0TIJXboKS/CNWd42J5O7JA/l8QxEvZm2uZQciInIsGhL2KvzbXWaWDnQCUpusIvHkzoc3vwu9x8N599Xa5ImP1vO7t1dz8fBu/P7y4Qp6IiLBUWFm04BrgTf952qfXO3JAvqZWZqZtQOuAN4IbGBmSWZm/v1MvOP1jkavvLlFdIQrp8PeHfDcVCgrPqLJ10f1YHRaPPfPXMm24rIgFCki0nY0JOz9zczigHvwDkYrgN82aVUnu5J874Qs0ckw9WkIPfKyhk/O2cADs1Zx4bBu/F49eiIiwXQ9MAa43zmXbWZpwL/qauz3At4OvAOsBF5yzi03s1vM7Ba/2VRgmZktBv4IXOHayrjGbqfC5f+EghXw0jVQWX7I6pAQ4zdThrC/spqfv7E8SEWKiLQN9V4c3cxCgGLn3E7gY6B3s1R1MqvcDy9+A8p2w42zoX38EU3WF5bywKxVnD+4K3+4fBhhoQ3J7CIi0hSccyuAOwD8H0ejnXMPHGWbmcDMw557IuD+n4A/NX61LUS/8+CiP8Lrt8F/7oBL/gJ28EfL3p078p3z+vG7t1fz9rJ8JqYnBbFYEZHWq96U4Jyrxvv1UZrLrB9C7jy45M+QlF5rk1+/tZLI8FDuv3SIgp6ISJCZ2YdmFmNm8cBi4GkzezjYdbV4p14N4++GxS/A+786YvU3x/ZmYHIMP3t9Gbv3VdSyAxEROZqGJIXZZnanmfUws/iapSE7N7OJZrbazNaZ2V21rB9vZrvNbJG//OyY30FbMv8pWPAMnPk9GHxprU3mrC3kv6u2cfs5fUnsGNG89YmISG06OeeKgSnA0865kcB5Qa6pdTjrhzDiWpjzkHcMDBAeGsJvLxvC9tL9/PbtVUEqUESkdat3GKfvBv/2toDnHEcZ0mlmocDjwAS8M49lmdkb/nCXQHOccxc0sN62a+NnMPMH0HcCnHNPrU0qq6r51Zsr6RnfnuvPSG3e+kREpC5hZpYMXA78JNjFtCpm8NWHvbnqb33fO1PngMkHVg9NieXGM9P4+5xsLh7WjdG9E4JYrIhI63PUnj3nXFotS0Pm7mUC65xzG5xz5cB04OITLbhN2r0FXvoGxPaCy56EkNBam03P2szqghLunjyAiLDa24iISLP7Bd7JVtY757LMrDewNsg1tR6hYfC1pyF5OMy4ATZnHbL6uxNOoUd8FD9+dSllFVVBKlJEpHU6atgzs2tqWxqw7+5A4EVycv3nDjfGzBab2SwzG1xHDTeb2Xwzm19YWNiAl25FKsrgxau822kvQFRsrc1276vg4dlrGJ0Wz/mDNVFdRKSlcM697Jwb6py71X+8wTl3WbDralXadYArX4LoJHjh67Bj/YFV7duF8ZtLh7Jh+x4ee18ZWkTkWDRkzl5GwDIWuBe4qAHb1XYtgMNPG70Q6OWcGwY8BrxW246cc39zzo1yzo3q3LlzA166lXDOu5be1i9hyl+hc/86mz7+wTp27i3npxcMwkyXWRARaSnMLMXM/m1m28yswMxeMbOUYNfV6nTsDFe/4t3/1xQoPfjj7pn9Epk6MoW/frSBFVuPvDafiIjUriHDOL8VsHwTOBVo14B95wI9Ah6nAFsP23exc67Uvz8TCDezxAZX39p98QQsfh7G/xgGfLXOZjnb9/D0J9l8bWQK6d07NWOBIiLSAE/jXYe2G94Ilv/4z8mxSujj9fCVFMDzl0P5ngOrfjJ5ILHtw7nr1SVUVbeNSw6KiDS14zlv/16gXwPaZQH9zCzNzNoBV+AdDA8wsyTzu6nMLNOvZ8dx1NT6bPgI3vkJDLgAxv2w3qa/nrmS8NAQ7vxK3T1/IiISNJ2dc0875yr95RmgDQ1DaWYpo7w5fHmL4OXroKoSgLgO7fj5hYNZkrubpz/JDm6NIiKtREPm7P3HzN7wlzeB1cDrR9vOOVeJd42+d4CVwEvOueVmdouZ3eI3mwosM7PFwB+BK5xzbf/nup0bvQNYYj+49AkIqfs/w6frt/PuigJuO7svXWIim69GERFpqO1mdrWZhfrL1ZwsP1w2lf6T4Ku/h7Xvwlvf9aY9ABcMTebcAV34/btr2Fy0N8hFioi0fA259MJDAfcrgY3OudyG7NwfmjnzsOeeCLj/J+BPDdlXm1G+F6ZfBdVVcMXzEBFdZ9Oqascv31xJ99gobjwzrRmLFBGRY3AD3rHsD3hz0z8Frg9qRW3BqBu8s1XPeQhiUmD8jzAzfnlJOl/5w8fc/e+l/POGTM1jFxGpR0OGcW4CvnDOfeSc+wTYYWapTVpVW+UcvH4bFCyDqf/w5ibU4+X5m1mZV8xdkwYQGa5LLYiItETOuU3OuYucc52dc12cc5fgXWBdTtQ598CwK+HDX8PCZwHoFhvFjyb2Z87a7by6cEuQCxQRadkaEvZeBqoDHlf5z8mx+uRRWP4qnPsz6Deh3qYlZRU89O4aRvaK44Khyc1UoIiINJLvBbuANsEMLvr/9u47vsry/v/468reCdlkQMLIYAQIYQmyohYcoGjFVau2pVpH++23/Wpbf35tHdVq+3VW66671lEVwUVwsgSEMBL2CglZSNgjyfX74z5AwAABzslJTt7Px+M8zrjv+7o+9yGH63zOdd3X9Qh0Hwvv/xJWfQrAlUO6MrBrJ+76YDk1O/d5OUgRkbarJclegGtRdABcj1syG6c0tepT+PRO6H0RjPivE+7+98/WULNzH3doqQURkfZI/3G7i38gXPoiJPWCN66G8m/x8zPcN6kvu/c18Kf3l3s7QhGRNqslyV61MebQunrGmIlAjedC8kG1a+Ct6yCpN0x83Pml8jg2bd3Ns1+uY9KAVPqlN7/IuoiItGm+P9lYawqOhCvfhLA4eOVS+G49PZMiuXFMD95bXE5RaaW3IxQRaZNakuxdD/zeGLPRGLMRuBX4uWfD8iH7dsDrV4Dxg8tegaDwEx5y3/RS/P0Mvx2npRZERNoqY8wOY8z2Zm47cNbcE3eKTIar3oSG/fDyxbCrlhtGdycrKYLb31nKzn313o5QRKTNacmi6mustUOBXkBvaw4Vxi0AACAASURBVO0Z1trVng/NBzQ2wjvXQ81K+OEL0CnjhIfMW7eVD5ZUcP2o7nSODvV4iCIicmqstZHW2qhmbpHW2pbMdi0nKyEbrvgXbNsEr11GUONe/jwpj4rte3ngw1JvRyci0ua0ZJ29e40xMdbandbaHcaYTsaYu1sjuHbvywehdCqcczd0G33C3RsbLXdNXU7n6BCmjOzm8fBERETanS5D4eJnoOwbePtnDEyP4sfDMnhxzgYWbNjq7ehERNqUlgzjHG+t3XbwibX2O+Bcz4XkI0qnwcx7IG8yDP1Fiw55+9vNLNlcx63jcggN0lILIiIizeo1Acbf7/ygOv1WfnNOFinRodz61hL21Td4OzoRkTajJcmevzEm+OATY0woEHyc/aV6Jbw9BTr3hwsePuGELAC79tXzlw9L6Zcew4R+utRDRETkuIb8HM64Bb55moj5j3H3RX1YXbWTJz5b4+3IRETajJYkey8DM4wxPzHG/AT4BPinZ8Nq5z7+A/gHOBOyBLbsursnP19D1Q5nqQU/P83YLSIickJn/RH6XAyf3smYfZ8xsX8Kj89czarKHd6OTESkTWjJBC1/Ae4GcnEmafkQ6OrhuNqvymWw6mMYeiNEp7XokM3b9vDUF2uZ0C+FgV07eThAERERH+HnBxc+ARlnwn9+wZ/61hARHMCtbxXT0KjVL0REWtKzB7AFaAQuBgqBEo9F1N7NehQCw2DQT1p8yP3TnRnEbh2f46moREREfFNAMEx+GeJ7Ev3utfx1pD8LN27jhpcXsPeArt8TkY7tmMmeMSbLGHOHMaYEeAzYBBhr7Rhr7WOtFmF7UlcGS/4N+T+GsNgWHbJgw3e8t7icKSO7kRqjpRZEREROWmiMs+h6SBRjF9zIg2fH8klJJVc9M5dtu/d7OzoREa85Xs9eKU4v3gXW2hHW2kcB/UR2PHOeAGthWMtm3zy41EJiZDDXj+ru4eBERER8WHSqk/Dt380lpb/ihQtiKC6r44dPzqZ82x5vRyci4hXHS/Yuxhm+OdMY87QxphDQzCHHsmcbLHjBuVA8pkuLDnlvcTmLNm3jtz/IJjxY6++KiIiclqRezuRo28sZ9elEZvYvYvv275j091ms2KJJW0Sk4zlmsmetfcdaOxnIAT4D/gtIMsY8YYw5p5Xiaz/mPwf7d8LwW1q0+579Ddz/YSl9UqO4OL9lE7mIiIjICWSeCTcvgLzJpC57iq/Cfss5DZ/zwye/Zu7aWm9HJyLSqloyG+cua+0r1trzgTRgEXCbxyNrTw7shblPQvexkNy3RYc89cVaKur2csf5vbXUgoiIiDtFJMKFj8NPZxAYk8afGh7mFf8/cu9zbzB9SYW3oxMRaTUtnY0TAGvtVmvtP6y1Yz0VULtU/C/YWQnDf9mi3Svq9vDk52s4t28ygzNbNpGLiIiInKS0AvjpDJjwKL2DKnk74PfUvHET//p8sbcjExFpFSeV7EkzGhth1iPQuR9kjmrRIQ98uIKGRsvvxud6ODgREZEOzs8P8q/G75YF2IKfcoV/EWcXncvHL96Lbaj3dnQi0pHsrIKVH8H6r1utSs0KcrpWTIPa1XDJc2BOPBxz8aZtvP3tZm4Y3Z302LBWCFBEREQI7UTA+Q9Qn3812165iXPW3k/ZA++QfNkjBGQM83Z0IuJrdlZDxSIo/xbKXfc7yp1t2edCxvBWCUPJ3uma9QjEdIXciSfc1VrLn6YuJz4iiF+M1lILIiIirS0gpS+Z/z2Taf/6O/1L/0rAC+Oo73MpAT+4CyKTvR2eiLRHu2qchK7iYGK3CLaXHd4e19NJ7jr3h5QB0Dmv1UJTsnc6Ns6BTXPh3AfB/8Rv5dTiChZs+I77JvUlMiSwFQIUERGRoxk/P869/Cb+9fVZ1Ey/lylL38aunIYZfRsM/jkEBHk7RBFpq3bVNknqvoWKxVC36fD22O7QZSikuBK75DwIifJauEr2TsfXD0NoLPS/8oS77j3QwH3TS8ntHMUPC9JbITgRERE5nsnDc/g45i+c+9oY/uT/CsM+vh0Wvgjj73dm2BaRjm331qOGYi6Cuo2Ht8d2g7RBMHiKk9x17gch0d6LtxlK9k5V9Qrner1Rt0HQia+9e/ardWzetocHfpiHv5ZaEBERaRPO6Z1M3M8mct0LqYzx/5b7979K8EsXQc758IN7oVNXb4coIp5kLezdBts2wrZNULvqcHK3bcPh/TplQNpAGPQT11DMfhAa47WwW0rJ3qma9QgEhMLgn51w16rte3l85mrO6ZXEGd3jWyE4ERERaamBXWN564ZhXP2sP0PrevPOgG/JWPZ3eHwwDP8VjPgVBIZ6O0wRORXWOrNg1m1yJXQbXY83Hb7fv+PIY2K6OgldwbXOdXad+0FY+1wuTcneqdheAYv/BQOvgfATJ28PfryCAw2N/P5cLbUgIiLSFvVIjOTtXwznmufncfY3A3n8gg84p+wx+Pw+WPQqjLvX6e1rwczbItKKGuqdWS6bJm91Gw/31NWVQcO+I48JiYboLk5vXcaZEJMO0ekQ43qtnSZ2zVGydyrmPgG2AYbdeMJdl26u498LyvjpiEwy4sNbITgRERE5FcnRIfzr58OY8uJ8pvyngtvPu52fDrwWpt8K/7oKuo2B8X+BhCxvhyrSsdTvh6rlsGWJM7SyaWK3fbPzvbyp8EQngUvuCznnOoldjCuZi0736oQprU3J3snaux3mPw+9LoTYzOPuenCphU5hQdw0tmcrBSgiIiKnKjo0kH9eN5hfv7GIuz8ooWpkN26b8jl+C56DmffAE8NgyPUw6tYO9YVRpNU0HIDqUtd1c65b5TJo2O9sN34QleokbV2HuXrkDvbMdYXoVA27bkLJ3sla8ALs2w7Dbznhrh8t28K8dVu5+8I+RIdqqQUREZH2ICTQn0cvzyc+YhlPfbGWqu17+cslPyOoz8Uw448w+3FY8E/I+gHkXgA9zoLgCG+HLdL+NNRDzcqjErulUL/X2R4cDSn9YOgNh5cxiOkC/vpe3VJK9k5G/X6Y83fIHOn8wR3HvvoG7plWQlZSBJcN0lILIiIi7Ym/n+GPE3qTFBXCAx+toGbnfp780UAiJjwKA6+F+c85s3IvfRP8g6FHoXNNX/Z4n7reR8RtGhugdnWTxG4RbCmGA7ud7UERzmQog37qfM9OGQCdMsHPz7txt3NK9k7Gkn/DjgqY+NgJd33+6/Vs2rqHl34ymAB//ZGKiIi0N8YYbhzTg8TIYG57ewmXPTWb568ZTEJqPqTmO70Sm+ZAyVQoed9J/ow/ZAyHnAsg5zxnSJlIR9PYCFvXuhYdb7L4+P6dzvbAMGeGy/wfH07s4noosfMAY631dgwnpaCgwM6fP7/1K25sdMbp+wXA9V8ddzau6h37GPPgZwzJjOXZawa1YpAiIr7FGLPAWlvg7TjaC6+1kR3AzBVV/OLlhSREBvPP6waTefSka9Y6X2pL3neSv5oVzuupA52hnjkXQHyP1g9cxNOsdWa8LPvmcK9dxWLnsieAgBBnopSDSV3KAIjPAj9/78bdzrW0fVTPXkut+ti5WHTS0yecdvmZr9ay90ADvz9PSy2IiMj3GWPGAQ8D/sAz1tr7jrHfIGAOMNla+2YrhihHGZOdyGtThnLdC99wyROzeO6aQfRLb7KgsjGHv8gW3gHVK6H0fSf5+/RO55aQC7nnO8lfcp6WcZD2af9u54eNsm+c26ZvYOcWZ5t/ECT1gb4/PPx5SMjWNXZepGSvpWY94szy0/uiE+76yfJKhnWPo3uCLtYWEZEjGWP8gceBs4Ey4BtjzHvW2uXN7Hc/8FHrRynN6Z8ew5vXD+PHz8/j4idmceWQLvzyrCxiw4O+v3NCFiT8N5z538708KUfQOlU+PKv8MUDziQTORc4yV/6EPVySNtkLXy3zknoDiZ3lUuhsd7Z3inTmcsibRCkFTiJXkAznwfxGiV7LbHpG9jwNYy774S/TKyv2cXa6l1cPbRrKwUnIiLtzGBgtbV2LYAx5nVgIrD8qP1uBt4CdD1AG9ItIYJ3bxzB3z5ZwctzN/L2ws3cOLYH15yRQUjgMRK2mHQYer1z21UDK6Y7id83T8OcxyE8AbLPhdwJzhdnfVkWb9m3AzYvhLJ5UDbfSe521zrbgiKca1WH/9JJ7lILICLBu/HKCSnZa4lZD0NIDAz40Ql3LSqtAmBsTpKnoxIRkfYpFdjU5HkZMKTpDsaYVOAiYCxK9tqc2PAg7r6wLz8elsF900u5b3opL83ewP+My2ZCvxTM8YZnhsdD/o+c274dsOoTZ6jn0rdg4T8hOMpZ0iHnfCfx08ye4imNjVC7yjUU05XcVS0HXPN5xGdB1jhXr90gSMxVD3Q7pGTvRGpWOxdan/nfLVpDZ+aKKnokRtAlLqwVghMRkXaouUzg6NnSHgJutdY2HDdxAIwxU4ApAF26dHFLgNIyPZMiefaaQXy9uoZ7Pijhl68v4rmv13P7ebkMymhBkhYcCX0mObcDe2Hd54dn9Vzyb2ef6HTn+r7kvtDZdR+druv95OTt+Q7KFriGY85zHu+rc7aFRDs9dbkXQPogZ2Kh0E7ejVfcQsneicx+1LnYdMjPT7jrzn31zFlby7XDM1shMBERaafKgKYLsKYB5UftUwC87kr04oFzjTH11tr/HF2YtfYp4ClwZuP0SMRyXMN7xDP15hG8/e1mHvxoBT98cjbjeidz6/ic78/aeSyBIU6PXtYPXEs6zHW+lG9Z4qxFtmIah34TCIlxJX/9nPvkPKcXxl9f6zo8a2FnJdSschYrr1nl9N7VrIRtG519jB8k9oI+F7l67QZr2QMfpv8VjmdnFSx6DfpfARGJJ9z9q1U1HGiwjMk+8b4iItJhfQP0NMZkApuBy4Armu5grT30q6Ex5gVganOJnrQdfn6GSwamcV7fzjzz5Vqe+HwNn5ZU8qNhXbllbE86NTeJy7H4Bzhr9WUMP/za/l1QudxJ/LYUO0ngN89A/V7XMcGQ1Otw8pecB0m9WzQqSdqh+n3OOnbfS+pWHV7yAJz17OJ6OAld/o9d19rlO73K0iEo2Tueuf+Ahv1wxs0t2r2otJLIkAAKMtTtLSIizbPW1htjbsKZZdMfeM5au8wYc71r+5NeDVBOS2iQPzcX9mTy4HT+75NV/HPWet5aUMbNY3ty9RldCQ44xWuegsKd4XXpTS7hbKiH2tWHE8CKYmcY6MIXXTsYiOt+1DDQvBb9gC1tgLXO5CgHk7mmSd1368E2Ht43KtVJ6vImQ3xP1y0LIlPUY9fBaVH1Y9m3E/6vF2SOgskvnXD3xkbLkD/PYHBmLI9fke/5+EREOgAtqn5ytKh627Oycgf3TivhsxXVpMeGcuu4HM7r2/n4k7icDmth+2an56+i+HAieHAIH0BEsqsHsK8zU2hkCkR1du7D4pQctLb9u51FyQ8Ot6xZ7dzXrnKuszsoIMRJ6OJ6OIlcfBbEu56rp67D0aLqp2vhi7C3zpletgWWltdRvWMfhTn6tUxEREQcWUmRvHDtYL5cVc09H5Rw06vf8myXddx+Xi4Du3pgpk1jIDrNuWWPP/z6nu9gy9LDQ0ArimFNEdiGI4/3C4TIzq7krzNEpRx173o9MNT9sfuahnrn+rkdW2BHuXO/3XW/o+LwbW/dkcdFJDmJXK8Lj0zqotM1G6acNCV7zWk4ALMfh67DnQUiW6CotApjYFSW1hsRERGRI53ZM4EPbonnrYVlPPjRCi5+Yjbn9k3m1nE5dI1r4SQupyO0E2Se6dwOOpSMVLiSkKPuK5c6S0Mc2NV8eYd6BJOP7B309V5Ca53keUcFbK84MnFr+nxnFd+baNf4u96vzk6PXOZI53lUKsT1dJK6kGivnJb4JiV7zVn6Nmwvg/P/1uJDikqrGJAeQ1xEsAcDExERkfbK389waUE65+d15ukv1vGPL9bwyfJKrh6Wwc1jexAT1sqLqfsHQHSqczsWa50JP7ZXOL1TR9y7ksItS5pPbPwCnXUF/QKcHinj79z7Bbge+zV5fPB1v2b2P/i4mdcPHuNJthF2VR/ZM9ew7/v7hcU5SVxkZ2eIbNOe0IO38ATfTIClzVKydzRr4euHISEXepzdokOqduyluKyO35yT5eHgREREpL0LCwrgl2f15PLB6fztk5U8//U63lxQxs1je/CjYacxiYsnGOP0NIVEQ2LOsfdrOOD0Eh5MBg8mRrtrobHBGS7aWN/k8cFb/ZHP7f6jXm9sZp8mZTXW8/1lKt3+JjiJXFQKpA8+nLgdkcglQ4B+8Je2R8ne0VbPgKplcOETLf7l5bMV1QCMzUnyZGQiIiLiQxKjQrjv4jyuGZ7BvdNKufuDEl6cvYHbxucwvk+y5yZx8QT/wMPXCopIm6F+5KPNetgZZ97nkhYfUlRSRefoEHI7ayYkEREROTk5yVG8eN1g/nndYMKC/PnFKwu5+IlZfLmqmsbG9jVruoi0LerZa2rzQlj3BZxzNwS0bNz8/vpGvlxVzcQBqe3rFzgRERFpU0ZlJTCiRzxvLtjEgx+v5EfPziMjLozLB3fhkoFpmhdARE6akr2mZj0CwdGQ/+MWHzJv3VZ27W9gbLaWXBAREZHT4+9nmDyoCxP7p/LRsi28Mmcjf55eyl8/Xsm4PslcMaQLQzJj9QOziLSIR5M9Y8w44GHAH3jGWnvfMfYbBMwBJltr3/RkTMe0dS0sfxfOuAVColp8WFFpFUEBfpzRI86DwYmIiEhHEhLoz8T+qUzsn8qqyh28Mncjby8s473F5XRPCOeKIV25OD+19WfwFJF2xWPX7Blj/IHHgfFAL+ByY0yvY+x3P/CRp2JpkdmPO9P3Drn+pA4rKq3kjO5xhAWpk1RERETcr2dSJHdO6M3c35/FA5fkERUayF1TlzPk3hn8+o1FLNjwHdbq2j4R+T5PZiiDgdXW2rUAxpjXgYnA8qP2uxl4CxjkwViOb1cNfPsy5E12ptFtobXVO1lfu5vrRmR6MDgRERERCA3y54cF6fywIJ3l5dt5dd4G/vNtOW8v3ExOciRXDOnChQNSiQoJ9HaoItJGeHI2zlRgU5PnZa7XDjHGpAIXAU96MI4Tm/c01O91hnCehKLSKgDG6Ho9ERERaUW9UqK4+8K+zP19IX+e1JcAf8Md7y5jyD0zuPXNYorLtnk7RBFpAzzZs9fclcNHjzF4CLjVWttwvAuNjTFTgCkAXbp0cVuAAOzfBfOeguzzIOHkFkUvKq0iKymC9Ngw98YkIiIi0gLhwQFcPrgLlw/uQnHZNl6du5F3F5Xzr/mb6JMaxZVDujKhXwrhwbrcRKQj8mTPXhmQ3uR5GlB+1D4FwOvGmPXAJcDfjTEXHl2QtfYpa22BtbYgISHBvVF++wrs2QrDf3lSh23fe4B567ZqIXURERFpE/LSYrjv4jzm/qGQuyb2pr7B8ru3lzDk3hnc/p8lLC/f7u0QRaSVefJnnm+AnsaYTGAzcBlwRdMdrLWHLnYzxrwATLXW/seDMR2poR5mPwrpQ6HLkJM69KtVNdQ3WsbmaAiniIiItB1RIYH8aFgGVw3tysKN23hl7gb+Pb+Ml+dsZECXGK4Y3IXz81IIDfL3dqgi4mEeS/astfXGmJtwZtn0B56z1i4zxlzv2u7d6/QAlv8Htm2Ecc2uCHFcRaVVRIcGkt8lxgOBiYiIiJweYwwDu3ZiYNdO3HF+L95auJlX527gt28Wc9fU5UzKT2NSfip9U6O1bp+Ij/LoAG5r7TRg2lGvNZvkWWuv8WQszVQIXz8McT0ha/xJHdrYaPlsRRWjshII8PfkSFgRERGR0xcTFsRPRmRy3fAM5q7byqtzN/Lq3I28MGs9XePCuCAvhQv6pZCdHOntUEXEjTru1brrPoctxTDhUfA7uYSteHMdNTv3U5irIZwiIiLSfhhjGNotjqHd4qjbfYCPlm3h/eJy/v7Zah6buZrspEgu6NeZ8/NSyIgP93a4InKaOm6y9/XDEJHkrK13kopKKvEzMCrLzZPFiIiIiLSS6LBALh2UzqWD0qnesY/pSyt4f3E5D368kgc/XkleWjQX5KVwXl5nUmJCvR2uiJyCjpnsVRTDmiI4604ICD7pw4tWVDGwaydiwoLcHpqIiIhIa0uIDObqYRlcPSyD8m17+KC4gveLy7lnWgn3TCthUEYnLuiXwvg+nUmIPPnvTiLiHR0z2Zv1CARFwsBrT/rQyu17Wbp5O/8zLtsDgYmIiIh4V0pMKD8b2Y2fjezG+ppdTC0u5/3FFdzx7jLufG8ZZ3SP54J+nRnXuzPRYYHeDldEjqPjJXvfbYClb8OwX0Doyc+kObO0CkBLLoiIiIjPy4gP56axPblpbE9WbNnB1OJy3ltczq1vLeH2/yxlZM8ELuiXwlm9kojQwu0ibU7H+1TuqoHkPjDkhlM6fEZpFakxoWQnabYqERER6TiykyPJTs7m12dnsWRzHe8vLmdqcQUzSqsIDvCjMDeRCf1SGJ2dSEig1vATaQs6XrKXNhB+/sUpHbqvvoGvV9cwKT9V69GIiIhIh2SMIS8thry0GH43PpcFG7/j/cXlTFtSwbQlW4gIDuCcXklc0C+F4T3iCQrQMlUi3tLxkr3TMHftVnbvb6AwJ8nboYiIiIh4nZ+fYVBGLIMyYrnj/F7MWbuV9xeXM31pBW9/u5mokABGZSdSmJPIqKwEOoVrcjuR1qRk7yQUlVYREujHsO5x3g5FREREpE0J8PdjRM94RvSM564L+/DlqmqmL93CZyuqeH9xOX4G8rt0YmxuIoU5SWQlRWiklIiHKdlrIWstM0orGd49XuPQRURERI4jKMCPwtwkCnOTaGy0FG+uo6ikkqIVVfzlwxX85cMVpMaEMjYnkbE5iQzrHqfvVyIeoGSvhdZU72TT1j38fGR3b4ciIiIi0m74+Rn6p8fQPz2GX5+TTeX2vcwsrWJGaRVvLijjpTkbCAn0Y0SPeMa4kr/O0VrEXcQdlOy1UJGWXBARERE5bUlRIVw2uAuXDe7C3gMNzF23laKSSmaUVvFpifN9q1fnKKfXLzeRfmkx+PtpuKfIqVCy10IzSqrISY4kJUa/NImIiIi4Q0igP6OyEhiVlcCdEyyrq3Yyo7SKopIqnvh8DY/NXE1seBCjsxMozEnizKx4okK0kLtISynZa4G6PQeYv+E7rh/VzduhiIiIiPgkYww9kyLpmRTJ9aO6s233fj5fWU1RaRUzSqp4e+FmAlyzfxbmJjImJ5Fu8eGa5EXkOJTstcAXK6tpaLQawikiIiLSSmLCgpjYP5WJ/VOpb2jk203bmFFSxczSKu7+oIS7PyghIy6M0dlO4jckM1aTvIgcRcleC8wsraJTWCD90zt5OxQRERGRDifA3+/Qen63jc+h7LvdhyZ5eW3eRl6YtZ6QQD/O6B7P6OwERmcl0iUuzNthi3idkr0TaGi0zFxRxejsRF0cLCIiItIGpHUK40fDMvjRsAz27G9gzrpaPiutYuaKatekesvolhDO6KxExuQkMDgzluAA9fpJx6Nk7wQWbdrGd7sPMEZDOEVERETanNAgf8ZkJzImO5E7rWVdzS4+W1HNZyureXnuBp77eh2hgf6c0T2O0TmJjM5KID1WvX7SMSjZO4GZpVX4+xlG9UzwdigiIiIichzGGLolRNAtIYLrRmSyZ38Ds9fW8NmKamaucIZ9AvRIjGB0VgKjsxMZlNlJvX7is5TsncCM0ioGdu1EdJim+RURERFpT0KD/Bmbk8TYnCSstaw92Ou3oooXZ2/gma/WERbkf/hav+wE0jqp1098h5K946io20NJxXZ+Nz7H26GIiIiIyGkwxtA9IYLuCRH8ZEQmu/fXM3tNLTNXVPHZimo+LakEoGdiBKOzExiTnUhBRixBAX5ejlzk1CnZO44iV1e/llwQERER8S1hQQEU5iZRmOv0+q2p3unq9avmn7M28PSX6wgP8md4j3gKcxMZm5NEQmSwt8MWOSlK9o5jZmkV6bGh9EiM8HYoIiIiIuIhxhh6JEbSIzGSn57ZjV376pm1ppbPXL1+Hy+vxJglDEiP4axeSZydm0SPxAgt6C5tnpK9Y9h7oIGvVtcwuSBdH2QRERGRDiQ8OICzeyVxdi+n16+kYgefllTyaUklf/lwBX/5cAUZcWGclZvEWb2SKOjaiQB/DfeUtkfJ3jHMXlvL3gONjM1N8nYoIiIiIuIlxhh6pUTRKyWKWwp7UlG3hxklVXxaUnlokpeYsEDGZidyVq8kRmYlEBGsr9jSNugv8RiKSqoIDfRnSGast0MRERERkTaic3QoVw3tylVDu7JzXz1frqzmk5JKikqrePvbzQT5+zG0exxn90rirNxEOkeHejtk6cCU7DXDWktRaRUjesYTEqh1V0RExL2MMeOAhwF/4Blr7X1HbZ8I3AU0AvXAr6y1X7V6oCJyXBHBAYzv25nxfTtT39DIwo3b+GT5Fj5ZXsn/+89S/t9/oE9qFGfnJnNWr0R6dY7S5UHSqpTsNWNl5U42b9vDTWN7eDsUERHxMcYYf+Bx4GygDPjGGPOetXZ5k91mAO9Za60xJg94A9A6QCJtWIC/H4MzYxmcGcvvz81lTfUuPi2p5JPllTw0YyX/9+lKUmNCOSvXGe45JDNOyzqIxynZa8bBJRfGZGvJBRERcbvBwGpr7VoAY8zrwETgULJnrd3ZZP9wwLZqhCJyWpzZPSPokRjB9aO6U7NzH0WlVXyyvJJ/zd/EP2dvIDI4gJHZCZzTK4nRWYlEhwV6O2zxQUr2mjGztIreKVEkR4d4OxQREfE9qcCmJs/LgCFH72SMuQj4M5AInNc6oYmIJ8RHBHNpQTqXFqSz90ADX6+ucfX6VfFBcQX+foaCrp0OrefXPSFcwz3FLZTsHWXb7v3M37CVG8doCKeIryVaJQAAGmBJREFUiHhEc9/gvtdzZ619B3jHGDMS5/q9s5otzJgpwBSALl26uDFMEfGEkED/Q4u533OhZXHZNj4tqWRGSRX3Tivl3mmlZMSFMTYnicLcRAZlxGq4p5wyJXtH+XxlNY0WxuZoCKeIiHhEGZDe5HkaUH6sna21Xxhjuhtj4q21Nc1sfwp4CqCgoEDDPUXaET8/w4AunRjQpRO//UEOm7ftoai0iqKSSl6eu4Hnvl5HRHAAI7PiGZuTxOjsBOIjgr0dtrQjSvaOUlRaRVx4EP3SYrwdioiI+KZvgJ7GmExgM3AZcEXTHYwxPYA1rgla8oEgoLbVIxWRVpUaE8qPhnblR0O7snt/PbNW1zKj1On1m7ZkC8ZA//QYzspNYmxOIjnJkRruKcelZK+J+oZGPl9ZTWFOEn5++uCIyPcdOHCAsrIy9u7d6+1QfEpISAhpaWkEBvr+BAXW2npjzE3ARzhLLzxnrV1mjLnetf1J4GLgamPMAWAPMNlaq147kQ4kLCiAs3olcVavJKy1LCvfzoySKopKK3ngoxU88NEKUqJDGJubSGFOEsO6x2nJMPkeJXtNfLtpG9t2H9AQThE5prKyMiIjI8nIyNCvqW5iraW2tpaysjIyMzO9HU6rsNZOA6Yd9dqTTR7fD9zf2nGJSNtkjKFPajR9UqP55Vk9qdqxl89Kq/m0pJK3F27m5TkbCQn0Y0QPZ7jn2JxETTQogJK9IxSVVhHgZzgzK97boYhIG7V3714lem5mjCEuLo7q6mpvhyIi0i4kRoZw6aB0Lh3kzO45d91Wikoq+bSkik9LnCXE+qRGOZO85CTSNzVao9Y6KCV7TRSVVDEoI5aoEN8fRiQip06JnvvpPRUROTUhgf6MykpgVFYCd06wrKrayacllRSVVPFY0SoembGK+IhgRmUlMKx7HMO6x5EaE+rtsKWVKNlzKftuNysqd3D7ebneDkVE5Jhqa2spLCwEYMuWLfj7+5OQkADAvHnzCAoKOmEZ1157LbfddhvZ2dnH3Ofxxx8nJiaGK6+80j2Bi4iIxxljyEqKJCspkl+M7sHWXfv5fKXT2zejtJK3FpYB0CU2jGHdnMRvaLc4Dfn0YUr2XGaWOl3eY3S9noi0YXFxcSxatAiAO++8k4iICH7zm98csY+1Fmstfn7Nr8v0/PPPn7CeG2+88fSDFRERr4oND+KiAWlcNCCNxkZL6ZYdzF5by5y1tUxfWsG/5m8CIDM+nKGHkr9YEiOV/PkKJXsuRaVVdI0Lo1t8uLdDERE5aatXr+bCCy9kxIgRzJ07l6lTp/LHP/6RhQsXsmfPHiZPnswdd9wBwIgRI3jsscfo06cP8fHxXH/99UyfPp2wsDDeffddEhMTuf3224mPj+dXv/oVI0aMYMSIERQVFVFXV8fzzz/PGWecwa5du7j66qtZvXo1vXr1YtWqVTzzzDP079/fy++GiIgczc/P0Cslil4pUfxkRCYNjZaSiu3MXlPL7LW1vL+4nNfmbQSgR2LEoZ6/IZmxxGltv3ZLyR6wZ38Ds9bUcsWQLrpuRERa7I/vL2N5+Xa3ltkrJYr/vaD3KR27fPlynn/+eZ580pnU8b777iM2Npb6+nrGjBnDJZdcQq9evY44pq6ujlGjRnHffffx61//mueee47bbrvte2Vba5k3bx7vvfcef/rTn/jwww959NFHSU5O5q233mLx4sXk5+efUtwiItL6/P0Oz/D5s5HdqG9oZFn5dmavrWX2mlreWljGS3M2AJCdFHloyOfQbrHEhJ34kgFpG5TsAbPW1LCvvlFLLohIu9a9e3cGDRp06Plrr73Gs88+S319PeXl5Sxfvvx7yV5oaCjjx48HYODAgXz55ZfNlj1p0qRD+6xfvx6Ar776iltvvRWAfv360bv3qSWpIiLifQH+fvRLj6FfegzXj+rOgYZGisvqmOMa9vn6Nxt5YdZ6jIHc5ChnspducQzKjCU6VJMbtlVK9nCGcIYH+TM4M9bboYhIO3KqPXCeEh5+eBj6qlWrePjhh5k3bx4xMTFcddVVzS4E33RCF39/f+rr65stOzg4+Hv7aI1vERHfFejvx8CunRjYtRM3junB/vpGFpdtc4Z9rqnlpTkbePardfgZ6J0SzbDucQzs2om8tGiSo0I0Wq6N6PDJnrWWotIqRvSMJzjA39vhiIi4xfbt24mMjCQqKoqKigo++ugjxo0b59Y6RowYwRtvvMGZZ57JkiVLWL58uVvLFxGRtiMowI9BGbEMyojllsKe7D3QwKJN2w5d8/fC1+t56ou1AMRHBNMvLZq+adHkpUWTlxZDvK7784oOn+yVbtlBRd1e/uusLG+HIiLiNvn5+fTq1Ys+ffrQrVs3hg8f7vY6br75Zq6++mry8vLIz8+nT58+REdHu70eERFpe0IC/V3X8MXxX8DeAw2UVGynuKzOddtG0YoqDg4CSYkOcSV/MeSlRdM3NVrX/rUC096G4RQUFNj58+e7rbzHZ67mgY9WMO8PhZpmVkROqKSkhNxcrccJUF9fT319PSEhIaxatYpzzjmHVatWERBwar8jNvfeGmMWWGsL3BFvR+DuNlJE5HTs2lfPsvLtFJdto7isjiWb61hXs+vQ9q5xYfRNPdz71yc1mojgDt8X1SItbR87/LtZVFpFXlq0Ej0RkZO0c+dOCgsLqa+vx1rLP/7xj1NO9ERExPeEBwcwODP2iHkx6nYfYGn54d6/bzduY2pxBQDGQLf4cPqlxRwaAtqrczShQbrU6lR16FZ56679LNz4HbeM7entUERE2p2YmBgWLFjg7TBERKQdiQ4LZHiPeIb3iD/0Ws3OfSzZXMcSVwL45eoa3v52M+AsEdEzMcIZ+pkWQ8/ECHokRhAXHqRJYFqgQyd7n690xhEX5mrJBRERERERb4iPCGZMdiJjsg9/J6/cvpfFm7axZLPTC/jJ8kremF92aHt0aCA9EiPonhBO94QI1+MI0jqFEuDv543TaJM6dLI3o6SK+Ihg+qRoQgERERERkbYiKSqEc3onc07vZMCZQb+8bi+rq3aypmona6p3srpqJ0Wl1UckgUH+fmTGh9M98cgksFtCOGFBHS/18egZG2PGAQ8D/sAz1tr7jto+EbgLaATqgV9Za7/yZEwH1Tc08sXKasb1ScbPT13AIiIiIiJtlTGG1JhQUmNCGZWVcMS2ut0HWF3tJIBrqp1ksKRiBx8u3UJjk7koU2NC6XZUT2D3xHASIoJ9dkiox5I9Y4w/8DhwNlAGfGOMec9a23QhphnAe9Zaa4zJA94AcjwVU1MLNnzH9r31jM3REE4RERERkfYqOizw0ALwTe2rb2BD7e4jegLXVO/ijfmb2L2/4dB+USEBdHclfznJkQztFkdu5yj8faBDyJM9e4OB1dbatQDGmNeBicChZM9au7PJ/uFAq60DUVRaRaC/YUTPhBPvLCLSRowePZrf/e53/OAHPzj02kMPPcTKlSv5+9//3uwxERER7Ny5k/Lycm655RbefPPNZst98MEHKSg49izODz30EFOmTCEsLAyAc889l1dffZWYmJjTPCsRERH3Cw7wJyspkqykyCNet9ZSUbf3UC/g6uqdrKnaxRcrq3lzgTMkNDIkgCGZsYfWEmyvyZ8nk71UYFOT52XAkKN3MsZcBPwZSATO82A8RygqrWJIZpzW8hCRduXyyy/n9ddfPyLZe/3113nggQdOeGxKSkqziV5LPfTQQ1x11VWHkr1p06adclkiIiLeYowhJSaUlJhQzjyq42dL3V7mrqtlztpa5qzdyqclVYDT+zc4M46h3WLbVfLnyalqmjv77/XcWWvfsdbmABfiXL/3/YKMmWKMmW+MmV9dXX3agW3auptVVTsZoyGcItLOXHLJJUydOpV9+/YBsH79esrLy+nfvz+FhYXk5+fTt29f3n333e8du379evr06QPAnj17uOyyy8jLy2Py5Mns2bPn0H433HADBQUF9O7dm//93/8F4JFHHqG8vJwxY8YwZswYADIyMqipqQHgb3/7G3369KFPnz489NBDh+rLzc3lZz/7Gb179+acc845oh4REZG2Jjk6hIn9U/nzpDxm/mY0c35XyMOX9efcvp1ZXbWDuz8o4fxHvyL/rk/42YvzefardSwrr6OxsdUGKJ4UT3ZrlQHpTZ6nAeXH2tla+4UxprsxJt5aW3PUtqeApwAKCgpO+50sKnUy9EIleyJyOqbfBluWuLfM5L4w/r5jbo6Li2Pw4MF8+OGHTJw4kddff53JkycTGhrKO++8Q1RUFDU1NQwdOpQJEyYc84LzJ554grCwMIqLiykuLiY/P//QtnvuuYfY2FgaGhooLCykuLiYW265hb/97W/MnDmT+Pj4I8pasGABzz//PHPnzsVay5AhQxg1ahSdOnVi1apVvPbaazz99NNceumlvPXWW1x11VXuea9EREQ87GDyN7F/KgAVdXuYu3arq+evlk+WVwLOUhCDDw37jCU3OapNTALpyWTvG6CnMSYT2AxcBlzRdAdjTA9gjWuClnwgCKj1YEwAzCitolt8OBnx4Z6uSkTE7Q4O5TyY7D333HNYa/n973/PF198gZ+fH5s3b6ayspLk5ORmy/jiiy+45ZZbAMjLyyMvL+/QtjfeeIOnnnqK+vp6KioqWL58+RHbj/bVV19x0UUXER7u/J86adIkvvzySyZMmEBmZib9+/cHYODAgaxfv95N74KIiEjr6xwdyoUDUrlwgJP8lW/b4wz7XLOVOeuOTP6aXvOXkxzpleTPY8metbbeGHMT8BHO0gvPWWuXGWOud21/ErgYuNoYcwDYA0y21nq0D3TXvnrmrKnl6mFdPVmNiHQEx+mB86QLL7yQX//61yxcuJA9e/aQn5/PCy+8QHV1NQsWLCAwMJCMjAz27t173HKa6/Vbt24dDz74IN988w2dOnXimmuuOWE5x/tvOzg4+NBjf39/DeMUERGfkhITykUD0rhoQBpwOPmbvca55u/jZpK/ET3jvzdpjKd4dHYSa+00YNpRrz3Z5PH9wP2ejOFos9bUsr+hUUsuiEi7FRERwejRo7nuuuu4/PLLAairqyMxMZHAwEBmzpzJhg0bjlvGyJEjeeWVVxgzZgxLly6luLgYgO3btxMeHk50dDSVlZVMnz6d0aNHAxAZGcmOHTu+N4xz5MiRXHPNNdx2221Ya3nnnXd46aWX3H/iIiIibdzRyd/mbXuYu/bwhC8fL6/kvL6defzK/BOU5B4dbirKsTmJvHn9MPLSNFW4iLRfl19+OZMmTeL1118H4Morr+SCCy6goKCA/v37k5Nz/CVLb7jhBq699lry8vLo378/gwcPBqBfv34MGDCA3r17061bN4YPH37omClTpjB+/Hg6d+7MzJkzD72en5/PNddcc6iMn/70pwwYMEBDNkVEpMNLjQllUn4ak/Kd5K/su93sq29stfqNh0dNul1BQYGdP3++t8MQkQ6qpKSE3Nxcb4fhk5p7b40xC6y1x178T46gNlJEpGNoafvoyaUXRERERERExEuU7ImIiIiIiPggJXsiIiIiIiI+SMmeiMhJam/XOrcHek9FRETcT8meiMhJCAkJoba2VsmJG1lrqa2tJSQkxNuhiIiI+JQOt/SCiMjpSEtLo6ysjOrqam+H4lNCQkJIS0vzdhgiIiI+RcmeiMhJCAwMJDMz09thiIiIiJyQhnGKiIiIiIj4ICV7IiIiIiIiPkjJnoiIiIiIiA8y7W1GOWNMNbDhNIuJB2rcEI7qaPvl+0odvnAOvlKHL5xDa9ThrvK7WmsT3FBOh6A2stXKVx1tp3xfqcMXzqE16vCFc3BXHS1qH9tdsucOxpj51toC1eH9OnzhHFqjDl84B1+pwxfOoTXqaI1zEM/Q35/qaE/l+0odvnAOrVGHL5xDa9VxkIZxioiIiIiI+CAleyIiIiIiIj6ooyZ7T6mONlOHL5xDa9ThC+fgK3X4wjm0Rh2tcQ7iGfr7Ux3tqXxfqcMXzqE16vCFc2itOoAOes2eiIiIiIiIr+uoPXsiIiIiIiI+rUMle8aY54wxVcaYpR6sI90YM9MYU2KMWWaM+aWbyw8xxswzxix2lf9Hd5Z/VF3+xphvjTFTPVT+emPMEmPMImPMfA/VEWOMedMYU+r6NxnmxrKzXbEfvG03xvzKXeU3qee/XP/WS40xrxljQjxQxy9d5S9z1zk093kzxsQaYz4xxqxy3Xdyc/k/dJ1DozHmtGe5OkYdD7j+noqNMe8YY2I8UMddrvIXGWM+NsakuLP8Jtt+Y4yxxpj4Uy3/WHUYY+40xmxu8vk493TqEM/zdBvp6fbRVUertJFqH1tUvsfbSLWPJ11Hu2ojPd0+HquOJtt8o4201naYGzASyAeWerCOzkC+63EksBLo5cbyDRDhehwIzAWGeuhcfg28Ckz1UPnrgXgP/5v/E/ip63EQEOOhevyBLThrnriz3FRgHRDqev4GcI2b6+gDLAXCgADgU6CnG8r93ucN+Atwm+vxbcD9bi4/F8gGPgMKPHQO5wABrsf3n845HKeOqCaPbwGedGf5rtfTgY9w1mQ7rc/hMc7hTuA37vxb1c2zN0+3kZ5uH13ltkobqfbxpOtyexup9vGU6mhXbaSn28dj1eF63WfayA7Vs2et/QLY6uE6Kqy1C12PdwAlOP8huat8a63d6Xoa6Lq5/cJLY0wacB7wjLvLbi3GmCicD9izANba/dbabR6qrhBYY6093cWMmxMAhBpjAnAanHI3l58LzLHW7rbW1gOfAxedbqHH+LxNxPmCgev+QneWb60tsdauONUyW1jHx673CWAOkOaBOrY3eRrOaXzGj/P/3v8B/3M6ZbegDmlHPP3v6On20VWux9tItY+nxFNtpNrHk6ijvbWRnm4fj1WHi8+0kR0q2WttxpgMYADOL4vuLNffGLMIqAI+sda6tXyXh3D+yBs9UPZBFvjYGLPAGDPFA+V3A6qB513DbZ4xxoR7oB6Ay4DX3F2otXYz8CCwEagA6qy1H7u5mqXASGNMnDEmDDgX5xctT0iy1laA88UPSPRQPa3lOmC6Jwo2xtxjjNkEXAnc4eayJwCbrbWL3VluM25yDbd57nSHJIlv8VT76Crb022k2seT5/Y2Uu1ju+CRNtKT7aOrfJ9qI5XseYgxJgJ4C/jVUb9CnDZrbYO1tj/OryWDjTF93Fm+MeZ8oMpau8Cd5TZjuLU2HxgP3GiMGenm8gNwus2fsNYOAHbhDI1wK2NMEDAB+LcHyu6E82tfJpAChBtjrnJnHdbaEpyhFp8AHwKLgfrjHiQYY/6A8z694onyrbV/sNamu8q/yV3lur6w/AEPNJBHeQLoDvTH+SL2Vw/XJ+2EJ9tH8Gwbqfbx5HmqjVT72LZ5so30VPsIvtlGKtnzAGNMIE5D9oq19m1P1eMacvEZMM7NRQ8HJhhj1gOvA2ONMS+7uQ6steWu+yrgHWCwm6soA8qa/Kr7Jk7j5m7jgYXW2koPlH0WsM5aW22tPQC8DZzh7kqstc9aa/OttSNxhhqscncdLpXGmM4ArvsqD9XjUcaYHwPnA1da1+B7D3oVuNiN5XXH+XK02PUZTwMWGmOS3VgH1tpK15fuRuBp3P/5lnaotdpH8Fgbqfbx5HmqjVT72Ea1Yhvp7vYRfLCNVLLnZsYYgzMGvsRa+zcPlJ9wcGYjY0wozn92pe6sw1r7O2ttmrU2A2foRZG11q2/lhljwo0xkQcf41zQ69YZ4Ky1W4BNxphs10uFwHJ31uFyOR4YwumyERhqjAlz/W0V4lzn4lbGmETXfRdgEp47n/eAH7se/xh410P1eIwxZhxwKzDBWrvbQ3X0bPJ0Am78jFtrl1hrE621Ga7PeBnOpBlb3FUHHPqyctBFuPnzLe2Pp9tHVx0ebSPVPp4ST7WRah/bIE+3kZ5sH8FH20jbCrPAtJUbzge0AjiA84/3Ew/UMQJnrH0xsMh1O9eN5ecB37rKXwrc4eH3bDQemG0M53qBxa7bMuAPHoq/PzDf9X79B+jk5vLDgFog2oP/Bn/E+c9sKfASEOyBOr7EaegXA4VuKvN7nzcgDpiB88voDCDWzeVf5Hq8D6gEPvLAOawGNjX5fJ/uTGDN1fGW69+7GHgfSHVn+UdtX8/pzzTW3Dm8BCxxncN7QGd3/93q5t6bp9tIT7ePrjparY1U+9iiOjzaRqp9POk62lUb6en28Vh1HLW93beRxhWEiIiIiIiI+BAN4xQREREREfFBSvZERERERER8kJI9ERERERERH6RkT0RERERExAcp2RMREREREfFBSvZEWpExpsEYs6jJ7TY3lp1hjNFaZiIi0i6pjRRxvwBvByDSweyx1vb3dhAiIiJtkNpIETdTz55IG2CMWW+Mud8YM8916+F6vasxZoYxpth138X1epIx5h1jzGLX7QxXUf7GmKeNMcuMMR8bY0K9dlIiIiJuoDZS5NQp2RNpXaFHDVGZ3GTbdmvtYOAx4CHXa48BL1pr84BXgEdcrz8CfG6t7QfkA8tcr/cEHrfW9ga2ARd7+HxERETcRW2kiJsZa623YxDpMIwxO621Ec28vh4Ya61da4wJBLZYa+OMMTVAZ2vtAdfrFdbaeGNMNZBmrd3XpIwM4BNrbU/X81uBQGvt3Z4/MxERkdOjNlLE/dSzJ9J22GM8PtY+zdnX5HEDui5XRER8g9pIkVOgZE+k7Zjc5H626/Es4DLX4yuBr1yPZwA3ABhj/I0xUa0VpIiIiBeojRQ5BfpFQ6R1hRpjFjV5/qG19uDU0sHGmLk4P8Jc7nrtFuA5Y8xvgWrgWtfrvwSeMsb8BOfXyRuACo9HLyIi4jlqI0XcTNfsibQBrusRCqy1Nd6ORUREpC1RGyly6jSMU0RERERExAepZ09ERERERMQHqWdPRERERETEBynZExERERER8UFK9kRERERERHyQkj0REREREREfpGRPRERERETEBynZExERERER8UH/H9nT00f2izv5AAAAAElFTkSuQmCC\n", 644 | "text/plain": [ 645 | "
" 646 | ] 647 | }, 648 | "metadata": { 649 | "needs_background": "light" 650 | }, 651 | "output_type": "display_data" 652 | } 653 | ], 654 | "source": [ 655 | "# Run this cell to plot accuracy vs epoch and loss vs epoch\n", 656 | "\n", 657 | "plt.figure(figsize=(15,5))\n", 658 | "plt.subplot(121)\n", 659 | "plt.plot(history_dict['sparse_categorical_accuracy'])\n", 660 | "plt.plot(history_dict['val_sparse_categorical_accuracy'])\n", 661 | "plt.title('Accuracy vs. epochs')\n", 662 | "plt.ylabel('Accuracy')\n", 663 | "plt.xlabel('Epoch')\n", 664 | "plt.xticks(np.arange(len(history_dict['sparse_categorical_accuracy'])))\n", 665 | "ax = plt.gca()\n", 666 | "ax.set_xticklabels(1 + np.arange(len(history_dict['sparse_categorical_accuracy'])))\n", 667 | "plt.legend(['Training', 'Validation'], loc='lower right')\n", 668 | "\n", 669 | "plt.subplot(122)\n", 670 | "plt.plot(history_dict['loss'])\n", 671 | "plt.plot(history_dict['val_loss'])\n", 672 | "plt.title('Loss vs. epochs')\n", 673 | "plt.ylabel('Loss')\n", 674 | "plt.xlabel('Epoch')\n", 675 | "plt.xticks(np.arange(len(history_dict['sparse_categorical_accuracy'])))\n", 676 | "ax = plt.gca()\n", 677 | "ax.set_xticklabels(1 + np.arange(len(history_dict['sparse_categorical_accuracy'])))\n", 678 | "plt.legend(['Training', 'Validation'], loc='upper right')\n", 679 | "plt.show() " 680 | ] 681 | }, 682 | { 683 | "cell_type": "markdown", 684 | "metadata": {}, 685 | "source": [ 686 | "#### Write a text generation algorithm\n", 687 | "\n", 688 | "You can now use the model to generate text! In order to generate a single text sequence, the model needs to be rebuilt with a batch size of 1." 689 | ] 690 | }, 691 | { 692 | "cell_type": "code", 693 | "execution_count": 24, 694 | "metadata": {}, 695 | "outputs": [ 696 | { 697 | "data": { 698 | "text/plain": [ 699 | "" 700 | ] 701 | }, 702 | "execution_count": 24, 703 | "metadata": {}, 704 | "output_type": "execute_result" 705 | } 706 | ], 707 | "source": [ 708 | "# Re-build the model and load the saved weights\n", 709 | "\n", 710 | "model = get_model(len(tokenizer.word_index) + 1, batch_size=1)\n", 711 | "model.load_weights(tf.train.latest_checkpoint('./models/'))" 712 | ] 713 | }, 714 | { 715 | "cell_type": "markdown", 716 | "metadata": {}, 717 | "source": [ 718 | "An algorithm to generate text is as follows:\n", 719 | "\n", 720 | "1. Specify a seed string (e.g. `'ROMEO:'`) to get the network started, and a define number of characters for the model to generate, `num_generation_steps`.\n", 721 | "2. Tokenize this sentence to obtain a list containing one list of the integer tokens.\n", 722 | "3. Reset the initial state of the network. \n", 723 | "4. Convert the token list into a Tensor (or numpy array) and pass it to your model as a batch of size one.\n", 724 | "5. Get the model prediction (logits) for the last time step and extract the state of the recurrent layer.\n", 725 | "6. Use the logits to construct a categorical distribution and sample a token from it.\n", 726 | "7. Repeat the following for `num_generation_steps - 1` steps:\n", 727 | "\n", 728 | " 1. Use the saved state of the recurrent layer and the last sampled token to get new logit predictions\n", 729 | " 2. Use the logits to construct a new categorical distribution and sample a token from it.\n", 730 | " 3. Save the updated state of the recurrent layer. \n", 731 | "\n", 732 | "8. Take the final list of tokens and convert to text using the Tokenizer.\n", 733 | "\n", 734 | "Note that the internal state of the recurrent layer can be accessed using the `states` property. For the GRU layer, it is a list of one variable:" 735 | ] 736 | }, 737 | { 738 | "cell_type": "code", 739 | "execution_count": 25, 740 | "metadata": {}, 741 | "outputs": [ 742 | { 743 | "data": { 744 | "text/plain": [ 745 | "[]" 746 | ] 747 | }, 748 | "execution_count": 25, 749 | "metadata": {}, 750 | "output_type": "execute_result" 751 | } 752 | ], 753 | "source": [ 754 | "# Inspect the model's current recurrent state\n", 755 | "\n", 756 | "model.layers[1].states" 757 | ] 758 | }, 759 | { 760 | "cell_type": "markdown", 761 | "metadata": {}, 762 | "source": [ 763 | "We will break the algorithm down into two steps. First, you should now complete the following function that takes a sequence of tokens of any length and returns the model's prediction (the logits) for the last time step. The specification is as follows:\n", 764 | "\n", 765 | "* The token sequence will be a python list, containing one list of integer tokens, e.g. `[[1, 2, 3, 4]]`\n", 766 | "* The function should convert the list into a 2D Tensor or numpy array\n", 767 | "* If the function argument `initial_state` is `None`, then the function should reset the state of the recurrent layer to zeros.\n", 768 | "* Otherwise, if the function argument `initial_state` is a 2D Tensor or numpy array, assign the value of the internal state of the GRU layer to this argument.\n", 769 | "* Get the model's prediction (logits) for the last time step only.\n", 770 | "\n", 771 | "The function should then return the logits as a 2D numpy array, where the first dimension is equal to 1 (batch size).\n", 772 | "\n", 773 | "**Hint:** the internal state of the recurrent can be reset to zeros using the `reset_states` method." 774 | ] 775 | }, 776 | { 777 | "cell_type": "code", 778 | "execution_count": 26, 779 | "metadata": {}, 780 | "outputs": [], 781 | "source": [ 782 | "#### GRADED CELL ####\n", 783 | "\n", 784 | "# Complete the following function.\n", 785 | "# Make sure not to change the function name or arguments.\n", 786 | "\n", 787 | "def get_logits(model, token_sequence, initial_state=None):\n", 788 | " \"\"\"\n", 789 | " This function takes a model object, a token sequence and an optional initial\n", 790 | " state for the recurrent layer. The function should return the logits prediction\n", 791 | " for the final time step as a 2D numpy array.\n", 792 | " \"\"\"\n", 793 | " if initial_state is None:\n", 794 | " model.layers[1].reset_states() \n", 795 | " else:\n", 796 | " model.layers[1].reset_states(states=initial_state)\n", 797 | " # model.layers[1].initial_state = initial_state\n", 798 | " token_sequence = np.array(token_sequence)\n", 799 | " predicted = model.predict(token_sequence) # (1, 4, 65)\n", 800 | " # shape should be 2D\n", 801 | " pred = np.expand_dims(predicted[0,-1,:], 0) # (1, 65)\n", 802 | " # print(pred.shape)\n", 803 | " return pred" 804 | ] 805 | }, 806 | { 807 | "cell_type": "code", 808 | "execution_count": 27, 809 | "metadata": {}, 810 | "outputs": [ 811 | { 812 | "data": { 813 | "text/plain": [ 814 | "array([[-6.905809 , 1.156763 , 1.5588742 , 0.29829672, 2.034187 ,\n", 815 | " -0.10080466, -6.5318418 , 2.7604616 , 8.012833 , 3.2324765 ,\n", 816 | " 2.2604296 , -1.3730426 , 4.8131843 , 0.3714735 , 5.2470407 ,\n", 817 | " 6.258843 , -1.811632 , 0.34183514, 2.6086504 , -0.824508 ,\n", 818 | " 2.7813735 , -1.0732076 , -9.556668 , -1.7602841 , 2.939676 ,\n", 819 | " -2.0531266 , -5.32776 , -1.501653 , -1.3735346 , -3.3129911 ,\n", 820 | " 1.3639722 , -6.2925954 , -7.213993 , -2.7346692 , -0.98425233,\n", 821 | " -5.4954934 , -6.239683 , -5.477082 , -3.3806677 , -2.3722863 ,\n", 822 | " -6.9433165 , -7.1003103 , -4.306319 , -2.703047 , -3.3444567 ,\n", 823 | " -5.3171678 , -1.1046621 , -5.1544476 , -0.44752467, -7.810459 ,\n", 824 | " -4.1871576 , -4.3159685 , -6.7241583 , -6.5769987 , -5.6329894 ,\n", 825 | " -2.0669992 , -0.8374925 , -2.1317348 , -4.9491835 , -5.632787 ,\n", 826 | " -2.3372908 , -2.6463022 , -6.1446548 , -6.2574797 , -4.260248 ]],\n", 827 | " dtype=float32)" 828 | ] 829 | }, 830 | "execution_count": 27, 831 | "metadata": {}, 832 | "output_type": "execute_result" 833 | } 834 | ], 835 | "source": [ 836 | "# Test the get_logits function by passing a dummy token sequence\n", 837 | "\n", 838 | "dummy_initial_state = tf.random.normal(model.layers[1].states[0].shape)\n", 839 | "get_logits(model, [[1, 2, 3, 4]], initial_state=dummy_initial_state)" 840 | ] 841 | }, 842 | { 843 | "cell_type": "markdown", 844 | "metadata": {}, 845 | "source": [ 846 | "You should now write a function that takes a logits prediction similar to the above, uses it to create a categorical distribution, and samples a token from this distribution. The following function takes a 2D numpy array `logits` as an argument, and should return a single integer prediction that is sampled from the categorical distribution. \n", 847 | "\n", 848 | "**Hint:** you might find the `tf.random.categorical` function useful for this; see the documentation [here](https://www.tensorflow.org/api_docs/python/tf/random/categorical)." 849 | ] 850 | }, 851 | { 852 | "cell_type": "code", 853 | "execution_count": 28, 854 | "metadata": {}, 855 | "outputs": [], 856 | "source": [ 857 | "#### GRADED CELL ####\n", 858 | "\n", 859 | "# Complete the following function.\n", 860 | "# Make sure not to change the function name or arguments.\n", 861 | "\n", 862 | "def sample_token(logits):\n", 863 | " \"\"\"\n", 864 | " This function takes a 2D numpy array as an input, and constructs a \n", 865 | " categorical distribution using it. It should then sample from this\n", 866 | " distribution and return the sample as a single integer.\n", 867 | " \"\"\"\n", 868 | " return int(tf.random.categorical(logits, 1))" 869 | ] 870 | }, 871 | { 872 | "cell_type": "code", 873 | "execution_count": 29, 874 | "metadata": {}, 875 | "outputs": [ 876 | { 877 | "data": { 878 | "text/plain": [ 879 | "15" 880 | ] 881 | }, 882 | "execution_count": 29, 883 | "metadata": {}, 884 | "output_type": "execute_result" 885 | } 886 | ], 887 | "source": [ 888 | "# Test the sample_token function by passing dummy logits\n", 889 | "\n", 890 | "dummy_initial_state = tf.random.normal(model.layers[1].states[0].shape)\n", 891 | "dummy_logits = get_logits(model, [[1, 2, 3, 4]], initial_state=dummy_initial_state)\n", 892 | "sample_token(dummy_logits)" 893 | ] 894 | }, 895 | { 896 | "cell_type": "code", 897 | "execution_count": 30, 898 | "metadata": {}, 899 | "outputs": [ 900 | { 901 | "name": "stdout", 902 | "output_type": "stream", 903 | "text": [ 904 | "12 17\n" 905 | ] 906 | }, 907 | { 908 | "data": { 909 | "text/plain": [ 910 | "True" 911 | ] 912 | }, 913 | "execution_count": 30, 914 | "metadata": {}, 915 | "output_type": "execute_result" 916 | } 917 | ], 918 | "source": [ 919 | "logits_size = dummy_logits.shape[1]\n", 920 | "dummy_logits = -np.inf*np.ones((1, logits_size))\n", 921 | "dummy_logits[0, 20] = 0\n", 922 | "sample_token(dummy_logits)\n", 923 | "random_inx = np.random.choice(logits_size, 2, replace=False)\n", 924 | "random_inx1, random_inx2 = random_inx[0], random_inx[1]\n", 925 | "print(random_inx1, random_inx2)\n", 926 | "dummy_logits = -np.inf*np.ones((1, logits_size))\n", 927 | "dummy_logits[0, random_inx1] = 0\n", 928 | "dummy_logits[0, random_inx2] = 0\n", 929 | "sampled_token = []\n", 930 | "for _ in range(100):\n", 931 | " sampled_token.append(sample_token(dummy_logits))\n", 932 | " \n", 933 | "l_tokens, l_counts = np.unique(np.array(sampled_token), return_counts=True)\n", 934 | "len(l_tokens) == 2" 935 | ] 936 | }, 937 | { 938 | "cell_type": "markdown", 939 | "metadata": {}, 940 | "source": [ 941 | "#### Generate text from the model\n", 942 | "\n", 943 | "You are now ready to generate text from the model!" 944 | ] 945 | }, 946 | { 947 | "cell_type": "code", 948 | "execution_count": 31, 949 | "metadata": {}, 950 | "outputs": [], 951 | "source": [ 952 | "# Create a seed string and number of generation steps\n", 953 | "\n", 954 | "init_string = 'ROMEO:'\n", 955 | "num_generation_steps = 1000" 956 | ] 957 | }, 958 | { 959 | "cell_type": "code", 960 | "execution_count": 32, 961 | "metadata": {}, 962 | "outputs": [ 963 | { 964 | "name": "stdout", 965 | "output_type": "stream", 966 | "text": [ 967 | "ROMEO:\n", 968 | "The 'dway my extraments: and, let him well soldier,\n", 969 | "that supments your hands and the means to myself\n", 970 | "That many yielding their constable ang Bolingbroke or go;\n", 971 | "Which you shall strike from Launtaint\n", 972 | "Now, thou that's a ted, a wife, there's a little wind\n", 973 | "Out of any other's eye, at any thing still be a second,\n", 974 | "You have the mustard on the pearl'? and\n", 975 | "Our are a line, I'll deliver you all\n", 976 | "It is follow me with yourselves of; and, for my business,\n", 977 | "Proclaiming as I cauble much perforce of beef\n", 978 | "Take orbs: by this and my uncle's atcenting\n", 979 | "That in this stick in your own crowns, I'll rejouct\n", 980 | "With his beheful wonder in counsel house\n", 981 | "By Herciush and I repleing the sun\n", 982 | "that you are their dear and husbands' tale!\n", 983 | "Or as at Leckness, this page cracksing\n", 984 | "That valcaments my sons! elder part!\n", 985 | "\n", 986 | "PRINCE:\n", 987 | "No, 'tis no other, to the power than any thing\n", 988 | "Hake, ladies; when hangs and preposity\n", 989 | "With your wandrent for detisuded by man\n", 990 | "That shames Kate that proudly bow'd the apparent accused\n", 991 | "Would terror Valinanions, i\n" 992 | ] 993 | } 994 | ], 995 | "source": [ 996 | "# Use the model to generate a token sequence\n", 997 | "\n", 998 | "token_sequence = tokenizer.texts_to_sequences([init_string])\n", 999 | "initial_state = None\n", 1000 | "input_sequence = token_sequence\n", 1001 | "\n", 1002 | "for _ in range(num_generation_steps):\n", 1003 | " logits = get_logits(model, input_sequence, initial_state=initial_state)\n", 1004 | " sampled_token = sample_token(logits)\n", 1005 | " token_sequence[0].append(sampled_token)\n", 1006 | " input_sequence = [[sampled_token]]\n", 1007 | " initial_state = model.layers[1].states[0].numpy()\n", 1008 | " \n", 1009 | "print(tokenizer.sequences_to_texts(token_sequence)[0][::2])" 1010 | ] 1011 | }, 1012 | { 1013 | "cell_type": "markdown", 1014 | "metadata": {}, 1015 | "source": [ 1016 | "Congratulations for completing this programming assignment! In the next week of the course we will see how to build customised models and layers, and make custom training loops." 1017 | ] 1018 | } 1019 | ], 1020 | "metadata": { 1021 | "coursera": { 1022 | "course_slug": "tensor-flow-2-2", 1023 | "graded_item_id": "4eYSM", 1024 | "launcher_item_id": "HEV6h" 1025 | }, 1026 | "kernelspec": { 1027 | "display_name": "Python 3", 1028 | "language": "python", 1029 | "name": "python3" 1030 | }, 1031 | "language_info": { 1032 | "codemirror_mode": { 1033 | "name": "ipython", 1034 | "version": 3 1035 | }, 1036 | "file_extension": ".py", 1037 | "mimetype": "text/x-python", 1038 | "name": "python", 1039 | "nbconvert_exporter": "python", 1040 | "pygments_lexer": "ipython3", 1041 | "version": "3.7.1" 1042 | } 1043 | }, 1044 | "nbformat": 4, 1045 | "nbformat_minor": 2 1046 | } 1047 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TensorFlow 2 for Deep Learning Specialization 2 | 3 | ## About this Specialization 4 | 5 | This Specialization is intended for machine learning researchers and practitioners who are seeking to develop practical skills in the popular deep learning framework TensorFlow. 6 | 7 | The first course of this Specialization will guide you through the fundamental concepts required to successfully build, train, evaluate and make predictions from deep learning models, validating your models and including regularisation, implementing callbacks, and saving and loading models. 8 | 9 | The second course will deepen your knowledge and skills with TensorFlow, in order to develop fully customised deep learning models and workflows for any application. You will use lower level APIs in TensorFlow to develop complex model architectures, fully customised layers, and a flexible data workflow. You will also expand your knowledge of the TensorFlow APIs to include sequence models. 10 | 11 | The final course specialises in the increasingly important probabilistic approach to deep learning. You will learn how to develop probabilistic models with TensorFlow, making particular use of the TensorFlow Probability library, which is designed to make it easy to combine probabilistic models with deep learning. As such, this course can also be viewed as an introduction to the TensorFlow Probability library. 12 | 13 | Prerequisite knowledge for this Specialization is python 3, general machine learning and deep learning concepts, and a solid foundation in probability and statistics (especially for course 3). 14 | 15 | There are 3 Courses in this Specialization: 16 | 1. [Getting started with TensorFlow 2](https://www.coursera.org/learn/getting-started-with-tensor-flow2?specialization=tensorflow2-deeplearning) 17 | 2. [Customising your models with TensorFlow 2](https://www.coursera.org/learn/customising-models-tensorflow2?specialization=tensorflow2-deeplearning) 18 | 3. [Probabilistic Deep Learning with TensorFlow 2](https://www.coursera.org/learn/probabilistic-deep-learning-with-tensorflow2?specialization=tensorflow2-deeplearning) 19 | 20 | Link Specialization: https://www.coursera.org/specializations/tensorflow2-deeplearning 21 | --------------------------------------------------------------------------------