├── .gitignore ├── imgs ├── img1.png ├── img2.png └── cifar10.png ├── server.py ├── README.md ├── 3-omt ├── README.md └── pytorch-fgu-omt.ipynb ├── 2-Intermediate ├── 2.2-Pretrained-ResNet-Imagenet.ipynb ├── 2.4-Finetuning-Hymenoptera.ipynb ├── 2.3-TransferLearning-MNIST.ipynb ├── 2.5-CharRNN.ipynb └── 2.1-Convolutional-Neural-Networks.ipynb └── 1-Basics ├── 1.4-FizzBuzz.ipynb └── 1.2-Linear-Regression.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | *.ipynb_checkpoints 2 | *.html 3 | *.pdf 4 | .DS_Store 5 | *.pth 6 | data/ 7 | -------------------------------------------------------------------------------- /imgs/img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leriomaggio/PyCon9-Pytorch-from-Ground-Up/master/imgs/img1.png -------------------------------------------------------------------------------- /imgs/img2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leriomaggio/PyCon9-Pytorch-from-Ground-Up/master/imgs/img2.png -------------------------------------------------------------------------------- /imgs/cifar10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/leriomaggio/PyCon9-Pytorch-from-Ground-Up/master/imgs/cifar10.png -------------------------------------------------------------------------------- /server.py: -------------------------------------------------------------------------------- 1 | #!/bin/env python 2 | 3 | import SocketServer 4 | import BaseHTTPServer 5 | import SimpleHTTPServer 6 | 7 | class ThreadingSimpleServer(SocketServer.ThreadingMixIn, 8 | BaseHTTPServer.HTTPServer): 9 | pass 10 | 11 | import sys 12 | 13 | if sys.argv[1:]: 14 | port = int(sys.argv[1]) 15 | else: 16 | port = 8000 17 | 18 | server = ThreadingSimpleServer(('', port), SimpleHTTPServer.SimpleHTTPRequestHandler) 19 | try: 20 | while 1: 21 | sys.stdout.flush() 22 | server.handle_request() 23 | except KeyboardInterrupt: 24 | print "Finished" -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PyCon9: PyTorch from Ground Up 2 | 3 | ## Instructions: 4 | 5 | ### Part 0 - Introduction Presentation 6 | 7 | #### How to launch 8 | 9 | $ cd 0-pytorch-fgu 10 | $ source launch_me.sh 11 | 12 | ### Part 1-2 - Hands On 13 | 14 | #### How to launch 15 | 16 | - move to the root folder 17 | - run: 18 | $ jupyter notebook 19 | 20 | 21 | ### Notes 22 | 23 | we make use of [rise](https://github.com/damianavila/RISE) to make a presentation out of a notebook, so you should install it 24 | 25 | $ conda install -c damianavila82 rise 26 | 27 | ### TODO 28 | 29 | - @lantiga: 30 | - check and fix part-0 presentation 31 | - add notes on the notebooks for white board interventions 32 | - eventually add notes for other fixes 33 | - check 3-omt part (pytorch future roadmap) and remove it 34 | - @dnlcrl: 35 | - write 1.5-MNIST-intro.ipynb: 36 | - introduction of mnist dataset 37 | - approaching mnist with linear and/or logistic regression 38 | - write 2.1-Convolutional-Neural-Networks.ipynb: 39 | - introducing convolutional layers 40 | - approaching mnist with a simple convnet 41 | - fix remaining notebooks -------------------------------------------------------------------------------- /3-omt/README.md: -------------------------------------------------------------------------------- 1 | # pytorch from ground up 2 | This is the repository for **PyTorch From Ground Up** tutorial and [**Pycon 9**](https://pycon.it) talk slides. It features Jupyter Notebook Slides html file, and training notebooks 3 | 4 | ## Knowledge Prerequisites 5 | This tutorial assumes familiarity with Python and Numpy. 6 | 7 | ## Tutorial Prerequisites 8 | Python3 is required to run this tutorial. You also will need some libraries from SciPy package (NumPy, Matplotlib, Pandas), Jupyter Notebook support, and Pytorch 0.3.0 or newer. 9 | 10 | The simpliest way to maintain Python with all these libraries as well as many others is to install [Anaconda](https://www.anaconda.com/download). You can Find Pytorch installation instructions on the [Pytorch page](http://pytorch.org). 11 | 12 | CUDA availability is NOT required, but still, Life is short -- use a GPU! 13 | 14 | ## How to Use 15 | Our tutorial git has no submodules. To download the tutorial, use 16 | 17 | ``` 18 | git clone TODO 19 | ``` 20 | 21 | To run the Jupyter Notebook Slides as at Jupyter Day Atlanta 2018 talk, you can use following command: 22 | 23 | ``` 24 | jupyter nbconvert pytorch-fgu.ipynb --to slides --post serve 25 | ``` 26 | 27 | ## Table of Contents 28 | TODO -------------------------------------------------------------------------------- /3-omt/pytorch-fgu-omt.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "#
PyTorch From the Ground Up - One More Thing\n", 12 | "\n", 13 | "\n", 14 | "\n", 15 | "\n", 16 | "\n", 17 | "
[L. Antiga](http://twitter.com/lantiga), [D. Ciriello](http://twitter.com/dnlcrl) and [A. Paszke](http://twitter.com/apaszke)\n", 18 | "\n", 19 | "
[PyCon Nove (2018)](https://pycon.it/)." 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "slideshow": { 26 | "slide_type": "slide" 27 | } 28 | }, 29 | "source": [ 30 | "# Forward and Backward Function Hooks\n", 31 | "\n", 32 | "- how about inspecting / modifying the output and grad_output of a layer?\n", 33 | "- \"We introduce hooks for this purpose.\" \n", 34 | "- You can register a function on a Module or a Variable\n" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": { 41 | "collapsed": true, 42 | "slideshow": { 43 | "slide_type": "slide" 44 | } 45 | }, 46 | "outputs": [], 47 | "source": [ 48 | "def printnorm(self, input, output):\n", 49 | " # input is a tuple of packed inputs\n", 50 | " # output is a Variable. output.data is the Tensor we are interested\n", 51 | " print('Inside ' + self.__class__.__name__ + ' forward')\n", 52 | " print('')\n", 53 | " print('input: ', type(input))\n", 54 | " print('input[0]: ', type(input[0]))\n", 55 | " print('output: ', type(output))\n", 56 | " print('')\n", 57 | " print('input size:', input[0].size())\n", 58 | " print('output size:', output.data.size())\n", 59 | " print('output norm:', output.data.norm())\n", 60 | "\n", 61 | "\n", 62 | "net.conv2.register_forward_hook(printnorm)\n", 63 | "\n", 64 | "out = net(input)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "collapsed": true, 72 | "slideshow": { 73 | "slide_type": "slide" 74 | } 75 | }, 76 | "outputs": [], 77 | "source": [ 78 | "def printgradnorm(self, grad_input, grad_output):\n", 79 | " print('Inside ' + self.__class__.__name__ + ' backward')\n", 80 | " print('Inside class:' + self.__class__.__name__)\n", 81 | " print('')\n", 82 | " print('grad_input: ', type(grad_input))\n", 83 | " print('grad_input[0]: ', type(grad_input[0]))\n", 84 | " print('grad_output: ', type(grad_output))\n", 85 | " print('grad_output[0]: ', type(grad_output[0]))\n", 86 | " print('')\n", 87 | " print('grad_input size:', grad_input[0].size())\n", 88 | " print('grad_output size:', grad_output[0].size())\n", 89 | " print('grad_input norm:', grad_input[0].data.norm())\n", 90 | "\n", 91 | "\n", 92 | "net.conv2.register_backward_hook(printgradnorm)\n", 93 | "\n", 94 | "out = net(input)\n", 95 | "err = loss_fn(out, target)\n", 96 | "err.backward()" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": { 102 | "slideshow": { 103 | "slide_type": "slide" 104 | } 105 | }, 106 | "source": [ 107 | "# Nightly Builds!\n", 108 | "\n", 109 | "- recently added\n", 110 | "- install the last **unstable** master version of PyTorch\n", 111 | "- use all just implemented layers" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": { 117 | "slideshow": { 118 | "slide_type": "slide" 119 | } 120 | }, 121 | "source": [ 122 | "# Thank You!" 123 | ] 124 | } 125 | ], 126 | "metadata": { 127 | "celltoolbar": "Slideshow", 128 | "kernelspec": { 129 | "display_name": "Python 3", 130 | "language": "python", 131 | "name": "python3" 132 | }, 133 | "language_info": { 134 | "codemirror_mode": { 135 | "name": "ipython", 136 | "version": 3 137 | }, 138 | "file_extension": ".py", 139 | "mimetype": "text/x-python", 140 | "name": "python", 141 | "nbconvert_exporter": "python", 142 | "pygments_lexer": "ipython3", 143 | "version": "3.6.3" 144 | } 145 | }, 146 | "nbformat": 4, 147 | "nbformat_minor": 2 148 | } 149 | -------------------------------------------------------------------------------- /2-Intermediate/2.2-Pretrained-ResNet-Imagenet.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# Classifying images with a pretrained model" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "In this section we will learn how to use a pretrained model, to perform predictions without changing its parameters, specifically we will use a pretrained residual network model having 152 residual blocks.\n", 23 | "\n", 24 | "Residual models are a generation of convnets proposed in 2016 which obtained the best results for the ILSRVC (originally regarding classifcation and localization on ImageNet dataset) competition for that year, since then a lot of its variations are getting proposed, you can see a full list of available pretrained models on the pytorch documentation website, along with the performace obtained by the model on the ImageNet test set.\n", 25 | "\n", 26 | "The power of those residual model is given by the residual paths, which consists of a residual block's input value replicated and concatenated to its output, in a way that the model keeps a sort of track of what the orinal data contains. This fact permits the model to leaern better using the same number of parameters. More informations about the residual models can be found here:\n", 27 | "\n", 28 | "https://arxiv.org/abs/1512.03385" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": { 34 | "slideshow": { 35 | "slide_type": "notes" 36 | } 37 | }, 38 | "source": [ 39 | "Let's import our needed packages" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": { 46 | "slideshow": { 47 | "slide_type": "slide" 48 | } 49 | }, 50 | "outputs": [], 51 | "source": [ 52 | "import torch\n", 53 | "from torch.utils import data\n", 54 | "\n", 55 | "import numpy as np\n", 56 | "\n", 57 | "from torchvision.datasets.mnist import FashionMNIST\n", 58 | "from torchvision.models.resnet import resnet152\n", 59 | "from torchvision import transforms, utils\n", 60 | "\n", 61 | "from torch import nn\n", 62 | "from torch import optim\n", 63 | "from torch.autograd import Variable\n", 64 | "from torch.nn import functional as F\n", 65 | "\n", 66 | "from PIL import Image\n", 67 | "\n", 68 | "import matplotlib.pyplot as plt\n", 69 | "%matplotlib inline" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": { 75 | "slideshow": { 76 | "slide_type": "notes" 77 | } 78 | }, 79 | "source": [ 80 | "to get pretrained resnet, it is sufficient to pass **True** to the **pretrained** method parameter. It's VERY important to call **.eval()** on our method before using it for prediction, because otherwise we will obtain strange results. That's because inside our model, there are some layers which behaves differently during training and valuidation mode, and so by calling the **.eval()** method, thanks to how the nn.Module class, the call gets propagated to all the model'd submodules." 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "slideshow": { 88 | "slide_type": "slide" 89 | } 90 | }, 91 | "outputs": [], 92 | "source": [ 93 | "model = resnet152(pretrained=True).eval()" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": { 99 | "slideshow": { 100 | "slide_type": "notes" 101 | } 102 | }, 103 | "source": [ 104 | "This model has been trained on ImageNet dataset, which consists of high-res images for 1000 classes, so if we want to predict an image with the resnet model we need to map the class having the highest log probability, to the label name ('cat', 'dog', etc), so we just have to download the following json file and load it.\n", 105 | "\n", 106 | "json imagenet classes, https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": { 113 | "slideshow": { 114 | "slide_type": "slide" 115 | } 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "import json\n", 120 | "class_idx = json.load(open(\"../data/imagenet_class_index.json\"))" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": { 126 | "slideshow": { 127 | "slide_type": "notes" 128 | } 129 | }, 130 | "source": [ 131 | "Let's take an example image" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": null, 137 | "metadata": { 138 | "slideshow": { 139 | "slide_type": "slide" 140 | } 141 | }, 142 | "outputs": [], 143 | "source": [ 144 | "img = Image.open('../data/imgs/img.jpg')\n", 145 | "img = img.convert('RGB')" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": { 152 | "slideshow": { 153 | "slide_type": "slide" 154 | } 155 | }, 156 | "outputs": [], 157 | "source": [ 158 | "plt.imshow(img)" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": { 164 | "slideshow": { 165 | "slide_type": "notes" 166 | } 167 | }, 168 | "source": [ 169 | "As you can see this is a picture of a dog of labrador breed, in order to pass the image to the model and get its prediction, we should apply the same preprocessing applied to the training image used for training the model with ImageNet images. Firstly, we need to apply the same normalization process applied to images form the training set, which means we need to subtract the pixels mean and divide by the pixels std, as eplained in the PyTorch documentaiton\n", 170 | "\n", 171 | "http://pytorch.org/docs/master/torchvision/models.html\n", 172 | "\n", 173 | "\n", 174 | "All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using **mean = [0.485, 0.456, 0.406]** and **std = [0.229, 0.224, 0.225]**. \n", 175 | "\n", 176 | "On the same page you can also see other model implementation and their accuracies on ImageNet testset.\n", 177 | "\n", 178 | "Given that we should pass 224 images to the model we add a resize + crop transofrmation before creating the tensors and normalizing the image.\n" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": { 185 | "slideshow": { 186 | "slide_type": "slide" 187 | } 188 | }, 189 | "outputs": [], 190 | "source": [ 191 | "def preproc(x):\n", 192 | " x = img\n", 193 | " t = transforms.Compose([\n", 194 | " transforms.Resize(256),\n", 195 | " transforms.CenterCrop(224),\n", 196 | " transforms.ToTensor(),\n", 197 | " transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n", 198 | " ])\n", 199 | " x = t(x)\n", 200 | " x = torch.Tensor(x).unsqueeze(0)\n", 201 | " x = Variable(x)\n", 202 | " return x\n" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": { 208 | "slideshow": { 209 | "slide_type": "notes" 210 | } 211 | }, 212 | "source": [ 213 | "ok, now we have our preprocessing funciton ready and we can pass the image to the model in order to get its preditction for this image, so we pass the preprocessed image to the model, getting the model's log probabilities for the image belonging to each class (1000 classes), than we find the label having the highest probability, map its value to the class name and print it. We could actually have a master class too (dog, labrador) but this is a more simple map which just returns a sigle value." 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "metadata": { 220 | "slideshow": { 221 | "slide_type": "slide" 222 | } 223 | }, 224 | "outputs": [], 225 | "source": [ 226 | "output = model(preproc(img))\n", 227 | "m, argm = output.data.squeeze().max(0)\n", 228 | "class_id = argm[0]\n", 229 | "print(output)\n", 230 | "class_idx[str(class_id)]" 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "metadata": { 236 | "slideshow": { 237 | "slide_type": "notes" 238 | } 239 | }, 240 | "source": [ 241 | "Let's try with another image:" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": null, 247 | "metadata": { 248 | "slideshow": { 249 | "slide_type": "slide" 250 | } 251 | }, 252 | "outputs": [], 253 | "source": [ 254 | "img = Image.open('../data/imgs/cat.jpg')\n", 255 | "img = img.convert('RGB')" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "slideshow": { 263 | "slide_type": "slide" 264 | } 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "plt.imshow(img)" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "slideshow": { 276 | "slide_type": "slide" 277 | } 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "output = model(preproc(img))\n", 282 | "m, argm = output.data.squeeze().max(0)\n", 283 | "class_id = argm[0]\n", 284 | "print(output)\n", 285 | "class_idx[str(class_id)]" 286 | ] 287 | }, 288 | { 289 | "cell_type": "markdown", 290 | "metadata": { 291 | "slideshow": { 292 | "slide_type": "notes" 293 | } 294 | }, 295 | "source": [ 296 | "\"A tabby is any domestic cat\" [wikipedia]\n", 297 | "If we had the master class it would also gave us the 'cat' label" 298 | ] 299 | } 300 | ], 301 | "metadata": { 302 | "celltoolbar": "Slideshow", 303 | "kernelspec": { 304 | "display_name": "Python 3", 305 | "language": "python", 306 | "name": "python3" 307 | }, 308 | "language_info": { 309 | "codemirror_mode": { 310 | "name": "ipython", 311 | "version": 3 312 | }, 313 | "file_extension": ".py", 314 | "mimetype": "text/x-python", 315 | "name": "python", 316 | "nbconvert_exporter": "python", 317 | "pygments_lexer": "ipython3", 318 | "version": "3.6.3" 319 | } 320 | }, 321 | "nbformat": 4, 322 | "nbformat_minor": 2 323 | } 324 | -------------------------------------------------------------------------------- /2-Intermediate/2.4-Finetuning-Hymenoptera.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# FineTuning a Pretrained Model to Distinguish Between Bees and Ants" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "In this last but one section we will see how to perform the fine tuning of a pretrained model (for real this time).\n", 23 | "To do this we will use a simpler dataset, composed by images belonging to just 2 classes, bees and ants.\n", 24 | "\n", 25 | "Q: what does FineTuning actually consists of?\n", 26 | "\n", 27 | "Finetuning means that we downlaod and use a pretrained model as in the previous seciton, but instead of just training the newly added last layer, we should re-train *all* the model's parameters, but using a very low learning rate, in this way we won't change the parameters as much, but we'll permit our model to slowly adapt the parameters to the new task, thus the term FineTuning.\n", 28 | "\n", 29 | "Let's add our usual imports" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "slideshow": { 37 | "slide_type": "slide" 38 | } 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "import torch\n", 43 | "from torch.utils import data\n", 44 | "\n", 45 | "import numpy as np\n", 46 | "from tqdm import tqdm\n", 47 | "\n", 48 | "\n", 49 | "from torchvision.datasets.mnist import FashionMNIST\n", 50 | "from torchvision.models.resnet import resnet18\n", 51 | "from torchvision import transforms, utils, datasets\n", 52 | "\n", 53 | "from torch import nn\n", 54 | "from torch import optim\n", 55 | "from torch.autograd import Variable\n", 56 | "from torch.nn import functional as F\n", 57 | "\n", 58 | "import os\n", 59 | "from PIL import Image\n", 60 | "\n", 61 | "import matplotlib.pyplot as plt\n", 62 | "%matplotlib inline" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": { 68 | "slideshow": { 69 | "slide_type": "notes" 70 | } 71 | }, 72 | "source": [ 73 | "lets create our dataset objects" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": { 80 | "slideshow": { 81 | "slide_type": "slide" 82 | } 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "mean = [0.485, 0.456, 0.406]\n", 87 | "std = [0.229, 0.224, 0.225]\n", 88 | "\n", 89 | "transform_tr = transforms.Compose([\n", 90 | " transforms.RandomResizedCrop(224),\n", 91 | " transforms.RandomHorizontalFlip(),\n", 92 | " transforms.ToTensor(),\n", 93 | " transforms.Normalize(mean, std)\n", 94 | "])\n", 95 | "\n", 96 | "transform_val = transforms.Compose([\n", 97 | " transforms.Resize(256),\n", 98 | " transforms.CenterCrop(256),\n", 99 | " transforms.ToTensor(),\n", 100 | " transforms.Normalize(mean, std)\n", 101 | "])\n", 102 | "\n", 103 | "data_dir = '../data/hymenoptera_data'\n" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "slideshow": { 111 | "slide_type": "slide" 112 | } 113 | }, 114 | "outputs": [], 115 | "source": [ 116 | "image_datasets_tr = datasets.ImageFolder(\n", 117 | " os.path.join(data_dir, 'train'), \n", 118 | " transform_tr)\n", 119 | "image_datasets_val = datasets.ImageFolder(\n", 120 | " os.path.join(data_dir, 'val'), \n", 121 | " transform_val)\n", 122 | "\n", 123 | "dataloader_tr = data.DataLoader(image_datasets_tr, shuffle=True, batch_size=4, num_workers=4)\n", 124 | "dataloader_val = data.DataLoader(image_datasets_val, shuffle=False, batch_size=4, num_workers=4)\n", 125 | "\n", 126 | "dataset_sizes = {'train': len(image_datasets_tr), 'val': len(image_datasets_val)}\n", 127 | "\n", 128 | "class_names = image_datasets_tr.classes" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": { 134 | "slideshow": { 135 | "slide_type": "notes" 136 | } 137 | }, 138 | "source": [ 139 | "let's show some sample from the dataset and its label, for this purpose we will create a simple funciton" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": { 146 | "slideshow": { 147 | "slide_type": "slide" 148 | } 149 | }, 150 | "outputs": [], 151 | "source": [ 152 | "def imshow(inp, title=None):\n", 153 | " '''imshow for Tensor'''\n", 154 | " inp = inp.numpy().transpose((1, 2, 0))\n", 155 | " inp = np.array(std) * inp + np.array(mean)\n", 156 | " inp = np.clip(inp, 0, 1)\n", 157 | " plt.figure(figsize=[10,10])\n", 158 | " plt.imshow(inp)\n", 159 | " if title:\n", 160 | " plt.title(title)\n", 161 | " plt.pause(0.001)" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "metadata": { 168 | "slideshow": { 169 | "slide_type": "slide" 170 | } 171 | }, 172 | "outputs": [], 173 | "source": [ 174 | "inputs, classes = next(iter(dataloader_tr))\n", 175 | "out = utils.make_grid(inputs)\n", 176 | "imshow(out, title=[class_names[x] for x in classes])" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": { 182 | "slideshow": { 183 | "slide_type": "notes" 184 | } 185 | }, 186 | "source": [ 187 | "Bees, and also Ants.\n", 188 | "\n", 189 | "Let's get our pretrained model, replace the pooling layer and the last fc layer, we will also initialize the newly created fc layer weight data using a normal function having mu=0 and var=0.001" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "metadata": { 196 | "slideshow": { 197 | "slide_type": "slide" 198 | } 199 | }, 200 | "outputs": [], 201 | "source": [ 202 | "model = resnet18(pretrained=True)\n", 203 | "# replace the avgpool (it's a 7x7 pooling expecting Cx7x7 tensors)\n", 204 | "model.avgpool = nn.AdaptiveAvgPool2d((1,1))\n", 205 | "# replace the last layer (it has 1000 out features and we need new weights)\n", 206 | "model.fc = nn.Linear(model.fc.in_features, 2)\n", 207 | "model.fc.weight.data.normal_(0.0, 0.001)" 208 | ] 209 | }, 210 | { 211 | "cell_type": "markdown", 212 | "metadata": { 213 | "slideshow": { 214 | "slide_type": "notes" 215 | } 216 | }, 217 | "source": [ 218 | "we use a CrossEntropyLoss and SGD algorithm for classification as usual for classification problems. This time we will also make use of a *Learning Rate Scheduler*, which permits us to update the learing rate following a certain rule, in this case we will use the simple StepLR scheduler which accept a **step_size** parameter which corresponds to the amount of steps between each lr update, and a gamma parameter, which will get multiplied with our LR and the results will be our new LR.\n", 219 | "\n", 220 | "in brief the StepLR does:\n", 221 | "\n", 222 | " every step_size steps:\n", 223 | " LR *= gamma" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": { 230 | "slideshow": { 231 | "slide_type": "slide" 232 | } 233 | }, 234 | "outputs": [], 235 | "source": [ 236 | "loss = nn.CrossEntropyLoss()\n", 237 | "optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)\n", 238 | "scheduler = optim.lr_scheduler.StepLR(\n", 239 | " optimizer, \n", 240 | " step_size=7, \n", 241 | " gamma=0.1)" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": { 247 | "slideshow": { 248 | "slide_type": "notes" 249 | } 250 | }, 251 | "source": [ 252 | "Let's get our training started, as we don't want to wait and want to go drink some beer, we will train the model for just 1 epoch and evaluate the model on the validation dtaset" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "metadata": { 259 | "slideshow": { 260 | "slide_type": "slide" 261 | } 262 | }, 263 | "outputs": [], 264 | "source": [ 265 | "for epoch in range(1):\n", 266 | "\n", 267 | " for i, (x, y) in enumerate(dataloader_tr):\n", 268 | " x, y = Variable(x), Variable(y)\n", 269 | " l = loss(model(x), y)\n", 270 | "\n", 271 | " optimizer.zero_grad()\n", 272 | " l.backward()\n", 273 | " optimizer.step()\n", 274 | " \n", 275 | " print('Epoch: {}, batch_idx: {}/{}, loss: {}'.format(\n", 276 | " epoch, i, len(dataloader_tr)-1, l.data.numpy()[0]))" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "metadata": { 282 | "slideshow": { 283 | "slide_type": "notes" 284 | } 285 | }, 286 | "source": [ 287 | "Q: Why this time the pretrained model is training faster??\n", 288 | "\n", 289 | "because the dataset is much more small, in terms of number of samples, even if the images are much larger, we roughly have almost a 10% of its number of samples, thus the faster epochs" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": { 295 | "slideshow": { 296 | "slide_type": "notes" 297 | } 298 | }, 299 | "source": [ 300 | "Let's evaluate the model on the validation dataset as usual" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": { 307 | "slideshow": { 308 | "slide_type": "slide" 309 | } 310 | }, 311 | "outputs": [], 312 | "source": [ 313 | "model.eval()\n", 314 | "preds = []\n", 315 | "ys = []\n", 316 | "for x, y in dataloader_val:\n", 317 | " x, y = Variable(x), Variable(y)\n", 318 | " preds.extend(model(x).max(1)[1].data.tolist())\n", 319 | " ys.extend(y.data)\n", 320 | "\n", 321 | "corrects = (np.array(preds) == np.array(ys))\n", 322 | "print('Accuracy: {}'.format(corrects.mean()))" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": { 328 | "slideshow": { 329 | "slide_type": "notes" 330 | } 331 | }, 332 | "source": [ 333 | "that's actually an awesome result with just one epoch of training, IMHO. But let's count how many images the model classifies wrongly" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "metadata": { 340 | "slideshow": { 341 | "slide_type": "slide" 342 | } 343 | }, 344 | "outputs": [], 345 | "source": [ 346 | "print('correct predicitons: {}'.format(\n", 347 | " np.sum(np.array(preds) == np.array(ys))))\n", 348 | "\n", 349 | "print('wrong predicitons: {}'.format(\n", 350 | " np.sum(np.array(preds) != np.array(ys))))" 351 | ] 352 | } 353 | ], 354 | "metadata": { 355 | "celltoolbar": "Slideshow", 356 | "kernelspec": { 357 | "display_name": "Python 3", 358 | "language": "python", 359 | "name": "python3" 360 | }, 361 | "language_info": { 362 | "codemirror_mode": { 363 | "name": "ipython", 364 | "version": 3 365 | }, 366 | "file_extension": ".py", 367 | "mimetype": "text/x-python", 368 | "name": "python", 369 | "nbconvert_exporter": "python", 370 | "pygments_lexer": "ipython3", 371 | "version": "3.6.3" 372 | } 373 | }, 374 | "nbformat": 4, 375 | "nbformat_minor": 2 376 | } 377 | -------------------------------------------------------------------------------- /1-Basics/1.4-FizzBuzz.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# Fizz Buzz with Pytorch" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "this is a very simple task often chosen by an interviewer to get an idea of a candidate's ability to write simple functions. We will break it with a very simple feed forward neural network composed by a total of 3 weighted layers (2 hidden and 1 output layer).\n", 23 | "\n", 24 | "This task usually consists of writing a function that takes an integer and returns the string 'fizz' if the number is divisible by (is a multiple of) 3, 'buzz' if the number is divisible by 5, 'fizzbuzz' if the number is divisible by 3*5=15 and returning the number itself otherwise.\n", 25 | "\n", 26 | "We'll approach this task by first converting the decimal integer numbers to binary inputs, so our model will have `num_bits` values per sample and will output 4 values, corresponding to the possible classes for each sample (fizz, buzz, fizzbuzz, x).\n", 27 | "\n", 28 | "So we will start by writing the **fizz_buzz_encode** method, and other two convenience methods for encoding/decoding binary and fizz buzz, obviously after importing the usual modules, and defining the number of possible digits for representing the numbers (bits). We will set it to 12.\n", 29 | "\n", 30 | "source:\n", 31 | "http://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": { 38 | "slideshow": { 39 | "slide_type": "slide" 40 | } 41 | }, 42 | "outputs": [], 43 | "source": [ 44 | "import torch\n", 45 | "from torch.utils import data\n", 46 | "\n", 47 | "import torch.nn as nn\n", 48 | "\n", 49 | "# functional module, fuctional implementations for \n", 50 | "# unparameterized neural network modules\n", 51 | "import torch.nn.functional as F\n", 52 | "\n", 53 | "from torch.autograd import Variable\n", 54 | "import torch.optim as optim\n", 55 | "\n", 56 | "import numpy as np\n", 57 | "\n", 58 | "NUM_DIGITS = 12" 59 | ] 60 | }, 61 | { 62 | "cell_type": "markdown", 63 | "metadata": { 64 | "slideshow": { 65 | "slide_type": "notes" 66 | } 67 | }, 68 | "source": [ 69 | "write our solution and convenience methods" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": null, 75 | "metadata": { 76 | "slideshow": { 77 | "slide_type": "slide" 78 | } 79 | }, 80 | "outputs": [], 81 | "source": [ 82 | "# Represent each input by an array of its binary digits.\n", 83 | "def binary_encode(i, num_digits):\n", 84 | " return np.array([i >> d & 1 for d in range(num_digits)])\n", 85 | "\n", 86 | "# One-hot encode the desired outputs: [number, \"fizz\", \"buzz\", \"fizzbuzz\"]\n", 87 | "def fizz_buzz_encode(i):\n", 88 | " if i % 15 == 0: return 3\n", 89 | " elif i % 5 == 0: return 2\n", 90 | " elif i % 3 == 0: return 1\n", 91 | " else: return 0\n", 92 | "\n", 93 | "#printable and coherent labels\n", 94 | "def fizz_buzz_decode(i, prediction):\n", 95 | " return [str(i), \"fizz\", \"buzz\", \"fizzbuzz\"][prediction]" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": { 101 | "slideshow": { 102 | "slide_type": "notes" 103 | } 104 | }, 105 | "source": [ 106 | "Now I will show you a way to build a custom dataset class, that subclasses data.Dataset. \n", 107 | "\n", 108 | "That's a simple way to create a sort of singleton object for our data, so that if and when we instantiate the dataset object multiple times, the data doesn't get created/loaded multiple times. To do this, we create an empty dictionary **DATA_CACHE** at global scope (it will be created when we import the module). Then when we instantiate the dataset object and its init method gets called, we first check if our **DATA_CACHE** actually contains the data, if not we fill the cache with our data, otherwise we simply compute the data at each index (we need to split in train/val/test). This way we could load even a gazillion samples without actually copying data and thus using more memory." 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": { 115 | "slideshow": { 116 | "slide_type": "slide" 117 | } 118 | }, 119 | "outputs": [], 120 | "source": [ 121 | "DATA_CACHE = {} \n", 122 | "\n", 123 | "def fill_cache(num_bits):\n", 124 | " DATA_CACHE.update({\n", 125 | " 'X': [binary_encode(i, NUM_DIGITS) for i in range(2 ** NUM_DIGITS)],\n", 126 | " 'y': [fizz_buzz_encode(i) for i in range(2 ** NUM_DIGITS)]\n", 127 | " })\n", 128 | "\n", 129 | "class FizzbuzzDataset(data.Dataset):\n", 130 | " def __init__(self, num_bits=NUM_DIGITS, mode='train'):\n", 131 | " super(FizzbuzzDataset, self).__init__()\n", 132 | " \n", 133 | " if not DATA_CACHE:\n", 134 | " fill_cache(num_bits)\n", 135 | " \n", 136 | " start, end = (0, 100) if mode == 'val' else (100, len(DATA_CACHE['y']))\n", 137 | " self.idxs = list(range(start, end))\n", 138 | " \n", 139 | " def __len__(self):\n", 140 | " return len(self.idxs)\n", 141 | " \n", 142 | " def __getitem__(self, idx):\n", 143 | " x = DATA_CACHE['X'][self.idxs[idx]]\n", 144 | " x = x.astype(np.float32)\n", 145 | " y = DATA_CACHE['y'][self.idxs[idx]]\n", 146 | " return x, y" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": { 152 | "slideshow": { 153 | "slide_type": "notes" 154 | } 155 | }, 156 | "source": [ 157 | "As said above, we create a simple model composed by a total of 3 feed forward layers, where the first one has `num_digits` inputs for each sample, followed by an activation fuction. We say our model must have 50 \"neural units\" for each one of the hidden layers and, given that we have 4 classes, an output dimension of 4." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": { 164 | "slideshow": { 165 | "slide_type": "slide" 166 | } 167 | }, 168 | "outputs": [], 169 | "source": [ 170 | "# http://pytorch.org/docs/master/nn.html#torch.nn.LeakyReLU\n", 171 | "class FizzbuzzModel(nn.Module):\n", 172 | " def __init__(self, h_dim=50, input_dim=NUM_DIGITS, num_classes=4):\n", 173 | " super(FizzbuzzModel, self).__init__()\n", 174 | " self.linear1 = nn.Sequential(\n", 175 | " nn.Linear(input_dim, h_dim),\n", 176 | " nn.LeakyReLU()\n", 177 | " )\n", 178 | " self.linear2 = nn.Sequential(\n", 179 | " nn.Linear(h_dim, h_dim),\n", 180 | " nn.LeakyReLU()\n", 181 | " )\n", 182 | " self.classifier = nn.Linear(h_dim, num_classes)\n", 183 | " \n", 184 | " def forward(self, x):\n", 185 | " x = self.linear1(x)\n", 186 | " x = self.linear2(x)\n", 187 | " x = self.classifier(x)\n", 188 | " return x " 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": { 195 | "slideshow": { 196 | "slide_type": "slide" 197 | } 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "# SAME AS PREVIOUS BUT USING F (nn.functional)\n", 202 | "class FizzbuzzModel(nn.Module):\n", 203 | " def __init__(self, h_dim=50, input_dim=NUM_DIGITS, num_classes=4):\n", 204 | " super(FizzbuzzModel, self).__init__()\n", 205 | " self.linear1 = nn.Linear(input_dim, h_dim)\n", 206 | " self.linear2 = nn.Linear(h_dim, h_dim)\n", 207 | " self.classifier = nn.Linear(h_dim, num_classes)\n", 208 | " \n", 209 | " def forward(self, x):\n", 210 | " x = F.leaky_relu(self.linear1(x))\n", 211 | " x = F.leaky_relu(self.linear2(x))\n", 212 | " x = self.classifier(x)\n", 213 | " return x " 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": { 219 | "slideshow": { 220 | "slide_type": "notes" 221 | } 222 | }, 223 | "source": [ 224 | "let's instantiate our dataset objects, their corresponding dataloaders, our FizzbuzzModel, the usual SGD optimizing algorithm and a cross entropy loss" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": { 231 | "slideshow": { 232 | "slide_type": "slide" 233 | } 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "dataset_tr = FizzbuzzDataset(mode='train')\n", 238 | "dataset_val = FizzbuzzDataset(mode='val')\n", 239 | "dataloader_tr = data.DataLoader(dataset_tr, batch_size=128, shuffle=True)\n", 240 | "dataloader_val = data.DataLoader(dataset_val, batch_size=128, shuffle=False)\n", 241 | "\n", 242 | "model = FizzbuzzModel()\n", 243 | "optimizer = optim.SGD(model.parameters(), lr=.05, momentum=0.9)\n", 244 | "loss = nn.CrossEntropyLoss()\n" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": { 250 | "slideshow": { 251 | "slide_type": "notes" 252 | } 253 | }, 254 | "source": [ 255 | "we thus train our model for 500 epochs" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "slideshow": { 263 | "slide_type": "slide" 264 | } 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "for epoch in range(500):\n", 269 | " # train loop\n", 270 | " for x, y in dataloader_tr:\n", 271 | " x, y = Variable(x), Variable(y)\n", 272 | " l = loss(model(x), y)\n", 273 | " \n", 274 | " optimizer.zero_grad()\n", 275 | " l.backward()\n", 276 | " optimizer.step()\n", 277 | " if not epoch % 100:\n", 278 | " print('Epoch: {}, loss: {}'.format(epoch, l.data.numpy()[0]))\n", 279 | " " 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": { 285 | "slideshow": { 286 | "slide_type": "notes" 287 | } 288 | }, 289 | "source": [ 290 | "Finally we run the prediction on our evaluation set, which contains numbers from 0 to 99, we then fizz buzz encode the model's predictions and print the fizz buzz encoded values." 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": null, 296 | "metadata": { 297 | "slideshow": { 298 | "slide_type": "slide" 299 | } 300 | }, 301 | "outputs": [], 302 | "source": [ 303 | "preds = []\n", 304 | "ys = []\n", 305 | "model.eval()\n", 306 | "for x, y in dataloader_val:\n", 307 | " x = Variable(x)\n", 308 | " preds.extend(model(x).max(1)[1].data.tolist())\n", 309 | " ys.extend(y)\n", 310 | " \n", 311 | "correct = np.array(preds) == np.array(ys)\n", 312 | "predictions = zip(range(0, 100), preds)\n", 313 | "\n", 314 | "print('Accuracy: ', correct.mean(), ', Errors: ', np.logical_not(correct).sum())\n", 315 | "print ([fizz_buzz_decode(i, x) for (i, x) in predictions])" 316 | ] 317 | } 318 | ], 319 | "metadata": { 320 | "celltoolbar": "Slideshow", 321 | "kernelspec": { 322 | "display_name": "Python 3", 323 | "language": "python", 324 | "name": "python3" 325 | }, 326 | "language_info": { 327 | "codemirror_mode": { 328 | "name": "ipython", 329 | "version": 3 330 | }, 331 | "file_extension": ".py", 332 | "mimetype": "text/x-python", 333 | "name": "python", 334 | "nbconvert_exporter": "python", 335 | "pygments_lexer": "ipython3", 336 | "version": "3.6.3" 337 | } 338 | }, 339 | "nbformat": 4, 340 | "nbformat_minor": 2 341 | } 342 | -------------------------------------------------------------------------------- /2-Intermediate/2.3-TransferLearning-MNIST.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# Transfer Learning and Training on (FASHION) MNIST" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "Now that we saw how to load and use a pretrained model for our aims, lets look at how to apply transfer learning in order to use a pretrained model as a feature extractor. Since the pretrained models available in torchvision are all trained on ImageNet, our model's last layer will provides 1000 output values, which are meant to be interpreted as log probabilities for the input to belong to every class, so we have to get rid of that layer and use a new layer with the needed number of parameters. The MNIST dataset contains digits from 0 to 9, so we need 10 output values, than we will replace the pretrained model's last layer to a new layer having 10 output features." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "slideshow": { 29 | "slide_type": "notes" 30 | } 31 | }, 32 | "source": [ 33 | "As usual, we will import all required packages, torch, numpy, etc" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": { 40 | "slideshow": { 41 | "slide_type": "slide" 42 | } 43 | }, 44 | "outputs": [], 45 | "source": [ 46 | "import torch\n", 47 | "import torchvision\n", 48 | "from torch.utils import data\n", 49 | "\n", 50 | "import numpy as np\n", 51 | "\n", 52 | "from torch import nn\n", 53 | "from torch import optim\n", 54 | "from torch.autograd import Variable\n", 55 | "from torch.nn import functional as F\n", 56 | "\n" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": { 62 | "slideshow": { 63 | "slide_type": "notes" 64 | } 65 | }, 66 | "source": [ 67 | "We should obviously also import the dataset and model classes from the torchvision package" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": { 74 | "slideshow": { 75 | "slide_type": "slide" 76 | } 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "from torchvision.datasets.mnist import FashionMNIST\n", 81 | "from torchvision.models.resnet import resnet18\n", 82 | "from torchvision import transforms, utils\n", 83 | "\n", 84 | "from PIL import Image\n", 85 | "\n", 86 | "import matplotlib.pyplot as plt\n", 87 | "%matplotlib inline" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": { 93 | "slideshow": { 94 | "slide_type": "notes" 95 | } 96 | }, 97 | "source": [ 98 | "Instead of using the boring MNIST, we will use a better, less boring version of it, its called Fashion MNIST and consists of the same number of images, having the same dimensions and same calsses from 0 to 9 but representing dresses types instead of digits.\n", 99 | "\n", 100 | "The images are W/B 28x28, having the pixel mean and std corresponding to 0.1307 and 0.3081, we can also use a data augmenting function from torchvision.transforms, which permit to \"increase\" the training dataset size by modifying the images applying transformations which won't change the image label, in this case we can use for example a random horizontal flipping function, which unsurprisingly applies horizontal flipping to images with probability p=0.5.\n", 101 | "\n", 102 | "After creating the dataset objects we can finally pass them to the DataLoader init method to get our iterable dataloaders." 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": { 109 | "slideshow": { 110 | "slide_type": "slide" 111 | } 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "transfs_tr = transforms.Compose([\n", 116 | " transforms.RandomHorizontalFlip(),\n", 117 | " transforms.ToTensor(),\n", 118 | " transforms.Normalize([0.1307], [0.3081])\n", 119 | "])\n", 120 | "\n", 121 | "transfs_val = transforms.Compose([\n", 122 | " transforms.ToTensor(),\n", 123 | " transforms.Normalize([0.1307], [0.3081])\n", 124 | "])\n", 125 | "\n", 126 | "dset_tr = FashionMNIST(root='../data/fmnist', train=True, download=True, transform=transfs_tr)\n", 127 | "dset_val = FashionMNIST(root='../data/fmnist', train=False, download=True, transform=transfs_val)\n", 128 | "dataloader_tr = data.DataLoader(dset_tr, batch_size=64)\n", 129 | "dataloader_val = data.DataLoader(dset_val, batch_size=64)\n", 130 | "\n" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": { 136 | "slideshow": { 137 | "slide_type": "notes" 138 | } 139 | }, 140 | "source": [ 141 | "we can actually show some of the images from the dataset using the torchvision's utils package, containing a couple of functions useful for visualization of tensor objects containing images." 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": { 148 | "slideshow": { 149 | "slide_type": "slide" 150 | } 151 | }, 152 | "outputs": [], 153 | "source": [ 154 | "dset_temp = FashionMNIST(root='../data/fmnist', train=True, download=True, transform=transforms.ToTensor())\n", 155 | "dataloader_temp = data.DataLoader(dset_temp, batch_size=64)\n", 156 | "batch_img, batch_label = next(iter(dataloader_temp))\n", 157 | "\n", 158 | "grid = utils.make_grid(batch_img)\n", 159 | "plt.figure(figsize=(10,10))\n", 160 | "plt.imshow(grid.numpy().transpose(1,2,0))" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": { 166 | "slideshow": { 167 | "slide_type": "notes" 168 | } 169 | }, 170 | "source": [ 171 | "As you can see we really have a fashion dataset, certainly less boring than the \"always the same\" MNIST dataset" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": { 177 | "slideshow": { 178 | "slide_type": "notes" 179 | } 180 | }, 181 | "source": [ 182 | "we can now get our model object, as seen in previous sections we will use a resnet model from the torchvision package, but given that we are on CPU and we still need to train a model, we will use a resnet18 (the smallest) instead of resnet152 (the largest). So we create the [pretrained] model and will apply some edits to it. \n", 183 | "\n", 184 | "For example, if I'm not wrong, the model is being deveoped to work with 224x224 images and it makes uses of a average pooling layer in order to average the features for each channel before enetring to the last, classifier layer. Since we have different size images, we can simply replace that layer with and AdaptiveAvgPool2d, a great pytorch module, which will compute the pooling window by itself in order to obtain the required output dimension. Since the standard resnet wants one value for each channel in the last but one layer (before the classifier), we will replace the resnet's avg pool with a nn.AdaptiveAvgPool2d((1,1)).\n", 185 | "\n", 186 | "An other change we need to apply to our pretrained models is due to the fact that the model has been trained on RGB images, having thus 3 channels for image, but our images are W/B, so in attidion to adding a dimension for the color channel, we also need to adapt the first layer of the model. Since it contains 3channel filters, in order to make them work for our aims we simply need to perform a sum over the channel dimension for that parameters, and given that **sum(1)** will actually remove that dimension, we will readd it with the **unsqueeze** method.\n", 187 | "\n", 188 | "Finally, as already explained above, we will replace the last (linear/affine/fully connected) layer having 1000 output features, with a much simpler layer having 10 output features. \n", 189 | "\n", 190 | "Last but not least, transfer learning requires that we don't update the already trained parameters, so we can put them in eval mode, we will also pass to our optimizer, not the entire model's parameters but only the parameters from our last fully connected layer." 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": { 197 | "slideshow": { 198 | "slide_type": "slide" 199 | } 200 | }, 201 | "outputs": [], 202 | "source": [ 203 | "model = resnet18(pretrained=True)\n", 204 | "# replace the avgpool (it's a 7x7 pooling expecting Cx7x7 tensors so it would rise an error)\n", 205 | "model.avgpool = nn.AdaptiveAvgPool2d((1,1))\n", 206 | "# sum over input channels and read its dimension\n", 207 | "model.conv1.weight.data = model.conv1.weight.data.sum(1).unsqueeze(1)\n", 208 | "# replace the last layer (it has 1000 out features and we need new weights)\n", 209 | "model.fc = nn.Linear(model.fc.in_features, 10)\n", 210 | "model.eval()\n", 211 | "model.fc.train()" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": { 217 | "slideshow": { 218 | "slide_type": "notes" 219 | } 220 | }, 221 | "source": [ 222 | "Ok, our pretrained model is ready to give us features from the new data, lets define our loss function (Cross entropy as usual for classification problems) and our optimizer/ccriterion/optimization algorithm, SGD. We will use a learning rate of 0.1 and a momentum of 0.5" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": { 228 | "slideshow": { 229 | "slide_type": "notes" 230 | } 231 | }, 232 | "source": [ 233 | "That's how you perform transfer learning, our problem now is that even if the model is less big respect othe models, our dataset contains a lot of samples, so we should wait to much time to train it on CPU, so we will train a much much simpler model from scratch to end this part of the tutorial, just remember that if you want to actually use the pretrained model and train only the last fully connected layer, you just need to pass **model.fc.parameters()** to the optimizer, instead of passing all model's parameters." 234 | ] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "metadata": { 239 | "slideshow": { 240 | "slide_type": "notes" 241 | } 242 | }, 243 | "source": [ 244 | "So we can now see also how to build a convolution network, which applies shifting filters of window size **k**, storing the filter output values for each shifting. For more information about the convolution layers you can firstly look at this awesome gif below, then go straight to the PyTorch documentation website." 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": { 250 | "slideshow": { 251 | "slide_type": "notes" 252 | } 253 | }, 254 | "source": [ 255 | "source: https://github.com/vdumoulin/conv_arithmetic/blob/master/gif/no_padding_no_strides.gif\n", 256 | "\n", 257 | "PyTorch convolutions documentation: http://pytorch.org/docs/stable/nn.html?highlight=convolution#convolution-layers" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": { 263 | "slideshow": { 264 | "slide_type": "notes" 265 | } 266 | }, 267 | "source": [ 268 | "we can now define our loss function and optimizer objects" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "slideshow": { 276 | "slide_type": "slide" 277 | } 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "loss = nn.CrossEntropyLoss()\n", 282 | "optimizer = optim.SGD(model.fc.parameters(), lr=0.02, momentum=0.5)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": { 288 | "slideshow": { 289 | "slide_type": "notes" 290 | } 291 | }, 292 | "source": [ 293 | "We can now start our training, lets run the model for 10 epochs and see how it performs later" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": { 300 | "slideshow": { 301 | "slide_type": "slide" 302 | } 303 | }, 304 | "outputs": [], 305 | "source": [ 306 | "# how much batches the dataloader will iterate through??\n", 307 | "\n", 308 | "len(dataloader_tr)" 309 | ] 310 | }, 311 | { 312 | "cell_type": "code", 313 | "execution_count": null, 314 | "metadata": { 315 | "slideshow": { 316 | "slide_type": "slide" 317 | } 318 | }, 319 | "outputs": [], 320 | "source": [ 321 | "for epoch in range(1):\n", 322 | " for i, (x, y) in enumerate(dataloader_tr):\n", 323 | " x, y = Variable(x), Variable(y)\n", 324 | " l = loss(model(x), y)\n", 325 | "\n", 326 | " optimizer.zero_grad()\n", 327 | " l.backward()\n", 328 | " optimizer.step()\n", 329 | " if i % 10 == 0:\n", 330 | " print('Epoch: {}, iter:{}, loss: {}'.format(epoch, i, l.data.numpy()[0]))\n", 331 | " if i > 100:\n", 332 | " break" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": { 338 | "slideshow": { 339 | "slide_type": "notes" 340 | } 341 | }, 342 | "source": [ 343 | "Ok, finally we can evaluate our model accuracy performace on the validation dataset (not doing it on the training set because the model already saw those samples many times)" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": { 350 | "slideshow": { 351 | "slide_type": "slide" 352 | } 353 | }, 354 | "outputs": [], 355 | "source": [ 356 | "model.eval()\n", 357 | "preds = []\n", 358 | "ys = []\n", 359 | "for i, (x, y) in enumerate(dataloader_val):\n", 360 | " x, y = Variable(x), Variable(y)\n", 361 | " preds.extend(model(x).max(1)[1].data.tolist())\n", 362 | " ys.extend(y.data)\n", 363 | " if not i % 30:\n", 364 | " print(i)\n", 365 | "\n", 366 | "corrects = (np.array(preds) == np.array(ys))\n", 367 | "print('Accuracy: {}'.format(corrects.mean()))\n" 368 | ] 369 | }, 370 | { 371 | "cell_type": "markdown", 372 | "metadata": { 373 | "slideshow": { 374 | "slide_type": "notes" 375 | } 376 | }, 377 | "source": [ 378 | "And we can show the confusion matrix, which will show how many samples the model classifies correctly." 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "metadata": { 385 | "slideshow": { 386 | "slide_type": "slide" 387 | } 388 | }, 389 | "outputs": [], 390 | "source": [ 391 | "from sklearn.metrics import confusion_matrix\n", 392 | "\n", 393 | "plt.matshow(confusion_matrix(np.array(preds), np.array(ys)))\n" 394 | ] 395 | }, 396 | { 397 | "cell_type": "code", 398 | "execution_count": null, 399 | "metadata": {}, 400 | "outputs": [], 401 | "source": [] 402 | } 403 | ], 404 | "metadata": { 405 | "celltoolbar": "Slideshow", 406 | "kernelspec": { 407 | "display_name": "Python 3", 408 | "language": "python", 409 | "name": "python3" 410 | }, 411 | "language_info": { 412 | "codemirror_mode": { 413 | "name": "ipython", 414 | "version": 3 415 | }, 416 | "file_extension": ".py", 417 | "mimetype": "text/x-python", 418 | "name": "python", 419 | "nbconvert_exporter": "python", 420 | "pygments_lexer": "ipython3", 421 | "version": "3.6.3" 422 | } 423 | }, 424 | "nbformat": 4, 425 | "nbformat_minor": 2 426 | } 427 | -------------------------------------------------------------------------------- /2-Intermediate/2.5-CharRNN.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# Simple CharRNN with PyTorch" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "We are now at the last section of this training, as I get you may want to see something else than image classification tasks, in this section we will approach a NLP (Natural Language Processing) task, that is, to learn to generate names in different languages, just by looking at, guess what?, names! The news is that we will make use of recurrent neural networks, but that's not as much exiting as the obtained results we will get." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "slideshow": { 29 | "slide_type": "notes" 30 | } 31 | }, 32 | "source": [ 33 | "Given that I'm not an NLP expert, this part comes straight from the PyTorch documentation, let call the jupyter magic method to show plots" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": { 40 | "slideshow": { 41 | "slide_type": "slide" 42 | } 43 | }, 44 | "outputs": [], 45 | "source": [ 46 | "%matplotlib inline" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": { 52 | "slideshow": { 53 | "slide_type": "notes" 54 | } 55 | }, 56 | "source": [ 57 | "\n", 58 | "\n", 59 | "Download the data from https://download.pytorch.org/tutorial/data.zip and extract it to the current directory.\n", 60 | "\n", 61 | "In short, there are a bunch of plain text files ``data/names/[Language].txt`` with a name per line. We split lines into an array, convert Unicode to ASCII, and end up with a dictionary ``{language: [names ...]}``.\n", 62 | "\n", 63 | "\n" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": { 70 | "slideshow": { 71 | "slide_type": "slide" 72 | } 73 | }, 74 | "outputs": [], 75 | "source": [ 76 | "from __future__ import unicode_literals, print_function, division\n", 77 | "from io import open\n", 78 | "import glob\n", 79 | "import unicodedata\n", 80 | "import string\n", 81 | "\n", 82 | "all_letters = string.ascii_letters + \" .,;'-\"\n", 83 | "n_letters = len(all_letters) + 1 # Plus EOS marker\n" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "slideshow": { 91 | "slide_type": "slide" 92 | } 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "def findFiles(path): return glob.glob(path)\n", 97 | "\n", 98 | "# Turn a Unicode string to plain ASCII, thanks to \n", 99 | "# http://stackoverflow.com/a/518232/2809427\n", 100 | "def unicodeToAscii(s):\n", 101 | " return ''.join(\n", 102 | " c for c in unicodedata.normalize('NFD', s)\n", 103 | " if unicodedata.category(c) != 'Mn'\n", 104 | " and c in all_letters\n", 105 | " )\n", 106 | "\n", 107 | "# Read a file and split into lines\n", 108 | "def readLines(filename):\n", 109 | " lines = open(filename, encoding='utf-8').read().strip().split('\\n')\n", 110 | " return [unicodeToAscii(line) for line in lines]\n" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": { 117 | "slideshow": { 118 | "slide_type": "slide" 119 | } 120 | }, 121 | "outputs": [], 122 | "source": [ 123 | "# Build the category_lines dictionary, a list of lines per category\n", 124 | "category_lines = {}\n", 125 | "all_categories = []\n", 126 | "for filename in findFiles('../data/names/*.txt'):\n", 127 | " category = filename.split('/')[-1].split('.')[0]\n", 128 | " all_categories.append(category)\n", 129 | " lines = readLines(filename)\n", 130 | " category_lines[category] = lines\n", 131 | "\n", 132 | "n_categories = len(all_categories)\n", 133 | "\n", 134 | "print('# categories:', n_categories, all_categories)\n", 135 | "print(unicodeToAscii(\"O'Néàl\"))" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": { 141 | "slideshow": { 142 | "slide_type": "slide" 143 | } 144 | }, 145 | "source": [ 146 | "Creating the Network\n", 147 | "====================\n", 148 | "\n", 149 | "We will interpret the output as the probability of the next letter. When\n", 150 | "sampling, the most likely output letter is used as the next input\n", 151 | "letter.\n", 152 | "\n", 153 | "this simple scheme shows how we will implement our recurrent model\n", 154 | "\n", 155 | "https://i.imgur.com/jzVrf7f.png\n", 156 | "\n", 157 | "\n", 158 | "\n" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": { 165 | "slideshow": { 166 | "slide_type": "slide" 167 | } 168 | }, 169 | "outputs": [], 170 | "source": [ 171 | "import torch\n", 172 | "import torch.nn as nn\n", 173 | "from torch.autograd import Variable\n", 174 | "\n", 175 | "class RNN(nn.Module):\n", 176 | " def __init__(self, input_size, hidden_size, output_size):\n", 177 | " super(RNN, self).__init__()\n", 178 | " self.hidden_size = hidden_size\n", 179 | " \n", 180 | " out_f = n_categories + input_size + hidden_size\n", 181 | " self.i2h = nn.Linear(out_f, hidden_size)\n", 182 | " self.i2o = nn.Linear(out_f, output_size)\n", 183 | " self.o2o = nn.Linear(hidden_size + output_size, output_size)\n", 184 | " self.dropout = nn.Dropout(0.1)\n", 185 | " self.softmax = nn.LogSoftmax(dim=1)\n", 186 | "\n", 187 | " def forward(self, category, input, hidden):\n", 188 | " input_combined = torch.cat((category, input, hidden), 1)\n", 189 | " hidden = self.i2h(input_combined)\n", 190 | " output = self.i2o(input_combined)\n", 191 | " output_combined = torch.cat((hidden, output), 1)\n", 192 | " output = self.o2o(output_combined)\n", 193 | " output = self.dropout(output)\n", 194 | " output = self.softmax(output)\n", 195 | " return output, hidden\n", 196 | "\n", 197 | " def initHidden(self):\n", 198 | " return Variable(torch.zeros(1, self.hidden_size))" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": { 204 | "slideshow": { 205 | "slide_type": "slide" 206 | } 207 | }, 208 | "source": [ 209 | "Training\n", 210 | "=========\n", 211 | "Preparing for Training\n", 212 | "----------------------\n", 213 | "\n", 214 | "First of all, helper functions to get random pairs of (category, line):\n", 215 | "\n", 216 | "\n" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "metadata": { 223 | "slideshow": { 224 | "slide_type": "slide" 225 | } 226 | }, 227 | "outputs": [], 228 | "source": [ 229 | "import random\n", 230 | "\n", 231 | "# Random item from a list\n", 232 | "def randomChoice(l):\n", 233 | " return l[random.randint(0, len(l) - 1)]\n", 234 | "\n", 235 | "# Get a random category and random line from that category\n", 236 | "def randomTrainingPair():\n", 237 | " category = randomChoice(all_categories)\n", 238 | " line = randomChoice(category_lines[category])\n", 239 | " return category, line" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": { 245 | "slideshow": { 246 | "slide_type": "notes" 247 | } 248 | }, 249 | "source": [ 250 | "For each timestep (that is, for each letter in a training word) the\n", 251 | "inputs of the network will be ``(category, current letter, hidden state)`` and the outputs will be ``(next letter, next hidden state)``. So for each training set, we'll need the category, a set of input letters, and a set of output/target letters.\n", 252 | "\n", 253 | "Since we are predicting the next letter from the current letter for each timestep, the letter pairs are groups of consecutive letters from the line - e.g. for ``\"ABCD\"`` we would create (\"A\", \"B\"), (\"B\", \"C\"),\n", 254 | "(\"C\", \"D\"), (\"D\", \"EOS\").\n", 255 | "\n", 256 | "https://i.imgur.com/JH58tXY.png\n", 257 | "\n", 258 | "The category tensor is a [one-hot tensor](https://en.wikipedia.org/wiki/One-hot) of size\n", 259 | "``<1 x n_categories>``. When training we feed it to the network at every timestep - this is a design choice, it could have been included as part of initial hidden state or some other strategy.\n", 260 | "\n", 261 | "\n" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": null, 267 | "metadata": { 268 | "slideshow": { 269 | "slide_type": "slide" 270 | } 271 | }, 272 | "outputs": [], 273 | "source": [ 274 | "# One-hot vector for category\n", 275 | "def categoryTensor(category):\n", 276 | " li = all_categories.index(category)\n", 277 | " tensor = torch.zeros(1, n_categories)\n", 278 | " tensor[0][li] = 1\n", 279 | " return tensor\n", 280 | "\n", 281 | "# One-hot matrix of first to last letters (not including EOS) for input\n", 282 | "def inputTensor(line):\n", 283 | " tensor = torch.zeros(len(line), 1, n_letters)\n", 284 | " for li in range(len(line)):\n", 285 | " letter = line[li]\n", 286 | " tensor[li][0][all_letters.find(letter)] = 1\n", 287 | " return tensor\n", 288 | "\n", 289 | "# LongTensor of second letter to end (EOS) for target\n", 290 | "def targetTensor(line):\n", 291 | " letter_indexes = [all_letters.find(line[li]) for li in range(\n", 292 | " 1, len(line))]\n", 293 | " letter_indexes.append(n_letters - 1) # EOS\n", 294 | " return torch.LongTensor(letter_indexes)" 295 | ] 296 | }, 297 | { 298 | "cell_type": "markdown", 299 | "metadata": { 300 | "slideshow": { 301 | "slide_type": "notes" 302 | } 303 | }, 304 | "source": [ 305 | "For convenience during training we'll make a ``randomTrainingExample``\n", 306 | "function that fetches a random (category, line) pair and turns them into\n", 307 | "the required (category, input, target) tensors.\n", 308 | "\n", 309 | "\n" 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": null, 315 | "metadata": { 316 | "slideshow": { 317 | "slide_type": "slide" 318 | } 319 | }, 320 | "outputs": [], 321 | "source": [ 322 | "# Make category, input, and target tensors from a random category, line pair\n", 323 | "def randomTrainingExample():\n", 324 | " category, line = randomTrainingPair()\n", 325 | " category_tensor = Variable(categoryTensor(category))\n", 326 | " input_line_tensor = Variable(inputTensor(line))\n", 327 | " target_line_tensor = Variable(targetTensor(line))\n", 328 | " return category_tensor, input_line_tensor, target_line_tensor" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": { 334 | "slideshow": { 335 | "slide_type": "notes" 336 | } 337 | }, 338 | "source": [ 339 | "Training the Network\n", 340 | "--------------------\n", 341 | "\n", 342 | "In contrast to classification, where only the last output is used, we\n", 343 | "are making a prediction at every step, so we are calculating loss at\n", 344 | "every step.\n", 345 | "\n", 346 | "The magic of autograd allows you to simply sum these losses at each step\n", 347 | "and call backward at the end.\n", 348 | "\n", 349 | "\n" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": { 356 | "slideshow": { 357 | "slide_type": "slide" 358 | } 359 | }, 360 | "outputs": [], 361 | "source": [ 362 | "criterion = nn.NLLLoss()\n", 363 | "\n", 364 | "learning_rate = 0.0005\n", 365 | "\n", 366 | "def train(category_tensor, input_line_tensor, target_line_tensor):\n", 367 | " hidden = rnn.initHidden()\n", 368 | "\n", 369 | " rnn.zero_grad()\n", 370 | "\n", 371 | " loss = 0\n", 372 | "\n", 373 | " for i in range(input_line_tensor.size()[0]):\n", 374 | " output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)\n", 375 | " loss += criterion(output, target_line_tensor[i])\n", 376 | "\n", 377 | " loss.backward()\n", 378 | "\n", 379 | " for p in rnn.parameters():\n", 380 | " p.data.add_(-learning_rate, p.grad.data)\n", 381 | "\n", 382 | " return output, loss.data[0] / input_line_tensor.size()[0]" 383 | ] 384 | }, 385 | { 386 | "cell_type": "markdown", 387 | "metadata": { 388 | "slideshow": { 389 | "slide_type": "notes" 390 | } 391 | }, 392 | "source": [ 393 | "To keep track of how long training takes I am adding a\n", 394 | "``timeSince(timestamp)`` function which returns a human readable string:\n", 395 | "\n", 396 | "\n" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": null, 402 | "metadata": { 403 | "slideshow": { 404 | "slide_type": "slide" 405 | } 406 | }, 407 | "outputs": [], 408 | "source": [ 409 | "import time\n", 410 | "import math\n", 411 | "\n", 412 | "def timeSince(since):\n", 413 | " now = time.time()\n", 414 | " s = now - since\n", 415 | " m = math.floor(s / 60)\n", 416 | " s -= m * 60\n", 417 | " return '%dm %ds' % (m, s)" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": { 423 | "slideshow": { 424 | "slide_type": "notes" 425 | } 426 | }, 427 | "source": [ 428 | "Training is business as usual - call train a bunch of times and wait a\n", 429 | "few minutes, printing the current time and loss every ``print_every``\n", 430 | "examples, and keeping store of an average loss per ``plot_every`` examples\n", 431 | "in ``all_losses`` for plotting later.\n", 432 | "\n", 433 | "\n" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": null, 439 | "metadata": { 440 | "slideshow": { 441 | "slide_type": "slide" 442 | } 443 | }, 444 | "outputs": [], 445 | "source": [ 446 | "rnn = RNN(n_letters, 128, n_letters)\n", 447 | "\n", 448 | "n_iters = 100000\n", 449 | "print_every = 5000\n", 450 | "plot_every = 500\n", 451 | "all_losses = []\n", 452 | "total_loss = 0 # Reset every plot_every iters\n", 453 | "\n", 454 | "start = time.time()\n", 455 | "\n", 456 | "for iter in range(1, n_iters + 1):\n", 457 | " output, loss = train(*randomTrainingExample())\n", 458 | " total_loss += loss\n", 459 | "\n", 460 | " if iter % print_every == 0:\n", 461 | " print('%s (%d %d%%) %.4f' % (\n", 462 | " timeSince(start), iter, iter / n_iters * 100, loss))\n", 463 | "\n", 464 | " if iter % plot_every == 0:\n", 465 | " all_losses.append(total_loss / plot_every)\n", 466 | " total_loss = 0" 467 | ] 468 | }, 469 | { 470 | "cell_type": "markdown", 471 | "metadata": { 472 | "slideshow": { 473 | "slide_type": "notes" 474 | } 475 | }, 476 | "source": [ 477 | "Plotting the Losses\n", 478 | "-------------------\n", 479 | "\n", 480 | "Plotting the historical loss from all\\_losses shows the network\n", 481 | "learning:\n", 482 | "\n", 483 | "\n" 484 | ] 485 | }, 486 | { 487 | "cell_type": "code", 488 | "execution_count": null, 489 | "metadata": { 490 | "slideshow": { 491 | "slide_type": "slide" 492 | } 493 | }, 494 | "outputs": [], 495 | "source": [ 496 | "import matplotlib.pyplot as plt\n", 497 | "import matplotlib.ticker as ticker\n", 498 | "\n", 499 | "plt.figure()\n", 500 | "plt.plot(all_losses)" 501 | ] 502 | }, 503 | { 504 | "cell_type": "markdown", 505 | "metadata": { 506 | "slideshow": { 507 | "slide_type": "notes" 508 | } 509 | }, 510 | "source": [ 511 | "Sampling the Network\n", 512 | "====================\n", 513 | "\n", 514 | "To sample we give the network a letter and ask what the next one is,\n", 515 | "feed that in as the next letter, and repeat until the EOS token.\n", 516 | "\n", 517 | "- Create tensors for input category, starting letter, and empty hidden\n", 518 | " state\n", 519 | "- Create a string ``output_name`` with the starting letter\n", 520 | "- Up to a maximum output length,\n", 521 | "\n", 522 | " - Feed the current letter to the network\n", 523 | " - Get the next letter from highest output, and next hidden state\n", 524 | " - If the letter is EOS, stop here\n", 525 | " - If a regular letter, add to ``output_name`` and continue\n", 526 | "\n", 527 | "- Return the final name\n", 528 | "\n", 529 | ".. Note::\n", 530 | " Rather than having to give it a starting letter, another\n", 531 | " strategy would have been to include a \"start of string\" token in\n", 532 | " training and have the network choose its own starting letter.\n", 533 | "\n", 534 | "\n" 535 | ] 536 | }, 537 | { 538 | "cell_type": "code", 539 | "execution_count": null, 540 | "metadata": { 541 | "slideshow": { 542 | "slide_type": "slide" 543 | } 544 | }, 545 | "outputs": [], 546 | "source": [ 547 | "max_length = 20\n", 548 | "\n", 549 | "# Sample from a category and starting letter\n", 550 | "def sample(category, start_letter='A'):\n", 551 | " category_tensor = Variable(categoryTensor(category))\n", 552 | " input = Variable(inputTensor(start_letter))\n", 553 | " hidden = rnn.initHidden()\n", 554 | "\n", 555 | " output_name = start_letter\n", 556 | "\n", 557 | " for i in range(max_length):\n", 558 | " output, hidden = rnn(category_tensor, input[0], hidden)\n", 559 | " topv, topi = output.data.topk(1)\n", 560 | " topi = topi[0][0]\n", 561 | " if topi == n_letters - 1:\n", 562 | " break\n", 563 | " else:\n", 564 | " letter = all_letters[topi]\n", 565 | " output_name += letter\n", 566 | " input = Variable(inputTensor(letter))\n", 567 | "\n", 568 | " return output_name\n" 569 | ] 570 | }, 571 | { 572 | "cell_type": "code", 573 | "execution_count": null, 574 | "metadata": { 575 | "slideshow": { 576 | "slide_type": "slide" 577 | } 578 | }, 579 | "outputs": [], 580 | "source": [ 581 | "# Get multiple samples from one category \n", 582 | "# and multiple starting letters\n", 583 | "def samples(category, start_letters='ABC'):\n", 584 | " for start_letter in start_letters:\n", 585 | " print(sample(category, start_letter))\n", 586 | "\n", 587 | "samples('Russian', 'RUS')\n", 588 | "print('----')\n", 589 | "samples('German', 'GER')\n", 590 | "print('----')\n", 591 | "samples('Spanish', 'SPA')\n", 592 | "print('----')\n", 593 | "samples('Chinese', 'CHI')\n", 594 | "print('----')\n", 595 | "samples('Italian', 'ITA')\n", 596 | "print('----')\n", 597 | "samples('Italian', 'BCDFG')" 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "metadata": { 603 | "slideshow": { 604 | "slide_type": "notes" 605 | } 606 | }, 607 | "source": [ 608 | "Italian names are not very strong, but you get the idea, if we were using LTSMs we would actually get much better results but sould need to wait a lot to train the model on CPU." 609 | ] 610 | }, 611 | { 612 | "cell_type": "markdown", 613 | "metadata": { 614 | "slideshow": { 615 | "slide_type": "notes" 616 | } 617 | }, 618 | "source": [ 619 | "Exercises\n", 620 | "=========\n", 621 | "\n", 622 | "- Try with a different dataset of category -> line, for example:\n", 623 | "\n", 624 | " - Fictional series -> Character name\n", 625 | " - Part of speech -> Word\n", 626 | " - Country -> City\n", 627 | "\n", 628 | "- Use a \"start of sentence\" token so that sampling can be done without\n", 629 | " choosing a start letter\n", 630 | "- Get better results with a bigger and/or better shaped network\n", 631 | "\n", 632 | " - Try the nn.LSTM and nn.GRU layers\n", 633 | " - Combine multiple of these RNNs as a higher level network\n", 634 | "\n", 635 | "\n" 636 | ] 637 | } 638 | ], 639 | "metadata": { 640 | "celltoolbar": "Slideshow", 641 | "kernelspec": { 642 | "display_name": "Python 3", 643 | "language": "python", 644 | "name": "python3" 645 | }, 646 | "language_info": { 647 | "codemirror_mode": { 648 | "name": "ipython", 649 | "version": 3 650 | }, 651 | "file_extension": ".py", 652 | "mimetype": "text/x-python", 653 | "name": "python", 654 | "nbconvert_exporter": "python", 655 | "pygments_lexer": "ipython3", 656 | "version": "3.6.3" 657 | } 658 | }, 659 | "nbformat": 4, 660 | "nbformat_minor": 1 661 | } 662 | -------------------------------------------------------------------------------- /1-Basics/1.2-Linear-Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "slideshow": { 7 | "slide_type": "slide" 8 | } 9 | }, 10 | "source": [ 11 | "# 1.2 Linear Regression with PyTorch" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": { 17 | "slideshow": { 18 | "slide_type": "notes" 19 | } 20 | }, 21 | "source": [ 22 | "we will see now how to implement a really simple task with PyTorch, such as performing a linear regression on two small sets of points. \n", 23 | "\n", 24 | "The task consists of: given a set of *independent* x values, learn to estimate the relationship (beta) with the corresponding *dependent* y values.\n", 25 | "\n", 26 | "more info: https://en.wikipedia.org/wiki/Regression_analysis\n", 27 | "\n", 28 | "First we import our usual packages" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 1, 34 | "metadata": { 35 | "slideshow": { 36 | "slide_type": "slide" 37 | } 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "import torch\n", 42 | "import torch.nn as nn\n", 43 | "import numpy as np\n", 44 | "import matplotlib.pyplot as plt\n", 45 | "from torch.autograd import Variable\n", 46 | "\n", 47 | "%matplotlib inline" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": { 53 | "slideshow": { 54 | "slide_type": "notes" 55 | } 56 | }, 57 | "source": [ 58 | "Lets create our datapoints, you can copy the same numbers or change them a bit, that wouldn't change our aims to show how the process works" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 2, 64 | "metadata": { 65 | "slideshow": { 66 | "slide_type": "slide" 67 | } 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "# Toy Dataset \n", 72 | "x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168], \n", 73 | " [9.779], [6.182], [7.59], [2.167], [7.042], \n", 74 | " [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)\n", 75 | "\n", 76 | "y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573], \n", 77 | " [3.366], [2.596], [2.53], [1.221], [2.827], \n", 78 | " [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 3, 84 | "metadata": { 85 | "slideshow": { 86 | "slide_type": "slide" 87 | } 88 | }, 89 | "outputs": [ 90 | { 91 | "data": { 92 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFNxJREFUeJzt3X+Q1PV9x/HX+87TyyHViqciyC2lpFrIgXqIlkxHQzTWxuhESexsbXDS3EySNthxbIyXSH4MmTo6sRgSM2f8gboTYzCJ1jFtbTRjzEyMB4IoOP4od3hqdSHlgK7GA97947sgLHfsd7nd+373s8/HzM5397Of232z3L3ue5/ve79r7i4AQFiaki4AAFB9hDsABIhwB4AAEe4AECDCHQACRLgDQIAIdwAIEOEOAAEi3AEgQEck9cTHH3+8ZzKZpJ4eAOrS6tWrt7h7e7l5iYV7JpNRX19fUk8PAHXJzAbizGNZBgACRLgDQIDKhruZtZrZ78xsnZm9YGbfGGHOYjPLm9na4uXva1MuACCOOGvuf5D0EXffaWYtkp4ys1+4+29L5v3Y3f9hLMUMDw9rcHBQ77777lgeBlXS2tqqqVOnqqWlJelSAFSobLh7dML3ncWbLcVLTU4CPzg4qIkTJyqTycjMavEUiMndtXXrVg0ODmr69OlJlwOgQrHW3M2s2czWSnpb0mPu/vQI0y4zs+fMbJWZnTLK43SbWZ+Z9eXz+YPuf/fddzVp0iSCPQXMTJMmTeKvKKCacjkpk5GamqJtLlezp4oV7u6+293nSpoq6Swzm10y5d8kZdy9U9J/SVo5yuP0unuXu3e1t4/cpkmwpwf/F0AV5XJSd7c0MCC5R9vu7poFfEXdMu6+TdKvJF1YMr7V3f9QvHm7pDOrUh0AhKKnRyoUDhwrFKLxGojTLdNuZscWr39A0kclvVgyZ/J+Nz8haWM1ixxPg4ODuuSSSzRz5kzNmDFDS5Ys0XvvvTfi3DfeeEOXX3552ce86KKLtG3btsOq5+tf/7puvvnmsvOOPvroQ96/bds2ff/73z+sGgBUwebNlY2PUZw998mSnjCz5yQ9o2jN/REz+6aZfaI450vFNsl1kr4kaXFNqi1V5fUrd9cnP/lJXXrppXr55Zf10ksvaefOneoZ4Tfrrl27dPLJJ2vVqlVlH/fRRx/VscceO6baxopwBxI2bVpl42NUNtzd/Tl3P93dO919trt/szh+g7s/XLz+FXef5e5z3P08d3/x0I9aBTVYv3r88cfV2tqqq666SpLU3NysW265RXfeeacKhYLuvvtuLVq0SBdffLEuuOAC9ff3a/bs6PBDoVDQpz71KXV2durTn/605s+fv+/0CplMRlu2bFF/f79OO+00fe5zn9OsWbN0wQUX6J133pEk3X777Zo3b57mzJmjyy67TIXSP99KbNq0Seecc47mzZunr33ta/vGd+7cqYULF+qMM87Qhz70IT300EOSpOuuu06vvvqq5s6dq2uvvXbUeQBqZNkyqa3twLG2tmi8Ftw9kcuZZ57ppTZs2HDQ2Kg6OtyjWD/w0tER/zFKLF++3K+++uqDxufOnevr1q3zu+66y6dMmeJbt251d/dNmzb5rFmz3N39pptu8u7ubnd3X79+vTc3N/szzzxTLLXD8/m8b9q0yZubm/3ZZ591d/dFixb5vffe6+7uW7Zs2fd8PT09fuutt7q7+9KlS/2mm246qKaLL77YV65c6e7uK1as8AkTJri7+/DwsA8NDbm7ez6f9xkzZviePXsOqPVQ80pV9H8C4NDuuy/KKLNoe999FT+EpD6PkbGJnThszGqwfuXuI3aI7D9+/vnn67jjjjtozlNPPaUlS5ZIkmbPnq3Ozs4Rn2P69OmaO3euJOnMM89Uf3+/JOn555/XV7/6VW3btk07d+7Uxz72sUPW+pvf/EYPPvigJOnKK6/Ul7/85X21Xn/99XryySfV1NSk119/XW+99daI/6aR5p100kmHfF4AY5DNRpdxUL/nlqnB+tWsWbMOOlPl9u3b9dprr2nGjBmSpAkTJoz4tdEv1PKOOuqofdebm5u1a9cuSdLixYu1YsUKrV+/XkuXLo3VXz7SL6JcLqd8Pq/Vq1dr7dq1OvHEE0d8rLjzANSn+g33GqxfLVy4UIVCQffcc48kaffu3brmmmu0ePFitZU+V4kPf/jDeuCBByRJGzZs0Pr16yt67h07dmjy5MkaHh5WLsZxgwULFuj++++XpAPmDw0N6YQTTlBLS4ueeOIJDQxEZwedOHGiduzYUXYeEJRxfNNQ2tRvuGezUm+v1NEhmUXb3t4x/cljZvrZz36mn/zkJ5o5c6Y++MEPqrW1Vd/+9rfLfu0XvvAF5fN5dXZ26sYbb1RnZ6eOOeaY2M/9rW99S/Pnz9f555+vU089tez85cuX63vf+57mzZunoaGhfePZbFZ9fX3q6upSLpfb91iTJk3SggULNHv2bF177bWjzgOCMc5vGkobi7ucUG1dXV1eugSyceNGnXbaaYnUM1a7d+/W8PCwWltb9eqrr2rhwoV66aWXdOSRRyZd2pjU8/8JGlwmEwV6qY4OqXisqx6Z2Wp37yo3r34PqKZMoVDQeeedp+HhYbm7brvttroPdqCujfObhtKGcK+SiRMn8rGBQJpMmzbynnuN3jSUNqlbc09qmQgH4/8CdW283zSUMqkK99bWVm3dupVQSQEvns+9tbU16VKAw1ODpot6kqoDqnwSU7rwSUxA+tTlAdWWlhY+9QcAqiBVyzIAgOog3AEgQIQ7AASIcAeAABHuABAgwh0AAkS4A5Vo4FPIor6kqs8dSLW9p5Dd+/m2e08hKzXMux5RP9hzB+Lq6Xk/2PcqFKJxIGUIdyCuBj+FLOoL4Q7EVYPP7QVqhXAH4mrwU8iivhDuQFwNfgpZ1Be6ZYBKZLOEOeoCe+4AECDCHQACRLgDQIAIdwAIEOEOAAEi3AEgQIQ7AASIcAeAABHuABCgsuFuZq1m9jszW2dmL5jZN0aYc5SZ/djMXjGzp80sU4tiAQDxxNlz/4Okj7j7HElzJV1oZmeXzPmspP919z+VdIukG6tbJgCgEmXD3SM7izdbihcvmXaJpJXF66skLTQzq1qVAICKxFpzN7NmM1sr6W1Jj7n70yVTpkh6TZLcfZekIUmTRnicbjPrM7O+fD4/tsoBAKOKFe7uvtvd50qaKuksM5tdMmWkvfTSvXu5e6+7d7l7V3t7e+XVAgBiqahbxt23SfqVpAtL7hqUdIokmdkRko6R9Psq1AcAOAxxumXazezY4vUPSPqopBdLpj0s6TPF65dLetzdD9pzBwCMjzgf1jFZ0koza1b0y+ABd3/EzL4pqc/dH5Z0h6R7zewVRXvsV9SsYgBAWWXD3d2fk3T6COM37Hf9XUmLqlsaAOBw8Q5VIHS5nJTJSE1N0TaXS7oijAM+QxUIWS4ndXdLhUJ0e2Agui3xWbCBY88dCFlPz/vBvlehEI0jaIQ7ELLNmysbRzAIdyBk06ZVNo5gEO5AyJYtk9raDhxra4vGETTCHaiVNHSpZLNSb6/U0SGZRdveXg6mNgC6ZYBaSFOXSjZLmDcg9tyBWqBLBQkj3IFaoEsFCSPcgVqgSwUJI9yBWqBLBQkj3BtFGjo3GgldKkgY3TKNIE2dG42ELhUkiD33RkDnBtBwCPdGQOcG0HAI90ZA5wbQcAj3RkDnBtBwCPdGQOcG0HDolmkUdG4ADYU9dwAIEOEOAAEi3AEgQIQ7AASIcAeAABHuABAgwh0AAkS4I3yc7hgNiDcxIWyc7hgNij13hI3THaNBEe4IG6c7RoMi3BE2TneMBkW4I2yc7hgNinBH2EI63TFdP6gA3TIIXwinO6brBxUqu+duZqeY2RNmttHMXjCzJSPMOdfMhsxsbfFyQ23KBRoUXT+oUJw9912SrnH3NWY2UdJqM3vM3TeUzPu1u3+8+iUCoOsHlSq75+7ub7r7muL1HZI2SppS68IA7IeuH1SoogOqZpaRdLqkp0e4+xwzW2dmvzCzWaN8fbeZ9ZlZXz6fr7hYoGHR9YMKxQ53Mzta0oOSrnb37SV3r5HU4e5zJH1X0s9Hegx373X3Lnfvam9vP9yagcYTUtcPxoW5e/lJZi2SHpH0H+7+nRjz+yV1ufuW0eZ0dXV5X19fBaUCAMxstbt3lZsXp1vGJN0haeNowW5mJxXnyczOKj7u1spKBgBUS5xumQWSrpS03szWFseulzRNktz9B5Iul/R5M9sl6R1JV3icPwkAADVRNtzd/SlJVmbOCkkrqlUUAGBsOP0AAASIcAeAABHuABAgwh0AAkS4A0CACHcACBDhDgABItwBIECEOwAEiHAHgAAR7gAQIMIdAAJEuANAgAh3AAgQ4Q4AASLcASBAhDsABIhwB4AAEe4AECDCHQACRLgDQIAIdwAIEOEOAAEi3AEgQIQ7AASIcAeAABHuABAgwh3Jy+WkTEZqaoq2uVzSFQF174ikC0CDy+Wk7m6pUIhuDwxEtyUpm02uLqDOseeOZPX0vB/sexUK0TiAw0a4I1mbN1c2DiAWwh3JmjatsnEAsRDuSNayZVJb24FjbW3ROIDDRrgjWdms1NsrdXRIZtG2t5eDqcAY0S2D5GWzhDlQZWX33M3sFDN7wsw2mtkLZrZkhDlmZrea2Stm9pyZnVGbcgEAccTZc98l6Rp3X2NmEyWtNrPH3H3DfnP+StLM4mW+pNuKWwBAAsruubv7m+6+pnh9h6SNkqaUTLtE0j0e+a2kY81sctWrBQDEUtEBVTPLSDpd0tMld02R9Np+twd18C8AmVm3mfWZWV8+n6+sUgBAbLHD3cyOlvSgpKvdfXvp3SN8iR804N7r7l3u3tXe3l5ZpQCA2GKFu5m1KAr2nLv/dIQpg5JO2e/2VElvjL08AMDhiNMtY5LukLTR3b8zyrSHJf1dsWvmbElD7v5mFesEAFQgTrfMAklXSlpvZmuLY9dLmiZJ7v4DSY9KukjSK5IKkq6qfqkAgLjKhru7P6WR19T3n+OSvlitogAAY8PpBwAgQIQ7AASIcAeAABHuABAgwh0AAkS4A0CACHcACBDhDgABItwBIECEOwAEiHAHgAAR7gAQIMIdAAJEuANAgAh3AAgQ4Q4AASLcASBAhDsABIhwr6ZcTspkpKamaJvLJV0RxhvfA0iJOB+QjThyOam7WyoUotsDA9FtScpmk6sL44fvAaSIRZ9tPf66urq8r68vkeeuiUwm+mEu1dEh9fePdzVIAt8DGAdmttrdu8rNY1mmWjZvrmwc4eF7AClCuFfLtGmVjTeaRliL5nsAKUK4V8uyZVJb24FjbW3ReKPbuxY9MCC5v78WHVrA8z2AFCHcqyWblXp7o/VVs2jb28uBNEnq6Xn/IONehUI0HhK+B5AiHFBF7TU1RXvspcykPXvGvx6gjnFAFenBWjQw7gh31B5r0cC4I9xRe6xFA+OOcA9F2lsNs9nojTx79kRbgh2oKU4/EALe9g6gBHvuIWiUVkMAsRHuIeBt7wBKEO4hoNUQQAnCPQS0GgIoUTbczexOM3vbzJ4f5f5zzWzIzNYWLzdUv0wcEq2GAErE6Za5W9IKSfccYs6v3f3jVakIhyebJcwB7FN2z93dn5T0+3GoBQBQJdVacz/HzNaZ2S/MbNZok8ys28z6zKwvn89X6akBAKWqEe5rJHW4+xxJ35X089Emunuvu3e5e1d7e3sVnhoAMJIxh7u7b3f3ncXrj0pqMbPjx1wZAOCwjTnczewkM7Pi9bOKj7l1rI8LADh8ZbtlzOxHks6VdLyZDUpaKqlFktz9B5Iul/R5M9sl6R1JV3hSnwACAJAUI9zd/W/K3L9CUaskACAleIcqAASIcAeAABHuABAgwh0AAkS4A0CACHcACBDhDgABItwBIECEOwAEiHCvVC4nZTJSU1O0zeWSrggADhLnk5iwVy4ndXdLhUJ0e2Agui3xKUgAUoU990r09Lwf7HsVCtE4AKQI4V6JzZsrGweAhBDulZg2rbJxAEgI4V6JZcuktrYDx9raonEASBHCvRLZrNTbK3V0SGbRtreXg6kAUqe+wj0NbYjZrNTfL+3ZE20JdgApVD+tkLQhAkBs9bPnThsiAMRWP+FOGyIAxFY/4U4bIgDEVj/hThsiAMRWP+FOGyIAxFY/3TJSFOSEOQCUVT977gCA2Ah3AAgQ4Q4AASLcASBAhDsABMjcPZknNstLGogx9XhJW2pcTj3idRkdr83IeF1GV0+vTYe7t5eblFi4x2Vmfe7elXQdacPrMjpem5HxuowuxNeGZRkACBDhDgABqodw7026gJTidRkdr83IeF1GF9xrk/o1dwBA5ephzx0AUKFUhruZnWJmT5jZRjN7wcyWJF1TmphZs5k9a2aPJF1LmpjZsWa2ysxeLH7vnJN0TWlhZv9U/Fl63sx+ZGatSdeUFDO708zeNrPn9xs7zsweM7OXi9s/TrLGakhluEvaJekadz9N0tmSvmhmf55wTWmyRNLGpItIoeWS/t3dT5U0R7xGkiQzmyLpS5K63H22pGZJVyRbVaLulnRhydh1kn7p7jMl/bJ4u66lMtzd/U13X1O8vkPRD+mUZKtKBzObKumvJf0w6VrSxMz+SNJfSrpDktz9PXfflmxVqXKEpA+Y2RGS2iS9kXA9iXH3JyX9vmT4Ekkri9dXSrp0XIuqgVSG+/7MLCPpdElPJ1tJavyrpH+WtCfpQlLmTyTlJd1VXLL6oZlNSLqoNHD31yXdLGmzpDclDbn7fyZbVeqc6O5vStHOpaQTEq5nzFId7mZ2tKQHJV3t7tuTridpZvZxSW+7++qka0mhIySdIek2dz9d0v8pgD+tq6G4fnyJpOmSTpY0wcz+NtmqUGupDXcza1EU7Dl3/2nS9aTEAkmfMLN+SfdL+oiZ3ZdsSakxKGnQ3ff+hbdKUdhD+qikTe6ed/dhST+V9BcJ15Q2b5nZZEkqbt9OuJ4xS2W4m5kpWjvd6O7fSbqetHD3r7j7VHfPKDog9ri7swcmyd3/R9JrZvZnxaGFkjYkWFKabJZ0tpm1FX+2FoqDzaUelvSZ4vXPSHoowVqqIq2fobpA0pWS1pvZ2uLY9e7+aII1If3+UVLOzI6U9N+Srkq4nlRw96fNbJWkNYo60Z5VgO/IjMvMfiTpXEnHm9mgpKWS/kXSA2b2WUW/DBclV2F18A5VAAhQKpdlAABjQ7gDQIAIdwAIEOEOAAEi3AEgQIQ7AASIcAeAABHuABCg/wfIgd/kvMMxQwAAAABJRU5ErkJggg==\n", 93 | "text/plain": [ 94 | "
" 95 | ] 96 | }, 97 | "metadata": {}, 98 | "output_type": "display_data" 99 | } 100 | ], 101 | "source": [ 102 | "plt.plot(x_train, y_train, 'ro', label='Original data')\n", 103 | "plt.legend()\n", 104 | "plt.show()" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": { 110 | "slideshow": { 111 | "slide_type": "notes" 112 | } 113 | }, 114 | "source": [ 115 | "Now we create a really simple model, subclassing **nn.Module** and thus reimplementing its **forward** method, which gets called everytime a call is performed on the instantiated object (like as in **x = model(x)**, which triggers the dunder method **\\__call__**, which returns the results of the forward method). \n", 116 | "\n", 117 | "Our model for the linear regression will consits in just a single linear layer, also known as affine layer or fully connected layer, which applies a linear transformation to the incoming data: `y = Wx + b`. \n", 118 | "\n", 119 | "So we initialize the layer object and call it with **x** as argument in the **forward** method.\n", 120 | "\n", 121 | "Our Linear layer will have a single input and output value, thats because torch always assumes you send the data in batches through modules, so we will at each step send x_train, and that's the reason of the arrays having an additional dimension." 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 4, 127 | "metadata": { 128 | "slideshow": { 129 | "slide_type": "slide" 130 | } 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "# Linear Regression Model\n", 135 | "class LinearRegression(nn.Module):\n", 136 | " def __init__(self, input_size, output_size):\n", 137 | " super(LinearRegression, self).__init__()\n", 138 | " self.linear = nn.Linear(input_size, output_size) \n", 139 | " \n", 140 | " def forward(self, x):\n", 141 | " out = self.linear(x)\n", 142 | " return out" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": { 148 | "slideshow": { 149 | "slide_type": "notes" 150 | } 151 | }, 152 | "source": [ 153 | "We can now insantiate our model object, loss function and optimizing algorithm\n", 154 | "\n", 155 | "We use an MSELoss (which stands for Mean Squared Error Loss), that computes the mean squared error between two inputs (the model's output and the actual target/ground truth/etc, the one which should be the correct output).\n", 156 | "\n", 157 | "As optimization algorithm we use the standard gradient descent algorithm (SGD) which consists of the computations of the error and the derivative of each one of the models' parameters with respect to it (the gradients). The algorithm updates than each parameters applying `w' = w - lr * dl/dw`." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 5, 163 | "metadata": { 164 | "slideshow": { 165 | "slide_type": "slide" 166 | } 167 | }, 168 | "outputs": [], 169 | "source": [ 170 | "# single \"neural unit\" layer\n", 171 | "model = LinearRegression(1, 1)\n", 172 | "# same as:\n", 173 | "# model = nn.Linear(1, 1)\n", 174 | "\n", 175 | "# Loss and Optimizer\n", 176 | "criterion = nn.MSELoss()\n", 177 | "optimizer = torch.optim.SGD(model.parameters(), lr=0.001)" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": { 183 | "slideshow": { 184 | "slide_type": "notes" 185 | } 186 | }, 187 | "source": [ 188 | "In order to train our simplest model, we loop trough the desired number of epochs, performing an optimization step at every run in the for loop, remember that the model actually sees every data point and its respective y in a single forward pass, so the loss (and the gradients) will be averaged at each step." 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 6, 194 | "metadata": { 195 | "slideshow": { 196 | "slide_type": "slide" 197 | } 198 | }, 199 | "outputs": [ 200 | { 201 | "name": "stdout", 202 | "output_type": "stream", 203 | "text": [ 204 | "Epoch [5/60], Loss: 22.0868\n", 205 | "Epoch [10/60], Loss: 9.0634\n", 206 | "Epoch [15/60], Loss: 3.7874\n", 207 | "Epoch [20/60], Loss: 1.6500\n", 208 | "Epoch [25/60], Loss: 0.7840\n", 209 | "Epoch [30/60], Loss: 0.4332\n", 210 | "Epoch [35/60], Loss: 0.2910\n", 211 | "Epoch [40/60], Loss: 0.2334\n", 212 | "Epoch [45/60], Loss: 0.2100\n", 213 | "Epoch [50/60], Loss: 0.2005\n", 214 | "Epoch [55/60], Loss: 0.1966\n", 215 | "Epoch [60/60], Loss: 0.1949\n" 216 | ] 217 | } 218 | ], 219 | "source": [ 220 | "# Train the Model \n", 221 | "for epoch in range(60):\n", 222 | " # Convert numpy array to torch Variable\n", 223 | " inputs = Variable(torch.from_numpy(x_train))\n", 224 | " targets = Variable(torch.from_numpy(y_train))\n", 225 | "\n", 226 | " # Forward + Backward + Optimize\n", 227 | " optimizer.zero_grad() \n", 228 | " outputs = model(inputs)\n", 229 | " loss = criterion(outputs, targets)\n", 230 | " loss.backward()\n", 231 | " optimizer.step()\n", 232 | " \n", 233 | " if (epoch+1) % 5 == 0:\n", 234 | " print ('Epoch [%d/60], Loss: %.4f' \n", 235 | " %(epoch+1, loss.data[0]))\n", 236 | " " 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": { 242 | "slideshow": { 243 | "slide_type": "notes" 244 | } 245 | }, 246 | "source": [ 247 | "We run the SGD algorithm for 60 epochs, let's see what the model has learnt by plotting the regression line (remember y = Wx + b from above?) and the ground truth points" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 7, 253 | "metadata": { 254 | "slideshow": { 255 | "slide_type": "slide" 256 | } 257 | }, 258 | "outputs": [ 259 | { 260 | "data": { 261 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt4VNW9//H3AgLhpiigIhAmYpQ7AYJCQYtyEQEvB0VpqVaPLVU5Sn9VFAkKoiBUD2ofL5xYLXpM9SAKqKD1AgiCIgG5Y9HIABFEQAPEgARYvz8mDNlDQiZkJnvPzOf1PDzJXrOS+TrETxZr7/luY61FRETiSzW3CxARkchTuIuIxCGFu4hIHFK4i4jEIYW7iEgcUriLiMQhhbuISBxSuIuIxCGFu4hIHKrh1hM3atTI+nw+t55eRCQmrVixYre1tnF581wLd5/PR05OjltPLyISk4wxW8KZp20ZEZE4pHAXEYlDCncRkTjk2p57aYqKisjLy+PgwYNulyJAcnIyzZo1Iykpye1SRKSCPBXueXl51K9fH5/PhzHG7XISmrWWPXv2kJeXR2pqqtvliEgFeWpb5uDBgzRs2FDB7gHGGBo2bKh/RYnEKE+FO6Bg9xD9XYjELs+Fu4hIvDpYdISpH25ie/6BqD+Xwj1EXl4e11xzDWlpabRs2ZKRI0dy6NChUudu376d66+/vtzvOWDAAPLz80+pnvHjx/PEE0+UO69evXonfTw/P5/nnnvulGoQkcqbkbONVg++z98+/ppFm3ZF/fliO9yzs8Hng2rVAh+zsyv17ay1DB48mGuvvZavv/6aTZs2UVBQQGZm5glzDx8+zLnnnsvMmTPL/b7z5s2jQYMGlaqtshTuIu7Ye6AI3+i53DdzDQDXpp/L0ItSov68sRvu2dkwfDhs2QLWBj4OH16pgJ8/fz7JycnceuutAFSvXp0nn3ySl156icLCQqZPn86QIUO46qqr6NevH36/n3bt2gFQWFjIDTfcQIcOHbjxxhu5+OKLg+0VfD4fu3fvxu/307p1a/74xz/Stm1b+vXrx4EDgX+evfDCC3Tt2pWOHTty3XXXUVhYeNJaN2/eTPfu3enatSsPPvhgcLygoIDevXvTuXNn2rdvz5w5cwAYPXo0ubm5pKenM2rUqDLniUjkTPskl44PfxA8XjTqMp4a2qlKnjt2wz0zE0IDsLAwMH6K1q9fT5cuXRxjp512GikpKXzzzTcAfPbZZ7z88svMnz/fMe+5557jjDPOYM2aNTz44IOsWLGi1Of4+uuvGTFiBOvXr6dBgwa8+eabAAwePJjly5ezevVqWrduzYsvvnjSWkeOHMkdd9zB8uXLOeecc4LjycnJzJo1i5UrV7JgwQLuuecerLVMnjyZli1bsmrVKh5//PEy54lI5f2w7yC+0XOZ/N5XAPzp0vPwTx5ISsM6VVaDp65zr5CtWys2HgZrbalXiJQc79u3L2eeeeYJcz799FNGjhwJQLt27ejQoUOpz5Gamkp6ejoAXbp0we/3A7Bu3TrGjh1Lfn4+BQUFXHHFFSetdcmSJcFfDDfddBP3339/sNYxY8awaNEiqlWrxnfffcfOnTtL/W8qbV7JXxQiUnGPvLuBFz/dHDxentmHxvVrVXkdsRvuKSmBrZjSxk9R27Ztg4F5zL59+9i2bRstW7ZkxYoV1K1bt9SvDXfVW6vW8b/k6tWrB7dlbrnlFmbPnk3Hjh2ZPn06CxcuLPd7lfaLKDs7m127drFixQqSkpLw+XylXqse7jwRCY9/98/0emJh8DhzQGv+eOl5rtUTu9syEydCnZB/4tSpExg/Rb1796awsJBXXnkFgCNHjnDPPfdwyy23UCf0uUL07NmTGTNmALBhwwbWrl1boefev38/TZo0oaioiOwwzhv06NGD119/HcAxf+/evZx11lkkJSWxYMECthT/Aqxfvz779+8vd56IVNxdr33pCPY14/u5GuwQy+E+bBhkZUGLFmBM4GNWVmD8FBljmDVrFm+88QZpaWlccMEFJCcnM2nSpHK/9s4772TXrl106NCBKVOm0KFDB04//fSwn/uRRx7h4osvpm/fvrRq1arc+U8//TTPPvssXbt2Ze/evcHxYcOGkZOTQ0ZGBtnZ2cHv1bBhQ3r06EG7du0YNWpUmfNEJHzrvtuLb/Rc3lm9HYAnhnTEP3kgpyW734/JuHUSLSMjw4berGPjxo20bt3alXoq68iRIxQVFZGcnExubi69e/dm06ZN1KxZ0+3SKiWW/05EouXoUcvQrM/5wv8jAGfUSeKzB3qTnFQ96s9tjFlhrc0ob17s7rl7TGFhIZdddhlFRUVYa3n++edjPthF5ERLc3fz2xeWBY9fuiWDy1ud7WJFpVO4R0j9+vV120CROFZ05Ch9pn7Clj2BS7BbnVOfuXdfQvVq3uzBpHAXESnH++t2cPurK4PHM2/vTobvxEuivUThLiJShgOHjtDpkQ84WHQUgEsvaMzLt3aNiY6pCncRkVL8c9lWxsw6fknzv/58KReeU9/Fiiqm3HA3xiQDi4BaxfNnWmvHhcy5BXgc+K546Blr7d8jW6qISPTlFx4ifcKHweMhXZrx+JCOLlZ0asK5zv0X4HJrbUcgHehvjOlWyrz/s9amF/+J2WCvXr066enpwT9+v5+cnBzuvvtuABYuXMjSpUuD82fPns2GDRsq/Dxlteg9Nh5uO2ERiZxn5n/tCPbF910Wk8EOYazcbeBC+ILiw6TiP3HbYap27dqsWrXKMebz+cjICFxWunDhQurVq8evfvUrIBDugwYNok2bNhGtI9x2wiJSed/vPUi3xz4OHo+4rCWjrojtN/aF9Q5VY0x1Y8wq4AfgQ2vtslKmXWeMWWOMmWmMaR7RKl22cOFCBg0ahN/vZ9q0aTz55JOkp6fzySef8PbbbzNq1CjS09PJzc0lNzeX/v3706VLFy655BK++irQFa6sFr1lKdlOePr06QwePJj+/fuTlpbGfffdF5z3wQcf0L17dzp37syQIUMoKCgo61uKSCnGzVnnCPYVY/tEL9gjfA+KkwnrhKq19giQboxpAMwyxrSz1q4rMeUd4DVr7S/GmNuBl4HLQ7+PMWY4MBwgpZwGXw+/s54N2/eF918Rpjbnnsa4q9qedM6BAweCXRtTU1OZNWtW8DGfz8ftt99OvXr1uPfeewG4+uqrGTRoUHALpXfv3kybNo20tDSWLVvGnXfeyfz584Mtem+++WaeffbZCte+atUqvvzyS2rVqsWFF17IXXfdRe3atXn00Uf56KOPqFu3LlOmTGHq1Kk89NBDFf7+Iokmd1cBvf/7k+DxQ4Pa8J89U6P3hMfuQXGsVfmxe1BApdqmlKVCV8tYa/ONMQuB/sC6EuN7Skx7AZhSxtdnAVkQaD9Q0WKrQmnbMuEqKChg6dKlDBkyJDj2yy+/AGW36A1X7969g71q2rRpw5YtW8jPz2fDhg306NEDgEOHDtG9e/dTql0kUVhruePVlby//vvg2LqHr6BerShfPHiye1C4Ee7GmMZAUXGw1wb6EBLexpgm1todxYdXAxsrW1h5K2wvOnr0KA0aNCjzl0Nlro0NbRV8+PBhrLX07duX11577ZS/r0giWZOXz9XPLAkePz00nWvSm1bNk0fhHhQnE86eexNggTFmDbCcwJ77u8aYCcaYq4vn3G2MWW+MWQ3cDdwSlWo9ILR1bsnj0047jdTUVN544w0gsEJYvXo1UHaL3sro1q0bS5YsCd4lqrCwkE2bNkXke4vEk6NHLdc+uyQY7GfVr8W/H+1fdcEOZd9rohL3oDiZcsPdWrvGWtvJWtvBWtvOWjuhePwha+3bxZ8/YK1ta63taK29zFr7VVSq9YCrrrqKWbNmkZ6ezuLFixk6dCiPP/44nTp1Ijc3l+zsbF588UU6duxI27Ztg/cmLatFb2U0btyY6dOn85vf/IYOHTrQrVu34AlcEQn457KtnDdmHqu25QMw/daufJHZh1o1ot/B0SEK96A4GbX8lZPS34nEqsJDh2nz0L+Cx+2bns7sET3cbfSVnR3YY9+6NbBinzixwvvtavkrIgnrzuwVzFt7/ITp+KvacEuPKF4JE65hw6Jy8rQ0sXsnJhGRELsLfsE3eq4j2De/PoJbLmkZ9evKvcZzK3drbUx0XEsEbm3ZiZyK/k8t4qvvj1/s8HzKz1w56tYqu67cazwV7snJyezZs4eGDRsq4F1mrWXPnj0kJye7XYrISX27q4DLS7wZCcA/eWBgpV6F15V7jafCvVmzZuTl5bFr1y63SxECv2ybNWvmdhkiZfKNnus4fvOO7nRpUXwTjSq+rtxrPBXuSUlJpKZ64KSHiHjaii0/ct3znznG/JMHOielpAS2YkJF6bpyr/FUuIuIlCd0tf7xPb+mZeNSWmhPnOjs5QJRva7ca3S1jIjEhPfX7XAEe9pZ9fBPHlh6sENgXz0rC1q0AGMCH7OyEmK/HbRyFxGPs9aS+sA8x9jyzD40rl+rjK8ooQqvK/cahbuIeNY/lmzm4XeO3+nsynbn8PzvurhYUexQuIuI5xQdOUpa5nuOsQ0TrqBOTUVWuPRKiYinTHhnAy8t2Rw8vv3XLRl9ZWzf8s4NCncR8YSCXw7Tbty/HGPfTLySGtV13cep0KsmUhFVeA/MRHLb9OWOYH/k2nb4Jw9UsFeCVu4i4arie2Amgh/2HeSiSR87xjY/NkDtRyLAU/3cRTzN5yv9HY8tWoDfX9XVxLxfP76ALXuOv8Ho7zdn0KfN2S5WFBvUz10k0hK8V0mkfL1zP32fXOQYO6F1gFSawl0kXAneqyQSQlsHzB7Rg/TmDVyqJr7pbIVIuKr4Hpjx5PNv9ziCvVaNavgnD1SwR5FW7iLhOnbStJL3wEw0oav1T0b1okXDui5VkzgU7iIVkcC9SirqndXbueu1L4PH7Zuezjt39XSxosSicBeRiCqt0dfKB/tyZt2aLlWUmBTuIhIx//NJLo+991Xw+Nr0c3lqaCcXK0pcCncRqbRDh49ywVhno6+vHulPclJ1lyoShbuIVMrY2Wt59fPj1/rf3TuNv/S9wMWKBBTuInKK9h0sosP4DxxjuZMGUL2aWgd4QbnhboxJBhYBtYrnz7TWjguZUwt4BegC7AFutNb6I16tiHjC7/6+jE+/2R08nnJde27sqjdzeUk4K/dfgMuttQXGmCTgU2PMe9baz0vMuQ34yVp7vjFmKDAFuDEK9YqIi3bsPUD3x+Y7xtQ6wJvKDXcb6CxWUHyYVPwntNvYNcD44s9nAs8YY4x1qyuZiETcxZM+Yue+X4LH02/tSq8Lz3KxIjmZsPbcjTHVgRXA+cCz1tplIVOaAtsArLWHjTF7gYbAbkQkpm3csY8rn17sGNNq3fvCCndr7REg3RjTAJhljGlnrV1XYkppZ1BOWLUbY4YDwwFS1GxJxPNCWwe8e1dP2jU93aVqpCIq1DjMWpsPLAT6hzyUBzQHMMbUAE4Hfizl67OstRnW2ozGjRufUsEiEn1LvtntCPbTayfhnzxQwR5DwrlapjFQZK3NN8bUBvoQOGFa0tvA74HPgOuB+dpvF4lNoav1xfddRvMz65QxW7wqnG2ZJsDLxfvu1YAZ1tp3jTETgBxr7dvAi8D/GmO+IbBiHxq1ikUkKt5amcdfZqwOHnf1ncEbt//KxYqkMsK5WmYNcEJzCGvtQyU+PwgMiWxpIhIR2dknbVN89KjlvDHORl+rH+rH6XWSqrpSiSC9Q1UknpVzU+9n5n/NEx9sCk6/IaMZf72+owuFSqQp3EXiWWbm8WA/prCQgw+Oo9Va512Q1OgrvijcReJZKTfvvu/Ku5nRoV/w+N5+F/Bfl6dVZVVSBRTuIvGsxE2985PrkT7ydcfD304aQDU1+opLukG2SLRkZ4PPB9WqBT5mZ1d9DcU39fbd/64j2J9s9jP+yQMV7HFMK3eRaCjnRGZV2XDZVQy4a4ZjzN8+X/eBTQDGrfcaZWRk2JycHFeeWyTqfL7gdohDixbg91dNCSFvRpo8uD1DL1Lbj1hnjFlhrc0ob55W7iLRUMqJzJOOR9D8r3byn9OdCyc1+ko8CneRaChxIvOE8SgKXa2/etvF9ExrFNXnFG/SCdVE4YWTe4mk+ESmQ506gfEomL5k8wnB7p88UMGewLRyTwQeObmXUI69rid5238kWGtJfcDZOuDD/3cpaWfXj+jzSOzRCdVE4IGTexJ5D85ex/9+7vx71d56/NMJVTnOxZN7EnmHjxzl/Mz3HGM5Y/vQqF4tlyoSL1K4JwKXTu5J5F377BJWbcsPHjdtUJsloy93sSLxKoV7Ipg40bnnDlE9uSeRl194iPQJHzrG1OhLTkbhngiq6OSeREfoVTCtm5zGeyMvcakaiRUK90QxbJjCPMZ880MBfaZ+4hhToy8Jl8JdxINCV+v9257DtJu6uFSNxCKFu4iHLNq0i5tf+sIxpssb5VQo3EU8InS1rptoSGUo3EVc9vJSP+PeXu8Y02pdKku9ZST+ebivjm/0XEewT/tdZwW7RIRW7hLfPNpX54G31vDaF9scYwp1iST1lpH45rG+OqU1+nr3rp60a3p6ldcisUm9ZUTAU311+j+1iK++3+8Y02pdokXhLvHNA311fjl8hAvHvu8Y+2JMb846LbnKapDEoxOqEt+q+KYZoXyj554Q7P7JA08t2D18Yli8p9yVuzGmOfAKcA5wFMiy1j4dMqcXMAfYXDz0lrV2QmRLFTkFLvXV2V3wCxmPfuQYq1SjL4+eGBbvKveEqjGmCdDEWrvSGFMfWAFca63dUGJOL+Bea+2gcJ9YJ1QlXoW+GSm1UV0W3Nurkt/U56kTw+KeiJ1QtdbuAHYUf77fGLMRaApsOOkXiiSYlVt/YvBzSx1jmx8bgDERaPTloRPDEhsqdELVGOMDOgHLSnm4uzFmNbCdwCp+fSlzROJS6Gr9mvRzeXpop8g9gQdODEtsCTvcjTH1gDeBP1tr94U8vBJoYa0tMMYMAGYDJzTFMMYMB4YDpOiHUuLAGznbGDVzjWMsKpc36oYrUkFhvYnJGJMEvAv8y1o7NYz5fiDDWru7rDnac5dYF7pav61nKg8OahO9J8zO1g1XJHJ77iawYfgisLGsYDfGnAPstNZaY8xFBC6x3FPBmkViwrg563j5M+cWSZW8GUk3XJEKCGdbpgdwE7DWGLOqeGwMkAJgrZ0GXA/cYYw5DBwAhlq3+hqIRFHoan3qDR0Z3LmZS9WIlC2cq2U+BU56ut9a+wzwTKSKEvGaAU8vZsMO56kmtQ4QL1P7AZGTOHrUct4YZ6Ov2SN6kN68gUsViYRH4S5ShtAtGNBqXWKHwl0kxM+/HKbtuH85xpaN6c3ZavQlMUThLlKCVusSLxTuIsC2Hwu55K8LHGOVavQl4jKFuyQ8rdYlHincJWF9lruH37zwuWMsYo2+RFymcJeEFLpa/1XLhvzzj91cqkYk8hTuklBe+czPQ3OcDUu1BSPxSOEuCSN0tX7X5edzT78LXapGJLoU7hL3nvpoE0999LVjTKt1iXcKd4lroav1Z3/bmYEdmrhUjUjVUbhLXPrDyzl8tHGnY0yrdUkkCneJK0eOWlqGNPqaf8+vOa9xPZcqEnGHwl3iRqcJH/BTYZFjTKt1SVQKd4l5Bb8cpl1Io6/VD/Xj9DpJLlUk4j6Fu8Q0tQ4QKZ3CXWJS3k+F9JzibPT19cQrSapezaWKRLxF4S4xJ3S1fpHvTGbc3t2lakS8SeEuMWPFlh+57vnPHGPaghEpncJdYkLoav0PPVMZO6iNS9WIeJ82KMV92dng80G1aoGP2dnBh95amXdCsPsnD1Swi5RDK3dxV3Y2DB8OhYWB4y1bAseAb20Dx9S/Xt+BGzKaV3WFIjFJ4S7uysw8HuzFHut6A/8TEuzaWxepGIW7uGvrVseh7/53Hccz/tSdi1LPrMqKROKCwl3clZICW7bw2xsnstTX0fGQVusip07hLq46/OhEzl/n3IJZ/PIImj8x0aWKROJDuVfLGGOaG2MWGGM2GmPWG2NGljLHGGP+Zoz5xhizxhjTOTrlSjxJy5x3QrD7Xy8O9mHDXKpKJD6Es3I/DNxjrV1pjKkPrDDGfGit3VBizpVAWvGfi4Hniz+KnGDvgSI6PvyBY2zt+H7UT04CbcWIRES54W6t3QHsKP58vzFmI9AUKBnu1wCvWGst8LkxpoExpknx14oEhV6zXq9WDdY9fIVL1YjErwrtuRtjfEAnYFnIQ02BbSWO84rHFO4CwPd7D9LtsY8dY7mTBlC9mnGpIpH4Fna4G2PqAW8Cf7bW7gt9uJQvsaV8j+HAcICUlJQKlCmxLHS13uvCxky/9SKXqhFJDGGFuzEmiUCwZ1tr3yplSh5Q8q2DzYDtoZOstVlAFkBGRsYJ4S/xZf32vQz826eOMV3eKFI1yg13Y4wBXgQ2WmunljHtbeC/jDGvEziRulf77YktdLU+5br23NhV/1oTqSrhrNx7ADcBa40xq4rHxgApANbaacA8YADwDVAI3Br5UiUWfLxxJ7e9nOMY02pdpOqFc7XMp5S+p15yjgVGRKooiU2hq/XsP1xMj/MbuVSNSGLTO1Sl0v6xZDMPv7PBMabVuoi7FO5yyqy1pD4wzzH20V8u5fyz6rtUkYgco3CXUzJ29lpe/dzZ0VGrdRHvULhLhRw+cpTzM99zjOWM7UOjerVcqkhESqNwl7Bd9/xSVmz5KXjc/MzaLL7vchcrEpGyKNylXPsPFtF+vLPR11eP9Cc5qbpLFYlIeRTuclJpmfMoOnL8zcRXtjuH53/XxcWKRCQcCncpVd5PhfScssAx9u2kAVRToy+RmKBwlxOEvhnp7t5p/KXvBS5VIyKnQuEuQau35XPNs0scY7q8USQ2KdwFOHG1/tSN6VzbqalL1YhIZSncE9z763Zw+6srHWNarYvEPoV7Agtdrc/4U3cuSj3TpWpEJJIU7glo2ie5TH7vK8eYVusi8aWa2wXElexs8PmgWrXAx+xstytysNbiGz3XEewL7u2lYI8kj/8MSOLQyj1SsrNh+HAoLAwcb9kSOAYYNsy9uordM2M1b67Mc4wp1CPM4z8DklhM4D4bVS8jI8Pm5OSUPzFW+HyB/5lDtWgBfn9VVxN06PBRLhjrbPS16qG+NKhT06WK4phHfwYkvhhjVlhrM8qbp22ZSNm6tWLjVeDKpxc7gr3VOfXxTx7oTrAnwnaFB38GJHFpWyZSUlJKX7WlVP1NofcWFtFxgrPR178f7U+tGi41+kqU7QoP/QyIaOUeKRMnQp06zrE6dQLjVcg3eq4j2P+jU1P8kwe6F+wAmZnHg/2YwsLAeDzxyM+ACGjlHjnHVqCZmYF/hqekBP6nrqKV6Q/7D3LRxI8dY5sfG4AxHmj0lSjbFS7/DIiUpBOqcaD3fy8kd9fPweP7+l/Inb3Od7GiEDrRKBIx4Z5Q1co9hn3zQwF9pn7iGPPk5Y0TJzr33EHbFSJRpj33GOUbPdcR7G+etx//6yO8eTXKsGGQlRVYqRsT+JiVpe0KkSjSyj3GLPf/yJBpnwWPjYHN7fK9fzXKsGHeqUUkASjcY0hoo68F9/YitVHdwEq9rKtRFKgiCUnhHgPmrtnBiH8eb8vb6pz6vP/nS49PSJSrUUQkbOWGuzHmJWAQ8IO1tl0pj/cC5gCbi4festZOiGSRicpaS+oD8xxjOWP70KheLedEvXlGREKEc0J1OtC/nDmLrbXpxX8U7BHw98XfOoJ9YPsm+CcPPDHYQW+eEZETlLtyt9YuMsb4ol+KABQdOUpaprPR14YJV1Cn5kn+qvTmGREJEak99+7GmNXAduBea+36CH3fhDL+7fVMX+oPHt/ZqyX39W8V3hfrahQRKSES4b4SaGGtLTDGDABmA2mlTTTGDAeGA6RoPzho/8Ei2o93NvrKnTSA6tU80DpARGJSpd/EZK3dZ60tKP58HpBkjGlUxtwsa22GtTajcePGlX3quPD7l75wBPuk/2iPf/JABbuIVEqlV+7GmHOAndZaa4y5iMAvjD2VrizOfb/3IN0e82ijLxGJeeFcCvka0AtoZIzJA8YBSQDW2mnA9cAdxpjDwAFgqHWrG1mM6DllPnk/HQgev/j7DHq3PtvFikQk3oRztcxvynn8GeCZiFUUxzbt3E+/Jxc5xjzZ6EtEYp7eoVpFQlsHzBnRg47NG7hUjYjEO4V7lC3N3c1vX1gWPK5bszrrJ5T3njARkcpRuEdR6Gp90ajLSGlYp4zZIiKRo3CPgjmrvmPk66uCxx2bN2DOiB4uViQiiUbhHkGlNfr68sG+nFG3pksViUii0p2YImTOqu8cwT64U1P8kwcq2EXEFVq5V1R2tqNBV9GjE0lb57zq5d+P9qdWjeouFSgionCvmOxsx+3sss7uzKQSwf749R0YktHcrepERIIU7hWRmQmFhfyclEzbv8x0PPTtpAFUUz8YEfEI7blXxNatzGx3uSPY//HGOPx/vUrBLiKeopV7mPYdLKLDfe8Ej2sfOsjGJ68PHLRo4VJVIiKlU7iHIWtRLpPmfRU8Xvg/f8SXvyNwoNvZiYgHxda2THY2+HxQrVrgY3Z2VJ/uh/0H8Y2eGwz223qm4m+fj+/0mmBMYMWelaU7IImI58TOyj3kShW2bAkcQ1TCdeLcDbyweHPw+IsxvTnrtGSgjcJcRDzPuNV6PSMjw+bk5IT/BT5fINBDtWgBfn+kymLLnp/59eMLg8f392/FHb1aRuz7i4hUhjFmhbU2o7x5sbNy37q1YuOnYOTrXzJn1fbg8epx/Ti9dlLEvr+ISFWJnXBPSSl95R6BG22v376XgX/7NHj81+s7cIPejCQiMSx2wn3iROeeO1T6ShVrLUOzPmfZ5h8BqJ9cg+WZfUhOUusAEYltsRPux05ilujrwsSJp3xy8/Nv9zA06/Pg8Qs3Z9C3je5jKiLxIXbCHQJBXskrVQ4fOUrfJxexeffPAJx/Vj3eH3kJNarH1lWhIiInE1vhXknvr/ue219dETye8afuXJR6posViYhER0KE+8GiI3R+5EMKDx3f19+JAAAFF0lEQVQBoMf5DXn1tosxRv1gRCQ+xX24/9/yrdz/5trg8XsjL6F1k9NcrEhEJPriNtz3FhbRccIHwePBnZsy9YZ0FysSEak6cRnuzy74hsf/9e/g8eL7LqP5mXVcrEhEpGrFVbjv3HeQiyd9HDy+/dctGX1lKxcrEhFxR9yE+/i31zN9qT94vDyzD43r13KvIBERF5Ub7saYl4BBwA/W2nalPG6Ap4EBQCFwi7V2ZaQLLcvm3T9z2RMLg8djB7bmD5ecV1VPLyLiSeGs3KcDzwCvlPH4lUBa8Z+LgeeLP0aVtZb/+ueXzF27Izi2dnw/6ier0ZeISLnhbq1dZIzxnWTKNcArNtA7+HNjTANjTBNr7Y6TfE2lrM3by1XPHG/0NfWGjgzu3CxaTyciEnMisefeFNhW4jiveCwq4b7tx8JgsDesW5Mloy9Xoy8RkRCRCPfS3uZZ6h1AjDHDgeEAKafYqrderRr0OL8ht/VM5fJWavQlIlKaSIR7HlCy+XkzYHtpE621WUAWBO7EdCpPdkbdmmT/odupfKmISMKIRCvEt4GbTUA3YG8099tFRKR84VwK+RrQC2hkjMkDxgFJANbaacA8ApdBfkPgUshbo1WsiIiEJ5yrZX5TzuMWGBGxikREpNJ0hwoRkTikcBcRiUMKdxGROKRwFxGJQwp3EZE4ZAIXu7jwxMbsAraEMbURsDvK5cQivS5l02tTOr0uZYul16aFtbZxeZNcC/dwGWNyrLUZbtfhNXpdyqbXpnR6XcoWj6+NtmVEROKQwl1EJA7FQrhnuV2AR+l1KZtem9LpdSlb3L02nt9zFxGRiouFlbuIiFSQJ8PdGNPcGLPAGLPRGLPeGDPS7Zq8xBhT3RjzpTHmXbdr8ZLiWzzONMZ8Vfyz093tmrzCGPP/iv9fWmeMec0Yk+x2TW4xxrxkjPnBGLOuxNiZxpgPjTFfF388w80aI8GT4Q4cBu6x1rYGugEjjDFtXK7JS0YCG90uwoOeBt631rYCOqLXCABjTFPgbiDDWtsOqA4MdbcqV00H+oeMjQY+ttamAR8XH8c0T4a7tXaHtXZl8ef7CfxP2tTdqrzBGNMMGAj83e1avMQYcxpwKfAigLX2kLU2392qPKUGUNsYUwOoQxl3S0sE1tpFwI8hw9cALxd//jJwbZUWFQWeDPeSjDE+oBOwzN1KPOMp4D7gqNuFeMx5wC7gH8VbVn83xtR1uygvsNZ+BzwBbCVw4/q91toP3K3Kc84+dge54o9nuVxPpXk63I0x9YA3gT9ba/e5XY/bjDGDgB+stSvcrsWDagCdgeettZ2An4mDf1pHQvH+8TVAKnAuUNcY8zt3q5Jo82y4G2OSCAR7trX2Lbfr8YgewNXGGD/wOnC5MeZVd0vyjDwgz1p77F94MwmEvUAfYLO1dpe1tgh4C/iVyzV5zU5jTBOA4o8/uFxPpXky3I0xhsDe6UZr7VS36/EKa+0D1tpm1lofgRNi8621WoEB1trvgW3GmAuLh3oDG1wsyUu2At2MMXWK/9/qjU42h3ob+H3x578H5rhYS0SUew9Vl/QAbgLWGmNWFY+NsdbOc7Em8b67gGxjTE3gW3SzdgCstcuMMTOBlQSuRPuSOHxHZriMMa8BvYBGxpg8YBwwGZhhjLmNwC/DIe5VGBl6h6qISBzy5LaMiIhUjsJdRCQOKdxFROKQwl1EJA4p3EVE4pDCXUQkDincRUTikMJdRCQO/X/8F7Oc5j0LqAAAAABJRU5ErkJggg==\n", 262 | "text/plain": [ 263 | "
" 264 | ] 265 | }, 266 | "metadata": {}, 267 | "output_type": "display_data" 268 | } 269 | ], 270 | "source": [ 271 | "# Plot the graph\n", 272 | "model.eval()\n", 273 | "predicted = model(Variable(torch.from_numpy(x_train))).data.numpy()\n", 274 | "plt.plot(x_train, y_train, 'ro', label='Original data')\n", 275 | "plt.plot(x_train, predicted, label='Fitted line')\n", 276 | "plt.legend()\n", 277 | "plt.show()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": { 283 | "slideshow": { 284 | "slide_type": "notes" 285 | } 286 | }, 287 | "source": [ 288 | "We are done with our task and we can export the model parameters to a file so that we could eventually load it later when neeeded." 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 8, 294 | "metadata": { 295 | "slideshow": { 296 | "slide_type": "slide" 297 | } 298 | }, 299 | "outputs": [], 300 | "source": [ 301 | "# Save the Model\n", 302 | "torch.save(model.state_dict(), 'reg-model.pth')" 303 | ] 304 | } 305 | ], 306 | "metadata": { 307 | "celltoolbar": "Slideshow", 308 | "kernelspec": { 309 | "display_name": "Python 3", 310 | "language": "python", 311 | "name": "python3" 312 | }, 313 | "language_info": { 314 | "codemirror_mode": { 315 | "name": "ipython", 316 | "version": 3 317 | }, 318 | "file_extension": ".py", 319 | "mimetype": "text/x-python", 320 | "name": "python", 321 | "nbconvert_exporter": "python", 322 | "pygments_lexer": "ipython3", 323 | "version": "3.6.3" 324 | } 325 | }, 326 | "nbformat": 4, 327 | "nbformat_minor": 2 328 | } 329 | -------------------------------------------------------------------------------- /2-Intermediate/2.1-Convolutional-Neural-Networks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Conv Network and Fashion MNIST" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": { 14 | "slideshow": { 15 | "slide_type": "slide" 16 | } 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "import torch\n", 21 | "import torchvision\n", 22 | "from torch.utils import data\n", 23 | "\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "from torch import nn\n", 27 | "from torch import optim\n", 28 | "from torch.autograd import Variable\n", 29 | "from torch.nn import functional as F" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 3, 35 | "metadata": { 36 | "slideshow": { 37 | "slide_type": "slide" 38 | } 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "from torchvision.datasets.mnist import FashionMNIST\n", 43 | "from torchvision.models.resnet import resnet18\n", 44 | "from torchvision import transforms, utils\n", 45 | "\n", 46 | "from PIL import Image\n", 47 | "\n", 48 | "import matplotlib.pyplot as plt\n", 49 | "%matplotlib inline" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 5, 55 | "metadata": { 56 | "slideshow": { 57 | "slide_type": "slide" 58 | } 59 | }, 60 | "outputs": [], 61 | "source": [ 62 | "transfs_tr = transforms.Compose([\n", 63 | " \n", 64 | " transforms.ToTensor(),\n", 65 | " transforms.Normalize([0.1307], [0.3081])\n", 66 | "])\n", 67 | "\n", 68 | "transfs_val = transforms.Compose([\n", 69 | " transforms.ToTensor(),\n", 70 | " transforms.Normalize([0.1307], [0.3081])\n", 71 | "])\n", 72 | "\n", 73 | "dset_tr = FashionMNIST(root='../data/fmnist', train=True, download=True, transform=transfs_tr)\n", 74 | "dset_val = FashionMNIST(root='../data/fmnist', train=False, download=True, transform=transfs_val)\n", 75 | "dataloader_tr = data.DataLoader(dset_tr, batch_size=64)\n", 76 | "dataloader_val = data.DataLoader(dset_val, batch_size=64)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 6, 82 | "metadata": { 83 | "slideshow": { 84 | "slide_type": "slide" 85 | } 86 | }, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "" 92 | ] 93 | }, 94 | "execution_count": 6, 95 | "metadata": {}, 96 | "output_type": "execute_result" 97 | }, 98 | { 99 | "data": { 100 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAJCCAYAAADQsoPKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnWm8bFV5p5/qJJ3uGGNGcEAFmRRERpF5kNmBQUQRnKIS0SCihIDR5JeIQYg/FYIaRVRAZFBBBb3MgoLKdAVkRiYJaGucojGdTps+/UGfWm+ts2rfOudUnVPc+z5fTp2qVXtYe+1Ve/3fqTczM0OSJEmSJEnS5r8t9QEkSZIkSZJMM/mwlCRJkiRJ0kE+LCVJkiRJknSQD0tJkiRJkiQd5MNSkiRJkiRJB/mwlCRJkiRJ0kE+LCVJkiRJknQwsYelXq+3Z6/Xu7vX693b6/WOmdR+kiRJkiRJJklvEkkpe73ebwD3ALsBDwM3AC+bmZm5Y+w7S5IkSZIkmSC/OaHtbgncOzMzcz9Ar9c7B9gHaD4s9Xq9TCOeJEmSJMli88OZmZk/WVGjSZnhngT8c/j/4V+/16fX6/1Zr9e7sdfr3TihY0iSJEmSJOniO6M0mpSy1Gu8N6AezczMnAKcAqksJUmSJEkyvUxKWXoYeHL4fw3guxPaV5IkSZIkycSYlLJ0A7Bur9dbC3gEOBA4aEL7SgK93q9EvVEc95/xjGf0X5988skAfOYznwHgpptuAuA///M/+23+7//9vwA885nPBGC//fYD4L777uu3ec973gPAT3/60/mdwBKw2mqrAfDqV78agDPOOAOA//W//tectrPJJpsA8PSnPx2A8847Dyj9Ns2stdZaAOy4444A7LPPPv3PfvSjHwFw5plnAvDNb34TKOcJsP/++wOwyy67APDv//7vA98BOOWUUyZy7NPOE5/4xP7r7353utaMc5kvvE+e+9znAvC6172u/5n3+1133QXA//k//weA3//93++32WabbQC49tprAfirv/orAP73//7fYzm+ZDJ4DWSUa+E8AuX34eGHHx7a3vlniy22AMrv0DQxkYelmZmZX/Z6vcOAS4DfAD4+MzNz+yT2lSRJkiRJMkkmpSwxMzOzDFg2qe0nSZIkSZIsBhPJszTng0gH73kziky96aabAvDSl74UKCaT//qv/+q3+d3f/V0A/sf/+B8A/NEf/dEK933PPfcA8P/+3//rv7f++usD8P3vfx+ASy65BID3vve9/Ta33nrrCrc9aTxfgAMPPBCAI444AigmhB/+8If9Npoj/fvYxz4WgN/+7d/ut1ljjTUA+MIXvgDAN77xDWD6JOW99tqr//otb3kLUEwh//2//3cA/uM//qPfxnPV/Lr66qsD8OCDD/bb/PKXvwTge9/7HgD/+q//Cgz2z5Oe9KuA2CuuuAKAww8/fBynM3Y8vj/4gz/ov6cp8pBDDgEGz71Gs9uVV14JwP/8n/+z/9lDDz0EwB577AHAL37xizEd9ehEs0o9b/zxH/8xAG9+85v77+26665AuZaaWB0rUEyyjhWJJmjNMI4R++XHP/5xv81Xv/pVoLgF/OQnP5nDmSWT4L/9t1+5Nsd5XpzzXvOa1wBw5JFHAvB7v/d789qXv0nOJ0cffXT/s5NOOqnz+IYd4wgsn5mZ2WJFjbLcSZIkSZIkSQepLK1k+ESvkzLAs571LKA8gf/bv/0bMOhY6QrQJ/Pf/M1fWWgf97jH9du4CrZN19hRoXL1GFeh11xzDQAvf/nL53Bmk+OAAw4ASn+8/e1vBwYdc1VTXF274rUvAS677DIAzj77bKCoV5///OcnduxzYe211wbgb//2b/vvqQD+zu/8DtBeRbrKe/KTY4DrYBtfqyj5Hf9CUWdUmHQK/ou/+It5n9MkuOqqq4DSX1Cuu+P55z//OVCc+KGM59/4jd8AijoXgx0cYxtvvPEkDn0kWsqS53rhhRcCZVxAOQ/nCFf/KrBQ1CHHfKuNc8Cf/Mmv8v85x8S5wdeqVx/5yEcAOP/88+d+osmCGKYoGeABsN566wGzVceomPpb4Jzp/fCEJzyh38b5x+97n0ULgGPs8ssvB+Dggw8e+ZhXQCpLSZIkSZIkC2WVV5a67Pfa37fbbrv+exdddFHz+64mYXA1Pcp+W/ueLz51P/WpT+2/54q+Vo3icdbH4xN69DmItuHW/y1aPlWuKPbcc08A7rzzzhVuZ5K4QvnBD34AlBDp6FOj/4orKFdHy5cv77f5xCc+AcCaa64JwL/8y78AcPHFF0/q0OfEhz70IWDQH8kxUfusxbHhas/3VI9sG7cTfZRg0C/O77t/faCiCvqlL31pHmc2XlSLDGOG0gd/+Id/CBR1JN4D+tuo5KrOeL8BfOc7v0oWbPj9tPDpT38aKD5L0Y/ot37rt4ByD9cqNBQFqf4bVSNVardXzzlQ+tPv2Xbfffftt4lqbjJeun4P9cGM94Vj3Ovld+Lvoe/V6rX3FJR5wuvdSifhZ45RfUPj2KjPY8Tf1VSWkiRJkiRJFsrEUgc8WogrQ59u11lnHaAkXYtPudpiXR1ff/31QFtN8unWfcSn9rq9T+JxJT4XNt98c6AoSjGSy5Wt+9AeHH1y6qd+V49xVeyxeR4+6cdz0ZfDyJdWv7id1772tcDS+6y4UnXFYsTSW9/61n4boz5UFB544AGgqHbx+/ZZa+W8lJx22mlAiYCDon65QlRNbSXSNBLQPoioNnUlGPT7Kgz//M+/Kh85DWpS5P777wdgq6226r9X++C0rq0Rcttvvz0AjzzyCDAYDed9Ni2o8j7+8Y8H4Gc/+xkwqAh5D3vsj3nMY4B2FJL95N+oPvq9OuIpznnei86vfmfvvffutznrrLPmfqLJSLSUGJMPP+c5zwEGk0s6BvwtaPmz+trfhvp3Mb7nWPCeaflOOj/vvvvuwGB0r5afSVjMUllKkiRJkiTpIB+WkiRJkiRJOljlzXDREU0JUOdLk7FF2VEHViXp3XbbDYBTTz2130aThlJgy7SmQ60yY3R2mw8777zzwPFFR1v3UYc0x4Rf1qzyXDXRmUAOZpvolOpjeOdmm20GwJve9CagmAOjOc/jefGLXwwsvRmuNhW2EnJ6HtaL8/obBg/lOnvdpyF4IqLJWEdNKOaN6667DijXKZqLNDVqRtN0Fx3Fbe/3Nee0THa2PeaYYxZ0PpPCgIOWk6pmePtCZ+6IpkhNC3Hs2y/TgoELmuEcw9EMpynM+8S5JZpIatNKy9HXNn6vNr1AGS/ebx6HczGkGW4SdLmBmLbBaxITjxroUrttxLnPbddjo0X9mxnb1u4fmv6XLSuFQjQrO0+3gpnmSypLSZIkSZIkHazyypIrxMizn/1soISAx9WRT8eW8bCUyD/8wz/029x4441AKevhSnXLLbectY+vf/3rQFnt+7Q8V1RpfIJuKWY6W7qPj370o/02OsvpKP7xj38cgNe//vX9NrfddhtQwqfdhyH3AO9///sBeOMb3wiUJ/vo6KmKZokEE5tZPmWxqVc89lfsw1g9fRh1uGpUFKaJf/zHf+y/tqyFTpOqRjGpnNdLB02J5+f3fM/VX/yOjt06YU6byiKt4ITakVXFNSbo81x17Hb8RGfw+d7fk0JlzGNVYYrOt75WSVSFtpo8FOf2OgAmjiPVB5WpjTbaCIAXvvCF/Taqct5vqtaqW8lkaClKhuarHul8H9PS+JlqYUvBGSXFTE1Lna/nZcdWDCrZaaedADjnnHMGvjMOUllKkiRJkiTpYDqXvotAK2mV/kcm3XKlGFc1qiD+veGGGwC49957+21cDW2zzTYAvOhFLwIGw7H9nukJVLi+/OUvz+t8LJ9gOHZURepEga0ihyZO9Gn9Gc94BjDoT/S5z30OKCtBVYS4ulaZcoVRhwtDWYWoZmy99dbA0ilLXi/7yVVx7MPa76sroZ5/o5o2DbTs9yZc/fu///uBttGHzvaG87qSi/3jZ4bWt1aTvmdJjWlF1SiqzrW/jWPkjjvu6LdRdfI8W8WEpy2dhCvwq6++GigJWk0YCnDccccBcNdddw3djn5ojgP/xrnT+8E5Rt+jt73tbf02zouWF3IcPu1pT5vjmSULxXlZ9B+LY7hWbrr8Necy9lvbqX3cPJ44z/rb7bgep99oKktJkiRJkiQdrDLK0ihPtcceeywwWOAPBiODXGW76nRlHlPAu/q86aabAPj2t7898F2Aww47DIC11loLKD5Hc0W7vz4j7iOu7OtklDGRoriSVBmwD6LiYB+qkPl/vQKB4tdgtFir6Kqrc5P4nX766Ss428lQJ5HsSpo2SqJRP4vKyzTQ8idQRdH/xPEYI91UWOvrFvtHfwajmVrjUCVx2vFe0mcRiqriubci3aS+P+Lqu5XscynR19Jre+WVVwJl7oKiRNsHnlf0OXNOqaOjWsqAvmsbbrghMOj7pLLleHK7sSDvNND6PakjAOOcV/sxdkVnzaUYbCsx8LjUFBVkFZyW/0/9m9A6HvvD46qjIWG232jr3O27upROVMEdP5OIsE5lKUmSJEmSpIN8WEqSJEmSJOlglTHDjSJN/uQnPwGKCUoZMjpoKjPqFKwsH+s/KSFqotNMFU0SVrZfaEV6E0u6f+XrKJn6mceqRBpNhyZiNC2A56mjJRSp1e0og8aw+pe+9KVASXZnHyq9Q5Ff/X48jqWgroJdJ1GDtkkF2uNq2kwGo+C5mnAuyuCOf81xXrdoqqtTcLQke5O1TjsmtIvUqQNaDuy1qcVxFE11zjHTgilQdtllFwD2339/oKQSgWIeNx2I97I1NKHMh3Uf2F9Qxohj68wzzwQG00s4n9nW/jJIBkrgzI9//OM5nu346Po96ap432V+s3/f/va3A4MJb4cxbrOugUJQal1qbtWROt7rvleb5uP9XwdH2C/RDFc7dNduD1D6zvf8jYnHM47kk8NIZSlJkiRJkqSDVUZZGgUduWtlITqQGQ7sqkYn0K7U/263FT7/5Cc/eUHHbFJLFSBXezE9gOG7Opp7HNdee+2s46mrh0cn5doR2jZxle0q0TQA7ruV2FMn8M9//vNzOeWxU6sELQfNltpUUzsgqh5OG61q8SZSNElhbOP52NbVZOwf31NJdKUZS8e4DxlnKYJJ0FIIa7Ug/j/sHoqr42lLxHn88ccDRaHwnjSRLpRUIX/zN38z8N2oathXdZmKlqOvyqRzQ1TbLMmjuqfDeUzNspSKUotaSeoazwcddBAAm2yyCQAHHHBA/zPvHcuKnH322QC87GUvG7o9+/Iv//Iv+++9613vmtsJBKIKWjtmt9LA1CVIuoJjaifueF/U6pPE7aggeVz2czyeNdZYY7QTnQepLCVJkiRJknSwyihL9dOtT6OxCKzFY10V+yQbi0r6nonVtN/HcHyVJL+nH1FUe771rW8N7F+/HUuljMqHPvShgb/acdddd91+mze84Q0A7LjjjkBZmVm+BErIryuEUcLeW6sI+85+ueWWW4AS0jkt2E8wO9GkK59R0vTHlZCrMvvAlVhMmhb9fKYJy1V4znHM21ff+c53gLKii6qR6kCdWiP24bQqSMPoCt3uWh3Xn0X1KZb/mAZMNGvxcOchS9IAXHDBBUBRSk0BEecI5w39I1vzh9dfpV5lKhZmtZTGEUccMfC/ZSygpDWI6Q0Wi9Y1rdVG596oGum3qi+Y6RJikXZVR60Vz3ve81Z4PAceeCAAz3nOc0Y/iQ4shA7lmtbzYfQRUg3zd6zlQ1X7I3UlrKzTL7TGkcfhWIs+b/7W2h8WCB8HqSwlSZIkSZJ0sMooS/UTq8qS0VtQouAsDNvyzVAt0NfIp+wYMefTdV1ENq7EP/jBDwLFdj2uoquu8LX9Q/EncPVoX0T1oPYtaq2q6wiF2ocFSn/4nj5V00b0R/F1V4RL/VlLVRP7UP+2aVWTIq72W9fd9+oyLrGt486klFGxlTjeHg10KYte/66Vb12YGabPj82yRioE+gpFf8Ztt90WKIlr67k00hXxVPeZbWP0oSVQbr75ZgDuv/9+oJRxArj77rvndI4rok4C2bIkSGuOMBrYBL7+pkRfV5O/Oi/XShyUpJ/63ZgkOeL4cR/ve9/7gFKUHErJqeXLlzfOtptWokj7pSvyrk5OGX8P61JR9XzSwn6O23E+rX+romJte5XJLn+vuZLKUpIkSZIkSQf5sJQkSZIkSdLBKmOG08xVy6rRyblOttiq0aMMalsdu2PyNc0UdWhsdOYzhPQ973kPMCh7z4c6hDOep5KmjnBdtYvq7c21zlAtzes43mpTS/aLSdznuGq4taTjaaRlalPKti5aHD91IkX/j200J5h4UnOcDpePRlr1v7rqB9ZJ81qpEWK9uWngaU97GlCOVRNQNI1pTvI8nEda5951b9svBsBosnGsxH3p9O3xxMS3j3/844FiopsvrcSHMPs3IlIn74Qyl/tbcMcddwCD193gHl0xNHtGU53O9fa92z3qqKP6bfzerbfeCpS5JrpCRIfnudK6X2vH7miOG+aO0DK/jkJtzovuKfX8qlmulV4g9se4SGUpSZIkSZKkg0eNslQ7B7aSXvk02rVyrlm2bFn/tWG9dbXluDpy5e1x+ATbcn6rjycqGCb/8+l4oXiMreMwTNV91ckTW9vpUpbqlULcTlTYoJ2Er5USf7Hpck7tcjwcpU19fq0kkEtJ63hc+ZomIK54Y2AClHtAhQBKqoh6VR739ZSnPGXgs2lPJdBaEXs+XaqT1MEkMH3KkuejUu6xRnXC61zPY/EeqhWGVmX5+t5xfo3bMSGjWH4pKgymeFmostRywK85/PDDATj00EOBkvw3Wgm0TjieY4koqZW2ltXC+yqmmIHBIJn99ttv4LN3vOMdQCmVAiW1w8tf/nJgMKHnivirv/qr/mt/SzwvFR2vCZTrNRf1qIt67myVXfI3xjEaneSdt/bdd9+B4xqH9SKVpSRJkiRJkg6mXlmqV2fzXY3usMMOQLE1Gw6rigTF5uyKx9VMXHnUxVZbNmOfYuPqPG4XioplgcgLL7xwXudV01JtPMc6zUHsy7qUSVfobx0aHZUlV6HDCs9OC63rVZ9zPHbPucu/qe67VoqGaUgj0FK3XNW6So6h2l5Tj92Vc1SRTFhpG1fHhkzDaIVBp4H11lsPGLxu9lmd4qOlcNd/431mcdJpoVbKPM/op+bKvVaGWqv1YfcSzA4rty9jH+rzVitd8b6LSSznSky6uNtuuwGw/vrrA2VOULmCkgJD30tL9sTC4J6P77Xm/7ociOcVx5P9W5cM2nLLLfttLEfjcalwWcoKyv16yCGHAKU48SistdZa/dfO656ff73X477GqeDE7cU5xnO2z1opLPzMJLvj9IdNZSlJkiRJkqSDqVeWhikT0W7qSsAVocklVW2grB58Wnc1E8sP6Jvh03sdHQclGs4nXp+so13ZJ2DVLFcM0T/J72+11VbN85svrSfpurBnK+187YPT8rOSYQpT/F7LJt91jItNSzGbi5/WqNuG0cqmLDXbb789UPxA4urR+0AfAVf2MULJVbTj2nswoiLlPWTy12nz6TJRY/RLURWpffJafjvieUXl1T7YZpttgOlJ2tqVKDL6hERa/ki1WhSvba16t+YW+2pYeaph31sRhx12GDD4m+B51SpGvMaOa9s4t8dx6m+I6pN90Co07XZUaeK5eDy29Tii36f9UJcXitdoPsqbqm/0Q9Qfyffsn3he9TzfmveHjY1I7aPUundU7rwXaxUbSn8stEB9i+mfxZMkSZIkSZaQfFhKkiRJkiTpYOrNcFZrfuc73wmUBGbRBFDLusqh0bFSKVMpUTk0Ongrib/kJS8B4MYbbwQGZU1lwToEeKONNuq/tr1Oskq5USpVzrWq9mKg1KqEGyXgOqR1LmanKKsqkXbVzpoG5npcLYfV+v9hFbPHVfdvobQkcuXqDTbYAChmOFMIQDFPG4JsstXoDOo9V4c9R0x4Z7K9E088cdbxTAMmHmyZqbsCIGrzrdc/vm8ajze84Q3A0pvhhiWjjQ7edfX5VloA59raNN8yaddt43acIx1PrQSv80k4+MlPfhKAG264of+eQT4bbrghUObiON97H9TBPnHO8zfJvy0XhjpoqGWK8v7QrOdvVfwdsz9rF5HoTuJv1Je+9KVZ+xiGZviI5+o+PJ4YpKI7jPN+PUbie8P+7yI6eNf1K71OcX712Cbxu5PKUpIkSZIkSQfTseStiE+FJ510ElCcuH3KboXzi0/CrfB50VksKjvHH3/8QFtXfzp8Q3lyveKKK4CyEl933XX7bVyJ1w6DrfIAhmyPi66n9tpZPjqu+9mw8Oe47TrMODpEuqqxbe0Qu6JjXCziedWO78NS+Mc2rf+HpfyPYcatJJ2LRUvB2WOPPYBSosFVewxG8B4xbNoK53F7OkObbNUQ8JjQUrVChdN7JoY9TwMGXcQEr85JtbLUtYJ1PEQlxPtDxfzRgMdfK0ot523pCjSpy2fE76osOSY22WSTgbZx/3PB78TyVtddd91AG1WsqJius846QLEk+DsUr+mwhJwxwaaqkelpVM7ifeZ7/p7Vv2tQ5uy6D+K+VJnmMs+2khk7VmsFMFp1/Mzvt9Tr2tm/dd3rtt5X8br72+t7qlrxHpxkottUlpIkSZIkSTqYSmXpVa96Vf+1q1pt/fr6+BcG0whAUTPiil7/IVUiwyFdAQOcfvrpQEmVbqLIuNLQX2PzzTcHYOeddwbaxfxcqUQFR3wC9lj1HYnJAMdNbc+NT+HDCmDGp3bPy8/q5GAwezUUVyHTRFS8hvlpdfmjdFErVZMo6jguVIK+9a1vAaUvoq9I7TfS8sFy3PjXsRZDeFXV/Ou9PW3KkipC9NsZloix5fdXE9vYlxaD9f9W6aHFwHQQzmut1b5qT33/d6UFaSnTtQ9X7d8Y92/JDovLxv6Zjz+Kqo3nCSW9RX3f//jHP+6/vuqqq4Duslb1ebUURdv4W+D8E9v4m6bvkz45ca6qC8z6OxbL09jGPrTobhdf+cpXZr1XX+eWv5a/IV6flp+ex1onmO4qhdO6xn7m9uyXlk/XJEhlKUmSJEmSpIOpVJai2qPSYpSNK9aowPhE7lO7beMKwSR7tq1TykN5Qv3c5z4HlCfyGPmmiuUqyxVLXHH45Fz7LMWVmE/VHrMJNSepLHVFHQ1LyNgq5yBdEUGtZGl1m6UkqiN15N58j6+2l7dWztNAVEotR+IKV9+K2D/DrmU8X8dWrUJFpdGEjKq7rqCnBSOfLEli0kwo59VVaLpeedf3OMCll14KwAEHHAAUhXoxo+Li8dRqSMunzvmrVlXiudumpRpIHVFWq9ixjeUq6u3G9+ZDjBqLryNxnNf79/cjjvNhiUrj3Fn7i7aUSvte30D7MN6L9fG0ijV7XtHXdkU8//nPn/Wev1/+9X6Nv8+1/1HL2lD/HrbOve6zls9rbRlpnfskI2tTWUqSJEmSJOkgH5aSJEmSJEk6mEoznDIkFDlO85QOerF6t6YwwycNx4/ypbJp7VgXE5ApAboda0RFudbj0PnT7cbQTeVqzRT+H+VdHTwNHTVE1pQEk6CrRtkw09MoZrjYpjbDxVpD00TL6b52aJxvTbfagTU6lU4D0em6rttkv0TH01aFdBhMXOn1to1/H3jggX4bUwUo4xuAEQM0oul8sdl0002BMs6jmdH+6HLite9q81LcjjUq7R/nmMU0w8V7vQ7WiHOv1ObplqljWMqRVo2w2tQSzSjOx/fcc8/AccVjnrRZO6aZqVPORKf/lYk999xz1nvOXzpve21MqQNw5plnAmXs62ger7vmt9pM3RqHtTk/3l/OFzqjGyDi738LTf/RdDhfUllKkiRJkiTpYCqVpZtvvrn/WmfrP/3TPwWK05rJIKE4ftWO3q1Vn6skn5bjqsanW51Srbwdn5LrVXa9b5jt/N1yAne1qbPtOJ584zl00RV6O6ysR+t7rbajVBafBlpOrl6T+a5c63P3eq+99tr9NjfddNO8tj1OokLkMTvmVQKjY2Vdbdy/cczXIcQmnrRkEMAOO+wAFKdyjyMqVEupLL3gBS8Aikoc79dh5x7HSq1a67Abt6OibH/FMklLQa2UtZSlYU63UXmtU49IK/ijywlc9eD2228f2EdsO20BEysDtTIERRGvr6m/yQAnn3wyUEoYqT7FZLT+ZtfBH62SKN4rbif+PptE1ETVO+6446zt1I70e++9NwAf/ehH61OeM6ksJUmSJEmSdDCVylLkuOOOA4radOSRRwKD4c/6KKng6GPUVciwLl8As1dOrcRhvteVwFCVyNWnPhnxCdgVpskAtf0ulGEpAKAoBF1+RHURyPik3kplX+/L/XcpS9OQOsCyBZF6BR3Pc1hagbi6rstBqB5Ef7ZpIK76vC+8h575zGcCg2NehcS2nlf09/MzlVaTXcZint6ftq2LlC41KoCel/colOus8uVnL3zhC/ttvvjFLwLFz6WVMFBctVvEdamolSUTGUZUCx0jnk+rtMSw9AnxtX9bfinOmXX4fLwXp2W8rEw4DuI93eULJMccc8zA3xa1f3DrN8rX/kbNpSxUHGOODe9B789UlpIkSZIkSSbMVD6it1bry5YtG/j73Oc+t99G9UnveO3eLZt6nRitpQj5lOvqJqbaN2nfMP8dKHZX/UA8jssuu6zf5s477wQWNwpGat8amF0MsrUyHObP1BWpMq0+SzEZqWphXeKlVWy3Pp9WsdXar6W1Wl9KYjJIr7MFPr134updHyMVISOCYpTosMhB75f4PfvH71t2AuDuu++e8/mMC5WhnXbaCWiP/ToxZzw/UXGJRUDFceT4G6UUxbjp8v9pqWAqQP51zEeF0nPu8vur5xjVg1YJEvuntgjE95Lx8brXvQ6A/fffv/+eymjr92IueC3jnDsOjLRdbbXV+u+phqlmfe1rXxvb/lJZSpIkSZIk6SAflpIkSZIkSTqYSjPcKPVdvvzlL/dfb7XVVgOfPf3pTwcGzQ2aANZYYw2g1IqLUvl99903zyOeHrqcpw3htA5dq7aXf1v17Opke8Piy5ZiAAAgAElEQVSSFcY20+rgff311/df2x+///u/D8xORAeznba7zkFTgv21lKalFtHsoak4hu/DoNOt94jX2ftKh9+4TT/zb0ybUCcl9P/oVLqU6AR6yimnAIOmJJ3067mpNVfZVpNmNNV6rtavNAx6MYn3pNe2y3x23nnnAeWYve5xO7WJpmXKrp213aeJeWEw1URs0wo0ScaH5itdWaC4iHjdzz777JG3F69R7drRmjvr9+rfI5jtGH7JJZcAxYQI5f7SXeeEE04Y+ZhXRI66JEmSJEmSDqZSWVood91118DfyG233bbYhzM1qJyoAkRFyPIx9Sqgy5nSVV9cYVoORufAqCzIsBQEi4mKCsAZZ5wBwM477wyUvogKjOdYh0vHc/czHQ+vvPLKWfuaBiw7AuVYo5IEgytDr6UOmq44TUQHZSxZrqceR1DGn47ddT9NC6Y9MK1HJAZ7wKBzqVhiQWfweJ+58t1jjz2AonAvJtFJvb5OXqPIu9/97sU5sAatoJLWMSbjIQajOPc7ZrXKRJwjY7AHtC0SC6Weg00nFJVbg2o+8IEPjGWfkVSWkiRJkiRJOlgplaVVma6klJbauOOOO4DBpGOxvAWUlWYMja5XeT7hx5WDT/mu/qJvkCyloiRxpapictFFFw20iQVeTUKoH4p9YUmc+LoOkW2FoC8lb3zjG/uvvYZe73PPPRcYVARVPyzAqyJU+5dE9HOJfOYzn1nIYS8ahvPH67b99tsDpfCtqUtaockf/OAHgaI62adQfCmWklhSRn86FWFLSkS6UoVMmk996lMAPO1pT+u/981vfnPR9r+qEa/1UUcdBZTxYgqRSK20TpJ63Ok7F31M69JM4ySVpSRJkiRJkg5607DS7fV6S38QSZIkSZKsaiyfmZnZYkWNUllKkiRJkiTpIB+WkiRJkiRJOsiHpSRJkiRJkg7yYSlJkiRJkqSDTB2QrFKYRO3YY4/tv7fNNtsAJTnlhz70oQXt44ADDgBKGv6YkuDEE09c0LanifXXX7//es899wRKmHGdwBLgkUceGXnbXSkwkiRJFptUlpIkSZIkSTrI1AHJKsGHP/xhAHbYYQdgsEzJ97//fQA22GADoBRCNVEfwD333APAz372M6AkrFSVglIewMKTFi42BX/c5p/92Z8BcP/99y/wzCbPMJUnFrPecsstgVLa47d/+7dnbefUU08FYOONNwZKGZWvfvWr/TZHHnkkUBLNeZ3qQq1JkiRjIlMHJEmSJEmSLJRVXlmK6d3rAq+tvplP6n/Vh+i/ob+HisU0XIdIfZ6wsGM888wz+6/f9773AaVsgSrEJFLnW5bi6KOPBuBHP/oRUNQfKOdqgdE/+ZM/AYryAaWUyfLlywHYYotfLURiAdp//dd/BYpSZbmLWF7CMjA///nPAdhvv/0WcnqLwrDCx7fffnv/tf2pumbZgVj0VJXIfrY0Tiy1c/LJJwNw+OGHD7SNJQ2SJEnGSCpLSZIkSZIkCyUflpIkSZIkSTrI1AENusxNo5iidtppJwA22mgjANZdd10AjjvuuH4bTT+77747sDjVm7vCsevP/BvNccO+H80omlY8989+9rMArLfeev02Ojzvu+++Q49nXOy2224APPjgg0Ax+XmcUI5fx+5f/vKXwOC5a0LSCdzQ+F/84hf9NprWnvSkJwHw7//+70AxY0EJn9dste222wLt6vXTQm2G09T21Kc+td/GfrCf/uAP/gCAf/u3f+u30RxpBXlNdbGf3//+9w/sexLVw5MkSeZKKktJkiRJkiQdrDLK0jBVJP4/LDz5la98Zf/1tddeC8D2228PFEdUw8QBnvWsZwHw7W9/GyiOzEcccUS/zc033zyPs1gYLbWo/iyG1MOgKmJYuM62fhZVGkPzzz///IHP7rrrrn6bP//zPx/YR/z+uHniE58IlJD/lrLkOfuZikdURVSf7DvHSnQU1yFcRUmlKY4x9+V7jqNpU5ZagQ+i07wJPqGcq6qTRNVRRdF+djzdeuut/TZu8/GPfzxQHOuHOZknSZIsBqksJUmSJEmSdLDKKEtz4RnPeAZQVr76IEEJGTcp4emnnw7AV77ylX4blSTb+lfFAmCdddYB4N577x378a+ILh+hWl2L/9cKkKv8Jz/5yf33li1bBhSlQSXFZINQ/HYmVdIiKiEqP4b1+zeG/IvX278xmaTveQ1VTFrKm+/ZphX2bt9FX65pIl6TOG4Bnv3sZwNF9QH46U9/CpTz8fuqbAB//Md/PPCZY+QLX/hCv40+fKZocB8tNTRJkkcfloN6/etf33/vjjvuAOCKK64ABueEaSGVpSRJkiRJkg5WGWVpmHoREw+aPNLVrCrExz72sX6bt7zlLUDxUTLBogkI477009lss82AEpkFJZpqKZSlUfw/Vl99daAoaPG1SpltVFSgRDzZh4973OMAuPHGG8dy7KOw1lpr9V97riY39Jr+5Cc/6bfx+P/oj/4IKNFw0f9GZUNFyv+j2uZ2/KyV3DQqLVAi56aNqOTU945Ka3zf/rz88suBEvEW25js86abbgJg0003BQb9ms477zwAvvOd7wzs89FU7mTNNdcEYI011gDgmmuuWcKjSZLp4jnPeQ4w6O+pWv2mN70JgJNOOgkY9PMdRvSdfMc73gGU3+NDDz0UGI9fbCpLSZIkSZIkHeTDUpIkSZIkSQerjBlOR+PaNBKdeE0M+cxnPhMo5oboiLbnnnsCcMkllwxs/wc/+MGsfda1waLJ5TWveQ1QQsZvu+22OZ/TfKn7AmDttdcG4MQTTwRm1zAD2HDDDYHioO3/V111Vb+Nn2nCsk+jqW4uxzgf84th53H/nqvmpWjmcV+mCrBNlHc1zbkdZd14XprYbKNj9Pe+971+G82+j33sY4FSq04TFcC//Mu/zOFsJ0NMIeG5i2Ml9s/WW28NlPOxD03ZAGWcaJ46++yzAXj7298+a/+Tcv6fJDquHnvssQBcfPHFwKDJN9bTmysHH3xw/7VpSa6//vp5by9JJkWcP+o5fLvttgOKSwSU+dA54s1vfjMAn/zkJ/ttDPoQf6Pi74+uFLpdnHHGGcBgANZ8SWUpSZIkSZKkg1VGWWo528JgWLerWZPunXnmmUBxEpsrPuXqyBafjFUdTNBnW1fmk6Tl7HbfffcB8OpXv3rOxxGVEB2gTTT46U9/GhhM2lkrW/4fnYprNWMuGKIORdXR0dwkkJ/61Kf6bTy2JzzhCUC5JjrhQ7lejh9XSzGsXkdl+/f73/8+AFtttVW/jed85513AmVsPP3pT++3mQZlqaXo2XeqYFEl0fnfMieqKTHwQad/02bYB48G6qAIVWIdUaEoZvfffz9QSv6ccsop/TaWt6mJCreqs+PYVXJMkhrVysViFLXPJL2mT4mKu/eB4/tb3/oWUNToufK2t70NKOPwggsumNd2kvHTGiPOEQbgxETFWiJUog18ioFBls7SKmA6Gu83KHOM86olrMZBKktJkiRJkiQdrDLK0rDVUPTJ+epXvzrwV1zZQVEb6u21Qq1VKlxlR/+Niy66aKCNRUkXQ1nqwv27ko6252Hhl1deeWX/9Yte9CKgnPOOO+4IwAknnNBv05X4UhaitEX/H1fsO++8M1BW66Y/gHK9LVNjgsWoGtWlXVwJRZ8lVTVXUA899BAwmC7AsFnb/vM//zMAG2+8cb/N1VdfPYeznQyt++XlL3850B4b+uV5f9hPMS1AnZzzM5/5DADvfe97+++5WqxL8yy171KdFNNrHBVBizWrpnito7pmH3rPvOAFLwBgv/3267dxvnEcnHbaacDC/J3Ggde7pfruuuuuAJxzzjlAUY/ieXl/OQ7e8IY3AIPKgEqCKrzqo+kYAHbZZRcAnvKUpwDlXno0KEuOI/srnrvq/rSM+YXQSkvzspe9DCjza0zo62+A95Vz5t13391vs9deewHFV9JElnGe1oKgb6gJk8dx76SylCRJkiRJ0sEqoyyNQu1LUxcQje+NEqWlwlFHWcXtqHwsxEdnnNSrmpaapJriMRtxACUiyPPTPyWqc7XCsMEGGwDwwQ9+sP+efgyuxOfCqaee2n992WWXAcWXRp8K/UKgqAOqIq5UYlJKx0RdUNdoOyirISM7TLT2kpe8pN/mrW99K1B8XvSHi9tZSrqiEE2qqtqnjw6UlZz9U5dIgaIWipEucV+WOdhnn32AxVldO1bdV2ufdX/ok6eiBmUcq6rqtxPP++STTwbg4YcfBuCWW24BBtU1I2Nrv6Q4f3gPTqoIdZz7HPve75aD8l6HMp6NFjbSKR6f5+x2VM5iVJRKgMqvClXsf/0gVeXXXXfdeZ3jfJiL6mNiVoC/+Zu/AYr6qOJ+4YUX9tuY4Hg+Yz4WJ7dI+7QV5zZhpNc7JqV0nAxL/gtFqbd//F2Nvy2OE+ch/eSMTF0IqSwlSZIkSZJ0kA9LSZIkSZIkHaQZLjDM8TiajaJTK3TLsjqivepVrwLgi1/8Yv+zs846CyhSYqsy/VIwigRcO+/F89IsoaOdkqvpGKDI8eeff/7AdjSVARx00EFzOeyhGGaq47nEJKCGxHtcrWvqe5onWk7OOvBrdvGzaKpRip5WWtdfx1xDfnVKNcUCFBOmUrmJK+1TmH1/eW1iOH1M6TAJWnURu2okroijjjqq/9qK6ZoQDR7RiR9KOonDDjsMmFuyvHhtxmV+c1zXf1tmWE1s1uuKZnOdk9dbb72B77RqZmqyde6LJj/nQd/7xS9+AZRgACjXSzOw80Y0C8dxN1dqZ/547K37Q5OPSXr33ntvoJgJIyY81hk9znnOQ6MEeGy++eYAfOhDHxrYLhRT9lKY4Vpzp/OGyYI1L2tqgzKeR6m9aX/HZK/1dkxYabLccZDKUpIkSZIkSQcrpbI0ydBLV1y1wtRaiZkQyyrrMVz9Ix/5CFBW4F//+tfHfqxzYVifxVXWKP3qik7nPVdOUX3y+4ZYuxqIaesXknSv5Uhfh/7roAtlhetx2TaGvdflTiSOA7+vo3dc6daMMn6WgpbKsvvuuwOl73Tejkk7XfXp3K7qFK+jAQ9ux9Bvy4NEDJc3SeoodI1PP2udnyveV7ziFUAJUY5q6DCuu+66/utzzz134Pte06icODZ0jm4pS44N1VmDQKIj6xOf+ESgrK516p0rdb94rOuvv36/jeHbf/3Xfw3A6173OmCw3I1q4yjKoKv+PfbYAxhMm6FTtPekilVMB7L66qsDRaFyPHYpS7VyBsPVolF+Nxy7AMcddxxQ+k51VWduKOqyauMLX/hCoITRA+y7775ASTlhIEVUVwxGcf/XXHMNMJhaIapMi4Vj00CV2Ic6t+usbx/EObCepyWmZvG1Y8P5ObapE7hasmwcpLKUJEmSJEnSQW8aEl/1er2lP4gxsskmmwBlZWnyLBPQQbHNumo0jLoVcr2UtFZio2ApA0s+HHjggf3PVJ3+7u/+Dih9EQuq/tM//dM8j7hNrTpEfwLt/vqTeDxRhXAV5PdbhYHrsFVXybHcybDjWWrqlAHRH8mirZYSUHWIqRUct64aDSX//Oc/32+z5ZZbAkUZ0B8lhhDHAsdQfHuiMjmMLvWg/t+C0VBSPHjs+tlce+21/TZvfOMbV7h/r7vJ9wwPj6v+urzNl7/8ZaCkuICikKjSud2YXkS1wWvzgQ98YIXH1zXmvDYqwTEU3QSap59++sD5RBXJ9BgqtnV6kVFRFbG/W8lNVdUcf6prpmWAwWs3KnU6FyhJbFVyTJoY0xWolJkGwvk/JiF+8YtfDBQl6fLLL5+1/1oh83rF+0zlxn5V3Yv+P95fqpfeZ5Ng2JhSOYPin2UJE1WfqKbXKQOklcJCdda2sX9EX1nvbf3tLrnkktZpLJ+Zmdmi9cHAsayoQZIkSZIkyarMSumzNEmGJe07+uij+69dfXz4wx8Gii9ELN2xbNkyoKzSplVRiisGV4uee12SAsrKx1VVK7JEVJLs0xjxMm7qUg0tfyRXsa4sY4SiSottW1FD2su9lvfcc8/Q45k2ZakezzFqz0SBroqN7op+LfZnLPZaU/vF1CtFKKtEx5H+PyoyRpG26IpYqonlDw4++GCgXC/9ZPQhATj++OOB4o/SwvGjwqA6Ele++ibqx6gS88ADD/TbXH/99cCgugeDKo1zTFfR5VGSbVpyxHJLquDRl8rSHPoUbrfddkCZw2D2WG/NDaOUsHnTm94EFPXI+1aVDcpYcDw65mKx7mFEpaL2/7FP9RWD4hel6ugcGM/LPjOaTf8klWoo18nzaEXrqRI5flSsojJU+685bqKKpUqoKjaKstRSZeu5u1WMvr6GFjeO84cla5wjvKYxEa+fdfks1YkrnbPifVGPLedw/eKGKEsjkcpSkiRJkiRJB/mwlCRJkiRJ0kGa4eaI0p/ms7/9278FBuVCQ+L3339/oDhhtpwUx21+q5MnxvdEOXWuyfhaMizADTfc0H+tM6hhwS00SyjH6tRrqoXFQAdkmJ0QTWJodJ0E0D6I17SWiaUV0txlnlxMWkkaYTBkXwdaz0OThE78UBxedRA2bDo6y9ZjvU5SCMWkYoLH6Gi8IhxXcXs6etZj9qMf/Wj/tQ7Zmpne+c53AoNOwo5nv6dpMjrv69DvOLJ/4v2hSUQTm2M/phWxfzXDaPqLodZei2gKqxnl/nY8ahZ0HnPOgpLA1X1a8y7eQ3Vtw65UGPW1iHPFIYccApRaXo6r6MKg2dZr6766zHDWmNScCvDxj38cKCYyTVlxX85JjktTGETzp+Y7zWf2XUz1oAO+9733RTS1ei/GRJVxn/G1c41mvdgmBhSMylxM2BETcP7DP/wDUEzz9gXMHguO7zh32lea1DyOVvJYx6j3ezTj159phvMaLYRUlpIkSZIkSTp41CpLcZU1roR+brMO2YxPrlbcfs973gMUx0ZXmgBHHnkkMPsp3ZBSKKvQb3zjGyMfX1cZjnplMIkkh/VK9bzzzgMGEzz+6Z/+6UCblnLh078KgKvaSVJfi5gG31Wa19tjjqvlOuma/RvVKMeJ48jvxJIPruTdx2Imo2yNn/qaGvIb1TAdaT0fHWyjauRK0j7UYTjep27HfbbO3eSGJj6cCyZ4jUpFrbS60oyO6K7udej2s6iEnXLKKUBZoTqGo+J41113DWxPdcTwZYBHHnlk4Jh1Jo4lLiwvo7rmtYjqtYkiF1KqJR5brWYYfg4l+agqmKq4/d3CY40pOhw3no/KrduDEmKvmqaqGZ2U6+SoziNdaQouuuiiWW08d8uUtKgr21u6I153r7fHYds49utEo7aN94DHZt+pwMRz93jq6x7HqtfOkih1Oo5RsWyTDv7+fsUUOKZ68HdQFbWVKHKY8h4ZFkAFs1XZ1jxiP6haOR+Nw4KTylKSJEmSJEkHC1KWer3eg8DPgf8CfjkzM7NFr9f7Q+BcYE3gQeAlMzMzsyveLZDWk2f9xDpX+2tdykSlINrm3/rWtwIlmZyp6U0A1kU8nnofo9AVjtuFIbKvec1rgKKKtcKOayUoKieuMN/1rncBRTHRN6tFa+Xre/aBodqRcYfW18cRV8Wu6FztuSqJylKdZK81/uwrr6mrmhhir7/HUqQMGGWf+uvEUg36YKieep7RN8JwcleY9k8sN+B7jqPosyC135iMMh5cdavwxOPxHtDPxRInAJ/4xCeAcp6qByeddFK/jck1vX51igMo/jWu5E3IGgt+uir2+/ZBTHKoAmcouqpd9ENUjdE/soUFivfbbz+ghLLHEhsqbbUqEtNmqCDahx5HLBBs0r/6Pol+f96DfqZSEK+XbRxrG2ywATA4VnytUuE89LGPfazfpu4X255zzjn99+LrUXHfcT7xmtbKSWssew/YNipU454TYrqFYXh/WpIkWkjsV9VQtxeVLhVRj91x3Sr67DhqqT11agZ/G+I4rH+bHMfxvvD7tnGfc7HgDGMcytLOMzMzm4QMmMcAV8zMzKwLXPHr/5MkSZIkSR6VTMJnaR9gp1+/Ph24Cjh6WONxMp8n85ZKU6sGRrxBibjQr+ClL33pyPuKqxETho1iS3U10yoG6ipNtUdfD0tTRFwx77PPPsCg4lEfo/uKxVJddaiiPe95z5v1/Xpl2vJZ0j/C9ywGGRmXslTv35VH9COqiz+2bOquHr1e9ntc1biCqov2dvXzUtAqIaD/hsnbYmSin6mQmAjP8gVQVp2bbbYZUFaKX/va1/ptjByrVZnYFyo/NaOMA8dcVIL1MdJfxoSB0UdHhVVfGovSxmKpKkq1WuR5QpkbVH1UFlrJTf3rtYhKiGO0VsGiH4hjtEuZNvGm52W/6IsCRW2y4LHHHEvQ2N5jVi2K48gEsyqSKlWtskB1AkT7NJ5zHcXYUiHr8zvjjDNmtREj3eJ5+br2k4l+TcN8aOJ4tF+cK1sFlKWV/LH+rC4qGyPm6sTAEvfl/eWYimNU9Guy1JNzVbQ2+Np92YfxN8v7q0582YrGrue82Ad1glr7Nx6756UfXCu5pePOPnN7X/3qV1koC1WWZoBLe73e8l6v92e/fm/1mZmZ7wH8+u9qQ7+dJEmSJEky5SxUWdp2Zmbmu71ebzXgsl6vd9cKv/Frfv1w9WcrbJgkSZIkSbKELOhhaWZm5ru//vuDXq/3OWBL4Pu9Xu8JMzMz3+v1ek8Amh6IMzMzpwCnAPR6vTnbWlrmM5O4KTHGsFUTzjWOY+g+/u7v/g4YlGU1v+k02aKWnv1+DCXVDDcKdULEFptuuilQzj2eVy31m8AsVoa+8MILB7bX6pezzz4bKCHRLcfsaHIYhmYFJVLrZU2CWu5Weo+J5+wPJV9NSrHfh9Usitv3+tp3tllnnXVmHVdt7pyEw3ct53clIz3hhBOA2SZJKOYFzR06dkf521B262Q5Dk0hACW5oWZJjyP2c3SGniuaCVpmPfvZY47yvmPBY9WpOF5bzbY6gXutoxOv/azjqY7LMSGe5i6dpT2euB2PWZOhn8W6dH6/qxafx3HuuecObVOfhya2mBbC8/Le8Zhjzbt6DvY7MVGk46hOHeJ34me1U3A0O3mdnT9Mx2F/dRFrqMXXqxp13UL7IibSrBNDes9Ep32vi+/5nfibaRt/G/wsjnnHoeY8jyumMPF3Q9OxYyte93oOaNXXmy/zNsP1er3H9Hq9x/oa2B24DbgAeNWvm70K+MJCDzJJkiRJkmSpWIiytDrwuV+vvn4TOGtmZubiXq93A/DpXq/3WuAhYMUx9fOgtRJ3Ragjclw5uIoZJVRfR8htttkGGHwCNpx3lGOrV/DxmKPz6IrYYYcdBr7z2c9+tv+Zq7WY2A0GHWV98vbJ3qfsE088sd+mVpbkC18oz7omb9NBfL6YmK3rWoyrHEi9HcdGDKv1utRh3fF6+V7tzBnbuCpzBeYKKjpE1mHP40pO6XlGR1j32+VMftRRRwElBYbV5h37UM7DMeUqMCoLqrjRcR4Gk0vq4G1yuzpcGNrpLEbF841jXwXGz2qHbyjqTO1gG6+t/er3VYaj861jyu/7nZZjdr0Cjiq4Y9T7o6X26YS+0EABx51zgn+7UhIkj368p1Xu/G2Ic5W/e4471aOWY7ZjtE7aC7Od5N1uK/2CSpL3Z7QI/fVf/zVQ0lR4HFGZdl8eawxmWCjzfliamZm5H9i48f6PgF0WclBJkiRJkiTTwlSWOxkl6WKrzbh8XyxtsN566wGD6d1HwSfoYQVsoax4R8HSKB/5yEcAOPbYY/ufuSpWWfL/+LStIqViViffhFII8dRTTwWKD8vOO+/cb3PZZZcBg/4+88FVdJfPwKSSNtrvqlswO5zc1VVUBHyteqSyFFdi2sd9zzZRmXS/huaPS0Gzv1qpKLzOjoPDDjus/5lJVr139AuI95LpAFyFuo+usikW2IyK5V577TXQpvbbgtmpA+bj03X++ef3X6vymPTR84sKmPeXqorXK/pdeFyOA9MnRH8J7z2/Z/qO6Ic1TAmK96LH5j3sqjuu1oelWEiSUfA3RH/N5z73ucBgstZa/Wz9tniv1OkSWilVvIcd61H1cR/ve9/7gEGrR80rX/lKoPj/xX2pbNXlTsZBljtJkiRJkiTpYCqVpVFWka02rkKXLVsGDCanO/744wE466yzhm7TlO/aRC13EAvFLoS4eqwLV3Zx2mmnAXDIIYcAg4Uf3Y5P9q5mY8SCiodqhquBuKLXd8W/+o7E6LaYnBPaCSdHweOJZRdqJpW0UZ+T6GviykfVR+UsKkuOt9ofJUYj+X19cbze8bq7cotJH8eJhUihlPGoC/vGe0eFwoKYy5cvBwYTBZp80vfsl6hieV5GibZ84FoJCuvjidFTsHCfLseR0Xr+nTbi+bliTpJJc/jhhwNlPjviiCP6n6ngaLVoJaB0rnMu8Pci3uv+3vh7rC/e3//93/fbHHfccSMfsxHp+vbF3wrnOOdX56VxFNRNZSlJkiRJkqSDfFhKkiRJkiTpYCrNcLFSubKZcp8OlVEK1KlLh1r/xsryOrJefvnlQAmN3X333fttlCQNnz7mmIXVAK5NhS1HtLlg3SVDsKFU/lZGNRlg3JemIk1PLROm/VoflwnAYLY5chRzqfuM5jzNVXHbMOgIHWvSLYTagdr6eFGOtY2my/vvv3/g2COGtOq0G7dj6LhSsH0Zj6GuBD4uB2+d5t/znvf039PR2Hun5RSsic5z3XrrrQG49tpr+210gHY7OiBHU69O1Z///OeHHmMtgWt6iuOoNs2Oq3+SJJlNXccyzh/xNRQncAM+oJjvTegaE4yKJrEPfOADQHGJGeW4Wu4Y/i5rzovzir91ziO6FYyDVJaSJEmSJEk6mEplyXIK8bUhjq7sY/iiqohPoaotn/rUp/ptvvWtbwGw6667AiXpXnRktUL6kUceCZQn1qgwzEcRkqiuXHLJJXP+vk/kB7Bo9vQAACAASURBVB10UP89k3i5AvfJOqoIdXJCVaiYwNAneZUGw8QPPvjgWccxF8fuljKgglQnvmtV6R43qhnxOqoE2U+OrRh2qoqik6Oh463Q1NopOfZzl9q4EAzVj6U1amd/99UqM1Cnu3j2s5/db2M5iRtuuAGAzTffHBi8T/fff/+B4/Geif1clxxo9V2tNiZJMjnmEkjz5S9/eeDvJOk6rtNPP33i+2+RylKSJEmSJEkHU6ksGSrfRUxopbriqrpWW6DYVFWUVKhMMwAlrYDKlCxETYpEZektb3kLMJhgckXoMxTPyzQH73znO4GiCMRyDvPh6quvBuDKK69c0HZaKwSvQR0mPqlElBHVoxiqrfKj0lWn8I9t/J4+S7HwpKpel/JW+2KNK0XCGWecAcABB5TqQs94xjOA4idVpz+Acj4eh34AMd2Bvn+qu/olxISlNTGho9QFof0/hhnXhWE9jtb2kiRJFotUlpIkSZIkSTqYSmVpFGLJjYWW31gsjGYD+OAHPziWbV588cUDf8VSLVB8TEzmZXKw6N+i6vDII48AcOihh87al4rWXNSQlipnaZU6QeBCEoaNiv0SozZUOHzPRJ/Rp8bCqSp266yzDjBYNmPTTTcFSqkQFZ2oBE7qHFUt9cmDorC+6lWvAkrZnhjNMpdyAPo6Pf/5zwcGi1yOwre//e2B/+3v++67r//e7bffPtBmoQWGkyRJxkEqS0mSJEmSJB3kw1KSJEmSJEkHvcVwql3hQfR6S38QyUpJHRpvWgjNalAcuzUZWhcvmoCsj2Tyx29+85vAYFoJgwi8p3SW3mSTTfpt/uIv/gIo6S7q41tsNEuaeFITpMcHJUlnbUbrouu8TDprv8d9me4gSZJkkVg+MzOzxYoapbKUJEmSJEnSQSpLSZIkSZKsqqSylCRJkiRJslDyYSlJkiRJkqSDfFhKkiRJkiTp4FGblHKuWAbkFa94BVASWcbyCpZUsJSK/lyx/MnGG28MlGSEloB47nOfO7FjXww8Zwvwjqu8hAkZY2LGcZX4qGmVGRlWeiQmY3zKU54CwIYbbgjAddddB8w9MstouA022AAoiUK7/ALnUpR4Egzbv4WUofSLfy1KHROOGiVoIdxbbrll1r4cA1PiJ9l/Pex4YjkXowU/9rGPLWi/b3zjG4HSh9dcc82CtrcYWNInlmsaJ7HczbSWtWkVBJd6/Jj01ySwUKJMLZtkMVqLt7f25XZHGavJ5EllKUmSJEmSpIN8WEqSJEmSJOlglTHDmQjvmc98JlDMDmuttVa/jbW8TFhosjxNUwA//elPgWLGW3PNNSd30GNGOXePPfYABivUa3LQvGgdsA9/+MP9NtYU03RjVfu77rqr3+aQQw4BiplB2TjKx5Myx7g9jw9mm5c+8pGPAIPJJDUnrb766gAcfvjhs45Ps91NN90EFNOEdeWgmKl+/vOfA7DnnnsCg3XoLrjgAgDOO++8gePrOuZJUu9r/fXXB8q9ACVxpbUFvR9+8pOf9Nt4jo4br/HNN9/cbzPtJgRNJV6beN00D9kHJt1smXytuej5Pv7xj++3cYz5vf/4j/8AYMsttxznqYwVzW/eA56PdSRhtpnK+8PxEN/T5O/8+p3vfGcShz1RWmPZmomvf/3rgcFkqya6tZ/+/M//HCj3G8DHP/7xoduWaTJlr2qkspQkSZIkSdLBKqMs/c7v/A5QSje4unn44Yf7bVwZ3n333UBRH+KqSWXJVYOrragwPfjgg+M+/DmjszHAueeeC8BjHvMYAB73uMcBg6tilbJf/OIXA99XkYPi5OqqZvny5cCgCnHWWWcBZUV56qmnAnD88cf329SOi+NaJbm9ljLz7ne/GyjqwXe/+93+Z15DHfntH52WAc455xwA/umf/gmAb3zjG0BxaI7b/OEPfwgUx1XLngC85CUvAYpT+fvf//6BY18q1l57bQDWWGMNYHC1bz94P3jOcZxb1sRxpCqzxRYl19uNN944iUOfF60x57VwtX/vvff2P1NZ8nycN+LYv+iiiwDYaqutgKLIxDY/+9nPgDJG1113XQBe/epX99ucdtpp8zmliaPKrDIdFXfHryqtY18FDkqf2x9+xzE3zdSqdRw/++23HwCvfOUrgeLEHUv9OPd6f9x3333AYGCQ86nBEUsd/JEMkspSkiRJkiRJB6uMsqTfhaH+hkb7xA9FfdK+7Koohpn/3u/9HlCe+l1F7LDDDv0206AsfeITn+i/Vk3Rx0T1KK5YfK1y9sADDwCDBWevuOIKoKwoVWBi+oVaLdprr72AwTDa7bbbbqDNuGitxFTD9FV76KGHgEGfJY/D7+mLEduoBOnnpVrkWIHiq+S4cbvRr0n1yeNpFZxdiuK6KkGmS4hpAVRRTLvhSvpLX/pSv41j48477wSK+hQV10mHoM8Xx42KkqpaHPu1Wuh84r0EsOOOOw5sx+veSpvhdr73ve8B8La3va3fZlqVJces1zbev94rqpDOq95vUFQ1r/+0jYMuulRrfZW0VtgHWjGgKO2OEecYC3QDvOlNbwJKeon//M//BCbjzzgpVb8r3UHt59flx9o6vqX210plKUmSJEmSpINVRllylai9vPbfgeKH5BOwT/QqTlCeal1J2Vb1ZqkxGs2oNijqh6vZ1urEp3aVspYKYJ/V5x4VEF8b5eO+VeQA9t9/f6BEHY2LVkK7XXbZBSjn7Dl4fDCYFA/KGHHVD2X8vPCFLwRKVFz0R7HP3JfKQlwZ2s+qldtvvz0AV1111aw2kyIej8qbSusmm2wCDCZidRWsX5PnFZU3V8jbbLMNUJQ4vwNFoTr77LMH/l9qTjjhBKBcE8/PlT2Uce394LWO49o+9PrZz1F9cqypNDhmozo7qftjoRgF5zwZj1m1Sf8jlek4f9if/nWsPBroUjP0cfVa1pGuUOYb73PHWvzdULnzN8m5cxLKUn0+G220ETCogjueR/E1rLfX6q8upXyU7y91BGAqS0mSJEmSJB3kw1KSJEmSJEkHq4wZTmlT04qSoIkEoUii0UQDgzKo6OCr5G6CxqXm0EMPBQZNObWZqeV4Kq3EjqLMrPlNOT2av+w7zXluJ7Z5+ctfDiyOmcE6bZ6rZrhoYqkdBx0bngMUh2dNKpqgoiO07euEg9HUq/nFfSm9RzPcpOtjaXqDYi5zPBsub/JFgOuvvx6Y7bStCTG2MbmiZjxrYEHp12233RaAe+65BygmzcUkju+tt94aKNdJZ+54bcWx4jgwyWTE+0Nn/nifab5z/5o6fB+Kg++0meE0HTk+oxnW+8kx4rWOqQPq+8J7IKY5eTQmqHTuNxmp91D8bTFxr32ma0dM2mmAiKbdlhluXE7O7t9UJnvvvTdQkglDuV4GL+msH5O1eqzf/va3gRJAFQNfxO95X0WTouNEdwcDjeL4qe9H+yDO0762n92eAU8xZcxcSWUpSZIkSZKkg5VaWYorHx1xb7vtNqCoK/4P5clXJ0VXfTH5mitwV5+qUTGB4TQQn8jrp+1aFYPZjoOt8E5fu7L0/6hcuUJ29aiDZ9yXfaVT8EKe9leEDsYec+3AHo/VMeGqOB6z/VmrctFp0X3UK+/Yt+7X77sSW0ziylAlwPe8ppdeemm/jSHfOrdfcsklwOAYM3VArSgYTg1FjfEaOA5clcKg0/AkidfEVBZnnHEGAM9+9rOBsrqFsrr3vLzGsaSFc4HqtW1igIj7VVHwO1deeWW/jav9acFjdayqMMSx7xzg9fO+jwqu86nYdzF8fhqUpXjfD0sZ8NrXvrb/2kCF22+/HShjJc5r/v7YT5aXuuOOO/pt7M99990XgPe+973AYB+2kmLOB+9lAzre8Y53AINqseWanB8tXRRLhDkfmojVpJtRcXUOcEyoOsVSL44FP1PZjgEC3o8qTCpecY7xGFXyHHMmf01lKUmSJEmSZEKs1MqSNmQoKx4VIW2ZUX3Sn6Ve/X3ta1/rt/GzOkR+qctVfOxjHwPKKjauZlXKtImrJkQbsP4ItaLUFe7Z8ksQVxyGG8cEfx6HCdoMJR8X0Ybtdde27irtSU96Ur+N/jVey1pFiMTxAoMJS4eF9cbvOCbdZ/QfmjSqWvGYVT9UG20TFS/vA1f99q9+SlBWbPqIud3ob+FrV9etchexKPNiY7mKz372swDsuuuu/c9csdY+Z/H8XB37mfNJVDHtF8ejZXje9773jfNUxopj1vvd5La1UgSDKggM+n+qrjg3mAA1+vRNA1G1qRMpWp7EouJQ1A+vrW2d56Cco2PkwgsvHPgfypzk/XDSSScB8OY3v7nfxm0v1HfJ+7Uu46OqCsWi4l+VnK985Sv9NqqCJqy9+OKLgcFktB6zJaNUnWJCaNUh7xX9wCwrBUW1MiGsqmxMd6AK7vylUhaTNM+XVJaSJEmSJEk6WKmVpZjwyxWPT7murqO64irCCCVt0UYMQSll4mrCJ9n4dLsUfOADHwBg9913BwbPS5XJc/aJPkZduWrsSg5m37ma9pxj0j1XkioV9mWM+nA7PvWPW1mK/mOeu/t0NRxVR5PKeV4tZcnv16VMWoqibb0GcRVa++1E/6FJo7oXj9nr5fVxpRvVMFd7HuvrXve6gbZQFES37f0W/dkcb14f20T/hqVQluoyOS9+8YuBwdXxDTfcAJQCqN5LUVmqo3NaEaWumFWWYpHlacV51DkvKrdSR9y2fGscG44574WoOk8btbJu5GT0Z6vVav1go0+O7zn/OHdq6YDiC6TPm2MkzlXec/bvfEsieWyqWP7GRR9e/T29D/Qjiv513sveF17L+JtQ+6E5L8bEt94XHk+0jIgWEf2t/D8e8zrrrAMUpcw+jOrufEllKUmSJEmSpIN8WEqSJEmSJOlgpTbDRdNPlAWhSMmxtpeSqNKxUmt0UjSBmnKopoXoNLsUmNjvyU9+MgCf+cxn+p/pPKdUquSq+QmKCaJLatc0Yxul99i3yp72j6ZMnfOgOC5q2hg30ezleWgCUN5tmYfqpJLRYXuYI2XLTFmnFYjmTh09dW61X6LJR1PvuPF+iJK049+xXpstocjmmoxMYBcdPT1mz8/+jaZM96t0r9O0JrzFpJU2o651GE0kw+q+tep2uR3/tlJQPBrMb1Kbmp0HWslT/aw1N9SmI9ss9dxZE8+3vu8d37EmoNQBNNF9w/vK7Wmuiq4ijh+3bb8Y3g8lyetCzXDOzzpCOx/FWo7uwzaeV2xjmgPrx2lGi8ktdYrXzKjZLDqTf/3rXwdK0I/zUZzLPVePyzkzBqM4x/h927Z+z+ZKKktJkiRJkiQdrNTKUlwVxORWUJ44Y8JJw57FENmYKM8Eeq4aXEnFMNFp4IADDpj13plnngnAaqutBgw68dYlCForl9qp2ZVhXGHaz64w9tprr4WeypyJDsMeq+qIK7x43V111OHucYVZO6y2HOLrhJVuNzrbmyrAUh+2javHSSlLrorjeFZZ8j2dVesUCVBWbSagjA6adTJSVawYSl4nRfWzqACPq5zDihilqnmr3EmthsQ2w1JOtFTMmi41Y6mp04o49uM48tq6ole1juqBwTCeX729aSGqhR6b6oXzR7xHdez2Pc85XkfV1FtvvRUoFoCouNcBRqo9G220Ub+NytJCx4hBPg888AAA11xzDVASUUK5pw268PrF+fXEE08ESpJN+2mXXXbpt3Hb/rW/li1b1m+j87iO3qYZMBUBFCVJ1cpEmNEBXkz26bGrii2EVJaSJEmSJEk6WKmVpVHKecQVdJ1kTR+fjTfeuP+eioC2eFfr07Y6alHbfKPaZj94Xq2w57oMSEuBcXVUhxJHapVm3CvpaFNX3VHx0H/L6whlLAwLf47H2PJDqbdjG1feUWmoUw+4jxhmPG5cDdeJOaH4EXjdVQZaZW5UGFRRWwkn6+vfKvWiv4aftZLTRX+hxaIehzEdiH1Wq5CR2j+uvt/g0eWrJM6LdcLJOOfVKqrnHK+tY6suK9RK/rqUtO5t/fQsxxH9PevyRqoaKvgwO2WJ46CVtNM+U8lp+fQttNi26pC+Syrb0RfL6+R7fif+HqoyezzOY0ceeWS/jb8zFlDX6hATRer/qEJl/0b/SlN6mMJEK0/8DVe18ntei+ibPF9SWUqSJEmSJOlgpVaW4krRJ1+fcv0bE6LVqz6fXLfZZpv+e64EtIGa7n3aVkctWhEcohLgas+VYlxN1spLawXm9+vow8iwsiDjIhbm1B+mjvaKviaeV60stNQ1z7lrZee2VWL0fYPZCQtdRU6yEHPtZxVX+46Jln+O1CqRq7baDxBmR41FdcYyBa7+7Is4Vly9LqayNIqflCtdleQ6uSjMjoJrKZbRV+7RgveQ17Lla+Q5qzq2CpbWUW9+fxwJA8dJaxzoi+NY2Wmnnfqf2R8mX1T5iIqQ/aI/UkvBVXVyjDjG4v2qitJ1v47C8uXLgRLNdu+99wIlISaUyDTP3Sjm6LP0l3/5lwPHc9RRRwGDPkKWa1E1dhyZ4BPgggsuAODkk08GSv/GPrzllluAci++4AUvAAajDo20875UBYtlU+ZLKktJkiRJkiQd5MNSkiRJkiRJByu1GS5SO+hqTosyeqz3A8U5LFLX19Lhb9rCfVso89ZJJWF23beWWbE2L7QcNJVYFyoTL4QYSlonI9N81nK0rRMNdpnh6gSWUMaU5gbHRAyxro9LM1g0HY4b9+U5xz7xOhnCrFQex7PjxP7QPBDNcJ6z17/l4K85QROb5oaYOmCxTDJzDdXXvFinl4jj3DaaXFoJKL1nTEaoifbRkDqgJppY65qAzqWjOLS30lQsJa1gHc1n3/zmN4HB66552XM3EW+8jpqQ6rkgpt/Q8dn7oZWexNpnt99++9xOqsJ72NQubi/W6XQucD71WA866KB+G+cvTWHXXXcdUIKjAD75yU8C8KIXvQgo9459CSWlimPB+yPOrx6PCZg9rpjY86KLLgLg1a9+NVDmk9ZcPldSWUqSJEmSJOlgpVaW4mrN1VGdLC8+udZP/ZbjiE+ltbpSJ9qbZnwyNyQ1ruLrlAF1iHtE1UBlIW7HVXVLTVks4vE4Buqw9ehUPMw5P46Nul9iyG+9nTrUPiowdSLGVvX6caNTe+uaujpzHNeJR2F4ss6o4NmfdfLP2LduRwdR/8bEfFFlmiStce35ed1f+9rX9j9zzKuYOJ6iCuFrFZeWA7sOq+9+97sBOPTQQwf2OY14DWsFKPZhrSw6DqKKWSe69bNpU5YiqhbOZ45ZlSYoSRJNIqlKEu8h7wcdn3WaNnUHFGXSfnr6058ODI6NLbfcEli4smSIv+qO1yQmZr766quBcp223XZbYLCUib8lJpN86KGHADj44INn7euLX/wiUBTm7bbbrt/GcWMJJOdMLTdQfmOf//znAyX9i4kxoah8dVCKqt1CSGUpSZIkSZKkg5VaWYqr2rpEhyuG+PRf+yjVPkww2/epfn+p6QqDrlURw6ChnGtX0kU/q5WGVtmUOqx8MXwyWitUlQoVJVcu8dq6WnQ1XCeXhHKuXT45tS+YxxPDaFUZat+3VoK/6BOyEFQUXf3H86oV0VZhaPujTkoY1TCvuz4MfiduRwWp9iOIKp1lIJaCelxaABRmj3mJSpj9U6uFrf7efPPNx3XYE2dYYeCoeNgP+mCpssTrbx/Wc8s4ipxOCo/fcX3ppZcCgwknbaOvknNLDLHX18h+8p5UiYFyXzgO/T2yLYyv6LRpcdyn1ysm23zFK14xcBx33nknAO94xzv6bQzJ97ie97znAYPFbfVnUjnzfo++T6YOsM+cB2IZMdOr2Nb5db/99uu30WfK1Aj77LMPMJiEeL6kspQkSZIkSdLBSq0sxRWLT9Cmdzf6KKoRrgzEp9qYgLCODPKzetU9jdgfnkNrVaxt3dVfXBXXhXPr8icw2/dlMYlREeLxm+7e42qtZuvok3heLaUtfgfKmHClaZ/GpHIqS9rWVbriCtxV6yOPPNLc51xx2/5dd911Zx2zK0t9MaLPWe1HVCe5hHKu9rMKwxZbbNFvY/SbSpuryNi3MUnsYlOP2agM+Jn3gKpfvG51Ile/E+cP3+uaL2rfqWnBc64L4UJRDUxGqXIR77NWeRuY7oS++++/P1DuB693PAf9iIwssy+iSnrccccB8NKXvhQoKnS8P1T6L7/8cqDcSyZxhZL4cqF4XfRLct623AgU9dNrqiJ0//3399vUZZocGxb8hXIeqk0q+CaQBLj++uuB8jvtnBN/nx0nRuU5j0VlyX187nOfA+DCCy8ceH8hpLKUJEmSJEnSQT4sJUmSJEmSdLBSm+Eidci4cmqUia2PUxOdzOpwcOXCrlpoi0mXg7fHWtd6gmIqqMPLo0Nr7STvd6KZpk66J4thlrMmU5RuPWZlc2Xs2KY+d80F8ZiH9UvLVFKnDogOzIb86vSoOSaaImozxULxOrdCwHW69lg1w7ZSP3hcyuiaG2B2jTmdXNdcc81+Gx1Fldz33HNPAG699dZ+G6+XYdN33XXXyOc5X4bdM7Fen+PasW7/xPujNk/W90v8TJPLtJrcIrVp3fHUqhupycaQ7zhG/N6wFAvTSH0/aKaONdQ23XRToPSPTvzxvvd+8Jw1N7WCkHQG11wd749oQl8Ijm1D/71OMfBFM5ltdPiO5mnnD38PTS8QTc86XddO5daBg2LyqxNPRvOZc4mBFyag1Jkbym+A95Umu2FuFHMhlaUkSZIkSZIOVmplKTqgukKoK55HB81h4YU6hUN5cnWl4WpiWlIHdGGYqg53ceVTO7C2lKU6dUArVN82S7FarBPjQVFDPNZLLrkEgGc961n9Nravw8KjIuD3a7WgVTKmVu5iP7m6OuCAA4Cyoour9HErS3UCxXg811xzzcCxq5C2nG697vVYibiq9j5pqbWWO3FVGs/dbS+mo/cwZSmqIjqn2y+tkhj16rU1N/jaa6LCpEP8NFKP/Vbwh+ehCv+DH/wAGJwH6n52/Dg3TyMeW53UODo2twJdYFBdqfuuywLg2FcVUbWFwTG5EFSLTNZo6P+NN97Yb6NKuPbaawNFTXvwwQf7bVR7VJSvvPJKYPB31XQEJrH19zQqVI4T54SnPvWpA/9DUdqcW1SxYrqDZcuWAeX6qFQZwLIQUllKkiRJkiTpYKVWllrUq/a4GogKUiSmFDCte53Of1wJBBdKywfCc3SV5IonHnOdcK5l43W14PddGcZV9rCknZGuVdVCiCu5el9+5nWLqojXve672EbFpVZnWuqayonnF0OIVXJUKhw/0UcoriTHgcqN+4iK4jBVLeI5esx1yRYoK1WVW7cXw4z9rPZniatQV9PT4AMY73uvSa2utXz6vKatFBS1P5ur7WlWlurzaqnGXsuoOkAZF1DGi+pjV9HuacEUM1//+teBcr9stNFG/Tb1PNjyP6vn1VaxbseW/asqstlmm/Xb2M/+js33PqnLdW299dZAO62ISo7h+PEab7PNNkDxb9K/Ks6LhxxyCFDmDdWi+Fus4q+ydfTRRwODZWVOOeUUoJRbOeaYY4DBIuTepypmKvnjmFNTWUqSJEmSJOlglVGWfEpWXfFv9JcYtrrT/g4lSsenbf+OK4HgJKhVo9YKqPY/8TstxaFeCUZFx36NK8rFwn1HxUwVxZWO/0c1Q3u9CpP+JLHMgGNAJcBxEyMlbW+km5El0c9AZcF9uhKLak8sBDwOah+TeG1Um1zlOTZaK97aT6tVaLguOBtXmEa22PdGycQVplE1sV8nTa2i2k/xOgzz6WtFTNbE9+0z92lU0n333Tf/E1gk6rInrYjSqCTCoFrvdW6VkZomYhko712vl0Vy49wwCsMU+1bBan1KHX9XXHFFv83uu+8+cFzzVZb0//F+s5RJvKb+tukH9JWvfAUo0X8A1157LVDGr3Nw3I5KlD5KtR8plLlBJUmlKvosOWc6NzjW4v1Vl1tyflPtXwipLCVJkiRJknSQD0tJkiRJkiQdrDJmuLpquHJhlPuH1WuKUrJt6ursLefiaUHHSs9dU0ur5l3toN0yIdQmF52moZjAukJcJ+Xg7bWIErmSuudc14iDIgfXju/RkVWZWDOs8nM00ZqI0/5xX9FcZQirYbgmXYyOldFEOA48d69XlL+Vp2ONqhqvr+flPRBNCMrfmhNb6Q+UxHV4N1XHDjvsMGtfmgAWg3oc1vcLDA/9jgwLC49tazPcuE2uk6BOnFrXAYRy79RmoWhyc05wOy1z7jQQEyHqXmEtR+eEmJTSOUHzdmt+q+vptRLfes9obnK7BkRAmRs22GADoJjT5oqh9QceeCBQ0gTE30P3e9BBBwElhUBMkrnWWmsBxaH60ksvBQZNdc7HdaLbmLjYRJya3Vo1Kv1sk002AUr6l+hWULsTOK/qwG5S4PmQylKSJEmSJEkHK7WyFEObXfHUYfQ+UXcRQyVdWUSHXJie1AGtFe+wJIethJN1yYaoLLVSBdRtVAaWItGcK9eWquV1e85zngMMrtZUOlwx+/14np6jSowrnrgv+0clcsMNNwQGV9e77bYbUNQdHcajOheTtU0CnTpb79lP0eFf1cDVcK1UQek7lRLHnOkGoKwAXR3bL3H8OP5ifyw2dXoAmJ1WoqUw1apsS1mqVYd6HplGHBu1Kh8VIRWJem6ICoxqhmPD7U3L3ClRmVYxeeCBB4ASih4r3TtW6+seqQNmbBMVVJ2kVVy9X+L96thcaPkOr5dKkL+VMVTf8zIQozW3qy56XJYtiXNe/fvj3BlVHueSWGIIBudCE2DalyZZdg6Nx+hvtn9j4sr5kspSkiRJkiRJByu1shR9Tmofo3p120X0S6lDht3HtNndI3XRzlZCPV/XSRzjCqZeHbX8mmQpyp3oaxBLbHju+tToMxRVR1cjqiKudOO51wVUXUG3xphqiqpTVEkcdyqdkSsLeQAAIABJREFUrraj/9ikSueoZrkig9IvqmD6I7Suu6s//4+KQN2H/h/vi7r8S8vPry5qvBR47PH4vP6jXJuuNrUi0EqxMY6in5PAUO1WmZI6ZYDElCqOMfvH/p02dS36LPn7oMLhOIhqj8pJfd26Er225tfTTjsNgC9+8YsAXHbZZUBJ1BpplRqaC973KuzOY7vssku/jcVsLXytqr7ddtv12zh+HQv6IZnAEoraZEoV54Ro1XE7tqmTPsPsVCyqRVG9tji36Racb1Wlrr766mZ/jEIqS0mSJEmSJB2scsqS+NTf8t/o8ivQN0NlwSfiaVkddfks1cpSSw2rV/StFb7KQB0lFdt3raomhccVo8m8Xn7muVtyIX7mNWz1ob4F+i5IbOtYsg9UJKPSoDJV+3hE9WnckZX6FeibdfPNN/c/s2ClK69bbrll4Bzi8dTRlHFlqJ+Hn9m/scyA76222mrAbIUBShHRVqHaxSZeExXKLgW59mOybcsny89ayfKmrSi3EU9eG33yor/NN77xjeZ3oypiP6jOOC4de9NCVCo8fo/Zc24lmq2vbYxUsyRHrTrHpItvectbAHjXu94FlHtRPymAXXfdFRhMnDkf7rjjDqD8NngPfvazn+238XoZeaf/WSxKa+mRF7zgBUBRqqKvkb+RqtaOn/j7rCKtEum+4nacE1SY9CeLlh+Ta1paybH76U9/emhfjEoqS0mSJEmSJB3kw1KSJEmSJEkHK7UZLlLXfdPBtmWGq80NUSqtw4ofDSHASq21U2CU++vEcxL/9xx9rxXeXZtoJpWAsoXXNJp+DB1VttZ5M8rofs/PPIeYYE/zmSa7VjLBuqZTq36YDox1Asw4PmtT30KxzpLbjWYGzWdf+MIXgOKo2apZ5Xte95jE1f7VvFgnh4Nyr2nO0Vx6/vnn99vYh9MQTl6HMUM7IafUgRN1AsLYxvOL5mCZNgdvHWkdP5phrJMGxWRUo5kGyhh3/DmP6Mg8LcR72/FoeLrjPJoXvadt6/WPyV91fPZ+8/84zn1tf2u+MjklFOfv+daEE01iMcHkQjjjjDPGsp1pJpWlJEmSJEmSDlZqZSmGgPpalUiHvZYiVCtL8em/dhR2ddxV3mMxaSk4deVsVzc6HUJZxdQpBKIaVScjtE+j+uTq033oEFkre5PAJGdxtW5K/Le//e1AUUli4jmdbF1RmiJ/77337rdRoVIlsPxBLIWjw6KJ3uzL6Izp+PO9VhK3r33tayOe8Wh4bVth6pttttnA/y21sE4qZx/EcheODdvG1AxSl7twRR5TPcRyC4vFMNUzXltLUKiOeZ7xvnduqINAIn6/K/R72hy8VTP8OxesVA+w8847A9PhvN9FvAcsA+RYePjhh4FSAiSy8cYbA2VejXOMc5N96NwQy3nYxvvB+yQqr257+fLl8zizZCGkspQkSZIkSdLBSq0sRXvsl770JWB2SYqrrrpq1vfq8OAYKmkYpzZnwxYXUqBvnLTCzi+++GIA9tprL6CE6uo7AkUJcKWsahDDO13x2ocqFbGkgbZ0k9S1FKVJrZz1zTnhhBP672277bYAXHDBBcDwYsktjj322DEe3SAqSyeddBIA11xzTf+zSRdljqqGq2gV1lbSvTrE3+9HVdbPVBsNIY6qVF2+oaV01f4+i8Gw8RgTKno/qFo6f1iiAoq/V+3jFc9FpdV+bvXBtKF6Psq943nZB3EsDyuT1BqPS0lUw4444gigWBeOOuqood/Tb2uY/xaURI+jYH+pZkEZd9Pm57UqkMpSkiRJkiRJB71psI/3er2lP4gkSZIkSVY1ls/MzGyxokapLCVJkiRJknSQD0tJkiRJkiQd5MNSkiRJkiTJ/2fvTaMlq6q03Tf+VI1qHGXpV1IIgjTSSi+JtNJIIyAgICCtAtIoolCIgGhpfiKoFI2ilqBIowhKjzTSSU8CJj0kYEEBAipW3TEc3nHrz/3uuT/k2euNGSt2niaac9L5jJHjREasiNh7rbVX7PnOueZsIW+WkiRJkiRJWliiUwfUIJnY1ltvLak7cSXb5CmXQbJCTxL4jW98YyTHOWo458lsZSaR4lxOjLbJJps0j9/3vvdJkj72sY9Jks455xxJ0q9+9aumDUkjSa2w/PLLS5L222+/pg0J7L797W9Lki699FJJgy9fMlmmU2pmu+22kyQdfvjhzXPveMc7JJXzIB2Afy7bwEno+cwzz0iSvv71rzdt7r777qmdwF8AtVQJoywRNChI0UHqELbae+khts3P5tJQyeLxcjxxjrL2sYZK0qJFixb7meutt54k6dBDD5UkffKTn5zxcQ6aVJaSJEmSJElaWKJSB7QltCM5YlRQPKEiqgEp6Ek86dYRFnQs59B2tz3bQB3baaedmudIDEcfLliwQJK0xhprNG1QDUjMR7K0s846q2lz8cUXD+uwpwUFMNddd11JRUXyUgQkH1x22WUlSUceeaSkyRU0vfXWW5vH3//+9yVJq666qqSSgNCLbt52222SuguMjhqSkkrFAtx4440lFdXIy68wJ1yF7QdqLNeAv4dSLySLRcmjlMy4iEoO1/YOO+zQtNl2220lSauvvrqk0j+e2JUEjLxGkkquJUm6/PLLJZUEquAlg2Z7ORB4+umnm8eUrmE9ZQ2mD6SiKLGOAPNLqhefTmYXtd86xvC0006T1F38l99TErjiwVlppZV6Pociwl/4wheGcux9yNQBSZIkSZIkM2WJUpYiK6+8cvOY+JrnnntOUrH6vaQB1hBWHoqSp+DHKv/hD38oSTrxxBN7vnccJRsmw/z58yWVorKeRh9rgWKgWMlu8XI+xCVggb/tbW9r2hDz4iUDhg2qGMe8++67N6/94Ac/kFTUC8bbi6RiyaMIoK54oeFYguSll17q+RyUKa6p2ufQhmKZxx57rKTJ+fWnSuwXCn1S+kUqsSWcB33hpS0ozRLV2ddee61pg5rGtfLyyy9L6laWUBl4P8f34Q9/uGlDfOCw43a8jA99QLFcYqtcPX7ggQckSe9+97sllWvBFTiuFdYWyrpQIkkq1jTj7TFvMMqYpX7fRYkTqbfMyfHHHy9J+tSnPtU8x7gzpvRprVwSpXDmzZvX97h4H3N3NvxOJX+mpizx3MknnyypXANSrxeGOeIeG9Zl5tpJJ53U93uHMBdSWUqSJEmSJJkpebOUJEmSJEnSwpxNHdAWUL3CCitI6q4aT0A3LhFk3r/5m79p2iA94y4gMNerYuOu+MAHPiBJ+uMf/yipBLZJRaKfbe44jhnXgbuW6E/kc/7vwaY8R9/hZnBXxJe+9CVJ0lZbbTXw4+9HdJF5AOl5550nSbr++usllTF2N0OUdwlgr40bbZgTuNok6eGHH+76PFwTr7/+etMGd9Wuu+4qSdpwww0lDccNF/vl9NNPl9Q953GtcV64Yd31QpA8KQQI2GSuSEVS5zn60OV4PpP3494744wzmjYbbbRR1/EMmjjPnb322ktScZEynlKZ66SHgN///vfNY1ysuP9x/XsAOy4oNhwss8wykrrDAUbpcooBusz56HqTpPPPP19S2eb9u9/9rnkNd+uf/vQnSWW95fyk4vbH3XnPPfdI6g7m/eUvfympd3zm0gaaJR3v/7h2ck37bwJzgrnFusRaI5UNNz5fZhupLCVJkiRJkrSwRAV4E8xLkDHbmB0PRpa6A5jZ9rriiitKKtaRB6gR3IyFiNXNdmhJ2m233bq+Y7YoTFiCKEPc8Uv9g+f8mKOqgnLiKg3qg/frqNhiiy0kdQcwX3jhhZKKxcKxxqBDqZwrfVEbL17z7c6AxeRqk9StTJLokrFAWfKg9EGz1FJLSSpKiStdWPCMJePm4xdVFRJOevAu/cvnPPTQQ5K6k3+inqBW0r9cQ1JRJD3dwrA54YQTJJVUIbXxRx1ijjFeHuD/pje9SVLpu+OOO05Sd5A784Y5wXvuu+++pg3JTEdBv+ue5L2StP/++0sqijvrBolZpXI9MZYoDJ6ahTnP9YH66PPosccekyRdcMEFkorSlMwe2lQ+gv9Rj6UyX1C0mfuuXjInUKjdKzQCMsA7SZIkSZJkpszZmCUHa3TzzTeXVNIDuHWMBY0yhAXsd8bc1eJL5//uR8Uq5m6ZmAWS1UnFB8/d8bgVJUBh4JhrvmeOldf82KMigGXgliGWM0nq2EI+CrD+Sesg9VrwTz31VM9xxfIL8Tyl3v7gr6tG8f0oFW6BoyRhpY+iFMpRRx0lqcSK+PminpJckbnBdSKVMUUZ4nrwxIprrbWWpLJNHoWRmD7/HK495hzJYCXpkEMOkVRi34aFjwnKzwsvvCCpqM++tRlFiXgmVBL/HObEgw8+KKn0iSffu//++yUV1Zv5yOdLJdFpTRkfNFEZ+N73viepOxktcW2MP+qsXx8kHGWMUVldYeBaYY4xDz0uknlDHNtNN90kqZ6iJRkPbcoSaqGr61zvzBd+iz0ujfilWiwhjNtDk8pSkiRJkiRJC0uEsnTAAQdIKnelHmUP3AFjzXB36gknYxI/LB7851KxfFZZZRVJxQ+L9SV1q0yzAVd+pN7YHKmcc7QMvU2M1+GvxywBhTVHqSxtv/32krp3w1111VWSigWP2uNjxLyp9Qv02+FIPI/Um8wUSPMvSd/61rcklZgcYkOYT1JRRgcFyivxNT7neY5dou9973t7joExRD2gD7kWpNJnKFKoNR7TE68rlE7+Su2JCgeJ72pDJcIqpuSCnx/KG7t8UKHcgub9KCeMO4qe1Hs98H7fMUdJniuuuGLqJzZNmJf0yymnnNK8hkq8yy67SCpz18udMOeZG1wXrmKyLqPy1nadMn9QXNdcc01J0oc+9KGmDdd0Mh7alB3WFt8FG9dMfmfd88NvSFsh93HHV6eylCRJkiRJ0sJib5Y6nc75nU7n9U6n86Q995ZOp3NLp9P59Rt///GN5zudTuebnU7nPzqdzuOdTmf9YR58kiRJkiTJsJmMG+4CSedIusieO0HSbRMTE6d1Op0T3vj/5yR9QNK73vi3kaTvvvF3qOBmQt5j26q7PzyI1PEA3SgPIvuRSNC/C9cBAYyeANDl+9lATPSFHOqBlVHirG39p01MKlZr666HUUHdPt+K/slPflJScbWxdd+3asegdv62VX9nrrjLF1cU7gXm3J577tm0ISAYN84222wjSbryyiubNoN2w+EqxjWCq0QqQfqkA8ANgutFKufIcVETjlpzUulDPhtXln8OmyLoJ8bJEzJ6Pcdh4m6CmBKjBv1DADuB0KQpkYrLkbQJuKA8XQn9FLfPu+t36aWXnvL5TBe+FzcwiSN94wH18Fj7uN49uSnuF64Z3Cm+UYANAZwf3+3rB2NAKgJSCXz5y19u2qQbbvbCeuKB2jGsoZZ+hWvQk1lGZr0bbmJi4i5J/1d4eldJF77x+EJJu9nzF038mQWS3tzpdEZ35SdJkiRJkgyY6QZ4LzUxMfFbSZqYmPhtp9PBdFpGkmeTe+WN536rIULwJVYbqo9XReeOF8sFC8jvernT5Tne49YxFhRWMltsXQ3A4iLQ2KuOj4O111676/815SQG4WFF+t18VKSwju+4446mzd577y2pe7v0qCAAecGCBc1zbNHHykfF2GGHHXraQM3iBeYP/eUBrLzGX5TNSy65pGmzzz77SCpBxbxW25QwKGIiVg9KZ0wJ0OW4PDAb1YHg38cff7yrrVTmPH/XX//PHnhX+VAkCQzmuxm32H6YxMShUhlTrnsff+YEqgjHjkLo70NRROH2seUz4/rjmzBGqcoS0M8aylrlqRsI1ueYOT5fGwjaZoz568HAjC3njArp1xDvQ4FFefM0CgQD10qyJOOFsWE+SCV9CGMbE+D6+8atHrUx6N1wvduIpOrZdzqdwyQdNuDvT5IkSZIkGSjTvVn6fafTWfoNVWlpSdRPeEWSB+wsK+m1nndLmpiYOFfSudLMy534XaxU7lLdQqM8Cb7UmmqAhYOVF5Po+fuxoPGtuwWFUkGs0LiVJd9KL5Xjq1lmWI9Yw7U7fSxD+uDuu+9uXkNZaov/GBY777yzpO7iplj5jOUee+whqbuMAuPD+NMvPu5RVav1C/2BakGSVI95Y7488sgjkko/U5xUkm688cZJnO3kocAr8SiudHCtcO6MrbehPyl2iWrk5XJijArqFd8plesU1ZH3uLLHsQ4bPz/Gi/FDVfN1hfOKSUl9jeGcsaRjDJxU1hbmFmuE90FN9RoWxPl5/JHUnZSS64E5whroZVyINSGGk/Py0jr0Oe9DNfJ+jrErqPrezyjlxB8mswdSsjC2UlkzowfI12nWnfhbNZuYbuqAayUd9MbjgyRdY88f+MauuPdK+iPuuiRJkiRJkrnIYs3/TqfzE0lbSvpfnU7nFUn/Kuk0ST/tdDqHSHpZ0offaH6DpB0l/Yek/0fSx4ZwzD1guWPF1JIkoixg3bTF5MT3u0rCHXBUGGqJDCl34DE94yDuziNGxOMkOB/+Yj16/8RElbxW22lYKzQ7bGoFXokR2m+//SRJ//7v/y6pJLCUijVM3E4siCoVRYj5U9sJSJ9RPBTlBMVLKjuCSPAX2w4DrouaWshcRzlB4fAYEeJ0KPXB9eaJWGlP3zMnXDEhro++pK2PV1u5g0HiiUNjbCKJI30nF2pTTEJb21HKOdT6kjGIsZO+xgxzLkSI72QMUA89BpPHqE30gces0Z+c38KFCyV175hELWKdRWHyYrvEw2255ZaSSl+6Cj6OtSUptJU7YWxcdWaNJPEt//eYYpQo36U621jszdLExMRH+ry0TaXthKRPzvSgkiRJkiRJZgt5i54kSZIkSdLCnK0N51uMCYhsS5ZIG1wASN38XyoSItIi73fZMboOakGcPN5ggw2me3oDxWtvSdLpp58uSTrppJOa5zhXZO9abbhYEyhulXZGGaSKWwD3mQcO4ua47rrrJJUEe0ceeWTTBvcdQbe4F1xK5jF/kY09UBgXBi6JWrDyt7/9bUnFzXDLLbdIGk5/cTyMLe5GT/zGPOY8CMwl8FeSnnjiCUm9gbnuhkE+x61HP3ngMK4mxgn3YC0Aeth4UDHuAI4D9xmB2lK5Hji/uFb4c/H/7oaLiW/pQ4Jfpd7agsNkhRVWkFTGiTnj1xDzmOuEtdfrGca+I9CXjTVSWVtwafI5Ph8J3ua5mLZAGk/CW1h33XUlSauttlrzHOkjqJnHuugpMfrR5tKardSOOdYE9DnP78+DDz4oqcwR/w3n95h0G1wf/pvD9dSWLHiYpLKUJEmSJEnSwpxVljxRZCxPArXA0Zg0zYlV52uBhLSJSdhqAd6zpexJTJNw6623SpI+97nPNc+hsLE9uFZZOvYLVjIWldRrYY4CAulJC/Dwww83r3HuKB1sN54/f37Thm38KGScu/cb1j5tYnoJSVq0aJGk3sSVDtb5//7f/1tSsa422WSTps2PfvQjSZOzTNvgGkEtwOpzyxx1B2WBc3DVYNNNN5VUlDc+x7eOEwyN6sT4u7JEYHhMzVBT1bAwXd0ZBMxzV29QlrBYuaZ9/Jj7sTSOXydx4wPzoLbWcJ3FsZF6yxMNk5hIlT7wkjiMBc8xfh7gzzjHTR8esB+vK9QHVxhYV1FB6VNXaaejLMXzq1FTdlhLOPbzzz9fkrTxxhs3begfVDTUR/dasO5cffXVkko5oJoyOVOFadgKVe3zWYM5Zx93VFPSiDDGPu5sHmFdXXPNNSWVgP/ZQCpLSZIkSZIkLcxZZcnjcLiL5e6/dofer1BszXqM73ELIVpVWDmupMQtxOMmKhxY9G5dexIxqV4OJqZNoF9qBRFr6RuGBVb5hRde2PPdxE4AFuYXv/jF5jmsPKhZn7GIcG1u8Fxb/NGjjz4qqVjVKDu3335704byJDNVlmLJGc7LrXRUQWKW6C+P14KYNqG2zReFCSvSE3tigTMG8Xp12GY8aGUJq7Y2P7FqOU4/9raUEfH9vMZ5+ndF9Ynv8jasJaMo64GKxRhw7LXvJK4kFgP2Y+X9jJsri6hpcX307yJWKs4VbzOddbWm7vXDC19zrpT6ge985zvNY8aU0jErrriipO4UEFzThxxyiKSSQuSyyy5r2vh8mwnRAzDoGJ+asoQSXZs3XHMcD3PF1w+OkTnC2uXK0rhilSCVpSRJkiRJkhbmrLLku0f6ldZwC4S74ag6talP4FYJ34USg+VU2zU2yl0tbUTFDCXGj88Lp9be48/FxJUeb4FlMcrEcZQVwWLxuRGtNawZLDupxCVgFaOO+Hs5rxiP4nFxcdz/8z//s+dYUSRRdJhH3hbVlPih6RJ3bLLDiCKlTpzXrujQP8QcMe5+3fGY+B/67qWXXmraoBYR60Rci19fMSZw0NTUUGBucE17UkrguPgcP07ibKLK4jsm6Rfa1FS6uENxmMoSu0S32GILSSWGyksYMSbEmHCerqZHNYP3uPKK2sj58X5fPyCqcx4bxjyaCuxM9qK9vgvPz+GnP/1p8xy7BaEWh8b1df/990sq163vuKWk0te+9jVJZVfu4Ycf3rS55po/F8Lwa2Y61AqlDwvmKGp13C0qleuJvuP/HhfH53B9eDzkbCGVpSRJkiRJkhbyZilJkiRJkqSFOeuG822HnhxPKi4lfx45OLqXagm2ojvOJeAYtEmiLT8eXnOZcZzE8+GcPUgVybbNTRnfT194oCefM6hgxclAYCVBlGxjlXoTcuIiu/POO5vn2M6PFFyrERhTIdAH7oZDmo8V5WuQKoAq255+YeWVV5Y085qCjAuBpjVZnjmAK4Rj96R7nBeuQlyJ66+/ftPm6aefllTcUiQXdEh0iasOd6lvQa8FDw8SxsvdQ3ErO65IXxtisr1a7cToMiJI3tvgUmNu8TkENPtzzLlasP2gYE5wzByHp47ArcQ41dICxABxriVcdlJvgkk2MLgLks/kOo0pCaTpuWj32msvSdJWW23VPEdCWGrT4XY955xzmjZcB9S4i6kW/NiYz7Wt8aTfoL4e17iHQlCDkc0e9GFtGz7zl+NwVyZznGSy/EYNA465beNDPC7G369xXmMejjL1zGRJZSlJkiRJkqSFOass+ZZUrDTu8AlO9VT7WErcgWMJ1bbGx8RzHqzGXTHfz927Wwjc9bfdZY+SuGW3prLVEjH6e5yYir5mZc00SHEqHHjggZKKtebK0mOPPdbVFsXjE5/4RPMcyhJjiJLiVjFqQ5w33oexgnybInDWWWdJKhaZJ9L0xzMBZZPzYR54cCpWPWosfejB5agNWHsoBH5+9BXXF9/tigltCCKmn3yO8f5hKUu1wPwYtMv14eOPQslrbG12q591iJIYzz//vKRuJSSqTcwZ7yfWLV/jhgXb5On3GMAuFYUkBqfXAvyjuuYKAcpNTC/ifdgvPYGnNpmOYk8C3qOPPrp5buedd5ZU5iPf6d/FWsK4EVz+4osvNm3imhm3yPtj1guue5/7JLqMKlZtExLHypz1Y2YsSUsyTLgu6INaQleI4127xukf5pzPQw/OHwepLCVJkiRJkrQwZ5Ulv8uM1iJJBj2pHxYTqlOtJEVUC7h795ilGM+Cv9uTlsW4H45rXEkq+X7Oy61YiBZLLWlnTFSJNcOWcqm+HXxUkDKfvzVIGeDbeokXitafz7EY5xXjtqTSd4wzCkONz3/+84s9n0ERlVKPXcKSJ96LLc6efoHYqzh/3aJmizXWLJahb5uPZYlqx8Nz/r5BUku6iIVLP9W2sqO8Mb9vu+02Sd3XPXOe+C0Ui1pqBJQKrkUvRo0qO6w+cPgOrgG22Psxc+6cD++pKUucXy12kX5myz6f4woMawrqEePlcW0z6ZdvfvObPY85dhQdjzVCOUGBjYWQpXKuzF3WBld2GG9+f+hTX2Pocz67VnqKeRvTtozrt4WYpago1RR3xp//+zjGtZf/8/lSPRXLKEllKUmSJEmSpIU5qyy5H7df8r1akULuamvFd/slUvQ7+3iXjHrlFhSxL1gG+LlHGcfjsLviueeek1S3zKKqVis8GfsQa8tjw1BMPOnjsIk7HGtqIZYPu2FOOeWUpg1xZ8RX1OZG3O3RVmgYK4/dX3fddVfT5t577538ic2QqO6hYnh//eQnP5EkHXHEEZLKnPV4GaxhYk64vrDEpWKNY/liVbt1zfGQkK8taV4tKeogYIw94SSqDvOGa9nVA8aUNvSPzwPa8Jd+q6kr9FMsfyKVfhrFjiDGIBYRdouemD7OnWvBy3nwGuoT51lTTjgv+qUWkxOTW3p83KAVN45r4cKFA/3cvwSIX2Q+1zwSUS3m2nMPR/Ra8H4vmpzKUpIkSZIkySwmb5aSJEmSJElamLNuOHcxxIRoBF3XtrTzvpggy4kScC3AGxmewEOkaqm4X+I26nG54b73ve91/Z+EiO7qiOkX4lZQqfRLrA3mdeU8gHJUxPQGbe4dXKKe8BF3AttUOWcP9OW5NvdQ3HqMS+Id73hH3/fU3DCDIn42MrifAy7C4447TlKRvb02HP1J+g3cKF4RnJQMbKmn7zzZJq4UD+iVuq8vjm1Y2+ZjkLFUNn9Q8b5W3zDWe6u56qKbitd8i3RM41FzKTEfh5U+wYnfwXm5CzCmxKi5qfkcXCyMo2/9Zp2IG3L8c3CJ8lpta/woUiokk4NQDH4TYh04qTdVBOuAp9SIm1BYP1hPpFJ7b1ykspQkSZIkSdLCnFWWfOtvtNJeffVVSaWUhLePd7Bu1cY7YCwo35YZt9bzHg8+4/2xhMC4iEn3OD7fjhsD63hPW4qGuKXYoX9GUfk6UithA2yR92BOrNaakgj9gv+dfon59thjj6bNpZdeWj3WYShLKJpxW7EHy6KwxE0JL7/8ctMmKiUcq89rFCT6csUVV5RUklxKRYl69NFHJRXFzccLZcIDjAcJc9/VUOZvTCvgqg/dhOdEAAAgAElEQVTXzmQCWbk+ausHSgvPoXQ6jElU4IZB3NjB+XmZoJgCIa6TUjlnFCmUAZ9r/UpZLFiwoHm89dZbd3123CIvpbI0myDAm9JirDl+XcRkr7zmqmNcB/m73HLLDe3Yp0oqS0mSJEmSJC3MWWWplugNawQr15MCYpnEhJM1xSCqId4GyyuWB3BiXIsn+BsHUbWIpTv8cVSYXGWJljJtagkoh6GUzAS2q++4446SuhNXxpQBtcRz0fKJBUj9Naxp5oiroKMEa5/0FihBKDtSsdKj2ojFKBWVgdQTqCGuzhAnSOzKAw880PW5UrkOKOdCXJSrT8yt5ZdffopnOzlqSh7jFMt6uOWLykR/MaY+trzG59WSUvI5FBNm/fB5VIuHGjbMZ64FTweC0kXiUcaoFsvJtUMiUy9mTvkYzovvcBWLucoaxed5P09G5U2Gh8cjxRQa0Svjr8USL7WYvBhL7OvQuMlZlyRJkiRJ0sKcVZZqu0jiThW3QLCGorLkKlBMulgrpEv7fkqVVO6OOQ4vZTAbqBU7jLvgYl/E9lKxMMaVar8fNVWLmJyzzz5bUneMWUzAWVMfYgmbGL8h9cazMUe9JAZxdJQVGSbxmFE6fv7znzdt4tih/rj1yE5P5jVFRFEavD3qAaqMx5qgohCXhLpCfJO/Nqw5VYuboZ9QOBg3b8N4M0dqJYywqjl3L9YLzC0+mza+24vXRlEyKMbpce4+P1FKWccYY99RSN9xzPz1eUT/8H7es+yyyzZtfvWrX0kqKuhGG20kqa7yJuPB4wnjjkaupbZYPnDVOa6rXFd+DUbldtSkspQkSZIkSdJC3iwlSZIkSZK0MGfdcB4chnTHcyTWc2k7SrcxkNmJ2xf9vbH+F1sl3V2F+42gRA8UnQ1EF6LUK58ie9fOPbrqvPbVbOUzn/mMpOLy8TGJ4xz/SmV8Y//UiJKy1yF76qmnJI3GDVcLxJW6k6NSF49gW4K2/T0xdQB9gTtOKv0Zt6B7H3I89913nyRptdVWkyRtscUWPd81rLpoHINfr3wXySBx0bqbgJQKzPWai4zrKlaCJ2jZv+v555+XVDYaeJ29GAYwTOJGDubqs88+27RhHeW8+L/3D+f1X//1X5JK8L73E3OB/iDhIIkNHVy0HJ8HeNc21SSjw7fzx9CDmuuZMWS+1H5z4zyM6Teksnazho6aVJaSJEmSJElamLPKklusWIRsN+Zu1C16ggq5c60lS+yXGKtm4VFNncDhRYsWNa/FatqDrpI9UzzRJPRTlmrlHKIq54GesxUseY7V1TCs4qg++rnXVEZ/XirWL4oCc84t56jCDVM9YHwI2iYtAFa7wxZdUgD4ucdklLUyFcwpXouJUKWiJLzyyiuSSlkgvz5QuGrHOAg4ZtYMqQSO3nnnnZKK0uWB57GUCUlNXQFDcSFwmRIwDnOEdAkk0F1vvfWaNsyxmpU+aFBpmLMxiF8q8zcG39aqxvN+ztPVfZR/5hpj7es0x0PKgcsuu0xSPQVFMh58PYvj3qYe8VotNUv0ZHBN+m8V6ShSWUqSJEmSJJmFzFllycGC564UFWG77bZr2mAFxYRvvrUV2qz9WIgX6xOrVCp323zHKMoWTAXiCmrbs1HeOE+/s8eKjlvs3QqFcWzvbSsdQmxZtI6lEqPSVpA3xjHF5JRSr28+WuRSd2FI/xy3sgZVIgYFB8sda82TScKaa64pqcyNWh9OZkzj+/zcURRIbomaUku/4CUwBkktIStrwM033yxJ+spXviKpu4wPfUjcDsqHj1vsZ95fi7eh1Muxxx4rqZT5kOqlmIYFxYNJC8A5MEZSKRG0zTbbSOpNfyCVmLWY5NTXj1gmh35xpQjFnkLXFAF3NYH1PRkPJPiVer0vsRyQVH5zWYNp40W2UWyZL8wnvwbmzZsnqVynoyaVpSRJkiRJkhbmrLLkygDR+dzNohT84he/aNpsvvnmkoplWSuEicWEZV+z8KN1xM4e393EXTGW5bh97NFCJckcKoJU/MHRaqQEhFQUPF6j72o+5FFYxZFaIk1Ugz333FNSiTVZffXVmzZYNVE1qiXCi+qKf1c8Z+aoW+kbbrihJGn+/PlTOLPpwa4jxou+uP/++3vaPv3000M/Hi/cLBU19qtf/WrzHNeKl8kYJLUiwIwz37nbbrsN5Lu+8Y1vLLYNO3d9HnFsoyjrscoqq0gqsWrMES9C/vnPf15SiQnF+nfVMO4W5bVauSTWH9YTj2s6//zzJUkXX3yxpHq8px9bMnp8rWAdZUxIMOpxrFzTvIaa6So7SivKJq95bOELL7wwwLOYOqksJUmSJEmStDBnlSUv2fD+979fUrFq8Ht/97vfbdr442Fz7bXXdh3PFVdcMbLvrtEvBuvcc89tHhMzwbE/8sgjkqS99967aUOcFhYhCh79PW5qahYW7pZbbimpWNAeb8H7aooStO2MjJ8Ty8m4ddxvZ+QwlDgK1l511VWSSuxAbfciikKtDNCgduzFQsyosZdccknTBsvSi/0OEuJvPM8WuY7GUUaBa8lzX7HDdhRlhI4//nhJ0gc/+EFJRSU+77zzetp+8pOfHPrxAMoSfeB9QUmUZDxcd911zeNbb71VUlGY2Am6wQYbNG1uvPFGSWWNYQ0+7bTTmjb77befpPKbwjV5ww03NG1437hIZSlJkiRJkqSFvFlKkiRJkiRpoTOKlPqLPYhOZ/wHkSRJkiTJXxoLJyYm3rO4RqksJUmSJEmStJA3S0mSJEmSJC3kzVKSJEmSJEkLczZ1gBMTBk5mG/Ymm2wiqSS0lMqWXZLTsc2YZFpSSYx10UUXTfr42EI+jkSNNWplQdhK/eyzz0oqqei9HMwaa6whSfrmN78pSTr55JMn9dmzHc6LFBTf+ta3JE39HL7//e9Lkk488URJpVDoXCcWxSUB3YUXXti0ITWDl/ZI+kNyUkq+MAclaemll5ZUksaefvrpksq26ulSK1cTC0RPZs5T4olEv1L3XOjHe97z57AQEt22la1gzayl7JgNawxla6SSVJXyQqTIqCXkZGxJheDlYEhQevTRR0sqyTtnG22JnIFzkUoS3Ji41FOYfO5zn5Mk3XLLLV2fUytmPq7f0VSWkiRJkiRJWpizylKbldSm5FDokYSMixYtal7jbpgyEWuvvbak7jTrKFIUc7z33nv7Hs9sU5SgZpFhKfEaSpPf2V9zzTWSestWLO6zxwnjQqkGxtitGhSgD3/4w5KkzTbbTFL3uZOy30szeFtJOueccySVQrWrrbaapG51DnUGBW+29Re49cgxMydQFA455JCmDQkLzzzzzK73z7a5Pw4+8YlPNI+Zf7HUh5dmQnn5l3/5F0nSgw8+KKkUIp4uUUXyx3Ee/t3f/V3zeN1115VUlCHGdtttt23a7LjjjpKkjTfeWFIpX3Hbbbc1bVBK+EtJCy+7xLl6ks7ZyKGHHto8PuqooxbbPhZXB0+2SdJfygBdcMEFMz3MgRIV5ja81M873/lOSSUpLnNthRVWaNqwbniRXqlecmwqxzFIUllKkiRJkiRpIW+WkiRJkiRJWpizbrg290Xba9TJohL8m9/85ua1l19+WZL0j//4j5JKTTd3JWy//faSpMMOO0xSCeqrycZIiDUX3TgggJ3zwyUpSc8884ykUqfrhBNOkNQtddL+uOOOkyT91V/9laQiuUvFvfXHP/6x5/3jYJ111pFUXAeME64yqQQlb7HFFpJKAPOpp57atOE13K+33367JGnllVdu2lBNG+hTDwZ961vfKqnUTppLda4IgGeM3YVNgC9yerrfCmuuuWbzGPcbgb3Ux3M3HBAGsP/++0sq89LfNxXaAqOXWWYZSWWer7LKKj1tOHYqwbubCHcStby47t2NgsuRdahWR2zevHmSiqvuoYcekiTddNNNTZvZ4Lr2a5q5zm8KLkx3v7P+4IajLzzAG7cU6/O4ifOltpbvvvvukspvAm5YXwtfe+01SSWUgc8l3EWS/uEf/kFS6Q9qzlGrVCqhM+P6TUllKUmSJEmSpIU5qyy1QQVt7nKlojBg1WANUC1ZKnesr7/+uqRiLe21115NG6ohs+X3Bz/4gaTuIMWf//znkqQf/ehHksZjCXGeUgkqpaI75+l39nDPPfdIKoGWTz75ZPMaAb4EuWMNu3WEVYWCgoU4yorRbM+VigXHcaAioYpJ0t/+7d9KKoHrWIh77LHHlL73r//6ryV1W5QRjgNLk+DHF198cUrfNWw8wJtr5X3ve5+keuDp//zP/0jqPZ/a5/ylQfC7VObILrvsIqlY5vPnz2/aEBTN+oOa+YEPfKBpMx1lqbYOsY6hGrI2+HzkfagiKAOuytMGJYnPcfWa9gT6olC5CsF30HajjTbq+Ry8A+NUmFxRBtZcxtg9CqxDPEfbmvq06qqrDuGIp07sX9ZDn6ustYx7bR3jN4E1grmBUukwJzbddFNJRemUpM985jOSpGOOOUZSXY0dJqksJUmSJEmStLBEFdI94ogjJEkHHXSQpO4EeSgb/OUOlmR6kvTwww9LKioUvlZPDhZjk/DDsq1WKhYhqgx3wqOErfJSUZSw6GrKh1tuUrH2als3UQ+wDNyHzPZXwEJErZNKPNOg4bt9Oz9xQ2y7Rk3zmCUsOv5iBdbOPX6XqyWxPW1QriTp7//+77vaELfx6quvNs/F2KfZwuWXXy5J2nPPPXte+/SnPy2pJDO98sorJf1lKkvTSZrocSoka3ziiSckFTXbr1u37qcKqQmksgWeOLSYGsNhftfmPmttVKFcwY3jT5taag1eQ43weC1iV0gePA4WLFjQPGa7O78TnI+nCeA8eI2+8DWDOYCyiPo4blDKSIrqx8w5M87MfT93flsYW34f3fvB58Q11FMrsI6iVJHSgljjGZCFdJMkSZIkSWbKEhGzxN0sigLqEYn/pLJrjVga1Ba3TrjzvfTSSyVJxx57rCTpxhtvbNpgeXG3jEWGCiWVHU5YQyRvi6nchwG7CmqqCDFHtQRpnBd/UTxcceJcic+qWY8xGSEKnlsRw1KWOGaPxcJa4zgYfx+vqSgAnDMWj8clsKMI1YpYLreoaY91TxvGTRqvslRL+EYSyrZyG+xe2WGHHbqen4yyNJfVJx//WmkOqXu3KDFKrFVrrbWWpGK9S6WMEBZz3KU7UzxOM65jzG8/h/gcc9ZVZNYA5g3j6MoA4xwT3ta+i89BTfDPYdfYOJUlSmFJRWHnnGu7xzjnqLz53OccWaNmC3hGmCvuJWDc/Tyk4sWQynkxx+kn/x3oFxfnoM6hzn/lK1+RJB144IFTP6lpkMpSkiRJkiRJC3mzlCRJkiRJ0sIS4YYjRQDSNkGmbLmVpLe//e2SipsJ2dCDHZH5+PvjH/9YUnewNG4Kgpwfe+wxSSWBnFS2T7LFnuDdUbjhOD6XMZFPOa+a2wn5E3cQ/8dd6Z8da0zVXCexTt8oEnMyTl7Lj62tSLcEzR5wwAFNm6233lqSdNddd0kqbjB3L+Jiw9WHi9cD/plL66+/vqQip//sZz9r2pCegH7mWPnccRPldKm4ju6+++6+73vqqackSbvuumvX8z43ootvSagf59dSPB/+f+SRRzZtcNGef/75krpd/JGPf/zjkqRHHnlEUn2r9XTwbe+sCbjUGBuel3rXTHDXWHSf1FxRuGFw0dRCBXgfaxbuKw88p+Yi1ezHgadLicHNnLOHHsTAZ1xQPn8YX9+YNBuI9dp8XWQOMDc4H3crx3Nn3H0+8VxcC7xN/A1xd/IoSGUpSZIkSZKkhSVCWULBIYibu3a/I0ZlitvoPcHeaaedJkk6+uijJZVUBBtuuGHTBmshJlTz6tqk/F9jjTUk1UsHDAvu0N06ITU/1hB36271xbt+8EC9aEGB/x9LA6WkFuQ8LEjV4EkpOUeeo9I5KpI/RiX64he/KKnbgjrnnHO6/qIW7bfffk0bFEQCD+lnL3eB2oTlzth4UtNxUrNqSQx44oknLvb9XBck1vNNFlGZmMuKEtQCvIHzI8GrJG211VaSpO22205Sqc7OdmhJuuiiiySVUh8oDoNSljyZJMoNyhLqXy29CNcya0RNGWpTkmNQc02F4Jqj7zgOV7pcsRkXPnc5D64dVDG/lmJQOufF9eLtfc2dDUTV29fF6EFoG/8YAF/7HM695v2IKRl8U8woSGUpSZIkSZKkhSVCWWKLPvFD+DI9Hglrirt3to5/7GMfa9qQ/I0iufjJsfCkknzy7LPPliTtuOOOkkp8k1SUCY4nJiIcJtx9u1XD9lvu7NlaHxNRSuXOHkvTrbioTEVLQSp3/VjKsbzHMEFRdKsPhQMrJhZslMq4A0VhGVupxJpcdtllkoqiiIrk34u6hmrkW75J33/fffdJKsoLsUyzBVfDmFPeZ1J9yz9jQLkCV5ZiLM+SoCxNJu3E1VdfXX3seGFWEnwSg0n5pDYVayq4IsRawBqB8upqAnOVaxs1xFOP8Jkx8W2tDcfO2uLzilhQYkz5bo+PGuV62g9SLUhlTWHNZSz93LkuUNFikkqp9KurTbMBVHnGrZZENJa7cXgtJiz2ZM/APIyfK5XfK9aNUc+DVJaSJEmSJElaWCKUpWWXXVZStxUrSQsXLmwec8dLMrGjjjpKUncCw1gEkB0+Xq7ikksukVRiDoiTosifVHav8L5RqCqAguK72IA7+1pK+hgjQH+5Nct5xMSMHnOAlYjVSCyO++E5DrcWB4lbLMQRbb/99pKKFeiqI32Gksic+O53v9u0oV+IVcKqqe14YdzZdeRjccUVV3Qd12xTlGD//fdvHt97771dr7UpQ4sWLZIkffSjH5VUdn31a++f522WBPWJ+VDb9QPEsBEfKRVlgrUmfp40PWUpzk+pqBgcV9ztKRUVPu728vUjfkfbuHEt1VRn5k9MzOhrDArDsNeRNjwxIzuhY7HcWvxfjNPyODRKZ/G7M1tgTtR+EyCOt7eJ10Htuoi7J2njY8s6zW/JqJN3prKUJEmSJEnSQt4sJUmSJEmStLBEuOGQ82JitR/+8IdNmw996EOSSgqBnXfeWVK3G44gvH/+53+WJO20006Sumt14aq75557JEkHHXSQpBIYKZVK98jN1IYbBbHGjh9HrBruUn5MFEhbl0FjpWza+rZe3FOxDlAtSeYo5HPqBJ5yyimSStJEHy/OHZcE5+Aushic6NuvgXNlOzbuDq+p5gkzZzNcH1JJ2gltLpaHH35YknTqqadO+rtqnzeX3W9QC3rdZ599JJX0JFwn11xzTdPma1/7mqSyqWBQLknWBOanJP3hD3+QVObzddddJ6kk9pXKhhlcT7W0AnFtaEtCG10svgWc6+P222/v+u7nnnuuacOxElKBS3uUsMZLZU2hD3AX1VzsuDRJcuvgVvJkyrMBQg1iDTx/zJyYyhx112r8vBrxt6k2D4dJKktJkiRJkiQtzFllyS19LCWsEJJAkgBOKooQFZQPPfRQScXSk0p1cKwbqn2TXFIqd//z5s2TVILxFixY0LTBgiMtP5/nSbiGldK+tk3Tk09KxaJrK1NSS04Xg1NrJVEIcoylLVzFGkXpk36gEjI2UlEXY1kH384d+xArqxaczByJgaxOLAsxDCi7wvURk5NKZS4QWIsa66kDCDwlySsKg58Xj1EJUNN8CzrpJLAoY0kL/67HH39cUu+mi7nI/Pnzm8cEA7PBBCWuxqDnCOuPX8dR+UX58GuaecNrNRWBORU3tcTrxp/jOHyTBNfg008/LamkT/DrjO/3QPVRQwJcqXfjC39JQOmgNqEs+VgwPr/+9a+HcMTTJ6aBqG0MaktUHIP+a/M6qkW1FARxgwH/9xQ47uUYNKksJUmSJEmStDBnlSWPd+EuFCuU11CKJGmvvfaSJJ188smSpH333VdSSTIpFYuXRGi77LKLpJKqXipWEKoTqhFKk1SsokcffVRSKcfiPtphK0t+Z853sV2eJIwed8NdeoxH8mMmpgtqbbA6sJJiKQD/rnFQS2VQs34jrgpKxUqqqXOoLKMsc1ODBH/E4KEWuLIUSyug/lBwWCoKLdcXiT59THkf30EfvP/972/aoEygPmERumpAMWQs8NmuLLVZ0KwVxB5J0m677Vb9nLbkjYOipgjF0hG08fFHfSduiPHya4JjZh6gvHr/0J55w3F40lZSscTPc4U7rjHjgPQwDusIqj7xYA4qNvGRNSWEZMbjhpgwYNxc0WMs2uZqjOVq82hE1cjhdyZeH6xHUvnNHQapLCVJkiRJkrQwZ5UlhztMFBMsenYpSMVSYlcclo8nrkSJ4g74gQcekFSsHanEUhDHxI45T5DF+3/xi19IKnEKcUfVIIkJv2rW4z/90z91tXX1Kd7J06euQmDdRfXKv4tzRzWoWcnD7IdI9I+jeBAbIfXuquCc/dw5L9rGwpFSsbywGnnPZJSrYYAVi0VGDIUfD/EovIb6U0sUyPtQo/xzUHOZa8Q1eTxBVAliUWupxCyNI9FgG9OJH+K68J1l/fBrcd111+36S7HvWnzcVIjKjlT6Ps4DHxOulVgeyWP6OP5YEsffE5PixhIpUu+crX3XsFT5qcA1IZW5ynFRCqWmjtCGeeTznNdmi7LksbpSGa9azFIsqFuLUe23mzq2d7xNLQmqVMqdSaksJUmSJEmSjI28WUqSJEmSJGlhzrrhXN4lkBLJDjnzzjvvbNoQkA1sIUfqlkrwL+4FXAIeqEc1bAL8kFE92BAXBIFwBE96wJwHfQ6CuL3TZX0kTs7Zt2pDTC4G7mqJNY+inO7v59yjTFv7jlHCsbu7oF8Aors6+tX38+ej2w2X32xwG0i9bhSpjBPB/gTxeyJN5jzng9vE+4dz5P0kYvVNBGx8wN0dt1xLJeC1dqzjhPk7GXcc54Wr312+uO1x1VDR3RMzsibRX/Fzpem54XCD1two9LuPOxBsTyA21w7uZqm42JgHsUK8P+b7cbXV0pzwHbGWmjMu93aE8I8YnkByWyduf3c80eVswAOnpXJePhb89sbNP211DHmttjkiXl+1wP74+8H6NGxSWUqSJEmSJGlhzipLbDGWeu84sWC8onNM1Y6K5AoPVhFbrtdbbz1J0q233tq0oT1K1UYbbSSpO4iTu2IsC47PlaVBWxHxjtwtOlIGxIBatxBiMjACKt0qwHqIiS/dSkKFi6rBbLECaxZ5LCeBQlQL3u73XqlYwzGQ1rfaMgbDTEYJKIikDuD8/JjjNnDmt6fLoA1KLf9HqfDHzBs2VLiKyfxHoYhKsD/nSRFnE1FhcvqpPR7ATDJK5kotqR/PRXVtpuVOGCMf/xhkXdvKThvGkmvZg5NjgD//9/GvBQjH/8d1opbUlnWnVnJoHOBlWGeddSSV86wlzURlrCnuvtloNoACHEtouScBLwpjwfzxMY2bj2qbY6Kazxi7B4C5wO8Wx8XGpWGTylKSJEmSJEkLc1ZZ8oR/KDikxufO061TfKvcnaIe+ZZU7qRXXnllSdLll18uqTtpGp+51lprSapvn8eaIvaANp5eYFhgpfvdPxYqKRBqpUwiWAj+OVHBw3r0mAPiG1Du6HcvRjzObeGMv6sZnBfW9XQLl9IvMS7Ki2bWEtUNC74LtYgx9fg6xm7ZZZeVJN19992SpM0337xpQ3/ccMMNksp8cuUjbkFn/GvWI9YoSoePBVbiMMsWSHVlKKp9bh3HWJOaMsg6RNs999xTUllrpKJIx9IzXvLjpZdeklRSj/Q7vqnCdxIv5fBcrcwE6x9tGBu/7hlnlBPOp3adxfhKP6947TGHvbROVL/HDV6LuJ7WtrrTH/SdrzG1skjjBCU4qoSkF5Gke++9V5K03377SSpxbLVYI8a55v0AnuNa8tIv/IZssMEGXcc1it9VKZWlJEmSJEmSVuassoR16o+588RKdss3pqLnNbf6UJauv/56SaUUipdNufbaayWVO1/utj0inzvvm2++WZK0/fbbSxrurijOK+40kYoaRsxR2x19TCpW29XQ5ntGtSCGy+Na4ueMgvhd+NZd3Yqp9mPiyamC5Y3F6TudRgnnEVWM2u5FrFqsSY8JZC5RmJfEhX59xflTs/pRn6JV7QowfTXsHYRtc7C2y7MflE+SpM9+9rOSpGOPPVZSiUukOLEknXnmmV3vp0+/9a1vNc9dcsklUz7myVCbh7E8Se07WFMYk5piQp8xtjHORepV4WtJMplTzAmUpVr5ppnGcA0KzpUYpbgz0InKkl9D/ZIujgsUG9RCFEbUZ6ms87XdihAVpbbfFqB/PN7vV7/6laTiQWIdGlXsWipLSZIkSZIkLeTNUpIkSZIkSQtz1g3nsh/uBdxwG264oaRu1xgyHwHI/K3VHKImXNu2d9wUBKS6pLjUUktJKrIhcuEwifXe/LyQUXEPIRfXktNxrrTxfsaNF7fIu7uK9//2t7+VVNyck6kDNAoYr1pSyuiGmwzuCuB9jD/97cnp4vbb2nbcQYErgzGsBfZHtwfj5G5KXDW46Kjf5S5Wl8uluouEoN8YYOyfw/fHzxsmMSEex15bP3DJf+QjH5FUNoNIxZXG+rPvvvtK6r7OCIDFdUDdt5rrrV8txunC9eoB/oxz3OZeSxjYtoWc56KLtZYWIG4Pb0srQoiFb5Kg/XTd5IOGvotuuAULFvS0feqppyRJu+22m6Ru12gtieU44XrFRUsKmueff75pE9fKNtdoW4hHDOxnbvGdknTXXXdJko4//viutrUUDcNgdsy2JEmSJEmSWcqcVZbc6mObMhYvW6Vd0UFJIjU9d/9rrrlm0wYLnJIEJBvzrYmUA8ASR8Eh8Z/UnWpAKta7txk03GVjKbq1jorCX15r28JfqyLOc1gEvN8tBPo1qiquJo1TWWIsasoS1k3Ncu2XTqBmFWNtEZzqKh8WMlvIhxmkihLI35olzrEybg00J7oAACAASURBVDGxq1QsN5SlZ599tudzaM81hKLg3xnnJtdQLcC7tjFgOkymPEkMPCXlBwqzJF111VWSpBNOOEGSdMYZZ0jqDvAmSJ7+JgWAW77MBa6l733ve32Pa9CJXOln3/IPsX88lQGbWGIixZrS1dbf/QKYa9vMAVXDk1ty7Y4zBYmD2hj79b777utp+9hjj0nqTewrlY1JswXW7rhG+Xb+bbbZRlLvb0Nb8HZtbsTviCkEJOnxxx9fbJthkspSkiRJkiRJC3NWWXJfL7FGq6++uqQSE4HSJPWmSqeNJwLD8iLeZuutt5ZUrAGpWDMoVVgVL774YtPm6quvllRiF7jLHubWUO7s+euWeb8SDW4ZogDEVPIeOxKtxpoVgZVIX2IRepthxOdMllrhxxirVCsLAtECamuDiuAWMMWZUZaGqbKttNJKksq1wlh6YVeOn3N/17ve1fUeqVjylEChwKsrH5xXVDi9TSyuWtsCjio86GLLMc6GdB5SUZJQzmh78MEH9xz7FVdcIUm6//77u45XKuNN36Ewe3+jLl933XWSpOWWW05SWSukoqZwrKQgIKnsdGHN83WI+CXWM6htx47pV1wViWpTba2LKQiYVx4XSWFhYrt4jyvBMFtK4hC/GkvXvPrqqz1tiVmqqYZce7OFfjFht9xyS/P4wAMPlFQ8CbXrtt/vRVsh3VqpqPjZzLWaUjoMUllKkiRJkiRpYc4qS16OAYsXKwlLzC067nyxePFzup8YS4VddQ8//HDX50vlLhYlCQXHLXG+Y8stt5RUrCXipYZBLLXhd9soCvEOv1bOId6t+918LAcCtSKXMe29W49e4HhcuJoRz6emLEVLMCoyUrF+o1Xk6sOodm5IZW5vt912kkoMjZ9LLEGC8uHxfjHOBjW1FtPFZzNXUJxqr4GrfFw7rgrPhKh4MM8peioVJZlkkieddJKkumWNsoQa5X2Jiso8YP3xdYjH+++/vyTp4x//uKTuaxGVh52bp556qiTp0ksvncwp94V56UpnVHvAk/UyR2IcoisDrL1tqixjEMtRuULUL8Gg7+Crncc4QXVkrtV2lEJci33tnG274frh1ybHzLnWxj2q55NRlrhOvEiu74j0tqNKTprKUpIkSZIkSQt5s5QkSZIkSdLCnHXDLVq0qHmMdIu7jP/XEtshmbKt26VAXAYEdONaW3HFFZs2bJtEUo5Bq5K0xx57SJIWLlwoqUiKuEGGQazs7LI1LsOYFMzl0eimwF1Uq8kUXRvu8otBl/HzpHrV82ETaxfVAlBjyoC2pHe1NtENQ3/5WDBfJrOlfabgSmMTA+4dPx7OOV4Xnh4gboXnOvHrK26giMcglb7iu3BPugsIN6DXzJoJm266qaRSg4356dcrY4BLjHXEXeskx8MVxTZmd+Oz7uCaxV3priVc8rShf7wPYhLR2hb0meDXaz9Xhq95XK/Mm1gb0mlLK8B8iedHn0q9NQX5nNpaNc6NIg5zijUuplhwWHfoQ3fju7t2NkD/xnFmDktlwwxjG89PKv3AudLW28TfL9x6tXWA12rzZ5ikspQkSZIkSdLCnFWW/G6SoMsttthCUrHo3DqKyfKw5FZbbbWmDVbe3nvvLalYuR6gu+qqq0qSdthhB0n1pIvAc2whHfR2aCcGyLmF3896rB0z6gh3/wRqO6gI9KWrSVgdVKOm71zN4H2jDND0tPlS91ZpxpBjxMJzaz8GdMeSL/5aHOdaYsZRJObk+ONY+DyI226x7L0N54hCwmueniKWsuBzPDVH3DSA1eifw/s9CeFMYGy5XpmrlKSRSj8x7hyzqxxY2Rw7FrUnrKV/YskP70vWmFjWwdvE1wZVLikGIPv5kNgXPLEu/RKt/FqAbtsaF7fW83ke2BwT99YUyzZlaxy89tprksraxxzz5I3AWNbKzLi3ZDZAWRP/HZW65wFjxzVTKxEWfy9qv0exlE4tVUTcxMTnjCqFxOyYbUmSJEmSJLOUOassoR453PEST+Dbg5944glJ5c6eu1FiNCTpwQcflCTNmzdPUrFybrzxxqYN5U74Dsql+OdgKWMdc9ccy6AMklh6xO/MeRzv6GtbLqMVUCuAG7cJuxWBxR0L8bqKFJNjjgLmBlaJl3OI26bBzz2ea638Cc/F/qkVrkV9GKa6RjqAGEfkahLHjIrCX1camM8x5qCmqgHn6VYobZhb8bikco1cdNFFizu9SUEakahq+LhxzmxN5nxrcW1xbGtJbSNtCiXUlCXU837zc6q0xRjF7yChqVRSK7QpSzHpZy2eiPe3KcseD+PH5f3MGAwzye9U4DeFMUWJq8Vm0i+kT3FlkkS1swV+Myn/ReyrjxtjwXO1sWW94bV4LUq9iYFrY0v/xITHnnpkmKSylCRJkiRJ0sKcVZZczcCnyh3s2WefLUm64IILmjakpGcnD7E47i8nfonXjjnmGEmlWKDUu0uM43j3u9/dtOH7t9pqq67vfPLJJ6d8npOFO3wsFvf9ozAQs8Jdu1u3cQdH224t2vI5HnOC8oZVhVLgluagC4ROhrhzopaYsV/iSW+DSodVM5nPqfnvUbYGlXyxRix4u8Yaa0gqY+TQLzXlA6s+xmTV1BCoFSPG4ub9taLPfM6g4jewRldeeWVJxQr1MYnxebzmcVNx3rAO+OfEnWBR0ZVKf8T4DbfEa2VyBgFrRE31iTvdnJh4sGb1x0SM/Uos+XfFvpR6FTfaugpZ2001Tl544YWu/08m6S4xk578c7bFLMVyRIyXK6hcu3EXWy0ujtf43FpcYvxOH2P6GSWPnaiucA+TVJaSJEmSJElayJulJEmSJEmSFuasG+64445bbJvjjz++eUxdJbb3IoO6JI27ja2gBx10kKQS4ChJm2yyiaQiISJRU0Vckq666qquv6MgBm+6exFZnzYcey3hJIn4aq4WpNaY7NNl9ChBI5V6m1Fsm494IGUkugVqAfD96g/VXHVRFq69F/l6mG44+PrXvy6p1Dwj/YVUJPWYBqImoxOg2yZ7c65RTpfKvGE+8rnUYpSkL3zhC1M4s8XDXMNN/tnPflZSqc0mFZc15xWT6Em9Cfpq9a2gXxB37X3MOXfBMBa//OUvJ3WOk4Vr28efcSJEgHM/6qijmjb9qsZPl/h5tbkGjImvH7jkYgLLcRFTj7hbuR+08b7sVxdvXMSkqoyN1zilbuGJJ54oqYR8+Bof3x83SUilD1kP+T/pC5y1115bUm+twWGTylKSJEmSJEkLncVZ+Z1O53xJO0t6fWJi4t1vPPclSR+XxH75kyYmJm5447UTJR0i6f9IOnpiYuIXiz2ITmdkUsPb3vY2ScW68krG3KHyl4rHnlyMLdUEydaCZWcDbqHFgGPu7L1N3EJOsLwnpeQ5Pi+W9ZB6g0cJ8PZttONQlpZbbrmu/7s60k8p8XIXEJMIunVEf9Cv9J1ve47bi/21YcMYs/FAKqqKqw1St7UW+wfrz48d1SgqnJ6mgP5EvcIKvfjii6d6KgOFc0d9Ytu8rw1ReSPg268PnqMN88HVVtrwXK0N14onTh0EbCog2N2/A3VvtpXcYHOMpzLgGnr66aclDb6fpsrqq68uSfrKV74iSbr66qsltc/r3XffXZJ04IEHNs+h0jzwwANDOc6p8va3v12SdNhhh0kq6XG+/e1v97RFheK8SKkj9ZYwwrPhGxj4TWETEmM7f/78nu/CA8SGFbxGfoxTZOHExMR7FtdoMsrSBZJ2qDx/5sTExLpv/ONGaQ1J+0ha8433fKfT6cyOZBhJkiRJkiTTYLHKkiR1Op13Svp5UJb+74mJidNDuxMlaWJi4tQ3/v8LSV+amJi4fzGfP3qpIUmSJEmSv3QGpiz146hOp/N4p9M5v9PpkJp6GUm/sTavvPFcD51O57BOp/OrTqfzqxkcQ5IkSZIkyVCZ7s3SdyWtJGldSb+V9G9vPF/bJlFVjSYmJs6dmJh4z2Tu6JIkSZIkScbFtG6WJiYmfj8xMfF/JiYm/j9J50ma98ZLr0jyom3LSnptZoeYJEmSJEkyPqZ1s9TpdLza4YckUcfjWkn7dDqdv+50OitIepekB2d2iEmSJEmSJONjsUkpO53OTyRtKel/dTqdVyT9q6QtO53Ouvqzi+1FSYdL0sTExFOdTuenkp6W9P9K+uTExERvIaIR4Qm/YiD7u971LknSeeed1zxH3Si2wpNQcc8992zavPrqq4v9LrZP12owJUkyd4k1E6lLKZVt06effnrvG2cRbetiG5zrscceK0maN29e89qFF14oqaQg2GWXXSR1pyk455xzJEl33HGHpJL8N5n7fOADH5BUUmMwxlPl85//vKSSfuOss86a+cENiMXeLE1MTHyk8vQPWtqfIumUmRxUkiRJkiTJbGFSqQOGfhADSh1A0ivubtsgARglTSTp1ltvlVQS7B188MGSpJ/85CdNm0984hOSupPR9YNEdqkwJcmSiScQ3HDDDSWV9aNWVX06nHnmmZKKoiP1KlyDgkS8e+21V/PcpptuKqko5iTNfM97yt4cVCfKkqAMPPbYY02bRx99VFJRm0jwe8899zRtKBE17kSTSX88qe03v/lNSWXcSUJLcklJevnllyWVsSUx9K677tq04bebxKngv50k8FywYMEAzqKLoacOSJIkSZIkWeKZs8rSZPzuW265ZfN4n332kSTttttukkpxXC8z8Mwzz0gqFhTlD0j7LpU7Xf4+8sgjkqRrr722aXPJJZdI6i0zkiTJ3IQ1gfI273//+yVJP/zhD5s2rAlYxxRL9bIOqM18Tm3t4jmKiVJuaZVVVhnEqVTXzo022kiSdMopf46gQAWSilJPKRSO3a1+yglxzqyvrhChWqE6Uf6GPpHKOX7xi1+UJC1cuHBa55gMD5/z2223nSTplVdekVQUpre85S1NG+YEv4fEBPt1gaeGuUVbSqVI0uWXXy5J+vSnPz2oU4FUlpIkSZIkSWbKnFWWanzmM5+RVArtuSLE3SzxBPhW3/rWtzZtKOzJ3S274by4JL7YaP15UVost6OPPlqSdNddd3U97+9PkmTugaJMoVeprBusCbWYRZ6Lr/nagMVNzBPFQVdYYYUBn0WBuEwKg3tBUophs8ZRWJkiwFJZV6FWrJvziuqaF9mmP1l7DznkEEndxbqT8XLFFVc0jymYi4eGAtz++8ZzzPHonXGiaukxTL/5zZ+Lg7jHaECkspQkSZIkSTJT8mYpSZIkSZKkhcXmWZoL7LTTTpKkj370o5KKO80TSCIF/ulPf5IkLb30n5OQuxSITIj8jbToqQgIPENm5LtcdiQobf78+ZKKbJiutyRZMmA79DLLlDrhrB+4EnBBeQBzdMO5+y22wW31zne+U1J30CzhADNNIbDJJptIkv77v/+763MIyvXzwBXGd/oGFtZM1lMC4vnrbXDf8XkEeku9rrmdd95ZknTllVc2bYaVNiGZHO56ji7Z2qYGfkeZ14ybXxfMkzj3ff4M0w09GVJZSpIkSZIkaWGJUJaOPPJIScXaItgQNUkqFg/Bl1hSL730UtMGK5GyJwQ7+p00d8VYQNwJewI67qSXX355SSVYke2VyZIDJS6kMg9JdJrlHOY2bRsyCGz15HusE7wPRcbVa9aNqCj557P5BHXlzW9+s6RSUkKSfvzjH1ePa6qwNgGKO+ujVNZDlCTWPFQFqVdNY1MMW8m9Desyn+frNOfDX1Q1JxWl8eLjzu8eChCJRv03s99c9+d5zG8naSa8jW86GAepLCVJkiRJkrSwRChLWENYQG1bdnkOa8b9piRLwwJ605ve1PV/qTcugTtq0rVLxReP5URyy1SWZif9YiDalIXrrrtOkrT55ps3zzG3iJVjPnqxZlL1Y7ljLbE9Wyrz58EHH2w9vmT0ME4rrbSSpBIDKZXxR9muxWaAx2JEooIDJ598cvMYZWmmkF6FY+b/zF1/jXOPyQX9WDkvFKXaln/6BRXCr6HnnntOUllf3/GOd0zzzJJBw5j6dn7WJmKV+A31+R2VJV7z52M8U+29/MaivHpc1ChIZSlJkiRJkqSFJUJZ4k4Ti4e4JLf6eI07VawlvwOOlhx3rn63i782WlDum+d7ed9SSy01zTMbH5NRM9Zbbz1J3UU311lnHUnSiy++KKlYwPfee+8wDnPksHtos802k9Qds8K8o8QD/vxjjjmmaROtqlp8HQWdt912W0mpKM0mKAJLTA5lPaSS4LZfXKPUm5iP//s6xPuYP6iQq622WtOGcivMlelCUVPWL+aj74aLyQM5Lrf64y5h/u/nFXfTodyvscYaTZvHH39cUol9InYlGT/8jnl8LtcB40bZnFqpL+YCc8TnRkxC+fzzz0uS3vWudzVt+O0ltpjfmFGRylKSJEmSJEkLebOUJEmSJEnSwhLhhsOFEV0auEWkbneJVKRAr10Ug7ZrtWtIvkWwGd/hbaPE7pL2XKHN9bPLLrtIkq655hpJRTKVSv8gn7Kd3hN7PvDAA5Kk//iP/5BUXAmXXXbZQI59qkRZOLpKnNNOO63r/z5/4iaCWEk7tvfvZnu41F0razZAv/h5SNLGG2/cPMb1+Mwzz0z6c9tcvTFpY03WHza14/rgBz8oqbitfGNHTMjIsdcCWaHWB7yfc64Fsl5yySWSihttusRkgPH//v24zVhn/fqILhbWRT8v+grXDdeCb3whpIJ1BJePu+PGvYVcKscp9Q80riUcbVtXYzhJ7T24onBpjjLYGRdZbW4QrF+rG9hvzfP+YW3h/QsXLpQkrbLKKj2f49fcKEllKUmSJEmSpIU5qyz59kXgjhzLxxNjoWywtbFm0WEpcZfLa7WkcrESt1fd5rOjRTYXaLP2URK+9KUvSSoqAonE/DGW5VNPPSWp27pgezLbgt/73vdKkm666aamjW9dHjbxnKOCIhUlgW3OVMB2RSgqipxzLQUB1lkMjJVmR1CrW49RYSPw/OCDD26ew5JEbbz66qt7PpP+iEqVf1dNtZhN7L333pJKuRMPdo3nU1MWIrW5FhVu5p4n0CXh7bx58ySVNBNTBSs9jo2XVuE51k7e4wpInNesfZ7AEBUkbm5AlZTK3Odc6QsP9J0NypKfe1TVoKacwKGHHiqpOwWEq+/9QFGin0mA+9Of/rRpg6I9aCjj5dcrx8z1yvXg6njsn9qmBh4zD0kH5DB/KHuyaNGiGZ3PVEllKUmSJEmSpIU5qyy59R2tGe5ufTs2d6685koQYC1gDdWUAR5zl8t3u9Ll24ml7rQCs4ma4hGtI7bKS9IVV1whqViYxB64lYVFGv3K/rlYI6gRFNJcffXVmzYkbxwFk9maf9FFF0kq58y4u8U4GQsKeB996fF1nqByXNSUnX/913+VVBTBJ598snmN2IIDDzxQUimAigUt9Y87qj2//vrrSyrzj7nxs5/9rGlzxx13TOJMBsPaa68tqZwniUdrsSsxVsnHP55rrfwJcwsVgbnh38U1tOOOO0qavrLEWsd4c/16MkmuZeY1qVFiLIpU+oC/vs7SHuUDhYC+laQbbrih67s4rprSME58zei3fvjzjOlyyy0nqZSu2W+//Zo2X/nKVyRJt91222K///DDD5dUts/778+wlCU8AP67ynfF38NaWgDmAm18PnNdxPhej8XiGsEzMWpSWUqSJEmSJGlhzipLtZiluIvJ415iAd2aMsDjGLvkxJ13scSFVO6qsc48dmpYRGu2duxYdrU4mcjZZ58tqSS/k6Tf/e53kooiRHySx+3E768dD30YjxErWRqtsoR1TX9gzWyxxRZNmz322ENSUQ05B+/DqCT0U+ukMm+wpHz3COVxRkncZUMsjCR9//vflyRde+21ksouxq222qppQ9LRX//615JKf918881Nm+uvv16S9Prrr0sqKp2fL3ERzC2uM+ae746ZibJUK0HSFidFTAjjxBj7uMVYrFr8X1xj6HdXYKL6RF+4JY7aM52dQa5ioeqyVpJYkyLifjzM+bi+SWUeE6tSK7YbS1qwZjK2UklQef/993e19RiqUTKdUkP0qSfr3XrrrSWVAszPPvuspO54z89+9rOSyo5L2rzwwgtNG5Kicp1xfX30ox9t2rB2TmVn6mRg56XH6fE7zI5G/s/uOKnMATwsrK+uUPHbxPVx1113SZL23HPPpg1jMY71UUplKUmSJEmSpJW8WUqSJEmSJGmhMxvqTnU6nSkfxHbbbdc8/s53viNJeuyxx/g8SdIvfvGLps1JJ50kSXr44Ycl9bqkpCK1xqDHWoAuAYcEWhL8LJXtnLxGQOT2228/pXOcClMJKgV3RRx22GGSpBNOOEFSkVHdlYnUHrcb1xLqAX1YS9EQ+9eD+9zdMmxiMknwY6Y/cJfgeqy5bvpV0JZ6ZX3GxscIty21mHBbTYZa0P505gauN6kEdDN/2Vxx/vnnN22Q1En/QODpqquu2rR55zvfKanXTfX73/++5/txbyHZU3vK3XqDJo7Nsssu27z2xBNPSCpuk9qmDa53+reW4BGiy64WEMtzuHV8PtE/BLu6O2dxuGvs4osvllTGgIBj1kmprG3U5MId6ClROA9cqzUYS96P+23DDTds2lD/jsS1rAm4fiVp/vz5iz3Hfng/x+uT9WwyCR5JIeLv4zoheNvXsIceekhS+S0gHMTXDzYzkCahlqQyHhtpFAgcl6Rzzz1XkvRv//Zviz2PqcC8WXHFFZvnGHfWx9dee01St+sQ1248H+9/fmdw8TEnfNMP38XaQvqOAbBwYmLiPYtrlMpSkiRJkiRJC3M2wNuDwwg05M6Xu1OCxCTpa1/7mqTe7a41iw5q5QqAgHHubknPLpVAPcqAzDTAOyY59GNG6YjBxLUA7/e85883z0cffbSkUiFaKgF6nAeJ4txioX/iln+3nGNCPo7DLSLGC4WKoFCvqk7w8C9/+cue8xg0cdyxjj1on3NGUeS83OqLQYpt8L62pIQEN6KcToaamjWZuQE77LCDpO70BVH5wZolEFUqwZ8cK9bx3Xff3bRhnLEw+Q4UNKkk+7z99tslFYVhFMS+82s6KtHMZ587nFfb5oZYvqWmLKFa8h2sMagSUpmbU1GUwDfHoNww57k2UZH8mLm26QtXkWJC1lj2RCrbzHmOc/fr7CMf+YikMu7000wTtfYr2ePUAo8p7UTKCtId7Lbbbk0b0mRw7s8995wk6YILLmjaoMZwHiQVdcXskUcekdSdkkPq7kPUJ8aC3zov5zWsrfXMjVrg+Oc+9zlJZQOMB/+zXriiKXVfb/yOsW6wsciTdo6bVJaSJEmSJElamLPKklu++L6x7Ign8HToWLVx678TraKo6Pj7+Bzumr0YJIoJd9d+lz0d4pb2yeCW2AYbbCBJOuaYYySV8yEOxMHqxP/uljPnTL+gtviW5picLL5X6rW8GS8vyMvW88koS23lJWLpmprywrZkSixgpfk50J98HmNaS1ja9l2AxRyLrkpljlFWpk1Z6peuwI8V6GdPB8GW5mj5EnsgdW8V7gdxVahhZ555piRp3333bdqgFmApE8twxhlnNG2efvrp1u+pxWRNh7ZyLo8++qik7ng9vgs1lfgk/5wYrxeVGH+OZKSoSN7HrGesKVxnrlDPpISSv5djQ91B/SEuxEFJYn1zVSsm8OU8a+oT85u55koF1xWKF2veTJNS1hQl4unYjo+yvemmmzZtdt11V0lFUaLUj695qB8of8wDj8WKiWbpFxQmqYxzjItcb731mjZx3an9JpDQddDU1hgeb7bZZpLKGNcSlsZyN66U8jnMO2KSXVmaTEqWYZLKUpIkSZIkSQtzVlnyRIhY6VjOXsQPuFuPSeBqO93a7ly5K+ZzsAY8iVv0j8cU7tOFu21PyoWlS3+gDLkVilWDxcpdfy12AYuwlrQzprKv7a6KZQpqMUuxLAR96N/Vz5KsKQvTKbpKjI1UYnKI0+Jza2oYc62t1APvr6lGUHs/oFpMJk5jMn3wqU99SlKx/nx3HXMCFXadddaR1G2d9tvh1KbyoGJ6olEsZHbOoCjU1KR+yQCnak3Wdr1K9f767ne/K6n0gZcton2Mj/TjoZ/iTjlXn5hjKLfg/cTj3XffXVJR+XzO9NvFOBlquyE5RtYREiL696IoxV16teOhv/zc4+5Z1igvEEy8TkwwPJk4wDb4LhRPP37OC+XDd1mRjHSfffaR1LubTSrjHctluTpLDBjnzrXt6hzfS2JW1F5X+ehDlEhPigoxNmhQtI1BjNdsu05ru0PjGukxc5P5/lGQylKSJEmSJEkLebOUJEmSJEnSwpx1w7kkjTyHvFdzw+HaiPK5S9J8Zqz6XQN52Ld1xs/huPj/TGV0tv4TkCiVbaJIwbUq5kAwIX1Qc0HGPnDJNAa8174jyq/0kwdE0q+4rZCUPcjVg//aPn9x4M5jy+2HP/xhSdLee+/dtInJzehLdz/RV9EtUJs/9AvnXHN7tLnoosuPGnWeCiOC+8RrheFGwsWK24tNCVIJsqbv2bLP9SIVF0SkNhZsJsCF4O5Oxh0XBkkfCWSXSvAx/UywPQHIJFGcLJO5zkjEeMQRR0gqLkncIVLvhpDa+OPWxjVC//h1j/sN1y9bpX2+ExQfg7h9Pk4nZQC4e5nrO9Z2cxcSbvyYtNHXx+i2Zz7Vam/SP7RxtxHuP86dc/a5Rj8TjD4ZSBTprkPOlWD7WAPPj5kwC47Lx5Q5z3nxuV4fjeOnn5k3XvOO78UFzjrkKVXi+sM4uevXXYSDpK1OXgy38N+Ntt8koD/i+tjWdtRuuVSWkiRJkiRJWpizyhLWgFTuMPlbS4OOZYg1NJkAtDblhDtoDzQHrOFY7sCThU0lVTvBgF/96ld7XkPZQm3Covc08VizWDFYZh54znlgocQ0AVKxplwlktoTK/LXrUDGAvWC/3vyw2uuuabnLBIBUgAAIABJREFUMyNsKyVwmTFxq8+35koluNkt56gAcjw1VS2qRa5cxDIuUfGUyvyJlngtBUG/4GTn7LPPllTGz7dhM+4EcbMV3rcxYw1j3VP13VM2rLTSSpJK4DHnzjZqqcxDrGkCmVGupKIkMFdrGzJiKQTmLNa2X/dXXnllb4cESJb3oQ99SFK5Bj3Amu/68pe/LKkEp7uSExPU1kra8Bh1j7a+keKWW26R1F2uKYLKFJNa+ne58jdVXMlh/FHRmPs+Jow341Ur48J1EbeO+8YOVBnmCGPpbUgfQkoL5qz3IXNiKsoSfUjJHamolWuuuaakMvd8A0RMk8C66EoOc53rpJYUknNmbOk7krBKJVXEr3/9a0lFAfbNCFHdoQ9d5VuwYEFvBwyAmrLEXCCxbFxLpd70BrVkzzzmPPg8V1dj36WylCRJkiRJMouYs8pSbcskd7wU1HWwCNoKPfJ+7m6jOiIVC4q7XI9rAPzbfGfNPz0VZYnvZEuz+9RJl//UU09JKgn1BoX7jidTIDQm9Kxt0Y79OhULgbgtSdprr70klb7EYnaLpV8/e5toqdT85fE8sJw8Zi0mXaulVoD4Xd4HWKpw3333Vc9Bkn7yk59IKuV8vHAt1iz9gsLkfcJ3ceyoP64+UTD34IMPllQsb99aj5WPaoTa59cHx4Eqwt9agVhew6JHofIt7f1AlZCk0047TVKx1vlcVw9YEw4//PCu/zv9xtTnRVQNmRtXXXVV04Z0APFzvQ/oS5RWFA+P6Yvq7lTwz+FcUSY5di9BQtwf58z64+pWVARp67FeMUaJddbnCOfO59E/rij6+jdZuE48po8SJjzHeXpMKMqGxzrFY0D9oj9QiFzl9aTFowKly5P9zgTWPFex2OJPH9ZiVLmmo6fG53wsZs7n+XqP2j2dVDGDIJWlJEmSJEmSFuasslRTI7jzfeihh3pew3LC2m5TRWLytJq6gk+/5p+mkCLxH3yO+92nAvE1fKdb/dzZ40OvFXON8VpYRW4dxV1wWEltSSlr6e9pg2VRS/DJ9/KdWApuqXLOUUnwnVyoaey4qo1pHEuo7c7D4uVvrUwFSgfv8V1JPIclhWpTs8A5Z/z5rmbwOVi6zKNa8kbiE973vvdJKrE5UlGCKB2DdeyqLMeDkoCV7PEgKCSMN/N43rx5TRv6k91dxO157EuMR+D/3oa+p79RcIk1mcxuyI033rjnuah4OjHGKKoa/v4Yz+gqDWNJf6EMRTXJqSlLqGAxDshpS2q6OGoxS5wHY+tqxOabb971fsbEVZp4LbPm+HHyXbyPNj73WTspVBuLbUvTU9VYx37+8583z912222Sisr4wgsvSJIefvjhpg3XLmPC2uXrIvOZPkQp88S6MdappqoxB1CzWFu8nyHONZ8/lFkhXm9QylIN1t6otNaU+/ha7Xc17obzUi8oS6MucwKpLCVJkiRJkrSQN0tJkiRJkiQtzFk3XC1hJDIfcqqDRBol9jZJj9d863YM1q3V7yLAnC2pUEtgORWQq122xn0St/W7dMtzcbuzS8mcV6w9VHNFRMnVgzijiy5uCZVKf/KX1/xzYhJAWHvttZvHSOQEQONq9ZQIseZVbdsqfcX5kdSt5j6L6QD8OHkftZwISvZzjy6VGBQuFXcFfUjAac0NF/FgYn/seN23ddddt+t8SN7p50XtLlw0uEYff/zxpo27LhYHFdxx2bkrO9ZRxJ13wgknSJpcH9SI9cicfmPr7p64JRo3iF9DMcXDkUceOa1jjalHYtoKqX2jyuKobfnnvHwTSmwTr0l31fJ+jjnWkZNK/8SAX18XGV/mAW18+/lMam2yyUEqwcOEMuBu9LQArOEESzN3cRdK0iqrrNJ1jJyDzzX6B1c4myM8pAK3fVyDffs87kDmHeu/J8C8/vrrJUn33ntvv26YFjEFgFTciLV1FWK6Dc7Lg/ZZF2NoB/3upBsuSZIkSZJkFjJnlaWamkGiudqdJ0GNbQn+4nb3WlBo3GYeSxJI5c7+gAMO6Hq+Fqg3KKJaM5OkdbOZ6667rnmMgvPe975XUtk274kQseSwvGoB34xpTPXgfUhQK1bjnXfeKUlauHBh0yaWGUBh9JIffE60xN3aRzXA4o7blmeKq0BRETrvvPMG+l01SK3B30FTC36OyVE9yDm2ryVk5f1R0fH1hPItpGbop+w5NUschSFa204tdcpkqW17RyHnunDVKKrOtWOOCWFrG036pV2ozf249vrnzFShj6DOXHrppQP93CWNmjeGeRM3XPmcjZuGUNprSSn5HOaGJxEdN6ksJUmSJEmStDBnlSX3n/aLP6qVq4ht2spM1NpwxxwLBnrZEyxmYk1QQKaTTC3pzxNPPNH1tw3GDZXHFUF86Kg+WNWuXrp/fbKwddfTHcTUEzGFgLchfUItViDpT62/4vZlXwdictRa6oCoptTUJx6TIBQ8vUBUhGoqDUpHW9HlmRTSdTUMhY04IP5fK09En9W27nOObSWDiAVifOJf/37g3L3QcGyTjIaax4byWtFjUyugHAvUu6clqk3MiXEloKyRylKSJEmSJEkLc1ZZcguNO9YYM+K7JvpZ57W75Rgj4He33EHHu+TllluuaYNqgPUXd5wkowfrht0wvitmWNx0001D/46kl7adslDbCco1jXLha0NUgGo71Ig1+s53vtPVdqrKILEdMWbJj6HfbtHJUIv3jPFEHheEas76Gndr+rHy2TX1B3U2qnKu8vI+dl7WktrmOjoear+VzI2Y0LkWsxTnlv+Gxxg1rpnpJnIeBqksJUmSJEmStJA3S0mSJEmSJC3MWT3TJUHkvSeffLKrjW/9rtWskeqBaDGg0SXFKAuTnKwmDbMtm5QGtYR4SZIMFk/nACTPw0XnKQC43uMWed+QEV1irC3uJjjrrLOqx9MWpNqWOiC6/H0dos10oKaiVFwhnCv10jzJYaz/xTrZVjU+Pi+VIPCYqsNTY5BQmOSLtWSHzzzzzORONBk6/LbhNos1QaXeRK5cgz6f4+8n82jQaSJmQipLSZIkSZIkLcxZZYlSCVJJjEWVb/DAykWLFknqrSxeS1LJczVlqWblSWW7r4P1SRDcoJMLJknSC9XJJWmttdaSJB1zzDGSpG233VZSKVsh9W7n57p3lYTrPQYg+0aBL3zhC1M+1lrQLOtGLA/hweS1IPbJQjkV/w7KbnzpS1+S1H3uq622miTp0UcflVRPpBrLVKA+taVdQdFbffXVmzasvSSf/chHPiKpe21HdUrGD3OVsUQt9N9e5lIsGeS/oTGpaVuainGRylKSJEmSJEkLc1ZZQimSpHvuuUdSe4kPtq2i7mCZLb/88k2b6FMFvwN+/fXXJZXtrhQTjaqWVJIl4v9PX3uSjBbiGA855JCu5ylNIhW16aCDDpIkbbjhhpLa4yV++tOfSpL23nvvGR1fW+oSSj0Q++iJImcSs+SgiFPG5/nnn+86Bqmsr7/5zW8klWSCtYK8rJ21Ui204S9tbr755p7jYq0kvmomhYOT4UERbBQl5oaXnIqF0mOyVX9/VJ+WWmqppg0q8WSSEA+DVJaSJEmSJEla6NQsm5EfRKczsoMgeeTSSy8tqdwZS72qE+n5vdQFafexwFzhSpJkdhF3cCVJMjhuueUWSSVhaSw0L5X4vlh03u89UJuInUPx9Pi0c889V1JRQQfIwomJifcsrlEqS0mSJEmSJC3kzVKSJEmSJEkLf3FuuCRJkiRJkjdIN1ySJEmSJMlMyZulJEmSJEmSFvJmKUmSJEmSpIU5m5RyMqy66qrN4w022EBSSWy11VZbSZJuvfXWps1NN90kqSTEIrHWoYce2rQhUd3555/f9X4Sts1V9thjD0klQd+FF14oSVpmmWWaNrvvvrsk6aGHHpIkXXnllZKkl156abGf7wnIZsM2bi/DEOP2SKzmJR/6lZegQKvUXfpirvHxj39cUr3gNM+RboP/e/+85S1vkVTGmYKoXgboj3/8o6SSouOGG24Y8Fkko+CAAw6QJO22227Nc2zxvvfeeyWVa8GLh++6666SpD/84Q+SpFNPPVXS3E84STka0tJQXJbUM1IpqcPWelLQcJ1IJXmxlwqZa+y0006SpM0226x57re//a2kkmiUVDwUaJakjTbaSFJZR+644w5J0iWXXNL3u1ijRhV3ncpSkiRJkiRJC0vEbriVVlpJkrTvvvtKKne1XriWUigvvviipJL8auONN27aoChxd4sl/MorrzRtKCaJ+kDZBLeO7rzzTknS17/+9Z7XRoWXInAFQCqK0Morr9w8R3+gHrz22muSugsZ0i9YjVhAJB2TpCOOOKLrO+ZCUsBtttlGknTGGWdIKhYPfSKVucF50L+k8JeK5YRicvLJJw/zsAcChUpPPPH/b+/No2WpyjztJ1ZXf/0HdpVfl0pRqKCoTMpUiCiCTAKicHEAcQIVBcURcUAtBS21LBWVUgFFHFCZLEFABkVAUQERFBlFJgUB0cJS2+pa37eqO/uPe57cb+6zT3jOPVPm4X3WuuvkjdwZGbFjx87Yv3d6O1BWw3HV5z1j4jhVtjh3WK5A7C+VXIDf//73QClhYX9F5TYZD/bcc8/h62OOOQYYnS9gNFnvWmutBZQyLJZo8d6CMl84bixKHOenj33sYwC85S1vWYCzWDyi4u6cpyXipz/9KVAUJyjzqJYJ55ZYUsffkptvvhmAL3/5ywBceeWV075/KVSVmb7D3z6At73tbQA873nPA8pvZmzzuMc9Dii/o5YI23vvvYdtVNqca1Tn4tg44YQTgFLseQHJaLgkSZIkSZL5kg9LSZIkSZIkPUysGU6ZDop067n86U9/AorpDYoErJnhvvvuA4ppAeChD30oUEwsVvmO+9HkpMTq56PsqPlPk82RRx4JFLPDcnHWWWcBxbk9OqV7rEqvmlVi/+ikqInFtkrw8XM6imtyiea8cXBgjA7Md955JwAPfvCDgdHrLV53+8m+8C+UfnE/73vf+wB473vfu6DHvpBY8b0+9sitt94KjI5xGB0byuV+3n6J110cI5rzdttttzU/gWRBOfroowF405veNO09r6mm53gP+Z5zr64Hsfam87NjwnlWc1x87/rrrwdgiy22AKa7Eiw30Uz52Mc+FijHvu222wKj/aN5yjY6PcfaoraxD70nW4EQi2WG6wt8efOb3wzADjvsMNzmvOj19jdTcyyU66yLhybI2EYznGZJ55o4f/hd9o9uDpdeeumczrFBmuGSJEmSJEnmy8QqS4buQ1mZ+HTa99TtNldHsU3tuKoKFZ9ua4fl1srZFbdVln/zm98A7dXaUvK9730PKE/4MRzePutzyK771b9RiTGU/sQTTwTg3e9+98h+Z9r3UhMd+3VG/93vfgeUFUx0WvaYPWdXfXHF66pKbPPoRz96QY99vhx66KHD167ODP021Nu+gBLS7AqxdnaP79kfMcBAdBD2fvU7Tj/99GEbw8mTpcUxb1h/SzX0Hm7Nr17v+m9UXuv9SVQz/F4VyrPPPhsoaQeWmnhsUM55xx13HG4zRY3zq07tMfBFBUk1xd+E2Bf+3jg/q9ZccsklMx7XUvx+77XXXgAcdthhQFs18p5W9YkKtfe542fttdcGSsBHbO/n7YvoAO/Y1HLj/LzNNtsM26zhb0sqS0mSJEmSJPNl4pJSrrfeesCondsQVp9GVY3i6sgncJ9utaXHFbBtfNr3ydWVdfwuv191Jq5AfE/fnEc96lFASYwJcPXVV8/2lBcMla56hQjTn8h9r2XD9q+fib4s9l1MCNra/3LjNYHif+a1VCFyBQSlH+qEjHFlWPvp1GMlfsdyElUjk+LddtttQFnhubqNeF+1zsFz9q/3Tlxdf/7znwfKPazvQjyeZHnQx9Cx2vLFq1WWeE87JpxbnCPiZ3zPeUhiCgLHjarK9ttvP3Jc8buWgpmUm8c//vHD1wcccABQQv/tu4svvnjYxnQr3g/OmTFxpf3jucaUNbM9rsXAlDxabuJ19zicQ/3Naylmd9xxB1DOff311x+2cf5x3ypK0SJR+zo6f7z61a8etvnUpz61Jqc4K1JZSpIkSZIk6WHilCVtxVHN8Gm0TnrWevpWLTAtvxFiUJ7+je7yyfdFL3rRsI2KlGqT3xFVLFdiPlH7f1O6w9IqS6bhd1Xj6j+u+o0OlJayJJ6PSkOMhpO473HEJIxQVnImYjQxWvRZqvvB/99///3DbSbt9NxdibkyA/jCF76wEIc/L0499dTm60g8L30UYgJOGF0V1/5rJuiLJWA+8IEPzOewk0Ukzk0wOnfO5LfTUo1ExT6qEM7ZlktSYYrJLmsfJ/1TYpLUPsVlsXCO2GSTTYBR/0TvadUU1VrnAygqiL9R9k+cJ1XYjK6rfSAjS+Gz5DyvYqYCHJNter28/qo+0WJjG/tH1TD6LLnNffvdrcS3vud3xcjEVJaSJEmSJEmWiXxYSpIkSZIk6WHizHAm/IoOZDqD1Y7H0THbkHbDxC+44AIAnv/85w/bKAcrlZqwz/o3AIcffjgAW221FQC//OUvgVGTjVKi0q2SqZ9ZapR1Tbp52WWXAaMJ0Qwn19GuZYZTavX8lI2/+tWvDttosozVxseRWI/soIMOAmDfffcFipNzHGP2g+ZW/x9DWw0HVlr/4Q9/CJQ+nSRiPSpNz5oVNeNGE0ttqrFfvv3tby/qcSYLw2zSW8zF5FPPxVDGyI033ggU0/9b3/rWYRvnXk0tzjnRoXo5zHDPetazgGKKjgmGNSdqMrQvY0qVOvWIxDYmqjSNh/P2dtttN2yj+8hSOHhrStcM7zlEB31NaV435wbnCii/F3Vi32hmdK71N7PVX7p7aH7zWthPi00qS0mSJEmSJD1MnLKks1lMjBUd6aA46MbQ75/85CcAfOlLXwLgPe95DzD6lLzBBhsAJTHWxz/+cQC+/vWvD9tY8dgK9T4tR4dYK1L7RK2DWzyepaROx+/K5cILLxy2ef3rXw+UJ/xWuHDtxKnD5mmnnTbcZhV71RnVLBOKjQuqPvG1CqJKUAxpr1MHuJKKKRIMYY0JUyeVuHKOpYWgvaq1f3Ts9HqPQ9Lb5M9Tp4popQyZi1NxPR6gzIMvfvGLR96L6spMaQo222yz4WutAkvB5ptvDhSVJarx4ryq4uXvT1TVnDtt6/yhYzOU3wvnTkPtt9xyy2Eb1Wvn8MXEsiYeu3NfdPDWulCfX7x+qsy+V6tRMBogBe1UM/V32Hc6z0Mp5dVK5DlfUllKkiRJkiTpYWKUJUNHfSqNZUa0Xbo6siChqwEoq33tySpKURnw6dan9/PPPx8o6d4B3vnOdwKl0OPee+8NjK4QTLrnNn2nYqi1Nnj3s5hstNFGwPTVWvTp8j2f6FtlT+oyBa4Mr7rqquE2V4muJvQDGzdlqVV+RbUy2tvFMWYfOP7ifup0CbUSF79r3In3jj4Crvqjf56oNtQh4yqLk0Rr7NfJFvvUFe9/fU1OPvnkGdsuZdmKPuoiyXGumOkY4//75g1xfq39/lr7rL/LPl1q/J3Q2uB9ES0JnrM+NXVqFSj9USfUjG1U3rSUOIfGvtQ6sRTKkuO3Lm8UE4R67t739kGcI+rP+5k4P9aKov+POOf6ntfE32soPs2pLCVJkiRJkiwxE6MsPfWpTwXKk360d/qE6sreNjFSyZWTytQ999wDjCal1BZb+6Vce+21047HCCHbxBW0r3/+858D5Yk6Pi0/7nGPA5ZGWbJfXK2deeaZQInki7SiWKT2Z2pFLBjhYPShCtrll18+z7NYWPpW8ipMUQmsE416nnfeeeewjSsdcVzW9vhJwESmUFa/daK4OJ7rVaefib4mNeOiqtTUK+C4rUYfRijzhCrqEUccAYwmXXzve9878vk+/y/fq30OAW6++WZgutq7pjj3tRJO1sfYFyk7m2ta92/0a5opaiyWJ1psYpJd+14/Ru+LqAhpyfDYW2Wg6qSNRr7tvPPOwzZeA4sHa/WIFoD4m7bY7LHHHkBJLFsXoYfyW6fvUp1UEqYnZ7YvYv+oqqlIRfVK3LfKv22i33EsobLQpLKUJEmSJEnSQz4sJUmSJEmS9DAxZjiduJTgWlKppjUT6u23337DNkqBmlisJxMdB5W5lfUMG73hhhuGbU466SSg1AgSE1hCcajWFKUsH02HsdbRYqNU6rlfdNFFwGhIqtJx7WAXZf467LVVE87++fCHPwyUVA+TgCG7nqemVijnqmOlcnoMo910001H9rdQJpLlwLELJYVCNJfAqBnG9+qkdDGsV8dV9zeuZjhpOXgfdthhQEkwG9toUvE+u+KKK4DRpHmnn346MDo31dTmN+eaWGdPs9Stt946t5OagfpebgUnLBS1Ga/PFClxPC42u++++/C1/aIpbJtttgFGHbV9T/NUKy2ArhDOF46bmGzTa6qbhEkuYwoT76F6Ll4MTNbrmHcuj/OaLgZ9zv+21+zWCgKoHftbY6IOsjFFz8UXXzxsc8ghh8z29OZMKktJkiRJkiQ9TIyy9P3vf3/kb0SFxFXW/vvvDxQnOihJF4877jigJN3bfvvth21cCfi0blr+Y489dtjGCvI62unUG8PGt9hiCwCe8IQnAHDdddfN/kQXAVdldQmSJz/5ycPXPsnXCcjiCs+VgA6NKmiRn/70pyP/X0qHxLnQcmA1iMDw4JgsL64SoTgixhX4PvvsA5SVWN2nMP6pA1QhY0Vwj1n1yPOKDseOl3rcqPZCSdsRV8rjTLw/LHMk9klUFk0iqEKpMhRXyb6nsqTS1IdBKPFeqhOFzpc6dUifg3dr9V9va6UFqJWkusRF3FbvbylTB8RUKFtvvTVQjlEH4jgf1GlRWnNDfd+rFn72s58dbjNoxPukVmLicWj9WMx7ydIq/v3GN74BjKbbMcDFuUHFraVMOhd4XlEtVDWqPxfHmN/h772/5ZbqWmxSWUqSJEmSJOlhYpSlPvu2pUzEsPwY1q0Pjiu6E044AYAjjzxy2OZlL3sZUFYKKksxjHrVqlVAsVOrQrVs6sutKIlP8nVCtNe97nXD166UoloAoyuiOqzXVYBKGhRVxuu1lL5Zc6Gl8FhI11VODJEVz9nrHvfjGHP1admDVmj0uFD7D6iuxbQJKomumPVTiKvA2g+lHmsAT3ziE4Gycl9oX6W+OaJOhNeXzsGVavRd8Vrqg2XajbjKtu/sJ/0sY4JP1W7vPcsDOddA6TtVa0s0xeOxEPTTnva0Gc9jLtT+KH1FklvUfd9SmOr9eA2iOlfPLf6/Lmm1mMQ50ONQ0VHd018G4O677wbKdbfvom+P84X7c66IBYzr1AO+5/6hjI2WP9NiYzHh448/frhN64m/ta2Ek15nf4e8H1pjrFY447XwtYmgzznnnPmf1BxIZSlJkiRJkqSHiVGWZkqMBmXl7hOsBQ1jokijc3wSdiUV06L71O7+XPXFpHI+yftdta22RV9kyVIoDfaVKxRX+DFK56677gLK+dRJ5mD6qtHVUlSoTC5mfyy0b8ViYr/USRhhup9FK1rQbfq6qEaMc1ScqzXP2THRKkVQj/GWgtMX1RLH20yfXwhaY7Zetbd47nOfCxSlNBbrdr7w8y960YuA4oMExSdDH4/77rsPGE1Wqoqij6O+OM985jOHbWpl4tnPfva08/J7nZvWNCrOaK9aGWr1YV8pk9lcy5n8mVpRh63El0tFVLqc45zvVXSi1aJW4+sklRG36Q8Z+83EjqqXFnZ3PEF/yaGl4lWvetXw9XnnnQcU5a113erf1Xouje+5H/s9qrLHHHMMsPSKkqSylCRJkiRJ0kM+LCVJkiRJkvQwMWa4mlbSK9FBOzroKltqojP0UqfeuE8lROXzWJunlvNb1ZFrZhOOu9AoF0NxeNcM9453vAMYDQ/vq/Zco1SqmcFKz1DkVMNgowlz3KmTnkWpu3b27guR1hHylFNOAcbPqTtSH9s666wzbbvn6LVtpZWYKeVE3E9M8rkYzOae8hhi2guDPnQi9lxiqL5mL51UNXvF+or2j3OL90UMjXf+8d7xb6wi7/fr4GuywmgabVV3XxPq+7NOEwHFjFKbnltjfzbXoHbejtTmQM0xrVphi0VM0Kn5TXOQQTtxPvB6ey09Vs1qUM6rzyXD77W/TR4cg2TcZwzAWCxmkzTW49D9op4roIwpzYveQ3UtTShjzP3FQIxdd90VgI9//ONzPZUFIZWlJEmSJEmSHiZWWerD1Uh8uq0dB32qjU+39VO/K6i4XWc+Q+1bicNqlqOsg86B8fvtD5/QY9hpvdrrSzvvfuzn6OCoouXKoE6EOY5YckCnWx00+2gpJ75WyZsE6nIJMU2GeL3r1WPEba40XRHG/S9VyQpDiwG22247oCgwa6+9NjA6rlVBvX4qybfffvuwjeO6Tp8Ry7moPnjO7idWRdfJXbXAEiZRCVbtcTw6D8VVtk7I8fvXBI+jdt5vKYv1XNAa+63AkHo/fQ7e0jef2s/333//jG3mQ3Twtu+dxyxrZTAIFEWodvSOalj9u1EncYUyTgws0EIS1T+31d+1GMzk2B+tD/7+9f1eeO85xlTHYlvbuM2+i4mB63JSS00qS0mSJEmSJD2sSGVJe3JfGn3fa6Wkr5+kWwns6qKiMYR4HIh27vpp3/OJ5+VqYS4hwPZTK3mjq8flDHGdLapwjolaPYDp4fMt27yKQqs/JoU+paIe8/HeqdWPlu9LXTJmoXnKU54CjCZvVDlWFTMhZhyXqk2u6D2XqPaI/eN5xfMzCaVKQ6tsjm3sH9WRWFBVFaNWbqNC5Xisr8lcqf1kWr5Bc0k4OZdQ/9rPDcr59O3P67JYylJL1VLlcwzHtBL6ujmm/Buvu/1bl0Jp3RP2hwpMK43HUihLMpekpK30Eh6zbZwf4zmoSNffFZVp72VTAqkqrl7DAAAgAElEQVSyLZUvcCpLSZIkSZIkPfxZZanrukcAJwF/A/wf4DODweCYruv+B3AasD7wC2C/wWDwb93qx7xjgD2B/wW8dDAY/HhxDr9NSzWq0823VmQzJb6M231i9jtcBcRV3zhgBA1MP49WdF5N3xN6vbKMfVlHJsZooXFFXxpXOn2RbjXRfu/qMfo8TBqee59i0VJV6lIorZVvjCpdDIxqM6IPyirW4zPCzCS1UNSeWg2NEUf2ixGk+mrEiDlVJ69/7ScV96k/lIW8LawKRTGp/SvjMetn9d3vfrfZF7PFpJh1ceTWnFffF3F+rd/r82GROroSpqsQrXtwKe+vWiV2DEdV3jGgSud1isXVoxIFRelsRYTV3926F5dSsa99yqLiWl+L2cwbrQLjtaLZUh2957bZZhsAzj33XGC8lKX/BA4fDAYbA9sCr+m6bhPgCOCiwWDwWOCiqf8DPAN47NS/g4HjFvyokyRJkiRJlog/+7A0GAzuVRkaDAb/E7gJWBdYBXxxqtkXgX2mXq8CThqs5grgwV3XrUOSJEmSJMkEMicH767r1ge2BH4IrD0YDO6F1Q9UXdfpUbwucFf42K+mtt0734OdLcqhrcRYSnatyumiFNwyV9Whsa06N+PA5ptvPnxtMso+h/X6fFr1wFoVpWvqZHLKtDGVwW233TaXU1l0NM3Mxlm2Lw1EbcZRrp5NKoJxQafJmEKgThngtY0mqPq+qBMZwuKb4b75zW8C8MIXvnC47aabbgJKjUKvdXSo9nrVTtMtU5TU1eNh+tgwAaWJDKGYz6wtZn/H+cMac9YWtI1JD2HhUnK4H90IPI54vjOlE5mNyaPPlN2XfqPvPltsM1zLvKyZSLNZKzDI+8TPxGvqNj/vd0RTr9dAx27/RpO2995y1MyTODc4VvvMprXZ1r/R9Fa7iNS/wVD609Q3muGWKunvrB+Wuq57EPA14I2DweCPPRer9ca0Ed913cGsNtMlSZIkSZKMLbN6WOq67r+y+kHpK4PB4Iypzfd1XbfOlKq0DvCbqe2/AmKp+YcD99T7HAwGnwE+M7X/eXllzaSKtBSQvqrWMyWlbCkOdahtnyqxHEkpoyKkg6kOg63VUY3nFfdTH3/LWd5VkCtWnWdrB8dxwpVSvWLuK/nRh+NPNWOSlKW77lotCj/pSU8abquTs/o3hvXWq8VW+RzHwmJx/vnnA6NlVbbccksALr30UgBOO+00YLQUhRgCbtqNvvtDJ96Y2FUFSAftOo0CFIXB/vF+0Xk1bjNEWufvmNbBBH2e1+mnnz7jsfZh6HpdRiOqN3UZoFZqjdkwmwAaz92/dbJMWPxgmug87Vzp8bTmPO/vOogoHrtpBNxPnQwWijLlPeO1iepTa7wsJC1n6fo6RSvBTGMg3jv1nNlSbuvfY8dYVOcch0996lNnfT4LyZ/1WZqKbjsRuGkwGHw0vHU2cODU6wOBs8L2A7rVbAv8QXNdkiRJkiTJpDEbZWk74CXAdV3XXTO17R3AB4HTu647CLgTsCLteaxOG3Arq1MHvGxBj3gOtJ6SW6qB1CU/Wj5LdZp6245buPjrX//64euf/exnQHki32uvvYCyEobpiltrBVWHhfukH0OjzzhjtfBoyoBvf/vbwOgKfNxQUahXN7PxC2gljFNdiYlBxxWP2WurX1FUhOp7xesf28yUsLK1ApfFUlxPPPHEadt22203oJQbiX4Xfr/3g2M1nrdJZz13/bXcHxT1QEXA/olKRZ24VEUnKg31/GMbVb94ji3lZS7o31nPdbGwr31VJ67t80/pS09Sfz765KjK+fk4t4h+Z/NNmzATrWSJzveqe/He1resLgMVfXJMI+B1b6lzdUoO56U4DmvlbaHp88WU6C9X3/d9x1X7dMb5Y6bPxfnVsa5i3/cdi2HF+bMPS4PB4Pu0/ZAAdmm0HwCvmedxJUmSJEmSjAUrotxJ/RTpk320d9ZqQSuyy/Z1VExcvdUFQ30vRgaNG8ceeyxQogf2339/oEQywPQn+5ZvTv3Ubt9F+7m+Se9+97sX5NiXAq9dXYKgpUzOlKAv4n4mQVmquffe1RbzVhFqUYWKbepVY6tf6mi4pfTlswBqMorKWD1/RR+oQw89FBj1nZkPtcof96sK7jhctWrVtM9vvPHGC3IcM9GKFnZO8J6Ovy2+p3oUFRPxXnH+0BIRx77Kq3PxZpttBowmI5WlLKlUR3rHSOvaT1ha/n514tPWHNNKWCn+rqv86psYf8cWkyx3kiRJkiRJ0kM+LCVJkiRJkvSwIsxwM5mHopQ3k+TfCl8UTQqxjRJtnYyyL8x4OWglTXvxi18MlMR4rQrjdR/Gc6+32V8xLcBOO+20MCewhKy77rpAkYfrawxFMrZNrCgudbj8fCvCLwW1xK7E3bov6kSMLTNl7aQazbutCuvJ8uLYN2xdk/o111wzbOM254tWYsR67pxNKpU6ASGUUP2bb7555DNxPFrLcbGIx+OY9a/O7nF+9XfCOaGV2NPX/tWMFl08atOafbDjjjtOO8bFcvCOeI51gMdGG200fK2JULNibW6EMrbsH8dR7Od6HmolfbWvHI877LADAKeeeuqwzWKa9sfrFz5JkiRJkmTMWBHKUk3r6dYnTp9qWyn769VQX7mTPoVqHGilRthwww2Bdmh8nXTRvy31oC854UyrvqWqDL0mxGR/UFYzMR1Efa7+v7XCazm+jyueq+flKrCvfI8rvOiY6zmryrVKWUxScs4HCia+rLnkkkuGr722tZramj/mUgqlNUepUtfKUhyPJhpdLFpJL50PdYS/7LLLhu/ZP3X6mNgHqka2qe8XKP3rvei1iYqTv18xHcViUZe80rE7Ho/XsP7tjL8JqoX2h+cQz93+cE5pqfuOMftJS0lUlhaz9EkqS0mSJEmSJD2sSGWp5VMxUzr1VqG+WlWJK6raNutTd98T7XIWPYyst956wPTVf6Q+95YvVv25+PQ/U/LJVnLLccGCp3XyPVUWmNmm3pdiwUK640x9Xo7nluLqirJV4qcuMeR78Vqnz9L4YYj+Yx7zGKCoACaHhOKXUhdJbjFTSZMWrTnTJJTnnHMOMD35L8BPf/rTGfe5EMRQfed+1WdL9uy7777DNt/5zneAkuCzpTbXfkz+PyZ4rPvOeyf2d+v+XCzqedpEx/F4VLhqhSmq6vad/eM8EOce5496PmrNMS3VaSlIZSlJkiRJkqSHFaEs1U+YdTHP2MYn11Zh1NrmXBfLhbJyrlc8fWrJuPjo/PVf/zUwPeU+TD9GVZWoBtS+SvZLXA24irAkwe233z7ymXGkLvmgz0I8r7p/WtFetU9Xn9/PuPLzn/8cGL3u9YrO84pjvk44J/H+uu+++xbhiJP54P35tKc9DSiJayPOG153/VFaSQX7VOeaunwSlALOlp5R+Xr4wx8+bHPMMcf0n9Q8iT6M+nlaakblLZaTuummm4Dpc1ycX2dSReI9ZL+qSOu3FX0Dn/CEJwBwxx13rMGZzQ8jJA877LDhNiOr69/TVhS1faDfV/R9qiOsW7/Pvrb00EknnTT/k5oD4/sLliRJkiRJMgbkw1KSJEmSJEkPK8IMV5s9lFGjWU2TmiaXukI8FOlPE4SyX2xTO6IpnbbqAY0bOuNpZrIv4rZaRrfyNUyXSu2nKCXr2LfzzjsDReYfZzQzeOyaBaIc7/WuHSxjCK/blJnrlASTiveXY8QxEeuJ2cYK7LIUIc7JmuP1uvXWWwG4/vrrp7XxvlhnnXWAMi/GOc+aaZqM/v3f/33G7/Qesk0MpKjnC9NNRKfrPgfzhUAnbpju2O3fn/zkJ8M2mug0D/UlRqwdvKOZyfc8Z83WMR2L/dtKirvQ1Mf/iU98AhhNrfCCF7wAKHOo5xCDW7y+9W9L/N1wnqh/l+NnvO4f/OAHAfjIRz4y95OaB6ksJUmSJEmS9NCNg/Nx13ULehD7778/MJomXkc0qavHw/SEjC2n5DoppSqE1bsBdtttt+ZnFjNh1lx461vfCow+2btSuvHGG4ESGvt3f/d3wzauoH7wgx8AZSW1zTbbDNtcddVVAJxyyikj3znOSSk9ZldyKidxNesxG9rstYwrPJUWx8RRRx0FjFZwHzdmWgWff/75w9eubL2HTj75ZABuueWWYRvH/O677z6yv+jE+eQnP3lW350ky8mee+45fO28fueddwJw5plnLvnx7LfffsPX66+/PgBXX301ABdddNGife9c7s9nPOMZAGy66aZAOU4o6pxWmFbiUudT07g4t3zzm98ctvnXf/3XuZ/E7Lh6MBhs/ecapbKUJEmSJEnSw4pUlpIkSZIkSWZBKktJkiRJkiTzJR+WkiRJkiRJesiHpSRJkiRJkh7yYSlJkiRJkqSH8c+kOAPzDUU//vjjAdhoo42G2y6//PKR/a233noAPP7xjx+22WOPPYBSs2ilYdi7YZ4mHYOSeOyTn/wkML3e0aRixfV3vOMdQKlqPtcaVCZtM1mn/WTKhUnljW98IwBPfOITgZLI8OKLLx62efaznw2UBIYnnHACAFdcccVSHWaygBgKDiUthIkC1113XaDURAP41re+BcAll1wy4z6dTw8//HAAtttuOwD++Z//edjGFCbXXnvtyGfHOfVI8sAglaUkSZIkSZIeVnTqgF133XX42irRz3zmM4GSgDCm7HdV7CrGpIQxHb+rI9WHo48+GoAvf/nLC38CS4irRlPKr1q1CiirP4DbbrsNgE9/+tMjn530Vd8RRxwBwEte8hKgJEaLCTmtAG4SSsfBv/3bvw3b+LktttgCgFe/+tXAaGK1SeHUU08dvt5+++2BUhLDpK2tJKuWsDBR6cEHHzx872tf+9riHGwyJ1r362abbQbAQQcdBIyWqzAh4+9//3ugqM6HHnrosI3JbPfee28AfvaznwGj8+t1110HwOMe9zig3Bcxcek+++wDwAUXXADA2972tpHjTJJFIFMHJEmSJEmSzJcVqSxdc801QFn9Q1m9qBa5Ko6rLBWkushhLHviSsmChhbrjcUXVa9++ctfjnzHOPT1TNT+R/ogWLIDYMMNNwRK2QuVglhoeLGLXC4mKieuqqNyYqFHt3mesY394FjYZJNNFvmIFw99jqCUfPjtb38LFL+2ujAmFGXJUjhRTTrxxBMX52CTNcbyFB/60IeAUj4jlruxTI0FVFVTVefjtq985SsA/O3f/i1Q5lKApz3taUBRbN3fD3/4w2Eb58hddtkFgBe96EUAPOtZzxq2UeFOkgUilaUkSZIkSZL5sqKi4V71qlcBZVUTo5BUTlSJVI2iP9Jf/uVfjrR1VaStPn7+T3/6E1CK+1ksEODcc88FShTduCpKUTGrI9oe9rCHAXDZZZcNtxkJWKsG9uWk84c//AGAtdZaCyhRbVCut0pSqxCzBXj1fZtkfv3rXw9fP+lJTxrZpoIWVbW6WPTDH/5wYOWpALVKHH1y3BbVlJoNNtgAmO4P6dwDJQLVv1tttdW0/XjP9X3XbDjyyCOBoiSrDMYo2G984xtAiYLbeOONgdH7Y+eddwZKZKnFlvXjg6K4+zkjJZ1X4vf/6Ec/Asq9FKPsHvnIR879RJNknqSylCRJkiRJ0kM+LCVJkiRJkvQwsWa4lmnrla98JVCcDaPjsWYmzQV+PprzNCEoBds2tvF1vT9NOFBMWK94xSsA+OxnPzvX01sSdNSF4gD/nOc8Bxh13pTrr78egH/4h38Y2W44PUyGM/tMaEZrmVPcVptxI567odaTzD333DN8vfbaawMlHLx2dofSP5po7ScTvU4iLVN/Pa5bCVkNLHnrW9868n+A9ddfHygBAobRR3Ol/RuDRhaSN7zhDcPXznWaxDSVaXKDYoY1wEOie4KO4TfccAMAW2+92l82zovu0/PSVLvDDjsM29RpSe64446R/UJJS3HllVf2n2iSLCCpLCVJkiRJkvQwscpSRCdkVyo6XUdlyURq0lI+VFpUWVQPoorg52pH39jGVeO73vUuAM4++2xg8VaKa0or9Nsw4eiYKfazTpyGHcdVnw6vdX9PAq6yTQcRHeAdG15nx1YMEPDcY6LKSeXrX//68PVHPvIRoIR616k1Ijrk3n777cBowsGVQF0O6OUvf/nwPRO4GmCiOhvLJTlGnEccc3/zN38zbOO4M0DlTW96EwAf/ehHp7VZEwfvqPYZFHP11VePHHtUwwxu8D5XBVPhgdIfqupXXXUVUNIFAHzxi18EipO791lUuA888ECgpF3RmdzSQVASVT73uc+d7SknybxJZSlJkiRJkqSHiVWW4orFVP0mzXN1G0NbaxXFVXH0OXB1VPusxDZ1CgLbxtWR7+nrcdxxxwHjtxJqrUr1oYjFLUW1yfPzGqw0ZelRj3oUMDpmVJLsM/tA/xIoK+WoNk0qsVC0SpmJWO2n2D9eb6+/oegrnf3222/4evfddwdKclPVmujb5RhxHPlenGMsrWMyUJXyhSL6+hjyb1kaQ/ZVBmF6egPV54c+9KHDNjvttBMA999/P1CKbsdj1/fJc7V/YsFqlSRVeL9TFRvggAMOmMPZJsnCkMpSkiRJkiRJDxOrLH33u98dvr7wwgsB2HbbbUfaxEgMFQBXNf5tRbq5Om6t+kRFyRW1UUAAD3rQg4CyQlxuRalW1fx/S/0xuabRLS1UHVxhHnvsscP39FGJ/j7QLro6briaVSWM5+Br+6w1flSfVDhXCqoF3kP6sLSixYz2uvTSS5fwCJeO+p5RZYNSGFb1yBIdFt2GoiipeqvAxPIilvi4+OKLgVG/KGnNSWuC97L+aEY8qhRBie702B3nHh+UyDj9tJwDLawLsOOOOwLwnve8BygK7tOf/vRhm1p5M3nnJz7xiWGb2OdJslSkspQkSZIkSdJDPiwlSZIkSZL0MLFmuMib3/xmoFSvbplRarNJK3VA/Z7/b5kbdBDXJBEdfTVlve9971vzk1pA6nNtnbuVwGfjnKwDq471kUkwt83ELbfcAhSzQDwX00FohmkFCNg+VlFfCWha0aG3laCx7o+77757KQ9xUZhNYtU+E7umo+gsrRO4jtTf//73gWLShuJOELfNdGwLlQT2F7/4BVDMgtF0aOoDTWSaWPfaa69hG9MReBymEIgmerdpqnWe1o0CSvoF+6BOBzPOzOZaeH6mP9h3332H79UuDJM8l65EUllKkiRJkiTpYUUoSyZJ8+8Tn/hEoDgZQnH2djXjKqD19O62VnX1OqmcaQpcdUFxFI5J5Mad5z3vecDsVqimCoiVyWvsexUZ/44zjh+dS1vHXI+fqDrqtLsSklJGHOPifVE78UO7jNADlde97nXTtqm8WVpnq622AuDDH/7wsM2uu+468hnHXFQxVbIXKumnStduu+0GwB//+Mfhe6pfKkI/+MEPgFHla6211gKKiqbSaJJJKOH/qrMmvvS7oTiaq3Sdd9550451XEsqtdRmMQXC8ccfDxQF/9WvfvWwzcc//nGgX1Gqzz1VqKUjlaUkSZIkSZIeJlZZiqtan6otpKu9PSaKtE0d+ttKzFinBYhtXDW4TaUprr7jKnEc+NSnPgWUY95yyy0B+MpXvjJsY4LJm266CSgrV0sTANx4440A/O53vwPKNYhpHCzbcO655wLlGrRW2eOG6mNLFamLKvetIidBRZsLJiq1qGndB1DuFa+3fi7f+c53luowl4WWP2MfqiumGTB83vIeMD1tRyvFx3wSn7aO2QSRr3nNa4DR+/7zn/88AC984QuBkibl2muvHbZRUTLxpL5GMXGlvk7u26S9qkjxvPRZeu973zvt+MdNUZJ6Lvirv/qr4WvLB9nGa7znnnsO22y//fYAnHXWWQCcdNJJ076jPvdUlJaOVJaSJEmSJEl6mFhlqfVE7UrHJGraiaFE9Pg5V8BRNXLFVRepjCqW/hpGwbl6cP8AH/jAB9bspBaJhzzkIcB0pWyjjTYatrGUhcrQBz/4QWA0safqmW1VGvS/gOL340o5RgmOO0YEqhZGZdJzrsddXE06NowSjFE+k4ZRW1D8UfTF0j8vrnLtF1UCI6VWurLUUjlU3LzPHvGIRwzfc9645pprADjjjDOA4q/Sx4tf/OLh63/8x38Eiq/hfCMwPWZ9llQ3oCiLJ598MlCU5VgY3DHvuZ5++ukA/Md//MewTT3nGhEWE2CqWlnqpTXP1/07l/Nr7W82StVMvw0R7//DDjsMKHMowBZbbAGMRhlCKWQc2+jH9JznPAcY9R/7+c9/DpRIy3XXXRcoqn/rmBdaiYtRkBtuuCFQCjDH3wLxGJ0//N2Ivp3OlSqc82Uxzj2VpSRJkiRJkh7yYSlJkiRJkqSHiTXDRerwyV122QWACy64YFobzSZKptFJdaZwzGiOqR193Y9OkOOI0q8yr+f5k5/8ZNjGuk3K5r/+9a+BUt8KiglB04p9odkKSn9qqmk5p44rOrlbjypKuJ67ODZiG2tWrVq1CoCjjz568Q52kfEcYLpDf8sB3jGgecKxNkkslHRfm4lM9Ajw2te+FoB11lln5DMf+tCHhq81bTinaMbQFAzlWsS5aT6YgkDH45gEUnPQ2WefPfKdMTXL5ptvDpTAjkc+8pFAmUeg9MsTnvAEAA466KCRtlDuoeuuu27acchczG/SCrGvr3NrXNcJWFvfrenJNAd1agQoriE1phKA8luio7dzr/0FxXXC+doEyNHk99WvfhXoNxnORMv5/5BDDgGKg78mNyjBPZ6HrjDR/eLggw8G4CMf+QhQfo9MSQHwlre8BSi/Py972cvmdKx/rs1CmONSWUqSJEmSJOlhRShLtdOe5Ti+973vDbeZQK1OGNgXeul7UX3ySbVOGXDUUUetyaEvCYb1Gs6r8haf/nWsNEmejrpWEwf41a9+BcAmm2wClJVudEB0P6544nvjjikVPE9XelDGQL3SNBEllFXxU5/61MU/2EUmOv87BlQLHfutVbqrWQMFxo3lSOJngAXAl770JaCswHWMjg7aOkm7yvZvVFlUEkwQuVA4Jzz60Y8eblMxueeee4ByL0RHXwM7VJsMlojHrBPyb3/7W6DMS62yQqqZC4VqTyuRqvSpD15DS1jFuVMHeK/hG97whmmfn43C4bmr7tk/p5122rCN18IAGhOYxmtx/vnnA9PV8NkQj2/jjTce2bfK+2yI6VNU2P1roMI73/nOYRsdu005ctxxxwGjSTv7jnUpSGUpSZIkSZKkhxWhLEn99O5KCIoCdP/99wPt8FM/58rZFV1cRejzpN2+LwHhuKTlV/FwdaQ/wDOf+cxhm8997nNAOVbDX02mBsU2bzkZy7nEcFHLG6g6XXnllQt5KouC4dLi9Y7+IHUZD1eoUX1yvJhYb//99wfg1FNPXYzDXlRUCKGsyutzj7jNv9GfZZxo3Yu17533+JqqT7WPyL333jt8rZ+Gf00CGdMLeM/87Gc/G3kv+sCY0HFN5pa+z3zta18DRv2sTB3h3GlSyc985jPDNvrkOM+adNMUB1DUSv2z9MWKSSn1z1FhaLEm82qrvNVs9vPyl78cKCkaLAZs+DuUhLs//vGPZ9zPTN/RSq4sWgDiHON8bOoJ/dr23nvvYZvoQzoT/haYXscxH32rTAhqygiJv4eWp3GO8LtbCWtVqL/85S+P/IUyX6jKqTpFhdFtHrvzbCs9TT0f+Ru4JmrbcJ9r/MkkSZIkSZIHACtaWYolSFzt6XfRiuyp1QM/E59cjRrx8zHp2rhiCRMVIM8rrljcpmrkCiMmpXSF7Opq3333BeDyyy8ftjGSTFyVjjP6aUkrgV29MnQlFldxKlGODfc7icpSayVcFxFuJWu178Y1CrJecUI51oUqU1MrSy95yUuGr105GxFkOZAjjjhi2EYlSXVGn5GYBFKfjoXGBJEWzYWicHlv61+irw7AS1/6UgC+8IUvAEWZOvPMM4dt9thjD6CoCKpHJlqE4osTfSVr5qOm9UVQqZSbEBPK/FUX+HXua1ErlJG6AHffuXgc0e/TSDSVJTFScbbUCpDHE8/dhJm+pw9TvHc8x/o3s4W/G84j8bdT37ZPfvKTQDtJq/eFxxzLyYjv1VHZ/o6lspQkSZIkSbJI5MNSkiRJkiRJDyvKDFfTcuYTTWtR4qylUf/GZHCmHtDsUKciGEdMpWB/aJ68/vrrh21q53aTyUXHU/tHWV6zXjR31o7z0dQ3rlj53T7Q1BpNbMrN0XExbo+vPeeYvG3SiPdLnXCyz3SgxB5TKiwn9X3vObTMBTvssAMAl1566bT31iTlwLve9S6gOHFDCZvXPLXpppsC8K1vfWvYRnPHU57yFKCY4awHBot3X9kvMWGt3+U8qGnD/oJSG/Pd7343UMxo55xzzrCN25xTTJ+geQbKWPM7FjpIpm8/XoM4dk3+qHO74/u5z33usI1O8dIyv9Xf33ccpm1omQ5jsMB8cH7X/OU1juPba+FvpQFTrUSoddBHnxm/lcZBc2dt1osuMPVvi/+P+/G9OkhjIX6nU1lKkiRJkiTpYUUpS/UqMq4Q6idd1YO4vX5y1YHMEMXWfqJzYs1ypwyo0blNZ8WomKkOuXpwdRMT6hnG6TZVuegE7pO8K5VxdfSNmECxHhNRffB17aAZz08HYVc14xo+Pxui86XX2/5prfZrZTLeM0tFy/Hc43Jc2ubwww8fttluu+0A2HnnnYGSPNFEtvHzs+H9738/AG9605uA0fQZ7scEsd6TpuqAktpDdcVxFNWexcL7NqojXkvn02233RYYDTPXwVgHYRMCR/XaUHjnEcucREfdu+++GygKnHOUod8wP7UpqsWqFqbJsDyWCXWhBLyoTDh3WsIDinJfz3WtucHv9JpGhdAwfvvZ+y2q+6qO/u54b8bkn46x2QQfeZ371LD5OEWvJFJZSpIkSZIk6WFFKUvaRFUBYpd9SMIAAA3aSURBVGIzk7jVoYXR36a2u/r0H1eV2tddEcSEjDIuyShrXCF67DFU2vfsA1dAJ5xwwrCNq4/dd98dKKu0aMO2TZ3sbJypS5k4fuIq1D6rr21LOVuTApbjxi233DJ8bYi4q+s6hQBMT9Yak/YtFX3FTuWss84Cij8QFH8bw9af9KQnAaPh4RYnrf0uIibxe/Ob3wyUFXkM+bd0hW1UcmN/ffaznwVK0j59aaLiofKy0FimJBYG10/GpJgmcTVNAJQEt4Z+O0/G1CGmMKn9ZUxOCUVVUYXy81FZWhMsXPz0pz99uM35z3HjnBfTFhjC7vVSRY9j7AUveAFQFES/Sz8nmO6L4xwRfyP8nfG4brjhBmC03IkqmEWI/Uy0oqg2ObZi8sdkzUllKUmSJEmSpIcVpSzVSeU+8IEPDF9r4/XJXHt59MnRRmwb1ZZYNsXVkKuPWKx33NHe7qpI9QeKH4ErFVfZ9hOUFZMrcaPIou9TndCxr3DluKCCpHrkCrMVcVRHs8SVsz4CtTo3ibj6hzJuvKatAtP1Nn1PlhJ9fKKPh/epSozHFf05POZaFTOaDYqyVCtKMTmkfiz6HKlsx6SnFhN1PlLxvvnmm4dtLAei/8/f//3fA6P3mcyn9Ef8nGPd///oRz8atlE9t+8856jKO0Y233xzoCTUjAWV9XF0m4kVo0qmP5RtFkqddc7yuKD0w0zKMpTfhGc961lAW02v7w/HYVTV3U8djRmtFo5Jt+k3qiIHRV2sk6tGFdx+7kvsmcyd8f8lS5IkSZIkWUbyYSlJkiRJkqSHFWWGq8N7r7rqqubrmdAkp0OlFbDnyrg5dou1naxHFaVb+055WMk/hgdrKjDEuhUerjTvfiYh7FSnzTrJWSvZWZ06oFU1fDZ1ksadaBpp1VGM26GYHB37t91222If4jSOPPJIAF75ylcOt2lO1OH4rrvuAkbvUc3RmlR1sH7b2942bOO5H3jggQB87nOfm/b9OkJrwv70pz8NFIdxKEkot956a6A4z2t+isfjfkwrEB19xTE730AKa7l5j8caYR7PZZddBpQAj2jS0mzme459TUlQzHg62TtmTN0AxTS32WabAaVGXXSWXhO8NzXHLSS6MCw00fyWLD+pLCVJkiRJkvSwopQlcRUYnfBcgamG2CaqUdtvvz0w6iA6075FdWYSki/qeChRWfJ17ZgdSzW4MnQFbuV0nReh9LnfNQlOzq7gpS57AtNVNJW3OH5q5+9JSJswEzHJnWOhVt6iA3wdXGEpi6Xk7W9/OzAatGG4u6HnhsbH62ayx0022QQoySSjQqUTr2kA9tlnH2DUgVmF0v0ZHPGWt7xl2EZlqnbCjQquDt3Pe97zgKLSnXrqqdPOeU1U7JZiagJOndtjst1tttkGgFWrVgGljE/sZ4NinC+22GILYPT+33DDDYHSl16bmLRTRVNFcBLKJSUPDFJZSpIkSZIk6WFFKkuuturVbqQOcYcSIqttvm/fMgmKkrjKq/2ToKy8Xf1ZgiAmy7N/9ENy9ajCBNPDX1vhzuNG7cfgOUTlrb7utol96DmbzLRWISeJyy+/fPja81ARUDFrFRq2nxbDN2S2xCSQHuOxxx4LlPGtbw3Al770JaDMF/rtRF+avffeGyh94H2hwgTFF9AkhSoxkV/84hdAmTdMv2FxaihlN1RnVXJaiRnnUoZFWr50G2ywwch3REXZEhsf+9jHRj4f7wlVLxNWOh5iag2VaecE96vfFhTfKUPkW75v4+oTmqxsUllKkiRJkiTpYUUqS7OhtSJTOYm+TiuBOoKrVQTW1ZrnbqFPU+5DUZRcEUafHtHHQP+NSVCW6vIkKgxxHNg/dRLAqB7VStIkqY410WelTrLZUiZVaj3nmNRyqTnnnHOGr1XIvF4ml7z66quHbfRVMvLKsR8TTloo9vnPfz5QFCYTtUJRSFSC/M6oMHls5557LlCiziynAUXR8vMqMkazRhZKZTHazGsd/Rv9Xq+3/lWxzXOe8xwA/uVf/gWAnXbaCRiNqjRZp+qR/lpRWTJq2X41avC8886bz+klybxJZSlJkiRJkqSHfFhKkiRJkiTp4QFrhtPEEk1JhgG3ki1OMrX5RKk9mol0hDXsecsttwSKSQGKU6vbvvGNbwAlDBpKUkol+lg/bFypHV41JbZST/ieDt5xrPjaPpjk1AER64R5f2hujA7e3keabKKT9VLhuL7iiiuG23Q4dqwecMABANx4443DNjMlnz3iiCOmbdOMdvrppwMlySSUlCMms7TGW4t1110XKP0WEzy+4hWvAGDXXXcFyv125plnDtvY9ws1xtyfjtUxnN8EkTqBe86x0v0ll1wClHlD83uswbfjjjsC5Vp4DZxzoMwb3meTnNg1WVmkspQkSZIkSdLDA1ZZajl4G+Zap69vlbSYJDxmnThNIBfPS5XJBHQqA9HR19WeTpiPecxjgNHq1qpPcd/jjgkYVYZU4qKyVJc3aSXdVFFqOdBPMo4FU0XYX/Hc7Y/zzz9/iY+u0Lo3L7zwQqAcq8rHS1/60mEb73eVM++PlvP+HnvsAcBGG20ElJImAO9///tnfayWXWnxT//0T0BxIj/qqKOmtVno++uQQw4B4D3veQ8wWqJFB/gf/OAHAPz4xz8GRlVVA0F06Pbeede73jVsc+ihhwJFxdp3332B0SAQFVzbeJ5ReUuS5WByftGSJEmSJEmWgQesstSy9buSi+GuMJlqUsRQeP0kTJroXyj+CPaLiSYNAYainFjs0sSV+jRACQvWj8kw7HHGVWztBxL92RwbjgXPPY4j0yVIX1LUScKEjJ/85CcB+Pa3vw2UwrFQfIOOP/74JT66uaF/TPSTWQ722muveX1+oceWiTAN4zcxJxQfJd9TUYrqmCkMnv70pwMlNYNFt6HMDSp4+kV5b0Ep2mvqgOgXlSTLSSpLSZIkSZIkPTzglKU6qWDkjDPOAEaTpK0kjFh5ylOeAoyWWNCfyf6xSOZ3vvOdYRtLITz60Y8GSpRUVGBMInfrrbcCo8n/xhWLvqqG6R/x3e9+d9hGhcyV7tprrw0U/xYofaZip2/XpKOaps9Ji1geJZk83vnOdwLFZzEm2zzllFOAokzfdNNNAOyyyy7DNg972MOAci/p0xX3Y2SbPm8q1SarBNhhhx0AuP766wE4+eST53lmSbIwpLKUJEmSJEnSQzcORQm7rlv+g0iSJEmS5IHG1YPBYOs/1yiVpSRJkiRJkh7yYSlJkiRJkqSHfFhKkiRJkiTpIR+WkiRJkiRJehiX1AH/Cvz71N9kcnkIeQ0nmbx+k01ev8knr+HSs95sGo1FNBxA13VXzcYjPRlf8hpONnn9Jpu8fpNPXsPxJc1wSZIkSZIkPeTDUpIkSZIkSQ/j9LD0meU+gGTe5DWcbPL6TTZ5/SafvIZjytj4LCVJkiRJkowj46QsJUmSJEmSjB1j8bDUdd0eXdfd3HXdrV3XHbHcx5P8ebqu+0XXddd1XXdN13VXTW37H13XXdh13S1Tf//f5T7OpNB13ee6rvtN13XXh23Na9at5p+n7slru67bavmOPIEZr99RXdfdPXUfXtN13Z7hvbdPXb+bu67bfXmOOpGu6x7Rdd0lXdfd1HXdDV3XvWFqe96DE8CyPyx1XfdfgE8BzwA2AV7Qdd0my3tUySzZaTAYbBFCXY8ALhoMBo8FLpr6fzI+fAHYo9o20zV7BvDYqX8HA8ct0TEmM/MFpl8/gI9N3YdbDAaD8wCm5tD9gU2nPnPs1FybLB//CRw+GAw2BrYFXjN1nfIenACW/WEJ2Aa4dTAY3D4YDP5/4FRg1TIfU7JmrAK+OPX6i8A+y3gsScVgMLgU+F21eaZrtgo4abCaK4AHd123ztIcadJihus3E6uAUweDwf83GAzuAG5l9VybLBODweDewWDw46nX/xO4CViXvAcngnF4WFoXuCv8/1dT25LxZgB8q+u6q7uuO3hq29qDweBeWD0xAA9btqNLZstM1yzvy8nhtVNmms8F03devzGm67r1gS2BH5L34EQwDg9LXWNbhuiNP9sNBoOtWC0Vv6bruh2W+4CSBSXvy8ngOGADYAvgXuDoqe15/caUruseBHwNeONgMPhjX9PGtryGy8Q4PCz9CnhE+P/DgXuW6ViSWTIYDO6Z+vsb4ExWS/z3KRNP/f3N8h1hMktmumZ5X04Ag8HgvsFg8L8Hg8H/AU6gmNry+o0hXdf9V1Y/KH1lMBicMbU578EJYBweln4EPLbrukd1Xff/sNop8exlPqakh67r1uq67r/7GtgNuJ7V1+3AqWYHAmctzxEmc2Cma3Y2cMBURM62wB80FSTjQ+XD8mxW34ew+vrt33Xdf+u67lGsdhK+cqmPLyl0XdcBJwI3DQaDj4a38h6cAP5iuQ9gMBj8Z9d1rwW+CfwX4HODweCGZT6spJ+1gTNX3/v8BXDyYDC4oOu6HwGnd113EHAnsO8yHmNS0XXdKcCOwEO6rvsVcCTwQdrX7DxgT1Y7Bv8v4GVLfsDJCDNcvx27rtuC1eaZXwCHAAwGgxu6rjsduJHVUVivGQwG/3s5jjsZsh3wEuC6ruuumdr2DvIenAgyg3eSJEmSJEkP42CGS5IkSZIkGVvyYSlJkiRJkqSHfFhKkiRJkiTpIR+WkiRJkiRJesiHpSRJkiRJkh7yYSlJkiRJkqSHfFhKkiRJkiTpIR+WkiRJkiRJevi/X+KrFMgF3SkAAAAASUVORK5CYII=\n", 101 | "text/plain": [ 102 | "
" 103 | ] 104 | }, 105 | "metadata": {}, 106 | "output_type": "display_data" 107 | } 108 | ], 109 | "source": [ 110 | "dset_temp = FashionMNIST(root='../data/fmnist', train=True, download=True, transform=transforms.ToTensor())\n", 111 | "dataloader_temp = data.DataLoader(dset_temp, batch_size=64)\n", 112 | "batch_img = next(iter(dataloader_temp))[0]\n", 113 | "\n", 114 | "grid = utils.make_grid(batch_img)\n", 115 | "\n", 116 | "plt.figure(figsize=(10,10))\n", 117 | "plt.imshow(grid.numpy().transpose(1,2,0))" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": { 123 | "slideshow": { 124 | "slide_type": "slide" 125 | } 126 | }, 127 | "source": [ 128 | "### Labels\n", 129 | "Each training and test example is assigned to one of the following labels:\n", 130 | "\n", 131 | "| Label | Description |\n", 132 | "| --- | --- |\n", 133 | "| 0 | T-shirt/top |\n", 134 | "| 1 | Trouser |\n", 135 | "| 2 | Pullover |\n", 136 | "| 3 | Dress |\n", 137 | "| 4 | Coat |\n", 138 | "| 5 | Sandal |\n", 139 | "| 6 | Shirt |\n", 140 | "| 7 | Sneaker |\n", 141 | "| 8 | Bag |\n", 142 | "| 9 | Ankle boot |\n", 143 | "\n", 144 | "https://github.com/zalandoresearch/fashion-mnist/blob/master/README.md" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": { 150 | "slideshow": { 151 | "slide_type": "slide" 152 | } 153 | }, 154 | "source": [ 155 | "\n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | "
No padding, no stridesArbitrary padding, no stridesHalf padding, no stridesFull padding, no strides
No padding, stridesPadding, stridesPadding, strides (odd)
\n", 181 | "\n" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": 8, 187 | "metadata": { 188 | "slideshow": { 189 | "slide_type": "slide" 190 | } 191 | }, 192 | "outputs": [], 193 | "source": [ 194 | "class Net(nn.Module):\n", 195 | " def __init__(self):\n", 196 | " super(Net, self).__init__()\n", 197 | " self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n", 198 | " self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n", 199 | " self.conv2_drop = nn.Dropout2d()\n", 200 | " self.fc1 = nn.Linear(320, 50)\n", 201 | " self.fc2 = nn.Linear(50, 10)\n", 202 | "\n", 203 | " def forward(self, x):\n", 204 | " x = F.relu(F.max_pool2d(self.conv1(x), 2))\n", 205 | " x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n", 206 | " x = x.view(-1, 320)\n", 207 | " x = F.relu(self.fc1(x))\n", 208 | " x = F.dropout(x, training=self.training)\n", 209 | " x = self.fc2(x)\n", 210 | " return x\n", 211 | " \n", 212 | "model = Net()" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": 9, 218 | "metadata": { 219 | "slideshow": { 220 | "slide_type": "slide" 221 | } 222 | }, 223 | "outputs": [], 224 | "source": [ 225 | "loss = nn.CrossEntropyLoss()\n", 226 | "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 10, 232 | "metadata": { 233 | "slideshow": { 234 | "slide_type": "slide" 235 | } 236 | }, 237 | "outputs": [ 238 | { 239 | "name": "stdout", 240 | "output_type": "stream", 241 | "text": [ 242 | "938\n", 243 | "Epoch: 0, loss: 0.7718479633331299\n", 244 | "Epoch: 1, loss: 0.45067089796066284\n" 245 | ] 246 | }, 247 | { 248 | "ename": "KeyboardInterrupt", 249 | "evalue": "", 250 | "output_type": "error", 251 | "traceback": [ 252 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 253 | "\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)", 254 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 9\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Epoch: {}, loss: {}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mepoch\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ml\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 255 | "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/torch/autograd/variable.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph, retain_variables)\u001b[0m\n\u001b[1;32m 165\u001b[0m \u001b[0mVariable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 166\u001b[0m \"\"\"\n\u001b[0;32m--> 167\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 169\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 256 | "\u001b[0;32m~/miniconda3/lib/python3.6/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(variables, grad_variables, retain_graph, create_graph, retain_variables)\u001b[0m\n\u001b[1;32m 97\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 98\u001b[0m Variable._execution_engine.run_backward(\n\u001b[0;32m---> 99\u001b[0;31m variables, grad_variables, retain_graph)\n\u001b[0m\u001b[1;32m 100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", 257 | "\u001b[0;31mKeyboardInterrupt\u001b[0m: " 258 | ] 259 | } 260 | ], 261 | "source": [ 262 | "print(len(dataloader_tr))\n", 263 | "for epoch in range(5):\n", 264 | " for i, (x, y) in enumerate(dataloader_tr):\n", 265 | " x, y = Variable(x), Variable(y)\n", 266 | " l = loss(model(x), y)\n", 267 | ",\n", 268 | " optimizer.zero_grad()\n", 269 | " l.backward()\n", 270 | " optimizer.step()\n", 271 | " print('Epoch: {}, loss: {}'.format(epoch, l.data.numpy()[0]))" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 11, 277 | "metadata": { 278 | "slideshow": { 279 | "slide_type": "slide" 280 | } 281 | }, 282 | "outputs": [ 283 | { 284 | "name": "stdout", 285 | "output_type": "stream", 286 | "text": [ 287 | "157\n", 288 | "0\n", 289 | "30\n", 290 | "60\n", 291 | "90\n", 292 | "120\n", 293 | "150\n", 294 | "Accuracy: 0.8392\n" 295 | ] 296 | } 297 | ], 298 | "source": [ 299 | "model.eval()\n", 300 | "preds = []\n", 301 | "ys = []\n", 302 | "print(len(dataloader_val))\n", 303 | "for i, (x, y) in enumerate(dataloader_val):\n", 304 | " x, y = Variable(x), Variable(y)\n", 305 | " preds.extend(model(x).max(1)[1].data.tolist())\n", 306 | " ys.extend(y.data)\n", 307 | " if not i % 30:\n", 308 | " print(i)\n", 309 | "\n", 310 | "corrects = (np.array(preds) == np.array(ys))\n", 311 | "print('Accuracy: {}'.format(corrects.mean()))\n" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": 12, 317 | "metadata": { 318 | "slideshow": { 319 | "slide_type": "slide" 320 | } 321 | }, 322 | "outputs": [ 323 | { 324 | "data": { 325 | "text/plain": [ 326 | "" 327 | ] 328 | }, 329 | "execution_count": 12, 330 | "metadata": {}, 331 | "output_type": "execute_result" 332 | }, 333 | { 334 | "data": { 335 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAECCAYAAADesWqHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAC5dJREFUeJzt3U2MVfUZx/Hfb15gGNQgtonI0IKJ0RpjhY6tSmITMLFqq5suNJGkbmbTKhoTo02Nu66M0UVjMsGaphJdIAtrjNr4smhtqSPQIo5NDFreI61FKAZhmKeLuTQW6dwz9Pzvmcvz/SQmMF6fPBnm67lzOfc/jggByKWn6QUAdB7hAwkRPpAQ4QMJET6QEOEDCTUWvu3v2f6r7Q9sP9jUHlXZXmL7DdvjtrfbXtv0TlXY7rW9xfaLTe9She0FtjfYfr/1ub626Z3asX1f62viXdvP2h5oeqd2Ggnfdq+kX0i6SdLlku6wfXkTu8zAhKT7I+Ibkq6R9OMu2FmS1koab3qJGXhC0ssRcZmkb2qW7257saR7JA1HxBWSeiXd3uxW7TV1xf+2pA8iYkdEHJP0nKTbGtqlkojYFxGbW78+rKkvyMXNbjU920OSbpG0ruldqrB9nqTrJT0lSRFxLCIONrtVJX2S5tnukzQoaW/D+7TVVPiLJe36wu93a5ZH9EW2l0paLmlTs5u09bikByRNNr1IRRdLOiDp6da3J+tsz296qelExB5Jj0raKWmfpE8j4tVmt2qvqfB9mo91xb3Dts+R9LykeyPiUNP7/C+2vy/p44h4p+ldZqBP0gpJT0bEcklHJM3q139sn6+pZ6vLJF0kab7tO5vdqr2mwt8tackXfj+kLnh6ZLtfU9Gvj4iNTe/TxkpJt9r+SFPfSq2y/UyzK7W1W9LuiDj5TGqDpv5HMJvdIOnDiDgQEcclbZR0XcM7tdVU+G9LusT2MttzNPViyAsN7VKJbWvqe8/xiHis6X3aiYiHImIoIpZq6vP7ekTM6itRROyXtMv2pa0PrZb0XoMrVbFT0jW2B1tfI6s1y1+QlKaeWnVcREzY/omkVzT1KugvI2J7E7vMwEpJayRts7219bGfRsRLDe50Nrpb0vrWBWGHpLsa3mdaEbHJ9gZJmzX1Nz9bJI02u1V75m25QD7cuQckRPhAQoQPJET4QEKEDyTUePi2R5reYSa6bV+JnTuh2/ZtPHxJXfUJU/ftK7FzJ3TVvrMhfAAdVuQGngULe2PRULWbAg9+ckILFvZWeuyebQXfqOXTvW/oy47HUfXP4JwF91SbeybiRLU33R3X5+rX3GJ7VHXigup/fhNHj6hvoPrje/9x5ExWqs1s+Rwf1REdi8/bftEVuWV30VCfnv7NotrnPrzs6tpnnuT+OUXm9swrdxjLiUOz9s2Bp/XPH5Q7TOf8X/2h2Oxusileq/Q4nuoDCRE+kBDhAwkRPpAQ4QMJVQq/287ABzC9tuF36Rn4AKZR5YrfdWfgA5helfC7+gx8AF9WJfxKZ+DbHrE9Znvs4Ccn/v/NABRTJfxKZ+BHxGhEDEfEcNV77wE0o0r4XXcGPoDptX2TTpeegQ9gGpXendf6oRH84AjgLMGde0BChA8kRPhAQoQPJET4QEJFDts8zwvjO15d+9xX9m5t/6AzdONFV5UZXPEQzzPCTzrGKTbFazoUn7T9ouOKDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBChA8kRPhAQpV+aOZM2VbPwEDtc4sdgS3Jry8uMjdW7y0yV5L6Fl1YZO7Evv1F5vZceVmRuZI0+Zf3yww+S49H54oPJET4QEKEDyRE+EBChA8kRPhAQoQPJNQ2fNtLbL9he9z2dttrO7EYgHKq3MAzIen+iNhs+1xJ79j+bUS8V3g3AIW0veJHxL6I2Nz69WFJ45LK3OYGoCNm9D2+7aWSlkvaVGIZAJ1R+V592+dIel7SvRFx6DT/fkTSiCQNeH5tCwKoX6Urvu1+TUW/PiI2nu4xETEaEcMRMTxHc+vcEUDNqryqb0lPSRqPiMfKrwSgtCpX/JWS1khaZXtr65+bC+8FoKC23+NHxO8kFXxTMoBO4849ICHCBxIifCAhwgcSInwgoSKn7MqWenvrHzu33I1BcUOZk2V/vqPc3c0Pr7ix2OwiPtjZ9AZo4YoPJET4QEKEDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBChA8kRPhAQoQPJET4QEKEDyRE+EBCRY7XjslJTX72WYnRxUysWlFk7s+uHCwyV5LueHtbkbnrLxsqMvfEVZcUmStJfuvPxWafjbjiAwkRPpAQ4QMJET6QEOEDCRE+kBDhAwlVDt92r+0ttl8suRCA8mZyxV8rabzUIgA6p1L4tock3SJpXdl1AHRC1Sv+45IekDRZcBcAHdI2fNvfl/RxRLzT5nEjtsdsjx3X57UtCKB+Va74KyXdavsjSc9JWmX7mVMfFBGjETEcEcP9mlvzmgDq1Db8iHgoIoYiYqmk2yW9HhF3Ft8MQDH8PT6Q0Izejx8Rb0p6s8gmADqGKz6QEOEDCRE+kBDhAwkRPpBQkVN2JUkRxUaX0PfatDcmnrGS9ziXOg3317t+X2TumiVFxk7p6S0zNwr+Cdr1z6yYHVd8ICHCBxIifCAhwgcSInwgIcIHEiJ8ICHCBxIifCAhwgcSInwgIcIHEiJ8ICHCBxIifCAhwgcSInwgIcIHEiJ8ICHCBxIifCChIqfsuqdHPeecW/vcycOHa595Ut/SrxWZO7n/4yJzJcnz5hWZu2bJyiJz9997XZG5knTh428Vmeu+ggdRT0wUm90OV3wgIcIHEiJ8ICHCBxIifCAhwgcSInwgoUrh215ge4Pt922P27629GIAyql6d8ITkl6OiB/aniNpsOBOAAprG77t8yRdL+lHkhQRxyQdK7sWgJKqPNW/WNIBSU/b3mJ7ne35hfcCUFCV8PskrZD0ZEQsl3RE0oOnPsj2iO0x22PH4mjNawKoU5Xwd0vaHRGbWr/foKn/EfyXiBiNiOGIGJ7jgTp3BFCztuFHxH5Ju2xf2vrQaknvFd0KQFFVX9W/W9L61iv6OyTdVW4lAKVVCj8itkoaLrwLgA7hzj0gIcIHEiJ8ICHCBxIifCAhwgcSKnJ2cExOFj0Ku4SJj3Y2vcLMHS1za3TPufUfjS6VOwJbkl7as7nI3JuHvlVkriT1XrCw9pk+2FvpcVzxgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0iI8IGEipyyK1vun1P72Dh+rPaZ/2GXmRtRZq5UbOdSJySX+Jo46ebFK4rMfWHPn4rMlaRbF19d+8yIE5UexxUfSIjwgYQIH0iI8IGECB9IiPCBhAgfSKhS+Lbvs73d9ru2n7U9UHoxAOW0Dd/2Ykn3SBqOiCsk9Uq6vfRiAMqp+lS/T9I8232SBiXtLbcSgNLahh8ReyQ9KmmnpH2SPo2IV0svBqCcKk/1z5d0m6Rlki6SNN/2nad53IjtMdtjx+No/ZsCqE2Vp/o3SPowIg5ExHFJGyVdd+qDImI0IoYjYrif1/6AWa1K+DslXWN70LYlrZY0XnYtACVV+R5/k6QNkjZL2tb6b0YL7wWgoErvx4+IRyQ9UngXAB3CnXtAQoQPJET4QEKEDyRE+EBChA8kVOZ47YiyR2GXUPIY7FIK7dwzf36RuZNHjhSZK0k9g4NF5pY4Avuk53f/sfaZ373pX5UexxUfSIjwgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0iI8IGECB9IiPCBhAgfSIjwgYQIH0jIUeCkVtsHJP2t4sO/IunvtS9RTrftK7FzJ8yWfb8eEV9t96Ai4c+E7bGIGG50iRnotn0ldu6EbtuXp/pAQoQPJDQbwh9teoEZ6rZ9JXbuhK7at/Hv8QF03my44gPoMMIHEiJ8ICHCBxIifCChfwNG8ppeSzgZ/QAAAABJRU5ErkJggg==\n", 336 | "text/plain": [ 337 | "
" 338 | ] 339 | }, 340 | "metadata": {}, 341 | "output_type": "display_data" 342 | } 343 | ], 344 | "source": [ 345 | "from sklearn.metrics import confusion_matrix\n", 346 | "\n", 347 | "plt.matshow(confusion_matrix(np.array(preds), np.array(ys)))\n", 348 | "\n" 349 | ] 350 | }, 351 | { 352 | "cell_type": "markdown", 353 | "metadata": { 354 | "slideshow": { 355 | "slide_type": "slide" 356 | } 357 | }, 358 | "source": [ 359 | "shirt get confused for t-shirt" 360 | ] 361 | } 362 | ], 363 | "metadata": { 364 | "celltoolbar": "Slideshow", 365 | "kernelspec": { 366 | "display_name": "Python 3", 367 | "language": "python", 368 | "name": "python3" 369 | }, 370 | "language_info": { 371 | "codemirror_mode": { 372 | "name": "ipython", 373 | "version": 3 374 | }, 375 | "file_extension": ".py", 376 | "mimetype": "text/x-python", 377 | "name": "python", 378 | "nbconvert_exporter": "python", 379 | "pygments_lexer": "ipython3", 380 | "version": "3.6.3" 381 | } 382 | }, 383 | "nbformat": 4, 384 | "nbformat_minor": 2 385 | } 386 | --------------------------------------------------------------------------------