├── .gitattributes ├── .gitignore ├── README.md └── notebooks ├── 1-Pytorch-Introduction.ipynb ├── 2-Pytorch-Autograd.ipynb ├── 3-Pytorch-Optimizers.ipynb ├── 4-Pytorch-Modules.ipynb ├── 5-Pytorch-Dataloader.ipynb ├── 6-Pytorch-Alexnet-Example.ipynb ├── autograd-graph.png ├── dataloader.png ├── how-to-read-pytorch.png ├── ipynb_drop_output.py └── setup_notebooks.sh /.gitattributes: -------------------------------------------------------------------------------- 1 | *.ipynb filter=clean_ipynb 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | /datasets 3 | /models 4 | /checkpoints 5 | /notebooks/datasets 6 | /notebooks/models 7 | /notebooks/checkpoints 8 | __pycache__ 9 | .DS_Store 10 | .__* 11 | .ipynb* 12 | .nfs* 13 | .*swp 14 | .idea 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | David's Tips on How to Read Pytorch 2 | =================================== 3 | 4 | ![Figure thumbnails](notebooks/how-to-read-pytorch.png) 5 | 6 | These five python notebooks are an illustrated introduction to core pytorch idioms. Click below to run them on Colab. 7 | 8 | 1. [Tensor arithmetic](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks/1-Pytorch-Introduction.ipynb): the notation for manipulating n-dimensional arrays of numbers on CPU or GPU. 9 | 2. [Autograd](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks//2-Pytorch-Autograd.ipynb): how to get derivatives of any scalar with respect to any tensor input. 10 | 3. [Optimization](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks//3-Pytorch-Optimizers.ipynb): ways to update tensor parameters to reduce any computed objective, using autograd gradients. 11 | 4. [Network modules](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks//4-Pytorch-Modules.ipynb): how pytorch represents neural networks for convenient composition, training, and saving. 12 | 5. [Datasets and Dataloaders](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks//5-Pytorch-Dataloader.ipynb): for efficient multithreaded prefetching of large streams of data. 13 | 14 | Pytorch is a numerical library that makes it very convenient to train deep networks on GPU hardware. It introduces a new programming vocabulary that takes a few steps beyond regular numerical python code. Although pytorch code can look simple and concrete, much of of the subtlety of what happens is invisible, so when working with pytorch code it helps to thoroughly understand the runtime model. 15 | 16 | For example, consider this code: 17 | 18 | ``` 19 | torch.nn.cross_entropy(model(images.cuda()), labels.cuda()).backward() 20 | optimizer.step() 21 | ``` 22 | 23 | It looks like it computes some function of `images` and `labels` without storing the answer. But actually the purpose of this code is to update some hidden parameters that are not explicit in this formula. This line of code moves batches of image and label data from CPU to the GPU; runs a neural network to make a prediction; constructs a computation graph describing how the loss depends on the network parameters; annotates every network parameter with a gradient; then finally it runs one step of optimization to adjust every parameter of the model. During all this, the CPU does not see any of the actual answers. That is intentional for speed reasons. All the numerical computation is done on the GPU asynchronously and kept there. 24 | 25 | The berevity of the code is what makes pytorch code fun to write. But it also reflects why pytorch can be so fast even though the python interpreter is so slow. Although the main python logic slogs along sequentially in a single very slow CPU thread, just a few python instructions can load a huge amount of work into the GPU. That means the program can keep the GPU busy churning through massive numerical computations, for most part, without waiting for the python interpreter. 26 | 27 | Is is worth understanding five idioms that work together to make this possible. The five notebooks in this directory are a quick overview of these idioms. 28 | 29 | The key ideas are illustrated with small, runnable, tweakable examples, and there are links to other reference material and resources. 30 | 31 | All the notebooks can be run on Google Colab where GPUs can be used for free. Or they can be run on your own local Jupyter notebook server. The examples should all work with python 3.5 or newer and pytorch 1.0 or newer. 32 | 33 | [Start with the first notebook here!](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks/1-Pytorch-Introduction.ipynb) 34 | 35 | --- *David Bau, July 2020* 36 | 37 | ([David](https://people.csail.mit.edu/davidbau/home/) is a PhD student at MIT and former Google engineer. His research pursues transparency in deep networks.) 38 | -------------------------------------------------------------------------------- /notebooks/1-Pytorch-Introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "How to Read Pytorch\n", 8 | "===================\n", 9 | "\n", 10 | "These five python notebooks are an introduction to core pytorch idioms.\n", 11 | "\n", 12 | "Pytorch is a numerical library that makes it very convenient to train deep networks on GPU hardware. It introduces a new programming vocabulary that takes a few steps beyond regular numerical python code. Although pytorch code can look simple and concrete, much of of the subtlety of what happens is invisible, so when working with pytorch code it helps to thoroughly understand the runtime model.\n", 13 | "\n", 14 | "For example, consider this code:\n", 15 | "\n", 16 | "```\n", 17 | "torch.nn.cross_entropy(model(images.cuda()), labels.cuda()).backward()\n", 18 | "optimizer.step()\n", 19 | "```\n", 20 | "\n", 21 | "It looks like it computes some function of `images` and `labels` without storing the answer. But actually the purpose of this code is to update some hidden parameters that are not explicit in this formula. This line of code moves batches of image and label data from CPU to the GPU; runs a neural network to make a prediction; constructs a computation graph describing how the loss depends on the network parameters; annotates every network parameter with a gradient; then finally it runs one step of optimization to adjust every parameter of the model. During all this, the CPU does not see any of the actual answers. That is intentional for speed reasons. All the numerical computation is done on the GPU asynchronously and kept there.\n", 22 | "\n", 23 | "The berevity of the code is what makes pytorch code fun to write. But it also reflects why pytorch can be so fast even though the python interpreter is so slow. Although the main python logic slogs along sequentially in a single very slow CPU thread, just a few python instructions can load a huge amount of work into the GPU. That means the program can keep the GPU busy churning through massive numerical computations, for most part, without waiting for the python interpreter.\n", 24 | "\n", 25 | "Is is worth understanding five core idioms that work together to make this possible. This tutorial has five Colab notebooks, one for each topic:\n", 26 | "\n", 27 | " 1. GPU Tensor arithmetic ([this notebook on colab](https://colab.research.google.com/github/davidbau/how-to-read-pytorch/blob/master/notebooks/1-Pytorch-Introduction.ipynb)): the notation for manipulating n-dimensional arrays of numbers on CPU or GPU.\n", 28 | " 2. [Autograd](./2-Pytorch-Autograd.ipynb): how to build a tensor computation graph and use it to get derivatives of any scalar with respect to any input.\n", 29 | " 3. [Optimization](./3-Pytorch-Optimizers.ipynb): ways to update tensor parameters to reduce any computed objective, using autograd gradients.\n", 30 | " 4. [Network modules](./4-Pytorch-Modules.ipynb): how pytorch represents neural networks for convenient composition, training, and saving.\n", 31 | " 5. [Datasets and Dataloaders](./5-Pytorch-Dataloader.ipynb): for efficient multithreaded prefetching of large streams of data.\n", 32 | "\n", 33 | "The key ideas are illustrated with small, illustrated, hackable examples, and there are links to other reference material and resources.\n", 34 | "\n", 35 | "All the notebooks can be run on Google Colab where some GPU compuation can be used for free, or they can be run on your own local Jupyter notebook server.\n", 36 | "\n", 37 | "The examples should all work with python 3.5 or newer and pytorch 1.0 or newer.\n", 38 | "\n", 39 | "The original [code on github can be found here](https://github.com/davidbau/how-to-read-pytorch).\n", 40 | "\n", 41 | "--- [*David Bau, July 2020*](http://davidbau.com/archives/2020/07/05/davids_tips_on_how_to_read_pytorch.html)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "Topic 1: pytorch Tensors\n", 49 | "===============\n", 50 | "\n", 51 | "The first big trick for doing math fast on a modern computer is to do giant array operations all at once.\n", 52 | "\n", 53 | "To faciliate this, pytorch provides a [`torch.Tensor`](https://pytorch.org/docs/stable/tensors.html) class that is a lookalike to the older python numerical library [`numpy.ndarray`](https://numpy.org/doc/1.18/reference/arrays.ndarray.html). Just like a numpy `ndarray`, the pytorch `Tensor` stores a d-dimensional array of numbers, where d can be zero or more, and where the contained numbers can be any of the usual selection of float or integer types. Pytorch is designed to feel just like numpy: almost all the numpy operations are also available on torch tensors. But if something is missing, torch tensors can be directly converted to and from numpy using `x.numpy()` and `torch.from_numpy(a)`. So what is different and why did the pytorch authors bother to reimplement this whole library?\n", 54 | "\n", 55 | "**There are two things that pytorch Tensors have that numpy arrays lack:**\n", 56 | "\n", 57 | " 1. pytorch Tensors can live on either **GPU or CPU** (numpy is cpu-only).\n", 58 | " 2. pytorch can automatically track tensor computations to enable **automatic differentiation**.\n", 59 | "\n", 60 | "In the following sections on this page we talk about the basics of the Tensor API as well as point (1) - how to work with GPU and CPU tensors. A discussion of (2) can be found in the next notebook, [2. Autograd](https://colab.research.google.com/github/davidbau/pytorch-tutorial/blob/master/notebooks/2-Pytorch-Autograd.ipynb).\n" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "Basic operations in the Tensor API\n", 68 | "----------------------------------\n", 69 | "\n", 70 | "Pytorch is not very different from numpy, although the pytorch API has more convenience methods such as `x.clamp(0).pow(2)` (supporting a chained method style, as is popular in Javascript libraries, so you don't need to say the verbose `numpy.pow(numpy.clip(x, 0), 2)`). So code is often shorter in pytorch. A brief overview:\n", 71 | "\n", 72 | "**Elementwise operations.** Most tensor operations are simple (embarassingly parallelizable) elementwise operations, where the same math is done on every element of the array. `x+y`, `x*y`, `x.abs()`, `x.pow(3)`, etc. Unlike Matlab, `*` is for element-wise multiplication, not matrix-multiplication.\n", 73 | "\n", 74 | "**Copy semantics by default.** Almost all operations, including things like `x.sort()`, return a new copy of the tensor without overwriting the input tensors. The exceptions are functions that end in an underscore such as `x.mul_(2)` which doubles the contents of x in-place.\n", 75 | "\n", 76 | "**Common reduction operations.** There are some common operations such as `max`, `min`, `mean`, `sum` that reduce the array by one or more dimension. In pytorch, you can specify which dimension you want to reduce by passing the argument `dim=n`.\n", 77 | "\n", 78 | "**Why does min return two things?** Note that `[data, indexes] = x.sort(dim=0)` and `[vals, indexes] = x.min(dim=0)` return the pair of both the answer and the index values, so you do not need to separately recompute `argsort` or `argmin` when you need to know where the min came from.\n", 79 | "\n", 80 | "**What about linear algebra?** It's there. `torch.mm(a,b)` is matrix multiplication, `torch.inverse(a)` inverts, `torch.eig(a)` gets eigenvalues, etc.\n", 81 | "\n", 82 | "The other thing to know is that pytorch tends to be very fast, often much faster than numpy even on CPU, because its implementation is aggressively parallelized behind-the-scenes. Pytorch is willing to use multiple threads in situations where numpy just uses one.\n", 83 | "\n", 84 | "See the [reference for Tensor methods](https://pytorch.org/docs/stable/tensors.html#torch.Tensor) for what comes built-in. A simple demo of some vectors:\n", 85 | "\n" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 1, 91 | "metadata": {}, 92 | "outputs": [ 93 | { 94 | "name": "stdout", 95 | "output_type": "stream", 96 | "text": [ 97 | "tensor([0.0000, 0.0500, 0.1000, 0.1500, 0.2000])\n" 98 | ] 99 | } 100 | ], 101 | "source": [ 102 | "import math, numpy, torch\n", 103 | "from matplotlib import pyplot as plt\n", 104 | "\n", 105 | "# Make a vector of 101 equally spaced numbers from 0 to 5.\n", 106 | "x = torch.linspace(0, 5, 101)\n", 107 | "\n", 108 | "# Print the first five things in x.\n", 109 | "print(x[:5])" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": {}, 115 | "source": [ 116 | "### Exercise\n", 117 | "\n", 118 | "Print the last five things in x." 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": 6, 124 | "metadata": {}, 125 | "outputs": [ 126 | { 127 | "name": "stdout", 128 | "output_type": "stream", 129 | "text": [ 130 | "TODO\n" 131 | ] 132 | } 133 | ], 134 | "source": [ 135 | "# TODO: Print the last five things in x (instead of the first five)\n", 136 | "\n", 137 | "print('TODO')" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 3, 143 | "metadata": {}, 144 | "outputs": [ 145 | { 146 | "name": "stdout", 147 | "output_type": "stream", 148 | "text": [ 149 | "The shape of x is torch.Size([101])\n", 150 | "The shape of y1=x.sin() is torch.Size([101])\n", 151 | "The shape of y2=x ** x.cos() is torch.Size([101])\n", 152 | "The shape of y3=y2 - y1 is torch.Size([101])\n", 153 | "The shape of y4=y3.min() is torch.Size([]), a zero-d scalar\n" 154 | ] 155 | }, 156 | { 157 | "data": { 158 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA5vElEQVR4nO3dd3hU1dbA4d9OgST0EpASCB0BKYIgRUARRaQIiBUEpXzY69ULKhd75YrtKtiQDiLSQRGkSQ1VuvQOAUIJgZSZ9f2xg7QAKTNzZpL1Ps95MjPnzDlrksyaPbsaEUEppVT2F+R0AEoppXxDE75SSuUQmvCVUiqH0ISvlFI5hCZ8pZTKIUKcDuBqihYtKtHR0U6HoZRSAWPFihVHRCQyrX1+nfCjo6OJiYlxOgyllAoYxphdV9qnVTpKKZVDaMJXSqkcQhO+UkrlEJrwlVIqh9CEr5RSOYQmfKWUyiE04SulVA6R5YRvjIkyxvxhjNlojFlvjHk2jWOaG2NOGGNWp279s3pdpZTKjn7b9hufL/2cJFeSx8/tiYFXKcCLIrLSGJMPWGGMmSUiGy45boGItPHA9ZRSKlsSEV75/RVOJ53m8Zse9/j5s5zwReQAcCD19iljzEagFHBpwldKKXUVkzZPYvXB1Qy7ZxghQZ6fCMGjdfjGmGigDrA0jd0NjTFrjDEzjDHVr3KO3saYGGNMTGxsrCfDU0opv+UWNwPmDqBS4Uo8eMODXrmGxxK+MSYv8DPwnIicvGT3SqCsiNQCPgcmXuk8IjJEROqJSL3IyDTn/1FKqWxn0qZJrDm0hv7N+nuldA8eSvjGmFBssh8pIhMu3S8iJ0UkPvX2dCDUGFPUE9dWSqlA5xY3A+YNoHKRyjxQ4wGvXSfLHyPGGAN8B2wUkf9e4ZjrgEMiIsaY+tgPmqNZvbZSSmUHv2z8hbWH1jKiwwivle7BM710GgNdgb+MMatTH+sHlAEQka+Be4HHjTEpwBngARERD1xbKaUCmsvtov/c/lQpUsWrpXvwTC+dhYC5xjFfAF9k9VpKKZXdDFszjA2xGxjfeTzBQcFevZaOtFVKKYecST5D/7n9qV+qPh2v7+j16/n1ildKKZWdfbn8S/ae3MvwDsOxzaHepSV8pZRywPGzx3l3wbu0qtiK5tHNfXJNTfhKKeWADxZ+QNzZON5r8Z7PrqkJXymlfGxH3A4+WfIJXWp2ofZ1tX12XU34SinlYy///jLBQcE+Ld2DJnyllPKpeTvnMX7DeF5p/Aql85f26bU14SullI+43C6e//V5ovJH8VKjl3x+fe2WqZRSPjJ09VBWHVzF6E6jiQiN8Pn1tYSvlFI+cOzMMfrO7kujqEbcX/1+R2LQhK+UUj7Qb3Y/jp05xv9a/88ng6zSoglfKaW8bOnepQxZMYRnGjxDretqORaHJnyllPKiFHcKj097nBL5SjCg+QBHY9FGW6WU8qKvln/FqoOrGHvvWPLnzu9oLFrCV0opL9l9Yjf95vTjjgp30LlaZ6fD0YSvlFLeICL839T/Q0QY3GawYw21F9IqHaWU8oIRa0cwc+tMPm31KdEFo50OB9ASvlJKedyh+EM89+tzNIpqxFP1n3I6nH9owldKKQ8SEZ6c/iTxSfF81+47goz/pNksR2KMiTLG/GGM2WiMWW+MeTaNY4wx5jNjzFZjzFpjzI1Zva5SSvmjkX+N5OeNP/NG8zeoWrSq0+FcxBN1+CnAiyKy0hiTD1hhjJklIhsuOOYuoFLq1gD4KvWnUkplG3tO7OGp6U/ROKox/2r0L6fDuUyWS/gickBEVqbePgVsBEpdclh7YJhYS4CCxpgSWb22Ukr5C7e46T6pOy5xMazDMIKDgp0O6TIerVwyxkQDdYCll+wqBey54P5eLv9QOHeO3saYGGNMTGxsrCfDU0opr/l86efM2TGHT+78hPKFyjsdTpo8lvCNMXmBn4HnROTkpbvTeIqkdR4RGSIi9USkXmRkpKfCU0opr1lzcA2v/P4KbSu3pUedHk6Hc0UeSfjGmFBssh8pIhPSOGQvEHXB/dLAfk9cWymlnHQ66TT3j7+fwuGF+a7dd34xwOpKPNFLxwDfARtF5L9XOGwy8Ehqb52bgRMiciCr11ZKKac9PeNpthzdwsiOI4nM49+1Ep7opdMY6Ar8ZYxZnfpYP6AMgIh8DUwHWgNbgQTgUQ9cVymlHDXqr1H8sPoHXm/6OreWu9XpcK4pywlfRBaSdh39hccI8GRWr6WUUv5i05FN/N/U/6NJmSb0b9bf6XDSxX+GgCmlVICIT4qn49iOhIeEM7rTaEKCAmNassCIUiml/ISI0HNyTzYf3cysrrMonb+00yGlmyZ8pZTKgM+Xfc7Y9WN5r8V73FbuNqfDyRCt0lFKqXSat3MeL/72Iu2qtOPlxi87HU6GacJXSql02Hl8J/f+dC8VC1dk2D3D/GoWzPQKvIiVUsrHTiedpv2Y9iS7kpn0wCQKhBVwOqRM0Tp8pZS6inOToq07vI7pD02ncpHKToeUaZrwlVLqKl6f8zrjN4zn45Yfc2fFO50OJ0u0Skcppa5g6OqhvLvwXXrd2IsXGr7gdDhZpglfKaXSMHfnXHpP6U3L8i35svWXfj0pWnppwldKqUtsiN1Ax7EdqVSkEuM6jyM0ONTpkDxCE75SSl1g38l9tBrRitwhuZn20DQKhhV0OiSP0UZbpZRKdeLsCe4aeRdxZ+OY330+0QWjnQ7JozThK6UUkJiSSIexHdh4ZCPTH5pOnRJ1nA7J4zThK6VyvBR3Cg9NeIg/dv7B8A7DaVmhpdMheYXW4SulcjQRoc/UPkzYOIFBdw6iS80uTofkNZrwlVI52iu/v8J3q77j9aav8+zNzzodDmfOgNvtnXNrwldK5VjvzH+HjxZ9xBP1nuCN5m84HQ4Ab78NNWvC2bOeP7cmfKVUjjRoySBe++M1utbsyuetP/eLgVWnT8NXX0GVKhAW5vnzeyThG2O+N8YcNsasu8L+5saYE8aY1albYCwAqZTKloasGMLzvz5Pp+s78X377/1mquOhQyEuDl7w0iwOnuqlMxT4Ahh2lWMWiEgbD11PKaUy5cfVP9Jnah9aV2rNqE6j/GY9WpcLPvkEGjSARo28cw2PfKyJyHzgmCfOpZRS3jJy7UgenfQoLcq3YHzn8eQKzuV0SP+YMgW2bYMXXwRv1S758ntMQ2PMGmPMDGNM9SsdZIzpbYyJMcbExMbG+jA8pVR2NnbdWB6Z+AjNo5sz6YFJhIeGOx3SRQYOhOho6NDBe9fwVcJfCZQVkVrA58DEKx0oIkNEpJ6I1IuMjPRReEqp7Gzc+nE8POFhmpRpwpQHpxARGuF0SBdZtgwWLoTnnoMQL9Yw+SThi8hJEYlPvT0dCDXGFPXFtZVSOduYdWN46OeHaBTViKkPTiVPrjxOh3SZjz6CAgXgsce8ex2fJHxjzHUmtc+TMaZ+6nWP+uLaSqmca9Rfo3h4wsM0LtOY6Q9PJ1/ufE6HdJmNG+Hnn+GppyCfl8PzyJcHY8xooDlQ1BizF/gPEAogIl8D9wKPG2NSgDPAAyIinri2UkqlZcTaEXSb2I2mZZv6bcke4N13ISLCVud4m0cSvog8eI39X2C7bSqllNd9v+p7ek7uya3lbmXyA5P9Ntlv2wajRtl+90V9UMntH6MNlFLKQ76O+Zoek3twR4U7/LpkD/DeexAaarti+oImfKVUtjFoySAen/Y4bSq3YeIDE/2u6+WFdu+GH3+EXr3guut8c01N+EqpbOGd+e/w/K/P0/H6jvx838+EhXhhMhoP+uADO8Dq5Zd9d01N+EqpgCYi9Jvdj9f+eI0uNbsw9t6xfjWCNi3bt8M330CPHhAV5bvr+sckEkoplQlucfP8zOf5bNln9L6xN1+1+cpvJkK7mv/8xw6wev11315XE75SKiC53C56TenFD6t/4Pmbn2fgHQP9Yorja/nrLxg5Ev71LyhZ0rfX1oSvlAo4Sa4kukzowk8bfmJAswH0b9Y/IJI9wGuvQf788Morvr+2JnylVEBJSE7g3nH3MmPrDAbeMZAXGnpp8ngvWLQIJk+2q1oVLuz762vCV0oFjBNnT9B2dFsW7l7IkDZD6FW3l9MhpZuILdUXKwbPOrR0riZ8pVRAiD0dS6uRrVh7aC2jO43m/hr3Ox1Shvz0k50R8+uvIW9eZ2LQhK+U8nt7Tuyh5fCW7Dqxi0kPTKJ1pdZOh5QhZ87YRtpataBnT+fi0ISvlPJrW45u4fZht3Mi8QS/dfmNW8re4nRIGTZw4PmRtcHBzsWhCV8p5bdWHVjFnSPuBGBut7nUKVHH4Ygybt8+O2dOx47QvLmzsfj/CAWlVI60YNcCmv/YnPDQcBY+tjAgkz3Av/9tFyj/6COnI9GEr5TyQ1O3TOWOEXdQMl9JFj66kMpFKjsdUqbMng0jRtj6+/LlnY5GE75Sys+MXDuSe8bcQ41iNVjw6AKiCvhwshkPOnMG+vSBihWhXz+no7G0Dl8p5Tc+W/oZz858lubRzZn0wCTy587vdEiZ9u67sHUr/P47hPvJLM2a8HOYxES7hua6dbYx6fBhuyUm2qlajbF9hIsXtwNEoqOhWjX7ddTJ3gUqexMRBswdwJvz3+SeqvcwutNov5/e+Go2bLDTH3ftCi1aOB3NeZrws7nTp2H+fJg1y9Ynrl9vG5DOyZMHIiMhLMyOBBSBU6fsh8CFx4WFQc2acMst0LSp/VmokO9fj8p+3OLm6elP87+Y//Fo7UcZ0nYIIUGBm5pcLruoSf78tjumP/HUIubfA22AwyJSI439BvgUaA0kAN1FZKUnrq0ul5wMv/5qG4smTYKzZyF3bmjSxPYYuOEGqFHDlt7zXGH1N7cbjh2z83avX2+3Zcvgiy/sP3FwsE387dpBhw5QtqxPX6LKJpJcSXSb2I0x68bwYsMX+ajlRwEzCdqVDBxo58wZPtwWpvyKiGR5A5oCNwLrrrC/NTADMMDNwNL0nLdu3bqi0u/wYZEBA0SKFbNl9SJFRJ58UmTWLJGEBM9c48wZkfnzRfr1E6le/dx3ApGmTUW++Ubk+HHPXEdlf/GJ8dJqRCthAPLBwg+cDscj1qwRyZVLpFMnEbfbmRiAGLlCTjV2f9YZY6KBqZJ2CX8wMFdERqfe3ww0F5EDVztnvXr1JCYmxiPxZWf79sE778APP9jSfOvWtndAq1Z2gWSPcrnsBffuhUOH2Lb+LGMXlODHmBpsOVaUiJBEHq4cw5M1F1CrxGHbIJAvHxQsaBfuLFECSpWytwO8JKcy79iZY7QZ1Yal+5YyuM1get7o4HwDHpKYCPXrw6FDds57p0r3xpgVIlIvrX2+qigrBey54P7e1McuS/jGmN5Ab4AyZcr4JLhAdfw4fPghDBoEKSnQrRu88AJcf70HTi4Cf/8NK1fa/961a2HzZti509YZpaoA9AP6YlgW3oxv3I8xYkMnvtnQmFuC/uRf7ve5m2kEcUnBIiLC9lerVMlOMFKrFtSpA6VL6wdBNrf/1H7uHHEnW45u4afOP9Hx+o5Oh+QRb7xh3yaTJ/thVU4qXyX8tN7BaX61EJEhwBCwJXxvBhWoRGxp/uWX4ehRePhheOstKFcuCyd1uWD1atuHbMECWLLEnhzsWmxVq9qk3LGj7bJTpsz5rjyFC2PCwmhgDA2AD4/Z+D77rDHtdk+hWlUXL/c6zsMNtxMSewD27IFt22yftTVrYMIE+6LAlv6bNLHb7bdDlSr6AZCNbD22lZbDW3Ik4QgzHp7BbeVuczokj5g1C95/Hx57DNq2dTqaq7hSXU9GNyCaK9fhDwYevOD+ZqDEtc6pdfiX27DB1peDSJMmIitXZuFkx4+LjBolcv/9IoULn6+Qr1pV5LHHbKX8qlUiZ89m6vRJSSIjRojccIM9bYUKIj/8IJKcfMmBp06JLFok8vnnIg88IBIVdT6WMmVEevYUmTLFNiCogLVy/0op9lExKfphUVm+b7nT4XjMvn0ikZEi1aqJxMc7Hc3V6/B9lfDv5uJG22XpOacm/PNSUkQ+/FAkNFSkUCGRb78VcbkycaKTJ0V+/FGkVSt7MrCtvN27i4wcKXLggMdjd7tFJk0SqVPnfOIfNeoa8W/fLjJ4sEjHjiL589sn5s1rP5wmThRJTPR4nMp75u6YK/nfyy9R/42SjbEbnQ7HY5KTRZo1E4mIEFm/3uloLK8nfGA0tj4+GVs/3wPoA/RJ3W+AL4FtwF9AvfScVxO+tWuXSPPm9q/VsaPIoUMZPIHLJfL777b0HBZmT1S2rMhLL4n8+af9NPGBc4m/Zk0bQq1aItOmpaM3Q2KiyIwZIr1726IU2G8kjz8usmKFL0JXWTBp0yTJ/VZuuf6L62X38d1Oh+NRr75q/x1//NHpSM7zSQnfG5smfFuYLVDAFm5/+CGDXb2OHBF5/32R8uXtn7pQIZEnnrBJ3qk+Y2I/f0aNOh/WrbdmIG8nJdlPiQs/vOrVs195Tp/2atwq435Y9YMEvxEsNw25SWJPxzodjkeNH2///R57zOlILqYJPwAlJ4u88sr5fLZtWwaevH69LQ2Hh8s/neRHjPC7OvDERJEvvhApWtSG2aWLyO6MFADj4kQ+++z8gIAiRWyRa/9+b4WsMuCjPz8SBiAth7WUU4mnnA7Ho1atstU4N9/sd28rTfiB5vBhW+oFkf/7vwy0mS5ZItK+vX1iWJhIr14if/3lzVA94vhxkX//WyR3bvsZ9frrth033dxukXnzRDp0EDHGtk307CmydavXYlZX5na75ZVZrwgDkM7jOsvZ5Mw1+vurgwdtv4LSpb3S5JVlmvADyPr1IuXK2eQ3dGg6n7RwoUiLFvJPtc2AASKxgff1eccO2yYLIiVLigwblomG6a1b7fDi3LlFgoJEHn5YZNMmb4Sr0pDsSpYek3oIA5A+U/pIiss37UO+kpAg0qiRLZhkqYecF2nCDxAzZ9oOKcWL28L6NcXEiNx1l/zT0+ajj2wvnAD355+2GgtEGjRI5+/iUvv320bpPHls4n/0UZGdOz0eqzrvTPIZuWfMPcIApP+c/uJ2sJ3IG5KTRdq1s18ix493Opor04QfAL75RiQ42PZg2bXrGgdv324bLc/1VvngA//oAOxBLpf9hnPddfZlduuWyar5Q4dEnn/elvhDQ0Wee07k6FFPh5vjnTh7QpoPbS4MQD5d8qnT4Xic220bZ0Hkyy+djubqNOH7Mbfb1sCALaxftYB+/LjIiy/a2ZnCw0Vee03kxAmfxeqEkydt43WuXLan0nvvZXIc2J49Ij162NJ+wYIiAwdmekCZutjBUwelztd1JOTNEBm5dqTT4XhF3772Pdq/v9ORXJsmfD+VnGw705wrwSYlXeFAl0vku+9stY0xtqixd68vQ3Xc33+fb4+uUMF2V81UjcHatSJ33mlPVLmyrUdTmbb92Hap+FlFiXgnQmb8PcPpcLziP/+x/y69ezvamzndNOH7ocREkXvvtX+Bvn2v8o+0YoVI/fr2wMaN/belyEd++80OYQeR22/PQiek6dNFKlWyJ7rnHq3fz4Q1B9dIiY9LSKH3C8mi3YucDscrziX7Rx/N5Mh2B2jC9zOnT59va/344yscdPKkrW8OCrIV2cOHB0bxwgeSkmz3+4IF7a/niScy2Snp7FmRd9+1HaojIuwf47KJflRa5u+cLwXeKyClBpaSdYfWOR2Ox7ndgZnsRTTh+5WTJ+04KGPsVDFpmjLFdvI1xmazuDhfhhgwYmNtD8zgYJv8//vfTE6xs2uXSJs29u1Qu7bt/aSuaPKmyRL2dphU/ryy7IzLft+MXC6RZ54JzGQvognfbxw/LtKwoU1Qo0alccCRI3a4KYjUqCGyeLHPYwxE69aJ3HGH/bVVrCgyYUImvgy53bavXYkS9g/Ut6826qbh+5XfS/AbwVJvSD05HH/Y6XA8LjHxfAe4558PvGQvognfL8TF2ar4kBCbkC7zyy+2UTYkxH6X1NkgM2zGjPP1+7fcksn++3FxtlgHItdfL7J0qafDDEhut1s+WPiBMAC5fdjtcvJs4I/3uFRcnG0XAtvTOVBrUDXhO+zYMZG6dW038IkTL9l5/LjtonOuOmHNGidCzDaSk0W++ur8ur6dO9sePhk2Y4atVgsOth/AV+xClf253C55YeYLwgDkgfEPSGJK9iuMbNpkO22FhNhJCgOZJnwHxcXZZJ8rl62av8jcuXZSjuBg26deS/Uec/Kk7TMdEWHfxH36ZGLgVlycSNeu8s8Mdhuzzzzu6ZWYkigP//ywMAB5evrT4nIHYB3HNcycaWekLVrUTskU6DThO+T4cVuNExp6SbJPShLp1882ylasmMm6B5UeBw7Yht2QEDtW7eWXM9GjZ/x4OxNnRIQdEh2o3/Uz6FTiKblz+J3CAOSd+e9ku6kSUlLsoMegIDvCfccOpyPyDE34Djhxwk6dGhJiF/34x7Zt5/vVP/ZYBqeFVJm1daudR80YO2L3tddsVVu67dt3voL33nsz+OTAczj+sNw05CYJeiNIvl3xrdPheNy+fecXFeraNXu9DTXh+1h8vF1vNjj4kgbacePs7GgFC9rbyufWr7f1+iCSL5+dPv/IkXQ+2eWyrXkhIXat3Wzai2rbsW1S6bNKEvZ2mEzaNOnaTwgwEybYhdMiIjIwI20A0YTvQwkJdqbioCCRsWNTHzxzxi7HB7bYr6M6HbdmjU3850r8L71kS33psmyZSHS0Tfwff5ytqnhW7l8pxT8qLoXeLyR/7v7T6XA86sgRkQcftG/DOnX8Zw1aT9OE7yOJiSKtW9skMmxY6oPbtp1fvftf/8rRvT380bp1Ig89ZD+gc+Wy66aka/r8uDi7wDDYOXOzweC4WdtmSb5380nUf6Nkw+ENTofjMW63HfdSvLj9jH7jjez9NvTFIuatgM3AVuDfaexvDpwAVqdu/dNz3kBK+MnJ59//Q4akPjhliq2+KVhQZPJkR+NTV7dtmx3UfG6Z3DZtRObMuUbh3e0W+fRTm0XKl7fr3gWoEWtGSMibIVLjfzVkz4k9TofjMevXn189rm7dgP4TpZtXEz4QDGwDygO5gDVAtUuOaQ5Mzei5AyXhu1y2QRBEBg0S2/z/2mv2gRtvtPPXq4Bw6JDtuREZaf98N9xgp8C46nIDixaJlCplPy0CrBP3hQOqmg9tLnFn4pwOySMOHTrfO6tQITs2IyV7Lb51Rd5O+A2BXy+43xfoe8kx2Tbhu93npzh+5x2xvTfOzYzWo4f/rXCs0uXMGZFvv7Vj4cD2037mmavMznnokMhtt9mDn3wyIMZUpLhS5MlpTwoDkPt/uj9brD174oStssmb13aa6NPHrhGdk3g74d8LfHvB/a7AF5cc0xw4mlr6nwFUv8r5egMxQEyZMmW8/svJCrdb5IUX5J8pjmXtWjtZe2joVWZGU4HE7bZLBj/4oK3jP9fu/s03dpzFRZKTbesv2G5aBw86EnN6nE46Le1HtxcGIC/++mLAD6iKi7OJvlAh++vv2DHnLmXs7YTfOY2E//klx+QH8qbebg38nZ5z+3sJv39/+xt8+mkR988T7PqpJUrYr/gq24mNtQtlVa1q/+5hYfaDYPr0SxoBR4+2o7xKlRJZvtyxeK/kcPxhufnbm8UMMPLZks+cDidLdu60n7H588s/7efLljkdlbMcr9JJ4zk7gaLXOrc/J/wPPrC/vccedYur/wB7p379DPTtU4HK7baDox9//HyJsmhRW30wZ07qlPqrV9u++mFhV5ga1Rmbj2yWCp9WkLC3w+TnDT87HU6muFwis2fb8W9BQbbq5r77ckaDbHp4O+GHANuBchc02la/5JjrAJN6uz6w+9z9q23+mvC//NL+5u7vlCwpne6zdx55ROvrc6CzZ+2EeA88YAfynEv+PXqITBkeJwmNWpyv83N4rt0FuxZI4Q8KS+SHkbJ4T+ANGtu1y7aTlS9vf6WFCtn1jnfvdjoy/+KLbpmtgS2pvXVeTX2sD9An9fZTwPrUD4MlQKP0nNcfE/7Qofa31rZlgiTVqW873WezwTcqc+Lj7bQ7Dz10voohPNwtbcqskf/RR7a27HON7j7eM2rtKMn9Vm6p/Hll2Xp0qyMxZMaBA7aA1aSJ/X2C7WY5cqQd5Kgud7WEf67U7Zfq1asnMTExTofxj3Hj4MEHocVNJ5i8qzZh8Udg9Gho08bp0JSfSUyE+fNh6lSYMkXYscMAUD7XHm6/txDN2+SlWTMoWdK7cYgIb89/m/5z+9O0bFMm3DeBIhFFvHvRLBCB9eth2jSYOBGWLrWPVasGDz0EDzwAFSo4HaV/M8asEJF6ae7ThJ8+U6ZAx45wc6UjzNxRlTzF89p3c40aToem/JwIbN0Kv32ynt++2clc1y2clPwAVKwIjRrZrWFDm9hCQjxz3cSURHpN6cXwtcPpWrMr37T9htwhuT1zcg8RgR07YMECmDMHZs2CAwfsvrp1oX17uOce+zYzxtFQA4Ym/Cz67Tdo21aoWewQs/dWIX+D62HSJChe3OnQVKBZtw7X3e1YfbgEcx8YzPxjNVi8GGJj7e7wcKhTxya7WrXsVr26fTwjYk/H0mFsB/7c8ydvNn+T15q+hvGDjHnyJKxaBcuW2W3xYti3z+4rUgRatoQ77rBbqVLOxhqoNOFnwbx5cNddQuWIvcw5WovCnW+HH3/M+DtQqXMOHoR27SAmBgYORJ59ju07DIsXw4oV9uFVq+D0aXu4MVCuHFx/vd0qVbLfDCpUsEnx0m8E6w6vo+3othyMP8iP9/zIfdXv8+nLE4Fjx+Dvv2HzZrutXw9r18LOneePK1cOGjSAW26Bpk3tt5ugIJ+Gmi1pws+kxYuhZUuhDHuYd7oukf/uCe+8o/+VKusSEqBrV5gwAZ56CgYNguDgf3a73bB9u02Sa9fCxo1227LFtg+cExxsk35UlP15tsxUfs37EOHBeXm90iQalb2JggWhQAG7hYdnrmpEBM6cgePH4cQJOHoUjhyx30wOHLCl9P37Ydcum9RPnTr/3JAQ+yFVqxbUrGl/3nQTREZm8nenrkoTfiYsWwYtb3dTLHEv81MaUeKr/tC7tyOxqGzK7YaXX4aBA22Jf9QoyJPnqk9xuWxy3boVtm2D3bttkt21W9hQ6H2O1HoVDtwIYybCydKXPT8oCCIi7JY7N4SGQq5cF5dhXC5ITrbb2bM20Sck2HCvJDLSNkBHRdmSe7ly9ltIlSr2dmhoJn9HKsM04WfQypXQormLQgl7mZf7TqJ+HgStWvk8DpVDfPEFPPusrbifOhWKFcvQ0xOSE+g5uSej143mwRoP8ult3xF/PPyfEvi5UvmJE7aa6PRpm8CTks5v59KAiC2Rn9vCw+2HQ3g45Mt3/ptCkSI2yRctasPN7V9twTna1RK+h/oDZB9r1kDLW5PJH3+QOUU7E/XbGKhd2+mwVHb21FNQpoztc9iwIcycaetA0mFH3A46juvImoNreL/F+7zc+GWMMUQWsiVrpS6kldEXWLsWWjRJJOLkQf4o34Po5T9psle+0a4d/PGH7cbSqBEsWXLNp8zaNot639Rj5/GdTHtoGq80ecUveuIo/6UJP9XatXBbwzOExcfyR72XKb98LJQt63RYKidp0AAWLbJ1JrfdZgd/pMEtbt5b8B6tRraiZL6SLO+1nLsq3eXjYFUg0oQP/LXGTYub4wlLOMrcO9+n4oIfoFAhp8NSOVGlSjbpV69uRxx9++1Fu+POxHHPmHvoN6cfnat1ZnGPxVQsXNGZWFXAyfEJf/XyZG5tcJrcZ44zt8t3VJz2KYSFOR2WysmKFbPVOy1bQq9e8OabIMLKAyup9009ZmydwWetPmN0p9HkzZXX6WhVAMnRjbYx805zx+0u8qYcZ86L06n4UX8dv638Q968tkqnVy/kP//hyxOzeLHQMorlKcb87vNpGNXQ6QhVAMqxCX/pjGPc2TaUQq6jzPlwBeX+1cfpkK4qPimePSf2sPfkXo4kHOHYmWPEnY3jbMpZklxJJLmSCAkKIVdwLnIF56JgWEGKRhSlaERRSucvTbmC5ciT6+p9vJWfCQ0l7n//pWeZxUwIXkibuOsY+uQSihTSOQdU5uTIhD9/7AHufjAfxTnEnO93UubRe50OCbAzG+4/tZ81h9aw9tBaNh3ZxOajm9lydAvHzhxL8znBJphcwbkIDQ7F5XaR5Eoi2Z2c5rGREZFULVqVmsVrUrN4TeqVrEfN4jUJCcqR/wZ+b/6u+XSZ0IUDoQf4OFc7XhgwGbOyi51GskABp8NTASjHvdN//Wo7HZ4oQXTQbn7/5RQl27VwLJZjZ46xZO8Slu1bxrJ9y1i+fzlHEo78s79kvpJUKVKFztU6E10wmqj8UUQViCIyIpLC4YUpFF6IXMG5LjuvW9wcP3ucIwlHOJJwhN0ndrMjbgfb47az4cgGhq0ZxqkkO/Y9IjSCBqUa0KxsM1pWaEn9UvX1A8Bhya5k3pj3Bu8ueJcKhSuw6LFF3FTqJogaBd26QbNmtq/+ddc5HaoKMDlqpO2kd9Zx32uVqBb6N7/NCSWySRWPnftaRIRdJ3axYNcCFuxewJ97/mRD7AYADIbqxapzU8mbqHNdHWpfV5uaxWtSIMw7pTgRYefxnSzdt5RFexbx554/WXVgFYJQIHcBWlZoyT1V7qF1pdYUCtfeSr60IXYDj/zyCCsOrOCx2o/x6V2fXtww++uv0KmTnan1t990cnh1GZ1aARjxXAzdP61NvbD1zFhWhEI3XD7PiCeJCH8f+5t5O+cxb9c85u+az56TewAoGFaQRlGNaBzVmIalG1KvZD3y5c7n1Xiu5WjCUebsmMNv235j2t/TOBB/gJCgEG6NvpUHazxIh+s7UDCsoKMxZmcut4tBSwbx6pxXyZc7H1/f/TWdqnVK++ClS+Huu+3MaTNn2vmUlUqV4xP+l10W89TIhtyWfzkT15QnX7TnV/xxuV38dfgvFu5eyPxd85m/az6HTh8CoHie4jQt25RmZZvRtGxTqherTpDx3x6xbnGzfN9yftn0Cz9t+IntcdvJFZyL1pVa071Wd1pXak1osM6G5SkbYzfSc0pPFu1ZRPsq7RncZjDF815jrYVNm+yk8SdOwOTJtppHKXJwwhe38F7rBbz6a1PaF1vEmPU1CSvqmX7LRxKOsGzfMpbuXcrivYtZsnfJP/XiZQqUoWnZptxS5haalW1G5SKVA3bIu4gQsz+G0etGM+qvURw6fYiiEUXpckMXet7Yk+rFqjsdYsBKciXx4Z8f8tb8t8ibKy+D7hxEl5pd0v+/snevTfrbt8OYMXaglsrxvJ7wjTGtgE+BYOBbEXn/kv0mdX9rIAHoLiIrr3XerCR8d4qbl25ewCcrmtGl7AK+X9+A0DyXN3Bei8vtYsfxHWyI3cDaQ2tZdXAVKw+sZOfxnQAEmSBqFKtB46jGNCnThMZRjSlbMHtOyZDiTuHXrb8ydM1QJm2aRLI7mYalG9K7bm/uq34fEaERTocYMObtnMcT059gQ+wG7q9+P5/d9RnF8mRslkzATkx/992wfDkMGQI9eng+WBVQvJrwjTHBwBagJbAXWA48KCIbLjimNfA0NuE3AD4VkQbXOndmE35yQjI9ay5l2LYmPFNzLp+saEpQyOVVKCLC6eTTHDtzjIPxB9l3ch/7Tu1j5/GdbIvbxrZj2/j72N+cTTn7z3MqFa5EnRJ1uPG6G2lQugH1StbLkaMdY0/HMmzNML5Z+Q2bj26mYFhButfqTp96fahS1HeN4YHmYPxB/jXrX4xYO4LogtF8ftfntKncJmsnPX0a7r3X1ue//76dYz9Av1GqrPN2wm8IDBCRO1Pv9wUQkfcuOGYwMFdERqfe3ww0F5EDVzt3ZhL+mWNnKP/8XRx0h1Gx8FHK1SuES1wku5JJdidzJvkM8UnxxCfFE3c2jiRX0mXnCAsJo3yh8lQoVIFKhStRvVh1qkVWo1pkNfLnzp+heLI7EWH+rvl8FfMVEzZOINmdTItyLXjipidoV6WddvFMdSb5DJ8s+YT3Fr5HkiuJlxu9TN9b+nruW1FSEnTvDqNHw4svwocf6spsOZS358MvBey54P5ebCn+WseUAi5L+MaY3kBvgDJlymQ4GHELCblOEl3kMEUq5udU0ilCgkIIDQolPDSc4nmKkydXHvKE5vlnNGqR8CIUz1ucUvlKUSp/KYpGFPXrRlV/YoyhWXQzmkU341D8Ib5b9R1fx3xNp3GdKJWvFL3r9qbXjb0oka+E06E6wi1uxqwbQ7/Z/dh1Yhftq7Tno5YfUalI+ua7T7dcuWDECLsyycCBdv3Bb77RpabUxUQkSxvQGVtvf+5+V+DzS46ZBjS54P5soO61zl23bl3JjJTElEw9T3lGiitFJm6cKHcOv1MYgIS8GSL3jrtXft/2u7jcLqfD8wm32y2TNk2SG/53gzAAqf11bZmzfY4vLizyxhsiINKmjUhCgvevqfwKECNXyKmeKOHvBaIuuF8a2J+JYzwmOFfwtQ9SXhMcFEz7qu1pX7U9W49t5euYr/lh9Q+M3zCeioUr0vvG3nSr3S1zjZR+zi1uJm+ezDsL3iFmfwyVCldiTKcxdK7e2TffGo2B/v3t+oNPPml78UyZAgULev/ayu95og4/BNto2wLYh220fUhE1l9wzN3AU5xvtP1MROpf69xOLmKuPOtsylnGbxjP4BWDWbh7ISFBIbSr0o4edXpwR4U7Ar6u/2zKWcauG8uHiz5kQ+wGKhSqQN8mfXmk1iPOjVkYNw66dIGqVe0I3RI5s1otp/FFt8zWwCBst8zvReQdY0wfABH5OrVb5hdAK2y3zEdF5JqZXBN+9rQxdiPfrfqOH9f8yJGEIxTPU5yHbniIrjW7Uvu62gE1ZmHvyb0MjhnM4BWDiU2I5YZiN9C3SV86V+/sHx9is2ZBhw62xP/bb+leK1cFrhw78Er5tyRXEtO2TGP42uFM3TKVZHcylYtU5r5q93Ff9fuoUayGXyb/sylnmbRpEj+s/oFZ22chIrSr0o5nGjzDrdG3+l/My5fDXXfZqRhmzIAbb3Q6IuVFmvCV3zuacJTxG8YzbsM45u6ci1vclCtYjnZV2tG2clsal2lMWIhzK5ElJCcwc+tMxm8Yz9QtUzmVdIoyBcrQrVY3utfuTvlC5R2LLV02b7b1+XFxdnrl225zOiLlJZrwVUA5FH+IiZsmMmXLFH7f/juJrkTCQsJoUqYJLcq1oHFUY+qWrOvVkb1JriTWHFzDnB1zmLV9Fgt2LyDJlUSR8CJ0qNqB+2vcz23lbgus7rv79sGdd8Lff8PIkXawlsp2NOGrgHU66TRzdsxh9o7ZzN4xm3WH1wF24ZdzC7lUj6xO9WLVKV+oPFH5ozK0spfL7eLw6cNsObqFjUc2sjF2IzEHYlixfwWJrkQAbih2Ay3Lt+TuynfTtGxT/6ibz6xjx6BtW1i8GL78Eh5/3OmIlIcFbMLPVy6f1P1P3Yseu6/6fTxx0xMkJCfQemTry57TvXZ3utfuzpGEI9w77vISzOP1Huf+Gvez58Qeuv7S9bL9LzZ8kbZV2rL5yGb+b+r/Xbb/taavcXv521l9cDXPzXzusv3vtniXRlGNWLRnEf1m97ts/6BWg6h9XW1+3/47b89/+7L9g9sMpkrRKkzZPIWBiwdetn94h+FEFYhi7LqxfBXz1WX7x983nqIRRRm6eihDVw+9bP/0h6cTERrB/5b/j3Hrx122f273uQB8vOhjpm6ZetG+8NBwZjw8A4C35r3F7B2zL9pfJKIIP9/3MwB9f+/L4r2LL9pfOn9pRnQcAcBzM59j9cHVF+2vXKQyQ9oOAaD3lN5sObrlov21r6vNq7e8ytJ9S+n7e1/2ndrH6aTTJLkvHi1dJLwIia5EDIaQoBCCTBAGQ7E8xShXqBzxSfGsOrjqn+UhL5QnNA+1r6vNruO7yJ87PwXCCvyzyEy2+d/bMJW3v+8GR49B2bIQHQ3o/961/vcGtRoEQJcJXdh7cu9F+xuWbsh7t9vJBTqN68TRhKMX7W9RrgWvN3sdgLtG3sWZ5DMX7W9TuQ0vNXoJgOZDm3OpjPzvReaJ9OpIW6V8JjJPJG0qt2HMujEUibDTXKe4UzidfJqo/FE0LN2Q3Sd2M2XLFE4nnybZnWwHnSDEJsQSHhpO3lx5yR2cm/CQcHKH5CZXcC7CQ8JpW6Utb936FkEmKM03XbYRFgbVa8CWLbBrl52WQXvv5Ah+XcLXKh2lvEgEXn8d3nnHVvOMGQMROuNpoLtalU4AtTgppTzKGHj7bfjiC5g6FVq0sHPwqGxLE75SOd2TT8L48bBqFTRpAjt2OB2R8hJN+Eop6NjRjso9dAgaNoQVK5yOSHmBJnyllHXLLfDnn5A7t10jd8YMpyNSHqYJXyl1XrVqto9+pUq2Iffbb52OSHmQJnyl1MVKloT58+H226FXL3j1VXC7nY5KeYAmfKXU5fLls/Po9+oF775rp1lOTHQ6KpVFOvBKKZW20FAYPBgqVIB//xv27IFffoGiRZ2OTGWSlvCVUldmDLzyCowda6dZvvlm2LTJ6ahUJmnCV0pd2333wdy5cPKk7bY5Z47TEalM0ISvlEqfm2+GZcugVCk7zfLgwU5HpDJIE75SKv2io2HRIruYSp8+8MwzkJLidFQqnTThK6UyJn9+mDwZXngBPv8cWre28+wrv5elhG+MKWyMmWWM+Tv1Z6ErHLfTGPOXMWa1MUanv1Qq0AUHw8CB8N13tm6/fn1Yv97pqNQ1ZLWE/29gtohUAman3r+SW0Wk9pWm7VRKBaDHHrMJPz7e1vFPnOh0ROoqsprw2wM/pt7+Ebgni+dTSgWaRo0gJgaqVoUOHeC118DlcjoqlYasJvziInIAIPVnsSscJ8BvxpgVxpjeVzuhMaa3MSbGGBMTGxubxfCUUj5RujQsWGBL/O+8A23aaL2+H7pmwjfG/G6MWZfG1j4D12ksIjcCdwFPGmOaXulAERkiIvVEpF5kZGQGLqGUclRYmJ1sbfBgmD0b6tbVaZb9zDUTvojcLiI10tgmAYeMMSUAUn8evsI59qf+PAz8AtT33EtQSvkNY6B3b1vad7lsdc/gwXY5ReW4rFbpTAa6pd7uBky69ABjTB5jTL5zt4E7gHVZvK5Syp81aAArV8Ktt9r++o88Yht2laOymvDfB1oaY/4GWqbexxhT0hgzPfWY4sBCY8waYBkwTURmZvG6Sil/V7QoTJ8Ob7wBo0bZKp7Vq52OKkcz4sdfterVqycxMdptX6mAN3cuPPSQbcgdOBCeeMJW/yiPM8asuFL3dx1pq5TyvubNYc0auO02eOopaNcODqfZ5Ke8SBO+Uso3IiNh6lT49FO7YHrNmrpuro9pwldK+U5QkJ1wbfly+wHQurVt1NUGXZ/QhK+U8r0bbrBJ/6WXYMgQW9qfP9/pqLI9TfhKKWeEhcFHH8G8ebYBt3lzePppOHXK6ciyLU34Siln3XKLbdB9+mn48kuoUQN+/dXpqLIlTfhKKeflzWsbcxcuhIgIaNXKduM8eNDpyLIVTfhKKf/RqJEdnPXGGzBhAlSpAl98oatqeYgmfKWUf8mdG/r3h7/+slM0PP20HaU7d67TkQU8TfhKKf9UqZKtyx8/Hk6csPPydO4M27Y5HVnA0oSvlPJfxkCnTrBxI7z5pp2b5/rr4dlnQdfLyDBN+Eop/xceDq+/Dlu3wqOP2nr9ChXgP/+B48edji5gaMJXSgWOEiXs/Prr1sEdd9hSf7ly8NZbmvjTQRO+UirwXH+9rdtftQqaNbONvGXKwCuvwIEDTkfntzThK6UCV+3aMHGiTfx33w0ffwzR0dCtm12ARV1EE75SKvDVrg2jR8OWLdCrF/z8s+3K2aQJDBsGCQlOR+gXNOErpbKPChVsg+6+ffDJJ3bO/W7dbN3/44/Dn3+C2+10lI7RhK+Uyn4KFIDnnoPNm+2ArfbtYehQW+IvV87W9S9dmuOSvyZ8pVT2ZYxt1B02DA4dguHD7eRs//0v3HwzlCwJPXvCuHH+068/MdFWTXlBlta0NcZ0BgYA1wP1RSTNBWiNMa2AT4Fg4FsReT8959c1bZVSXhEXBzNnwuTJdtWtEyfs4zVrQuPG9sOgQQM72jfIi+ViEdi/3zY6L1kCCxbYbx5FisDevZla9/dqa9pmNeFfD7iBwcBLaSV8Y0wwsAVoCewFlgMPisiGa51fE75SyutSUmDFCpg9G+bMgWXLzs/JHxEB1arZbwUVK9oeQNHRtk0gMtLO8nmtpJyUZL897N9vtx07bFXTli12PMG5tX2Dg+HGG+100bfcYtf9zcSHzdUSfkiGz3YBEdmYeoGrHVYf2Coi21OPHQO0B66Z8JVSyutCQmxpvkED6NcPXC7YtMmWtP/6C9avt3P6DB16+XNz5YJ8+exI4PBwm6BdLrslJNhvDmfPXv68ggXtTKCtW9skf+ONUKuW/QDx5kv16tmtUsCeC+7vBRpc6WBjTG+gN0CZMmW8G5lSSl0qOBiqV7fbhRISYPdu2LnTztMfG2u3+Hg4c8ZuIvb5wcH2A6BAAbsVLQqlStk2gzJl7P1MVNdk1TUTvjHmd+C6NHa9KiKT0nGNtF7VFeuRRGQIMARslU46zq+UUt4XEQFVq9otQF0z4YvI7Vm8xl4g6oL7pYH9WTynUkqpDPJFt8zlQCVjTDljTC7gAWCyD66rlFLqAllK+MaYDsaYvUBDYJox5tfUx0saY6YDiEgK8BTwK7ARGCci67MWtlJKqYzKai+dX4Bf0nh8P9D6gvvTgelZuZZSSqms0ZG2SimVQ2jCV0qpHEITvlJK5RCa8JVSKofI0lw63maMiQV2ZfLpRYEjHgwnEOhrzv5y2usFfc0ZVVZEItPa4dcJPyuMMTFXmkAou9LXnP3ltNcL+po9Sat0lFIqh9CEr5RSOUR2TvhDnA7AAfqas7+c9npBX7PHZNs6fKWUUhfLziV8pZRSF9CEr5RSOUS2S/jGmFbGmM3GmK3GmH87HY8vGGO+N8YcNsasczoWXzDGRBlj/jDGbDTGrDfGPOt0TN5mjAkzxiwzxqxJfc1vOB2Trxhjgo0xq4wxU52OxReMMTuNMX8ZY1YbYzy6qHe2qsPPyoLpgcwY0xSIB4aJSA2n4/E2Y0wJoISIrDTG5ANWAPdk57+zsQtH5xGReGNMKLAQeFZEljgcmtcZY14A6gH5RaSN0/F4mzFmJ1BPRDw+2Cy7lfD/WTBdRJKAcwumZ2siMh845nQcviIiB0RkZertU9h1Fko5G5V3iRWfejc0dcs+pbUrMMaUBu4GvnU6luwguyX8tBZMz9aJIKczxkQDdYClDofidalVG6uBw8AsEcn2rxkYBLwMuB2Ow5cE+M0Ys8IY09uTJ85uCT9DC6arwGaMyQv8DDwnIiedjsfbRMQlIrWx60LXN8Zk6+o7Y0wb4LCIrHA6Fh9rLCI3AncBT6ZW2XpEdkv4umB6DpFaj/0zMFJEJjgdjy+JyHFgLtDK2Ui8rjHQLrVOewxwmzFmhLMheV/qioGIyGHsioL1PXXu7JbwdcH0HCC1AfM7YKOI/NfpeHzBGBNpjCmYejscuB3Y5GhQXiYifUWktIhEY9/Lc0Ski8NheZUxJk9qRwSMMXmAOwCP9b7LVgk/py6YbowZDSwGqhhj9hpjejgdk5c1BrpiS3yrU7fW13pSgCsB/GGMWYst2MwSkRzRTTGHKQ4sNMasAZYB00RkpqdOnq26ZSqllLqybFXCV0opdWWa8JVSKofQhK+UUjmEJnyllMohNOErpVQOoQlfKaVyCE34SimVQ/w/l3gd2N/VcY4AAAAASUVORK5CYII=\n", 159 | "text/plain": [ 160 | "
" 161 | ] 162 | }, 163 | "metadata": { 164 | "needs_background": "light" 165 | }, 166 | "output_type": "display_data" 167 | } 168 | ], 169 | "source": [ 170 | "# Do some vector computations.\n", 171 | "y1, y2 = x.sin(), x ** x.cos()\n", 172 | "y3 = y2 - y1\n", 173 | "y4 = y3.min()\n", 174 | "\n", 175 | "# Print and plot some answers.\n", 176 | "print(f'The shape of x is {x.shape}')\n", 177 | "print(f'The shape of y1=x.sin() is {y1.shape}')\n", 178 | "print(f'The shape of y2=x ** x.cos() is {y2.shape}')\n", 179 | "print(f'The shape of y3=y2 - y1 is {y3.shape}')\n", 180 | "print(f'The shape of y4=y3.min() is {y4.shape}, a zero-d scalar')\n", 181 | "\n", 182 | "plt.plot(x, y1, 'red', x, y2, 'blue', x, y3, 'green')\n", 183 | "plt.axhline(y4, color='green', linestyle='--')\n", 184 | "plt.show()" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "### Exercise\n", 192 | "\n", 193 | "Plot y3 clamped between 0.0 and 1.0." 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": 62, 199 | "metadata": {}, 200 | "outputs": [ 201 | { 202 | "data": { 203 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAD4CAYAAAAKA1qZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAJYUlEQVR4nO3cW4jmd33H8c+32YYSWrFmt9YmadfWC6lQYhxCL1IIVUTSoC30UtrihQqKPVDbYBCk0IvEirbggSDYSBVvekQsbQ2FFoqR2ZxEo00itdUmTdILtbRFbL+9mP+WcZ3ZnXnm8Mzk+3rBn3nmf3jy+/KHvHf+z+xWdweAmb5n3QsAYH1EAGAwEQAYTAQABhMBgMHOrHsB+3H27Nk+f/78upcBcKpcuHDh2e4+t9OxUxWB8+fPZ3Nzc93LADhVquorux3zOAhgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDBLhuBqrq2qh5atqeq6mvbvv/Rqvrzqnqsqp6oqt+vqquX626tqq9X1YNV9aWq+ruquv2S935jVX1x2T5bVbcc5aAAfLczlzvY3f+e5MYkqap3JfmP7v69qqok9yf5YHe/rqquSnJPkt9N8vbl8r/v7tuXa29M8mdV9V/dfd8ShDcluaW7n62qm5bjN3f3U4c+JQA7WvVx0M8m+e/u/kiSdPf/JPn1JG+oqmsuPbm7H0ryO0neuuz67SRv7+5nl+MPJLk3yVtWXA8AK1g1Ai9LcmH7ju7+RpJ/TvKSXa55IMlLd7s+yeay/zssj402q2rzmWeeWXG5AOxk1QhUkt7H/ovH9v2e3X1Pd29098a5c+f2t0oALmvVCHw+ycb2HVX1vCQ3JHlil2tenuTR5fUXkrzikuM3LfsBOCarRuC+JNdU1S8lyfLB8HuS/GF3/+elJ1fVTyV5Z5L3L7vuTnJXVV27HL8xya8k+cCK6wFgBZf97aDddHdX1S8k+UBVvTNbMflUkndsO+1nqurBJNckeTrJ27r7vuX6v6iq65L8Q1V1km8meX13P3mAWQDYp+re7RH+ybOxsdGbm5vrXgbAqVJVF7p7Y6dj/sYwwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMJgIAg4kAwGAiADCYCAAMVt297jXsWVU9k+Qr617HCs4meXbdizhmZp7BzKfDj3X3uZ0OnKoInFZVtdndG+tex3Ey8wxmPv08DgIYTAQABhOB43HPuhewBmaewcynnM8EAAbzkwDAYCIAMJgIHJKqekFV/U1VPbZ8/cFdzntNVX2pqh6vqjt2OP6bVdVVdfboV30wB525qt5dVV+sqkeq6k+r6vnHtvh92MM9q6r6g+X4I1V1016vPalWnbmqbqiqv62qR6vq81X1q8e/+tUc5D4vx6+qqger6pPHt+pD0N22Q9iS3J3kjuX1HUnu2uGcq5I8keTHk1yd5OEkP7nt+A1J/ipbfyHu7LpnOuqZk7w6yZnl9V07Xb/u7Ur3bDnntiR/maSS/HSS+/d67UncDjjzi5LctLz+gST/+Fyfedvx30jy8SSfXPc8+9n8JHB4Xpfk3uX1vUl+fodzbk7yeHd/ubu/leQTy3UXvTfJbyU5LZ/WH2jm7v7r7v72ct5nklx/tMtdyZXuWZbvP9pbPpPk+VX1oj1eexKtPHN3P9ndDyRJd38zyaNJrjvOxa/oIPc5VXV9kp9L8uHjXPRhEIHD88LufjJJlq8/tMM51yX5l23ff3XZl6p6bZKvdffDR73QQ3SgmS/xhmz9Keuk2cv6dztnr7OfNAeZ+f9V1fkkL09y/+Ev8dAddOb3ZesPcP97ROs7MmfWvYDTpKo+neSHdzh0517fYod9XVXXLO/x6lXXdlSOauZL/ht3Jvl2ko/tb3XH4orrv8w5e7n2JDrIzFsHq74/yR8n+bXu/sYhru2orDxzVd2e5OnuvlBVtx72wo6aCOxDd79qt2NV9W8XfxxefkR8eofTvpqt5/4XXZ/kX5P8RJIXJ3m4qi7uf6Cqbu7upw5tgBUc4cwX3+OXk9ye5JW9PFg9YS67/iucc/Uerj2JDjJzqup7sxWAj3X3nxzhOg/TQWb+xSSvrarbknxfkudV1R919+uPcL2HZ90fSjxXtiTvznd+SHr3DuecSfLlbP0P/+KHTy/b4bx/yun4YPhAMyd5TZIvJDm37lkuM+MV71m2ngVv/8Dws/u53ydtO+DMleSjSd637jmOa+ZLzrk1p+yD4bUv4LmyJbk2yX1JHlu+vmDZ/yNJPrXtvNuy9RsTTyS5c5f3Oi0RONDMSR7P1jPWh5btQ+ueaZc5v2v9Sd6c5M3L60ry/uX455Js7Od+n8Rt1ZmT3JKtxyiPbLuvt617nqO+z9ve49RFwD8bATCY3w4CGEwEAAYTAYDBRABgMBEAGEwEAAYTAYDB/g9I3ziJUe7pHQAAAABJRU5ErkJggg==\n", 204 | "text/plain": [ 205 | "
" 206 | ] 207 | }, 208 | "metadata": { 209 | "needs_background": "light" 210 | }, 211 | "output_type": "display_data" 212 | } 213 | ], 214 | "source": [ 215 | "# TODO: Plot y3 clamped between 0.0 and 1.0.\n", 216 | "\n", 217 | "plt.plot('TODO')\n", 218 | "plt.show()\n" 219 | ] 220 | }, 221 | { 222 | "cell_type": "markdown", 223 | "metadata": {}, 224 | "source": [ 225 | "Subscripts and multiple dimensions\n", 226 | "----------------------------------\n", 227 | "\n", 228 | "Pytorch code is full of multidimensional arrays. The key to reading this kind of code is stopping to think about the careful, sometimes tangled, use of multiple array subscripts.\n", 229 | "\n", 230 | "**Slicing.** As normal in python, you can use `[min:max:stride]` to slice ranges, and multidimensional subscripts like `x[2,0,1,9]` work as you would expect (selecting the 9th entry of the of the 1st of the 0th of the 2nd entry of `x`; and can be used with slices like `x[0:3,2,:,:]`. The special slice `:` selects the whole range in that dimension.\n", 231 | "\n", 232 | "**Unsqueezing to add a dimension, and broadcasting.** While a single integer subscript like `x[0]` eliminates a dimension, the special subscript `x[None]` does the reverse and adds an extra dimension of size one.\n", 233 | "\n", 234 | "An extra dimension of size one is more useful than you might imagine, because pytorch (similar to numpy) can combine different-shaped arrays as long as the shape differences appear only on dimensions of size one by **broadcasting** the singleton dimensions. An example that uses broadcasting to calculate an outer product is illustrated below.\n", 235 | "\n", 236 | "**Fancy indexing.** Lots more can be done by passing numerical arrays or boolean array masks as subscripts. The reshuffling possibilities can get quite intricate; the rules are modeled on the capabilties in numpy. For details see [Numpy fancy indexing](https://numpy.org/doc/stable/user/basics.indexing.html).\n", 237 | "\n", 238 | "Here is a demonstration of simple tensor reshaping." 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 12, 244 | "metadata": {}, 245 | "outputs": [ 246 | { 247 | "name": "stdout", 248 | "output_type": "stream", 249 | "text": [ 250 | "m is tensor([[0.1353, 1.2838, 0.2440, 0.5774, 1.3416],\n", 251 | " [0.9628, 0.1760, 0.4458, 0.9256, 1.6327]]), and m[1,2] is 0.445751816034317\n", 252 | "\n", 253 | "column zero, m[:,0] is tensor([0.1353, 0.9628])\n", 254 | "row zero m[0,:] is tensor([0.1353, 1.2838, 0.2440, 0.5774, 1.3416])\n", 255 | "\n", 256 | "The dot product of rows (m[0,:] * m[1,:]).sum() is 3.1897406578063965\n", 257 | "\n", 258 | "The outer product of rows m[0,:][None,:] * m[1,:][:,None] is:\n", 259 | "tensor([[0.1302, 1.2361, 0.2349, 0.5560, 1.2917],\n", 260 | " [0.0238, 0.2259, 0.0429, 0.1016, 0.2361],\n", 261 | " [0.0603, 0.5723, 0.1088, 0.2574, 0.5980],\n", 262 | " [0.1252, 1.1883, 0.2258, 0.5345, 1.2417],\n", 263 | " [0.2208, 2.0961, 0.3983, 0.9428, 2.1904]])\n" 264 | ] 265 | }, 266 | { 267 | "data": { 268 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgEAAAG6CAYAAACRA5VKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAwKklEQVR4nO3de5wsdX3n/9fbQwDFc44RvCGQmLjLrohBgY2yEdAEg8YYwSQaNQZzwYWQaEy8EF1E4yUkPo74U7xFVlhiook/xXAzCEq8kQsoEoTgkkUiEiGKnjkqHBA/+0dVc/r0mUvPnJ6p6anX8/Hox0xXV1d9uqen+l3f77eqUlVIkqT+uU/XBUiSpG4YAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAnoqyXFJaui214SXf+7Qsq+Z5LIlSZNhCNCxwBOAb094uS9vl/uFCS9XkjQhu3RdgDr3har6yqQXWlVfBkgyA0y0lUGSNBm2BEyxJKe2ze2PSfLXSTYnuT3JpiS7JNk/yceSbEnylSQv77pmSdLqYQhYG/4K+CLwLODPgN8D3gKcC1wAHAN8AjgtybELLSzJkW24OHWMeX+0nfesJVcvSeqE3QFrw3uqalP7+yVJngKcBBxbVR8BSHIZ8HTgecCHF1heAfcAPxhj3YN571lC3ZKkDhkC1obzR+5fB/wEcNFgQlV9P8kNwI8stLCq+jvG/GxU1U3jzitJWl3sDlgbbh+5fxfwvaq6c5bpu69MSZKk1c4QIElSTxkCJEnqKUOAdpDkiCTfT3LKyPTBIYlHDk37kXbeM1e6TknSzjEEaDYB1rHj5+P+NEcDfH2WedetTGmSpElJVXVdgzqQ5DjgfcAjgZuq6vtjPOcf23l/aYx570MTIi4F9qyqR+9cxZKkSbMlQDcAdy90AaEkG2gOOzxlvvmGfBi4Gzh858qTJC0XWwJ6KsmewCOGJl01TmvAIpb/48APt3fvqKovTWrZkqTJMARIktRTdgdIktRThgBJknrKECBJUk+t+IVfkgTYG9iy0uuW1oj1NP8/W8pBPZJ2QhdXf9sbuLmD9UprzUZgpusiJE2vLkLAFmguZZcOVr4YX9/8vK5LGMvTNr6/6xIW9O6uCxjTvptf33UJ85qZuZN99309wD7YmtaJoRNtDTyoqr4x9PiPAW8Gnkyzjb0ceEVVfX4n1nkVzXk6AC6oqqe3008FXtNO/25V3X/oOZcBRwB/W1VHjyzvR4EbgZdV1ZuXWlcXkhwEfGFo0i9V1Yc6KmfqdXYd+LD6Q8CGDbt2XcJYOvsjLsL6rgsY04YNU3OlZbsCuncs8O/AtwcTkjwI+DTwLeDXgTuBk4HLkhxaVdcvcV2/CuwBfGSOx58A3DPHYz+b5MlV9Yklrnu1+TLN630ccEbHtUy9afj+kKTV6AtV9ZWRaS8DHgQcVlU3AST5DPCvwOuAZy9lRVX1z+2yts7x+N/P8dQv02zn/6QNIVMfHKvqe8DfJ5maxL6aeXSApN4ZuiLmY5L8dZLNSW5PsinJLkn2T/KxJFuSfCXJy8dc9DHAJwYBAKCqZmhOo/3zSVZ6x+tu4FXAwYwRQJI8OslHk3wryZ1JrkryayPzHNm+d7+S5A1Jbkkyk+SSJPvPssyfSXJpO8/3knw2yU9P7BVqpxgCJPXZXwFfBJ4F/Bnwe8BbgHOBC2i/1IHTkhw734KS3Bf4ceDqWR6+Grgv8GMLLGPwBXvqol7F/D4IXAm8PskPzbPu/YHPAQcAv0vT3XEtcNYcIeiNwI8AvwkcD/wn4Lwk915RNMnzgYtpBrD+GvDLwO3A344TBJKc1b4fPzrG69QS2B0gqc/eU1Wb2t8vSfIU4CTg2Kr6CNw7wO7pwPNo9ujn8sM0Q51un+WxwbQ9F6inaPr2fzBW9WOoqkryCuAS4EXA2+eY9VRgV+BJVfXVdtqFSR4AvCbJu6tq89D811bV8wd3ktxDE6oOpWmuvx/wVuD8qjpmaL4Lgc/ThIifXKD8e9rb1HdjrFa2BEjqs/NH7l9H84Vz0WBCe2GtG2j2escx3xfWvF9mVfV3VbVLVb1uzHWNV1DVpTR75KckmWuc7pOBS4cCwMBZwP1oBuMN+5uR+4MWkMH7dBjwQODstotll7Y75D7Ax4BDk+yxQN2/0b4fN803n5bOECCpz0b32u8CvldVd84yfaGBaN+i+ZKfbW//gXOsbyW9AtgL+IM5Ht+T5miHUbcMPT7smyP3B4MW79v+fEj780M0YxOGb6+gaTV5IOqU3QGSNAFVdUeSG4ADZ3n4QOAO4P+ubFXbVNVVSf4SeClw4SyzfBN42CzT925/fmOWx+YzmP93gLmOXrh1kcvUhBkCJGlyPgK8JMm+g2b1tvn9WOBv2q6FLr0a+EW2nWBo2KXAMUn2rqpbhqa/APgec3+Rz+WzNOdQeFRVzTUOQR2zO0CSJufNNHvUFyR5ZpKn0ow72J1m4N29Zhv5nuSIJN9PcspyFFdVNwLvBJ46y8OvpWmq/2SS5yV5apI/B34OOHVkUOA46/oOTSvAi5J8IMkvJjk8ybOSvC7JO4fnb9+Ly0amndm+H+OOx9AiGQIkaUKq6j+AJ9KcHOhstvWHH1lV/zIy+/1pugi+PTQtwDqWd9v8ema55kR7NsPDgOtpzsR3LvBo4IVV9adLWVFV/TnwJJrX+m6aIxTeSnO2v0sH8yUZnO54dEzCuva22k8wO7XsDpDUO1V1KiN75u3044DjZpl+5CyLWZdkl9Em/qr6V5rzCyzkp4AzqurbQ8+9jFm+8Npj7+f8ImxH3VdV3Xvq4Dlqpr3OwcY5HrsGeMZ8Rc9VY3v2xNmmfwr41HzLBA6nGVT5xpHnHscsf4/29a4bna7FsyVAkpbmBuDuJHst9olJDqA57O60MZ9yJU2LwlzN4ncDi2quX2WeBHxgcHrk+bQXELqbplVBO8mWAElanPNoTogz8O3FLqCqvgRsWMRTnksTGkbX9x62netgrgsIrXpV9bJFzH4927///zrhcnrFECBJi1BV32THY+SXe53XzjH9FrYdx98LVXUHcEXXdawVS+oOSHJikhvbC0xcmeSJky5MkiQtr0WHgCTPBk4H3gA8luba2Rcl2W+ypUmSpOW0lJaAlwJnVtV7q+q6qnoJ8FXghIlWJkmSltWixgQk2ZXmutR/PPLQxTTHl872nN2A3YYmzXXxCklTIkloTie7petapCm2Hrilqjq7SuJiBwbuRXNs5uj5nm8FHjrHc05m9lNUSppeewM3d12EtAbsA3ytq5Uv9eiA0dSSWaYNvAnYNHR/PW48pGm3BZpz4a72U7l9ffPzui5hLE/b+P6uS1jQu7suYEz7bn591yUsaGbmTvbd9/XQcWvaYkPAN2iORR3d638wc1wNqqq2su0SkzStiJLWgrD6Q8CGDbt2XcJYpuF47Wnpy92wYaGrPmtgUQMDq+oumjNXHTXy0FHA5yZVlCRJWn5LCZ+bgHOSXAFcDhwP7Ae8a5KFSZKk5bXoEFBVH0yyJ3AK8DDgGuBpVXXTpIuTJEnLZ0ndUFX1DuAdE65FkiStIK8iKElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOA1FNJTkxyY5I7k1yZ5Ild1yRpZRkCpB5K8mzgdOANwGOBTwMXJdmvy7okrSxDgNRPLwXOrKr3VtV1VfUS4KvACbPNnGS3JBsGN2D9CtYqaZks6SqCk/D1d8OG+3a19vHsm/d1XcJY3th1AWPY7/SuKxjX73ddwAJmgFfv1BKS7AocDPzxyEMXA4fN8bSTgdfs1IolrTq2BEj9sxewDrh1ZPqtwEPneM6bgI1Dt32WrTpJK6azlgBJnauR+5llWjNj1VZg670zJstYlqSVYkuA1D/fAO5hx73+B7Nj64CkNcwQIPVMVd0FXAkcNfLQUcDnVr4iSV2xO0Dqp03AOUmuAC4Hjgf2A97VaVWSVpQhQOqhqvpgkj2BU4CHAdcAT6uqm7qtTNJKMgRIPVVV7wDe0XUdkrrjmABJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8tOgQkOTzJeUluSVJJnrkMdUmSpGW2lJaAPYAvAidNuBZJkrSCFn0Boaq6CLgIIMmC8yfZDdhtaNL6xa5T0upUXRcwhpmZu7ouYSzf77qAMWzpuoAxzczc2XUJC1otNa7EVQRPBl6zAuuRtHLWA6yOzdj8Nm58f9clrBmP7rqAcW18ddcVLMZ6YKarla9ECHgTsGno/nrg5hVYr6TlcwuwD5PdORxsGya93EmzzsmZhhph+epcT/O/1JllDwFVtRXYOrg/TheCpNWtqgr42iSXObRt2FJVne0ZLcQ6J2caaoRlrbPz1+whgpIk9ZQhQJKknlp0d0CS+wOPHJr0iCQHAbdX1b9NqjBJvbMVeC1D3YerlHVOzjTUCNNT56ItZUzAIcAnh+4PBv2dDRy3swVJ6qd2/NCpXdexEOucnGmoEaanzqVYynkCLgMc3SdJ0pRzTIAkST1lCJAkqacMAZIk9ZQhQJKknjIESOpckhOT3JjkziRXJnli1zWNmobLqCc5Ock/JdmS5LYk5ybZv+u6RiU5IcnVSWba2+VJntp1XfNp39tKcnrXtUySIUBSp5I8GzgdeAPwWODTwEVJ9uuyrllMw2XUjwDOAB4PHEVzBNjFSfbotKod3Qy8kuaQ80OATwAfTXJAp1XNIcmhwPHA1V3XMmkrcQEhSZrPS4Ezq+q97f2XJPlZ4ASaq5CuCou9jHoXquro4ftJXgjcBhwMfKqTomZRVeeNTHpVkhNowsuXOihpTu0J8t4P/BYwVZcnHIctAZI6k2RXmi+oi0ceuhg4bOUrWnM2tj9v77SKeSRZl+Q5NC0tl3ddzyzOAC6oqku6LmQ52BIgqUt7AeuAW0em3wo8dOXLWTvSNFdsAj5TVdd0Xc+oJAfSfOnvDnwHOKaqru22qu214eRxwKFd17JcDAGSVoMauZ9Zpmlx3g48BviprguZw/XAQcADgGcBZyc5YrUEgST7Am8FnlJVd3Zdz3IxBEjq0jeAe9hxr//B7Ng6oDEleRvwDODwqrq563pmU1V3ATe0d69oB9+9GHhRd1Vt52Caz+GVQ2NA1gGHJzkJ2K2q7umquElxTICkzrRfBFfSjGQfdhTwuZWvaLql8XbgWODJVXVj1zUtQoDdui5iyKXAgTStFYPbFTSDBA9aCwEAOmwJmLmjqzWP7wddFzCmKXgrmZmWxrSZma4rmNfMtvrWJ9lSVWuhyXwTcE6SK2j6iI8H9gPe1WlVI6bkMupnAM8FfgHYkmTQwrK5qlbNpiLJG2mOtPgqsB54DnAkcPQ8T1tRVbUF2G4sRZLvAt9cjWMsliorvQ1J8nCaY0Ql7ZyNVbW6U8uYkpwIvBx4GM2G9/eqatUc0gaQ5Ei2v4z6wNlVddyKFjOHJHNt0F9YVWetZC3zSXIm8NM0f+/NNMffn1ZVH++0sAUkuQy4qqpe0nEpE9NFCAiwN7BlQotcTxMq9pngMpeDdU5Wn+tc3y5rrbQESOrIincHtButr01qeUMDNras5r0i65ysnte5al+vpOniwEBJknrKECBJUk+thRCwFXht+3M1s87Jsk5J2kkrPjBQkiStDmuhJUCSJC2BIUCSpJ4yBEiS1FOGAEmSesoQIElST019CEhyYpIbk9yZ5MokT+y6pmFJDk9yXpJbklSSZ3Zd02ySnJzkn5JsSXJbknOT7N91XcOSnJDk6iQz7e3yJE/tuq6FtO9tJTm961okadhUh4AkzwZOB94APBb4NHBRkv26rGvEHsAXgZO6LmQBR9BcgezxNJdx3QW4OMkenVa1vZuBVwKHtLdPAB9NckCnVc2jvUb68TQXSJGkVWWqzxOQ5B+Az1fVCUPTrgPOraqTu6tsdu0Vvo6pqnO7rmUhSR4E3AYcsdqu5jYsye3Ay6rqzK5rGdVeevbzwInAq1ljVx+TNP2mtiUgya7AwcDFIw9dDBy28hWtORvbn7d3WsUckqxL8hyalpbLu65nDmcAF1TVJV0XIkmzWfGrCE7QXsA64NaR6bcCD135ctaO9nLPm4DPVNU1XdczLMmBNF/6uwPfoWlZubbbqnbUBpTHAYd2XYskzWWaQ8DAaH9GZpmmxXk78Bjgp7ouZBbXAwcBDwCeBZyd5IjVFASS7Au8FXhKVd3ZdT2SNJdpDgHfAO5hx73+B7Nj64DGlORtwDOAw6vq5q7rGVVVdwE3tHevaAfevRh4UXdV7eBgms/hlU2jCtC0Wh2e5CRgt6q6p6viJGlgascEtF8GV9KMZB92FPC5la9ouqXxduBY4MlVdWPXNY0pwG5dFzHiUuBAmhaLwe0K4P3AQQYASavFNLcEQNNvfU6SK2j6iY8H9gPe1WlVQ9oR4o8cmvSIJAcBt1fVv3VT1azOAJ4L/AKwJcmghWVzVd3RXVnbJHkjcBHwVWA98BzgSODoDsvaQVVtAbYbS5Hku8A3V9sYC0n9NtUhoKo+mGRP4BTgYTQb3qdV1U3dVradQ4BPDt3f1P48GzhuxauZ2+Awy8tGpr8QOGtFK5nbQ4BzaP7Wm2mOvT+6qj7eaVWSNKWm+jwBkiRp6aZ2TIAkSdo5hgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCFAkqSeMgRIktRThgBJknrKELBCkhyXpIZuew09dkCSdyS5PMl328ePnMA6rxpa3/lD008dmv6dnVj+We0yvpRk3SyPV5K3L3X5k5TkoJH3/xe7rkmSumYIWHnHAk8Avj007RDgmcDtwKUTXNevtuv6+hyPPwF40gTW8yjguAksZzl9meb1/nbXhUjSarFL1wX00Beq6isj086pqrMB2j3Un5/Eiqrqn9tlbp3j8b+fwGq+C3weeG2Sv6iqOyawzImrqu8Bf59k965rkaTVwpaARRhqRn9Mkr9OsjnJ7Uk2Jdklyf5JPpZkS5KvJHn5OMutqh8sd+3L7BXAw4EXLzRjkv2S/HmS25JsTXJdkt9Pcp+heX60fZ//IMlLk9yY5Dttd8njZ1nmIUn+pv1b3JnkC0l+ebIvUZLWHkPA0vwV8EXgWcCfAb8HvAU4F7gAOAb4BHBakmOXu5gkR7ZfmqdOYFmDsQvHjfucqroc+AjwiiQPnGfZDwI+BzwF+J/AM4BLgDcDs40d+G3gKOAlwPOAPYALk2wcWuaTgM8CDwD+B/ALwFXABxfzGiSpjwwBS/Oeqnp9VV1SVa+g+dI5CfjDqnpbVV0CHA/8B82X13Ir4B5gEi0KP1jisk4G1gN/OM88L6VpMXh6Vb27qv62qn4HeCfwP5L855H5t7TzfrSqPgr8BvDDwFOH5nkH8CXgyVX1V+0yXwicD7xxuIVBizffgNb28R9L8uEk325baz6e5HE7uc5FD2hNclk7/WOzLO/elqWdqasLSx3Q2u4YnLXM5Q2v77LZ/mbtYy9I8oEk1yf5QZKvTGB9c74vbSvsYPqSByYPLeOVszw2+L84ZKnLn6Qkpy91sLcbyKU5f+T+dTRfxBcNJlTV94EbgB9Z7mKq6u+qapeqet0ElvW/22X970U+73rgTOCkJPvNMduTgWur6h9Hpp8FpH182AVVdc/Q/avbnz8CkOSRwH8B3t/e32VwAy4EHgbsv5jXoTntMKC1bdn5NPCfgV8HfhnYHbgsyc687zszoPVnk4x+jqbZ2ANakzwyye8m+aGR6U9I8mvLVeCQL9DU+vsj038VOAD4R+BfJ7Suhd6XC9vH3zyBdb1yvhbOVeItNK/3wsU+0YGBS3P7yP27gO9V1Z2zTN+wMiWtCqcCzwf+CJhto7Mn8JVZpt8y9Piwbw7fqaqtSQDu2056SPvzzcz9z77XHNO1OLMNaH0Z8CDgsKq6CSDJZ2g29K8Dnr2UFe3EgNYv02zT/iTJoVVVS1n/arLIAa23Aw8FLqc5ymjvJB8A7kfzv7ncZub42/zsYNxT20rw6J1d0Rjvy39MaODzJcCRwKvYMdysGu3/301J/mOxz7UlQBNTVf8OnA48P8ljZpnlmzR756P2bn9+Y5GrHMz/JuDQOW5XLXKZa0qWaTBr6xjgE4MAAFBVM8CHgZ9vW2RW0t00G+uDGSOAJHl0ko8m+VY7oPSq0T3mofE2v5LkDUluSTKT5JLZWjuS/EySS9t5vpfks0l+emKvcB5VdXtV/SHwHOC5wM8An6yqZ1TV5+d7bvu3Pz/J09MMrL0jzaDdp7ePH9fe/26Sf1xMM/iUD3wetHD+dpIFW3WTPCPNAObvtf9TH0/yhJF5Bv+TByT5y/Z/8tYk/ytD453aeZPkxPazeUf7Wf1Qkh+b1As0BGjSTqPZI/njWR67FHhUduwzfgFNd8onF7Oitgvi/wA/UVVXzHHbsoTXsBZNdDBrkvsCP862LpphV9O01sy7ocoEB7QO+SBwJfD60WbxkXXvTzNI9QDgd2m6O64FzpojCL2RphvqN2nG+/wn4LwMnSQryfOBi4EZmpawX6b5X/jbcYJAtp1860fHeJ2zPf8BSV4H/CXwFzR7sU9ug8444zR+giZQn0bzfmwGPpzktTSv+w9pxjhtBM5vPwPLbmffl5FlXZZksS1Ep9KMk/qjBZb9XOCjNH//X2HbGKbLkvzULE/5/2lar55Fs718Ls3/5LB30+xYXUJzLpkTaT6zn0vyECbA7oBVIMn9gKe1dweHwB2RZhDWd6vqoqF5z6LZwDxi0Dyb5AiaL9jX7ey4gCQvAP4X8OvD4wJmW+9sqmomyRvY8cNMO+0FwAVJTgFuAn6O5oP9zqr68hJKfhFwUZK/pRlb8DXggcB/BR5XVb+0hGWuRe+pqk3t75ckeQrNYNZjq+oj0GwggafTbOg/vMDyfphmHMdo1xhD00a7d0ZNckBrs8CqSvIKmo3mi5j9qBNoNuy7Ak+qqq+20y5M8gDgNUneXVWbh+a/tqqeP7iT5B6aYHUoTbP0/YC3AudX1TFD811Icx6NNwI/uUD597S3pXZjPIimdeww4L8DD6mq49o90QPbOuazJ/D4qvpaW/stNC1pvwU8sm2Cp/0SPZempeG8Jda6GDv7vsy2rLFV1deTvAU4Ocmbq2qH4JtmAPKfAv8MPHWo++NCmu6x02j+JsPOrKo/bX+/JM0Yp19P8hvt5/jxNO/97w/975Lk0zTh4aU0h2fvFFsCVocHA3/d3gb9Tqe29985Mu/9gTvY/oyDAdYxmb/nfeZY1mzrncs7gBtHJ1bVf9BsoD5Bs8dxPvCzwMuB31lKsVX1SeC/tXWdTrPxfyfNBuqSpSxzjVquwazzbZjn3WhPckDryHIvpdkjPyXJ+jlmezJw6VAAGDiLpg/9CSPT/2bk/naDVGk+1w8Ezs72A1TvA3wMODTJHgvU/Rvt+3HTfPPN8/z/U1X/X1XdPTL98sHJyBZw1SAAtK5rf142CAAj05d90DPs/Psysqyfrqql7Pz+CU24PW2Ox/en6dY8Z7j7o6q+Q7PH//g2KA6b7TO1O833ATSBvIA/H/lMfZ2mVe/IJbyOHdgSsAhVdSqzDLCpquOY5bS5VXXkLItZl2SXdoM7mO8rNF/k4/gp4Iyq+vbQ8y+b7fltU+Wcy20/UDU8Ar+qzqLZEI6z3uOY/XXfxRxNwVX1byxw2OR870dV7TC9Tebj9AHvQhNw+mjSg1m/RbOBmm1vfzCSerZWgpXyCpo93z8A3jfL43sC/z7L9LEGqQKDQYujg1Q/NE9ND6Q5w+aya7cJly3yadv9varqrjQDcWf77EDzhdULbQvn64HT05ybZNTg8zLXZ+o+NK1nw2FqnM9UgFvnKOv/LlT3OAwBK+8GaA6vqqpFDYRLcgDNXspcaXTUlTT9fADXzPL43TQbpftPeL2rTpKDaA5h0gRU1R1JbqBpZh51IE2r0UQ2UktRVVcl+UuaJtPZDptarkGqvwPMNSp9ro25psM7ac6Keho7ttAOvtDn+kz9gCY4L8Y3aIL2E9kWEIbNevTMYhkCVs55NP2HA99e7AKq6kss7pDD59J8eY+u7z1sax5esH9sCetdja5n+/d/Uscr99lHgJck2XfQrN42vx8L/M1wa1dHXg38IvCaWR67FDgmyd5VdcvQ9BfQ7K0t9vCyz9L8jz2qqlbFlTM1WW3LyKtpzksyGhKvpxmP9Nx23EABtF1AzwIuH+lSGcf5wCuBh1fVX+1c9XMzBKyQqvomOzb/LPc6r51j+i1sa/bshWoubHRF13WsMW+mORHMYKDnVpqN1u6MdJst94DW2VTVjUkGe2+jXkvT5/rJdkT97TTdVD8HvHxkUOA46/pOkt+hGRPwQJpugdtoBuv9BPCgqjphMH87uO7vhrsMk5xJ8x79+CT6v1eLJI+iudIoNOcxuF+2neHv2uHt1HK/L0kuBY4YHRcw23rn8Jc0XUzDZy2lqn7QHlXyfpojJ94N7EZzLo0H0PxfLEpVfTbJe4D3pTkk81M0LbcPo+me/eeqGm2RWDQHBkpaknag5xNpWlXOpvniuxs4sqr+ZWT25R7QOpfX0xyytZ328NLDaPbgzqAZ7f5o4IVDI7YXpar+nOZMhvenObTrEpojBh7H0CXCkwy630b7j9e1t3HHB02LX2bbwOeDaYLR4P69F/paofdlsKx7zbPeHbR7+LOOyK+qv6A5jG9PmkNV30fz2XtSVX1mKcVW1YtojuI5HPgAzeG8r6O5jsromVeXJDX9J9WStEzSXITpfcAjgZuW2sSf5Os0I6dfNsa8gw3+DcA1VTU4Yc2pNE37P8TIgNZpkuRpNE29P1Ht2REXmH8X4AiaUPFLVTXf4MNOtIeXBvhp4Ae1hBMETep9SXNtgr+jOU5/wVoWu97VqD1E8T40JzZ6VlXNO85rmN0BUyrNsN29aS6yMy3WA7eUyXMaTd2A1lXsScAHxvyiO4jpGdB6OM3f5gKarpbFmuT78oL2dgbNnvRE1ruKbWJbt9eijkCxJWBKJXk4cHPXdSzBPiPHImsVS7In8IihSVct94C/tg/53gGtVTUIIHuzbfT+PVU1LV+OS9aele+AoUn/WlWLHWW+7NozMA7OyXDv32wZ1zfn+5LkQJr+eIDb2sOS17Qk+7LtMNVF/W8YAqZUkg3A5q+eBhum4GjdmTth36YnbWM155eXJHXM7oApt2F32LAiZ/CWJK01Hh0gSVJP2RIgadGmdGCqtNp0PljaECBpKfZmOgemSqvNPjRnG+yEIUDSUmyB5iw4q/2KTJdsPrrrEsby0o0f67qEBS3pLEodWLf5xK5LWNDMzF3su+97oePWNEOApCVbx+rfiGzY8ENdlzCWXbsuYAzTcgGRdRt2W3gmAQ4MlCSptwwBkiT1lCFAkqSeMgRIktRThgBJknrKECBJUk8ZAiRJ6ilDgCRJPWUIkCSppwwBkiT1lCGgQ0lOTHJjkjuTXJnkiV3XJEnqD0NAR5I8GzgdeAPwWODTwEVJ9uuyLklSfxgCuvNS4Myqem9VXVdVLwG+CpzQbVmSpL4wBHQgya7AwcDFIw9dDBw2x3N2S7JhcAPWL3OZkqQ1zhDQjb1orsJ668j0W4GHzvGck4HNQ7ebl6069YJjUiQZArpVI/czy7SBNwEbh277LGNdWuMckyIJDAFd+QZwDzvu9T+YHVsHAKiqrVU1M7gBW5a5Rq1tjkmRZAjoQlXdBVwJHDXy0FHA51a+IvWJY1IkDRgCurMJ+M0kv57kvyZ5C7Af8K6O69La55gUSQDs0nUBfVVVH0yyJ3AK8DDgGuBpVXVTt5WpRxY7JmXT0P31GASkqWcI6FBVvQN4R9d1qHeWNCYF2Dq4n2TZipO0cuwOkHrGMSmSBmwJkPppE3BOkiuAy4HjcUyK1DuGAKmHHJMiCQwBUm85JkWSYwIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqae8dsCUe+iLYRqu7F5dFyBJ2oEtAZIk9ZQhQJKknkqVDbXTKMkGYPN9mZ7ugDuaXzdW1UynxWinTdPn7/ldFzCmD3VdwBj27rqAMT2x6wLGcBdwZvNrp9tEWwIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4aAjiQ5PMl5SW5JUkme2XVNkqR+MQR0Zw/gi8BJXRciSeonQ0BHquqiqnp1VX2461rUP7ZESQJDwNRIsluSDYMbsL7rmjTVbImSxC5dF6CxnQy8pusitDZU1UXARQBJOq5GUldsCZgebwI2Dt326bYc9YktUdLaZAiYElW1tapmBjdgS9c1qVdOBjYP3W7uthxJk2AIkDQOW6KkNcgxAR1Jcn/gkUOTHpHkIOD2qvq3bqqSZldVW4Gtg/uOI5DWBkNAdw4BPjl0f1P782zguBWvRpLUO4aAjlTVZYC7U+qELVGSwBAg9ZUtUZIMAVIf2RIlCTw6QJKk3jIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopryI45V4L3LfrIsZwB/DyrouQJG3HECBpyTay+psT3/1DXVcwnn+4u+sKFvYzXRcwpjdv7LqChc0UnDnTdRWr//9XkiQtE0OAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCOpLk5CT/lGRLktuSnJtk/67r0trnZ0/SgCGgO0cAZwCPB44CdgEuTrJHp1WpD/zsSQKaf351oKqOHr6f5IXAbcDBwKc6KUq94GdP0oAhYPXY2P68fbYHk+wG7DY0af2yV6S+mPezB37+pLXK7oBVIEmATcBnquqaOWY7Gdg8dLt5hcrTGjbmZw/8/ElrkiFgdXg78BjgV+aZ5000e2yD2z4rUJfWvnE+e+DnT1qT7A7oWJK3Ac8ADq+qOfeuqmorsHXoeStQndaycT974OdPWqsMAR1pm2HfBhwDHFlVN3ZcknrCz56kAUNAd84Angv8ArAlyUPb6Zur6o7uylIP+NmTBDgmoEsn0PStXgb8+9Dt2R3WpH7wsycJsCWgM1Vlp6o64WdP0oAtAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPGQIkSeopQ4AkST1lCJAkqacMAZIk9ZQhQJKknjIESJLUU4YASZJ6yhAgSVJPeRVBSUv2cKZgI3LXo7quYCw/mWu7LmFBp3RdwLi+fUzXFSxs5m7YeH7XVaz+/1/N70X/Ezbs3nUVC5u5E17+R11XIUkaZneAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOA1ENJTkhydZKZ9nZ5kqd2XZeklWUIkPrpZuCVwCHt7RPAR5Mc0GlVklaUIaAj7ompS1V1XlVdWFVfbm+vAr4DPL7r2iStnF26LqDHBntiN7T3f41mT+yxVfWl7spS3yRZB/wSsAdw+Rzz7AbsNjRp/QqUJmmZGQI6UlXnjUx6VZITaPbEDAFadkkOpPnS352mFeCYqrp2jtlPBl6zUrVJWhl2B6wCSdYleQ4L7Ikl2TC44Z6Ydt71wEE0wfOdwNlJHjXHvG8CNg7d9lmJAiUtL1sCOuSemLpUVXexrTvqiiSHAi8GXjTLvFuBrYP7SVakRknLy5aAbrknptUkbN/vL2mNsyWgQ+6JqStJ3ghcBHyVpmvpOcCRwNEdliVphRkCVhf3xLRSHgKcAzwM2AxcDRxdVR/vtCpJK8oQ0BH3xNSlqvqNrmuQ1D1DQHfcE5MkdcoQ0BH3xCRJXfPoAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKa8dIGnJHgXs2nURC/pS1wWM5emk6xIWtOFDXVcwrg93XcAYZoCNXRdhS4AkSX1lCJAkqafsDph2f7AZNmzouoqFzczAH3Xf9CVJ2saWAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUDquSQnJ6kkp3ddi6SVZQiQeizJocDxwNVd1yJp5RkCpJ5Kcn/g/cBvAd9aYN7dkmwY3ID1K1GjpOVlCJD66wzggqq6ZIx5TwY2D91uXs7CJK0MQ8AqYJ+sVlqS5wCPo/lyH8ebgI1Dt32WqTRJK2iXrgvoO/tktdKS7Au8FXhKVd05znOqaiuwdWgZy1SdpJVkS0CH7JNVRw4GHgxcmeT7Sb4PHAH8bnt/XbflSVophoBu2SerLlwKHAgcNHS7giaQHlRV93RVmKSVZXdAR4b6ZA8d8ylvAjYN3V+PQUBLUFVbgGuGpyX5LvDNqrpm9mdJWosMAR2wT1aStBoYArox3Cc7mLYOODzJScBuNslqJVXVkV3XIGnlGQK6MeiTHfY+4F+A0wwAkqSVYAjogH2ykqTVwKMDJEnqKVsCVgn7ZCVJK82WAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOeLEjSkt3VdQFjmJmZ6bqEsXyv6wLGMDMNRQJMwd98tXwuU1Vd16AlSLIB2Lx582Y2bNjQdTkLmpmZYePGjQAbq2p1fPq1ZEkeDtzcdR3SGrBPVX2tq5XbEiBpKW4B9gG2THCZ62mCxaSXO2nWOTnTUCMsX53raf6XOmMIkLRo1TQhTnTvJcng1y2rubXIOidnGmqEZa2z89fswEBJknrKloApt1oGlyxkWuqUpD4xBEyv9QD77rtv13Us1npWQROYVqWtwGvbn6uZdU7ONNQI01Pnonl0wJRK00m1N5MfTLOcA3XWA7eUHzpJWhUMAdrO4NBDPJRPktY8BwZKktRThgBJknrKEKBRa3YAjCRpe44JkCSpp2wJkNS5JCcmuTHJnUmuTPLErmsaleTwJOcluSVJJXlm1zWNSnJykn9KsiXJbUnOTbJ/13WNSnJCkquTzLS3y5M8teu65tO+t5Xk9K5rmSRDgKROJXk2cDrwBuCxwKeBi5Ls12Vds9gD+CJwUteFzOMI4Azg8cBRNOeCuTjJHp1WtaObgVcCh7S3TwAfTXJAp1XNIcmhwPHA1V3XMml2B0jqVJJ/AD5fVScMTbsOOLeqTu6usrklKeCYqjq361rmk+RBwG3AEVX1qa7rmU+S24GXVdWZXdcyLMn9gc8DJwKvBq6qqpd0WtQE2RIgqTNJdgUOBi4eeehi4LCVr2jN2dj+vL3TKuaRZF2S59C0tFzedT2zOAO4oKou6bqQ5eBpgyV1aS9gHXDryPRbgYeufDlrR3tW0U3AZ6rqmq7rGZXkQJov/d2B79C0rFzbbVXba8PJ44BDu65ludgSoHtNw+AsrVmj/ZKZZZoW5+3AY4Bf6bqQOVwPHEQzfuGdwNlJHtVpRUOS7Au8FXh+Vd3ZdT3LxRAgYKoGZ2lt+QZwDzvu9T+YHVsHNKYkbwOeATypqm7uup7ZVNVdVXVDVV3Rjv34IvDirusacjDN5/DKJN9P8n2agZe/295f1215k2EI0MBLgTOr6r1VdV078OWrwAnzP01auqq6C7iSZiT7sKOAz618RdMtjbcDxwJPrqobu65pEQLs1nURQy4FDqRprRjcrgDeDxxUVfd0VdgkOSZAw4Oz/njkIQdnaSVsAs5JcgVNH/HxwH7AuzqtakQ7SvyRQ5MekeQg4Paq+rduqtrBGcBzgV8AtiQZtLBsrqo7uitre0neCFxEs6OxHngOcCRwdIdlbaeqtgDbjaVI8l3gm6txjMVSGQIEDs5Sh6rqg0n2BE4BHkaz4X1aVd3UbWU7OAT45ND9Te3Ps4HjVrya2Q1a7i4bmf5C4KwVrWR+DwHOofl7b6Y5/v7oqvp4p1X1kOcJEEn2Br4GHFZVlw9NfxXwq1X1XzorTpK0bBwTIHBwliT1kiFADs6SpJ5yTIAGpmJwliRpcgwBAqZqcJYkaUIcGChJUk85JkCSpJ4yBEiS1FOGAEmSesoQIElSTxkCJEnqKUOAJEk9ZQiQJKmnDAGSJPWUIUCSpJ4yBEiS1FOGAEmSeur/AQI9gGEAR5unAAAAAElFTkSuQmCC\n", 269 | "text/plain": [ 270 | "
" 271 | ] 272 | }, 273 | "metadata": { 274 | "needs_background": "light" 275 | }, 276 | "output_type": "display_data" 277 | } 278 | ], 279 | "source": [ 280 | "import torch\n", 281 | "from matplotlib import pyplot as plt\n", 282 | "\n", 283 | "# Make an array of normally distributed randoms.\n", 284 | "m = torch.randn(2, 5).abs()\n", 285 | "print(f'm is {m}, and m[1,2] is {m[1,2]}\\n')\n", 286 | "print(f'column zero, m[:,0] is {m[:,0]}')\n", 287 | "print(f'row zero m[0,:] is {m[0,:]}\\n')\n", 288 | "dot_product = (m[0,:] * m[1,:]).sum()\n", 289 | "print(f'The dot product of rows (m[0,:] * m[1,:]).sum() is {dot_product}\\n')\n", 290 | "outer_product = m[0,:][None,:] * m[1,:][:,None]\n", 291 | "print(f'The outer product of rows m[0,:][None,:] * m[1,:][:,None] is:\\n{outer_product}')\n", 292 | "\n", 293 | "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(5, 5), dpi=100)\n", 294 | "def color_mat(ax, m, title):\n", 295 | " ax.set_title(title)\n", 296 | " ax.imshow(m, cmap='hot', vmax=1.5, interpolation='nearest')\n", 297 | " ax.get_xaxis().set_ticks(range(m.shape[1]))\n", 298 | " ax.get_yaxis().set_ticks(range(m.shape[0]))\n", 299 | "color_mat(ax1, m, 'm[:,:]')\n", 300 | "color_mat(ax2, m[0,:][None,:], 'm[0,:][None,:]')\n", 301 | "color_mat(ax3, m[1,:][:,None], 'm[1,:][:,None]')\n", 302 | "color_mat(ax4, outer_product, 'm[0,:][None,:] * m[1,:][:,None]')\n", 303 | "fig.tight_layout()\n", 304 | "fig.show()" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": {}, 310 | "source": [ 311 | "### Exercise\n", 312 | "\n", 313 | "Use `torch.mm` to compute `outer_product` and `dot_product`.\n", 314 | "\n", 315 | "Explain to yourself why order matters when using torch.mm but not when using `*`. " 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": 32, 321 | "metadata": {}, 322 | "outputs": [ 323 | { 324 | "name": "stdout", 325 | "output_type": "stream", 326 | "text": [ 327 | "False\n", 328 | "False\n" 329 | ] 330 | } 331 | ], 332 | "source": [ 333 | "# TODO Use torch.mm to compute outer_product and dot_product.\n", 334 | "\n", 335 | "outer = 'TODO'\n", 336 | "print(outer == outer_product)\n", 337 | "dot = 'TODO'\n", 338 | "print(dot == dot_product)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": {}, 344 | "source": [ 345 | "Devices and types\n", 346 | "-----------------\n", 347 | "\n", 348 | "One of the big reasons to use pytorch instead of numpy is that pytorch can do computations on the GPU. But because moving data on and off of a GPU device is more expensive than keeping it within the device, pytorch treats a Tensor's **computing device** as pseudo-type that requires explicit declaration and explicit conversion. Here are some things to know about pytorch devices and types:\n", 349 | "\n", 350 | "**Single precision CPU default.** By default a torch tensor will be stored on the CPU and will store single-precision 32-bit `torch.float` values.\n", 351 | "\n", 352 | "**Specifying data type.** To store a different data type such as integers, use the argument `dtype=torch.long` when you create the Tensor. For example, `z = torch.zeros(10, dtype=torch.long)`. This is similar to numpy with minor differences. See the [Tensor reference](https://pytorch.org/docs/stable/tensors.html) for all the types.\n", 353 | "\n", 354 | "**Specifying GPU.** To store the tensor on the GPU, specify `device='cuda'` when you make it, for example `identity_matrix = torch.eye(5, device='cuda')`. (Instead `device='cpu'` indicates the default CPU storage).\n", 355 | "\n", 356 | "Even on a multi-GPU machine it is fine to pretend there is only one GPU. Setting the environment variable `CUDA_VISIBLE_DEVICES=3` before you start the program will set up the process to see GPU\\#3 as the only visible GPU when it runs.\n", 357 | "\n", 358 | "As an aside, in principle you could instead target one of many GPUs with `device='cuda:3'`, but if you want to use multiple GPUs for the same computation your best bet is to a use a multiprocess utility class that manages data distribution between forked processes automatically, while each python process touches only one GPU. When this becomes an issue, read the [DistributedDataParalllel docs](https://pytorch.org/docs/stable/distributed.html).\n", 359 | "\n", 360 | "**Copying a tensor to a different device or type.** You cannot directly combine tensors that are on different devices (e.g., GPU vs CPU or different GPUs); this is similar to how most different-data-type combinations are also prohibited. In both cases you will need to convert types and move devices explicitly to make tensors compatible before combining them. The `x.to(y.device)` or `x.to(y.dtype)` function can be used to do the conversion.\n", 361 | "\n", 362 | "There are also commonly-used convenience synonyms `x.cpu()`, `x.cuda()`, `x.float()`, `x.long()`, etc. for making a copy of `x` with the specified device or type. There is a bit of cost, so move data only when needed.\n", 363 | "\n", 364 | "**GPU rounding is nondeterministic.** Computationally the GPU is **not** perfectly equivalent to the CPU. To speed parallelization, the GPU does not do associative operations such as summations in a deterministic sequential order. Since changing the order of summations can alter rounding behavior in fixed-precision arithmetic, GPU rounding can be different from CPU results an even nondeterministic. When the numerical algorithm is well-behaved, the difference should be small enough that you do not care, but you should know it is different. You can see this gap in the code example below.\n", 365 | "\n", 366 | "**float is fastest.** All commodity GPU hardware is fast at single-precision 32-bit floating-point math, about 20x CPU speed. Be aware that only expensive cards are fast at 64-bit double-precision math. If you change `torch.float` in the below example to `torch.double` on an Nvidia Titan or consumer card without hardware double-precision support, you will slow down to just-slightly-faster-than-CPU speeds. Similarly 16-bit `torch.half` or `torch.bfloat16` or other cool options will only be faster on newer hardware, and with these data types you need to take care that the reduced precision is not damaging your results.\n", 367 | "\n", 368 | "So `float` is the default and usually the best.\n", 369 | "\n", 370 | "Also note that some operations (like linear algebra) are floating-point only and cannot be done on integers.\n", 371 | "\n", 372 | "An example of some CPU versus GPU speed comparisons is below." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": 35, 378 | "metadata": {}, 379 | "outputs": [ 380 | { 381 | "name": "stdout", 382 | "output_type": "stream", 383 | "text": [ 384 | "time using the CPU alone: 1.39 seconds\n", 385 | "time using GPU, moving data from CPU: 0.135 seconds\n", 386 | "time using GPU on pinned CPU memory: 0.0728 seconds\n", 387 | "time using the GPU alone: 0.0174 seconds\n" 388 | ] 389 | }, 390 | { 391 | "data": { 392 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAEpCAYAAACUS/YHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAABcSAAAXEgFnn9JSAAAoyElEQVR4nO3debhkVXno/+8r80xHwKgMDY1iaEVFHECGVoZERREF54uAmvycQNBw1SDXJJqrRlH4KcFoAGeZBUQJMjSjCZExMqexGWQQaMYGmsH3/rFW0UVZZ16nT5+u7+d56tlda9hrV+2qOm/vvYbITCRJkgbNs6b6ACRJkqaCQZAkSRpIBkGSJGkgGQRJkqSBZBAkSZIGkkGQJEkaSAZBkiRpIBkESZKkgWQQJEmSBpJBkCRJGkgGQZIkaSAZBEmSpIFkECRJkgaSQZAkSRpI0z4Iiog/j4ivR8QNEfFoRCyIiEsj4itDlN8rIi6JiIdr2V9ExDYjtLFNLbeg1rskIt4/Oa9IkiQtCZGZU30M4xYRWwO/ANYGrgF+C6wBbA6sn5nL95Q/FDgAeBQ4E1gZ2BEIYM/MPLlPG7sDx1MCxvOBe2qdtYGvZ+aBk/DSJEnSJJu2QVBEPA+4GlgJeG9vABMRr8rMS7qevx44G7gX2Dozb6zpWwNzKYHRxpl5X1edGcDvgLWAt2fmSTX9OcCFwKbA6zPz3Ml6nZIkaXJM59thX6JcjTmo3xWc7gCo+mTdfqETANVyvwaOpAQ6+/bU+WBNP6UTANU6dwEH1adeCZIkaRqalleC6hWaO4DHgD/PzMdGKL8ycD/lqtEGmXlbT/52lFtd52XmnK7084Dtgf+VmT/sqbMi8EB9OmOkY5AkSUuX5UcuslR6LSWgOQt4IiL2ALYFVgCuA46rV2s6XlTL390bAFWX1e0WPelb9OQ/LTMfj4jfAlsBmwFXjvO1EBF3AqsCt453H5IkDagNgEcy88/HWnG6BkGz6/Yu4AJg6578/xsR+2Tm8fX5hnXbLwAiMxdGxP3AjIhYIzMfiog1KbfbhqxX07eq+x93EASsutJKK60xa9aszSewD0mSBs68efNYtGjRuOpO1yBoRt3uBSwCPgCcCqwOfJzST+eHEXF9Zl5V0wEeGWafCylBz+rAQ111hqu3sG5XHyL/GSLi6iGyVpw1axZXXz1UtiRJ6mf27Nlcc80147qTMl07Ri9Xt8sDB2bmUZl5T2bOz8xPAicAK7K483LU7XAdoGKE56OpI0mSponpeiXoobr9I/C9PvlHAXsAc3rKrzbMPlet24d76nTyHhxFnWFl5ux+6fUKkbfCJElagqbrlaD5dXtnZva7EdjJX69ub6nb9fvtLCJWo9wKuz8zHwLIzAdZPPqrb72u9FuGyJckSUup6RoEXV63MyKi3y2pZ9dt5wrN9ZS+Q+tGRL+AZsu6vaon/cqe/KdFxArAi+t+rx/lcUuSpKXEtAyCMvO/KTM5rwK8uk+ROXV7WS3/KHBOTdujT/lO2s970k8fps6ulGU3znaOIEmSpp9pGQRVX67bwyNinU5iRLyCxbNDH9lV/tC6PTgiXtBVfmvgbyh9fv6tp43v1vTdIuJtXXXWAzoLtB6KJEmadqZrx2iA71AWMt0TuD4iLqYMVd+GMjLsO5l5QqdwZp4VEYcB+wNXRMSvarmdKcHgezNzQXcDmbkgIvYFjgNOqDNI3wPsROlDdHhmnj25L1OSJE2GaRsEZeYfI+JdlMVPPwi8njIE/jfAkZn5gz51PhERVwAfowQ/T1AWVf1CZl44RDsnRsT2wMHAayiB07XAtzLz6NavS5IkLRnTNgiCEggBR9THaOscAxwzxnYuAt4wljqSJGnpNq2DIEmSWpr56dNHLqQJmf+lN031ITxtOneMliRJGjeDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA6lZEBQRsyPikIh4+TBlXl7L/EWrdiVJksaj5ZWg/YG/A+4cpsydwMHAxxu2K0mSNGYtg6AdgMsz846hCtS8y4DXNWxXkiRpzFoGQesDvxtFufm1rCRJ0pRpGQQ9DqwxinKrA9mwXUmSpDFrGQRdDWwXEWsPVSAiZgDbAdc2bFeSJGnMWgZBP6ZcCTo+Ip7bm1nTjqVcCfpRw3YlSZLGbPmG+/pX4F3AjsCNEfELYB7l1temwBuBVYGLgX9p2K4kSdKYNQuCMvPJiPgr4HDg/cAePUWeAo4G9s/MJ1q1K0mSNB4trwSRmY8AH4yIgynD4DeoWbcCc4cbPi9JkrQkNQ2COjLzTuAnk7FvSZKkFiYlCAKIiBcA6wD3ZuYNk9WOJEnSeDRdQDUiVomIL0fEvcB1wIXAp7vy94mIyyLiZS3blSRJGquWC6iuBpwHfApYBJwORE+x84GXAe9s1a4kSdJ4tLwS9L+BrYDvABtn5lt6C2TmPMoVop0atitJkjRmLYOgd1LWBftoZi4aptzNuHaYJEmaYi2DoA2BSzPzqRHKPQjMaNiuJEnSmLUMghZSRoONZGPg3obtSpIkjVnLIOhS4FURscFQBSJiNvBy4NcN25UkSRqzlkHQN4FVgJMiYtPezIjYCPh+bfObDduVJEkas2ZBUGaeBnwdeAVwfUT8lrJ46i4R8RvgRspVoK9k5txW7UqSJI1H08kSM/OTlJXk/xvYnDJP0POALSkryv+vzPxMyzYlSZLGo/myGZl5HHBcRKwLbAQsB9yWmb9v3ZYkSdJ4TdraYZl5N3D3ZO1fkiRpIiYtCOqIiOWBDwAvAW4Bvp2ZD0x2u5IkScNpuXbYIRHxVETs0JUWwDnAEcBHgP8L/FdErNmqXUmSpPFo2TF6Z+D3mXleV9rbgG0pHaX/BjgZ2BT4aMN2JUmSxqxlELQJcG1P2h6UYfLvyszvAHtSbont2bBdSZKkMWsZBD2bP+0IvR1wQ2ZeB5CZCfyGMmpMkiRpyrQMgu4G1u08iYhNKHMEnddT7nFgxYbtSpIkjVnLIOgaYLuutcM+RLkV9ouecjOBOxq2K0mSNGYtg6BDgZWBqyLiMuB/A78DzugUiIi1KLNHX9mwXUmSpDFruXbYv1OGwT8AbAZcCOyemY93FduLcivs7FbtSpIkjUfTyRIz80jgyGGKfJeykvzDLduVJEkaq0mfMbpbZj4KPLok25QkSeqn6SrykiRJ04VBkCRJGkgGQZIkaSAZBEmSpIFkECRJkgZSsyAoItaMiDVa7U+SJGkytbwSdD9wZsP9SZIkTZqWQdADwE0N9ydJkjRpWgZBlwOzGu5PkiRp0rQMgr4MvDIi9mi4T0mSpEnRMgh6lLI22LERcUpEfDAidomI7fs9GrZLRPxZRPwhIjIirhuh7F4RcUlEPBwRCyLiFxGxzQh1tqnlFtR6l0TE+1u+BkmStGS1XDtsLpBAAG8Gdh2h/HIN2z4UWGekQhFxKHAAJWA7E1gZ2BnYJSL2zMyT+9TZHTieEjCeD9wD7AgcExEvzcwDm70KSZK0xLQMgr5PCYKWqIjYEXg/8K/AXw9T7vWUAOheYOvMvLGmb00J4I6OiLmZeV9XnRnA0ZSA7e2ZeVJNfw5wIXBARJyWmedOxmuTJEmTp1kQlJl7t9rXaEXEKsCRwDXAVxkmCAI+Wbdf6ARAAJn564g4EtgP2Bf4WledDwJrAad0AqBa566IOAg4CTgQMAiSJGmame4zRv8fyoi0DwNPDFUoIlam3MICOKFPkU7am3vSd+3J73Y68BiwU92/JEmaRiYlCKodlXeOiHeP1Ol4Am1sQbm6c3Rmnj9C8RcBKwF3Z+ZtffIvq9stetK36Ml/WmY+DvyW0q9os9EetyRJWjo0DYIi4jkRcSxwF3AG8EPKLaVO/kfqCKvtJtjOs4DvUGapPmgUVTas234BEJm5sO5rRmfpj4hYE1h7uHpd6RsOkS9JkpZSzfoERcQ6wMXAxpSJEy8CPtZT7GfAYcAewAUTaO7jwKuAfTLz3lGUX71uHxmmzEJK0LM68FBXneHqLezZ/7Ai4uohspxkUpKkJazllaDPUQKgQzLzFZm5X2+BzLwduBYY9zxBEbEB8AXgvMw8ZrTVOocwijJDPR9NHUmSNE20HCL/FuDazPzCCOVuBl4zgXaOAFakdIYerYfqdrVhyqxatw/31OnkPTiKOsPKzNn90usVos1Hsw9JktRGyyDoucApoyj3GLDGBNrZldJ/518innEhpjNCa8OImNspm5kPA7fU5+v322FErEa5FXZ/Zj4EkJkPRsQDlCHy61OG4ffq7O+WPnmSJGkp1jIIegB4/ijKvQC4c4JtrQ3sMETeKl15ndd3PbAIWDci1u8zQmzLur2qJ/1Kyq27LekJgiJiBeDFdb/Xj/H4JUnSFGvZJ+hi4FUR0feWD0BEvJYy7HykIe1Dyszo96D0RwK4viv9/lrnUeCcmt9vgddO2s970k8fps6ulKtPZ2fmY+N9PZIkaWq0DIK+Rlle4tSI2LEOY39aRGwL/AB4Evh6w3ZH69C6PTgiXtB1XFsDf0Pp8/NvPXW+W9N3i4i3ddVZD/hKz34lSdI00iwIyswLKWtzbURZnHQBZTTW2yLibuA8ynw6n8jMy1u1O4bjO4syPP/ZwBUR8bOI+AXlqtQKwL6ZuaCnzgLKUhp/BE6IiHMj4njK7a9NgcMz8+wl+TokSVIbTSdLzMzDgW2B0+q+A1iTMo/OmcDrMvOIlm2O8fg+AexDGaa/M7ANcDawQ2aeOESdEyn9gv4deBnwRmAeJWjaf/KPWpIkTYaWHaMByMz/AN4aZejWsym3yO7JzKdat9XT7nxGMW9PnVvomDHu+yLgDeM5LkmStHRqHgR1ZGYC90zW/iVJkiZiUoKgiHg15bbY82rS7cBF9SqRJEnSlGsaBEXES4CjWDzvzjOWq4iIKyjrffXOxyNJkrREtVxAdTPKCLC1gVuBE4H5lEBoQ+DtwMuB8yJi68y8rlXbkiRJY9XyStA/UQKgL1EWUX2yOzMiDgL+AfgM8EVKUCRJkjQlWg6Rfx1wdWZ+tjcAAsjMpzLz74Cra1lJkqQp0zIIWoE/XXurn6tqWUmSpCnTMgi6Epg1inKzallJkqQp0zII+iLwyojYd6gCEbEP8EpK/yFJkqQp07Jj9ELgX4DvRMTewLHAzTVvI+CdwGtrmYcjYvvuypk57pXlJUmSxqplEDSXMh9QUCZKfG1PfmfOoA/XR6/lGh6LJEnSsFoGQd+nToooSZK0tGsWBGXm3q32JUmSNNladoyWJEmaNgyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA2kZkFQRDwnIraPiOf0pG8cET+JiN9GxOkR8apWbUqSJI1XyytBnwbOBdbuJETE6sCFwDuAzYE3AGdHxCYN25UkSRqzlkHQHODazLy+K21v4LnAT4DNgAOA1YBPNWxXkiRpzFoGQc8HbupJ2xV4Etg/M2/MzMOAK4DXNWxXkiRpzFoGQWsAD3WeREQArwYuzcx7u8pdD6zfsF1JkqQxaxkE/R7YuOv5VsBawNyecssDjzdsV5IkacxaBkG/Bl4VEbtFxJrAwUACp/WU+wtKwCRJkjRlWgZBXwQWAScB9wFvBuZm5sWdAhExkzJK7D8btitJkjRmy7faUWZeFxHbAvsD6wKXAv/cU+wvgSuBn7VqV5IkaTyaBUEAmXk5ZVj8UPnfBr7dsk1JkqTxcNkMSZI0kJpeCeqIiA0pkySuNFSZzDx/MtqWJEkajaZBUETsC3wO2HAUxZdr2bYkSdJYNAuCImIf4Lv16X8DNwAPt9q/JElSSy2vBB1IWSLj7ZnZOzeQJEnSUqVlx+gXAOcbAEmSpOmgZRC0AG9/SZKkaaJlEHQKZdmMVRruU5IkaVK0DII+CzwIHBMRazfcryRJUnMtO0Z/DbgG2APYJSJ+A9xGWUS1V2bmBxq2LUmSNCYtg6C9u/69FrDjMGUTMAiSJElTpmUQ9LqG+5IkSZpULVeRP6/VviRJkiabC6hKkqSB1HwB1YhYAdgd2A54HqX/zx3ABcDJmflE6zYlSZLGqvUCqq8FfgysD0RP9keAWyPiPZl5cct2JUmSxqrlAqovBH4JrA5cCvwQmF+zNwLeB2wF/DIitsrMG1u1LUmSNFYtrwT9HSUAOiAzD+uTf3hE7Ad8o5bdu2HbkiRJY9KyY/SOwOVDBEAAZObhwOXATg3blSRJGrOWQdC6wHWjKHcdsE7DdiVJksasZRB0L/DCUZR7IWXFeUmSpCnTMgg6F9gyIj40VIGa9wrgnIbtSpIkjVnLjtFfAN4KHBkR76EMlZ9PmSdoY+C9lLmDHgG+2LBdSZKkMWu5bMa1EfEW4EfADsD2PUUCuAt4b2Ze26pdSZKk8Wg6WWJmnh0RmwDvYPGM0QC3U2aMPi4zH2nZpiRJ0ng0XzajBjnH1IckSdJSyQVUJUnSQBp3EBQRG9bHcj3PR/WYyEFHxKoR8daI+LeIuCoiHoyIhRFxZUQcEhGrD1N3r4i4JCIejogFEfGLiNhmhPa2qeUW1HqXRMT7J/IaJEnS1JrI7bD5wB+BzYEbWDwSbDRygm2/B/hO/ffVwBnAmsA2wN8D746IHTLzD92VIuJQ4ADgUeBMYGVgZ2CXiNgzM0/ubSgidgeOpwSM5wP3UGbHPiYiXpqZB07gdUiSpCkykUDkfEow80jP8yXhceBfgK93L8QaEc8FTgdeTlmj7D1dea+nBED3Alt36kXE1sBc4OiImJuZ93XVmQEcDSwHvD0zT6rpzwEuBA6IiNMy89zJe6mSJGkyjDsIysw5wz2fTJn5feD7fdLviIiPAhcDb4uIFTPz8Zr9ybr9QnfglJm/jogjgf2AfYGvde3yg8BawCmdAKjWuSsiDgJOAg6kTBQpSZKmkWWxY/SVdbsS8GyAiFiZcgsL4IQ+dTppb+5J33WYOqcDjwE71f1LkqRppFkQFBHn1KsjI5X7VERM5rIZm9TtEyxeo+xFlKDo7sy8rU+dy+p2i570LXryn1avMP2W0q9os4kcsCRJWvJaXgmaQwk2RrIZZUbpybJ/3Z6RmYvqvzuj0foFQGTmQuB+YEZErAEQEWsCaw9Xryt9QqPdJEnSktd8ssRRWBl4cjJ2HBFvBD5AuQr0ua6szpD54WarXkgJelYHHuqqM1y9hT37H+n4rh4ia9Zo6kuSpHaWaJ+genVlG+COSdj3XwA/pKxR9reZeWV3dt0ON3otRng+mjqSJGmamNCVoIi4qSdpj4iYM0xbz6nbb06k3T7HsT5lrqAZwKGZeVhPkYfqdrVhdrNq3T7cU6eT9+Ao6gwrM2f3S69XiDYfzT4kSVIbE70dNrPr30m5LTTUraEnKAupngp8ZoLtPi0i1gF+RemXczTwqT7Fbqnb9YfYx2qUW2H3Z+ZDAJn5YEQ8QBkivz5wTZ+qnf3d0idPkiQtxSZ0Oywzn9V5UG4NHdOd1vNYKTM3zsz9W60kXzsx/5LSIfsk4EOZ2e+W1/XAImDdetWo15Z1e1VP+pU9+d1trwC8uO73+rEfvSRJmkot+wTtA/xbw/0NKyJWAk4BtgL+HXh3Zj7Vr2xmPgp0huXv0adIJ+3nPemnD1NnV0on77Mz87ExHLokSVoKNAuCMvN7mXlRq/0Npy7a+hPgdcAFwNu6ZoYeyqF1e3BEvKBrX1sDf0Pp89MbxH23pu8WEW/rqrMe8JWe/UqSpGlkUobI19tUs4A1GGIEVWaeP4EmPgbsXv99D3BERN9mPpWZ99T2zoqIwyjzCF0REb8CVqQsoPos4L2ZuaC7cmYuiIh9geOAEyLivNreTpQ+RIdn5tkTeB2SJGmKNA2CIuLFlIVL5zDy8PHlJtDUjK5/7z5kKfg8JWgBIDM/ERFXUIKonSmdtc+mrCd2Yb8dZOaJEbE9cDDwGkrgdC3wrcw8egKvQZIkTaFmQVC9xXQhsCZwEfBcYGPgp5SlLLas7Z1KmZ153DLz85QAZzx1jwGOGWOdi4A3jKc9SZK0dGrZMfpgyu2vfTJzO0pfHTLzvZm5NTCbEiRtTll5XZIkacq0DIJeD1ybmd/rl5mZ/wPsBqwL/GPDdiVJksasZRC0Hs+cUPAJgIhYuZOQmfcDcynDyyVJkqZMyyBoAWXenO7nABv1Kbtew3YlSZLGrGUQ9DtKR+iOKygjxN7VSahLXMzBZSYkSdIUaxkEnQlsHhGdQOg0yvD0QyLi2Ij4GvBflLW4jmvYriRJ0pi1nCfoB8BKlI7Pv8vMhRHxLkrAs2dXuV8BX2zYriRJ0pg1C4Iycx49q8Nn5jkRsRGwHWWCwxsy89JWbUqSJI1Xy8kS3wI8kZm/7E7PzIXAGa3akSRJaqFln6CTgf0a7k+SJGnStAyC7gbua7g/SZKkSdMyCJoLvCqGWM5dkiRpadIyCPocsA7w9e5ZoiVJkpZGLYfIvxv4BfBx4F0RcRZlUsTH+pTNzHT9MEmSNGVaBkGfB5IyS/R6wHuGKZu4iKqkZdDMT58+1YewzJv/pTdN9SFoGdEyCNqn4b4kSZImVcvJEr/Xal+SJEmTrWXHaEmSpGmj5e0wACJieWBX4JWU0WL/mZlH1bzn1bRrMvPJ1m1LkiSNVtMgKCJ2oCyk+nxKB+kEVgCOqkV2BI4B3gGc2LJtSZKksWh2OywiXkIZIr8ecBhl5fjeiRNPBB4B3t6qXUmSpPFoeSXoEGAlYJfMPAegd/LozHwkIq4FXt6wXUmSpDFr2TF6B+A/OgHQMG4BntewXUmSpDFrGQStCfx+FOVWApZr2K4kSdKYtQyC7gD+YhTlXgzc3LBdSZKkMWsZBJ0JzI6I3YcqEBF7AxsBzisvSZKmVMsg6J+Ah4GfRMQ/RsRWNX3ViHhxRBwMHAHcCxzasF1JkqQxaxYEZebNwJuA+4C/A/6TMk/QnsCVwD8ADwG7ZeadrdqVJEkaj6aTJWbmhRHxQuADwE7ATEon6NuAs4BvZ+b9LduUJEkaj+bLZmTmQ8A36kOSJGmp1HLG6EMi4i2jKPfmiDikVbuSJEnj0bJj9OeBt46i3FuA/9OwXUmSpDFrGQSN1nLAH6egXUmSpKdNRRA0mzKCTJIkacpMqGN0RBzVk7Rtn7TutjYDtgJ+NpF2JUmSJmqio8P27vp3ApvWx3CuAv52gu1KkiRNyESDoNfVbQDnAGcAXx6i7OPA7XVSRUmSpCk1oSAoM8/r/Dsivgdc0J0mSZK0tGo2WWJm7tNqX9Kgm/lp1xiebPO/9KapPgRJU2wqRodJkiRNuXFfCYqImyidoXfKzN/V56OVmTlrvG1LkiRN1ERuh82s2xV6nkuSJC31xh0EZeazhnsuSZK0NDNwkSRJA8kgSJIkDSSDIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgkYhIlaOiL+PiBsi4rGIuD0ijoqI9af62CRJ0vgYBI0gIlYGzgYOAVYHTgFuBfYBLouIWVN4eJIkaZwMgkb2WWAb4NfACzPznZn5auCTwLrAUVN5cJIkaXwMgoYRESsAH69PP5qZD3fyMvNQ4Cpg+4h4xVQcnyRJGj+DoOFtC6wNzMvMy/vkn1C3b15iRyRJkppYfqoPYCn30rq9bIj8y3rKLVVmfvr0qT6EZd78L71pqg9BkjROBkHD27Bubxsi/7aecsOKiKuHyHrRvHnzmD179liObUS33/XwyIU0IbNPW31S9uu5m3yeu+lrss4deP6WhNbnb968eQAbjKeuQdDwOmfqkSHyF/aUG68/Llq0aOE111xz6wT3M511RtnNm9KjGKNr7p3qI1hqTLvz57l7mudu+pp25w4m5fxtwNB/p4dlEDS8qNscIX9UMrPtpZ5lSOcqme/R9OT5m748d9OX527i7Bg9vIfqdrUh8letW6+fSpI0zRgEDe+Wuh1qZuj1e8pJkqRpwiBoeFfW7ZZD5HfSr1oCxyJJkhoyCBreRcADwKyIeHmf/D3q9udL7pAkSVILBkHDyMzHgW/Wp9+MiKf7BkXEgcAWwIWZ+V9TcXySJGn8InOogU+CpxdQnQu8GrgDuADYqD6/F3hNZv7PlB2gJEkaF4OgUYiIVYDPAO+hzEdwH3AG8LnMHOS5fSRJmrYMgiRJ0kCyT5AkSRpIBkGSJGkgGQRJkqSBZBAkSZIGkkGQJEkaSAZBkoYVEXMjIiNi5lQfS7d6TPOn+jiWtKX1fIxXRHy+vp69p/pYNHgMgiRJ0kBafqoPQNJSby9gVeD3U30gAjwfUjMGQZKGlZm3TPUxaDHPh9SOt8PUVERsGBHfjIgbI+KxiLg3Ii6JiM/W5Uee0achIt4XEZdGxCMR8YeI+F5EPL/Pfo+pdeYM0e5A9g/pp76vWd/n1SLi0Ii4NSIejYjLIuLNXWX3rOdnYUTcFRGHd85TV5k/6YMSEV+pacf2aX+9iLgzIp6MiK178l4SET+KiN9HxKKIuD0ijh6qf0s9/i9HxC3183RdRBwYETHR92lp0HOu1oyIw+q5eiwiro2IAyLiWT11+vYJ6nwHImK5iDgoIm6o7/Gt9T1cqU/78yMi678/GBFX1c/JnRHx7YhYe4jjXjEi9o+I/4qIh+rn55KI+MBQ5yYidqjH/nD9XTg5Il403vduWRAR76jv4aP1+3d0RDyn3+9d1/ldMSL+PiLm1c/JTRHxD1HWuezd/9Pnt0/enLrPYybtBU4HmenDR5MHsD1wP5DAPOBY4OfATTVtZi03tz7/JvBH4DzgJ8DvavqtwPo9+z6m5s0Zou0E5k/1e7A0PICZ9f24GPgP4G7gNOBc4CngSWAn4ADgiVruZOCeWu9HPfvrnK+ZXWkrApfV9L16yp9W0z/fk/52YFHN+w1wfNc+7gFm95RfCbio5t9dy58BPA58a1k4513n6tf1PbkPOLG+h4/UvKNHOh81PYH5wE+Bh4Fz6n4638kf9ml/fs37Sj03F9bPwl01/Xzq8kpddVar6Z3z8kvgdGBBTTuyTzu71c9d1nP6E8pvxAPAD2v63lN9Ppbwuf9Efd1PAmfV83Yb5XfwFHp+7+rzm4FT62fjtPpZ6Zzfs4Dl+p3fIdqfU+sdM9XvxZSeh6k+AB/LxgOYAfyhfqk+0eeHc3tgrfrvzo/4E8Abu8qs0PWDeFJP/WN6fxR68qf9H8SG56LzhzUpgc+Mrry9a/qNwL3Adl15z+v647dJV3rnfM3saedF9cf4ARYHuB9m8R/15bvKbgwsrD/Y2/fsZ69a55Ke9M/U9P/sfHZq+pa1zWl/znvO1ZXAOl15syj9fhJ4yyjOR2c/1/DMgHVjFgcos3rqzK/ptwMv60pfp35GEnh9T50javr3gdW70telBN0JvKkrfQ1KsJTAu7vSl+/6Xg9UEARsQgk6H+35Dq5M+Y9j5z2Z0+f83trz/VwX+O+at1+/8zvEMczBIMggyEebB3BQ/UKdNoqynR/xH/XJezblf7FPAc/vSu/8WM4ZYp/T/g9iw3PR+cP6JLBpT96zWBys/n2fuof2/kEa6o9uzftIzbsAmE0JdB7q88f2G7XcXw9xzCfX/C270m6padv0Kf9Py8I555lB0M598v+/mvfvI52Prv3s2Gc/h/cLNFgcBH2gT50D6bmiB6xHuRJ3E7BSnzovrXVO7Urbt6ad2af8jPp5GbQg6AsMfdVsVv39GyoI+lCfOn9V867vd36HOIY5GATZJ0jN7FS33x5DnZ/2JmTmvcCvKH+st2lwXINsfmb+T3dCZv6Rckkdyvvca17dPnc0DWTmEZRbIdtSbqutCuyfmfN6iu5ct6cMsasL6/aVUPqWARsAv8/Mi/uU/8lojm8aWZCZ/c7Hj+t2m1H2g3qCEiT1uqFuhzqvZ46yzg6UK7ZnZOai3gqZeSUlqHllV/K2dXtcn/L3DdH2sq7z23Z8b0b97lw+TN1+v5tnUG6lvjAi1m1yhAPCIEitbFC3vX/8hnPzEOnz6/Z54z4awdBDqBcOk9/J+5NOtMP4APAYsCblSuBRfcrMrNs7a2fMZzyAr9b8deq2c+6HGgm1rI2Q6vtdyMwHKbcQV6e8vyO5IzOf6pP+cN0OdV5vG2WdmXX74X7nsZ7LNVh8HmHwzuVodN6TW4fIH+o9uS8zHxoir/MZ8ndzDBwir9aywT7GNPKnd/SMnjbSuWhxrgDeQunLAPCiiFgtMxf2lFmOxf1IhnN13XY+A0MdY6tjnw7G8n0Y1/uS9f7IKCxXt5cDV42yzkjncpAN9Z6MZ/TjWOv4u4lBkNq5ldJRdlPgulHW2Yj+P6Qb1u3tXWmP1+3qfcpv0CdNS0BEvAD4OuUK0q+At9bnf91T9DZKX4f96tWNkXTO/UZD5A+VPl1t2C8xItYE1qK8v6N53yZb54rR3Mw8cJR1RjqXfV/7Mu4OYDPKa7+xT/5Qv2kzImKNIa4Gdd7HO7rSHgeIiNUz8+Ge8v5uYiSods6q294/fsN5Z29CRPwZsAuLRxh1dL7YL+yzn13G0KYaiYjlgR9Rhkx/Angf5Qf9QxHx1p7inc9Hb3pfmXkz5Q/u83vnGqreNfYjXqo9OyJ26pP+7rq9eAxXaybTuZROu7tGxHIjFa46/b327M2o8xAN4ve3089tj96MiNgEePkwdfv9bv4lpZP5jZn5h64sfzdHYBCkVr5LmevlzRHxsd5OnBGxXUSs1VPnHfXL2ymzPOUqwmqU0SXd/RTOq9sPR8Szu+psCfxjw9eh0fs8pQPsKZn53XoL7H2UUWnfiYg/7yr7Ncpw4K9H12SNHRHxZxHxkXjmRI2dTvZfq1dEOmVfBny06StZOvxzz2d7Y+Bz9ekRU3NIz5SZv6eM1HwB8IOIWKe3TERsExFv7Eo6njJEf5eIeEdXueUon4t+V3eXdUdTOrHvHRFPDwCpEx5+g+H/Nh8Sz5y4dB3KPE/wp5+Tzu/mZ7qD1oh4H8vefyTGxSBITWTmAuAdlJEh/z9wY0QcGxGnRcRNlMnVZvRU+1fgl3UW2R9TRqPsRbl8vl9P2XMpX+hNgWsi4qSIuIAyL8kPJut1qb+I2Bb4NGVeoQ910jPzEkpQug5wdCcYzswbKQHSKsCpUWZ+PjkifhYRl1P+x/otntkJ958pcwRtDcyLiOMi4pc17ccsW/6DMnHojRFxQkScCvwWeD5lksOfTeXB9diP8n18N3BTRJwfET+t3+PbKJMhPn2Vod7+/GvK6zs2Ii6s3/frKVdCfrTEX8EUq6M2P0vpS3d+RPwqIn5KuZL6UspEiLC4G0DHLZT5pK6OiFMj4sRaZwvKOflmT/lvUeZo2oPyu3l8RFxBCWQPa/26piODIDWTmecCL6MEN8tTbn28hjIvzWeAO3uqfBXYh9LnYXfK6JcfAK/OnvWR6q2A3YAjKbfK3kgJqvbLzL+dlBekvupVmR9QOsnum5l39xT5IuVW5l8BH+skZuZJlB/4b1OGWb+BMlfJSpQ/hLtSJkHslF9EmXrhq5SJ5XajTDJ3cPd+lxGLgNdThv5vDfwlpZ/dpygTXC41MvMRSpDzQcqM3y+mfH9nUUaHHsTi0X6dOidSpkm4gHKr5w2USR23Bp4xjcOgyMyvUgLJK4HtKOf/XMpvZueK6L291SgBzTeAl7D4O/NFygSVT/a0cRdlotqfU6Y6eEMtvzNl5umBF0vHbWYNkoiYS5lvZOPMnD+1RyNNnXpb43fAeZk5Z2qPRkuDiFiNMk3IKpSZ0p+q6QncnJkzp+7olj1eCZIkaQmLiE16+0lGxOqUq93rAMcOMeeTGnKIvCRJS947gM9HxKWUkZAzKLcK16FcCfrs1B3a4DAIkiRpyTub0ofyNZTgJygdn78HfLlPXztNAvsESZKkgWSfIEmSNJAMgiRJ0kAyCJIkSQPJIEiSJA0kgyBJkjSQDIIkSdJAMgiSJEkDySBIkiQNJIMgSZI0kAyCJEnSQDIIkiRJA8kgSJIkDSSDIEmSNJAMgiRJ0kD6f2fzpPF1P8mgAAAAAElFTkSuQmCC\n", 393 | "text/plain": [ 394 | "
" 395 | ] 396 | }, 397 | "metadata": { 398 | "needs_background": "light" 399 | }, 400 | "output_type": "display_data" 401 | }, 402 | { 403 | "name": "stdout", 404 | "output_type": "stream", 405 | "text": [ 406 | "Your GPU is 80.1x faster than CPU but only 10.3x if data is repeatedly copied from the CPU\n", 407 | "When copying from pinned memory, speedup is 19.1x\n", 408 | "Numerical differences between GPU and CPU: 0.0002938236575573683\n" 409 | ] 410 | } 411 | ], 412 | "source": [ 413 | "import torch, time\n", 414 | "from matplotlib import pyplot as plt\n", 415 | "\n", 416 | "# Here is a demonstration of moving data between GPU and CPU.\n", 417 | "# We multiply a batch of vectors through a big linear opeation 10 times\n", 418 | "r = torch.randn(1024, 1024, dtype=torch.float)\n", 419 | "x = torch.randn(32768, 1024, dtype=r.dtype)\n", 420 | "iterations = 10\n", 421 | "\n", 422 | "def time_iterated_mm(x, matrix):\n", 423 | " start = time.time()\n", 424 | " result = 0\n", 425 | " for i in range(iterations):\n", 426 | " result += torch.mm(matrix, x.to(matrix.device).t())\n", 427 | " torch.cuda.synchronize()\n", 428 | " elapsed = time.time() - start\n", 429 | " return elapsed, result.cpu()\n", 430 | "\n", 431 | "cpu_time, cpu_result = time_iterated_mm(x.cpu(), r.cpu())\n", 432 | "print(f'time using the CPU alone: {cpu_time:.3g} seconds')\n", 433 | "\n", 434 | "mixed_time, mixed_result = time_iterated_mm(x.cpu(), r.cuda())\n", 435 | "print(f'time using GPU, moving data from CPU: {mixed_time:.3g} seconds')\n", 436 | "\n", 437 | "pinned_time, pinned_result = time_iterated_mm(x.cpu().pin_memory(), r.cuda())\n", 438 | "print(f'time using GPU on pinned CPU memory: {pinned_time:.3g} seconds')\n", 439 | "\n", 440 | "gpu_time, gpu_result = time_iterated_mm(x.cuda(), r.cuda())\n", 441 | "print(f'time using the GPU alone: {gpu_time:.3g} seconds')\n", 442 | "\n", 443 | "plt.figure(figsize=(4,2), dpi=150)\n", 444 | "plt.ylabel('iterations per sec')\n", 445 | "plt.bar(['cpu', 'mixed', 'pinned', 'gpu'],\n", 446 | " [iterations/cpu_time,\n", 447 | " iterations/mixed_time,\n", 448 | " iterations/pinned_time,\n", 449 | " iterations/gpu_time])\n", 450 | "plt.show()\n", 451 | "\n", 452 | "print(f'Your GPU is {cpu_time / gpu_time:.3g}x faster than CPU'\n", 453 | " f' but only {cpu_time / mixed_time:.3g}x if data is repeatedly copied from the CPU')\n", 454 | "print(f'When copying from pinned memory, speedup is {cpu_time / pinned_time:.3g}x')\n", 455 | "print(f'Numerical differences between GPU and CPU: {(cpu_result - gpu_result).norm() / cpu_result.norm()}')" 456 | ] 457 | }, 458 | { 459 | "cell_type": "markdown", 460 | "metadata": {}, 461 | "source": [ 462 | "### Exercise\n", 463 | "\n", 464 | "Repeat the benchmark using type `torch.double`. What does that tell you about your GPU hardware?" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": 38, 470 | "metadata": {}, 471 | "outputs": [], 472 | "source": [ 473 | "# TODO: Repeat the benchmark using type torch.double.\n", 474 | "r = 'TODO'\n", 475 | "x = 'TODO'\n", 476 | "\n", 477 | "# Benchmark and plot" 478 | ] 479 | }, 480 | { 481 | "cell_type": "markdown", 482 | "metadata": {}, 483 | "source": [ 484 | "Performance tips\n", 485 | "----------------\n", 486 | "\n", 487 | "**GPU operations are async.** When pytorch operates on GPU tensors, the python code does not wait for computations to complete. Sp GPU calculations get queued up, and they will be done as quickly as possible in the background while your python is free to work on other things like loading the next batch of training data.\n", 488 | "\n", 489 | "**Moving data to cpu waits for computations.** You do not need to worry about the GPU asynchrony, because as soon as you actually ask to look at the data, e.g., when you move GPU data to CPU (or print it or save it), pytorch will block and wait for the GPU operations to finish computing what you need before proceeding. The call seen above to `torch.cuda.synchronize()` flushes the GPU queue without requesting the data, but you will not need to do this unless you are doing performance timing.\n", 490 | "\n", 491 | "**Pinned memory transfers are async and faster.** Copying data from CPU to GPU can be sped up if the CPU data is put in pinned memory (i.e., at a fixed non-swappable block of RAM). Therefore when data loaders gather together lots of CPU data that is destined for the GPU, they should be configured to stream their results into pinned memory. See the performance comparison above." 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "metadata": {}, 497 | "source": [ 498 | "pytorch Tensor dimension-ordering conventions\n", 499 | "---------------------------------------------\n", 500 | "\n", 501 | "**Multidimensional data convention.** As soon as you have more than one dimension, you need to decide how to order the axes. To reduce confusion, most data processing follows the same global convention. In particular, much image-related data in pytorch is four dimensional, and the dimensions are ordered like this: `data[batch_index, channel_index, y_position, x_position]`, that is:\n", 502 | "\n", 503 | "* Dimension 0 is used to index separate images within a batch.\n", 504 | "* Dimension 1 indexes channels within an image representation (e.g., 0,1,2 = R,G,B, or more dims for more channels).\n", 505 | "* Dimension 2 (if present) indexes the row position (y-value, starting from the top)\n", 506 | "* Dimension 3 (if present) indexes the column position (x-value, starting from the left)\n", 507 | "\n", 508 | "There a way to remember this ordering: adjacent entries that vary only in the last dimensions are stored physically closer in RAM; since they are often combined with each other, this could help with locality, whereas the first (batch) dimension usually just groups separate independent data points which are not combined much, so they do not need to be physically close.\n", 509 | "\n", 510 | "Stream-oriented data without grid geometry will drop the last dimensions, and 3d grid data will be 5-dimensional, adding a depth z before y. This same 4d-axis ordering convention is also seen in caffe and tensorflow.\n", 511 | "\n", 512 | "Separate tensors can be put together into a single batch tensor using `torch.cat([a, b, c])` or `torch.stack([a, b, c])`. (The difference: `cat` doesn't add any new dimensions but just concatenates along the existing 0th dimension. `stack` adds a new 0th dimension for the batch.)\n", 513 | "\n", 514 | "**Multidimensional linear operation convention.** When storing matrix weights or convolution weights, linear algebra conventions are followed\n", 515 | "* Dimension 0 (number of rows) matches the output channel dimension\n", 516 | "* Dimension 1 (number of columns) matches the input channel dimension\n", 517 | "* Dimension 2 (if present) is the convolutional kernel y-dimension\n", 518 | "* Dimension 3 (if present) is the convolutional kernel x-dimension\n", 519 | "\n", 520 | "Since this convention assumes channels are arranged in different rows whereas the data convention puts different batch items in different rows, some axis transposition is often needed before applying linear algebra to the data.\n", 521 | "\n", 522 | "**Permute and view reshape an array without moving memory.** The `permute` and `view` methods are useful for rearranging, flattening, and unflatteneing axes. `x.permute(1,0,2,3).view(x.shape[1], -1)`. They just alter the view of the block of numbers in memory without moving any of the numbers around, so they are fast.\n", 523 | "\n", 524 | "**Reshaping sometimes needs copying.** Some sequences of axis permutations and flattenings cannot be done without copying the data into the new order in memory; the `x.contiguous()` method copies the data iinto the natural order given by the current view; also `x.reshape()` is similar to `view` but will makea copy if necessary so you do not need to think about it. See [the Tensor.view method documentation](https://pytorch.org/docs/master/tensors.html#torch.Tensor.view).\n" 525 | ] 526 | }, 527 | { 528 | "cell_type": "markdown", 529 | "metadata": {}, 530 | "source": [ 531 | "### Exercise\n", 532 | "\n", 533 | "Use `torch.randn` to create a four-dimensional tensor `x` of size (2,3,4,5), which could store two 5x4 RGB images.\n", 534 | "\n", 535 | "Then print three things:\n", 536 | " * print `x`.\n", 537 | " * Use `x.permute` to switch the horizontal and vertical (last two) dimensions.\n", 538 | " * Use `x.view` to see each image as a flat vector of 60 numbers." 539 | ] 540 | }, 541 | { 542 | "cell_type": "code", 543 | "execution_count": 60, 544 | "metadata": {}, 545 | "outputs": [ 546 | { 547 | "name": "stdout", 548 | "output_type": "stream", 549 | "text": [ 550 | "TODO\n", 551 | "TODO\n", 552 | "TODO\n" 553 | ] 554 | } 555 | ], 556 | "source": [ 557 | "# TODO make x of size (2,3,4,5), and print three rearrangements of x\n", 558 | "x = 'TODO'\n", 559 | "print(x)\n", 560 | "print('TODO')\n", 561 | "print('TODO')" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "metadata": {}, 567 | "source": [ 568 | "## Special topic: einsum notation\n", 569 | "\n", 570 | "Matrix multiplication can be generalized to tensors of arbitrary number of dimensions, but keeping tensor dimensions straight can be confusing. The solution to this is [Einstein notation](https://en.wikipedia.org/wiki/Einstein_notation): assign letter variables to each axis of the input tensors, and then explicitly write down which axes end up in the output tensor. For example, an outer product might be written as `i, j -> ij`, whereas matrix multiplication could be `ij, jk -> ik`.\n", 571 | "\n", 572 | "Einstein notation is a topic of active development and programming language design: [here is a recent paper on the history and future of Einstein APIs.](https://openreview.net/pdf?id=oapKSVM2bcj)\n", 573 | "\n", 574 | "\n", 575 | "In pytorch, Einstein notation is available as `einsum`. Here is how ordinary matrix multiplication looks as einsum:" 576 | ] 577 | }, 578 | { 579 | "cell_type": "code", 580 | "execution_count": 58, 581 | "metadata": {}, 582 | "outputs": [ 583 | { 584 | "name": "stdout", 585 | "output_type": "stream", 586 | "text": [ 587 | "tensor([[ 3.2591, -0.9139, 3.3531],\n", 588 | " [ 4.6914, -1.4011, 5.6399]])\n" 589 | ] 590 | } 591 | ], 592 | "source": [ 593 | "A = torch.randn(2,5)\n", 594 | "B = torch.randn(5,3)\n", 595 | "\n", 596 | "# Uncomment to see ordinary matrix multiplication\n", 597 | "# print(torch.mm(A, B))\n", 598 | "\n", 599 | "# Ordinary matrix multiplication written as an einsum\n", 600 | "print(torch.einsum('ij, jk -> ik', A, B))" 601 | ] 602 | }, 603 | { 604 | "cell_type": "markdown", 605 | "metadata": {}, 606 | "source": [ 607 | "### Exercise\n", 608 | "\n", 609 | "Make A in the shape (5, 6, 2) and B in the shape (5, 6, 3); we can think of A as a 5x6 grid of 2-dimensional vectors and B as a 5x6 of 3-dimesnsional vectors.\n", 610 | "\n", 611 | "Covariances (un-normalized) of vectors in A and B could be computed by flattening and transposing the tensors into (2,30) and (30,3) matrices and then doing a matrix multiplication of these batches as follows:\n", 612 | "\n", 613 | "```\n", 614 | "print(torch.mm(A.reshape(30, 2).t(), B.reshape(30, 3)))\n", 615 | "```\n", 616 | "\n", 617 | "Instead use einsum to compute the same thing.\n" 618 | ] 619 | }, 620 | { 621 | "cell_type": "code", 622 | "execution_count": 56, 623 | "metadata": {}, 624 | "outputs": [ 625 | { 626 | "name": "stdout", 627 | "output_type": "stream", 628 | "text": [ 629 | "TODO\n" 630 | ] 631 | } 632 | ], 633 | "source": [ 634 | "# TODO: use einsum to compute a covariance statistic over vectors in A and B.\n", 635 | "A = torch.randn(5,6,2)\n", 636 | "B = torch.randn(5,6,3)\n", 637 | "\n", 638 | "\n", 639 | "print('TODO')" 640 | ] 641 | }, 642 | { 643 | "cell_type": "markdown", 644 | "metadata": {}, 645 | "source": [ 646 | "### [On to topic 2: Autograd →](2-Pytorch-Autograd.ipynb)" 647 | ] 648 | } 649 | ], 650 | "metadata": { 651 | "accelerator": "GPU", 652 | "kernelspec": { 653 | "display_name": "Python 3 (ipykernel)", 654 | "language": "python", 655 | "name": "python3" 656 | }, 657 | "language_info": { 658 | "codemirror_mode": { 659 | "name": "ipython", 660 | "version": 3 661 | }, 662 | "file_extension": ".py", 663 | "mimetype": "text/x-python", 664 | "name": "python", 665 | "nbconvert_exporter": "python", 666 | "pygments_lexer": "ipython3", 667 | "version": "3.9.9" 668 | } 669 | }, 670 | "nbformat": 4, 671 | "nbformat_minor": 4 672 | } 673 | -------------------------------------------------------------------------------- /notebooks/2-Pytorch-Autograd.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Topic 2: Pytorch Autograd\n", 8 | "================\n", 9 | "\n", 10 | "If you flag a torch Tensor with the attribute `x.requires_grad=True`, then pytorch will automatically keep track the computational history of all tensors that are derived from `x`. This allows pytorch to figure out derivatives of any scalar result with regard to changes in the components of x.\n", 11 | "\n", 12 | "\n", 13 | "\n", 14 | "The function `torch.autograd.grad(output_scalar, [list of input_tensors])` computes `d(output_scalar)/d(input_tensor)` for each input tensor component in the list. For it to work, the input tensors and output must be part of the same `requires_grad=True` compuation.\n", 15 | "\n", 16 | "In the example here, `x` is explicitly marked `requires_grad=True`, so `y.sum()`, which is derived from `x`, automatically comes along with the computation history, and can be differentiated." 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 37, 22 | "metadata": {}, 23 | "outputs": [ 24 | { 25 | "data": { 26 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABJbklEQVR4nO2dd3iUVdqH75NeSU8gBUIJvRMQQVAQUbCjoq69LOra1111V9ey6lbL6tqWte+6on6KDURUVECl906oCYH0XkgmOd8fZyYEmCRT3pl3Jjn3deV6J/O2Z1J+88xzniKklGg0Go2m8xNgtgEajUaj8Q5a8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9FoughBZhvQHomJiTIzM9NsMzQajcZvWLt2bbGUMsnePp8W/MzMTNasWWO2GRqNRuM3CCEOtLVPh3Q0Go2mi6AFX6PRaLoIDgu+EOINIUShEGJLq+fihRBfCyF2W7dxbZx7jhBipxAiRwjxoBGGazQajcY5nInhvwW8CLzT6rkHgW+llH+xCvmDwAOtTxJCBAIvAWcBecBqIcRnUsptrhjc2NhIXl4e9fX1rpzut4SFhZGenk5wcLDZpmg0Gj/FYcGXUi4VQmSe8PSFwBnWx28D33OC4APjgBwp5V4AIcQ863kuCX5eXh7R0dFkZmYihHDlEn6HlJKSkhLy8vLo3bu32eZoNBo/xd0YfoqU8jCAdZts55g0ILfV93nW5+wihJgjhFgjhFhTVFR00v76+noSEhK6jNgDCCFISEjocp9qNBqNsXhj0daeMrfZolNKOVdKmS2lzE5KsptK2qXE3kZXfM0ajcZY3BX8AiFEDwDrttDOMXlARqvv04F8N++r0Wg0/sf+5ZC/wbTbuyv4nwHXWR9fB3xq55jVQJYQorcQIgS4wnqeRqPRdC3m3wqf/Mq02zuTlvke8DMwQAiRJ4S4CfgLcJYQYjcqC+cv1mNThRALAaSUFuAO4CtgO/CBlHKrsS9Do9FofJyaYqjIhcKtUGCOBDqTpXNlG7vOtHNsPjCz1fcLgYVOW+eD/OEPfyAxMZG7774bgIceeoiUlBTuuusuky3TaDQ+zeENxx5v+gDOetzrJvh0L52OePzzrWzLrzT0moNTu/Ho+UPa3H/TTTcxa9Ys7r77bpqbm5k3bx6rVq0y1AaNRtMJscXuM8bD5v+DMx+FAO82O9CtFZwkMzOThIQE1q9fz+LFixk1ahQJCQlmm6XRaHydwxshvg+MvQkq8+Dgz143wa89/PY8cU9y880389Zbb3HkyBFuvPFGU2zQaDR+xuENkDYGBsyE4AjY/AFkTvSqCdrDd4GLL76YRYsWsXr1as4++2yzzdFoNL5ObSmUH4QeIyE0CgaeC1s/AUuDV83Qgu8CISEhTJkyhdmzZxMYGGi2ORqNxtc5vFFte4xQ22Gzob4ccr72qhla8F2gubmZFStWcNNNN5ltikaj8QdOFPy+UyAiAbZ85FUztOA7ybZt2+jXrx9nnnkmWVlZZpuj0Wj8gcMbILYnRMSr7wODIeMUKNrpVTP8etHWDAYPHszevXvNNkOj0fgThzeq+H1rolIg17sp3drD12g0Gk9SXwGleyF15PHPR6VAbTE0NXrNFC34Go1G40kOb1JbW/zeRnSK2tac3AbeU2jB12g0Gk9ia6lgL6QDUHXEa6ZowddoNBpPcngjdEuHyMTjn4/qrrbV9rrKewYt+G7y2GOP8fTTT7d7zHvvvcdTTz110vOZmZkUFxd7yjSNpmuw+GH48kE4WmW2JfY5vPHkcA4cC+lUaw+/U7Fo0SLOOeccs83QaDofDTWw4hVY+Qq8PAH2LDHbopOpOATxdmZRR1onwlYVeM0ULfgu8NRTTzFgwACmTZvGzp07aWpqYvTo0S37d+/ezZgxYwA1gHzDhg2MHj2akpISpk+fzqhRo7jllluQUk16XL16NcOHD6e+vp6amhqGDBnCli1bTHltGo1fkbcGmi1wxu8gOAz+czH8+ILZVh2joQYaayDSzrjWoBAIj4dq7wm+f+fhf/kgHNls7DW7D4MZf2lz99q1a5k3bx7r16/HYrEwevRoxowZQ0xMDBs2bGDkyJG8+eabXH/99QCsX7+eESNGIITg8ccf57TTTuORRx5hwYIFzJ07F4CxY8dywQUX8PDDD1NXV8fVV1/N0KFDjX1dGk1n5ODPgIBTboWJ98A7F8CG/8FEH5lPYcvAsSf4oBZuvSj42sN3kmXLlnHxxRcTERFBt27duOCCCwDVQfPNN9+kqamJ999/n1/84heACufMmDEDgKVLl3L11VcDcO655xIXF9dy3UceeYSvv/6aNWvWcP/993v5VWk0fsqBnyBlKITHKg+/10Qo2e31pmRtUm0V/Khk+/ujvSv4bnv4QogBwPutnuoDPCKl/EerY85AzbvdZ33qYynlH929d3ueuCcRQpz03CWXXMLjjz/O1KlTGTNmTEuP/MWLF/PRRx+1ey5AaWkp1dXVNDY2Ul9fT2RkpGeM12g6C02NkLcaRl197LmUISrEU7JbPTabFg8/0f7+qO7qTctLuO3hSyl3SilHSilHAmOAWmC+nUOX2Y4zROxNYvLkycyfP5+6ujqqqqr4/PPPAQgLC+Pss8/mtttu44YbbgCgoqICi8XSIv6TJ0/m3XffBeDLL7+krKys5bpz5szhiSee4KqrruKBBx7w8qvSaPyQI5ugsRZ6nnrsueTBaluwzRybTqTGmnLZZkgnWWXpWNfzPI3RIZ0zgT1SygMGX9dnGD16NJdffjkjR47kkksuYdKkSS37rrrqKoQQTJ8+HYCvv/6aadOmtex/9NFHWbp0KaNHj2bx4sX07NkTgHfeeYegoCB+8Ytf8OCDD7J69WqWLPHBbAONxpc4YJ0Y1WvCsecS+kFAkBoU7gt0FMOP7g5NDapVshcwetH2CuC9NvadKoTYCOQDv5FS+shvxHkeeughHnrooZOeX758OTfeeGNLj/xFixZx8803t+xPSEhg8eLFLd8/99xzAFx77bVce+21AAQGBrJy5UpPmq/RdA4O/gxxvZVo2ggKgcT+vuPhVxdBaAwEhdrf31JtWwDhcfaPMRDDBF8IEQJcAPzOzu51QC8pZbUQYibwCWC3t7AQYg4wB2jxgP2Biy++mD179hznmb/22msmWqTRdGKkVILf3059S/JgyPURp6mmCKLa8O7hmOBXF0DyQI+bY2RIZwawTkp50pKzlLJSSlltfbwQCBZC2F3FkFLOlVJmSymzk5La+UH5GPPnz2fTpk0kJraxOKPRaIyjeBfUlkDP8SfvSxkMFbmqS6XZ1BS1Hc6BY59OvJSpY6TgX0kb4RwhRHdhTU8RQoyz3rfE1RtJLy1w+BJd8TVrNG1iy2zpOeHkfcnW7JzCHd6zpy06EnxbuqaXGqgZIvhCiAjgLODjVs/dKoS41frtpcAWawz/BeAK6aKChYWFUVJS0qUEUEpJSUkJYWFhZpui0fgGB1coIU3oe/K+FGumji8s3HYk+KHdICjcax6+ITF8KWUtkHDCc6+2evwi8KIR90pPTycvL4+iIu/1kPYFwsLCSE9PN9sMjcY3OPizSse0V9cSkwEh0eYv3DZZoLa07aIrUPZHJfuX4HuT4OBgeve204hIo9F0DSxHofwAjLzK/n4hIHkQFJos+LUlgGy76MpGdHe/jOFrNBqN56nIU9vYjLaPSRkMBVu9VtBkl5aiq3Y8fFCZOl7qmKkFX6PR+Bfl1rrO2HbStpOHqGKmqsNeMckuHRVd2YhK8VpPfC34Gk1XpXAHvDsb9i012xLnKM9V2/YEv2Xh1sSwTrWDgh+dolJIG+s9bpIWfI2mqyElrPsPzD0Ddn8FC+5TC4z+QvlBEIEQndr2Mb7QU8fm4bdXeAXHF195GC34Gk1XorkJ5t8Kn90BGWPh3GdVEdPGtjqi+CDlB6FbGgS2k3MSEa86UZrp4dcUQWCISr1sDy/OttWCr9F0JfYvg03z4LR74ZpPIPtGSMuG7//slZCCIVTkth/OsZE8CIpMLL6qKVILtm20RG/BlrbphTi+FnyNpiux7TMIjoDJ90NAoBKjaY9B5SFY7Se9n8oPtp+hYyMmzWsVrHapKeo4JROOtVfwgq1a8DWarkJzM+z4ArLOgpCIY8/3ngR9p8KyZ3yj/0x7WBqgMt8xDz/Kmt/e3OR5u+xRXdh+0ZWNyCQQATqko9FoDCRvlRLAQRecvO/MR6CuFNa84X27nKHyECAdE/zo7iCboabY42bZpaa44wwdUJ+0IhJ1SEej0RjIts/UImLW9JP3pY5Suev7lnnfLmcoP6i2MQ6EdFpCJSbk4kupCq8cEXywzrbVHr5GozECKWH75yp0E9ZG1kjGWMhbo0I/vopN8B3y8HuorRlx/KOVapKVo4IflaJj+BqNxiDy10PFQRh0ftvHpI+DoxUqTdNXqchV8e5uaR0f29Jr3gTBtxVdORLDBxXSqSv1nD1WtOBrNF2B7Z+pYqUBM9s+JmOc2uat8o5NrlB+UHnuQSEdHxvp3V7zx9HSVsHBgUhhMV5ZMNeCr9F0dqRU8fvek1RBUlsk9FNzVX1lPKA9yg86Fs4B9aYQkWiS4NsapzkY0gmLgfpKj4fTtOBrNJ2d4t1Quqf9cA6onPz0sZC72jt2uUK5g0VXNqK7m+zhOxjSCYsBpIr9exAt+BpNZyfPKuCZkzs+NmMcFO+EujLP2uQKTRaVlulIho6N6O7mZOnUFAMCIhI6PBSA8Fi19XBYRwu+RtPZObwBQqJUyKYj0m1x/LUeNcklKg+BbHLew/fScJHjqC5U4bP2+v20JixGbf1B8IUQ+4UQm4UQG4QQa+zsF0KIF4QQOUKITUKI0UbcV6PROED+eugxAgIc+HdPG6OyYHxx4bbCgbbIJ2JWtW1Hs2xPxJ8E38oUKeVIKWW2nX0zgCzr1xzgFQPvq9Fo2qKpEY5sVoVVjhAapQqwcn1Q8J3JwbdhVrVtFxD89rgQeEcqVgCxQogeXrq3RtN1KdoBlnrHBR9aFWCZ1IOmLVqqbNMdP8esattOLvgSWCyEWCuEmGNnfxqQ2+r7POtzJyGEmCOEWCOEWFNUVGSQeRpNFyV/vdo6I/jp46ChytzWwvYoz7Xm4Ic6fo5Z1bbVzgp+rNrWl3vCmhaMEvyJUsrRqNDN7UKIE9MB7DWEtjtdWEo5V0qZLaXMTkpy4gem0WhOJn+DGsAR19vxc2wFWL4W1ik/4FyGDphTbdtkURXLjmbogHVIivAPD19KmW/dFgLzgXEnHJIHtP5NpQP5Rtxbo9G0gzMLtjbi+0B4PBzysUwdZ4qubJhRbWsTbVuqpSMEBCjR93XBF0JECiGibY+B6cCWEw77DLjWmq0zHqiQUpo4Tl6j6QJYGqBgi3PhHFAFWEkDoWSPZ+xyheZmlZbpyOCT1phRbWsLy9jCNI7ihfYKDiaJtksKMF+oMV5BwP+klIuEELcCSClfBRYCM4EcoBa4wYD7ajSa9ijcpjo2Oiv4AAl9YddXxtvkKrXF0Gxpf3B5W3i72rauXG3D45w7zx8EX0q5Fxhh5/lXWz2WwO3u3kuj0TjB4Q1qmzrS+XMT+kHNf5QA2TJIzMRWPBWd4vy53q62tVUpOxPSsR1ve7PwEEZ4+BqNb5PzDRzeCMGREByuiou6DzXbKs+Tv16JtTMLtjZsVbkleyDNB+okq6yCH9Xd+XOju0PBVmPtaQ93Qjql+4y25ji04Gs6L3Xl8OUDsGne8c8HBMH5z8Ooq00xy2vkr1fhHGEvSa4DWgQ/xzcE35Zl42h/+da0rrYNCDTWLnu46uH7Q0hHo/FJ9i2F+beq2O3pD8CEu1Q8u74CFvwaPr1ddZE881HnMlj8hcZ6KNgGE+5w7fz43oBQgu8L2EI6US6GdGzVtq6EhJylky/aajS+xZHN8O5lKmf75q9VCMdGRDz84kP48rfw4z+gMh9mzXXNC/ZlCrdCcyP0GOna+UGhKgXSVwS/qkClLYZEOH9u62pbbwh+XbkKHzoypKU1YTGq4K3J4njTNSfphK6NpktTXwEfXKu8qxsWHi/2NgKD4Nxnlee/+QPY+aXXzfQ4hdvVtvsw16+R0M93BL+6wDXvHrxfbVtX7nw4B459IvBgT3wt+JrOg5Twya9Ugc7st9uP9woBk3+r8s2/+p0KgXQminZAYCjEZbp+jcQstWgr7RbFexe3BN/L1bb15c6Hc6BVP51yA405Hi34ms7DTy/Aji/grCeg5/iOjw8Mhhl/g7L98NM/PW6eVynapQTbnUXKhH7QUG1OP/kTqS5wPRzj7Wpblz18zzdQ04Kv6RyU7oVvn4BBF8D42xw/r8/pMPhCWPaMas7VWSjaAUkD3LtGQl+19YWwTpUbHr63q23rypwvugIt+BqNw3z9KASGKI/d2QXY6U+q7eKHjbfLDBpqVVgr0V3Bb5WaaSZHq6GxxnXBB+9W27od0tGCr9G0zYGfYPtncNo90M2FMQuxPVX64rZP1CcFf6dkNyDd9/C7pat1ALMFv6XK1oWiKxverLZ1NaRjO8eD1bZa8DX+TXMzfPV71WPlVBdzzgGyb1Sj/da9Y5xtZlG0U23dFfyAABXWMbuJWksOvgtFVzaiUtScWU/T1Kg+jWgPX6PxAJs/VBWl0x51LUfbRrdUyDob1r+r/mn9maKdIAIhvq/710roa76HbwvFuNJWwUZEAtSWeD7jqKVxWqzz54ZEKadDC75GYwfLUfj2j6p9wLDZ7l9vzHVQUwi7Frl/LTMp2qGE2tnCH3skZKn+Lk0W96/lKu5U2dqISICmo9BQY4xNbdHSVsGFRVshPF5tqwVf479snAeVeXDmI8a0R+h3lirSWfu2+9cyk6KdkNjfmGsl9FMVu+UHjLmeK1QXQECwayJqwzZ9qrbEGJvawtW2Cja04Gs0dmhugh+fV60D+kwx5pqBQTDyKtVd019TNC0NauE5aaAx12vdNdMsqgpU/N6dN3VvCb47IR1QbxS68EqjOYHtn0HpHjjtXmP74Iy+BpCw4V3jrulNSveAbPKA4JsYx3enytZGZKLaag9fo/EzpITlzykxGnS+sdeOy1SfGNb9R2UA+RtFO9Q2yaCQTkS8Ei+zBd+dlEzwoofvRgwftOBrNCexZ4kaaDLxbs/0Nx9xpVob8LUh3o5QtAsQarHVCIRQQ83LPDuYo12qC9xLyQT1xgXeC+m4OiXM1wVfCJEhhPhOCLFdCLFVCHG3nWPOEEJUCCE2WL8ecfe+mi7M8udU3v3wyz1z/ayzVFrjLj/solm0A+J6uZeieiKxGeataTRZVB97d1IyAUJj1O/UGyGdkGjX2xv7uuADFuA+KeUgYDxwuxBisJ3jlkkpR1q//mjAfTVdkfwNsH8ZnPor1bPdE0TEQ89T/bNtctFO91sqnEhMBlTkmdM1s6YIkO57+AEBKqxTU2yIWW3iapWtjfBYaKxVi+8ewG3Bl1IellKusz6uArYDae5eV6Oxy6q5arjE6Gs9e58BM6Bwm+qk6S80WVSs3d0K2xOJ7QmWOs+LpT1sLY3djeHDseIrT1JX5vqCLRw710NevqExfCFEJjAKWGln96lCiI1CiC+FEEPaucYcIcQaIcSaoqIiI83T+DvVRaqyduSVrsdIHWXADLXd6UdFWOUHVHGRURk6NmIy1LbioLHXdYQqA4qubEQkQG2p+9dpj/py9zx8D7dXMEzwhRBRwEfAPVLKE0e2rAN6SSlHAP8EPmnrOlLKuVLKbClldlJSklHmaToD695Sc2nHzfH8vRL6quIlf4rjG9VD50RirYJvRhzfiCpbGxHx3lm07eyCL4QIRon9u1LKj0/cL6WslFJWWx8vBIKFEIlG3FvTRWhqhNWvQ9+pxgtaW/Q/B/b/6PHB0oZRvEttEw3K0LHR4uGbKfhuxvBB5eJ7Y9HWrZCOZ6deGZGlI4DXge1SymfbOKa79TiEEOOs9/XwT17Tqdj+uWpvO+4W791zwEzVViDnW+/d0x1KctR0J6PDXeGxaoC4WR5+eJwxC/QRCVBX6tn6iroyNz1867kecjKMGI0+EbgG2CyE2GB97vdATwAp5avApcBtQggLUAdcIaUvDMrU+A0r/wVxvSFruvfumTEOwuNVM7Whs7x3X1cpyTlWGWs0MRnmePhVR9xPybQRkQCyWXnPtrx8I2msB0u9T3v4bgu+lHI50G5tu5TyReBFd++l6aIc3gi5K+DsPxnTJM1RAgKh/9kqPbPJ4nputbcoyTm22Gw0ZuXiVxcaE86B46ttPSH4NpF2p8mbP8TwNRqPsvo1CApXjc28TdZ09Y+cv97793aGunKVs+5RD9+ELJ3qI8akZILn2yu42zgNIDhcdQbVgq/pktSVw6YPYfhl7v0juUrvyWq7f6n37+0Mtm6WRrVUOJHYDCVC9Scm4HkQKT3j4XuqnsDdxmng8Z74WvA1vs3G91TRT/ZN5tw/MhGSB8O+Zebc31Fszc086eGDd+P49RUqJm5ESiZ4wcO3NU6Lde864bFa8DVdEClVKmZaNqSONM+OzEmQu9Jj5e6GULJb9YqJy/TM9WN7qq034/i2GbRGLtqCF0I6bsTwQXn4HhpkrgVf47vsW6qEbOzN5trRe5Lqb+LL3TNLclTTNCPGGtrDDA+/xib4BhVghkRAcITnBN+IkA7okI6mi7L6NeUtDbnYXDt6TQSEatrmqxTneC5+DxCZBIEhUO7FhVubhx9pUAwfPNtPx93WyDa04Gu6HJX5sGMBjLoGgsPMtSUiHroPVZ84fJHmZjXpylPxe1DpsDHp3vXwW0I6BsXwwbPtFerKVBtmd2c0aMHXdDnWvKmKZLJvNNsSReZkyF2limt8jap8FXJK9KDggwrreDOGX1Oo1iXcjYm3xpMefn05hBtQ5Tz+V3D5f92/jh204Gt8D0sDrH1LDSKJ7222NYrek1QnyrzVZltyMp7O0LER6+Vq2+pCFUoystguwoP9dOrKjXlzShoAPU9x/zp28PHSQdeY/erPHLU0OXawEC1lwkKokuEAIQgQAiHU48AAQUCAIMj6FRwYQFCgICQwgJCgAEKDAgkLDiA8OJDwkEAiQoKICgsiKjSQmPBgYsKD6RYeTFxECMGB+j22Q7Z/pry7sb802xIAGpua2RY4mGEEsPiLD/koJpwGSzPNUtIvOYqhqTEMT4+hX3IUwsiB6o5SvFttPRjDL69t4HBdLIOqC5jzxo+kJsaSmRDB2N7xDEn1UKtqI3PwgdoGC0X1YaRUFHLH22s4Z2h3zhnanahQg2TQ3cZpQHOz5L8rD7D+YDnPzh5h+N9TpxT82IhgGpo6jqNJCbLlsWx5rlnKlq2luZmjFkmThKbmZixNksamZizNkgZLc8tXvaWJxqaO2wPFRgSTEBlCSrcwuncLo3tMGGlx4WTERdAzPoK0uHD9prD6NZVe2G+aqWZszqvg7Z/3s2DTYeoam/g0JJPupavJbbqU0KAAmiXMW5VLXeN+AAZ2j+a6CZlcODKViBAv/muV7IGQKOMqUltxuKKOP3yylSU7CrhYWHgmBJrKc/lwfxU1DcqpmjUqjQdmDCSlm8FrLTXGCH5Ts+Svi3bw1k/7mSPr+E1wLTn5JfxmewEPf7KZ84en8scLhxIe4mbsva4Mkge5fHpOYTW/+3gTq/eXMSkrkbrGJsP/jjql4M+9NtuU+zY2NVPX2ETNUQs1Ry1U1VuorLdQUddIeW0DpTUNlFQ3UFx9lILKelbsLaGw6iiW5mNvFEEBgl4JEfRLjqJ/SjSDe3RjcGo3MuIiCAgwwXv0Nke2wMGf4awnvNs3pxXf7Szkn9/uZt3BciJCArlwZCqT+yfR78AMItfNZdGvsltmxjY1S/YWVbNiXyn/W3mQ3328mT8v3M5dZ2Zx/YRMgrzx5l2yW/XvN9AblFIyf/0hHv1sK5Ymya2n92VWfAAsfJXXL0xG9plCUdVR3vppP68t28eirUe4d1p/bp7U2zivtLpIFb25QX1jE3fPW89XWwuYNTqNi6OGw6oP+e72YawrC+PjdYd4b9VBDpTW8sb1Y93z9uvKXfbwP1idy8OfbiE8OJCnLxvBJaPTPPJpsVMKvlkEBwYQHBhAt7Bgh89papYUVNaTW1rLwdJa9hXXkFNYze7Car7eVoDtvaBbWBAje8YxMiOWMb3iyO4VR6RRH0V9idX/hqAwGHW112+9u6CKJxds54ddRfSMj+DR8wdzyZj0Y7/P0DNgzUsqjt/ndAACAwRZKdFkpURz9Sk9WXOgjJe+y+HJBdv5ZMMh/nzxcIale3g6V0kOpI0x7HL1jU3c98FGFmw+zNjMOP5+6QgyEyOhzOrBl+cihCC5Wxj3nzOQK8b25I9fbOOphdvJr6jjkfMGuy9WUioPP9L1HPzSmgZufns163PL+cN5g7nptN6w7QCsAlFbypheQxnTK55T+iRw7/sbuPb1lbx14zin/n+Ps7e+3KUY/toDpfx+/mbG90ng2ctHkBztuay0TqgY/kVggCA1NpzU2HBO6ZNw3L76xiZ2FVSxNb+STXkVrD9YxotLdtMs1SeB4ekxnNo3gdP7JzO6Z6x3vElPUl8Bmz6AoZd6ppthW7dtbOK5r3fx2vJ9RIQE8vC5g7j21ExCgk74eWaMVdvclS2C3xohBGMz43nz+rF8ueUIj362lQtfWs5dZ2Zx19Qsz3xCsxxVufHDrzDkcrUNFua8s5blOcU8cM5A5kzuQ6DN7m5pIAJOWrjtmRDBv68dwxNfbOeNH/dRe7SJP80aduw8V6gvV9PNXAzp1Dc2cd0bq9hZUMVLvxjNzGE91I6Wattj/XQuGJFKSGAAd763jqv+vZIPbjnV+fBOY52y18m2CqU1Ddzxv/Wkxobz8tWjXXuzcQIt+D5MWHAgw9NjGZ4ey5Xj1HPVRy2sP1jGz3tKWLG3hFd/2MtL3+0hOiyIyf2TOHtId6YMSCLaw384HmHdf1R64TjvVdZuyivn1x9sJKewmsuzM7j/nAEkRLUxbCM8DpIGKcFvByEEM4f1YGK/RB77bCv/+GY3W/MreXb2CON/L6X7VPqqARk61Uct3PjmatYcKOXpy0Zw6Zj04w8IDIboHnZTM4UQ/OG8QUSFBvLCkhzqLU08N3uk629ybubgP/rpVjYfquDf12Zz1uBW12ijvcI5Q7vz8lVj+OU7a3hywTaeuniYczd0ocq2uVly7/sbKKlu4ONfTfC42IMWfL8jKjSISVlJTMpSH3Ur6hr5MaeY73cWsmRHEQs2HSYkMICJ/RK4YGQqZw02MAvBkzRZ1JCTnhMgdZTnb9cseXFJDi8s2U1SVChv3ziO0/s7ED7oeQpsma+KnTpYY4gJD+bZ2SMYlhbDUwu3c/HLP/HatdkqPGIUJdYMHTdz8GsbLFz7+ko25lXw/BWjOH9Eqv0DYzKgIs/uLiEEv54+gNDgQP7+1U56xkdw33QXx1G2VNk6H9J5f/VB3l+Tyx1T+h0v9tBK8E8eZn7W4BTmTO7D3KV7OWNA8snntocLjdP+tXQvP+wq4smLhjI0zcNhPyt+oASa9ogJD2bmsB7MHNaDpmbJ+oNlfLX1CAs3H+He9zcSFryZswZ359Ix6ZzWL9G9j9meZOcC1W/97Kc8fquCynrumbeBn/eWcNHIVB6/cCgx4Q56VxnjVY1A0XZIGdLh4UIIbjytNwN7RHP7u+u45JWfePvGccb9g9tSMuP7unwJS1Mzt7+7jg255bx81WjOGdqj7YNj0jrsKfSrM/pysKSWfy7JISslmgvaevNoj5Y+Os6FdDbnVfCHT7cyKSuRe8/qf/IB4dZQYRu5+PdN78/y3cU88NEmRqRPItnRzKOWtgqxDh1+oKSG577Zxcxh3bnqlJ6O3cMAjBpifo4QYqcQIkcI8aCd/UII8YJ1/yYhxGgj7qs5nsAAQXZmPA+dO5hl90/h/249lcvGZLBsdxHXvbGK0/66hGcW7ySvrNZsU0/m55chthcMPNejt1m6q4iZzy9jQ245T182gn9cMcpxsQc19hA6DOucyIS+iXx02wTCggO5cu4KVu41qPinJEeFWcK6uXS6lJKH5m/hu51FPHHR0PbFHlQcvzK/3bmwQgieuGgoYzPj+O2HG9mUV+68YS6EdGqOWrj9f+tIjAzh+StG2XduAoOUKLch+KFBgbxw5UhqGyzc9+FGmps7TrUGWk27iu3wUCklj322lZDAAB49f4hXazeMGGIeCLwEzAAGA1cKIU7MpZoBZFm/5gCvuHtfTfsEWMX/iYuGsvL3Z/LyVaMZ0D2al77LYfLfvuOmt1bz3Y5Cx/+gPcmhtWqE4Sm3ut+HpA2amiXPfr2L695cRWJUKJ/fOfHkGLUjxPdRYYaDzgk+QJ+kKD689VSSu4Vy7Rur+HZ7gfP3P5HiXW7F75//djfvr8nlzqn9uOqUXh2fEJOuFidr2x8iEhIUwCtXjyExKpQ576yluPqoc4ZVF0JAkFMx8ScXbCO3rJbnrxxFfGQ7XUMjEtodgtIvOVo5TbuLeX+Ng5XFTrRG/npbAd/tLOKeaVnG1y50gBEe/jggR0q5V0rZAMwDLjzhmAuBd6RiBRArhOjAlXCDynyoOHTsqzJfDUOuLoSaEvXLOVqtSvi7wCz10KBAZg7rwVs3jGPZA1P51Rn92JhXwQ1vrWbqM9/z5o/7qKpvNM/AFa9ASLTHUjFLqo9y3RureOHb3cwalc4nt0+kX3K0axcTAjJOUW9QLpAaG86Ht05gQPdobv3vWhZtOeKaHaD+dot3Q6Kd0IUDfLAml398s5tLRqfza3vhD3t0S1PbNuL4rUmMCmXutWMoq23g7nnraXLGuahxrq3Ct9sLeG9VLrdM7svYzA4yvBzop3P1KT05pXc8f1q4ncJKB/onObhoW9fQxOOfb2NAiirS8zZGxPDTgNZvg3nAiY0g7B2TBhw24P4n888xKtvDEUSAyvsOjoDQaPUVFqP+KCITVWvWmDT1hx7bU30F+mEGjJW02HB+c/YA7jozi0Vbj/Dmj/t4/PNtPLN4F5ePzeCGiZmkx0V4z6DKfNg6H8bNcTks0R4r95Zw17z1lNU28tdLhjE7O8P9j9A9x8OOL6CqAKKdzyKJjwzhvzefwnVvrOKO/63jhStHHUsbdIaaYiU0ic63VPgxp5jff7yZ0/ol8pdLhjn+M4mxCn7lIUjrODI7JDWGP144hAc+2szz3+52/I3FibYKJdVHeeCjTQzsHs29Zznws4hM7LAJnBCCP88axjnPL+Oxz7fy8lUd1DnUlQECQtv/G375+xwOldfx/pzxplTUGyH49v5STnwrd+QYdaAQc1BhH3r2dHExY+bT0Gw5dhsp1ba5ydo7wQLNjeqjaWO9GqPWUAMN1XC0Sn0CKNiqPrbaVt9tBASpsv+ELNUyt/tw6DFCvRGY0UfFRUKCArhgRCoXjEhlQ245byzfx1s/7eetn/YzY2h3bpnc1/MFQwA/v6TSCsfNMfSyzc2Sl7/P4dmvd9ErIZI3rh9rXM+XDKs/k7sSBl/g0iW6hQXzzo3juPGt1dz53noam5q5cGSacxcp3qW2Tgr+ziNV3PqftfRJiuTlq0c7JzzdrGGwikMOnzI7O4NV+8r455LdjOkV51g2VHWhQ33wpZT87uPNVNZZ+O/NpxAa5EBIMCIe8jd0eFifpCjumtqPpxfv4uttBe1n7dSVK0exnU8kOYXVvPrDHi4amXpSzY23MELw84CMVt+nA/kuHAOAlHIuMBcgOzvbtXjLqKtcOs0ulqNQdVj9gZcfUH1LSnKgaCfs/kqJFaiFs57j1bCMvlNVqbufMDIjlheuHMWDMwby1k/7eW/lQb7YdJhT+yQw5/Q+nNE/yTMLS7Wlqg3y0EsN7Yp5pKKe33y4keU5xVwwIpU/zRpmbGpqjxEQGOqW4ANEhwXz1g3juOnt1dzz/gaONjYze2xGxyfaaEnJdDykc7iijhvfWk1YSCBv3uBCVWlkonrtlR2HdGwIIXjyoqFsOVTBPfPW8/mdp3X8KbKmyKEsqP+uOMDibQU8NHMQA7s7+AkxIkE5c1J26KTNmdyXzzce5pFPtzC+T3zbdRT15e0u2KrF8c1EhATx8HnutYtwByP+C1YDWUKI3sAh4ArgFycc8xlwhxBiHircUyGl9Ew4x2iCQpVHH5cJTDx+X2MdFGyD/HVwcIXqAbN1vtoX1xuypsOg86HXBI8tRhpJamw4v585iDun9uO9VQd5Y/l+bnhzNQNSovnl5D6qIvHE6lN3WPEKNNbApF8bdskvNuXz0PwtNFia+fOsYVwx1oAQzokEhapwhpOZOvaIDA3izevHcct/13L/R5uobbBw/UQH3/yKd0NQ+DGvu6PDq49y1WsrqahrZN6c8aTFhjtvsBDQLdUpDx8gPCSQV64ezYUv/cgv31nLR7ed2nZjMCkdCulsza/giQXbOWNAkmqb4CgRCerTfUMNhEa1e2hIUAB/uWQYl7zyE499to1nZo+wf2AHfXT+b20eK/eV8pdZw0hsq7DPC7j93yultAB3AF8B24EPpJRbhRC3CiFutR62ENgL5AD/Bn7l7n19guBwSB8D434Jl74O926Fu9arkFJif1j3Drx9Hjw7GBbeD/nrzbbYIaLDgpkzuS9L75/C05eNQAj4zYcbmfS3Jbzy/R4qag1Y4K2vhFX/goHnudVh0EZx9VHunreeO/63nt6JkSy8exJXjuvpuZS3jFNUWKCxzu1LhYcE8u9rxzB9cAqPfb6NF5fsbune2i7Fu1WGjgMLmxW1jVzz+iryy+t44/qx7tUBxKSrGL6T9EmK4oUrR7HjSCW//XBT26+xrkyFXNsJ6dQctXDn/9YTFxHMM5eNcK6it4Nc/BMZ1TOOO6b046N1eSzY1Iaf2k4fndKaBv60cDvZveKYne3EJzgPYIi7JqVcKKXsL6XsK6V8yvrcq1LKV62PpZTyduv+YVLKNUbc1+cQQqXtjfslXPUB3L8HLn0D0rNVsc7cM+DVSar9b32l2dZ2SEhQAJeOSefLuyfx9o3j6JccxV8X7WD8n7/lkU+3sLeo2vWLr35N9c6Z/Bu3bGxulry36iBnPvMDCzcf5p5pWfzfrafS28hqVntknKJEyaA38dCgQF66ajQXjUzl6cW7eOCjTTRY2s51B1QM34EK24q6Rm54axV7Cqv51zXZjOvtZp+ibmlOe/g2pgxI5nczBrJg82FeXJJj/6Dq9ouubOGR/SU1/OPyUW23wmiLNtortMedZ2YxIiOW38/fzOEKO2/ydeV2QzpSSp74YhtV9Rb+NGuY6R1v/bzblo8TEglDL4Er3oXf7FKePxIW3AfPDYHFDzuU3mY2QghO75/EuzePZ+Fdkzh3eA/mrcpl6jM/cM3rK/lq6xEsTR2IU2saatVibd8z3WqjsGJvCZe8+hO/+3gzA7tH8+Xdk7hnWn/vNJGzFWAZOAErODCAZ2eP5K6p/fhgTR7XvbGq7U9TlqNqTamD+H1uaS2XvPITm/IqeOHKUY4tmHZETJpa12p2cMjQCfxyUh9mjUrjma938e7KAycf0E6VrZTS2ok0n3um9efUvi4sftoEv+7k9gptERwYwPOXj6SxqZn7PrBTkFVXZjek89ZP+5m//hC/mtKP/ikupgIbiBZ8bxEeqzz/W5bBzUvUcI+fX4bnR8D829RisB8wOLUbT182gh8fnMp9Z/Unp7CaW/6zlol/XcJfF+0gp9ABr3/NG2rRbPJvXbJh/cEyrnl9JVfMXUF+eR1PXzaCeXPGu55b7wqRierTXO4qQy8bEKD60Tw7ewRrD5Rx3ovL+HmPHU+0dK+1aVrbGTprD5Ry0Us/UlR1lHduGsc5Qw0akNItDWSTqm1xASEEf5o1jKkDk3lo/hbmLj3hb7+lj87Jgv/04p28vnwf10/I5M6pLhac2Tqx2umn0x6ZiZE8ev5gftpTwkOfbD5WV9DSGjn2uOO/3V7AE19sY/rgFO4503PTyJxB99LxNkKouP9lb6q2tj+/DGvfhE3vw4grlAj6yhzXdkiKDuXOM7O47Yy+fLujkA9W5zJ36V5e+X4Pw9Nj1Pi4Id3pk3TColhdOSx7GvpMgV6nOny/uoYmPt+Uz7srD7Ixt5z4yBAePncQV4/vRViwSQvi6WNh7/cOZXs4y6zR6fRKiOTXH2zgyn+v4JrxvXhwxsBjMxDaScmsbbDwrx/28soPe+gRE8Yb14+l74m/B3eIsS4SVx46lpfvJGHBgbx69Rju/WADf1q4g+qjTdw7LUutudgJ6TQ3S57/djcvfbeHK8dl8Oj5bvTcdyGkY2N2dgYHS2t56bs91DY08cxlIwiy1KpU71Ye/rb8Su58bz2DU7vxjyvc6BpqMFrwzSS2J8z4C5x2L/z4D+X5bvpA5aRP/o1Xe8K7SlBgAGcP6c7ZQ7pTWFXPp+vz+WJTPn9btJO/LdpJn6RITu2TwPg+CYzNjCdl1XOIunI464/tXldKyZHKepbuKuL7nUUs311M1VELWclRPHr+YC7LzjC/C2j6WPVGXZGrfpcGM6ZXHIvunszTi3fyxo9qqtQVYzO4fGwG6S1zbI95ufWNTSzYdJi/f7WTI5X1nDu8B09cOLT9NgOu0Lra1hbacoGQoABeuGIUkSGBvPDtbr7fWcivz+rP6dWFiIDglkXQn/eU8OSCbWzNr2TWqDSeusiJQjF7hMWogksnPXxQn05+e/ZAIkKC+PtXO6lraOIv0+KJBwiPpa6hiXdXHuDl7/fQLSyY168b691xlx0gHMoGMIns7Gy5Zk3nXN+1S+Vh+O4pWP9fVXV6+oMqDOSHlb355XV8tfUIS3cVsXp/GdVHLaRSzHdh9/Fz2CQ+6/MoseEhxIQHExESSH1jEzUNTVTUNbCnqIbdBVWUWePX3buFccaAJGaNTmdsZpw5g8LtcXgj/GsyXPI6DLvUo7dae0ANv/l+VxEAb8W8znDLJl4d/TkIWHegjI25FTQ0NTMiPYY/nDeY7I5aDLhKXRn8NROmPwkT7nT7cs3Nkv9bl8fz3+zmUHkdr8e+yRjLBv7Q90OKqupZsbeUtNhw7j9nAOcPTzXGW/5rbxhyMZz3rMuXsFWpDxQHWRT6IO/2epLnDg2iuPooE/om8PgFQ8gyIW4vhFgrpbQ759V33no00K0HXPgijL9NLeh+9TtY/x+Y+XfIPM1s65wiNTacGyb25oaJvbE0NbM1v5LIL+8k8DD8X7frWLenhIq6xpZB2KA8vujQIPokRXLO0O5kJUczoV8CA1KifUfkW5M8RLXkyFvtccEf0yuON28YR15ZLe+vzqXHqlxymnvw5k/7aWqWDE2L4fqJmWoCWlaSZ0MIYbEQHOlyps6JBAQIZmdncNHINN5fk0v0N6Ucae7G1kMVhAQF8NuzB3DTab2NDd050E+nI26Y2JtxvePZtXIRbISv9tbTv2cUL/1ilGmVtB2hBd8XSRkCV38MOxfClw/CW+fCsNlw9p8gyoAsCy8TFBjAiOBcyP8cJtzBi9MvatlnsQ5+DwsONKW3iFsEBkHqaEMzdToiPS6C+87qD2uPwKjL2TnzHCzN0rs/OyFU7N6JaltHCAkK4JrxvWCjBaL6suSqMwy9/nFEJDiVpdMWQ1JjGDIwAjbC27edhUgd6b5tHsTP/sO6EEKo3vC3r1QLuVvnw0tjYcP//K/DZ3MzLPiNip2ednxVbVBgANFhwf4n9jYyxsLhTaonk7eoLoCjlZDYHyGEOT87N3LxO6S6yKE+Om4REe9SDN8u1tbIwoUB5t7GT//LuhAhETD1Ybh1ucq5/uQ2+M9FUGYnf9lXWfO6aid89p/8YiHaKdLHqgKswxu8d0/bgq2bYw3dIibNpWrbDmluVnn4Lg4vd5iIeLdDOi04MfzEbLTg+wvJA+GGRXDuM5C3Bl6ZoKpV25k85BOU58I3j6k0zJEntljqBKQbX4DVIS0pma71wTeEbukqfdLSYOx168tViqPHBd8awzfi03Jducr6CTG/sKojtOD7EwEBMPZm+NXPyrNccB+8cwGU7TfbMvtICQt+rQqEzv+HX7WPdpioJNVYz+ACrHYp2qkWTaNdmBVrFDFpgIQqu01vXafaOgXMheHlTtG6gZq71JerhWwHh7WYie9bqDmZ2J5wzXy44J+qgdfLE2D1674X29/0AexeDFP/YO022klJH6s8fG/9/Au3qYZzZgpMSy6+wWGdKmtzsm4efjNzsoFau7TRR8cX0YLvrwgBo69V3n7GWOVJ/+ciVb3rCxRshS/ugYzxcMotZlvjWdLHKaHyREz7RKRUP9sU83qqA8dX2xqJrV1DtEFtINrCjWrbk2ijj44vogXf34nNgGs+gXOfVbH9lyeooSJmevt1ZTDvKjXubfbbfjELwC0yxqqtAf3xO6S6QKUTJnc8HMSjODHb1ilsHn6UlwTfgNTMjoaf+BJa8DsDQsDYm+C2nyBtlPKs/3OxObH95ib46GYlBLPf8byn5gukDFUFWN6I4xdstd7TZA8/NEql2XrCww+LUdlpnsTFBmp26WD4iS+hBb8zEdcLrv3M6u2vhpdPVW2IXWxj6zRSqgrhnG9g5t+g54mz7DspgcGQNkZNPfM0hdvU1mwPH1SmjtEx/Mp87yxGGxnS0R6+xjRs3v7tK6H3ZPjq9/DamZC31rP3bW6Ghb+BFS/DuFtgzA2evZ+v0XM8HNkMR90YCuMIBdsgKgUifaB03wPVtlQd8c6nQjcaqB2HlNZFW98vugIt+J2XmHS4ch5c+qbyml6bCp/cfqz1rJE0WeDT21VdwIS7YMZfO2cKZntknKJ6xB/y8Btr4VZINjmcYyMmQ9VZGEnVEYjuYew17REQqMIw7nr4DdXq994VQjpCiL8LIXYIITYJIeYLIWLbOG6/EGKzEGKDEKILtb80GSFg6Cy4c60S4k3vwz/HwHd/bikHd5vKw/DeFbDxf3DG71Xb464m9qBSMxGeXbhtblI5+Ck+EM4BlTBQXw5Hq4y5XnMzVHvJwwdDGqhRV6a2XSSk8zUwVEo5HNgF/K6dY6dIKUe21bZT40FCo2H6EyqFs/dk+OEv8Pxw+OFvUFPs2jWlhA3vwcunwP5lanzjGQ90TbEH9Q+fPMizgl+6Fyz1vuXhg3Fefm2JqrL1hocPBgl+udp2BQ9fSrlYSmmxfrsCSHffJI3HSMxS83VvWQq9TlO9958ZCB9eD3uWQFMb81NbYzkKWz5SHTw/uRWSBsGtP6q+/V2djFMgd7Xn2l3YMnSSB3nm+s5iG/pSYZDg26p2u3lR8G0euqv4UR8dMLY98o3A+23sk8BiIYQE/iWlnNvWRYQQc4A5AD17Gj9FSAP0GAFX/g8Kt8Pat2HTPNWNMzhSZdb0mgCxmeqPOCxGfQooyYHinbBjocpdjukJM/6mWj109jx7R+k5Xo2rLNrumbBL4TZAQNJA46/tCi0evkHFfi1FV94S/DjIX+feNfzMw+9Q8IUQ3wD2gmoPSSk/tR7zEGAB3m3jMhOllPlCiGTgayHEDinlUnsHWt8M5oKaeOXAa9C4SvIgNWJx2mOw+yvYtwwO/AhLnrR/fHg89J4Eo69TzdD8oHeIV7GN+zu4wjOCX7BVDU73dI66o0SlQGCIgR6+tejK2zF8d2YSt3j4/pGl06HgSymntbdfCHEdcB5wpmxjXqKUMt+6LRRCzAfGAXYFX2MCwWEw+EL1BepjbnWh8l7qK9Q/RkIfv/mjNo243qqPe+5KlRprNIXbzS+4ak1AgKq4NSqGX3UEEOqNxBu0bqAW6uKQd5uH3xVCOkKIc4AHgNOllLVtHBMJBEgpq6yPpwPtT7DWmEt4nBZ3VxBChcQ8UYDVUKsWbYddZvy13SE2w1gPPzLJezOcWzdQc1nwy0AEQoiL53sZdz+TvwhEo8I0G4QQrwIIIVKFEAutx6QAy4UQG4FVwAIp5SI376vR+CYZ46H8wLF4tFEU7QCkb3n4oNZyjPLwKw97txWHEdW2tipbP8lOc8vDl1LaHbljDeHMtD7eC4xw5z4ajd/Qc7zaHlwBQy4y7rq+1FKhNbEZKnfechSCQt27VtVh7y3YQivBd6Pa1o/66ICutNVojKX7cJXttH+5sdct2ApBYRDf29jruostU8eIrpneaqtgw9ZAzZ2OmX7URwe04Gs0xhIUAr1OhX0G5yTkrYbUUb6XAhtrE3w3wzpNjVBTZJKH70ZIx4/66IAWfI3GeHqfrmoWKg8bc73GejXZLH2sMdczEqOqbasLAOm9oito1UDNHcH3n+EnoAVfozGe3pPVdv8yY653eCM0Nx7L8/cluqUBwn0P39tFV6A+LYXHuRfD1yEdjaaL03248vr2/mDM9fKsg1XSfVDwg0KUSLvr4Xu76MpGeLzrHn5zs6pT0R6+RtOFCQhQFcn7fjBm1GTuSojtBdFeKkhyFiNy8c3w8MG9BmoNVSCbtYev0XR5ep+uRLBsn3vXkVI1ZPPFcI6NmAz3++lUHYaAIIhINMYmR3GngVpLa2S9aKvRdG36nKG27oZ1KnJVnnuGD4+LjM1Qs23dGaVZeVgNLvd2f6aIONc9/BrreZFJxtnjYbTgazSeIKGfCk+4m55pG4zuixk6NmIyVB97d6qLq7xcZWujdQM1Z6kpsl7Dy59K3EALvkbjCYRQYZ19S93rj5+7CoIjIGWocbYZjRF98b1ddGUjIlE1UDta6fy5NsGP1IKv0Wh6T4baYtUf31XyVkHaGAg0cnSFwRiRi+/ttgo2bJ05q4ucP1cLvkajaaHP6Wq75zvXzm+ohSObfTucA62qbV1cuG2sU/nsZnj4UclqW13g/Lm1JaqNRkiksTZ5EC34Go2niEmHlGGw/TPXzs9fr2LjvpyhA0rwwuNd9/BtOfjdUo2zyVFaPHwXBL+mCCITjLXHw2jB12g8yZCLVB69K83F8vxgwdaGO7n4ldZZtqZ6+IXOn1tT5FcZOqAFX6PxLEMuVtttnzp/7t4fICHLP2LEsW70xS+11irEmdAJNDxeDTCp0YKv0WjcJaGvarWwdb5z59UUqwyfQed7xi6jictUg19cycUv26eKrmyLv94kIEB5+S6FdEr84824FVrwNRpPM+Ri1d7YmWrUbZ+CbIKhszxnl5EkZIGl3rWwTule1TrCrEykqGTnQzpSKg/fj3LwQQu+RuN5bJOvnAnrbJ2vRNSX8+9bk5iltsU5zp9butfcwS6RLgh+fYXqYNqVQjpCiMeEEIes82w3CCFmtnHcOUKInUKIHCHEg+7cU6PxO+L7QI+RsOVjx46vOqImZg2d5TezUknsr7bFu5w7T0oVw4/vY7xNjhKV4rzg1xSrbVcSfCvPSSlHWr8WnrhTCBEIvATMAAYDVwohfGwSs0bjYYZcDPnroGx/x8du+xSQMMRPwjmgWhSExULJbufOqy1VVa6mCn6yWrR1piLaD4uuwDshnXFAjpRyr5SyAZgHXOiF+2o0voMtrLP5w46P3fIxJA+G5IEeNclQhFBefrGTgl+6V23N9vCbLc51zay1efhdT/DvEEJsEkK8IYSw1yc0DWi9kpNnfc4uQog5Qog1Qog1RUUulDtrNL5IXCb0nQo/vdj+hKWKPMhd4V/evY3ELNcF34yUTBtR1rCMM6mZLR5+JwvpCCG+EUJssfN1IfAK0BcYCRwGnrF3CTvPtdmaTko5V0qZLaXMTkryrx+mRtMu059U4YulT7d9zNZP1NZfsnNak5ilWjnXO9GIrHQvICCul8fM6hBXqm1tMXw/y9LpMA9KSjnNkQsJIf4NfGFnVx7QOsE2Hch3yDqNpjORMgRGXQ2r5sLYm1SOfmtqimHFy5A66uR9/kCCNVOnZLdq+OYIpXtV/n1QqOfs6ogWwXfSww+LUSMe/Qh3s3Rat7e7GNhi57DVQJYQorcQIgS4AnCxuYhG4+dMeRgCQ+CbR49/vrkJPrpJif75z5tjm7u4kppZts/clExwrYFaTbHfeffgfgz/b0KIzUKITcAU4F4AIUSqEGIhgJTSAtwBfAVsBz6QUm51874ajX8SnQKn3QvbP4edXx4bvPHdn2Dv93DuM9BjhKkmukxcb9WmwJnUzNK95i7YAoR2g8BQ5z18P4vfgwMhnfaQUl7TxvP5wMxW3y8ETkrZ1Gi6JKfeDuvfgfeuUCKZORHW/xdGXwuj7f5L+QdBIWpx2tHUzLpy1WLYbA9fCOdz8WuK/TLspittNRpvExIBc36A819Q3u3Geaowa8bfzbbMfZxJzbQNeDfbwwfn++l0RQ9fo9G4SEQ8jLlOfdVXqLh+cJjZVrlPYj/Ys0StSQQEtn9sqS8JfopjRXGgXlttiV8KvvbwNRqzCYuB4HCzrTCGxP7QdNSxRnEtOfiZHjXJIaKSHM/DrysDpN8VXYEWfI1GYyQtqZkOZOqU7oOo7r4xIjAqRcXlmywdH+unbRVAC75GozGSltRMBzJ1fCFDx0ZUMiCPtUxoDz+tsgUt+BqNxkgiEiA8zrGF2zKTu2S2xplqWy34Go1Gg0pxTMjqOKTTUKOGl5udkmkj0lZ85UD/rpoSte2ChVcajUZzPIn9oWhn+8fYMmJ8xsN3otq2pggQKtPKz9CCr9FojKXHCJXxYku7tEfhdrX1leIlZwU/IqHjtFMfRAu+RqMxlr5T1Xbvd20fs/d7lY7qKyMcQyIhJPpYfL49/LToCrTgazQao0noqzpg7llif7+UsOc76HOGb3nJUUkOevjFfpmSCVrwNRqN0QgBfafA3qX289qLd0NlHvSZ4n3b2sPRfjq1WvA1Go3mGH2mwNEKyF9/8j6b59/X1wTfwX46OqSj0Wg0rehzBiDsh3X2LIH4vr7RUqE1jnj4lgbV+0gLvkaj0ViJiIfUkScv3FqOwv5lxxZ2fYnIZKgvVza2ha0SNyLBKyYZjRZ8jUbjGfpOhdxVx8+4zV0FjbW+KfjdUtW2Iq/tY2whH1sap5+hBV+j0XiGPlNANsH+5cee27MEAoIg8zTz7GqLxP5q217RmG18Y7yP1A84iRZ8jUbjGTLGQXDk8XH8PUsgfRyEdTPPrrZIsgp+cXuCvxNEgO8UjDmJWwNQhBDvAwOs38YC5VLKkXaO2w9UAU2ARUqZ7c59NRqNHxAUqjz5XV/BgBkq7n14I0z5vdmW2ScsRrVrLmqn02fxLrXYHBTqNbOMxN2ZtpfbHgshngEq2jl8ipTSgd6jGo2m0zDofNj9Ffx31rHnfDF+byOpf/seftEuSBzQ9n4fx5ARh0IIAcwGfPg3qdFovM7oa6DfNNX7vmwfNDVC2hizrWqbxAFqxrCUqoCsNU0W1QW0/3RzbDMAo2baTgIKpJRtNcGWwGIhhAT+JaWc29aFhBBzgDkAPXv2NMg8jUZjGt16qK/MiWZb0jFJA6ChSrVutmXt2Cg/AM2NxxZ3/ZAOBV8I8Q3Q3c6uh6SUn1ofXwm8185lJkop84UQycDXQogdUsql9g60vhnMBcjOzpYd2afRaDSG0TpT50TBt2XvdOaQjpRyWnv7hRBBwCygzc9pUsp867ZQCDEfGAfYFXyNRqMxjSSrmBfvOrn1gy22bxvj6IcYkZY5DdghpbRbrSCEiBRCRNseA9OBLQbcV6PRaIwlKgVCY+zn4hfvVvvDY71ullEYIfhXcEI4RwiRKoRYaP02BVguhNgIrAIWSCkXGXBfjUajMRYhrJk6dlIzi3b6dfweDFi0lVJeb+e5fGCm9fFeYIS799FoNBqvkDgAdi8+/jkp1ZvA8Nnm2GQQutJWo9FoWpPUX41orCs79lx1ARyt9HsPXwu+RqPRtMaWhVPcKsu8JUNHC75Go9F0HpLsNFGzxfST/DclE7TgazQazfHE9oLA0ONbLBTvUkPOo3uYZ5cBaMHXaDSa1gQEqlz71k3Uinaq505st+BnaMHXaDSaE0nsf7KH7+fhHNCCr9FoNCeTNADKDsCRzVBTonrr+PmCLRjXPE2j0Wg6D6mjAAmvngZYwzha8DUajaYT0v9suO1nKNwGRTugphj6nG62VW6jBV+j0WjskTJYfXUidAxfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLIKSUZtvQJkKIIuCAi6cnAsUGmuMP6Nfc+elqrxf0a3aWXlLKJHs7fFrw3UEIsUZKmW22Hd5Ev+bOT1d7vaBfs5HokI5Go9F0EbTgazQaTRehMwv+XLMNMAH9mjs/Xe31gn7NhtFpY/gajUajOZ7O7OFrNBqNphVa8DUajaaL0OkEXwhxjhBipxAiRwjxoNn2eAMhxBtCiEIhxBazbfEGQogMIcR3QojtQoitQoi7zbbJ0wghwoQQq4QQG62v+XGzbfIWQohAIcR6IcQXZtviDYQQ+4UQm4UQG4QQawy9dmeK4QshAoFdwFlAHrAauFJKuc1UwzyMEGIyUA28I6UcarY9nkYI0QPoIaVcJ4SIBtYCF3Xm37MQQgCRUspqIUQwsBy4W0q5wmTTPI4Q4tdANtBNSnme2fZ4GiHEfiBbSml4sVln8/DHATlSyr1SygZgHnChyTZ5HCnlUqDUbDu8hZTysJRynfVxFbAdSDPXKs8iFdXWb4OtX53HW2sDIUQ6cC7wmtm2dAY6m+CnAbmtvs+jkwtBV0cIkQmMAlaabIrHsYY2NgCFwNdSyk7/moF/APcDzSbb4U0ksFgIsVYIMcfIC3c2wRd2nuv0XlBXRQgRBXwE3COlrDTbHk8jpWySUo4E0oFxQohOHb4TQpwHFEop15pti5eZKKUcDcwAbreGbA2hswl+HpDR6vt0IN8kWzQexBrH/gh4V0r5sdn2eBMpZTnwPXCOuZZ4nInABdaY9jxgqhDiv+aa5HmklPnWbSEwHxWqNoTOJvirgSwhRG8hRAhwBfCZyTZpDMa6gPk6sF1K+azZ9ngDIUSSECLW+jgcmAbsMNUoDyOl/J2UMl1KmYn6X14ipbzaZLM8ihAi0pqIgBAiEpgOGJZ916kEX0ppAe4AvkIt5H0gpdxqrlWeRwjxHvAzMEAIkSeEuMlsmzzMROAalMe3wfo102yjPEwP4DshxCaUY/O1lLJLpCl2MVKA5UKIjcAqYIGUcpFRF+9UaZkajUajaZtO5eFrNBqNpm204Gs0Gk0XQQu+RqPRdBG04Gs0Gk0XQQu+RqPRdBG04Gs0Gk0XQQu+RqPRdBH+H2CPrSMV0KsfAAAAAElFTkSuQmCC\n", 27 | "text/plain": [ 28 | "
" 29 | ] 30 | }, 31 | "metadata": { 32 | "needs_background": "light" 33 | }, 34 | "output_type": "display_data" 35 | } 36 | ], 37 | "source": [ 38 | "import torch\n", 39 | "from matplotlib import pyplot as plt\n", 40 | "\n", 41 | "x = torch.linspace(0, 5, 100,\n", 42 | " requires_grad=True)\n", 43 | "y = (x**2).cos()\n", 44 | "s = y.sum()\n", 45 | "[dydx] = torch.autograd.grad(s, [x])\n", 46 | "\n", 47 | "plt.plot(x.detach(), y.detach(), label='y')\n", 48 | "plt.plot(x.detach(), dydx, label='dy/dx')\n", 49 | "plt.legend()\n", 50 | "plt.show()" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "(Note that in the example above, because the components of the vector space are independent of each other, we happen to have `dy[j] / dx[i] == 0` when `j != i`, so that `d(y.sum())/dx[i] = dy[i]/dx[i]`. That means computing a single gradient vector of the sum `s` is equiavlent to computing elementwise derivatives `dy/dx`.)\n", 58 | "\n", 59 | "**Detaching tensors from the computation history.** Every tensor that depends on `x` will be `requires_grad=True` and connected to the complete computation history. But if you were to convert a tensor to a regular python number, pytorch would not be able to see the calculations and would not be able to compute gradients on it.\n", 60 | "\n", 61 | "To avoid programming mistakes where some computation invisibly goes through a non-pytorch number that cannot be tracked, pytorch disables requires-grad tensors from being converted to untrackable numbers. You need to explicitly call `x.detach()` or `y.detach()` first, to explicitly say that you want an untracked reference, before plotting the data or using it as non-pytorch numbers." 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "### Exercise\n", 69 | "\n", 70 | "Plot the polynomial y=x3-6x2+8x and its derivative, instead of cos(x2)." 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 69, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "# TODO: set y to the given polynomial of x\n", 80 | "y = 'TODO'\n", 81 | "\n", 82 | "# TODO: use autograd to compute the derivative\n", 83 | "\n", 84 | "# TODO: plot the results.\n" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": {}, 90 | "source": [ 91 | "Backprop and In-place gradients\n", 92 | "-------------------------------\n", 93 | "\n", 94 | "In a typical neural network we will not just be getting gradients with regard to one input like `x` above, but with regard to a list of dozens or hundreds of tensor parameters that have all been marked with `requires_grad=True`. It can be inconvenient to keep track of which gradient outputs go with which original tensor input. But since the gradients have exactly the same shape as the inputs, it is natural to store computed gradients in-place on the tensors themselves.\n", 95 | "\n", 96 | "**Using `backward()` to add `.grad` attributes.** To simplify this common operation, pytorch provides the `y.backward()` method, which computes the gradients of y with respect to every tracked dependency, and stores the results in the field `x.grad` for every original input vector `x` that was marked as `requires_grad=True`." 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 70, 102 | "metadata": {}, 103 | "outputs": [ 104 | { 105 | "name": "stdout", 106 | "output_type": "stream", 107 | "text": [ 108 | "tensor([-0.0000e+00, -2.5765e-04, -2.0612e-03, -6.9560e-03, -1.6485e-02,\n", 109 | " -3.2185e-02, -5.5575e-02, -8.8145e-02, -1.3133e-01, -1.8650e-01,\n", 110 | " -2.5487e-01, -3.3752e-01, -4.3528e-01, -5.4869e-01, -6.7791e-01,\n", 111 | " -8.2262e-01, -9.8193e-01, -1.1543e+00, -1.3373e+00, -1.5279e+00,\n", 112 | " -1.7218e+00, -1.9138e+00, -2.0978e+00, -2.2665e+00, -2.4118e+00,\n", 113 | " -2.5246e+00, -2.5954e+00, -2.6144e+00, -2.5720e+00, -2.4592e+00,\n", 114 | " -2.2684e+00, -1.9940e+00, -1.6330e+00, -1.1861e+00, -6.5843e-01,\n", 115 | " -5.9786e-02, 5.9438e-01, 1.2829e+00, 1.9791e+00, 2.6508e+00,\n", 116 | " 3.2620e+00, 3.7737e+00, 4.1467e+00, 4.3434e+00, 4.3315e+00,\n", 117 | " 4.0872e+00, 3.5983e+00, 2.8676e+00, 1.9159e+00, 7.8273e-01,\n", 118 | " -4.7262e-01, -1.7729e+00, -3.0265e+00, -4.1327e+00, -4.9894e+00,\n", 119 | " -5.5028e+00, -5.5970e+00, -5.2252e+00, -4.3782e+00, -3.0925e+00,\n", 120 | " -1.4526e+00, 4.1007e-01, 2.3249e+00, 4.0956e+00, 5.5192e+00,\n", 121 | " 6.4094e+00, 6.6222e+00, 6.0798e+00, 4.7897e+00, 2.8560e+00,\n", 122 | " 4.7794e-01, -2.0646e+00, -4.4405e+00, -6.3087e+00, -7.3680e+00,\n", 123 | " -7.4080e+00, -6.3531e+00, -4.2917e+00, -1.4813e+00, 1.6739e+00,\n", 124 | " 4.6748e+00, 7.0040e+00, 8.2157e+00, 8.0255e+00, 6.3823e+00,\n", 125 | " 3.5034e+00, -1.3781e-01, -3.8789e+00, -6.9824e+00, -8.7814e+00,\n", 126 | " -8.8286e+00, -7.0156e+00, -3.6318e+00, 6.6059e-01, 4.9416e+00,\n", 127 | " 8.2239e+00, 9.6828e+00, 8.8724e+00, 5.8738e+00, 1.3235e+00])\n" 128 | ] 129 | }, 130 | { 131 | "data": { 132 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABJbklEQVR4nO2dd3iUVdqH75NeSU8gBUIJvRMQQVAQUbCjoq69LOra1111V9ey6lbL6tqWte+6on6KDURUVECl906oCYH0XkgmOd8fZyYEmCRT3pl3Jjn3deV6J/O2Z1J+88xzniKklGg0Go2m8xNgtgEajUaj8Q5a8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9FoughBZhvQHomJiTIzM9NsMzQajcZvWLt2bbGUMsnePp8W/MzMTNasWWO2GRqNRuM3CCEOtLVPh3Q0Go2mi6AFX6PRaLoIDgu+EOINIUShEGJLq+fihRBfCyF2W7dxbZx7jhBipxAiRwjxoBGGazQajcY5nInhvwW8CLzT6rkHgW+llH+xCvmDwAOtTxJCBAIvAWcBecBqIcRnUsptrhjc2NhIXl4e9fX1rpzut4SFhZGenk5wcLDZpmg0Gj/FYcGXUi4VQmSe8PSFwBnWx28D33OC4APjgBwp5V4AIcQ863kuCX5eXh7R0dFkZmYihHDlEn6HlJKSkhLy8vLo3bu32eZoNBo/xd0YfoqU8jCAdZts55g0ILfV93nW5+wihJgjhFgjhFhTVFR00v76+noSEhK6jNgDCCFISEjocp9qNBqNsXhj0daeMrfZolNKOVdKmS2lzE5KsptK2qXE3kZXfM0ajcZY3BX8AiFEDwDrttDOMXlARqvv04F8N++r0Wg0/sf+5ZC/wbTbuyv4nwHXWR9fB3xq55jVQJYQorcQIgS4wnqeRqPRdC3m3wqf/Mq02zuTlvke8DMwQAiRJ4S4CfgLcJYQYjcqC+cv1mNThRALAaSUFuAO4CtgO/CBlHKrsS9Do9FofJyaYqjIhcKtUGCOBDqTpXNlG7vOtHNsPjCz1fcLgYVOW+eD/OEPfyAxMZG7774bgIceeoiUlBTuuusuky3TaDQ+zeENxx5v+gDOetzrJvh0L52OePzzrWzLrzT0moNTu/Ho+UPa3H/TTTcxa9Ys7r77bpqbm5k3bx6rVq0y1AaNRtMJscXuM8bD5v+DMx+FAO82O9CtFZwkMzOThIQE1q9fz+LFixk1ahQJCQlmm6XRaHydwxshvg+MvQkq8+Dgz143wa89/PY8cU9y880389Zbb3HkyBFuvPFGU2zQaDR+xuENkDYGBsyE4AjY/AFkTvSqCdrDd4GLL76YRYsWsXr1as4++2yzzdFoNL5ObSmUH4QeIyE0CgaeC1s/AUuDV83Qgu8CISEhTJkyhdmzZxMYGGi2ORqNxtc5vFFte4xQ22Gzob4ccr72qhla8F2gubmZFStWcNNNN5ltikaj8QdOFPy+UyAiAbZ85FUztOA7ybZt2+jXrx9nnnkmWVlZZpuj0Wj8gcMbILYnRMSr7wODIeMUKNrpVTP8etHWDAYPHszevXvNNkOj0fgThzeq+H1rolIg17sp3drD12g0Gk9SXwGleyF15PHPR6VAbTE0NXrNFC34Go1G40kOb1JbW/zeRnSK2tac3AbeU2jB12g0Gk9ia6lgL6QDUHXEa6ZowddoNBpPcngjdEuHyMTjn4/qrrbV9rrKewYt+G7y2GOP8fTTT7d7zHvvvcdTTz110vOZmZkUFxd7yjSNpmuw+GH48kE4WmW2JfY5vPHkcA4cC+lUaw+/U7Fo0SLOOeccs83QaDofDTWw4hVY+Qq8PAH2LDHbopOpOATxdmZRR1onwlYVeM0ULfgu8NRTTzFgwACmTZvGzp07aWpqYvTo0S37d+/ezZgxYwA1gHzDhg2MHj2akpISpk+fzqhRo7jllluQUk16XL16NcOHD6e+vp6amhqGDBnCli1bTHltGo1fkbcGmi1wxu8gOAz+czH8+ILZVh2joQYaayDSzrjWoBAIj4dq7wm+f+fhf/kgHNls7DW7D4MZf2lz99q1a5k3bx7r16/HYrEwevRoxowZQ0xMDBs2bGDkyJG8+eabXH/99QCsX7+eESNGIITg8ccf57TTTuORRx5hwYIFzJ07F4CxY8dywQUX8PDDD1NXV8fVV1/N0KFDjX1dGk1n5ODPgIBTboWJ98A7F8CG/8FEH5lPYcvAsSf4oBZuvSj42sN3kmXLlnHxxRcTERFBt27duOCCCwDVQfPNN9+kqamJ999/n1/84heACufMmDEDgKVLl3L11VcDcO655xIXF9dy3UceeYSvv/6aNWvWcP/993v5VWk0fsqBnyBlKITHKg+/10Qo2e31pmRtUm0V/Khk+/ujvSv4bnv4QogBwPutnuoDPCKl/EerY85AzbvdZ33qYynlH929d3ueuCcRQpz03CWXXMLjjz/O1KlTGTNmTEuP/MWLF/PRRx+1ey5AaWkp1dXVNDY2Ul9fT2RkpGeM12g6C02NkLcaRl197LmUISrEU7JbPTabFg8/0f7+qO7qTctLuO3hSyl3SilHSilHAmOAWmC+nUOX2Y4zROxNYvLkycyfP5+6ujqqqqr4/PPPAQgLC+Pss8/mtttu44YbbgCgoqICi8XSIv6TJ0/m3XffBeDLL7+krKys5bpz5szhiSee4KqrruKBBx7w8qvSaPyQI5ugsRZ6nnrsueTBaluwzRybTqTGmnLZZkgnWWXpWNfzPI3RIZ0zgT1SygMGX9dnGD16NJdffjkjR47kkksuYdKkSS37rrrqKoQQTJ8+HYCvv/6aadOmtex/9NFHWbp0KaNHj2bx4sX07NkTgHfeeYegoCB+8Ytf8OCDD7J69WqWLPHBbAONxpc4YJ0Y1WvCsecS+kFAkBoU7gt0FMOP7g5NDapVshcwetH2CuC9NvadKoTYCOQDv5FS+shvxHkeeughHnrooZOeX758OTfeeGNLj/xFixZx8803t+xPSEhg8eLFLd8/99xzAFx77bVce+21AAQGBrJy5UpPmq/RdA4O/gxxvZVo2ggKgcT+vuPhVxdBaAwEhdrf31JtWwDhcfaPMRDDBF8IEQJcAPzOzu51QC8pZbUQYibwCWC3t7AQYg4wB2jxgP2Biy++mD179hznmb/22msmWqTRdGKkVILf3059S/JgyPURp6mmCKLa8O7hmOBXF0DyQI+bY2RIZwawTkp50pKzlLJSSlltfbwQCBZC2F3FkFLOlVJmSymzk5La+UH5GPPnz2fTpk0kJraxOKPRaIyjeBfUlkDP8SfvSxkMFbmqS6XZ1BS1Hc6BY59OvJSpY6TgX0kb4RwhRHdhTU8RQoyz3rfE1RtJLy1w+BJd8TVrNG1iy2zpOeHkfcnW7JzCHd6zpy06EnxbuqaXGqgZIvhCiAjgLODjVs/dKoS41frtpcAWawz/BeAK6aKChYWFUVJS0qUEUEpJSUkJYWFhZpui0fgGB1coIU3oe/K+FGumji8s3HYk+KHdICjcax6+ITF8KWUtkHDCc6+2evwi8KIR90pPTycvL4+iIu/1kPYFwsLCSE9PN9sMjcY3OPizSse0V9cSkwEh0eYv3DZZoLa07aIrUPZHJfuX4HuT4OBgeve204hIo9F0DSxHofwAjLzK/n4hIHkQFJos+LUlgGy76MpGdHe/jOFrNBqN56nIU9vYjLaPSRkMBVu9VtBkl5aiq3Y8fFCZOl7qmKkFX6PR+Bfl1rrO2HbStpOHqGKmqsNeMckuHRVd2YhK8VpPfC34Gk1XpXAHvDsb9i012xLnKM9V2/YEv2Xh1sSwTrWDgh+dolJIG+s9bpIWfI2mqyElrPsPzD0Ddn8FC+5TC4z+QvlBEIEQndr2Mb7QU8fm4bdXeAXHF195GC34Gk1XorkJ5t8Kn90BGWPh3GdVEdPGtjqi+CDlB6FbGgS2k3MSEa86UZrp4dcUQWCISr1sDy/OttWCr9F0JfYvg03z4LR74ZpPIPtGSMuG7//slZCCIVTkth/OsZE8CIpMLL6qKVILtm20RG/BlrbphTi+FnyNpiux7TMIjoDJ90NAoBKjaY9B5SFY7Se9n8oPtp+hYyMmzWsVrHapKeo4JROOtVfwgq1a8DWarkJzM+z4ArLOgpCIY8/3ngR9p8KyZ3yj/0x7WBqgMt8xDz/Kmt/e3OR5u+xRXdh+0ZWNyCQQATqko9FoDCRvlRLAQRecvO/MR6CuFNa84X27nKHyECAdE/zo7iCboabY42bZpaa44wwdUJ+0IhJ1SEej0RjIts/UImLW9JP3pY5Suev7lnnfLmcoP6i2MQ6EdFpCJSbk4kupCq8cEXywzrbVHr5GozECKWH75yp0E9ZG1kjGWMhbo0I/vopN8B3y8HuorRlx/KOVapKVo4IflaJj+BqNxiDy10PFQRh0ftvHpI+DoxUqTdNXqchV8e5uaR0f29Jr3gTBtxVdORLDBxXSqSv1nD1WtOBrNF2B7Z+pYqUBM9s+JmOc2uat8o5NrlB+UHnuQSEdHxvp3V7zx9HSVsHBgUhhMV5ZMNeCr9F0dqRU8fvek1RBUlsk9FNzVX1lPKA9yg86Fs4B9aYQkWiS4NsapzkY0gmLgfpKj4fTtOBrNJ2d4t1Quqf9cA6onPz0sZC72jt2uUK5g0VXNqK7m+zhOxjSCYsBpIr9exAt+BpNZyfPKuCZkzs+NmMcFO+EujLP2uQKTRaVlulIho6N6O7mZOnUFAMCIhI6PBSA8Fi19XBYRwu+RtPZObwBQqJUyKYj0m1x/LUeNcklKg+BbHLew/fScJHjqC5U4bP2+v20JixGbf1B8IUQ+4UQm4UQG4QQa+zsF0KIF4QQOUKITUKI0UbcV6PROED+eugxAgIc+HdPG6OyYHxx4bbCgbbIJ2JWtW1Hs2xPxJ8E38oUKeVIKWW2nX0zgCzr1xzgFQPvq9Fo2qKpEY5sVoVVjhAapQqwcn1Q8J3JwbdhVrVtFxD89rgQeEcqVgCxQogeXrq3RtN1KdoBlnrHBR9aFWCZ1IOmLVqqbNMdP8esattOLvgSWCyEWCuEmGNnfxqQ2+r7POtzJyGEmCOEWCOEWFNUVGSQeRpNFyV/vdo6I/jp46ChytzWwvYoz7Xm4Ic6fo5Z1bbVzgp+rNrWl3vCmhaMEvyJUsrRqNDN7UKIE9MB7DWEtjtdWEo5V0qZLaXMTkpy4gem0WhOJn+DGsAR19vxc2wFWL4W1ik/4FyGDphTbdtkURXLjmbogHVIivAPD19KmW/dFgLzgXEnHJIHtP5NpQP5Rtxbo9G0gzMLtjbi+0B4PBzysUwdZ4qubJhRbWsTbVuqpSMEBCjR93XBF0JECiGibY+B6cCWEw77DLjWmq0zHqiQUpo4Tl6j6QJYGqBgi3PhHFAFWEkDoWSPZ+xyheZmlZbpyOCT1phRbWsLy9jCNI7ihfYKDiaJtksKMF+oMV5BwP+klIuEELcCSClfBRYCM4EcoBa4wYD7ajSa9ijcpjo2Oiv4AAl9YddXxtvkKrXF0Gxpf3B5W3i72rauXG3D45w7zx8EX0q5Fxhh5/lXWz2WwO3u3kuj0TjB4Q1qmzrS+XMT+kHNf5QA2TJIzMRWPBWd4vy53q62tVUpOxPSsR1ve7PwEEZ4+BqNb5PzDRzeCMGREByuiou6DzXbKs+Tv16JtTMLtjZsVbkleyDNB+okq6yCH9Xd+XOju0PBVmPtaQ93Qjql+4y25ji04Gs6L3Xl8OUDsGne8c8HBMH5z8Ooq00xy2vkr1fhHGEvSa4DWgQ/xzcE35Zl42h/+da0rrYNCDTWLnu46uH7Q0hHo/FJ9i2F+beq2O3pD8CEu1Q8u74CFvwaPr1ddZE881HnMlj8hcZ6KNgGE+5w7fz43oBQgu8L2EI6US6GdGzVtq6EhJylky/aajS+xZHN8O5lKmf75q9VCMdGRDz84kP48rfw4z+gMh9mzXXNC/ZlCrdCcyP0GOna+UGhKgXSVwS/qkClLYZEOH9u62pbbwh+XbkKHzoypKU1YTGq4K3J4njTNSfphK6NpktTXwEfXKu8qxsWHi/2NgKD4Nxnlee/+QPY+aXXzfQ4hdvVtvsw16+R0M93BL+6wDXvHrxfbVtX7nw4B459IvBgT3wt+JrOg5Twya9Ugc7st9uP9woBk3+r8s2/+p0KgXQminZAYCjEZbp+jcQstWgr7RbFexe3BN/L1bb15c6Hc6BVP51yA405Hi34ms7DTy/Aji/grCeg5/iOjw8Mhhl/g7L98NM/PW6eVynapQTbnUXKhH7QUG1OP/kTqS5wPRzj7Wpblz18zzdQ04Kv6RyU7oVvn4BBF8D42xw/r8/pMPhCWPaMas7VWSjaAUkD3LtGQl+19YWwTpUbHr63q23rypwvugIt+BqNw3z9KASGKI/d2QXY6U+q7eKHjbfLDBpqVVgr0V3Bb5WaaSZHq6GxxnXBB+9W27od0tGCr9G0zYGfYPtncNo90M2FMQuxPVX64rZP1CcFf6dkNyDd9/C7pat1ALMFv6XK1oWiKxverLZ1NaRjO8eD1bZa8DX+TXMzfPV71WPlVBdzzgGyb1Sj/da9Y5xtZlG0U23dFfyAABXWMbuJWksOvgtFVzaiUtScWU/T1Kg+jWgPX6PxAJs/VBWl0x51LUfbRrdUyDob1r+r/mn9maKdIAIhvq/710roa76HbwvFuNJWwUZEAtSWeD7jqKVxWqzz54ZEKadDC75GYwfLUfj2j6p9wLDZ7l9vzHVQUwi7Frl/LTMp2qGE2tnCH3skZKn+Lk0W96/lKu5U2dqISICmo9BQY4xNbdHSVsGFRVshPF5tqwVf479snAeVeXDmI8a0R+h3lirSWfu2+9cyk6KdkNjfmGsl9FMVu+UHjLmeK1QXQECwayJqwzZ9qrbEGJvawtW2Cja04Gs0dmhugh+fV60D+kwx5pqBQTDyKtVd019TNC0NauE5aaAx12vdNdMsqgpU/N6dN3VvCb47IR1QbxS68EqjOYHtn0HpHjjtXmP74Iy+BpCw4V3jrulNSveAbPKA4JsYx3enytZGZKLaag9fo/EzpITlzykxGnS+sdeOy1SfGNb9R2UA+RtFO9Q2yaCQTkS8Ei+zBd+dlEzwoofvRgwftOBrNCexZ4kaaDLxbs/0Nx9xpVob8LUh3o5QtAsQarHVCIRQQ83LPDuYo12qC9xLyQT1xgXeC+m4OiXM1wVfCJEhhPhOCLFdCLFVCHG3nWPOEEJUCCE2WL8ecfe+mi7M8udU3v3wyz1z/ayzVFrjLj/solm0A+J6uZeieiKxGeataTRZVB97d1IyAUJj1O/UGyGdkGjX2xv7uuADFuA+KeUgYDxwuxBisJ3jlkkpR1q//mjAfTVdkfwNsH8ZnPor1bPdE0TEQ89T/bNtctFO91sqnEhMBlTkmdM1s6YIkO57+AEBKqxTU2yIWW3iapWtjfBYaKxVi+8ewG3Bl1IellKusz6uArYDae5eV6Oxy6q5arjE6Gs9e58BM6Bwm+qk6S80WVSs3d0K2xOJ7QmWOs+LpT1sLY3djeHDseIrT1JX5vqCLRw710NevqExfCFEJjAKWGln96lCiI1CiC+FEEPaucYcIcQaIcSaoqIiI83T+DvVRaqyduSVrsdIHWXADLXd6UdFWOUHVHGRURk6NmIy1LbioLHXdYQqA4qubEQkQG2p+9dpj/py9zx8D7dXMEzwhRBRwEfAPVLKE0e2rAN6SSlHAP8EPmnrOlLKuVLKbClldlJSklHmaToD695Sc2nHzfH8vRL6quIlf4rjG9VD50RirYJvRhzfiCpbGxHx3lm07eyCL4QIRon9u1LKj0/cL6WslFJWWx8vBIKFEIlG3FvTRWhqhNWvQ9+pxgtaW/Q/B/b/6PHB0oZRvEttEw3K0LHR4uGbKfhuxvBB5eJ7Y9HWrZCOZ6deGZGlI4DXge1SymfbOKa79TiEEOOs9/XwT17Tqdj+uWpvO+4W791zwEzVViDnW+/d0x1KctR0J6PDXeGxaoC4WR5+eJwxC/QRCVBX6tn6iroyNz1867kecjKMGI0+EbgG2CyE2GB97vdATwAp5avApcBtQggLUAdcIaUvDMrU+A0r/wVxvSFruvfumTEOwuNVM7Whs7x3X1cpyTlWGWs0MRnmePhVR9xPybQRkQCyWXnPtrx8I2msB0u9T3v4bgu+lHI50G5tu5TyReBFd++l6aIc3gi5K+DsPxnTJM1RAgKh/9kqPbPJ4nputbcoyTm22Gw0ZuXiVxcaE86B46ttPSH4NpF2p8mbP8TwNRqPsvo1CApXjc28TdZ09Y+cv97793aGunKVs+5RD9+ELJ3qI8akZILn2yu42zgNIDhcdQbVgq/pktSVw6YPYfhl7v0juUrvyWq7f6n37+0Mtm6WRrVUOJHYDCVC9Scm4HkQKT3j4XuqnsDdxmng8Z74WvA1vs3G91TRT/ZN5tw/MhGSB8O+Zebc31Fszc086eGDd+P49RUqJm5ESiZ4wcO3NU6Lde864bFa8DVdEClVKmZaNqSONM+OzEmQu9Jj5e6GULJb9YqJy/TM9WN7qq034/i2GbRGLtqCF0I6bsTwQXn4HhpkrgVf47vsW6qEbOzN5trRe5Lqb+LL3TNLclTTNCPGGtrDDA+/xib4BhVghkRAcITnBN+IkA7okI6mi7L6NeUtDbnYXDt6TQSEatrmqxTneC5+DxCZBIEhUO7FhVubhx9pUAwfPNtPx93WyDa04Gu6HJX5sGMBjLoGgsPMtSUiHroPVZ84fJHmZjXpylPxe1DpsDHp3vXwW0I6BsXwwbPtFerKVBtmd2c0aMHXdDnWvKmKZLJvNNsSReZkyF2limt8jap8FXJK9KDggwrreDOGX1Oo1iXcjYm3xpMefn05hBtQ5Tz+V3D5f92/jh204Gt8D0sDrH1LDSKJ7222NYrek1QnyrzVZltyMp7O0LER6+Vq2+pCFUoystguwoP9dOrKjXlzShoAPU9x/zp28PHSQdeY/erPHLU0OXawEC1lwkKokuEAIQgQAiHU48AAQUCAIMj6FRwYQFCgICQwgJCgAEKDAgkLDiA8OJDwkEAiQoKICgsiKjSQmPBgYsKD6RYeTFxECMGB+j22Q7Z/pry7sb802xIAGpua2RY4mGEEsPiLD/koJpwGSzPNUtIvOYqhqTEMT4+hX3IUwsiB6o5SvFttPRjDL69t4HBdLIOqC5jzxo+kJsaSmRDB2N7xDEn1UKtqI3PwgdoGC0X1YaRUFHLH22s4Z2h3zhnanahQg2TQ3cZpQHOz5L8rD7D+YDnPzh5h+N9TpxT82IhgGpo6jqNJCbLlsWx5rlnKlq2luZmjFkmThKbmZixNksamZizNkgZLc8tXvaWJxqaO2wPFRgSTEBlCSrcwuncLo3tMGGlx4WTERdAzPoK0uHD9prD6NZVe2G+aqWZszqvg7Z/3s2DTYeoam/g0JJPupavJbbqU0KAAmiXMW5VLXeN+AAZ2j+a6CZlcODKViBAv/muV7IGQKOMqUltxuKKOP3yylSU7CrhYWHgmBJrKc/lwfxU1DcqpmjUqjQdmDCSlm8FrLTXGCH5Ts+Svi3bw1k/7mSPr+E1wLTn5JfxmewEPf7KZ84en8scLhxIe4mbsva4Mkge5fHpOYTW/+3gTq/eXMSkrkbrGJsP/jjql4M+9NtuU+zY2NVPX2ETNUQs1Ry1U1VuorLdQUddIeW0DpTUNlFQ3UFx9lILKelbsLaGw6iiW5mNvFEEBgl4JEfRLjqJ/SjSDe3RjcGo3MuIiCAgwwXv0Nke2wMGf4awnvNs3pxXf7Szkn9/uZt3BciJCArlwZCqT+yfR78AMItfNZdGvsltmxjY1S/YWVbNiXyn/W3mQ3328mT8v3M5dZ2Zx/YRMgrzx5l2yW/XvN9AblFIyf/0hHv1sK5Ymya2n92VWfAAsfJXXL0xG9plCUdVR3vppP68t28eirUe4d1p/bp7U2zivtLpIFb25QX1jE3fPW89XWwuYNTqNi6OGw6oP+e72YawrC+PjdYd4b9VBDpTW8sb1Y93z9uvKXfbwP1idy8OfbiE8OJCnLxvBJaPTPPJpsVMKvlkEBwYQHBhAt7Bgh89papYUVNaTW1rLwdJa9hXXkFNYze7Car7eVoDtvaBbWBAje8YxMiOWMb3iyO4VR6RRH0V9idX/hqAwGHW112+9u6CKJxds54ddRfSMj+DR8wdzyZj0Y7/P0DNgzUsqjt/ndAACAwRZKdFkpURz9Sk9WXOgjJe+y+HJBdv5ZMMh/nzxcIale3g6V0kOpI0x7HL1jU3c98FGFmw+zNjMOP5+6QgyEyOhzOrBl+cihCC5Wxj3nzOQK8b25I9fbOOphdvJr6jjkfMGuy9WUioPP9L1HPzSmgZufns163PL+cN5g7nptN6w7QCsAlFbypheQxnTK55T+iRw7/sbuPb1lbx14zin/n+Ps7e+3KUY/toDpfx+/mbG90ng2ctHkBztuay0TqgY/kVggCA1NpzU2HBO6ZNw3L76xiZ2FVSxNb+STXkVrD9YxotLdtMs1SeB4ekxnNo3gdP7JzO6Z6x3vElPUl8Bmz6AoZd6ppthW7dtbOK5r3fx2vJ9RIQE8vC5g7j21ExCgk74eWaMVdvclS2C3xohBGMz43nz+rF8ueUIj362lQtfWs5dZ2Zx19Qsz3xCsxxVufHDrzDkcrUNFua8s5blOcU8cM5A5kzuQ6DN7m5pIAJOWrjtmRDBv68dwxNfbOeNH/dRe7SJP80aduw8V6gvV9PNXAzp1Dc2cd0bq9hZUMVLvxjNzGE91I6Wattj/XQuGJFKSGAAd763jqv+vZIPbjnV+fBOY52y18m2CqU1Ddzxv/Wkxobz8tWjXXuzcQIt+D5MWHAgw9NjGZ4ey5Xj1HPVRy2sP1jGz3tKWLG3hFd/2MtL3+0hOiyIyf2TOHtId6YMSCLaw384HmHdf1R64TjvVdZuyivn1x9sJKewmsuzM7j/nAEkRLUxbCM8DpIGKcFvByEEM4f1YGK/RB77bCv/+GY3W/MreXb2CON/L6X7VPqqARk61Uct3PjmatYcKOXpy0Zw6Zj04w8IDIboHnZTM4UQ/OG8QUSFBvLCkhzqLU08N3uk629ybubgP/rpVjYfquDf12Zz1uBW12ijvcI5Q7vz8lVj+OU7a3hywTaeuniYczd0ocq2uVly7/sbKKlu4ONfTfC42IMWfL8jKjSISVlJTMpSH3Ur6hr5MaeY73cWsmRHEQs2HSYkMICJ/RK4YGQqZw02MAvBkzRZ1JCTnhMgdZTnb9cseXFJDi8s2U1SVChv3ziO0/s7ED7oeQpsma+KnTpYY4gJD+bZ2SMYlhbDUwu3c/HLP/HatdkqPGIUJdYMHTdz8GsbLFz7+ko25lXw/BWjOH9Eqv0DYzKgIs/uLiEEv54+gNDgQP7+1U56xkdw33QXx1G2VNk6H9J5f/VB3l+Tyx1T+h0v9tBK8E8eZn7W4BTmTO7D3KV7OWNA8snntocLjdP+tXQvP+wq4smLhjI0zcNhPyt+oASa9ogJD2bmsB7MHNaDpmbJ+oNlfLX1CAs3H+He9zcSFryZswZ359Ix6ZzWL9G9j9meZOcC1W/97Kc8fquCynrumbeBn/eWcNHIVB6/cCgx4Q56VxnjVY1A0XZIGdLh4UIIbjytNwN7RHP7u+u45JWfePvGccb9g9tSMuP7unwJS1Mzt7+7jg255bx81WjOGdqj7YNj0jrsKfSrM/pysKSWfy7JISslmgvaevNoj5Y+Os6FdDbnVfCHT7cyKSuRe8/qf/IB4dZQYRu5+PdN78/y3cU88NEmRqRPItnRzKOWtgqxDh1+oKSG577Zxcxh3bnqlJ6O3cMAjBpifo4QYqcQIkcI8aCd/UII8YJ1/yYhxGgj7qs5nsAAQXZmPA+dO5hl90/h/249lcvGZLBsdxHXvbGK0/66hGcW7ySvrNZsU0/m55chthcMPNejt1m6q4iZzy9jQ245T182gn9cMcpxsQc19hA6DOucyIS+iXx02wTCggO5cu4KVu41qPinJEeFWcK6uXS6lJKH5m/hu51FPHHR0PbFHlQcvzK/3bmwQgieuGgoYzPj+O2HG9mUV+68YS6EdGqOWrj9f+tIjAzh+StG2XduAoOUKLch+KFBgbxw5UhqGyzc9+FGmps7TrUGWk27iu3wUCklj322lZDAAB49f4hXazeMGGIeCLwEzAAGA1cKIU7MpZoBZFm/5gCvuHtfTfsEWMX/iYuGsvL3Z/LyVaMZ0D2al77LYfLfvuOmt1bz3Y5Cx/+gPcmhtWqE4Sm3ut+HpA2amiXPfr2L695cRWJUKJ/fOfHkGLUjxPdRYYaDzgk+QJ+kKD689VSSu4Vy7Rur+HZ7gfP3P5HiXW7F75//djfvr8nlzqn9uOqUXh2fEJOuFidr2x8iEhIUwCtXjyExKpQ576yluPqoc4ZVF0JAkFMx8ScXbCO3rJbnrxxFfGQ7XUMjEtodgtIvOVo5TbuLeX+Ng5XFTrRG/npbAd/tLOKeaVnG1y50gBEe/jggR0q5V0rZAMwDLjzhmAuBd6RiBRArhOjAlXCDynyoOHTsqzJfDUOuLoSaEvXLOVqtSvi7wCz10KBAZg7rwVs3jGPZA1P51Rn92JhXwQ1vrWbqM9/z5o/7qKpvNM/AFa9ASLTHUjFLqo9y3RureOHb3cwalc4nt0+kX3K0axcTAjJOUW9QLpAaG86Ht05gQPdobv3vWhZtOeKaHaD+dot3Q6Kd0IUDfLAml398s5tLRqfza3vhD3t0S1PbNuL4rUmMCmXutWMoq23g7nnraXLGuahxrq3Ct9sLeG9VLrdM7svYzA4yvBzop3P1KT05pXc8f1q4ncJKB/onObhoW9fQxOOfb2NAiirS8zZGxPDTgNZvg3nAiY0g7B2TBhw24P4n888xKtvDEUSAyvsOjoDQaPUVFqP+KCITVWvWmDT1hx7bU30F+mEGjJW02HB+c/YA7jozi0Vbj/Dmj/t4/PNtPLN4F5ePzeCGiZmkx0V4z6DKfNg6H8bNcTks0R4r95Zw17z1lNU28tdLhjE7O8P9j9A9x8OOL6CqAKKdzyKJjwzhvzefwnVvrOKO/63jhStHHUsbdIaaYiU0ic63VPgxp5jff7yZ0/ol8pdLhjn+M4mxCn7lIUjrODI7JDWGP144hAc+2szz3+52/I3FibYKJdVHeeCjTQzsHs29Zznws4hM7LAJnBCCP88axjnPL+Oxz7fy8lUd1DnUlQECQtv/G375+xwOldfx/pzxplTUGyH49v5STnwrd+QYdaAQc1BhH3r2dHExY+bT0Gw5dhsp1ba5ydo7wQLNjeqjaWO9GqPWUAMN1XC0Sn0CKNiqPrbaVt9tBASpsv+ELNUyt/tw6DFCvRGY0UfFRUKCArhgRCoXjEhlQ245byzfx1s/7eetn/YzY2h3bpnc1/MFQwA/v6TSCsfNMfSyzc2Sl7/P4dmvd9ErIZI3rh9rXM+XDKs/k7sSBl/g0iW6hQXzzo3juPGt1dz53noam5q5cGSacxcp3qW2Tgr+ziNV3PqftfRJiuTlq0c7JzzdrGGwikMOnzI7O4NV+8r455LdjOkV51g2VHWhQ33wpZT87uPNVNZZ+O/NpxAa5EBIMCIe8jd0eFifpCjumtqPpxfv4uttBe1n7dSVK0exnU8kOYXVvPrDHi4amXpSzY23MELw84CMVt+nA/kuHAOAlHIuMBcgOzvbtXjLqKtcOs0ulqNQdVj9gZcfUH1LSnKgaCfs/kqJFaiFs57j1bCMvlNVqbufMDIjlheuHMWDMwby1k/7eW/lQb7YdJhT+yQw5/Q+nNE/yTMLS7Wlqg3y0EsN7Yp5pKKe33y4keU5xVwwIpU/zRpmbGpqjxEQGOqW4ANEhwXz1g3juOnt1dzz/gaONjYze2xGxyfaaEnJdDykc7iijhvfWk1YSCBv3uBCVWlkonrtlR2HdGwIIXjyoqFsOVTBPfPW8/mdp3X8KbKmyKEsqP+uOMDibQU8NHMQA7s7+AkxIkE5c1J26KTNmdyXzzce5pFPtzC+T3zbdRT15e0u2KrF8c1EhATx8HnutYtwByP+C1YDWUKI3sAh4ArgFycc8xlwhxBiHircUyGl9Ew4x2iCQpVHH5cJTDx+X2MdFGyD/HVwcIXqAbN1vtoX1xuypsOg86HXBI8tRhpJamw4v585iDun9uO9VQd5Y/l+bnhzNQNSovnl5D6qIvHE6lN3WPEKNNbApF8bdskvNuXz0PwtNFia+fOsYVwx1oAQzokEhapwhpOZOvaIDA3izevHcct/13L/R5uobbBw/UQH3/yKd0NQ+DGvu6PDq49y1WsrqahrZN6c8aTFhjtvsBDQLdUpDx8gPCSQV64ezYUv/cgv31nLR7ed2nZjMCkdCulsza/giQXbOWNAkmqb4CgRCerTfUMNhEa1e2hIUAB/uWQYl7zyE499to1nZo+wf2AHfXT+b20eK/eV8pdZw0hsq7DPC7j93yultAB3AF8B24EPpJRbhRC3CiFutR62ENgL5AD/Bn7l7n19guBwSB8D434Jl74O926Fu9arkFJif1j3Drx9Hjw7GBbeD/nrzbbYIaLDgpkzuS9L75/C05eNQAj4zYcbmfS3Jbzy/R4qag1Y4K2vhFX/goHnudVh0EZx9VHunreeO/63nt6JkSy8exJXjuvpuZS3jFNUWKCxzu1LhYcE8u9rxzB9cAqPfb6NF5fsbune2i7Fu1WGjgMLmxW1jVzz+iryy+t44/qx7tUBxKSrGL6T9EmK4oUrR7HjSCW//XBT26+xrkyFXNsJ6dQctXDn/9YTFxHMM5eNcK6it4Nc/BMZ1TOOO6b046N1eSzY1Iaf2k4fndKaBv60cDvZveKYne3EJzgPYIi7JqVcKKXsL6XsK6V8yvrcq1LKV62PpZTyduv+YVLKNUbc1+cQQqXtjfslXPUB3L8HLn0D0rNVsc7cM+DVSar9b32l2dZ2SEhQAJeOSefLuyfx9o3j6JccxV8X7WD8n7/lkU+3sLeo2vWLr35N9c6Z/Bu3bGxulry36iBnPvMDCzcf5p5pWfzfrafS28hqVntknKJEyaA38dCgQF66ajQXjUzl6cW7eOCjTTRY2s51B1QM34EK24q6Rm54axV7Cqv51zXZjOvtZp+ibmlOe/g2pgxI5nczBrJg82FeXJJj/6Dq9ouubOGR/SU1/OPyUW23wmiLNtortMedZ2YxIiOW38/fzOEKO2/ydeV2QzpSSp74YhtV9Rb+NGuY6R1v/bzblo8TEglDL4Er3oXf7FKePxIW3AfPDYHFDzuU3mY2QghO75/EuzePZ+Fdkzh3eA/mrcpl6jM/cM3rK/lq6xEsTR2IU2saatVibd8z3WqjsGJvCZe8+hO/+3gzA7tH8+Xdk7hnWn/vNJGzFWAZOAErODCAZ2eP5K6p/fhgTR7XvbGq7U9TlqNqTamD+H1uaS2XvPITm/IqeOHKUY4tmHZETJpa12p2cMjQCfxyUh9mjUrjma938e7KAycf0E6VrZTS2ok0n3um9efUvi4sftoEv+7k9gptERwYwPOXj6SxqZn7PrBTkFVXZjek89ZP+5m//hC/mtKP/ikupgIbiBZ8bxEeqzz/W5bBzUvUcI+fX4bnR8D829RisB8wOLUbT182gh8fnMp9Z/Unp7CaW/6zlol/XcJfF+0gp9ABr3/NG2rRbPJvXbJh/cEyrnl9JVfMXUF+eR1PXzaCeXPGu55b7wqRierTXO4qQy8bEKD60Tw7ewRrD5Rx3ovL+HmPHU+0dK+1aVrbGTprD5Ry0Us/UlR1lHduGsc5Qw0akNItDWSTqm1xASEEf5o1jKkDk3lo/hbmLj3hb7+lj87Jgv/04p28vnwf10/I5M6pLhac2Tqx2umn0x6ZiZE8ev5gftpTwkOfbD5WV9DSGjn2uOO/3V7AE19sY/rgFO4503PTyJxB99LxNkKouP9lb6q2tj+/DGvfhE3vw4grlAj6yhzXdkiKDuXOM7O47Yy+fLujkA9W5zJ36V5e+X4Pw9Nj1Pi4Id3pk3TColhdOSx7GvpMgV6nOny/uoYmPt+Uz7srD7Ixt5z4yBAePncQV4/vRViwSQvi6WNh7/cOZXs4y6zR6fRKiOTXH2zgyn+v4JrxvXhwxsBjMxDaScmsbbDwrx/28soPe+gRE8Yb14+l74m/B3eIsS4SVx46lpfvJGHBgbx69Rju/WADf1q4g+qjTdw7LUutudgJ6TQ3S57/djcvfbeHK8dl8Oj5bvTcdyGkY2N2dgYHS2t56bs91DY08cxlIwiy1KpU71Ye/rb8Su58bz2DU7vxjyvc6BpqMFrwzSS2J8z4C5x2L/z4D+X5bvpA5aRP/o1Xe8K7SlBgAGcP6c7ZQ7pTWFXPp+vz+WJTPn9btJO/LdpJn6RITu2TwPg+CYzNjCdl1XOIunI464/tXldKyZHKepbuKuL7nUUs311M1VELWclRPHr+YC7LzjC/C2j6WPVGXZGrfpcGM6ZXHIvunszTi3fyxo9qqtQVYzO4fGwG6S1zbI95ufWNTSzYdJi/f7WTI5X1nDu8B09cOLT9NgOu0Lra1hbacoGQoABeuGIUkSGBvPDtbr7fWcivz+rP6dWFiIDglkXQn/eU8OSCbWzNr2TWqDSeusiJQjF7hMWogksnPXxQn05+e/ZAIkKC+PtXO6lraOIv0+KJBwiPpa6hiXdXHuDl7/fQLSyY168b691xlx0gHMoGMIns7Gy5Zk3nXN+1S+Vh+O4pWP9fVXV6+oMqDOSHlb355XV8tfUIS3cVsXp/GdVHLaRSzHdh9/Fz2CQ+6/MoseEhxIQHExESSH1jEzUNTVTUNbCnqIbdBVWUWePX3buFccaAJGaNTmdsZpw5g8LtcXgj/GsyXPI6DLvUo7dae0ANv/l+VxEAb8W8znDLJl4d/TkIWHegjI25FTQ0NTMiPYY/nDeY7I5aDLhKXRn8NROmPwkT7nT7cs3Nkv9bl8fz3+zmUHkdr8e+yRjLBv7Q90OKqupZsbeUtNhw7j9nAOcPTzXGW/5rbxhyMZz3rMuXsFWpDxQHWRT6IO/2epLnDg2iuPooE/om8PgFQ8gyIW4vhFgrpbQ759V33no00K0HXPgijL9NLeh+9TtY/x+Y+XfIPM1s65wiNTacGyb25oaJvbE0NbM1v5LIL+8k8DD8X7frWLenhIq6xpZB2KA8vujQIPokRXLO0O5kJUczoV8CA1KifUfkW5M8RLXkyFvtccEf0yuON28YR15ZLe+vzqXHqlxymnvw5k/7aWqWDE2L4fqJmWoCWlaSZ0MIYbEQHOlyps6JBAQIZmdncNHINN5fk0v0N6Ucae7G1kMVhAQF8NuzB3DTab2NDd050E+nI26Y2JtxvePZtXIRbISv9tbTv2cUL/1ilGmVtB2hBd8XSRkCV38MOxfClw/CW+fCsNlw9p8gyoAsCy8TFBjAiOBcyP8cJtzBi9MvatlnsQ5+DwsONKW3iFsEBkHqaEMzdToiPS6C+87qD2uPwKjL2TnzHCzN0rs/OyFU7N6JaltHCAkK4JrxvWCjBaL6suSqMwy9/nFEJDiVpdMWQ1JjGDIwAjbC27edhUgd6b5tHsTP/sO6EEKo3vC3r1QLuVvnw0tjYcP//K/DZ3MzLPiNip2ednxVbVBgANFhwf4n9jYyxsLhTaonk7eoLoCjlZDYHyGEOT87N3LxO6S6yKE+Om4REe9SDN8u1tbIwoUB5t7GT//LuhAhETD1Ybh1ucq5/uQ2+M9FUGYnf9lXWfO6aid89p/8YiHaKdLHqgKswxu8d0/bgq2bYw3dIibNpWrbDmluVnn4Lg4vd5iIeLdDOi04MfzEbLTg+wvJA+GGRXDuM5C3Bl6ZoKpV25k85BOU58I3j6k0zJEntljqBKQbX4DVIS0pma71wTeEbukqfdLSYOx168tViqPHBd8awzfi03Jducr6CTG/sKojtOD7EwEBMPZm+NXPyrNccB+8cwGU7TfbMvtICQt+rQqEzv+HX7WPdpioJNVYz+ACrHYp2qkWTaNdmBVrFDFpgIQqu01vXafaOgXMheHlTtG6gZq71JerhWwHh7WYie9bqDmZ2J5wzXy44J+qgdfLE2D1674X29/0AexeDFP/YO022klJH6s8fG/9/Au3qYZzZgpMSy6+wWGdKmtzsm4efjNzsoFau7TRR8cX0YLvrwgBo69V3n7GWOVJ/+ciVb3rCxRshS/ugYzxcMotZlvjWdLHKaHyREz7RKRUP9sU83qqA8dX2xqJrV1DtEFtINrCjWrbk2ijj44vogXf34nNgGs+gXOfVbH9lyeooSJmevt1ZTDvKjXubfbbfjELwC0yxqqtAf3xO6S6QKUTJnc8HMSjODHb1ilsHn6UlwTfgNTMjoaf+BJa8DsDQsDYm+C2nyBtlPKs/3OxObH95ib46GYlBLPf8byn5gukDFUFWN6I4xdstd7TZA8/NEql2XrCww+LUdlpnsTFBmp26WD4iS+hBb8zEdcLrv3M6u2vhpdPVW2IXWxj6zRSqgrhnG9g5t+g54mz7DspgcGQNkZNPfM0hdvU1mwPH1SmjtEx/Mp87yxGGxnS0R6+xjRs3v7tK6H3ZPjq9/DamZC31rP3bW6Ghb+BFS/DuFtgzA2evZ+v0XM8HNkMR90YCuMIBdsgKgUifaB03wPVtlQd8c6nQjcaqB2HlNZFW98vugIt+J2XmHS4ch5c+qbyml6bCp/cfqz1rJE0WeDT21VdwIS7YMZfO2cKZntknKJ6xB/y8Btr4VZINjmcYyMmQ9VZGEnVEYjuYew17REQqMIw7nr4DdXq994VQjpCiL8LIXYIITYJIeYLIWLbOG6/EGKzEGKDEKILtb80GSFg6Cy4c60S4k3vwz/HwHd/bikHd5vKw/DeFbDxf3DG71Xb464m9qBSMxGeXbhtblI5+Ck+EM4BlTBQXw5Hq4y5XnMzVHvJwwdDGqhRV6a2XSSk8zUwVEo5HNgF/K6dY6dIKUe21bZT40FCo2H6EyqFs/dk+OEv8Pxw+OFvUFPs2jWlhA3vwcunwP5lanzjGQ90TbEH9Q+fPMizgl+6Fyz1vuXhg3Fefm2JqrL1hocPBgl+udp2BQ9fSrlYSmmxfrsCSHffJI3HSMxS83VvWQq9TlO9958ZCB9eD3uWQFMb81NbYzkKWz5SHTw/uRWSBsGtP6q+/V2djFMgd7Xn2l3YMnSSB3nm+s5iG/pSYZDg26p2u3lR8G0euqv4UR8dMLY98o3A+23sk8BiIYQE/iWlnNvWRYQQc4A5AD17Gj9FSAP0GAFX/g8Kt8Pat2HTPNWNMzhSZdb0mgCxmeqPOCxGfQooyYHinbBjocpdjukJM/6mWj109jx7R+k5Xo2rLNrumbBL4TZAQNJA46/tCi0evkHFfi1FV94S/DjIX+feNfzMw+9Q8IUQ3wD2gmoPSSk/tR7zEGAB3m3jMhOllPlCiGTgayHEDinlUnsHWt8M5oKaeOXAa9C4SvIgNWJx2mOw+yvYtwwO/AhLnrR/fHg89J4Eo69TzdD8oHeIV7GN+zu4wjOCX7BVDU73dI66o0SlQGCIgR6+tejK2zF8d2YSt3j4/pGl06HgSymntbdfCHEdcB5wpmxjXqKUMt+6LRRCzAfGAXYFX2MCwWEw+EL1BepjbnWh8l7qK9Q/RkIfv/mjNo243qqPe+5KlRprNIXbzS+4ak1AgKq4NSqGX3UEEOqNxBu0bqAW6uKQd5uH3xVCOkKIc4AHgNOllLVtHBMJBEgpq6yPpwPtT7DWmEt4nBZ3VxBChcQ8UYDVUKsWbYddZvy13SE2w1gPPzLJezOcWzdQc1nwy0AEQoiL53sZdz+TvwhEo8I0G4QQrwIIIVKFEAutx6QAy4UQG4FVwAIp5SI376vR+CYZ46H8wLF4tFEU7QCkb3n4oNZyjPLwKw97txWHEdW2tipbP8lOc8vDl1LaHbljDeHMtD7eC4xw5z4ajd/Qc7zaHlwBQy4y7rq+1FKhNbEZKnfechSCQt27VtVh7y3YQivBd6Pa1o/66ICutNVojKX7cJXttH+5sdct2ApBYRDf29jruostU8eIrpneaqtgw9ZAzZ2OmX7URwe04Gs0xhIUAr1OhX0G5yTkrYbUUb6XAhtrE3w3wzpNjVBTZJKH70ZIx4/66IAWfI3GeHqfrmoWKg8bc73GejXZLH2sMdczEqOqbasLAOm9oito1UDNHcH3n+EnoAVfozGe3pPVdv8yY653eCM0Nx7L8/cluqUBwn0P39tFV6A+LYXHuRfD1yEdjaaL03248vr2/mDM9fKsg1XSfVDwg0KUSLvr4Xu76MpGeLzrHn5zs6pT0R6+RtOFCQhQFcn7fjBm1GTuSojtBdFeKkhyFiNy8c3w8MG9BmoNVSCbtYev0XR5ep+uRLBsn3vXkVI1ZPPFcI6NmAz3++lUHYaAIIhINMYmR3GngVpLa2S9aKvRdG36nKG27oZ1KnJVnnuGD4+LjM1Qs23dGaVZeVgNLvd2f6aIONc9/BrreZFJxtnjYbTgazSeIKGfCk+4m55pG4zuixk6NmIyVB97d6qLq7xcZWujdQM1Z6kpsl7Dy59K3EALvkbjCYRQYZ19S93rj5+7CoIjIGWocbYZjRF98b1ddGUjIlE1UDta6fy5NsGP1IKv0Wh6T4baYtUf31XyVkHaGAg0cnSFwRiRi+/ttgo2bJ05q4ucP1cLvkajaaHP6Wq75zvXzm+ohSObfTucA62qbV1cuG2sU/nsZnj4UclqW13g/Lm1JaqNRkiksTZ5EC34Go2niEmHlGGw/TPXzs9fr2LjvpyhA0rwwuNd9/BtOfjdUo2zyVFaPHwXBL+mCCITjLXHw2jB12g8yZCLVB69K83F8vxgwdaGO7n4ldZZtqZ6+IXOn1tT5FcZOqAFX6PxLEMuVtttnzp/7t4fICHLP2LEsW70xS+11irEmdAJNDxeDTCp0YKv0WjcJaGvarWwdb5z59UUqwyfQed7xi6jictUg19cycUv26eKrmyLv94kIEB5+S6FdEr84824FVrwNRpPM+Ri1d7YmWrUbZ+CbIKhszxnl5EkZIGl3rWwTule1TrCrEykqGTnQzpSKg/fj3LwQQu+RuN5bJOvnAnrbJ2vRNSX8+9bk5iltsU5zp9butfcwS6RLgh+fYXqYNqVQjpCiMeEEIes82w3CCFmtnHcOUKInUKIHCHEg+7cU6PxO+L7QI+RsOVjx46vOqImZg2d5TezUknsr7bFu5w7T0oVw4/vY7xNjhKV4rzg1xSrbVcSfCvPSSlHWr8WnrhTCBEIvATMAAYDVwohfGwSs0bjYYZcDPnroGx/x8du+xSQMMRPwjmgWhSExULJbufOqy1VVa6mCn6yWrR1piLaD4uuwDshnXFAjpRyr5SyAZgHXOiF+2o0voMtrLP5w46P3fIxJA+G5IEeNclQhFBefrGTgl+6V23N9vCbLc51zay1efhdT/DvEEJsEkK8IYSw1yc0DWi9kpNnfc4uQog5Qog1Qog1RUUulDtrNL5IXCb0nQo/vdj+hKWKPMhd4V/evY3ELNcF34yUTBtR1rCMM6mZLR5+JwvpCCG+EUJssfN1IfAK0BcYCRwGnrF3CTvPtdmaTko5V0qZLaXMTkryrx+mRtMu059U4YulT7d9zNZP1NZfsnNak5ilWjnXO9GIrHQvICCul8fM6hBXqm1tMXw/y9LpMA9KSjnNkQsJIf4NfGFnVx7QOsE2Hch3yDqNpjORMgRGXQ2r5sLYm1SOfmtqimHFy5A66uR9/kCCNVOnZLdq+OYIpXtV/n1QqOfs6ogWwXfSww+LUSMe/Qh3s3Rat7e7GNhi57DVQJYQorcQIgS4AnCxuYhG4+dMeRgCQ+CbR49/vrkJPrpJif75z5tjm7u4kppZts/clExwrYFaTbHfeffgfgz/b0KIzUKITcAU4F4AIUSqEGIhgJTSAtwBfAVsBz6QUm51874ajX8SnQKn3QvbP4edXx4bvPHdn2Dv93DuM9BjhKkmukxcb9WmwJnUzNK95i7YAoR2g8BQ5z18P4vfgwMhnfaQUl7TxvP5wMxW3y8ETkrZ1Gi6JKfeDuvfgfeuUCKZORHW/xdGXwuj7f5L+QdBIWpx2tHUzLpy1WLYbA9fCOdz8WuK/TLspittNRpvExIBc36A819Q3u3Geaowa8bfzbbMfZxJzbQNeDfbwwfn++l0RQ9fo9G4SEQ8jLlOfdVXqLh+cJjZVrlPYj/Ys0StSQQEtn9sqS8JfopjRXGgXlttiV8KvvbwNRqzCYuB4HCzrTCGxP7QdNSxRnEtOfiZHjXJIaKSHM/DrysDpN8VXYEWfI1GYyQtqZkOZOqU7oOo7r4xIjAqRcXlmywdH+unbRVAC75GozGSltRMBzJ1fCFDx0ZUMiCPtUxoDz+tsgUt+BqNxkgiEiA8zrGF2zKTu2S2xplqWy34Go1Gg0pxTMjqOKTTUKOGl5udkmkj0lZ85UD/rpoSte2ChVcajUZzPIn9oWhn+8fYMmJ8xsN3otq2pggQKtPKz9CCr9FojKXHCJXxYku7tEfhdrX1leIlZwU/IqHjtFMfRAu+RqMxlr5T1Xbvd20fs/d7lY7qKyMcQyIhJPpYfL49/LToCrTgazQao0noqzpg7llif7+UsOc76HOGb3nJUUkOevjFfpmSCVrwNRqN0QgBfafA3qX289qLd0NlHvSZ4n3b2sPRfjq1WvA1Go3mGH2mwNEKyF9/8j6b59/X1wTfwX46OqSj0Wg0rehzBiDsh3X2LIH4vr7RUqE1jnj4lgbV+0gLvkaj0ViJiIfUkScv3FqOwv5lxxZ2fYnIZKgvVza2ha0SNyLBKyYZjRZ8jUbjGfpOhdxVx8+4zV0FjbW+KfjdUtW2Iq/tY2whH1sap5+hBV+j0XiGPlNANsH+5cee27MEAoIg8zTz7GqLxP5q217RmG18Y7yP1A84iRZ8jUbjGTLGQXDk8XH8PUsgfRyEdTPPrrZIsgp+cXuCvxNEgO8UjDmJWwNQhBDvAwOs38YC5VLKkXaO2w9UAU2ARUqZ7c59NRqNHxAUqjz5XV/BgBkq7n14I0z5vdmW2ScsRrVrLmqn02fxLrXYHBTqNbOMxN2ZtpfbHgshngEq2jl8ipTSgd6jGo2m0zDofNj9Ffx31rHnfDF+byOpf/seftEuSBzQ9n4fx5ARh0IIAcwGfPg3qdFovM7oa6DfNNX7vmwfNDVC2hizrWqbxAFqxrCUqoCsNU0W1QW0/3RzbDMAo2baTgIKpJRtNcGWwGIhhAT+JaWc29aFhBBzgDkAPXv2NMg8jUZjGt16qK/MiWZb0jFJA6ChSrVutmXt2Cg/AM2NxxZ3/ZAOBV8I8Q3Q3c6uh6SUn1ofXwm8185lJkop84UQycDXQogdUsql9g60vhnMBcjOzpYd2afRaDSG0TpT50TBt2XvdOaQjpRyWnv7hRBBwCygzc9pUsp867ZQCDEfGAfYFXyNRqMxjSSrmBfvOrn1gy22bxvj6IcYkZY5DdghpbRbrSCEiBRCRNseA9OBLQbcV6PRaIwlKgVCY+zn4hfvVvvDY71ullEYIfhXcEI4RwiRKoRYaP02BVguhNgIrAIWSCkXGXBfjUajMRYhrJk6dlIzi3b6dfweDFi0lVJeb+e5fGCm9fFeYIS799FoNBqvkDgAdi8+/jkp1ZvA8Nnm2GQQutJWo9FoWpPUX41orCs79lx1ARyt9HsPXwu+RqPRtMaWhVPcKsu8JUNHC75Go9F0HpLsNFGzxfST/DclE7TgazQazfHE9oLA0ONbLBTvUkPOo3uYZ5cBaMHXaDSa1gQEqlz71k3Uinaq505st+BnaMHXaDSaE0nsf7KH7+fhHNCCr9FoNCeTNADKDsCRzVBTonrr+PmCLRjXPE2j0Wg6D6mjAAmvngZYwzha8DUajaYT0v9suO1nKNwGRTugphj6nG62VW6jBV+j0WjskTJYfXUidAxfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLoAVfo9Fougha8DUajaaLIKSUZtvQJkKIIuCAi6cnAsUGmuMP6Nfc+elqrxf0a3aWXlLKJHs7fFrw3UEIsUZKmW22Hd5Ev+bOT1d7vaBfs5HokI5Go9F0EbTgazQaTRehMwv+XLMNMAH9mjs/Xe31gn7NhtFpY/gajUajOZ7O7OFrNBqNphVa8DUajaaL0OkEXwhxjhBipxAiRwjxoNn2eAMhxBtCiEIhxBazbfEGQogMIcR3QojtQoitQoi7zbbJ0wghwoQQq4QQG62v+XGzbfIWQohAIcR6IcQXZtviDYQQ+4UQm4UQG4QQawy9dmeK4QshAoFdwFlAHrAauFJKuc1UwzyMEGIyUA28I6UcarY9nkYI0QPoIaVcJ4SIBtYCF3Xm37MQQgCRUspqIUQwsBy4W0q5wmTTPI4Q4tdANtBNSnme2fZ4GiHEfiBbSml4sVln8/DHATlSyr1SygZgHnChyTZ5HCnlUqDUbDu8hZTysJRynfVxFbAdSDPXKs8iFdXWb4OtX53HW2sDIUQ6cC7wmtm2dAY6m+CnAbmtvs+jkwtBV0cIkQmMAlaabIrHsYY2NgCFwNdSyk7/moF/APcDzSbb4U0ksFgIsVYIMcfIC3c2wRd2nuv0XlBXRQgRBXwE3COlrDTbHk8jpWySUo4E0oFxQohOHb4TQpwHFEop15pti5eZKKUcDcwAbreGbA2hswl+HpDR6vt0IN8kWzQexBrH/gh4V0r5sdn2eBMpZTnwPXCOuZZ4nInABdaY9jxgqhDiv+aa5HmklPnWbSEwHxWqNoTOJvirgSwhRG8hRAhwBfCZyTZpDMa6gPk6sF1K+azZ9ngDIUSSECLW+jgcmAbsMNUoDyOl/J2UMl1KmYn6X14ipbzaZLM8ihAi0pqIgBAiEpgOGJZ916kEX0ppAe4AvkIt5H0gpdxqrlWeRwjxHvAzMEAIkSeEuMlsmzzMROAalMe3wfo102yjPEwP4DshxCaUY/O1lLJLpCl2MVKA5UKIjcAqYIGUcpFRF+9UaZkajUajaZtO5eFrNBqNpm204Gs0Gk0XQQu+RqPRdBG04Gs0Gk0XQQu+RqPRdBG04Gs0Gk0XQQu+RqPRdBH+H2CPrSMV0KsfAAAAAElFTkSuQmCC\n", 133 | "text/plain": [ 134 | "
" 135 | ] 136 | }, 137 | "metadata": { 138 | "needs_background": "light" 139 | }, 140 | "output_type": "display_data" 141 | } 142 | ], 143 | "source": [ 144 | "x = torch.linspace(0, 5, 100, requires_grad=True)\n", 145 | "y = (x**2).cos()\n", 146 | "y.sum().backward() # populates the grad attribute below.\n", 147 | "print(x.grad)\n", 148 | "\n", 149 | "plt.plot(x.detach(), y.detach(), label='y')\n", 150 | "plt.plot(x.detach(), x.grad, label='dy/dx')\n", 151 | "plt.legend()\n", 152 | "plt.show()" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "### Exercise\n", 160 | "\n", 161 | "1. Introduce a new vector x2 which also ranges from zero to five (same as x, but not cloned from x).\n", 162 | "2. Plot the polynomial y=x23-6x2+8x\n", 163 | "3. Plot dy/dx and dy/dx2.\n", 164 | "\n", 165 | "Which x contributes most to the gradient at zero? At five?" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 72, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "# TODO: define x2 just like x, but not cloned from x\n", 175 | "x2 = 'TODO'\n", 176 | "\n", 177 | "# TODO: Plot the given polynomial which depends on both x and x2\n", 178 | "\n", 179 | "# TODO: Plot both dy/dx and dy/dx2. Explain what you get." 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "Accumulating and Zeroing grad\n", 187 | "-----------------------------\n", 188 | "\n", 189 | "**Gradient accumulation.** If you find that your data batches are too large to get gradients of the\n", 190 | "whole thing, then it is usually possible to split the batches into smaller pieces and add the\n", 191 | "gradients. Because gradient accumulation is a common pattern, if you call `.backward()` when parameters\n", 192 | "`x.grad` already exists, it is not an error. The new gradient will be *added* to the old one.\n", 193 | "\n", 194 | "**zero_grad().** That means that you need to set any previous value of `x.grad` to zero before\n", 195 | "running `backward()`, or else the new gradient will be added to the old one. Optimizers have a\n", 196 | "utility `optim.zero_grad()` to do this to all the optimized parameters at once." 197 | ] 198 | }, 199 | { 200 | "cell_type": "markdown", 201 | "metadata": {}, 202 | "source": [ 203 | "Saving memory on inference\n", 204 | "--------------------------\n", 205 | "\n", 206 | "**Avoid autograd when you don't need it.** Normally, all the parameters of a neural network are set to `requires_grad=True` by default, so they are ready to be trained. But that means that whenever you run a network, you will get output which is also requires-grad, and it will be attached to a long computation history that consumes a lot of precious GPU memory.\n", 207 | "\n", 208 | "To avoid all this expense when you have no intention of training the network, you could go through all the network parameters to set `requires_grad=False`.\n", 209 | "\n", 210 | "Another way to avoid the computation history is to enclose the entire computation within a `with torch.no_grad():` block. This will suppress all the autograd mechanics (which means, of course, `.backward()` will not function).\n", 211 | "\n", 212 | "Note that this is different from the role of `net.eval()` which puts puts the network in inference mode computationally (batchnorm, dropout, and other operations behave differently in training and inference); `net.eval()` does not have any effect on `requires_grad`." 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "More tricks\n", 220 | "-----------\n", 221 | "\n", 222 | "**Gradients over intermediate values.** Normally gradients with respect to intermediate values are not stored in `.grad` - just original input variables - but you can ask for intermediate gradients to be stored using `v.retain_grad()`.\n", 223 | "\n", 224 | "**Second derivatives.** If you want higher-order derivatives, then you want pytorch to build the computation graph when it is computing the gradient itself, so this graph can be differentiated again. To do this, use the `create_graph=True` option on the `grad` or `backward` methods.\n", 225 | "\n", 226 | "**Gradients of more than one objective.** Usually you only need to call `y.backward()` once per computation tree, and pytorch will not let you call it again. To save memory, pytorch will have deallocated the computation graph after you have computed a single gradient. But if you need more than one gradient (e.g., if you have different objectives that you want to apply to different subsets of parameters, as with happens with GANs sometimes), you can use `retain_graph=True`.\n" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "metadata": {}, 232 | "source": [ 233 | "### Exercise\n", 234 | "\n", 235 | "1. Plot the polynomial y=x3-6x2+8x, just as in the first exercise.\n", 236 | "2. Use `y.sum().backward(create_graph=True)` to compute the gradient, and plot dy/dx. Why is `x.grad.detach()` needed now?\n", 237 | "3. Now set `dy = x.grad.clone()` and then `x.grad.zero_()`, before using `dy.sum().backward()` to compute a 2nd gradient. Plot d2y/dx." 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": 74, 243 | "metadata": {}, 244 | "outputs": [], 245 | "source": [ 246 | "# TODO: define the polynomial just like the first exercise\n", 247 | "y = 'TODO'\n", 248 | "\n", 249 | "# TODO: Use `backward(create_graph=True)` to compute dy/dx. Plot it.\n", 250 | "\n", 251 | "# TODO: Use `backward()` a second time to compute the second derivative. Plot it." 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "### [On to topic 3: Optimizers →](3-Pytorch-Optimizers.ipynb)" 259 | ] 260 | } 261 | ], 262 | "metadata": { 263 | "accelerator": "GPU", 264 | "kernelspec": { 265 | "display_name": "Python 3 (ipykernel)", 266 | "language": "python", 267 | "name": "python3" 268 | }, 269 | "language_info": { 270 | "codemirror_mode": { 271 | "name": "ipython", 272 | "version": 3 273 | }, 274 | "file_extension": ".py", 275 | "mimetype": "text/x-python", 276 | "name": "python", 277 | "nbconvert_exporter": "python", 278 | "pygments_lexer": "ipython3", 279 | "version": "3.9.9" 280 | } 281 | }, 282 | "nbformat": 4, 283 | "nbformat_minor": 4 284 | } 285 | -------------------------------------------------------------------------------- /notebooks/5-Pytorch-Dataloader.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# To save time, start this download first, before reading through the examples.\n", 10 | "import torch, torchvision, os\n", 11 | "if not os.path.isfile('datasets/miniplaces/train/yard/00001000.jpg'):\n", 12 | " torchvision.datasets.utils.download_and_extract_archive(\n", 13 | " 'http://dissect.csail.mit.edu/datasets/miniplaces.zip',\n", 14 | " 'datasets', md5='bfabeb497c7eca01c74cd8441a9ac108')" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "Datasets and Dataloaders in pytorch\n", 29 | "===================================\n", 30 | "\n", 31 | "Data sets can be thought of as big arrays of data. If the data set is small enough (e.g., MNIST, which has 60,000 28x28 grayscale images), a dataset can be literally represented as an array - or more precisely, as a single pytorch tensor. With one number per pixel, MNIST takes about 200 megabytes of RAM, which fits comfortably into a modern computer.\n", 32 | "\n", 33 | "But larger-scale datasets like ImageNet or Places365 have more than a million higher-resolution full-color images. In these cases, an ordinary python array or pytorch tensor would require more than a terabyte of RAM, which is impractical on most computers.\n", 34 | "\n", 35 | "Instead, we need to load the data from disk (or SSD). Unfortunately, the latency of loading from disk is very slow compared to RAM, so we need to do the loading cleverly if we want to load the data quickly.\n", 36 | "\n", 37 | "To solve the problem, pytorch provides two classes:\n", 38 | " * `torch.utils.data.Dataset` - This very simple base class represents an array where the actual data may be slow to fetch, typically because the data is in disk files that require some loading, decoding, or other preprocessing. Pytorch provides a variety of different `Dataset` subclasses. For example, there is a handy one called `ImageFolder` that treats a directory tree of image files as an array of classified images.\n", 39 | " * `torch.utils.data.DataLoader` - This fancy class wraps a `Dataset` as a stream of data batches. Behind the scenes it uses a few techniques to feed the data faster. You do not need to subclass `DataLoader` - its purpose is to make a `Dataset` speedy." 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## Looking at an image data set using ImageFolder\n", 47 | "\n", 48 | "The most common `Dataset` used in computer vision is `ImageFolder`, which loads a set of image files from a directory tree. It treats every subdirectory of images as a classification category. To demonstrate it, we will use it to load images from the miniplaces dataset loaded above.\n", 49 | "\n", 50 | "**Directory layout.** Notice that `datasets/miniplaces/val` contains a set of 100 directories with names like `golf_course`. Each of these directories contains 100 images, each stored as a jpeg file: 10000 images in total." 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "ls datasets/miniplaces/val/golf_course" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "**Constructing an ImageFolder.** Making an ImageFolder at the root directory of the data set creats an object that behaves like an array: it has a length, and each entry contains a tuple with an image and a number. The image is stored as a `PIL` object which is a standard python object for images, and the number denotes the classification class - with one class for each folder, numbered in alphabetical order." 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "val_set = torchvision.datasets.ImageFolder('datasets/miniplaces/val')\n", 76 | "print('Length is', len(val_set))\n", 77 | "item = val_set[5100]\n", 78 | "print('5100th item is a pair', item)\n", 79 | "\n", 80 | "# Display the PIL image and the class name directly.\n", 81 | "display(item[0])\n", 82 | "print('Class name is', val_set.classes[item[1]])" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "**Transforming the PIL image into a pytorch tensor.** A PIL image is not convenient for training: we would prefer our data set to return pytorch tensors. So we can tell `ImageFolder` to do this by specifying the `transform` function on construction. Pytorch comes with a standard transform function `torchvision.transforms.ToTensor()` which converts an image to a pytorch tensor.\n", 90 | "\n", 91 | "Now when indexing into the data set, we will get a pytorch tensor instead of a PIL image." 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "val_set = torchvision.datasets.ImageFolder(\n", 101 | " 'datasets/miniplaces/val',\n", 102 | " transform=torchvision.transforms.ToTensor())\n", 103 | "print(val_set[1000])\n", 104 | "\n", 105 | "# There is an inverse transform that can be used to convert it back to a PIL image,\n", 106 | "# handy if we want to see it.\n", 107 | "as_image = torchvision.transforms.ToPILImage()\n", 108 | "display(as_image(val_set[1000][0]))" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "## Fast Dataset Access using DataLoader\n", 116 | "\n", 117 | "When we use a dataset for training, we will usually run through the whole dataset in batches. We could do this ourselves, as in line 6-8 below, by just fetching the images one at a time and grouping them.\n", 118 | "\n", 119 | "But a faster way to iterate through the dataset is to wrap our `val_set` object in a `torch.utils.data.DataLoader` object, as shown on line 14-18 below. The `val_loader` we get can magically pull data out of the Dataset much faster than doing it in the smiple way; the `DataLoader` class does this by using several threads to load and prefetch the data.\n", 120 | "\n", 121 | "The speedup will depend on the system and the number of threads you use (the number of threads to use is specified using `num_workers`). In practice using `DataLoader` will typically be 5-20 times faster than direct `Dataset` access." 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "import time\n", 131 | "\n", 132 | "print('Going over the data set as an array.')\n", 133 | "start = time.time()\n", 134 | "summed_image_dataset = 0\n", 135 | "batch_size = 100\n", 136 | "for i in range(0, len(val_set), batch_size):\n", 137 | " image_batch = torch.stack([val_set[i+j][0] for j in range(batch_size)])\n", 138 | " summed_image_dataset += image_batch.sum(0)\n", 139 | "end = time.time()\n", 140 | "print(f'Took {end - start} seconds')\n", 141 | "\n", 142 | "print('Going over the same dataset using a dataloader.')\n", 143 | "start = time.time()\n", 144 | "val_loader = torch.utils.data.DataLoader(\n", 145 | " val_set, batch_size=batch_size, num_workers=10)\n", 146 | "summed_image_loader = 0\n", 147 | "for image_batch, label_batch in val_loader:\n", 148 | " summed_image_loader += image_batch.sum(0)\n", 149 | "end = time.time()\n", 150 | "print(f'Took {end - start} seconds')\n", 151 | "\n", 152 | "print('Numerical difference is exactly', (summed_image_loader - summed_image_dataset).norm().item())" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "### Exercise\n", 160 | "\n", 161 | "1. Try adjusting `num_workers` down to 1 and up to 100. How does this affect the speed?\n", 162 | "2. Try changing `batch_size` down to 1 or up to 1000.\n", 163 | "\n", 164 | "**Note**: the speed differences you see will depend on the specifics of your system setup.\n", 165 | "If you are running on Google Colab, you may not see much of a speedup from DataLoader.\n", 166 | "This is because Colab provides a very low-latency virtual disk (so direct Dataset access\n", 167 | "is faster than on a regular computer), and a virtual CPU with very slow concurrency\n", 168 | "(so DataLoader multithreading is slower than normal)." 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [ 177 | "# TODO: copy the code above and alter:\n", 178 | "# 1. num_workers and note the changes in speed\n", 179 | "# 2. batch_size and note the changes in speed." 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "**Other common dataloader tricks.** `DataLoader` can do a few more useful things.\n", 187 | "\n", 188 | " * Although a DataLoader does not put batches on the GPU directly (because of multithreading limitations), it *can* put the batch in pinned memory, which is faster to copy to the GPU later after you get it out of the DataLoader. Make the DataLoader with `pin_memory=True` for this.\n", 189 | " * During training you usually do not want the batches in alphabetical order. The DataLoader can shuffle the batches so that they are randomized, instead of sequential. `shuffle=True` for this.\n", 190 | "\n", 191 | " \n", 192 | " " 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "## Using a DataLoader for Training\n", 200 | "\n", 201 | "We can put everything together by using the data from a data loader to train a classifier.\n", 202 | "\n", 203 | "The following is a simplistic example of training an image classifier. It uses the Adam optimizer and the ResNet-18 neural network architecture, and trains for a couple minutes, just passing once over the training set." 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "from tqdm import tqdm\n", 213 | "\n", 214 | "# Create a Dataset of miniplaces training images.\n", 215 | "train_set = torchvision.datasets.ImageFolder(\n", 216 | " 'datasets/miniplaces/train',\n", 217 | " torchvision.transforms.ToTensor())\n", 218 | "\n", 219 | "# Wrap the Dataset in a high-speed DataLoader with batch_size 100.\n", 220 | "train_loader = torch.utils.data.DataLoader(\n", 221 | " train_set, batch_size=100, num_workers=10,\n", 222 | " shuffle=True,\n", 223 | " pin_memory=True)\n", 224 | "\n", 225 | "# Create an untrained neural network using the ResNet 18 architecture.\n", 226 | "model = torchvision.models.resnet18(num_classes=100).cuda()\n", 227 | "\n", 228 | "# Set up the model for training using the Adam optimizer.\n", 229 | "model.train()\n", 230 | "optimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n", 231 | "\n", 232 | "# To train, optimize an objective on batches of training data.\n", 233 | "# Here we look at every training image once.\n", 234 | "for batch in tqdm(train_loader):\n", 235 | " images, labels = [d.cuda() for d in batch]\n", 236 | " optimizer.zero_grad()\n", 237 | " scores = model(images.cuda())\n", 238 | " loss = torch.nn.functional.cross_entropy(scores, labels)\n", 239 | " loss.backward()\n", 240 | " optimizer.step()" 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "metadata": {}, 246 | "source": [ 247 | "## Checking Accuracy with a Held-Out Dataset\n", 248 | "\n", 249 | "To check if network has learned anything useful, we can check whether the model can make good predictions on unseen images. The easy way to do this is to create a second `ImageFolder` dataset (and `DataLoader`) with a second set of images that was **not** used for training.\n", 250 | "\n", 251 | "While the achieved accuracy after a couple minutes of training is not perfect, it is already much better than random." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "metadata": {}, 258 | "outputs": [], 259 | "source": [ 260 | "# Create a validation dataset and data loader.\n", 261 | "val_set = torchvision.datasets.ImageFolder(\n", 262 | " 'datasets/miniplaces/val',\n", 263 | " torchvision.transforms.ToTensor())\n", 264 | "val_loader = torch.utils.data.DataLoader(\n", 265 | " val_set, batch_size=100, num_workers=10,\n", 266 | " pin_memory=True)\n", 267 | "\n", 268 | "# This function runs over the validation images and counts accurate predictions.\n", 269 | "def accuracy():\n", 270 | " model.eval()\n", 271 | " correct = 0\n", 272 | " for iter, batch in enumerate(val_loader):\n", 273 | " images, labels = [d.cuda() for d in batch]\n", 274 | " with torch.no_grad():\n", 275 | " scores = model(images.cuda())\n", 276 | " correct += (scores.max(1)[1] == labels).float().sum()\n", 277 | " return correct.item() / len(val_set)\n", 278 | "\n", 279 | "print(f'Accuracy on unseen images {accuracy() * 100}% (random guesses would be 1%)')" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "### Exercise\n", 287 | "\n", 288 | "1. For every 10th batch, display the first image in the batch.\n", 289 | "2. Also print the predicted class name and the true class name for that image.\n", 290 | "\n", 291 | "Hints:\n", 292 | "* Use the `as_image` function defined in a previous cell.\n", 293 | "* Use `images[0].cpu()` to move the image to the CPU before displaying it.\n", 294 | "* The prediction of the network for the 0th item of the batch is `scores.max(1)[1][0]`\n", 295 | "* Use `val_set.classes[pred]` to convert the numerical prediction to a readable label." 296 | ] 297 | }, 298 | { 299 | "cell_type": "markdown", 300 | "metadata": {}, 301 | "source": [ 302 | "## Improving Training using Data Augmentation\n", 303 | "\n", 304 | "One of the main ways to stretch a data set to make it more effective for training is to randomly adjust the images. For example if we randomly adjust the crop, color, or orientation of the image while loading, using the same image file multiple times will produce different training examples for the network. This is an easy way to increase the amount of training diversity in the data set without requring more actual images.\n", 305 | "\n", 306 | "To do data augmentation in a pytorch `Dataset`, you can specify more operations on `transform=` besides `ToTensor()`.\n", 307 | "\n", 308 | "In particular, there is a `Compose` transform that makes it easy to chain a series of data transformations; and `torchvision.transforms` includes a number of useful image transforms such as random resized crops and image flips.\n", 309 | "\n", 310 | "Here is an example:" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [ 319 | "# Create a Dataset of miniplaces training images.\n", 320 | "train_set = torchvision.datasets.ImageFolder(\n", 321 | " 'datasets/miniplaces/train',\n", 322 | " torchvision.transforms.Compose([\n", 323 | " torchvision.transforms.RandomCrop(112),\n", 324 | " torchvision.transforms.RandomHorizontalFlip(),\n", 325 | " torchvision.transforms.ToTensor(),\n", 326 | " ]))\n", 327 | "train_loader = torch.utils.data.DataLoader(\n", 328 | " train_set, batch_size=100, num_workers=10,\n", 329 | " shuffle=True,\n", 330 | " pin_memory=True)\n", 331 | "\n", 332 | "# Now let's train for one more epoch, and test the accuracy\n", 333 | "model.train()\n", 334 | "for batch in tqdm(train_loader):\n", 335 | " images, labels = [d.cuda() for d in batch]\n", 336 | " optimizer.zero_grad()\n", 337 | " scores = model(images.cuda())\n", 338 | " loss = torch.nn.functional.cross_entropy(scores, labels)\n", 339 | " loss.backward()\n", 340 | " optimizer.step()\n", 341 | "print(f'Accuracy on unseen images {accuracy() * 100}% (random guesses would be 1%)')" 342 | ] 343 | }, 344 | { 345 | "cell_type": "markdown", 346 | "metadata": {}, 347 | "source": [ 348 | "### Exercise\n", 349 | "\n", 350 | "1. Print out the same images as before, with updated predictions for the newly tuned network parameters.\n", 351 | "2. Repeat training for a few more epochs. How does the accuracy evolve?" 352 | ] 353 | }, 354 | { 355 | "cell_type": "markdown", 356 | "metadata": {}, 357 | "source": [ 358 | "### Epilog\n", 359 | "\n", 360 | "Almost all the pytorch code you will find will be variations and extensions of the patterns we have covered. You're ready to explore.\n", 361 | "\n", 362 | "Have fun!\n", 363 | "\n", 364 | "### [Back to the introduction →](1-Pytorch-Introduction.ipynb)" 365 | ] 366 | } 367 | ], 368 | "metadata": { 369 | "accelerator": "GPU", 370 | "kernelspec": { 371 | "display_name": "Python 3 (ipykernel)", 372 | "language": "python", 373 | "name": "python3" 374 | }, 375 | "language_info": { 376 | "codemirror_mode": { 377 | "name": "ipython", 378 | "version": 3 379 | }, 380 | "file_extension": ".py", 381 | "mimetype": "text/x-python", 382 | "name": "python", 383 | "nbconvert_exporter": "python", 384 | "pygments_lexer": "ipython3", 385 | "version": "3.9.9" 386 | } 387 | }, 388 | "nbformat": 4, 389 | "nbformat_minor": 4 390 | } 391 | -------------------------------------------------------------------------------- /notebooks/6-Pytorch-Alexnet-Example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "Pytorch Alexnet Example\n", 8 | "=======================\n", 9 | "\n", 10 | "This is a complete example of training an alexnet on pytorch, fully within notebook, and using nothing but widely-used library functions.\n", 11 | "\n", 12 | "Warning: this notebook download a full large-scale dataset (places365). That is too large to do in a practical way on Google Colab, so you need to host this notebook on your own server." 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": null, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "import torch, torchvision, os\n", 22 | "\n", 23 | "def train_alexnet_places(num_steps=100000):\n", 24 | " print(\"Making alexnet...\")\n", 25 | " alexnet = make_untrained_alexnet_places()\n", 26 | " alexnet.train()\n", 27 | " print(\"Loading datasets...\")\n", 28 | " train_loader, val_loader = get_train_and_val_data_loaders()\n", 29 | " print(\"Training classifier...\")\n", 30 | " checkpointer = make_checkpointing_function(val_loader, checkpoint_dir='checkpoints')\n", 31 | " train_classifier(alexnet, train_loader,\n", 32 | " max_iter=num_steps,\n", 33 | " momentum=0.9,\n", 34 | " init_lr=2e-2,\n", 35 | " weight_decay=5e-4,\n", 36 | " monitor=checkpointer)\n", 37 | " return alexnet" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "# Untrained Alexnet\n", 45 | "-----------------\n", 46 | "\n", 47 | "This function creates an untrained alexnet, with randomized parameters." 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "from torch import nn\n", 57 | "from collections import OrderedDict\n", 58 | "def make_untrained_alexnet_places():\n", 59 | " # channel widths\n", 60 | " w = [3, 96, 256, 384, 384, 256, 4096, 4096, 365]\n", 61 | " # Alexnet splits channels into groups\n", 62 | " groups = [1, 2, 1, 2, 2]\n", 63 | " model = nn.Sequential(OrderedDict([\n", 64 | " ('conv1', nn.Conv2d(w[0], w[1], kernel_size=11,\n", 65 | " stride=4,\n", 66 | " groups=groups[0], bias=True)),\n", 67 | " ('relu1', nn.ReLU(inplace=True)),\n", 68 | " ('pool1', nn.MaxPool2d(kernel_size=3, stride=2)),\n", 69 | " ('conv2', nn.Conv2d(w[1], w[2], kernel_size=5, padding=2,\n", 70 | " groups=groups[1], bias=True)),\n", 71 | " ('relu2', nn.ReLU(inplace=True)),\n", 72 | " ('pool2', nn.MaxPool2d(kernel_size=3, stride=2)),\n", 73 | " ('conv3', nn.Conv2d(w[2], w[3], kernel_size=3, padding=1,\n", 74 | " groups=groups[2], bias=True)),\n", 75 | " ('relu3', nn.ReLU(inplace=True)),\n", 76 | " ('conv4', nn.Conv2d(w[3], w[4], kernel_size=3, padding=1,\n", 77 | " groups=groups[3], bias=True)),\n", 78 | " ('relu4', nn.ReLU(inplace=True)),\n", 79 | " ('conv5', nn.Conv2d(w[4], w[5], kernel_size=3, padding=1,\n", 80 | " groups=groups[4], bias=True)),\n", 81 | " ('relu5', nn.ReLU(inplace=True)),\n", 82 | " ('pool5', nn.MaxPool2d(kernel_size=3, stride=2)),\n", 83 | " ('flatten', nn.Flatten()),\n", 84 | " ('fc6', nn.Linear(w[5] * 6 * 6, w[6], bias=True)),\n", 85 | " ('relu6', nn.ReLU(inplace=True)),\n", 86 | " ('dropout6', nn.Dropout()),\n", 87 | " ('fc7', nn.Linear(w[6], w[7], bias=True)),\n", 88 | " ('relu7', nn.ReLU(inplace=True)),\n", 89 | " ('dropout7', nn.Dropout()),\n", 90 | " ('fc8', nn.Linear(w[7], w[8]))\n", 91 | " ]))\n", 92 | " # Setup the initial parameters randomly\n", 93 | " for n, p in model.named_parameters():\n", 94 | " if 'bias' in n:\n", 95 | " torch.nn.init.zeros_(p)\n", 96 | " else:\n", 97 | " torch.nn.init.kaiming_normal_(p, nonlinearity='relu')\n", 98 | " model.cuda()\n", 99 | " model.train()\n", 100 | " return model" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "We can call the function to make a network, and then list all the network's trainable parameters." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "a = make_untrained_alexnet_places()\n", 117 | "for n, p in a.named_parameters():\n", 118 | " print(n, tuple(p.shape))" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "And we can save the uninitialized neural network." 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "torch.save(a.state_dict(), 'checkpoints/uninitialized_alexnet.pth')" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "Main Training Loop\n", 142 | "------------------\n", 143 | "\n", 144 | "This is a generic training loop for a classifier." 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "def train_classifier(model, train_data_loader, max_iter,\n", 154 | " momentum=0.9, init_lr=2e-2, weight_decay=5e-4,\n", 155 | " monitor=None):\n", 156 | " if monitor is not None:\n", 157 | " monitor(model, 0, 0.0, 0.0, 0)\n", 158 | " optimizer = torch.optim.SGD(\n", 159 | " model.parameters(),\n", 160 | " lr=init_lr, momentum=momentum, weight_decay=weight_decay)\n", 161 | " scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, init_lr, max_iter)\n", 162 | " iter_num = 0\n", 163 | " while iter_num < max_iter:\n", 164 | " for t_input, t_target in train_data_loader:\n", 165 | " # Copy data into the gpu\n", 166 | " input_var, target_var = [d.cuda() for d in [t_input, t_target]]\n", 167 | " # Evaluate model\n", 168 | " output = model(input_var)\n", 169 | " loss = torch.nn.functional.cross_entropy(output, target_var)\n", 170 | " # Perform one step of SGD\n", 171 | " optimizer.zero_grad()\n", 172 | " loss.backward()\n", 173 | " optimizer.step()\n", 174 | " scheduler.step() # Learning rate schedule\n", 175 | " # Check training set accuracy\n", 176 | " _, pred = output.max(1)\n", 177 | " batch_size = len(t_input)\n", 178 | " accuracy = target_var.detach().eq(pred).float().sum().item() / batch_size\n", 179 | " # Advance, and print out some stats\n", 180 | " iter_num += 1\n", 181 | " if monitor is not None:\n", 182 | " monitor(model, iter_num, loss, accuracy, batch_size)\n", 183 | " if iter_num >= max_iter:\n", 184 | " break" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "Data set\n", 192 | "--------\n", 193 | "\n", 194 | "This is the definition of the places data set used for training.\n", 195 | "If we do not have the files, we download them. And then we make a\n", 196 | "DataSet object that defines how to resize, crop, and normalize the images.\n", 197 | "\n", 198 | "The DataLoader objects wrap the dataset in a multithreaded streaming\n", 199 | "object that batches the image data and loads it quickly." 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "metadata": {}, 206 | "outputs": [], 207 | "source": [ 208 | "def get_places_data_set(split, crop_size=227, download=True):\n", 209 | " dirname = f'datasets/places/{split}'\n", 210 | " nfs_source = '/data/vision/torralba/datasets/places/files'\n", 211 | " web_source = 'https://dissect.csail.mit.edu/datasets/'\n", 212 | " if not os.path.exists(dirname) and download:\n", 213 | " if os.path.exists(nfs_source):\n", 214 | " os.symlink(nfs_source, 'datasets/places')\n", 215 | " else:\n", 216 | " os.makedirs(dirname, exist_ok=True)\n", 217 | " torchvision.datasets.utils.download_and_extract_archive(\n", 218 | " 'web_sources' +\n", 219 | " 'places_%s.zip' % split,\n", 220 | " 'datasets',\n", 221 | " md5=dict(val='593bbc21590cf7c396faac2e600cd30c',\n", 222 | " train='d1db6ad3fc1d69b94da325ac08886a01')[split])\n", 223 | " if split == 'train':\n", 224 | " cropping_rule = [\n", 225 | " torchvision.transforms.RandomCrop(227),\n", 226 | " torchvision.transforms.RandomHorizontalFlip() ]\n", 227 | " else:\n", 228 | " cropping_rule = [torchvision.transforms.CenterCrop(crop_size)]\n", 229 | " places_transform = torchvision.transforms.Compose([\n", 230 | " torchvision.transforms.Resize(256)\n", 231 | " ] + cropping_rule + [\n", 232 | " torchvision.transforms.ToTensor(),\n", 233 | " torchvision.transforms.Normalize(\n", 234 | " [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n", 235 | " ])\n", 236 | " return torchvision.datasets.ImageFolder(\n", 237 | " dirname, transform=places_transform)\n", 238 | "\n", 239 | "def get_train_and_val_data_loaders():\n", 240 | " return [\n", 241 | " torch.utils.data.DataLoader(\n", 242 | " get_places_data_set(split),\n", 243 | " batch_size=256, shuffle=(split == 'train'),\n", 244 | " num_workers=48, pin_memory=True)\n", 245 | " for split in ['train', 'val']\n", 246 | " ]" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "metadata": {}, 252 | "source": [ 253 | "Generic Evaluation and Checkpointing Utilities\n", 254 | "----------------------------------------------\n", 255 | "\n", 256 | " * **measure_val_accuracy_and_loss** evaluates the model on the holdout set and reports its performance.\n", 257 | " * **save_model_iteration** saves the current model parameters in a pytorch file.\n", 258 | " * **make_training_monitor** makes a callback function for periodically evaluating and saving a model during training.\n", 259 | " * **AverageMeter** tracks averages (e.g., average accuracy, average loss)." 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": null, 265 | "metadata": {}, 266 | "outputs": [], 267 | "source": [ 268 | "def measure_val_accuracy_and_loss(model, val_data_loader):\n", 269 | " '''\n", 270 | " Evaluates the model (in inference mode) on holdout data.\n", 271 | " '''\n", 272 | " model.eval()\n", 273 | " val_loss, val_acc = AverageMeter(), AverageMeter()\n", 274 | " for input, target in val_data_loader:\n", 275 | " input_var, target_var = [d.cuda() for d in [input, target]]\n", 276 | " with torch.no_grad():\n", 277 | " output = model(input_var)\n", 278 | " loss = torch.nn.functional.cross_entropy(output, target_var)\n", 279 | " _, pred = output.max(1)\n", 280 | " accuracy = (target_var.eq(pred)\n", 281 | " ).data.float().sum().item() / input.size(0)\n", 282 | " val_acc.update(accuracy, input.size(0))\n", 283 | " val_loss.update(loss.data.item(), input.size(0))\n", 284 | " return val_acc, val_loss\n", 285 | "\n", 286 | "def save_model_iteration(model, iter_num, checkpoint_dir):\n", 287 | " '''\n", 288 | " Saves the current parameters of the model to a file.\n", 289 | " '''\n", 290 | " torch.save(model.state_dict(), os.path.join(checkpoint_dir, f'iter_{iter_num}.pth'))\n", 291 | " \n", 292 | "def make_checkpointing_function(val_data_loader, checkpoint_dir=None, checkpoint_freq=100):\n", 293 | " '''\n", 294 | " Makes a callback to monitor training and make checkpoints.\n", 295 | " '''\n", 296 | " avg_train_accuracy, avg_train_loss = AverageMeter(), AverageMeter()\n", 297 | " def monitor(model, iter_num, loss, accuracy, batch_size):\n", 298 | " avg_train_accuracy.update(accuracy, batch_size)\n", 299 | " avg_train_loss.update(loss, batch_size)\n", 300 | " if iter_num % checkpoint_freq == 0:\n", 301 | " val_accuracy, val_loss = measure_val_accuracy_and_loss(model, val_data_loader)\n", 302 | " if checkpoint_dir is not None:\n", 303 | " save_model_iteration(model, iter_num, checkpoint_dir)\n", 304 | " print(f'Iter {iter_num}, ' + \n", 305 | " f'train acc {avg_train_accuracy.avg:.3g} loss {avg_train_loss.avg:.3g}, ' +\n", 306 | " f'val acc {val_accuracy.avg:.3g}, loss {val_loss.avg:.3g}')\n", 307 | " model.train()\n", 308 | " return monitor \n", 309 | " \n", 310 | "class AverageMeter(object):\n", 311 | " '''\n", 312 | " To keep running averages.\n", 313 | " '''\n", 314 | " def __init__(self):\n", 315 | " self.reset()\n", 316 | " def reset(self):\n", 317 | " self.val = 0.\n", 318 | " self.avg = 0.\n", 319 | " self.sum = 0.\n", 320 | " self.count = 0\n", 321 | " def update(self, val, n=1):\n", 322 | " self.val = val\n", 323 | " self.sum += val * n\n", 324 | " self.count += n\n", 325 | " if self.count:\n", 326 | " self.avg = self.sum / self.count" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": {}, 332 | "source": [ 333 | "Now do the work\n", 334 | "---------------\n", 335 | "\n", 336 | "Try loading alexnet from a checkpoint. If we have not yet saved a checkpoint snapshot with the number of iterations we want, then train it. " 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": null, 342 | "metadata": {}, 343 | "outputs": [], 344 | "source": [ 345 | "num_iterations = 100\n", 346 | "try:\n", 347 | " a = make_untrained_alexnet_places()\n", 348 | " a.load_state_dict(torch.load(f'checkpoints/iter_{num_iterations}.pth'))\n", 349 | "except:\n", 350 | " a = train_alexnet_places(num_iterations)" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "Now view one image - reverse the dataset normalization to get a nice image." 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [ 366 | "dsv = get_places_data_set('val')\n", 367 | "im, label = dsv[5000]\n", 368 | "im = im.cuda()\n", 369 | "# Reverse the normalization\n", 370 | "unnormalized = (im.cpu().permute(1, 2, 0)\n", 371 | " * torch.tensor([0.229, 0.224, 0.225])\n", 372 | " + torch.tensor([0.485, 0.456, 0.406]))\n", 373 | "\n", 374 | "from matplotlib import pyplot as plt\n", 375 | "plt.imshow(unnormalized)\n", 376 | "plt.axis('off')\n", 377 | "plt.show()" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "Finally, run the network on the function and print the prediction.\n", 385 | "\n", 386 | "Note the network expexts to work in batches, so `im[None]` forms an image batch of size one." 387 | ] 388 | }, 389 | { 390 | "cell_type": "code", 391 | "execution_count": null, 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "a.eval()\n", 396 | "output = a(im[None])\n", 397 | "pred = output.max(1)[1][0]\n", 398 | "\n", 399 | "print('prediction: ', dsv.classes[pred])\n", 400 | "print('groundtruth: ', dsv.classes[label])" 401 | ] 402 | } 403 | ], 404 | "metadata": { 405 | "accelerator": "GPU", 406 | "kernelspec": { 407 | "display_name": "Python 3", 408 | "language": "python", 409 | "name": "python3" 410 | }, 411 | "language_info": { 412 | "codemirror_mode": { 413 | "name": "ipython", 414 | "version": 3 415 | }, 416 | "file_extension": ".py", 417 | "mimetype": "text/x-python", 418 | "name": "python", 419 | "nbconvert_exporter": "python", 420 | "pygments_lexer": "ipython3", 421 | "version": "3.6.10" 422 | } 423 | }, 424 | "nbformat": 4, 425 | "nbformat_minor": 4 426 | } -------------------------------------------------------------------------------- /notebooks/autograd-graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/davidbau/how-to-read-pytorch/bbefbbd51834ac766f9d0a0ad09b69c1337521be/notebooks/autograd-graph.png -------------------------------------------------------------------------------- /notebooks/dataloader.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/davidbau/how-to-read-pytorch/bbefbbd51834ac766f9d0a0ad09b69c1337521be/notebooks/dataloader.png -------------------------------------------------------------------------------- /notebooks/how-to-read-pytorch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/davidbau/how-to-read-pytorch/bbefbbd51834ac766f9d0a0ad09b69c1337521be/notebooks/how-to-read-pytorch.png -------------------------------------------------------------------------------- /notebooks/ipynb_drop_output.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Suppress output and prompt numbers in git version control. 5 | 6 | This script will tell git to ignore prompt numbers and cell output 7 | when looking at ipynb files UNLESS their metadata contains: 8 | 9 | "git": { 10 | "keep_outputs": true 11 | }, 12 | 13 | The notebooks themselves are not changed. 14 | 15 | See also this blogpost: http://pascalbugnion.net/blog/ipython-notebooks-and-git.html. 16 | 17 | Usage instructions 18 | ================== 19 | 20 | 1. Put this script in a directory that is on the system's path. 21 | For future reference, I will assume you saved it in 22 | `~/scripts/ipynb_drop_output`. 23 | 2. Make sure it is executable by typing the command 24 | `chmod +x ~/scripts/ipynb_drop_output`. 25 | 3. Register a filter for ipython notebooks by 26 | putting the following line in `~/.config/git/attributes`: 27 | `*.ipynb filter=clean_ipynb` 28 | 4. Connect this script to the filter by running the following 29 | git commands: 30 | 31 | git config --global filter.clean_ipynb.clean ipynb_drop_output 32 | git config --global filter.clean_ipynb.smudge cat 33 | 34 | To tell git NOT to ignore the output and prompts for a notebook, 35 | open the notebook's metadata (Edit > Edit Notebook Metadata). A 36 | panel should open containing the lines: 37 | 38 | { 39 | "name" : "", 40 | "signature" : "some very long hash" 41 | } 42 | 43 | Add an extra line so that the metadata now looks like: 44 | 45 | { 46 | "name" : "", 47 | "signature" : "don't change the hash, but add a comma at the end of the line", 48 | "git" : { "keep_outputs" : true } 49 | } 50 | 51 | You may need to "touch" the notebooks for git to actually register a change, if 52 | your notebooks are already under version control. 53 | 54 | Notes 55 | ===== 56 | 57 | Changed by David Bau to make stripping output the default. 58 | 59 | This script is inspired by http://stackoverflow.com/a/20844506/827862, but 60 | lets the user specify whether the ouptut of a notebook should be kept 61 | in the notebook's metadata, and works for IPython v3.0. 62 | """ 63 | 64 | import sys 65 | import json 66 | 67 | nb = sys.stdin.read() 68 | 69 | json_in = json.loads(nb) 70 | nb_metadata = json_in["metadata"] 71 | keep_output = False 72 | if "git" in nb_metadata: 73 | if "keep_outputs" in nb_metadata["git"] and nb_metadata["git"]["keep_outputs"]: 74 | keep_output = True 75 | if keep_output: 76 | sys.stdout.write(nb) 77 | exit() 78 | 79 | 80 | ipy_version = int(json_in["nbformat"]) - 1 # nbformat is 1 more than actual version. 81 | 82 | 83 | def strip_output_from_cell(cell): 84 | if "outputs" in cell: 85 | cell["outputs"] = [] 86 | if "prompt_number" in cell: 87 | del cell["prompt_number"] 88 | if "execution_count" in cell: 89 | cell["execution_count"] = None 90 | 91 | 92 | if ipy_version == 2: 93 | for sheet in json_in["worksheets"]: 94 | for cell in sheet["cells"]: 95 | strip_output_from_cell(cell) 96 | else: 97 | for cell in json_in["cells"]: 98 | strip_output_from_cell(cell) 99 | 100 | json.dump( 101 | json_in, 102 | sys.stdout, 103 | sort_keys=True, 104 | indent=1, 105 | separators=(",", ": "), 106 | ensure_ascii=False, 107 | ) 108 | # https://stackoverflow.com/questions/729692/why-should-text-files-end-with-a-newline 109 | sys.stdout.write("\n") 110 | -------------------------------------------------------------------------------- /notebooks/setup_notebooks.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Start from directory of script 4 | cd "$(dirname "${BASH_SOURCE[0]}")" 5 | 6 | # Set up git config filters so huge output of notebooks is not committed. 7 | git config filter.clean_ipynb.clean "$(pwd)/ipynb_drop_output.py" 8 | git config filter.clean_ipynb.smudge cat 9 | git config filter.clean_ipynb.required true 10 | 11 | # Set up symlinks for the example notebooks 12 | for DIRNAME in datasets checkpoints 13 | do 14 | mkdir -p ../${DIRNAME} 15 | ln -sfn ../${DIRNAME} . 16 | done 17 | --------------------------------------------------------------------------------