├── LICENSE ├── README.md ├── attention ├── __init__.py └── attention.py ├── examples └── Pointer-Network-Argmin-Argmax.ipynb ├── setup.py ├── svgs ├── 190083ef7a1625fbc75f243cffb9c96d.svg ├── 1eb39a281b1e66935a51005b6beb9dbe.svg ├── 28e6b84adb66aca59d04ec9e227bfd3f.svg ├── 39c9d05724010ea29be9eb321b1422ec.svg ├── 39d2a848a943a7f5ec27272dad27c784.svg ├── 3cf4fbd05970446973fc3d9fa3fe3c41.svg ├── 5397f1268e113895a997a61e51165ffc.svg ├── a5a09669219f681bb51e176b190b0e4a.svg ├── a5d4c0a87edcc90e9dc7bb8a1845e86a.svg ├── da2cf8b162672dc46adcace06ec2740a.svg └── e73485aa867794d51ccd8725055d03a3.svg └── test └── test_attention.py /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2017, thom lake 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ```python 2 | def attend(query, context, value=None, score='dot', normalize='softmax', 3 | context_sizes=None, context_mask=None, return_weight=False 4 | ): 5 | """Attend to value (or context) by scoring each query and context. 6 | 7 | Args 8 | ---- 9 | query: Variable of size (B, M, D1) 10 | Batch of M query vectors. 11 | context: Variable of size (B, N, D2) 12 | Batch of N context vectors. 13 | value: Variable of size (B, N, P), default=None 14 | If given, the output vectors will be weighted 15 | combinations of the value vectors. 16 | Otherwise, the context vectors will be used. 17 | score: str or callable, default='dot' 18 | If score == 'dot', scores are computed 19 | as the dot product between context and 20 | query vectors. This Requires D1 == D2. 21 | Otherwise, score should be a callable: 22 | query context score 23 | (B,M,D1) (B,N,D2) -> (B,M,N) 24 | normalize: str, default='softmax' 25 | One of 'softmax', 'sigmoid', or 'identity'. 26 | Name of function used to map scores to weights. 27 | context_mask: Tensor of (B, M, N), default=None 28 | A Tensor used to mask context. Masked 29 | and unmasked entries should be filled 30 | appropriately for the normalization function. 31 | context_sizes: list[int], default=None, 32 | List giving the size of context for each item 33 | in the batch and used to compute a context_mask. 34 | If context_mask or context_sizes are not given, 35 | context is assumed to have fixed size. 36 | return_weight: bool, default=False 37 | If True, return the attention weight Tensor. 38 | 39 | Returns 40 | ------- 41 | output: Variable of size (B, M, P) 42 | If return_weight is False. 43 | weight, output: Variable of size (B, M, N), Variable of size (B, M, P) 44 | If return_weight is True. 45 | """ 46 | ``` 47 | 48 | Install 49 | ------- 50 | ```bash 51 | python setup.py install 52 | ``` 53 | 54 | Test 55 | ---- 56 | ```bash 57 | python -m pytest 58 | ``` 59 | Tested with pytorch 1.0.0 60 | 61 | About 62 | ----- 63 | Attention is used to focus processing on a particular region of input. 64 | The `attend` function provided by this package implements the most 65 | common attention mechanism [[1](#1), [2](#2), [3](#3), [4](#4)], which produces 66 | an output by taking a weighted combination of value vectors with weights 67 | from a scoring function operating over pairs of query and context vectors. 68 | 69 | Given query vector `q`, context vectors `c_1,...,c_n`, and value vectors 70 | `v_1,...,v_n` the attention score of `q` with `c_i` is given by 71 | 72 | ``` 73 | s_i = f(q, c_i) 74 | ``` 75 | 76 | Frequently `f` takes the form of a dot product between query and context vectors. 77 | 78 | ``` 79 | s_i = q^T c_i 80 | ``` 81 | 82 | The scores are passed through a normalization functions `g` (normally the softmax function). 83 | 84 | ``` 85 | w_i = g(s_1,...,s_n)_i 86 | ``` 87 | 88 | Finally, the output is computed as a weighted sum of the value vectors. 89 | 90 | ``` 91 | z = \sum_{i=1}^n w_i * v_i 92 | ``` 93 | 94 | In many applications [[1](#1), [4](#4), [5](#5)] attention is applied 95 | to the context vectors themselves, `v_i = c_i`. 96 | 97 | Sizes 98 | ----- 99 | This `attend` function provided by this package accepts 100 | batches of size `B` containing 101 | `M` query vectors of dimension `D1`, 102 | `N` context vectors of dimension `D2`, 103 | and optionally `N` value vectors of dimension `P`. 104 | 105 | Variable Length 106 | --------------- 107 | If the number of context vectors varies within a batch, a context 108 | can be ignored by forcing the corresponding weight to be zero. 109 | 110 | In the case of the softmax, this can be achieved by adding negative 111 | infinity to the corresponding score before normalization. 112 | Similarly, for elementwise normalization functions the weights can 113 | be multiplied by an appropriate {0,1} mask after normalization. 114 | 115 | To facilitate the above behavior, a context mask, with entries 116 | in `{-inf, 0}` or `{0, 1}` depending on the normalization function, 117 | can be passed to this function. The masks should have size `(B, M, N)`. 118 | 119 | Alternatively, a list can be passed giving the size of the context for 120 | each item in the batch. Appropriate masks will be created from these lists. 121 | 122 | Note that the size of output does not depend on the number of context vectors. 123 | Because of this, context positions are truly unaccounted for in the output. 124 | 125 | References 126 | ---------- 127 | #### [[1]](https://arxiv.org/abs/1409.0473) 128 | 129 | @article{bahdanau2014neural, 130 | title={Neural machine translation by jointly learning to align and translate}, 131 | author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, 132 | journal={arXiv preprint arXiv:1409.0473}, 133 | year={2014} 134 | } 135 | 136 | #### [[2]](https://arxiv.org/abs/1410.5401) 137 | @article{graves2014neural, 138 | title={Neural turing machines}, 139 | author={Graves, Alex and Wayne, Greg and Danihelka, Ivo}, 140 | journal={arXiv preprint arXiv:1410.5401}, 141 | year={2014} 142 | } 143 | 144 | #### [[3]](https://arxiv.org/abs/1503.08895) 145 | 146 | @inproceedings{sukhbaatar2015end, 147 | title={End-to-end memory networks}, 148 | author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others}, 149 | booktitle={Advances in neural information processing systems}, 150 | pages={2440--2448}, 151 | year={2015} 152 | } 153 | 154 | #### [[4]](https://distill.pub/2016/augmented-rnns/) 155 | 156 | @article{olah2016attention, 157 | title={Attention and augmented recurrent neural networks}, 158 | author={Olah, Chris and Carter, Shan}, 159 | journal={Distill}, 160 | volume={1}, 161 | number={9}, 162 | pages={e1}, 163 | year={2016} 164 | } 165 | 166 | #### [[5]](https://arxiv.org/abs/1506.03134) 167 | 168 | @inproceedings{vinyals2015pointer, 169 | title={Pointer networks}, 170 | author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep}, 171 | booktitle={Advances in Neural Information Processing Systems}, 172 | pages={2692--2700}, 173 | year={2015} 174 | } 175 | -------------------------------------------------------------------------------- /attention/__init__.py: -------------------------------------------------------------------------------- 1 | from . attention import attend 2 | -------------------------------------------------------------------------------- /attention/attention.py: -------------------------------------------------------------------------------- 1 | from torch import FloatTensor 2 | from torch.autograd import Variable 3 | from torch.nn.functional import sigmoid, softmax 4 | 5 | 6 | def mask3d(value, sizes): 7 | """Mask entries in value with 0 based on sizes. 8 | 9 | Args 10 | ---- 11 | value: Tensor of size (B, N, D) 12 | Tensor to be masked. 13 | sizes: list of int 14 | List giving the number of valid values for each item 15 | in the batch. Positions beyond each size will be masked. 16 | 17 | Returns 18 | ------- 19 | value: 20 | Masked value. 21 | """ 22 | v_mask = 0 23 | v_unmask = 1 24 | mask = value.data.new(value.size()).fill_(v_unmask) 25 | n = mask.size(1) 26 | for i, size in enumerate(sizes): 27 | if size < n: 28 | mask[i,size:,:] = v_mask 29 | return Variable(mask) * value 30 | 31 | 32 | def fill_context_mask(mask, sizes, v_mask, v_unmask): 33 | """Fill attention mask inplace for a variable length context. 34 | 35 | Args 36 | ---- 37 | mask: Tensor of size (B, N, D) 38 | Tensor to fill with mask values. 39 | sizes: list[int] 40 | List giving the size of the context for each item in 41 | the batch. Positions beyond each size will be masked. 42 | v_mask: float 43 | Value to use for masked positions. 44 | v_unmask: float 45 | Value to use for unmasked positions. 46 | 47 | Returns 48 | ------- 49 | mask: 50 | Filled with values in {v_mask, v_unmask} 51 | """ 52 | mask.fill_(v_unmask) 53 | n_context = mask.size(2) 54 | for i, size in enumerate(sizes): 55 | if size < n_context: 56 | mask[i,:,size:] = v_mask 57 | return mask 58 | 59 | 60 | def dot(a, b): 61 | """Compute the dot product between pairs of vectors in 3D Variables. 62 | 63 | Args 64 | ---- 65 | a: Variable of size (B, M, D) 66 | b: Variable of size (B, N, D) 67 | 68 | Returns 69 | ------- 70 | c: Variable of size (B, M, N) 71 | c[i,j,k] = dot(a[i,j], b[i,k]) 72 | """ 73 | return a.bmm(b.transpose(1, 2)) 74 | 75 | 76 | def attend(query, context, value=None, score='dot', normalize='softmax', 77 | context_sizes=None, context_mask=None, return_weight=False 78 | ): 79 | """Attend to value (or context) by scoring each query and context. 80 | 81 | Args 82 | ---- 83 | query: Variable of size (B, M, D1) 84 | Batch of M query vectors. 85 | context: Variable of size (B, N, D2) 86 | Batch of N context vectors. 87 | value: Variable of size (B, N, P), default=None 88 | If given, the output vectors will be weighted 89 | combinations of the value vectors. 90 | Otherwise, the context vectors will be used. 91 | score: str or callable, default='dot' 92 | If score == 'dot', scores are computed 93 | as the dot product between context and 94 | query vectors. This Requires D1 == D2. 95 | Otherwise, score should be a callable: 96 | query context score 97 | (B,M,D1) (B,N,D2) -> (B,M,N) 98 | normalize: str, default='softmax' 99 | One of 'softmax', 'sigmoid', or 'identity'. 100 | Name of function used to map scores to weights. 101 | context_mask: Tensor of (B, M, N), default=None 102 | A Tensor used to mask context. Masked 103 | and unmasked entries should be filled 104 | appropriately for the normalization function. 105 | context_sizes: list[int], default=None, 106 | List giving the size of context for each item 107 | in the batch and used to compute a context_mask. 108 | If context_mask or context_sizes are not given, 109 | context is assumed to have fixed size. 110 | return_weight: bool, default=False 111 | If True, return the attention weight Tensor. 112 | 113 | Returns 114 | ------- 115 | output: Variable of size (B, M, P) 116 | If return_weight is False. 117 | weight, output: Variable of size (B, M, N), Variable of size (B, M, P) 118 | If return_weight is True. 119 | 120 | 121 | About 122 | ----- 123 | Attention is used to focus processing on a particular region of input. 124 | This function implements the most common attention mechanism [1, 2, 3], 125 | which produces an output by taking a weighted combination of value vectors 126 | with weights from by a scoring function operating over pairs of query and 127 | context vectors. 128 | 129 | Given query vector `q`, context vectors `c_1,...,c_n`, and value vectors 130 | `v_1,...,v_n` the attention score of `q` with `c_i` is given by 131 | 132 | s_i = f(q, c_i) 133 | 134 | Frequently, `f` is given by the dot product between query and context vectors. 135 | 136 | s_i = q^T c_i 137 | 138 | The scores are passed through a normalization functions g. 139 | This is normally the softmax function. 140 | 141 | w_i = g(s_1,...,s_n)_i 142 | 143 | Finally, the output is computed as a weighted 144 | combination of the values with the normalized scores. 145 | 146 | z = sum_{i=1}^n w_i * v_i 147 | 148 | In many applications [4, 5] the context and value vectors are the same, `v_i = c_i`. 149 | 150 | Sizes 151 | ----- 152 | This function accepts batches of size `B` containing 153 | `M` query vectors of dimension `D1`, 154 | `N` context vectors of dimension `D2`, 155 | and optionally `N` value vectors of dimension `P`. 156 | 157 | Variable Length Contexts 158 | ------------------------ 159 | If the number of context vectors varies within a batch, a context 160 | can be ignored by forcing the corresponding weight to be zero. 161 | 162 | In the case of the softmax, this can be achieved by adding negative 163 | infinity to the corresponding score before normalization. 164 | Similarly, for elementwise normalization functions the weights can 165 | be multiplied by an appropriate {0,1} mask after normalization. 166 | 167 | To facilitate the above behavior, a context mask, with entries 168 | in `{-inf, 0}` or `{0, 1}` depending on the normalization function, 169 | can be passed to this function. The masks should have size `(B, M, N)`. 170 | 171 | Alternatively, a list can be passed giving the size of the context for 172 | each item in the batch. Appropriate masks will be created from these lists. 173 | 174 | Note that the size of output does not depend on the number of context vectors. 175 | Because of this, context positions are truly unaccounted for in the output. 176 | 177 | References 178 | ---------- 179 | [1](https://arxiv.org/abs/1410.5401) 180 | @article{graves2014neural, 181 | title={Neural turing machines}, 182 | author={Graves, Alex and Wayne, Greg and Danihelka, Ivo}, 183 | journal={arXiv preprint arXiv:1410.5401}, 184 | year={2014} 185 | } 186 | 187 | [2](https://arxiv.org/abs/1503.08895) 188 | 189 | @inproceedings{sukhbaatar2015end, 190 | title={End-to-end memory networks}, 191 | author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others}, 192 | booktitle={Advances in neural information processing systems}, 193 | pages={2440--2448}, 194 | year={2015} 195 | } 196 | 197 | [3](https://distill.pub/2016/augmented-rnns/) 198 | 199 | @article{olah2016attention, 200 | title={Attention and augmented recurrent neural networks}, 201 | author={Olah, Chris and Carter, Shan}, 202 | journal={Distill}, 203 | volume={1}, 204 | number={9}, 205 | pages={e1}, 206 | year={2016} 207 | } 208 | 209 | [4](https://arxiv.org/abs/1409.0473) 210 | 211 | @article{bahdanau2014neural, 212 | title={Neural machine translation by jointly learning to align and translate}, 213 | author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, 214 | journal={arXiv preprint arXiv:1409.0473}, 215 | year={2014} 216 | } 217 | 218 | [5](https://arxiv.org/abs/1506.03134) 219 | 220 | @inproceedings{vinyals2015pointer, 221 | title={Pointer networks}, 222 | author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep}, 223 | booktitle={Advances in Neural Information Processing Systems}, 224 | pages={2692--2700}, 225 | year={2015} 226 | } 227 | """ 228 | q, c, v = query, context, value 229 | if v is None: 230 | v = c 231 | 232 | batch_size_q, n_q, dim_q = q.size() 233 | batch_size_c, n_c, dim_c = c.size() 234 | batch_size_v, n_v, dim_v = v.size() 235 | 236 | if not (batch_size_q == batch_size_c == batch_size_v): 237 | msg = 'batch size mismatch (query: {}, context: {}, value: {})' 238 | raise ValueError(msg.format(q.size(), c.size(), v.size())) 239 | 240 | batch_size = batch_size_q 241 | 242 | # Compute scores 243 | if score == 'dot': 244 | s = dot(q, c) 245 | elif callable(score): 246 | s = score(q, c) 247 | else: 248 | raise ValueError(f'unknown score function: {score}') 249 | 250 | # Normalize scores and mask contexts 251 | if normalize == 'softmax': 252 | if context_mask is not None: 253 | s = context_mask + s 254 | 255 | elif context_sizes is not None: 256 | context_mask = s.data.new(batch_size, n_q, n_c) 257 | context_mask = fill_context_mask(context_mask, 258 | sizes=context_sizes, 259 | v_mask=float('-inf'), 260 | v_unmask=0 261 | ) 262 | s = context_mask + s 263 | 264 | s_flat = s.view(batch_size * n_q, n_c) 265 | w_flat = softmax(s_flat, dim=1) 266 | w = w_flat.view(batch_size, n_q, n_c) 267 | 268 | elif normalize == 'sigmoid' or normalize == 'identity': 269 | w = sigmoid(s) if normalize == 'sigmoid' else s 270 | if context_mask is not None: 271 | w = context_mask * w 272 | elif context_sizes is not None: 273 | context_mask = s.data.new(batch_size, n_q, n_c) 274 | context_mask = fill_context_mask(context_mask, 275 | sizes=context_sizes, 276 | v_mask=0, 277 | v_unmask=1 278 | ) 279 | w = context_mask * w 280 | 281 | else: 282 | raise ValueError(f'unknown normalize function: {normalize}') 283 | 284 | # Combine 285 | z = w.bmm(v) 286 | if return_weight: 287 | return w, z 288 | return z 289 | -------------------------------------------------------------------------------- /examples/Pointer-Network-Argmin-Argmax.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Pointer Network Attention Demo\n", 8 | "\n", 9 | "The below code trains a [pointer network](https://arxiv.org/abs/1506.03134) like architecture that takes as input a sequence of vectors and outputs the vector with the minimum or maximum value along a particular coordinate. In other words, a neural network version of argmax/argmin.\n", 10 | "\n", 11 | "### Setup\n", 12 | "\n", 13 | "Let $\\{\\mathbf{c}_i\\}_{i=1}^n = (\\mathbf{c}_1, \\ldots, \\mathbf{c}_n)$ be a sequence of $n$ vectors with each $\\mathbf{c}_i \\in \\mathbb{R}^d$. The minimum and maximum target positions, $i_\\min$ and $i_\\max$, for the sequence are given by\n", 14 | "\n", 15 | "$$\n", 16 | "\\begin{align*}\n", 17 | " i_\\min &= \\text{argmin}_i \\left\\{ x_{i, k_\\min}\\right\\} \\\\\n", 18 | " i_\\max &= \\text{argmax}_i \\left\\{ x_{i, k_\\max}\\right\\} \\\\\n", 19 | "\\end{align*}\n", 20 | "$$\n", 21 | "\n", 22 | "where $1 \\leq k_\\min \\neq k_\\max \\leq d$ are a priori chosen coordiantes along which to compute the minimum or maximum.\n", 23 | "\n", 24 | "### Model\n", 25 | "\n", 26 | "The model has the following form\n", 27 | "\n", 28 | "$$\n", 29 | "\\begin{align*}\n", 30 | " \\mathbf{u}_i &= A \\mathbf{c}_i & \\; i = 1, \\ldots, n \\\\\n", 31 | " \\mathbf{v} &= B \\mathbf{q} \\\\\n", 32 | " \\mathbf{p} &= \\text{softmax}_i(\\mathbf{v}^T \\mathbf{u}_i) \\\\\n", 33 | " \\mathbf{z} &= \\sum_i p_i \\mathbf{c}_i\n", 34 | "\\end{align*}\n", 35 | "$$\n", 36 | "\n", 37 | "where $A, B \\in \\mathbb{R}^{p \\times d}$ and $\\mathbf{q} \\in \\{\\mathbf{q}_\\min, \\mathbf{q}_\\max\\} \\subseteq \\mathbb{R}^d$ is a query vector indicating whether the model should output the minimum or maximum.\n", 38 | "\n", 39 | "The loss is defined as the mean squared error between the output and target vector.\n", 40 | "\n", 41 | "$$\n", 42 | "\\begin{align*}\n", 43 | " l &= \\frac{1}{n}\\sum_j (z_j - c_{t,j})^2\n", 44 | "\\end{align*}\n", 45 | "$$\n", 46 | "\n", 47 | "where $t$ is either $i_\\min$ or $i_\\max$.\n", 48 | "\n", 49 | "### Details\n", 50 | "The vectors $\\mathbf{q}_\\min$ and $\\mathbf{q}_\\max$ are initialized to random values sampled from $N(\\mathbb{0}, \\mathbb{I}_d)$ and held constant throughout training. The code below uses 10 dimensional context and query vectors and 7 dimensional hidden representations. The model is optimized over 1,600 training instances using RMSProp with mini batches of size 8. Each training instance contains betwen 5 and 14 context vectors. To better assess generalization validation instances are longer, containing between 15 and 24 context vectors." 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 1, 56 | "metadata": {}, 57 | "outputs": [ 58 | { 59 | "name": "stdout", 60 | "output_type": "stream", 61 | "text": [ 62 | "Seed: 1273\n" 63 | ] 64 | } 65 | ], 66 | "source": [ 67 | "%matplotlib inline\n", 68 | "import matplotlib.pyplot as plt\n", 69 | "import numpy as np\n", 70 | "import seaborn as sns\n", 71 | "\n", 72 | "import torch\n", 73 | "from torch.nn import Linear, Module\n", 74 | "\n", 75 | "from attention import attend\n", 76 | "\n", 77 | "\n", 78 | "seed = sum(map(ord, 'les bons mots'))\n", 79 | "np.random.seed(seed)\n", 80 | "torch.manual_seed(seed)\n", 81 | "print(f'Seed: {seed}')" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 2, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "class Data(object):\n", 91 | " dim = 10\n", 92 | " min_position, max_position = 3, 7\n", 93 | " q_min = np.random.normal(0, 1, dim).astype(np.float32)\n", 94 | " q_max = np.random.normal(0, 1, dim).astype(np.float32)\n", 95 | " query = np.row_stack([q_min, q_max])\n", 96 | "\n", 97 | " @staticmethod\n", 98 | " def create_minibatches(n, m, min_length, max_length):\n", 99 | " assert 0 < min_length <= max_length\n", 100 | "\n", 101 | " minibatches = []\n", 102 | " for i in range(n):\n", 103 | " lengths = np.random.randint(min_length, max_length + 1, m)\n", 104 | " context = np.zeros((m, lengths.max(), Data.dim), dtype=np.float32)\n", 105 | " target = np.zeros((m, 2, Data.dim), dtype=np.float32)\n", 106 | " target_indices = []\n", 107 | "\n", 108 | " for j, length in enumerate(lengths):\n", 109 | " c = np.random.normal(0, 1, (length, Data.dim))\n", 110 | " k_min = np.argmin(c[:,Data.min_position])\n", 111 | " k_max = np.argmax(c[:,Data.max_position])\n", 112 | " target_min = c[k_min]\n", 113 | " target_max = c[k_max]\n", 114 | " context[j,:length] = c\n", 115 | " target[j,0] = target_min\n", 116 | " target[j,1] = target_max\n", 117 | " target_indices.append((k_min, k_max))\n", 118 | "\n", 119 | " query = torch.from_numpy(np.tile(Data.query, (m, 1, 1)))\n", 120 | " context = torch.from_numpy(context)\n", 121 | " target = torch.from_numpy(target)\n", 122 | " minibatches.append((query, context, target, lengths, target_indices))\n", 123 | " return minibatches" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 3, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "class PointerNet(Module):\n", 133 | " def __init__(self, n_hidden):\n", 134 | " super().__init__()\n", 135 | " self.n_hidden = n_hidden\n", 136 | " self.f = Linear(Data.dim, n_hidden)\n", 137 | " self.g = Linear(Data.dim, n_hidden)\n", 138 | "\n", 139 | " def forward(self, q, x, lengths=None, **kwargs):\n", 140 | " batch_size_q, n_queries, dim_q = q.size()\n", 141 | " batch_size_x, n_inputs, dim_x = x.size()\n", 142 | " assert batch_size_q == batch_size_x\n", 143 | " assert dim_q == dim_x\n", 144 | " batch_size = batch_size_q\n", 145 | " dim = dim_q\n", 146 | "\n", 147 | " q_flat = q.view(batch_size*n_queries, dim)\n", 148 | " u_flat = self.f(q_flat)\n", 149 | " u = u_flat.view(batch_size, n_queries, self.n_hidden)\n", 150 | "\n", 151 | " x_flat = x.view(batch_size*n_inputs, dim)\n", 152 | " v_flat = self.g(x_flat)\n", 153 | " v = v_flat.view(batch_size, n_inputs, self.n_hidden)\n", 154 | "\n", 155 | " return attend(u, v, value=x, context_sizes=lengths, **kwargs)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 4, 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [ 164 | "batch_size = 8\n", 165 | "\n", 166 | "n_train = 200\n", 167 | "min_length_train, max_length_train = 5, 14\n", 168 | "train_batches = Data.create_minibatches(n_train, batch_size, min_length_train, max_length_train)\n", 169 | "\n", 170 | "n_valid = 100\n", 171 | "min_length_valid, max_length_valid = 15, 24\n", 172 | "valid_batches = Data.create_minibatches(n_valid, batch_size, min_length_valid, max_length_valid)" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 5, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "net = PointerNet(7)\n", 182 | "opt = torch.optim.RMSprop(net.parameters(), lr=0.001)\n", 183 | "mse = torch.nn.MSELoss()" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": 6, 189 | "metadata": {}, 190 | "outputs": [ 191 | { 192 | "name": "stdout", 193 | "output_type": "stream", 194 | "text": [ 195 | "[ 1] 0.647\n", 196 | "[ 2] 0.264\n", 197 | "[ 3] 0.162\n", 198 | "[ 4] 0.120\n", 199 | "[ 5] 0.096\n", 200 | "[ 6] 0.081\n", 201 | "[ 7] 0.070\n", 202 | "[ 8] 0.063\n", 203 | "[ 9] 0.057\n", 204 | "[10] 0.052\n" 205 | ] 206 | } 207 | ], 208 | "source": [ 209 | "epoch = 0\n", 210 | "max_epochs = 10\n", 211 | "while epoch < max_epochs:\n", 212 | " sum_loss = 0\n", 213 | " for query, context, target, lengths, target_indices in train_batches:\n", 214 | " net.zero_grad()\n", 215 | " output = net(query, context, lengths=lengths)\n", 216 | " loss = mse(output, target)\n", 217 | " loss.backward()\n", 218 | " opt.step()\n", 219 | " sum_loss += loss.item()\n", 220 | " epoch += 1\n", 221 | " print('[{:2d}] {:5.3f}'.format(epoch, sum_loss / n_train))" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": 7, 227 | "metadata": {}, 228 | "outputs": [ 229 | { 230 | "name": "stdout", 231 | "output_type": "stream", 232 | "text": [ 233 | "valid loss: 0.059\n", 234 | "valid error min: 0.018\n", 235 | "valid error max: 0.015\n" 236 | ] 237 | } 238 | ], 239 | "source": [ 240 | "sum_loss = 0\n", 241 | "sum_error_min = 0\n", 242 | "sum_error_max = 0\n", 243 | "\n", 244 | "for query, context, target, lengths, target_indices in valid_batches:\n", 245 | " with torch.no_grad():\n", 246 | " weight, output = net(query, context, lengths=lengths, return_weight=True)\n", 247 | " loss = mse(output, target)\n", 248 | "\n", 249 | " sum_loss += loss.item()\n", 250 | " weight = weight.data.numpy()\n", 251 | "\n", 252 | " for i, (i_min_true, i_max_true) in enumerate(target_indices):\n", 253 | " w_min, w_max = weight[i]\n", 254 | " i_min_pred = w_min.argmax()\n", 255 | " i_max_pred = w_max.argmax()\n", 256 | " sum_error_min += int(i_min_true != i_min_pred)\n", 257 | " sum_error_max += int(i_max_true != i_max_pred)\n", 258 | "\n", 259 | "print('valid loss: {:5.3f}'.format(sum_loss / n_valid))\n", 260 | "print('valid error min: {:5.3f}'.format(sum_error_min / (n_valid * batch_size)))\n", 261 | "print('valid error max: {:5.3f}'.format(sum_error_max / (n_valid * batch_size)))" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 8, 267 | "metadata": {}, 268 | "outputs": [ 269 | { 270 | "data": { 271 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr8AAALICAYAAAB2PpiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8VNX9//HXmSUECCCrQFhV9kWWqLijCIJVoBYtLS6ttfitxa39fbVq/f60/bba2vprrbWKy5dWqYooghsCflFEsRLQKqsoBNlJWEKALLOc3x+ThITJMsncWch9Px8PyMzce88952bymc+ce+65xlqLiIiIiIgbeFJdARERERGRZFHyKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn7lhGOMecIYc5/T64qISP2MMYeNMaekuh4ijWU0z6+kE2NMHtAV6GqtLajy+mfA6UBva21eamonIpL+FEdF6qaeX0lHW4DvVTwxxgwBmqeuOiIiJxzFUZFaKPmVdPQccF2V59cD/6h4YoyZZYz57/LHo40x240xPzfG7DXG7DLG/LCede+ssu5kY8xlxpgvjTH7jTH31LRt1e2rPM8zxvynMeZzY8wRY8wzxpiTjTFvG2OKjDFLjDFtE3KERETqVl8c/ZYx5lNjzCFjzDZjzP1Vln3XGLPZGNO6/PkEY8xuY0zH8ufWGHNa+eNZxpjHy+PeYWPMh8aYzsaYPxljDhhjNhhjhlcpu3LbKts3KkaLNJaSX0lHHwOtjTEDjDFe4LvA83Ws3xloA2QDPwL+WkfS2RnILF/3v4CngGuAkcD5wH81cCzbd4CxQF/gCuBt4B6gA5G/r1sbUJaIiFPqi6NHiCTHJwHfAn5ijJkMYK19CVgBPGqMaQ88A9xorc2vZV9XA78kEvdKy7ddXf58LvBIA+rtdIwWiaLkV9JVRa/FWGADsKOOdQPAr6y1AWvtW8BhoF8d6/7GWhsAXiQSnP9srS2y1q4F1gJDG1DPv1hr91hrdwAfAP+y1n5qrS0F5gHD695cRCRhao2j1tr3rLVfWGvD1trPgReAC6ts+1PgYuA94HVr7Rt17GeetXaVtbaESNwrsdb+w1obAl6iYXHQ6RgtEsWX6gqI1OI5YBnQmyqn6mqxz1obrPL8KJBVx7qh8sfF5T/3VFleXMe2NTl+23jKEhFxUq1x1BhzFvAQMBjIAJoBL1cst9YeNMa8DPyMyBmuujgZB52O0SJR1PMraclau5XIBRuXAa+mqBpHgBZVnndOUT1ERBqsnjj6T2AB0N1a2wZ4AjAVC40xw4AbiPQIP+pgtY6iuCoppuRX0tmPgIuttUdStP/PgMuMMe2MMZ2B21NUDxGRxqotjrYC9ltrS4wxZwLfr1hgjMkkMj74HuCHQLYx5maH6vMZ8H1jjNcYM57qQy1EkkLJr6Qta+3X1trcFFbhOeDfQB6wiMjYNRGRE0YdcfRm4FfGmCIiF5bNqbLsQWC7tfZv5dcvXAP8tzGmjwNVuo3IxcEHgWnAaw6UKdIgusmFiIiIiLiGen5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6RkptcjB8/3i5cuDAVu5YUm7GoEIDHxrVJcU1E0oKpf5X6KaZKU6DPB4lTzPE0JT2/BQUFqditiEiTpJgqIhI7DXsQEREREddQ8isiIiIirqHkV0RERERcI+4L3owx3YF/AJ2BMDDTWvvneMsVkdQIBAJs376dkpKSVFelycjMzKRbt274/f5UV0VEkkBxNHGciKdOzPYQBH5urV1tjGkFrDLGLLbWrnOgbBFJsu3bt9OqVSt69eqFMY5MRuBq1lr27dvH9u3b6d27d6qrIyJJoDiaGE7F07iHPVhrd1lrV5c/LgLWA9nxlisiqVFSUkL79u0VsB1ijKF9+/bqARJxEcXRxHAqnjo65tcY0wsYDvyrhmXTjTG5xpjc/Px8J3crIg5TwHZWIo6nYqpIelMcTQwnjqtjya8xJgt4BbjdWnvo+OXW2pnW2hxrbU7Hjh2d2q2IiCsppoqINI4jya8xxk8k8Z1trX3ViTJFROqyYMECHnrooYTuIysrK6Hli4i4xXvvvcfll1+e6moAzsz2YIBngPXW2kfir5KISP0mTpzIxIkTU10NERE5wTjR83sucC1wsTHms/J/lzlQroi4UF5eHv379+fGG29k8ODBTJs2jSVLlnDuuefSp08fPvnkEwBmzZrFjBkzAPjBD37ArbfeyjnnnMMpp5zC3Llzo8q96667ePzxxyuf33///fzxj3/k8OHDjBkzhhEjRjBkyBDmz58fte3xPRYzZsxg1qxZAKxatYoLL7yQkSNHcumll7Jr1y4nD4eISIPFGkc/+eQTzjnnHIYPH84555zDxo0bAXjkkUe44YYbAPjiiy8YPHgwR48erbaPs846i7Vr11Y+Hz16NKtWraq1zKruv/9+/vCHP1Q+Hzx4MHl5eQA8//zznHnmmQwbNoybbrqJUCjk6LEBB3p+rbXLAY3qlgb79oIRMa87b+LqBNZEavPn3CNs2h90tMw+7XzcltOyznW++uorXn75ZWbOnMkZZ5zBP//5T5YvX86CBQv47W9/y2uvvRa1za5du1i+fDkbNmxg4sSJTJkypdryqVOncvvtt3PzzTcDMGfOHBYuXEhmZibz5s2jdevWFBQUMGrUKCZOnBjTRRWBQIBbbrmF+fPn07FjR1566SXuvfdenn322QYcEZGmL9Z4n525NME1Sb50jqP9+/dn2bJl+Hw+lixZwj333MMrr7zC7bffzujRo5k3bx6/+c1vePLJJ2nRokW18qdOncqcOXN44IEH2LVrFzt37mTkyJEcOnSoxjJjsX79el566SU+/PBD/H4/N998M7Nnz+a6665r9LGqiRPz/IpII8X6geC25L93794MGTIEgEGDBjFmzBiMMQwZMqSyd+B4kydPxuPxMHDgQPbs2RO1fPjw4ezdu5edO3eSn59P27Zt6dGjB4FAgHvuuYdly5bh8XjYsWMHe/bsoXPnzvXWc+PGjaxZs4axY8cCEAqF6NKlS+MbLiLikFjiaGFhIddffz2bNm3CGEMgEADA4/Ewa9Yshg4dyk033cS5554bVf7VV1/N2LFjeeCBB5gzZw5XXXVVnWXG4t1332XVqlWcccYZABQXF9OpU6d4DkONlPyKSK3q61lIlGbNmlU+9ng8lc89Hg/BYM09KFW3sdbWuM6UKVOYO3cuu3fvZurUqQDMnj2b/Px8Vq1ahd/vp1evXlFzSPp8PsLhcOXziuXWWgYNGsSKFSsa0UoRcYN0jqP33XcfF110EfPmzSMvL4/Ro0dXbrNp0yaysrLYuXNnjeVnZ2fTvn17Pv/8c1566SWefPLJesusUFdMvf7663nwwQfjant9lPymkHr9RJJr6tSp/PjHP6agoID3338fiPRSdOrUCb/fz9KlS9m6dWvUdj179mTdunWUlpZSUlLCu+++y3nnnUe/fv3Iz89nxYoVnH322QQCAb788ksGDRqU7KaJiDRYYWEh2dmR+5JVXMdQ8fptt93GsmXLmDFjBnPnzo0aSgaRmPr73/+ewsLCyl7m2sqsqlevXrzxxhsArF69mi1btgAwZswYJk2axB133EGnTp3Yv38/RUVF9OzZ06kmAw7f5EKkPsZaqKVXTiTRBg0aRFFREdnZ2ZXDE6ZNm0Zubi45OTnMnj2b/v37R23XvXt3rr76aoYOHcq0adMYPnw4ABkZGcydO5e77rqL008/nWHDhvHRRx8ltU0iTUGb0kyu+WoYV679gHD+gVRXxzXuvPNO7r77bs4999xqF5bdcccd3HzzzfTt25dnnnmGX/ziF+zduzdq+ylTpvDiiy9y9dVX11tmVd/5znfYv38/w4YN429/+xt9+/YFYODAgfz3f/8348aNY+jQoYwdOzYhFxGb2k4PJlJOTo7Nzc1N+n7TjZt6fm3YEnrvE/YtXkmJL4NuV41mSt7UmLdvCsegJun4Hli/fj0DBgxI2v7copbj6sjFwoqpku7qinVdj7Tm7s8vpENp+fCA5s3I+NGVeE7pnqTaOU9xNLHijafq+ZWEsyWlBGbNI/jG+7QpPcrJRw4SmPUak7YOBHUCi4i4Vt/CDjzw6ZhjiS9AcSllT8wh9PmXqauYNGlKfiWhwnv2Ufan5wiv+Spq2dQtQ/nJhrPwhfU2FBFxm5z8bO7992iygs2iFwZDBP7+GsHlTfOsn6SWLnhzgVSdWg99/iWBF96E0tqnOblgT286lWTxyKDlFGWUOrr/ZEvHIQwiIunokh2n8sNNI/HU1QdnIfjqEuyhw/gmnB/T3NsisVDyK46z4TDBhcsJLfk4atmOVu05qeQwLQPHEt3+hR359eqxPDxkGTtaHkpmVUVEEk439KnCwlV5Q7hya/SMKHN6fYHNmsF31y2D8LExcaElH2MLD+O/+lKM15vM2koT1eSTXwWd5LJHignMfoPwhi1Ry7xnDOYvJ53FSSVHuHvDO9gqV/SeXJLFr1Zfwp8GfcQX7XYns8oiIpIE3rDhR1+ewUW7T6n2eogwz/TNZWnXzWRn9ueaUR0I/GMBlB07axheuYZA0RH810/CNMtIdtWliWnyya8kT3jHXgKzXsPuO1h9gceD79tj8J4zjODiQxS0bEPGbdey6g/3MvjgyZWrtQhlcNfnF/D3PqtZnB09RlhExC3SueOmMUO8moW83Lb2XIbv71ptnVJPkD8P/IhPOxy7kYJ34KmYn0yl7Om5cKS48vXwhi2UPf4iGTd+B9MqNTeOkKbhhEt+3T6uMl3bH1q9jsBLCyFw3N23WrUk4/pJeE7pVu1l0yKTh4a+xw835TBm16mVr3vxcMOmHLoebc1zp35K2KPpIOSY3/72t9xzzz0AHDx4kH/+85/cfPPNjS5v1qxZjBs3jq5dIx/IN954Iz/72c8YOHCgI/UVEbCHj/LLzy7mtKL21V4v8pXy+yHL+KrNvqhtPD27kHHrNAJPvozdX3isrG27KXt0Nv6brsLToW3C694UKY6egMlvukrnb+mJZENhgm+8R+j96DlGTa+uZFw/CdOmVY3bhjyWp/uuZGeLQ0z7ehieKlP0jd/Rl5OLs/jLwI8o9h1LqN16nCXi+KD9+OOPxx20Bw8eXBm0n376aUfqKSIR4X0HCcx8OSrx3Zt5mIeGvs+uFkW1buvp2I6M266h7Km52O17Kl+3+w5S9uhsMn78HTzdu1TbJl07iNKJ4qiS3xopwYqNPXyUwD8WEP7qm6hl3nOG4Zs8BuOr5+IEA29138ju5kXcsu5sMsP+ykXD93flgdWX8PCQD5yuer0UQKHkZ79PaPmZj9xZ5/LJkyezbds2SkpKuO2229i8eTPFxcUMGzaMQYMGEQqF+Prrrxk2bBhjx47l4Ycf5uGHH2bOnDmUlpby7W9/mwceeIC8vDwmTJjAeeedx0cffUR2djbz58/nzTffJDc3l2nTptG8eXNWrFjBhAkT+MMf/kBOTg4vvPACv/3tb7HW8q1vfYvf/e53AGRlZXHbbbfxxhtv0Lx5c+bPn8/JJ59cZ1vSlWKdJFJ4+27KnnoFio5Ue31L1n5+N2QZhc1K6i3DtGpJxs1TCfx9PuGNeccWHD5K2V9fxP+DyXj793a45s5RHE3POOpI8muMGQ/8GfACT1trH3Ki3IbyhA3djrYGILwzchu+HofbxLx9um5TsX7Ct9m2C1tSBqWRf7a0DErKf9bwmt27v9p4LAC8XnxTxuI7a2hM+6ywusNO/u+Id/nPL86vNtl596Mn8evVYwkN+RpzUqtGHefGSLvfTfk2pmM7jN8d31mfffZZ2rVrR3FxMWeccQbvv/8+jz32GJ999hkAeXl5rFmzpvL5okWL2LRpE5988gnWWiZOnMiyZcvo0aMHmzZt4oUXXuCpp57i6quv5pVXXuGaa67hscceqwzSVe3cuZO77rqLVatW0bZtW8aNG8drr73G5MmTOXLkCKNGjeI3v/kNd955J0899RS//OUvk358kqFVWQZty5oD8f09uVKVEVs9Dp8U82bhHRWfDw3YJsm/m1jiVuejrSj76wtRU11+0XY3/2/Q8mpn9OpjMpvh/9F3CMxZSDh37bEFZQECT7+CvfISPL26xlw3KD9mPh+eTu1irseJSHG0ZnF/ihpjvMBfgbHAdmClMWaBtXZdvGU3VIuQn9/lTgCgLHcWAL9jQszbp+s2FesnfpvnYlqvVie1IuMHk/H06FL/ujX4Jusg941YzM/XnF/tFFmbQCaBp18BGnecGyP9fjeRbTJ+8SNMp/Z1r9xEPProo8ybNw+Abdu2sWnTpjrXX7RoEYsWLWL48OEAHD58mE2bNtGjRw969+7NsGHDABg5ciR5eXl1lrVy5UpGjx5Nx44dAZg2bRrLli1j8uTJZGRkcPnll1eWtXjx4niamdZG5ffghk2RD7R4/p7c7neMj3ndY58PDd8mWWKPw9UT3+Wd8nii/yeEPOEG79P4vPi/dxnB1lmE/vdfxxaEwwTnLmpw3cpyZ2G6dqLZ//lBg+tyIlEcrZkTXUhnAl9ZazcDGGNeBCYBSU9+JXU8p3bHf93EuK/APdishF8N+19u3nAWo/J7OFQ7OdG89957LFmyhBUrVtCiRQtGjx5NSUndp0ittdx9993cdNNN1V7Py8ujWbNjd5Dyer0UFxcfv3lUWbXx+/2Vk+17vV6Cwdh7sETc6vXu63nhlH9j47hPhTEG/+UXYtpkEXzt3Wq96xJNcbR2TiS/2cC2Ks+3A2cdv5IxZjowHaBHj8YnNXWNO7NHiin98C+NLlsawefDe/4IfJddwJVv5tS/PpCduRSo53cZtgTfWU5o8QpHqikNV99YskQqLCykbdu2tGjRgg0bNvDxx5Ebpvj9fgKBAH6/n1atWlFUdOximUsvvZT77ruPadOmkZWVxY4dO/D7/bXtAiCqjApnnXUWt912GwUFBbRt25YXXniBW265xdlGxsmJmFrfON7gh58S3NR0e7YlOXyTLuaqC+/kqnrWm7GosNrzusaknzWgOz9dPwq/Te+bXiiOpmccdSL5rel7XFS6b62dCcwEyMnJScz3NY/BdOmYkKJdwWOgWQYmM4PlBUsp9gYp8QYiP32BY8+rPH7kyrcxzWu4L3ucjMfgn3A+nq6dCC7LhZIyx/dxQvK5Y7zv+PHjeeKJJxg6dCj9+vVj1KhRAEyfPp2hQ4cyYsQIZs+ezbnnnsvgwYOZMGECDz/8MOvXr+fss88GIhdUPP/883jruCPUD37wA/7jP/6j8kKNCl26dOHBBx/koosuwlrLZZddxqRJkxLb6AZKRkw1LTJPqJi69VDdp3Qr9GzdJ+nbJEPataV5M3wXnYl30GkN37Ye/+q0jf3NjnJV3hCGZg5pVBmmY9OeKk1xtHamrm7pmAow5mzgfmvtpeXP7waw1j5Y2zY5OTk2Nzd6aixJH42Z7SDWbbIzl/LYuNgvXJPkWr9+PQMGDEh1NZqcWo5rHCeBj1FMjUhk3Ip3m2Q40dtS0fNb8fmQTnVrKMXRxIo3njrRjbQS6GOM6Q3sAKYC33egXBEREYlRY5LAdEwcRRIt7uTXWhs0xswA3iEy1dmz1tq19WwmIiIiIpJ0jgwgtNa+BbzlRFkiIiIiIonijqtnRESkydMpfBGJhSfVFRARERERSRYlvyIiIiLiGhr2IDXS6UOpEOt0Q7FKxHsrKyuLw4cPO16uiIgTFEfTi3p+RURERMQ1lPyKSNq56667ePzxxyuf33///TzwwAOMGTOGESNGMGTIEObPnx+13Xvvvcfll19e+XzGjBnMmjULgFWrVnHhhRcycuRILr30Unbt2gXAo48+ysCBAxk6dChTp05NbMNERJJEcbR2GvYgjonlNMzx924XqcnUqVO5/fbbufnmmwGYM2cOCxcu5I477qB169YUFBQwatQoJk6ciDH139QnEAhwyy23MH/+fDp27MhLL73Evffey7PPPstDDz3Eli1baNasGQcPHkx000QkRhp+Fx/F0dop+RWRtDN8+HD27t3Lzp07yc/Pp23btnTp0oU77riDZcuW4fF42LFjB3v27KFz5871lrdx40bWrFnD2LFjAQiFQnTp0gWAoUOHMm3aNCZPnszkyZMT2i5pGpSUyYlAcbR2Sn5FJC1NmTKFuXPnsnv3bqZOncrs2bPJz89n1apV+P1+evXqRUlJSbVtfD4f4XC48nnFcmstgwYNYsWKFVH7efPNN1m2bBkLFizg17/+NWvXrsXnU2gUkROf4mjNNOZXRNLS1KlTefHFF5k7dy5TpkyhsLCQTp064ff7Wbp0KVu3bo3apmfPnqxbt47S0lIKCwt59913AejXrx/5+fmVQTsQCLB27VrC4TDbtm3joosu4ve//z0HDx50zdXOItL0KY7WLH3TchFJC6k6xTto0CCKiorIzs6mS5cuTJs2jSuuuIKcnByGDRtG//79o7bp3r07V199NUOHDqVPnz4MHz4cgIyMDObOncutt95KYWEhwWCQ22+/nb59+3LNNddQWFiItZY77riDk046KdlNlRTSEAZJBsXR9GKstUnfaU5Ojs3NzU36fiX1Ki54e2xcmxTXRGqzfv16BgwYkOpqNDm1HNf6rzKJgWKqNAVN6fNBcTSx4o2nGvYgIiIiIq6h5FdEREREXCOu5NcY87AxZoMx5nNjzDxjTHoP8hCRmKRiOFRTpuMp4j76u08MJ45rvD2/i4HB1tqhwJfA3XHXSERSKjMzk3379ilwO8Ray759+8jMzEx1VUQkSRRHE8OpeBrXbA/W2kVVnn4MTImrNiKSct26dWP79u3k5+enuipNRmZmJt26dUt1NUQkSRRHE8eJeOrkVGc3AC/VttAYMx2YDtCjRw8HdysiTvL7/fTu3TvV1ZB6KKaKpC/F0fRW77AHY8wSY8yaGv5NqrLOvUAQmF1bOdbamdbaHGttTseOHZ2pvYiISymmiog0Tr09v9baS+paboy5HrgcGGM1uEVERERE0lhcwx6MMeOBu4ALrbVHnamSiIiIiEhixDvbw2NAK2CxMeYzY8wTDtRJRERERCQh4p3t4TSnKiIiIiIikmi6w5uIiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4KZsxJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLxU7HT9+vF24cGEqdl2rGYsKAXhsXJsU10REXMQ4UUi6xVTFUxFJgZjjaUp6fgsKClKxWxGRJkkxVUQkdhr2ICIiIiKuoeRXRERERFwj7jG/xpjuwD+AzkAYmGmt/XO85YpI6nx7wYiY1503cXUCayIiIuIsJy54CwI/t9auNsa0AlYZYxZba9c5ULaIiIiIiGPiHvZgrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnEk+TXG+IkkvrOtta86UaaIiIiIiNOcmO3BAM8A6621j8RfJRERSVexzgSSnbk0wTUREWkcJ3p+zwWuBS42xnxW/u8yB8oVEREREXFU3D2/1trlOHR/ehERERGRRHJinl8RERERqUWsw4XivWlQsvZzolPyKycU3XlMRERE4qHkN4X0DU1EREQkuU645FcJo4iIiIg0lqN3eBMREQFoFixLdRVERGp0wvX8iohI+up2uA03bTyT04r+h9It3fFPnYCn/UmprpaISCUlvyIi4oiRBV356fqzaR7yA2C/3kbZn54j4/pJeE7rkeLaiUTTRdTupGEPQGjDFr7/73f51saPsYcOp7o6IiInFguTtg7gZ2vOr0x8Kx0ppuyJOQRXfJaauomIHMfVPb+26AiB194l/OkGKr77lT60Ad8Vo/GeNRTj0b07RETq4g95mb7xDM7b26v2lcJhgi8vwu4qwDfpIozXm7T6iYgcz5XJr7WW0Mo1BOcvheKS6gtLSgm+/A6hVWvxX30pnk7tU1NJEZE017a0OT9bcx6nFVWPkyHCrDn5NE7fs7n668tXY/cU4L9uEqZl82RWVUSkkuuGPYTzDxB4Yg7BF9+OTnyrsJu3U/aHWQQXr8AGQ0msoYhI+jv1UDt+s2pcVOJ72FfKQ0Pf57nhY/FfPwkyqg+DCG/6hrI/PUd4d0EyqysiUsk1Pb82FCL0Xi7Bdz6EYDBq+e6strQrLiIjVGVZMETw7Q8Ifboe/3fH4+nZNYk1FhFJT+fu6cn0DWeSYasPX9jRopA/DP6A3S0Okw14T++H6dCWsmdfhQOHKtez+w5S9ufn8V97Bd6Bpya59tU1Zu54zTcvcmJzRfIb3rabwJyF2B17oxf6ffguPZdHyvrSpuQI9+1dQXjDlmqr2N0FlD36PN7zRuKbcB4ms1mSai4iADYUxubvx7RrgzmuJ1GSx4bDTP16KJO2DYxa9mm7nfxl4AqKfYFqr3uyO9Hs9mspmzUfu2X7sQWlZQSeeQX7rQvxXnQmxlS/xkJJqYgkSpNOfm1pGcGFywktWwXWRi339O2J76pL8bQ/ifCiQg60aIX/x1MIr15P4LV34UhxlcIg9MEqQl98iX/KuJT1Vii4S1NnS0qxuwoI79yL3bGH8I692F0FEAzi/8l38fbpmeoqupItKSXw/Bs1Jr6vd1/PC6d8jjXRcRbgyqXn4+3u4YbikVy8u0rstBB8433+d+UWIGZhAAAgAElEQVQsnu67kjnfzk1U9cUl9BkpsXAk+TXGjAf+DHiBp621DzlRbjxCG7dEri7eXxi9sEUm/kkX48kZFNXbYIzBO3Ignn69CCxYSjh3bfVtDxYRePoVQsP74588BtOqZQJb0bRpfkVn2GAQjpZgi0uhuAR7tCQynt0CzZthmmdW+0mGP+p9n5J6WwuFh8uT3L2Rnzv3YgsOROpe0zY79oKS36QL7ztI4JlXsceN0w2YEE/3W8myznn1lhHyhHmq30q+yTrIdV8Nx1PlkpML9vSmy9FW2DGHMa2znK6+iFRhrMGWlkFZAFo2x3hcd/lX/MmvMcYL/BUYC2wHVhpjFlhr18VbdmO0Ksvg2q9HEHjv5RqXe0YMwD/p4nqTVpPVgozvf4vQyIE1JtHhTzdQuiEP75mDoZHT9kz9emhM6wXeeD+ubdJVrG2BY+1pzDZNQiCILS4pT26PS3ID0WPY6+TxlCfDzaB55rGfmc3AHwkJ128ZgS3PQK2h8jFE8tKKHj5LI45zIIjdU0B4x97qZ1diEN5Zw9AlSajQpq0E/j4fjla/QPigv5g/Dl7OV232xV6YgXe6bWJniyJuXXcOWcGMykV9ijpQ+v/+gXfEQDAmafHR7XG4qWno7yahnynWQjAEgSA/2XIW/rAXf9hDRtiHP+zBH/aSEfZW+emh5OM/gc8HPi/G74s89lc894PfW/maKX8djyfyORAIcsuWs2kW9pER8lb56SUj5Kv8mWG9lL7/JwAyfnkTpl2bhrWrCTC2huEADSrAmLOB+621l5Y/vxvAWvtgbdvk5OTY3NzGnd6qq7dw1N7u/HDTSFoHMqMXtm0dGa4w4JQat52xKJLcPjYu+k1gS8sIvvMhofdzaxw+ISJJ0Kol3qF98X9nbKpr4iRHuuDjial1CX74KcF570I4XO31zVn7+ePgD9ifWfuXl+zMpZXxtKa43floK/7PF+eTXdza2UqLSMwy7rwBT+cOqa6GU2KOp04Me8gGtlV5vh04K6pGxkwHpgP06NH421zWdfo7uGwVwXXvVnstTJiF3TYxp9cXlG6aCZtqLzs7cylQe4Lde0RbfrzxDHofbtfwiotITMJYdrU4xNasg+RlHWBr1kG2tjzIrKs+SnXV0ooTMbWuzoTmQR8Pr7yM9uEW1V73DOvHgKmX8UwdFx5WdCZUqC1u20klBJ57PeoiYxFJkkCg/nWaICeS35oy7ajuUWvtTGAmRHopHNhvFO95wwmtXof9ZlekYl06cl+X2Wxuvd+R8re0OsAvRyzmsu39uCpvMBnhJn29oJwgQoQ56gtw2F/GEd+xf+d1uxSKy7AlJWzbu56WwQxaBv1p9b4t8QT5JusgW8uT3LysA2xvWUipV3Nr1yfRMbXYF+SPgz/g/346hmbl7xnfhPPwXnK2Y2PGTfNM/Dd+h+Dr70XOrIlI4hnA74/MwR1259lsJz4FtwPdqzzvBux0oNwGMx4P/qvHU/aX2fguGYV39BlsfvMvju4j7LG80WMDH3Xayoh9Xbmp788BeG5d7Pu5duAtjtYpEWJtT9W2NGabZEjW7yah7fd6ysfnZnLfp7dypEqiW+IN1vgV9OKJf618/J9Vevj8IQ8tQn5aBDNoEfTTsvxnViCDmwb/ovyrq43+WfHDVnlc25VpdTKYdq0x2Z34/opxtc4QIKm3pdUBnuj/L6ZvPJPW116Fd2hfx/dhPB78ky7Ge+YQwhu3QChc/0YnmCYRg+LcT1P6TGkQn69ynK7xV4zf9WF83kgCevwyr6dynPBPF008Nk445COjfJyw/7hxwjcMuKM8mfVFpoL0+yIXNpe/ht9f7XV83rS46DmVnEh+VwJ9jDG9gR3AVOD7DpTbKJ6uHWn2Xz+JXMTTSI2ZWWDBkZtjXveHY0Y1uPxki7U9VdvSmG2SIVm/m2S1f31efBd9BbxhCr2lFGaURi376fkj4yq7oZT4plassc4WHUn4zDaeLh3xdOmY0H2kSlOLQY3ZT2P2ma6fKcmyq0VRTOtNvzhqpKnUI+7k11obNMbMAN4hMtXZs9batfVsllDxJL4iIlKdpnQUkabEkcF/1tq3gLecKEvkRKW5iEVERNJf+lz5Iic8JX8iIqmjGCwSGyW/0uTpA0FEREQqKPkVERERSTPquEkc993QWURERERcSz2/DtE3NBEREZH0p+RXaqRkXkRERJoiDXsQEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLngTERGRtKMLryVRlPyKuIA+RERERCI07EFEREREXEPJr4iIiIi4RlzDHowxDwNXAGXA18APrbUHnaiYiNRMQxhEREQaL94xv4uBu621QWPM74C7gbvir5aIuIESeRERSba4kl9r7aIqTz8GpsRXHefF+uE6Y1FhgmsiIiIiIqnm5JjfG4C3HSxPRERERMRR9fb8GmOWAJ1rWHSvtXZ++Tr3AkFgdh3lTAemA/To0aNRlRURkQjFVEkVDVeSE129ya+19pK6lhtjrgcuB8ZYa20d5cwEZgLk5OTUup6IiNRPMVVEpHHine1hPJEL3C601h51pkoiIiIiIokR75jfx4BWwGJjzGfGmCccqJOIiIiISELEO9vDaU5VREREREQk0XSHNxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa8R1wZuIiIjUTTeFEEkv6vkVEREREddQ8isiIiIirmHquCNx4nZqTD6wNcG76QAUJHgf6czt7QcdA7U//dtfYK0dH28hiqlJofa7u/2gY5Du7Y85nqYk+U0GY0yutTYn1fVIFbe3H3QM1H53t99pbj+ear+72w86Bk2p/Rr2ICIiIiKuoeRXRERERFyjKSe/M1NdgRRze/tBx0DtFye5/Xiq/eL2Y9Bk2t9kx/yKiIiIiByvKff8ioiIiIhUo+RXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENXyp2On78eLtw4cJU7FpqMWNRIQCPjWuT4pqIuIpxohDFVGkK9DkkcYo5nqak57egoCAVuxURaZIUU0VEYqdhDyIiIiLiGkp+RURERMQ14k5+jTHdjTFLjTHrjTFrjTG3OVExERERERGnOXHBWxD4ubV2tTGmFbDKGLPYWrvOgbJFRERERBwTd8+vtXaXtXZ1+eMiYD2QHW+5IiIiIiJOc3TMrzGmFzAc+FcNy6YbY3KNMbn5+flO7lZExHUUU0VEGsex5NcYkwW8AtxurT10/HJr7UxrbY61Nqdjx45O7VZExJUUU0VEGseR5NcY4yeS+M621r7qRJkiIiIiIk5zYrYHAzwDrLfWPhJ/lUREREREEsOJnt9zgWuBi40xn5X/u8yBckVEREREHBX3VGfW2uU4dH96ETlxfXvBiJjXnTdxdQJrIiIiUjvd4U1EREREXEPJr4iIiIi4hhN3eBMBYj/trVPeIiIikipKfqXJ01hUERERqaDkV0RERBKqvk6I3kVtuf7r02nF6YR6XIC3f+8k1UzcSMmviIiIpMyZe7vx0w2jyAj7gD0EnpqLnXQR3vNHErmVgIizlPyKiIhI8lm4Ylt/vr952HGvW4Kv/S82/wC+yWMwXl2bL85S8iuEt+7iB6uX0aq0mGCrYXjPOp0r3xgZ07YaIysiJzJdqJsa3rCHH32Zw0W7T6l1ndCHn2L3FeK/7gpMZrMk1k6aOiW/LmaLSwm+tYzQR58y2EZeC768iNAna+jRsQ3fZBWmtoIiIpJQqUj+Wwb83LH2PAYdPLna60ETJuhpRmYoUPlaeMNmyv7yTzJu/A6mbWvH6iDupnMJLmStJfTpekofeprQh5+CPW751p38NvdSvv/16TQLeVNTSRERaXI6FWfxwOqxUYnvYV8ZDw59j7+OmgQntaq2zO7Kp/RPzxHetiuZVZUmTD2/LhPed5Dg3MWEN26pcz0vHq7YNoBRe3swq88qVnfYmaQa1k3TlomIpFZje4v7HezAz9ecT6tg9SEMuzOL+P3QZexqUUR2Znua3X4tZc+8it22+9hKRUcoe+wF/NdcgXdIn7jbIO6mnl+XsMEQwSUrKPvdszUmvvktWrOuY4+o1zuWtuQ/11zAHWvOpV1J82RUVU4Qdn8hoS82Edq0FRsIpro6IpLGztnTk3v/fVFU4ruhdT7/NWIxu1oUVb5mWmeR8dPv4Tk+yQ0ECcyaR3DpJ1h73ClLkQZQz68LhL/eRmDuIuyefdELvV68Y87ij3YAQa+PP3fdS+DVJXCwqNpqZxZ0Z8iBzrzc6wveyd5E2KPAc7ymfuGMLSkl/PU2whvzCG/cgs0/cGyhz8cvWl3IF+1283nb3WxrWQiaoUjE9ay1hBZ9xC3rz45atrxTHk/2/4SgJxy1zGT48V8/meCb7xNa+kmVAiH4+nvYggP4rrwE49XQPGk4Jb9NmD1STPD19wh98kWNyz2ndsc3ZRyek9sTXBS5uM07uA+ePj155a//wYTtffFWOTnQPOTnuq9HcP6eXjzdN5fNrfcnpR2SGjYcxm7bTfjLPEIb87B5OyEc/SEFQDDI6Qe6cPqBLgAcyCjmi7a7y//tobBZSRJrLk1BU/8yWZ+mMMTLBoMEXlxIePW6qGVze33BKz3X1vkl2XgM/itGYzq0JfjKIggf63QJrfh3ZCaI6ydhmmsmCGkYR5JfY8x44M+AF3jaWvuQE+VKtJgCooW5Xf9BYMFSOFIcvbxlc/wTL8KTM6jGCcRNswxmn/YZH3TO48aNOfQp6lBtee/D7fj16rEszt6ELS5V4GlCwvsLIz27X+YR/nIrFDcuaW1b1pwL9vTmgj2RuzTltTzAF+0iyfCGNgVOVlnEGRY81uCzHrzWgy9s8FY+9hAuOBDpZfR6oOKnp/yxxzSNmzFY8Npj7fZagzcceWz3F5a310PLQAZhEyZoLGETJmRsVBLbqiyDsr/NwW7ZXu31gAnxZP9P+PDkrTFXy3f26Zh2bQj8/TUoKat8PfxlHmWPPo//x1PwtGsTV9PFXUy842aMMV7gS2AssB1YCXzPWhv9Va9cTk6Ozc3NjWu/NbFHSyj9w/84Xm46KSjeXe86vrCHkwI1j8/1njkE3xWjMS2rL59R3vP72LhIAKlIso01XLLzVKZuHkqLUEZ0gc0yoDz5jaVuAB2ad45pvZrEuo+q+4l1G4OhffMqVyDX9LdRx5/L/pK9Na9y3IdCtX2kk1AYio7Evr7Hg+nRGXuwKGqYTF3KPEEyWrdtRAUbx3/5hXhHDEza/hrIkYwpUTE1tHINgbc/cLzcWCQ0noRt5CxGMERJ2RF81uCzcZ4+r0yMKxJiD3ga/+ttTKxrkFAYQiGOlhZVJrw+2/jLgEKECXkiiXDIhPGFvWSGq/evFflK+ePgD9h4Uu1fgLMzl1Z+Dh0vvLuAsqfmwoFD1Rdk+KFFZqPrLg3T7M4b0nXe5Zj/4Jzo+T0T+MpauxnAGPMiMAmoNflNGGsb9CF8IupAy0ZtZ05uj3/KODyndm/QdtZYFmd/xcoO27n2q+Gck9+z+gqlZZF/DalbaeN/Rw1qf/l+GrNNY7SjRcL3kWqmUzs8fXvh6dcLz6ndMZnNsNZi8/czc+7NDDnQmUEHOpEZ9tdaRkbYl9S/U1sWqH8lqZEtC6QspiYjngBkOjX6LxSK/HNIsuJWC2r/W20ILx684dqT5x3ND/H7ocvY2/xwo/fh6dzh2EwQ31SZ9qwsEPknEiMn/uqzgW1Vnm8Hzjp+JWPMdGA6QI8e0bMKxKqu0/5ZgQye4spGl90k+Xy82H01b3TfQGjtX2Ft7atmZy6tfFzbGLLQhi0EX1mM3XfQ6ZpKGirylbKm7Z7I2N12uynIPBpZ8HX5PyLvFdOpPf9x80tAZGYRu3UnoYoL47bvrrPHXBrHiZha3zCqsTtO4wZyGlW2SAXPaT045QeTebKe3tmKM5AVant/+nt6ubn4LEblNz6XEHdzIvmtqZs56qPOWjsTmAmRU3QO7FfqYgyeAb3xTR7D/I+ed6xYb//eeP7zhwSXrCD0werKXl9pIjweTO9sXgy/yedtd7Ol1QGsadifq/F5Mad2j5xluOx87JFiwpu2Et6YR+jLvOhTltIoiqkJYEz08AWvJzLW11psOAyhMEXF+yNjg8MefNbgaUqzhnoMeKqOaS7/Z8yx4SKhMIRD5T/D1S5EO573nGH4Jo/B+JyblSHgDfHowI/YkXeIyVsHVrswWyQWTiS/24Gq59K7ASm5I8IRX4AZoxYA8NTYtwD48eLLYt4+2dskVGazhF2IZjL8+C+7AN+4cxs2RvQEcmPl7zM6qNsqX/eeGbuw7oLqGVMf6/um6numMdvErGVzTIaf1xY83PBta2FaNsc7rD/eYf3xWQuHj0Iwcno4kX9rle3XWMBG++DkPFa3j4TzRh//Rm7TGDHvZ9zb1S9c83rA48XEOEb3e8f1SBpL5YVxkbGzkQvFnh77doPqBfEdgwYd56oX7FU89nhiPgZV2bAFW54UVyTEoRBk+BM2NtQamNt7Da9338ALFy1JyD5qk5S4lc7bZNRw/c8JxonkdyXQxxjTG9gBTAW+70C5NWrolC77Kk7TxqDivuHJ2uZEZ3xeaCJtOd7+GH+f5rjbcDZUrO+bqu+ZxmyTLowx0OrYWMbG/N08ffVyx+slNSvxBSnxRW5gEs/xT9Z7Nub9xPl3ezxrIGjCUfPVJvvzIVWxIZIweyOJdJKV+oJJj3WJzBHijfVu+0xprLiTX2tt0BgzA3iHyFRnz1pr6xhZmlzpOv+hCOj9KSIikmyOXOZqrX0LSMK5fBFpSpT8u4N+zyKSTnSHNxERkSagqX3JaGrtSVduPM5KfmvgxjeCnDj0/hSn6L0kIm6k5Nch+hAREZGa6PNBJL0o+RURERGJkb7MnPg0M7SIiIiIuIZ6fkVEpElQj5ykK70304t6fkVERETENdTzK1IDfUsXkdooPoic2NTzKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIamu1BRKLoanYREWmq4kp+jTEPA1cAZcDXwA+ttQedqJg4J5ZEZsaiwiTURERERCS14u35XQzcba0NGmN+B9wN3BV/tURERETcS2fgEieuMb/W2kXW2mD504+BbvFXSUREREQkMZy84O0G4O3aFhpjphtjco0xufn5+Q7uVkTEfRRTRUQap97k1xizxBizpoZ/k6qscy8QBGbXVo61dqa1Nsdam9OxY0dnai8i4lKKqSIijVPvmF9r7SV1LTfGXA9cDoyx1lqnKiYiIiIisdM44djEO9vDeCIXuF1orT3qTJVERERERBIj3jG/jwGtgMXGmM+MMU84UCcRERERkYSIq+fXWnuaUxUREREREUk03d5YRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4I7ExJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcw5eKnY4fP94uXLgwFbsWSakZiwoBeGxcmxTXRNKEcaIQxVSR9KN4n3Qxx9OU9PwWFBSkYrciIk2SYqqISOw07EFEREREXEPJr4iIiIi4hpJfEREREXGNuJNfY0x3Y8xSY8x6Y8xaY8xtTlRMRERERMRpTsz2EAR+bq1dbYxpBawyxiy21q5zoGwREREREcfE3fNrrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnHkDm/GGD+RxHe2tfZVJ8oUERERSTffXjAi5nWzM5cmsCbSWHEnv8YYAzwDrLfWPhJ/lURERBou1qRk3sTVCa6JiKQzJ4Y9nAtcC1xsjPms/N9lDpQrIiIiIuKouHt+rbXLAeNAXUREREREEkp3eBMRERER13DkgjcREXEHjasVkROdkl8RERGX0pcZcSMlvyJJYkvLGP/lJ3Q6cpBQzxy8/Xonbd/6gBMREYlQ8iuSBLboCGVPzeWS7XsACDy5BTvpInwXnpHimolIU6EvuSKxUfIrkmDh/AMEZr6M3Xew2uvB+UuxhYfxXT4a49GEKSIiTYmxqa6B1EbJr0gChbftouypV+Dw0RqXh95biT10GP/UyzA+b5JrJyIiTutQ0oLrNo3g9AOd2XrS64SHXIqni25Bnk6U/IokSGj9ZgJ/nw9lgTrXC69eT6DoKP4fTsZkNktS7URExFEWLtzdm+u+GkGLkB+APvt3UvbIP/BNOA/v6DMwHs0wmw6U/IokQPCTLwjOWQjh6ue9crv25cMeg7ht7SIoOlL5enjTVsr++gIZP56CaZ2V7OpGaci96zV+UMR5+htsnFSNe25T1owbN55Bzr5u0QtDIYJvvE9o7df4vzcBT4e2ju5bGk7Jr6RMUwzu1lpC735M8K0PopZ5x4ziRd9QMIaMW6dFxgHnHzi27Y69lD06G//0KXg6tU9mtUVcSxeJSbzOzO/Gj77MoXUgs8717JbtHHrocWaf+hlLun5d471x9T5LDvW/izjEhsMEX10Snfga8H37EvzfugBMJNp52p9Exi3TMD26VC9jfyFlf/kn4bydyaq2iIg0QsuAn5+uG8Uda8+LSnxLPUHe7LaBI/7qQ9kyw35+tOkM7vriAtqWNk9mdaUKJb8iDrBlAQJ/X0Dow0+rL/B58V83Cd/50b1LJqsFGT/5Lp6Bp1RfcKSYsr+9SGjtVwmssYiINNaQ/Z353coJnLe3V9SyL1sX8IuchTx/2mf84dyr8Aw4JWqdYfu78vuVEzhnT0/QrBBJp2EPJxidoks/9mgJZc+8it2yvfqCzGZk/OhKPKd2r3Vb0ywD/w+vJPjyO4Q++eLYgkCQwLPzsFeNwzfq9ATVXKTp84YN3Y+cxKlF7TjlUDt6H25L6YZnMV074enRmT6F7dmadZAybyjVVZUTgC0t44YvRzJ2Z5+oZUET4uVea3i9xwZs+TxnRZkt8d/4HR578iqu+Xo4zcsvhAPICmZwy/qzOaMgm2f75FKUUZa0dridkl8XUMKcOPbAIcpmvozds6/6gjZZZEy/KqbpbYzXg++746FNFqHFK6oUbgnOeQcKD+Mddw7GaC5gkboYa+h6tFVlontKUTt6Hm5Lhq0+jaA9XIDdXUB49Tp+xVhChNnespCvW+1nc+v9fN1qP9taHiTkUZecHBPesp3AP99i7L7oxHdrywM8PuBffJN1MGqZMYb/7bqZNW338B8bzmJAYadqy0fl96D/wY7M7LcyYXWX6hxJfo0x44E/A17gaWvtQ06U21A2FMJ+sysVu06avoUdYlovXKUXMuZtNm+vZUljPwBqSdbKX461XlC9PenCFpcSeDmSnFZlTm5PxvSrMG1bx1yWMQb/hPMxbVoRfGUx2GPHPPjOh9iDRXjPHAzA3ctviKnMB897tvJxQ983Df7dNMvA07VT/SvLCcUeOhy5OYuFijjQ/2DkC50BsObY48qfkf9DX249VlDVUGDMsacVX+hMlZUa+B3PFh4m/M0uwtt280zeldV61mLlxUPPI23peaQtF+8+FYAyE2Jr1gE2t97P5lb7I/Exid8/Y/0bNPZY7O53MNZYv63yccXvE2ponj32Suirb2per+qXclPDGkn+zt7QWBdrPB1ZkM3E7QOrxWaAMGEW9NjAK73WEPSE6yxjb/Mj/HrYUi7b3perNw+t9qXspEBz7lxzAYGX3sY36WJNe5lgxtr4vtkaY7zAl8BYYDuwEvietXZdbdvk5OTY3NzcuPZbE3ukmNL7/uJ4uSKxMqd0I+OGKzEtar7qd8aiQgAeG9em1jJCX2wi8NzrEAwmpI6JYHpl0+zWaamuxonIkdQgUTE1uHw1wVeXOF6uSFOwq3kRf+v/MZva7Kt1nezMpTw2rk3UGdhuR1rzk/WjOOVwu+iN2rbG/73L8J7Ww+kqN3Uxx1MnLng7E/jKWrvZWlsGvAhMcqBckROKZ0hfMm66utbEN1beIX3I+MnV0Dy+ckQkYl/GUVZ22M6LvT/nt0OXkvHT7+GbeBGe4f3ZnVmU6urJCeidrl9yd87COhPfumxveYj/GrGYV3quIcRxPcYHDhH4n9ewJaUO1FRq4sSwh2xgW5Xn24Gzjl/JGDMdmA7Qo0fjv83UNX41K5DBU1zZ6LJFGst77nC+6/sF9u1f1btudubSysd1vZ+7Dm7NLz6/kI6lLR2pozQtTsTU+q4HGLvjNG4gp1Flp0zL5ni6d8Z074yne2c8PbqQ3TqLbOD8KqtVXIjai4nYI8WEt+/GbttN+JvdhLftihrOJAJAmyz8Uy9jUr9e9fbyVZzpg7qvqQl/s4vAP9/E7t1f+Zr/22M09CGBnEh+a+pmjhpLYa2dCcyEyCk6B/YbJWQsG1vnA9C/3TAANuz/LObtK7ZpjMbsJ9Zt4qlXg5X/Ntfv+6zG1483oN3wysfr91eZ5quW37Ah+cc5kfsZcPJIPMMG4B01FPu6s2/rneU9A1dtGcKYzAscLdtpni6xjxEWZyQjph7MKDkWU3nukL0AACAASURBVNsPo8Zxuab6uF+Mqb7cwhcFkQt5jDVRocRgKuPFgEb8zZoMX/nMDV0w3Ttj2rVp8MWhpmVzvP16Q7/ex6p96NhYYrt9D7b4WC9c2sXuRg2eiXFc7vHHsrahkjbqQbWHjTlmaXWcPQbPqd3xjT4D4/BZOU+PLmT87HqCb39AaFkuniF98Ywc6Og+pDonxvyeDdxvrb20/PndANbaB2vbJp7xaQ2duSBZdxFrzH7SeRaGxtQtGe1Jt99nY9pfMQassfuRJiWlY36b4l0Wk0F/tw2Xrp8piRTLNR7HC3/1DaZzB0xWi0RVqymLOZ460fO7EuhjjOkN7ACmAt93oNwapeubPF3rJSIiciJy4+eqRxe5JUXcya+1NmiMmQG8Q2Sqs2ettWvjrpmIiIiIiMMcmefXWvsW8JYTZTnNjd8cJX5634iIiDRNTkx1JiIiIiJyQtDtjUVSSD3MIiIiyaWeXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa+iCNxERkQbQhaoiJzb1/IqIiIiIa/z/9u48Tq66zPf496mlO/tGNukQkrBvgUCzGTaRJSCyyggqoijRYeLgdkcZx6tXr3d0FjdwmbAMClFGCZsCMQEjmwRpkgAJYQmEJSQknX3v2p77R1Wn9+7qqlNV3XU+79eru0+d9fmdrvqdp37nd86h5beCaD0AAAAoL1p+AQAAEBokvwAAAAgNkl8AAACEBn1+AQBASXGNC/oSkl/0K1SgAACgGEV1ezCzfzezl83sBTO718xGBBUYAAAAELRiW34XSLrB3VNm9gNJN0j6WvFhoT+iVRYAAPR1RbX8uvt8d0/lXi6SNKH4kAAAAIDSCPJuD9dIeririWY208wazKyhsbExwM0CQPhQpwJAYXrs9mBmj0ga38mkb7j7/bl5viEpJWlOV+tx99mSZktSfX29FxQtyoYuDEDfFkSdyuccQBj1mPy6+1ndTTezqyVdIOmD7k5SCwAAgD6rqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqQd7tAQAAAOjTSH4BAAAQGsU+4Q1ATj63jZo1f2sZIgEAAF2h5RcAAAChQfILAACA0CD5BQAAQGhYJR7KZmaNkt4q8WZGS9pQ4m30ZWEvv8Q+oPx9v/wb3H1GsSuhTi0Lyh/u8kvsg75e/rzr04okv+VgZg3uXl/pOCol7OWX2AeUP9zlD1rY9yflD3f5JfZBNZWfbg8AAAAIDZJfAAAAhEY1J7+zKx1AhYW9/BL7gPIjSGHfn5QfYd8HVVP+qu3zCwAAALRXzS2/AAAAQBskvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEIjVomNzpgxw+fNm1eJTSNAs+ZvlSTddM7wCkcC9FsWxEqoU/se6keg7PKuTyvS8rthw4ZKbBYAqhJ1KgDkj24PAAAACA2SXwAAAIRG0cmvme1nZgvNbIWZLTez64MIDAAAAAhaEBe8pSR9xd0Xm9lQSc+Z2QJ3fymAdQMAAACBKbrl193Xuvvi3PB2SSsk1RW7XgAAACBogfb5NbNJkqZJeqaTaTPNrMHMGhobG4PcLACEDnUqABQmsOTXzIZImivpi+6+rf10d5/t7vXuXj9mzJigNgsAoUSdCgCFCST5NbO4sonvHHe/J4h1AgAAAEEr+oI3MzNJt0pa4e4/LD4kAAD6rkseODav+eoGLCxxJAAKEUTL73RJV0k608yW5n7OD2C9AAAAQKCKbvl19ycV0PPpAQAAgFLiCW8AAAAIjSAecgGgQPn2Hbz3wsUljgQAgHAg+QUA9Dl8MUQheN8gH3R7AAAAQGjQ8gsAQIDev25/ffidQ7VnwEPKHH2OIuP2qXRIAFqh5RcAgCC49HdvHKUvrDhZk3aM1KEb3lHip3cq8/o7lY4MQCskvwAAFCmaiei6l0/SJW8f0XbC7iYlfvk7pZesqExgADqg2wMCw4UGADpT7XXDwFRcX152io7cMq7zGdJpJe/4g3zLdkXPOF7ZB6P2X9X+/0T1I/lFQTzjmrxpraKekaeHyqKcROCAAITPqD2D9LUXT9PEnSPajN8VTWhQuqbNuNQf/iLfvE2xi8+URagzgUoh+UWvuLsyL72u1ENP6B/WNkqSEm8+pdh5p8pc8v7doAEAeZu4Y7j+6YXTtU9iUJvxawdu1/enPqYTd/2nPrb8MSmT2Tst/eRi+Zbtin/iAllNvNwhAxB9ftELmZVvK3Hjb5S89R55LvGVJG/crOSvH9D3njtHR28cL3kFgwSAMjhy0zh9a8lZHRLfV4dt0LemPaL1A3docd3Bis+8XBrQtgU4s+w1JX5+l3zHrnKGDCCn6lp+U396Spn1mxQ9fIoih06RDR5Y6ZD6vczq95R68AllXlnV7XyTd4zS1188QyuGr9dvpzyv14ZvLFOEwaMLA4CunPbeJF37ygmKedv2o7+Nfkc/O2yREtH03nHRg/eXzfq4Ejf/Xtq6Y+94f3utEj+9U/FrL1dkzMiyxQ6gCpPf9OKX5I2blVmyQjKTTapT9PADFDl8imz86LwuNMg38ZGqO/nJrN+o1LwnlVn6SufTJbmZot62qfewrWP1nSVn67l93tXvJr+gt4dsLUO0AFBa7q70gqf19y+f1GHaw3Wv6I4Dl8qt46mvyL5jVHv9VUrcfHfbs2Ybtijx0ztV85nLFJm0b0ljB9CiqpLfzPpN8sbNLSPc5atWK7VqtfTgY9LIYblE+ABFDpwoi1dV8QPjm7cpNf8ppZ9dJmU678MQOeIA/eeIaUpGovrn7Us7TZCP21inaRv31V/HvqXfT16m9QN3dLImAOj7PJ1W6u4FSj/zQodpdxywRA9NeEXqpm3FRgxVzayPKXn7fcq89lbLhJ27lfjFXYp/4sOKHnVQCSIH0F4g2Z+ZzZD0E0lRSbe4+/eDWG9vZV7u/rS8Nm9T+qklSj+1RKqJK3Lw/oocdoCih0+RDR9aniD7MN+xS6lHF2X3Tyrd6Tw2ZYLiHzpdkcl1Wjs/26Jbc8lFypz5nv52+/d1zKa2rRcRmU5ZP0knNU7Uwve9rnv3X15UjEG1yns6Le1JyJsS0p6m7HAiKauJSbW1GrN7sHbHktodTSodoRMzEGa+p0nJXz/Q4RiTtLR+ftgiLRqb30MsbGCt4td+RMnfzVOmoVVdmEwpefu98ovPUuzU/Os4AIUpOvk1s6ikn0k6W9JqSc+a2QPu/lKx6+6t6CnTFNlvvNIvva7MS6+3Ob3UQSKpzLKVyixbqZQkmzBONn60ZKbPvXOCpOx1Wy6XrOUaLs8NNZ/aSt49v/ug2nUJyK7U9/54q+GWH2VbXD3T8tpMMtPT7/1Zbq6MXG5SxlwuV6bVOJfr3MkfyS6Tr2RK6edflpqSnU62urGKnX+aIodO7rTrSGTCeP1g6uM6dMsYXfHGVB2ybUyb6TGP6Ow1B+m09yYrkXxQVlujh1f9Lu/wzpv8d5KkT686Lq/5TVLiNw+2JLZ7mqSmhHxPLtlNprpd/qf68N7hRCSl3dGUduWS4d17/6aUTMzfu5/zjS05d8He4d4uEzvn/bKhg/NaBuhJT18mD90yRievnyip5TNYiLzf53fP31v3fe6tExRxk8kUccsNq9Vw9m/ivd9LEcvVkZGW4b1/I9k6q3lc8y3GelE/Zt5Y3eF4siPWpP848km9MqKb40wnLBZV/MrzlRo5TOkFT7dMcCl17yPKrHxbNqxvf8YLqetKpvkYmztWXvvW8ZJnG14kyXLvG5PJvOVvYsv9slEjFL/g9NLHiD7HvH1y1tsVmJ0s6dvufm7u9Q2S5O7/2tUy9fX13tDQUNR28+Gbt+1NhDOvvdVlaya6ZmNGKnbeqYpMPUQWaXuwmJVr+b3pnOGSWh1IXZq2cV99dNVR2n8nF3IEqebrn1Fk7D6VDgPBCeTmgIXWqT0lv2e/e6Cuea2+0LCq1voBO/SDox7TmsHbu52vbsDCvfVjZ1JPP6/U3Plddi9DaVndWNV+5VOVDgPBybs+DaLbQ52k1ud8Vks6sUNEZjMlzZSkiRMnFryx3lyFbyOH6fKNn5HGSbWjozpi8zhN27ivpm3ct8PtadDWxtpdmrv/Mj02fpUyb/9Cervz+eoGLNw73L6bgWdcmSUrlJr3pHzjllKGC4ROEHVqTxfspp5aotRrZWi960dswjjt99nr9LNh3+l2vubGAan749YxR7xP1y9/vwZkuOdv2fGdI7SCSH47y7Q7vKXcfbak2VK2lSKA7fZKUzStxaPXaPHoNZJL++8YoWNzifAB2/fZe4ok9AYPVOyDJ+lLm/9eyWim5/m7YRFT9LjDFTnmEKWfeUGp+X+Vtu0MKNDiZOQdujA0RVKaOuI4aU9Tm77AHbquAH1ApevUMIocNkXxT14oq63peeZWevqSkXnnPSVumStt7xv1Y2hQt4dWEMnvakn7tXo9QdKaANZbOia9NXSL3hq6RfdOeknDErU6dMsY/a+p38tOb+5nq/Z/2w277+03NvuFf22e2om2Yz93zDcki+im5/9Prv9utg9xc99dz73O9uF13XDCj7ILZlr1C8609Af2vcPedp7e7pYhg7J9egfUKvlAcYlvm/VGo4q9f5qi9Ucq8/Iq+bby3PXBamuk2hp9e/H1rZLcpHZFU2qKpjr92nbvhT9s89rdpUQy11+4uf9wLjFuyl0wV0Y2pG/3BUR1iUyeoNilZ7UZN/vF/K5nnnnU1wvcaK7vbq6PrrXuq9vcd7d1n16ztnVfc33Yqi70Tsb1lo0anr13fCT4hpLIfuNV+0/XKL18Zba+Uf77WWrZ1yX/3xShkNjyXmbqDdkBs2y9btlevs3De/t8t59nQG3e8aO6BJH8PivpIDObLOldSVdI+lgA6y2bbTVN+tvY1YqdcFTB67huev4XbzV7bH0Pd6fIiR5ZHbe/sZq4olMPLvt2l725ruBlzUzKJdE2bEiAUQF9X2TfMYrs2/bi1etO6X1dh57Z4IFtjkELNq3Me9nrTjm2V8s0z19OhcSW9zLTpxUUE8Kr6OTX3VNmNkvSn5S91dlt7l7c/awAAACAEgjkPr/u/pCkh4JYV9Cq+QlsAAA043gH5IdHnFUQFRUAAEB5kfwCANDH0DjCPkDpkPyi6lGBAgCAZv0u+SWRAQAAQKH6XfKL8sjnS0brJxgBAAD0ByS/AACgpDhri74kUukAAAAAgHIh+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFRVPJrZv9uZi+b2Qtmdq+ZjQgqMAAAACBoxbb8LpB0pLtPlfSqpBuKDwkAAAAojaKSX3ef7+6p3MtFkiYUHxIAAABQGkH2+b1G0sNdTTSzmWbWYGYNjY2NAW4WAMKHOhUACtNj8mtmj5jZsk5+Lmo1zzckpSTN6Wo97j7b3evdvX7MmDHBRA8AIUWdCgCFifU0g7uf1d10M7ta0gWSPujuHlRgAAAAQNB6TH67Y2YzJH1N0unuviuYkAAAAIDSKLbP702ShkpaYGZLzeyXAcQEAAAAlERRLb/ufmBQgQAAAAClxhPeAAAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaFglnkhsZo2S3irxZkZL2lDibfRlYS+/xD6g/H2//BvcfUaxK6FOLQvKH+7yS+yDvl7+vOvTiiS/5WBmDe5eX+k4KiXs5ZfYB5Q/3OUPWtj3J+UPd/kl9kE1lZ9uDwAAAAgNkl8AAACERjUnv7MrHUCFhb38EvuA8iNIYd+flB9h3wdVU/6q7fMLAAAAtFfNLb8AAABAGyS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQiNWiY3OmDHD582bV4lNA6iwWfO3SpJuOmd4hSPpEyyIlVCnVgc+G0BR8q5PK9Lyu2HDhkpsFgCqEnUqAOSPbg8AAAAIjYp0ewDQt13ywLF5z3vvhYtLGAkAAMEquuXXzPYzs4VmtsLMlpvZ9UEEBgAAAAQtiJbflKSvuPtiMxsq6TkzW+DuLwWwbgAAACAwRbf8uvtad1+cG94uaYWkumLXCwAAAAQt0AvezGySpGmSnulk2kwzazCzhsbGxiA3CwChQ50KAIUJ7II3Mxsiaa6kL7r7tvbT3X22pNmSVF9f70FtFwDCiDq1/+jNBaR1AxaWMBIAUkAtv2YWVzbxnePu9wSxTgAAACBoRbf8mplJulXSCnf/YfEhAQhavi1P3LYMAFDtgmj5nS7pKklnmtnS3M/5AawXAAAACFTRLb/u/qQCej49AAAAUEo83hgAAAChQfILAACA0CD5BQAAQGgEdp9fAOiJ79qjY9au1Jqh+0gaXulwgD5jSLJGx22oU2LkOvHZAEqr6pPf3txcnNs8AaWTeXONErfO1Sd27lZGptToMxU77bhKh4Ve4rZ5wTtg2yj904unaVhygKT7lBx0omIfOk3ZO4kCCFq/S36peIH+J73sNSXv+IOUTEmSInKl7ntUvnW7Yh86XRbhII9wmrbxffrH5dM1INNyOE7/+Rn5th2Kf3SGLBqtYHRAdep3yW81IZFHGKSeXqrU3Qsk7/gE3vTCv8m37lD8ivNkMQ7yCJcz1k7WZ185XtFOLr/JNCxXcvsuxT91kay2pgLRAdWLC94AlIS7KznvSaV+P7/TxLdZZvFLSt58t3xPUxmjAyrIpUvePFyfe+XEThPfZplXVinxs9/Kt+8sY3BA9aPlF0DgPJ1R6u4/Kf3Mi20nmOmRycfoxNUva2hi997RmdfeUuKm36rm2stkw4eWOVqgfMxNn37tWJ295qA24zNyzZ20TB9YN12jd2/bO95Xr1Pip3MUn3m5ImNGljtcBKSQM72cHS4dWn4BBMqbEkredk/HxDcWU/zTF2vewSfoxpMulrU7kPua9Wr66Rxl1m0sY7RA+cTTUX1p+fQOiW/C0vrREU/qnknLddNJF8kmjGsz3TduUeLGOcq8s7ac4QJVi5ZfVAx34qg+vn2nErfMlb/zXtsJgweq5jOXKjKpTlqzVZsGDVPNFz6enfftVgf0zduUuHGOaj5zmSKT68obPFBCg5M1+uqLp+rQbWPajN8RS+g/jnxcr4zYkH1dO0g1112h5K/uV+aVN1vNuEuJn92l+NUXKXrYlDJGjvY4dvV/tPwCCERmw2YlbpzTIfG1UcNV84WPZRPf1uOHZA/ykcMPaLuiXXuU+MX/KP3ia6UOGSiLffYM0reXfLBD4ruhdqe+Pe2RvYlvMxtQq/hnLlOk/oi2K0oklbz1HqWfXVbqkIGqRvILoGiZd9Yq8dM58g1b2oy3urGq+cePKzJ2n06Xs5q44p++RNGTpradkEopeft9Sv11SalCBsois6ZR31l8libsavvgircHb9G3pj2idwdv63Q5i0UVv/J8Rc88sd0KM0r+9iGlHl0k7+ZCUgBdo9sDgKKkV7yh5K/ulxLJNuMjB++v+Kculg2o7XZ5i0YUu/xc2fChSv3pqZYJ7krdvUC+ZYdi553CDf/R76RXvq3kbfdoVGJQm/Erhq/Xfxz5hHbFk10smWVmil9wumz4EKXue1RqleumHnxcvnWHYhefKYuUtx2Li7fQ3wXyiTGzGWb2ipmtNLOvB7FOAH3faWsnK3nr3I6J73GHK/7Zj/SY+DYzM8XOna7Y382Q2j3wIv3I00re9bA8nQ4sbqDU0ktfVvK/fi/tSbQZv2jM2/rXqX/pMfFtLXbqcYpfdaHU7oEX6ScXK3nHH+S5h8cAyI8Ve9rEzKKSXpV0tqTVkp6VdKW7v9TVMvX19d7Q0FDQ9rr79jg4WaMfP/MhSdKQePYU047k1rzXXcwyhch3O8Vsoy8r137uewr4zHnul7ca0Tzc/BnuME+OSYlMk3zvlnO/reW1514PiQ+T1Iv3Zqpjchs980TFzj+ty6e2zZqfXfdN53T+P02/9LqSv36gY0J9yORqvOF/IM3ZxdSp3UktekGpPyxsM25Hcnteyw6JF3rLuipp4d+9p8OoeXWv6tcHLpFb13VA3YCFXX82Vr6t5G33Su3viV0T75AYl1Ihx65qOt6VMq8o9z6r+cLHFBk/uuDl+5i8K48guj2cIGmlu78hSWZ2l6SLJHWZ/JaKqdXBOJWteIYov5anYpcpRN7bKWIbfVm59jOkmnw/6oV8BpqZFLv4LMVOzf9K6M5EDz9A9vdXKHHL3dLOVvcCzt3wv2bm5bIhg7pZAwKTTku72yZaQ5Tnl48UDy1p7TdTluoP+71cVG4fPXCibNbHlLj599LWHS0TEklJ+bckF6uQY1c1He9KmleUe59lMoUv248FkfzWSXqn1evVkk5sP5OZzZQ0U5ImTpxY8Ma66w/kO3er6akbC143gALFovrRIY/pb5vvkh7oefa6AdnWxO7O5Iw/Yoi+/sIZGrdnSJvtqCZebLRVIYg6tad+mGe/e6CuUX1B60ZOJKL4Fefpmvp/0jU9zNp8VkTq/n8z6rBBuuGF0ztcRAcgP0H0+e3se2yHczruPtvd6929fsyYMZ0sAqBfGjlMNZ//O/1tzOpAV/veoB361rEL9MaQTZIkGztKNZ+5TEbyK4k6tV8YNljxay9TtP0ty4q0acAufXvao1oyak2g6wXCIoiW39WS9mv1eoKkynwiBw5Q7Xe/UJFNXzXvA3nPe8eMhT3PVOR2itlGocoRW9Xs59ydC656+PT8tnPeY7mvmaaPP3yqJLX04TVv6f6b++0m/e6CRbmZmvsEt/pO2lk/4UL6/5tJA2uzd2Iowa1Ht9Y06TvH/FnXvHaczrr2c7LBA4PfSIj1dGW9J1NSABdT5f0+n/GX4rZTwOe2pMtc+nTJ7lKyM57Qv019XLWpmH577mO9i6sCx4dCVNMxpVx6vc/yvCi52gSR/D4r6SAzmyzpXUlXSPpYAOvtNYuYVKGD4854oueZcoo5gN952aKCly21fPdBMeUv134uR1my28mvn54NGrB3eHcsv2TEYtVxJ8OmWEq/OOwZnTOKU7zlZvGYFC/+fZT3+7zoz1PvP7clXaYMt+driqX2xtaXjw+FqKZjSrmU69jV3xVdq7l7ysxmSfqTpKik29x9edGRAQAAAAELpGnI3R+S9FAQ6+qvuDE3AABA31cd50URGnzJAAAAxSjvMxEBAACACqLlFwBQUpyx6T32GVA6JL8AAKDP4QsASoXkFwAAoArwhSE/9PkFAABAaNDyC3SCb88AAFQnkl8EhoQRQH9DvQWED8kvAAC9QMIM9G/0+QUAAEBo0PILVFDYW5DCXn4AQPnR8gsAAIDQIPkFAABAaNDtAQAAhBJdr8KpqOTXzP5d0oclJSS9LunT7r4liMAAdI7KGgCAwhXb7WGBpCPdfaqkVyXdUHxIAAAAQGkUlfy6+3x3T+VeLpI0ofiQAAAAgNII8oK3ayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeL3hz97O6m25mV0u6QNIH3d2DCgxA/5LvhXiz5m8tcSQIKy4GBZCPYu/2MEPS1ySd7u67ggkJAAAAKI1i+/zeJGmopAVmttTMfhlATAAAAEBJFNXy6+4HBhUIAAAAUGo83hgAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2rxEPZzKxR0lsl3sxoSRtKvI2+LOzll9gHlL/vl3+Du88odiXUqWVB+cNdfol90NfLn3d9WpHktxzMrMHd6ysdR6WEvfwS+4Dyh7v8QQv7/qT84S6/xD6opvLT7QEAAAChQfILAACA0Kjm5Hd2pQOosLCXX2IfUH4EKez7k/Ij7PugaspftX1+AQAAgPaqueUXAAAAaIPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEKD5BcAAAChQfILAACA0CD5BQAAQGjEKrHRGTNm+Lx58yqxaXRh1vytkqSbzhle4UiAULEgVkKdGl7U3cBeedenFWn53bBhQyU2CwBViToVAPJHtwcAAACERkW6PQBAoS554Ni85rv3wsUljgQA0B8V3fJrZvuZ2UIzW2Fmy83s+iACAwAAAIIWRMtvStJX3H2xmQ2V9JyZLXD3lwJYNwAAABCYopNfd18raW1ueLuZrZBUJ4nkF31CvqfJJU6VA+gbelNv1Q1YWMJIgOoT6AVvZjZJ0jRJzwS5XgAAACAIgV3wZmZDJM2V9EV339bJ9JmSZkrSxIkTg9osAIQSdSqA9rggOD+BJL9mFlc28Z3j7vd0No+7z5Y0W5Lq6+s9iO0CQFhRp6I/ISlDXxLE3R5M0q2SVrj7D4sPCQAAACiNIPr8Tpd0laQzzWxp7uf8ANYLAAAABCqIuz08qYCeTw8AAACUEo83BgAAQGiQ/AIAACA0ArvVGQAAQH/CQ5DCiZZfAAAAhAYtv5C7a8qmNRqS2C1PTpXFeVsAAIDqRJYTcp5KK/n7P+m6Z5dJkhJrFit+7UcU2WdEhSMDEGY8FKEHLp2ybpLqN9Rp7egX5amTZbFopaMC+gWS3xDzXXuUvP0+ZVa+3TJu/SYlfnKnaq65VJFJ+1YwOgBAZ2rSUX3u5RP0/sb9syM2/FWJX76tmk9dLBsyqLLBAf0AyW9IZTZtVfLmu+XrNnacuGOXEj+/S/GPf0jRow8pWQxcaAAA9WFJZwAAHilJREFUvbPPnkH6yrJTNHnHqDbj/Y3VavrRr1VzzSWK1I2rUHRA/8AFbyGUeWutEj++o/PEt1kqpeSv7lfqz8/I3csXHACgU4dsGaPvPXdOh8R3r83blPjpHKWXvlzewIB+huQ3ZNIvvqbEz38r7djVZvyqEeO0cPLRHeZP/fExpe6eL09nyhUiAKCdD645QP/y/Ac0PDmgzfimSKrtjMmUkr9+QMmHnpBnaLgAOkPyGxLurtRjDUrefq+UbFtZRo45VP91/AV68JCTFLviPCnS9m2Rfvp5JW+dK9/TVM6QASD0opmIrnn1OH321eMV87Z187OjV+sfTr5fT0w8ssNy6UeeVvK/76HeBjpBn98Q8ExGqfv+rPSTHfvNRs88UbHzT1PqkW2SpNgJR8lGDMsmyXsSe+fLvLxKiRt/o5prPyIbMbRssQNAWA1N1OpLy6frsK1jO0ybu/8yzZ20TG7S/YdP1wdOmqDU3PlSq7N0meWvK/GTOxW/5hJFxnTRVQKhF8Y7q9DyW+W8KaHkf9/bMfGNmGKXn6v4BafLItZmUvTg/VXzj5+QRg5ru661jWr68R3KvLuu1GEDQKjtv32EvvfcOR0S3z2RpH50xJO6e3I28W0WO2mqaq67Uho6uM38vm6jEj++Q+lXVpUjbKBfoOW3ivm2HUrcMle+ul2yWluj+NUXKXro5C6XjYwfrdrrP6HErffI33mvZcK2HUrc+BvFP3mhoocfUKLIERbc8QPoKL30Zf2fJWepNtP2EL1+wA7955FP6O0hWztdLjK5TrVfvEqJ/763bb2/u0nJ2XfLP3yGoqfXy8w6XR4IC5LfKpVZ26jELXOlzdvaThgxVDWfvUyRfTueRmvPhg1RzXVXKDnnj8osW9kyIZFU8tZ75Jeepdj0aQFHjv6KRDYcwniKtFw840rNe1LpR55WbbvD8/IR6/STw5/S9ppEF0tn2chhqpn1MSV/N0+ZxStardyVemChMmvWK375uf3iSZ6811Aqgbz7zWyGpJ9Iikq6xd2/H8R6e8tTaWVee6sSm9Z3F83Kc07TN0+6sePoQm8n1sk3eN++U6n7Hm3TZ1eSrG6saj57mWx4/n12rbZG8U9drNQDC5V+/Lk28abmLpCv26jIwft3HksPr4/eOD7vONIr3sh73vbKtZ2qkMlIqbSUSslTaSmdzr3O/nhumtKZlr8mKRbTJ946RqlIRinLKBnJKBVJK2nZv6lIRknLKJkbbt7Phfxv8l2m3P/LyAH7yWriZd1mqfnmbcq8t6HNuEL2f7714zdPuqmo7fR36b8uUWb56x3G/6nuVd1xwBKlI/kdJ6wmrvjHL1B637FKPfiY1GqxTMNyJdZtVOzc6Z0ePwKVyWQvsE6mdNa7ByieiaomE1NNJqradFQ1mWh2XG64NhNT4t3/keIxqSamz689UYloSolIOvsTTbcMR9J7p4X++JDJSImkPJHUuasP0oB0TLXpWMvfTLvX6Zialt8s1cSl2hp9bctp2hNNqSmS1p5oUk3RdPZ17ic7LVXW8tuAWkUm15Vu/cXew9XMopJelXS2pNWSnpV0pbu/1NUy9fX13tDQUND2uvsmOCRZo5ufurSg9Va7yOFTFL/qQlltTafTZ83Pnka76ZzhXa4j9cTibFLNfX+BDmr++VpFRo/s7WKBZB/F1KndST21RKm5CwJfL/KTsrRuO+g5Ldy3+6SjbsBC3XTO8E6Pj0dvHK8vvPR+DU53XvcDfZFNGKfaL1/d68XynTGIC95OkLTS3d9w94SkuyRdFMB6EZDo+49R/NOXdpn45it26rGKX3NJ9tsiAKBktsR36zvH/LnHxLcnz+/znr553AK9O3BbzzMDIRFEt4c6Se+0er1a0ontZzKzmZJmStLEiRML3lh3fXt85241PdVJl4KwMin24TN0+dbPSw/2PHvdgIV7h7trYZ901Ej904unaWRiYBBRAihAEHVqT30qz373QF2j+oLWjcLZhHEa9+lL9G8jv9XjvM1n7aQejo8XNSl55x+U6aun7oEyCiL57ayZucN5cXefLWm2lD1FF8B2O4pGFDmk7R0Mlqz/a96LTxv7/oKX6ZXcHlu8/qk2o7vaKceNPWXv8HPrn2y/mk4dW3eGoicfnb2jwwO9D7E7bw7drG8cN18XvX2Yzhtxvpojf/a9x7NxeXN8bSM0SeZW2D7LKfn/ply6eA905dix03u/jVZvqHz327T3TZeiMT2xbkGu726uv24krZRl2gwnIxmlLaPrjv6XbJ/gdOt+wa37C6c6jitzz5m8y1/Ee6bcFxCVo07dWLtLz49cK6mPf55yyvF/Lul2oqbIwZMUPenowPuP28BaxT9zqdINy7MXMLd72FFv5F3+8dOz/XfjseznIx5vNdzqdU3LsEx7+wl7MiUlkm1fJ5NSIiVPpXLjk1K65e1fjvdAKfOK1nHlvcy46VJtPPueyfXjtZq4VJt9bTU1ueGalnmiESmRkJqyfYWVSMqbEtn93Twu9zo7PSGlWu4hXdLyjH2/bPSIvOYtVBB9fk+W9G13Pzf3+gZJcvd/7WqZUvVP60whV6CX66r1Qq5kLeUyzf3Gyhlbb1XbHQX66tXM7OeyqWifX/7PfXs7+cjneo2g9aXyt9dXj0N99ZhaqD5anrzr0yCaKp6VdJCZTZb0rqQrJH0sgPVWTH+o5AEAANB7RSe/7p4ys1mS/qTsrc5uc/flRUeGfqccXxr4YgIAQP/Tl47fgXRSc/eHJD0UxLqC1pd2NgAA6N/KlVeQv5RO33/EC9rgwwAAQP/CsbtvIfkFgJDigAwgjEh+AQBVgdPRAPIRxBPeAAAAgH6Bll8AVY+WOgBAM5JfAACQN75Mor8j+QUAAEDe+vsXIJJfoIL6ewUCAH0JdSryQfJbQXxIAQAAyovkF50iMQcAANWIW50BAAAgNGj5BdABLf8AgGpFyy8AAABCg+QXAAAAoVFU8mtm/25mL5vZC2Z2r5mNCCowAAAAIGjF9vldIOkGd0+Z2Q8k3SDpa8WHhSDl039z1vytZYgEAACgsopq+XX3+e6eyr1cJGlC8SEBAAAApRFkn99rJD3c1UQzm2lmDWbW0NjYGOBmASB8qFMBoDA9Jr9m9oiZLevk56JW83xDUkrSnK7W4+6z3b3e3evHjBkTTPQAEFLUqQBQmB77/Lr7Wd1NN7OrJV0g6YPu7kEFBgAAAAStqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqPOENAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNq8RD2cysUdJbJd7MaEkbSryNvizs5ZfYB5S/75d/g7vPKHYl1KllQfnDXX6JfdDXy593fVqR5LcczKzB3esrHUelhL38EvuA8oe7/EEL+/6k/OEuv8Q+qKby0+0BAAAAoUHyCwAAgNCo5uR3dqUDqLCwl19iH1B+BCns+5PyI+z7oGrKX7V9fgEAAID2qrnlFwAAAGiD5BcAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBoxCqx0RkzZvi8efMqselQmDV/qyTppnOGVzgSAD2wIFZCnQrkj2Nk1cq7Pq1Iy++GDRsqsVkAqErUqQCQP7o9AAAAIDRIfgEAABAaRSe/ZrafmS00sxVmttzMrg8iMAAAACBoQVzwlpL0FXdfbGZDJT1nZgvc/aUA1g0AAAAEpuiWX3df6+6Lc8PbJa2QVFfsegEAAICgBdrn18wmSZom6ZlOps00swYza2hsbAxyswAQOtSpAFCYwJJfMxsiaa6kL7r7tvbT3X22u9e7e/2YMWOC2iwAhBJ1KgAUJpDk18ziyia+c9z9niDWCQAAAAQtiLs9mKRbJa1w9x8WHxIAAABQGkG0/E6XdJWkM81sae7n/ADWCwAAAASq6FudufuTCuj59Pm45IFj85rv3gsXlzgSAAAA9Dc84Q0AAAChEcRDLgAAACou37PDdQMWljgS9GUkvwAAoKTosoi+hG4PAAAACA1afquMb9qqa557WGN3blEqOk3RD5wgi5TtekQAAIA+jeS3imRWv6fEzXN1+PadkqTUg48ps2a94leeJ4vxrwbQf3CaHECpkBFVifSKN5T81f1SItlmfGbJCiW27VDNpy+RDRqQ9/o48KCa8H4GADQj+a0CqUXPK3X3fCnjnU73199R4sY5qrn2I7JRw8scHdC1fJNSicQUQPEiGdOU7aM0ILNdEsfDQvX3BgWS337M3ZWa96TSC57uMG1HfICGJPe0zLtuo5p+cqdqrr1MkQnjyxkmAAAVd+SmcfrUymNVt2u4pN8oseNwxS88QzZsSKVDQ5mR/PZTnkor+bt5yjQsbzvBpHsPna4l7ztQ3131iHzVuy3Ttu9U4qbfKn71RYoeNqW8AQMAUAH77Bmkq1ZO04kb9mszPrP4JTUtX6nYudMVPfVYWTRaoQhRbiS//ZDvblLy9vuUee2tthNiMcWv+rCeWjtWklTz+Y8q+ZsHlXn+lZZ5Ekklb50r/8g5ip10dBmjBoC+p7+fvkXXYpmIPvTOobr4rcM1INNFutOUUOqBhUo/84Jil56l6EH7lzfIPiCMnwGS337Gt2xX4ua75Wsb204YPFA1n7lMkUn7Smu3SpIsHlP8qguVGrFQ6ccaWubNuFK/+5N88zbFZpwiM26F1p+EsaICUP2GJmrk23ZIQwcXfVw6euN4Xb3yWL1v97C85vd1G5X8xf8ofcyhil/4AdmIoUVtH30byW8/klmzXomb75a27mgz3kaPUPzayxUZM7LDMhYxxS86UzZyuFL3Pyq1uiYuveBp+eZtiv/dDFms+NM9JGUAgHzF01EdvmWMjto8Xkdvep8m7Bqupr/+XDZquCKHTlbk4EmKHLS/bGBt3uscvXuwPrlymo7fOKHT6SuHbtSvD1ysaXu+rUtWNUh7mtpMzyx9WU0vva7YOe9X9LT6QI6N6HtIfvuJ9CtvKnn7fVJTos14239f1XzmUtmQQd0uHzvtONmIoUre+Ucpldo7PtOwXMmtOxT/1MW9qmAAAOgVl/bbOVxTN43X1M3v06FbxqjGOyaXvmmr0n9dqvRfl0oRk+1fp+ihkxQ5eLJsv3GySMeH03oypfSfn9F/Pnueajrp4rA91qS7pjyvhe97Q27SrnFH6oorjlHqj48p/eyytjMnktnxf3tRsUvOUvSQSUHtAfQRgSS/ZjZD0k8kRSXd4u7fD2K9yEo/u0zJ/5knZTJtxkeOOkjxj18gq4nntZ7o1INl131UiVvvkXbu3js+89pbStz0m+yt0DjVAwDdqqazXKUui+/Ypcyrb+rzK07U1M3jNTIxsHcryLh81WqlVq2WHn5SGjQg2yJ8yCRFD5ksGzFU6eUrlbrvz/KNW1TTLq3JyPXoviv1P5Nf1M54u8ajoYMVv/J8RU8+Wsm5C+Tvrm8b+/pNSv7X75SeenDuDGp+XSjQ9xWd/JpZVNLPJJ0tabWkZ83sAXd/qdh199aAVEyffu04SVJix4Ol36B7rhtB7q97y7CUu+9u63la9Tkwk2SS5Yab+zdZbpxy4xJJZZa91mHT0VOPU+yiD3T6Dbg7kUl1qvnHjys5+275xi0tRVnbqKYf36HIwdnO/n//zol5ra/1fi5kmbx4J/cv7jCq83sct9VJH7L2o/pB/+dC9vNf3vljXsucsd8FvQvGc+/rTPPfTKthlzz3OpORt54v957/5qYzlTHP/WSUUXbYTXuHm3+K+Uz3dp/FzjxRkfGjC95etUi/+qbS7e8oU0oZlzJpKZ3RV9ecqlgmoqhHFPOIohlrGXZTNDdtz9JfZOvBaESKRrN/Iy3D1jyueXokIkVa6tw+VddVQCnrbV+3Uf7uOsml0zW5x0UyyigSjUvpdNcz7dqjzNKXlVn6slKSNGKotGV7p7O+OmyD/vug5/Tm0M3dbjcyqU41X/qk0k8/r9RDT0i797SZnnnhVTWteEORIw/MvoeqTN7vge2544hL/7D6pDbTLHcwbXMEdSmx5QFZbY3iH50RQKTBMe8ssejNCsxOlvRtdz839/oGSXL3f+1qmfr6em9oaOhqcre6+5Y6JFmjm5+6tKD19iexCz+g6On1XV4QMGt+9oK3m87p+gbevmOXErfMlb+9tiQxAv1Z/PMfVfTgslz1Hci3rWLq1O6knlqi1NwFga8XaNZYu1PPj1qrF0a9p+Uj1unOC55Q5o3Vyry8SplXVsnXb+r1OrfG9+i3U57X4+NXybv4hNUNWNjpMdJ37FLqoceVfuaF/NpU0LPBAzXgu18ox5byrk+D6PZQJ+mdVq9XS+rwNcLMZkqaKUkTJ04seGPdnXrxnbvV9NSNBa+7z4tF9eODH9cz2+6S/tD9rHUDFu4d7uoLQ83+Uc3aeXKXFwYA6LuCqFN7OuV99rsH6hrVF7RuoFM1cUUO3E+RQyYrcshkTRgzUvuZqfU5p+hhUxQ9bIoueeBY7TNlUK6P8HgduXm8hqRqul63maLTp2nseafoiwMH6ItdzNbcQCR18RkYIB0wbZQ+9dpxOnD7PgUVE60U2chaCkEkv51l2h1PSrvPljRbyrZSBLDdcBk0QDXXXKpnls0JbJWJaFo/OvIpfXLlNM149+DA1gug9KhT0V9Y3dhcsjtJkcl1slj+qcfGAbu0cN83tHDfN2RuOmDbKB29ebyO2jReB23bRxFlu/7Z5AmKX3qWInVjA4n59WGb9L+PXaAPrJ2iK944WkNTXBBeTYJIfldLav3YlAmS1gSw3t6rjSt+5flFr+YnS/533vNef+x3JUk/WvIvcnm2669lj0PZHr/Z/oue+z7gJt1w/A8ld/3g2a/KPNtXpvkbhHl22GTKrsb0xRP+nyIHTsze0aHdRanFcnP96qDFemLcm/q3Q34c7Mp7UMh+/vHib7Ya2/Z43/701peO/b+SpB8t/pfcvmzPOry6ftp38o6pvXzL03obhSxTDr3+30RM/7H4hmz/3L39dr1Dv93m198/5b+zF7JkMi39g1sPe+vXuX7DvYytmH0WGU9rjyQtG7lOPz90kaSW/VnS9/mx323VZ7dVv91WfXjb9N2NRmRm8nRGymT0pUc/oqhH9vYHjrm1em25/sIRfemY7/ZyT5Rfrz6DRfxvCpH3dk76fjbZHTq4qO01c3OtHL5RK4dv1NxJyzUoGdfB20brm6f9UjZlQuD3rHeT/rzvG/rr2Ld1xJaxuuGowq/lL9fxoeTvgc72sbUbaD2LWZ/sJx1E8vuspIPMbLKkdyVdIeljAay31ywWU/T4I4tezxNr38x73i/XHyFJemrNWz3M2SJ61EGSpGffXJ3X/F855tC8112oN4ZtCmTf9UYh+/nJNfkv89Xjcv+bd/P/33y5iH2Qb3lab6OQZcqhkP/NotXv9DBni8jkwrva9NV9Vo3WDtqutYOyFxM178+Svs9z76Xeaj7Wvj1ka7fzNftqP3hvfPn4e3q9TLk+G3lvZ2ppzyjuiie1dJ+1ihywX88zd6Fcd+Qo1/GB+jE/RSe/7p4ys1mS/qTsrc5uc/cyXh4MAChEf7gVF/oe3jflwX4unUDu8+vuD0l6KIh19QW84QCgc9SPfRf/GyA/POENFUNF3Xf3QV+NCwD6I+rUvoXkFwgIlVt5sJ8BAMUg+Q0BkgUAAICs3j0bFwAAAOjHaPmtIFpky4P9DAD9D3U3SoXkFwAAoArwhSE/dHsAAABAaNDyCwAoKVqjAPQltPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFR1K3OzOzfJX1YUkLS65I+7e5bgggM6Ay3TALQFeoHAPko9j6/CyTd4O4pM/uBpBskfa34sPofKl0AAIC+r6jk193nt3q5SNJHigsHQH/FF0AAQH8QZJ/fayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeuz24+1ndTTezqyVdIOmD7u5BBQYAAAAErdi7PcxQ9gK30919VzAhAQAAAKVR7N0ebpJUK2mBmUnSInf/fNFRoUv5XFQ0a/7WMkQCAADQ/xR7t4cDgwoEAAAAKDWe8AYAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAELDKvFEYjNrlPRWiTczWtKGEm+jLwt7+SX2AeXv++Xf4O4zil0JdWpZUP5wl19iH/T18uddn1Yk+S0HM2tw9/pKx1EpYS+/xD6g/OEuf9DCvj8pf7jLL7EPqqn8dHsAAABAaJD8AgAAIDSqOfmdXekAKizs5ZfYB5QfQQr7/qT8CPs+qJryV22fXwAAAKC9am75BQAAANog+QUAAEBoVF3ya2YzzOwVM1tpZl+vdDyVYGZvmtmLZrbUzBoqHU+pmdltZrbezJa1GjfKzBaY2Wu5vyMrGWOpdbEPvm1m7+beB0vN7PxKxlhKZrafmS00sxVmttzMrs+ND9X7oBSoU6lTc+NC81miPq3++rSqkl8zi0r6maTzJB0u6UozO7yyUVXMB9z9mGq5J18PbpfU/sbWX5f0qLsfJOnR3Otqdrs67gNJ+lHufXCMuz9U5pjKKSXpK+5+mKSTJP1D7rMftvdBoKhT26BODc9n6XZRn1Z1fVpVya+kEyStdPc33D0h6S5JF1U4JpSYuz8uaVO70RdJ+lVu+FeSLi5rUGXWxT4IDXdf6+6Lc8PbJa2QVKeQvQ9KgDo1hMJep1KfVn99Wm3Jb52kd1q9Xp0bFzYuab6ZPWdmMysdTIWMc/e1UvaDLGlsheOplFlm9kLuNF6/PUXVG2Y2SdI0Sc+I90GxqFOzqFP5LEnUp1XzHqi25Nc6GRfGe7lNd/djlT1V+Q9mdlqlA0JF/ELSAZKOkbRW0n9WNpzSM7MhkuZK+qK7b6t0PFWAOjWLOhXUp1Wk2pLf1ZL2a/V6gqQ1FYqlYtx9Te7vekn3KnvqMmzWmdn7JCn3d32F4yk7d1/n7ml3z0i6WVX+PjCzuLIV9Rx3vyc3OvTvgyJRp4o6NSfUnyXqU0lV9B6otuT3WUkHmdlkM6uRdIWkByocU1mZ2WAzG9o8LOkcScu6X6oqPSDp6tzw1ZLur2AsFdFcSeVcoip+H5iZSbpV0gp3/2GrSaF/HxSJOpU6tVmoP0vUp5Kq6D1QdU94y91+5MeSopJuc/fvVTiksjKzKcq2TEhSTNJvqn0fmNlvJZ0habSkdZK+Jek+Sb+TNFHS25Iud/eqvYChi31whrKn6FzSm5I+19xfq9qY2SmSnpD0oqRMbvQ/K9tPLTTvg1KgTqVOVcjqVOrT6q9Pqy75BQAAALpSbd0eAAAAgC6R/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS9Cx8w+b2afzA1/ysz2bTXtFjM7vHLRAUD/QX2K/ohbnSHUzOwvkr7q7g2VjgUA+jPqU/QXtPyiXzGzSWb2spn9ysxeMLO7zWyQmX3QzJaY2YtmdpuZ1ebm/76ZvZSb9z9y475tZl81s49Iqpc0x8yWmtlAM/uLmdXn5rsyt75lZvaDVjHsMLPvmdnzZrbIzMZVYl8AQDGoTxFWJL/ojw6RNNvdp0raJunLkm6X9FF3P0rZpzD9vZmNUvYxlEfk5v2/rVfi7ndLapD0cXc/xt13N0/Lnbr7gaQzlX2qz/FmdnFu8mBJi9z9aEmPS7q2ZCUFgNKiPkXokPyiP3rH3Z/KDd8p6YOSVrn7q7lxv5J0mrIV+R5Jt5jZpZJ29WIbx0v6i7s3untK0pzcOiUpIemPueHnJE0qtCAAUGHUpwgdkl/0R3l1VM9VsidImivpYknzerEN62Za0ls6y6eVbRkBgP6I+hShQ/KL/miimZ2cG75S0iOSJpnZgblxV0l6zMyGSBru7g9J+qKyp9va2y5paCfjn5F0upmNNrNobjuPBVkIAOgDqE8ROnzDQn+0QtLVZvZfkl6TdL2kRZJ+b2YxSc9K+qWkUZLuN7MByrY8fKmTdd0u6ZdmtltS8wFA7r7WzG6QtDC37EPufn/pigQAFUF9itDhVmfoV8xskqQ/uvuRFQ4FAPo16lOEFd0eAAAAEBq0/AIAACA0aPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhMb/B0Hm1BpCSN+pAAAAAElFTkSuQmCC\n", 272 | "text/plain": [ 273 | "
" 274 | ] 275 | }, 276 | "metadata": { 277 | "needs_background": "light" 278 | }, 279 | "output_type": "display_data" 280 | } 281 | ], 282 | "source": [ 283 | "query, context, target, lengths, target_indices = valid_batches[0]\n", 284 | "\n", 285 | "with torch.no_grad():\n", 286 | " weight, output = net(query, context, lengths=lengths, return_weight=True)\n", 287 | "\n", 288 | "context = context.numpy()\n", 289 | "weight = weight.data.numpy()\n", 290 | "\n", 291 | "colors = sns.color_palette('husl', 3)\n", 292 | "\n", 293 | "fig, axs = plt.subplots(batch_size, 2, figsize=(10, 10), sharex=True, sharey=True)\n", 294 | "sns.despine(fig=fig)\n", 295 | "axs_min, axs_max = axs[:,0], axs[:,1]\n", 296 | "\n", 297 | "for i, (i_min, i_max) in enumerate(target_indices):\n", 298 | " length = lengths[i]\n", 299 | " w_min, w_max = weight[i]\n", 300 | " c_min = [context[i,j,Data.min_position] for j in range(length)]\n", 301 | " c_max = [context[i,j,Data.max_position] for j in range(length)]\n", 302 | "\n", 303 | " axs_min[i].axvline(i_min, zorder=1, color=colors[2], label='min value')\n", 304 | " axs_min[i].bar(np.arange(length) - 0.4, c_min, zorder=2, color=colors[1], lw=0, label='values')\n", 305 | " axs_min[i].plot(np.arange(length), w_min[:length], zorder=3, color=colors[0], lw=4, label='attention')\n", 306 | "\n", 307 | " axs_max[i].axvline(i_max, zorder=1, color=colors[2], label='max value')\n", 308 | " axs_max[i].bar(np.arange(length) - 0.4, c_max, zorder=2, color=colors[1], lw=0, label='values')\n", 309 | " axs_max[i].plot(np.arange(length), w_max[:length], zorder=3, color=colors[0], lw=4, label='attention')\n", 310 | "\n", 311 | "axs_min[0].set_title('Minimum')\n", 312 | "axs_max[0].set_title('Maximum')\n", 313 | "axs_max[0].legend(loc='best') \n", 314 | "axs_min[0].legend(loc='best')\n", 315 | "axs_max[0].legend(loc='best')\n", 316 | "axs_min[-1].set_xlabel('position')\n", 317 | "axs_max[-1].set_xlabel('position')\n", 318 | "plt.tight_layout()" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": {}, 325 | "outputs": [], 326 | "source": [] 327 | } 328 | ], 329 | "metadata": { 330 | "kernelspec": { 331 | "display_name": "Python 3", 332 | "language": "python", 333 | "name": "python3" 334 | }, 335 | "language_info": { 336 | "codemirror_mode": { 337 | "name": "ipython", 338 | "version": 3 339 | }, 340 | "file_extension": ".py", 341 | "mimetype": "text/x-python", 342 | "name": "python", 343 | "nbconvert_exporter": "python", 344 | "pygments_lexer": "ipython3", 345 | "version": "3.7.1" 346 | } 347 | }, 348 | "nbformat": 4, 349 | "nbformat_minor": 1 350 | } 351 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | 3 | setup( 4 | name='attention', 5 | version='0.1.0', 6 | author='tllake', 7 | author_email='thom.l.lake@gmail.com', 8 | packages=['attention'], 9 | description='An attention function for PyTorch.', 10 | long_description=open('README.md').read()) -------------------------------------------------------------------------------- /svgs/190083ef7a1625fbc75f243cffb9c96d.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /svgs/1eb39a281b1e66935a51005b6beb9dbe.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /svgs/28e6b84adb66aca59d04ec9e227bfd3f.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | -------------------------------------------------------------------------------- /svgs/39c9d05724010ea29be9eb321b1422ec.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | -------------------------------------------------------------------------------- /svgs/39d2a848a943a7f5ec27272dad27c784.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /svgs/3cf4fbd05970446973fc3d9fa3fe3c41.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /svgs/5397f1268e113895a997a61e51165ffc.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | -------------------------------------------------------------------------------- /svgs/a5a09669219f681bb51e176b190b0e4a.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /svgs/a5d4c0a87edcc90e9dc7bb8a1845e86a.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | -------------------------------------------------------------------------------- /svgs/da2cf8b162672dc46adcace06ec2740a.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | -------------------------------------------------------------------------------- /svgs/e73485aa867794d51ccd8725055d03a3.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /test/test_attention.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pytest 3 | 4 | import torch 5 | from attention import attention 6 | 7 | 8 | def test_apply_mask_3d(): 9 | batch_size, m, n = 3, 4, 5 10 | sizes = [4, 3, 2] 11 | values = torch.randn(batch_size, m, n) 12 | masked = attention.mask3d(values, sizes=sizes).data 13 | assert values.size() == masked.size() == (batch_size, m, n) 14 | for i in range(batch_size): 15 | for j in range(m): 16 | for k in range(n): 17 | if j < sizes[i]: 18 | assert masked[i,j,k] == values[i,j,k] 19 | else: 20 | assert masked[i,j,k] == 0 21 | 22 | 23 | @pytest.mark.parametrize('v_mask, v_unmask', [(0, 1), (float('-inf'), 0)]) 24 | def test_fill_context_mask(v_mask, v_unmask): 25 | batch_size, n_q, n_c = 3, 4, 5 26 | query_sizes = [4, 3, 2] 27 | context_sizes = [3, 2, 5] 28 | mask = torch.randn(batch_size, n_q, n_c) 29 | mask = attention.fill_context_mask( 30 | mask, sizes=context_sizes, 31 | v_mask=v_mask, v_unmask=v_unmask) 32 | 33 | for i in range(batch_size): 34 | for j in range(n_q): 35 | for k in range(n_c): 36 | if k < context_sizes[i]: 37 | assert mask[i,j,k] == v_unmask 38 | else: 39 | assert mask[i,j,k] == v_mask 40 | 41 | 42 | def test_dot(): 43 | batch_size, n_q, n_c, d = 31, 18, 15, 22 44 | q = np.random.normal(0, 1, (batch_size, n_q, d)) 45 | c = np.random.normal(0, 1, (batch_size, n_c, d)) 46 | 47 | s = attention.dot(torch.from_numpy(q), 48 | torch.from_numpy(c) 49 | ) 50 | s = s.data.numpy() 51 | 52 | assert s.shape == (batch_size, n_q, n_c) 53 | 54 | for i in range(batch_size): 55 | for j in range(n_q): 56 | for k in range(n_c): 57 | assert np.allclose(np.dot(q[i,j], c[i,k]), s[i,j,k]) 58 | 59 | 60 | @pytest.mark.parametrize( 61 | 'batch_size,n_q,n_c,d', [ 62 | (1, 1, 6, 11), 63 | (20, 1, 10, 3), 64 | (3, 10, 15, 5)]) 65 | def test_attention(batch_size, n_q, n_c, d): 66 | q = np.random.normal(0, 1, (batch_size, n_q, d)) 67 | c = np.random.normal(0, 1, (batch_size, n_c, d)) 68 | 69 | w_out, z_out = attention.attend(torch.from_numpy(q), 70 | torch.from_numpy(c), 71 | return_weight=True 72 | ) 73 | w_out = w_out.data.numpy() 74 | z_out = z_out.data.numpy() 75 | 76 | assert w_out.shape == (batch_size, n_q, n_c) 77 | assert z_out.shape == (batch_size, n_q, d) 78 | 79 | for i in range(batch_size): 80 | for j in range(n_q): 81 | s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)] 82 | max_s = max(s) 83 | exp_s = [np.exp(si - max_s) for si in s] 84 | sum_exp_s = sum(exp_s) 85 | 86 | w_ref = [ei / sum_exp_s for ei in exp_s] 87 | assert np.allclose(w_ref, w_out[i,j]) 88 | 89 | z_ref = sum(w_ref[k] * c[i,k] for k in range(n_c)) 90 | assert np.allclose(z_ref, z_out[i,j]) 91 | 92 | 93 | @pytest.mark.parametrize( 94 | 'batch_size,n_q,n_c,d,p', [ 95 | (1, 1, 6, 11, 5), 96 | (20, 1, 10, 3, 14), 97 | (3, 10, 15, 5, 9)]) 98 | def test_attention_values(batch_size, n_q, n_c, d, p): 99 | q = np.random.normal(0, 1, (batch_size, n_q, d)) 100 | c = np.random.normal(0, 1, (batch_size, n_c, d)) 101 | v = np.random.normal(0, 1, (batch_size, n_c, p)) 102 | 103 | w_out, z_out = attention.attend(torch.from_numpy(q), 104 | torch.from_numpy(c), 105 | value=torch.from_numpy(v), 106 | return_weight=True 107 | ) 108 | w_out = w_out.data.numpy() 109 | z_out = z_out.data.numpy() 110 | 111 | assert w_out.shape == (batch_size, n_q, n_c) 112 | assert z_out.shape == (batch_size, n_q, p) 113 | 114 | for i in range(batch_size): 115 | for j in range(n_q): 116 | s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)] 117 | max_s = max(s) 118 | exp_s = [np.exp(si - max_s) for si in s] 119 | sum_exp_s = sum(exp_s) 120 | 121 | w_ref = [ei / sum_exp_s for ei in exp_s] 122 | assert np.allclose(w_ref, w_out[i,j]) 123 | 124 | z_ref = sum(w_ref[k] * v[i,k] for k in range(n_c)) 125 | assert np.allclose(z_ref, z_out[i,j]) 126 | 127 | 128 | @pytest.mark.parametrize( 129 | 'batch_size,n_q,n_c,d,context_sizes', [ 130 | (1, 1, 6, 11, [3]), 131 | (4, 1, 10, 3, [7, 5, 10, 9])]) 132 | def test_attention_masked(batch_size, n_q, n_c, d, context_sizes): 133 | q = np.random.normal(0, 1, (batch_size, n_q, d)) 134 | c = np.random.normal(0, 1, (batch_size, n_c, d)) 135 | 136 | w_out, z_out = attention.attend(torch.from_numpy(q), 137 | torch.from_numpy(c), 138 | context_sizes=context_sizes, 139 | return_weight=True 140 | ) 141 | w_out = w_out.data.numpy() 142 | z_out = z_out.data.numpy() 143 | 144 | assert w_out.shape == (batch_size, n_q, n_c) 145 | assert z_out.shape == (batch_size, n_q, d) 146 | 147 | w_checked = np.zeros((batch_size, n_q, n_c), dtype=int) 148 | z_checked = np.zeros((batch_size, n_q, d), dtype=int) 149 | 150 | for i in range(batch_size): 151 | for j in range(n_q): 152 | n = context_sizes[i] if context_sizes is not None else n_c 153 | 154 | s = [np.dot(q[i,j], c[i,k]) for k in range(n)] 155 | max_s = max(s) 156 | exp_s = [np.exp(sk - max_s) for sk in s] 157 | sum_exp_s = sum(exp_s) 158 | 159 | w_ref = [ek / sum_exp_s for ek in exp_s] 160 | for k in range(n_c): 161 | if k < n: 162 | assert np.allclose(w_ref[k], w_out[i,j,k]) 163 | w_checked[i,j,k] = 1 164 | else: 165 | assert np.allclose(0, w_out[i,j,k]) 166 | w_checked[i,j,k] = 1 167 | 168 | z_ref = sum(w_ref[k] * c[i,k] for k in range(n)) 169 | for k in range(d): 170 | assert np.allclose(z_ref[k], z_out[i,j,k]) 171 | z_checked[i,j,k] = 1 172 | 173 | assert np.all(w_checked == 1) 174 | assert np.all(z_checked == 1) 175 | --------------------------------------------------------------------------------