├── .gitignore ├── LICENSE ├── README.md ├── ttq_densenet_big ├── get_densenet.py ├── model_ternary_quantization.pytorch_state ├── test_and_explore.ipynb └── train.ipynb ├── ttq_densenet_small ├── get_densenet.py ├── model_ternary_quantization.pytorch_state ├── test_and_explore.ipynb └── train.ipynb ├── ttq_squeezenet ├── get_squeezenet.py ├── model_ternary_quantization.pytorch_state ├── test_and_explore.ipynb └── train.ipynb ├── utils ├── __init__.py ├── diagnostic.py ├── input_pipeline.py ├── move_tiny_imagenet_data.py ├── quantization.py ├── show_augmentation.ipynb └── training.py ├── vanilla_densenet_big ├── README.md ├── densenet.py ├── get_densenet.py ├── model_step5.pytorch_state ├── train_step1.ipynb ├── train_step2.ipynb ├── train_step3.ipynb ├── train_step4.ipynb └── train_step5.ipynb ├── vanilla_densenet_small ├── densenet.py ├── get_densenet.py ├── model.pytorch_state └── train.ipynb └── vanilla_squeezenet ├── error_analysis.ipynb ├── get_squeezenet.py ├── model.pytorch_state ├── squeezenet.py └── train.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | __pycache__ 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Dan Antoshchenko 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Trained Ternary Quantization 2 | `pytorch` implementation of [Trained Ternary Quantization](https://arxiv.org/abs/1612.01064), a way of replacing full precision weights of a neural network by ternary values. I tested it on [Tiny ImageNet](https://tiny-imagenet.herokuapp.com/) dataset. The dataset consists of 64x64 images and has 200 classes. 3 | 4 | The quantization roughly proceeds as follows. 5 | 1. Train a model of your choice as usual (or take a trained model). 6 | 2. Copy all full precision weights that you want to quantize. Then do the initial quantization: 7 | in the model replace them by ternary values {-1, 0, +1} using some heuristic. 8 | 3. Repeat until convergence: 9 | * Make the forward pass with the quantized model. 10 | * Compute gradients for the quantized model. 11 | * Preprocess the gradients and apply them to the copy of full precision weights. 12 | * Requantize the model using the changed full precision weights. 13 | 4. Throw away the copy of full precision weights and use the quantized model. 14 | 15 | ## Results 16 | I believe that this results can be made better by spending more time on hyperparameter optimization. 17 | 18 | | model | accuracy, % | top5 accuracy, % | number of parameters | 19 | | --- | --- | --- | --- | 20 | | DenseNet-121 | 74 | 91 | 7151176 | 21 | | TTQ DenseNet-121 | 66 | 87 | ~7M 2-bit, 88% are zeros | 22 | | small DenseNet | 49 | 75 | 440264 | 23 | | TTQ small DenseNet | 40 | 67 | ~0.4M 2-bit, 38% are zeros | 24 | | SqueezeNet | 52 | 77 | 827784 | 25 | | TTQ SqueezeNet | 36 | 63 | ~0.8M 2-bit, 66% are zeros | 26 | 27 | ## Implementation details 28 | * I use pretrained DenseNet-121, but I train SqueezeNet and small DenseNet from scratch. 29 | * I modify the SqueezeNet architecture by adding batch normalizations and skip connections. 30 | * I quantize all layers except the first CONV layer, the last FC layer, and all BATCH_NORM layers. 31 | 32 | 33 | ## How to reproduce results 34 | For example, for small DenseNet: 35 | 1. Download [Tiny ImageNet](https://tiny-imagenet.herokuapp.com/) dataset and extract it to `~/data` folder. 36 | 2. Run `python utils/move_tiny_imagenet_data.py` to prepare the data. 37 | 3. Go to `vanilla_densenet_small/`. Run `train.ipynb` to train the model as usual. 38 | Or you can skip this step and use `model.pytorch_state` (the model already trained by me). 39 | 4. Go to `ttq_densenet_small/`. 40 | 5. Run `train.ipynb` to do TTQ. 41 | 6. Run `test_and_explore.ipynb` to explore the quantized model. 42 | 43 | To use this on your data you need to edit `utils/input_pipeline.py` and to change 44 | the model architecture in files like `densenet.py` and `get_densenet.py` as you like. 45 | 46 | ## Requirements 47 | * pytorch 0.2, Pilllow, torchvision 48 | * numpy, sklearn, tqdm, matplotlib 49 | -------------------------------------------------------------------------------- /ttq_densenet_big/get_densenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.optim as optim 3 | 4 | import sys 5 | sys.path.append('../vanilla_densenet_big/') 6 | from densenet import DenseNet 7 | 8 | 9 | def get_model(learning_rate=1e-3): 10 | 11 | # get DenseNet-121 12 | model = DenseNet() 13 | 14 | # set the first layer not trainable 15 | model.features.conv0.weight.requires_grad = False 16 | 17 | # the last fc layer 18 | weights = [ 19 | p for n, p in model.named_parameters() 20 | if 'classifier.weight' in n 21 | ] 22 | biases = [model.classifier.bias] 23 | 24 | # all conv layers except the first 25 | weights_to_be_quantized = [ 26 | p for n, p in model.named_parameters() 27 | if 'conv' in n and ('dense' in n or 'transition' in n) 28 | ] 29 | 30 | # parameters of batch_norm layers 31 | bn_weights = [ 32 | p for n, p in model.named_parameters() 33 | if 'norm' in n and 'weight' in n 34 | ] 35 | bn_biases = [ 36 | p for n, p in model.named_parameters() 37 | if 'norm' in n and 'bias' in n 38 | ] 39 | 40 | params = [ 41 | {'params': weights, 'weight_decay': 1e-4}, 42 | {'params': weights_to_be_quantized}, 43 | {'params': biases}, 44 | {'params': bn_weights}, 45 | {'params': bn_biases} 46 | ] 47 | optimizer = optim.Adam(params, lr=learning_rate) 48 | 49 | loss = nn.CrossEntropyLoss().cuda() 50 | model = model.cuda() # move the model to gpu 51 | return model, loss, optimizer 52 | -------------------------------------------------------------------------------- /ttq_densenet_big/model_ternary_quantization.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/ttq_densenet_big/model_ternary_quantization.pytorch_state -------------------------------------------------------------------------------- /ttq_densenet_big/train.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%load_ext autoreload\n", 10 | "%autoreload 2" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "data": { 20 | "text/plain": [ 21 | "True" 22 | ] 23 | }, 24 | "execution_count": 2, 25 | "metadata": {}, 26 | "output_type": "execute_result" 27 | } 28 | ], 29 | "source": [ 30 | "from math import ceil\n", 31 | "import torch\n", 32 | "from torch.utils.data import DataLoader\n", 33 | "from torch.autograd import Variable\n", 34 | "import torch.optim as optim\n", 35 | "\n", 36 | "import matplotlib.pyplot as plt\n", 37 | "%matplotlib inline\n", 38 | "\n", 39 | "import sys\n", 40 | "sys.path.append('..')\n", 41 | "from utils.input_pipeline import get_image_folders\n", 42 | "from utils.training import train\n", 43 | "from utils.quantization import optimization_step, quantize, initial_scales\n", 44 | "\n", 45 | "torch.cuda.is_available()" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 3, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "torch.backends.cudnn.benchmark = True" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 4, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "LEARNING_RATE = 1e-4 # learning rate for all possible weights\n", 64 | "HYPERPARAMETER_T = 0.15 # hyperparameter for quantization" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "# Create data iterators" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 5, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "batch_size = 64" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 6, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "100000" 92 | ] 93 | }, 94 | "execution_count": 6, 95 | "metadata": {}, 96 | "output_type": "execute_result" 97 | } 98 | ], 99 | "source": [ 100 | "train_folder, val_folder = get_image_folders()\n", 101 | "\n", 102 | "train_iterator = DataLoader(\n", 103 | " train_folder, batch_size=batch_size, num_workers=4,\n", 104 | " shuffle=True, pin_memory=True\n", 105 | ")\n", 106 | "\n", 107 | "val_iterator = DataLoader(\n", 108 | " val_folder, batch_size=256, num_workers=4,\n", 109 | " shuffle=False, pin_memory=True\n", 110 | ")\n", 111 | "\n", 112 | "# number of training samples\n", 113 | "train_size = len(train_folder.imgs)\n", 114 | "train_size" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "# Model" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 7, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "from get_densenet import get_model" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 8, 136 | "metadata": { 137 | "scrolled": true 138 | }, 139 | "outputs": [], 140 | "source": [ 141 | "model, loss, optimizer = get_model(learning_rate=LEARNING_RATE)\n", 142 | "\n", 143 | "# load pretrained model, accuracy ~73%\n", 144 | "model.load_state_dict(torch.load('../vanilla_densenet_big/model_step5.pytorch_state'))" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "#### keep copy of full precision kernels" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 9, 157 | "metadata": { 158 | "scrolled": true 159 | }, 160 | "outputs": [], 161 | "source": [ 162 | "# copy almost all full precision kernels of the model\n", 163 | "all_fp_kernels = [\n", 164 | " Variable(kernel.data.clone(), requires_grad=True) \n", 165 | " for kernel in optimizer.param_groups[1]['params']\n", 166 | "]\n", 167 | "# all_fp_kernels - kernel tensors of all convolutional layers \n", 168 | "# (with the exception of the first conv layer)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": {}, 174 | "source": [ 175 | "#### initial quantization " 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": 10, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "# scaling factors for each quantized layer\n", 185 | "initial_scaling_factors = []" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 11, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "# these kernels will be quantized\n", 195 | "all_kernels = [kernel for kernel in optimizer.param_groups[1]['params']]" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 12, 201 | "metadata": {}, 202 | "outputs": [], 203 | "source": [ 204 | "for k, k_fp in zip(all_kernels, all_fp_kernels):\n", 205 | " \n", 206 | " # choose initial scaling factors \n", 207 | " w_p_initial, w_n_initial = initial_scales(k_fp.data)\n", 208 | " initial_scaling_factors += [(w_p_initial, w_n_initial)]\n", 209 | " \n", 210 | " # do quantization\n", 211 | " k.data = quantize(k_fp.data, w_p_initial, w_n_initial, t=HYPERPARAMETER_T)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "#### parameter updaters" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 13, 224 | "metadata": {}, 225 | "outputs": [], 226 | "source": [ 227 | "# optimizer for updating only all_fp_kernels\n", 228 | "optimizer_fp = optim.Adam(all_fp_kernels, lr=LEARNING_RATE)" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": 14, 234 | "metadata": {}, 235 | "outputs": [], 236 | "source": [ 237 | "# optimizer for updating only scaling factors\n", 238 | "optimizer_sf = optim.Adam([\n", 239 | " Variable(torch.FloatTensor([w_p, w_n]).cuda(), requires_grad=True) \n", 240 | " for w_p, w_n in initial_scaling_factors\n", 241 | "], lr=LEARNING_RATE)" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": {}, 247 | "source": [ 248 | "# Train" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": 15, 254 | "metadata": {}, 255 | "outputs": [ 256 | { 257 | "data": { 258 | "text/plain": [ 259 | "1563" 260 | ] 261 | }, 262 | "execution_count": 15, 263 | "metadata": {}, 264 | "output_type": "execute_result" 265 | } 266 | ], 267 | "source": [ 268 | "from torch.optim.lr_scheduler import ReduceLROnPlateau\n", 269 | "\n", 270 | "class lr_scheduler_list:\n", 271 | " \"\"\"ReduceLROnPlateau for a list of optimizers.\"\"\"\n", 272 | " def __init__(self, optimizer_list):\n", 273 | " self.lr_scheduler_list = [\n", 274 | " ReduceLROnPlateau(\n", 275 | " optimizer, mode='max', factor=0.1, patience=3, \n", 276 | " verbose=True, threshold=0.01, threshold_mode='abs'\n", 277 | " ) \n", 278 | " for optimizer in optimizer_list\n", 279 | " ]\n", 280 | " \n", 281 | " def step(self, test_accuracy):\n", 282 | " for scheduler in self.lr_scheduler_list:\n", 283 | " scheduler.step(test_accuracy)\n", 284 | "\n", 285 | "n_epochs = 15\n", 286 | "n_batches = ceil(train_size/batch_size)\n", 287 | "\n", 288 | "# total number of batches in the train set\n", 289 | "n_batches" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": 16, 295 | "metadata": { 296 | "scrolled": false 297 | }, 298 | "outputs": [ 299 | { 300 | "name": "stdout", 301 | "output_type": "stream", 302 | "text": [ 303 | "0 3.605 2.637 0.213 0.369 0.454 0.656 674.036\n", 304 | "1 2.571 2.173 0.384 0.473 0.670 0.744 668.391\n", 305 | "2 2.276 2.121 0.444 0.487 0.724 0.751 667.756\n", 306 | "3 2.116 1.922 0.478 0.520 0.751 0.784 668.168\n", 307 | "4 1.995 1.934 0.505 0.522 0.772 0.783 668.347\n", 308 | "5 1.914 1.824 0.522 0.546 0.786 0.802 668.592\n", 309 | "6 1.843 1.749 0.538 0.567 0.797 0.812 668.261\n", 310 | "7 1.788 1.845 0.549 0.541 0.806 0.795 668.585\n", 311 | "8 1.737 1.718 0.561 0.571 0.814 0.812 668.550\n", 312 | "9 1.693 1.861 0.571 0.547 0.819 0.792 668.989\n", 313 | "10 1.655 1.689 0.579 0.583 0.825 0.819 668.097\n", 314 | "11 1.619 1.647 0.587 0.590 0.831 0.824 668.579\n", 315 | "12 1.586 1.545 0.594 0.616 0.836 0.844 668.487\n", 316 | "13 1.552 1.699 0.602 0.575 0.842 0.814 668.539\n", 317 | "14 1.521 1.600 0.609 0.601 0.846 0.833 668.464\n", 318 | "CPU times: user 2h 56min 43s, sys: 27min 18s, total: 3h 24min 2s\n", 319 | "Wall time: 2h 47min 11s\n" 320 | ] 321 | } 322 | ], 323 | "source": [ 324 | "%%time\n", 325 | "optimizer_list = [optimizer, optimizer_fp, optimizer_sf]\n", 326 | "\n", 327 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 328 | " return optimization_step(\n", 329 | " model, loss, x_batch, y_batch, \n", 330 | " optimizer_list=optimizer_list,\n", 331 | " t=HYPERPARAMETER_T\n", 332 | " )\n", 333 | "all_losses = train(\n", 334 | " model, loss, optimization_step_fn,\n", 335 | " train_iterator, val_iterator, n_epochs,\n", 336 | " lr_scheduler=lr_scheduler_list(optimizer_list) \n", 337 | ")\n", 338 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": 17, 344 | "metadata": {}, 345 | "outputs": [], 346 | "source": [ 347 | "# backup\n", 348 | "model.cpu();\n", 349 | "torch.save(model.state_dict(), 'model_ternary_quantization.pytorch_state')" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "# Continue training" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": 18, 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "# reduce learning rate\n", 366 | "for optimizer in optimizer_list:\n", 367 | " for group in optimizer.param_groups:\n", 368 | " group['lr'] = 1e-5" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "execution_count": 22, 374 | "metadata": {}, 375 | "outputs": [], 376 | "source": [ 377 | "n_epochs = 5\n", 378 | "model.cuda();" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": 23, 384 | "metadata": {}, 385 | "outputs": [ 386 | { 387 | "name": "stdout", 388 | "output_type": "stream", 389 | "text": [ 390 | "0 1.312 1.362 0.660 0.654 0.875 0.866 649.648\n", 391 | "1 1.243 1.341 0.675 0.661 0.885 0.868 651.109\n", 392 | "2 1.222 1.366 0.678 0.657 0.889 0.866 651.324\n", 393 | "3 1.207 1.334 0.682 0.664 0.890 0.870 651.090\n", 394 | "4 1.195 1.350 0.685 0.659 0.891 0.869 651.020\n", 395 | "CPU times: user 57min 55s, sys: 8min 45s, total: 1h 6min 41s\n", 396 | "Wall time: 54min 14s\n" 397 | ] 398 | } 399 | ], 400 | "source": [ 401 | "%%time\n", 402 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 403 | " return optimization_step(\n", 404 | " model, loss, x_batch, y_batch, \n", 405 | " optimizer_list=optimizer_list,\n", 406 | " t=HYPERPARAMETER_T\n", 407 | " )\n", 408 | "all_losses = train(\n", 409 | " model, loss, optimization_step_fn,\n", 410 | " train_iterator, val_iterator, n_epochs \n", 411 | ")\n", 412 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 413 | ] 414 | }, 415 | { 416 | "cell_type": "markdown", 417 | "metadata": {}, 418 | "source": [ 419 | "# Final save" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": 24, 425 | "metadata": {}, 426 | "outputs": [], 427 | "source": [ 428 | "model.cpu();\n", 429 | "torch.save(model.state_dict(), 'model_ternary_quantization.pytorch_state')" 430 | ] 431 | } 432 | ], 433 | "metadata": { 434 | "kernelspec": { 435 | "display_name": "Python 3", 436 | "language": "python", 437 | "name": "python3" 438 | }, 439 | "language_info": { 440 | "codemirror_mode": { 441 | "name": "ipython", 442 | "version": 3 443 | }, 444 | "file_extension": ".py", 445 | "mimetype": "text/x-python", 446 | "name": "python", 447 | "nbconvert_exporter": "python", 448 | "pygments_lexer": "ipython3", 449 | "version": "3.5.2" 450 | } 451 | }, 452 | "nbformat": 4, 453 | "nbformat_minor": 2 454 | } 455 | -------------------------------------------------------------------------------- /ttq_densenet_small/get_densenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.optim as optim 3 | 4 | import sys 5 | sys.path.append('../vanilla_densenet_small/') 6 | from densenet import DenseNet 7 | 8 | 9 | def get_model(learning_rate=1e-3): 10 | 11 | model = DenseNet( 12 | growth_rate=12, block_config=(8, 12, 10), 13 | num_init_features=48, bn_size=4, drop_rate=0.25, 14 | final_drop_rate=0.25, num_classes=200 15 | ) 16 | 17 | # set the first layer not trainable 18 | model.features.conv0.weight.requires_grad = False 19 | 20 | # the last fc layer 21 | weights = [ 22 | p for n, p in model.named_parameters() 23 | if 'classifier.weight' in n 24 | ] 25 | biases = [model.classifier.bias] 26 | 27 | # all conv layers except the first 28 | weights_to_be_quantized = [ 29 | p for n, p in model.named_parameters() 30 | if 'conv' in n and ('dense' in n or 'transition' in n) 31 | ] 32 | 33 | # parameters of batch_norm layers 34 | bn_weights = [ 35 | p for n, p in model.named_parameters() 36 | if 'norm' in n and 'weight' in n 37 | ] 38 | bn_biases = [ 39 | p for n, p in model.named_parameters() 40 | if 'norm' in n and 'bias' in n 41 | ] 42 | 43 | params = [ 44 | {'params': weights, 'weight_decay': 1e-4}, 45 | {'params': weights_to_be_quantized}, 46 | {'params': biases}, 47 | {'params': bn_weights}, 48 | {'params': bn_biases} 49 | ] 50 | optimizer = optim.Adam(params, lr=learning_rate) 51 | 52 | loss = nn.CrossEntropyLoss().cuda() 53 | model = model.cuda() # move the model to gpu 54 | return model, loss, optimizer 55 | -------------------------------------------------------------------------------- /ttq_densenet_small/model_ternary_quantization.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/ttq_densenet_small/model_ternary_quantization.pytorch_state -------------------------------------------------------------------------------- /ttq_squeezenet/get_squeezenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.optim as optim 3 | 4 | import sys 5 | sys.path.append('../vanilla_squeezenet/') 6 | from squeezenet import SqueezeNet 7 | 8 | 9 | def get_model(learning_rate=1e-3): 10 | 11 | model = SqueezeNet() 12 | 13 | # set the first layer not trainable 14 | model.features[0].weight.requires_grad = False 15 | 16 | # all conv layers except the first and the last 17 | all_conv_weights = [ 18 | (n, p) for n, p in model.named_parameters() 19 | if 'weight' in n and not 'bn' in n and not 'features.1.' in n 20 | ] 21 | weights_to_be_quantized = [ 22 | p for n, p in all_conv_weights 23 | if not ('classifier' in n or 'features.0.' in n) 24 | ] 25 | 26 | # the last layer 27 | weights = [model.classifier[1].weight] 28 | biases = [model.classifier[1].bias] 29 | 30 | # parameters of batch_norm layers 31 | bn_weights = [ 32 | p for n, p in model.named_parameters() 33 | if ('bn' in n or 'features.1.' in n) and 'weight' in n 34 | ] 35 | bn_biases = [ 36 | p for n, p in model.named_parameters() 37 | if ('bn' in n or 'features.1.' in n) and 'bias' in n 38 | ] 39 | 40 | params = [ 41 | {'params': weights, 'weight_decay': 1e-4}, 42 | {'params': weights_to_be_quantized}, 43 | {'params': biases}, 44 | {'params': bn_weights}, 45 | {'params': bn_biases} 46 | ] 47 | optimizer = optim.Adam(params, lr=learning_rate) 48 | 49 | # loss function 50 | criterion = nn.CrossEntropyLoss().cuda() 51 | # move the model to gpu 52 | model = model.cuda() 53 | return model, criterion, optimizer 54 | -------------------------------------------------------------------------------- /ttq_squeezenet/model_ternary_quantization.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/ttq_squeezenet/model_ternary_quantization.pytorch_state -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/utils/__init__.py -------------------------------------------------------------------------------- /utils/diagnostic.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from sklearn.calibration import calibration_curve 3 | import matplotlib.pyplot as plt 4 | from sklearn.metrics import confusion_matrix 5 | from tqdm import tqdm 6 | from torch.autograd import Variable 7 | import torch.nn.functional as F 8 | 9 | 10 | """Tools for diagnostic of a learned model. 11 | 12 | Arguments: 13 | true: a numpy array of shape (n_samples,) of type int 14 | with integers in range 0..(n_classes - 1). 15 | 16 | pred: a numpy array of shape (n_samples, n_classes) of type float, 17 | represents probabilities. 18 | 19 | decode: a dict that maps a class index to a human readable form. 20 | """ 21 | 22 | 23 | NUM_CLASSES = 200 24 | NUM_VAL_SAMPLES_PER_CLASS = 50 # number of samples per class in the validation dataset 25 | 26 | 27 | def top_k_accuracy(true, pred, k=[2, 3, 4, 5]): 28 | n_samples = len(true) 29 | hits = [] 30 | for i in k: 31 | hits += [np.equal(pred.argsort(1)[:, -i:], true.reshape(-1, 1)).sum()/n_samples] 32 | return hits 33 | 34 | 35 | def per_class_accuracy(true, pred): 36 | 37 | true_ohehot = np.zeros((len(true), NUM_CLASSES)) 38 | for i in range(len(true)): 39 | true_ohehot[i, true[i]] = 1.0 40 | 41 | pred_onehot = np.equal(pred, pred.max(1).reshape(-1, 1)).astype('int') 42 | 43 | per_class_acc = (true_ohehot*pred_onehot).sum(0)/float(NUM_VAL_SAMPLES_PER_CLASS) 44 | return per_class_acc # shape: [NUM_CLASSES] 45 | 46 | 47 | def most_inaccurate_k_classes(per_class_acc, k, decode): 48 | most = per_class_acc.argsort()[:k] 49 | for i in most: 50 | print(decode[i], per_class_acc[i]) 51 | 52 | 53 | def entropy(pred): 54 | 55 | prob = pred.astype('float64') 56 | log = np.log2(prob) 57 | result = -(prob*log).sum(1) 58 | 59 | return result 60 | 61 | 62 | def model_calibration(true, pred, n_bins=10): 63 | """ 64 | On Calibration of Modern Neural Networks, 65 | https://arxiv.org/abs/1706.04599 66 | """ 67 | 68 | hits = np.equal(pred.argmax(1), true) 69 | 70 | fraction_of_positives, mean_predicted_value = calibration_curve( 71 | hits, pred.max(1), n_bins=n_bins 72 | ) 73 | 74 | plt.plot(mean_predicted_value, fraction_of_positives, '-ok') 75 | plt.plot([0.0, 1.0], [0.0, 1.0], '--') 76 | plt.xlim([0.0, 1.0]) 77 | plt.ylim([0.0, 1.0]) 78 | plt.xlabel('confidence') 79 | plt.ylabel('accuracy') 80 | plt.title('reliability curve') 81 | 82 | 83 | def count_params(model): 84 | # model - pytorch's nn.Module object 85 | count = 0 86 | for p in model.parameters(): 87 | count += p.numel() 88 | return count 89 | 90 | 91 | def most_confused_classes(val_true, val_pred, decode, min_n_confusions): 92 | 93 | conf_mat = confusion_matrix(val_true, val_pred.argmax(1)) 94 | 95 | # not interested in correct predictions 96 | conf_mat -= np.diag(conf_mat.diagonal()) 97 | 98 | # confusion(class A -> class B) + confusion(class B -> class A) 99 | conf_mat += conf_mat.T 100 | 101 | confused_pairs = np.where(np.triu(conf_mat) >= min_n_confusions) 102 | confused_pairs = [(k, confused_pairs[1][i]) for i, k in enumerate(confused_pairs[0])] 103 | confused_pairs = [(decode[i], decode[j]) for i, j in confused_pairs] 104 | 105 | return confused_pairs 106 | 107 | 108 | def predict(model, val_iterator_no_shuffle, return_erroneous=False): 109 | 110 | # predictions on the validation dataset 111 | val_predictions = [] 112 | # it will have shape [n_val_samples, n_classes] 113 | 114 | # corresponding true targets 115 | val_true_targets = [] 116 | # it will have shape [n_val_samples] 117 | 118 | # also you can return samples where 119 | # the model is making mistakes 120 | if return_erroneous: 121 | erroneous_samples = [] 122 | erroneous_targets = [] 123 | erroneous_predictions = [] 124 | 125 | model.eval() 126 | 127 | for x_batch, y_batch in tqdm(val_iterator_no_shuffle): 128 | 129 | x_batch = Variable(x_batch.cuda(), volatile=True) 130 | y_batch = Variable(y_batch.cuda(), volatile=True) 131 | logits = model(x_batch) 132 | 133 | # compute probabilities 134 | probs = F.softmax(logits) 135 | 136 | if return_erroneous: 137 | _, argmax = probs.max(1) 138 | hits = argmax.eq(y_batch).data 139 | miss = 1 - hits 140 | if miss.nonzero().numel() != 0: 141 | miss = miss.nonzero()[:, 0] # indices of errors 142 | erroneous_samples += [x_batch[miss].cpu().data.numpy()] 143 | erroneous_targets += [y_batch[miss].cpu().data.numpy()] 144 | erroneous_predictions += [probs[miss].cpu().data.numpy()] 145 | 146 | val_predictions += [probs.cpu().data.numpy()] 147 | val_true_targets += [y_batch.cpu().data.numpy()] 148 | 149 | val_predictions = np.concatenate(val_predictions, axis=0) 150 | val_true_targets = np.concatenate(val_true_targets, axis=0) 151 | 152 | if return_erroneous: 153 | erroneous_samples = np.concatenate(erroneous_samples, axis=0) 154 | erroneous_targets = np.concatenate(erroneous_targets, axis=0) 155 | erroneous_predictions = np.concatenate(erroneous_predictions, axis=0) 156 | return val_predictions, val_true_targets,\ 157 | erroneous_samples, erroneous_targets, erroneous_predictions 158 | 159 | return val_predictions, val_true_targets 160 | -------------------------------------------------------------------------------- /utils/input_pipeline.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from PIL import Image, ImageEnhance 3 | from torchvision.datasets import ImageFolder 4 | import torchvision.transforms as transforms 5 | 6 | 7 | TRAIN_DIR = '/home/ubuntu/data/tiny-imagenet-200/training' 8 | VAL_DIR = '/home/ubuntu/data/tiny-imagenet-200/validation' 9 | 10 | 11 | """It assumes that training image data is in the following form: 12 | TRAIN_DIR/class4/image44.jpg 13 | TRAIN_DIR/class4/image12.jpg 14 | ... 15 | TRAIN_DIR/class55/image33.jpg 16 | TRAIN_DIR/class55/image543.jpg 17 | ... 18 | TRAIN_DIR/class1/image6.jpg 19 | TRAIN_DIR/class1/image99.jpg 20 | ... 21 | 22 | And the same for validation data. 23 | """ 24 | 25 | 26 | def get_image_folders(): 27 | """ 28 | Build an input pipeline for training and evaluation. 29 | For training data it does data augmentation. 30 | """ 31 | 32 | enhancers = { 33 | 0: lambda image, f: ImageEnhance.Color(image).enhance(f), 34 | 1: lambda image, f: ImageEnhance.Contrast(image).enhance(f), 35 | 2: lambda image, f: ImageEnhance.Brightness(image).enhance(f), 36 | 3: lambda image, f: ImageEnhance.Sharpness(image).enhance(f) 37 | } 38 | 39 | # intensities of enhancers 40 | factors = { 41 | 0: lambda: np.clip(np.random.normal(1.0, 0.3), 0.4, 1.6), 42 | 1: lambda: np.clip(np.random.normal(1.0, 0.15), 0.7, 1.3), 43 | 2: lambda: np.clip(np.random.normal(1.0, 0.15), 0.7, 1.3), 44 | 3: lambda: np.clip(np.random.normal(1.0, 0.3), 0.4, 1.6), 45 | } 46 | 47 | # randomly change color of an image 48 | def enhance(image): 49 | order = [0, 1, 2, 3] 50 | np.random.shuffle(order) 51 | # random enhancers in random order 52 | for i in order: 53 | f = factors[i]() 54 | image = enhancers[i](image, f) 55 | return image 56 | 57 | def rotate(image): 58 | degree = np.clip(np.random.normal(0.0, 15.0), -40.0, 40.0) 59 | return image.rotate(degree, Image.BICUBIC) 60 | 61 | # training data augmentation on the fly 62 | train_transform = transforms.Compose([ 63 | transforms.Lambda(rotate), 64 | transforms.RandomCrop(56), 65 | transforms.RandomHorizontalFlip(), 66 | transforms.Lambda(enhance), 67 | transforms.ToTensor(), 68 | transforms.Normalize( 69 | mean=[0.485, 0.456, 0.406], 70 | std=[0.229, 0.224, 0.225] 71 | ), 72 | ]) 73 | 74 | # for validation data 75 | val_transform = transforms.Compose([ 76 | transforms.CenterCrop(56), 77 | transforms.ToTensor(), 78 | transforms.Normalize( 79 | mean=[0.485, 0.456, 0.406], 80 | std=[0.229, 0.224, 0.225] 81 | ), 82 | ]) 83 | 84 | # mean and std are taken from here: 85 | # http://pytorch.org/docs/master/torchvision/models.html 86 | 87 | train_folder = ImageFolder(TRAIN_DIR, train_transform) 88 | val_folder = ImageFolder(VAL_DIR, val_transform) 89 | return train_folder, val_folder 90 | -------------------------------------------------------------------------------- /utils/move_tiny_imagenet_data.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import os 3 | import shutil 4 | from tqdm import tqdm 5 | import argparse 6 | 7 | 8 | """The purpose of this script is to arrange tiny-imagenet data in the following way: 9 | data_dir/training/n03444034/image_name10.JPEG 10 | data_dir/training/n03444034/image_name13.JPEG 11 | data_dir/training/n03444034/image_name15.JPEG 12 | ... 13 | data_dir/training/n04067472/image_name16.JPEG 14 | data_dir/training/n04067472/image_name123.JPEG 15 | data_dir/training/n04067472/image_name93.JPEG 16 | ... 17 | And in the same way arrange validation data. 18 | """ 19 | 20 | 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument( 23 | '--data_dir', type=str, 24 | default='/home/ubuntu/data/tiny-imagenet-200/', 25 | help='A path to the tiny-imagenet-200 folder.' 26 | ) 27 | args = parser.parse_args() 28 | 29 | 30 | # a folder from tiny-imagenet-200.zip file 31 | data_dir = args.data_dir 32 | # inside there are folders 'train' and 'val'. 33 | 34 | 35 | # load validation metadata 36 | annotations_file = os.path.join(data_dir, 'val', 'val_annotations.txt') 37 | val_data = pd.read_csv(annotations_file, sep='\t', header=None) 38 | val_data.drop([2, 3, 4, 5], axis=1, inplace=True) # drop bounding boxes info 39 | val_data.columns = ['img_name', 'img_class'] 40 | unique_classes = val_data.img_class.unique() 41 | 42 | 43 | print('moving validation data') 44 | 45 | # create new folders to move the data into 46 | validation_dir = os.path.join(data_dir, 'validation') 47 | os.mkdir(validation_dir) 48 | for name in unique_classes: 49 | os.mkdir(os.path.join(validation_dir, name)) 50 | 51 | # loop over all classes 52 | for name in tqdm(unique_classes): 53 | # choose images only from a specific class 54 | class_images = val_data.loc[val_data.img_class == name, 'img_name'].values 55 | # copy these images to a new folder 56 | for img in class_images: 57 | shutil.copyfile( 58 | os.path.join(data_dir, 'val', 'images', img), 59 | os.path.join(validation_dir, name, img) 60 | ) 61 | 62 | 63 | print('\nmoving training data') 64 | 65 | # create new folders to move data into 66 | training_dir = os.path.join(data_dir, 'training') 67 | os.mkdir(training_dir) 68 | for name in unique_classes: 69 | os.mkdir(os.path.join(training_dir, name)) 70 | 71 | # loop over all classes 72 | for name in tqdm(unique_classes): 73 | # choose images only from a specific class 74 | class_images = os.listdir(os.path.join(data_dir, 'train', name, 'images')) 75 | # copy these images to a new folder 76 | for img in class_images: 77 | shutil.copyfile( 78 | os.path.join(data_dir, 'train', name, 'images', img), 79 | os.path.join(training_dir, name, img) 80 | ) 81 | 82 | print('\nvalidation data is in', validation_dir) 83 | print('training data is in', training_dir) 84 | -------------------------------------------------------------------------------- /utils/quantization.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.autograd import Variable 3 | import torch.nn.functional as F 4 | from .training import _accuracy 5 | 6 | 7 | def initial_scales(kernel): 8 | return 1.0, 1.0 9 | 10 | 11 | def quantize(kernel, w_p, w_n, t): 12 | """ 13 | Return quantized weights of a layer. 14 | Only possible values of quantized weights are: {zero, w_p, -w_n}. 15 | """ 16 | delta = t*kernel.abs().max() 17 | a = (kernel > delta).float() 18 | b = (kernel < -delta).float() 19 | return w_p*a + (-w_n*b) 20 | 21 | 22 | def get_grads(kernel_grad, kernel, w_p, w_n, t): 23 | """ 24 | Arguments: 25 | kernel_grad: gradient with respect to quantized kernel. 26 | kernel: corresponding full precision kernel. 27 | w_p, w_n: scaling factors. 28 | t: hyperparameter for quantization. 29 | 30 | Returns: 31 | 1. gradient for the full precision kernel. 32 | 2. gradient for w_p. 33 | 3. gradient for w_n. 34 | """ 35 | delta = t*kernel.abs().max() 36 | # masks 37 | a = (kernel > delta).float() 38 | b = (kernel < -delta).float() 39 | c = torch.ones(kernel.size()).cuda() - a - b 40 | # scaled kernel grad and grads for scaling factors (w_p, w_n) 41 | return w_p*a*kernel_grad + w_n*b*kernel_grad + 1.0*c*kernel_grad,\ 42 | (a*kernel_grad).sum(), (b*kernel_grad).sum() 43 | 44 | 45 | def optimization_step(model, loss, x_batch, y_batch, optimizer_list, t): 46 | """Make forward pass and update model parameters with gradients.""" 47 | 48 | # parameter 't' is a hyperparameter for quantization 49 | 50 | # 'optimizer_list' contains optimizers for 51 | # 1. full model (all weights including quantized weights), 52 | # 2. backup of full precision weights, 53 | # 3. scaling factors for each layer 54 | optimizer, optimizer_fp, optimizer_sf = optimizer_list 55 | 56 | x_batch, y_batch = Variable(x_batch.cuda()), Variable(y_batch.cuda(async=True)) 57 | # forward pass using quantized model 58 | logits = model(x_batch) 59 | 60 | # compute logloss 61 | loss_value = loss(logits, y_batch) 62 | batch_loss = loss_value.data[0] 63 | 64 | # compute accuracies 65 | pred = F.softmax(logits) 66 | batch_accuracy, batch_top5_accuracy = _accuracy(y_batch, pred, top_k=(1, 5)) 67 | 68 | optimizer.zero_grad() 69 | optimizer_fp.zero_grad() 70 | optimizer_sf.zero_grad() 71 | # compute grads for quantized model 72 | loss_value.backward() 73 | 74 | # get all quantized kernels 75 | all_kernels = optimizer.param_groups[1]['params'] 76 | 77 | # get their full precision backups 78 | all_fp_kernels = optimizer_fp.param_groups[0]['params'] 79 | 80 | # get two scaling factors for each quantized kernel 81 | scaling_factors = optimizer_sf.param_groups[0]['params'] 82 | 83 | for i in range(len(all_kernels)): 84 | 85 | # get a quantized kernel 86 | k = all_kernels[i] 87 | 88 | # get corresponding full precision kernel 89 | k_fp = all_fp_kernels[i] 90 | 91 | # get scaling factors for the quantized kernel 92 | f = scaling_factors[i] 93 | w_p, w_n = f.data[0], f.data[1] 94 | 95 | # get modified grads 96 | k_fp_grad, w_p_grad, w_n_grad = get_grads(k.grad.data, k_fp.data, w_p, w_n, t) 97 | 98 | # grad for full precision kernel 99 | k_fp.grad = Variable(k_fp_grad) 100 | 101 | # we don't need to update the quantized kernel directly 102 | k.grad.data.zero_() 103 | 104 | # grad for scaling factors 105 | f.grad = Variable(torch.FloatTensor([w_p_grad, w_n_grad]).cuda()) 106 | 107 | # update all non quantized weights in quantized model 108 | # (usually, these are the last layer, the first layer, and all batch norm params) 109 | optimizer.step() 110 | 111 | # update all full precision kernels 112 | optimizer_fp.step() 113 | 114 | # update all scaling factors 115 | optimizer_sf.step() 116 | 117 | # update all quantized kernels with updated full precision kernels 118 | for i in range(len(all_kernels)): 119 | 120 | k = all_kernels[i] 121 | k_fp = all_fp_kernels[i] 122 | f = scaling_factors[i] 123 | w_p, w_n = f.data[0], f.data[1] 124 | 125 | # requantize a quantized kernel using updated full precision weights 126 | k.data = quantize(k_fp.data, w_p, w_n, t) 127 | 128 | return batch_loss, batch_accuracy, batch_top5_accuracy 129 | -------------------------------------------------------------------------------- /utils/training.py: -------------------------------------------------------------------------------- 1 | from torch.autograd import Variable 2 | import torch.nn.functional as F 3 | import time 4 | 5 | 6 | def optimization_step(model, loss, x_batch, y_batch, optimizer): 7 | """Make forward pass and update model parameters with gradients.""" 8 | 9 | # forward pass 10 | x_batch, y_batch = Variable(x_batch.cuda()), Variable(y_batch.cuda(async=True)) 11 | logits = model(x_batch) 12 | 13 | # compute logloss 14 | loss_value = loss(logits, y_batch) 15 | batch_loss = loss_value.data[0] 16 | 17 | # compute accuracies 18 | pred = F.softmax(logits) 19 | batch_accuracy, batch_top5_accuracy = _accuracy(y_batch, pred, top_k=(1, 5)) 20 | 21 | # compute gradients 22 | optimizer.zero_grad() 23 | loss_value.backward() 24 | 25 | # update params 26 | optimizer.step() 27 | 28 | return batch_loss, batch_accuracy, batch_top5_accuracy 29 | 30 | 31 | def train(model, loss, optimization_step_fn, 32 | train_iterator, val_iterator, n_epochs=30, 33 | patience=10, threshold=0.01, lr_scheduler=None): 34 | """ 35 | Train 'model' by minimizing 'loss' using 'optimization_step_fn' 36 | for parameter updates. 37 | """ 38 | 39 | # collect losses and accuracies here 40 | all_losses = [] 41 | 42 | running_loss = 0.0 43 | running_accuracy = 0.0 44 | running_top5_accuracy = 0.0 45 | n_steps = 0 46 | start_time = time.time() 47 | model.train() 48 | 49 | for epoch in range(0, n_epochs): 50 | 51 | # main training loop 52 | for x_batch, y_batch in train_iterator: 53 | 54 | batch_loss, batch_accuracy, batch_top5_accuracy = optimization_step_fn( 55 | model, loss, x_batch, y_batch 56 | ) 57 | running_loss += batch_loss 58 | running_accuracy += batch_accuracy 59 | running_top5_accuracy += batch_top5_accuracy 60 | n_steps += 1 61 | 62 | # evaluation 63 | model.eval() 64 | test_loss, test_accuracy, test_top5_accuracy = _evaluate( 65 | model, loss, val_iterator 66 | ) 67 | 68 | # collect evaluation information and print it 69 | all_losses += [( 70 | epoch, 71 | running_loss/n_steps, test_loss, 72 | running_accuracy/n_steps, test_accuracy, 73 | running_top5_accuracy/n_steps, test_top5_accuracy 74 | )] 75 | print('{0} {1:.3f} {2:.3f} {3:.3f} {4:.3f} {5:.3f} {6:.3f} {7:.3f}'.format( 76 | *all_losses[-1], time.time() - start_time 77 | )) 78 | 79 | # it watches test accuracy 80 | # and if accuracy isn't improving then training stops 81 | if _is_early_stopping(all_losses, patience, threshold): 82 | print('early stopping!') 83 | break 84 | 85 | if lr_scheduler is not None: 86 | # possibly change the learning rate 87 | lr_scheduler.step(test_accuracy) 88 | 89 | running_loss = 0.0 90 | running_accuracy = 0.0 91 | running_top5_accuracy = 0.0 92 | n_steps = 0 93 | start_time = time.time() 94 | model.train() 95 | 96 | return all_losses 97 | 98 | 99 | def _accuracy(true, pred, top_k=(1,)): 100 | 101 | max_k = max(top_k) 102 | batch_size = true.size(0) 103 | 104 | _, pred = pred.topk(max_k, 1) 105 | pred = pred.t() 106 | correct = pred.eq(true.view(1, -1).expand_as(pred)) 107 | 108 | result = [] 109 | for k in top_k: 110 | correct_k = correct[:k].view(-1).float().sum(0) 111 | result.append(correct_k.div_(batch_size).data[0]) 112 | 113 | return result 114 | 115 | 116 | def _evaluate(model, loss, val_iterator): 117 | 118 | loss_value = 0.0 119 | accuracy = 0.0 120 | top5_accuracy = 0.0 121 | total_samples = 0 122 | 123 | for x_batch, y_batch in val_iterator: 124 | 125 | x_batch = Variable(x_batch.cuda(), volatile=True) 126 | y_batch = Variable(y_batch.cuda(async=True), volatile=True) 127 | n_batch_samples = y_batch.size()[0] 128 | logits = model(x_batch) 129 | 130 | # compute logloss 131 | batch_loss = loss(logits, y_batch).data[0] 132 | 133 | # compute accuracies 134 | pred = F.softmax(logits) 135 | batch_accuracy, batch_top5_accuracy = _accuracy(y_batch, pred, top_k=(1, 5)) 136 | 137 | loss_value += batch_loss*n_batch_samples 138 | accuracy += batch_accuracy*n_batch_samples 139 | top5_accuracy += batch_top5_accuracy*n_batch_samples 140 | total_samples += n_batch_samples 141 | 142 | return loss_value/total_samples, accuracy/total_samples, top5_accuracy/total_samples 143 | 144 | 145 | def _is_early_stopping(all_losses, patience, threshold): 146 | """It decides if training must stop.""" 147 | 148 | # get current and all past (validation) accuracies 149 | accuracies = [x[4] for x in all_losses] 150 | 151 | if len(all_losses) > (patience + 4): 152 | 153 | # running average with window size 5 154 | average = (accuracies[-(patience + 4)] + 155 | accuracies[-(patience + 3)] + 156 | accuracies[-(patience + 2)] + 157 | accuracies[-(patience + 1)] + 158 | accuracies[-patience])/5.0 159 | 160 | # compare current accuracy with 161 | # running average accuracy 'patience' epochs ago 162 | return accuracies[-1] < (average + threshold) 163 | else: 164 | # if not enough epochs to compare with 165 | return False 166 | -------------------------------------------------------------------------------- /vanilla_densenet_big/README.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | Here I finetuned a pretrained DenseNet-121. Tiny ImageNet consists of 64x64 images, 3 | but the pretrained model expects much bigger images (224x224). So before finetuning 4 | I made some changes to the network architecture. 5 | 6 | ## Preparing the model for finetuning 7 | 1. Changed the first CONV layer: resized filters (7 -> 3), reduced the stride (2 -> 1). 8 | To resize filters I treated them like small 7x7 images and then used `PIL.Image.resize`. 9 | 2. Removed the first pooling layer. 10 | 3. Randomly reinitialized the last FC layer. 11 | 12 | ## Training steps 13 | 14 | | step | layers | optimizer | epochs | time, in hours | accuracy, % | 15 | | --- | --- | --- | --- | --- | --- | 16 | | 1 | only the last FC layer | Adam, lr=1e-3 | 5 | 1 | 45 | 17 | | 2 | the whole network | SGD with nesterov=0.9, lr=1e-5 | 5 | 3 | 65 | 18 | | 3 | the whole network | lr=1e-4 | 5 | 3 | 71 | 19 | | 4 | the whole network | lr=1e-4 | 5 | 3 | 72.8 | 20 | | 5 | the whole network | lr=5e-5 | 3 | 1.5 | 73.6 | 21 | 22 | I used `p2.xlarge` for training. 23 | -------------------------------------------------------------------------------- /vanilla_densenet_big/densenet.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from collections import OrderedDict 5 | 6 | 7 | NUM_CLASSES = 200 8 | FINAL_DROPOUT = 0.25 9 | 10 | 11 | class _DenseLayer(nn.Sequential): 12 | def __init__(self, num_input_features, growth_rate, bn_size, drop_rate): 13 | super(_DenseLayer, self).__init__() 14 | 15 | self.add_module('norm.1', nn.BatchNorm2d(num_input_features)) 16 | self.add_module('relu.1', nn.ReLU(inplace=True)) 17 | self.add_module('conv.1', nn.Conv2d(num_input_features, bn_size*growth_rate, 18 | kernel_size=1, stride=1, bias=False)) 19 | 20 | self.add_module('norm.2', nn.BatchNorm2d(bn_size*growth_rate)) 21 | self.add_module('relu.2', nn.ReLU(inplace=True)) 22 | self.add_module('conv.2', nn.Conv2d(bn_size*growth_rate, growth_rate, 23 | kernel_size=3, stride=1, padding=1, bias=False)) 24 | 25 | self.drop_rate = drop_rate 26 | 27 | def forward(self, x): 28 | new_features = super(_DenseLayer, self).forward(x) 29 | 30 | if self.drop_rate > 0: 31 | new_features = F.dropout(new_features, p=self.drop_rate, training=self.training) 32 | 33 | return torch.cat([x, new_features], 1) 34 | 35 | 36 | class _DenseBlock(nn.Sequential): 37 | def __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate): 38 | super(_DenseBlock, self).__init__() 39 | for i in range(num_layers): 40 | layer = _DenseLayer(num_input_features + i*growth_rate, growth_rate, bn_size, drop_rate) 41 | self.add_module('denselayer%d' % (i + 1), layer) 42 | 43 | 44 | class _Transition(nn.Sequential): 45 | def __init__(self, num_input_features, num_output_features): 46 | super(_Transition, self).__init__() 47 | self.add_module('norm', nn.BatchNorm2d(num_input_features)) 48 | self.add_module('relu', nn.ReLU(inplace=True)) 49 | self.add_module('conv', nn.Conv2d(num_input_features, num_output_features, 50 | kernel_size=1, stride=1, bias=False)) 51 | self.add_module('pool', nn.AvgPool2d(kernel_size=2, stride=2)) 52 | 53 | 54 | class DenseNet(nn.Module): 55 | """Densenet-BC model class, based on 56 | `"Densely Connected Convolutional Networks" ` 57 | 58 | Args: 59 | growth_rate (int) - how many filters to add each layer (`k` in paper). 60 | block_config (list of 4 ints) - how many layers in each pooling block. 61 | num_init_features (int) - the number of filters to learn in the first convolution layer. 62 | bn_size (int) - multiplicative factor for number of bottle neck layers 63 | (i.e. bn_size * k features in the bottleneck layer). 64 | drop_rate (float) - dropout rate after each dense layer. 65 | num_classes (int) - number of classification classes. 66 | """ 67 | def __init__(self, growth_rate=32, block_config=(6, 12, 24, 16), 68 | num_init_features=64, bn_size=4, drop_rate=0, 69 | final_drop_rate=FINAL_DROPOUT, num_classes=NUM_CLASSES): 70 | 71 | super(DenseNet, self).__init__() 72 | 73 | # First convolution 74 | self.features = nn.Sequential(OrderedDict([ 75 | ('conv0', nn.Conv2d(3, num_init_features, 76 | kernel_size=3, stride=1, padding=1, bias=False)), 77 | ('norm0', nn.BatchNorm2d(num_init_features)), 78 | ('relu0', nn.ReLU(inplace=True)) 79 | ])) 80 | # this is different from the original model, because 81 | # i removed pooling layer and changed the first conv layer. 82 | # in the original: kernel_size=7, stride=2, padding=3 83 | 84 | # i use input images with size 56, 85 | # so after first layers spatial size will be also 56 86 | 87 | # Each denseblock 88 | num_features = num_init_features 89 | for i, num_layers in enumerate(block_config): 90 | block = _DenseBlock( 91 | num_layers=num_layers, num_input_features=num_features, 92 | bn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate 93 | ) 94 | self.features.add_module('denseblock%d' % (i + 1), block) 95 | num_features = num_features + num_layers*growth_rate 96 | if i != len(block_config) - 1: 97 | trans = _Transition( 98 | num_input_features=num_features, 99 | num_output_features=num_features // 2 100 | ) 101 | self.features.add_module('transition%d' % (i + 1), trans) 102 | num_features = num_features // 2 103 | 104 | # Final batch norm 105 | self.features.add_module('norm5', nn.BatchNorm2d(num_features)) 106 | 107 | # Linear layer 108 | self.dropout = nn.Dropout(p=final_drop_rate) 109 | self.classifier = nn.Linear(num_features, num_classes) 110 | 111 | def forward(self, x): 112 | features = self.features(x) 113 | out = F.relu(features, inplace=False) 114 | out = F.avg_pool2d(out, kernel_size=7).view(features.size(0), -1) 115 | out = self.dropout(out) 116 | out = self.classifier(out) 117 | return out 118 | -------------------------------------------------------------------------------- /vanilla_densenet_big/get_densenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import numpy as np 3 | import torch 4 | import torch.optim as optim 5 | from torch.nn.init import normal 6 | from densenet import DenseNet 7 | import torch.utils.model_zoo as model_zoo 8 | from PIL import Image 9 | 10 | 11 | NUM_CLASSES = 200 12 | 13 | 14 | def resize_filter(f): 15 | """Treats a filter like a 16 | small image and resizes it. 17 | 18 | Arguments: 19 | f: a numpy float array with shape 20 | [filter_size, filter_size, 3]. 21 | Returns: 22 | a numpy float array with shape 23 | [new_size, new_size, 3], 24 | where new_size = 3. 25 | """ 26 | min_val = f.min() 27 | max_val = f.max() 28 | 29 | # convert to [0, 255] range 30 | f = (f - min_val)/(max_val - min_val) 31 | f *= 255.0 32 | 33 | img = Image.fromarray(f.astype('uint8')) 34 | img = img.resize((3, 3), Image.LANCZOS) 35 | f = np.asarray(img, dtype='float32')/255.0 36 | 37 | # back to the original range 38 | f *= (max_val - min_val) 39 | f += min_val 40 | 41 | return f 42 | 43 | 44 | def get_model(): 45 | """Get the model, the loss, and an optimizer""" 46 | 47 | # get DenseNet-121 48 | model = DenseNet() 49 | 50 | state_dict = model_zoo.load_url( 51 | 'https://download.pytorch.org/models/densenet121-241335ed.pth' 52 | ) 53 | 54 | # in the original pretrained model the first conv layer has 55 | # 64 filters with size (7, 7), but because i use 56 | # images of smaller size i resize filters to (3, 3) 57 | first_conv = state_dict['features.conv0.weight'].cpu().numpy() 58 | n_filters = first_conv.shape[0] 59 | new_filters = np.zeros((n_filters, 3, 3, 3), 'float32') 60 | for i, f in enumerate(first_conv): 61 | f = f.transpose(1, 2, 0) 62 | f = resize_filter(f) 63 | f = f.transpose(2, 0, 1) 64 | new_filters[i] = f 65 | state_dict['features.conv0.weight'] = torch.FloatTensor(new_filters).cuda() 66 | 67 | # reset weights of the last fc layer 68 | weight = torch.zeros(NUM_CLASSES, 1024) 69 | normal(weight, std=0.01) 70 | state_dict['classifier.weight'] = weight.cuda() 71 | state_dict['classifier.bias'] = torch.zeros(NUM_CLASSES).cuda() 72 | 73 | # load pretrained model 74 | model.load_state_dict(state_dict) 75 | 76 | # make all params not trainable 77 | for p in model.parameters(): 78 | p.requires_grad = False 79 | 80 | # make the last fc layer trainable 81 | model.classifier.weight.requires_grad = True 82 | model.classifier.bias.requires_grad = True 83 | 84 | # make the last batch norm layer trainable 85 | model.features.norm5.weight.requires_grad = True 86 | model.features.norm5.bias.requires_grad = True 87 | 88 | # create different parameter groups 89 | weights = [model.classifier.weight] 90 | biases = [model.classifier.bias] 91 | bn_weights = [model.features.norm5.weight] 92 | bn_biases = [model.features.norm5.bias] 93 | 94 | params = [ 95 | {'params': weights, 'weight_decay': 1e-4}, 96 | {'params': biases}, 97 | {'params': bn_weights}, 98 | {'params': bn_biases} 99 | ] 100 | optimizer = optim.Adam(params, lr=1e-3) 101 | 102 | loss = nn.CrossEntropyLoss().cuda() 103 | model = model.cuda() # move the model to gpu 104 | return model, loss, optimizer 105 | -------------------------------------------------------------------------------- /vanilla_densenet_big/model_step5.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/vanilla_densenet_big/model_step5.pytorch_state -------------------------------------------------------------------------------- /vanilla_densenet_big/train_step1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "\n", 36 | "import sys\n", 37 | "sys.path.append('..')\n", 38 | "from utils.input_pipeline import get_image_folders\n", 39 | "from utils.training import train, optimization_step\n", 40 | "from utils.diagnostic import count_params\n", 41 | " \n", 42 | "torch.cuda.is_available()" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": { 49 | "collapsed": true 50 | }, 51 | "outputs": [], 52 | "source": [ 53 | "torch.backends.cudnn.benchmark = True" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "# Create data iterators" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 4, 66 | "metadata": { 67 | "collapsed": true 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "batch_size = 256" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 5, 77 | "metadata": {}, 78 | "outputs": [ 79 | { 80 | "data": { 81 | "text/plain": [ 82 | "100000" 83 | ] 84 | }, 85 | "execution_count": 5, 86 | "metadata": {}, 87 | "output_type": "execute_result" 88 | } 89 | ], 90 | "source": [ 91 | "train_folder, val_folder = get_image_folders()\n", 92 | "\n", 93 | "train_iterator = DataLoader(\n", 94 | " train_folder, batch_size=batch_size, num_workers=4,\n", 95 | " shuffle=True, pin_memory=True\n", 96 | ")\n", 97 | "\n", 98 | "val_iterator = DataLoader(\n", 99 | " val_folder, batch_size=256, num_workers=4,\n", 100 | " shuffle=False, pin_memory=True\n", 101 | ")\n", 102 | "\n", 103 | "# number of training samples\n", 104 | "train_size = len(train_folder.imgs)\n", 105 | "train_size" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 6, 111 | "metadata": {}, 112 | "outputs": [ 113 | { 114 | "data": { 115 | "text/plain": [ 116 | "10000" 117 | ] 118 | }, 119 | "execution_count": 6, 120 | "metadata": {}, 121 | "output_type": "execute_result" 122 | } 123 | ], 124 | "source": [ 125 | "# number of validation samples\n", 126 | "val_size = len(val_folder.imgs)\n", 127 | "val_size" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "# Model" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 7, 140 | "metadata": { 141 | "collapsed": true 142 | }, 143 | "outputs": [], 144 | "source": [ 145 | "from get_densenet import get_model" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 8, 151 | "metadata": { 152 | "collapsed": true, 153 | "scrolled": true 154 | }, 155 | "outputs": [], 156 | "source": [ 157 | "model, loss, optimizer = get_model()" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 9, 163 | "metadata": {}, 164 | "outputs": [ 165 | { 166 | "data": { 167 | "text/plain": [ 168 | "7151176" 169 | ] 170 | }, 171 | "execution_count": 9, 172 | "metadata": {}, 173 | "output_type": "execute_result" 174 | } 175 | ], 176 | "source": [ 177 | "# number of params in the model\n", 178 | "count_params(model)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "metadata": {}, 184 | "source": [ 185 | "# Train" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 10, 191 | "metadata": {}, 192 | "outputs": [ 193 | { 194 | "data": { 195 | "text/plain": [ 196 | "391" 197 | ] 198 | }, 199 | "execution_count": 10, 200 | "metadata": {}, 201 | "output_type": "execute_result" 202 | } 203 | ], 204 | "source": [ 205 | "n_epochs = 5\n", 206 | "n_batches = ceil(train_size/batch_size)\n", 207 | "\n", 208 | "# total number of batches in the train set\n", 209 | "n_batches" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 11, 215 | "metadata": { 216 | "scrolled": false 217 | }, 218 | "outputs": [ 219 | { 220 | "name": "stdout", 221 | "output_type": "stream", 222 | "text": [ 223 | "0 3.882 2.772 0.208 0.392 0.427 0.659 776.920\n", 224 | "1 3.110 2.502 0.308 0.425 0.565 0.690 761.085\n", 225 | "2 2.974 2.403 0.330 0.441 0.588 0.704 761.199\n", 226 | "3 2.925 2.367 0.338 0.444 0.596 0.706 761.763\n", 227 | "4 2.891 2.329 0.344 0.455 0.603 0.719 760.764\n", 228 | "CPU times: user 50min 25s, sys: 13min 12s, total: 1h 3min 38s\n", 229 | "Wall time: 1h 3min 41s\n" 230 | ] 231 | } 232 | ], 233 | "source": [ 234 | "%%time\n", 235 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 236 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 237 | "\n", 238 | "all_losses = train(\n", 239 | " model, loss, optimization_step_fn,\n", 240 | " train_iterator, val_iterator, n_epochs\n", 241 | ")\n", 242 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 243 | ] 244 | }, 245 | { 246 | "cell_type": "markdown", 247 | "metadata": {}, 248 | "source": [ 249 | "# Save" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": 15, 255 | "metadata": { 256 | "collapsed": true 257 | }, 258 | "outputs": [], 259 | "source": [ 260 | "model.cpu();\n", 261 | "torch.save(model.state_dict(), 'model_step1.pytorch_state')" 262 | ] 263 | } 264 | ], 265 | "metadata": { 266 | "kernelspec": { 267 | "display_name": "Python 3", 268 | "language": "python", 269 | "name": "python3" 270 | }, 271 | "language_info": { 272 | "codemirror_mode": { 273 | "name": "ipython", 274 | "version": 3 275 | }, 276 | "file_extension": ".py", 277 | "mimetype": "text/x-python", 278 | "name": "python", 279 | "nbconvert_exporter": "python", 280 | "pygments_lexer": "ipython3", 281 | "version": "3.6.1" 282 | } 283 | }, 284 | "nbformat": 4, 285 | "nbformat_minor": 2 286 | } 287 | -------------------------------------------------------------------------------- /vanilla_densenet_big/train_step2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "import torch.optim as optim\n", 36 | "import torch.nn as nn\n", 37 | "\n", 38 | "import sys\n", 39 | "sys.path.append('..')\n", 40 | "from utils.input_pipeline import get_image_folders\n", 41 | "from utils.training import train, optimization_step\n", 42 | " \n", 43 | "torch.cuda.is_available()" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "torch.backends.cudnn.benchmark = True" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "# Create data iterators" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "batch_size = 32" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 5, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "data": { 82 | "text/plain": [ 83 | "100000" 84 | ] 85 | }, 86 | "execution_count": 5, 87 | "metadata": {}, 88 | "output_type": "execute_result" 89 | } 90 | ], 91 | "source": [ 92 | "train_folder, val_folder = get_image_folders()\n", 93 | "\n", 94 | "train_iterator = DataLoader(\n", 95 | " train_folder, batch_size=batch_size, num_workers=4,\n", 96 | " shuffle=True, pin_memory=True\n", 97 | ")\n", 98 | "\n", 99 | "val_iterator = DataLoader(\n", 100 | " val_folder, batch_size=256, num_workers=4,\n", 101 | " shuffle=False, pin_memory=True\n", 102 | ")\n", 103 | "\n", 104 | "# number of training samples\n", 105 | "train_size = len(train_folder.imgs)\n", 106 | "train_size" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "10000" 118 | ] 119 | }, 120 | "execution_count": 6, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "# number of validation samples\n", 127 | "val_size = len(val_folder.imgs)\n", 128 | "val_size" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "# Model" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": 7, 141 | "metadata": { 142 | "collapsed": true 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "from densenet import DenseNet\n", 147 | "model = DenseNet()\n", 148 | "# load the model from step1\n", 149 | "model.load_state_dict(torch.load('model_step1.pytorch_state'))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 8, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "# create different parameter groups\n", 161 | "weights = [\n", 162 | " p for n, p in model.named_parameters()\n", 163 | " if 'conv' in n or 'classifier.weight' in n\n", 164 | "]\n", 165 | "biases = [model.classifier.bias]\n", 166 | "bn_weights = [\n", 167 | " p for n, p in model.named_parameters()\n", 168 | " if 'norm' in n and 'weight' in n\n", 169 | "]\n", 170 | "bn_biases = [\n", 171 | " p for n, p in model.named_parameters()\n", 172 | " if 'norm' in n and 'bias' in n\n", 173 | "]" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 9, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "params = [\n", 185 | " {'params': weights, 'weight_decay': 1e-4},\n", 186 | " {'params': biases},\n", 187 | " {'params': bn_weights},\n", 188 | " {'params': bn_biases}\n", 189 | "]\n", 190 | "optimizer = optim.SGD(params, lr=1e-5, momentum=0.9, nesterov=True)\n", 191 | "loss = nn.CrossEntropyLoss().cuda()\n", 192 | "model = model.cuda()" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "# Train" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 10, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "data": { 209 | "text/plain": [ 210 | "3125" 211 | ] 212 | }, 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "output_type": "execute_result" 216 | } 217 | ], 218 | "source": [ 219 | "n_epochs = 5\n", 220 | "n_batches = ceil(train_size/batch_size)\n", 221 | "\n", 222 | "# total number of batches in the train set\n", 223 | "n_batches" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 11, 229 | "metadata": { 230 | "scrolled": false 231 | }, 232 | "outputs": [ 233 | { 234 | "name": "stdout", 235 | "output_type": "stream", 236 | "text": [ 237 | "0 2.450 1.717 0.426 0.581 0.689 0.813 2330.438\n", 238 | "1 2.141 1.577 0.487 0.607 0.745 0.839 2312.572\n", 239 | "2 2.021 1.495 0.510 0.626 0.766 0.847 2309.705\n", 240 | "3 1.939 1.443 0.530 0.638 0.778 0.855 2307.756\n", 241 | "4 1.881 1.398 0.540 0.650 0.789 0.862 2305.410\n", 242 | "CPU times: user 2h 56min 54s, sys: 16min 15s, total: 3h 13min 10s\n", 243 | "Wall time: 3h 12min 45s\n" 244 | ] 245 | } 246 | ], 247 | "source": [ 248 | "%%time\n", 249 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 250 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 251 | "\n", 252 | "all_losses = train(\n", 253 | " model, loss, optimization_step_fn,\n", 254 | " train_iterator, val_iterator, n_epochs\n", 255 | ")\n", 256 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "# Save" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 15, 269 | "metadata": { 270 | "collapsed": true 271 | }, 272 | "outputs": [], 273 | "source": [ 274 | "model.cpu();\n", 275 | "torch.save(model.state_dict(), 'model_step2.pytorch_state')" 276 | ] 277 | } 278 | ], 279 | "metadata": { 280 | "kernelspec": { 281 | "display_name": "Python 3", 282 | "language": "python", 283 | "name": "python3" 284 | }, 285 | "language_info": { 286 | "codemirror_mode": { 287 | "name": "ipython", 288 | "version": 3 289 | }, 290 | "file_extension": ".py", 291 | "mimetype": "text/x-python", 292 | "name": "python", 293 | "nbconvert_exporter": "python", 294 | "pygments_lexer": "ipython3", 295 | "version": "3.6.1" 296 | } 297 | }, 298 | "nbformat": 4, 299 | "nbformat_minor": 2 300 | } 301 | -------------------------------------------------------------------------------- /vanilla_densenet_big/train_step3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "import torch.optim as optim\n", 36 | "import torch.nn as nn\n", 37 | "\n", 38 | "import sys\n", 39 | "sys.path.append('..')\n", 40 | "from utils.input_pipeline import get_image_folders\n", 41 | "from utils.training import train, optimization_step\n", 42 | " \n", 43 | "torch.cuda.is_available()" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "torch.backends.cudnn.benchmark = True" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "# Create data iterators" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "batch_size = 64" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 5, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "data": { 82 | "text/plain": [ 83 | "100000" 84 | ] 85 | }, 86 | "execution_count": 5, 87 | "metadata": {}, 88 | "output_type": "execute_result" 89 | } 90 | ], 91 | "source": [ 92 | "train_folder, val_folder = get_image_folders()\n", 93 | "\n", 94 | "train_iterator = DataLoader(\n", 95 | " train_folder, batch_size=batch_size, num_workers=4,\n", 96 | " shuffle=True, pin_memory=True\n", 97 | ")\n", 98 | "\n", 99 | "val_iterator = DataLoader(\n", 100 | " val_folder, batch_size=256, num_workers=4,\n", 101 | " shuffle=False, pin_memory=True\n", 102 | ")\n", 103 | "\n", 104 | "# number of training samples\n", 105 | "train_size = len(train_folder.imgs)\n", 106 | "train_size" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "10000" 118 | ] 119 | }, 120 | "execution_count": 6, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "# number of validation samples\n", 127 | "val_size = len(val_folder.imgs)\n", 128 | "val_size" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "# Model" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": 7, 141 | "metadata": { 142 | "collapsed": true 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "from densenet import DenseNet\n", 147 | "model = DenseNet()\n", 148 | "# load the model from step2\n", 149 | "model.load_state_dict(torch.load('model_step2.pytorch_state'))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 8, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "# create different parameter groups\n", 161 | "weights = [\n", 162 | " p for n, p in model.named_parameters()\n", 163 | " if 'conv' in n or 'classifier.weight' in n\n", 164 | "]\n", 165 | "biases = [model.classifier.bias]\n", 166 | "bn_weights = [\n", 167 | " p for n, p in model.named_parameters()\n", 168 | " if 'norm' in n and 'weight' in n\n", 169 | "]\n", 170 | "bn_biases = [\n", 171 | " p for n, p in model.named_parameters()\n", 172 | " if 'norm' in n and 'bias' in n\n", 173 | "]" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 9, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "params = [\n", 185 | " {'params': weights, 'weight_decay': 1e-4},\n", 186 | " {'params': biases},\n", 187 | " {'params': bn_weights},\n", 188 | " {'params': bn_biases}\n", 189 | "]\n", 190 | "optimizer = optim.SGD(params, lr=1e-4, momentum=0.9, nesterov=True)\n", 191 | "loss = nn.CrossEntropyLoss().cuda()\n", 192 | "model = model.cuda()" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "# Train" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 10, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "data": { 209 | "text/plain": [ 210 | "1563" 211 | ] 212 | }, 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "output_type": "execute_result" 216 | } 217 | ], 218 | "source": [ 219 | "n_epochs = 5\n", 220 | "n_batches = ceil(train_size/batch_size)\n", 221 | "\n", 222 | "# total number of batches in the train set\n", 223 | "n_batches" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 11, 229 | "metadata": { 230 | "scrolled": false 231 | }, 232 | "outputs": [ 233 | { 234 | "name": "stdout", 235 | "output_type": "stream", 236 | "text": [ 237 | "0 1.748 1.295 0.568 0.671 0.810 0.877 2096.644\n", 238 | "1 1.631 1.224 0.592 0.689 0.827 0.886 2080.292\n", 239 | "2 1.558 1.184 0.609 0.697 0.838 0.892 2084.515\n", 240 | "3 1.493 1.154 0.622 0.704 0.848 0.893 2083.987\n", 241 | "4 1.449 1.118 0.632 0.713 0.855 0.899 2083.928\n", 242 | "CPU times: user 2h 38min 32s, sys: 15min 17s, total: 2h 53min 50s\n", 243 | "Wall time: 2h 53min 49s\n" 244 | ] 245 | } 246 | ], 247 | "source": [ 248 | "%%time\n", 249 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 250 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 251 | "\n", 252 | "all_losses = train(\n", 253 | " model, loss, optimization_step_fn,\n", 254 | " train_iterator, val_iterator, n_epochs\n", 255 | ")\n", 256 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "# Save" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 12, 269 | "metadata": { 270 | "collapsed": true 271 | }, 272 | "outputs": [], 273 | "source": [ 274 | "model.cpu();\n", 275 | "torch.save(model.state_dict(), 'model_step3.pytorch_state')" 276 | ] 277 | } 278 | ], 279 | "metadata": { 280 | "kernelspec": { 281 | "display_name": "Python 3", 282 | "language": "python", 283 | "name": "python3" 284 | }, 285 | "language_info": { 286 | "codemirror_mode": { 287 | "name": "ipython", 288 | "version": 3 289 | }, 290 | "file_extension": ".py", 291 | "mimetype": "text/x-python", 292 | "name": "python", 293 | "nbconvert_exporter": "python", 294 | "pygments_lexer": "ipython3", 295 | "version": "3.6.1" 296 | } 297 | }, 298 | "nbformat": 4, 299 | "nbformat_minor": 2 300 | } 301 | -------------------------------------------------------------------------------- /vanilla_densenet_big/train_step4.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "import torch.optim as optim\n", 36 | "import torch.nn as nn\n", 37 | "\n", 38 | "import sys\n", 39 | "sys.path.append('..')\n", 40 | "from utils.input_pipeline import get_image_folders\n", 41 | "from utils.training import train, optimization_step\n", 42 | " \n", 43 | "torch.cuda.is_available()" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "torch.backends.cudnn.benchmark = True" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "# Create data iterators" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "batch_size = 64" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 5, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "data": { 82 | "text/plain": [ 83 | "100000" 84 | ] 85 | }, 86 | "execution_count": 5, 87 | "metadata": {}, 88 | "output_type": "execute_result" 89 | } 90 | ], 91 | "source": [ 92 | "train_folder, val_folder = get_image_folders()\n", 93 | "\n", 94 | "train_iterator = DataLoader(\n", 95 | " train_folder, batch_size=batch_size, num_workers=4,\n", 96 | " shuffle=True, pin_memory=True\n", 97 | ")\n", 98 | "\n", 99 | "val_iterator = DataLoader(\n", 100 | " val_folder, batch_size=256, num_workers=4,\n", 101 | " shuffle=False, pin_memory=True\n", 102 | ")\n", 103 | "\n", 104 | "# number of training samples\n", 105 | "train_size = len(train_folder.imgs)\n", 106 | "train_size" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "10000" 118 | ] 119 | }, 120 | "execution_count": 6, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "# number of validation samples\n", 127 | "val_size = len(val_folder.imgs)\n", 128 | "val_size" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "# Model" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": 7, 141 | "metadata": { 142 | "collapsed": true 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "from densenet import DenseNet\n", 147 | "model = DenseNet()\n", 148 | "# load the model from step3\n", 149 | "model.load_state_dict(torch.load('model_step3.pytorch_state'))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 8, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "# create different parameter groups\n", 161 | "weights = [\n", 162 | " p for n, p in model.named_parameters()\n", 163 | " if 'conv' in n or 'classifier.weight' in n\n", 164 | "]\n", 165 | "biases = [model.classifier.bias]\n", 166 | "bn_weights = [\n", 167 | " p for n, p in model.named_parameters()\n", 168 | " if 'norm' in n and 'weight' in n\n", 169 | "]\n", 170 | "bn_biases = [\n", 171 | " p for n, p in model.named_parameters()\n", 172 | " if 'norm' in n and 'bias' in n\n", 173 | "]" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 9, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "params = [\n", 185 | " {'params': weights, 'weight_decay': 1e-4},\n", 186 | " {'params': biases},\n", 187 | " {'params': bn_weights},\n", 188 | " {'params': bn_biases}\n", 189 | "]\n", 190 | "optimizer = optim.SGD(params, lr=1e-4, momentum=0.9, nesterov=True)\n", 191 | "loss = nn.CrossEntropyLoss().cuda()\n", 192 | "model = model.cuda()" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "# Train" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 10, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "data": { 209 | "text/plain": [ 210 | "1563" 211 | ] 212 | }, 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "output_type": "execute_result" 216 | } 217 | ], 218 | "source": [ 219 | "n_epochs = 5\n", 220 | "n_batches = ceil(train_size/batch_size)\n", 221 | "\n", 222 | "# total number of batches in the train set\n", 223 | "n_batches" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 11, 229 | "metadata": { 230 | "scrolled": false 231 | }, 232 | "outputs": [ 233 | { 234 | "name": "stdout", 235 | "output_type": "stream", 236 | "text": [ 237 | "0 1.407 1.100 0.642 0.715 0.862 0.904 2095.310\n", 238 | "1 1.373 1.084 0.650 0.721 0.866 0.904 2081.717\n", 239 | "2 1.343 1.063 0.656 0.726 0.870 0.908 2085.719\n", 240 | "3 1.314 1.060 0.662 0.726 0.874 0.907 2087.037\n", 241 | "4 1.288 1.042 0.669 0.728 0.879 0.908 2085.568\n", 242 | "CPU times: user 2h 38min 5s, sys: 15min 52s, total: 2h 53min 58s\n", 243 | "Wall time: 2h 53min 55s\n" 244 | ] 245 | } 246 | ], 247 | "source": [ 248 | "%%time\n", 249 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 250 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 251 | "\n", 252 | "all_losses = train(\n", 253 | " model, loss, optimization_step_fn,\n", 254 | " train_iterator, val_iterator, n_epochs\n", 255 | ")\n", 256 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "# Save" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 12, 269 | "metadata": { 270 | "collapsed": true 271 | }, 272 | "outputs": [], 273 | "source": [ 274 | "model.cpu();\n", 275 | "torch.save(model.state_dict(), 'model_step4.pytorch_state')" 276 | ] 277 | } 278 | ], 279 | "metadata": { 280 | "kernelspec": { 281 | "display_name": "Python 3", 282 | "language": "python", 283 | "name": "python3" 284 | }, 285 | "language_info": { 286 | "codemirror_mode": { 287 | "name": "ipython", 288 | "version": 3 289 | }, 290 | "file_extension": ".py", 291 | "mimetype": "text/x-python", 292 | "name": "python", 293 | "nbconvert_exporter": "python", 294 | "pygments_lexer": "ipython3", 295 | "version": "3.6.1" 296 | } 297 | }, 298 | "nbformat": 4, 299 | "nbformat_minor": 2 300 | } 301 | -------------------------------------------------------------------------------- /vanilla_densenet_big/train_step5.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "import torch.optim as optim\n", 36 | "import torch.nn as nn\n", 37 | "\n", 38 | "import sys\n", 39 | "sys.path.append('..')\n", 40 | "from utils.input_pipeline import get_image_folders\n", 41 | "from utils.training import train, optimization_step\n", 42 | " \n", 43 | "torch.cuda.is_available()" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 3, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "torch.backends.cudnn.benchmark = True" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "# Create data iterators" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "batch_size = 64" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 5, 78 | "metadata": {}, 79 | "outputs": [ 80 | { 81 | "data": { 82 | "text/plain": [ 83 | "100000" 84 | ] 85 | }, 86 | "execution_count": 5, 87 | "metadata": {}, 88 | "output_type": "execute_result" 89 | } 90 | ], 91 | "source": [ 92 | "train_folder, val_folder = get_image_folders()\n", 93 | "\n", 94 | "train_iterator = DataLoader(\n", 95 | " train_folder, batch_size=batch_size, num_workers=4,\n", 96 | " shuffle=True, pin_memory=True\n", 97 | ")\n", 98 | "\n", 99 | "val_iterator = DataLoader(\n", 100 | " val_folder, batch_size=256, num_workers=4,\n", 101 | " shuffle=False, pin_memory=True\n", 102 | ")\n", 103 | "\n", 104 | "# number of training samples\n", 105 | "train_size = len(train_folder.imgs)\n", 106 | "train_size" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "10000" 118 | ] 119 | }, 120 | "execution_count": 6, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "# number of validation samples\n", 127 | "val_size = len(val_folder.imgs)\n", 128 | "val_size" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "# Model" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": 7, 141 | "metadata": { 142 | "collapsed": true 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "from densenet import DenseNet\n", 147 | "model = DenseNet()\n", 148 | "# load the model from step4\n", 149 | "model.load_state_dict(torch.load('model_step4.pytorch_state'))" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": 8, 155 | "metadata": { 156 | "collapsed": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "# create different parameter groups\n", 161 | "weights = [\n", 162 | " p for n, p in model.named_parameters()\n", 163 | " if 'conv' in n or 'classifier.weight' in n\n", 164 | "]\n", 165 | "biases = [model.classifier.bias]\n", 166 | "bn_weights = [\n", 167 | " p for n, p in model.named_parameters()\n", 168 | " if 'norm' in n and 'weight' in n\n", 169 | "]\n", 170 | "bn_biases = [\n", 171 | " p for n, p in model.named_parameters()\n", 172 | " if 'norm' in n and 'bias' in n\n", 173 | "]" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 9, 179 | "metadata": { 180 | "collapsed": true 181 | }, 182 | "outputs": [], 183 | "source": [ 184 | "params = [\n", 185 | " {'params': weights, 'weight_decay': 1e-4},\n", 186 | " {'params': biases},\n", 187 | " {'params': bn_weights},\n", 188 | " {'params': bn_biases}\n", 189 | "]\n", 190 | "optimizer = optim.SGD(params, lr=5e-5, momentum=0.9, nesterov=True)\n", 191 | "loss = nn.CrossEntropyLoss().cuda()\n", 192 | "model = model.cuda()" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "# Train" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 10, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "data": { 209 | "text/plain": [ 210 | "1563" 211 | ] 212 | }, 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "output_type": "execute_result" 216 | } 217 | ], 218 | "source": [ 219 | "n_epochs = 3\n", 220 | "n_batches = ceil(train_size/batch_size)\n", 221 | "\n", 222 | "# total number of batches in the train set\n", 223 | "n_batches" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 11, 229 | "metadata": { 230 | "scrolled": false 231 | }, 232 | "outputs": [ 233 | { 234 | "name": "stdout", 235 | "output_type": "stream", 236 | "text": [ 237 | "0 1.257 1.029 0.676 0.734 0.882 0.910 2051.787\n", 238 | "1 1.245 1.017 0.679 0.736 0.884 0.911 2033.188\n", 239 | "2 1.226 1.014 0.683 0.736 0.887 0.913 2035.694\n", 240 | "CPU times: user 1h 32min 32s, sys: 9min 22s, total: 1h 41min 54s\n", 241 | "Wall time: 1h 42min\n" 242 | ] 243 | } 244 | ], 245 | "source": [ 246 | "%%time\n", 247 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 248 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 249 | "\n", 250 | "all_losses = train(\n", 251 | " model, loss, optimization_step_fn,\n", 252 | " train_iterator, val_iterator, n_epochs\n", 253 | ")\n", 254 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "# Save" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 12, 267 | "metadata": { 268 | "collapsed": true 269 | }, 270 | "outputs": [], 271 | "source": [ 272 | "model.cpu();\n", 273 | "torch.save(model.state_dict(), 'model_step5.pytorch_state')" 274 | ] 275 | } 276 | ], 277 | "metadata": { 278 | "kernelspec": { 279 | "display_name": "Python 3", 280 | "language": "python", 281 | "name": "python3" 282 | }, 283 | "language_info": { 284 | "codemirror_mode": { 285 | "name": "ipython", 286 | "version": 3 287 | }, 288 | "file_extension": ".py", 289 | "mimetype": "text/x-python", 290 | "name": "python", 291 | "nbconvert_exporter": "python", 292 | "pygments_lexer": "ipython3", 293 | "version": "3.6.1" 294 | } 295 | }, 296 | "nbformat": 4, 297 | "nbformat_minor": 2 298 | } 299 | -------------------------------------------------------------------------------- /vanilla_densenet_small/densenet.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.functional as F 4 | from collections import OrderedDict 5 | 6 | 7 | class _DenseLayer(nn.Sequential): 8 | def __init__(self, num_input_features, growth_rate, bn_size, drop_rate): 9 | super(_DenseLayer, self).__init__() 10 | 11 | self.add_module('norm.1', nn.BatchNorm2d(num_input_features)) 12 | self.add_module('relu.1', nn.ReLU(inplace=True)) 13 | self.add_module('conv.1', nn.Conv2d(num_input_features, bn_size*growth_rate, 14 | kernel_size=1, stride=1, bias=False)) 15 | 16 | self.add_module('norm.2', nn.BatchNorm2d(bn_size*growth_rate)) 17 | self.add_module('relu.2', nn.ReLU(inplace=True)) 18 | self.add_module('conv.2', nn.Conv2d(bn_size*growth_rate, growth_rate, 19 | kernel_size=3, stride=1, padding=1, bias=False)) 20 | 21 | self.dropout = nn.Dropout(drop_rate) 22 | 23 | def forward(self, x): 24 | new_features = super(_DenseLayer, self).forward(x) 25 | new_features = self.dropout(new_features) 26 | return torch.cat([x, new_features], 1) 27 | 28 | 29 | class _DenseBlock(nn.Sequential): 30 | def __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate): 31 | super(_DenseBlock, self).__init__() 32 | for i in range(num_layers): 33 | layer = _DenseLayer( 34 | num_input_features + i*growth_rate, 35 | growth_rate, bn_size, drop_rate 36 | ) 37 | self.add_module('denselayer%d' % (i + 1), layer) 38 | 39 | 40 | class _Transition(nn.Sequential): 41 | def __init__(self, num_input_features, num_output_features): 42 | super(_Transition, self).__init__() 43 | self.add_module('norm', nn.BatchNorm2d(num_input_features)) 44 | self.add_module('relu', nn.ReLU(inplace=True)) 45 | self.add_module('conv', nn.Conv2d(num_input_features, num_output_features, 46 | kernel_size=1, stride=1, bias=False)) 47 | self.add_module('pool', nn.AvgPool2d(kernel_size=2, stride=2)) 48 | 49 | 50 | class DenseNet(nn.Module): 51 | """Densenet-BC model class, based on 52 | `"Densely Connected Convolutional Networks" ` 53 | 54 | Arguments: 55 | growth_rate (int) - how many filters to add each layer (`k` in paper). 56 | block_config (list of 4 ints) - how many layers in each pooling block. 57 | num_init_features (int) - the number of filters to learn 58 | in the first convolution layer. 59 | bn_size (int) - multiplicative factor for number of bottleneck layers 60 | (i.e. bn_size * k features in the bottleneck layer). 61 | drop_rate (float) - dropout rate after each dense layer. 62 | final_drop_rate (float) - dropout rate before final fc layer. 63 | num_classes (int) - number of classification classes. 64 | """ 65 | def __init__(self, growth_rate=12, block_config=(8, 12, 10), 66 | num_init_features=48, bn_size=4, drop_rate=0.25, 67 | final_drop_rate=0.25, num_classes=200): 68 | 69 | super(DenseNet, self).__init__() 70 | 71 | # first convolution 72 | self.features = nn.Sequential(OrderedDict([ 73 | ('conv0', nn.Conv2d(3, num_init_features, 74 | kernel_size=3, stride=1, padding=1, bias=False)), 75 | ('norm0', nn.BatchNorm2d(num_init_features)), 76 | ('relu0', nn.ReLU(inplace=True)), 77 | ('pool0', nn.MaxPool2d(kernel_size=3, stride=2, padding=1)) 78 | ])) 79 | 80 | # each denseblock 81 | num_features = num_init_features 82 | for i, num_layers in enumerate(block_config): 83 | block = _DenseBlock( 84 | num_layers=num_layers, num_input_features=num_features, 85 | bn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate 86 | ) 87 | self.features.add_module('denseblock%d' % (i + 1), block) 88 | num_features = num_features + num_layers*growth_rate 89 | if i != len(block_config) - 1: 90 | trans = _Transition( 91 | num_input_features=num_features, 92 | num_output_features=num_features // 2 93 | ) 94 | self.features.add_module('transition%d' % (i + 1), trans) 95 | num_features = num_features // 2 96 | 97 | # final batch norm 98 | self.features.add_module('norm4', nn.BatchNorm2d(num_features)) 99 | 100 | # linear layer 101 | self.relu = nn.ReLU(inplace=True) 102 | self.avg_pool = nn.AvgPool2d(kernel_size=7) 103 | self.dropout = nn.Dropout(p=final_drop_rate) 104 | self.classifier = nn.Linear(num_features, num_classes) 105 | 106 | def forward(self, x): 107 | features = self.features(x) 108 | out = self.relu(features) 109 | out = self.avg_pool(out).view(features.size(0), -1) 110 | out = self.dropout(out) 111 | logits = self.classifier(out) 112 | return logits 113 | -------------------------------------------------------------------------------- /vanilla_densenet_small/get_densenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.optim as optim 3 | from torch.nn.init import constant, kaiming_uniform 4 | from densenet import DenseNet 5 | 6 | 7 | def get_model(): 8 | 9 | model = DenseNet( 10 | growth_rate=12, block_config=(8, 12, 10), 11 | num_init_features=48, bn_size=4, drop_rate=0.25, 12 | final_drop_rate=0.25, num_classes=200 13 | ) 14 | 15 | # create different parameter groups 16 | weights = [ 17 | p for n, p in model.named_parameters() 18 | if 'conv' in n or 'classifier.weight' in n 19 | ] 20 | biases = [model.classifier.bias] 21 | bn_weights = [ 22 | p for n, p in model.named_parameters() 23 | if 'norm' in n and 'weight' in n 24 | ] 25 | bn_biases = [ 26 | p for n, p in model.named_parameters() 27 | if 'norm' in n and 'bias' in n 28 | ] 29 | 30 | # parameter initialization 31 | for p in weights: 32 | kaiming_uniform(p) 33 | for p in biases: 34 | constant(p, 0.0) 35 | for p in bn_weights: 36 | constant(p, 1.0) 37 | for p in bn_biases: 38 | constant(p, 0.0) 39 | 40 | params = [ 41 | {'params': weights, 'weight_decay': 1e-4}, 42 | {'params': biases}, 43 | {'params': bn_weights}, 44 | {'params': bn_biases} 45 | ] 46 | optimizer = optim.SGD(params, lr=1e-1, momentum=0.95, nesterov=True) 47 | 48 | loss = nn.CrossEntropyLoss().cuda() 49 | model = model.cuda() # move the model to gpu 50 | return model, loss, optimizer 51 | -------------------------------------------------------------------------------- /vanilla_densenet_small/model.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/vanilla_densenet_small/model.pytorch_state -------------------------------------------------------------------------------- /vanilla_squeezenet/get_squeezenet.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch.optim as optim 3 | from torch.nn.init import constant 4 | from squeezenet import SqueezeNet 5 | 6 | 7 | def get_model(): 8 | 9 | model = SqueezeNet() 10 | 11 | # create different parameter groups 12 | weights = [ 13 | p for n, p in model.named_parameters() 14 | if 'weight' in n and 'bn' not in n and 'features.1.' not in n 15 | ] 16 | biases = [model.classifier[1].bias] 17 | bn_weights = [ 18 | p for n, p in model.named_parameters() 19 | if ('bn' in n or 'features.1.' in n) and 'weight' in n 20 | ] 21 | bn_biases = [ 22 | p for n, p in model.named_parameters() 23 | if ('bn' in n or 'features.1.' in n) and 'bias' in n 24 | ] 25 | 26 | for p in bn_weights: 27 | constant(p, 1.0) 28 | for p in bn_biases: 29 | constant(p, 0.0) 30 | 31 | params = [ 32 | {'params': weights, 'weight_decay': 3e-4}, 33 | {'params': biases}, 34 | {'params': bn_weights}, 35 | {'params': bn_biases} 36 | ] 37 | optimizer = optim.SGD(params, lr=4e-2, momentum=0.95, nesterov=True) 38 | 39 | loss = nn.CrossEntropyLoss().cuda() 40 | model = model.cuda() # move the model to gpu 41 | return model, loss, optimizer 42 | -------------------------------------------------------------------------------- /vanilla_squeezenet/model.pytorch_state: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/TropComplique/trained-ternary-quantization/4cd4132124c30e0e868a78eb1b2a2798df5e2a90/vanilla_squeezenet/model.pytorch_state -------------------------------------------------------------------------------- /vanilla_squeezenet/squeezenet.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | import torch.nn.init as init 4 | 5 | 6 | class Fire(nn.Module): 7 | 8 | def __init__(self, inplanes, squeeze_planes, 9 | expand1x1_planes, expand3x3_planes, 10 | residual=False): 11 | super(Fire, self).__init__() 12 | self.inplanes = inplanes 13 | # squeeze 14 | self.squeeze_bn = nn.BatchNorm2d(inplanes) 15 | self.squeeze_activation = nn.ReLU(inplace=True) 16 | self.squeeze = nn.Conv2d( 17 | inplanes, squeeze_planes, kernel_size=1, bias=False 18 | ) 19 | # expand 20 | self.expand1x1_bn = nn.BatchNorm2d(squeeze_planes) 21 | self.expand1x1_activation = nn.ReLU(inplace=True) 22 | self.expand1x1 = nn.Conv2d( 23 | squeeze_planes, expand1x1_planes, kernel_size=1, bias=False 24 | ) 25 | # expand 26 | self.expand3x3_bn = nn.BatchNorm2d(squeeze_planes) 27 | self.expand3x3_activation = nn.ReLU(inplace=True) 28 | self.expand3x3 = nn.Conv2d( 29 | squeeze_planes, expand3x3_planes, kernel_size=3, padding=1, bias=False 30 | ) 31 | self.residual = residual 32 | if self.residual: 33 | assert (inplanes == (expand1x1_planes + expand3x3_planes)) 34 | 35 | def forward(self, x): 36 | residual = x 37 | x = self.squeeze(self.squeeze_activation(self.squeeze_bn(x))) 38 | out = torch.cat([ 39 | self.expand1x1(self.expand1x1_activation(self.expand1x1_bn(x))), 40 | self.expand3x3(self.expand3x3_activation(self.expand3x3_bn(x))) 41 | ], 1) 42 | 43 | if self.residual: 44 | return out + residual 45 | else: 46 | return out 47 | 48 | 49 | class SqueezeNet(nn.Module): 50 | 51 | def __init__(self, num_classes=200): 52 | super(SqueezeNet, self).__init__() 53 | self.num_classes = num_classes 54 | 55 | self.features = nn.Sequential( 56 | nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False), 57 | nn.BatchNorm2d(64), 58 | nn.ReLU(inplace=True), 59 | nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True), 60 | Fire(64, 16, 64, 64), 61 | Fire(128, 16, 64, 64, residual=True), 62 | nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True), 63 | Fire(128, 32, 128, 128), 64 | Fire(256, 32, 128, 128, residual=True), 65 | nn.MaxPool2d(kernel_size=3, stride=2, ceil_mode=True), 66 | Fire(256, 48, 192, 192), 67 | Fire(384, 48, 192, 192, residual=True), 68 | Fire(384, 64, 256, 256), 69 | Fire(512, 64, 256, 256, residual=True) 70 | ) 71 | 72 | # Final convolution is initialized differently form the rest 73 | final_conv = nn.Conv2d(512, self.num_classes, kernel_size=1) 74 | self.classifier = nn.Sequential( 75 | nn.Dropout(p=0.6), 76 | final_conv, 77 | nn.ReLU(inplace=True), 78 | nn.AvgPool2d(7) 79 | ) 80 | 81 | for m in self.modules(): 82 | if isinstance(m, nn.Conv2d): 83 | if m is final_conv: 84 | init.normal(m.weight, mean=0.0, std=0.01) 85 | else: 86 | init.kaiming_uniform(m.weight) 87 | if m.bias is not None: 88 | m.bias.data.zero_() 89 | 90 | def forward(self, x): 91 | x = self.features(x) 92 | x = self.classifier(x) 93 | return x.view(x.size(0), self.num_classes) 94 | -------------------------------------------------------------------------------- /vanilla_squeezenet/train.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%load_ext autoreload\n", 12 | "%autoreload 2" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 2, 18 | "metadata": {}, 19 | "outputs": [ 20 | { 21 | "data": { 22 | "text/plain": [ 23 | "True" 24 | ] 25 | }, 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "output_type": "execute_result" 29 | } 30 | ], 31 | "source": [ 32 | "from math import ceil\n", 33 | "import torch\n", 34 | "from torch.utils.data import DataLoader\n", 35 | "\n", 36 | "import matplotlib.pyplot as plt\n", 37 | "%matplotlib inline\n", 38 | "\n", 39 | "import sys\n", 40 | "sys.path.append('..')\n", 41 | "from utils.input_pipeline import get_image_folders\n", 42 | "from utils.training import train, optimization_step\n", 43 | "from utils.diagnostic import count_params\n", 44 | " \n", 45 | "torch.cuda.is_available()" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 3, 51 | "metadata": { 52 | "collapsed": true 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "torch.backends.cudnn.benchmark = True" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "# Create data iterators" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 4, 69 | "metadata": { 70 | "collapsed": true 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "batch_size = 128" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 5, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "data": { 84 | "text/plain": [ 85 | "100000" 86 | ] 87 | }, 88 | "execution_count": 5, 89 | "metadata": {}, 90 | "output_type": "execute_result" 91 | } 92 | ], 93 | "source": [ 94 | "train_folder, val_folder = get_image_folders()\n", 95 | "\n", 96 | "train_iterator = DataLoader(\n", 97 | " train_folder, batch_size=batch_size, num_workers=4,\n", 98 | " shuffle=True, pin_memory=True\n", 99 | ")\n", 100 | "\n", 101 | "val_iterator = DataLoader(\n", 102 | " val_folder, batch_size=256, num_workers=4,\n", 103 | " shuffle=False, pin_memory=True\n", 104 | ")\n", 105 | "\n", 106 | "# number of training samples\n", 107 | "train_size = len(train_folder.imgs)\n", 108 | "train_size" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": 6, 114 | "metadata": {}, 115 | "outputs": [ 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "10000" 120 | ] 121 | }, 122 | "execution_count": 6, 123 | "metadata": {}, 124 | "output_type": "execute_result" 125 | } 126 | ], 127 | "source": [ 128 | "# number of validation samples\n", 129 | "val_size = len(val_folder.imgs)\n", 130 | "val_size" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "# Model" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 7, 143 | "metadata": { 144 | "collapsed": true 145 | }, 146 | "outputs": [], 147 | "source": [ 148 | "from get_squeezenet import get_model" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 8, 154 | "metadata": { 155 | "collapsed": true, 156 | "scrolled": true 157 | }, 158 | "outputs": [], 159 | "source": [ 160 | "model, loss, optimizer = get_model()" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 9, 166 | "metadata": {}, 167 | "outputs": [ 168 | { 169 | "data": { 170 | "text/plain": [ 171 | "827784" 172 | ] 173 | }, 174 | "execution_count": 9, 175 | "metadata": {}, 176 | "output_type": "execute_result" 177 | } 178 | ], 179 | "source": [ 180 | "# number of params in the model\n", 181 | "count_params(model)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "# Train" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": 10, 194 | "metadata": {}, 195 | "outputs": [ 196 | { 197 | "data": { 198 | "text/plain": [ 199 | "782" 200 | ] 201 | }, 202 | "execution_count": 10, 203 | "metadata": {}, 204 | "output_type": "execute_result" 205 | } 206 | ], 207 | "source": [ 208 | "from torch.optim.lr_scheduler import ReduceLROnPlateau\n", 209 | "\n", 210 | "n_epochs = 200\n", 211 | "n_batches = ceil(train_size/batch_size)\n", 212 | "\n", 213 | "lr_scheduler = ReduceLROnPlateau(\n", 214 | " optimizer, mode='max', factor=0.1, patience=4, \n", 215 | " verbose=True, threshold=0.01, threshold_mode='abs'\n", 216 | ")\n", 217 | "\n", 218 | "# total number of batches in the train set\n", 219 | "n_batches" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 11, 225 | "metadata": { 226 | "scrolled": false 227 | }, 228 | "outputs": [ 229 | { 230 | "name": "stdout", 231 | "output_type": "stream", 232 | "text": [ 233 | "0 4.768 4.074 0.053 0.118 0.165 0.327 115.271\n", 234 | "1 3.948 3.709 0.145 0.175 0.363 0.419 113.210\n", 235 | "2 3.634 3.363 0.193 0.228 0.437 0.498 113.220\n", 236 | "3 3.436 3.255 0.227 0.253 0.485 0.520 113.171\n", 237 | "4 3.294 3.139 0.252 0.275 0.515 0.552 113.301\n", 238 | "5 3.193 3.086 0.268 0.284 0.538 0.565 113.149\n", 239 | "6 3.089 2.988 0.288 0.305 0.562 0.583 113.123\n", 240 | "7 3.025 2.940 0.300 0.312 0.576 0.597 113.105\n", 241 | "8 2.952 2.947 0.314 0.315 0.590 0.597 113.212\n", 242 | "9 2.903 2.900 0.324 0.329 0.601 0.602 113.163\n", 243 | "10 2.862 2.767 0.332 0.349 0.610 0.624 113.071\n", 244 | "11 2.822 2.732 0.340 0.357 0.616 0.639 113.108\n", 245 | "12 2.782 2.738 0.346 0.352 0.626 0.634 113.138\n", 246 | "13 2.754 2.819 0.351 0.341 0.632 0.620 113.156\n", 247 | "14 2.734 2.681 0.355 0.363 0.636 0.646 113.206\n", 248 | "15 2.709 2.693 0.362 0.363 0.642 0.642 113.168\n", 249 | "16 2.690 2.761 0.364 0.352 0.644 0.627 113.054\n", 250 | "17 2.673 2.637 0.365 0.379 0.648 0.655 112.994\n", 251 | "18 2.653 2.666 0.373 0.368 0.650 0.653 113.018\n", 252 | "19 2.635 2.696 0.376 0.359 0.654 0.648 113.105\n", 253 | "20 2.628 2.585 0.377 0.388 0.655 0.664 113.004\n", 254 | "21 2.620 2.640 0.379 0.374 0.658 0.654 113.123\n", 255 | "22 2.602 2.596 0.383 0.386 0.661 0.660 113.102\n", 256 | "Epoch 22: reducing learning rate of group 0 to 4.0000e-03.\n", 257 | "Epoch 22: reducing learning rate of group 1 to 4.0000e-03.\n", 258 | "Epoch 22: reducing learning rate of group 2 to 4.0000e-03.\n", 259 | "Epoch 22: reducing learning rate of group 3 to 4.0000e-03.\n", 260 | "23 2.202 2.154 0.469 0.475 0.732 0.743 113.071\n", 261 | "24 2.074 2.116 0.493 0.485 0.753 0.749 113.004\n", 262 | "25 2.024 2.116 0.503 0.486 0.760 0.754 113.051\n", 263 | "26 1.995 2.082 0.509 0.496 0.767 0.755 112.939\n", 264 | "27 1.962 2.079 0.516 0.495 0.773 0.757 113.034\n", 265 | "28 1.944 2.073 0.519 0.497 0.776 0.758 113.073\n", 266 | "29 1.930 2.084 0.521 0.495 0.778 0.755 113.067\n", 267 | "30 1.910 2.051 0.526 0.498 0.781 0.764 113.023\n", 268 | "31 1.896 2.056 0.529 0.500 0.784 0.760 112.984\n", 269 | "32 1.891 2.073 0.528 0.499 0.784 0.757 113.065\n", 270 | "33 1.876 2.082 0.534 0.499 0.787 0.759 112.994\n", 271 | "Epoch 33: reducing learning rate of group 0 to 4.0000e-04.\n", 272 | "Epoch 33: reducing learning rate of group 1 to 4.0000e-04.\n", 273 | "Epoch 33: reducing learning rate of group 2 to 4.0000e-04.\n", 274 | "Epoch 33: reducing learning rate of group 3 to 4.0000e-04.\n", 275 | "34 1.775 1.978 0.556 0.517 0.803 0.771 112.831\n", 276 | "35 1.746 1.980 0.563 0.521 0.807 0.773 112.862\n", 277 | "36 1.731 1.977 0.567 0.520 0.810 0.771 113.006\n", 278 | "37 1.727 1.970 0.566 0.524 0.811 0.773 113.034\n", 279 | "38 1.722 1.979 0.567 0.525 0.811 0.774 113.045\n", 280 | "39 1.721 1.975 0.568 0.523 0.811 0.773 113.064\n", 281 | "Epoch 39: reducing learning rate of group 0 to 4.0000e-05.\n", 282 | "Epoch 39: reducing learning rate of group 1 to 4.0000e-05.\n", 283 | "Epoch 39: reducing learning rate of group 2 to 4.0000e-05.\n", 284 | "Epoch 39: reducing learning rate of group 3 to 4.0000e-05.\n", 285 | "40 1.707 1.968 0.571 0.526 0.814 0.774 113.096\n", 286 | "41 1.700 1.967 0.572 0.525 0.816 0.774 112.965\n", 287 | "42 1.704 1.972 0.571 0.525 0.815 0.774 113.022\n", 288 | "43 1.704 1.971 0.572 0.524 0.813 0.775 112.907\n", 289 | "44 1.697 1.968 0.575 0.524 0.816 0.774 112.902\n", 290 | "early stopping!\n", 291 | "CPU times: user 1h 12min 2s, sys: 13min 11s, total: 1h 25min 14s\n", 292 | "Wall time: 1h 24min 50s\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "%%time\n", 298 | "def optimization_step_fn(model, loss, x_batch, y_batch):\n", 299 | " return optimization_step(model, loss, x_batch, y_batch, optimizer)\n", 300 | "\n", 301 | "all_losses = train(\n", 302 | " model, loss, optimization_step_fn,\n", 303 | " train_iterator, val_iterator, n_epochs,\n", 304 | " patience=8, threshold=0.01, # for early stopping\n", 305 | " lr_scheduler=lr_scheduler\n", 306 | ")\n", 307 | "# epoch logloss accuracy top5_accuracy time (first value: train, second value: val)" 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": {}, 313 | "source": [ 314 | "# Loss/epoch plots" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 12, 320 | "metadata": {}, 321 | "outputs": [ 322 | { 323 | "data": { 324 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW9//HXJ5PJvpCNEJKw7yiLIKK4IG6Aa93QunZT\nW2/Ve9VWe9vetr/2/ry/9ra3XutVXG61LtW61A3cUdECCsgmhFUgYcsG2fd8f3+cIQQIISyTSTLv\n5+MxjzNzzpkznzkPmHfO93zP95hzDhEREYCIUBcgIiJdh0JBRERaKBRERKSFQkFERFooFEREpIVC\nQUREWigURESkhUJBRERaKBRERKRFZKgLOFLp6eluwIABoS5DRKRbWbJkSbFzLuNw63W7UBgwYACL\nFy8OdRkiIt2KmW3pyHpqPhIRkRYKBRERaaFQEBGRFt3unIKIyNFoaGigoKCA2traUJcSVDExMeTk\n5OD3+4/q/QoFEQkLBQUFJCYmMmDAAMws1OUEhXOOkpISCgoKGDhw4FFtQ81HIhIWamtrSUtL67GB\nAGBmpKWlHdPRkEJBRMJGTw6EvY71O4ZNKOTtLOc/3s6jrKYh1KWIiHRZYRMK+aU1/M9HG9lcXBXq\nUkQkDO3Zs4eHH374iN83c+ZM9uzZE4SK2hY2oZCbGgtA/u7qEFciIuHoUKHQ2NjY7vvmzJlDr169\nglXWQcKm91FuShwAW0sVCiLS+e677z42btzIuHHj8Pv9xMTEkJKSQl5eHuvWreOyyy4jPz+f2tpa\n7rzzTm655RZg39A+lZWVzJgxg9NPP51//OMfZGdn89prrxEbG3tc6wybUIiPjiQtPor80ppQlyIi\nIfbLN75i9fby47rNUX2T+LeLRx9y+QMPPMCqVatYtmwZH330ERdeeCGrVq1q6Tr65JNPkpqaSk1N\nDSeffDJXXHEFaWlp+21j/fr1PP/88zz22GNcffXVvPzyy1x//fXH9XuETSgA5KbGka8jBRHpAiZN\nmrTftQQPPvggr776KgD5+fmsX7/+oFAYOHAg48aNA2DChAls3rz5uNcVdqGwoqDzTtiISNfU3l/0\nnSU+Pr7l+UcffcT777/PggULiIuLY+rUqW1eaxAdHd3y3OfzUVNz/Fs+wuZEM0C/1Fi27a6hsak5\n1KWISJhJTEykoqKizWVlZWWkpKQQFxdHXl4eCxcu7OTq9gmvI4WUOBqbHTvKaslNjQt1OSISRtLS\n0pgyZQonnHACsbGxZGZmtiybPn06jzzyCCNHjmT48OFMnjw5ZHWGVSj0CwRB/u5qhYKIdLrnnnuu\nzfnR0dHMnTu3zWV7zxukp6ezatWqlvn33HPPca8Pwqz5aG8Q6GSziEjbwioUspJj8EWYuqWKiBxC\nWIVCpC+C7F6xuoBNROQQwioUwBvuQkNdiIi0LexCoZ8uYBMROaSgh4KZ+czsSzN7s41lU82szMyW\nBR4/D3Y9OSlxFFfWU1XX/iBUIiLhqDO6pN4JrAGSDrF8vnPuok6oA9jXLbVgdw3D+yR21seKiByR\nhIQEKisrO/1zg3qkYGY5wIXA48H8nCOxNxR0sllE5GDBPlL4L+BHQHt/kp9mZiuAbcA9zrmvglmQ\nrlUQkVC47777yM3N5fbbbwfgF7/4BZGRkcybN4/du3fT0NDAr3/9ay699NKQ1hm0UDCzi4BC59wS\nM5t6iNWWAv2cc5VmNhP4OzC0jW3dAtwC0K9fv2OqKyXOT0J0pI4URMLZ3Ptg58rju80+J8KMBw65\neNasWdx1110tofDiiy/yzjvvcMcdd5CUlERxcTGTJ0/mkksuCem9pIPZfDQFuMTMNgN/BaaZ2TOt\nV3DOlTvnKgPP5wB+M0s/cEPOudnOuYnOuYkZGRnHVJSZkZMSS4G6pYpIJxo/fjyFhYVs376d5cuX\nk5KSQp8+ffjJT37CmDFjOPfcc9m2bRu7du0KaZ1BO1Jwzt0P3A9eLyO8pqH97gZhZn2AXc45Z2aT\n8EKqJFg17dUvNY7NJbpXs0jYaucv+mC66qqreOmll9i5cyezZs3i2WefpaioiCVLluD3+xkwYECb\nQ2Z3pk4fEM/MbgNwzj0CXAl838wagRrgGuecC3YNualxzF9fjHMupIdpIhJeZs2axfe+9z2Ki4v5\n+OOPefHFF+nduzd+v5958+axZcuWUJfYOaHgnPsI+Cjw/JFW8x8CHuqMGlrrlxpHTUMTxZX1ZCRG\nH/4NIiLHwejRo6moqCA7O5usrCyuu+46Lr74Yk488UQmTpzIiBEjQl1ieA2dvVduqnej662l1QoF\nEelUK1fuO8Gdnp7OggUL2lwvFNcoQBgOcwGtL2DTyWYRkdbCMhRyUgIXsJUoFEREWgvLUIjx++id\nGK3RUkXCTCf0Ywm5Y/2OYRkK4DUh6QI2kfARExNDSUlJjw4G5xwlJSXExMQc9TbC8kQzeN1SP/+6\nNNRliEgnycnJoaCggKKiolCXElQxMTHk5OQc9fvDOhReW7aN+sZmoiLD9oBJJGz4/X4GDhwY6jK6\nvLD9NcxNiaXZwfY9ul+ziMheYRsKe7ul6mSziMg+YRsKubqvgojIQcI2FDKTYojyRZBfquYjEZG9\nwjYUfBHeENq62Y6IyD5hGwoAOalxOqcgItJKWIdCv9RYnVMQEWklrEMhNyWOPdUNlNc2hLoUEZEu\nIaxDoaVbqo4WRESAMA+FXIWCiMh+FAqgbqkiIgFhHQrJsX6SY/062SwiEhDWoQDerTnVLVVExBP2\noaD7KoiI7BM+obDhfXjoZKgq2W92bkocBaU1NDf33BtviIh0VPiEQmwKFK/zwqGV3NQ46puaKayo\nC1FhIiJdR9BDwcx8Zvalmb3ZxjIzswfNbIOZrTCzk4JWSNZ4iO8N69/Zb7ZGSxUR2aczjhTuBNYc\nYtkMYGjgcQvwP0GrIiIChp7vHSk0NbbM1gVsIiL7BDUUzCwHuBB4/BCrXAo87TwLgV5mlhW0goZd\nALVlkL+oZVbfXjGY6UhBRASCf6TwX8CPgOZDLM8G8lu9LgjMC47BZ0OEH9a93TIrOtJHVlKMuqWK\niBDEUDCzi4BC59yS47CtW8xssZktLioqOvoNRSdC/9Ng/bv7zc5NjVPzkYgIwT1SmAJcYmabgb8C\n08zsmQPW2QbktnqdE5i3H+fcbOfcROfcxIyMjGOrath0KMqD3ZtbZuXqWgURESCIoeCcu985l+Oc\nGwBcA3zonLv+gNVeB24M9EKaDJQ553YEqybAO68AsG7f0UK/1Dh2lddR29AU1I8WEenqOv06BTO7\nzcxuC7ycA2wCNgCPAT8IegFpgyF18H5dU0/MTgbgHxuLg/7xIiJdWWRnfIhz7iPgo8DzR1rNd8Dt\nnVHDfoZNhy8eh/oqiIpnypB0kmP9vLF8B9NGZHZ6OSIiXUX4XNHc2rDzoakONn0MQFRkBNNH9+G9\n1bvUhCQiYS08Q6HfaRCVuF8T0kVjs6isa+SjtYUhLExEJLTCMxQio7xrFta9C84bCO/UQWmkxUfx\nxvLgnucWEenKwjMUwOuFVLEddq4EINIXwcwTs/ggbxdVdY2HebOISM8UvqEw9Hxvuq5VE9KYLGob\nmnl/za4QFSUiElrhGwoJvaHvSfudVzh5QCqZSdG8uUJNSCISnsI3FMBrQipYDFXe9QkREcZFY/ry\n8doiymoaQlyciEjnUyjgYP17LbMuGpNFfVMz7361M3R1iYiESHiHQp+xkJC5XxPSuNxe5KTEqglJ\nRMJSeIdCy413PoAmr7nIzGtC+nRDMaVV9SEuUESkc4V3KIDXhFRXDlsXtsy6eGwWTc2Ot1epCUlE\nwotCYdDUg268MyoriUEZ8byxfHvIyhIRCQWFQnQiDDh9vxvv7G1CWvh1CYXltSEsTkSkcykUwBs1\ntXgdlGxsmXXxmCycgzkrdcJZRMKHQgFg+AxvunZOy6yhmYmM6JPIG+qFJCJhRKEAkNIfMk+EvLf2\nm33x2L4s2bKbbXtqQlSYiEjnUijsNWIm5C+CyqKWWReNyQLgrRU64Swi4UGhsNeIC8E179cLqX9a\nPGNykjWctoiEDYXCXn3GQHLufucVAC4fn83KbWW6f7OIhAWFwl5mMHwmbPzQu3dzwDWT+tE3OYYH\n5ubR3OxCWKCISPApFFobMRMaa2HjvJZZMX4f/3L+cFYUlDFnlZqRRKRnUyi01n8KxCQf1AvpG+Oz\nGdEnkd++s5b6xuYQFSciEnwKhdZ8fu9CtnVvQ9O+W3L6IowfzxjBlpJqnv98awgLFBEJrqCFgpnF\nmNnnZrbczL4ys1+2sc5UMyszs2WBx8+DVU+HDZ8JNaWQv3C/2VOHZXDqoDQe/GA9FbW6AY+I9EzB\nPFKoA6Y558YC44DpZja5jfXmO+fGBR6/CmI9HTPkHPBFH9SEZGbcN2MEJVX1PPbJphAVJyISXEEL\nBeepDLz0Bx5dv/tOdCIMOssLBbd/uWNze3HhmCwem/+1BsoTkR4pqOcUzMxnZsuAQuA959yiNlY7\nzcxWmNlcMxsdzHo6bMSFsGcL7PrqoEX3nj+chqZm/uuD9SEoTEQkuIIaCs65JufcOCAHmGRmJxyw\nylKgn3NuDPDfwN/b2o6Z3WJmi81scVFRUVurHF/DZgB20IVsAAPS47nulH688EU+G4sqD36viEg3\n1im9j5xze4B5wPQD5pfvbWJyzs0B/GaW3sb7ZzvnJjrnJmZkZAS/4MRMyDkZ8t5sc/EPzxlKTGQE\nv317bfBrERHpRMHsfZRhZr0Cz2OB84C8A9bpY2YWeD4pUE9JsGo6IiNmwo7lUFZw0KL0hGhuPWsw\nb3+1kyVbdoegOBGR4AjmkUIWMM/MVgBf4J1TeNPMbjOz2wLrXAmsMrPlwIPANc65rnEyesRF3jTv\n4CYkgO+eMZCMxGh+89ZqmjT8hYj0ENZVfoM7auLEiW7x4sWd82EPnQxJfeHG19pc/OqXBfzzC8u5\n45yh/Mt5wzqnJhGRo2BmS5xzEw+3nq5obs/wmbD5U6jZ0+bib4zP4aoJOfz3h+v5ZF0nnAAXEQky\nhUJ7RlwEzY2w/r1DrvKrS09gWO9E7nphGTvKdIc2EeneFArtyZ4ACZmH7IUEEBvl4+HrT6KuoYkf\nPvclDU0aME9Eui+FQnsiImD4DFj/rtcT6RAGZyTw75efyOItu/ndO+qmKiLdl0LhcM64G+LS4OlL\n2w2GS8dlc/3kfjz6ySbeW72rEwsUETl+OhQKZnanmSWZ5wkzW2pm5we7uC6hVz+46Q2ISjhsMPz0\nwlGckJ3E3S8uI7+0uhOLFBE5Pjp6pPBt51w5cD6QAtwAPBC0qrqa1IEdCoYYv4+HvzkBB/zTc0up\na2zq3DpFRI5RR0PBAtOZwF+cc1+1mhceOhgM/dLi+N1VY1leUMYvXl9Nd7sORETCW0dDYYmZvYsX\nCu+YWSIQft1sOhgMF4zuw/enDub5z7fyr39fpSueRaTb6GgofAe4DzjZOVeNd2+EbwWtqq6sg8Hw\nowuG84Opg3lu0Vb+5cVl6qoqIt1CR0PhVGCtc26PmV0P/BQoC15ZXdyBwVCYd9AqZsaPpo/g3guG\n89qy7Xz/maXUNugcg4h0bR0Nhf8Bqs1sLHA3sBF4OmhVdQepA+Gm18EXBX/5BuzZ2uZqt589hF9d\nOpr31+ziO099QVVdYycXKiLScR0NhcbA6KWXAg855/4EJAavrG4idRBc/wo0VMHTl0FlYZur3Xjq\nAH531VgWbCzhhicWUVbT0MmFioh0TEdDocLM7sfrivqWmUXgnVeQPifAN/8G5dvhmcuhtu1WtSsn\n5PCnb57Eym1lXDt7IcWVdZ1cqIjI4XU0FGYBdXjXK+zEu73mb4NWVXfT7xSY9Yx3buG5a6Ch7YHx\nZpyYxeM3ncym4kpmPbqAnWW1nVyoiEj7OhQKgSB4Fkg2s4uAWudceJ9TONDQc+HyR2HrAvjbzdDU\ndhPRWcMyePrbp7CzrJZZsxdQsFtXPotI19HRYS6uBj4HrgKuBhaZ2ZXBLKxbOuEKuPA/Yd3b8Nrt\n0Nx2N9RJA1N55runsLuqnlmPLmRzcVUnFyoi0raONh/9K941Cjc5524EJgE/C15Z3djJ34FpP4MV\nL8BrP4CKtgfHG98vhee+N5nq+kaufnQBGworOrlQEZGDdTQUIpxzrbvWlBzBe8PPGXfDmfd6wfDH\nMTD3PijfcdBqJ2Qn88Ktp9LsYNajC1m9vTwExYqI7NPRH/a3zewdM7vZzG4G3gLavqO9gBlM+yn8\n02KvSenz2fDHsTDnXijbtt+qwzITefHWyURFRnDtYwtZnt/2rT9FRDqDdXTANjO7ApgSeDnfOfdq\n0Kpqx8SJE93ixYtD8dFHr/RrmP+fsPx5sAgYf4N3NJGc3bJKfmk133x8IburGph9wwROG5IewoJF\npKcxsyXOuYmHXa+7jeLZLUNhr91b4NPfw5fPQkwSXPcSZJ/UsnhHWQ3XPb6ITUVVXHFSDvfNGEFG\nYnQICxaRnuK4hIKZVQBtrWCAc84lHX2JR6dbh8Jexeu9C92qS73rGwaf3bKour6Rhz7cwGPzNxHj\n93H3ecO4fnJ/In06hSMiR6+jodDuL41zLtE5l9TGI/FwgWBmMWb2uZktN7OvzOyXbaxjZvagmW0w\nsxVmdlJb2+px0ofCt9+FXv3h2atg1csti+KiIvnR9BG8fdeZjMvtxS/eWM3FD33G4s2lISxYRMJF\nMP/8rAOmOefGAuOA6WY2+YB1ZgBDA49b8AbeCw9JWfCtOZBzMrz0HVj06H6LB2ck8PS3J/E/153E\nnup6rnxkAXe/uFzDY4hIUAUtFJynMvDSH3gc2BR1KfB0YN2FQC8zywpWTV1ObC+44RUYPhPm/gg+\n/DW0as4zM2acmMUHd5/F96cO5vXl2zj/D58wZ+XB3VtFRI6HoDZUm5nPzJYBhcB7zrlFB6ySDeS3\nel0QmBc+/LFw9dMw/nr45Lfw5l3QvP99F+KiIvnx9BHMueMMclJi+cGzS7nj+S/ZU10foqJFpKcK\naig455qcc+PwBtCbZGYnHM12zOwWM1tsZouLioqOb5FdgS8SLnnI66a65M/wxPmw5s2DwmFoZiIv\nf/807j5vGHNW7uC8P3zCB2vavmJaRORodEqXFufcHmAeMP2ARduA3FavcwLzDnz/bOfcROfcxIyM\njOAVGkpmcM7P4dKHoaoQXrgOHjoZvngc6vcNmuf3RfDDc4by2j9NIS0+iu88tZh7/7ac8lrdo0FE\njl3QQsHMMsysV+B5LHAecOB9K18Hbgz0QpoMlDnnwrvBfPx18MMv4cr/hZhkeOtu+MNomPfvULnv\nKGl0Zhyv3TyUn50SweYvP+CXv/sDcxet0L2gReSYBO3iNTMbAzwF+PDC50Xn3K/M7DYA59wjZmbA\nQ3hHENXAt5xz7V6E0COuU+go52DLP2DBQ7B2DviiISETanZD/cED6NW5SN6LPJO6ibdxwbRzSIiO\nDEHRItIV6YrmnqZ4vdeUVFsGsakQm+L1XopNgdgUmn3RbP/sWdI3vkyMq2MhJ7J12M2cMfNasnrF\nh7p6EQkxhUK4qi5lx4ePELPsCVIai9no+vJl1iwGX/B9xg3ojXdwJiLh5rhc0SzdUFwqWRf9hJT7\n8yi54E/ExCdz5c4/sPXJmznv9x8z+5ONFFXoAjgRaZuOFHo656ib9x9Ef/J/eTrhO/y8+Bx8Eca0\nEb25emIuU4dn4Ne4SiI9XkePFHQmsqczI/rsH0PxGm5c/STnXnk2TxUN4ZWl23hv9S7SE6K5bFxf\nrpiQw8isTh/fUES6GB0phIv6KnjiAtizFb73IY0pg/h4XREvLs7nw7xCGpoco7KSuPykbC4dl60h\nu0V6GJ1oloPt3gKPnQ1xafDd973rIIDSqnreWL6dV5YWsLygDF+EcdawDK44KYdzRvYmxu87ss9p\nboKII3yPiASVQkHatvlTePpSGHIuXPM8ROx/PmH9rgpe+XIbry7dxs7yWpJiIrl4rNe8ND63V/u9\nl+qrYM6PIO8NuOlNyBoT5C8jIh2lUJBD+/wxmHMPnHEPnPOzNldpanb8Y2MxLy8p4O2vdlLb0Myg\n9HiumJDDN8Zn07dX7P5vKMyDv90ERWu9u8rF9IJbP/auoxCRkFMoyKE5B2/cAUuf9obTOOHydlev\nqG1g7sqdvLS0gM+/LsUMJg9MY+rwDKYMSWdU4ZtEzLkHouLh8se86f/OhMHT4Nq/HnQ0IiKdT6Eg\n7Wusg6cuhp0r4fR/gVGXQsaww75ta0k1r3xZwJsrdrCtsJhfRf6ZqyI/YV3sOFae8p+cdMJIBqTF\nYZ8/BnPvhbN/Cmfde2y1lm+HhhpIG3xs2xEJYwoFObyKXfDSt2HLp97rjBFeOIy8BDJHeyO3Hkph\nHo0v3IivZB3vpd/EL8svZFu5N1JrVnIMo7OSuLvyt4wofpfiy54lfezMI7+auqkRFj0C834DGHzr\nLeg7/ui+q0iYUyhIx5VvhzVvwOrXYes/wDVD6mAYNt1r+qmrgNpyb1oXmJZuguhEr7lo8Nk45/i6\nuIrPNpbwxdelfLW9jB3FJbzi/zcybTfX2v8jpe9gTshOYkL/FCb0T22/2+uO5fD6HbBjGQy9AArX\nQFOd12uqV7/O2zciPYRCQY5OZSHkvekFxOb5EBHp/fhHJ3nTmCTveWIfOPNeb3oI1fWNbF67giGv\nXUyhP4d/jn+A5TtrqW/0hvfunxbHhP4pTOyfysQBKQzJSCCisQY++r+w4E9e19mZ/w9GXQZFed51\nFklZ8O13vMEARaTDFApy7Jqbj89J4ry34K/fhAk3Uzfj96zaVs6SLaUs3rybJVt2U1Ll3Vb0/Oiv\n+D+RT5DZtJN1OVdQPuWnDO2fS3Kc39vO15/AXy6HfpPh+lcgMuroa2qo8W6FKhImFArStbz/C/j0\nD3DuLyF1EFTsgPJtuPLt1JYU0LSngITqfLb7svlp0/f4sHbfSe8+STEM65PI4Ix4zq2fx5SV/0rN\nqKuIuXI21tHQamqEbYthwwew8UPYvhQGTYVrnlM4SFhQKEjX0tQIz3zD+2t/L18UJGZBUrbXLJQ1\nDibdgouMZmd5LXk7K1gbeKwvrGBjYRU1DU380PcKd/tf4hGu5J3e32ZY70RGZycxKiuJkVlJxO+9\nuVDp114AbPzQ+9y6crAIyJ4AvUd5XXIHTwsEQ0xo9otIJ9GAeNK1+CLhmy/C1gXeuYKkbG/aRo8k\nA7KSY8lKjuXs4b1b5jc3O3aW17Jh18ms+7iR27a/hKvvx+zVp/LC4nx6UcEU31dMj13LZFtJRsN2\nABoT+uIbdRk25BwYeCbEpXobzJkIr//Qu+ju6r8cW3OUSA+hIwXpnpoa4NmrYPN83Ek30bD1C/yF\nKzEcNRbPFzaa9+tG8lnzCWx0fYn1RzKkdwJDeycwJDOBIRkJZCXHkr3hOVI/ug834iLsqj+Dz3/o\nz6zYBSv+ChkjYcg5Gt9JuhU1H0nPV1vuXTldlAe5k7xzBIOmQt+TwBdJWXUD6worWL+rkvWFFWwo\nrGRDYSU7ymr328y3fHP5N/9feJtTeSD2bhLiYugVG0VKfBQpcX6y/FWcvutZRha8QGST917Xqx82\n4WYYfwMk9Eakq1MoSHhorIPmRm9ojQ4qr21gY2ElRRV1lNU0UFbTwNAN/8tZWx5kcdJ5PNzrbnbX\nNtNUVcrFVS9zrZtLHHX8vXkKDzdewjAr4Eb/B0y2r2gkks29p1E++gYyx5xLdkpcEL+syNFTKIgc\nqU9+Cx/+GsZd510gt+BPUFdO8+jLKT/lbopjBlBaVc+mokrW7aqkcttXjC/8OzObPiTZqlnXnE3R\nJc8wZeJJof4mIgfRiWaRI3Xmvd65io//w3s98mKYej8RmaPpBey9XG7SwMCJakYBV1FWVs7Xnz/P\nkM9+zObPnwGFgnRjCgWR1qbe740BlTYYssZ26C3JyUkkn3cr+V88SkrxYpxzRz7Ok0gXoTGNRVoz\n84YS72AgtFbVZxKjm/LYUlgWhMJEOkfQQsHMcs1snpmtNrOvzOzONtaZamZlZrYs8Ph5sOoRCbaU\nUWcTZ3Ws+fKTw68s0kUF80ihEbjbOTcKmAzcbmaj2lhvvnNuXODxqyDWIxJUvUdPBaB6nUJBuq+g\nhYJzbodzbmngeQWwBsgO1ueJhJolZlIY3Z/00iU0NjWHuhyRo9Ip5xTMbAAwHljUxuLTzGyFmc01\ns9GHeP8tZrbYzBYXFRUFsVKRY1OTdQrjXR4r8ktDXYrIUQl6KJhZAvAycJdzrvyAxUuBfs65McB/\nA39vaxvOudnOuYnOuYkZGRnBLVjkGKSNPpskqyZv+cJQlyJyVIIaCmbmxwuEZ51zrxy43DlX7pyr\nDDyfA/jNLD2YNYkEU8KwMwGo2zg/xJWIHJ1g9j4y4AlgjXPu94dYp09gPcxsUqCekmDVJBJ0yTns\nie5L3z1LqaprDHU1IkcsmEcKU4AbgGmtupzONLPbzOy2wDpXAqvMbDnwIHCN627jbogcoC57MhNt\nDYs2FYe6FJEjFrQrmp1zn+INjd/eOg8BDwWrBpFQSB05Ff+mV1izcgnTRl4Y6nJEjoiuaBY5zvyD\nTgegcZPOK0j3o1AQOd5SB1EVlcGAquXsPODeDSJdnUJB5HgzozH3VCZF5PHZel1XI92LQkEkCBKH\nn0WWlbJmzcpQlyJyRBQKIkEQMWAKAE2bP0Ud6qQ7USiIBEPGCGqjUhhVt5K1uypCXY1IhykURILB\nDBc4r/Dpel2vIN2HQkEkSGKHnEH/iEJWrVkT6lJEOkyhIBIs/U8DwFewgLrGphAXI9IxCgWRYOlz\nIg3+BE5qXs3SLXtCXY1IhygURIIlwgf9JjPJl8enG3S9gnQPCgWRIPIPPJ2hto2VazeEuhSRDlEo\niARTf+96hbhdX7Cnuj7ExYgcnkJBJJiyxtEUGcsky+M/3s7ThWzS5SkURIIpMgpf7iQuTNrE85/n\n8+Rnm0NdkUi7FAoiwdZ/Cr2r13P5iBh+89Zq5uUVhroikUNSKIgE25BzMBz/uf1GZic+wfPP/5m1\n23eHuir3B94jAAAO7klEQVSRNikURIItZyJ8ay42+jKm2RfMtt+QPnscNa/fAwWLQecZpAux7nbi\na+LEiW7x4sWhLkPk6DTUsnnhq+S99yTTIr4kigZIzILoRG/5gf8fo+JgyHkw8mLIGgvW7h1uRQ7J\nzJY45yYebr2g3aNZRNrgj2HAGdeyKvksJjz3GT8ZtJ5r0jZhzY2tVmr1w19ZCJ/+Aeb/DpL7wciL\nvIDIPcW7OE7kOFMoiITARWP6srFwHPe/H8fuYVfzg6lDDr1yVQmsmwtr3oAvnoCFD0N8Bgw4A6Li\nwRcFkdHg83vPfVEQGQMxyRCTFJgmQ3Rg6pqhYjuUBx4VO/Y9j+0Fw2bA0PMhPu3Yv2h9NVQVet+h\nugSqi71pVTHUV8KEm6HPicf+OXLcqPlIJEScc9zx12W8sXw7U4dncOuZg5k8KBVrr4morgLWvwd5\nb3rnI5oaoKkuMK2HxjrgCP9Pm89rwkrKgrICLyQsAnInw/AZMOJCSBvc/jZq9kDxOijKg6K1+x5l\nW9teP8LvHen4ouH6lyB30pHVLEeso81HCgWREKptaOLx+Zv43882U1JVz9icZG49azAXjO6DL+Io\nzx80NUJDNdSVQ20Z1O6dBh5mgRDo6z3iM/Y1RTU3w45lsHau99gVuJ1o+jBIHextt6Em8KjypvXV\nUFe27/MjYyBtKGQM9x5JfSEuHeLSvKOPuDSITvIC6OlLoGIXXPciDDj92HamtCvkoWBmucDTQCbe\nny6znXN/PGAdA/4IzASqgZudc0vb265CQXqi2oYmXl5awGOfbGJzSTX90+L43hmDuHJCDjH+EJ47\n2LMV1r7tNV9VFYM/DvyxraaB50l9IWMEZAyDXv07fr6jYic8fSns3gLXPAtDzgnu9wljXSEUsoAs\n59xSM0sElgCXOedWt1pnJvBDvFA4Bfijc+6U9rarUJCerKnZ8e5XO3nkk00sz99DcqyfM4dlcObQ\ndM4clkFmUkyoSzz+qorh6cugeC1c/bTXZCXHXchD4aAPMnsNeMg5916reY8CHznnng+8XgtMdc7t\nONR2FAoSDpxzfP51KS8szmf++mKKKuoAGJ6ZyJnD0jljaAaTBqaG9ijieKrZDc9cATuWwxWPw+hv\nhLqiHqdLdUk1swHAeGDRAYuygfxWrwsC8/YLBTO7BbgFoF+/fsEqU6TLMDNOGZTGKYPScM6xZkcF\n89cX8cn6Ip76xxYem/81Mf4ITh+SwXmjenP2iN70TuzGRxGxKXDD3+G5q+Glb3snzMde4y1rbgbX\nBM2NgUeT19Mqwu9Nde3GcRX0IwUzSwA+Bn7jnHvlgGVvAg845z4NvP4A+LFz7pCHAjpSkHBXXd/I\noq9LmZdXyAdrCtm2pwaAcbm9OG9UJueOzGRYZkL7vZi6qvoqeP5a+Ppj70e/uZHD9qbaGw4+v9eb\nKSYJYnp53Wtjkvc9j4yF2j2BrrGtH7u97rF7z5FExrY6XxLrdfttvZ3W06h4L5QsAghMzbznzYET\n/vVVgWm1d3K+vjrwvWgVaLbvdURkq27Gga7GkdHevN4jIXP0Ue3aLtF8ZGZ+4E3gHefc79tYruYj\nkWPgnCNvZwXvr97F+2t2sbzA6wWUmRTN8D5JDM9MYFhmIsP7JDKkdwJxUd3g0qSGWlj0iNdTKsLn\n/Ui2TCO9H97mRq8LblNg2tzgdcttrA30ttrjdZOt3eNtp2aPd7Thj/d6P8WlBKaBR1S8d3SyX++q\nGmisgbrKVtsr87ZzLCzCC7K9YdfyG+y85+1t//R/hnN/cXQfG+pQCPQsegoodc7ddYh1LgT+iX0n\nmh90zrXbYVmhIHJou8pr+WBNIV9sLmXdrgrWF1ZS39gMeH+E5qbEMTgjntzUOHJT4shJiW15nhzn\nD3H1QeScFxqRUce+nbqKfSHRUB34UXfeRYEuMMV5AeaP8wKn9TQyuv0mL+cCgVcPjfWB61ACz2OS\nITHzqErvCqFwOjAfWAk0B2b/BOgH4Jx7JBAcDwHT8bqkfqu9piNQKIgciaZmx5aSKtbtqmTdrgrW\n7qzg6+Iq8ndXU1HbuN+6iTGRDEyPZ1RWEiOzkhjVN4kRfRJJjOnBYRFGQh4KwaJQEDk+yqobyN9d\nTcHuavJLa8jfXc2GwkrW7Chnd3VDy3q5qbGMykpiSO8E+qd6Rxn90uLokxRz9BfYSafrUr2PRKTr\nSY7zkxyXzAnZyfvNd86xs7yWNTvKWbOjgtU7ylmzvZz31xTS1Lzvj8goX0RL81NafBRx0T7ioyOJ\nj4okLsp7Hhflo09SDP3T4umdGE2EQqTLUyiIyH7MjKzkWLKSY5k2Yl/7dWNTMzvKatlSUs3WUu+R\nX1rNltIqNhZVUl3fRFVdI3WNzW1uNzoygn6pcfRPi6d/WhwD0uO5ZEzfnn0uoxtSKIhIh0T6IryT\n0qlx7a7X2NRMdUMT1XVNVNY1sH1PLVtKq9laUsXmkmq2llTz6YYiahua+cuCzTzznVPo3ROv1O6m\nFAoiclxF+iJI8kWQFOMHYhjSO/GgdZxzfLqhmNv+soQrH1nAs9895bBhI51Dt+MUkU5nZpwxNINn\nvnsKZTUNXPXIAjYUVoS6LEGhICIhNL5fCi/cOpnGZsfVjy5k1bayw79JgkqhICIhNaJPEi/ddiqx\nfh/Xzl7IF5tLQ11SWFMoiEjIDUiP52+3nUpGUjQ3PLGIj9cVhbqksKVQEJEuoW+vWF689VQGpSfw\n3ae+4PbnlvL851vJL60OdWlhRb2PRKTLSE+I5vlbJvPA3DXMyyvirRXe2Jj9UuOYMiSdM4amc+qg\nNFLij3EMIzkkDXMhIl2Sc46NRVV8tqGYTzcUs3BjCRV13nhN6QnR9E+Lo39gyI3+ad5FcTkpsSTF\n+ImOjOieQ4cHkcY+EpEepbGpmeUFZSz6uoTNxVUtV1bvLK/lwJ8xM4iPiiQ2ykdclI+4qEjio3zE\nRvlahuGIi/bmx0X5iPX78EUYZobPwBdhREQYPjNio3zkpMSSkxJHRkL3HapDYx+JSI8S6YtgQv8U\nJvRP2W9+bUMTBbur2VJSzfY9NVTWNVFd30h1vTetqmtqeV5R28iu8trAa29ebUPbw3K0JSoygpxe\nsWQHQiIzKZrkWD/JsX6SYvzeeFKB536fUdvYTF1DE3WNzdQGpnWNzTQ17/vMAwMt1u8jKbDN5Dg/\nidGRnXrUo1AQkW4txu9jSO/ENq+c7oimZkdtQxPNztHcDE3O0dTscM7R5ByVtY0U7KmhYHcNBaXV\n3nR3Ne9u30lJVf1x/jYHizBIivXTK9bP9ZP7890zBgX18xQKIhLWfBFGfHQ7P4XJMDSz7cBpbGqm\nvLaRspoGymsavGmtN21scsT4I4jx+4iOjCA6MjD1RxAZsX/Hz70HAs5BTUMTZTUNlFV722n9SE+I\nPl5f+5AUCiIiRynSF0FqfBSpPag3lK5TEBGRFgoFERFpoVAQEZEWCgUREWmhUBARkRYKBRERaaFQ\nEBGRFgoFERFp0e0GxDOzImDLUb49HSg+juX0FNovB9M+OZj2ycG60z7p75zLONxK3S4UjoWZLe7I\nKIHhRvvlYNonB9M+OVhP3CdqPhIRkRYKBRERaRFuoTA71AV0UdovB9M+OZj2ycF63D4Jq3MKIiLS\nvnA7UhARkXaETSiY2XQzW2tmG8zsvlDXEwpm9qSZFZrZqlbzUs3sPTNbH5imtLeNnsbMcs1snpmt\nNrOvzOzOwPyw3S9mFmNmn5vZ8sA++WVgftjuk73MzGdmX5rZm4HXPW6fhEUomJkP+BMwAxgFXGtm\no0JbVUj8GZh+wLz7gA+cc0OBDwKvw0kjcLdzbhQwGbg98G8jnPdLHTDNOTcWGAdMN7PJhPc+2etO\nYE2r1z1un4RFKACTgA3OuU3OuXrgr8ClIa6p0znnPgFKD5h9KfBU4PlTwGWdWlSIOed2OOeWBp5X\n4P2HzyaM94vzVAZe+gMPRxjvEwAzywEuBB5vNbvH7ZNwCYVsIL/V64LAPIFM59yOwPOdQGYoiwkl\nMxsAjAcWEeb7JdBMsgwoBN5zzoX9PgH+C/gR0NxqXo/bJ+ESCtIBzuuKFpbd0cwsAXgZuMs5V956\nWTjuF+dck3NuHJADTDKzEw5YHlb7xMwuAgqdc0sOtU5P2SfhEgrbgNxWr3MC8wR2mVkWQGBaGOJ6\nOp2Z+fEC4Vnn3CuB2WG/XwCcc3uAeXjnosJ5n0wBLjGzzXjNz9PM7Bl64D4Jl1D4AhhqZgPNLAq4\nBng9xDV1Fa8DNwWe3wS8FsJaOp2ZGfAEsMY59/tWi8J2v5hZhpn1CjyPBc4D8gjjfeKcu985l+Oc\nG4D3+/Ghc+56euA+CZuL18xsJl6boA940jn3mxCX1OnM7HlgKt7IjruAfwP+DrwI9MMbffZq59yB\nJ6N7LDM7HZgPrGRfW/FP8M4rhOV+MbMxeCdNfXh/OL7onPuVmaURpvukNTObCtzjnLuoJ+6TsAkF\nERE5vHBpPhIRkQ5QKIiISAuFgoiItFAoiIhIC4WCiIi0UCiIdCIzm7p3hE2RrkihICIiLRQKIm0w\ns+sD9xRYZmaPBgaIqzSzPwTuMfCBmWUE1h1nZgvNbIWZvbp3TH0zG2Jm7wfuS7DUzAYHNp9gZi+Z\nWZ6ZPRu4qlqkS1AoiBzAzEYCs4ApgUHhmoDrgHhgsXNuNPAx3hXhAE8DP3bOjcG7Mnrv/GeBPwXu\nS3AasHc0zfHAXXj39hiEN66OSJcQGeoCRLqgc4AJwBeBP+Jj8QY6awZeCKzzDPCKmSUDvZxzHwfm\nPwX8zcwSgWzn3KsAzrlagMD2PnfOFQReLwMGAJ8G/2uJHJ5CQeRgBjzlnLt/v5lmPztgvaMdI6au\n1fMm9P9QuhA1H4kc7APgSjPrDS334e2P9//lysA63wQ+dc6VAbvN7IzA/BuAjwN3cSsws8sC24g2\ns7hO/RYiR0F/oYgcwDm32sx+CrxrZhFAA3A7UIV3w5mf4jUnzQq85SbgkcCP/ibgW4H5NwCPmtmv\nAtu4qhO/hshR0SipIh1kZpXOuYRQ1yESTGo+EhGRFjpSEBGRFjpSEBGRFgoFERFpoVAQEZEWCgUR\nEWmhUBARkRYKBRERafH/ARCHV18Tu01TAAAAAElFTkSuQmCC\n", 325 | "text/plain": [ 326 | "" 327 | ] 328 | }, 329 | "metadata": {}, 330 | "output_type": "display_data" 331 | } 332 | ], 333 | "source": [ 334 | "epochs = [x[0] for x in all_losses]\n", 335 | "plt.plot(epochs, [x[1] for x in all_losses], label='train');\n", 336 | "plt.plot(epochs, [x[2] for x in all_losses], label='val');\n", 337 | "plt.legend();\n", 338 | "plt.xlabel('epoch');\n", 339 | "plt.ylabel('loss');" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": 13, 345 | "metadata": {}, 346 | "outputs": [ 347 | { 348 | "data": { 349 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8lfXZx/HPlb1JSMJKgIS9h4RRR8GNaMWF4qqzal3Y\nTddT21prn/axS62rVK0oKoiiotaBqEWUMGSDzAwIGZC9zrieP+5DTMIKkJOT5Fzv1+u8zrnHOefK\n/YLzPef3u3+/W1QVY4wxBiAk0AUYY4xpPywUjDHGNLBQMMYY08BCwRhjTAMLBWOMMQ0sFIwxxjTw\nayiIyFQR2SIi20Rk9hH2mSIia0Rkg4gs9Wc9xhhjjk78NU5BREKBrcC5QB6wArhaVTc22icRWAZM\nVdUcEemmqoV+KcgYY8wx+fOXwgRgm6ruUNV6YB4wvdk+1wCvqmoOgAWCMcYEVpgfXzsNyG20nAdM\nbLbPICBcRD4C4oG/qupzR3vRlJQUzcjIaMUyjTGm81u5cmWxqqYeaz9/hkJLhAHjgLOBaOAzEVmu\nqlsb7yQitwG3AfTp04fs7Ow2L9QYYzoyEdndkv382XyUD/RutJzuW9dYHvCuqlapajHwMTC6+Qup\n6pOqmqWqWampxww6Y4wxJ8ifobACGCgimSISAcwEFjXb53XgdBEJE5EYnOalTX6syRhjzFH4rflI\nVd0icjfwLhAKzFHVDSJyh2/746q6SUTeAdYCXuBpVV3vr5qMMcYcnd9OSfWXrKwsbd6n4HK5yMvL\no7a2NkBVtZ2oqCjS09MJDw8PdCnGmA5ERFaqatax9gt0R3OryMvLIz4+noyMDEQk0OX4japSUlJC\nXl4emZmZgS7HGNMJdYppLmpra0lOTu7UgQAgIiQnJwfFLyJjTGB0ilAAOn0gHBQsf6cxJjA6RfOR\nMcZ0RqrK3rJaNheUs7mgglFpiZw+MMWv72mh0ApKS0t54YUXuPPOO4/redOmTeOFF14gMTHRT5UZ\nY9qCqlJe46awopbCijoKK2pxuZW4qDBiI8OIO3iLCiMuIgwE6twe6lxeal0eal1eat0eal0ecvZX\ns3lvBVsKKthUUE5Frbvhfe6Y3N9CoSMoLS3lscceOyQU3G43YWFHPsSLFy/2d2nGmFZQ7/ayp7SG\n3APV5O4/eF/NntIaXwjUUe/2ttr7xUeGMbhHPNPH9GJwjwSG9ohnUI94EqL8f9ahhUIrmD17Ntu3\nb2fMmDGEh4cTFRVFUlISmzdvZuvWrVxyySXk5uZSW1vLrFmzuO222wDIyMggOzubyspKLrjgAk4/\n/XSWLVtGWloar7/+OtHR0QH+y4zpOKrr3eTsr8brhZAQEIQQAZGv++Lq3V7q3F7qXB7q3N6G5ep6\nN2U1roZbabWL0hoXZdX1FFXUUVBei7fR2fvhoUJaYjS9EqMZn9GVbvGRpPpu3eKj6JYQSWRYCFV1\nHirrXFTWeaisdVNV56aizo2qEhUeSlR4KJFhIb7Hzn3PLlGkJUYHrP+w04XCr9/YwMY95a36msN6\nJfCrbw0/4vaHHnqI9evXs2bNGj766CMuvPBC1q9f33Da6Jw5c+jatSs1NTWMHz+eyy+/nOTk5Cav\n8dVXX/Hiiy/y1FNPceWVV7JgwQKuu+66Vv07jOkMVJX80ho27a1g095yNheUs2lvBbtKqjjZYVci\n0CU6nMTocLrERJAYE0H/1DjSu8bQOyma3l1j6NM1hu4JUYSGdM6TPjpdKLQHEyZMaDKO4G9/+xsL\nFy4EIDc3l6+++uqQUMjMzGTMmDEAjBs3jl27drVZvca0d6rKmtxSXs7O4+31eymtdjVsy0iOYUiP\nBKaP6cWAbnGEhYSgqijgVcWrzvMBIsNCiAwLJSIshMiwEN99KNHhoXSJCSc+MoyQTvph31KdLhSO\n9o2+rcTGxjY8/uijj3j//ff57LPPiImJYcqUKYcdZxAZGdnwODQ0lJqamjap1Zj2rKiijoWr83g5\nO49thZVEh4cydUQPsjKSGNozgcHd44mN7HQfYwFlR7MVxMfHU1FRcdhtZWVlJCUlERMTw+bNm1m+\nfHkbV2dMx7NkcyFzP89hyZZCPF7llD6JPHTZSC4c1ZP4NuhsDWYWCq0gOTmZ0047jREjRhAdHU33\n7t0btk2dOpXHH3+coUOHMnjwYCZNmhTASo1p/174PIefLVxHSlwkt56RyYxx6QzoFh/osoJGp5gQ\nb9OmTQwdOjRAFbW9YPt7TfDI3rWfq59azjf6p/DPG7IID+00ky4EXEsnxLMjboxpFwrKarnj+VX0\nSozm7zPHWiAEiDUfGWMCrtbl4fZ/Z1NT7+aF70ykS4z1GwSKhYIxJqBUlZ8vXM+XeWU8cf04BnW3\n/oNAst9nxpiAembZLhasymPW2QM5f3iPQJcT9CwUjDEBs2x7MQ+8tYlzh3Vn1tkDA12OwULBGBMg\nufuruWvuKjJTYnn4ytFBP5K4vbA+hQCIi4ujsrIy0GUYExA5JdV8tLWQZ5btwuNVnvp2lg1Ia0cs\nFIwxflXr8vD5zv18tKWQpVuK2FFcBUDf5Bj+cd04MlNij/EKpi1ZKLSC2bNn07t3b+666y4A7r//\nfsLCwliyZAkHDhzA5XLxwAMPMH369ABXasyJK6txkVNSza6SKnL2V7OruIrd+6vJKammzu0hwjfB\nXERoCOGhzoRzISHCpr3l1Lq8RIaFMKlfMtd/oy9TBnezMGinOl8ovD0bCta17mv2GAkXPHTEzVdd\ndRX33XdfQyi8/PLLvPvuu9x7770kJCRQXFzMpEmTuPjii+0ay6bDqKh18d9tJSzdWsTHW4vIL206\nSWO3+EgykmM5fWAK0eGh1Lu9uDxe6jzehscuj5eZ4/sweXAq3+iXTFR4aID+GqC2HOrKob4K6iqh\n/uCtCrxuiEmBuFSI9d3Cvp6kEo8bynKgZDuUbPPdtkPNAYiIdW7hMU0fh0dDaITzOg33kRAWAeGx\nEBl/6C3U14zmcUFdxdf1Hay3Szqk+LdDvvOFQgCMHTuWwsJC9uzZQ1FREUlJSfTo0YPvfe97fPzx\nx4SEhJCfn8++ffvo0cNOuTPtk6qyaW8FS7cW8dGWQlbuPoDbq8RFhnHagGS+/Y2+9E2OJSPFuaZA\nTEQ7/fhQhfI9sPdL2LvGud+zBioLju91IrtAbApICBzYBV5Xo20JkDzA2e6qgcpC58PbVf31vaf+\n+GsPiwb1HPm5p90H5/76+F/3eErw66sHwlG+0fvTjBkzmD9/PgUFBVx11VXMnTuXoqIiVq5cSXh4\nOBkZGYedMtuYtuT1KvsqatlVXM3uEqf5Z3dJVcNyVb0HgKE9E7j1jH5MGZzKuL5JgZtyQhVKc6Bg\nLexd69wXbwWk2Tdw373XDfs2QFWR83wJgZTB0G8KdBsK0YkQEed8K4+I/fqxCFTvdz7cq4qgqhCq\nip1lrxuGXgRd+ztBcDAMjvWr3+v7cHfXHXrvqnZ+tdRVNLuVg4T66opz7iN8vyoi4iCxt58PeGcM\nhQC56qqr+M53vkNxcTFLly7l5Zdfplu3boSHh7NkyRJ2794d6BJNEKl1edhZXMX2okq2F/ruiyrZ\nUVRFjcvTsF94qNA7KYa+yTFMyOzKsJ4JTB6cSveEqMAV76qBTx6GnM+cpuDaUme9hEDKIOg52vng\n9NSBu/7r++r9zn4Dz4OeY5z9eoxwPlRbomu/1v07QkIhJNppRupALBRayfDhw6moqCAtLY2ePXty\n7bXX8q1vfYuRI0eSlZXFkCFDAl2i6eQOVNXz1rq9vLY6n5U5BxouTSkCaYnR9E+NY2JmMpmpsWQm\nx9I3OYaeXaIIa08Tz9VXw7xrYMdH0GsMDJsOPUc5H/LdhkFETKAr7PQsFFrRunVfd3CnpKTw2Wef\nHXY/G6NgWkuty8MHmwpZuDqfpVsLcXmUgd3iuPvMAQzqHk//1DgyU2KJjghgB29L1VfDizNh58dw\nyWMw5ppAVxSULBSM6WA8XuWz7SW8viafd9YXUFHnpntCJDeemsElY9MY1jOh453lVl8FL1wFu/8L\nlz4Oo2cGuqKg5ddQEJGpwF+BUOBpVX2o2fYpwOvATt+qV1X1N/6syZiOSFVZlXOARWv28Na6Aoor\n64iLDGPqiB5cOjaNSf2SCe2o00TUVTqBkLMMLn0CRl0Z6IqCmt9CQURCgUeBc4E8YIWILFLVjc12\n/URVLzrZ91PVjvft6AR0tCvlmROnqmzcW86iL/fw5pd7yS+tISIshLOHdOPi0b04c0i3wJ733xrq\nKmHuDMhdDpc9BSOvCHRFQc+fvxQmANtUdQeAiMwDpgPNQ+GkRUVFUVJSQnJycqcOBlWlpKSEqKgA\nnhli2sSWggoeeGsjn3xVTFiIcPrAFH5w3iDOHda988wTVFcBz18BeSvg8qdhxOWBrsjg31BIA3Ib\nLecBEw+z36kishbIB36oqhuO943S09PJy8ujqKjoxCrtQKKiokhPTw90GcZPSirrePi9rbz4RQ5x\nkWH8bNoQrhjXm66xEYEpSBUqCmDfeijLc0bwHu7m9TjjBBqP2j14jwAK6nVeT9VZPjgq+Ip/wvBL\nA/P3mUMEuqN5FdBHVStFZBrwGnDIGG4RuQ24DaBPnz6HvEh4eDiZmZl+LtUY/6lze3jmv7t45MNt\nVLs8XD+pL/edM4ikI4VB+V6nUzY8ptlUCQnOPQpl+VCWC+X5zgd6WZ7z2F0P8T0goSfE+24JvZx1\n9dVOABSsdcYIFKyH6uKm7x0WBdFdITrJuXXt55yT7677eoBWbfnX4wdQZ4wB4tyLOI8j4uDK55yB\nYabd8Gco5AONh9+l+9Y1UNXyRo8Xi8hjIpKiqsXN9nsSeBIgKyvLGtVNp6GqvLuhgAcXbyZnfzVn\nDk7l5xcOZUC3I1ySsno/fPowfPEUuI9zhHxcd0hIc77N52fDpr3OB/fhhEY6I4AHT4Ueo5z5vxL7\nQkzXDjcYyxwff4bCCmCgiGTihMFMoMmJxyLSA9inqioiE3Au+lPix5qMaVee/mQnv1u8iUHd43ju\n5gl8c1Dq4Xesq4Dl/4Blf3cej54JE28H5NBpEuoqAHUCoEu6c5/Qq+kEb+A049QcgIq9zq18r7NP\n9xHOpGuhnaTvwhwXv4WCqrpF5G7gXZxTUueo6gYRucO3/XHgCuC7IuIGaoCZaqfXmCDx+Y4SHnpn\nM1OH9+CRa8YefmSxqxay58An/+c04wy5CM76hfMt/mSJON/8Y7pC9+En/3qmU5CO9hmclZWl2dnZ\ngS7DmJNSWF7LhX//lMQI5fWLIMZ1AGpKobbMmeunptS537Pa6QfoNwXO+h9IHxfo0k0HJSIrVTXr\nWPsFuqPZmKDj9ni554Vszqr7kN9GvUHESzlNdwiPhaguzoye3YbBJf+AfpMDU6wJOhYKxrQlVV6b\n9yS/2fN3BofkQewomPZ7SB0MUYlOGIQF6PRTY7BQMKbt7PiI0jd+yRUH1lIU3QcufgaGToeQdjRL\nqQl6FgrG+Ft9Fbx0HWz/kBpN5rn4Wdx+788hIvLYzzWmjVkoGONvX/0Htn/Is1HX8WjdBbx6y1lE\nWiCYdspCwRg/0/zVeCScB8vO5YkbJ5CeZBeKMe2XhYIxflaxcwU7Pb25/cyhTBncLdDlGHNU1sNl\njD95vUQWrWWdN5MbTs0IdDXGHJOFgjH+dGAnke5K8mKGkBxn/Qim/bNQMMaf9qwGQHuOCXAhxrSM\n9SkY40fVu1YQouF06zc60KUY0yIWCsb4UV3OSnZqX0b3PcLsp8a0M9Z8ZIy/eD3ElmxgvfZjeK8u\nga7GmBaxUDDGX0q2EeGtpih+ONERoYGuxpgWsVAwxk+8+asACO09NsCVGNNy1qdgjJ9U7FhBmEaS\n1t86mU3HYaFgjJ+4c1eyWTMZ1Sc50KUY02LWfGSMP3jcxJduYrP0Z0C3uEBXY0yLWSgY4w9Fm4nQ\nOsqTRhAaIoGuxpgWs1Awxg9ceSsBiOhj11Q2HYv1KRjjB2XbviBCo+kzYESgSzHmuNgvBWP8Yc9q\n1nszGdO3a6ArMea4WCgY09rc9SRWbGV7+EB6JEQFuhpjjouFgjGtrXAjYeqiOmUUItbJbDoWCwVj\nWln1rmwAYjKyAlyJMcfPOpqNaWVlO76gXmPpN3B4oEsx5rjZLwVjWlno3jWs9fZjZO/EQJdizHGz\nUDCmNblq6Vq1jbzowSREhQe6GmOOm4WCMa1I960nDA/13WwSPNMx+TUURGSqiGwRkW0iMvso+40X\nEbeIXOHPeozxt9JtXwAQ3298gCsx5sT4LRREJBR4FLgAGAZcLSLDjrDfH4D/+KsWY9pK5c4VFGsC\nAwcOCXQpxpwQf/5SmABsU9UdqloPzAOmH2a/e4AFQKEfazGmTUQWfskG7ceQnnb5TdMx+TMU0oDc\nRst5vnUNRCQNuBT4hx/rMKZt1FeTUrOTfXFDiQiz7jrTMQX6X+5fgJ+oqvdoO4nIbSKSLSLZRUVF\nbVSaMcfHvedLQvDi6TEm0KUYc8L8OXgtH+jdaDndt66xLGCebyqAFGCaiLhV9bXGO6nqk8CTAFlZ\nWeq3io05CcVbP6cHkDRgYqBLMeaE+TMUVgADRSQTJwxmAtc03kFVMw8+FpFngDebB4IxHUXN7mz2\naSJDBg0KdCnGnDC/NR+pqhu4G3gX2AS8rKobROQOEbnDX+9rzElbPRf2fnncT4stXssm6U/f5Bg/\nFGVM2/Dr3EequhhY3Gzd40fY90Z/1mJMi2xcBK/fCaERcP6DMP5WaMlMp/t3kFKXQ0mXyTYzqunQ\nAt3RbEz7UVMKi38E3UdCvzNh8Q/hlRuhtvyIT6mtrWXPWw/hfuQb1GgEVZlT265eY/zAZkk1HZu7\nDhAIizj513rvl1BVCNfMgx6jYdnf4IPfQMFa6i/7F3mRA9i9v5otBRVs3FMOuV/w3cpHGBqSw388\n4/hj6C38cdxpJ1+HMQFkoWA6Lo8L/jUNSnPg/N/ByBkta+rx8XqVijo35TUu3NuXkrnqOXYMuoXs\n/BTyN2wj98AUYpNiuWf/g3R56hyecn+bFz1nkUA198cu4BLPu1RFpbJq3CMMybqcd5OiCQmxpiPT\nsYlqxzrDMysrS7OzswNdhmkPlv4RljwAyQOgZBtknAHT/gTdvp5ios7tIXd/NduLqthZXMWOokp2\nFjuPS6rqUYVI6nkn4icIMLX+IWqJRAR6JkSR3jWGIfH13FL4IH1Ll3Og99l02b+OkOpimHgHnPkz\niIwP3DEwpoVEZKWqHvPKT/ZLwXRMBetg6R9gxOVw2VOw6ll4/9fw+Gm4J97F612u5bnsItbll+Ft\n9L0nJS6SfqmxnD2kO927RJEQFcZpux4hc/s+NpzzPAv6fZMu0eGkxkcSGRb69RO9Z8An/0fSRw9C\nj5Fw7cvQa2zb/93G+Jn9UjAdj7senj4LKvbBXZ9DTFcA9uTnUPzaTxlV9Cb5mszTsbcRN/oS+neL\nJzMllszU2EOvcbBnDTx1Foy5BqY/cuz3riiA2FQICT32vsa0I/ZLwXRen/yf80th5gtodBKfbSvm\nmWW7eH/TPuAavpt5HnfWPMavDvwe9nwCmbMg/dxD+xs8blh0D8SmwHm/bdl7x/do9T/HmPakRaEg\nIq8C/wTePtY8RcYckdcLRZvAXQteD3jdzs3jcpa7pEF357rGlXVudhVXsbukml0lVewrr6W8xkVi\n6UZ+UfBHPgqfws9fjaSs5h1qXV66xkZwx+T+XDupL2mJ0eC5FrL/CZ/+BV6YAalD4NR7nM7osEin\nns8egYK1cOVzEJ0UwANjTPvRouYjETkHuAmYBLwC/EtVt/i5tsOy5qMOShUW3ALrFxx1t/9ET+N3\n9Vezu6pp80xiTDgpUfBU3Y9I1HIe6DuHsNhkEqLDGNozgWkjexIVfpgmHXc9bFgIy/4O+9ZBXA+Y\neDv0m+ycuTTgHLjq+eM6a8mYjqilzUfH1acgIl2Aq4Gf40yL/RTwvKq6TrTQ42Wh0EEdPFPo1Hsh\n43Q8hLJpXxVLtx1g2c5SqlzCFVEruEbfojwsmf8O/hky5AIykmPpmxxDbGQYfPBb+ORPcPVLMPg4\nB4mpwo4lTjhs/9BZF5kAd30BCT1b/+81pp1p9T4FEUkGrgOuB1YDc4HTgRuAKSdWpmmXXLWw8TXY\n8rZzumfvidB7/Ik3sWxc5ATCqJlsG/1jXl2dz8LV+ewtCyM+sicXjj6Fy05JZ3zGvUj+KhIX3c2F\nG74H8ilM/YPz4Z2/Cj79M4y+5vgDAZxfAv3Pcm4F652mpQHnWCAY00xLm48WAoOBfwPPqOreRtuy\nW5I+rcV+KfhRWR5kz4GVz0J1McR1h6piUI+zPXUo9JkIvSdB31Mhqe8RX0pVyd1fQ87G5UxccjU5\n4ZncrL9id7mX0BDhmwNTuOyUdM4d1v3QZh93vRMAH//RGQNw3gPO6OLaMrhzOUQn+vEgGNM5tWrz\nkYicqapLWqWyk2Sh0MpUYdcn8MWTsPktZ92gC2DCd6DfFHBVQ/5KyPkccpdD7gqoK3P2G3cjnHM/\nRCfh9nhZk1vKx1uLWL5zP5v2lBNZV8Lrkb8gBOW+hD/TM60vY3onMm1UT7rFRx27tsJNztlBeSuc\n5Wvnw8BzW/0QGBMMWjsU7gLmqmqpbzkJuFpVHzvpSo+ThUIrOrAb5l0D+9ZDdFcYdwNk3QyJfY78\nHK8XijbDmrno8n9QG96FuYnf5a/7RlFR5yFEYGRaF8b0iubunO+TXLEZ1w1vE9nnlBOr0euBlf9y\nTh+dZDOuG3OiWjsU1qjqmGbrVqtqmw/ptFBoJVXFMOd8qCqC83/vjAwOP/a3990lVSxYlc+ba/cQ\nXbyB34U/zZiQHWyNG0/+qb9j7JixJEaHw+t3w5rnYcYzMPxS//89xpijau2O5lAREfUliIiEAq0w\nLaUJiLoKmHuF04fw7dehz6Sj7l5R62Lxur0sWJnPF7v2IwLf6JfM2ROnETfgWnT3Swz64LcM+vBi\ncP/IGe275nmY/BMLBGM6mJaGwjvASyLyhG/5dt8609G46+Gl62HvWpg594iB4PUqy7aXMH9lLu9s\nKKDW5aVfSiw/On8wl45No1di9Nc797gdhn4L3v4JfOgbGTz0Ypg8uw3+IGNMa2ppKPwEJwi+61t+\nD3jaLxUZ//F64bXvOufrT38UBl9wyC6VdW4WrMzj2WW72FFcRXxUGJefks7l49IZ2zvxyFcVS+gF\nV/0btrzjjAM451cQYtdwMqajsQnxgoUqvDMbPn/cOWPo9O812byruIrnPtvNK9m5VNS5Gd07kZtO\nzWDqiB6HHylsjOlQWrVPQUQGAr8HhgENvZGq2u+EKzRt69OHnUCYdCecdh/gjCX4dFsxz/x3Fx9u\nKSRUhAtH9eTGUzMY28fmAjImGLW0+ehfwK+APwNn4syDZG0DHcWq55zLSo6cAef9jhqXl4Wr8/nX\nf3fyVWElKXER3HPWQK6d2IfuCS0YP2CM6bRaGgrRqvqB7wyk3cD9IrIS+B8/1mZOltfjXIhm6f9C\n/7PZe+b/8dx/tvLiFzmUVrsY3iuBP80YzbdG92x6QRljTNBqaSjUiUgI8JWI3A3kA3H+K8uctPK9\nsOBW2P0pJQOv4AG9hUV/+i+qynnDenDz6ZmMz0g6csexMSYotTQUZgExwL3Ab3GakG7wV1HmJG17\nH169DW99NU93/TEPrhtDfFQFN5+Wwbe/kUHvrjGBrtAY004dMxR8A9WuUtUfApU4/QmmPfK4ndlI\nP/0zeRGZ3FA1m/3eDGZf0J/rJ/V1pp82xpijOOanhKp6ROT0tijGnISyPGrm3Uj03hW84DmLh+tu\n5tvnDOGm0zKIb35dYmOMOYKWfnVcLSKLcK66VnVwpaq+6peqzNG566FwI+xZDXtW48lfjRZuxOMN\n4wd6Lz1Ov5b3z+hHYozNRGKMOT4tDYUooAQ4q9E6BSwU2oLX60xbvXER5H7uzGrqqQfAE5nIancG\nX7im4RlzPT+dOpmUuMgAF2yM6ahaFAqqav0Ibe1gEGx4DTYtgoq9EBoJvSfApO9Cr7F8VJHOXYuL\niQoP4+83juXUASmBrtoY08G1dETzv3B+GTShqjcf43lTgb8CocDTqvpQs+3Tcc5m8gJu4D5V/bRl\npXdS+zbCymeaBsHAc53ZRgedD5HxuD1e/vjuFp74eAdjeify2LWnNJ2gzhhjTlBLm4/ebPQ4CrgU\n2HO0J/jOWnoUOBfIA1aIyCJV3dhotw+ARaqqIjIKeBkY0tLiO529X8KcC8DrPiQIDiqurOOeF1bz\n2Y4SrpvUh19eNMwGnhljWk1Lm48WNF4WkReBY32jnwBsU9UdvufMA6YDDaGgqpWN9o/lML9GgkZZ\nHsy9EqKT4Nb3nFlHm1mVc4A7n1/Fgep6/jRjNFeMSw9AocaYzuxET1wfCHQ7xj5pQG6j5TxgYvOd\nRORSnMn2ugEXnmA9HVttuRMIrmq4+Z1DAqHW5eFvH3zFEx/voFdiFK/eeSrDe3UJULHGmM6spX0K\nFTT9Fl+Ac42Fk6aqC4GFIvJNnP6Fcw7z/rcBtwH06XOU6wd3RB4XvHIDFG9xLkzffXiTzatyDvDj\n+WvZVljJjHHp/OKiYXSJtnEHxhj/aGnzUfyx9zpEPtC70XK6b92R3uNjEeknIimqWtxs25PAk+Bc\nT+EEammfVOGt7zsXpbn4Eeh/ZsOmmnoPD7+3hX9+upMeCVE8e/MEJg9KDWCxxphg0NJfCpcCH6pq\nmW85EZiiqq8d5WkrgIEikokTBjOBa5q97gBgu6+j+RQgEmc8RHD49M/OtNZn/BBOub5h9Rc79/Pj\n+V+yq6Saayf2YfYFQ2xUsjGmTbS0T+FXvmYeAFS1VER+BRwxFFTV7ZtR9V2cU1LnqOoGEbnDt/1x\n4HLg2yLiAmpw5ljqPL8EjmbdfPjg1841Ds76BeBc9OahdzbzxNId9O4azQu3TrSxB8aYNtXSUDjc\nBXVaMm/SYmBxs3WPN3r8B+APLayh89j5Cbx2J/Q51blWsm/66v99dwtPLN3B1RN688uLhhETYRPY\nGWPaVks+/1CWAAAQzklEQVQ/dbJF5GGccQcAdwEr/VNSJ1ayHZY8COvnQ/IAmDkXwpwpKZ5Yup1/\nfLSdayb24XeXjLDrHBhjAqKloXAP8EvgJZyzkN7DCQbTEmX58PH/wqp/OyFw+vfhtFkQnQjAvC9y\n+P3bm7loVE9+O90CwRgTOC09+6gKmO3nWjqfqhL49GH44ilQL4y/xelUju/esMvidXv52cJ1TB6U\nysNXjiE0xALBGBM4LT376D1ghqqW+paTgHmqer4/i+vQVj8Pb88GVxWMmglTfgJJGU12+eSrImbN\nW83YPkn847pTiAg7XNeNMca0nZY2H6UcDAQAVT0gIsca0Ry8vnoPFt0DfU+DaX+CbodO57Qq5wC3\n/3sl/VPjmHPDeOtUNsa0Cy39JPKKSB9VzQEQkQyCeZ6ioyncBK/c5IxMvuYliIg9ZJctBRXc9K8V\npMZH8twtE+gSY2MQjDHtQ0tD4efApyKyFBDgDHzTTphGqorhhasgIgauPnwgfJlbyk3PrCAqPITn\nb5lIt/ioABRqjDGH16JGbFV9B8gCtgAvAj/AGWxmDnLXwUvXQeU+mPkidEk7ZJePtxZx9VPLiY0M\nZd5t36B315gAFGqMMUfW0o7mW4FZOPMXrQEmAZ/R9PKcwUsV3rgPcj6DK+ZA+rhDdnl9TT4/ePlL\nBnaP59mbxtMtwX4hGGPan5ae7jILGA/sVtUzgbFA6dGfEkT++1f48gWYPBtGXH7I5jmf7mTWvDWM\n65vES7dPskAwxrRbLe1TqFXVWhFBRCJVdbOIDPZrZR3F5rfg/fth+GUwpelQDlXlf9/dwj8+2s7U\n4T34y8wxRIXbVdKMMe1XS0Mhzzcz6mvAeyJyANjtv7I6iIJ1sOA70GssXPJYwxxGAG6Pl9mvrmP+\nyjyumdiH304fYQPTjDHtXktHNF/qe3i/iCwBugDv+K2qjqC2HF66HqIS4OoXITy6YZOq8qP5a1m4\nOp9ZZw/kvnMG2tQVxpgO4bhHTKnqUn8U0qEcvDhO6W648S2I79Fk87wVuSxcnc/3zhnErHMGBqhI\nY4w5fjavwolYMxfWvQJTfgZ9T22yaXNBOfcv2sAZA1O456wBASrQGGNOjIXC8SraAot/BJnfhDO+\n32RTVZ2bu+auIiE6nIevHEOI9SEYYzoYm3DneLhqnCkswmPg0ichpOmZRP/z+gZ2FFcx95aJpMZH\nBqhIY4w5cRYKx+Pdn0HhBrh2AST0bLJp/so8FqzKY9bZA+0SmsaYDsuaj1pqw0LIngOn3gsDz2my\naVthBb98bT2T+nXl3rOtY9kY03FZKLTEgV2waBakjYOzftlkU63Lw11zVxMTEcpfZ461sQjGmA7N\nmo+OxeOC+bcA6sxrFBbRZPOv39jAln0VPHfzBLrb9BXGmA7OQuFYPn8c8rNhxjOHXDntzbV7ePGL\nXO6c0p9vDkoNSHnGGNOarPnoWNa8CL0nwvBLm6yudXl48K1NjEzrwvfPHRSg4owxpnVZKBxN4Sbn\nbKMRVxyy6fnlu9lTVstPpw0hLNQOozGmc7BPs6NZvwAkBIZf0mR1Ra2LR5ds44yBKZza304/NcZ0\nHhYKR6IK6+Y7I5fjujXZ9NQnOzlQ7eLH5w8JUHHGGOMfFgpHsmc1HNh5yEVziivrePqTHVw4sicj\n07sEqDhjjPEPC4UjWb8AQsJh6LearH50yTbq3F6+f551LhtjOh8LhcPxemH9qzDgHIhOalidu7+a\nuctzuDIrnf6pcQEs0Bhj/MOvoSAiU0Vki4hsE5HZh9l+rYisFZF1IrJMREb7s54Wy/kMKvbAyKZn\nHf3l/a9AsKksjDGdlt9CQURCgUeBC4BhwNUiMqzZbjuByao6Evgt8KS/6jku6+c7M6EOvqBh1ZaC\nCl5dnceNp2bQs0v0UZ5sjDEdlz9/KUwAtqnqDlWtB+YB0xvvoKrLVPWAb3E5kO7HelrG44KNr8Og\nqRAR27D6T//ZQlxEGN+d3D+AxRljjH/5MxTSgNxGy3m+dUdyC/D24TaIyG0iki0i2UVFRa1Y4mHs\nWArVJU2ajlblHOC9jfu4fXI/kmIjjvJkY4zp2NpFR7OInIkTCj853HZVfVJVs1Q1KzXVz3MMrV8A\nkV2cTmbnvfnD25tJiYvkptMy/fvexhgTYP4MhXygd6PldN+6JkRkFPA0MF1VS/xYz7G5amHzm85p\nqGHOldM++aqYz3fu556zBhAbafMHGmM6N3+GwgpgoIhkikgEMBNY1HgHEekDvApcr6pb/VhLy2x7\nD+rKYcRlDaueWbaL7gmRXD2hTwALM8aYtuG3r76q6haRu4F3gVBgjqpuEJE7fNsfB/4HSAYeExEA\nt6pm+aumY1o3H2JTIXMyAPvKa/loSyF3TO5PRFi7aGkzxhi/8mt7iKouBhY3W/d4o8e3Arf6s4YW\nq6uAre/C2Osg1Dksr67Kx6twxbjAnxRljDFtwb7+HrTlbXDXNMx1pKq8kp3L+Iwk+tnoZWNMkLBQ\nOGj9AkhIdy6og3Ma6o7iKmaM632MJxpjTOdhoQBQvR+2fQAjLoUQ55C8kp1HdHgo00b1DHBxxhjT\ndiwUwDkN1etqaDqqrnfzxpd7uHBUT+LsNFRjTBCxUADYuAiSMqDnGADeXldAVb2HGdbBbIwJMhYK\ntWWwcykMuQic02J5ZWUufZNjmJDZNcDFGWNM27JQ+Oo98NTD0IsByCmpZvmO/cwYl45v7IQxxgQN\nC4VNiyCuO6SPB2D+ylxE4LJTrOnIGBN8gjsUXDXOL4UhF0JICB6vMn9lHmcMTKVXol0zwRgTfII7\nFLYvAVd1w3WYl20vZk9ZrXUwG2OCVnCHwqY3IKoLZJwBOGMTEqLCOHdY9wAXZowxgRG8oeBxwZbF\nMHgahIZTVu3inQ0FXDI2jajw0EBXZ4wxARG8obDrU6gtdU5FBRat3UO922vTWhhjglrwhsLmNyE8\nBvqfBcD87FyG9IhnRFpCgAszxpjACc5Q8Hph05vOJTcjYthSUMGXeWXMyOptYxOMMUEtOEMhPxsq\nCxrOOnp/0z4Apo/pFciqjDEm4IIzFDYtgpBwGHgeAOvzy+ibHENKXGSACzPGmMAKvlBQdZqO+k2G\n6EQA1uWXMSKtS4ALM8aYwAu+UNi3AQ7sbGg6OlBVT96BGkZaKBhjTBCGwqY3AHHGJ+D8SgAYZaFg\njDFBGAqb34Q+34C4bsDXoTDcQsEYY4IsFEq2w771DU1H8HUnc5fo8AAWZowx7UNwhcLmN537IRc2\nrLJOZmOM+VpwhcKmN6DnaEjqC1gnszHGNBc8oVC+F/JWNGk6OtifYKFgjDGO4AmF7R86977LbsLX\noTCil4WCMcYAhAW6gDYz5hpIz4LUwQ2rGjqZY6yT2RhjIJh+KYg0CQSwTmZjjGkueEKhGetkNsaY\nQ/k1FERkqohsEZFtIjL7MNuHiMhnIlInIj/0Zy3Nrd9jnczGGNOc3/oURCQUeBQ4F8gDVojIIlXd\n2Gi3/cC9wCX+quNI1uZZJ7MxxjTnz18KE4BtqrpDVeuBecD0xjuoaqGqrgBcfqzjsNbnl9Gnq3Uy\nG2NMY/4MhTQgt9Fynm/dcROR20QkW0Syi4qKWqW4dfll1nRkjDHNdIiOZlV9UlWzVDUrNTX1pF+v\noZM53ULBGGMa82co5AO9Gy2n+9YFnHUyG2PM4fkzFFYAA0UkU0QigJnAIj++X4vZSGZjjDk8v519\npKpuEbkbeBcIBeao6gYRucO3/XER6QFkAwmAV0TuA4aparm/6gLrZDbGmCPx6zQXqroYWNxs3eON\nHhfgNCu1qbV5ZYxOT2zrtzXGmHavQ3Q0t6aDncw2vYUxxhwq6ELBOpmNMebIgi4U7BoKxhhzZEEX\nCtbJbIwxRxZ0oWAjmY0x5siCKhQOVNWTu986mY0x5kiCKhSsk9kYY44uqEKhYSRzWkKAKzHGmPYp\nqEJhfX4ZvbtGkxgTEehSjDGmXQqqULBOZmOMObqgCYXSaqeTeWSaTW9hjDFHEjShYIPWjDHm2IIm\nFKLCQzlnaDfrZDbGmKPw6yyp7cn4jK6Mz+ga6DKMMaZdC5pfCsYYY47NQsEYY0wDCwVjjDENLBSM\nMcY0sFAwxhjTwELBGGNMAwsFY4wxDSwUjDHGNBBVDXQNx0VEioDdJ/j0FKC4FcvpLOy4HMqOyaHs\nmByqIx2TvqqaeqydOlwonAwRyVbVrEDX0d7YcTmUHZND2TE5VGc8JtZ8ZIwxpoGFgjHGmAbBFgpP\nBrqAdsqOy6HsmBzKjsmhOt0xCao+BWOMMUcXbL8UjDHGHEXQhIKITBWRLSKyTURmB7qeQBCROSJS\nKCLrG63rKiLvichXvvukQNbY1kSkt4gsEZGNIrJBRGb51gftcRGRKBH5QkS+9B2TX/vWB+0xOUhE\nQkVktYi86VvudMckKEJBREKBR4ELgGHA1SIyLLBVBcQzwNRm62YDH6jqQOAD33IwcQM/UNVhwCTg\nLt+/jWA+LnXAWao6GhgDTBWRSQT3MTloFrCp0XKnOyZBEQrABGCbqu5Q1XpgHjA9wDW1OVX9GNjf\nbPV04Fnf42eBS9q0qABT1b2qusr3uALnP3waQXxc1FHpWwz33ZQgPiYAIpIOXAg83Wh1pzsmwRIK\naUBuo+U83zoD3VV1r+9xAdA9kMUEkohkAGOBzwny4+JrJlkDFALvqWrQHxPgL8CPAW+jdZ3umARL\nKJgWUOdUtKA8HU1E4oAFwH2qWt54WzAeF1X1qOoYIB2YICIjmm0PqmMiIhcBhaq68kj7dJZjEiyh\nkA/0brSc7ltnYJ+I9ATw3RcGuJ42JyLhOIEwV1Vf9a0O+uMCoKqlwBKcvqhgPianAReLyC6c5uez\nROR5OuExCZZQWAEMFJFMEYkAZgKLAlxTe7EIuMH3+Abg9QDW0uZERIB/AptU9eFGm4L2uIhIqogk\n+h5HA+cCmwniY6KqP1XVdFXNwPn8+FBVr6MTHpOgGbwmItNw2gRDgTmq+rsAl9TmRORFYArOzI77\ngF8BrwEvA31wZp+9UlWbd0Z3WiJyOvAJsI6v24p/htOvEJTHRURG4XSahuJ8cXxZVX8jIskE6TFp\nTESmAD9U1Ys64zEJmlAwxhhzbMHSfGSMMaYFLBSMMcY0sFAwxhjTwELBGGNMAwsFY4wxDSwUjGlD\nIjLl4AybxrRHFgrGGGMaWCgYcxgicp3vmgJrROQJ3wRxlSLyZ981Bj4QkVTfvmNEZLmIrBWRhQfn\n1BeRASLyvu+6BKtEpL/v5eNEZL6IbBaRub5R1ca0CxYKxjQjIkOBq4DTfJPCeYBrgVggW1WHA0tx\nRoQDPAf8RFVH4YyMPrh+LvCo77oEpwIHZ9McC9yHc22Pfjjz6hjTLoQFugBj2qGzgXHACt+X+Gic\nic68wEu+fZ4HXhWRLkCiqi71rX8WeEVE4oE0VV0IoKq1AL7X+0JV83zLa4AM4FP//1nGHJuFgjGH\nEuBZVf1pk5Uiv2y234nOEVPX6LEH+39o2hFrPjLmUB8AV4hIN2i4Dm9fnP8vV/j2uQb4VFXLgAMi\ncoZv/fXAUt9V3PJE5BLfa0SKSEyb/hXGnAD7hmJMM6q6UUR+AfxHREIAF3AXUIVzwZlf4DQnXeV7\nyg3A474P/R3ATb711wNPiMhvfK8xow3/DGNOiM2SakwLiUilqsYFug5j/Mmaj4wxxjSwXwrGGGMa\n2C8FY4wxDSwUjDHGNLBQMMYY08BCwRhjTAMLBWOMMQ0sFIwxxjT4f0St9J0fkBqAAAAAAElFTkSu\nQmCC\n", 350 | "text/plain": [ 351 | "" 352 | ] 353 | }, 354 | "metadata": {}, 355 | "output_type": "display_data" 356 | } 357 | ], 358 | "source": [ 359 | "plt.plot(epochs, [x[3] for x in all_losses], label='train');\n", 360 | "plt.plot(epochs, [x[4] for x in all_losses], label='val');\n", 361 | "plt.legend();\n", 362 | "plt.xlabel('epoch');\n", 363 | "plt.ylabel('accuracy');" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": 14, 369 | "metadata": {}, 370 | "outputs": [ 371 | { 372 | "data": { 373 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEKCAYAAAAW8vJGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcVNW16PHfqurqeZ4YuplpZRAEQcQZp4gahwwKXuPN\nYCQaTTR5NzcmMTFmNC/vxuiNxqAxaoKiiRPX4EU0gqKizMo8Q0/QAz3PXbXeH6cauroLKLCrq7tr\nfT+f+pxp1+lVR6lVZ++z9xZVxRhjjOnginQAxhhj+hZLDMYYYwJYYjDGGBPAEoMxxpgAlhiMMcYE\nsMRgjDEmgCUGY4wxASwxGGOMCWCJwRhjTICYSAdwMrKzs3XkyJGRDsMYY/qVNWvWVKhqzvHK9cvE\nMHLkSFavXh3pMIwxpl8RkX2hlLOqJGOMMQEsMRhjjAlgicEYY0yAftnGEExbWxtFRUU0NzdHOpSw\nio+PJz8/H4/HE+lQjDED1IBJDEVFRaSkpDBy5EhEJNLhhIWqUllZSVFREaNGjYp0OMaYAWrAVCU1\nNzeTlZU1YJMCgIiQlZU14O+KjDGRNWASAzCgk0KHaPiMxpjIGjBVScYY09t8PqW2uY2apjaqG/3L\nJmdZ09iKKiTEuon3uEnwuEmIdZbxHjdpCR5yUuLITIrF7Qr+g09VqWpso6iqkaKqJgoPNXL2mCwm\n56eH9XNZYugh1dXVPPvss3zzm988ofddeeWVPPvss6Snh/c/tDHm5NU0trGrop495Q3sqXBeu8rr\n2VvZQHOb71Od2yWQmRRHTorzyk6KpaapjaKqJoqqGmlo9QaU/+GV4/p/YhCR2cBDgBt4QlUf6HI8\nDfgbMNwfz/9T1b+EO66eVl1dzaOPPtotMbS3txMTc/TLvHjx4nCHZsyAVN/STnFVE8XVzq/p4qom\niqqbKK9rod3rw6vg9fnw+jqWik/xLxVfx7YqqorXpwFlvP4yzvEjf9ftEoZnJjIqO4lzx2YzND2B\n9AQPaQke0hOdZZp/6RKhuc1LU5uX5lYfTf71xtZ2ahrbKK9vobyuhQr/sryuhV1l9aTExzAsM5Fz\nxmaRn5FIfkYCwzISyctIIC0h/E8khjUxiIgbeAS4DCgCVonIIlXd3KnYHcBmVb1aRHKAbSKyQFVb\nwxlbT7vnnnvYtWsXU6ZMwePxEB8fT0ZGBlu3bmX79u1cd911FBYW0tzczF133cW8efOAI8N71NfX\nc8UVV3Deeefx/vvvk5eXx6uvvkpCQkKEP5kxkdXY2s72g/VsKa1la2ktW0rr2F5WR3VjW0C5WLeL\noenx5KbGkxgbg9slR14iuN2CSwS3gMvVsS64XOAS/3bHfpdTxu3flxIfw+jsZEblJDEsI5HYmNCb\nZz1uFynx/evx8nDfMcwAdqrqbgARWQhcC3RODAqkiNOqmgwcAto/zR+9/382sbmk9tOcopsJQ1O5\n7+qJRz3+wAMPsHHjRtavX8+yZcu46qqr2Lhx4+HHSp988kkyMzNpamrizDPP5Atf+AJZWVkB59ix\nYwfPPfccjz/+ODfccAMvvvgiX/rSl3r0cxjTH+ypaODBpdv5pLiGvZUNh3+xJ8W6GTcklStOG8Lw\nTOcXdF56AsMyEshOjsN1lLp6c2LCnRjygMJO20XAWV3K/AFYBJQAKcAcVf10lXZ9wIwZMwL6Gjz8\n8MO8/PLLABQWFrJjx45uiWHUqFFMmTIFgGnTprF3795ei9eYvuLldUXc+/JGXC7h3DHZXDtlKOOH\npDJ+cCr5GQn25d8L+kLj8+XAeuBiYAywVETeVdWAn/wiMg+YBzB8+PBjnvBYv+x7S1JS0uH1ZcuW\n8eabb/LBBx+QmJjIrFmzgvZFiIuLO7zudrtpamrqlViN6QsaWtr5yaubeHFtEWeOzOChuVMZmm5V\nqZEQ7n4MxcCwTtv5/n2dfRV4SR07gT3AuK4nUtX5qjpdVafn5Bx3OPFel5KSQl1dXdBjNTU1ZGRk\nkJiYyNatW1m5cmUvR2dM37a5pJar/7CCl9YV8e2Lx/LcrTMtKURQuO8YVgEFIjIKJyHMBf6tS5n9\nwCXAuyIyCDgV2B3muHpcVlYW5557LqeddhoJCQkMGjTo8LHZs2fz2GOPMX78eE499VRmzpwZwUiN\n6TtUlb+t3MfP/7mFtAQPC245i3PGZkc6rKgn2vk5rHD8AZErgd/jPK76pKr+UkRuA1DVx0RkKPAU\nMAQQ4AFV/duxzjl9+nTtOlHPli1bGD9+fBg+Qd8TTZ/VDBzNbV4O1jZzoKaZA7XNHKxtZuXuQ/xr\naxkXnpLDf91wOtnJccc/kTlpIrJGVacfr1zY2xhUdTGwuMu+xzqtlwCfCXccxpie4fMpB+ua2V/Z\nyL5DjeyvbKSwqpGGlnZavUq710eb13d4vbXdR0V9C1VdHi8FSImP4QdXjOPW80dbo3If0hcan40x\nfURTq5cDtc2HO1uV1zVT5l8vq2uhqKqRwqomWtuPPDjodglD0uJJjffgiXHhcQket4uE2CPrM0Zl\nMjg1nsFp/ldqPIPS4kmJi7Hxv/ogSwzGRJGO3sIdY+8UV3dar2qisqF7v9IYl5Cd7AzXUJCbwiXj\nBzE8M5ERWYkMz0xkaHoCHveAGo8z6lliMGaAafP6WF9YzabiGv94O00U+YeN6NZbOMZFvr+T2MSh\naeRnJDA4NZ7cVCcR5KbEk57giWw1j88LTVXQUA71Zc6yuQYS0iExG5JyICkbEjLBHRP4vuYa571N\n1c6yudrZ11IHLbXQXOssW+rA2wqeRIhNOrLsWBdxyhwu3/HeOvC2gcsF4gZXDLjc/nX/S1z+V8e6\nOPu9bdDaAG1N/lfjkZcrBmISwBPfaRkPngSY9hUYd1VYL7klBmP6OVVlT0UD7+6o4N0dFazcXUl9\nizN4QLzHdXisndPz0w+v52UkkJ+RQHZSH+wt7G2D9Qtg9V+gtgQaKyCkPq8CCRkQl+x8aTfX4Ays\ncLTibohPhbgUiEsDtwfaiqC1EdoanC/t9s79jcRf1v+e+FRIzAR3rJOE1Ossfe1OvO3NzlJ9/uM+\nUHXKqc/5e55EiE10Epsnwdn2JDjl25udhNF5WV/mxBdmlhiM6YeqGlp5b1cFK/zJoLja6Qw5PDOR\na6cM5fyCbM4YkUFOclz/qcP3tsPHz8Py30D1PhhyOoy70n9HkOt8eSbnOttxqc6v/4YK5w6isfLI\nems9xKc5SaLrKz79SDLouBM4Fp/X+QWvCrHJzp1BFLDEECHJycnU19dHOgzTTzS3eVmzr4p3d1Tw\n3s4KNpbUoOo81XPumGxunzWG8wuyGZGVdPyTnSifD/Z/ABtfdLZzx/tfE5xfzJ/6/F7n3MsegEO7\nnIRw5W+h4DPH/uJOHfLp//bxuNxOEokylhiM6aOKq5t4Y9MB/rW1jFV7D9Hc5iPGJZwxPIPvXHoK\n5xVkMzkvjZhwNfxW7oINC+HjhVC9HzxJTh1+c82RMsmDIGeckyQyR0HaMEjLd14JGcG/2Ntb/L/u\ny6BsK6x4ECq2waDTYM4Cp/68v9zlDFCWGHrIPffcw7Bhw7jjjjsA+OlPf0pMTAxvv/02VVVVtLW1\n8Ytf/IJrr702wpGavkpV2X6wniWbDvDG5gNsLHaGCxuTk8TcM4dzfkE2Z43OIjkuxqnaaK6B2v1H\nGkEDGlP9dexN/sbW5mr/un/pjj3yBZ4+3L8+DFKHQukG2PAcFH4ICIyeBRf/2PnC9iRCXSmUbXa+\n1Mu2OOtrn3aqXDqLTXbOmzrUqSNvKIf6cmipCSyXMw6ufxrGXxM1VTV9Xdh7PofDcXs+v34PHPik\nZ//o4ElwxQNHPbxu3Truvvtuli9fDsCECRNYsmQJaWlppKamUlFRwcyZM9mxYwci8qmqkqzn88BS\nXtfC4+/uZsmmA+yrdL5cL8h3cX1eFeckFpPVvA8aD0HTocCleo99YnesU6eekO7UuR9eTwdvC1QX\nQk2R82rvMmBjzjg4/UaYfIPzxX48qs5dQM1+53yHz13oNCDHJvnbCvyv5I7lYBg6xamyMWHXZ3o+\nR4upU6dSVlZGSUkJ5eXlZGRkMHjwYL7zne/wzjvv4HK5KC4u5uDBgwwePDjS4Zo+ovBQI7c98RZ5\nNev5dnYZM4YXMrRpB+6KIqjwF0oe5DyWmZgJOadCYpaznpDpb1BNPfK0THzakfWYuNCqZFSdxtsa\n/5d5Wj4MmXJi1Tkizpd9cg7kTTupa2H6joGZGI7xyz6crr/+ev7xj39w4MAB5syZw4IFCygvL2fN\nmjV4PB5GjhwZdLhtE2W8bVC0isqP/5eatYtZpDtxexRqBLLGwoiZMGSy0wg7eHLPNPAei4jzxE9S\nNgydGt6/ZfqFgZkYImTOnDnceuutVFRUsHz5cl544QVyc3PxeDy8/fbb7Nu3L9Ihmkjx+WDtU7Bj\nKex5F1rrSEcollOomnY32ZM/4ySBuORIR2qMJYaeNHHiROrq6sjLy2PIkCHcdNNNXH311UyaNInp\n06czbly3aSZMtNjwLLz2HUgfwYERV/OrbUPYkTSVP339ErKzEiMdnTEBLDH0sE8+OdLonZ2dzQcf\nfBC0nPVhiDI7lkLKEN64dAl3LlzPqKwknrllBoNS4yMdmTHd2LNhxoSbzwu7l7E3/Sxuf3YdE4ak\n8vw3ZlpSMH2W3TEYE24l66G5mv/alc/Zo7P4083TSIqzf3qm7xpQ/3eqav8ZF+Yk9cd+J1Fv11v4\nENZ7prD0y9OJ99gz+6ZvGzBVSfHx8VRWVg7oL05VpbKykvh4q4LoT3TnW2xlFNPGj7WkYPqFAXPH\nkJ+fT1FREeXl5ZEOJazi4+PJz8+PdBgmVM21ULSKt9uvYvZp1rHR9A8DJjF4PB5GjRoV6TCMCbT3\nXUS9fCin87VTciMdjTEhGTCJwZi+SHe+RRPxpBScR0KsVSOZ/mHAtDEY0xe1bHuT973j+czkYZEO\nxZiQhT0xiMhsEdkmIjtF5J4gx78nIuv9r40i4hWRMA8OY0wvOLSb+Lp9fMDpXDzOqpFM/xHWxCAi\nbuAR4ApgAnCjiEzoXEZVf6uqU1R1CvADYLmqHgpnXMb0Bt31NgCNwy4gJd4T4WiMCV247xhmADtV\ndbeqtgILgWPNVHMj8FyYYzKmV9RueoMizWbqlDMjHYoxJyTciSEPKOy0XeTf142IJAKzgRePcnye\niKwWkdUD/ZFUMwB424gvfJcVvslcOtEeUzX9S19qfL4aeO9o1UiqOl9Vp6vq9JycnF4OzZgTVLyG\nOG8DB3LOITMpNtLRGHNCwp0YioHOj2Pk+/cFMxerRjIDROWG1/GqMGTK5ZEOxZgTFu7EsAooEJFR\nIhKL8+W/qGshEUkDLgReDXM8xvSK1u1vsUHHcNGUUyIdijEnLKyJQVXbgTuBJcAW4AVV3SQit4nI\nbZ2Kfg54Q1UbwhmPMb2iqYrcuk3sTJlBrg2tbfqhsPd8VtXFwOIu+x7rsv0U8FS4YzGmN5R/vJQc\nfCSMuyzSoRhzUmxIDGN6WMWGxcRpAlNmXhLpUIw5KX3pqSRj+j9Vsg68x8a4KQzLSYt0NMacFEsM\nxvSg8n2byfWV0TpiVqRDMeakWWIwpgftXuk8WDdyxtURjsSYk2eJwZiuvG2w6RUo337Cb43Zs4xi\nGczIgolhCMyY3mGNz8Z0VrETXroVStYCAuOvhvO/C0OnBi3e5vVRVNXEnop69h6sZk7zenYMuTr4\nuC/G9BOWGIwBUIU1T8GSH4I7Fq57DA7tgg/nw5ZFNA27gO2nzGOD+zT2VDayt6KBvZWNHDpUwTl8\nzCzXBq50byBJWhg07bOR/jTGfCqWGEz/t+cdKN0A8emQkA7xaZ3W0yEuBUQC3tLY2s7u8gYONbTS\nXnuQcat+xNCDyyjMOIvXRt3LgX2ZFFaNpcxzBhc0LuKr+xdzeuGX8PrGsoEruTCxmh+xntGxm3Dh\npT02Fe+oi9CJVzNk0nURuhDG9AxR1UjHcMKmT5+uq1evjnQYpi/Y+RYs+CKo76hF2uKzqUidyE5P\nAavbRvFWbR4bq52B7S5yreP/ev5EKk080D6Xp7yX43K5SYp1k5eRyMisREZmJzEm3cW0Q4sZvvUJ\n3LX+AYMHT4aCy2DsZZB/Jrjtd5bp20RkjapOP245SwymP2rz+ijavp78F6/hUOwg/nvob2hsbMLX\nVAVN1bhbavC01pBGPQWuYibLLsZKCS5x/n+vjRtMW+pwsso/ojlzPFWzHyV26ESS4mKIi3EhXe4w\nDvO2wf6VkF0AKTactulfQk0M9hPH9K4Nz8MnL0DmGMgdBznjIedUSOw+m6uqUtPUxoHaZg7UNLO7\nvIEtpbVsLq2lvKyUF1z3Ui3CnKa78OIiIzGDtORc0nNjSUnwkJ7oIS3BQ3JGIjGDkvEleXGVfQLF\na0ktWQtlW+Dcu4i/6EcMiYkLLX63B0ad38MXxZi+xRKD6T1bXoOXvwGpec6v7tb6w4daE3I4GDeK\ndTGnszDmaorrfByoaaalPbCKKDs5ltMGJ/BoxqPkN1RReM0LLJ10IR53iE9ejzzPeRljjsoSgzm6\nfe/DugWQcwoMOwuGTAHPSY4WWrgKXrwF8qZRM+cl1hY3s2X7Fg7t2UBM5TbG1BVyakMh17ge5wz3\n//KPod+jceIMBqXGMzg1nsFpcQzLTCQ3OQ7+59tQtA4+/zgjJ1/Us5/ZGGOJwQTRUg9v3Q8fzQdP\nErT5R0N3xzrJYfhZTqIYfjYkZR/zVDWNbezatoHxi79IvWRyW81drP31ClQhxiVMyp/OjHM+Q+bI\nTEaMyITi5eS/9h3uLvw25H4Vpt/vPGXU4YNHYO0zcP5/wOQbwngRjIle1vhsAu1eBou+BdWFcNY3\n4JKfQGsDFH4EhSudZck68LaCywPn3AkXfA9ik/D6lM0ltazcXcmafVVsLKmhseogL8XeR4o0clvs\nr0nLH8ekvHTOHJXB1GEZJMS6u8fQUg9v/wo+/CMk5cKVv4UJ18D2JfDcXBj3Wbj+aXBZx31jToQ9\nlWROTHMtLP2x08krcwxc+wiMODt42fYWKFmPb/WTuD5eSF3cYJ5KvZ35ZeOoa/ECMCIrkalD4vjB\nwe+R3biD+jkvk3bKuScWU/EaWHQXHPwECi53qrayRsNXX4fYpE/3eY2JQvZUkgndjjedevu6Ujjn\nW3DRj8CTcNTiu6raeHJNIos2fJ5xrafwc99f+FbLfVyWejZFM3/K5ElTyE32wPM3Q90mmPPXE08K\nAHnTYN7b8MEfYNkDTme1uc9ZUjAmzCwxRLsVD8KbP4WccXDDM5Af/MeEqrJy9yH+vGI3b24pIzbG\nxdWThzLr1ElkjpgHm59m3LJfM+5f10Dbd6GhHLb9E674v854QyfL7YHzvgOT54C4rO+AMb3AEkO0\nUoVlv4blv4HTvgDX/RGCPMvf5vWx+JNSHn93NxuLa8lMiuWuSwq4+ewRZCd3Kn/OnXDa52HJj5zz\nApx9p9NO0RNSh/bMeYwxx2WJIRqpwtKfwPsPw5QvwTUPg+tII3Btcxsrd1Xy3s4K3th8kNKaZkbn\nJPGrz03i82fkEe8J0mAMzpf39X+BaV+GA5/AzDt66QMZY3qSJYZo4/PB6/8Jqx6HM78OV/yWNoV1\new6xYmcFK3aUs6GoBq9PSfC4OXtMFr/83GnMOiUXl+sow0R0NXqW8zLG9EthTwwiMht4CHADT6jq\nA0HKzAJ+D3iAClW9MNxxRSWfF/7nLlj3Vzj7TrZN/j5/XbSJV9aVUN/SjktgUn46t184hvMKspk6\nPJ24mKPcHRhjBqywJgYRcQOPAJcBRcAqEVmkqps7lUkHHgVmq+p+EckNZ0xRy9sOr9wGn/yd7eNu\n5949V/HR2+8SG+Pis5OH8JkJgzh7dDZpiZ5IR2qMibBw3zHMAHaq6m4AEVkIXAts7lTm34CXVHU/\ngKqWhTmm6NJcCyVraV7xKPG7l/CI6yZ+u/58hmU284MrxnHD9GFkJMVGOkpjTB8SUmIQkUmq+slJ\nnD8PKOy0XQSc1aXMKYBHRJYBKcBDqvrMSfwt422Hss1QvBqK1kDxarR8G4ISoy7u997MnjH/zpNn\nj+DCU3Jxh9pmYIyJKqHeMTwqInHAU8ACVa3p4RimAZcACcAHIrJSVQNmYheRecA8gOHDh/fgn+9H\ntr0Ob97vdERT35GXz+tftgNOT/a2uAw2uwp4q+0LbHYVUDD1Qr5y4WRGZFnnMGPMsYWUGFT1fBEp\nAL4GrBGRj4C/qOrS47y1GBjWaTvfv6+zIqBSVRuABhF5BzgdCEgMqjofmA/OkBihxD1gVO2F1++B\n7a87HdFOn+t09hKXM2WluEFcqLjY2DqEP+5MZ3FxPBmJsXzlolH89uwRVl1kjAlZyG0MqrpDRO4F\nVgMPA1PFmebqh6r60lHetgooEJFROAlhLk6bQmevAn8QkRggFqeq6cET+xgDVHuL09fgnf/nfPlf\n9jOY+U2nN3AXK3dXct+rm9h2sI78jATuv2Y0N0wfFnyQOmOMOYZQ2xgmA18FrgKWAler6loRGQp8\nAARNDKraLiJ3AktwHld9UlU3icht/uOPqeoWEflf4GPAh/NI68ZP+8H6vV3/gsXfg8qdMOFauPxX\nkJbfrVhVQyu/WryFv68pYlhmAg/NncJVk4YQE+rENcYY00VIo6uKyHLgCeAfqtrU5djNqvrXMMUX\n1IAeXbW5Fl77Dmz8B2SOdoacHntpt2Kqyktri/nl4i3UNrUx74LRfOviArtDMMYcVU+PrnoV0KSq\nXv/JXUC8qjb2dlIY0Mq3w/M3QeUumPUDOPfuoDOm7S6v595XNvL+rkqmjcjgV5+bxKmDUyIQsDFm\nIAo1MbwJXAp0TNKbCLwBnBOOoKLSlv+Bl293BrL791dg1AXdirS0e3ls2W4eeXsncR4Xv/zcadx4\n5vDQh6owxpgQhJoY4lX18MztqlovIolhiim6+Lzw9i/h3f+CoWfAnL8GbUt4d0c5P3l1E3sqGrhq\n8hDu++wEclNPcv5lY4w5hlATQ4OInKGqawFEZBrQdJz3mONpPAQvfh12vQVn/Dtc8dtuVUcHapr5\n+T8388+PSxmZlcgzX5vBBafkRChgY0w0CDUx3A38XURKAAEGA3PCFlU0KN3gn+GsFK5+CKZ9JeBw\nu9fHU+/v5cGl22nzKd+97BTmXTD66ENeG2NMDwm1g9sqERkHnOrftU1V28IX1gDWUOFMZLP6L5A8\nyJm/uMusaWv3V/HDlz5h64E6Zp2aw/3XTLQey8aYXnMig+idCkwA4oEzRAQb0+gEtDXDR39yOqu1\nNsCZtzhPHiVmBhRbtq2MW59ZTXZyHI99aRqXTxyE04/QGGN6R6gd3O4DZuEkhsXAFcAKwBLD8ajC\n5ldg6X1QvQ9OmQ2X/RxyTulW9INdlXzjr2soyE3huVtn2hDYxpiICPWO4Ys44xetU9Wvisgg4G/h\nC2uAKP0YFv8HFH4Ig06Dm1+BMRcFLbpmXxW3PL2K4ZmJ/PWWGZYUjDERE2piaFJVn4i0i0gqUEbg\n4Himq7oD8My14IqBa/4bptwUMK9yZxuLa/jKXz4iNyWOBV8/i6zkuF4O1hhjjgg1Maz2z7T2OLAG\np6PbB2GLqr9ThVe+CW1N8I13glYbddh+sI6b//whqfEeFtw60/omGGMi7riJwT+C6q9VtRp4zD/g\nXaqqfhz26Pqrj+Y7fROu+q9jJoU9FQ3c9MSHeNwuFnz9LPLSE3oxSGOMCe64iUFVVUQWA5P823vD\nHVS/VrYFlv4ECj4D0285arGiqkZuenwlXp/y/LyZjMy2x1GNMX1DqGMzrxWRM8MayUDQ3gIv3Qqx\nyXDtI84kOkGUVDfxb49/SH1LO898bQYFg2wAPGNM3xFqG8NZwE0isg9owOn9rKo6OWyR9Uf/+gUc\n+ARuXAjJuUGLlFQ3MXf+SqoaWnnmlhmclpfWy0EaY8yxhZoYLg9rFAPBnnfg/f+GaV+FU68IWqRr\nUpg6PKOXgzTGmOMLNTFE1xzLJ6qpCl6+DbLGwOW/DFqkpLqJGx+3pGCM6ftCTQz/xEkOgjMkxihg\nGzAxTHH1H6rw2neh/iDcshRiuzcil9Y4SeFQvSUFY0zfF+ogepM6b4vIGcA3wxJRf7P+Wdj0Elx8\nL+Sd0e1waY1TfXSovpWnLSkYY/qBk5ox3j8vw1k9HEv/s2EhLPoWjDwfzvtut8Ndk8IZlhSMMf1A\nqIPodf7WcwFnACVhiai/+OhxZxykURfC3Ge7DXfR2NrOl5/8yJKCMabfCbWNofOD9u04bQ4v9nw4\n/cS7v4O37odTr4IvPtlt1jWAn7y6iR1l9TzzNUsKxpj+JdQ2hvtP9g+IyGzgIcANPKGqD3Q5Pgt4\nFdjj3/WSqv7sZP9eWKk6CWHFgzDperjuj+DuPgrq31cX8o81RXz74rGcX2DTcBpj+peQ2hhEZKl/\nEL2O7QwRWRLC+9zAIzjzN0wAbhSRCUGKvquqU/yvvpkUfD6n6mjFg05fhc/ND5oUth+s48evbmTm\n6EzuuvTo4yQZY0xfFWrjc45/ED0AVLUKCN61N9AMYKeq7lbVVmAhcO2Jhxlh3nZ45XZY9QSc8234\n7IPg6n7pGlra+eaCtSTHeXh47lTcLpt5zRjT/4SaGLwiMrxjQ0RGEFqntzygsNN2kX9fV+eIyMci\n8rqIBO0bISLzRGS1iKwuLy8PMewesvwB+Hih80jqZT8LOgaSqvLjVzayq7yeh+ZOseGzjTH9VqiN\nzz8CVojIcpxObucD83oohrXAcFWtF5ErgVeAgq6FVHU+MB9g+vTpvdsTe+NLMOZiuOB7Ry3y99VF\nvLSumLsuKeDcsdm9GJwxxvSskO4YVPV/cR5RfR6nOmiaqh63jQEoJnCmt3z/vs7nrlXVev/6YsAj\nIn3nm7VyFxza5czVfBRbD9Ty41c3cu7YLL59SbecZowx/Uqojc+fA9pU9TVVfQ1oF5HrQnjrKqBA\nREaJSCwwF1jU5dyD/ZMBISIz/DFVnsiHCKudbzrLsZcGPdzQ0s4dC9aSmuDh93OsXcEY0/+F2sZw\nn6rWdGzl4o30AAARtElEQVT4G6LvO96bVLUduBNYAmwBXlDVTSJym4jc5i/2RWCjiGwAHgbmqmrf\nGbRvxxuQNdYZIC+IXy3ewp6KBh6aO4WcFJur2RjT/4XaxhAsgYTaB2IxsLjLvsc6rf8B+EOIcfSu\n1kbY8y5M/1rQw8XVTTy/qpCbzhrBOWP6Tu2XMcZ8GqHeMawWkd+JyBj/63fAmnAG1ifsXQHeFii4\nLOjhx9/ZDcA3Lhzdm1EZY0xYhZoYvgW04jQ+Pw+0AHeEK6g+Y8cb4EmEEed2O1RR38LCVfu5bmoe\n+RmJEQjOGGPCI9TqoAbgnjDH0reowo4lziB5QcZC+st7e2hp93H7rOBtD8YY01+FOrpqDvCfOBPz\nHP6WVNWLwxRX5FXsgOr9cO7d3Q7VNrfxzPv7uPK0IYzJSY5AcMYYEz6hViUtALbizNx2P7AX51HU\ngWvHG84ySPvCXz/YR11Lu90tGGMGpFATQ5aq/hmnL8NyVf0aMHDvFgB2LoWccZA+PGB3U6uXP6/Y\nw6xTczgtLy1CwRljTPiEmhja/MtSEblKRKYCmWGKKfJa6mHve0HvFhau2s+hhlbuuGhsBAIzxpjw\nC7Ufwy9EJA34P8B/A6nAd8IWVaTtWQ6+Nij4TMDu1nYf89/ZzYyRmZw5cuDmRWNMdAv1qaTX/Ks1\nwEVdj4vID1T11z0ZWETteANiU2DYzIDdr6wrprSmmV9/flKEAjPGmPALtSrpeK7vofNEnirsWApj\nZkFM7OHdXp/yx+W7mDg0lQtPsVnZjDEDV08lhoEzclzZFqgthrGB7QuvbyxlT0UDd1w0FgkyH4Mx\nxgwUPZUY+s6gd59WkMdUVZVH3t7F6JwkLp84OEKBGWNM77A7hq52LIVBkyB16OFdy7aVs6W0ltsv\nHGPDahtjBryeSgx/76HzRFZzDez/oNtjqgs+3M+g1DiumxpsVlJjjBlYjpkYus6kJiJfEpGH/fMv\nH/7prKq/CleAvWr3MlBvwGOqTa1eVuws5/KJg/G4eyqPGmNM33W8b7o3OlZE5F7gZpzhti8DfhfG\nuCJjxxsQnwb5Zx7etWJnBc1tPi6bMCiCgRljTO85Xj+GzhXqnwfOV9UGEXkWWBu+sCLg8GOqF4P7\nyGVZuvkAKXExnDUqK4LBGWNM7zleYkjwD3/hAjz+4bdR1TYR8YY9ut504GOoPxhQjeT1KW9tKWPW\nuFxiY6wayRgTHY6XGEo5UmVUISJDVLVURLKA9vCG1st2LHWWYy89vGvd/ioqG1qtGskYE1WOmRhU\ntdvwF35VwAU9H04E7V0BuRMhOffwrqWbDxLjEmadaj2djTHRI9RB9BCRzwPn4XRmW6GqL4ctqt7m\nbYPCj2DqTQG7l245yMzRWaTGeyIUmDHG9L6QKs5F5FHgNuATYCPwDRF5JMT3zhaRbSKyU0SOOj2o\niJwpIu0i8sVQztujStZDW0PA3M67yuvZXd5g1UjGmKgT6h3DxcB4VVUAEXka2HS8N4mIG3gE5/HW\nImCViCxS1c1Byv2GTo/H9qp9K5zliHMO71q6+SAAl1piMMZEmVAftdkJdJ7KbJh/3/HMAHaq6m5V\nbQUWAtcGKfct4EWgLMR4etbe9yD7lG7tCxOHppKXnhCRkIwxJlJCTQwpwBYRWSYiy4DNQKqILBKR\nRcd4Xx5Q2Gm7yL/vMBHJAz4H/DHkqHuSzwv7VwZUI5XXtbB2f5VVIxljolKoVUk/CWMMvwe+r6q+\nYw1nLSLzgHkAw4cPP2q5E3bgY2itg5HnHd71r60HUYVLx1tiMMZEn1BncFsuIoOAjrEiPlLVUKp9\ninGqnTrk+/d1Nh1Y6E8K2cCVItKuqq90iWE+MB9g+vTpPTfM9973nGWnO4alm8vIS09g4tDUHvsz\nxhjTX4T6VNINwEc4M7XdAHwY4tNDq4ACERklIrHAXCCg6klVR6nqSFUdCfwD+GbXpBBW+96DzNGQ\nOgQ4MmjepeNzbUIeY0xUCrUq6UfAmR13CSKSA7yJ80V+VKraLiJ3AksAN/Ckqm4Skdv8xx876ch7\ngs8H+96H8Vcf3vXujnL/oHk2IY8xJjqFmhhcXaqOKgnxbkNVFwOLu+wLmhBU9SshxtMzyjZBc3WX\naqSDpMTHcNbozF4NxRhj+opQE8PrIrIEeM6/PYcuX/b9Ukf7wkgnMXh9yr+2ljHr1Fybe8EYE7VC\n/fZT4E/AZP9rftgi6k373oO04ZDuPOW01gbNM8aYkO8YLlPV7wMvdewQkfuB74clqt6g6rQvdJrG\n883NB/G4bdA8Y0x0O2ZiEJHbgW8Co0Xk406HUoD3whlY2JVvg8aKbu0LNmieMSbaHe+O4VngdeDX\nQOcB8OpU9VDYouoNHeMj+dsXdpbVs7uiga+cOzJyMRljTB9wvPkYaoAa4MbeCacX7X0PUoZCxigA\n3t7qPHRlvZ2NMdEuOh+9UXUankecA/5ObLsr6slKimWoDZpnjIly0ZkYKnc58zuPPNK+UFLdbEnB\nGGOI1sSwr2N8pCMD55VUNzEkLT5CARljTN8RvYkhKReyCwBQVUqqm+yOwRhjiMbEoOo0PHdqX6ht\nbqeh1cvQdLtjMMaY6EsM1fugtihg/oXSmiYAhqTZHYMxxkRfYggy/0JpdTOAVSUZYwzRmBj2vQcJ\nGZAz7vCu4mrnjsGqkowxJhoTw94Vzt2C68hHL61pwu0SclMsMRhjTHQlhpoip42hUzUSOH0YBqfG\n43bZjG3GGBNdiWHf+85yZNfE0GTVSMYY4xddiSExCyZcB4NOC9hdUtNkTyQZY4xfqPMxDAxjL3Fe\nnfh8yoGaZoZOssRgjDEQbXcMQVTUt9DmVatKMsYYv6hPDCU1Th8Gq0oyxhhH2BODiMwWkW0islNE\n7gly/FoR+VhE1ovIahE5L9h5wqXE+jAYY0yAsLYxiIgbeAS4DCgCVonIIlXd3KnYW8AiVVURmQy8\nAIzrfrbwOJwY7I7BGGOA8N8xzAB2qupuVW0FFgLXdi6gqvWqqv7NJEDpRaU1zSR43KQn2jzPxhgD\n4U8MeUBhp+0i/74AIvI5EdkK/BP4WphjClBS3cSQ9HhErHObMcZAH2l8VtWXVXUccB3w82BlRGSe\nvw1idXl5eY/97ZKaZvJs8DxjjDks3ImhGBjWaTvfvy8oVX0HGC0i2UGOzVfV6ao6PScnp8cCtJnb\njDEmULgTwyqgQERGiUgsMBdY1LmAiIwVfz2OiJwBxAGVYY4LgJZ2L+V1LTbctjHGdBLWp5JUtV1E\n7gSWAG7gSVXdJCK3+Y8/BnwB+HcRaQOagDmdGqPD6mBNC2BPJBljTGdhHxJDVRcDi7vse6zT+m+A\n34Q7jmBKOmZusz4MxhhzWJ9ofI6UI53b7I7BGGM6RHViKPUPh2FVScYYc0RUJ4aS6iYyEj0kxLoj\nHYoxxvQZUZ8YbPA8Y4wJFNWJobSm2doXjDGmi6hODMU2pacxxnQTtYmhrrmNuuZ2u2MwxpguojYx\nlB6eoMfuGIwxprOoTQzWh8EYY4KL4sTg78NgicEYYwJEbWIorWnCJTAoJS7SoRhjTJ8StYmhpLqZ\nQanxxLij9hIYY0xQUfutaPMwGGNMcFGbGEprmqx9wRhjgojKxKCqlFivZ2OMCSoqE0NlQyut7T6G\nWlWSMcZ0E5WJoaMPwxC7YzDGmG6iNDHYPAzGGHM0UZoYOno9W1WSMcZ0FZWJobSmibgYF5lJsZEO\nxRhj+pyoTAwdTySJSKRDMcaYPic6E4N1bjPGmKMKe2IQkdkisk1EdorIPUGO3yQiH4vIJyLyvoic\nHu6YSqutD4MxxhxNWBODiLiBR4ArgAnAjSIyoUuxPcCFqjoJ+DkwP5wxtXl9HKxrtj4MxhhzFOG+\nY5gB7FTV3araCiwEru1cQFXfV9Uq/+ZKID+cAR2sbUbVhts2xpijCXdiyAMKO20X+fcdzS3A68EO\niMg8EVktIqvLy8tPOqCOPgzWuc0YY4LrM43PInIRTmL4frDjqjpfVaer6vScnJyT/julNU4fhjzr\nw2CMMUHFhPn8xcCwTtv5/n0BRGQy8ARwhapWhjWgjuEwrNezMcYEFe47hlVAgYiMEpFYYC6wqHMB\nERkOvATcrKrbwxwPpdXNpMbHkBQX7pxojDH9U1i/HVW1XUTuBJYAbuBJVd0kIrf5jz8G/ATIAh71\ndzhrV9Xp4YrJ5mEwxphjC/vPZlVdDCzusu+xTutfB74e7jg6FFsfBmOMOaY+0/jcW5w7Bmt4NsaY\no4mqxNDY2k51Y5s1PBtjzDFEVWLo6MOQZ1VJxhhzVFGWGDoeVbWqJGOMOZqoSgwdndus8dkYY44u\nqhJDbko8l00YxKBUu2MwxpijiapeXheNy+WicbmRDsMYY/q0qLpjMMYYc3yWGIwxxgSwxGCMMSaA\nJQZjjDEBLDEYY4wJYInBGGNMAEsMxhhjAlhiMMYYE0BUNdIxnDARKQf2neTbs4GKHgxnILBrEpxd\nl+7smnTXn67JCFXNOV6hfpkYPg0RWR3OGeL6I7smwdl16c6uSXcD8ZpYVZIxxpgAlhiMMcYEiMbE\nMD/SAfRBdk2Cs+vSnV2T7gbcNYm6NgZjjDHHFo13DMYYY44hqhKDiMwWkW0islNE7ol0PJEgIk+K\nSJmIbOy0L1NElorIDv8yI5Ix9jYRGSYib4vIZhHZJCJ3+fdH7XURkXgR+UhENvivyf3+/VF7TTqI\niFtE1onIa/7tAXdNoiYxiIgbeAS4ApgA3CgiEyIbVUQ8Bczusu8e4C1VLQDe8m9Hk3bg/6jqBGAm\ncIf//41ovi4twMWqejowBZgtIjOJ7mvS4S5gS6ftAXdNoiYxADOAnaq6W1VbgYXAtRGOqdep6jvA\noS67rwWe9q8/DVzXq0FFmKqWqupa/3odzj/6PKL4uqij3r/p8b+UKL4mACKSD1wFPNFp94C7JtGU\nGPKAwk7bRf59Bgapaql//QAwKJLBRJKIjASmAh8S5dfFX2WyHigDlqpq1F8T4PfAfwK+TvsG3DWJ\npsRgQqDOY2pR+aiaiCQDLwJ3q2pt52PReF1U1auqU4B8YIaInNbleFRdExH5LFCmqmuOVmagXJNo\nSgzFwLBO2/n+fQYOisgQAP+yLMLx9DoR8eAkhQWq+pJ/d9RfFwBVrQbexmmbiuZrci5wjYjsxamK\nvlhE/sYAvCbRlBhWAQUiMkpEYoG5wKIIx9RXLAK+7F//MvBqBGPpdSIiwJ+BLar6u06Hova6iEiO\niKT71xOAy4CtRPE1UdUfqGq+qo7E+f74l6p+iQF4TaKqg5uIXIlTR+gGnlTVX0Y4pF4nIs8Bs3BG\nhDwI3Ae8ArwADMcZtfYGVe3aQD1gich5wLvAJxypO/4hTjtDVF4XEZmM05DqxvkB+YKq/kxEsojS\na9KZiMwC/kNVPzsQr0lUJQZjjDHHF01VScYYY0JgicEYY0wASwzGGGMCWGIwxhgTwBKDMcaYAJYY\njOllIjKrY2ROY/oiSwzGGGMCWGIw5ihE5Ev+OQnWi8if/IPK1YvIg/45Ct4SkRx/2SkislJEPhaR\nlzvG5BeRsSLypn9eg7UiMsZ/+mQR+YeIbBWRBf7e18b0CZYYjAlCRMYDc4Bz/QPJeYGbgCRgtapO\nBJbj9BwHeAb4vqpOxulB3bF/AfCIf16Dc4COUTinAnfjzA0yGmccHmP6hJhIB2BMH3UJMA1Y5f8x\nn4AzOJoPeN5f5m/ASyKSBqSr6nL//qeBv4tICpCnqi8DqGozgP98H6lqkX97PTASWBH+j2XM8Vli\nMCY4AZ5W1R8E7BT5cZdyJzumTEundS/2b9H0IVaVZExwbwFfFJFcODyv7wicfzNf9Jf5N2CFqtYA\nVSJyvn//zcBy/2xwRSJynf8ccSKS2KufwpiTYL9SjAlCVTeLyL3AGyLiAtqAO4AGnElr7sWpWprj\nf8uXgcf8X/y7ga/6998M/ElEfuY/x/W9+DGMOSk2uqoxJ0BE6lU1OdJxGBNOVpVkjDEmgN0xGGOM\nCWB3DMYYYwJYYjDGGBPAEoMxxpgAlhiMMcYEsMRgjDEmgCUGY4wxAf4/v0VVPvCZ828AAAAASUVO\nRK5CYII=\n", 374 | "text/plain": [ 375 | "" 376 | ] 377 | }, 378 | "metadata": {}, 379 | "output_type": "display_data" 380 | } 381 | ], 382 | "source": [ 383 | "plt.plot(epochs, [x[5] for x in all_losses], label='train');\n", 384 | "plt.plot(epochs, [x[6] for x in all_losses], label='val');\n", 385 | "plt.legend();\n", 386 | "plt.xlabel('epoch');\n", 387 | "plt.ylabel('top5_accuracy');" 388 | ] 389 | }, 390 | { 391 | "cell_type": "markdown", 392 | "metadata": {}, 393 | "source": [ 394 | "# Save" 395 | ] 396 | }, 397 | { 398 | "cell_type": "code", 399 | "execution_count": 15, 400 | "metadata": { 401 | "collapsed": true 402 | }, 403 | "outputs": [], 404 | "source": [ 405 | "model.cpu();\n", 406 | "torch.save(model.state_dict(), 'model.pytorch_state')" 407 | ] 408 | } 409 | ], 410 | "metadata": { 411 | "kernelspec": { 412 | "display_name": "Python 3", 413 | "language": "python", 414 | "name": "python3" 415 | }, 416 | "language_info": { 417 | "codemirror_mode": { 418 | "name": "ipython", 419 | "version": 3 420 | }, 421 | "file_extension": ".py", 422 | "mimetype": "text/x-python", 423 | "name": "python", 424 | "nbconvert_exporter": "python", 425 | "pygments_lexer": "ipython3", 426 | "version": "3.6.1" 427 | } 428 | }, 429 | "nbformat": 4, 430 | "nbformat_minor": 2 431 | } 432 | --------------------------------------------------------------------------------