├── .gitignore ├── LICENSE ├── README.md ├── plot_csv.ipynb ├── results ├── diagram_17.png ├── diagram_17_clr.png ├── diagram_bees.png └── stats_bees.csv └── retrain.py /.gitignore: -------------------------------------------------------------------------------- 1 | **data/ 2 | .idea/** 3 | .ipynb_checkpoints/** 4 | __pycache__/** 5 | .DS_Store 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2017, Alexander Hirner 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pytorch-retraining 2 | Transfer Learning shootout for PyTorch's model zoo (torchvision). 3 | 4 | * **Load** any pretrained model with custom final layer (num_classes) from PyTorch's model zoo in one line 5 | ```python 6 | model_pretrained, diff = load_model_merged('inception_v3', num_classes) 7 | ``` 8 | 9 | * **Retrain** minimal (as inferred on load) or a custom amount of layers on multiple GPUs. Optionally with _Cyclical Learning Rate_ [(Smith 2017)](http://arxiv.org/abs/1506.01186). 10 | ```python 11 | final_param_names = [d[0] for d in diff] 12 | stats = train_eval(model_pretrained, trainloader, testloader, final_params_names) 13 | ``` 14 | 15 | * **Chart** `training_time`, `evaluation_time` (fps), top-1 `accuracy` for varying levels of retraining depth (shallow, deep and from scratch) 16 | 17 | | ![chart](https://raw.githubusercontent.com/ahirner/pytorch-retraining/master/results/diagram_bees.png) | 18 | |:---:| 19 | | *Transfer learning on example dataset [Bee vs Ants](http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html)* with 2xV100 GPUs| 20 | 21 | ## Results on more elaborate Dataset 22 | *num_classes = 23, slightly unbalanced, high variance in rotation and motion blur artifacts* with 1xGTX1080Ti 23 | 24 | | ![chart_17](https://raw.githubusercontent.com/ahirner/pytorch-retraining/master/results/diagram_17.png) | 25 | |:---:| 26 | | *Constant LR with momentum* | 27 | 28 | | ![chart_17_clr](https://raw.githubusercontent.com/ahirner/pytorch-retraining/master/results/diagram_17_clr.png) | 29 | |:---:| 30 | | *Cyclical Learning Rate* | 31 | -------------------------------------------------------------------------------- /plot_csv.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 197, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import matplotlib.pyplot as plt\n", 10 | "import matplotlib.cm as cm\n", 11 | "import numpy as np\n", 12 | "\n", 13 | "import csv" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 198, 19 | "metadata": { 20 | "collapsed": true 21 | }, 22 | "outputs": [], 23 | "source": [ 24 | "with open('stats_bees.csv') as csvfile:\n", 25 | " reader = csv.DictReader(csvfile)\n", 26 | " stats_full = list(reader)" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 200, 32 | "metadata": {}, 33 | "outputs": [], 34 | "source": [ 35 | "eval_examples = 153\n", 36 | "\n", 37 | "for s in stats_full:\n", 38 | " train_mode = None\n", 39 | " if s['retrained'] == 'True':\n", 40 | " if s['shallow_retrain'] == 'True':\n", 41 | " train_mode = 'shallow'\n", 42 | " else: \n", 43 | " train_mode = 'deep'\n", 44 | " else:\n", 45 | " train_mode = 'from_scratch'\n", 46 | " s['train_mode'] = train_mode\n", 47 | " \n", 48 | " s['fps'] = 1.0 / (float(s['eval_time']) / eval_examples)" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 255, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "#Color\n", 58 | "N = 6\n", 59 | "cmx = cm.rainbow(np.random.rand(3))\n", 60 | "c_map = {'shallow' : cmx[0], 'deep' : cmx[1], 'from_scratch': cmx[2]}\n", 61 | "cmx = cm.rainbow(range(0, N *50, 50))" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 291, 67 | "metadata": { 68 | "scrolled": false 69 | }, 70 | "outputs": [ 71 | { 72 | "data": { 73 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAGaCAYAAAC2Q8+BAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FMUbwPHvpCckpNEJhN57QpNeRESKKCIC/gBB6SBS\nLHQURMSGSFFErBRBaSKIoBSpCdJ7JwkQSggB0jO/P/YSj5CQQi6XhPfzPHng9nZm3tvb23tvZ2dW\naa0RQgghhBA5n421AxBCCCGEEOkjiZsQQgghRC4hiZsQQgghRC4hiZsQQgghRC4hiZsQQgghRC4h\niZsQQgghRC4hidtjQiml0/F33tpxpkQpNUApdUYpFauUumLBdiqZtkM3S7WRFZRSV5RS86wdR3JK\nqelKqTgrtDtKKdUxG9rpopQaloH1qymltiilIkz7VVsLxrbL7HOcoJQKUUr9ppTyy0AdbdN5nMi2\nfU8ptVIpdTgd672ulLqVHTFZilLq08y8BqVULdP78qwl4hI5j521AxDZpmGyx78CB4BJZsuisy2a\ndFJKlQK+ABYC3wL3rBlPDtEOCLN2EDnIKGAtsNrC7XQB/IFZ6Vx/FlDMVC4cOGahuBLtBYYBCigN\njAe2KqVqaq1Pp6P8Tu4/TvgCSzCOERvMll/NkmiFEJkiidtjQmu9y/yxUioauJ58eWqUUo5aa2sk\ndhUxzgx/o7Xe8aiVKaVsAaW1zvYzQylRSinAXmsdk94yWut9FgxJZJ3KwBqt9YY010yHdHwGb5t9\nnncqpQKAE8BrwJi06tdahwNJxwOzsz9n0nucSA8rHkuEyBOkq1Q8QCm1RCl1WinV1NQFEwlMMT33\nP1P3zzVTF1CgUqp7svJOplP345RSI5VSF0zrblJKVUy2bntTG7eVUneUUseUUm8lxgGsN636j3k3\njTIMVkodUkpFKaVClVLzlVLuKcQxQSk1Xil1AYgBymdwe7RWSv1tiu+OqQuqcrJ12iml1pu6Me+a\n4hqmlLJJtt4VpdQCU/fvSSAWaGXWTdtHKfW+ab0wU1dR0RTqmGf2eICprJ9SaplpWwcrpT5SSjkk\nK1tBKbVBKRVpqme6UmqIqXyRNLZDqu9VsvXKm9q4q5Q6p5R625Sgmq9TRSm1RikVborlH6VUqxTq\n6qCU2mNaJ0wptUIpVdZ8WwCFgb4qha68tMqntD1NyxL3HfN98UWgrFk7x1PZTm2VUhooArxqWjcq\ngzHtUkr9qZR6Til1QBk/tF5Jqb3UaK1PAhFAOaVUSaVUvFKqfwrxTjftM24ZqV8p5ayUmqmUuqSU\nilHG5QxjlfHjKHGdxG68/ymlPldKXQWiEvcH036/VBnHkyil1Cml1NQU2nrCtE3uKaWOK6V6phGb\nMu13J0313jSVfyqNciuVUoeVcezba3qPDiulWiilbJRxTAtSSt0yxe2RrLy3UuorpdRVpVS0Uupo\nKtv8CaXUblNsF5RSI1OJx1EpNVkZx+No07Z+TymVoZMuSqlnVepd35+arVfK9LpummILVEo9l0J9\nzZTxPXBPGceDdUqpGhmJSTwCrbX8PYZ/wHngh1SeWwLcAi4Ag4DmQF3Tc5OAIcBTwJPANCAe6G1W\n3gnQpjZ+AzpgfOldAo4CNqb1KmEkUotM9bUCBgBTTc+XA0aa6uoLNABKm5771FT2A6CN6fkrwDaz\n+hPjCAY2A52BpwHvVF53JdP63cyWPWd6fcuBjqY69gDXgKJm6w3F6LJ7BmgJvIXRrTspWRtXTPEc\nMG2T1kAps7bPY3QJtzW9pjBgQwp1zDN7PMBU9gQwwVTnZNOyt83WcwYumt7Xl4H2wBrTYw0Uecj+\n8tD3yrTOdNO2OgS8bopjjqnul8zW8zW9rpNAd9N23QTEAS3N1usEJADrMPahnqbtcxkoZFqnDnAD\nWGXaP8z3kTTLp7Q9k+07b5ntixsx9uHEdmqmsq3cTc/fAn4x/b9eBmPaZYrrDNALaAFUe8j7swv4\nM9mygqbXsND0eDWwL9k69hhdn/NSqDNxn+yZSpu/YVxe8Q7GZ3C66bXNMVunFv99Bn/C6OZ/HqM7\ntwpGF/JJjKS0JcY+/7VZ+ZXAdYxu5j6mdlaZ6qz9kO0x2BTbGIzjV3tgHGb7YSrlEts7gvEZeRoI\nNMX5ObDUtGwgEAl8aVbWAfjXtG7iMfIrU6xjzNYrAdw1rdvF9LfPtG/dMltPmfaT28CbGJ+5Uaay\nX6ewjZ99yOvy5L/9NvFvmqlcf9M6XqYYgk3b+hmMS2o08KJZXY0wfnBuxzgedgUOm+Isl5nvI/nL\n2J/VA5A/K73xaSduGngqjTpsMLrbvwd2my1P/NI7AtiaLe9pWl7H7HEC4PiQNtqbyjQwW1bBVG5M\nsnVbmdZtmyyO84BDOrbJfYmb6fVdAtYlW88L40t5eir1KNN2eRe4muy5KxhnQQqk0nbyJG2cablX\nsjpSStzeTlb2T+Cg2eNhpvVqmi2zAY6TduKWnvdqOg8maQrji3m12bLZGF+qJc2W2QPngB1myw6b\n9iEbs2UVMZLDacm2x4IU4slI+Ycmbmafi9MZ+IxdT6He9Ma0y7Sscjrb2oWR/NqZtmUFjC99889D\nW9PjumblupqW1XrI5+GBxA1obHru9WTLZ5riLmV6nJhU/JVCHatN28jzIa9rJcmSNMAN40fRjIeU\n+wHYnN73KoX2zD8jTU3L9iZbdyEQluwz8kAChfGjLwLIZ3r8BXAHsx+QgLdpmXni1sFUX8dk9Q3G\n+CyWSbaNU03cUnidtUztmSeAicca822tMK6dPGu27E+MHxrOZssKYySUCzO6zeUv43/SVSpSc0+n\ncG2OqWtjmVIqBOMMSSzGAati8nUxkpB4s8eHTP+WNP27D+MA9LOpS6hAOmN7CuOA8qNSyi7xD9iK\nkRA0Tbb+Op2Ba8jMVAV8gB+StXMb42CW1I5Sykcp9bVS6iLGNonFOBAWSt6dAmzTWl9Ppc3fkj1O\nvs0eJqWy5uUaACe11gcSF2itEzDOCqUlI+9VUhzaOKofSRZHU4xtcNFsvViMsxn1lNFN6YWx/Reb\nYkxc7wTGtm/2sGAftbwlZCKmE1rrjAxoaImx38VgnH2tA/TVWidebrAB4wyeedddf2CP1np/Rl4L\n/+37PyRb/gPGj4EmyZb/av7A1FXaGliutU5roM1lrfW/iQ+01hEYZ4kf9pnYCzQxdeU2V0o5pdGG\nuRDzzwjGDxuAP5KtdxzwUEq5mh43xUiGViVb7wfAFeP9AGMAyGat9Y3EFUz/35isXFuMM9O/Jzv+\n/IFx/GucgdeURBmXRKzG6DkYYPZUU+BIsm2tgR+B0kqpEqb3rTGwQmsdabbeVVP82f65ehxJ4iZS\n88C0G6YE5E+MX+KjMT7AdTE+2CkdGG8me5x4QbITgNb6KEa3gxNGN8pVZVzr1CiN2AqZ/g3ivyQp\n8QvLEePXq7nLadSXVjs/JmsnFuNLxxvAdDD9jf+6KJtjbJcPTeWTb5uHxfPQbZaGlMqalysKhKZQ\nLs1Rghl4r+K11rfTiMOLlLfBFcAWo6vRy7QstfW8Ulhu7lHLW0JGY8rofrsHY7/zw/jBUVRrvTDx\nSdOX8Dygm1Iqv1KqAkYXbGam9/AC4lL4AXLF7HlzyV+LC0bXfVA62kq+X8OD+1RyszG6FVthXCZx\nQym1WCW7XjQVyRPJmDSWJ8bhBVwxbWdzybdJUVL+zCVfVgijizOG+489J03PJz/OpUkp5YyRWN4D\nnjf9YEr0sM9l4vP5MI6xOelz9diRUaUiNckPPmD8ii6OcUo+IHGhUso+041ovRHYaPpF3BiYCqxT\nSpXUxii3lCT+Um2OcXo+uWvJm8lkeIntjMQ4m5dc4gXnlYEawAta6+WJTyqlXkil3szG86guY3yp\nJ1c4PYUz+V6l5CbGhfvJFcHoZgvH+ILiIeul9GWevI30lo/CuD7JXIa/FNMhIzFBxveTCPPPZSoW\nYnThv4wxZUg4xpnOjLoJ2CmlvM3PHPHfa0vrtdzDuEaseCbaTpPpTP9nwGems8PtgI/47xpNS7hJ\nyp+l5NvkcirrJV92A6Mr+elU2ruUkeBMZ8sWAWWB+imc6byJkVQmZx7/XYykObOfS5EF5IybyAgX\n079Jv9KUUoUwDoqPRGsdpbX+E+Pgmp+Hd4P8gfFF4KO1Dkjh78KjxmNyCAjBuM4opXYSJwZNabs4\nAi9lURxZZRdQQSlVM3GBMka9PjBq7GEy+F6lZAtGN1YxszjsMK632m2q/ybG9WBdTV84ieuVx5hL\n7W+z+qIxzt6Yx5iR8heAaslifCaFuB9oJyMyGJNFmGJYijHoqDfwvdY6M3MjbjH9m3yy6h4YXerb\n0ohDY3StvZDCpQRZSmt9XWv9HcZAnOTvc1baArgppZLvOz0wrnELND3eCbRUSiX9ODD9/8lk5dYD\nBTA2V0rHn4zOpzcJeBborLU+k0r8VZVSybdRd4xr3C6Z3rdtQGfTMS4x/oKm+P/OYEwiE+SMm8iI\nbRi/uOYrpaZgfGlPwDjF75PRypQxC31djANUEMYouHcwRj6mONUCGN12piHsX5oOMtswXeyOMers\nc50Fc75preOVUkMwrutyAVZg/AougjGy6qTWejZwECPBm6H+m/5jJP91peQUX2F0H61WSo3F6PoZ\nwH+JZ0JqBTP7XqViJsZ1kZuUUpMx9qlhGO9fP7P1xmFcG7VKKTUf8MA4W3QN42xKoqNAC6VUO4yu\n4FDT9XPpLb8EmKOU+gDjR0EdjJGcyR0F/qeU6ovxnt/TWh/J4GtPb0yWNAfYbfp/pu6CoLX+Rym1\nDphpusZrH0a36xvAfK31+XRU8zbGj4ldpm1/HmMfaKK17vewgmlRSv2EsW/uwThrVRVjNGt6rufM\nrJ8xPl/fK6XGAadNbXYB3jRLkD/ASJr/VEq9Z1qW+Hk0n5JlFcYlGL8rpT7C2MY2QBmMHxb9tNbp\nupOMMu7aMQGjCzlWKdXA7OnLph+7czGOB+uUUuMx9sm+GJ978wR9EkaC9odS6mOMs9XjMc6Wv5+e\neMQjssaICPmz/h9pjypNcfQcRjfDAYxujlMYw+KnA1Fm6ySOyBuXrGzyUZtNMH4FB2EkXiHAYsyG\nlJPCqFKz517BuAj5HsYv2iMYs9UXfVgcD9kmD0wHYhbn7xgH1iiM0Y8/YZriwbSOP8Yv6XsYXRjj\nMc5q3Ddak9RHQKY4go//RgI2SFZHSqNKfZKVve99MS2rgJGcRGIkOTNNsSYATg/ZNul5r6ZjXPeU\n0v50PNmyqqb6bpti2QG0TqFsB4wv3yiMkbwrgLLJ1qkO/GPa9jrZtklPeVuM5OkiRhL5G8Zgm+Sj\nSvNjfDnfMj13PLXtZVr/gVGlGYjpgek90mgro+tfwBggkp7PQ2rTgTib9p8gjB8pZzESEPOR5Ikj\nHrukUkcV0+u/adoep4B3zZ5fCRxOodx+YOVDYh+IMV3FdVO9p037p3Mar/mB9jCSaw2MSrb8ddPy\nAmbLvDF+IF01bZOjmKbbSFa2EUbyHG3a797AmOLoVrL17DGmFjpieh1hGAnc1MTXQvqmA0mMNaW/\nT83WK4VxRjbx/QgEnkuhvmYYl48kHnt/B2qkd/+Tv0f7U6Y3QQjxmFJK/YmR7Fa1dizC8pRS1THO\nGPbUWv9o7XiEEBkjXaVCPEaUUmMwfk2fwTiD9BLGyLs+1oxLWJ5SqgTGRMJTMc64/WzdiIQQmSGJ\nmxCPl1iM63BKYlwvcwzopY2Lt0XeNhhjGp/jQHedubkNhRBWJl2lQgghhBC5hEwHIoQQQgiRS0ji\nJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQ\nQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiR\nS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS0jiJoQQQgiRS9hZO4CsUKBAAV2qVClrhyGEEEII\nkabAwMDrWuuCmSmbJxK3UqVKERAQYO0whBBCCCHSpJS6kNmy0lUqhBBCCJFLSOImhBBCCJFLSOIm\nhBBCCJFLSOImhBBCCJFLZGvippRaqJQKVUodTuV5pZSapZQ6rZQ6qJSqk53xCSGEEELkZNl9xm0R\n0PYhzz8NlDf9vQbMzYaYhBBCCCFyhWxN3LTWW4GbD1mlE/CdNuwCPJRSRbMnOiGEEEKInC2nXeNW\nHLhk9jjItOwBSqnXlFIBSqmAa9euZUtwQgghhBDWlNMSt3TTWn+ptfbXWvsXLJipyYeFEEIIIXKV\nnJa4BQMlzB77mJYJIYQQQjz2clrithr4n2l0aQMgXGt92dpBCSGEEELkBNl6r1Kl1GKgOVBAKRUE\nTATsAbTW84B1QDvgNHAP6JOd8QkhhBBC5GTZmrhprV9K43kNDM6mcIQQQgghcpWc1lUqhBBCCCFS\nIYmbEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuIYmb\nEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuIYmbEEIIIUQuka33Ks2rtNbs27ePgIAAAnfvZF/AHq5e\nu050TAx2tra4ueajWrVq+NVvhJ+/P0888QRubm7WDjtXiY6O5sCBAwQGBhK4+yBhN8LRWuPhlZ86\n9arj5+dHrVq1cHZ2tnaoQgghhMUo477uuZu/v78OCAjI9nbDwsL4dtEi5s7+FGIieaKcJ34++fAr\n5UlxT2cc7WyIS9DcuhfDwUvhBFy4TcCluxy6eJNu3V5k4JDhVK9ePdvjzk1OnjzJ55/O5btvv8PT\nrgSFY/3wjqyFM16AIoowbjgd4KpDIDdiz9Kt20sMe2MQ1apVs3boQgghRIqUUoFaa/9MlZXELeOi\noqKYMnECc+fOoV3NYgxq5sMT5bxRSqWrfHBYJF9tPc+XWy9StVoN5i74hnLlylk46twlPDycYQPf\nYPXKtdSM60ut2NfwpNRDy9wmmP12C9hv/yXNWzdh/sLZFChQIHsCFkIIIdLpURI3ucYtg/bs2UOd\nGlU5sWUFR99twY/9atGofIF0J20AxT2dmdSpMhc+eJJnit+lQd06fPbpxyQkJFgw8txj06ZNVCpb\nnZO/ODAw8jQtYqelmbQB5Kc4TeMmMjDyNNc2+FC5XA3WrFlj+YCFEEKIbCKJWwZ89OEMOj79JBNb\nF2L5gDoU9Xi066ns7Wx4vU05dr7ViJ+//JgnWzQlPDw8i6LNnZYsWUqXDt158sbXtI2eiyMZvxbQ\nHmdaxcykY/jP9H5xAPPnfWWBSIUQQojsJ4lbOo1/522+nj2TgHFNeLF+iQydYUtL+SJubBndkMpO\nN2jZtBE3b97MsrpzkzVr1jDwleG8FPknZXnykesrSSN6RG5h7BtT+P67H7IgQiGEEMK6JHFLh+nT\npvLLj1+zZVRDfLxcLNKGrY3i85eq0by4pl2bVty5c8ci7eRUwcHB9Orely6RqylM1g3Y8KYcL0au\nZ8iA1zl16lSW1SuEEEJYgyRuafjzzz+Z/emHbBxRn4L5HS3allKKmV2rUMHlDq8PGWTRtnISrTW9\nu79K7ajB+FAvy+svRFUaRY+nR9c+xMfHZ3n9QgghRHaRxO0hIiIi6Nf7Zb56uTrFPLNnfjClFF90\nr8amDWtZv359trRpbWvWrOF4YBCN4t6xWBt1E4Zy8xT88IN0mQohhMi9ZDqQhxj4Wj9iTm3j6941\ns7zutGw6epU+3x/n0LGTuLu7Z3v72alZwzYU2NWLGvSwaDsn+Y3DlSZz4Ngei7YjhBA53ZEjR9ix\nYwe3bt3C3t6ewoUL065duzz/fZNTyHQgFnDmzBlWLF/Gx12rWKX9VlUK06K8O7M+/cQq7WeXkydP\ncuDAAarQxeJtlaMtly9eY+/evRZvSwghcpqYmBiWLl1Kszp1aFO/HjvHjOTqlAmcnTCWJYMHUqpo\nUfr3+h8HDhywdqjiIeSWV6mYN2c2fRqVxN3F3moxjGhVig5z5/D22HHY2eXNt+q3336jcsJz2GHZ\n6wcBbLClcvRLrF61lrp161q8PSGEyCmCgoJo17w5XtevMTQ2kk72NtiTYMoCEkBHc8Ves2DFMtqt\nWMFLfV5hxmefYWOTu8/vaK0JDQ3l7t271g4lOQelVBkgGristU73RK55Mxt4RJGRkSz6ZiG7325k\n1Thq+XpQ0tORNWvW0LlzZ6vGYik7twRSKLpltrVXJN6fXVsXZlt7QghhbZcuXaKxnx9DIyMYaatR\nDrYprlfERjHOBgYlxPPsd4voe/0aC39anKXTX2WnnTt38s033xAdHY27u3uOeh3169cv4+bm9uPd\nu3fV1atXI9zc3OZFRESsSE9ZSdxSsGbNGvxKeVGmkKu1Q6F/46Is+mpunk3cAgP20YbR2dZeMfzY\ndHBwtrUnhBDWFBUVRbvmzRkeeZs37BSQdvLiZaP4XcfS4rffmDZ5MmMnTbJ4nFnt+PHjzJkzh6FD\nh6Lj4gi7dYsKFStSqnTpHHEWcfPmzXEtW7a8DnD8+HGnIUOGvGNra3s7Pj5+Y1plrR99DrR75z80\nL5fxGfstoXmlQuzeG0BeGESSkuthV8iPT7a1lx8fbt6+mme3pxBCmFu6dCnFblxjRMon2VKVTymW\nq1g++vBDIiIiLBOcBa1atYq6/v6EXb+O960wGipN6MmTbNm8mdDQUGuHd59KlSpFvfPOO3dLliyZ\nrhF6krilIHD3Tvx8Pa0dBgAlvJxJiIsjODjY2qFYRFx8HDbZeOJXYQMauS+syDOio6MJDAxk0aJF\nzJkzh++++479+/cTGxtr7dBEDjDngw8YGheVqW7CkjaK5va2/JiLplGKj4/n66++Ys7nn+Pr4kIz\npSlnA+4K6qsEKsdEczgwgD07d+aoie4bNmx4WylVOT3rSuKWTEJCAvsOHqZOKQ9rhwIY87r5lSlI\nYGCgtUOxCEd7J2KJzLb24ojG1tYOW9sM/vwUIoc5duwYvfv1x7NAYZ7s0pvRX65nwrJ9jJy7lhad\nuuNRoDD9Bw3l9OnT1g5VWElgYCBXL13kabvMf9UPjoviixkzckUvxfbt26lXtSoLR42ksdLUdLDF\nySxfVUARG2imNN63bvLP1q0cPXw4R/zIcXZ21kC6RkPKNW7JhIeHY6PA29XyoxzTq4y3A5cuXbJ2\nGBZRxrc8148cw40i2dLedY7hW6x8trQlhCXExsYy+b2pfDJrDo6N++PxdiC2+Qs/sF5cWBA/7/yG\nH+s25J03R/Hm6FHyg+Uxs2XLFjqQgO0jXJTf0s6G8yHBhIeH4+GRM05opKTvS93YuGYNH+hYutnb\nMPQhL9kWKKvAB83xSxf5KyiIOv7+FChQINvifRRyxi2ZqKgonBysNwVISpztFZGR2XdWKjvVa+RH\nCNl3NjGEQOrWy9Sch0JYXXR0NE+178QXK7biPnI7+Z56M8WkDcDO04d87cbjNvwvPly0mude7E5c\nXFw2Ryys6VZYGN5xj3Y2SSmFt6MjYWFhWRSVZXyzdBlH7eJ5ycE23d3CjgpqKo1vfByhV65kuu3D\nhw/z1FNPUaBAgRTb7tmzJ0WKFCF//vxUqFCBBQsWZLotkMTtATY2NjnulHCCJs/+Un6iST2uuG7N\ntvYuO2+jUXOZw03kTj1692X/DXvyvboCW49i6SpjV6AU+fqvYtvJmwwYMtzCEYqcxM7OjrgsmAIj\nNkHnirlEXTJZ7lG/Xe3t7enatStff/11is+/9dZbnD17ltu3b7N69WrGjRvH8ePHM92sJG7JODs7\ncycyOkclb3diNM7O2XOv1OzWqVMnzsdv5zaWH3wRSRjH9Wq6du1q8baEyGorVqzgj617cOm+AGWb\nsV4BZe+E88uLWPrLav78808LRShymoKFCnHJ4dEu+4nUmrCYaLy8vLIoKuv5YNlyurw37b5lw+fN\nZ8r8+QQFBdG0aVPc3Nxo3bo1gwcPpmfPnknrfffdd/j6+uLt7c27775LqVKlkj5LFStWpG/fvlSt\nWjXFdqtVq4aLi5FWKqVQShEcHJzp/EsSt2Ty58+Pe343Lly/Z+1Qkhy5fJdKlSpZOwyLcHNzo3v3\n7vxr+6XF29qvFtHu6WcoVKiQxdsSIivFx8czZMRonF6YhXLI3I84G2d3HDvPpP/QETnqh6mwnA4d\nOrA6OpaIR3i/l8bE07xBQ/Lly5eFkVlHt2ZNWRcQSMQ94/s9Pj6eZVu306FZM8a8+Sb16tXjxo0b\nTJo0ie+//z6p3NGjRxk0aBA//vgjly9fJjw8PMMzPQwaNAgXFxcqVapE0aJFadiwYab7sCVxS4Ff\n7ZoEns8Z/flx8QkcOBdKnTp1rB2KxbwxZhj7HOZwiwsWa+MOoex2+oA3x71hsTaEsJQNGzYQ7eiF\nY7nGj1SPU9WnuR4RzT///JNFkYmczMfHh5bNmvNDbOanP5rj4MKgMWOyMCrr8S1ciDply/Lrjp0A\nbD5wEBdHRwp7eXH4yBGmTJmCg4MDjRs3pmPHjknlli9fTocOHWjcuDEODg5MmTIlw9OrzJkzh4iI\nCLZt28Zzzz2Hg4NDpl+HJG4p8G/QmMCL4dYOA4DjlyMoWqgg7u7u1g7FYipUqMDot9/g93z90GT9\nmQCN5g+XQfTt3ytPJ8Ai7/pl1VoSqj//yPUoGxtUjedZtWZtFkQlcoNBo0czy9aRyEycdfsrNp5Q\nB0eefvppC0RmHd1bNGPxFuO66p/+3kL3Fs24evMm7u7uSd2ZACVKlEj6f0hIyH2PXVxc8Pb2znDb\ntra2NG7cmKCgIH799ddMZ26SuKWgWfMWrD18I0d0J6w9cIVmzVtYOwyLe/Pt0biUus12u/eyvO4A\nmy+ILHiM996fnOV1C5EdduwJwKFk1vzosCtZh+278+a8kOJBLVu2xO/JNvTQdsRl4DvtRHwC3bU9\n8777Lk8NjnuhcWP+PniIoGvX+XXHTro3b04hLy/Cw8O5d++/S6TMp+AqWrQoQUFBSY8jIyO5ceNG\npmOIi4uoluZxAAAgAElEQVQjJCREBidkpWbNmhFj48Q/pzL/xmSF+ATNvK2XGDBkmFXjyA52dnb8\ntnElpwp+yw7bGVlWb6DNfAI8P2DDX2txcnLKsnqFyE5Xr1zG1qtkltRl5+XL5ZDLWVKXyPmUUnz9\n009E1fano7YnLCHt5G17XALN4+2YPns2bdu2zYYos09BD3ea16hOn08+pXThwlQuWYLihQpRtWpV\nJk2aRExMDDt37mTNmjVJZbp06cKaNWvYsWMHMTExTJo06b4TO1proqKiiImJAYxpxaKjowEIDQ1l\nyZIl3Llzh/j4eDZs2MDixYupU6eOXOOWlZRSDBzyOnO2WHfS298PXqZQ0eL4+z8e844VLVqUf/b8\nzVmfRax16kMktzJdVwx32eA4lH8LfsA/u/+mdOnSWRipENlLoSCLLiPQOiE99xkXeYijoyOrNm6k\nfLfulIlR9E+wZX/c/de9xWjNkph4muLAi/b5+Prnn+nVp4+VIs6cM+lISgG6N2/Gn//up3uLZmjg\nHvDB9Ons3LkTb29vxo0bx4svvoijozEit2rVqnz++ed069aNokWL4urqSqFChZKev3DhAs7Ozkmj\nSp2dnalYsSJg5BNz587Fx8cHT09PRo0axaeffkqTJk0yPamiJG6p6NW7NxsOX+VYyG2rtB+foJm2\n/hxDRoy2SvvW4uPjw75Du6n9kjNfu9TgKL+QQHy6yyeQwEnWscClBqU63ubgsUDKli2b4ThCQkL4\n448/+OWXX/j111/ZuHEjly/LWQphHcVLlCDu2rksqSv++llKlsyas3ci97C3t+ez+fM5du4cJUeO\noaOjGyXibKmBI5W0A4Wi4KtqtRi+YCHnr16lXbt21g45Qz6eMYOGCXa8Ga+4nUaX8MutWqJ/X0vf\n559nu7bhdj5XGjdpwrZt24iIiGDTpk2Eh4fj4+OTVKZ3795cvHiRGzduMGLECK5evZr0fKlSpdBa\n3/d3/vx5AAoWLMiWLVu4desWt2/f5tChQ7z66quP9Fpz/ox6VuLh4cG7U9+nz+fT+OfNJ7C1yd6f\nqJ//eQY7z+L3zSPzuHBzc+PLhXPo1rMLbwx5m78uvEHNqAGUT2hPASphm2y3TSCeG5zktPqdAy7z\n8CzswteffUb79u3T3WZsbCyrVq3im+8WsnfvXmJjYyldrSQu+Y2pF+7djuTsoQs4OjpSt25d+vbu\nR4cOHXLFpJQi93uinh+LLwbgVLHZI9cVfzGQJvX9siAqkRsVKVKEsRMm8NbYsQQFBREWFoaDgwMF\nCxakYMGC1g4v014fNYoXe/TgnREjqLRmDXUSUh/qFqnhmLLhhlJUrlaN4sWLExAQgJeXF6VLl+aP\nP/5g1apVvPXWW0ll1qxZQ6tWrdBaM2rUKKpXr06pUqWyLP6EhARI52l1+dZ5iAEDB7J86U98tOE0\nY57OvvtbnroSwXvrTrNr7z5sbB7fk6ItW7Zk/9HdBAYGMvvT+fy+6XmuXA+muEs1nDEmg4ziFsH3\nDuHtWYimzZrwy4hvadCgQbqHasfExPDhzA/5/ItZFCpdgOa969NhxkgKlvB6oA6tNdcu3uDItpNM\n+HAsg4cNYtiQ4Yx8YyT29jnrNmkib+n6fGcW9x6Cbv1GhqchMKcT4on7dxnPv78qC6MTuZGtrS2+\nvr74+vpaO5QsU7RoUb5ZsoS9e/fSq2dPtkXF4q7By/SRiQfOaDirFaV8falRoULSj+8rV67w3HPP\ncePGDXx8fJg7dy61a9dOqnvVqlW8/PLLaK3x9/dnyZIlj/RZTO727du2Sqno9KyrcsLIyUfl7++v\nAwICLFL3uXPnqOdXi2Wv1aZFZctP3Bp+L5bmM3fSZ9ibDBs+wuLt5Tbh4eEcOnSI27dvo7XGzc2N\n6tWr4+npmeG69u3bR8/ePclfwpkXJ7bHt5pP2oXMnD94iSWT1hJ5NZYfFv1AzZo1MxyDEOmhtaZM\nxWpEtJyEc7XMT81wb+9Sih/5ioMBu7IwOiFyngULFnDnzh2qVqqEt9YU1PGcwAYPb2+qVK9+39Qf\n1rB58+bwli1bbkt8vHLlSq9hw4YFXrx4Mc3RiI/v6Zx0Kl26NMtWrOTFL/9lx6nrFm3rdmQsz3y+\nl8ZPPcvQYa9btK3cyt3dncaNG9OuXTueeeYZmjZtmqmkbfbs2bRu24onhzdk9LL+GU7aAErVKMGb\nKwbQbIA/LZ5szrz58zJchxDpoZTiqzmfEf3LGyTcy9ygnfjboUSvGcuCObOyODohcp6nn36agIAA\n7kZFEe/tzcV8btSqVw//+vWtnrSZi4iIsFm1apXX5MmT7S5duvRNesrIGbd02rBhAz27vcDCXjXo\nUDt9N3fOiOCwSDp9EUC9lu2ZPXf+Y91FamkfzPiAz+d/xtg1QylcqkCW1HnlbChTO8xm1LDRjBgh\nd2cQljFgyHCW/rUfl1eWYeOY/lsQJdy7xd0vOzOw29O8/94UC0YoRM5x7tw5li5dyr59+4iMjMzS\nrs1H9e+//96zt7c/AcTEx8cHXLx4caHWel96ykrilgE7d+7kpReep1V5Nz7uWgV3l0e/rklrzbf/\nXGDMiuO8/sZo3h47LkftXHnNokXfMHbyO0zaOALvYhk/U/cw1y7dYNKTnzDz/Y/p0aNHltYtBBj3\nVuzZuy+//3MAp5e+xL5o5TTLxFz8l6gl/enWvhXzvpglxxfxWMppuY6NjU0gUFdnIjBJ3DIoIiKC\nMSNH8NuqFcx4riLP+/tgb5e5s2P7L9xi7KpThMQ4s+iHxXKNlIWdP3+eOnVrM+H31ylZJevPmoJx\n3dvUjrM5sO/gfUPJhcgqWmvmz/+S0W+Pw6FmJ+wa9sO+WNX7EjKtNbGX9hO3cwFxR9fzxWcf0aNH\nD0nahMghlFKBWutMTdKa7YmbUqot8BlgCyzQWk9P9rwnsBAoC0QBr2itDz+szuxM3BJt3ryZKePf\n5uSJE7zapAR9m/hS0jvtfvN70XH8EhjMnK0hXLoVzdDhbzBipIxKtDStNc1bN8e3eUGeHfmURdta\nPn0d1/ZE8MfvG+WLUljM5cuXmTNvPvO+Wsjdu/dwLVkNnNzQkbe5c/EQHh6eDO7fl/6vvZqrp3kQ\nIi/KNYmbUsoWOAk8CQQBe4GXtNZHzdb5ELijtZ6slKoEfKG1bvWweq2RuCU6cuQIc2fPYvHixTjZ\n2+BXugB+xZ0p7umIk70tsfEJhN+L5cDlSAIu3ObM5Zs0blCPQcNH0r59e5kHLJusWrWKURPfYNrW\n0djaWfa+e3Gxcbzd6ANmfzg3T92cWeRcISEhHDp0iHv37uHq6kqNGjUoXLiwtcMSQqQiNyVuDYFJ\nWuunTI/fBtBav2+2zm/AdK31NtPjM8ATWuurqdVrzcQtkdaac+fOsW/fPgL37iH0SjBRkZHY2zvg\nmt+d6rXq4OfnR/Xq1ZNukyGyT6s2Lan2YmmavdQgW9rb9O12zv52ld/Xrs+W9oQQQuQej5K4Zffp\nnuKA+Q1Ag4D6ydY5ADwHbFNK1QN8AR/gvsRNKfUa8BqQI27fopSiTJkylClThi5dulg7HGHm1KlT\n/Lv/X15dkn3vS+MX6vLT+HGcO3dO7pMqhBAiy+TEOSemAx5Kqf3AUOBfePBmlVrrL7XW/lprf7l+\nQzzM6tWradCpNg5O2XcdoaOLI/Xa12Lt2rXZ1qYQQoi8L7vPuAUDJcwe+5iWJdFa3wb6ACjjyu5z\nwNnsClDkPbsDdlGmWfaflS3tX4Lde3cxlKHZ3rYQmXXlyhV++OEHFi9dRkREBNWrVpHBDZlgZ2eH\nr68vrVq1ok6dOtYOR+Qh2Z247QXKK6VKYyRs3YDu5isopTyAe1rrGKAfsNWUzAmRKYH7Ahky4uVs\naevs/ouc3nee5t0bUra2LwvnLc+WdkXeFBcXx/Hjxzlx4gTR0dE4OjpSsWJFKlWqZJGBTadPn6ZX\n7974+fnTf+QEnD0Kwo1zuOd3pVLFCjlqxvmcLjY2lhMnTjB79my6detGmzZtrB2SyCOyNXHTWscp\npYYAGzCmA1motT6ilBpgen4eUBn4VimlgSNA3+yMUeQ9l85domg5y46wCw+9zU+TVxKw7iC+1X34\n9aP1vDS+IxfOXbBouyLviY+PZ/369Xw85wv+2bIFt2KFcatYFuXsiL4Xxe0TZ7lz+SqNmzfnjUGD\neeqpp7C1fbSR0lprfv75ZwYNHU6//oNp2WM4ysHZ9GRt4q+f4/TZc5QqWZIKFcrLaPh0Kl26NFWr\nVuWtt96iVatWj/w+CQHZf8YNrfU6YF2yZfPM/r8TqJDdcYm8KSEhgdjYOOwdLbOrx8bE8fvczfzy\n0e80e6khs/6dQj4PFw5sPso3Y5ahlebQoUNUr17dIu2LvGXXrl10f6U3kc4OFOzXhSbz3sHe3e2B\n9WLDI7i6ZhN9x72Jy+iRLP7mW+rVq5epNv/991/6DRrGmau3cS9Tl5pPv/xf0gagbLAtWBZbz+Jc\nvHqSi5v/okqlipQoUULmKUyHEiVK4OnpydmzZylfvry1wxF5QE4cnCBEllFKYWNjQ0J8QpbWq7Um\ncP1BRtSdxKEtx5n655v0+aAr+TyMrqSaLavw0a7x9JjcmWatmtJ/YH+uX7+epTGIvENrzdvjx9Gm\nU0fc3+xL7b++x6dHpxSTNgB7dzd8ej5Lrb++J/+oPrTu0J6xEydk6LY+oaGhvNynH41bPc350l1x\nfX0rNvaOuLjmT7mAnRO2xWtg41Obo6cvsGXrdm7evJmZl/vYcXd35+7du9YOQ+QRcr5b5GlKKbwK\nehJ2JZwCPl5ZUmfQ8cssemsZVy9cp88HL+LXNuWzabZ2trQb2JImL9Zn+bR1VKxSkQljJzBo0CC5\nU4ZIorWm38AB/Ba4m7o7luJYMP37qVKKos8/hVcTf75+aQTXrl9n/uwv0jwTFhoaSvlKVbHz64bH\nWwHYuHgY9ZnqfGibzh58POcrvD3cefmFjtSoWllu75YGOTMpspKccRN5Xs1aNTm7/+Ij13Mn7C4L\nRy9l/FMfUqt1VT7ZMzHVpM2cm1c++sx8gfHrhvHdmoVUrVmVDRs2PHI8Im94f8YM1u7dQfVfv8hQ\n0mbOsZA31X+dw6qd25gxc2aa64eGhmKbvxAund5PStoySjm5oTxKcOfOnUyVt4ZJkybRs2fP+5Yt\nW7aMJ554AhcXF5o3b/5Amfj4eMaNG0exYsVwc3Ojdu3a3Lp1C4Do6GhGjBhBsWLF8PT0ZNCgQcTG\nxmbHSxGPMTnjJvK8+n71OfXvIeq1r5Wp8vHxCfz5zTaWvrea+h1r82nAZNwL3t+FFXU3mkNbjhN8\nMJSYe6kfuMvXLM31AmH0H9QfTw9PGtRrgLu7e6biym729vZUrFiRVq1aUbRoUWuHkyccPnyY92fO\nwP+vH7DP7/pIddnnd6Xytx/wXsuX6di+PZUrV86iKPM2Ly8vXn/9dY4fP87mzZsfeH7ixIns2LGD\nnTt3UrJkSY4cOYKTkxMA06dPJyAggMOHDxMfH0+HDh147733mDx5cna/DPEYkcRN5HktWrTk51FL\neXFchwx3WRzacpxvxizD1dOFCatfp1SNEg+sExkRxc+T1lGxRBWeb/0i+VO5LslcQoLmZkgYN4Ju\nUaRIEXxL+ub47tPo6GgOHDjA6NGjmTBhAhUqyBiiR9V38CBKvTMQ55LFsqQ+F9/i+L7Vn1cGDWTn\nX39nSZ0Ap4/8y8dvvUbw+dPUa/40mH2O/vrrL7p168b58+epUqUK8+bNo0aNGgCUKlWKIUOG8N13\n33HhwgXatm3Lt99+i5OTE9evX6d3795s374dGxsbqlatypYtW7CxsSEkJIShQ4eydetWXF1dGTFi\nBMOGDQOMs2ZHjx7FycmJX3/9lZIlS/Ltt9/i72/cPSi1suvXr2fatGlorVm5ciVly5blwIEDtG7d\nGoAFCxY88LrDwsL49NNPOXDgAL6+vgBUq1Yt6fk1a9YwZswYvLyMM6XDhg3jzTfflMRNWJQkbiLP\na9myJTF34jm19xwV6pVJd7nP+n7N8Z2n+d+0LjToVCfVpG/fhsPUKFObASNey1hiWANio+MIOXmV\n0NBQ6vrXTfoCyKn8/Pzw9fXl22+/ZerUqdYOJ1c7cOAAJ06fokGvT7K0Xp/ez7Fz5tccPnz4viQj\ns2JjYpjUvwud+wyl0/8Gs2Pjat5/vSdd+4/mzImjTBw3jnXr1uHv788PP/xAx44dOXHiRNI9mZct\nW8b69etxcnKiUaNGLFq0iAEDBvDRRx/h4+PDtWvXAGNErVKKhIQEOnToQKdOnVi8eDFBQUG0bt2a\nihUr8tRTTwHG3VB++eUXvvnmG8aNG8eQIUPYtWvXQ8u2bduWd955h9OnT/PDDz+k67UfOnQIOzs7\nli9fzieffEL+/PkZPnw4gwcPTnF9rTVBQUGEh4fnmjPpIveRa9xEnmdjY8OQAUPYMG9rhsptXbKb\nTwMm0fBZv4cmZEH7r9K0VZNMXYBs72iHb/XiePl4EBoamuHy1tCkSRNOnDhBVFSUtUPJ1b74cj5F\nenXGJovnRLOxt6dor8588eX8LKnv2P7dxMXF8twrw7Gzt6dpu+epWMM4u/X7ymV07dqV+vXrY2tr\nS69evXB0dGTXrl1J5YcNG0axYsXw8vKiQ4cO7N+/HzC63i9fvsyFCxewt7enSRPjM7R3716uXbvG\nhAkTcHBwoEyZMrz66qssWbIkqc7GjRvTrl07bG1tefnllzlw4ABAuspmRGISdvLkSc6dO8fy5cuZ\nNGkSGzduBKBt27Z89tlnXLt2jStXrjBr1iwA7t27l6n2hEgPSdzEY6Ffv34c23aaI9tOZKicg7ND\nmutERkTj4ZW5C7wT2djknlFnDg4OuLi4yPQGj+ivbVvxerKRRer2atOYv7Zl7IdKam5cDaFA4WL3\n/TApVMzoNgy9EsKiRYvw8PBI+rt06RIhISFJ6xYpUiTp/y4uLkmDGUaPHk25cuVo06YNZcqUYfr0\n6QBcuHCBkJCQ++qcNm0aV69eTbXOqKgo4uLi0lU2I5ydjfnsJkyYgLOzMzVq1KBbt26sW2dMRTp2\n7Fhq165NrVq1eOKJJ3j22Wext7encGHLTvgtHm+SuInHgoeHB1/NW8C8gT8RdTc6y+t/3Ib7P26v\nN6tFRkZy8fQZ3KpZ5jpBt6rlOXfiZJacFfUuVJTrV0PumyMu9LIxSrtg4aL079+fW7duJf3du3eP\nl156Ke0Y3dz46KOPOHv2LKtXr+bjjz9m06ZNlChRgtKlS99XZ0RERFKy9DBplc3ofpt4rZ55OfP/\nOzs7M3v2bIKDgzl79ize3t74+flhYyNfrcJyZO8Sj40OHTrQokkL5g/6kfgsnpA3pzl//jxKKeLi\n4pKW/fXXX1SvXh0PDw+8vb3p3LkzwcHBD5S9efMmBQsWpHHjxtkZ8mMlODgY18IFsXVM+4xuZtg6\nO5GvgDeXL19+5Loq126Ara0dKxd9TlxsLNvX/8qJA3sBeKrTCyxdupTdu3ejtebu3bv89ttvRERE\npFnv2rVrOX36NFpr3N3dsbW1xcbGhnr16uHm5sYHH3xAZGQk8fHxHD58mL1796ZZZ1plCxcuzPnz\n50lI+O/zHx8fn3TGLiEhgaioqKQpPcqWLUuTJk2YOnUq0dHRHDt2jCVLltC+fXvAeB9DQoykdteu\nXbz77rsyMEFYnCRu4rHy1bwFJFyz5cvB2Ze8mSdP1lSlShXWrVtHWFgYISEhlC9fnoEDBz6w3ptv\nvilTSVhYXFwcNhYeRWxjb5clc4rZOzgwce4y/ljxHc/XKcSW336m8VOdAahQuRpTpkxhyJAheHp6\nUq5cORYtWpSuek+dOkXr1q1xdXWlYcOGDBo0iBYtWmBra8vatWvZv38/pUuXpkCBAvTr14/w8PA0\n60yr7AsvvACAt7c3derUAeD777/H2dmZgQMHsm3bNpydnXn11VeT6ly8eDEXLlzA29ubZ555hnff\nfZdWrVoBcObMGZ544gny5ctHr169mD59utxMXlicjCoVjxVnZ2d+X7ueDs+258MX5tF/Tg88i2T9\n6K96ZRvxv/49+XXxKs6cOMv2438xaeS77Nq2h3yuLrw6vC/9hvYB4N89+xn16ltcPH+JfPny0aNH\nDz7++GPOnz9P6dKlWbRoEePHj+fevXuMGDGCsWPHAsZ9WGfMmMFXX33FrVu3aNWqFfPmzcPLy4um\nTZsCRhcxwMaNG2nYsOF9Mdra2nL69On7lu3YsYPDhw/z2muv8fXXX2f5dhEGV1dXYm6nfVbqUURH\n3MHVNeW54ZRSxNwJw+n2VWzzp309VoUa/sxdG3D/Qp1AbPAhmjZtel+iY+78+fP3PZ40aVLS/0eM\nGMGIESNSLFesWDEWL16c4nPmdYAx5Yh5N+7Dynp7e7N9+/b7lvXu3ZvevXunuD5A8eLFWb9+fYrP\nNW3a9IHXKISlyRk38dhxdXVlw7o/aO3XljENp7F1ye4M3eMxvVYuXc13qxdy9Np++nbpT5Ualdl3\ncRfL/viJBbMW8veGLQBMGDGZbi+/SEBAAGfOnKFr16731bN9+3ZOnDjBpk2bmDJlCseOHQPg888/\nZ+XKlWzZsoWQkBA8PT2TpinYutW4MP3WrVvcuXMnKWm7ePEiHh4eODs7M3PmTMaMGZPUTnx8PEOG\nDGH27NlyDZuFFS9enPjoGKKvWeZen1FXrqHiE1KdKLly5cq82rsn4TPqc2/TJ+jYjF33mXDnOnFn\n/yG/bQwlSjw4t6EQwnIkcROPJQcHB6ZNfZ8/1m1k0+e7GFV3Kuvm/cXd8Kwbxt93SB+KlyjG8cMn\nuHHtJm+MH46DgwO+ZUrSo283Vi5bA4CdvT1BF4MICwvD1dWVBg0a3FfPxIkTcXZ2pmbNmtSsWTNp\n6oN58+YxdepUfHx8cHR0ZNKkSSxfvvyhXbMlS5bk1q1bXL9+nffee49KlSolPTdr1izq16+Pn59f\nlm0DkTKlFNXr1CY88LBF6g/fd4TqdWqnmoDb2NjwycwZ7N+7kzrRe7n9YT0iD65FKxsS4uNTrVfH\n3CX+UiA2V49Su1plGjVsgIuLi0VeQ14SHx+Pra2ttcMQeYR0lYrHmp+fHwf3HWLr1q18NvszBkx8\nB99KPpSq/eg3zS7mY5ztCLoYzNWQq1Ty/u++pvHxCdRvXBeAj776gCkjp9GuXTvKlSvHxIkTky5+\nhtSnU7hw4QKdO3e+bwSbra1tuqY+8PLyolevXtSsWZPg4GBCQ0OZNWsWgYGBj/aiRbq91Kkznyz9\nnUJtm2Z53TeXrmNkp85prle+fHk2rlvNH3/8Qf8hI4jRNgSfOkgFvyb3rxgfR/z10yTcCqZ8ubKU\nKe0viUg6JU5TUqxY1twdQwhJ3MRjTylFs2bNaNasGREREezfv5/AwEB+58H7Fma0XjASuJKlS/DP\n8b9TXK9M+dJM++g9nBNcOXr0KF26dOHGjRtp1l+iRAkWLlxIo0YPzgV24cKFNMvHxcURGhrK7du3\n2bNnD5cvX6ZKlSqAMV1FZGQkRYoUITg4WL6kLaBXr168M3ECpS+H4lS0UJbVGxl8lWtb9vC/b39O\nd5k2bdpw8sh+hgwZwoq5Uxn2znvk860BtvbEhwWRcO0URQoVokrzZkn36RRp01qzZMkSSpcujbe3\nt7XDEXmEJG5CmHFzc6NJkyY0adIk1QunM6p2vVrkc8vH7Blz6Tu0Dw4O9pw6dpqoyChq1a3Jih9/\npVK5yuTzyp80mCA980ANGDCAsWPH8u233+Lr68u1a9fYsWMHnTp1omDBgtjY2HD27Nmke4r+8ssv\nVK1alfLly3Pjxg3eeOMNateujZeXF08//fR9F1kvXbqUn376iVWrVknSZiHu7u70f+01Vrz9EVUW\nfZAldWqtOfvWhwzs35/8+fNnqKy9vT3z5s1j1qxZTHt7OAUKFsTV3RMnezvKlPbF1dWVzZv+zJI4\nHwexsbGcOHECb29vxo8fb+1wRB4iiZsQFmZra8t3qxYyefR7NCjXmJjoGMpWKMOYKaMA+GvDFiaM\nmEJ0VDSlSpViyZIlSTO2P8zw4cPRWtOmTRtCQkIoVKgQL774Ip06dcLFxYWxY8fSqFEjYmNjWb9+\nPcHBwYwcOZLQ0FDc3Nxo3rw5v/76KwCOjo73dcm6u7tjb29/3zKR9d6bNJkVtWsSsnw9xbq0feT6\nLi//HbvTQby7dE2myiulGD58OK+88gpr1qzh/PnzNGzYUAarZIKdnR09e/aUwRsiyylLjKbLbv7+\n/jogICDtFYXIAKUUy+/MT/NLa9HwX5g4bmLSNW2ZEXLqKvnIf99ggZysV69efPzxx9L9kwX27dtH\ni7ZtqLhgGgWa1890Pdc37+TEa+P4e8NGateunYURCiGymlIqUGvtn5myMqpUCCGsqE6dOqxZ/gsn\n+r3DpYXLMzw1jdaaS1//zInXxvPbLyslaRMij5PETYhHpGzUQ6dQyIvi4+PlfoxZqGnTpuz8eyv8\nsI5DnQcRvv9YusqF/3uUg50GYrN4A7u2bJXblAnxGJBr3IRIha2tLSd2n6VSg7IPXS9/oXwEXQjG\nxzdzU4gkxCUQGRGFa/6sv4ODJYSFhREXF4ebm5u1Q8lTqlSpwr+7dvPprFnMfHk0NkUK4N6xJflr\nVca1UllsnRyJj4rmzrEz3N5/lFurNqGvhTF6+AiGDx2KnZ0czoV4HMg1bkKkYsmSJYwY9TqVmpSj\n+5SOeBf3THG9Q1uOc/b3q0x4bywu+TIwGamGm5dvEXz8Kl5eXlSvVh1HR8csit4ytNbMmTOH2NhY\nXn/9dWuHk2fFx8ezbt061v3xBzsD93L25Clio6Kwd3KiTIXyNPSrS7s2bWjXrp2M+hUiF3qUa9wk\nccsGN2/eZOYH71O6bDle6dtPDrS5yJ07d5j2/lTmzJvLM0Na0H5YaxydHe5bR2vN5m92cHZ7MLXr\n1EZeh5EAACAASURBVMbT3QN4+ICGqHvR3AgKA63wLeGbK85eRUVFcfDgQTw8PJg4cSL58uWzdkhC\nCJErSeKWQxO3uLg4vpw/n0kTxtKpZmFOXL3HbZWPz76YT7NmzawdnsiAc+fOMWLUCHYH7KLH1E40\n7Oz3wGjT8GsRnD94iah7qd/38W7YPbYt2cv5A0G83ONlWrZsmWuuFXNwcKBChQqULVtWpocQQohH\nIIlbDkzcNm3axOtDBlDALppPu1aiZkkPtNb8vDeI0StOUK9hYz78ZBalSpWydqgiA/7++28GDx+M\nXX7434znKV0zfXM0xUbHsu6Lzaz+9E9e6dOXieMnZniCVCGEEHmDTAeSg5w9e5bnOj5Dv54vMLl1\nQTaPrE/NksZs+EoputYrwbEpzalucx6/WtUZP/Zt7t69a92gRbo1b96cg/sOMrDHUKZ3nsNXQxcT\nfi0i1fW11uxZu5+R/u9xbfdd9uzcy0cffiRJmxBCiEyRxC2LRERE8PaY0dStUxN/x2COTWnBc/4+\nKXYpuTjaMaFjJfZPaMbZrcupVK40P/74Y4bnbxLWYWtry4ABAzh1/DQV81dnpP+7rP38T2Jj4u5b\n7+LREKZ1nM2vk/9k4dxFrFvzO+XLl7dS1EIIIfIC6Sp9RAkJCXz/3Xe889ZoWlf05P3OFSnmmfbt\nisz9c+o6w5cew96jCJ99MZ969epZKFphCcePH2fYiKEcP3OMl6c/R4V6ZVg+9Td2rNjHxHETGThw\nIPb29tYOUwghRA4h17hZKXHbuXMnwwf3x+beDT7rWon6ZTN/+5+EBM23/1xg7MoTtHn6Gd6f8RFF\ni2b+Fkoi+61bt45hI4ZyOeQK//vfy7w3ZarcEkoIIcQDJHHL5sQtKCiIt0aN4K9NG5neuSI9GpbE\nxiZrRtndjoxl2rpTLNh2kZGjxjBi5CicnJyypG5heTExMVy/fp1ixYpZOxQhhBA5lCRu2ZS4RUZG\n8tGHM/jk45kMbF6Kt54uj6vTw2cr11pz+uodzl+/R1xCQrrbCgmLYsH2S5wLi+XVAYNo2LChTMGQ\nAU5OTlSrVo2CBQtaOxQhhBDiPpK4ZUPitnPnTl564Tn8S7jw4fOVKF0w7clHb96JYeK688Q6F6BS\nxYrY2zukWSa58MhYLt6MxN7BiXIVKshtbdLpzp07HDhwgHr16jF06FCZ9FgIIUSO8SiJm2QB6bTk\nx+/5n58nU56rlu4yMzddolHbLrzYqe0jnS1L0JodZ8KoULUmhQoVynQ9j5uYmBgmTZrEqlWreO65\n56wdjhBWFx0dzceffsrRkyd4b8JEfH19rR2SECKDZDqQDPB2Tf99JG/eifk/e/ce32P9/3H88d7H\nNjuxYTYbNrQcQyySU0kl53IWklMSUSpKSSVC9S2HftLBIjkUlXzFt8ihcq6lITkOm3LcbNj5+v0x\nfTK2+WD7bNPzfrvtdvtc1/t9vd+v66N47breBw4m2ujc9p7rfsXpYgzFbPqjulpubm707NmTtWvX\nFnQoIgXKsiyWLl1KlZrVmbl2JZtLFaNWvVt5/sUXtY6kSBGjJ2755M/4JILLldOrzQJWqVIl/vzz\nz4IOQ6TA7Nixg8dGDGfHgb2UatUMrzQL60Q8ofc0Y+F/l/LpvHnUqFaNoKAgjaPNgTEGf39/GjVq\nRKtWrfT3uhQo/deXTzIsS/9zFwI2m4309PSCDkPE6U6dOsVzY1/k04ULCBrUjXJlvHmwfmNuaXgb\nnt7/jNFNjU/g/J5oirvYqF6tGiVLlizAqAunjIwMYmJi+PLLL9mxYwfPPvusklwpMMosRERuIGlp\nafzfzJm8+MrLlGl/Nw03L+bkkm95oHo493XrdPkF5QKhWhhJR/7kwJ6DBJQtS41q1bUM0SWCg4O5\n9dZbGTJkCPv27eOmm24q6JDkX0oDp8TpQkND+e6777KcGzRoEFWrVsXFxYWIiAiH2omKiuK+++6j\nTJky+u1XBFi1ahXV6tbm9c8+4Zav/o+b33oOt9J+pP+2h7p3NMz12uLlA/FtehtxxQzfr13Dnr17\n9bT6Eq6urjRs2JDIyMiCDkX+xZS4SaFQp04d3n33XerVq+fwNa6urnTt2pUPP/wwHyMTKRqeGfM8\nXQY8QonnBlJ76Ux8av6zL651LgmvEj5XbMOlmA2vqpUoeXtdok8eZ826tUreLlGiRAlN6JACpcSt\nAEx6N4Lg2+7Hp3ozqt75IKt+2Mz5pCT6PjUOv1p3UaNFF6bMnEP5Bq3t1zRq3pIDBw7Yj/v27csL\nL7xgP162bBl169bF19eXO+64g+3bt9vLYmNj6dSpE/7+/lSqVImpU6fay3x9ffH29sbb2xsvLy+M\nMRw8ePCKbYaGhvLGG29Qu3ZtSpYsSbdu3UhKSrpiPL179+bQoUO0a9cOb29vJk+eDMDjjz/O3Xff\nfVWvZ6pWrUr//v2pWbOmw9eI3Kg2bt1ClTdGEdiuRbZPoC8916PO7Wxbuz7btmyeHvjUq0FycorT\nErfBgwfz6quvOqWv66Gn+1LQlLg52e59B5n+8Wds+XoOCbvWsXLudELLB/Hyf95nX/QR9v3wJSs/\nmcbHny9zuM1ffvmFfv368d5773Hy5EkeffRR2rdvT3JyMhkZGbRr1446deoQExPDqlWrePvtt1m5\nciUAcXFxJCYmkpiYyPDhw2natCnBwcG5tvm3RYsWsWLFCg4cOMD27dvtrzhzu3bu3LlUrFiRr7/+\nmsTERJ599tk8/X5F/tWuIqmY/+tG6jdvmo/B5CwiIoImTZpkOTdz5kxefPHFAoln586dhIeH4+fn\nh5+fHy1btmTnzp0FEovIlShxczKbzUZySgo79+wnNTWN0ApBVAktz6Jl3zFmWD9K+ZakQlAgTzzS\n3eE2Z82axaOPPkrDhg2x2Ww8/PDDuLu7s3HjRrZs2cLx48cZO3Ysbm5uVK5cmYEDB7JgwYIsbSxc\nuJBPP/2UxYsX4+rqmmubf3viiScICgqiVKlStGvXzj7uw5FrRUQKi6CgIBYuXMiJEyc4ceIE7du3\np3t3x/8OFnEmJW5OdlNoBd5+aSTj/jOLsrfeQ/fHnyP2z+PEHjtOhXIB9noh5cs53GZ0dDRvvvkm\nvr6+9p/Dhw8TGxtLdHQ0sbGxWcomTJjAX3/9Zb/+l19+YejQoXzxxRf2vT1za/NvgYGB9s+enp4k\nJiY6fK2IFKyOYbewedUa3n9lImN69OXlRx6lRany9KhzO7u2/WKvd/jwYR588EH8/f0pXbo0Q4cO\ntZd99NFHVK9eHT8/P+677z6io6PtZcYYpk6dSuXKlSlTpgzPPPMMGRkZ7Nq1i8GDB7Nhwwa8vb3x\n9fUFLh/+8f7773PTTTdRqlQp2rdvn+XvD2MMM2fOJCwsDF9fXx5//HFy274xOTkZX19foqKi7OeO\nHz+Oh4cHx44dw9fXlypVqmCz2bAsC5vNxt69e6/vCxbJJ0rcCkDPjq34YcmHRG/4GmMMoyZOpVzZ\nMhw++k8ydSgm66KxxYsX59y5c/bjixeVrVChAmPGjCEuLs7+c+7cOXr06EGFChWoVKlSlrKEhASW\nL18OwLFjx+jYsSMzZszg1ltvdajNK7nStRojIlK4rF/2Dfd07cS3x6Np2vZ+3hj+DADp6ek88MAD\nhISEcPDgQWJiYuxPor766ismTJjAkiVLOH78OE2bNr3s74cvvviCrVu38vPPP/PVV1/ZE72ZM2fS\nqFEjEhMTiYuLuyye1atX89xzz7Fo0SKOHj1KSEjIZU/Ali1bxpYtW9i+fTuLFi2yD//Ijru7Ow8+\n+CDz58+3n1u0aBHNmzfPso2gr68vxYsXZ9iwYTz//PNX/0WKOIHTEzdjTCtjzG5jzF5jzOhsyksa\nY742xvxqjNlhjHnE2THmp937DrL6xy0kJ6dQ3N0dj+LuuLi40LVNSybOiOB03BmOHP2LaRELs1wX\ndlMVlixZQnp6OitWrMiyjdPAgQOZOXMmmzZtwrIszp49y3//+18SEhJo0KABPj4+TJo0ifPnz5Oe\nnk5UVBRbtmwhLS2Nzp0706tXL7p27Zqlv9zavJIrXRsQEMD+/fuzXJOSkkJSUhKWZZGamkpSUhIZ\nGRm59mNZFklJSaSkpACQlJSUZQyeiDimduPbueP+e7HZbNz/UDf2bs98MvXHH39w9OhRpkyZgpeX\nF8WLF7ePTZs5cybPPfcc1atXp1ixYjz//PNERkZmeeo2atQoSpUqRcWKFRkxYkSWxCk38+bNo1+/\nftSrVw93d3cmTpzIhg0b7BOnAEaPHo2vry8VK1bkrrvuuuISHT179swyROTTTz+lZ8+eWerExcUR\nHx/P9OnTs/wiK1KYODVxM8bYgBnA/UANoIcxpsYl1R4HdlqWVQe4E3jTGOPmzDjzU3JKKqNfn0aZ\nui0JrH8fx06cZuKoobz05CBCggOp1KQ99z40lN4Pts5y3ZPDhvDtt9/i6+vLvHnz6Nixo70sPDyc\n999/n6FDh+Ln58dNN91knyhgs9lYtmwZkZGRVKpUiTJlyjBgwADi4+M5cuQI69ev5+2337bPLPX2\n9ubQoUO5tnklV7r2ueeeY/z48fj6+vLGG28AcO+99+Lh4cFPP/3EoEGD8PDwYN26dbn2Ex0djYeH\nh31WqYeHB1WrVnUoRhH5R+mAf4ZpuHt6kJyURFpaGieOH6dixYrZ7gITHR3N8OHD7cMhSpUqhWVZ\nxMTE2OtUqFDB/jkkJMTh4RKxsbGEhITYj729vSldunSWtnMaqpGTu+66i3PnzrFp0yYOHjxIZGQk\nDzzwwGX1vLy8GDx4MH369OHYsWMOxSviTM7eOaEBsNeyrP0AxpgFQAfg4uk7FuBjMt+neQOngDQn\nx5lvalcPY/PXc7Itm/P2K/bPazZszVJWvVpV1q1bl+Wx/sVatWpFq1atsi0LCgrK8Tfd3MaF5Nbm\nxb/5AowbN87hazt06ECHDh2ynFuzZk2OceQkNDQ01/hF5PqU8ffn8OHDpKWlXZa8/T0k4qGHHsrx\n+sOHD9t/sTp06BBBQUHAlYdLBAUFZXlyd/bsWU6ePElwcPC13go2m42uXbsyf/58AgICaNu2LT4+\n2a9tl5GRwblz54iJicnx71yRguLsV6XBwOGLjo9cOHex6UB1IBb4DRhuWdZl78yMMYOMMVuNMVuP\nHz+eX/Fel2zClkJk48aNNL6tBQ8/NEAb0Ytk4+abbyYwMJDRo0dz9uxZkpKS+PHHH4HMddcmTpzI\njh07AIiPj+ezzz7Lcv2UKVM4ffo0hw8f5p133qFbt25A5nCJI0eO2Ic5XKpHjx7Mnj2byMhIkpOT\nef7552nYsCGhoaHXdT89e/Zk4cKFzJs3L8tr0m+//ZZffvmF9PR0zpw5w1NPPYWfnx/Vq1e/rv5E\n8kNhnJxwHxAJBAF1genGmBKXVrIsa5ZlWeGWZYX/PROyMPF2L8bpuPiCDqPIu//++7O8xv37Z8KE\nCQ5df+rUqct+q46JiaF7p960ubsz/lt7c+CzUlSrUotJE6dojJzIRWw2G0uWLGHv3r1UrFiR8uXL\ns3Bh5vjbBx54gFGjRtG9e3dKlChBrVq1+Oabb7Jc36FDB+rXr0/dunVp06YN/fv3B6BFixbUrFmT\nwMBAypQpc1m/LVu25NVXX6VTp06UK1eOffv2XbaE0bVo2LAhXl5exMbGcv/999vPx8XF0aNHD0qW\nLEmVKlXYt28fK1as0H6tUigZZ75qMsY0AsZZlnXfhePnACzLmnhRnf8Cr1uWtf7C8WpgtGVZm3Nq\nNzw83Nq6dWtOxXli+NAhVD69geH3hl25MpmvIB+Zt4fnXxjLzZVDrnzBFWw8EEflarfosf1V+vTT\nTzl16hRDhw7l/PnzTJn0Jm9NeZtbUwfTKHU07ngDcJI9fO85koSSu5g6803atWun2a9SpDS97x6S\nHn0A/5aNLys7POBF3vm/GXiVuOx34FydXLWBli1a4OZ29cOMjTHs2bPnhtuM/bPPPuPcuXM8/PDD\nBR2KFGHGmG2WZYVfy7XOHuO2BQgzxlQCYoDuQM9L6hwC7gbWG2MCgKrAfgpYpcpVmP7mIm4q602b\nuldeY80Yw4AGpRj/+hTat+9AjZur4Op69V93umVx5NR5ok+ex92nFPHxeop3JX/PZN20aRObNm3i\ntdde4/PPP2fEkGcoc7Y+fc9twY9KWa4pTRidzy1l77mVPNbzSd6qM50Zs/6j7bSkyHArVoyj67dS\nunkDXFxdr7u91LgzWBnp+gVGpJBxauJmWVaaMWYosBKwAR9ZlrXDGDP4QvlM4FUgwhjzG2CAUZZl\nnXBmnNkZ8dRIbq5WnSeHDWH62sO81aUa1YNy/+21yc1lKON9hlWbvmDztxZpVzHkzcIi5vR5fj6U\nQElfP26tH84ObcHisOLFi3PLLbfQp08fOrfvyZHfT9Py7EdU4i57nXTSOM8pMki1nytLLbqcXc5v\nG+bSKLwZHR5sz6jnRuLn51cQt5Ejd3d3SpcurX9UxW7Gm/9hwOND2NakB5UmjqRMi0b/FNpspKU5\ntudoelIK5/84QNqpeOrWqYtrHiSBzjJ48GA++eSTy8736tWLmTNn5kkf2U3UEHEmp74qzS/OeFX6\nt5SUFGZMn8qE8a/S6/byjG0bhp9X3q5WsiMmnhGLfifmXDH+M+1d7rvvvjxt/9/g+PHjjH76BRZ/\n9iVNk16hnjUAF2xAZsL2m/tH/FV6Dd6+xXEtlv0/TBmkkuByhPPmFOUrlCcgoGyhSZSSkpJwcXGh\nffv2dOzYsdDEJQXLsiyWLl3KkKdGUKxaJULHP4lXlYocfXkGwzp1p+Zt9XO+Nj2D8wePkBQdQ2hI\nKGE33aQEJRuvvPIKzZo148477yzoUKQIu55XpUrcrtGxY8d48flRfLlkMa90uJkBzSphc7m+fzxP\nJabw0tLdLNx6lBdfepnBjw0pUr/tFgYpKSlMfWc6r708kZppvWiSPBYPsj4t2+YxlaB74ujbdQh+\nJS4fGH1Zm5zhuG0HuCdzS+2aFJbJMNHR0UyZMoVWrVrRtm3bgg5HCpHk5GTeevttJk6ZTLleHfBq\nUJtyP+1g5Mtjsx3nlvTncc7vPkApXz9q1aiBp6dnAURd+G3evJl33nmHWbNm4eXlVdDhSBGmxK0A\nEre/RUZGMmLoYE7HHuSdrtW4s/rVTx5IS8/gvTUHeHnZHrp07cbL4ydmO9NKcvfLL7/QqV13isdV\npsXZ/+BPtcvqpHCONRUeYfqbH+FZ/Gr+4rVI5E9O2nZS0q8E4Q3qYbPZ8i74a7R7927eeecd3n33\n3YIORQqhP//8k5HPjebrb5ZTplUzvM+c56awMHxK+GCMIf3seZIOHaVYegaVQkMpWbJkQYdcKKWn\np3PkyBHOnz/P6NGjtdC3XLeiNDnhhlO3bl2+X7+BxYsX03fEMMIrHGZKp2pU8ncsKfhux1+MWPQ7\nASE3sWrtj9xyyy35HPGNa8H8RQTEtKEVb+VYJ55oypcvf5VJG4DBm3J4pZfl8Ol1nDlzplCMe7v5\n5pv566+/SEpK0tIFcpnAwEDmzY5g69atDHpiGDFnz2DdfRtuZUrxZ8QSEjb9yqMDBtK2TZtC8YtI\nYWWMwd/fn7CwMA1LkAKnxC0PGGPo3Lkzbdq04c0pk7nttTcZ3DyE0feH4V08+69437FERn72O7/9\nlcKb77xHhw4d9BfC9bLAi9xfY6aTipub+zV3YbBhcyk8r6+NMbi5uZGamqrETXIUHh7Oth9/YsGC\nBQx/9mkSExIZ0K8fr8z/El9f34IOT0SugkML8BplFA7x8PDghbEv8WvULqI9qlPtxdV88tMhMjL+\neR2dcD6V0Z/vpOGEH7i9Y3927N6rweWF3JJVEfQc1aSgwxC5LsYYevTowcHde9i7cxdT3/qPkjaR\nIsjRnROijTEvGmOC8jWaG0RwcDBz5y/is6++Yermc9wx6Sc27j1JxPqDVH1hNX/51ua3nbsZ/fwY\nPSURh4wbN45evXoVdBhyA/D09LTvGSoiRY+jr0pXA6OBF40xy4CZlmX9L//CujE0atSIjdsi+WTu\nXB545ilCQyry1TffcdtttxV0aCIiIlIEOfTEzbKsvmTuHfo0cDOwwhizzxgzyhhTONZGKKRcXFzo\n8/DDxP51gp82/6ykrRCb9fnr3DOoCvW6+dDm8Rp8u+GLbOsdOhzNAw88QKlSpahatSqLFi0CMpci\nqVu3LtOmTQMyZ6I1btyYV155Bch8ata1a1f69OmDj48PNWvW5OLZ0LGxsXTq1Al/f38qVarE1KlT\nAVixYgUTJkxg4cKFeHt7U6dOnfz8GkREpBBzeJN5y7LiLcuaallWLaA58BMwDjhsjFlgjLkzf0K8\nMRhjNI6tkKsYWIVPJq5n6/x4Hu/+Es++1Ytjp45mqXMu6SzPPv8UnTt35tixYyxYsIAhQ4awc+dO\n3Nzc+OSTTxg7diy7du3i9ddfJz09nTFjxtivX7p0Kd27dycuLo727dszdOhQADIyMmjXrh116tQh\nJiaGVatW8fbbb7Ny5UpatWrF888/T7du3UhMTOTXX3916vciIiKFh8OJ2yV+BL4AIgE3oB2wyhiz\n2RhTPa+CE3GmVk26EFA6CBcXF1o37UZIUBi//bE5S501W5YREBDIQw89RLFixbj11lvp1KkTn332\nGQC1atXihRdeoGPHjrzxxhvMnTs3yzILTZo0oXXr1thsNnr37m1PwrZs2cLx48cZO3Ysbm5uVK5c\nmYEDB7JgwQLnfQEiIlLoXdVyIMaYCsBAoB8QCHwLdAD+S+bG8G8BHwMN8jZMkfz35eo5RHz1FjHH\nDgJw7nwip8+cwOWixCv2WDS/795JaGio/QlqWloavXv3ttd5+OGHGTNmDJ06dSIsLCxLH4GBgfbP\nnp6eJCUlkZaWRnR0NLGxsVlm+aWnp9O0adP8uFURESmiHErcjDHtgEeB+4B4YDbwf5Zl7b+o2rfG\nmKfITOJEipSYY9G8OH0gEeNXUbdqI2w2Gx2H18Ui684igWUqUPuWuny36n85LsA7ZMgQ2rZty8qV\nK/nhhx9o0uTKS4lUqFCBSpUqsWfPnmzL9ZpdRETA8VelXwH+wAAg2LKsZy5J2v62D5iXV8GJOMv5\npLMYY/ArkTnXZvF3s9kTHXVZvTtva8uRmMMsXLiQ1NRUUlNT2bJlC7t27QJg7ty5bNu2jYiICKZO\nncrDDz9MYmLiFftv0KABPj4+TJo0ifPnz5Oenk5UVBRbtmwBICAggIMHD5KRkZGHdy0iIkWNo4lb\nuGVZDS3L+tiyrOScKlmWtd+yrEfyKDYRp7mpYg0e6TiSHs82onGfAP6I/o1bqze+rJ63pw+TX3uL\nJUuWEBQURGBgIKNGjSI5OZlDhw4xYsQI5syZg7e3Nz179iQ8PJwnn3zyiv3bbDaWLVtGZGQklSpV\nokyZMgwYMID4+HgAunTpAkDp0qWpV69e3t68iIgUGQ5tMn9hyQ8/y7L+yKbsZuCUZVkn8iE+hxTk\nJvNSeIx65jm2vVGCpjyXY50/2U5i8wW8MHLCNfcT4/oDtzasWSj2KgXo0aMHs2bNwsfHp6BDERER\nB1zPJvOOPnF7FxiZQ9mTF8pFioRLx60VdY788iUiIjcGRxO3JsDKHMr+B1z+TknE2QycZj8ZpOdY\npRjuJCclXXMX6aSQlpFyzdfntYyMDJKTk3F3dy/oUERExAkcTdz8yJxNmp0zQOm8CUfk2g0Y2A+X\n2jv52KsB0fyQbR1fQomJOUrC2Zz+c85JBvEc5JBtDQFBZShZsuT1B5wHoqKiqFixIm5ubgUdioiI\nOIGjidsRoGEOZQ2BozmUiThNWFgYmyN/4PUPnuGb0j350qM7cRzKUqcY7pQ7fi/vfPAa0bF7HXrN\neI4THLKtI8U3ljua3E6durVxcbnWtavzRnp6OpGRkbzzzjt07NixQGMRERHncXRywuvA40B3y7L+\ne9H5NsB8Mtd0G5VvUV6BJifIpc6dO8fE1yYz9T/TqJ86jEZpz+KGJ5A5xm2X7XP+LP0tye7HKVbM\nNds20kgmweUwaS7nqBhSodBMRrAsi9TUVCpXrky7du246667CjokERG5CtczOcHRxM0T+I7Mp2t/\nAjFAMJm7J2wE7rEs69y1BJAXlLhJTg4dOsRTw0ax5rsfufPcJGrRHcM/i9mmkUIGaVmuSSGBjcWm\nsN11NiNGPsHwJ4dRvHhxZ4eeKzc3N4oVu6qNT0REpJDI98TtQieuQG/gHjLHtJ0gc2LCJ5ZlpeV2\nbX5T4iZXsn79eoYMGMH5GA/uPvsOQdS/rE4GGWxnLus8nue+ti154+2JBAUFFUC0IiJyI7uexM3h\nX9kty0oFPrrwI1KkNG3alMidm4mY/TGjnm5L5eTWNEt6DR8y9w49zAZWeQ3HL9SF5R8uoWHDnIZ0\nioiIFJyCHWEt4kQ2m43+A/qx/9Bumg4qzfsetfjBNpGvPXrztV8XXn53GFu3/6SkTURECi2HEzdj\nzL3GmC+MMTuNMfsv+dmXn0GK5KUSJUrw5juT2frrBkq2/o17hoaw79Dv9OnTu8Bni4qIiOTGoVel\nxpjWwNdkTlCoBqwAPMlceDcaWJ9fAYrkl7CwMD5f+mlBhyEiIuIwRx8vvAjMAFpfOH7Bsqw7gZqA\nDfgm70MTERERkYs5mrhVI/OJWwZgceFJ3YVN58eRmdiJiIiISD5yNHHLANKtzLVDjgMVLyqLBark\ndWAiIiIikpWjidtu/knOtgIjjDHljDH+wEjgYD7EJiIiIiIXcXQdt3nAzRc+v0TmJIUjF47TgZ55\nHJeIiIiIXMKhxM2yrBkXfd5mjLkFaEXmzNLvLMvamU/xiYiIiMgFV0zcjDFuwGPAKsuyogAsyzoC\nfJDPsYmIiIjIRa44xs2yrBTgdaBU/ocjIiIiIjlxdHLCLqByfgYiIiIiIrlzNHEbC7x4YWybtirt\n2QAAIABJREFUiIiIiBQAR2eVjgK8gV+MMQeBo2QuxPs3y7Ks5nkcm4iIiIhcxNHELR3QzFERERGR\nAuTociB35nMcIiIiInIFjo5xExEREZEC5tATN2NMsyvVsSxr3fWHIyIiInL1SpUowemEhOtux8/H\nh1NnzuRBRPnD0TFua8g6GSE7tusLRUREROTanE5IwPItft3tmLjrT/7yk6OJ213ZnCsNtAWaA0Pz\nLCIRERERyZajkxPW5lC0xBjzH6Ad8E2eRSUiIiIil8mLyQn/BbrmQTsiIiIikou8SNyqAhmOVjbG\ntDLG7DbG7DXGjM6m/BljTOSFnyhjTLoxRvukioiIyL+eo7NK+2Rz2g2oBfQHljjYjg2YAdwDHAG2\nGGOWWpZlX9zXsqwpwJQL9dsBT1qWdcqR9kVERERuZI5OTojI4XwysBAY7mA7DYC9lmXtBzDGLAA6\nkPOuDD2A+Q62LSIiInJDczRxq5TNuSTLsv66yv6CgcMXHR8BGmZX0RjjCbQihxmrxphBwCCAihUr\nXmUYIiIiIkWPo7NKo/M7kGy0A37M6TWpZVmzgFkA4eHhV1pjTkREclHCtxQJ8afzpC2fkn6cidMI\nF5H84OgYt7ZAqGVZ07Mpexw4YFnWcgeaigEqXHRc/sK57HRHr0lFRJwiIf405aflzcKjR4b55Ek7\nInI5R2eVvgh45VDmcaHcEVuAMGNMJWOMG5nJ2dJLKxljSpK5sO9XDrYrIiIicsNzNHGrBvycQ1kk\nUN2RRizLSiNzzNpKYBewyLKsHcaYwcaYwRdVfQD4n2VZZx2MT0REROSG5+jkBBfAO4cyH8DV0Q4v\nvFJdfsm5mZccR5DzTFYRERGRfyVHn7j9CjyUQ9lDwPa8CUdEREREcuLoE7c3gcXGmM+A98lcxiOY\nzOU4HgC65E94IiIiIvI3R5cD+cIYMxx4DXjwwmkDJAJPWJbl0M4JIiIiInLtHH3ihmVZ04wxEcAd\nQGngBPCTZVmJ+RSbiIiIiEP8fHwwcde/pI2fT+FezsbhxA3AsqwEMmeEityw/Er5EXc6rkD69vXz\n5fSpvFkEVUTk3+TUmTMFHYJTOLoA7yigvGVZw7IpmwocvrA5vEiRF3c6jsVnZxVI3528BhVIvyIi\nUjQ4Oqv0EXKeORp5oVxERERE8pGjiVtFYE8OZfuBkLwJR0RERERy4mjido7M5T+yUx5IzptwRERE\nRCQnjiZu64FnjDHuF5+8cDzyQrmIiIiI5CNHZ5WOA34C/jDGfALEkPkErheZS4P0zY/gREREROQf\nji7A+6sx5i7gDWAUmU/qMoAfgE6WZf2afyGKiIiICDj+qhTLsjZbltWMzE3lywM+lmXdCXgZYz7K\np/hERERE5AKHE7e/WZZ1HvAEnjPGHAC+B7rmdWAiIiIikpXDiZsxpqQxZpAx5kdgNzAGOA08BgTl\nU3wiIiIickGuY9yMMS5AK+BhoB1QHIgFZgCPAyMsy1qX30GKiEj+8inpx5FhebNHo09JvzxpR0Qu\nl2PiZox5E+gJlAWSgC+Aj4HvgBLAUGcEKCIi+e9M3KmCDkFEHJDbE7cnAQtYDvS1LOvk3wXGGCu/\nAxMRERGRrHIb4/YhkAC0AXYbY6YbYxo4JywRERERuVSOiZtlWQOBQOAhYCvwKLDBGLOLzLXc9NRN\nRERExIlynVVqWVaSZVnzLctqReZG888B6cBowACvG2N6GWOK53+oIiIiIv9uV7MA71HLsiZbllUL\naEDmzNIwYA5wNJ/iExEREZELrnoBXgDLsrZaljWMzPXbOgFr8jIoEREREbmco5vMZ8uyrFQylwn5\nIm/CEREREZGcXNMTNxERERFxPiVuIiIiIkWEEjcRERGRIkKJm4iIiEgRocRNREREpIhQ4iYiIiJS\nRChxExERESkilLiJiIiIFBFK3ERERESKCCVuIiIiIkWEEjcRERGRIuK69ioVuRH5+vnSyWtQgfUt\nIiKSEyVuIpc4fep0QYcgIiKSLb0qFRERESkilLiJiIiIFBFK3ERERESKCCVuIiIiIkWEEjcRERGR\nIkKJm4iIiEgRocRNREREpIhQ4iYiIiJSRDg9cTPGtDLG7DbG7DXGjM6hzp3GmEhjzA5jzFpnxygi\nIiJSGDl15wRjjA2YAdwDHAG2GGOWWpa186I6vsC7QCvLsg4ZY8o6M0YRERGRwsrZT9waAHsty9pv\nWVYKsADocEmdnsASy7IOAViWdczJMYqIiIgUSs5O3IKBwxcdH7lw7mI3A37GmDXGmG3GmD7ZNWSM\nGWSM2WqM2Xr8+PF8CldERESk8CiMkxOKAfWBNsB9wIvGmJsvrWRZ1izLssItywr39/d3dowiIiIi\nTufUMW5ADFDhouPyF85d7Ahw0rKss8BZY8w6oA7wh3NCFBERESmcnP3EbQsQZoypZIxxA7oDSy+p\n8xXQxBhTzBjjCTQEdjk5TinifEuUwhjj1B/fEqUK+rZFROQG59QnbpZlpRljhgIrARvwkWVZO4wx\ngy+Uz7Qsa5cxZgWwHcgAPrAsK8qZcUrRF59wmnFYTu1zXIJxan8iIvLv4+xXpViWtRxYfsm5mZcc\nTwGmODMuERERkcKuME5OEBEREZFsKHETERERKSKUuImIiIgUEUrcRESKqJKl8nf2dMlSmiktUtg4\nfXKCiIjkjTOnT3N/fGS+tf9Nybr51raIXBs9cRMREREpIpS4iYiIiBQRStxEREREigglbiIiIoXI\nmjVrMMZw4sSJgg4lV+PGjSMgIABjDBEREXnW7iOPPMIrr7ySZ+3ltxkzZtCuXTun9afETUREpBC5\n4447OHr0KKVLly7oUHIUFRXFyy+/zMyZMzl69CjdunXLk3Z/++03vvzyS0aMGJHl/B9//MGDDz6I\nr68vnp6e1KtXj127/tnGfODAgVSpUgUPDw/8/f3p0KFDlvKcDB8+nPDwcIoXL05oaOhl5QcPHsx2\nxvWKFSvsdQYMGMC2bdtYv379td/4VVDiJiIiUoi4ubkRGBiIMYV3/+O9e/cC0LFjRwIDA/Hw8MiT\ndqdNm0anTp0oUaKE/dyBAwdo3LgxlSpVYvXq1URFRTF+/Hi8vb3tdcLDw4mIiGDXrl2sXLkSy7Jo\n2bIlqampufaXkZHBww8/TJ8+fXKtt2LFCo4ePWr/adGihb3M3d2dnj17MnXq1Gu866ujxE1ERMTJ\n1q1bx+233463tzclS5akQYMGREVFAZe/Kg0NDc32qc/BgwcBiI+PZ9CgQZQtWxYfHx+aN2/O1q1b\n8y32cePG8cADDwDg4uJiTzD79u1L27ZtGT9+PAEBAXh7e/PII49w/vx5h+47PT2dRYsWXfbaccyY\nMdx77728+eab1KtXj8qVK9O6dWsqVKhgr/Poo4/StGlTQkNDqVevHuPHjyc2Npb9+/fnei/Tpk1j\n2LBh3HzzzbnWK126NIGBgfYfNze3LOXt27dn6dKlnDt37grf3vVT4iYiIuJEaWlpdOjQgSZNmvDr\nr7+yadMmRowYgc1my7b+li1bsjztadu2LdWqVSMgIADLsmjTpg0xMTEsW7aMX375hWbNmtGiRQuO\nHj2aYwyDBw/G29s7159Dhw5le+3TTz/N+++/D2CP6W9r167l119/ZdWqVSxevJj//e9/jBo1yqH7\n3r59O/Hx8YSHh9vby8jI4Ouvv6ZGjRq0atUKf39/brvtNhYuXJjjvZ09e5bZs2dTsWLFbF9/XosH\nH3yQsmXL0rhxYz7//PPLysPDw0lLS2PDhg150l9utACviIiIE505c4a4uDjatWtHlSpVAKhWrVqO\n9f39/e2fJ02axIYNG9i0aRMeHh6sXr2ayMhIjh8/bn9d+eqrr/L1118zd+5cnn322WzbfOWVV3j6\n6adzjTMoKCjb897e3vj6+gIQGBiYpcxmszF79my8vb2pVasWkyZNon///kycOJHk5ORc7zs6Ohpj\nDOXKlbOfO3bsGImJiUyYMIFXX32V119/ndWrV/PQQw/h7e1NmzZt7HXfffddnn32Wc6ePUvVqlVZ\ntWoV7u7uud7jlXh7e/PGG2/QuHFjihUrxtKlS+nWrRsff/wxvXr1stfz9PSkZMmS9qeg+UmJm4iI\niBOVKlWKvn37ct9993H33Xdz991307lzZypWrJjrdV9//TUvvfQSK1eutCc+27Zt49y5c1mSO4Ck\npCT27duXY1tly5albNmy138zl6hdu3aWsWeNGjUiJSWFffv2Ubt27Vzv+/z587i6uuLi8s/LwIyM\nDAA6dOjAU089BUDdunXZunUr06dPz5K4PfTQQ9xzzz0cPXqUN954gy5duvDjjz/i6enJ/fffb588\nEBISwo4dOxy6nzJlyjBy5Ej7cXh4OCdPnmTy5MlZEjcADw+PLK+F84telYqIiDjZ7Nmz2bRpE82a\nNWPp0qVUrVqVlStX5lg/KiqKhx56iBkzZtC8eXP7+YyMDAICAoiMjMzy8/vvv/Pqq6/m2N71vCq9\nHrndd5kyZUhJSckyTqxMmTIUK1aMGjVqZGmnevXql8VXsmRJwsLCaNasGZ9//jl//PEHixcvBuCD\nDz6wfzfLly+/rnto0KABe/bsuez8qVOnLkug84OeuImIiBSAOnXqUKdOHUaNGsX999/Pxx9/zH33\n3XdZvRMnTtCuXTsGDhxI//79s5TVq1ePv/76CxcXFypXruxw39fzqjQ3v/32G2fPnsXLywuAjRs3\n4ubmZn9CCDnfd926mXvj7ty50z7Ozc3Njdtuu43du3dn6eePP/4gJCQkxzgsy8KyLJKTkwEIDg6+\n6nvJSWRkZJbXuQD79u0jKSmJevXq5Vk/OVHiJiIi4kQHDhzgvffeo3379gQHB7N//362b9/OY489\nlm39Tp06ERwczMiRI/nzzz/t5/39/WnZsiWNGzemQ4cOTJ48mWrVqvHnn3+yYsUKWrZsSdOmTbNt\nM79elaalpdGvXz/Gjh1LbGwso0ePZuDAgXh5eV3xvv39/alXrx4//PBDlgkKzz77LF27dqVp06a0\naNGC77//ngULFvDll18CmUuTLF68mJYtW+Lv78+RI0d4/fXXcXd3p23btrnGu3fvXhITE4mNjSUl\nJYXIyEgAatSogZubGx9//DGurq7ceuutuLi48PXXXzNjxgwmTZqUpZ3169dTuXJlwsLC8vLrzJYS\nNxERESfy9PTkjz/+oEuXLpw4cYKAgAAeeugh++zLS61btw64/KnRgQMHCA0NZfny5bzwwgsMHDiQ\nY8eOERAQQOPGja+4Nll+aN68OTVr1uSuu+7i3LlzdOrUicmTJwOO3fegQYP44IMPsizA27FjR2bN\nmsWECRMYPnw4YWFhzJkzxz6+zd3dnTVr1vDmm28SFxdHQEAAzZo1Y8OGDZdNnrjUgAEDWLt2rf34\n1ltvBf75bgHGjx9PdHQ0NpuNm2++mY8++uiy8W3z589n4MCB1/7FXQVjWZZTOspP4eHhVn6uWSNF\njzGGcTj3v+1xGG6E/5+k6DDGcH98ZL61/03JuvpvWhzWt29fTpw4wbJly665jaSkJKpVq8bcuXNz\nfFpY2ERFRXH33Xfzxx9/ULJkSYeuMcZssywr/Mo1L6fJCSIiIlIoFC9enDlz5nDq1KmCDsVhsbGx\nzJkzx+Gk7XrpVamIiIgUGs2aNSvoEK7Kvffe69T+lLiJiIjIdYuIiCjoEP4V9KpURESkiLEsy744\nrfy76ImbiIhIIbd161a+/GIpG9dtI/K3nzl95hgZVgauNjdCg2/mtgb1adKiIV27dqV06dIFHa7k\nIyVuIiIihVBGRgbz5s3jjQnTOHr4ONWTulMufQC9qY8PQbhgIy09ieOHdhJ7aBsfLF/Ds089T/v2\nHXjuxaepVatWQd+C5AMlbnJDKunjx7gE4/Q+RUTywr59++jVrR/Hfk/h9rMv0ZFWuGC7rJ4rHgRR\nnyDqw7lBnOU4v3z+IY2/vovhTw3lxZeex9XVtQDuQPKL1nETESmiSpYqxZnTp/Ot/RJ+fsQXoWUZ\nipqMjAyOHDnC2bNns5z/5ptvGDdmAo1TnqdBxvBsE7YriecI3xQfgFUuhlkRM5yyh6Zk5eXlRfny\n5XFxuXw6wfWs46bETUREpAAcO3aM5ORkgoOD7f+4fxwxh6eGPEfX88sJpM51tW9h8UOx19hb9mM2\nbF132f6akn8yMjKIiYnB3d09263FtACviIhIEfP39kx/J23ffvstTw4ZRY/zq647aQMwGJqmvUDY\nsb60aHIf58+fv+42xTEuLi4EBAQQHx+f923neYsiIiJyRenp6fbxZ/Hx8fTp0Z+25+fiT7U87adx\n2vO4H63GmFFj87RdyZ2rqytpaWl53q4SNxERkQJiTOYkquFDRhJytjVVaJn3fWC49/wMIj74hI0b\nN+Z5+5K9v/9s85oSNxERkQJ08OBBliz5gruSJudbH1740/T8eF589tV860OcQ4mbiIhIAXp3+nvU\nzuhDcUrkaz+30JPNWzazf//+a25j3Lhx9OrV65qujYiIoEmTJvZjYwx79+695lj+rZS4iYiIFJCM\njAw+fP8jbk0ZnO99ueLBLem9ef+9j/K9L8k/StxEREQKyL59+3BJL04Zqjqlv9DUe1j73U9O6Uvy\nhxI3ERGRArJt2zbKu1zTcl7XpBz1+W3XzziyhuukSZMIDg7Gx8eHqlWrsmrVKgBSUlLo06cPPj4+\n1KxZk4vXUX399depUqUKPj4+1KhRgy+++MKhuOLj4+nTpw/+/v6EhIQwfvx4MjIyAAgJCWHbtm0A\nzJs3D2MMO3bsAODDDz+kY8eOV/UdFHVK3ERERApI1G878E24xWn9eVOWYhQnJiYm13q7d+9m+vTp\nbNmyhYSEBFauXEloaCgAS5cupXv37sTFxdG+fXuGDh1qv65KlSqsX7+e+Ph4XnrpJXr16sXRo0ev\nGNewYcOIj49n//79rF27ljlz5jB79mwAmjdvzpo1awBYu3YtlStXZt26dfbj5s2bX8M3UXQpcRMR\nESkgiWfO4YaPU/v0KFbism22LmWz2UhOTmbnzp2kpqYSGhpKlSpVAGjSpAmtW7fGZrPRu3dvfv31\nV/t1Xbp0ISgoCBcXF7p160ZYWBibN2/Ota/09HQWLFjAxIkT8fHxITQ0lJEjRzJ37lwgM3Fbu3Yt\nAOvXr+e5556zHytxExEREadxsblgkeHUPjOsdGy23Pc/vemmm3j77bcZN24cZcuWpXv37sTGxgIQ\nGBhor+fp6UlSUpJ9odk5c+ZQt25dfH198fX1JSoqihMnTuTa14kTJ0hNTSUkJMR+LiQkxP5UsHnz\n5qxfv56jR4+Snp5O165d+fHHHzl48CDx8fHUrVv3mr6HokqJm4iISAEp7e/H+WLHnNZfBhkkpJzA\nz8/vinV79uzJDz/8QHR0NMYYRo0alWv96OhoBg4cyPTp0zl58iRxcXHUqlXriuPpypQpg6urK9HR\n0fZzhw4dIjg4GMhMIj09PZk2bRrNmjWjRIkSBAYGMmvWLJo0aZLtJu43sn/X3YqIiBQi9erdygmv\nn53W32n2UdLHl9KlS+dab/fu3axevZrk5GSKFy+Oh4fHFROks2fPYozB398fgNmzZxMVFXXFmGw2\nG127dmXMmDEkJCQQHR3NW2+9lWW9uObNmzN9+nT7a9E777wzy/G/iRI3ERGRAlK/fn2OJP1MxlW+\nLrWwSCP5qvuLZRu31q1/xXrJycmMHj2aMmXKEBgYyLFjx5g4cWKu19SoUYORI0fSqFEjAgIC+O23\n32jcuLFDcU2bNg0vLy8qV65MkyZN6NmzJ/369bOXN2/enISEBJo1a5bt8b+JcWRKcGEXHh5uXTwd\nWUREpLDbtWsX1atXp0aVuoTvn0IV7nHour+IYgUjOMxPNOZZGvMsbng6dO0XXp0ZNPluhgx57HpC\nFwf9/Wd8KWPMNsuyrmkdGD1xExERKUBPPPMYv3q9e8V65zjJfxnKx7SgGh0ZQhQn2c10qvEbC7DI\n/UHMGWLYn7Ga3r2vbcsqKRycnrgZY1oZY3YbY/YaY0ZnU36nMSbeGBN54Wess2MUERFxll69HuKg\ntY7j7Mq2PJ1UNjGN6VTHYBjKLhoylFJUpjPz6cQ8fmQyH9GUWLbl2M9m1//Qo0cPfHycu/yI5K1i\nzuzMGGMDZgD3AEeALcaYpZZl7byk6nrLsto6MzYREZGC4O3tzYRJr/Lm6EfoffZHXPhnqY59fMsK\nRuBNOR5mNQHUuuz6EJoyiC1EEsGntCWM1rTgNXz4Z9mOI2xmp8cnfD7h18uul6LF2U/cGgB7Lcva\nb1lWCrAA6ODkGERERAqVx4YMJrimJz/ZJgFwkr3MpwPLGMzdTKAP32abtP3NBRv16M9QduNBad6l\nFj8wmTSSSSaR5V59eXfWOwQEBDjrliSfOPWJGxAMHL7o+AjQMJt6dxhjtgMxwNOWZe1wRnAiIiIF\nwcXFhXmfRRBetxGHTv/EYTbSmGfowiKK4e5wO8Upwb1Mpj4DWclItvEeHm7e3NPxDrp27ZqPdyDO\nUhgnJ/wMVLQsqzYwDfgyu0rGmEHGmK3GmK3Hjx93aoAiIiJ5KSMjg2+//Y601DTc8GEIv9GEUVeV\ntF2sNGH0ZClteJek9EQO7DvIzp2XjkqSosjZiVsMUOGi4/IXztlZlnXGsqzEC5+XA67GmDKXNmRZ\n1izLssItywr/e7E/ERGRomhAn8d4bfgsOicuozPz8aFcnrR7E/fxePrveG/uwB23NWfjxo150q4U\nHGcnbluAMGNMJWOMG9AdWHpxBWNMoDHGXPjc4EKMJ50cp4iIiNPs/eMAjc++QjC35XnbNlxpkDGM\nyq53cuTIkTxvX5zLqWPcLMtKM8YMBVYCNuAjy7J2GGMGXyifCXQGHjPGpAHnge7WjbBKsIiISBHW\nt29fypcvz/jx4ws6lH81Z09O+Pv15/JLzs286PN0YLqz4xIREREp7JyeuBVVpfxKcjruTEGHYefn\nW4JTp+MLOgwRERFxIiVuDjoddwYroktBh2Fn+n5W0CGIiMgN7JdffqF///7s2bOH1q1bc2H4OQDL\nli3jhRde4ODBg9SoUYOZM2dSu3ZtAGJjYxk2bBjr1q3D29ubJ598kieeeAKAcePGERUVhc1mY/ny\n5YSFhTF79mzq1KlTIPdYFBXG5UBERESkAKWkpNCxY0d69+7NqVOn6NKlC4sXLwYyE7p+/frx3nvv\ncfLkSR599FHat29PcnIyGRkZtGvXjjp16hATE8OqVat4++23Wblypb3tr776ii5dunDq1Cl69uxJ\nx44dSU1NLahbLXKUuImIiEgWGzduJDU1lREjRuDq6krnzp257bbMGa+zZs3i0UcfpWHDhthsNh5+\n+GHc3d3ZuHEjW7Zs4fjx44wdOxY3NzcqV67MwIEDWbBggb3t+vXr07lzZ1xdXXnqqadISkrSMiVX\nQa9KRUREJIvY2FiCg4OzvB4NCQkBIDo6mo8//php06bZy1JSUoiNjcVmsxEbG4uvr6+9LD09naZN\nm9qPK1T4ZzlXFxcXypcvT2xsbH7ezg1FiZuIiIhkUa5cOWJiYrAsy568HTp0iCpVqlChQgXGjBnD\nmDFjLrtuw4YNVKpUiT179uTY9uHD/+x8mZGRwZEjRwgKCsr7m7hB6VWpiIiIZNGoUSOKFSvG1KlT\nSU1NZcmSJWzevBmAgQMHMnPmTDZt2oRlWZw9e5b//ve/JCQk0KBBA3x8fJg0aRLnz58nPT2dqKgo\ntmzZYm9727ZtLFmyhLS0NN5++23c3d25/fbbC+pWixwlbiIiIpKFm5sbS5YsISIiglKlSrFw4UIe\nfPBBAMLDw3n//fcZOnQofn5+3HTTTURERABgs9lYtmwZkZGRVKpUiTJlyjBgwADi4/9ZvqpDhw4s\nXLgQPz8/5s6dy5IlS3B1dS2I2yyS9KpURERELhMeHs4vv/ySbVmrVq1o1apVtmVBQUHMnz8/x3aL\nFy/OJ598kicx/hspcRMRESlAviVKEZ9wmvV8m78dnYEuXRZT0sePuDOn8rcvyTdK3ERERApQfMJp\nxuG8LbnHJZgrV5JCS4mbiIiIOMW4ceMKOoQiT5MTRERERIoIJW4iIiIiRYQSNxEREZEiQombiIiI\nSBGhxE1ERESkiFDiJiIiIlJEKHETERGRLHbv3k3dunXx8fFh6tSpBR1OoWGMYe/evQUagxI3ERER\nyWLy5MncddddJCQk8MQTTxR0OPlmzZo1lC9fvqDDuCpK3ERERCSL6OhoatasmW1Zenq6k6O5NpZl\nkZGRUdBh5DklbiIiImLXokULvv/+e4YOHYq3tzc9e/bkscceo3Xr1nh5efH9998THx9Pnz598Pf3\nJyQkhPHjx9uTpIiICBo3bsyTTz6Jr68vlStX5qeffiIiIoIKFSpQtmxZPv744yvGsXz5cmrUqIGP\njw/BwcG88cYb9rKvvvqKunXrUqJECapUqcKKFSsAuPPOOxkzZgyNGzfG09OT/fv3M3v2bKpXr46P\njw+VK1fmvffeA+Ds2bPcf//9xMbG4u3tjbe3N7GxsaSnpzNhwgSqVKmCj48P9evX5/Dhw/a+v/vu\nO8LCwvD19eXxxx/Hspy3XRkocRMREZGLrF69mqZNmzJ9+nQSExNxc3Pj008/ZcyYMSQkJNCkSROG\nDRtGfHw8+/fvZ+3atcyZM4fZs2fb29i0aRO1a9fm5MmT9OzZk+7du7Nlyxb27t3LJ598wtChQ0lM\nTMw1jv79+/Pee++RkJBAVFQULVq0AGDz5s306dOHKVOmEBcXx7p16wgNDbVfN3fuXGbNmkVCQgIh\nISGULVuWZcuWcebMGWbPns2TTz7Jzz//jJeXF9988w1BQUEkJiaSmJhIUFAQb731FvPnz2f58uWc\nOXOGjz76CE9PT3v7y5YtY8uWLWzfvp1FixaxcuXKvP0DuAIlbiIiIpKrDh060LhxY1z7DkbnAAAQ\nN0lEQVRcXHB1dWXBggVMnDgRHx8fQkNDGTlyJHPnzrXXr1SpEo888gg2m41u3bpx+PBhxo4di7u7\nO/feey9ubm5XHOTv6urKzp07OXPmDH5+ftSrVw+ADz/8kH79+nHPPffg4uJCcHAw1apVs1/Xt29f\natasSbFixXB1daVNmzZUqVIFYwzNmzfn3nvvZf369Tn2+8EHHzB+/HiqVq2KMYY6depQunRpe/no\n0aPx9fWlYsWK3HXXXURGRl7r13pNlLiJiIhIripUqGD/fOLECVJTUwkJCbGfCwkJISYmxn4cEBBg\n/+zh4ZHtuSs9cVu8eDHLly8nJCSE5s2bs2HDBgAOHz5MlSpVHIoV4JtvvuH222+nVKlS+Pr6snz5\nck6cOJHj9VdqPzAw0P7Z09PziveR15S4iYiISK6MMfbPZcqUwdXVlejoaPu5Q4cOERwcnKd93nbb\nbXz11VccO3aMjh070rVrVyAzMdu3b59DsSYnJ9OpUyeefvpp/vrrL+Li4mjdurV9XNrFdf92pfYL\nmhI3ERERcZjNZqNr1672MW/R0dG89dZb9OrVK8/6SElJYd68ecTHx+Pq6kqJEiVwcclMWfr378/s\n2bNZtWoVGRkZxMTE8Pvvv+fYTnJyMv7+/hQrVoxvvvmG//3vf/bygIAATp48SXx8vP3cgAEDePHF\nF9mzZw+WZbF9+3ZOnjyZZ/d2vZS4iYiIyFWZNm0aXl5eVK5cmSZNmtCzZ0/69euXp33MnTuX0NBQ\nSpQowcyZM5k3bx4A/9/evQfbVZZ3HP/+kiDUQHIiBygSbuFiuHSaUgiMXMUxRkDE0kagFOnEaZlR\nR0yRgnUktaMgoHXaadWCCg3ENAgo4gQqSkWmAgbkFsL91pBwAoRLCBADefrH++5k7Z29zz45nH0W\ne6/fZ2bP3vtd71rrXc9+5+TJuy7v9OnTN9xkMHHiRI488si60b+i2gOEZ82axaRJk5g/fz7HH3/8\nhuVTp07l5JNPZsqUKfT19bF8+XLmzJnDrFmzmDFjBhMmTGD27Nm8/vrrI3psb4dG+zbWTjjwwANj\n8eLFHd2HJOKyv+joPjaHTr9q1G9BNjOzkbN06VL22WcfJDGX0ft7Phf5349RUvuNG0m6MyIOHM42\nPeJmZmZm1iWcuJmZmVkp9ttvvw0Pvy2+aqdFbVPjym6AmZlZVVX9lOWSJUvKbkLHdOq39YibmZlZ\nCcaOHcu6devKboZ1yLp16xg3buTHx5y4mZmZlaCvr4+BgYGym2EdsH79egYGBpg4ceKIb9unSs3M\nzErQ39/PsmXLStn30qVLS9lvlYwfP57+/v4R364TtyGa1DcBnX5V2c3YYFLfhLKbYGZmb8OYMWPY\nZZddStl3s0dUWHdw4jZEq158uX0lMzMzsw5y4mZmZlaiidtMYu7qTefM7OT+rHs5cTMzMyvRS6+s\nKrsJ1kV8V6mZmZlZl+iJuUolPQc0n2F2cP3A8yPcnG7meNRzPOo5HvUcj3qOx0aORT3Ho14/MD4i\nthvOyj2RuA2XpMXDneS1Fzke9RyPeo5HPcejnuOxkWNRz/Go93bj4VOlZmZmZl3CiZuZmZlZl6h6\n4vYfZTfgHcbxqOd41HM86jke9RyPjRyLeo5HvbcVj0pf42ZmZmbWTao+4mZmZmbWNZy4mZmZmXWJ\nSiVukp6UdJ+kuyUtzmXvkfRzSY/k956dC0TS9yWtlHR/oazl8Us6V9Kjkh6S9OFyWt05LeIxV9Iz\nuY/cLemYwrKejYeknSXdLOkBSUskfS6XV7J/DBKPqvaPrSTdIemeHI9/zOVV7R+t4lHJ/gEgaayk\n30m6Pn+vZN+oaRKPkesbEVGZF/Ak0N9QdiFwTv58DvD1stvZweM/AjgAuL/d8QP7AvcAWwK7A48B\nY8s+hlGIx1zgrCZ1ezoewI7AAfnzNsDD+Zgr2T8GiUdV+4eArfPnLYDbgUMq3D9axaOS/SMf4xxg\nPnB9/l7JvjFIPEasb1RqxK2FjwGX58+XAyeU2JaOiohbgMZJ8Vod/8eABRGxNiKeAB4Fpo9KQ0dJ\ni3i00tPxiIgVEXFX/rwaWArsREX7xyDxaKXX4xER8Wr+ukV+BdXtH63i0UpPx0PSZOBY4NJCcSX7\nBrSMRyubHY+qJW4B3CTpTkl/k8t2iIgV+fOzwA7lNK00rY5/J+D/CvWWMfg/XL3ks5LuzadSa8P7\nlYmHpN2APyGNIlS+fzTEAyraP/Kpn7uBlcDPI6LS/aNFPKCa/eNbwNnA+kJZZfsGzeMBI9Q3qpa4\nHRYR04CPAJ+WdERxYaRxy8o+H6Xqx599G5gCTANWAN8otzmjS9LWwNXAmRHxSnFZFftHk3hUtn9E\nxFv57+dkYLqk/RuWV6p/tIhH5fqHpOOAlRFxZ6s6Veobg8RjxPpGpRK3iHgmv68EriUNRw5I2hEg\nv68sr4WlaHX8zwA7F+pNzmU9LSIG8h/k9cAlbByy7vl4SNqClKRcGRHX5OLK9o9m8ahy/6iJiJeA\nm4GZVLh/1BTjUdH+cShwvKQngQXA0ZKuoLp9o2k8RrJvVCZxkzRe0ja1z8AM4H7gOuCTudongZ+U\n08LStDr+64CTJG0paXdgL+COEto3qmp/aLKPk/oI9Hg8JAn4HrA0Ir5ZWFTJ/tEqHhXuH9tJ6suf\n/wD4EPAg1e0fTeNRxf4REedGxOSI2A04CfhlRJxKRftGq3iMZN8Y14F2v1PtAFyb/h4zDpgfETdI\n+i2wUNJs4ClgVolt7ChJPwSOAvolLQPOAy6gyfFHxBJJC4EHgDeBT0fEW6U0vENaxOMoSdNIw/pP\nAn8LlYjHocBfAffl63YAvkh1+0ereJxc0f6xI3C5pLGk//AvjIjrJf2GavaPVvGYV9H+0UxV/3a0\ncuFI9Q1PeWVmZmbWJSpzqtTMzMys2zlxMzMzM+sSTtzMzMzMuoQTNzMzM7Mu4cTNzMzMrEs4cTPr\nYZJiCK8nR2hfW+XtnTOMdWfmdQ8ZibZsxn73lDRX0i5Nlj0r6Tuj2Z7B5CmWHpD0mbLb0oykkyQt\ny881M7MO8eNAzHpYk0ToWuAeYG6hbG1E/G4E9iXgYODpiFi+metOBPYB7i9M3t1xkmYCi4DDI+LW\nhmUHAC/miZ9LpzS/8peBPSJibdntaSRpDLAE+M+IOL/s9pj1qio9gNesciLituJ3SWuB5xvLW5G0\n5VCThDwf4ZC222Tdl4e7bqdExF1lt6HBWcAP3olJG0BErJd0CXC2pIsjYl3ZbTLrRT5VamYASFog\n6VFJR0i6TdLrwFfystMk/UrSc5JWS7pT0ikN629yqlTSBZLelLSXpBslrZH0hKRz8whdrd4mp0pz\nG26S9BFJd0t6TdJ9ko5t0vbTJD0s6Q1J9+R1bpN0wyDHWxttA/h14dTxIXl53alSSWfk5QdJulrS\nq7nO3+XlH837XiPpdkl/3GSfn5B0Rz6WF3PMdxrCb3MkaSqc+Q3l+0q6Lv8ub0h6WtJ/NdTZQdIl\nklZIWptPt/51k33sKWm+pJV5W49Juqiw/P2SfilpVW7/Y5K+1bCZBaRZaj7a7pjMbHg84mZmRf3A\nPODrpClY1uTyKcBVwCPAeuADwDxJ74qIy9psU8A1pLk/LwL+DPgaadqXH7ZZdx/gQuB84EXg74Fr\nJO0dEU8BSDoOuBz4EXAmKXH4NrAVcHezjWa/AT4P/DNp+pl7c/n9LddIrgAuy/v4S+BiSf3AMcBX\ngTeAi0lT7O1Vm75G0pnAN0kTTJ8H9JES45slTYuI1wbZ50zghYhYWivIie8iYFlu/wukCaqPLdSZ\nlI8T4EvA03n59ySNi4hLcr29gNuBl0hTez0O7EqaEq62nUXALcBppH6xG3BQsZERsVzSY7m91wwa\nRTMbnojwyy+/KvIiJUtXtFi2gDSP3ofbbGMM6T9984DbC+Vb5fXPKZRdkMtOLpQJeBi4rlA2M9c7\npFB2G7AW2LVQNjnXm1Mouwu4s6GN78/1bmhzLLX9HtZk2bPAdwrfz8h1zy6UvYuUUL4BTC6Uz8p1\nD87f+0jJzr837GNv0vyEZ7Rp583ALxrKarGYMch6X8373a2hfB6wHBiTvy8kJW3btdjOYXlfew+h\nj10F3Ft2X/fLr159+VSpmRW9FhE3NhZKmippoaTlpERjHXAq8L4hbvdntQ8REaSL2De5k7OJJZFH\n1vK6y0gJxi65XVsC00ijbRTq/S+wYoht21y106tExO+BJ3I7lxXqPJjfd87vhwPvBq6UNK72Io1s\nPQ4c0Waf7wWeayh7ljTadrGk2ZL2aLLeTOBWYFnDfm8kTZS+Z643A/hxRDTuo3g8q0kjdae0Ob37\nXG6vmXWAEzczK3q2sUBSH3ATMBX4Amn05SDgStIoWztvRcQrDWVrh7juqiZlxXX/kDSCt7JJvYEh\nbH84Xmz4/vsWZbCxndvn91tJSW/xtRewbZt9bkU67g0i4k3gaNIp3ouAR/M1irML1bYnJWWN+5yX\nl28raSwwkZQENhURz+d9vQB8l5QI3iPp+CbVXwf8SBCzDvE1bmZW1Oz5QIcDOwEnRMTiWqGkLUat\nVa0NkNq8fZNlO9C55G1zvZDfTyFdJ9ioMbFttv6kxsKIeAQ4VelRHNNI1/hdKunxiLg5r/coKeFu\n5sGIeEvSS6TfuKX825+Qf/eDSNfMXS1p39yOmvcAz7c5HjMbJo+4mVk7787vGx7vIGl70sX4pYqI\nN0g3IPx5sVzSoaRTge3URrE6PUJ0C2kkakpELG7yerjN+g+SbhBpKiLWR3p8yVm5aP/8fgPpBo/H\nW+y39sy8/wY+nm+yGFRErMunoueS/vM/taHK7sBD7bZjZsPjETcza+fXpAvcvyvpK8AE0oNgB0gX\nyJfty8BPJV0FfJ90+vQ8UvvWt1n3wVznU5LWkE5xLo2INYOvtnkiYlV+TMo3JL2XdI3ZatIo1weA\nRRHxo0E2cQvwCUnbRMRqAEnTSXfnLgQeA7YAPpWP4X/yeheSktpb86M7Hga2ISVzB0fEibnel0in\nVG+TdD7purudgaMj4nRJJ5KuafwJ8FTexudJ1xveUWtkPu36p6S7ks2sAzziZmaDijQLwomkUamr\ngX8C/pWGGwLKEhHXA6eTThX+GJgDfIZ03dnLbdZdAXyONOPDLcBvgT/qUDv/hZRE7U+6PvBnpAQz\ngPvarH4N8Bb1o5zPkG7A+ALw07zNbYFjIuK+vM9VpGO7CfgH0sjapaRHgvyi0LZHcr27SMneIjYm\n55AS3DdJo2yL8jbWAB+MiOLp6KOA8UDds+TMbOR4yisz6zmSdieNLn0xIi5qV78bSFoAbB0Rx5Xd\nllYk/YD0WJQPld0Ws17lxM3MuprSPKdfI40grQL2ID2odxKw7yCPuOgqkt5HuoP0wNqI2juJpJ1J\nN14cGRG3l90es17la9zMrNutI11r92+kU4WvAr8Czu2VpA0gIh7Kj/rYkfanVsuwK/BZJ21mneUR\nNzMzM7Mu4ZsTzMzMzLqEEzczMzOzLuHEzczMzKxLOHEzMzMz6xJO3MzMzMy6xP8DZApuX2kOBUwA\nAAAASUVORK5CYII=\n", 74 | "text/plain": [ 75 | "" 76 | ] 77 | }, 78 | "metadata": {}, 79 | "output_type": "display_data" 80 | } 81 | ], 82 | "source": [ 83 | "plt.figure(figsize=(10, 6))\n", 84 | "\n", 85 | "marker = {'shallow' : 'o', 'deep' : 'h', 'from_scratch' : 's'}\n", 86 | "props = dict(boxstyle='round', facecolor='w', alpha=0.7)\n", 87 | "\n", 88 | "for t in c_map.keys():\n", 89 | " stats = list(filter(lambda s: s['train_mode']==t, stats_full))\n", 90 | "\n", 91 | " colors = [cmx[j] for j in range(N)]\n", 92 | " train_mode = [s['train_mode'] for s in stats]\n", 93 | "\n", 94 | " #Size\n", 95 | " fps = [float(s['fps'])*20.0 for s in stats]\n", 96 | "\n", 97 | " #Labels\n", 98 | " names = [s['name'] for s in stats]\n", 99 | " #X\n", 100 | " training_time = [float(s['training_time']) for s in stats]\n", 101 | " #Y\n", 102 | " accuracy = [float(s['accuracy']) for s in stats]\n", 103 | " \n", 104 | " plt.scatter(training_time, accuracy, s=fps, c=colors,\\\n", 105 | " label=t, edgecolor='black', marker=marker[t])\n", 106 | " \n", 107 | " if t == 'deep':\n", 108 | " for i in range(len(stats)):\n", 109 | " plt.annotate(names[i],xy=(training_time[i], accuracy[i]), \\\n", 110 | " fontsize=12, va='center', bbox=props)\n", 111 | "\n", 112 | "\n", 113 | "\n", 114 | "plt.xlabel('Training time (secs)', fontsize=16)\n", 115 | "plt.ylabel('Accuracy', fontsize=16)\n", 116 | "\n", 117 | "plt.legend(loc='lower right', fontsize=12, )\n", 118 | "\n", 119 | "min_fps = min((s['fps'] for s in stats))\n", 120 | "max_fps = max((s['fps'] for s in stats))\n", 121 | "plt.text(400, 0.55, 'size = fps(%d-%d)' % (max_fps, min_fps), fontsize=14,\n", 122 | " va='bottom', ha='center')\n", 123 | "\n", 124 | "plt.title('Transfer learning shootout for PyTorch\\'s model zoo',y=1.05, fontsize=16)\n", 125 | "plt.show()\n" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": { 132 | "collapsed": true 133 | }, 134 | "outputs": [], 135 | "source": [] 136 | } 137 | ], 138 | "metadata": { 139 | "kernelspec": { 140 | "display_name": "Python 3", 141 | "language": "python", 142 | "name": "python3" 143 | }, 144 | "language_info": { 145 | "codemirror_mode": { 146 | "name": "ipython", 147 | "version": 3 148 | }, 149 | "file_extension": ".py", 150 | "mimetype": "text/x-python", 151 | "name": "python", 152 | "nbconvert_exporter": "python", 153 | "pygments_lexer": "ipython3", 154 | "version": "3.6.0" 155 | } 156 | }, 157 | "nbformat": 4, 158 | "nbformat_minor": 2 159 | } 160 | -------------------------------------------------------------------------------- /results/diagram_17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahirner/pytorch-retraining/94dd61177ee369d6724fdb29cdcf4630a0daa96d/results/diagram_17.png -------------------------------------------------------------------------------- /results/diagram_17_clr.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahirner/pytorch-retraining/94dd61177ee369d6724fdb29cdcf4630a0daa96d/results/diagram_17_clr.png -------------------------------------------------------------------------------- /results/diagram_bees.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ahirner/pytorch-retraining/94dd61177ee369d6724fdb29cdcf4630a0daa96d/results/diagram_bees.png -------------------------------------------------------------------------------- /results/stats_bees.csv: -------------------------------------------------------------------------------- 1 | variables_optimized,params_optimized,training_time,training_loss,training_losses,accuracy,eval_time,name,retrained,shallow_retrain 2 | 2,8194,132.27440738677979,0.3337305188179016,"[0.6449973697463671, 0.799626004199187, 0.4865968585014343, 0.5339775641759237, 0.7758354211846987, 0.762413190305233, 0.33923502415418627, 0.4132471902916829, 0.44662663439909617, 0.7605774372816085, 0.6177145212888717, 0.6748861173788706, 0.2680120279391607, 0.40952806174755096, 0.32828037440776825, 0.3411690669755141, 0.5101253290971121, 0.2958725616335869, 0.2770718668897947, 0.6957354908188184, 0.37219979166984557, 0.4904374713699023, 0.1039210965236028, 0.2812492909530799, 0.5081507921218872, 0.2843585185706615, 0.2735815142591794, 0.22060134013493857, 0.25705449680487313, 0.3337305188179016]",0.9411764705882353,3.6459550857543945,alexnet,True,True 3 | 2,3330,277.8260440826416,0.5314074479043484,"[0.7803391536076864, 1.0151035790642102, 0.6096459249655406, 0.71614589219292, 1.0304175108671187, 0.5891895038386186, 0.5866806777815025, 0.6755906095107397, 0.6227317874630293, 0.7099033010502657, 0.5689165299137433, 0.5437799210349719, 0.8015171070893605, 0.6678334052364031, 0.5985828551153342, 0.4999093475441138, 0.710104750096798, 0.7836596677700679, 0.5328871063888073, 0.44145535404483477, 0.46230312883853913, 0.5844454544285933, 0.47636913855870566, 0.6630046005050342, 0.6307528525590896, 0.5341838262975216, 0.632778454075257, 0.6141219452023506, 0.6268344536423683, 0.5314074479043484]",0.9215686274509803,7.851351261138916,densenet169,True,True 4 | 4,5636,271.20917892456055,2.134380352497101,"[1.8291524430116017, 1.8966477513313293, 1.5457304696242014, 1.962473878264427, 1.5311813334623972, 1.4712112734715144, 1.4740882496039072, 1.1312819381554922, 1.4500905454158783, 1.6148350318272908, 1.3559898629784584, 1.3147228678067526, 1.3842623422543208, 1.5068770463267962, 1.879888591170311, 1.7435520976781844, 1.3778608818848928, 1.59798597296079, 1.306519621113936, 1.4390670364101728, 1.5454671382904053, 1.2700662851333617, 1.3067057773470878, 1.2154812504847845, 1.301005790134271, 1.5278363784154256, 1.200030130147934, 1.3327435905734697, 1.2067295412222545, 2.134380352497101]",0.869281045751634,5.174986362457275,inception_v3,True,True 5 | 2,1026,151.41002368927002,0.8697185007234415,"[0.6588224162658055, 0.6182808289925258, 0.6056827455759048, 0.9534439203639825, 1.3672176629304886, 0.7642186850309372, 0.5063450093070666, 0.5990918685992559, 0.5700537902613481, 0.4351017480095228, 0.688445183634758, 0.6358402222394943, 0.5046178514758746, 0.8893822446465492, 0.8708702484766643, 0.7534642721215884, 0.6487383291125297, 0.5105993986129761, 0.571998796860377, 0.7673505996664365, 0.6745530679821968, 0.5809230287869771, 0.519818802177906, 0.5375595656534036, 0.6444296479225159, 0.6124498384694258, 0.7683310935894648, 0.4757927045226097, 0.5544534859557947, 0.8697185007234415]",0.8888888888888888,3.183588981628418,resnet34,True,True 6 | 2,1026,67.87700986862183,0.13191987425088883,"[0.48623288770516715, 0.4485887433091799, 0.2691540131966273, 0.2209258442123731, 0.23335402309894562, 0.20383766839901607, 0.3693067381779353, 0.17284807215134304, 0.18281655609607697, 0.14947874297698338, 0.16509772290786107, 0.14073311537504196, 0.1375660330057144, 0.1792406365275383, 0.13293640414873759, 0.17345188756783803, 0.17141750653584797, 0.21743928492069245, 0.24442363977432252, 0.19766809940338134, 0.10855132043361664, 0.10980408986409505, 0.21591163178284964, 0.14524314006169636, 0.14545256197452544, 0.2930703043937683, 0.15860677063465117, 0.14155150552590687, 0.09201516608397166, 0.13191987425088883]",0.934640522875817,2.2872278690338135,squeezenet1_1,True,True 7 | 2,8194,382.0219066143036,0.19555958012739819,"[0.4001276771227519, 0.310506143172582, 0.23651925325393677, 0.18007902751366298, 0.18294470881422362, 0.1772422303756078, 0.32518666932980217, 0.15456459845105808, 0.1438033568362395, 0.15566440671682358, 0.19824622323115668, 0.26507751444975536, 0.14460683663686116, 0.16236888604859512, 0.17159015734990438, 0.10164796014626822, 0.11712987770636876, 0.22701976038515567, 0.06703527470429739, 0.2144110011557738, 0.40434248447418214, 0.2274980825682481, 0.23861868505676587, 0.09473046114047369, 0.14760183145602543, 0.20219092269738514, 0.06310451328754425, 0.233037564655145, 0.04047793100277583, 0.19555958012739819]",0.934640522875817,9.78577995300293,vgg13,True,True 8 | 16,57012034,147.88123750686646,0.6490961054960886,"[0.6932636777559916, 0.694413940111796, 0.6928423265616099, 0.6936238706111908, 0.6931376377741496, 0.6934309899806976, 0.6930472016334533, 0.6926954646905263, 0.6913259943326314, 0.6928755978743235, 0.6895035644372304, 0.6926084061463674, 0.6883116761843363, 0.6916109144687652, 0.6890701989332835, 0.6883516212304434, 0.686809488137563, 0.6821940978368123, 0.6829461753368378, 0.6869771520296732, 0.679001122713089, 0.6722644845644633, 0.6520185828208923, 0.6664509057998658, 0.6661665479342143, 0.6722269376118978, 0.5952397326628367, 0.6772902309894562, 0.7036851187547047, 0.6490961054960886]",0.5686274509803921,3.558337926864624,alexnet,False,False 9 | 508,14149480,295.5555033683777,0.6394421021143596,"[3.758546610673269, 0.735620665550232, 0.7424758195877075, 0.6638825178146363, 0.7221163352330525, 0.644890832901001, 0.6815452257792155, 0.7064140717188517, 0.7258037646611532, 0.647605037689209, 0.6827737013498942, 0.7231933116912842, 0.6148008187611897, 0.7050951480865478, 0.6375233173370362, 0.646649964650472, 0.6387109676996867, 0.6555142800013224, 0.6029780467351278, 0.6045679092407227, 0.5591812372207642, 0.6446251392364502, 0.5901364167531331, 0.7067879438400269, 0.6704655885696411, 0.6535025517145793, 0.6707207918167114, 0.5821940183639527, 0.6732496658960978, 0.6394421021143596]",0.6797385620915033,7.686006784439087,densenet169,False,False 10 | 292,24348900,283.5886549949646,2.0593234837055205,"[1.9816147605578105, 2.151238536834717, 1.9586544593175252, 1.8817228436470033, 1.8949224571386973, 2.1144465227921803, 1.7105423301458358, 1.9395068109035491, 2.147524134318034, 2.3047763387362163, 1.8329116801420848, 2.0384798189004263, 2.198727269967397, 1.9577059944470723, 1.6804665486017862, 2.304601134856542, 2.4338692585627237, 2.023895623286565, 1.6520379881064098, 1.8802327811717987, 1.800662056605021, 1.8977745612462362, 1.9112554252147675, 2.0675816257794697, 2.046035075187683, 2.079533820350965, 2.057239713271459, 1.8226490914821625, 2.01467671195666, 2.0593234837055205]",0.5620915032679739,5.32808780670166,inception_v3,False,False 11 | 110,21285698,160.13548517227173,0.7010385453701019,"[0.94874327480793, 1.092853771150112, 0.8724816049138705, 0.8953985025485357, 0.7325556933879852, 1.0162584781646729, 0.9359619110822678, 0.7451163043578466, 0.7827076544364293, 0.8569934864838918, 0.8505562404791515, 0.8296958049138387, 0.8872446556886037, 0.8547310719887415, 1.0265008916457494, 1.0036779016256332, 0.7647147918740909, 0.8305339803298314, 0.7238698263963064, 0.7955250819524129, 0.9017803331216176, 0.8808030058940252, 0.8449353764454524, 0.8015545040369034, 0.9266935169696808, 0.7978487660487493, 0.8536848386128744, 0.9325247868895531, 0.8030649850765864, 0.7010385453701019]",0.6535947712418301,3.1701908111572266,resnet34,False,False 12 | 52,723522,71.20375370979309,0.6931471824645996,"[0.7200395142038664, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996, 0.6931471824645996]",0.45751633986928103,2.3948652744293213,squeezenet1_1,False,False 13 | 26,128959042,425.7545166015625,0.5962312410275141,"[0.7187932193279266, 0.6987431387106577, 0.6889206151167552, 0.6943059821923574, 0.6995935201644897, 0.6909210940202077, 0.6923192103703817, 0.6815319299697876, 0.6842368513345718, 0.6594259450833003, 0.6203869849443435, 0.6567952136198679, 0.6159723569949468, 0.640097905198733, 0.6382866154114405, 0.6094239433606465, 0.6413209572434425, 0.5826000889142354, 0.6356521238883336, 0.629578513900439, 0.6123635977506637, 0.6444858074188232, 0.6064973920583725, 0.6132155150175095, 0.6016938954591751, 0.6048149541020393, 0.6670526494582494, 0.5485491409897805, 0.6236189002792041, 0.5962312410275141]",0.6993464052287581,9.827883243560791,vgg13,False,False 14 | 16,57012034,145.4061324596405,0.3348389188448588,"[0.9058507854739825, 0.8422722816467285, 0.6832410077253978, 0.6694998492797216, 0.6638702193895976, 0.652396152416865, 0.5959371795256933, 0.6602226515611013, 0.5815990924835205, 0.5585591087738673, 0.5489123597741127, 0.6074581888814767, 0.5729220708211263, 0.6321026116609574, 0.608658962448438, 0.6026650200287501, 0.5889960353573164, 0.6382711047927538, 0.5468498448530833, 0.4546732917428017, 0.5000830709934234, 0.41874629954497017, 0.5144528771440188, 0.497105872631073, 0.2949995664258798, 0.6161103417476018, 0.43020301361878716, 0.40001582851012546, 0.39930335332949957, 0.3348389188448588]",0.7712418300653595,3.461834669113159,alexnet,True,False 15 | 508,12487810,292.345685005188,0.3186530863245328,"[0.6880594919125239, 0.5986983786026637, 0.4366995764275392, 0.8134673890968164, 0.887651317814986, 0.961603298286597, 0.6706142778197924, 0.7791942156851291, 0.7763946240146955, 0.6667415117224057, 0.43815006216367086, 0.6352087785800298, 0.560955377916495, 0.5578066587448121, 0.5151483791569869, 0.47221326182285944, 0.45165164371331534, 0.5341445614894231, 0.4635127648711205, 0.4361243749658267, 0.4244649996360143, 0.516990474363168, 0.3033191055059433, 0.41304358740647634, 0.36657139559586843, 0.37128164718548456, 0.40548308144013084, 0.35532402296861015, 0.3607709174354871, 0.3186530863245328]",0.8758169934640523,7.3409364223480225,densenet169,True,False 16 | 292,24348900,284.9803819656372,1.3199121822913489,"[1.8060940543810526, 1.5766649305820466, 1.5660042454799017, 1.6844182809193928, 1.8788533687591553, 1.7865262349446616, 2.113397620121638, 1.9967733124891918, 1.6930860122044882, 1.6454144696394601, 1.8579031467437743, 1.3389902373154958, 1.422914340098699, 1.6296266744534174, 1.3542779405911765, 1.7145804385344188, 1.6303958157698313, 1.3544182101885478, 1.3934075593948365, 1.3053396438558897, 1.2429701541860898, 1.4396466116110485, 1.3725314676761626, 1.3084830582141875, 1.2546294589837392, 1.3633785138527552, 1.4531818360090256, 1.3683309962352117, 1.5537205696105958, 1.3199121822913489]",0.803921568627451,5.308340787887573,inception_v3,True,False 17 | 110,21285698,160.45222806930542,0.4099969615538915,"[0.7902858893076579, 0.6412704070409139, 0.6075814704100291, 0.6192416568597158, 0.4730793411533038, 0.7957594017187755, 0.7570452819267909, 0.5804953868190448, 0.43717803110678993, 0.7785061250130335, 0.5714593350887298, 0.5947339942057928, 0.8420850122968356, 0.6745557328065236, 0.5376502513885498, 0.545789486169815, 0.6476305951674779, 0.5324482609828313, 0.5018530562520027, 0.7059176231424014, 0.59167833228906, 1.0117873390515646, 0.6448751380046208, 0.8215991308291754, 0.47594111661116284, 0.44203995565573373, 0.5368657032648723, 0.39744592358668646, 0.5040965192019939, 0.4099969615538915]",0.8627450980392157,3.0336742401123047,resnet34,True,False 18 | 52,723522,70.21874809265137,0.29626329789559047,"[0.6143193090955417, 0.6626703063646953, 0.6443698267141978, 0.5974467446406683, 0.6635206907987594, 0.6043845295906067, 0.47457945843537647, 0.4070695127050082, 0.521741617222627, 0.5705462197462717, 0.5362809658050537, 0.43556203742822014, 0.27457153995831807, 0.5104670559366544, 0.3998982186118762, 0.37296838412682215, 0.3276210208733877, 0.3635485038161278, 0.408823694785436, 0.4283299664656321, 0.4040239264567693, 0.39194182952245077, 0.3939064621925354, 0.42542093197504677, 0.3820832873384158, 0.34889175593852995, 0.3046407972772916, 0.3264291614294052, 0.20972305188576382, 0.29626329789559047]",0.8104575163398693,2.2634453773498535,squeezenet1_1,True,False 19 | 26,128959042,424.3832137584686,0.12049836541215579,"[0.5031461690862974, 0.4505705595016479, 0.37233323554197945, 0.4654947402576605, 0.2699298625191053, 0.2949039816856384, 0.16165028139948845, 0.3186729773879051, 0.259407972296079, 0.31026825631658234, 0.12120863124728203, 0.19890003502368928, 0.1764173058172067, 0.24973918870091438, 0.1773315489292145, 0.09509727160135904, 0.22233242044846216, 0.1466854323943456, 0.12553381125132243, 0.08609176427125931, 0.06346224993467331, 0.23509883309404056, 0.13204210102558137, 0.10656530782580376, 0.1179654265443484, 0.07381095265348753, 0.04487273693084717, 0.0732261930902799, 0.13789080331722894, 0.12049836541215579]",0.9281045751633987,9.507586240768433,vgg13,True,False 20 | -------------------------------------------------------------------------------- /retrain.py: -------------------------------------------------------------------------------- 1 | import csv 2 | import time 3 | import os 4 | from glob import glob 5 | 6 | import torch 7 | import torch.optim as optim 8 | from torch import nn 9 | from torch.optim.lr_scheduler import CyclicLR 10 | 11 | import torchvision 12 | import torchvision.models as models 13 | import torchvision.transforms as transforms 14 | from torchvision import datasets 15 | 16 | from itertools import accumulate 17 | from functools import reduce 18 | 19 | input_sizes = { 20 | 'alexnet' : (224,224), 21 | 'densenet': (224,224), 22 | 'resnet' : (224,224), 23 | 'inception' : (299,299), 24 | 'squeezenet' : (224,224),#not 255,255 acc. to https://github.com/pytorch/pytorch/issues/1120 25 | 'vgg' : (224,224) 26 | } 27 | 28 | # ### Configuration 29 | models_to_test = ['alexnet', 'densenet169', 'inception_v3', \ 30 | 'resnet34', 'squeezenet1_1', 'vgg13'] 31 | #Todo: inception_v3 hangs at model construction at some 32 | #scipy innards 33 | models_to_test = ['alexnet', 'densenet169', \ 34 | 'resnet34', 'squeezenet1_1', 'vgg13'] 35 | 36 | #Todo: Argparse 37 | data_dir = 'xxx' 38 | 39 | train_subfolder = os.path.join(data_dir, 'train') 40 | classes = [d.split(train_subfolder, 1)[1] for d in \ 41 | glob(os.path.join(train_subfolder, '**'))] 42 | 43 | batch_size = 8 44 | epoch_multiplier = 4 #per class and times 1(shallow), 2(deep), 4(from_scratch) 45 | use_gpu = torch.cuda.is_available() 46 | use_clr = True 47 | 48 | #Assume 50 examples per class and CLR authors' middle ground 49 | clr_stepsize = (len(classes)*50//batch_size)*4 50 | 51 | 52 | print("Shootout of model(s) %s with batch_size %d running on CUDA %s " % \ 53 | (", ".join(models_to_test), batch_size, use_gpu) + \ 54 | "with CLR %s for %d classes on data in %s." % \ 55 | (use_clr, len(classes), data_dir)) 56 | 57 | 58 | # ### Generic pretrained model loading 59 | 60 | 61 | #We solve the dimensionality mismatch between 62 | #final layers in the constructed vs pretrained 63 | #modules at the data level. 64 | def diff_states(dict_canonical, dict_subset): 65 | names1, names2 = (list(dict_canonical.keys()), list(dict_subset.keys())) 66 | 67 | #Sanity check that param names overlap 68 | #Note that params are not necessarily in the same order 69 | #for every pretrained model 70 | not_in_1 = [n for n in names1 if n not in names2] 71 | not_in_2 = [n for n in names2 if n not in names1] 72 | assert len(not_in_1) == 0 73 | assert len(not_in_2) == 0 74 | 75 | for name, v1 in dict_canonical.items(): 76 | v2 = dict_subset[name] 77 | assert hasattr(v2, 'size') 78 | if v1.size() != v2.size(): 79 | yield (name, v1) 80 | 81 | def load_model_merged(name, num_classes): 82 | 83 | # Get model and state dict in idiomatic way 84 | model_cls = getattr(models, name) 85 | model = model_cls(num_classes=num_classes, pretrained=False) 86 | pretrained_state = model_cls(pretrained=True).state_dict() 87 | 88 | #Diff 89 | diff = [s for s in diff_states(model.state_dict(), pretrained_state)] 90 | print("Replacing the following state from initialized", name, ":", [d[0] for d in diff]) 91 | 92 | for name, value in diff: 93 | pretrained_state[name] = value 94 | 95 | assert len([s for s in diff_states(model.state_dict(), pretrained_state)]) == 0 96 | 97 | #Merge 98 | model.load_state_dict(pretrained_state) 99 | return model, diff 100 | 101 | 102 | def filtered_params(net, param_list=None): 103 | def in_param_list(s): 104 | for p in param_list: 105 | if s.endswith(p): 106 | return True 107 | return False 108 | #Caution: DataParallel prefixes '.module' to every parameter name 109 | params = net.named_parameters() if param_list is None else (p for p in net.named_parameters() if in_param_list(p[0]) and p[1].requires_grad) 110 | return params 111 | 112 | 113 | #Todo: split function into separate test and train data 114 | #To get the tutorial data (bee vs. ants), go to: 115 | #http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html 116 | def get_data(resize): 117 | 118 | data_transforms = { 119 | 'train': transforms.Compose([ 120 | transforms.RandomResizedCrop(max(resize)), 121 | transforms.RandomHorizontalFlip(), 122 | transforms.ToTensor(), 123 | transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) 124 | ]), 125 | 'val': transforms.Compose([ 126 | #Higher scale-up for inception 127 | transforms.Resize(int(max(resize)/224*256)), 128 | transforms.CenterCrop(max(resize)), 129 | transforms.ToTensor(), 130 | transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]) 131 | ]), 132 | } 133 | 134 | dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) 135 | for x in ['train', 'val']} 136 | dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=batch_size, 137 | shuffle=True) 138 | for x in ['train', 'val']} 139 | 140 | return dset_loaders['train'], dset_loaders['val'] 141 | 142 | 143 | def train(net, trainloader, epochs, param_list=None, CLR=False): 144 | #Todo: DRY 145 | def in_param_list(s): 146 | for p in param_list: 147 | if s.endswith(p): 148 | return True 149 | return False 150 | 151 | criterion = nn.CrossEntropyLoss() 152 | if use_gpu: 153 | criterion = criterion.cuda() 154 | 155 | #If finetuning model, turn off grad for other params and make sure to turn on others 156 | for p in net.named_parameters(): 157 | p[1].requires_grad = (param_list is None) or in_param_list(p[0]) 158 | 159 | params = (p for p in filtered_params(net, param_list)) 160 | 161 | #Optimizer as in tutorial 162 | optimizer = optim.SGD((p[1] for p in params), lr=0.001, momentum=0.9) 163 | if CLR: 164 | 165 | global clr_stepsize 166 | clr_wrapper = CyclicLR(optimizer, base_lr=0.0001, max_lr=0.002, 167 | step_size_up=clr_stepsize, step_size_down=clr_stepsize) 168 | 169 | losses = [] 170 | for epoch in range(epochs): 171 | 172 | running_loss = 0.0 173 | for i, data in enumerate(trainloader, 0): 174 | # get the inputs 175 | inputs, labels = data 176 | if use_gpu: 177 | inputs, labels = inputs.cuda(), labels.cuda() 178 | 179 | # zero the parameter gradients 180 | optimizer.zero_grad() 181 | 182 | # forward + backward + optimize 183 | outputs = net(inputs) 184 | 185 | loss = None 186 | # for nets that have multiple outputs such as inception 187 | if isinstance(outputs, tuple): 188 | loss = sum((criterion(o,labels) for o in outputs)) 189 | else: 190 | loss = criterion(outputs, labels) 191 | loss.backward() 192 | optimizer.step() 193 | if CLR: 194 | clr_wrapper.step() 195 | 196 | # print statistics 197 | running_loss += loss.item() 198 | if i % 30 == 29: 199 | avg_loss = running_loss / 30 200 | losses.append(avg_loss) 201 | 202 | lrs = [p['lr'] for p in optimizer.param_groups] 203 | 204 | print('[%d, %5d] loss: %.3f' % 205 | (epoch + 1, i + 1, avg_loss), lrs) 206 | running_loss = 0.0 207 | 208 | print('Finished Training') 209 | return losses 210 | 211 | 212 | def train_stats(m, trainloader, epochs, param_list=None, CLR=False): 213 | """ 214 | Get stats for training and evaluation in a structured way 215 | If param_list is None all relevant parameters are tuned, 216 | otherwise, only parameters that have been constructed for custom 217 | num_classes 218 | """ 219 | stats = {} 220 | params = filtered_params(m, param_list) 221 | counts = 0,0 222 | for counts in enumerate(accumulate((reduce(lambda d1,d2: d1*d2, p[1].size()) for p in params)) ): 223 | pass 224 | stats['variables_optimized'] = counts[0] + 1 225 | stats['params_optimized'] = counts[1] 226 | 227 | before = time.time() 228 | losses = train(m, trainloader, epochs, param_list=param_list, CLR=CLR) 229 | stats['training_time'] = time.time() - before 230 | 231 | stats['training_loss'] = losses[-1] if len(losses) else float('nan') 232 | stats['training_losses'] = losses 233 | 234 | return stats 235 | 236 | def evaluate_stats(net, testloader): 237 | stats = {} 238 | correct = 0 239 | total = 0 240 | 241 | before = time.time() 242 | for i, data in enumerate(testloader, 0): 243 | images, labels = data 244 | 245 | if use_gpu: 246 | images, labels = images.cuda(), labels.cuda() 247 | 248 | outputs = net(images) 249 | _, predicted = torch.max(outputs.data, 1) 250 | total += labels.size(0) 251 | correct += (predicted == labels).sum().cpu().item() 252 | accuracy = correct / total 253 | stats['accuracy'] = accuracy 254 | stats['eval_time'] = time.time() - before 255 | 256 | print('Accuracy on test images: %f' % accuracy) 257 | return stats 258 | 259 | 260 | def train_eval(net, trainloader, testloader, epochs, param_list=None, CLR=False): 261 | print("Training..." if not param_list else "Retraining...") 262 | stats_train = train_stats(net, trainloader, epochs, param_list=param_list, CLR=CLR) 263 | 264 | print("Evaluating...") 265 | net = net.eval() 266 | with torch.no_grad(): 267 | stats_eval = evaluate_stats(net, testloader) 268 | 269 | return {**stats_train, **stats_eval} 270 | 271 | 272 | if __name__=='__main__': 273 | stats = [] 274 | t = 0.0 275 | num_classes = len(classes) 276 | 277 | #Retraining shallow 278 | epochs = num_classes * epoch_multiplier * 1 279 | print("RETRAINING %d epochs" % epochs) 280 | 281 | for name in models_to_test: 282 | print("") 283 | print("Targeting %s with %d classes" % (name, num_classes)) 284 | print("------------------------------------------") 285 | model_pretrained, diff = load_model_merged(name, num_classes) 286 | final_params = [d[0] for d in diff] 287 | 288 | resize = [s[1] for s in input_sizes.items() if s[0] in name][0] 289 | print("Resizing input images to max of", resize) 290 | trainloader, testloader = get_data(resize) 291 | 292 | if use_gpu: 293 | print("Transfering models to GPU(s)") 294 | model_pretrained = torch.nn.DataParallel(model_pretrained).cuda() 295 | 296 | pretrained_stats = train_eval(model_pretrained, 297 | trainloader, testloader, epochs, 298 | final_params, use_clr) 299 | pretrained_stats['name'] = name 300 | pretrained_stats['retrained'] = True 301 | pretrained_stats['shallow_retrain'] = True 302 | stats.append(pretrained_stats) 303 | 304 | print("") 305 | 306 | #Training from scratch 307 | epochs = num_classes * epoch_multiplier * 4 308 | print("TRAINING %d epochs from scratch" % epochs) 309 | 310 | for name in models_to_test: 311 | print("") 312 | print("Targeting %s with %d classes" % (name, num_classes)) 313 | print("------------------------------------------") 314 | model_blank = models.__dict__[name](num_classes=num_classes) 315 | 316 | resize = [s[1] for s in input_sizes.items() if s[0] in name][0] 317 | print("Resizing input images to max of", resize) 318 | trainloader, testloader = get_data(resize) 319 | 320 | if use_gpu: 321 | print("Transfering models to GPU(s)") 322 | model_blank = torch.nn.DataParallel(model_blank).cuda() 323 | 324 | blank_stats = train_eval(model_pretrained, trainloader, testloader, epochs, None, 325 | CLR=use_clr) 326 | blank_stats['name'] = name 327 | blank_stats['retrained'] = False 328 | blank_stats['shallow_retrain'] = False 329 | stats.append(blank_stats) 330 | 331 | print("") 332 | 333 | #Retraining deep 334 | epochs = num_classes * epoch_multiplier * 2 335 | print("RETRAINING %d epochs deeply" % epochs) 336 | 337 | for name in models_to_test: 338 | print("") 339 | print("Targeting %s with %d classes" % (name, num_classes)) 340 | print("------------------------------------------") 341 | model_pretrained, diff = load_model_merged(name, num_classes) 342 | 343 | resize = [s[1] for s in input_sizes.items() if s[0] in name][0] 344 | print("Resizing input images to max of", resize) 345 | trainloader, testloader = get_data(resize) 346 | 347 | if use_gpu: 348 | print("Transfering models to GPU(s)") 349 | model_pretrained = torch.nn.DataParallel(model_pretrained).cuda() 350 | 351 | pretrained_stats = train_eval(model_pretrained, trainloader, testloader, 352 | epochs, None,CLR=use_clr) 353 | pretrained_stats['name'] = name 354 | pretrained_stats['retrained'] = True 355 | pretrained_stats['shallow_retrain'] = False 356 | stats.append(pretrained_stats) 357 | 358 | print("") 359 | 360 | 361 | for s in stats: 362 | t += s['eval_time'] + s['training_time'] 363 | print("Total time for training and evaluation", t) 364 | print("FINISHED") 365 | 366 | #Export 367 | with open(data_dir+('_clr' if use_clr else '')+'.csv', 'w') as csvfile: 368 | fieldnames = stats[0].keys() 369 | writer = csv.DictWriter(csvfile, fieldnames=fieldnames) 370 | 371 | writer.writeheader() 372 | for s in stats: 373 | writer.writerow(s) 374 | 375 | --------------------------------------------------------------------------------