├── .ipynb_checkpoints └── README-checkpoint.md ├── LICENSE ├── README.md ├── paper-code ├── C-discover-symmetry │ ├── .ipynb_checkpoints │ │ └── L-conv-discover-large-angle2021-06-04-checkpoint.ipynb │ ├── L-conv-discover-2021-06-04.ipynb │ └── L-conv-discover-large-angle2021-06-04.ipynb └── D-image-experiments │ ├── L-conv-extra-exp-2020-11-21.ipynb │ ├── L-conv-extra-exp-2020-11-24.ipynb │ ├── L-conv-usage-vis-2020-11-23.ipynb │ ├── lconv.py │ ├── run_LieConv_cifar100.py │ ├── run_test-v2.py │ └── run_test.py └── src ├── .ipynb_checkpoints ├── examples-checkpoint.ipynb └── lconv-checkpoint.py ├── __pycache__ └── lconv.cpython-38.pyc ├── examples.ipynb └── lconv.py /.ipynb_checkpoints/README-checkpoint.md: -------------------------------------------------------------------------------- 1 | # Lie Algebra Convolutional Network (L-conv) implementation 2 | __Paper:__ [Automatic Symmetry Discovery with Lie Algebra Convolutional Network](https://papers.nips.cc/paper/2021/file/148148d62be67e0916a833931bd32b26-Paper.pdf) _Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, Rose Yu_ NeurIPS 2021 3 | (find updated versions on [arxiv](https://arxiv.org/abs/2109.07103)) 4 | 5 | 6 | ## Contents 7 | A simple implementation of the L-conv layer in PyTorch (>=1.8) can be found in `src/lconv.py`. 8 | The L-conv layer acts similar to a graph convlutional layer (GCN), so prepare your input in a similar fashion (e.g. flatten the spatial dimensions). 9 | The input should have shape `(batch, channels, #nodes)` (e.g. on an image, # nodes = # pixels) 10 | This repository also contains code and notebooks for the experiemnts in the paper (appendix C and D) under `paper-code`. 11 | Most experiments in appendix D use an older (but identical) implementation in Tensoflow (>=2.1). 12 | Comparison with LieConv in appendix D requires the [LieConv](https://github.com/mfinzi/LieConv) packages. 13 | 14 | ### TBA soon: 15 | Exmaples of uses will be added soon. 16 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Nima Dehmamy 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lie Algebra Convolutional Network (L-conv) implementation 2 | __Paper:__ [Automatic Symmetry Discovery with Lie Algebra Convolutional Network](https://papers.nips.cc/paper/2021/file/148148d62be67e0916a833931bd32b26-Paper.pdf) _Nima Dehmamy, Robin Walters, Yanchen Liu, Dashun Wang, Rose Yu_ NeurIPS 2021 3 | (find updated versions on [arxiv](https://arxiv.org/abs/2109.07103)) 4 | 5 | 6 | ## Contents 7 | A simple implementation of the L-conv layer in PyTorch (>=1.8) can be found in `src/lconv.py`. 8 | The L-conv layer acts similar to a graph convlutional layer (GCN), so prepare your input in a similar fashion (e.g. flatten the spatial dimensions). 9 | The input should have shape `(batch, channels, #nodes)` (e.g. on an image, # nodes = # pixels) 10 | This repository also contains code and notebooks for the experiemnts in the paper (appendix C and D) under `paper-code`. 11 | Most experiments in appendix D use an older (but identical) implementation in Tensoflow (>=2.1). 12 | Comparison with LieConv in appendix D requires the [LieConv](https://github.com/mfinzi/LieConv) packages. 13 | 14 | ### TBA soon: 15 | Exmaples of uses will be added soon. 16 | -------------------------------------------------------------------------------- /paper-code/D-image-experiments/L-conv-extra-exp-2020-11-21.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "colab": { 8 | "base_uri": "https://localhost:8080/", 9 | "height": 34 10 | }, 11 | "colab_type": "code", 12 | "id": "L6Q9A-AvgeCn", 13 | "outputId": "c2755a01-3585-4ab1-e378-a3eb70478363" 14 | }, 15 | "outputs": [ 16 | { 17 | "name": "stdout", 18 | "output_type": "stream", 19 | "text": [ 20 | "Populating the interactive namespace from numpy and matplotlib\n" 21 | ] 22 | } 23 | ], 24 | "source": [ 25 | "%pylab inline\n", 26 | "\n", 27 | "%config InlineBackend.figure_format = 'retina'" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": 2, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "import lconv\n", 37 | "import tensorflow as tf\n", 38 | "\n", 39 | "import os\n", 40 | "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\" \n", 41 | "\n", 42 | "sess = tf.compat.v1.InteractiveSession()\n", 43 | "# K = tf.keras.backend\n", 44 | "\n", 45 | "from tensorflow.keras import Model, Sequential\n", 46 | "from tensorflow.keras.layers import Layer, Input, Flatten, Reshape, Dense, Conv2D, MaxPool2D\n", 47 | "\n", 48 | "import pickle as pk\n", 49 | "import json\n", 50 | "# import numpy as np\n", 51 | "\n", 52 | "\n", 53 | "from scipy import ndimage\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 3, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "\n", 63 | "def rotated_ims_rand(x):\n", 64 | " return np.float32([ndimage.rotate(i, (np.random.rand()-.5)*180, reshape=False, mode='nearest') for i in x])\n", 65 | " \n", 66 | "class Scramble_x:\n", 67 | " def __init__(self,x):\n", 68 | " s = x.shape[1:-1]\n", 69 | " self.idx = np.argsort(np.random.rand(np.prod(s)))\n", 70 | " r,c = np.int0(self.idx/s[0]), (self.idx % s[1]) \n", 71 | " self.x = np.float32([i[r,c].reshape(s+(x.shape[-1],)) for i in x])\n", 72 | "\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 4, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "\n", 82 | "# Defaults \n", 83 | "configs= {\n", 84 | " 'dataset': dict(name= 'cifar100', #'mnist' ,#'mnist',cifar100, \n", 85 | " rotate=True, \n", 86 | " scramble=True,\n", 87 | " ),\n", 88 | " 'net': dict(architecture= 'lconv', # 'cnn', 'fc', \n", 89 | " num_filters=32, \n", 90 | " kernel_size=9, \n", 91 | " L_hid= [8], #[16], \n", 92 | " activation = 'relu',\n", 93 | " L_trainable = True,\n", 94 | " num_layers = 1,\n", 95 | " ),\n", 96 | "}\n" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 5, 102 | "metadata": {}, 103 | "outputs": [ 104 | { 105 | "name": "stdout", 106 | "output_type": "stream", 107 | "text": [ 108 | "Rotating images\n", 109 | "Scrambling images\n" 110 | ] 111 | } 112 | ], 113 | "source": [ 114 | "\n", 115 | "dataset_name = configs['dataset']['name']\n", 116 | "\n", 117 | "dataset = eval(\"tf.keras.datasets.%s.load_data()\" %dataset_name) \n", 118 | "(x_train, y_train), (x_test,y_test) = dataset\n", 119 | "if len(x_train.shape) == 3:\n", 120 | " # mnist channel is missing\n", 121 | " x_train = x_train[...,np.newaxis]\n", 122 | " \n", 123 | "# normalize\n", 124 | "x_train = x_train/x_train[:100].max() -.5 \n", 125 | "# make categorical\n", 126 | "y_train = tf.keras.utils.to_categorical(y_train)\n", 127 | "\n", 128 | "\n", 129 | "results = {'configs':configs,}\n", 130 | "\n", 131 | "if configs['dataset']['rotate']:\n", 132 | " print('Rotating images')\n", 133 | " x_train = rotated_ims_rand(x_train)\n", 134 | " \n", 135 | "if configs['dataset']['scramble']:\n", 136 | " print('Scrambling images')\n", 137 | " scr = Scramble_x(x_train)\n", 138 | " x_train = scr.x\n", 139 | " results['scramble_idx']=scr.idx.tolist()\n", 140 | "\n", 141 | "##### Make model #####\n", 142 | "\n", 143 | "net = configs['net']\n", 144 | "# arch = net['architecture']\n" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 6, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "Model: \"functional_1\"\n", 157 | "_________________________________________________________________\n", 158 | "Layer (type) Output Shape Param # \n", 159 | "=================================================================\n", 160 | "input_1 (InputLayer) [(None, 32, 32, 3)] 0 \n", 161 | "_________________________________________________________________\n", 162 | "tf_op_layer_Reshape (TensorF [(None, 1024, 3)] 0 \n", 163 | "_________________________________________________________________\n", 164 | "l__conv (L_Conv) (None, 1024, 32) 132224 \n", 165 | "_________________________________________________________________\n", 166 | "flatten (Flatten) (None, 32768) 0 \n", 167 | "_________________________________________________________________\n", 168 | "dense (Dense) (None, 100) 3276900 \n", 169 | "=================================================================\n", 170 | "Total params: 3,409,124\n", 171 | "Trainable params: 3,409,124\n", 172 | "Non-trainable params: 0\n", 173 | "_________________________________________________________________\n" 174 | ] 175 | } 176 | ], 177 | "source": [ 178 | "net = configs['net']\n", 179 | "\n", 180 | "inp = Input(x_train[0].shape)\n", 181 | "\n", 182 | "x = inp\n", 183 | "for _ in range(net['num_layers']):\n", 184 | " x = tf.reshape(x, shape=(-1,np.prod(x.shape[1:-1]), x.shape[-1]))\n", 185 | " lay = lconv.L_Conv(num_filters= net['num_filters'], \n", 186 | " kernel_size= net['kernel_size'], \n", 187 | " L_hid = net['L_hid'], \n", 188 | " activation = net['activation'],)\n", 189 | "\n", 190 | " x = lay(x)\n", 191 | " lay.L.trainable = net['L_trainable']\n", 192 | "\n", 193 | "\n", 194 | "x = Flatten()(x)\n", 195 | "\n", 196 | "# x = Dense(100, activation = 'relu')(x)\n", 197 | "\n", 198 | "out = Dense(y_train.shape[-1], activation='softmax')(x)\n", 199 | "\n", 200 | "model = Model(inputs = [inp], outputs = [out])\n", 201 | "model.compile(loss = tf.keras.losses.categorical_crossentropy, metrics = ['accuracy'])\n", 202 | "\n", 203 | "model.summary()\n", 204 | "\n" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": 7, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "h = model.fit(x_train, y_train, validation_split=0.2, epochs=10)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "# Shallow FC with parameters matching the L-conv model " 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "metadata": {}, 226 | "source": [ 227 | "The parameters \n", 228 | "xs * u + u* 10" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": 8, 234 | "metadata": {}, 235 | "outputs": [ 236 | { 237 | "data": { 238 | "text/plain": [ 239 | "1075" 240 | ] 241 | }, 242 | "execution_count": 8, 243 | "metadata": {}, 244 | "output_type": "execute_result" 245 | } 246 | ], 247 | "source": [ 248 | "xs = np.prod(inp.shape[1:])\n", 249 | "# u = int(0.5+ lay.count_params()/ xs)\n", 250 | "u = int(.5+ model.count_params() / (model.output_shape[-1] + xs ))\n", 251 | "u" 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": 9, 257 | "metadata": {}, 258 | "outputs": [ 259 | { 260 | "name": "stdout", 261 | "output_type": "stream", 262 | "text": [ 263 | "Model: \"functional_3\"\n", 264 | "_________________________________________________________________\n", 265 | "Layer (type) Output Shape Param # \n", 266 | "=================================================================\n", 267 | "input_2 (InputLayer) [(None, 32, 32, 3)] 0 \n", 268 | "_________________________________________________________________\n", 269 | "flatten_1 (Flatten) (None, 3072) 0 \n", 270 | "_________________________________________________________________\n", 271 | "dense_1 (Dense) (None, 1075) 3303475 \n", 272 | "_________________________________________________________________\n", 273 | "dense_2 (Dense) (None, 100) 107600 \n", 274 | "=================================================================\n", 275 | "Total params: 3,411,075\n", 276 | "Trainable params: 3,411,075\n", 277 | "Non-trainable params: 0\n", 278 | "_________________________________________________________________\n" 279 | ] 280 | } 281 | ], 282 | "source": [ 283 | "net = configs['net']\n", 284 | "\n", 285 | "inp = Input(x_train[0].shape)\n", 286 | "\n", 287 | "x = inp\n", 288 | "\n", 289 | "x = Flatten()(inp)\n", 290 | "# FC comparable to L-conv, but no shared weights \n", 291 | "x = Dense(u, activation = net['activation'])(x)\n", 292 | "\n", 293 | "out = Dense(y_train.shape[-1], activation='softmax')(x)\n", 294 | "\n", 295 | "model = Model(inputs = [inp], outputs = [out])\n", 296 | "model.compile(loss = tf.keras.losses.categorical_crossentropy, metrics = ['accuracy'])\n", 297 | "\n", 298 | "model.summary()\n", 299 | "\n" 300 | ] 301 | }, 302 | { 303 | "cell_type": "code", 304 | "execution_count": 10, 305 | "metadata": {}, 306 | "outputs": [ 307 | { 308 | "name": "stdout", 309 | "output_type": "stream", 310 | "text": [ 311 | "Epoch 1/30\n", 312 | "1250/1250 [==============================] - 5s 4ms/step - loss: 4.3505 - accuracy: 0.1206 - val_loss: 4.1713 - val_accuracy: 0.1420\n", 313 | "Epoch 2/30\n", 314 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.9070 - accuracy: 0.1848 - val_loss: 4.2039 - val_accuracy: 0.1650\n", 315 | "Epoch 3/30\n", 316 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.7058 - accuracy: 0.2255 - val_loss: 4.1117 - val_accuracy: 0.1878\n", 317 | "Epoch 4/30\n", 318 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.5451 - accuracy: 0.2544 - val_loss: 4.8140 - val_accuracy: 0.1781\n", 319 | "Epoch 5/30\n", 320 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.3958 - accuracy: 0.2869 - val_loss: 4.3992 - val_accuracy: 0.1821\n", 321 | "Epoch 6/30\n", 322 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.2576 - accuracy: 0.3169 - val_loss: 4.5950 - val_accuracy: 0.1988\n", 323 | "Epoch 7/30\n", 324 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.1092 - accuracy: 0.3451 - val_loss: 4.9388 - val_accuracy: 0.1893\n", 325 | "Epoch 8/30\n", 326 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.9819 - accuracy: 0.3800 - val_loss: 5.1262 - val_accuracy: 0.1933\n", 327 | "Epoch 9/30\n", 328 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.8600 - accuracy: 0.4033 - val_loss: 5.4323 - val_accuracy: 0.1927\n", 329 | "Epoch 10/30\n", 330 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.7234 - accuracy: 0.4313 - val_loss: 5.6304 - val_accuracy: 0.1902\n", 331 | "Epoch 11/30\n", 332 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.6202 - accuracy: 0.4598 - val_loss: 5.6795 - val_accuracy: 0.2021\n", 333 | "Epoch 12/30\n", 334 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.4936 - accuracy: 0.4812 - val_loss: 6.2062 - val_accuracy: 0.1952\n", 335 | "Epoch 13/30\n", 336 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.3621 - accuracy: 0.5110 - val_loss: 6.3498 - val_accuracy: 0.2006\n", 337 | "Epoch 14/30\n", 338 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.2828 - accuracy: 0.5314 - val_loss: 6.8185 - val_accuracy: 0.2076\n", 339 | "Epoch 15/30\n", 340 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.1961 - accuracy: 0.5562 - val_loss: 6.9912 - val_accuracy: 0.1992\n", 341 | "Epoch 16/30\n", 342 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.0824 - accuracy: 0.5715 - val_loss: 7.3662 - val_accuracy: 0.2000\n", 343 | "Epoch 17/30\n", 344 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.0096 - accuracy: 0.5952 - val_loss: 7.7374 - val_accuracy: 0.1947\n", 345 | "Epoch 18/30\n", 346 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.9498 - accuracy: 0.6051 - val_loss: 8.0833 - val_accuracy: 0.2032\n", 347 | "Epoch 19/30\n", 348 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.8755 - accuracy: 0.6250 - val_loss: 8.6338 - val_accuracy: 0.1993\n", 349 | "Epoch 20/30\n", 350 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.7955 - accuracy: 0.6419 - val_loss: 8.6264 - val_accuracy: 0.2086\n", 351 | "Epoch 21/30\n", 352 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.7687 - accuracy: 0.6562 - val_loss: 8.9263 - val_accuracy: 0.2076\n", 353 | "Epoch 22/30\n", 354 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.6695 - accuracy: 0.6693 - val_loss: 9.6866 - val_accuracy: 0.2030\n", 355 | "Epoch 23/30\n", 356 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.6299 - accuracy: 0.6830 - val_loss: 9.7473 - val_accuracy: 0.2070\n", 357 | "Epoch 24/30\n", 358 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.5655 - accuracy: 0.6931 - val_loss: 10.7333 - val_accuracy: 0.2005\n", 359 | "Epoch 25/30\n", 360 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.5255 - accuracy: 0.7073 - val_loss: 10.8067 - val_accuracy: 0.2012\n", 361 | "Epoch 26/30\n", 362 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.4941 - accuracy: 0.7142 - val_loss: 10.9942 - val_accuracy: 0.2002\n", 363 | "Epoch 27/30\n", 364 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.4608 - accuracy: 0.7237 - val_loss: 11.6645 - val_accuracy: 0.2034\n", 365 | "Epoch 28/30\n", 366 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.3985 - accuracy: 0.7359 - val_loss: 11.6220 - val_accuracy: 0.2058\n", 367 | "Epoch 29/30\n", 368 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.3767 - accuracy: 0.7433 - val_loss: 12.3072 - val_accuracy: 0.2017\n", 369 | "Epoch 30/30\n", 370 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.3341 - accuracy: 0.7549 - val_loss: 12.7778 - val_accuracy: 0.1996\n" 371 | ] 372 | } 373 | ], 374 | "source": [ 375 | "h = model.fit(x_train, y_train, validation_split=0.2, epochs=30)" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": 11, 381 | "metadata": {}, 382 | "outputs": [ 383 | { 384 | "data": { 385 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAusAAAHwCAYAAAAfACH4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeXxU5b3H8e8z2ROyEEhYsrBDQCDsoBQVtYprQatUrSIurUtL983WqlXb21uvW6/euouI1KUqUlurUtQq+74FwhYSAiQhITvZJs/9Y8JASCKBDDmTzOf9evk6nd8zc/K7rwvJl5Pz/I6x1goAAACA/3E53QAAAACA5hHWAQAAAD9FWAcAAAD8FGEdAAAA8FOEdQAAAMBPEdYBAAAAP0VYBwAAAPwUYR0AAADwU4R1AAAAwE8R1gEAAAA/RVgHAAAA/BRhHQAAAPBTwU430J6MMXskxUjKcrgVAAAAdG59JZVaa/u15SQBFdYlxURERMQPHTo03ulGAAAA0HllZGToyJEjbT5PoIX1rKFDh8avWbPG6T4AAADQiY0dO1Zr167Naut5uGcdAAAA8FOEdQAAAMBPEdYBAAAAP0VYBwAAAPwUYR0AAADwU4R1AAAAwE8R1gEAAAA/RVgHAAAA/BRhHQAAAPBThHUAAADATxHWAQAAAD9FWAcAAAD8FGEdAAAA8FOEdQAAAMBPEdYBAAAAP0VYBwAAQKeVU1SpnfllTrdx2oKdbgAAAADwpbzSKn2w8YAWbdyvddnFunhYDz138zin2zothHUAAAB0eEUVNfrn5gNatGG/VuwpkrXH1j7dXqDSqlrFhIc41+BpIqwDAACgQyqrqtVHW/K0aON+fbHjkOrqbZP3BLuMJg/spsMVNYR1AAAA4Ew6UuPW4m15WrRhv5ZsL1BNXX2T9xgjnd2/m65M761pZ/VU16hQBzr1DcI6AAAA/Fp1nVv/yTykRRv36+OteaqscTf7vjGpcboyvbcuH9FLiTHh7dzlmUFYBwAAgN+pc9dr2e5CLdqwXx9uPqjSqrpm3zesV4yuGuUJ6Cnxke3c5ZlHWAcAAIBfqK+3WpN9WIs27Nc/Nh3QofKaZt/XPyFKV6X31hUje2tgYpd27rJ9EdYBAADgCHe9VWZemdZlF2t9zmH9Z8chHSipava9yV0jdGV6b105sreG9oqWMaadu3UGYR0AAADtIr+sSuuzi7Uup1jrsg9r476SFu8/l6TE6DBdPrKXrkzvrdEpcQET0I9HWAcAAIDPVdW6tWV/qdZlH9b6nGKtyy5WbvGRk34uLjJElw7vpavSe2tCv3gFuQIvoB+PsA4AAIA2sdYqu6jSG8rXZR/W1gOlqnU3nXt+oh4xYRqd0lWjU+M0KiVOY/p0VUiQqx267hgI6wAAADglVbVurc46fOyqeU6xiiqa3wx6vLBgl0Ymx2p0aleNSonT6NQ49YqNaIeOOy7COgAAAE6qvt5qVVaR3l2Xqw82HVBZC6MUj9e/e5Q3lI9O7aohPaO5an6KCOsAAABo0e6Ccr27LlfvrsvVvsMt33MeEx6sUaldNTolTqNS4zQqOa5DPznUXxDWAQAA0EhRRY0Wbdivd9blakNOcbPvSY2P1JRB3TU61XO/eb9uUXIF+GbQM4GwDgAAAFXVurU4I1/vrtunT7cXqK6+6ebQ2IgQXTGyl64ek6QxqV0DcpRieyOsAwAABKjW3IceEmQ0dUiirh6TrKlpCQoLDnKg08BFWAcAAAgwuwrK9e7aXL23vuX70MekxmnGmGRdMaIX9547iLAOAAAQAArLq7Vow369uy5XG/aVNPue1PhIzRidpBmjk9S3e1Q7d4jmENYBAAA6qfyyKn2eeUgfbj7wlfehXz6yl64enaSxfbgP3d8Q1gEAADqJOne91mYX67PMfH26vUBb9pc2+75j96EnaWpaIveh+zHCOgAAQAd2sKTKG86/2HnoKx9WNDo1TldzH3qHQlgHAADoQGrq6rV6b5E+yyzQZ9sLtO1gWYvvDXYZje3TVecNSdClw3upH/ehdziEdQAAAD+XW3xEn273XD1fuvOQKmrcLb63V2y4zh+SoPMGJ+qcgd0UEx7Sjp3C1wjrAAAAfqa6zq2Ve4r02fYCfZpZoJ355S2+NyTIaEK/eJ03OEHnD0nUoMQubBLtRAjrAAAADis5UqvNuSXauK9Eq7OKtHRXoY7Utnz1PLlrhM4fkqDzByfq7AHdFBVGpOus+P8sAABAO6qortOW/aXauK9YG/eVaFNuifYcqvjKz4QGuzSpf7eGq+cJ6t89iqvnAYKwDgAAcIZU1bq19UCpNu3zXDXfuK9YOwvKZZuOO2+ib7dInT8kUecNSdCkft0UEcp4xUBEWAcAAPCBmrp6ZeaVacO+Ym84z8wra/ZBRCcKdhml9YrWiKQ4jUyO1dn9u/EEUUgirAMAAJyWQ+XV+ve2/IZgXqyMA2Wqcdef9HMuIw1KjNbI5FiNTI7ViOQ4pfWMVngIV87RlM/CujEmWdLvJE2T1E3SAUnvSXrQWnu4FZ8/X9KSVnypVGttThtaBQAAOC3WWq3NLta8ZVn6x6aDrQrn/ROiNDIpViOTPVfNh/WOUWQo10vROj75k2KMGSBpqaRESQslbZM0QdIPJE0zxky21hae5DRZkh5sYW2EpKslbSaoAwCA9lZV69b76/fr1eVZ2pxb2uL7UuMjNSI5VunJsRqRFKezkmKYc4428dU/656RJ6jPsdb++WjRGPOYpB9JekTSnV91AmttlqQHmlszxixo+J/P+6BXAACAVskurNRrK/bqjVU5KjlS22R9TGqcLhzaQyOSPLe0xEWGOtAlOrM2h/WGq+oXy3Nl/OkTlu+X9B1JNxljfmKt/eq5RM2fv7ukGZKOSHq1bd0CAAB8tfp6q892FOjVpVn6NLOgyeSWsGCXvjGqt24+u6+GJ8U60yQChi+urE9tOH5krW1045a1tswY86U8YX6SpMWncf5ZksIkvWqtLW5TpwAAAC0orqzRW6v36bUVe7W3sLLJemp8pG6a1EfXjkvmCjrajS/C+pCGY2YL6zvkCeuDdXph/Y6G47Ot/YAxZk0LS2mn8fUBAEAntjm3RPOW7dXCDbmqqm28YdQY6bzBCZp1dl+dNzhBLhcPIkL78kVYP/r7n5IW1o/W4071xMaY8+T5x8Bma+3S0+gNAACgiZq6ev1z8wG9umyv1uxtOrQuJjxYM8en6NuT+qhPN+adwzn+PjfoOw3H507lQ9basc3VG664j2lrUwAAoGM6UHJEr6/I1oKVOTpUXt1kfVivGM06p4+uSk/iiaHwC74I60evnLe0w+Jo/ZTuNzfGxEu6Rp6NpfNOrzUAABDoKmvq9MWOQ3p3Xa4+2pon9wlPFA0JMrp0eC/NOqePxqR2lTHc6gL/4Yuwvr3hOLiF9UENx5buaW/J0Y2lc9lYCgAATkVeaZU+ycjT4ox8fbnzkKrrmj68qGdMuG6cmKqZE1KUGB3uQJfAyfkirB996ujFxhjX8RNhjDHRkiZLqpS0/BTPe3Rj6SndAgMAAAKPtVZbD5Tqk635WrwtTxv3tbSVTjq7fzfdfHYffX1YDwUHudqxS+DUtTmsW2t3GWM+kmfiyz2S/nzc8oOSoiQ9e/yMdWNMWsNntzV3TmPMFElDxcZSAADQguo6t5btKtTijHwtzsjT/pKqFt87uEcXXTi0h2aMTtLgHtHt2CXQNr7aYHq3pKWSnjLGXCgpQ9JEeWawZ0r69Qnvz2g4tnRT2GltLAUAAJ1bYXm1lmwv0OKMPH2eWaCKGnez7wt2GU3sH68L03rooqE9lNotsp07BXzDJ2G94er6OEm/kzRN0mWSDkh6UtKD1tqmM5FaYIzpKumbYmMpAAABz1qrXQXl+iQjX59szdPa7MM6YX+oV0x4sKamJeqioT103pAExYSHtG+zwBngs9GN1tocSbNb+d4Wt1k3BPsIX/UFAAA6ljp3vVZlHW7YIJqnrGaeJnpU326RumhoD104tIfG9e2qEO5BRyfj73PWAQBAgDhYUqU3VuXor6uydaCF+89dRhrbp6suHOq5vWVAQhSjFtGpEdYBAIBj6uutvth5SPNX7NUnGflNZqBLUlRokM4dnKCLhvbQ1LRExUeFOtAp4AzCOgAAaHeF5dV6e80+vb4yW3ubuc2lW1SoLhvRSxcN66FJ/eMVFszTRBGYCOsAAKBdWGu1Kuuw5q/Yq39uOqgad9MHFU3qH68bJ/bRJWf1VGgw958DhHUAAHBGlVbV6t21uZq/Yq8y88qbrMeEB+uascm6cWKqBiYyAx04HmEdAACcERv3FWv+8my9v2G/jtQ2nYc+KiVON05M1RUjeysilNtcgOYQ1gEAgM9U1tRp0Yb9em15tjblljRZjwwN0vTRSbphQqqGJ8U60CHQsRDWAQBAm20/WKbXV+zVO2tzVVZd12Q9rWe0bpzUR9NH9VY0DysCWo2wDgAATsuRGrf+ufmAFqzM1qqspg8rDw126YqRvXTjxD4akxrHPHTgNBDWAQBAq1lrtXFfid5YnaNF6/c3exW9X/co3TgxVdeMSVZXZqIDbUJYBwAAJ1VUUaP31uXqzdU52nawrMl6sMvokrN66saJqTp7QDeuogM+QlgHAADNcjc8XfTNVTn6eGtes3PR+3WP0rXjkvXNMclKjAl3oEugcyOsAwCARnKKKvXW6hy9vWaf9pdUNVmPCAnSZSN6aeb4FI3v25Wr6MAZRFgHAACqqnXrX1sO6s3VOfpyZ2Gz7xmVEqfrxqXoyvReTHQB2glhHQCAALY5t0Rvrc7Re+v3q+RIbZP1+KhQzRidpOvGpWhIT54uCrQ3wjoAAAGmpLJWCzfk6o1VOdqyv7TJujHSuYMSNHN8ii4a2kOhwS4HugQgEdYBAAgI1lqt3FOk11dm65+bD6qmrulm0ZT4CF03NkXXjE1W77gIB7oEcCLCOgAAndiRGrcWrs/V3GV7lXGg6VX0sGCXLh3eU9eNS9Gk/t3kcrFZFPAnhHUAADqhnKJKvbZ8r/66KqfZe9GHJ8Vo5rgUXZWepNhINosC/oqwDgBAJ2Gt1Zc7C/XK0iwt3pYnaxuvR4QEafroJN04MVXDk2KdaRLAKSGsAwDQwVVU1+mdtfs0d9le7cwvb7KeGh+pm8/uo2vHpnAVHehgCOsAAHRQew5V6NVlWXp79T6VVdc1WZ8yqLtuOaevzh+SqCDuRQc6JMI6AAAdSH291WeZBXplaZY+yyxosh4VGqRvjk3WTWf31cDELg50CMCXCOsAAHQApVW1emv1Ps1blqWswsom6/27R+nms/vomrHJPF0U6EQI6wAA+LEdeWWauyxL76zNVWWNu9GaMdLUIYmadU5fTRnYnbGLQCdEWAcAwM9Ya/X5jkN67vNd+nJnYZP16PBgXTcuRTef3Ud9ukU50CGA9kJYBwDAT1hr9Z8dh/TEJ5lam13cZH1wjy6adU5fzRidpMhQfoQDgYC/6QAAOMxaqy92HtITn+zQmr2HG625jPT1YT0065y+Ort/NxnDrS5AICGsAwDgkKMPMXrik0ytPiGkhwa5NHN8ir57Xn8ld410qEMATiOsAwDQzqy1WrarUI9/kqlVWY1DekiQ0czxKbr7/IHqHRfhUIcA/AVhHQCAdrR0l+d2l5V7ihrVQ4KMrh2XonumDlQSIR1AA8I6AADtYNkuz+0uK04I6cGuoyF9ALe7AGiCsA4AwBm0Yrfndpflu5sL6cm6+/yBSoknpANoHmEdAIAzYOWeIj3xSaaW7mo8Jz3YZfTNscm6ZyohHcDJEdYBAPChVVmekH7iw4yCXEbXjEnS96YOUmo3QjqA1iGsAwDgA2v2Funxj3foi52HGtWDXEZXj07S9y8gpAM4dYR1AADaYG9hhR76+1Z9kpHfqB7kMpoxOknfmzpQfbtHOdQdgI6OsA4AwGk4UuPWM5/u1LOf71ZNXb237jLS9NFJmnPBIEI6gDYjrAMAcAqstfrn5oN65IMM5RYfabQ2fVRvzblwkPondHGoOwCdDWEdAIBW2pFXpgcWbWmyeTQ9OVYPfmO4RqXEOdQZgM6KsA4AwEmUVdXqqcU79PKXWaqrt956fFSofjFtiK4dmyKXyzjYIYDOirAOAEALrLV6d12u/vDPbSooq/bWXUa6aVIf/fjrQxQbGeJghwA6O8I6AADN2Jxbogfe36LVew83qk/oG68HrjpLw3rHONQZgEBCWAcA4DjFlTV69KPten1Fto6740U9YsJ072VDdVV6bxnDLS8A2gdhHQAASe56q7+uytaj/9quw5W13npIkNGtX+un718wSF3C+LEJoH3xXQcAEPDW7D2s+9/frM25pY3qUwZ11wNXnaUBjGIE4BDCOgAgYBWUVeuPH27T22v2Naond43QfVcM08XDenDLCwBH+SysG2OSJf1O0jRJ3SQdkPSepAettYe/6rPNnGuMpJ9KOldSgqRiSdskvWitfdVXPQMAAlOtu16vLturJz7OVFl1nbceFuzSXecP0J3nDVB4SJCDHQKAh0/CujFmgKSlkhIlLZQnWE+Q9ANJ04wxk621hV9xiuPP9T1JT0o6LOkDSbmS4iUNl3SZJMI6AOC05JVW6dPt+Xrxiz3KzCtvtHbJWT30m8uHKSU+0qHuAKApX11Zf0aeoD7HWvvno0VjzGOSfiTpEUl3nuwkxpiLJT0l6WNJ37TWlp2wzjBbAECrueut1ucU69Pt+fr3tnxt2V/a5D39E6L0wJVn6dzBCQ50CABfrc1hveGq+sWSsiQ9fcLy/ZK+I+kmY8xPrLUVJzndnyQdkXTDiUFdkqy1tU0/AgDAMcWVNfoss0Cfbi/QZ5kFKqqoafZ9UaFBmnPhIM2e3E+hwa527hIAWscXV9anNhw/stbWH79grS0zxnwpT5ifJGlxSycxxgyXNFKe+9yLjDFTJY2VZCWtl7TkxPMDAGCt1baDZfr3tnx9uj1fa/YebjQf/XjBLqMJ/eJ1QVqirhrVW4nR4e3bLACcIl+E9SENx8wW1nfIE9YH6yvCuqTxDcd8SZ/Ks7n0eJuMMVdba3eerCFjzJoWltJO9lkAgP+rrKnTlzsLvQH9QElVi+9NiA7T1CEJuiAtUZMHdld0OHdUAug4fBHWYxuOJS2sH63HneQ8iQ3H2+TZVHq5pC8k9ZD0W0nflvSBMWaEtbb532kCADqtvYUV+vc2z73nK3YXqcbd/C9bjZHSk+N0QVqiLkhL1LBeMXK5GL8IoGPypznrR28YDJL0LWvtsobXpcaYm+W5Kj5O0jWSFnzViay1Y5urN1xxH+ObdgEAZ1p2YaVeXZalf2/L1+5DLW97igkP1nlDEjV1SILOG5ygbl3C2q9JADiDfBHWj145j21h/Wi9+CTnObp+8LigLkmy1lpjzEJ5wvoEnSSsAwA6NmutFqzM0UN/36ojte5m35PWM1rnD/FcPR+TGqfgIDaJAuh8fBHWtzccB7ewPqjh2NI97Seep6VQf/TBShGt7AsA0AEVllfrF3/bpE8y8hrVI0KCNHlgN50/JFFT0xKVFMePAwCdny/C+pKG48XGGNfxE1uMMdGSJkuqlLT8JOdZLqlCUl9jTFQzYx6HNxz3+KBnAIAfWrI9Xz97a6MOlVd7awMTu+jey9J0zoDuPFUUQMBp8+8MrbW7JH0kqa+ke05YflBSlKR5x4dvY0yaMabRZBZrbaWkFyWFS3rYGGOOe/8ISbdIqpP0dlt7BgD4lyM1bv124WbNfnlVo6B+yzl99ffvf00XpPUgqAMISL7aYHq3pKWSnjLGXCgpQ9JEeWawZ0r69Qnvz2g4nrg9/z55Rjb+UNLZDTPae0i6Wp4Q/8OGfxwAADqJzbkl+uEb67Uzv9xbS4gO039/c6SmDkn8ik8CQOfnk7Burd1ljBkn6XeSpkm6TNIBSU9KetBae/irPn/ceUqNMVMk/UrStZK+J88TTb+Q9Ki19iNf9AsAcJ673uq5z3frsY+3q9Z97ClGXx/WQ/919QgmugCAfDi60VqbI2l2K9/b4sBba225PFfiT7waDwDoJHKLj+jHb6zXij1F3lpESJDuv3KYZo5P0XF3QgJAQPOnOesAgACwcH2ufvPeZpVV1Xlr6SlxemLmKPXrHuVgZwDgfwjrAIB2UXKkVve9t1nvb9jvrbmM9L0LBun7FwxUCHPSAaAJwjoA4IxbvrtQP35jvfaXVHlrqfGRenzmKI3t09XBzgDAvxHWAQBnTE1dvR77OFPPfr5L9tgeUl07Nln3X3WWuoTxYwgAvgrfJQEAZ8TO/DLNWbBeWw+UemtxkSH6w4wRunRELwc7A4COg7AOAPApa63mLd+rRz7IUHWd96HW+trA7nr02nT1jA13sDsA6FgI6wAAn8kvq9LP396oT7cXeGuhwS79clqabjmnr1wuRjICwKkgrAMA2qy6zq331uXqjx9uV1FFjbee1jNaT3xrlNJ6xjjYHQB0XIR1AMBpK6ms1Wsr9uqVpVkqKKtutHb71/rpp5cMUXhIkEPdAUDHR1gHAJyynKJKvfjFHr25OkeVNe5Gaz1jwvU/16Vr8sDuDnUHAJ0HYR0A0Gqb9pXo2c936R+bDqjeNl7rGROu2ZP76oaJqYoOD3GmQQDoZAjrAICvVF9v9VlmgZ77fLeW7S5ssp7WM1p3TOmvK9N7KzSYp5ACgC8R1gEAzaquc2vh+v16/vPd2pFf3mT9awO7645z++vcQd1lDFNeAOBMIKwDABopOVKr+Sv26pUvs5R/wqbRIJfRFSN76Y4p/TU8KdahDgEgcBDWAQCSpH2HK/XSF1l6Y1W2Kk7YNBoVGqRvTUjVrV/rp6S4CIc6BIDAQ1gHgAC3ObdEz32+Wx9sOiD3CbtGE6PDNHtyP90wMVWxEWwaBYD2RlgHgABk7bFNo0t3Nd00OrhHF90xpb++MSqJTaMA4CDCOgAEmPyyKv3kzQ36z45DTdbOGdBNd5zbX+cPTmDTKAD4AcI6AASQzzML9OM31+tQeY23FuQyunyEZ9PoiGQ2jQKAPyGsA0AAqHXX67GPM/V/n+7y1oyRbprUR3dM6a+U+EgHuwMAtISwDgCdXE5Rpeb8dZ3WZRd7awnRYXpi5ihNHtjdwc4AACdDWAeATuzDzQf087c3qrSqzlubMqi7HrtulBKiwxzsDADQGoR1AOiEqmrdeuSDDM1bvtdbC3YZ/fSSIfrOlP5yudg8CgAdAWEdADqZnfnl+t7ra7XtYJm3ltw1Qk9dP1pjUrs62BkA4FQR1gGgk7DW6u01+/TbhVt0pPbYE0gvHd5T/3XNSB5qBAAdEGEdADqB8uo63ffeZr27LtdbCw126bdXDNONE1OZmQ4AHRRhHQA6uM25Jfre62uVVVjprQ1IiNL/3jBGQ3vFONgZAKCtCOsA0EFZa/XK0iz94R/bVOOu99avHZusB79xliJD+RYPAB0d38kBoAM6XFGjn729UZ9k5HlrUaFBemTGCE0fneRgZwAAXyKsA0AHsyqrSHMWrNOBkipvbXhSjP58/Rj16x7lYGcAAF8jrANAB+Gut3pmyU49/kmm6u2x+uzJffXLS9MUFhzkXHMAgDOCsA4AHUB+aZV++MZ6Ld1V6K3FRYboT99M19eH9XCwMwDAmURYBwA/Zq3VR1vzdO87m1RYUeOtT+gbryevH6VesREOdgcAONMI6wDgp7ILK/XAoi3697Z8b80Y6ftTB2rOhYMUHORysDsAQHsgrAOAn6mqdevZz3brmU93qrru2EjGxOgwPfGtUTpnQHcHuwMAtCfCOgD4kU+35+v+97do73EPODJG+tb4VP38kiHqGhXqYHcAgPZGWAcAP5BbfEQPLdqqD7ccbFQfnhSjh6eP0KiUOIc6AwA4ibAOAA6qqavXC1/s1p8X79SRWre3HhMerJ9NS9MNE1IV5DIOdggAcBJhHQAcsnTnId23cLN2FVQ0qn9zbLJ+eWmauncJc6gzAIC/IKwDQDvLK63Swx9kaNGG/Y3qaT2j9dD04RrfN96hzgAA/oawDgDtpNZdr7lLs/T4x5mqqDl2y0uXsGD96OuDNevsPoxjBAA0QlgHgHawck+Rfrtws7YdLGtUvyq9t359+VD1iAl3qDMAgD8jrAPAGVRQVq0//DND76zNbVQfkBClh74xXOcMZGY6AKBlhHUAOAPc9VbzV+zVn/61XWVVdd56REiQ5lw4SLd9rZ9Cg7nlBQDw1QjrAOBja7MP6773NmvL/tJG9UuH99R9VwxT77gIhzoDAHQ0hHUA8JH6eqv//td2/eWzXY3qfbtF6sFvDNd5gxMc6gwA0FER1gHAB6rr3PrpWxsbjWMMC3bpnqkD9Z1z+ys8JMjB7gAAHRVhHQDaqORIrb7z6mqt2FPkrU0Z1F2/nzFCKfGRDnYGAOjofLa7yRiTbIx5yRiz3xhTbYzJMsY8YYzpegrn+NQYY7/iP2abAfAr+4uP6Nq/LG0U1G+a1EevzJ5AUAcAtJlPrqwbYwZIWiopUdJCSdskTZD0A0nTjDGTrbWFp3DKB1uo17VQB4B2l3GgVLe8vFJ5pdXe2i+mpenO8/rLGONgZwCAzsJXt8E8I09Qn2Ot/fPRojHmMUk/kvSIpDtbezJr7QM+6gsAzoilOw/pu/PWqKzacw0hJMjoT99M1/TRSQ53BgDoTNp8G0zDVfWLJWVJevqE5fslVUi6yRgT1davBQD+4L11uZr18kpvUI8OC9YrsycQ1AEAPueLK+tTG44fWWvrj1+w1pYZY76UJ8xPkrS4NSc0xsyU1E9SjaQMSf+21lZ/9acafX5NC0tprT0HAJzIWqv/+2yX/vvD7d5az5hwvTx7vIb2inGwMwBAZ+WLsD6k4ZjZwvoOecL6YLUyrEv66wmv840x91hr3z6N/gCgzdz1Vg+8v0Xzlu/11gb36KJXZk/gIUcAgDPGF2E9tuFY0sL60XpcK861UNKjktZJKpTUR9IsST+R9IYx5nJr7YcnO4m1dmxz9YYr7mNa0QcAeB2pcWvOX9fp46153tqk/vF69qZxio0IcbAzAEBn51dz1q21j59Q2i7pXmPMfkl/lvQHSWT24tEAACAASURBVCcN6wDgK0UVNbpt7iqtyy721q5M761Hrx2psGAedAQAOLN8MWf96JXz2BbWj9aLW1hvjRfkGds4yhgT3YbzAECrZRdW6pr/W9ooqH/n3P56cuYogjoAoF344sr60Z1Wg1tYH9RwbOme9pOy1lYZY8okdZUUJansdM8FAK2xIadYt81dpUPlNZIkY6TfXjFMsyf3c7gzAEAg8UVYX9JwvNgY4zp+IkzDVfDJkiolLT/dL2CMGSJPUC+TdKgNvQLASS3Zlq+756/VkVq3JCk02KUnZ47SpSN6OdwZACDQtPk2GGvtLkkfSeor6Z4Tlh+U50r4PGttxdGiMSbNGNNojKIxpp8xJv7E8xtjEiS93PDyr9ZanmIK4Iz568ps3f7qam9Qj40I0eu3TySoAwAc4asNpndLWirpKWPMhfLMRp8ozwz2TEm/PuH9GQ3H45/HfZ6kvxhjvpC0W1KRpFRJl8lz3/tqST/3Ub8A0Ii1Vo9/skNPLd7hrSXFRWjurRM0MLGLg50BAAKZT8K6tXaXMWacpN9JmiZPwD4g6UlJD1prD7fiNGvkma8+VtJoSTHy3PaySdKbkp611tb4ol8AOF6tu173vrNJb63Z562d1TtGL88er8TocAc7AwAEOp+NbrTW5kia3cr3mmZqmyTd4qt+AKA1yqvrdPf8tfo8s8BbO3dwgp65cYy6hPnVdFsAQADiJxGAgJVfWqVb567S5txSb+3ascn6/dUjFBLki8m2AAC0DWEdQMCx1uqtNfv0yAcZKjlS663PuXCQfnTRIBnT5Jd/AAA4grAOIKBkHarQve9u0tJdhd5akMvo4enDdf2EVAc7AwCgKcI6gIBQ667Xc5/v1lOLd6i6zvs4CCV3jdAfrxmpyQO7O9gdAADNI6wD6PTW5xTrl3/bqG0Hjz382GWk26f01w8vGqTIUL4VAgD8Ez+hAHRaFdV1evSj7Zq7NEv19lj9rN4x+uM1IzU8Kda55gAAaAXCOoBOacm2fP3mvc3KLT7irYWHuPTjrw/WrZP7KZhpLwCADoCwDqBTKSir1u/+vlWLNuxvVJ8yqLsemT5Cqd0iHeoMAIBTR1gH0Cm0NI6xa2SI7rtimGaMTmIkIwCgwyGsA+jw9hyq0L3vbNKy3YWN6lePTtJvrhim+KhQhzoDAKBtCOsAOqyWxjGmxEfokekjdO7gBAe7AwCg7QjrADokxjECAAIBP80AdChHxzG+sjRLlnGMAIBOjrAOoMNoaRzjT74+RLMn92UcIwCg0yGsA/B7NXX1uv/9LVqwMrtRfcqg7vr9jBFKiWccIwCgcyKsA/BrxZU1uvO1NVq+u8hb6xoZot9eOUzTRzGOEQDQuRHWAfit3QXlum3uau05VOGtXT6ylx76xnDGMQIAAgJhHYBfWrrrkO56bW2jBxz97JIhuvv8AVxNBwAEDMI6AL/zxqps/frdzaqr94x7CQt26fGZo3TZiF4OdwYAQPsirAPwG+56q//+cJue/Xy3t5YQHaYXbh6n9JQ4BzsDAMAZhHUAfqGiuk4/fGO9Pt6a560N7RWjF2eNU++4CAc7AwDAOYR1AI47UHJEt72yWlsPlHprFw1N1JPfGq2oML5NAQACFz8FAThq475i3T53tfLLqr21O6b00y8vHaogFxtJAQCBjbAOwDH/3HRAP3pzvapq6yVJwS6jh6YP1/UTUh3uDAAA/0BYB9DurLV65tNd+tO/tntrMeHB+su3x+qcgd0d7AwAAP9CWAfQrqrr3Lr3nc3629p93lrfbpF68ZbxGpDQxcHOAADwP4R1AO2mqKJGd85bo5VZRd7axH7x+su3x6orTyQFAKAJwjqAdrEzv1y3zV2lvYWV3tq1Y5P1yIwRCg12OdgZAAD+i7AO4Iz7Ysch3TV/jcqq6iRJxki/mJam757bX8Yw8QUAgJYQ1gGcUfNX7NVvF26Ru95KkiJCgvT4zFGaNrynw50BAOD/COsAzgh3vdUjH2TopS/3eGs9YsL04qzxGp4U62BnAAB0HIR1AD5XXl2nHyxYp8Xb8r214UkxeuHm8eoZG+5gZwAAdCyEdQA+lVNUqTteXa1tB8u8tUvO6qHHZ45SZCjfcgAAOBX85ATgM6uyinTnvDUqrKjx1u48b4B+fskQuVxsJAUA4FQR1gH4xFurc3Tvu5tU6/ZsJA0JMvr9jBG6dlyKw50BANBxEdYBtIm73uq//pmh5/9zbCNpt6hQPXvTWI3rG+9gZwAAdHyEdQCnrayqVnMWrNOS7QXeWlrPaD1/8zilxEc62BkAAJ0DYR3AackurNRtc1dpR365t3bR0B564luj1CWMby0AAPgCP1EBnLLluwt112trdLiy1lu76/wB+tnFbCQFAMCXCOsATsmCldm6773Nqmt4ImlosEt/vGaEZoxOdrgzAAA6H8I6gFapc9frkX9k6OUvs7y17l3C9NzNYzUmtatzjQEA0IkR1gGcVMmRWn1/wTp9nnlsI+mwXjF6ftY4JcVFONgZAACdG2EdwFfac6hCt81dpd0FFd7atLN66rGZ6TyRFACAM4yftABatHTnId01f61KjhzbSDrngoH64UWD2UgKAEA7IKwDaNa85Xv1wPtb5G7YSBoW7NKfrk3XVem9He4MAIDAQVgH0Eidu16/+/tWvbpsr7eWGB2m528ep/SUOAc7AwAg8BDWAXiVVNbqntfX6oudh7y1EUmxev7mceoZG+5gZwAABCaXr05kjEk2xrxkjNlvjKk2xmQZY54wxpz2TDdjzLnGGLcxxhpjHvZVrwCa2lVQrunPfNkoqF8+spfe/O7ZBHUAABzikyvrxpgBkpZKSpS0UNI2SRMk/UDSNGPMZGtt4SmeM1rSXEmVkrr4ok8Azfs8s0D3vL5WZVV13tqPLhqsORcOlDFsJAUAwCm+urL+jDxBfY61drq19pfW2gskPS5piKRHTuOcT0qKlfQHH/UI4ATuequ/fLZLs19Z5Q3q4SEuPX3DGP3gokEEdQAAHNbmK+sNV9UvlpQl6ekTlu+X9B1JNxljfmKtrVArGGO+IWm2pJt80SOApnKKKvWTNzdoZVaRt9YzJlwvzBqn4UmxDnYGAACO8sWV9akNx4+stfXHL1hryyR9KSlS0qTWnMwYkyjpeUnvWWtf80F/AI5jrdWbq3N06ZP/aRTUR6XE6f3vTSaoAwDgR3xx1XpIwzGzhfUd8lx5HyxpcSvO97w8/4i483QbMsasaWEp7XTPCXQGh8qr9at3NunjrXneWpDL6J6pA/X9CwYqJMhne84BAIAP+CKsH70MV9LC+tH6SQc0G2NulXSVpJnW2ryTvR9A632yNU+/fGejDpXXeGv9u0fpsZmjNIr56QAA+CW/uR/cGNNX0hOS3rLWvtmWc1lrx7bwNdZIGtOWcwMdTXl1nR5atFVvrM5pVL/57D761aVDFREa5FBnAADgZHwR1o9eOW/pRtej9eKTnOclSUck3e2DngBIWpVVpB+/uV45RUe8tcToMP3p2nSdNzjBwc4AAEBr+CKsb284Dm5hfVDDsaV72o8aI0+wL2hhXNyvjTG/lrTQWjv9lLsEAkh1nVuPf7xDz36+S9Yeq18+spcemT5ccZGhzjUHAABazRdhfUnD8WJjjOv4iTANDzaaLM+DjZaf5DyvyjM15kSDJJ0rab2kNZLWtbljoBPbfrBMP3xjvTIOlHprMeHBemj6cF2V3pvZ6QAAdCBtDuvW2l3GmI/kmfhyj6Q/H7f8oKQoSc8eP2PdGJPW8Nltx51nTnPnN8bcIk9Y/8Ba+5u29gt0Vu56qxe/2K1H/5WpGvexKaqTB3bTo9emq1dshIPdAQCA0+GrDaZ3S1oq6SljzIWSMiRNlGcGe6akX5/w/oyGI5f4AB/IKarUT97aoJV7js1NDwt26ZeXpmnW2X3lcvFXDQCAjsgnYb3h6vo4Sb+TNE3SZZIOSHpS0oPW2sO++DoAGrPW6u01+/Tgoq0qr67z1kckxerxmekamBjtYHcAAKCtfDa60VqbI2l2K9/b6st81tpXJL1yel0BnVdhebXufXeT/rWFBxwBANBZ+c2cdQCttzgjT7/4W+MHHPXrHqXHrkvX6NSuDnYGAAB8ibAOdCAV1XV6+IOtWrCy8QOOvj0pVfdeNlSRofyVBgCgM+EnO9BBlFXV6tsvrtSGnGPPF0uMDtN/f3Okzh+S6GBnAADgTCGsAx1AZU2dbn1lVaOgfvmIXnp4+nB1jeIBRwAAdFaEdcDPVdW6dfvc1VqVdWyo0gNXDtOsc/rygCMAADo5wjrgx6rr3LrztTVauqvQW7vvimG6ZXI/B7sCAADthdlugJ+qddfr+6+v06fbC7y1n10yRLd9jaAOAECgIKwDfshdb/XjNzfoo63HZqjPuWCg7pk60MGuAABAeyOsA36mvt7q529v1KIN+72175zbXz/6+mAHuwIAAE4grAN+xFqr+xZu1t/W7vPWZp3dR7+6NI3NpAAABCDCOuAnrLV66O8Zmr8i21v71vgU3X/lWQR1AAACFGEd8APWWv3pX9v10pd7vLXpo3rrkRkj5HIR1AEACFSEdcAP/PnfO/XMp7u8ry8b0VOPXpuuIII6AAABjbAOOOy5z3fpsY8zva8vGpqoJ2aOVnAQfz0BAAh0pAHAQXOXZun3/9jmfT1lUHf97w1jFBrMX00AAEBYBxzz15XZuv/9Ld7XE/vF67mbxik8JMjBrgAAgD8hrAMOeHfdPv3q3U3e12NS4/TiLeMVEUpQBwAAxxDWgXb2j00H9JM3N8haz+vhSTF6efYEdQkLdrYxAADgdwjrQDv6ZGue5ixYp/qGoJ7WM1rzbp2o2IgQZxsDAAB+ibAOtJPPMwt09/y1qmtI6gMSovTa7RPVNSrU4c4AAIC/IqwD7WD57kJ9Z95q1bjrJUl9ukXq9TsmqXuXMIc7AwAA/oywDpxha/Ye1q2vrFJVrSeoJ8VFaP7tE9UjJtzhzgAAgL8jrANn0KZ9JbrlpZWqrHFLknrEhOn1OyYquWukw50BAICOgLAOnCFb95fqppdWqKy6TpLUvUuo5t8+SX26RTncGQAA6CiYFQecAUu25ev7C9apvCGox0WG6LXbJ2pgYheHOwMAAB0JYR3wIWutXvxij37/jwzveMbosGDNu3Wi0nrGONscAADocAjrgI/U1NXr/vc3a8HKHG8tKS5CL8wap6G9COoAAODUEdYBHzhcUaO75q/R8t1F3trYPl317E1jGc8IAABOG2EdaKOd+eW6be4q7S2s9NZmjE7SH64eofCQIAc7AwAAHR1hHWiD/+zwPJW0rKrOW/vZJUN09/kDZIxxsDMAANAZENaB0/Tqsiw9uGir3A07SSNCgvT4zHRNG97L2cYAAECnQVgHTlGdu14PLtqqecv3emu9YsP1/M3jNDwp1sHOAABAZ0NYB05BSWWt7nl9rb7YechbS0+J0/M3jVViTLiDnQEAgM6IsA600p5DFbpt7irtLqjw1q4Y2UuPXpvORlIAAHBGENaBVli665Duem2tSo7Uems/vGiQfnDhIDaSAgCAM4awDpzEgpXZuu+9zapr2EgaFuzSo9em68r03g53BgAAOjvCOtACd73VIx9k6KUv93hridFhev7mcUpPiXOwMwAAECgI60Azyqpq9f0F6/Tp9gJv7azeMXph1jj1io1wsDMAABBICOvACXKKKnXb3FXKzCv31qad1VOPzUxXZCh/ZQAAQPsheQDHWZVVpO/OW6Oiihpv7Z6pA/STrw+Ry8VGUgAA0L4I60CDt1bn6N53N6nW7dlIGhrk0h+/OUIzRic73BkAAAhUhHUEPGutHv1ou55esstb694lVM/eNE5j+3R1sDMAABDoCOsIaDV19frlOxv1ztpcby2tZ7RemDVOyV0jHewMAACAsI4AVl5dp7teW6P/7DjkrV2Qlqinrh+tLmH81QAAAM4jkSAg5ZdVafbLq7Rlf6m3dv2EFD30jeEKDnI52BkAAMAxhHUEnF0F5Zr10krtO3zEW/vRRYM158KBMoaJLwAAwH8Q1hFQ1uw9rNvmrlJxZa0kKchl9PsZwzVzfKrDnQEAADTls9/3G2OSjTEvGWP2G2OqjTFZxpgnjDGtHqdhjPmZMeYfDZ8tN8aUGmM2GWMeM8YwPw9t8tGWg7rh+eXeoB4REqTnbx5LUAcAAH7LJ1fWjTEDJC2VlChpoaRtkiZI+oGkacaYydbawlac6ruSyiV9JilPUoik0ZJ+JOk2Y8z51tp1vugZgeW15Xv124WbVe8Zoa74qFC9dMt4jUqJc7YxAACAr+Cr22CekSeoz7HW/vlo0RjzmDxB+xFJd7biPMOttVUnFo0xd0h6ruE8l/mkYwQEa63+56NM/e+Snd5an26Rmjt7gvp2j3KwMwAAgJNr820wDVfVL5aUJenpE5bvl1Qh6SZjzEmTUXNBvcGbDcdBp9kmAlCtu14/e3tjo6A+MjlWf7vrHII6AADoEHxxz/rUhuNH1tr64xestWWSvpQUKWlSG77GlQ3HjW04BwJIRXWdbpu7Wm+v2eetnT8kQQvumKTuXcIc7AwAAKD1fHEbzJCGY2YL6zvkufI+WNLi1pzQGHO7pGRJXSSNkHSRpL2SftnKz69pYSmtNZ9Hx1ZQVq1bX1mlTbkl3tp145L1yIwRCmGGOgAA6EB8EdZjG44lLawfrZ/KTr7bJU087vUqSTdYa3e28H5AkrS7oFyzXl6pnKJjM9TnXDhIP7poEDPUAQBAh+OXc9attZMkyRjTTdIYeTaWrjHGXGet/VcrPj+2uXrDFfcxvuwV/mNd9mHdNne1iipqJEkuIz08fYRumMhoRgAA0DH5IqwfvXIe28L60XrxqZ64Ydzjx8aYVfKMg5xnjOljrT1yko8iwCzOyNM9r69VVa1n20R4iEv/e/0YXTSsh8OdAQAAnD5f3MC7veE4uIX1oxNcWrqn/aSstcWSlklKkHTW6Z4HndPrK7J1x6urvUG9a2SIFtwxiaAOAAA6PF9cWV/ScLzYGOM6fiKMMSZa0mRJlZKWt/HrJDUc69p4HnQS1lo9/skOPbV4h7eWEh+hubMnqH9CFwc7AwAA8I02X1m31u6S9JGkvpLuOWH5QUlRkuZZayuOFo0xacaYRpNZjDGpxphmL4UaY74rabykHEmb2tozOr5ad71+8beNjYL68KQYvXPXZII6AADoNHy1wfRuSUslPWWMuVBShjzTXKbKc/vLr094f0bD8fjxHGMkvWWMWSZpp6Q8Sd3kmc8+QlK5pJustW4f9YwOqrKmTvfMX6sl2wu8tXMHJ+iZG8eoS5hf7pkGAAA4LT5JNtbaXcaYcZJ+J2mapMskHZD0pKQHrbWHW3GatQ3vnyLpcknxkqok7Zb0P5KetNbm+KJfdFylVbW65aWVWpt9bL/yNWOS9V/XMEMdAAB0Pj67DNkQpGe38r1NBl5ba7Ml/dRX/aDzOVxRo5tfWtnoYUffmzpQP7l4MDPUAQBAp8Q9A+gQDpVX69svrNC2g2Xe2v1XDtPsyf0c7AoAAODMIqzD7+WVVumG55drV4Fnj7Ix0u9njND1E3jYEQAA6NwI6/Br+w5X6sYXVmhvYaUkz1NJ/+e6dM0YnexwZwAAAGceYR1+a29hhW54foVyiz0PrA12GT35rdG6fGQvhzsDAABoH4R1+KWd+eW68YXlyiutliSFBrn09I1j9HWeSgoAAAIIYR1+Z9vBUn37hRU6VF4jSQoLdum5m8fpvMEJDncGAADQvgjr8Cubc0v07RdXqLiyVpIUGRqkF2aN0zkDujvcGQAAQPsjrMNvrM0+rFkvrVRZVZ0kqUtYsF6ZPV7j+sY73BkAAIAzCOvwCyt2F+rWV1aposYtSYqNCNGrt05Qekqcw50BAAA4h7AOx32x45Buf3WVqmrrJUnxUaF67baJGtY7xuHOAAAAnEVYh6P+vS1Pd762VjV1nqCeGB2m+bdP1KAe0Q53BgAA4DzCOhzz4eaD+v6Ctap1W0lS79hwzb9jkvp1j3K4MwAAAP9AWIcjFq7P1Y/f3CB3vSeop8RH6PXbJyklPtLhzgAAAPwHYR3t7q3VOfr53zbKenK6+neP0vw7JqpXbISzjQEAAPgZwjra1bzle3Xfe5u9rwf36KLXbp+oxOhwB7sCAADwT4R1tJsX/rNbD3+Q4X09rFeMXrt9ouKjQh3sCgAAwH8R1tEunl6yU3/613bv6/SUOL06e4JiI0Mc7AoAAMC/EdZxRllr9fjHmXrq3zu9tfF9u+qlW8YrOpygDgAA8FUI6zhj3PVWD3+wVS9/meWtTR7YTc/fPE6RofzRAwAAOBkSE86Iiuo6/eCv6/VJRp63NnVIgv7v22MVHhLkYGcAAAAdB2EdPpdXWqVbX1mlLftLvbXLRvTU4zNHKSyYoA4AANBahHX41Nb9pbpt7iodKKny1r57Xn/94pI0uVzGwc4AAAA6HsI6fGbJtnx97/W1qqhxS5KCXEYPTx+u6yekOtwZAABAx0RYh0/MW5al+9/fovqGp5JGhwXrmW+P0ZRBCY72BQAA0JER1tEm7nqrRz7I0Etf7vHWkuIi9PLs8RrcI9rBzgAAADo+wjpOW2VNneYsaDzxJT05Vs/PGqfE6HAHOwMAAOgcCOs4LXmlVbpt7iptzj028eXS4T312HWjFBHKxBcAAABfIKzjlDHxBQAAoH0Q1nFKlmzP1/fmM/EFAACgPRDW0WpMfAEAAGhfhHWclLve6vf/yNCLXzSe+PLSLeM1pCcTXwAAAM4Uwjq+EhNfAAAAnENYR4uam/gy7ayeenwmE18AAADaA2Edzco4UKpbXzlh4su5/fWLaUx8AQAAaC+EdTTR3MSXh74xXDdMZOILAABAeyKso5F5y/fq/oWbG018efrGMTp3MBNfAAAA2hthHV5PLd6hxz7O9L5m4gsAAICzCOuQJC3asL9RUGfiCwAAgPMI69CmfSX66VsbvK+/NrC7nr95HBNfAAAAHOZyugE4K7+0Sne8ulrVdfWSpP4JUXr6xjEEdQAAAD9AWA9gVbVu3TFvjQ6WesYzxoQH68VZ4xUbEeJwZwAAAJAI6wHLWqtfvbNJG3KKJXnGMz594xj16x7lcGcAAAA4irAeoJ79fLfeXZfrff2by4dqyiDGMwIAAPgTwnoA+mRrnv744Tbv6+snpOiWc/o61xAAAACaRVgPMJl5ZfrBX9fJNjz0aEK/eD141XAZY5xtDAAAAE0Q1gPI4Yoa3T53tSpq3JKk5K4R+r8bxyg0mD8GAAAA/og56wGi1l2vu+avUXZRpSQpKjRIL8wap25dwhzuDADQRHWZtOc/0v51ntfBYVJweMMxTAoKO64WemwtqKVamMRvUIEOibAeIB54f4uW7y6S5Pl+/fjMUUrrGeNwVwBaVJYn7V8r5a71HAt3Sl37Sn2/JvWdIvUe4wlk6Bzq66WDG6Sdi6Vd/5ZyVkj1db79GscH/NgkqWs/Kb5f42N0L8nlwG9brZWOHJZKcqSSfVJxjlRZ6Pkz3ytdShgiBTFWGIHJZ2HdGJMs6XeSpknqJumApPckPWitPdyKz0dJmi7pckljJKVIqpe0XdICSX+21tb4qt9AMm9ZluavyPa+/unFQ3TxWT2dawhAY1UlniuoR4N57jqpdF/T9x3OknZ/6vnfwRFS6kTCe0dWekDavcQT0Hcv8YTTM8ld7fmvulSqyD921f54weFSXJ+mIT6+nxSX6gn7p/W1a6WyA54QXrJPKsluOO47VqutaPnzQWFSj2FSz5Ge8N4rXepxlhQScXr9oGXuOqmq2POPp8oiz7HRf8fVaiqlLomePxuxKVJcSsMxVYqIc/r/kk7DJ2HdGDNA0lJJiZIWStomaYKkH0iaZoyZbK092XehKZJek1QkaYk8Qb+rpKskPSrpamPMhdbaKl/0HCiW7jykBxZt9b6+Kr237j5/gIMdAf/f3r3HV1He+x7//HIjCSEBEkgkgAiESxEqYL2iBUF0t9tbq909PXVrdzm1N2u7tZdz2p5u3KfdvarVtran1tpqu63aWrur9YaoINpaUEHlLgHkIhBIAoEQkjz7j2dW1splhVxW1kzI9/16zWvWmpm11pNMnpXvPPPMMwPcsSOwe01CMF8FVRu7/z6NR3xwTwzvY87wwX3cHCif1fNgdaJrboLqbZBbBHnD0tc95Fg9bFvhW843PQN73uh8+9LpMP69kFPgQ3Zj4lSfZFmDnzfG5kFAb+piW1djPexb76d2DArLgwA/rnWQLyyHun1By/j21iG8ZrsP6q65u7+xuKaj/uAi8QDDMqFkEpwUBPiyGVA2XSExUWODPzA79A4c2ts6aCcL40drU/PZgwrbBPiEID90LAweoa5ZXWQuNixIb97E7AlgIfA559wdCctvAb4A/Mw598njvMdpwDTgwcQWdDMbAjyLb22/yTn3g16Uc+WsWbNmrVy5sqdv0a9U7qvjsh+/QM2RYwDMGF3EA9edTW52ZsglE+mB3a/D1hVQMAJGzfStf1H/om9qhL1rWwfzPW92rXtDVp4PIOWzffAumeRfW7kMKl+AA1uO/3qFd6+5yR8gVS6DyuX+7ygWSLIHQ9Fo3y2kaLQPE4Wxx6P94+zcnn2uc7B3vQ/nm5f4/dZ4JPn2+SUw4QKYOB/Gz4MhpT373I40N8eDfMMhf7Cyf4v/O0qcH9mfus/sruzBQaALfvd5w2DvBti92gf+rop1nSmbASed5sN8wcg+K3baOedbvg++E4TwPXBod/zxwd3BsnfC3Z/Hk5Ubr3NDx0DR2Pj+zxvuD6ZziyBncPS/65OYPXs2q1atWuWcm92b9+l1WA9a1TcBlcAE5+KHzkHQ3gUYMNI518k5rk4/4yPAb4A/O+cu6UVZB0xYr60/xgd+soJNew4BMHLIIP702TmU9cuO8wAAHVhJREFUFfXwn45IGA5shdcfgjUP+aCaKG+4D+3ls/x81CwoPCmccoJvodq3wYezvet8MN/1WufhLCYjC0a+Kx7MR82CEVMgs5OTnzVv+/AXC6DHDe+5bcL77J6H91jwa2rw3RsSH7smKBwFg4b07L1TobkZ3nnd/14ql8HWF3xXo54aPCIe3IvGJIT74PHgkfF+3of3+7Mdm5/xU+2O5O+bkQ1jz4oH9NLp4fQXT1Rf03GIP1Dp/+boRWYoKE34/Y0Ouk6Mjge2zs5yHN7v69Pu1X6+a7W/jqOr5SkoC7rOvMv/3puP+b/X5sb4325PHzc3QWZO/OLfdhcAd3JRcGZO621iyzIyoW5v+/Adm7p6pqRHzJ+dyBvu90lsym/zPG+4P5Ct3eW7NVUHZ1Vi82OHU1SczHhw73Aa2vn6EMN+lML6IuDnwP93zl3XwfpYq/sC59ySHn7GVcADwB+dc1d0YftkaXzKrFmz8k/0sN7U7Fj0q5dZun4vADlZGTxw3dmcNkanBpM6vN//Q88bBqPPUN/fMNVVwRt/8AF9+0vde21BWUJ4D6bBJakrm3NQu9N3Edi7ofW8bm/X36e4wpezfLYP5mWn9r7vbWJ43/oC7H+r8+2zcqH8dB+qWwXvo+0DeNvHXTkzUFgOJRX+jEDiNKQs9f84m5v9GYwty+I//5HjXCqVX+y7pXTWT7qrMrLjByh73uy8u0fxRJgw3wf0cXNgUEHvPz9dGo8mb5E/uNvXtaLR8e4OicG8aHTqz+wcPeQPynYFAX73a7BnnQ/i4lmGP9gsKPVnF/KLW4ftVmE8mA8q6v1Bo3P+Gozqba0DfPX2YNm23h1Ad0cs7OcNhc+uTOsBcarCeir6rE8O5huSrN+ID+uTgB6FdeBfgvnjPXz9gPLdx9e1BHWA7105Q0E9mboqePEO+NvP/alhgJwhvp/oxPkwcYFv/ZG+dfQQrP8LrHnAt0Z2FAaz8qBigR/SbucrHX/RH9oN6x/zU0zRWCifGW99H3Wa/+LuTFOjb0nct963lMdazPdthIaD3fvZCkf7z48F8658fk8UjYZ3/5OfAGp2+NAaa3lvG94b62Hr8tSXI6Z2h59ifepjBhUmhPjYfLLv99zV0T5i3Usql8GW5/3PebyLMwtK42cVTjkfho/3y48c8OWMXewYm2LLanf6swWdaT4G1Vs7XjeoCMafHw/ow07u2s8YRVmDgn1WEXZJvEEF/szE2LPiyxqPwp61QQt8EOLfeT11rbxRkVMQBPBS310qFsYLyuKPh5T5cJ4RQtdXM3/wNrjEN0x0pL62TZDf5qeDu/z3e2zq7b5zTb47UNOx8M9c9VAqwnrsv06yQ6TY8h6lRTP7LH6EmVeBu7vymmRHMEGLe5K/mhPDQyvf5mfPx/8pf3ruBC47rTzEEkVU3T5YcTv87a72LWsNB2Hdn/0EPkhMXODD+8nn9rz/qrTWdMwH89UP+HDd0ReyZfqAM/0qmPL+eCukcz58xi44i3U56aiVtCZoxXnzkfiy4onxlvfSU32reEsg3+BPr3f3NHNWbhBkJvth5spm+H9SYfWVLSqHGR/yE/jQmdhtZv/m3r1/Zk4wZSc8zvGtyjXbk7e+H62FHSv9lCgjy1+sWDIJRiS2xlf4gL9vI1Q+H3RtWX78MxmDR8RHyhl3nn+fjlr084f7qWx6x+/T3ORbjWve9iP0tAT6HfELKRP7BVuGPyibON8H9PLZnXdnktTKGuQPiEedFl/W3ARVm/13RNUm/3eQme3PiGRm+7+92N9ybFmHj4Pt2i7PyPTfZ4kX9Ca98LeTi4GbEtY1N/m/y4LSIICPjAfzwSP71xmZZHILIXeaH9WnM40N/nujvsb31U8M8l2ZYv9b+qKRJE0i/Q1iZh8AbgN2Ax90zuncVidWbj3A//nDmpbnC6aWctPCyZ28YgA6tMeH9Jd/0T4clkzyX5LV21ovj42M8NKPfevuuDlQcaEP8MPH99sLX0LR3OzHj17zILzxcPKLn8ac6QP6tCs67sZiBsUT/DT9yuC9m3yg27kqHuB3r/H/ANuq2uSnNQ92/2fILfL9yUsm+VBeMtmHy6Kx0W61KRwFM67yE/jwvus1f+DTLnRnx+dZg9qvz8jq/O++6Zi/3mBfcPCzb2NwMLQBjiZp12lu9KPiVG2E9Y+2XpdTED/zlUx+cetwPmJyaupmRmbQP70cOLPjbRoO+5b4un3+c/OH9/5zJXUyMn0dHTEp7JJIT2TlQFZJz7s0xsJ+Pz67koqwHvvmTXbIElte3Z03NbPLgfuBPcA859xxOmAObDurj3DdvStpaPJ9JSeXDuG2D59GRoaCJOCvmn/hh/D3u9tf8Fd6Krz3SzDlEv/PvWoTbHraT5XLfYCPaTwCm57yE/hRByYugIkXpq7/aUNdfLizlqHP3o4PiVYbDIEWaxGKTYnPO1vX4bbZvpUjb3j8IqL84QnPh/fuIp133vRdXNb83rdyd2TEFB/Qp1/pf6/dlZEJI6f46bSP+GVNx/wp8cQA39XRWIaclBDIE4J5wcgT4wCtcJSf+kJmNpRM9BPvjy93zh8w7wv6+cdC/L6NnY/20VFQzxvmz3Sdcr6veyOmhnewlJMfre4hIhIXC/v9WCrCemww1mSHrLFvr2R92tsJLij9Lb5F/QLnXA8GIR44Djc08r9+/Xf2HfItiMMH53DXNadTMCjSJ07S4+BuWH4brPxl69AN/rT3e78Mk9/f+p987J/uWZ/yY2JXvhAP723Hwz5QCS/f5afMHBh7dhDeF8DIqe1DnXP+9H2sj15NQhiv3tb+lHpnGo/TjzbVMnNah/f8YW2et5mb+a5Eqx9MPp50YTmc+kHfVaP01NSH4MzsYAzmGTD7Wr/sWL3vwxrrQrN3nT+tPGJSvAtLSUW/PmUaWWb+NP6QUjjlvNbrjh7yB8r7NrZukY91ScotgpPnBH3Oz4OR06J9JkNEJEUiN3Sjmf1P4FfADlLcon4iDt3onOOzv32FR9fsAiArw7hv0ZmcNb445JKFrHZnENLvad8NomwGzP0KTH5f98PhgcoguC+Bt57rfCSJwnI/VjIkBPMdHXfLGEhyh8K0y30r+thzFLikc81NvnvJ4JJwLpQTEemhyIwG45zbbGZP4kd8+QxwR8LqxcBg/E2RWlKNmU0JXrsu8b3M7Br8RaRb8UE9yeX1EnP7kk0tQR3g5stO7XlQr94Gr/zGX0FecaEfXaK/qdkBy2+FVb9uH4pPOs2H9EkX97wFd9g4eM8iPzU2+KEFNz0NG59u33pcuwNeva/7n5GRnTCGc+KYxGPiN2zJzI6P89vcmPD4mA83HT6PjQvcmLAueN7U4C/cObzft+zH7mqX+Lw3BxlZuf7gaPpV/qyDhsaUrsrITO3NgURE+plU9ZP4NLACuN3M5gNr8VfizMN3f/lqm+3XBvOWxGRm8/BBPQNYCnzM2geqaufcbSkqc7/3lzW7uPXpeO+ia88Zx0fO7MEwg875Fugnv9a6b+jIaX6ovIqF/oK/rg6rFobq7T6kv3Jv+1E8Rs3yIb1iYWq7WWTl+P6yp5wPF97sW/M3LfHhffPS5BfS5Rb5ixFbhfDR8WUFpV1rbU7n/nDOX5zTKswnCfWxecMhfxZjxoeCkVxCvEmOiIhIP9XrbjAtb2Q2BrgZP8xiMb77y8PAYufcgTbbOgDnXGJYvxb45XE+ZqtzblwvynjCdIN5Y2cNV975IkeO+X7LcyaWcM/H3kNWZje7FNTsgD9d72+F3ZlBhTBhnr+QsuJC3/oeBdXbYNkt8Mp97W+EUX66D+kTF6T/gsCmRtjxd9j+N39hZmILeW5hessiIiIiaReZbjAxzrntwMe6uG275OScuwe4J1XlOZE1NztuenB1S1AfV5zPjz4ys3tB3Tl47X74y5dbtwAXV/hAufWF1i3UR2v9ONWxsarLZvjQXrHQh+J0jyN8YCss+wG8+tv2IX30GTD3y36M47BG7cjMan+zDhEREZFu0nAh/dCf1+xi7a5aAPKyM7nrmvcwNL8bfYAPvgN//nzruzxicPZn4IKv+dueHz3k7wy46SnY+FT7YdV2r/bTsh/4CwYnzvet7hMXQMGI3v+Q9TXtR0lpGb4wuKsgbc4KjT3bj+4yfu6JMbSeiIiIDHgK6/3MsaZmbnlyfcvzf5kzjokjuzG29+t/gEdvbD084LBxcPmdcPI58WWDCmDK+/zknB/ebuOTPrhve7H1ONX11fD67/2E+btCViz0Le+jZrYfwaG5CQ69k2T4wuBxsv7eHTn5XB/STzlfIV1EREROKArr/czvV75NZZW/C1dhbhafOG9C115YVwWP3ejvGpnoPYtgweLOb+Zj5scMHzkVzr0B6mvhrWfjre4HdyVs7IIb0KyC577t7yo44QJ/2/iat/0NcWp3du2mNJ0yP9by+V9qP16ziIiIyAlCYb0fqT/WxA+XxG/K88m5EyjK78KIIOseg/+6Aer2xJcVjobLfuQvGu2u3EJ416V+cs7fYGbjk374wu1/BZdws57DVT27pXtWbvyCzMQhDGMjpxSW+9ugi4iIiJzAFNb7kfte2squGn8XzpKCQVx7zrjOX3CkGh7/Crz2n62Xz/woXPSt1Nyh0czfCbRsOpx3ox/Gb/PSYOzxp1ofICTKL04I4GPbB/PBJerSIiIiIgOewno/cehoIz95dnPL8+svmEh+Tie7b9PT8Mj1cHBnfFlBKVx6B0y6qO8KmjcMTv2An5qbYfdrsO0lyM6PB/PCcsjJ77syiIiIiJwgFNb7iV8s28L+Oj+U4uhhefyPM5Lc/OjoQX9zo5X3tF4+/Sr4h+9C/vC+LWiijAx/gemomen7TBEREZETiMJ6P3CgroGfL3ur5fnnF0wiJ6uDMdW3LINHPu2HOozJL4F/vAXedVkaSioiIiIiqaSw3g/c+dxmDh31o6dMHFnAFTPLW2/QcBiWLIa//rT18qmXwPtvTc245yIiIiKSdgrrEbe7pp5frahseX7TwklkZiRceLntr/DHT8H+eH92cofC+74P06/URZoiIiIi/ZjCesTd8cxGjjY2AzC9vIiLppX5Fcfq4dlvwYo7wDXHX1CxEC65HQpPCqG0IiIiIpJKCusRtrWqjt+9vL3l+RcvmoyZwc5X4eFPwt618Y1zhsDF/+GHZVRruoiIiMgJQWE9wm59agONzQ6As8YP57zxRfDsd+D577a+A+j4uXDpj/wNg0RERETkhKGwHlHrdtfyyGvxMdK/fmYmdvdC2PlKfKPsfLjwZjj9436YRBERERE5oSisR9T3n9iAc2A08+2TljPtT7+EpqPxDcacCZffCcUTwiukiIiIiPQphfUIWrXtAE+vfYfRtocfZP+UMw+si6/MzIF5X4VzroeMzPAKKSIiIiJ9TmE9gr7/+Do+nPkMX8u6jwKrj68omw5X/AxKp4VXOBERERFJG4X1iPnba2+waPtXuCD71fhCy4Tz/hXO/xJk5YRXOBERERFJK4X1CHFrHmLqH29gSOah+MLiCt+aPnp2eAUTERERkVAorEdBXRU8diP2xsMMSVh8aOYnKHjfzZCdF1rRRERERCQ8CuthW/84/Ol6qNvTsuhtV8LSyYu5+rKPhlgwEREREQmbwnpY6mvhif8Nr9zXavH9jXO5JeMaHr/sH0MqmIiIiIhEhcJ6GLY8D3/8NNRsb1lUxTBualjE0uaZ3DC/guGDdSGpiIiIyECnsJ5ODYdhyWL4609bLa4sW8jllR+kmiEMy89m0XmnhFRAEREREYkShfV0efvv8PB1ULUpvixvGA0XfY+rHh1GNf7upJ+eO5EhudkhFVJEREREokRhva81NsBz34Hlt4Brji+vWAiX3sHdq+rYe9DfobS0cBBXn31ySAUVERERkahRWO9rtTvgpZ/Eg3pOAVz0LZj1z9TUN3Lns6tbNv3c/ApyszNDKqiIiIiIRE1G2AU44Q0/BRb+u3988hz41AqYfQ2Ycdeyt6g5csyvKs7nQ6ePCbGgIiIiIhI1allPh9M/DvklMPVSyPDHR/sOHeUXy7e0bPKvF04iO1PHTiIiIiISp7CeDmYw7fJWi368dBOHG5oAmFI2hEtmjAqjZCIiIiISYWrKDcGO6iP85qVtLc9vXDiZjAwLsUQiIiIiEkUK6yG4/emNNDT5C05njh3KgqkjQy6RiIiIiESRwnqabd57iAdXxu9c+sWLJmOmVnURERERaU9hPc1ueWoDzc4/njOxhHMmlIRbIBERERGJLIX1NHp9Rw2Prt7V8vyLF00OsTQiIiIiEnUK62n0/SfXtzy+aFop7x4zNMTSiIiIiEjUKaynyd+27OfZ9XsBP5LjjQvVqi4iIiIinVNYTwPnHN97Yl3L8ytmljOpdEiIJRIRERGR/kBhPQ2e3bCXlysPAJCdaXxhwaSQSyQiIiIi/YHCeh9rbnZ8/4l4X/UPv2csY4bnh1giEREREekvFNb72K7aemqOHAMgNzuD6y+YGHKJRERERKS/yAq7ACe68qF5PHPjXH738jYONzQxsjA37CKJiIiISD+hsJ4GOVkZXH32uLCLISIiIiL9jLrBiIiIiIhElMK6iIiIiEhEKayLiIiIiESUwrqIiIiISESlLKyb2Wgzu9vMdprZUTOrNLPbzGxYN97jQjP7gZktMbMqM3NmtjxVZRQRERER6U9SMhqMmU0AVgAjgUeAdcAZwA3AxWZ2rnOuqgtv9RngMqAe2AQMT0X5RERERET6o1S1rP8EH9Q/55y73Dn3FefcBcCtwGTgm118n+8ApwIFwCUpKpuIiIiISL/U67AetKovBCqBH7dZ/Q2gDrjazAYf772ccy86595wzjX1tlwiIiIiIv1dKlrW5wXzJ51zzYkrnHMHgReAfOCsFHyWiIiIiMiAkYo+65OD+YYk6zfiW94nAUtS8HnHZWYrk6yako7PFxERERFJhVS0rBcF85ok62PLh6bgs0REREREBoyUjAYTNc652R0tD1rcZ6W5OCIiIiIiPZKKlvVYy3lRkvWx5dUp+CwRERERkQEjFWF9fTCflGR9RTBP1qddREREREQ6kIqwvjSYLzSzVu9nZkOAc4HDwEsp+CwRERERkQGj12HdObcZeBIYh78DaaLFwGDgXudcXWyhmU0xM43MIiIiIiLSCXPO9f5N/I2RVuDvYvoIsBY4Ez8G+wbgHOdcVcL2DsA5Z23eZw6wKHhaAHwQ2AP8JbaNc+7aXpSzKi8vb/jUqVN7+hYiIiIiIse1du1ajhw5st85V9yb90lJWAcwszHAzcDFQDGwC3gYWOycO9Bm22Rh/Vrgl519TtvXdLOMW4BC/N1W0y12JmFdCJ8tXaN9FH3aR9GnfRR92kfRpv0TfV3dR+OAWufcKb35sJSFdelc7EZNyYaVlPBpH0Wf9lH0aR9Fn/ZRtGn/RF+691EqLjAVEREREZE+oLAuIiIiIhJRCusiIiIiIhGlsC4iIiIiElEK6yIiIiIiEaXRYEREREREIkot6yIiIiIiEaWwLiIiIiISUQrrIiIiIiIRpbAuIiIiIhJRCusiIiIiIhGlsC4iIiIiElEK6yIiIiIiEaWw3sfMbLSZ3W1mO83sqJlVmtltZjYs7LIJBPvDJZl2h12+gcLMrjSzO8xsmZnVBr//+47zmnPM7DEz229mR8xstZl93swy01XugaQ7+8jMxnVSr5yZ3Z/u8p/ozKzYzBaZ2cNmtimoEzVmttzMPm5mHf6/Vz1Kn+7uI9WjcJjZd8xsiZltD/bRfjN7xcy+YWbFSV7Tp/UoKxVvIh0zswnACmAk8AiwDjgDuAG42MzOdc5VhVhE8WqA2zpYfijdBRnAvga8G/87fxuY0tnGZnYZ8HugHvgdsB+4BLgVOBe4qi8LO0B1ax8FXgP+2MHy11NYLvGuAu4EdgFLgW1AKfAB4C7gH8zsKpdwJ0TVo7Tr9j4KqB6l1xeAVcBTwB5gMHAW8G/AJ8zsLOfc9tjGaalHzjlNfTQBTwAOuL7N8luC5T8Nu4wDfQIqgcqwyzHQJ2AeUAEYMDeoH/cl2bYw+AI9CpyesDwXf3DsgA+H/TOdaFM399G4YP09YZd7oEzABUFAyGizvAwfCh3wwYTlqkfR30eqR+Hsp9wky78Z7I+fJCxLSz1SN5g+ErSqL8SHwR+3Wf0NoA642swGp7loIpHjnFvqnNvogm+547gSGAHc75z7e8J71ONbfwE+1QfFHNC6uY8kzZxzzzjn/ss519xm+W7gp8HTuQmrVI/SrAf7SEIQ1IGOPBDMKxKWpaUeqRtM35kXzJ/soGIeNLMX8GH+LGBJugsnrQwys48CY/EHUauB551zTeEWS5K4IJg/3sG654HDwDlmNsg5dzR9xZIOjDKz64BioAp40Tm3OuQyDUTHgnljwjLVo2jpaB/FqB5FwyXBPPF3n5Z6pLDedyYH8w1J1m/Eh/VJKKyHrQy4t82yLWb2Mefcc2EUSDqVtG455xrNbAswDRgPrE1nwaSdC4OphZk9C1zjnNsWSokGGDPLAv45eJoYKFSPIqKTfRSjehQCM7sJKACKgNOBOfig/u2EzdJSj9QNpu8UBfOaJOtjy4emoSyS3C+B+fjAPhiYDvwM31fwL2b27vCKJkmobkXfYeDfgdnAsGB6L/6iurnAEnUBTJtvA6cCjznnnkhYrnoUHcn2kepRuG7Cd1v+PD6oPw4sdM7tTdgmLfVIYV0GNOfc4qAf4TvOucPOudedc5/EXwSch7/6W0S6wTm3xzn3f51zq5xz1cH0PP5s4l+BicCicEt54jOzzwE34kciuzrk4kgHOttHqkfhcs6VOecM35j3AXzr+CtmNivdZVFY7zuxo6miJOtjy6vTUBbpvtjFPueHWgrpiOpWP+Wca8QPUQeqW33KzD4L/BB4E5jnnNvfZhPVo5B1YR91SPUovYLGvIfxB0nFwK8TVqelHims9531wXxSkvWxq4mT9WmXcMVOc+kUY/QkrVtB389T8BdpvZXOQkmXqW71MTP7PHAHfhzuecFoI22pHoWoi/uoM6pHaeac24o/sJpmZiXB4rTUI4X1vrM0mC/s4K5kQ/AD5R8GXkp3waRLzgrm+kcVPc8E84s7WHc+kA+s0AgWkaW61YfM7Mv4m7G8ig+Be5JsqnoUkm7so86oHoVjVDCPjRaXlnqksN5HnHObgSfxFyp+ps3qxfij4Xudc3VpLpoEzGxqRxfnmNk44EfB005veS+heAjYB3zYzE6PLTSzXOD/BU/vDKNg4pnZrI5ub29m8/F3BwTVrZQzs6/jL1ZcCcx3zu3rZHPVoxB0Zx+pHqWfmU0ys3ZdWswsw8y+ib8j/Qrn3IFgVVrqken+Fn0nuDHSCvzOfQQ/bM+Z+DHYNwDnOOeqwivhwGZm/4a/sOd5YCtwEJgAvB9/97HHgCuccw1hlXGgMLPLgcuDp2XARfgWo2XBsn3OuZvabP8Q/vbO9+Nv73wpfhith4AP6eY9qdWdfRQMK1eB//57O1g/g/iYxF93zsX+kUkKmNk1wD34Fr876Hh0ikrn3D0Jr1E9SqPu7iPVo/QLuif9B7Ac2IIf174UPwrPeGA3/iDrzYTX9Hk9UljvY2Y2BrgZf4qkGNgFPAwsTjgykxCY2XuBTwIziQ/dWI0/NXkv/syHKkgaBAdO3+hkk63OuXFtXnMu8FXgbPzB1SbgbuB23dAq9bqzj8zs48AV+OHoSoBs4B3gReBHzrllyd5EeqYL+wfgOefc3DavUz1Kk+7uI9Wj9DOzU/G5YA4wGj/kYh2+gfVRfL1odyFwX9cjhXURERERkYhSn3URERERkYhSWBcRERERiSiFdRERERGRiFJYFxERERGJKIV1EREREZGIUlgXEREREYkohXURERERkYhSWBcRERERiSiFdRERERGRiFJYFxERERGJKIV1EREREZGIUlgXEREREYkohXURERERkYhSWBcRERERiSiFdRERERGRiFJYFxERERGJKIV1EREREZGI+m+npFsLufI6kgAAAABJRU5ErkJggg==\n", 386 | "text/plain": [ 387 | "
" 388 | ] 389 | }, 390 | "metadata": { 391 | "image/png": { 392 | "height": 248, 393 | "width": 373 394 | }, 395 | "needs_background": "light" 396 | }, 397 | "output_type": "display_data" 398 | } 399 | ], 400 | "source": [ 401 | "for k in ['accuracy', 'val_accuracy']:\n", 402 | " plot(h.history[k], )" 403 | ] 404 | }, 405 | { 406 | "cell_type": "code", 407 | "execution_count": 12, 408 | "metadata": {}, 409 | "outputs": [ 410 | { 411 | "name": "stdout", 412 | "output_type": "stream", 413 | "text": [ 414 | "./results-v2/cifar100/FC_shallow-u1075-n_lay-1-act-relu-rotate=False-scramble=False.json\n" 415 | ] 416 | } 417 | ], 418 | "source": [ 419 | "model_name = f\"FC_shallow-u{u}\" \n", 420 | "\n", 421 | "\n", 422 | "model_name += f\"-n_lay-{net['num_layers']}\"\n", 423 | "model_name += f\"-act-{net['activation']}\"\n", 424 | "num_params = model.count_params() \n", 425 | "out_file_name = f\"./results-v2/{dataset_name}/{model_name}-rotate={bool(configs['dataset']['rotate'])}-scramble={bool(configs['dataset']['scramble'])}.json\"\n", 426 | "\n", 427 | "configs['net']['architecture'] = 'fc_shallow'\n", 428 | "\n", 429 | "results = {}\n", 430 | "results.update({\n", 431 | " 'num_params':num_params,\n", 432 | " 'result':h.history,\n", 433 | " 'configs':configs,\n", 434 | "# 'result': {k: np.float32(v).tolist() for k,v in h.history.items()}, # bug in json or TF2\n", 435 | " })\n", 436 | "\n", 437 | "\n", 438 | "\n", 439 | "# for k,v in results['result'].items():\n", 440 | "# print(k,type(v))\n", 441 | "\n", 442 | "import os\n", 443 | "\n", 444 | "# print(h.history)\n", 445 | "\n", 446 | "dirs = os.path.split(out_file_name)[0]\n", 447 | "os.makedirs(dirs,exist_ok=True)\n", 448 | "\n", 449 | "print(out_file_name)\n", 450 | "\n", 451 | "json.dump(results, open(out_file_name, 'w'))" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "# CNN + Max pool" 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": 7, 464 | "metadata": {}, 465 | "outputs": [ 466 | { 467 | "data": { 468 | "text/plain": [ 469 | "{'dataset': {'name': 'cifar100', 'rotate': True, 'scramble': True},\n", 470 | " 'net': {'architecture': 'lconv',\n", 471 | " 'num_filters': 32,\n", 472 | " 'kernel_size': 9,\n", 473 | " 'L_hid': [8],\n", 474 | " 'activation': 'relu',\n", 475 | " 'L_trainable': True,\n", 476 | " 'num_layers': 1}}" 477 | ] 478 | }, 479 | "execution_count": 7, 480 | "metadata": {}, 481 | "output_type": "execute_result" 482 | } 483 | ], 484 | "source": [ 485 | "net['num_layers'] = 1\n", 486 | "configs\n" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": 8, 492 | "metadata": {}, 493 | "outputs": [ 494 | { 495 | "name": "stdout", 496 | "output_type": "stream", 497 | "text": [ 498 | "Model: \"functional_3\"\n", 499 | "_________________________________________________________________\n", 500 | "Layer (type) Output Shape Param # \n", 501 | "=================================================================\n", 502 | "input_2 (InputLayer) [(None, 32, 32, 3)] 0 \n", 503 | "_________________________________________________________________\n", 504 | "conv2d (Conv2D) (None, 30, 30, 32) 896 \n", 505 | "_________________________________________________________________\n", 506 | "max_pooling2d (MaxPooling2D) (None, 29, 29, 32) 0 \n", 507 | "_________________________________________________________________\n", 508 | "flatten_1 (Flatten) (None, 26912) 0 \n", 509 | "_________________________________________________________________\n", 510 | "dense_1 (Dense) (None, 100) 2691300 \n", 511 | "=================================================================\n", 512 | "Total params: 2,692,196\n", 513 | "Trainable params: 2,692,196\n", 514 | "Non-trainable params: 0\n", 515 | "_________________________________________________________________\n" 516 | ] 517 | } 518 | ], 519 | "source": [ 520 | "net = configs['net']\n", 521 | "\n", 522 | "kx = int(0.5+np.sqrt(net['kernel_size']))\n", 523 | "ky = int(0.5+net['kernel_size']/kx)\n", 524 | "kernel_size = (kx,ky)\n", 525 | "\n", 526 | "\n", 527 | "inp = Input(x_train[0].shape)\n", 528 | "\n", 529 | "x = inp\n", 530 | "for _ in range(net['num_layers']):\n", 531 | " x = Conv2D(filters=net['num_filters'], \n", 532 | " kernel_size=kernel_size, \n", 533 | " activation = net['activation'])(x)\n", 534 | "\n", 535 | " x = MaxPool2D(pool_size=(2, 2),strides=(1, 1), padding='valid')(x)\n", 536 | "\n", 537 | "\n", 538 | "x = Flatten()(x)\n", 539 | "\n", 540 | "# x = Dense(100, activation = 'relu')(x)\n", 541 | "\n", 542 | "out = Dense(y_train.shape[-1], activation='softmax')(x)\n", 543 | "\n", 544 | "model = Model(inputs = [inp], outputs = [out])\n", 545 | "model.compile(loss = tf.keras.losses.categorical_crossentropy, metrics = ['accuracy'])\n", 546 | "\n", 547 | "model.summary()\n", 548 | "\n" 549 | ] 550 | }, 551 | { 552 | "cell_type": "code", 553 | "execution_count": 9, 554 | "metadata": {}, 555 | "outputs": [ 556 | { 557 | "name": "stdout", 558 | "output_type": "stream", 559 | "text": [ 560 | "Epoch 1/30\n", 561 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.9167 - accuracy: 0.1236 - val_loss: 3.7271 - val_accuracy: 0.1473\n", 562 | "Epoch 2/30\n", 563 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.5135 - accuracy: 0.1927 - val_loss: 3.6933 - val_accuracy: 0.1582\n", 564 | "Epoch 3/30\n", 565 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.3421 - accuracy: 0.2239 - val_loss: 3.6467 - val_accuracy: 0.1758\n", 566 | "Epoch 4/30\n", 567 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.2010 - accuracy: 0.2546 - val_loss: 3.6889 - val_accuracy: 0.1663\n", 568 | "Epoch 5/30\n", 569 | "1250/1250 [==============================] - 5s 4ms/step - loss: 3.0754 - accuracy: 0.2845 - val_loss: 3.8055 - val_accuracy: 0.1799\n", 570 | "Epoch 6/30\n", 571 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.9545 - accuracy: 0.3071 - val_loss: 3.7576 - val_accuracy: 0.1786\n", 572 | "Epoch 7/30\n", 573 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.8452 - accuracy: 0.3301 - val_loss: 3.8578 - val_accuracy: 0.1732\n", 574 | "Epoch 8/30\n", 575 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.7313 - accuracy: 0.3489 - val_loss: 3.8588 - val_accuracy: 0.1688\n", 576 | "Epoch 9/30\n", 577 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.6140 - accuracy: 0.3762 - val_loss: 3.9497 - val_accuracy: 0.1739\n", 578 | "Epoch 10/30\n", 579 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.5057 - accuracy: 0.4000 - val_loss: 3.9963 - val_accuracy: 0.1631\n", 580 | "Epoch 11/30\n", 581 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.3924 - accuracy: 0.4230 - val_loss: 4.0768 - val_accuracy: 0.1632\n", 582 | "Epoch 12/30\n", 583 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.2901 - accuracy: 0.4454 - val_loss: 4.1800 - val_accuracy: 0.1581\n", 584 | "Epoch 13/30\n", 585 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.1808 - accuracy: 0.4697 - val_loss: 4.2969 - val_accuracy: 0.1600\n", 586 | "Epoch 14/30\n", 587 | "1250/1250 [==============================] - 5s 4ms/step - loss: 2.0804 - accuracy: 0.4894 - val_loss: 4.4049 - val_accuracy: 0.1651\n", 588 | "Epoch 15/30\n", 589 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.9885 - accuracy: 0.5112 - val_loss: 4.4964 - val_accuracy: 0.1609\n", 590 | "Epoch 16/30\n", 591 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.8966 - accuracy: 0.5336 - val_loss: 4.6298 - val_accuracy: 0.1590\n", 592 | "Epoch 17/30\n", 593 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.8040 - accuracy: 0.5528 - val_loss: 4.6630 - val_accuracy: 0.1627\n", 594 | "Epoch 18/30\n", 595 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.7130 - accuracy: 0.5712 - val_loss: 4.8223 - val_accuracy: 0.1548\n", 596 | "Epoch 19/30\n", 597 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.6312 - accuracy: 0.5905 - val_loss: 4.9647 - val_accuracy: 0.1609\n", 598 | "Epoch 20/30\n", 599 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.5499 - accuracy: 0.6095 - val_loss: 5.1244 - val_accuracy: 0.1548\n", 600 | "Epoch 21/30\n", 601 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.4663 - accuracy: 0.6308 - val_loss: 5.3002 - val_accuracy: 0.1550\n", 602 | "Epoch 22/30\n", 603 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.3956 - accuracy: 0.6469 - val_loss: 5.4341 - val_accuracy: 0.1556\n", 604 | "Epoch 23/30\n", 605 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.3252 - accuracy: 0.6654 - val_loss: 5.7170 - val_accuracy: 0.1451\n", 606 | "Epoch 24/30\n", 607 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.2559 - accuracy: 0.6827 - val_loss: 5.7569 - val_accuracy: 0.1495\n", 608 | "Epoch 25/30\n", 609 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.1830 - accuracy: 0.6993 - val_loss: 5.9255 - val_accuracy: 0.1458\n", 610 | "Epoch 26/30\n", 611 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.1174 - accuracy: 0.7147 - val_loss: 6.1386 - val_accuracy: 0.1498\n", 612 | "Epoch 27/30\n", 613 | "1250/1250 [==============================] - 5s 4ms/step - loss: 1.0562 - accuracy: 0.7307 - val_loss: 6.3020 - val_accuracy: 0.1465\n", 614 | "Epoch 28/30\n", 615 | "1250/1250 [==============================] - 5s 4ms/step - loss: 0.9958 - accuracy: 0.7465 - val_loss: 6.4740 - val_accuracy: 0.1424\n", 616 | "Epoch 29/30\n", 617 | "1250/1250 [==============================] - 5s 4ms/step - loss: 0.9368 - accuracy: 0.7599 - val_loss: 6.6097 - val_accuracy: 0.1445\n", 618 | "Epoch 30/30\n", 619 | "1250/1250 [==============================] - 5s 4ms/step - loss: 0.8849 - accuracy: 0.7746 - val_loss: 6.8493 - val_accuracy: 0.1477\n" 620 | ] 621 | } 622 | ], 623 | "source": [ 624 | "h = model.fit(x_train, y_train, validation_split=0.2, epochs=30)" 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": 10, 630 | "metadata": {}, 631 | "outputs": [ 632 | { 633 | "data": { 634 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAusAAAHzCAYAAACZlFNWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3iV5f3H8c99MiGETdgjCYSwp6CiIA5A3KvaVqtoqzjqaOuoW3+OOuqs1lFn1SrFgRYEHAwFkaXMhEBCIEhYAUIGmef+/XFODjmEQEgOec5J3q/r8nrK98558r16Cflwez/fx1hrBQAAACD4uJxuAAAAAMChEdYBAACAIEVYBwAAAIIUYR0AAAAIUoR1AAAAIEgR1gEAAIAgRVgHAAAAghRhHQAAAAhShHUAAAAgSBHWAQAAgCBFWAcAAACCFGEdAAAACFKEdQAAACBIEdYBAACAIBWwsG6M6WKMedMYs9UYU2yMyTTGPGeMaXWU9znJGDPN+/kiY8xmY8wMY8yEQPUKAAAAhAJjra37TYxJlLRQUpykaZJSJY2QNFbSOkmjrLU5NbjP9ZJellQg6VNJWyR1kXShpKaS7rXWPlqHPjdKai4ps7b3AAAAAGqgh6R91tr4utwkUGF9lqRxkm621r5Yqf6MpNskvWqtnXyEe0RI2ikpStJga+26Smt9JP0kyS2plbW2uJZ95jRp0qR1nz59avNxAAAAoEZSUlK0f//+3dbaNnW5T53DundXfYM8u9WJ1lp3pbVYSdmSjKQ4a23BYe7TXtI2SSuttYMOsb5S0gBJbWuyS1/N91g2dOjQocuWLavNxwEAAIAaGTZsmJYvX77cWjusLvcJxJn1sd7r7MpBXZKstXmSFshzhOX4I9xnhzw760nGmF6VF4wxSZJ6Sfq5tkEdAAAACDXhAbhHb+81rZr19fIckUmS9E11N7HWWmPMjZLek7TMGPOppK2SOku6QNIaSZfVpCFjTHVb58k1+TwAAAAQDAIR1lt4r7nVrFfUWx7pRtba/xpjtkr6j6TfVVraLuktSRm1bRIAAAAINUE1Z90Yc7mkryV9J6mPPMdn+sizI/8PSR/W5D7W2mGH+keeKTUAAABASAhEWK/YOW9RzXpFfe/hbuI9l/6mPMddrrDWplpr91trUyVdIWmZpEuMMafUvWUAAAAg+AUirFeMWEyqZr3iYdHqzrRXGCcpQtK8Qzyo6pY03/vLOj1RCwAAAISKQIT1Od7rOGOM3/28oxtHSSqUtOgI94nyXttVs15RL6lNkwAAAECoqXNYt9amS5otz1uabjxo+SFJMZL+XXnGujEm2Rhz8GSW77zXi40xAysvGGMGS7pYkpX0bV17BgAAAEJBIKbBSNINkhZKesEYc5qkFEkj5ZnBnibpnoO+PsV7NRUFa+1iY8xbkiZJWuId3bhJnr8EnC8pUtJz1to1AeoZAAAACGoBCevW2nRjzHBJD0uaIGmiPG8ufV7SQ9baPTW81TXynE2/StJ4SbGS9kn6XtLr1toaTYMBAAAAGoJA7azLWpslz654Tb7WVFO3kt72/gMAAAA0akE1Zx0AAADAAYR1AAAAIEgR1gEAAIAgRVgHAAAAghRhHQAAAA1W6rZ9ytiZ73QbtRawaTAAAABAMCgrd+urtdv19sJM/bhxt84d1Ekv/HqI023VCmEdAAAADcLughL9Z/Fmvb9ok7bmFvnqM1Zl696z+iiuebSD3dUOYR0AAAAhbdWWXL29MFNfrNyqkjK331qYy2h8vw4qKnVX8+ngRlgHAABAyCkpc+vL1dl6Z2Gmlm/eW2W9TUykfj2im357fDd1bNHEgQ4Dg7AOAACAkLEjr0gf/LhZ7/+4WTvziqusD+zSQlee0ENnDeyo6IgwBzoMLMI6AAAAgpq1Vss379U7CzP15epslZZbv/WIMKOJAzrqyhN7aEjXljLGONRp4BHWAQAAEJSKSsv1v5Weoy6rfsmtsh4XG6XfjuyuX4/sqrjY0Ht4tCYI6wAAAAgq2bn79d6iTfpwcZZyCkqqrA/r3kpXnthDE/p1UGR4w35tEGEdAAAAjrPWavHG3Xrnh0zNWrNd5W7/oy6R4S6dO6iTrjqxh/p3buFMkw4grAMAAMAxOfnF+vSnX/TRkiyt31H1TaOdWkTr8hO667Ljuql1TKQDHTqLsA4AAIB6Ve62+m79Tk1ZmqWv1m6v8sCoJB2f0FpXndhDp/dpr/Cwhn3U5XAI6wAAAKgXWbsL9d+lWZq6bIvfG0YrNI0M03mDO+vKE7sruUNzBzoMPoR1AAAAHDNFpeWavXa7pizJ0oL0XbJVN9E1pFtLXTq8q84e1EnNooinlfH/BgAAAAIuJXufPlqSpc9+/kV7C0urrLeOidSFQzrrV8d1VVL7WAc6DA2EdQAAAATEvqJSff7zVk1ZmqWVW6rORTdGGt2rnS49rqtO79O+wY9dDATCOgAAAGqtYuTiR0uzNGNVtopK3VW+pnPLJvrV8K66eHgXdW7ZxIEuQxdhHQAAAEdtR16RPl72i6YszdLGXQVV1iPDXBrfv4MuHd5VJya2kctlHOgy9BHWAQAAUGMrsvbqtfkZmrlmW5UXF0lScodYXXpcV50/uLNaNcK56IFGWAcAAMBhud1Wc9N26NV5Gfpx4+4q67FR4Tp3cCddelxXDejcQsawix4ohHUAAAAcUnFZuab9vFWvz8845NtFR8S31qXDu2rigI5qEhnmQIcNH2EdAAAAfnL3l+qDHzfrrQUbtSOv2G8t3GV0zqBO+sPJCerbiRcXHWuEdQAAAEiStu7drze/36gPl2Qpv7jMby0mMky/HtFNV58Ur05MdKk3hHUAAIBGLiV7n16bn6EvVmxV2UEPjcbFRmnSqHj9ZmQ3tWgS4VCHjRdhHQAAoBGy1mrBhhy9Oj9d363fVWW9V1wz/WF0gs4b3ElR4ZxHdwphHQAAoBEpK3dr+qpsvTY/Q2u27quyPjK+ta4bk6BTkuKYjR4ECOsAAACNQEFxmT5ckqU3v9+oX/bu91tzGenM/h117egEDera0qEOcSiEdQAAgAZsR16R3lmYqfcWbVbu/lK/tegIl341vKuuOSle3dvEONQhDoewDgAA0ABt3FWg1+Zn6OPlW1RS5vZbax0TqStP6KErTuiu1rxlNKgR1gEAABqQFVl79cq8dM1cs03Wf7CLerRpqt+fnKCLh3VRdAQPjYYCwjoAAECIs9ZqXtpOvTIvXYsydldZH9S1pSaPTtC4fh0UxkOjIYWwDgAAEKIqJru8Mi9DKdlVJ7uc0rudJo9J1Mj41jKGkB6KCOsAAAAhprCkTFOWZOn176pOdglzGZ07qJOuHZ2gPh2bO9QhAoWwDgAAECJ2F5TonYWZeveHTO0p9J/s0iQiTJeN8Ex26dKqqTMNIuAI6wAAAEEua3eh3vh+oz5csllFpYee7PK7E7qrFZNdGhzCOgAAQJBau3WfXp2frv+tzFa523+0S5dWTXTt6ARdMqyrmkQy2aWhIqwDAAAEEWutfsjI0SvzMjQ/bWeV9b4dm2vyKYma2L+DwsNcDnSI+kRYBwAACAJl5W7NWrNdr81P14otuVXWT0xso8ljEnVyr7ZMdmlECOsAAAAO2ldUqilLsvTWgswqk11cRjqzf0ddNyZBA7u0dKhDOImwDgAA4ICs3YV6a0GmpizNUn5xmd9aZLhLlwzroj+cnKAebWMc6hDBgLAOAABQT6y1Wr55j/713UbNWrNNBz0zqtYxkbp8ZDddcUIPtYuNcqZJBBXCOgAAwDFWVu7WzDXb9K/vNurnrL1V1nvGNdM1J8XrgiGdFR3BZBccELCwbozpIulhSRMktZGULekzSQ9Za/fU4POnSJpTg2/VzVqbVYdWAQAA6sW+olJ9tDhLby+seh5dkk7u1VbXnBSv0b3ayeXioVFUFZCwboxJlLRQUpykaZJSJY2QdIukCcaYUdbanCPcJlPSQ9WsDZB0oaTVBHUAABDssnYX6s0FGzVlSZYKSsr91iLDXDp/SCddfVK8kjs0d6hDhIpA7ay/LE9Qv9la+2JF0RjzjKTbJD0qafLhbmCtzZT04KHWjDH/8f7P1wPQKwAAQMBZa7Vs0x698f1hzqMf311XHN+d8+iosTqHde+u+jh5dsZfOmj5AUnXSrrCGPNna21BLe7fVtIFkvZLerdu3QIAAARWablbX67epje+36gV1ZxH//1J8Tqf8+iohUDsrI/1Xmdba92VF6y1ecaYBfKE+eMlfVOL+18pKUrSu9baqr8DAAAAHLCvqFQfLt6stxdkamtuUZX1ivPoY5La8RIj1Fogwnpv7zWtmvX18oT1JNUurP/Be321ph8wxiyrZim5Ft8fAADAJye/WG8u2Kh3F25S3sHz0b3n0a85KUG9O8Q61CEakkCE9Rbea9X34vrXj/q1W8aYMfL8ZWC1tXZhLXoDAAAIiOzc/Xp9/kb9Z/Fm7S/1f2i0jfc8+uWcR0eABfuc9Wu919eO5kPW2mGHqnt33IfWtSkAANB4bMop0Cvz0jV12RaVlvs/NZrQNkbXjk7gPDqOmUCE9Yqd8xbVrFfUj+q8uTGmtaSL5Hmw9N+1aw0AAKB20rbn6eU5G/T5iq1VJrv06dhcN45N1Jn9OyqM+eg4hgIR1td5r0nVrPfyXqs7016digdL3+HBUgAAUF9Wbtmrl+Zs0Kw126usDe3WUjed2lNje8fx0CjqRSDCesVbR8cZY1yVJ8IYY2IljZJUKGnRUd634sHSozoCAwAAUBs/ZuToH3M26Lv1u6qsjerZRjeO7akTEtoQ0lGv6hzWrbXpxpjZ8kx8uVHSi5WWH5IUI+nVyjPWjTHJ3s+mHuqexpiTJfURD5YCAIBjyFqreWk79dKcDVqSuafK+ul92uvGsYka0q2VA90BgXvA9AZJCyW9YIw5TVKKpJHyzGBPk3TPQV+f4r1W91fTWj1YCgAAUBNut9WsNdv00twNWv3LPr81l5HOGthJN5ySqD4dmzvUIeARkLDu3V0fLulhSRMkTZSULel5SQ9Za6v+VbUaxphWki4WD5YCAIAAKy1364sVW/Xy3HRt2JHvtxYRZnThkC6afEqi4tvGONQh4C9goxuttVmSJtXwa6s97OUN9k0C1RcAAEBxWbn+u3SLXpmXri179vutRYW79OsR3XTt6AR1akkEQXAJ9jnrAAAAtVZcVq4pS7L00px0bdtX5LfWLCpcV5zQXdecFK+2zXiREYITYR0AADQ4hwvpLZtG6OpR8bryhB5q0TTCoQ6BmiGsAwCABuNwIb1tsyhdNzpBvxnZTTFRRCCEBv5NBQAAIa8ipL88N13ZuVVD+uQxCfrtyO5qEhnmUIdA7RDWAQBAyCKko6EjrAMAgJBDSEdjQVgHAAAho7isXFOWbtHLczYQ0tEoENYBAEDQI6SjsSKsAwCAoHX4kB6pyWMSCelo0AjrAAAg6BDSAQ/COgAACBpl5W5NWbpFL367npAOiLAOAACCgLVWX67epqdnrVPGrgK/NUI6GjPCOgAAcNTC9F164stUrdiS61cnpAOEdQAA4JDVv+TqyVnrND9tp189Njpck8ckatKoHmoaSVRB48bvAAAAUK825xTq6dnr9PmKrX71yHCXrjqxh64fk6hWMZEOdQcEF8I6AACoFzvzivWPb9frg8WbVVpufXWXkS4a2kW3npGkzi2bONghEHwI6wAA4JjKLy7Ta/Mz9K/vMlRYUu63dkbf9rp9fG8ltY91qDsguBHWAQDAMVFcVq4Pftysf3y7QTkFJX5rx/VopbvOTNaw7q0d6g4IDYR1AAAQUG631ecrturp2eu0Zc9+v7Xe7WN1x4TeOjU5TsYYhzoEQgdhHQAABIS1VnPTdurJmeuUkr3Pb61zyyb60xlJOn9IZ4W5COlATRHWAQBAnf20eY/+9mWqfty426/eqmmEbhzbU5cf313REcxKB44WYR0AANRa2vY8PTM7TTPXbPOrN4kI0+9PjtcfRieoeXSEQ90BoY+wDgAAjtqSzN16ZW66vknd4VcPdxldNqKrbj6tl+Jiox3qDmg4COsAAKBG3G6rb1N36JV56Vq6aU+V9bMHdtRfxvVWj7YxDnQHNEyEdQAAcFglZW59vmKrXp2XrvU78v3WjJHG9W2vm8b20oAuLRzqEGi4COsAAOCQCorL9OGSLP3ruwxl5xb5rUWEGV0wpLOuHZ2onnHNHOoQaPgI6wAAwE9OfrHeWZipd37YpNz9pX5rMZFh+u3x3XX1qHh1aMGZdOBYI6wDAABJUtbuQr3+XYamLM1SUanbb61ts0hNGhWvy0d2V4umTHcB6gthHQCARm7t1n16dX66/rcyW+Vu67fWrXVTXTs6QRcP68KcdMABhHUAABoha61+3Lhb/5ybrnlpO6us9+vUXNefkqgz+3fkjaOAgwjrAAA0Im631ey12/XKvHT9nLW3yvqonm00eUyiTurZVsYQ0gGnEdYBAGgErLWasWqb/v7VOmXsLPBbM0Y6s38HTR6TqIFdWjrUIYBDIawDANDAZe4q0H3TVuu79bv86pFhLl00rIuuHZ2geF5kBAQlwjoAAA1UcVm5XpmboZfmblBJ2YHpLrFR4d7xiz0U15zxi0AwI6wDANAALdiwS/d9tloZuw4ceXEZ6Xcn9NBtZySpRRPGLwKhgLAOAEADsiOvSI9OT9G0n7f61Qd2aaFHzx+gAV1aONQZgNogrAMA0ACUu60++HGTnpy1TnlFZb56bFS4bp/QW78d2Z0RjEAIIqwDABDiVv+Sq3s+XaUVW3L96ucO6qR7z+rDuXQghBHWAQAIUXlFpfr77DS9+0OmKr94NL5tjP7vvP46qVdbx3oDEBiEdQAAQoy1VtNXZevhL9ZqR16xrx4Z7tINpyRq8phERUeEOdghgEAhrAMAEEI25RTovmlrND9tp1/95F5t9fB5/ZmXDjQwhHUAAEJAcVm5Xp2XoX/M8Z+Z3i42Svef3VdnD+woY3iAFGhoCOsAAAS5hRt26d5qZqb/aVySmkczMx1oqAjrAAAEqZ15xXp0+lp9xsx0oNEirAMAEGRKytx6/8dNeuarNGamA40cYR0AgCBhrdWXq7fpyZmpyswp9FtjZjrQOAUsrBtjukh6WNIESW0kZUv6TNJD1to9R3mvoZL+Imm0pHaS9kpKlfSGtfbdQPUMAECwWLZptx6dnqLlm/f61ZmZDjRuAQnrxphESQslxUmaJk+wHiHpFkkTjDGjrLU5NbzXTZKel7RH0nRJv0hqLam/pImSCOsAgAZj464CPfFlqmau2eZXb9EkQn88taeuOKG7osKZmQ40VoHaWX9ZnqB+s7X2xYqiMeYZSbdJelTS5CPdxBgzTtILkr6SdLG1Nu+gdR53BwA0CDn5xXrhm/V6/8fNKqv0+tHIMJeuPLG7bhzbUy2bRjrYIYBgUOew7t1VHycpU9JLBy0/IOlaSVcYY/5srS3Q4T0lab+k3xwc1CXJWlta134BAHBSUWm53vh+o16Zm6684jK/tXMGddId43ura+umDnUHINgEYmd9rPc621rrrrxgrc0zxiyQJ8wfL+mb6m5ijOkvaaA859x3G2PGShomyUr6WdKcg+8PAECocLutPvnpF/199jpl5xb5rY2Ib617JvbRoK4tHeoOQLAKRFjv7b2mVbO+Xp6wnqTDhHVJx3mvOyTNlefh0spWGWMutNZuOFJDxphl1SwlH+mzAAAE2nfrd+qxGalKyd7nV09sF6O7zuyj0/vE8fZRAIcUiLBe8UaG3GrWK+pH2i6I816vkeeh0rMkfS+pvaT7JV0uaboxZoC1tqT27QIAUD9St+3T4zNSNS9tp1+9bbNI3Xp6ki47rqvCw1wOdQcgFATTnPWKP63CJF1mrf3B++t9xpjfybMrPlzSRZL+c7gbWWuHHaru3XEfGph2AQA4tG25RXrmq3WaumyLKj07qugIl649OUHXjklUs6hg+hEMIFgF4k+Kip3z6t55XFHfW826DlrfVimoS5KstdYYM02esD5CRwjrAAA4Ib+4TK/OS9fr32WoqPTAY1YuI10yrKv+NC5J7XmpEYCjEIiwvs57TapmvZf3Wt2Z9oPvU12or3ixUpMa9gUAQL0oLXfrwyVZev7rNO3K9z+pOSapnf46MVnJHZo71B2AUBaIsD7Hex1njHFVnthijImVNEpSoaRFR7jPIkkFknoYY2IOMeaxv/e6MQA9AwBQZ9ZafZu6Q4/NSFH6Tv8fW307NtfdE/vw5lEAdVLnsG6tTTfGzJZn4suNkl6stPyQpBhJr1YO38aYZO9nUyvdp9AY84akmyU9Yoz5k7XWer9+gKSrJJVJmlrXngEAqKs1W3P16PQULUz3f0F3xxbR+su43rpgSGe5XEx4AVA3gXq65QZJCyW9YIw5TVKKpJHyzGBPk3TPQV+f4r0e/KfYffKMbLxV0gneGe3tJV0oKVrSrdba9AD1DADAUduWW6SnZ6/Tx8u3yFZ6eLRZVLiuPyVR15wUr+iIMOcaBNCgBCSse3fXh0t6WNIESRMlZUt6XtJD1to9h/t8pfvsM8acLOmvki6RdJM8bzT9XtLT1trZgegXAICjVVBcplfnZ+j1+RnaX1ruq4e5jH49oqtuPT1JbZtFOdghgIYoYHOjrLVZkibV8Gur/e+C1tp8eXbiD96NBwCg3pW7rT5etkVPz16nHXnFfmunJsfp7onJ6hkX61B3ABo6hrwCAFCN79fv0iPT1yp1W55fvU/H5rr3rD4a1ZOHRwEcW4R1AAAOsn57nh6bkaI56/zfPBoXG6W/jO+ti4Z2URgPjwKoB4R1AAC8duUX69mv0vThkiyVV3r1aJOIMF03JkHXjk5Q00h+dAKoP/yJAwBo9IpKy/XG9xv1z7npyi8u89WNkS4e2kV/Gd+bN48CcARhHQDQaLndVl+s3KonZ67TL3v3+62N6tlGd0/so36dWjjUHQAQ1gEAjdSSzN165H9rtWJLrl+9Z1wz3T0xWWN7x8kYzqUDcBZhHQDQqGzcVaAnvkzVzDXb/OptYiJ16xlJ+vVxXRUe5nKoOwDwR1gHADQK2/cV6YVv1uujJVkqq/TwaGS4S9ecFK/rT0lU8+gIBzsEgKoI6wCABi23sFSvzE/XWws2qqjU7bd23uBOun18b3Vp1dSh7gDg8AjrAIAGaX9Jud5emKl/zt2gfUVlfmvHJ7TWXWf20eCuLR3qDgBqhrAOAGhQSsvdmrI0S89/vV478or91vp1aq47JyTr5F5teXgUQEggrAMAGgS322r6qmz9ffY6ZeYU+q31aNNUfxnfWxP7d5SLN48CCCGEdQBASLPWav76XXpyZqrWbN3ntxYXG6VbTu+lXw3vqggmvAAIQYR1AEDIWr55j56cmapFGbv96s2jw3X9KT111Yk91CQyzKHuAKDuCOsAgJCzfnuenpq1TrPXbverR0e4NGlUvCaPTlSLpoxhBBD6COsAgJDxy979eu6rNH28fIsqjUpXmMvosuO66ubTeql982jnGgSAACOsAwCCXk5+sV6em65//7BJJeX+s9LPGdRJfzojSfFtYxzqDgCOHcI6ACBoFRSX6V/fbdTr32Uov9h/VvqYpHa6fXxv9e/cwqHuAODYI6wDAIJOWblbU5Zu0TNfpWlXvv+s9MFdW+rOCck6IbGNQ90BQP0hrAMAgoa1VnPX7dRjM1K0fke+31rPuGa6fXxvjevbnhcaAWg0COsAgKCwZmuuHpuRogUbcvzqHZpH60/jknTR0C4K44VGABoZwjoAwFHZufv19Kw0ffLTFtlKE15iIsN0/SmJuuakBGalA2i0COsAAEfkF5fplbnp+tf3GSoqPTDhpWIM462nJ6ldbJSDHQKA8wjrAIB6VVbu1odLsvTc12nalV/it3ZacpzuOjNZvdrHOtQdAAQXwjoAoF5Ya/Vt6g49/mWqNhz08Gi/Ts11z8Q+OrFnW4e6A4DgRFgHABxzq3/J1aPTU/RDhv/Dox1bROv28b11/uDOcvHwKABUQVgHABwzW/fu19Oz1umTn37xqzeLCvc+PBqv6AgeHgWA6hDWAQABl1dUqn/OTdcb329UcZn/w6O/GdFNt5zeS22b8fAoABwJYR0AEDCl5W59uHiznvt6vXIK/B8ePb1Pe911ZrJ6xjVzqDsACD2EdQBAQMxP26kHv1ijjJ0FfvUBnVvo7ol9dEJiG4c6A4DQRVgHANTJnoIS/d/0tfpkuf+59M4tm+j28b117qBOPDwKALVEWAcA1Iq1VjNWbdMDn6/2m5ceGxWuG8b21KRRPXh4FADqiLAOADhq2/cV6b7PVmv22u1+9bMHdtQD5/TjzaMAECCEdQBAjVlrNWVplh6ZnqK8ojJfvX3zKD1y/gCd0be9g90BQMNDWAcA1MjmnELd9clKLUz3f7HRb0Z2011nJqt5dIRDnQFAw0VYBwAcVrnb6q0FG/X07HUqKj0wM717m6b624UDmfICAMcQYR0AUK112/J0x8crtSJrr6/mMtIfTk7QracnqUkkD5ACwLFEWAcAVFFS5tZLczbo5bkbVFpuffXkDrF68uKBGtilpYPdAUDjQVgHAPj5afMe3fnxSqVtz/fVIsNc+uOpPXXdmERFhrsc7A4AGhfCOgBAklRYUqa/z07Tmws2yh7YTNfQbi31xEUD1at9rHPNAUAjRVgHAGjBhl2665OVytq931drEhGmOyb01u9O6KEw3kAKAI4grANAI5a7v1SPTU/RR0uz/Oon92qrxy4YoK6tmzrUGQBAIqwDQKM1a8023ffZau3IK/bVWjSJ0H1n99VFQzvLGHbTAcBphHUAaGR27CvSg1+s0YxV2/zqEwd00IPn9lNcbLRDnQEADkZYB4BGwu22+mDxZj0xM1V5RWW+ervYKP3fef01oX8HB7sDABwKYR0AGoG07Xn66yertGzTHr/6r4Z30T0T+6pF0wiHOgMAHE7AwroxpoukhyVNkNRGUrakzyQ9ZK3dc7jPVrrHXEljDvMlTay1RXVsFQAajaLScr08Z4P+OS/d7+VG8W1j9OgF/XViYlsHuwMAHElAwroxJlHSQklxkg1ZePgAACAASURBVKZJSpU0QtItkiYYY0ZZa3OO4pYPVVMvq6YOADjID+k5uufTVcrYVeCrRYQZXT8mUTeM7anoiDAHuwMA1ESgdtZflieo32ytfbGiaIx5RtJtkh6VNLmmN7PWPhigvgCg0dlbWKLHZqRoytItfvVh3Vvp8QsHKImXGwFAyKhzWPfuqo+TlCnppYOWH5B0raQrjDF/ttYWCABwTFhr9fmKrfq//63VrvwSXz02Klx3npms34zoJhcvNwKAkBKInfWx3utsa6278oK1Ns8Ys0CeMH+8pG9qckNjzKWS4iWVSEqR9K21tvjwn/L7/LJqlpJreg8ACCVZuwt172erNS9tp1/9zP6ecYztmzOOEQBCUSDCem/vNa2a9fXyhPUk1TCsS/rwoF/vMMbcaK2dWov+AKDBKit3680FG/XsV+u1v7TcV+/QPFoPn9dP4/oxjhEAQlkgwnoL7zW3mvWKessa3GuapKcl/SQpR1J3SVdK+rOkj4wxZ1lrZx7pJtbaYYeqe3fch9agDwAIeiu37NVfP1mlNVv3+WrGSFee0EN/Hpek2GjGMQJAqAuqOevW2mcPKq2TdLcxZqukFyU9LumIYR0AGrKC4jI981Wa3lqwUe4D0xiV3CFWj184QEO6tXKuOQBAQAUirFfsnLeoZr2ivrcO3+Nfkp6VNNgYE2utzavDvQAgZM1J3aF7P1utX/bu99Wiwl269fQk/f7keEWEuRzsDgAQaIEI6+u816Rq1nt5r9WdaT8ia22RMSZPUitJMZII6wAalR15RXr4i7X638psv/pJPdvq0Qv6q3ubGIc6AwAcS4EI63O813HGGFfliTDGmFhJoyQVSlpU229gjOktT1DPk7SrDr0CQEgpK3frP4s366lZ67Sv6MB74VrHROres/rogiGdZQzjGAGgoapzWLfWphtjZssz8eVGec6WV3hInp3wVyvPWDfGJHs/m1qpFi8p11q7u/L9jTHtJL3l/eWH1lreYgqgUZiftlOPTF+rtO35fvWLhnbRPWf1UeuYSIc6AwDUl0A9YHqDpIWSXjDGnCbPbPSR8sxgT5N0z0Ffn+K9Vt4OGiPpFWPM95IyJO2W1E3SRHnOvS+VdEeA+gWAoJW+M1+PTU/RN6k7/Ord2zTVYxcM0KiebR3qDABQ3wIS1r2768MlPSxpgjwBO1vS85IestbuqcFtlskzX32YpCGSmstz7GWVpCny7M6XVP9xAAhtuYWlev6b9Xr3h0yVVRrzEhMZphtP7amrR8UrOiLMuQYBAPUuYKMbrbVZkibV8GurHLC01q6SdFWg+gGAUFFW7tYHizfrma/StLew1Fc3RvrVsK768/gkxcXyBlIAaIyCas46ADQ289J26pH/rdX6Hf7n0kfEt9b9Z/dV/87VTcUFADQGhHUAcMCGHfl6dPpazVm306/etXUT3X1mH03o34EpLwAAwjoA1Ke9hSV67uv1em/Rpirn0m86tZcmjerBuXQAgA9hHQDqQWm5W+8v2qRnv16v3P3+59IvHd5VfxrHuXQAQFWEdQA4xuau26FHpqdow0Hn0kfGt9b95/RVv06cSwcAHBphHQCOkQ078vTI9BTNPehcerfWTXX3xD4a368959IBAIdFWAeAANtTUKLnv1mvfy/apPJK59KbRYXrplN7atKoHooK51w6AODICOsAECDWWk1ZmqXHZqRWOZd+2XHd9KczktQuNsrBDgEAoYawDgABkJNfrDs/XqWvU7b71Y9PaK37z+6nvp2aO9QZACCUEdYBoI6+Td2uO6au0q78Yl+tW+umuuesPhrXl3PpAIDaI6wDQC0VlpTp0ekpev/HzX71q07sobvOTGZeOgCgzgjrAFALK7L26raPflbGrgJfrV1slJ6+ZJDGJLVzsDMAQENCWAeAo1BW7tbLc9P1wjfr/d5AOqFfBz124QC1jol0sDsAQENDWAeAGtqUU6DbPvpZyzfv9dViIsP04Ln9dPGwLpxNBwAEHGEdAI7AWqv/Lt2ih75Yo4KScl99ePdWeuZXg9WtTVMHuwMANGSEdQA4jN0FJbrr45WavfbASMZwl9FtZyRp8phEhbnYTQcAHDuEdQCoxpx1O3TH1JXamXdgJGNCuxg9f+kQDejSwsHOAACNBWEdAA6yv6Rcj81I0b8XbfKr/+6E7vrrmX3UJJKRjACA+kFYB4BKVm3J1S0f/aSMnQdGMrZtFqWnLh6osclxDnYGAGiMCOsAIKncbfXPuRv03Nf+IxnH9W2vxy8coDbNohzsDgDQWBHWATR6WbsLddtHP2vppj2+WkxkmB44p58uGc5IRgCAcwjrABota62mLtuiBz/3H8k4tFtLPXvpYHVvE+NgdwAAENYBNFLb9xXp3s9W66tKIxnDXEa3nNZLN5ySqPAwl4PdAQDgQVgH0KhYa/XRkiw9OiNFeUVlvnpC2xg9e+lgDera0sHuAADwR1gH0GhsyinQXR+v0g8ZOX71y4/vprsn9lHTSP5IBAAEF34yAWjwyt1Wb36/UX//ap2KSt2+eo82TfW3iwbq+IQ2DnYHAED1COsAGrTUbft059SVWrEl11dzGekPoxN02+lJio7gBUcAgOBFWAfQIBWXleulOel6ec4Gv7npyR1i9eTFAzWwC2fTAQDBj7AOoMFZvnmP7py6Uut35PtqkWEu3XxaT103JlERTHoBAIQIwjqABqOwpExPz0rTWws3yh7YTNew7q30xEUD1DMu1rnmAACoBcI6gAbh+/W7dNcnK7Vlz35frWlkmO6ckKwrju8ul4u3kAIAQg9hHUBIyy0s1aMz1mrK0i1+9dFJ7fTYBf3VpVVThzoDAKDuCOsAQtbM1dm6b9oa7cwr9tVaNInQ/Wf31YVDO8sYdtMBAKGNsA4g5OzIK9ID09boy9Xb/OpnDeioB8/tp3axUQ51BgBAYBHWAYQMa62mLtuiR6anKHd/qa/eLjZK/3def03o38HB7gAACDzCOoCQkLW7UHd/ukrfrd/lV790eFfdPbGPWjSNcKgzAACOHcI6gKDmdlu9v3izHp+RosKScl+9a+sm+tuFAzWqZ1sHuwMA4NgirAMIWlm7C3Xnxyu1MD3HV3MZ6epR8frTuCQ1jeSPMABAw8ZPOgBBx+22ev/HTXr8y1S/3fTEdjF66pJBGtqtlYPdAQBQfwjrAIJK1u5C3TF1pX7I8N9Nv3Z0om49vZeiI8Ic7A4AgPpFWAcQFNxuq/d+3KS/HbSb3jOumZ6+ZJAGd23pYHcAADiDsA7AcZtzCnXHxyu0KGO3r+Yy0nVjEnXLaeymAwAaL8I6AMe43Vb/XuTZTd9femA3vVdcMz3FbjoAAIR1AM7YlFOgO6au1I8b/XfTJ49J1M3spgMAIImwDqCeud1W7/6QqSdmrvPbTU9q30xPXTxIg9hNBwDAxxWoGxljuhhj3jTGbDXGFBtjMo0xzxljaj1jzRgz2hhTboyxxphHAtUrAGdk7irQZa8v0oNfrPUF9TCX0Y1jE/XFH08iqAMAcJCA7KwbYxIlLZQUJ2mapFRJIyTdImmCMWaUtTbnMLc41D1jJb0jqVBSs0D0CcAZbrfVOz9k6omZqSoqdfvqSe09k14GdiGkAwBwKIE6BvOyPEH9ZmvtixVFY8wzkm6T9KikyUd5z+cltZD0uPfzAEJQ5i7P2fTFmQfOpoe5jK4fk6g/ntZTUeGcTQcAoDp1DuveXfVxkjIlvXTQ8gOSrpV0hTHmz9baghre8zxJkyRdEYgeAdQ/t9vq7YWZenKW/2567/axevqSQRrQpYWD3QEAEBoCEYTHeq+zrbXuygvW2jxjzAJ5wvzxkr450s2MMXGSXpf0mbX2PWPMVQHoEUA9ytxVoNunrtCSzD2+WpjL6IZTEnXTqeymAwBQU4EI672917Rq1tfLE9aTVIOwLk9Qd+noj834GGOWVbOUXNt7Ajgya63e/3GzHp2e4jfpJbmDZze9f2d20wEAOBqBCOsVP31zq1mvqB/xCTJjzNWSzpV0qbV2ewB6A1BPduwr0h0fr9TcdTt9tTCX0Y2nJOqmU3spMjxgw6cAAGg0guY8uDGmh6TnJP3XWjulLvey1g6r5nsskzS0LvcGUNX0ldm657NV2ltY6qv1imumZy8dzG46AAB1EIiwXrFzXt1P5Ir63iPc501J+yXdEICeANSD3MJSPfD5an3281a/+jUnxev28b15CykAAHUUiLC+zntNqma9l/da3Zn2CkPlCfY7jTGHWr/HGHOPpGnW2vOPuksAAfX9+l36y39XaNu+Il+tc8smeuqSgToxsa2DnQEA0HAEIqzP8V7HGWNclSfCeF9sNEqeFxstOsJ93pXU9BD1XpJGS/pZ0jJJP9W5YwC1tr+kXE/MTNXbCzP96hcN7aIHzu2r5tERzjQGAEADVOewbq1NN8bMlmfiy42SXqy0/JCkGEmvVp6xboxJ9n42tdJ9bj7U/b2jG0dLmm6tvbeu/QKovRVZe3XblJ+VsfPAKxNax0TqsQsGaEL/Dg52BgBAwxSoB0xvkLRQ0gvGmNMkpUgaKc8M9jRJ9xz09Sne6yHPuwAILqXlbr00Z4Ne/HaDyt3WVz8tOU6PXzRAcbHRDnYHAEDDFZCw7t1dHy7pYUkTJE2UlC3peUkPWWv3HO7zAILXhh35+vOUn7Viy4HprDGRYbrv7L669LiuquYZEwAAEAABG91orc2SNKmGX1vjn+7W2rclvV27rgDUlttt9e4PmXr8y1QVlx14OfFxPVrp75cMVrc2h3rEBAAABFLQzFkHEDyyc/fr9v+u1PcbdvlqkWEu/Wlckv5wcoLCXOymAwBQHwjrAHystfp8xVbd99lq7Ssq89WTO8Tq2UsHq0/H5g52BwBA40NYByBJ2lNQonunrdb0ldm+mjHSdaMTddsZvRQVzguOAACob4R1AJq7bofumLpSO/KKfbWurZvomV8N1nE9WjvYGQAAjRthHWjESsvdemxGit5akOlXv+y4rrr37L5qFsUfEQAAOImfxEAjtSOvSDe9/5MWZ+721do2i9TfLhyo0/u2d7AzAABQgbAONELLNu3W9e8t9zv2cnqf9nriogFq0yzKwc4AAEBlhHWgEbHW6r1Fm/Tw/9aqtNzzJlKXkW4fn6zJYxJ4wREAAEGGsA40EkWl5br701X6ZPkvvlqrphF64ddDdHKvdg52BgAAqkNYBxqBrN2FmvzeMq3Zus9X69+5uV65fJi6tOJNpAAABCvCOtDAzU/bqZs//El7C0t9tYuHddEj5/dXdASz0wEACGaEdaCBcrut/jkvXU/PXifrOZ6uiDCjB87pp9+O7Mb5dAAAQgBhHWiA9hWV6s9TVuirtdt9tfbNo/TPy4dpaLdWDnYGAACOBmEdaGDWb8/Tdf9epoxdBb7aiPjWeuk3Q9UulrGMAACEEsI60IBMX5mt26euUGFJua92zUnxuuvMZEWEuRzsDAAA1AZhHWgAysrdenLWOr02P8NXaxIRpr9dNEDnDe7sYGcAAKAuCOtAiMvJL9ZNH/ykHzJyfLXubZrq1SuGKblDcwc7AwAAdUVYB0LYiqy9uv69ZdqaW+SrnZocp2cvHawWTSIc7AwAAAQCYR0IUR8u3qz7p61RSblbkmSMdOtpSfrjqT3lcjGWEQCAhoCwDoSY4rJyPfj5Gv1ncZav1jw6XM9dNlinJrd3sDMAABBohHUghGzOKdQf/7NcK7bk+mrJHWL16hXD1L1NjIOdAQCAY4GwDoSA3QUlevHb9Xpv0SaVlltf/bzBnfT4hQPUNJLfygAANET8hAeC2P6Scr25YKNemZuuvOIyXz3MZXTPxD6aNKqHjOF8OgAADRVhHQhC5W6rqcuy9OxX67VtX5Hf2tBuLXXf2X01pFsrh7oDAAD1hbAOBBFrrb5N3aEnZqYqbXu+31pCuxjdMT5Z4/u1ZzcdAIBGgrAOBImfNu/R41+mavHG3X71ts2idNsZvXTp8K4KD3M51B0AAHACYR1wWOauAj01a52mr8r2q8dEhuna0Yn6/cnxionityoAAI0RCQBwyK78Yr3wzXp98ONmlbkPTHgJdxn9ZmQ3/fHUXmoXG+VghwAAwGmEdaCeFZaU6V/fbdSr89JVUFLutzZxQAfdPj5Z8W2ZmQ4AAAjrQL0pK3fro6VZeu7r9dqZV+y3NiK+tf56ZjITXgAAgB/COnCMWWs1e+12PTkzVek7C/zWesU1011nJuvU5DgmvAAAgCoI68AxtGzTbj02I1XLNu3xq7dvHqU/nZGki4Z2YcILAACoFmEdOAb2Fpbo4S/W6pOffvGrx0aFa/Ipibp6VLyaRIY51B0AAAgVhHUgwL5au113f7rK71x6RJjRFcf30E2n9lTrmEgHuwMAAKGEsA4EyN7CEj34+Rp99vNWv/pZAzvqzvHJ6tamqUOdAQCAUEVYBwJg9pptuvvT1dqVf2A3vV1slB67YIDO6Nvewc4AAEAoI6wDdbCnoEQPfrFG0w7aTb9gSGc9cE5ftWzKkRcAAFB7hHWglmat2aZ72E0HAADHEGEdOEp7Ckr0wOdr9PkK/930C4d01v3spgMAgAAirANHYebqbbr3s1XalV/iq8V5d9NPZzcdAAAEGGEdqIHd3t30Lw7eTR/aWQ+c3U8tmkY41BkAAGjICOvAEcxcna17P1tdZTf98QsH6LQ+7KYDAIBjh7AOVGN3QYnun7Za/1uZ7Ve/aGgX3X92X3bTAQDAMUdYBw7hy1We3fScggO76e2be3bTT01mNx0AANQPwjpQSU5+se7/fI2mH7SbfvGwLrrv7L5q0YTddAAAUH8I64DXjFXZuu8Qu+l/u3CgxibHOdgZAABorAIW1o0xXSQ9LGmCpDaSsiV9Jukha+2eGt7jdkljJfWV1FaSW9ImSV9JesZauyVQ/QIVduYV68HP12j6Kv/d9EuGddG97KYDAAAHBSSsG2MSJS2UFCdpmqRUSSMk3SJpgjFmlLU2pwa3uk5SvqR5krZLipA0RNJtkq4xxpxirf0pED0D1lpN+3mrHvxijfYWlvrqHZpH6/GLBmhsb3bTAQCAswK1s/6yPEH9ZmvtixVFY8wz8gTtRyVNrsF9+ltriw4uGmP+IOk1730mBqRjNGrbcot0z6er9E3qDr86u+kAACCYuOp6A++u+jhJmZJeOmj5AUkFkq4wxsQc6V6HCupeU7zXXrVsE5Dk2U3/cPFmnfHMPL+g3rllE7179Qg9dckggjoAAAgagdhZH+u9zrbWuisvWGvzjDEL5Anzx0v6ppbf4xzvdWUtPw8oa3eh/vrJKn2/YZdf/XcndNcdE5LVLIrnrQEAQHAJRDrp7b2mVbO+Xp6wnqQahnVjzO8ldZHUTNIASafL86DpXTX8/LJqlpJr8nk0LG631bs/ZOrJWetUWFLuq/do01RPXDRQIxPaONccAADAYQQirLfwXnOrWa+otzyKe/5e0shKv14i6TfW2g1H2RsauYyd+brz45VaknlgIJHLSL8/OUG3nZ6kJpFhDnYHAABweEH53/2ttcdLkjGmjaSh8jxYuswY8ytr7awafH7YoereHfehgewVwams3K03vt+oZ75KU3HZgdNZveKa6cmLB2pIt1YOdgcAAFAzgQjrFTvnLapZr6jvPdobe8c9fmWMWSLPOMh/G2O6W2v3H32baCxSt+3THVNXauWWA/+xJ9xldP0pibrp1J6KCmc3HQAAhIZAhPV13mtSNesVE1yqO9N+RNbavcaYHySdL6mfpKW1vRcarpIyt/45N13/mLNepeXWV+/XqbmevHig+nWq7u+TAAAAwSkQYX2O9zrOGOOqPBHGGBMraZSkQkmL6vh9OnuvZXW8DxqgVVtydfvUFUrdluerRYa5dMvpvXTt6ARFhNV5SikAAEC9q3NYt9amG2NmyzPx5UZJL1ZafkhSjKRXrbUFFUVjTLL3s6mVat0kFVtrtx/8PYwx10k6TlKWpFV17RkNR1FpuZ7/Zr1em5+hcveB3fQh3VrqyYsGqlf7WAe7AwAAqJtAPWB6g6SFkl4wxpwmKUWeaS5j5Tn+cs9BX5/ivZpKtaGS/us97rJB0nZJbeSZzz5AUr6kK6y15QIkLdu0W7dPXamMnb6/Byo6wqW/jOutSaPiFeYyh/k0AABA8AtIWPfurg+X9LCkCZImSsqW9Lykh6y1ew73ea/l3q8/WdJZklpLKpKUIenvkp631mYFol+Etv0l5XpyVqreXpgpe2AzXSPjW+uJiwaqR9sjviwXAAAgJARsdKM3SE+q4ddW2fK01m6W9JdA9YOGafu+Il399hKt2brPV4uJDNNfJ/bRb0Z0k4vddAAA0IAE5Zx14FBSsvfp6reXKDu3yFcbndROj184QJ1bNnGwMwAAgGODsI6QMC9tp258f7nyiz3DgMJcRg+e20+Xj+wmY9hNBwAADRNhHUHvgx83675pq33TXppFhevl3w7V6KR2DncGAABwbBHWEbTcbqsnZqXq1XkZvlqnFtF6c9JxSu7Q3MHOAAAA6gdhHUGpqLRcf56yQtNXZftq/Ts31xtXHqf2zaMd7AwAAKD+ENYRdHLyi/WHd5dq+ea9vtrpfeL0/GVDFBPFv7IAAKDxIPkgqKTvzNekt5Zo8+5CX+2qE3vovrP78pIjAADQ6BDWETR+zMjRtf9eptz9pZIkY6T7zuqrq0+Kd7gzAAAAZxDWERQ+++kX3TF1pUrK3ZKk6AiXnr9siMb36+BwZwAAAM4hrMNR1lq9+O0GPfNVmq/WtlmU3rhyuAZ1belgZwAAAM4jrMMxJWVu3f3pKk1dtsVX6xXXTG9edZy6tm7qYGcAAADBgbAOR+TuL9X17y3TwvQcX21UzzZ6+bfD1KJJhIOdAQAABA/COupd1u5CTXp7iTbsyPfVLhnWRY9eMECR4S4HOwMAAAguhHXUq5+z9ur37yzRrvwSX+0v45J049ieMobRjAAAAJUR1lFvZq7epls/+klFpZ6JL5FhLj11yUCdN7izw50BAAAEJ8I6jjlrrd74fqMenZEiaz21lk0j9NoVwzUivrWzzQEAAAQxwjqOqbJytx7+31q9+8MmX617m6Z666rjlNCumYOdAQAABD/COo6Z/SXl+uN/luvrlB2+2rDurfT674ardUykg50BAACEBsI6jomc/GJd885S/Zy111c7e2BHPX3JIEVHhDnYGQAAQOggrCPgNucU6sq3FmvjrgJf7boxCbpzfLJcLia+AAAA1BRhHQG1akuuJr292Dea0RjpwXP66coTezjbGAAAQAgirCNg5q7boRveX67CknJJUmS4Sy9cNlgT+nd0uDMAAIDQRFhHQPx3aZb++skqlbk9sxlbNInQv64cruN6MJoRAACgtgjrqBNrrV6as0FPz07z1Tq3bKJ3rj5OPeNiHewMAAAg9BHWUWtl5W7d//kaffDjZl8tuUOs3rl6hNo3j3awMwAAgIaBsI5a8cxQ/0lfp2z31Ub1bKN/Xj5MzaMjHOwMAACg4SCs46jtLijRNe8s0U+bD8xQP29wJz118SBFhrsc7AwAAKBhIazjqGTtLtSVby5WRuUZ6qMTdOcEZqgDAAAEGmEdNbb6l1xd9dYS7covluSZoX7/2X01aVS8w50BAAA0TIR11Mj8tJ26/r1lKqg0Q/25Swdr4gBmqAMAABwrhHUc0cfLtujOj1f6Zqg3jw7X678brpEJbRzuDAAAoGEjrKNa1lq9PDddT81a56t1ahGtt68eoaT2zFAHAAA41gjrOKRyt9WDn6/Rvxdt8tWSO8Tq7Ukj1KEFM9QBAADqA2EdVRSVluvm//yk2WsPzFA/IaGNXv0dM9QBAADqE2EdfvYUlOj37y7Vsk17fLVzBnXS05cMVFR4mIOdAQAAND6Edfhk7S7UlW8tVsbOAzPUrx2doLuYoQ4AAOAIwjokSSnZ+3Tlm4u1I+/ADPV7z+qra05ihjqA/2/vzsPsqsp8j3/fqkpVUnPmmKQyTwyCJpiEQSBwpUHRVpDb9kVERXFCcaDv9el2wm7u1X7URlFb7+NFENvm3rYbaAURDRGQKZKADBkJZCJJFRlqTGpe9493nzqnTp1TqaqcqVK/z/OsZ5+z9j77rNTOqnr32msQEZF8UbAuPLnjENf/7BlaOroBKC0u4jt/dSaXnzEzzyUTERERGdsUrI9xD7ywn8/e/RydPb0AVJWV8L8/cBZnL9Qc6iIiIiL5pmB9DLvryZ185T9fIvhaR0yrKuOOD63k1JnVeS2XiIiIiDgF62NQCIHv/G4btz38cl/egikV3PnhldRNKs9jyUREREQkkYL1Maa7p5cv3fsid/9pT1/emXW13H7tWUyuLMtjyUREREQkmYL1MeRYZw+f/tdn+f3m+GJHFy6dyg+vXk55qf4riIiIiBQaRWhjROPRTq67s/9iR1csn8U3rzyDccVFeSyZiIiIiKSjYH0M2Nd4jGtvX8/2hta+vI9d4IsdmWmxIxEREZFCpWD9JLetvoVrb1/P/qb2vrwvX67FjkRERERGg4z1fzCz2WZ2u5ntM7MOM9tpZrea2cQhfr7CzK42s1+Y2RYzazOzFjN7xsy+YGalmSrrWPHMzsNc9aMn+wL1ccXGd9/3JgXqIiIiIqNERlrWzWwh8AQwDbgP2AKsBG4ELjWzc0MIh45zmrcCPwcOA+uAe4GJwLuAbwFXmNnFIYT29KeQmN9tqueGX2yko9sXO6ooLeZH16zgrYun5rlkIiIiIjJUmeoG80M8UP9MCOG2WKaZfQf4HHAL8PHjnOMA8H7g30IInQnnuAn4A3AO8Cng2xkq80nr7vW7+dt7XqA3WuxoSmUpP/3gSt44uya/BRMRERGRYTnhbjBRq/olwE7gB0m7vwq0AdeYWcVg5wkhPBdC+JfEQD3KbyEeoF94ouU9mYUQuG3tdr74H/FARtN05AAAHchJREFUfc6kcv79E+coUBcREREZhTLRsr4m2j4UQuhN3BFCaDGzx/FgfjWwdoTf0RVtu4dysJltSLNr2Qi/v+D19Aa+9p8vcddTu/ryTp9VzU8/uJKpVVrsSERERGQ0ysQA06XRdlua/duj7ZIT+I4PR9sHT+AcJ632rh5u+MXGfoH6eYumcPf1ZytQFxERERnFMtGyHutf0ZRmfyy/diQnN7MbgEuB54Dbh/KZEMKKNOfaACwfSTkKVXN7Fx+98xmefvVwX967zpzJt646k9ISLXYkIiIiMpoV9DzrZnYFcCs++PTKEELXcT4yptQ3t3Pt7evZcqClL+/D587nS+84haIiLXYkIiIiMtplIliPtZynG8EYy28czknN7N3A3UADsCaE8MrIindy2nmwjat/8jSvNR7ry/viZcv42PkLtCqpiIiIyEkiE8H61mibrk/64mibrk/7AGZ2FfALvEX9ohDC9uN8ZEwJIXDj3c/2BerFRcY/XnkGV66YneeSiYiIiEgmZSJYXxdtLzGzosQZYcysCjgXOAo8NZSTmdnVwJ3Aa6hFPaV1Wxv4815/oFFaXMSPP7CCNUun5blUIiIiIpJpJzwCMYSwA3gImIcvWpToZqACuCuE0BbLNLNlZjZgGkUzuxb4GbAbOF+B+kAhBG79ffxBw9Wr5yhQFxERETlJZWqA6SeBJ4DvmdnFwGZgFT4H+zbg75KO3xxt+zpXm9kafLaXIry1/kMp+l43hhBuzVCZR6V1Wxt4PmpVLysp4hMXLMxziUREREQkWzISrIcQdpjZWcDX8WkW3w7sB74L3BxCODKE08wl3tL/4TTH7MJnhxmTklvV/9uqOUyrHp/HEomIiIhINmVs6sYQwh7gQ0M8dkCTeQjhDuCOTJXnZKRWdREREZGxRavmjBJqVRcREREZewp6USSJU6t65PArsPEu2PYg9HTBuPEwrhzGTYCSCb5NTH155fFjSxI+k/j5ceVQPgWKdA8rIiIihUHB+igw5lvVuztg869g453w6qPZ/a6S8TB5MUxZDFOWxLeTF0FpeXa/W0RERCSJgvVRIGet6h2tsHc9lFXDzDdDUXF2vmeoGrZ4gP7nf4VjQxmjnAHd7VD/gqdkNXUDg/gpS6ByOmjVWBEREckCBesFLiet6p1H4U8/gcdvhaOHPG98DSy4EBZeBAsvhtq6zH5n2rK0wUv3epC+5+mB+60IFr0Nln/AA+auo9DVHm2PebA9IO+Yb/vy0+R1NEN7Y/qyNe3xtOPh/vll1amD+InzoaQ0sz+fTAsBjh6G5tdg4jwYX53vEomIiEgCBesF7uEtWWxV72qHDT+Fx74DbQ3997U3wab7PIF3DVl4ESy6GOaeC2WVmSsHwL7nPEB/4ZceNCermQPLr4E3XQ01szL73YmOHoZDL8PBbVHa7tvDr0LoSf2ZjmZ4bYOnRFYMk+Z70F47BybOhdq50et5MGFi7lrkO1rh8A7/tx2KbaPXsRuUonEw/3xY9g5Y+naofkNuyiYiIiJpKVgvYANWK101NzOt6t0dsPFn8Ni3oWV//33VsyD0Dsw/tN3T+h97UDdnddTqfhHMOGNkgzLbm+CFf4MNd8KB5wfuLxoHy94Oy6+FBWtyM/CzfBKUr4S6lf3zuzvhyKsDg/iD21PfXIAH97GgOJXSqoQgfo4H8omvh9vK3d0BR3b2D8RjgXnrgeN/vrcLdqz1dP/nYdZZHrgvuxymLhleWURERCQjFKwXsIe3NPDCa/FW9Y9fsODETtjTBc/+HB79FjTv7b+vaiacfxO8+RooHgevb4GX13qXj12Pe/eSmN4u2PmYp7U3+wwqC9fEg/eqGenLEIJ3b9lwJ7x0j3dHSTZ5kQfoZ/41VE49sX9zppSUwtSlnhKFAK31qYP4pj2Dn7OzBRpe8pTK+NqBrfG1c6B6pn/noVcSAvOX/ftC7/D/beMq/Od8ZGf//Nee8bT2Zn+yEgvcZ63QjDkiIiI5YiGEfJchZ8xsw/Lly5dv2LDh+AfnWQiBd33/8b5g/cPnzucr7zx1ZCfr6Ybn74ZH/hEad/XfVzkd3voFD47HpWm17zoGu5/0wP3lh9MHlzHTTvPgfdHFMOdsnxax7ZAPFN34Mzi4deBnSsbDqe/2vuhzzzk5Bmx2tnnLduMuOLILGnf768bd/r6rLXdlKRoHkxbA5IVRWuRp0kK/uTKDxj2w9Tew5dew84/pu/1UTvduMssuh/lvhZKy3P07RERERokVK1awcePGjSGEFSdyHgXrBWrt5nquu/MZwFvVH/vva4bfBaa3x/uAP/INn588UfkUOO9z8JbrPJgejpYDsGOdB+87HoajB9MfWzLeu8nsfw56Ogfun3663yiccZX34R4rQvDBvOkC+cbd0NMxzJOaDwRODMQnL/LgvKYOiofxIO3oYdj+Ow/cX/69D8RNpbQKFr/NW90Xv80HJouIiEjGgnV1gylAJ9xXvbcXNt0Df/iGd8lINGESnHsjrPwolFaMrIBVM+BNf+2pt9enOYx1mdn9lHeTielu9+kgE5VWwulXwoprYebyk6MVfbjMoGKKp1kp6nBvrw/67Qved8Zft+z3m63EFvLJi7ybTLqnI8NVPgnO/CtPXcfglUc8cN/6m/43Z50t8NJ/eNIA1czo7vQbt+TBwE17/brU1PlNWc2caBu9142SiMhJSS3rBWjEreq9vbDlVx6kN2zqv298DZzzaVj1cSirykKpIx2t3sc91uqeeLMw6ywP0E+7IvOzyUhu9PbAnvUeuG/59cB+7olmLveZexJ/x4QAhIS8EM9L3J/uWMz77PfryrNg+E+H8q2316fL7DcY+GWfsefIrvRdkAZTVhMNTk4I4BMD+4opo+fGuKfbx2U074OWff4UavIiv7EdaSODiEiOqWX9JDWiVvUQvMXzD/8TDiQt5lNWDas/Cas/ARNqs1DiJGWVsOQvPIH3gz7wvAdU007J/vdLdhUVw9yzPV3yD9CwGbbc74H7/uf6H7tvo6dcqJ6d9KQhel07xwdM50MI0HYwHpAnTp15+JX+g7YzoaMp/YJeACUToGb2wCC+fJLfwCem0qrhdZsajs42aN7vQXjzfr9padnvgXnzPn/dWp96sLQVw4w3Qt0qmLPKtzWzs1PO4eju9IaJxt1+gzplaeaeconImKdgvcAMawaYELw/8bpbYN+z/feVVnor+tmf8j/G+VJbl7sFlSS3zGD6qZ4u+JuhD1DNhua9nl59pH9+UYl3D+rrw58Q0Fe94fiz2vR0QUeLT8/Z0ZKUBsk7dgQO7/QAeiRq6qIBwQndnGrr/LyNe6Bpd7TdE98eL/jvPhafgnUoxpUPDOLLqpO2yflVHqS2HUoIwl+LgvPodfsIfybg/6f2P+dp/Y89r3q2T7U6Z7Vvp78xezcaIfjPun6TD7Sv3+RPMQ9ug97u+HFW7P/Xpp3q43Kmn+qva+dqJiURGTYF6wVkyK3qra/D1gfg2btg75/67yuZAKuuh3NuhIrJWS6xSILaOv+/t+p6H6C652mf+72v64VFr6P3sdep9vfLI/6Z3p6oP/cQu430dqef675kggdUE+d5EJYq+E41tWimlE9JehIQ3UhMnA+l5ek/NzdFXqwVv3F36kC+cc/wbxy6jnpqrR/e5zKlYpqPe6ie5TcDB573Jzkkdd1s3gsv7fVxE+A3GbNWRMH7Kpj9lpE9VTzW6IF4/UvRdpN//1B+jqEnPp3rpnvj+aWVMHUZTD/N07RTfZvPBhURKXgK1gvIoK3qR3bC5l97l4M9Tw18RFwyHs66Ds77LFROy12hRVIpnwRLL8vNd/UbkJk4KHOHd7VI+7ljUP+ip2wprUpozU/YTlqY2W5pZj5XfuVUmJ2ma2R7U1IQvxuaXut/k9LeHH9CkBwUZ0rROA/Cq2b6+INYqnpD/HXlDF/bINmxRtj7jP8O3PM07N0wcArUrqPxdSAAMO+CV7cS6qLW90kL4jeEsS4syYF58loUx1MzBybN85/tkZ2k/Pl1tsbXL0hUOSMK4E/1qW+nn1qYXWlC8L89PV1+I5yYMP/bM1rGRYiMIgrWC8SAVvWVc5h2dDtsiAL0dP1Qi0thxQfhvM9r9g0Zm0pKYcpiT8k6Wr1/eGJ/8dj22OHjn9uKjt/tY8C+KK92TmEFL+NrYEYNzDj9+MeG4H3Lh9rtJ/F1Z5vfrFXP6h+AV0Wt5OWTR94VZEItLP4vnsAHota/6IH7nqdh99MpguzgAXjDJthwh2dVTPUpZVv2D+zCcjzja+IB9fTT/PW0U/qvONzR6gvL9QX/0fboodTnbD3gacfaeJ4V+83d1KU+gDoWKBNt+70Px9mf+B7f9nYnBN09PotXb7f/THu74+97exKO6xpY9kTVs2HppX6jPk9rMIhkimaDKRBrN9fz0TvXs8K28fZxz/CBiS9S3LQ7zdHmiw2dcjmc9h7/Qygiw3P0cHzRqpKyNP2vywsn2JahadobBe/rfSrZAy+MbPxE0TgPlBO7q0w71X/fjuT/RAjQ2hD1dX8p3u/99a2ZH2xcCEorfWG8JZfB4kvULVPGJM0Gc7Loaie88gd67v0/rC97iinW7PnJ3SKLS2HBGg/Ql1zmj7tFZOTKJ3mqe0u+SyKZVDPb0+lX+vvONnhtQ7zlfe/6gYNca+fEW8tjgfnkRZmdScgMqqZ7WnhRPL+3x5/+1L/UvyU+XVeafLMiH7hdVOI3NEXF/nPqbPNuPjGdrbDpPk9W5F2Qll7mKdVTMBFJSy3r+dDe5KtDbv6Vz+aS+AsuUVm1t0jEVofM5vzoIiJjQW8vHNzq3VSqZg7swlIoOtugYYt34Qq9+GDrIg/6+wZiFw18bUWDHxvbXzwuCriLo6C7JMpLeF9U4jPr9AXmJem7MPV0we4nYeuDsPX+wddgmLwoCtzfDrNXZm/2npiO1vgq0a0NUTetmX79K6f5vzmfutp9pqSmPT6OpGmvLz43YaKXr3IGVEY3ehXTCm8sg6SVqZZ1Beu50lLvv8C23O+rQabp+9daMonKM98Fy94J89XnT0RERpkQvHvP1gdg24PeJSndU4IJk7xRaull3m1mJI1SXe0e6B7Z5QF5467odRSgpxsrAD42oGrGwEHOfQOgo8HQIw2Qe3v8BqFpr5exOQrGE1PiqtBDMb4mCuCnedkrpyekhLwJE8duN74QfAxNa4OvBh5buyH2xC1H1A1mtGhvhn9576C/rHb2Tue3vWexjpV873PXU1kzyLRtIiIihcwMpi3z9NbP+3TD23/r6zDseNhn7Ik5dhiev9tTcakPTI11l4kteNXT5UFtchDeuNvftx4YeVlDT7QWwGuDHzdh0sDB0rFAvnwStL3ev2U81lLevG94A5iHor3J08Gtgx9XNG5gAF8zy9dxqKnzn2/1zPwtHDcSnUej4DsKwFsTXre9HuXV+/+55Kl3a+pyHqxnioL1bBtf7QPZkgP1GWcQll3ODc/O4v76WsC47rz5TFOgLiIiJ5PKqfDm93vqaodXH423urfsjx/X0+kz4uxYCw/cBFOWQNcxD3xTrWg7VMWl0aq9czxoPXo4WkF33+Ct7omOHfaUjalerdiD/5rZUTA922csOtYYzRTUAC0H4q3EQw3+e7viC8al/e4iv/momR0P4Gtm+88q9np8TWb+ncm6O30WqdjNR+z1scakgPz1eGDe2TLy72ut9xb3Ufi0QcF6LpxyOTz+XZhzjr9e9g6oncPDm+u5v97n2y0rKeJjg61WKiIiMtqNGw9LLvEUgq9Gu/U3HrwfSJqi+OC2oZ3Tij3IrZ3raeJcDzZro+1gqxV3tfsNQ8t+D96b9/Vfebd5nwfMJ9I6Xj45asWeHQ+A+1q4Z3uL91D7zff2+k1Dy4GEluWkgD72vqP5+OcLvfEnC3ueTn1MWU0UwCcE87HW+arpfkPV3uQ9CdqbfOGwWADel9c88H3iE5ZsKJkQ9fmfFn/C0NM5KrsXK1jPhdWfhLNvgIopfVnJ86q/f/VcplVp0IiIiIwRZjDzzZ7W/K0vKLXtQQ/edz7mgZUf6AF37ZwoEJ+b8HqOt0qPtCvHuPEwab6ndHp7vItFXyC/LyGw3+ct9RVT4kFsXyt5nXczGWxF4uEqKvLvqpgCHGfNhM6jCd1C6qObj6iffOMe3w6lC1FHEzQ0+VSj+VY0rn8AXjE1ob9+wuuKqT7+YRS2oqeiYD0XUqwomrxaqVrVRURkTKutg5Uf9dTRAg2bva94bV1+W0OLokGoVTPyV4aRKC0//o1Id0d80GssgG+KVjqODYDN1joAVuRdbMqqfZuYKqb2bxGviAL0MTpoVsF6HqhVXUREZBBlVVC3Mt+lOPmVlMGkBZ5SCQHaDiYE79G2cbdv216H0oqkoLs66X1tirxqXzhrDAbeI6FgPQ/WblaruoiIiBQ4s6h7yVSYtTzfpRmz0oy4kGwJIXDr2vigGbWqi4iIiEg6CtZzbO3mBl58zUdoq1VdRERERAajYD2H1KouIiIiIsOhYD2H1KouIiIiIsOhYD1H1KouIiIiIsOlYD1H1KouIiIiIsOlYD0H1KouIiIiIiOhYD0H1KouIiIiIiOhYD3L1KouIiIiIiOlYD3L9je1c7i1E1CruoiIiIgMT0m+C3Cym1k7gXV/cyG/3LCX5mPdalUXERERkSFTsJ4DZSXFXL1qbr6LISIiIiKjjLrBiIiIiIgUKAXrIiIiIiIFSsG6iIiIiEiBUrAuIiIiIlKgFKyLiIiIiBSojAXrZjbbzG43s31m1mFmO83sVjObOIxzvM3Mvm1ma83skJkFM/tjpsooIiIiIjKaZGTqRjNbCDwBTAPuA7YAK4EbgUvN7NwQwqEhnOpTwF8C7cDLwKRMlE9EREREZDTKVMv6D/FA/TMhhHeHEL4YQrgI+CdgKXDLEM/zTeB0oBJ4Z4bKJiIiIiIyKp1wsB61ql8C7AR+kLT7q0AbcI2ZVRzvXCGEJ0MIL4UQek60XCIiIiIio10mWtbXRNuHQgi9iTtCCC3A40A5sDoD3yUiIiIiMmZkos/60mi7Lc3+7XjL+xJgbQa+77jMbEOaXcty8f0iIiIiIpmQiZb1mmjblGZ/LL82A98lIiIiIjJmZGQ2mEITQliRKj9qcV+e4+KIiIiIiIxIJlrWYy3nNWn2x/IbM/BdIiIiIiJjRiaC9a3Rdkma/Yujbbo+7SIiIiIikkImgvV10fYSM+t3PjOrAs4FjgJPZeC7RERERETGjBPusx5C2GFmD+EzvnwKuC1h981ABfDjEEJbLNPMlkWf3XKi3z9M8zZv3syKFSm7tIuIiIiIZMTmzZsB5p3oeSyEcMKFiRZGegJfxfQ+YDOwCp+DfRtwTgjhUMLxASCEYEnnOQ/4SPS2ErgSaAB+EzsmhPDBEyjnq0A1voBTrsWmjcz1DYoMna5R4dM1Kny6RoVP16iw6foUvqFeo3lAcwhh/ol8WUaCdQAzqwO+DlwKTAb2A/cAN4cQjiQdmy5Y/yDw08G+J/kzo0Vs7vd0M9VI/ukaFT5do8Kna1T4dI0Km65P4cv1NcrY1I0hhD3Ah4Z4bMqAO4RwB3BHpsokIiIiIjKaZWKAqYiIiIiIZIGCdRERERGRAqVgXURERESkQClYFxEREREpUBmbDUZERERERDJLLesiIiIiIgVKwbqIiIiISIFSsC4iIiIiUqAUrIuIiIiIFCgF6yIiIiIiBUrBuoiIiIhIgVKwLiIiIiJSoBSsZ5mZzTaz281sn5l1mNlOM7vVzCbmu2wC0fUIadKBfJdvrDCz95rZbWb2mJk1Rz//nx/nM+eY2QNmdtjMjpnZ82b2WTMrzlW5x5LhXCMzmzdIvQpmdneuy3+yM7PJZvYRM7vHzF6O6kSTmf3RzK4zs5R/71WPcme410j1KD/M7JtmttbM9kTX6LCZPWtmXzWzyWk+k9V6VJKJk0hqZrYQeAKYBtwHbAFWAjcCl5rZuSGEQ3ksorgm4NYU+a25LsgY9iXgTPxnvhdYNtjBZvaXwL8D7cD/BQ4D7wT+CTgXuCqbhR2jhnWNIn8G7k2R/2IGyyXuKuCfgf3AOmA3MB24AvgJcJmZXRUSVkJUPcq5YV+jiOpRbn0O2Aj8DmgAKoDVwNeA681sdQhhT+zgnNSjEIJSlhLwWyAAn07K/06U/6N8l3GsJ2AnsDPf5RjrCVgDLAYMuDCqHz9Pc2x19Au0AzgrIX88fnMcgPfl+990sqVhXqN50f478l3usZKAi6IAoSgpfwYeFAbgyoR81aPCv0aqR/m5TuPT5N8SXY8fJuTlpB6pG0yWRK3ql+DB4A+Sdn8VaAOuMbOKHBdNpOCEENaFELaH6LfccbwXmArcHUJ4JuEc7XjrL8AnslDMMW2Y10hyLITwcAjhVyGE3qT8A8CPorcXJuxSPcqxEVwjyYOoDqTy/6Lt4oS8nNQjdYPJnjXR9qEUFbPFzB7Hg/nVwNpcF076KTOz9wNz8Juo54FHQwg9+S2WpHFRtH0wxb5HgaPAOWZWFkLoyF2xJIWZZvYxYDJwCHgyhPB8nss0FnVF2+6EPNWjwpLqGsWoHhWGd0bbxJ99TuqRgvXsWRptt6XZvx0P1pegYD3fZgB3JeW9amYfCiE8ko8CyaDS1q0QQreZvQqcBiwANueyYDLA26LUx8z+AFwbQtidlxKNMWZWAnwgepsYUKgeFYhBrlGM6lEemNlNQCVQA5wFnIcH6t9IOCwn9UjdYLKnJto2pdkfy6/NQVkkvZ8CF+MBewXwRuDHeF/B35jZmfkrmqShulX4jgJ/D6wAJkbpAnxQ3YXAWnUBzJlvAKcDD4QQfpuQr3pUONJdI9Wj/LoJ77b8WTxQfxC4JITwesIxOalHCtZlTAsh3Bz1I6wPIRwNIbwYQvg4Pgh4Aj76W0SGIYTQEEL4SghhYwihMUqP4k8TnwYWAR/JbylPfmb2GeAL+Exk1+S5OJLCYNdI9Si/QggzQgiGN+ZdgbeOP2tmy3NdFgXr2RO7m6pJsz+W35iDssjwxQb7nJ/XUkgqqlujVAihG5+iDlS3ssrMbgC+C2wC1oQQDicdonqUZ0O4RimpHuVW1Jh3D36TNBn4WcLunNQjBevZszXaLkmzPzaaOF2fdsmv2GMuPWIsPGnrVtT3cz4+SOuVXBZKhkx1K8vM7LPAbfg83Gui2UaSqR7l0RCv0WBUj3IshLALv7E6zcymRNk5qUcK1rNnXbS9JMWqZFX4RPlHgadyXTAZktXRVn+oCs/D0fbSFPvOB8qBJzSDRcFS3coiM/sf+GIsz+FBYEOaQ1WP8mQY12gwqkf5MTPaxmaLy0k9UrCeJSGEHcBD+EDFTyXtvhm/G74rhNCW46JJxMxOSTU4x8zmAd+P3g665L3kxS+Bg8D7zOysWKaZjQf+IXr7z/komDgzW55qeXszuxhfHRBUtzLOzL6MD1bcAFwcQjg4yOGqR3kwnGukepR7ZrbEzAZ0aTGzIjO7BV+R/okQwpFoV07qkWl9i+yJFkZ6Ar+49+HT9qzC52DfBpwTQjiUvxKObWb2NXxgz6PALqAFWAi8A1997AHgPSGEznyVcawws3cD747ezgD+Am8xeizKOxhCuCnp+F/iyzvfjS/v/C58Gq1fAv9Vi/dk1nCuUTSt3GL899/eaP8ZxOck/nIIIfaHTDLAzK4F7sBb/G4j9ewUO0MIdyR8RvUoh4Z7jVSPci/qnvS/gD8Cr+Lz2k/HZ+FZABzAb7I2JXwm6/VIwXqWmVkd8HX8EclkYD9wD3Bzwp2Z5IGZXQB8HHgz8akbG/FHk3fhTz5UQXIgunH66iCH7AohzEv6zLnA3wFn4zdXLwO3A9/TglaZN5xrZGbXAe/Bp6ObAowD6oEnge+HEB5LdxIZmSFcH4BHQggXJn1O9ShHhnuNVI9yz8xOx+OC84DZ+JSLbXgD6/14vRgwEDjb9UjBuoiIiIhIgVKfdRERERGRAqVgXURERESkQClYFxEREREpUArWRUREREQKlIJ1EREREZECpWBdRERERKRAKVgXERERESlQCtZFRERERAqUgnURERERkQKlYF1EREREpEApWBcRERERKVAK1kVERERECpSCdRERERGRAqVgXURERESkQClYFxEREREpUArWRUREREQKlIJ1EREREZEC9f8B/sGve67gGLYAAAAASUVORK5CYII=\n", 635 | "text/plain": [ 636 | "
" 637 | ] 638 | }, 639 | "metadata": { 640 | "image/png": { 641 | "height": 249, 642 | "width": 373 643 | }, 644 | "needs_background": "light" 645 | }, 646 | "output_type": "display_data" 647 | } 648 | ], 649 | "source": [ 650 | "for k in ['accuracy', 'val_accuracy']:\n", 651 | " plot(h.history[k], )" 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": 11, 657 | "metadata": {}, 658 | "outputs": [ 659 | { 660 | "name": "stdout", 661 | "output_type": "stream", 662 | "text": [ 663 | "Model: \"functional_3\"\n", 664 | "_________________________________________________________________\n", 665 | "Layer (type) Output Shape Param # \n", 666 | "=================================================================\n", 667 | "input_2 (InputLayer) [(None, 32, 32, 3)] 0 \n", 668 | "_________________________________________________________________\n", 669 | "conv2d (Conv2D) (None, 30, 30, 32) 896 \n", 670 | "_________________________________________________________________\n", 671 | "max_pooling2d (MaxPooling2D) (None, 29, 29, 32) 0 \n", 672 | "_________________________________________________________________\n", 673 | "flatten_1 (Flatten) (None, 26912) 0 \n", 674 | "_________________________________________________________________\n", 675 | "dense_1 (Dense) (None, 100) 2691300 \n", 676 | "=================================================================\n", 677 | "Total params: 2,692,196\n", 678 | "Trainable params: 2,692,196\n", 679 | "Non-trainable params: 0\n", 680 | "_________________________________________________________________\n" 681 | ] 682 | } 683 | ], 684 | "source": [ 685 | "model.summary()" 686 | ] 687 | }, 688 | { 689 | "cell_type": "code", 690 | "execution_count": null, 691 | "metadata": {}, 692 | "outputs": [], 693 | "source": [] 694 | }, 695 | { 696 | "cell_type": "code", 697 | "execution_count": 12, 698 | "metadata": {}, 699 | "outputs": [ 700 | { 701 | "name": "stdout", 702 | "output_type": "stream", 703 | "text": [ 704 | "./results-v2/cifar100/CNN_maxpool-nf32-ker(3, 3)-n_lay-1-act-relu-rotate=True-scramble=True.json\n" 705 | ] 706 | } 707 | ], 708 | "source": [ 709 | "configs['net']['architecture'] = 'cnn_maxpool'\n", 710 | "\n", 711 | "model_name = f\"CNN_maxpool-nf{net['num_filters']}-ker{kernel_size}\"\n", 712 | "\n", 713 | "model_name += f\"-n_lay-{net['num_layers']}\"\n", 714 | "model_name += f\"-act-{net['activation']}\"\n", 715 | "num_params = model.count_params() \n", 716 | "\n", 717 | "out_file_name = f\"./results-v2/{dataset_name}/{model_name}-rotate={bool(configs['dataset']['rotate'])}-scramble={bool(configs['dataset']['scramble'])}.json\"\n", 718 | "\n", 719 | "configs['net']['architecture'] = 'cnn_maxpool'\n", 720 | "\n", 721 | "results = {}\n", 722 | "results.update({\n", 723 | " 'num_params':num_params,\n", 724 | " 'result':h.history,\n", 725 | " 'configs':configs,\n", 726 | "# 'result': {k: np.float32(v).tolist() for k,v in h.history.items()}, # bug in json or TF2\n", 727 | " })\n", 728 | "\n", 729 | "\n", 730 | "\n", 731 | "# for k,v in results['result'].items():\n", 732 | "# print(k,type(v))\n", 733 | "\n", 734 | "import os\n", 735 | "\n", 736 | "# print(h.history)\n", 737 | "\n", 738 | "dirs = os.path.split(out_file_name)[0]\n", 739 | "os.makedirs(dirs,exist_ok=True)\n", 740 | "\n", 741 | "print(out_file_name)\n", 742 | "\n", 743 | "\n", 744 | "json.dump(results, open(out_file_name, 'w'))" 745 | ] 746 | } 747 | ], 748 | "metadata": { 749 | "kernelspec": { 750 | "display_name": "TF2.0 Py3", 751 | "language": "python", 752 | "name": "tf2" 753 | }, 754 | "language_info": { 755 | "codemirror_mode": { 756 | "name": "ipython", 757 | "version": 3 758 | }, 759 | "file_extension": ".py", 760 | "mimetype": "text/x-python", 761 | "name": "python", 762 | "nbconvert_exporter": "python", 763 | "pygments_lexer": "ipython3", 764 | "version": "3.7.5" 765 | } 766 | }, 767 | "nbformat": 4, 768 | "nbformat_minor": 4 769 | } 770 | -------------------------------------------------------------------------------- /paper-code/D-image-experiments/lconv.py: -------------------------------------------------------------------------------- 1 | from tensorflow import reduce_sum, concat, reduce_max 2 | from tensorflow.keras import Model 3 | from tensorflow.keras.layers import Layer 4 | from tensorflow.keras.activations import deserialize 5 | 6 | from numpy import newaxis,prod 7 | 8 | 9 | class L_module(Layer): 10 | def __init__(self, n_L, out_dim = None, hidden_units = [], activation = 'linear', **kws): 11 | super(L_module, self).__init__(**kws) 12 | self.params = dict(n_L = n_L, activation = activation, hidden_units = hidden_units) 13 | self.out_dim = out_dim 14 | 15 | 16 | def build(self, input_shape): 17 | # print(input_shape) 18 | d = prod(input_shape[1:]) 19 | 20 | space_dim, feature_dim = input_shape[-2:] 21 | out_dim = self.out_dim or space_dim #int(space_dim/self.stride) 22 | 23 | n_L = self.params['n_L'] 24 | hidden_units = self.params['hidden_units'] 25 | 26 | self.hidden_layers = [] 27 | in_dim = space_dim 28 | for i,u in enumerate(hidden_units): 29 | self.hidden_layers += [self.add_weight(shape=(n_L, u, in_dim), 30 | initializer='glorot_normal', trainable=True, name='Lh_%d' %i)] 31 | in_dim = u 32 | 33 | self.L = self.add_weight(shape=(n_L, out_dim, in_dim), initializer='glorot_normal', trainable=True, name='L') 34 | 35 | def call(self, inputs): 36 | n_L = self.params['n_L'] 37 | 38 | x = inputs 39 | 40 | for l in self.hidden_layers: 41 | x = l @ x 42 | x = self.L @ x 43 | act = deserialize(self.params['activation']) 44 | return act(x) 45 | 46 | class L_Conv(Model): 47 | def __init__(self, 48 | num_filters: int, 49 | kernel_size: int, 50 | #stride: int = 1, 51 | activation = 'relu', 52 | L_hid =[], L_act = 'linear', 53 | ): 54 | """Assumes channel last input x: (batch, space, features). 55 | Uses stride to to scale space dimension: out_dim = int(space/stride). 56 | call: L @ (x @ W + b) 57 | """ 58 | super(L_Conv, self).__init__() 59 | self.num_filters = num_filters 60 | self.kernel_size = kernel_size 61 | self.stride = 1 # stride 62 | self.activation = activation 63 | self.L_params = dict(n_L = kernel_size-1, hidden_units = L_hid, activation = L_act) 64 | 65 | def get_L(self, input_shape): 66 | # assume channel last 67 | space_dim, feature_dim = input_shape[-2:] 68 | out_dim = int(space_dim/self.stride) 69 | 70 | # num_L = kernel-1 b/c original input will be concat 71 | # L = self.add_weight(shape=(self.kernel_size - 1, out_dim, space_dim), 72 | # initializer='glorot_normal', 73 | # trainable=True, name='L') 74 | L = L_module(out_dim = out_dim, **self.L_params) 75 | return L 76 | 77 | def build(self, input_shape): 78 | self.L = self.get_L(input_shape) 79 | self.w = self.add_weight(shape=(self.kernel_size, input_shape[-1], self.num_filters), 80 | initializer='glorot_normal', 81 | trainable=True, name = 'w') 82 | self.b = self.add_weight(shape=(self.kernel_size,1, self.num_filters), 83 | initializer='zeros', 84 | trainable=True, name = 'b') 85 | self.activation_layer = deserialize(self.activation) 86 | 87 | def call(self, inputs): 88 | x0 = inputs[:,newaxis] 89 | 90 | # (batch, space, features) --> (batch, 1, space, features) 91 | #x = self.L @ x0 92 | x = self.L(x0) 93 | # (batch, 1, space, features) --> (batch, kernel_size, space/stride, features) 94 | 95 | x = concat([x, x0], axis = 1) # add back the original 96 | 97 | x = x @ self.w + self.b 98 | # (batch, kernel_size, space, features) --> (batch, kernel_size, space, num_filters) 99 | 100 | x = reduce_sum(x, axis = 1) 101 | 102 | return self.activation_layer(x) 103 | 104 | class L_Conv_max(L_Conv): 105 | def __init__(self, stride: int = 1, **kws): 106 | """Does max_i(L_i L_j x) 107 | """ 108 | super(L_Conv_max, self).__init__(**kws) 109 | 110 | def call(self, inputs): 111 | x0 = inputs[:,newaxis] 112 | 113 | #print(x0.shape) 114 | # (batch, space, features) --> (batch, 1, space, features) 115 | #x = self.L @ x0 116 | x = self.L(x0) 117 | # (batch, 1, space, features) --> (batch, kernel_size, space, features) 118 | 119 | #print(x.shape) 120 | x = concat([x, x0], axis = 1) # add back the original 121 | 122 | #print(x.shape) 123 | # apply L again 124 | x1 = self.L(x[:,:,newaxis]) 125 | # (batch, kernel_size, space, features) --> (batch, kernel_size, kernel_size, space, features) 126 | 127 | #print(x1.shape, x.shape) 128 | x = concat([x1, x[:,:,newaxis]], axis = 2) # add back the original 129 | 130 | x = x @ self.w + self.b 131 | # (batch, kernel_size, kernel_size, space, features) --> (batch, kernel_size, kernel_size, space, num_filters) 132 | 133 | x = reduce_sum(x, axis = 1) 134 | # (batch, kernel_size, space, features) 135 | 136 | # max pooling 137 | x = reduce_max(x, axis = 1) 138 | # (batch, space, features) 139 | 140 | return self.activation_layer(x) 141 | 142 | 143 | class L_Conv_strided(L_Conv): 144 | def __init__(self, stride: int = 1, **kws): 145 | """Assumes channel last input x: (batch, space, features). 146 | Uses stride to to scale space dimension: out_dim = int(space/stride). 147 | call: L @ (x @ W + b) 148 | """ 149 | super(L_Conv_strided, self).__init__(**kws) 150 | self.stride = stride 151 | self.L_params['n_L'] += 1 # to make up for lack of residual conn. 152 | 153 | def call(self, inputs): 154 | x0 = inputs[:,newaxis] 155 | 156 | # (batch, space, features) --> (batch, 1, space, features) 157 | #x = self.L @ x0 158 | x = self.L(x0) 159 | # (batch, 1, space, features) --> (batch, kernel_size, space/stride, features) 160 | 161 | # x = tf.concat([x, x0], axis = 1) # add back the original 162 | 163 | x = x @ self.w + self.b 164 | # (batch, kernel_size, space, features) --> (batch, kernel_size, space, num_filters) 165 | 166 | x = reduce_sum(x, axis = 1) 167 | 168 | return self.activation_layer(x) -------------------------------------------------------------------------------- /paper-code/D-image-experiments/run_LieConv_cifar100.py: -------------------------------------------------------------------------------- 1 | """LieConv Baseline experiments. 2 | requires: 3 | https://github.com/mfinzi/LieConv 4 | 5 | Usage: 6 | $ python3 run_LieConv_cifar100.py --epochs 100 --nlay 2 --ker 256 --lr 3e-3 --bn 0 --rot 1 --scr 1 7 | 8 | --rot: rotate images 9 | --scr: scramble images (fixed shuffling of pixels in all images) 10 | --nlay: number of layers 11 | --bn: batchnorm (0: no batchnorm; used in baseline experiments) 12 | 13 | """ 14 | 15 | 16 | import torch 17 | import torchvision.transforms as transforms 18 | import torch.nn as nn 19 | import torch.nn.functional as F 20 | 21 | from torch.utils.data import DataLoader 22 | from oil.utils.utils import LoaderTo, cosLr, islice 23 | from oil.tuning.study import train_trial 24 | from oil.datasetup.datasets import split_dataset 25 | from oil.utils.parallel import try_multigpu_parallelize 26 | from oil.model_trainers.classifier import Classifier 27 | from functools import partial 28 | from torch.optim import Adam 29 | from oil.tuning.args import argupdated_config 30 | import copy 31 | import lie_conv.lieGroups as lieGroups 32 | import lie_conv.lieConv as lieConv 33 | from lie_conv.lieConv import ImgLieResnet 34 | # from lie_conv.datasets import MnistRotDataset, RotMNIST 35 | from oil.datasetup.datasets import EasyIMGDataset 36 | 37 | from lie_conv.utils import Named, export, Expression, FixedNumpySeed, RandomZrotation, GaussianNoise 38 | from lie_conv.utils import Named 39 | import numpy as np 40 | from PIL import Image 41 | from torchvision.datasets.utils import download_url, download_and_extract_archive, extract_archive, \ 42 | verify_str_arg 43 | from torchvision.datasets.vision import VisionDataset 44 | import torchvision 45 | 46 | class RotCIFAR10(EasyIMGDataset,torchvision.datasets.CIFAR10): 47 | # """ Unofficial RotMNIST dataset created on the fly by rotating MNIST""" 48 | means = (0.5,) 49 | stds = (0.25,) 50 | num_targets = 10 51 | def __init__(self,*args,dataseed=0,transform=None,**kwargs): 52 | super().__init__(*args,download=True,**kwargs) 53 | N = len(self) 54 | with FixedNumpySeed(dataseed): 55 | angles = torch.rand(N)*2*np.pi 56 | with torch.no_grad(): 57 | # R = torch.zeros(N,2,2) 58 | # R[:,0,0] = R[:,1,1] = angles.cos() 59 | # R[:,0,1] = R[:,1,0] = angles.sin() 60 | # R[:,1,0] *=-1 61 | # Build affine matrices for random translation of each image 62 | affineMatrices = torch.zeros(N,2,3) 63 | affineMatrices[:,0,0] = angles.cos() 64 | affineMatrices[:,1,1] = angles.cos() 65 | affineMatrices[:,0,1] = angles.sin() 66 | affineMatrices[:,1,0] = -angles.sin() 67 | # affineMatrices[:,0,2] = -2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/w 68 | # affineMatrices[:,1,2] = 2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/h 69 | # self.data = self.data.unsqueeze(1).float() 70 | self.data = torch.as_tensor(self.data.transpose((0,3,1,2))).float() 71 | flowgrid = F.affine_grid(affineMatrices, size = self.data.size()) 72 | self.data = F.grid_sample(self.data, flowgrid) 73 | normalize = transforms.Normalize((127.5,) ,(255,)) 74 | self.data = normalize(self.data) 75 | def __getitem__(self,idx): 76 | return self.data[idx], int(self.targets[idx]) 77 | def default_aug_layers(self): 78 | return RandomRotateTranslate(0)# no translation 79 | 80 | 81 | class CIFAR100(EasyIMGDataset,torchvision.datasets.CIFAR100): 82 | # """ Unofficial RotMNIST dataset created on the fly by rotating MNIST""" 83 | means = (0.5,) 84 | stds = (0.25,) 85 | num_targets = 100 86 | def __init__(self,*args,dataseed=0,transform=None,**kwargs): 87 | super().__init__(*args,download=True,**kwargs) 88 | N = len(self) 89 | with torch.no_grad(): 90 | self.data = torch.as_tensor(self.data.transpose((0,3,1,2))).float() 91 | normalize = transforms.Normalize((127.5,) ,(255,)) 92 | self.data = normalize(self.data) 93 | def __getitem__(self,idx): 94 | return self.data[idx], int(self.targets[idx]) 95 | def default_aug_layers(self): 96 | return RandomRotateTranslate(0)# no translation 97 | 98 | 99 | class RotCIFAR100(EasyIMGDataset,torchvision.datasets.CIFAR100): 100 | # """ Unofficial RotMNIST dataset created on the fly by rotating MNIST""" 101 | means = (0.5,) 102 | stds = (0.25,) 103 | num_targets = 100 104 | def __init__(self,*args,dataseed=0,transform=None,**kwargs): 105 | super().__init__(*args,download=True,**kwargs) 106 | N = len(self) 107 | with FixedNumpySeed(dataseed): 108 | angles = torch.rand(N)*2*np.pi 109 | with torch.no_grad(): 110 | # R = torch.zeros(N,2,2) 111 | # R[:,0,0] = R[:,1,1] = angles.cos() 112 | # R[:,0,1] = R[:,1,0] = angles.sin() 113 | # R[:,1,0] *=-1 114 | # Build affine matrices for random translation of each image 115 | affineMatrices = torch.zeros(N,2,3) 116 | affineMatrices[:,0,0] = angles.cos() 117 | affineMatrices[:,1,1] = angles.cos() 118 | affineMatrices[:,0,1] = angles.sin() 119 | affineMatrices[:,1,0] = -angles.sin() 120 | # affineMatrices[:,0,2] = -2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/w 121 | # affineMatrices[:,1,2] = 2*np.random.randint(-self.max_trans, self.max_trans+1, bs)/h 122 | # self.data = self.data.unsqueeze(1).float() 123 | self.data = torch.as_tensor(self.data.transpose((0,3,1,2))).float() 124 | flowgrid = F.affine_grid(affineMatrices, size = self.data.size()) 125 | self.data = F.grid_sample(self.data, flowgrid) 126 | normalize = transforms.Normalize((127.5,) ,(255,)) 127 | self.data = normalize(self.data) 128 | def __getitem__(self,idx): 129 | return self.data[idx], int(self.targets[idx]) 130 | def default_aug_layers(self): 131 | return RandomRotateTranslate(0)# no translation 132 | 133 | class RotScramCIFAR100(RotCIFAR100): 134 | """ Scrambled""" 135 | def __init__(self,*args,**kwargs): 136 | super().__init__(*args,**kwargs) 137 | with torch.no_grad(): 138 | idx = torch.randperm(self.data[0,0].nelement()) 139 | self.data = self.data.view(*self.data.shape[:2], -1)[:,:,idx].view(self.data.size()) 140 | 141 | def makeTrainer(*, dataset=RotCIFAR100, network=ImgLieResnet, num_epochs=100, 142 | bs=50, lr=3e-3, aug=False,#True, 143 | optim=Adam, device='cuda', trainer=Classifier, 144 | split={'train':40000}, small_test=False, net_config={}, opt_config={}, 145 | trainer_config={'log_dir':None}): 146 | 147 | # Prep the datasets splits, model, and dataloaders 148 | datasets = split_dataset(dataset(f'~/datasets/{dataset}/'),splits=split) 149 | datasets['test'] = dataset(f'~/datasets/{dataset}/', train=False) 150 | device = torch.device(device) 151 | model = network(num_targets=datasets['train'].num_targets,**net_config).to(device) 152 | if aug: model = torch.nn.Sequential(datasets['train'].default_aug_layers(),model) 153 | model,bs = try_multigpu_parallelize(model,bs) 154 | 155 | dataloaders = {k:LoaderTo(DataLoader(v,batch_size=bs,shuffle=(k=='train'), 156 | num_workers=0,pin_memory=False),device) for k,v in datasets.items()} 157 | dataloaders['Train'] = islice(dataloaders['train'],1+len(dataloaders['train'])//10) 158 | if small_test: dataloaders['test'] = islice(dataloaders['test'],1+len(dataloaders['train'])//10) 159 | # Add some extra defaults if SGD is chosen 160 | opt_constr = partial(optim, lr=lr, **opt_config) 161 | lr_sched = cosLr(num_epochs) 162 | return trainer(model,dataloaders,opt_constr,lr_sched,**trainer_config) 163 | 164 | 165 | 166 | import argparse 167 | parser = argparse.ArgumentParser(description='LieConv Tests') 168 | parser.add_argument('--rot', type=int, default=1, metavar='N', 169 | help='rotated CIFAR100 (default: False)') 170 | parser.add_argument('--scr', type=int, default=0, metavar='N', 171 | help='scramble (default: False)') 172 | parser.add_argument('--ker', type=int, default=128, metavar='N', 173 | help='k in LieConv layer (default: 128)') 174 | parser.add_argument('--nlay', type=int, default=2, metavar='N', 175 | help='number of layers (default: 2)') 176 | parser.add_argument('--epochs', type=int, default=40, metavar='N', 177 | help='number of epochs to train (default: 40)') 178 | parser.add_argument('--lr', type=float, default=3e-3, metavar='N', 179 | help='learning rate (default: 3e-3)') 180 | parser.add_argument('--bn', type=int, default=1, metavar='N', 181 | help='batch normalization (default: True)') 182 | 183 | 184 | 185 | args = parser.parse_args() 186 | 187 | SCRAMBLE = args.scr 188 | ROTATE = args.rot 189 | 190 | ker = args.ker 191 | nlay = args.nlay 192 | batchnorm = bool(args.bn) 193 | EPOCHS = args.epochs 194 | 195 | 196 | 197 | 198 | Trial = train_trial(makeTrainer) 199 | defaults = copy.deepcopy(makeTrainer.__kwdefaults__) 200 | 201 | if ROTATE: 202 | defaults['dataset'] = RotCIFAR100 #MnistRotScrambleDataset 203 | elif SCRAMBLE: 204 | defaults['dataset'] = RotScramCIFAR100 205 | else: 206 | print("=============\n\n Using default CIFAR100\n\n=============") 207 | defaults['dataset'] = CIFAR100 208 | 209 | 210 | defaults['net_config'] = dict(chin=3, 211 | num_layers=nlay, 212 | k=ker, 213 | bn= batchnorm 214 | ) 215 | defaults['num_epochs'] = EPOCHS 216 | defaults['lr'] = args.lr 217 | 218 | print(defaults) 219 | fnam = f'./results/lie_conv-cifar100{"-rot" if ROTATE else ""}{"-scr" if SCRAMBLE else ""}-lay{nlay}-k{ker}.pkl' 220 | 221 | print('\n', fnam,'\n') 222 | 223 | results = Trial(defaults) 224 | 225 | net = ImgLieResnet(**defaults['net_config']) 226 | param_size_list = [(name,tuple(param.shape)) for name, param in net.net.named_parameters() if param.requires_grad] 227 | 228 | out = dict( net_configs = results[0], 229 | results = results[1].to_dict(), 230 | params = param_size_list, 231 | total_params = sum([np.prod(i[1]) for i in param_size_list]) 232 | ) 233 | 234 | print('# params: ', out['total_params']) 235 | import pickle 236 | pickle.dump(out, open(fnam, 'wb')) 237 | 238 | 239 | 240 | # if __name__=="__main__": 241 | # Trial = train_trial(makeTrainer) 242 | # defaults = copy.deepcopy(makeTrainer.__kwdefaults__) 243 | # defaults['save'] = False 244 | # Trial(argupdated_config(defaults,namespace=(lieConv,lieGroups))) 245 | -------------------------------------------------------------------------------- /paper-code/D-image-experiments/run_test-v2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import lconv 4 | import tensorflow as tf 5 | 6 | sess = tf.compat.v1.InteractiveSession() 7 | # K = tf.keras.backend 8 | 9 | from tensorflow.keras import Model, Sequential 10 | from tensorflow.keras.layers import Layer, Input, Flatten, Reshape, Dense, Conv2D, MaxPool2D 11 | 12 | import pickle as pk 13 | import json 14 | import numpy as np 15 | 16 | 17 | from scipy import ndimage 18 | 19 | def rotated_ims_rand(x): 20 | return np.float32([ndimage.rotate(i, (np.random.rand()-.5)*180, reshape=False, mode='nearest') for i in x]) 21 | 22 | class Scramble_x: 23 | def __init__(self,x): 24 | s = x.shape[1:-1] 25 | self.idx = np.argsort(np.random.rand(np.prod(s))) 26 | r,c = np.int0(self.idx/s[0]), (self.idx % s[1]) 27 | self.x = np.float32([i[r,c].reshape(s+(x.shape[-1],)) for i in x]) 28 | 29 | 30 | # Defaults 31 | configs= { 32 | 'dataset': dict(name='mnist' ,#'mnist',cifar100, 33 | rotate=False, 34 | scramble=False,), 35 | 'net': dict(architecture= 'lconv', # 'cnn', 'fc', 36 | num_filters=32, 37 | kernel_size=9, 38 | L_hid= [8], #[16], 39 | activation = 'relu', 40 | L_trainable = True, 41 | num_layers = 1, 42 | ), 43 | } 44 | 45 | 46 | import argparse 47 | 48 | parser = argparse.ArgumentParser(description='Run experiments using L-conv or baselines.') 49 | # Add the arguments 50 | # parser.add_argument('dataset_name',#action='store', 51 | # # metavar='--data', 52 | # type=str, #required=True, 53 | # help='mnist, cifar10, cifar100') 54 | 55 | # Add the arguments 56 | # parser.add_argument('config_json_file', #action='store', 57 | # # metavar='--config', 58 | # type=str, #required=True, 59 | # help='json file of model configuration') 60 | parser.add_argument('--architecture', action='store', type=str, help="lconv (default), cnn, fc") 61 | parser.add_argument('--dataset', action='store', type=str, help='mnist (default), cifar10, cifar100, fashion_mnist') 62 | parser.add_argument('--rotate', action='store_const', const=True) 63 | parser.add_argument('--scramble', action='store_const', const=True) 64 | parser.add_argument('--lrand', action='store_true', help='Whether L are random or trainable') 65 | parser.add_argument('--epochs', action='store', type=int) 66 | parser.add_argument('--num_layers', action='store', type=int, help='Number of layers before classification layer') 67 | parser.add_argument('--hid', action='store', type=int, help='# hidden units in L for low-rank encoding') 68 | 69 | # parser.add_argument('--test', action='store', type=bool, required=False) 70 | 71 | args = parser.parse_args() 72 | # print(args.architecture) 73 | # exit() 74 | 75 | # print(args.dataset_name) 76 | # print(args.config_json_file) 77 | # configs = json.load(open(args.config_json_file,'r')) 78 | 79 | if args.architecture: 80 | configs['net']['architecture'] = args.architecture 81 | 82 | if args.dataset: 83 | configs['dataset']['name'] = args.dataset 84 | 85 | if args.rotate!=None: 86 | configs['dataset']['rotate'] = args.rotate 87 | 88 | if args.scramble!=None: 89 | configs['dataset']['scramble'] = args.scramble 90 | 91 | 92 | print(args.lrand) 93 | 94 | if args.lrand: 95 | configs['net']['L_trainable'] = False 96 | 97 | if args.hid: 98 | configs['net']['L_hid'] = [args.hid] 99 | 100 | if args.num_layers: 101 | configs['net']['num_layers'] = args.num_layers 102 | 103 | EPOCHS = args.epochs or 30 104 | 105 | print(configs) 106 | 107 | # exit() 108 | 109 | # cf: conv filters, ker: kernel size, d: dense (FC) units 110 | # make_model_name = lambda cf,ker, d: ''.join([('c%s(k%d)' %(cf,ker) if len(cf) else ''),('d%s'%d if len(d) else '')]) or 'base' 111 | # cf: conv filters, ker: kernel size, d: dense (FC) units 112 | 113 | 114 | 115 | dataset_name = configs['dataset']['name'] 116 | 117 | dataset = eval("tf.keras.datasets.%s.load_data()" %dataset_name) 118 | (x_train, y_train), (x_test,y_test) = dataset 119 | if len(x_train.shape) == 3: 120 | # mnist channel is missing 121 | x_train = x_train[...,np.newaxis] 122 | 123 | # normalize 124 | x_train = x_train/x_train[:100].max() -.5 125 | # make categorical 126 | y_train = tf.keras.utils.to_categorical(y_train) 127 | 128 | 129 | results = {'configs':configs,} 130 | 131 | if configs['dataset']['rotate']: 132 | print('Rotating images') 133 | x_train = rotated_ims_rand(x_train) 134 | 135 | if configs['dataset']['scramble']: 136 | print('Scrambling images') 137 | scr = Scramble_x(x_train) 138 | x_train = scr.x 139 | results['scramble_idx']=scr.idx.tolist() 140 | 141 | ##### Make model ##### 142 | 143 | net = configs['net'] 144 | # arch = net['architecture'] 145 | 146 | kernel_size = net['kernel_size'] 147 | # L_hid = net['L_hid'] 148 | # activation = net['activation'] 149 | # L_trainable = net['L_trainable'] 150 | 151 | 152 | inp = Input(x_train[0].shape) 153 | 154 | x = inp 155 | for _ in range(net['num_layers']): 156 | if net['architecture']=='lconv': 157 | x = tf.reshape(x, shape=(-1,np.prod(x.shape[1:-1]), x.shape[-1])) 158 | lay = lconv.L_Conv(num_filters= net['num_filters'], 159 | kernel_size= net['kernel_size'], 160 | L_hid = net['L_hid'], 161 | activation = net['activation'],) 162 | 163 | x = lay(x) 164 | lay.L.trainable = net['L_trainable'] 165 | 166 | elif net['architecture']=='cnn': 167 | kx = int(0.5+np.sqrt(net['kernel_size'])) 168 | ky = int(0.5+net['kernel_size']/kx) 169 | kernel_size = (kx,ky) 170 | cnn = Conv2D(filters=net['num_filters'], 171 | kernel_size=kernel_size, 172 | activation = net['activation']) 173 | x = cnn(inp) 174 | 175 | elif net['architecture']=='fc': 176 | k = net['kernel_size'] 177 | nf = net['num_filters'] 178 | hid = net['L_hid'][0] 179 | act = net['activation'] 180 | xs = np.prod(inp.shape[1:-1]) 181 | 182 | x = Flatten()(inp) 183 | # FC comparable to L-conv, but no shared weights 184 | x = Dense(k*hid, activation = act)(x) 185 | x = Dense(xs*nf, activation = act)(x) 186 | 187 | 188 | if net['architecture']!='fc': 189 | x = Flatten()(x) 190 | 191 | # x = Dense(100, activation = 'relu')(x) 192 | 193 | out = Dense(y_train.shape[-1], activation='softmax')(x) 194 | 195 | model = Model(inputs = [inp], outputs = [out]) 196 | model.compile(loss = tf.keras.losses.categorical_crossentropy, metrics = ['accuracy']) 197 | 198 | model.summary() 199 | 200 | 201 | # exit() 202 | 203 | ##### Train model 204 | 205 | h = model.fit(x_train, y_train, validation_split=0.2, epochs=EPOCHS) 206 | 207 | ##### model name and results 208 | non_trainable = 0 209 | if net['architecture'] == 'lconv': 210 | model_name = f"L-conv-nf{net['num_filters']}-hid{net['L_hid']}-L_trainable{net['L_trainable']}-ker{net['kernel_size']}" 211 | non_trainable = (0 if net['L_trainable'] else lay.L.count_params()) 212 | elif net['architecture'] == 'cnn': 213 | model_name = f"CNN-nf{net['num_filters']}-ker{kernel_size}" 214 | elif net['architecture'] =='fc': 215 | model_name = f"FC-nf{net['num_filters']}-hid{net['L_hid']}-ker{kernel_size}" 216 | 217 | 218 | model_name += f"-n_lay-{net['num_layers']}" 219 | model_name += f"-act-{net['activation']}" 220 | num_params = model.count_params() - non_trainable 221 | 222 | out_file_name = f"./results-v2/{dataset_name}/{model_name}-rotate={configs['dataset']['rotate']}-scramble={configs['dataset']['scramble']}.json" 223 | 224 | results = {} 225 | results.update({ 226 | 'num_params':num_params, 227 | 'result':h.history, 228 | 'configs':configs, 229 | # 'result': {k: np.float32(v).tolist() for k,v in h.history.items()}, # bug in json or TF2 230 | }) 231 | 232 | 233 | 234 | # for k,v in results['result'].items(): 235 | # print(k,type(v)) 236 | 237 | import os 238 | 239 | # print(h.history) 240 | 241 | dirs = os.path.split(out_file_name)[0] 242 | os.makedirs(dirs,exist_ok=True) 243 | 244 | print(out_file_name) 245 | 246 | 247 | json.dump(results, open(out_file_name, 'w')) -------------------------------------------------------------------------------- /paper-code/D-image-experiments/run_test.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import lconv 4 | import tensorflow as tf 5 | 6 | sess = tf.compat.v1.InteractiveSession() 7 | # K = tf.keras.backend 8 | 9 | from tensorflow.keras import Model, Sequential 10 | from tensorflow.keras.layers import Layer, Input, Flatten, Reshape, Dense, Conv2D, MaxPool2D 11 | 12 | import pickle as pk 13 | import json 14 | import numpy as np 15 | 16 | 17 | from scipy import ndimage 18 | 19 | def rotated_ims_rand(x): 20 | return np.float32([ndimage.rotate(i, (np.random.rand()-.5)*180, reshape=False, mode='nearest') for i in x]) 21 | 22 | class Scramble_x: 23 | def __init__(self,x): 24 | s = x.shape[1:-1] 25 | self.idx = np.argsort(np.random.rand(np.prod(s))) 26 | r,c = np.int0(self.idx/s[0]), (self.idx % s[1]) 27 | self.x = np.float32([i[r,c].reshape(s+(x.shape[-1],)) for i in x]) 28 | 29 | 30 | # Defaults 31 | configs= { 32 | 'dataset': dict(name='mnist' ,#'mnist',cifar100, 33 | rotate=False, 34 | scramble=False,), 35 | 'net': dict(architecture= 'lconv', # 'cnn', 'fc', 36 | num_filters=32, 37 | kernel_size=9, 38 | L_hid= [8], #[16], 39 | activation = 'relu', 40 | L_trainable = True), 41 | } 42 | 43 | 44 | import argparse 45 | 46 | parser = argparse.ArgumentParser(description='Run experiments using L-conv or baselines.') 47 | # Add the arguments 48 | # parser.add_argument('dataset_name',#action='store', 49 | # # metavar='--data', 50 | # type=str, #required=True, 51 | # help='mnist, cifar10, cifar100') 52 | 53 | # Add the arguments 54 | # parser.add_argument('config_json_file', #action='store', 55 | # # metavar='--config', 56 | # type=str, #required=True, 57 | # help='json file of model configuration') 58 | parser.add_argument('--architecture', action='store', type=str, help="lconv (default), cnn, fc") 59 | parser.add_argument('--dataset', action='store', type=str, help='mnist (default), cifar10, cifar100, fashion_mnist') 60 | parser.add_argument('--rotate', action='store_const', const=True) 61 | parser.add_argument('--scramble', action='store_const', const=True) 62 | parser.add_argument('--lrand', action='store_true', help='Whether L are random or trainable') 63 | parser.add_argument('--epochs', action='store', type=int) 64 | parser.add_argument('--hid', action='store', type=int, help='# hidden units in L for low-rank encoding') 65 | 66 | # parser.add_argument('--test', action='store', type=bool, required=False) 67 | 68 | args = parser.parse_args() 69 | # print(args.architecture) 70 | # exit() 71 | 72 | # print(args.dataset_name) 73 | # print(args.config_json_file) 74 | # configs = json.load(open(args.config_json_file,'r')) 75 | 76 | if args.architecture: 77 | configs['net']['architecture'] = args.architecture 78 | 79 | if args.dataset: 80 | configs['dataset']['name'] = args.dataset 81 | 82 | if args.rotate!=None: 83 | configs['dataset']['rotate'] = args.rotate 84 | 85 | if args.scramble!=None: 86 | configs['dataset']['scramble'] = args.scramble 87 | 88 | 89 | print(args.lrand) 90 | 91 | if args.lrand: 92 | configs['net']['L_trainable'] = False 93 | 94 | if args.hid: 95 | configs['net']['L_hid'] = [args.hid] 96 | 97 | 98 | EPOCHS = args.epochs or 30 99 | 100 | print(configs) 101 | 102 | # exit() 103 | 104 | # cf: conv filters, ker: kernel size, d: dense (FC) units 105 | # make_model_name = lambda cf,ker, d: ''.join([('c%s(k%d)' %(cf,ker) if len(cf) else ''),('d%s'%d if len(d) else '')]) or 'base' 106 | # cf: conv filters, ker: kernel size, d: dense (FC) units 107 | 108 | 109 | 110 | dataset_name = configs['dataset']['name'] 111 | 112 | dataset = eval("tf.keras.datasets.%s.load_data()" %dataset_name) 113 | (x_train, y_train), (x_test,y_test) = dataset 114 | if len(x_train.shape) == 3: 115 | # mnist channel is missing 116 | x_train = x_train[...,np.newaxis] 117 | 118 | # normalize 119 | x_train = x_train/x_train[:100].max() -.5 120 | # make categorical 121 | y_train = tf.keras.utils.to_categorical(y_train) 122 | 123 | 124 | results = {'configs':configs,} 125 | 126 | if configs['dataset']['rotate']: 127 | print('Rotating images') 128 | x_train = rotated_ims_rand(x_train) 129 | 130 | if configs['dataset']['scramble']: 131 | print('Scrambling images') 132 | scr = Scramble_x(x_train) 133 | x_train = scr.x 134 | results['scramble_idx']=scr.idx.tolist() 135 | 136 | ##### Make model ##### 137 | 138 | net = configs['net'] 139 | # arch = net['architecture'] 140 | 141 | kernel_size = net['kernel_size'] 142 | # L_hid = net['L_hid'] 143 | # activation = net['activation'] 144 | # L_trainable = net['L_trainable'] 145 | 146 | 147 | inp = Input(x_train[0].shape) 148 | 149 | if net['architecture']=='lconv': 150 | x = tf.reshape(inp, shape=(-1,np.prod(inp.shape[1:-1]), inp.shape[-1])) 151 | lay = lconv.L_Conv(num_filters= net['num_filters'], 152 | kernel_size= kernel_size, 153 | L_hid = net['L_hid'], 154 | activation = net['activation'],) 155 | 156 | x = lay(x) 157 | lay.L.trainable = net['L_trainable'] 158 | 159 | elif net['architecture']=='cnn': 160 | kx = int(round(np.sqrt(kernel_size))) 161 | ky = int(round(kernel_size/kx)) 162 | kernel_size = (kx,ky) 163 | x = Conv2D(filters=net['num_filters'], kernel_size=kernel_size, activation = net['activation'])(inp) 164 | # x = cnn(inp) 165 | elif net['architecture']=='fc': 166 | k = net['kernel_size'] 167 | nf = net['num_filters'] 168 | hid = net['L_hid'][0] 169 | act = net['activation'] 170 | xs = np.prod(inp.shape[1:-1]) 171 | 172 | x = Flatten()(inp) 173 | # FC comparable to L-conv, but no shared weights 174 | x = Dense(k*hid, activation = act)(x) 175 | x = Dense(xs*nf, activation = act)(x) 176 | 177 | if net['architecture']!='fc': 178 | x = Flatten()(x) 179 | 180 | # x = Dense(100, activation = 'relu')(x) 181 | 182 | out = Dense(y_train.shape[-1], activation='softmax')(x) 183 | 184 | model = Model(inputs = [inp], outputs = [out]) 185 | model.compile(loss = tf.keras.losses.categorical_crossentropy, metrics = ['accuracy']) 186 | 187 | model.summary() 188 | 189 | 190 | 191 | ##### Train model 192 | 193 | h = model.fit(x_train, y_train, validation_split=0.2, epochs=EPOCHS) 194 | 195 | ##### model name and results 196 | non_trainable = 0 197 | if net['architecture'] == 'lconv': 198 | model_name = f"L-conv-nf{net['num_filters']}-hid{net['L_hid']}-L_trainable{net['L_trainable']}-ker{kernel_size}" 199 | non_trainable = (0 if net['L_trainable'] else lay.L.count_params()) 200 | elif net['architecture'] == 'cnn': 201 | model_name = f"CNN-nf{net['num_filters']}-ker{kernel_size}" 202 | elif net['architecture'] =='fc': 203 | model_name = f"FC-nf{net['num_filters']}-hid{net['L_hid']}-ker{kernel_size}" 204 | 205 | 206 | model_name += f"-act-{net['activation']}" 207 | num_params = model.count_params() - non_trainable 208 | 209 | out_file_name = f"./results/{dataset_name}/{model_name}-rotate={configs['dataset']['rotate']}-scrambled={configs['dataset']['scramble']}.json" 210 | 211 | results = {} 212 | results.update({ 213 | 'num_params':num_params, 214 | 'result':h.history, 215 | # 'result': {k: np.float32(v).tolist() for k,v in h.history.items()}, # bug in json or TF2 216 | }) 217 | 218 | # for k,v in results['result'].items(): 219 | # print(k,type(v)) 220 | 221 | import os 222 | 223 | # print(h.history) 224 | 225 | dirs = os.path.split(out_file_name)[0] 226 | os.makedirs(dirs,exist_ok=True) 227 | 228 | print(out_file_name) 229 | 230 | 231 | json.dump(results, open(out_file_name, 'w')) -------------------------------------------------------------------------------- /src/.ipynb_checkpoints/examples-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "id": "c71613f7-6f96-4c85-aa8f-e56eb35b30cf", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "from lconv import Lconv" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "id": "65d9c034-6cfe-4640-b86b-c0f14d0cf971", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import torch" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "id": "4124cfe2-6c17-47ea-aaa9-b59c10afeba0", 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "n = 10\n", 31 | "d = 7\n", 32 | "c = 3\n", 33 | "# channel first\n", 34 | "x = torch.rand(n,c,d)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "id": "67a78d5c-758d-49bd-96bc-c4abc7dd3160", 40 | "metadata": {}, 41 | "source": [ 42 | "## Basic usage" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 5, 48 | "id": "70f5dfcb-e608-4eed-8dab-cf6353463b23", 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "num_L = 2\n", 53 | "co = 5\n", 54 | "l = Lconv(d,num_L, cin=c, cout= co)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 6, 60 | "id": "b9f51920-6495-425f-94b9-2e9a51dbb5ec", 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "y = l(x)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 7, 70 | "id": "2e380c7d-5f1e-456e-b3a5-21042bb6ac4e", 71 | "metadata": {}, 72 | "outputs": [ 73 | { 74 | "data": { 75 | "text/plain": [ 76 | "torch.Size([10, 5, 7])" 77 | ] 78 | }, 79 | "execution_count": 7, 80 | "metadata": {}, 81 | "output_type": "execute_result" 82 | } 83 | ], 84 | "source": [ 85 | "y.shape" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "id": "5c6c2860-59e8-4a6f-a2f8-5673d885feb6", 91 | "metadata": {}, 92 | "source": [ 93 | "# Updated soon" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "id": "aad5bb07-2ac4-43bb-a678-4913219b9840", 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [] 103 | } 104 | ], 105 | "metadata": { 106 | "kernelspec": { 107 | "display_name": "Python 3", 108 | "language": "python", 109 | "name": "python3" 110 | }, 111 | "language_info": { 112 | "codemirror_mode": { 113 | "name": "ipython", 114 | "version": 3 115 | }, 116 | "file_extension": ".py", 117 | "mimetype": "text/x-python", 118 | "name": "python", 119 | "nbconvert_exporter": "python", 120 | "pygments_lexer": "ipython3", 121 | "version": "3.8.1" 122 | } 123 | }, 124 | "nbformat": 4, 125 | "nbformat_minor": 5 126 | } 127 | -------------------------------------------------------------------------------- /src/.ipynb_checkpoints/lconv-checkpoint.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class Lconv(nn.Module): 6 | """ L-conv layer with full L """ 7 | def __init__(self,d,num_L=1,cin=1,cout=1,rank=8): 8 | """ 9 | L:(num_L, d, d) 10 | Wi: (num_L, cout, cin) 11 | """ 12 | super().__init__() 13 | self.L = nn.Parameter(torch.Tensor(num_L, d, d)) 14 | self.Wi = nn.Parameter(torch.Tensor(num_L+1, cout, cin)) # W^0 = Wi[0], W^0\epsion^i = Wi[1:] 15 | 16 | # initialize weights and biases 17 | nn.init.kaiming_normal_(self.L) 18 | nn.init.kaiming_normal_(self.Wi) 19 | 20 | def forward(self, x): 21 | # x:(batch, channel, flat_d) 22 | # res = x W0 23 | residual = torch.einsum('bcd,oc->bod', x, self.Wi[0] ) 24 | # y = Li x Wi 25 | y = torch.einsum('kdf,bcf,koc->bod', self.L, x, self.Wi[1:]) 26 | return y + residual 27 | 28 | class Reshape(nn.Module): 29 | def __init__(self,shape=None): 30 | self.shape = shape 31 | super().__init__() 32 | def forward(self,x): 33 | return x.view(-1,*self.shape) 34 | 35 | -------------------------------------------------------------------------------- /src/__pycache__/lconv.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nimadehmamy/L-conv-code/5a8abfbff3f6564771234df3e177d1d4aafe371d/src/__pycache__/lconv.cpython-38.pyc -------------------------------------------------------------------------------- /src/examples.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "id": "c71613f7-6f96-4c85-aa8f-e56eb35b30cf", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "from lconv import Lconv" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "id": "65d9c034-6cfe-4640-b86b-c0f14d0cf971", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import torch" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 3, 26 | "id": "4124cfe2-6c17-47ea-aaa9-b59c10afeba0", 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "n = 10\n", 31 | "d = 7\n", 32 | "c = 3\n", 33 | "# channel first\n", 34 | "x = torch.rand(n,c,d)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "id": "67a78d5c-758d-49bd-96bc-c4abc7dd3160", 40 | "metadata": {}, 41 | "source": [ 42 | "## Basic usage" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 5, 48 | "id": "70f5dfcb-e608-4eed-8dab-cf6353463b23", 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "num_L = 2\n", 53 | "co = 5\n", 54 | "l = Lconv(d,num_L, cin=c, cout= co)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 6, 60 | "id": "b9f51920-6495-425f-94b9-2e9a51dbb5ec", 61 | "metadata": {}, 62 | "outputs": [], 63 | "source": [ 64 | "y = l(x)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": 7, 70 | "id": "2e380c7d-5f1e-456e-b3a5-21042bb6ac4e", 71 | "metadata": {}, 72 | "outputs": [ 73 | { 74 | "data": { 75 | "text/plain": [ 76 | "torch.Size([10, 5, 7])" 77 | ] 78 | }, 79 | "execution_count": 7, 80 | "metadata": {}, 81 | "output_type": "execute_result" 82 | } 83 | ], 84 | "source": [ 85 | "y.shape" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "id": "5c6c2860-59e8-4a6f-a2f8-5673d885feb6", 91 | "metadata": {}, 92 | "source": [ 93 | "# Updated soon" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "id": "aad5bb07-2ac4-43bb-a678-4913219b9840", 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [] 103 | } 104 | ], 105 | "metadata": { 106 | "kernelspec": { 107 | "display_name": "Python 3", 108 | "language": "python", 109 | "name": "python3" 110 | }, 111 | "language_info": { 112 | "codemirror_mode": { 113 | "name": "ipython", 114 | "version": 3 115 | }, 116 | "file_extension": ".py", 117 | "mimetype": "text/x-python", 118 | "name": "python", 119 | "nbconvert_exporter": "python", 120 | "pygments_lexer": "ipython3", 121 | "version": "3.8.1" 122 | } 123 | }, 124 | "nbformat": 4, 125 | "nbformat_minor": 5 126 | } 127 | -------------------------------------------------------------------------------- /src/lconv.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class Lconv(nn.Module): 6 | """ L-conv layer with full L """ 7 | def __init__(self,d,num_L=1,cin=1,cout=1,rank=8): 8 | """ 9 | L:(num_L, d, d) 10 | Wi: (num_L, cout, cin) 11 | """ 12 | super().__init__() 13 | self.L = nn.Parameter(torch.Tensor(num_L, d, d)) 14 | self.Wi = nn.Parameter(torch.Tensor(num_L+1, cout, cin)) # W^0 = Wi[0], W^0\epsion^i = Wi[1:] 15 | 16 | # initialize weights and biases 17 | nn.init.kaiming_normal_(self.L) 18 | nn.init.kaiming_normal_(self.Wi) 19 | 20 | def forward(self, x): 21 | # x:(batch, channel, flat_d) 22 | # res = x W0 23 | residual = torch.einsum('bcd,oc->bod', x, self.Wi[0] ) 24 | # y = Li x Wi 25 | y = torch.einsum('kdf,bcf,koc->bod', self.L, x, self.Wi[1:]) 26 | return y + residual 27 | 28 | class Reshape(nn.Module): 29 | def __init__(self,shape=None): 30 | self.shape = shape 31 | super().__init__() 32 | def forward(self,x): 33 | return x.view(-1,*self.shape) 34 | 35 | --------------------------------------------------------------------------------