├── 1-Multi-Class-Image-Classification-Fashion-MNIST.ipynb
├── 2-Multi-Class-Word-Language-Classification.ipynb
├── 3-Baby-Name-Generation.ipynb
├── 4-Neural-Style-Transfer.ipynb
├── 4-Neural-Style-Transfer.pdf
├── 5-Captcha-Text-Recognition-With-CRNN.ipynb
├── 6-Image-Segmentation-with-UNet.ipynb
├── LICENSE
└── README.md


/3-Baby-Name-Generation.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "**Project Repository:** https://github.com/GokulKarthik/deep-learning-projects-pytorch"
   8 |    ]
   9 |   },
  10 |   {
  11 |    "cell_type": "code",
  12 |    "execution_count": 1,
  13 |    "metadata": {},
  14 |    "outputs": [],
  15 |    "source": [
  16 |     "import os\n",
  17 |     "import time\n",
  18 |     "from tqdm.notebook import tqdm\n",
  19 |     "\n",
  20 |     "import numpy as np\n",
  21 |     "import pandas as pd\n",
  22 |     "import matplotlib.pyplot as plt\n",
  23 |     "import seaborn as sns\n",
  24 |     "\n",
  25 |     "import torch\n",
  26 |     "import torch.nn as nn\n",
  27 |     "import torch.nn.functional as F\n",
  28 |     "import torch.optim as optim\n",
  29 |     "from torch.utils.data import Dataset, DataLoader\n",
  30 |     "from torch.utils.tensorboard import SummaryWriter\n",
  31 |     "from torchsummary import summary\n",
  32 |     "\n",
  33 |     "import string\n",
  34 |     "from collections import Counter"
  35 |    ]
  36 |   },
  37 |   {
  38 |    "cell_type": "code",
  39 |    "execution_count": 2,
  40 |    "metadata": {},
  41 |    "outputs": [],
  42 |    "source": [
  43 |     "writer = SummaryWriter(os.path.join(\"runs\", \"baby-names\"))"
  44 |    ]
  45 |   },
  46 |   {
  47 |    "cell_type": "markdown",
  48 |    "metadata": {},
  49 |    "source": [
  50 |     "## 1. Load data"
  51 |    ]
  52 |   },
  53 |   {
  54 |    "cell_type": "markdown",
  55 |    "metadata": {},
  56 |    "source": [
  57 |     "[This Kaggle dataset](https://www.kaggle.com/kaggle/us-baby-names#NationalNames.csv) has names of the child born from 1880 to 2014 along with other features such as Gender and Count. I am going to use this to build a name generator model using sampling of the trained character level LSTM network"
  58 |    ]
  59 |   },
  60 |   {
  61 |    "cell_type": "code",
  62 |    "execution_count": 3,
  63 |    "metadata": {},
  64 |    "outputs": [
  65 |     {
  66 |      "name": "stdout",
  67 |      "output_type": "stream",
  68 |      "text": [
  69 |       "(1825433, 5)\n"
  70 |      ]
  71 |     },
  72 |     {
  73 |      "data": {
  74 |       "text/html": [
  75 |        "<div>\n",
  76 |        "<style scoped>\n",
  77 |        "    .dataframe tbody tr th:only-of-type {\n",
  78 |        "        vertical-align: middle;\n",
  79 |        "    }\n",
  80 |        "\n",
  81 |        "    .dataframe tbody tr th {\n",
  82 |        "        vertical-align: top;\n",
  83 |        "    }\n",
  84 |        "\n",
  85 |        "    .dataframe thead th {\n",
  86 |        "        text-align: right;\n",
  87 |        "    }\n",
  88 |        "</style>\n",
  89 |        "<table border=\"1\" class=\"dataframe\">\n",
  90 |        "  <thead>\n",
  91 |        "    <tr style=\"text-align: right;\">\n",
  92 |        "      <th></th>\n",
  93 |        "      <th>Id</th>\n",
  94 |        "      <th>Name</th>\n",
  95 |        "      <th>Year</th>\n",
  96 |        "      <th>Gender</th>\n",
  97 |        "      <th>Count</th>\n",
  98 |        "    </tr>\n",
  99 |        "  </thead>\n",
 100 |        "  <tbody>\n",
 101 |        "    <tr>\n",
 102 |        "      <th>0</th>\n",
 103 |        "      <td>1</td>\n",
 104 |        "      <td>Mary</td>\n",
 105 |        "      <td>1880</td>\n",
 106 |        "      <td>F</td>\n",
 107 |        "      <td>7065</td>\n",
 108 |        "    </tr>\n",
 109 |        "    <tr>\n",
 110 |        "      <th>1</th>\n",
 111 |        "      <td>2</td>\n",
 112 |        "      <td>Anna</td>\n",
 113 |        "      <td>1880</td>\n",
 114 |        "      <td>F</td>\n",
 115 |        "      <td>2604</td>\n",
 116 |        "    </tr>\n",
 117 |        "    <tr>\n",
 118 |        "      <th>2</th>\n",
 119 |        "      <td>3</td>\n",
 120 |        "      <td>Emma</td>\n",
 121 |        "      <td>1880</td>\n",
 122 |        "      <td>F</td>\n",
 123 |        "      <td>2003</td>\n",
 124 |        "    </tr>\n",
 125 |        "    <tr>\n",
 126 |        "      <th>3</th>\n",
 127 |        "      <td>4</td>\n",
 128 |        "      <td>Elizabeth</td>\n",
 129 |        "      <td>1880</td>\n",
 130 |        "      <td>F</td>\n",
 131 |        "      <td>1939</td>\n",
 132 |        "    </tr>\n",
 133 |        "    <tr>\n",
 134 |        "      <th>4</th>\n",
 135 |        "      <td>5</td>\n",
 136 |        "      <td>Minnie</td>\n",
 137 |        "      <td>1880</td>\n",
 138 |        "      <td>F</td>\n",
 139 |        "      <td>1746</td>\n",
 140 |        "    </tr>\n",
 141 |        "  </tbody>\n",
 142 |        "</table>\n",
 143 |        "</div>"
 144 |       ],
 145 |       "text/plain": [
 146 |        "   Id       Name  Year Gender  Count\n",
 147 |        "0   1       Mary  1880      F   7065\n",
 148 |        "1   2       Anna  1880      F   2604\n",
 149 |        "2   3       Emma  1880      F   2003\n",
 150 |        "3   4  Elizabeth  1880      F   1939\n",
 151 |        "4   5     Minnie  1880      F   1746"
 152 |       ]
 153 |      },
 154 |      "execution_count": 3,
 155 |      "metadata": {},
 156 |      "output_type": "execute_result"
 157 |     }
 158 |    ],
 159 |    "source": [
 160 |     "data_path = os.path.join(\"data\", \"baby-names\", \"NationalNames.csv\")\n",
 161 |     "data = pd.read_csv(data_path)\n",
 162 |     "print(data.shape)\n",
 163 |     "data.head()"
 164 |    ]
 165 |   },
 166 |   {
 167 |    "cell_type": "code",
 168 |    "execution_count": 4,
 169 |    "metadata": {},
 170 |    "outputs": [
 171 |     {
 172 |      "name": "stdout",
 173 |      "output_type": "stream",
 174 |      "text": [
 175 |       "<class 'pandas.core.frame.DataFrame'>\n",
 176 |       "RangeIndex: 1825433 entries, 0 to 1825432\n",
 177 |       "Data columns (total 5 columns):\n",
 178 |       "Id        int64\n",
 179 |       "Name      object\n",
 180 |       "Year      int64\n",
 181 |       "Gender    object\n",
 182 |       "Count     int64\n",
 183 |       "dtypes: int64(3), object(2)\n",
 184 |       "memory usage: 69.6+ MB\n"
 185 |      ]
 186 |     }
 187 |    ],
 188 |    "source": [
 189 |     "data.info()"
 190 |    ]
 191 |   },
 192 |   {
 193 |    "cell_type": "markdown",
 194 |    "metadata": {},
 195 |    "source": [
 196 |     "## 2. Clean data"
 197 |    ]
 198 |   },
 199 |   {
 200 |    "cell_type": "code",
 201 |    "execution_count": 5,
 202 |    "metadata": {},
 203 |    "outputs": [],
 204 |    "source": [
 205 |     "def clean(name):\n",
 206 |     "    \n",
 207 |     "    name = name.lower().strip()\n",
 208 |     "    name = \"\".join([c for c in name if c in string.ascii_lowercase])\n",
 209 |     "    name += \".\"\n",
 210 |     "    return name"
 211 |    ]
 212 |   },
 213 |   {
 214 |    "cell_type": "code",
 215 |    "execution_count": 6,
 216 |    "metadata": {},
 217 |    "outputs": [
 218 |     {
 219 |      "data": {
 220 |       "text/html": [
 221 |        "<div>\n",
 222 |        "<style scoped>\n",
 223 |        "    .dataframe tbody tr th:only-of-type {\n",
 224 |        "        vertical-align: middle;\n",
 225 |        "    }\n",
 226 |        "\n",
 227 |        "    .dataframe tbody tr th {\n",
 228 |        "        vertical-align: top;\n",
 229 |        "    }\n",
 230 |        "\n",
 231 |        "    .dataframe thead th {\n",
 232 |        "        text-align: right;\n",
 233 |        "    }\n",
 234 |        "</style>\n",
 235 |        "<table border=\"1\" class=\"dataframe\">\n",
 236 |        "  <thead>\n",
 237 |        "    <tr style=\"text-align: right;\">\n",
 238 |        "      <th></th>\n",
 239 |        "      <th>Id</th>\n",
 240 |        "      <th>Name</th>\n",
 241 |        "      <th>Year</th>\n",
 242 |        "      <th>Gender</th>\n",
 243 |        "      <th>Count</th>\n",
 244 |        "    </tr>\n",
 245 |        "  </thead>\n",
 246 |        "  <tbody>\n",
 247 |        "    <tr>\n",
 248 |        "      <th>0</th>\n",
 249 |        "      <td>1</td>\n",
 250 |        "      <td>mary.</td>\n",
 251 |        "      <td>1880</td>\n",
 252 |        "      <td>F</td>\n",
 253 |        "      <td>7065</td>\n",
 254 |        "    </tr>\n",
 255 |        "    <tr>\n",
 256 |        "      <th>1</th>\n",
 257 |        "      <td>2</td>\n",
 258 |        "      <td>anna.</td>\n",
 259 |        "      <td>1880</td>\n",
 260 |        "      <td>F</td>\n",
 261 |        "      <td>2604</td>\n",
 262 |        "    </tr>\n",
 263 |        "    <tr>\n",
 264 |        "      <th>2</th>\n",
 265 |        "      <td>3</td>\n",
 266 |        "      <td>emma.</td>\n",
 267 |        "      <td>1880</td>\n",
 268 |        "      <td>F</td>\n",
 269 |        "      <td>2003</td>\n",
 270 |        "    </tr>\n",
 271 |        "    <tr>\n",
 272 |        "      <th>3</th>\n",
 273 |        "      <td>4</td>\n",
 274 |        "      <td>elizabeth.</td>\n",
 275 |        "      <td>1880</td>\n",
 276 |        "      <td>F</td>\n",
 277 |        "      <td>1939</td>\n",
 278 |        "    </tr>\n",
 279 |        "    <tr>\n",
 280 |        "      <th>4</th>\n",
 281 |        "      <td>5</td>\n",
 282 |        "      <td>minnie.</td>\n",
 283 |        "      <td>1880</td>\n",
 284 |        "      <td>F</td>\n",
 285 |        "      <td>1746</td>\n",
 286 |        "    </tr>\n",
 287 |        "  </tbody>\n",
 288 |        "</table>\n",
 289 |        "</div>"
 290 |       ],
 291 |       "text/plain": [
 292 |        "   Id        Name  Year Gender  Count\n",
 293 |        "0   1       mary.  1880      F   7065\n",
 294 |        "1   2       anna.  1880      F   2604\n",
 295 |        "2   3       emma.  1880      F   2003\n",
 296 |        "3   4  elizabeth.  1880      F   1939\n",
 297 |        "4   5     minnie.  1880      F   1746"
 298 |       ]
 299 |      },
 300 |      "execution_count": 6,
 301 |      "metadata": {},
 302 |      "output_type": "execute_result"
 303 |     }
 304 |    ],
 305 |    "source": [
 306 |     "data['Name'] = data['Name'].apply(clean)\n",
 307 |     "data.head()"
 308 |    ]
 309 |   },
 310 |   {
 311 |    "cell_type": "code",
 312 |    "execution_count": 7,
 313 |    "metadata": {},
 314 |    "outputs": [
 315 |     {
 316 |      "name": "stdout",
 317 |      "output_type": "stream",
 318 |      "text": [
 319 |       "93889\n"
 320 |      ]
 321 |     },
 322 |     {
 323 |      "data": {
 324 |       "text/html": [
 325 |        "<div>\n",
 326 |        "<style scoped>\n",
 327 |        "    .dataframe tbody tr th:only-of-type {\n",
 328 |        "        vertical-align: middle;\n",
 329 |        "    }\n",
 330 |        "\n",
 331 |        "    .dataframe tbody tr th {\n",
 332 |        "        vertical-align: top;\n",
 333 |        "    }\n",
 334 |        "\n",
 335 |        "    .dataframe thead th {\n",
 336 |        "        text-align: right;\n",
 337 |        "    }\n",
 338 |        "</style>\n",
 339 |        "<table border=\"1\" class=\"dataframe\">\n",
 340 |        "  <thead>\n",
 341 |        "    <tr style=\"text-align: right;\">\n",
 342 |        "      <th></th>\n",
 343 |        "      <th>Count</th>\n",
 344 |        "    </tr>\n",
 345 |        "  </thead>\n",
 346 |        "  <tbody>\n",
 347 |        "    <tr>\n",
 348 |        "      <th>aaban.</th>\n",
 349 |        "      <td>72</td>\n",
 350 |        "    </tr>\n",
 351 |        "    <tr>\n",
 352 |        "      <th>aabha.</th>\n",
 353 |        "      <td>21</td>\n",
 354 |        "    </tr>\n",
 355 |        "    <tr>\n",
 356 |        "      <th>aabid.</th>\n",
 357 |        "      <td>5</td>\n",
 358 |        "    </tr>\n",
 359 |        "    <tr>\n",
 360 |        "      <th>aabriella.</th>\n",
 361 |        "      <td>10</td>\n",
 362 |        "    </tr>\n",
 363 |        "    <tr>\n",
 364 |        "      <th>aadam.</th>\n",
 365 |        "      <td>196</td>\n",
 366 |        "    </tr>\n",
 367 |        "  </tbody>\n",
 368 |        "</table>\n",
 369 |        "</div>"
 370 |       ],
 371 |       "text/plain": [
 372 |        "            Count\n",
 373 |        "aaban.         72\n",
 374 |        "aabha.         21\n",
 375 |        "aabid.          5\n",
 376 |        "aabriella.     10\n",
 377 |        "aadam.        196"
 378 |       ]
 379 |      },
 380 |      "execution_count": 7,
 381 |      "metadata": {},
 382 |      "output_type": "execute_result"
 383 |     }
 384 |    ],
 385 |    "source": [
 386 |     "names = data[['Name', 'Count']].groupby('Name').sum()\n",
 387 |     "del names.index.name\n",
 388 |     "print(len(names))\n",
 389 |     "names.head()"
 390 |    ]
 391 |   },
 392 |   {
 393 |    "cell_type": "code",
 394 |    "execution_count": 8,
 395 |    "metadata": {},
 396 |    "outputs": [
 397 |     {
 398 |      "data": {
 399 |       "text/plain": [
 400 |        "16"
 401 |       ]
 402 |      },
 403 |      "execution_count": 8,
 404 |      "metadata": {},
 405 |      "output_type": "execute_result"
 406 |     }
 407 |    ],
 408 |    "source": [
 409 |     "pd.Series(names.index).apply(len).max()"
 410 |    ]
 411 |   },
 412 |   {
 413 |    "cell_type": "code",
 414 |    "execution_count": 9,
 415 |    "metadata": {},
 416 |    "outputs": [
 417 |     {
 418 |      "name": "stdout",
 419 |      "output_type": "stream",
 420 |      "text": [
 421 |       "[True, True, True, True, True, True, True, True, True, True]\n",
 422 |       "(93889, 1)\n",
 423 |       "(92889, 1)\n"
 424 |      ]
 425 |     }
 426 |    ],
 427 |    "source": [
 428 |     "max_length = 11\n",
 429 |     "len_filter = pd.Series(names.index).apply(lambda x: len(x)<=max_length).tolist() # max length of 10 excluding '.'\n",
 430 |     "print(len_filter[:10])\n",
 431 |     "print(names.shape)\n",
 432 |     "names = names[len_filter]\n",
 433 |     "print(names.shape)"
 434 |    ]
 435 |   },
 436 |   {
 437 |    "cell_type": "code",
 438 |    "execution_count": 10,
 439 |    "metadata": {},
 440 |    "outputs": [
 441 |     {
 442 |      "data": {
 443 |       "text/plain": [
 444 |        "11"
 445 |       ]
 446 |      },
 447 |      "execution_count": 10,
 448 |      "metadata": {},
 449 |      "output_type": "execute_result"
 450 |     }
 451 |    ],
 452 |    "source": [
 453 |     "pd.Series(names.index).apply(len).max()"
 454 |    ]
 455 |   },
 456 |   {
 457 |    "cell_type": "code",
 458 |    "execution_count": 11,
 459 |    "metadata": {},
 460 |    "outputs": [
 461 |     {
 462 |      "data": {
 463 |       "text/html": [
 464 |        "<div>\n",
 465 |        "<style scoped>\n",
 466 |        "    .dataframe tbody tr th:only-of-type {\n",
 467 |        "        vertical-align: middle;\n",
 468 |        "    }\n",
 469 |        "\n",
 470 |        "    .dataframe tbody tr th {\n",
 471 |        "        vertical-align: top;\n",
 472 |        "    }\n",
 473 |        "\n",
 474 |        "    .dataframe thead th {\n",
 475 |        "        text-align: right;\n",
 476 |        "    }\n",
 477 |        "</style>\n",
 478 |        "<table border=\"1\" class=\"dataframe\">\n",
 479 |        "  <thead>\n",
 480 |        "    <tr style=\"text-align: right;\">\n",
 481 |        "      <th></th>\n",
 482 |        "      <th>Count</th>\n",
 483 |        "    </tr>\n",
 484 |        "  </thead>\n",
 485 |        "  <tbody>\n",
 486 |        "    <tr>\n",
 487 |        "      <th>james.</th>\n",
 488 |        "      <td>5129096</td>\n",
 489 |        "    </tr>\n",
 490 |        "    <tr>\n",
 491 |        "      <th>john.</th>\n",
 492 |        "      <td>5106590</td>\n",
 493 |        "    </tr>\n",
 494 |        "    <tr>\n",
 495 |        "      <th>robert.</th>\n",
 496 |        "      <td>4816785</td>\n",
 497 |        "    </tr>\n",
 498 |        "    <tr>\n",
 499 |        "      <th>michael.</th>\n",
 500 |        "      <td>4330805</td>\n",
 501 |        "    </tr>\n",
 502 |        "    <tr>\n",
 503 |        "      <th>mary.</th>\n",
 504 |        "      <td>4130441</td>\n",
 505 |        "    </tr>\n",
 506 |        "  </tbody>\n",
 507 |        "</table>\n",
 508 |        "</div>"
 509 |       ],
 510 |       "text/plain": [
 511 |        "            Count\n",
 512 |        "james.    5129096\n",
 513 |        "john.     5106590\n",
 514 |        "robert.   4816785\n",
 515 |        "michael.  4330805\n",
 516 |        "mary.     4130441"
 517 |       ]
 518 |      },
 519 |      "execution_count": 11,
 520 |      "metadata": {},
 521 |      "output_type": "execute_result"
 522 |     }
 523 |    ],
 524 |    "source": [
 525 |     "names = names.sort_values(by=['Count'], ascending=False)\n",
 526 |     "names.head()"
 527 |    ]
 528 |   },
 529 |   {
 530 |    "cell_type": "markdown",
 531 |    "metadata": {},
 532 |    "source": [
 533 |     "## 3. Set training data"
 534 |    ]
 535 |   },
 536 |   {
 537 |    "cell_type": "markdown",
 538 |    "metadata": {},
 539 |    "source": [
 540 |     "We need a list of names to start building the name generator model. One naive approach for this dataset could be to just take list of unique names. The number of uniques names is 93889, which is large. So, if we sample uniformly from the unique names, the model may learn to generate uncommon and less interesting names. Also if we use the exact counts the model will generate more common names. So we have to sample in between these two. Normalized counts can be used to sample for training."
 541 |    ]
 542 |   },
 543 |   {
 544 |    "cell_type": "code",
 545 |    "execution_count": 12,
 546 |    "metadata": {},
 547 |    "outputs": [
 548 |     {
 549 |      "data": {
 550 |       "text/plain": [
 551 |        "count    9.288900e+04\n",
 552 |        "mean     3.606536e+03\n",
 553 |        "std      5.557699e+04\n",
 554 |        "min      5.000000e+00\n",
 555 |        "25%      1.100000e+01\n",
 556 |        "50%      4.500000e+01\n",
 557 |        "75%      2.400000e+02\n",
 558 |        "max      5.129096e+06\n",
 559 |        "Name: Count, dtype: float64"
 560 |       ]
 561 |      },
 562 |      "execution_count": 12,
 563 |      "metadata": {},
 564 |      "output_type": "execute_result"
 565 |     }
 566 |    ],
 567 |    "source": [
 568 |     "names['Count'].describe()"
 569 |    ]
 570 |   },
 571 |   {
 572 |    "cell_type": "code",
 573 |    "execution_count": 13,
 574 |    "metadata": {},
 575 |    "outputs": [
 576 |     {
 577 |      "data": {
 578 |       "text/plain": [
 579 |        "count     92889.000000\n",
 580 |        "mean        342.499295\n",
 581 |        "std        3211.942902\n",
 582 |        "min           3.000000\n",
 583 |        "25%           6.000000\n",
 584 |        "50%          21.000000\n",
 585 |        "75%          80.000000\n",
 586 |        "max      233363.000000\n",
 587 |        "Name: Count, dtype: float64"
 588 |       ]
 589 |      },
 590 |      "execution_count": 13,
 591 |      "metadata": {},
 592 |      "output_type": "execute_result"
 593 |     }
 594 |    ],
 595 |    "source": [
 596 |     "alpha = 0.8\n",
 597 |     "names['Count'].apply(lambda x: np.power(x, alpha)).apply(np.int).describe()"
 598 |    ]
 599 |   },
 600 |   {
 601 |    "cell_type": "code",
 602 |    "execution_count": 14,
 603 |    "metadata": {},
 604 |    "outputs": [
 605 |     {
 606 |      "data": {
 607 |       "text/html": [
 608 |        "<div>\n",
 609 |        "<style scoped>\n",
 610 |        "    .dataframe tbody tr th:only-of-type {\n",
 611 |        "        vertical-align: middle;\n",
 612 |        "    }\n",
 613 |        "\n",
 614 |        "    .dataframe tbody tr th {\n",
 615 |        "        vertical-align: top;\n",
 616 |        "    }\n",
 617 |        "\n",
 618 |        "    .dataframe thead th {\n",
 619 |        "        text-align: right;\n",
 620 |        "    }\n",
 621 |        "</style>\n",
 622 |        "<table border=\"1\" class=\"dataframe\">\n",
 623 |        "  <thead>\n",
 624 |        "    <tr style=\"text-align: right;\">\n",
 625 |        "      <th></th>\n",
 626 |        "      <th>Count</th>\n",
 627 |        "      <th>count_normalized</th>\n",
 628 |        "    </tr>\n",
 629 |        "  </thead>\n",
 630 |        "  <tbody>\n",
 631 |        "    <tr>\n",
 632 |        "      <th>james.</th>\n",
 633 |        "      <td>5129096</td>\n",
 634 |        "      <td>233363</td>\n",
 635 |        "    </tr>\n",
 636 |        "    <tr>\n",
 637 |        "      <th>john.</th>\n",
 638 |        "      <td>5106590</td>\n",
 639 |        "      <td>232543</td>\n",
 640 |        "    </tr>\n",
 641 |        "    <tr>\n",
 642 |        "      <th>robert.</th>\n",
 643 |        "      <td>4816785</td>\n",
 644 |        "      <td>221924</td>\n",
 645 |        "    </tr>\n",
 646 |        "    <tr>\n",
 647 |        "      <th>michael.</th>\n",
 648 |        "      <td>4330805</td>\n",
 649 |        "      <td>203823</td>\n",
 650 |        "    </tr>\n",
 651 |        "    <tr>\n",
 652 |        "      <th>mary.</th>\n",
 653 |        "      <td>4130441</td>\n",
 654 |        "      <td>196244</td>\n",
 655 |        "    </tr>\n",
 656 |        "  </tbody>\n",
 657 |        "</table>\n",
 658 |        "</div>"
 659 |       ],
 660 |       "text/plain": [
 661 |        "            Count  count_normalized\n",
 662 |        "james.    5129096            233363\n",
 663 |        "john.     5106590            232543\n",
 664 |        "robert.   4816785            221924\n",
 665 |        "michael.  4330805            203823\n",
 666 |        "mary.     4130441            196244"
 667 |       ]
 668 |      },
 669 |      "execution_count": 14,
 670 |      "metadata": {},
 671 |      "output_type": "execute_result"
 672 |     }
 673 |    ],
 674 |    "source": [
 675 |     "names['count_normalized'] = names['Count'].apply(lambda x: np.power(x, alpha)).apply(np.int)\n",
 676 |     "names.head()"
 677 |    ]
 678 |   },
 679 |   {
 680 |    "cell_type": "code",
 681 |    "execution_count": 15,
 682 |    "metadata": {},
 683 |    "outputs": [
 684 |     {
 685 |      "name": "stdout",
 686 |      "output_type": "stream",
 687 |      "text": [
 688 |       "31814417\n"
 689 |      ]
 690 |     }
 691 |    ],
 692 |    "source": [
 693 |     "count_normalized_sum = names['count_normalized'].sum()\n",
 694 |     "print(count_normalized_sum)"
 695 |    ]
 696 |   },
 697 |   {
 698 |    "cell_type": "code",
 699 |    "execution_count": 16,
 700 |    "metadata": {},
 701 |    "outputs": [
 702 |     {
 703 |      "data": {
 704 |       "text/html": [
 705 |        "<div>\n",
 706 |        "<style scoped>\n",
 707 |        "    .dataframe tbody tr th:only-of-type {\n",
 708 |        "        vertical-align: middle;\n",
 709 |        "    }\n",
 710 |        "\n",
 711 |        "    .dataframe tbody tr th {\n",
 712 |        "        vertical-align: top;\n",
 713 |        "    }\n",
 714 |        "\n",
 715 |        "    .dataframe thead th {\n",
 716 |        "        text-align: right;\n",
 717 |        "    }\n",
 718 |        "</style>\n",
 719 |        "<table border=\"1\" class=\"dataframe\">\n",
 720 |        "  <thead>\n",
 721 |        "    <tr style=\"text-align: right;\">\n",
 722 |        "      <th></th>\n",
 723 |        "      <th>Count</th>\n",
 724 |        "      <th>count_normalized</th>\n",
 725 |        "      <th>p</th>\n",
 726 |        "    </tr>\n",
 727 |        "  </thead>\n",
 728 |        "  <tbody>\n",
 729 |        "    <tr>\n",
 730 |        "      <th>james.</th>\n",
 731 |        "      <td>5129096</td>\n",
 732 |        "      <td>233363</td>\n",
 733 |        "      <td>0.007335</td>\n",
 734 |        "    </tr>\n",
 735 |        "    <tr>\n",
 736 |        "      <th>john.</th>\n",
 737 |        "      <td>5106590</td>\n",
 738 |        "      <td>232543</td>\n",
 739 |        "      <td>0.007309</td>\n",
 740 |        "    </tr>\n",
 741 |        "    <tr>\n",
 742 |        "      <th>robert.</th>\n",
 743 |        "      <td>4816785</td>\n",
 744 |        "      <td>221924</td>\n",
 745 |        "      <td>0.006976</td>\n",
 746 |        "    </tr>\n",
 747 |        "    <tr>\n",
 748 |        "      <th>michael.</th>\n",
 749 |        "      <td>4330805</td>\n",
 750 |        "      <td>203823</td>\n",
 751 |        "      <td>0.006407</td>\n",
 752 |        "    </tr>\n",
 753 |        "    <tr>\n",
 754 |        "      <th>mary.</th>\n",
 755 |        "      <td>4130441</td>\n",
 756 |        "      <td>196244</td>\n",
 757 |        "      <td>0.006168</td>\n",
 758 |        "    </tr>\n",
 759 |        "  </tbody>\n",
 760 |        "</table>\n",
 761 |        "</div>"
 762 |       ],
 763 |       "text/plain": [
 764 |        "            Count  count_normalized         p\n",
 765 |        "james.    5129096            233363  0.007335\n",
 766 |        "john.     5106590            232543  0.007309\n",
 767 |        "robert.   4816785            221924  0.006976\n",
 768 |        "michael.  4330805            203823  0.006407\n",
 769 |        "mary.     4130441            196244  0.006168"
 770 |       ]
 771 |      },
 772 |      "execution_count": 16,
 773 |      "metadata": {},
 774 |      "output_type": "execute_result"
 775 |     }
 776 |    ],
 777 |    "source": [
 778 |     "names['p'] = names['count_normalized'] / count_normalized_sum\n",
 779 |     "names.head()"
 780 |    ]
 781 |   },
 782 |   {
 783 |    "cell_type": "code",
 784 |    "execution_count": 17,
 785 |    "metadata": {},
 786 |    "outputs": [
 787 |     {
 788 |      "name": "stdout",
 789 |      "output_type": "stream",
 790 |      "text": [
 791 |       "100000\n",
 792 |       "['joanna.' 'pearlie.' 'harrison.' 'kay.' 'kim.' 'braden.' 'bernard.'\n",
 793 |       " 'eugena.' 'essynce.' 'audrey.' 'sharla.' 'oliver.' 'susie.' 'meka.'\n",
 794 |       " 'patricia.' 'donald.' 'robert.' 'sheyla.' 'lizeth.' 'tayden.' 'casmira.'\n",
 795 |       " 'karmen.' 'dean.' 'danita.' 'ronald.' 'mona.' 'frank.' 'kareemah.'\n",
 796 |       " 'mackenzie.' 'thelma.' 'grace.' 'kami.' 'gertrude.' 'misty.' 'robert.'\n",
 797 |       " 'horace.' 'gerard.' 'salvatore.' 'iyannah.' 'hillary.' 'edna.' 'holly.'\n",
 798 |       " 'simone.' 'thomas.' 'ronda.' 'helena.' 'katherine.' 'helen.' 'madison.'\n",
 799 |       " 'dawn.']\n"
 800 |      ]
 801 |     }
 802 |    ],
 803 |    "source": [
 804 |     "np.random.seed(0)\n",
 805 |     "names_list = np.random.choice(names.index, size=10**5, p=names['p'], replace=True)\n",
 806 |     "print(len(names_list))\n",
 807 |     "print(names_list[:50])"
 808 |    ]
 809 |   },
 810 |   {
 811 |    "cell_type": "code",
 812 |    "execution_count": 18,
 813 |    "metadata": {},
 814 |    "outputs": [
 815 |     {
 816 |      "data": {
 817 |       "text/plain": [
 818 |        "james.       733\n",
 819 |        "john.        724\n",
 820 |        "robert.      677\n",
 821 |        "michael.     649\n",
 822 |        "william.     605\n",
 823 |        "            ... \n",
 824 |        "laveta.        1\n",
 825 |        "nakeysha.      1\n",
 826 |        "ramin.         1\n",
 827 |        "vega.          1\n",
 828 |        "soliyana.      1\n",
 829 |        "Length: 16785, dtype: int64"
 830 |       ]
 831 |      },
 832 |      "execution_count": 18,
 833 |      "metadata": {},
 834 |      "output_type": "execute_result"
 835 |     }
 836 |    ],
 837 |    "source": [
 838 |     "pd.Series(names_list).value_counts()"
 839 |    ]
 840 |   },
 841 |   {
 842 |    "cell_type": "code",
 843 |    "execution_count": 19,
 844 |    "metadata": {},
 845 |    "outputs": [],
 846 |    "source": [
 847 |     "del data, names"
 848 |    ]
 849 |   },
 850 |   {
 851 |    "cell_type": "markdown",
 852 |    "metadata": {},
 853 |    "source": [
 854 |     "## 3. Define utilities"
 855 |    ]
 856 |   },
 857 |   {
 858 |    "cell_type": "code",
 859 |    "execution_count": 20,
 860 |    "metadata": {},
 861 |    "outputs": [
 862 |     {
 863 |      "name": "stdout",
 864 |      "output_type": "stream",
 865 |      "text": [
 866 |       ".abcdefghijklmnopqrstuvwxyz\n",
 867 |       "27\n"
 868 |      ]
 869 |     }
 870 |    ],
 871 |    "source": [
 872 |     "chars = \".\" + string.ascii_lowercase\n",
 873 |     "num_chars = len(chars)\n",
 874 |     "print(chars)\n",
 875 |     "print(num_chars)"
 876 |    ]
 877 |   },
 878 |   {
 879 |    "cell_type": "code",
 880 |    "execution_count": 21,
 881 |    "metadata": {},
 882 |    "outputs": [
 883 |     {
 884 |      "name": "stdout",
 885 |      "output_type": "stream",
 886 |      "text": [
 887 |       "{'.': 0, 'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8, 'i': 9, 'j': 10, 'k': 11, 'l': 12, 'm': 13, 'n': 14, 'o': 15, 'p': 16, 'q': 17, 'r': 18, 's': 19, 't': 20, 'u': 21, 'v': 22, 'w': 23, 'x': 24, 'y': 25, 'z': 26}\n",
 888 |       "{0: '.', 1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f', 7: 'g', 8: 'h', 9: 'i', 10: 'j', 11: 'k', 12: 'l', 13: 'm', 14: 'n', 15: 'o', 16: 'p', 17: 'q', 18: 'r', 19: 's', 20: 't', 21: 'u', 22: 'v', 23: 'w', 24: 'x', 25: 'y', 26: 'z'}\n"
 889 |      ]
 890 |     }
 891 |    ],
 892 |    "source": [
 893 |     "char_to_id = {c:i for i, c in enumerate(chars)}\n",
 894 |     "id_to_char = {v:k for k, v in char_to_id.items()}\n",
 895 |     "print(char_to_id)\n",
 896 |     "print(id_to_char)"
 897 |    ]
 898 |   },
 899 |   {
 900 |    "cell_type": "code",
 901 |    "execution_count": 22,
 902 |    "metadata": {},
 903 |    "outputs": [
 904 |     {
 905 |      "name": "stdout",
 906 |      "output_type": "stream",
 907 |      "text": [
 908 |       "11\n"
 909 |      ]
 910 |     }
 911 |    ],
 912 |    "source": [
 913 |     "print(max_length)"
 914 |    ]
 915 |   },
 916 |   {
 917 |    "cell_type": "markdown",
 918 |    "metadata": {},
 919 |    "source": [
 920 |     "## 4. Define dataset"
 921 |    ]
 922 |   },
 923 |   {
 924 |    "cell_type": "code",
 925 |    "execution_count": 23,
 926 |    "metadata": {},
 927 |    "outputs": [],
 928 |    "source": [
 929 |     "class NamesDataset(Dataset):\n",
 930 |     "    \n",
 931 |     "    def __init__(self, names_list):\n",
 932 |     "        self.names_list = names_list\n",
 933 |     "        \n",
 934 |     "    def __len__(self):\n",
 935 |     "        return len(self.names_list)\n",
 936 |     "    \n",
 937 |     "    def __getitem__(self, idx):\n",
 938 |     "        x_str = self.names_list[idx].ljust(max_length, \".\")[:max_length]\n",
 939 |     "        y_str = x_str[1:] + \".\"\n",
 940 |     "        \n",
 941 |     "        x = torch.zeros((max_length, num_chars))\n",
 942 |     "        y = torch.zeros(max_length)\n",
 943 |     "        for i, c in enumerate(x_str):\n",
 944 |     "            x[i, char_to_id[c]] = 1\n",
 945 |     "        for i, c in enumerate(y_str):\n",
 946 |     "            y[i] = char_to_id[c]\n",
 947 |     "            \n",
 948 |     "        return x, y"
 949 |    ]
 950 |   },
 951 |   {
 952 |    "cell_type": "code",
 953 |    "execution_count": 24,
 954 |    "metadata": {},
 955 |    "outputs": [],
 956 |    "source": [
 957 |     "trainset = NamesDataset(names_list)"
 958 |    ]
 959 |   },
 960 |   {
 961 |    "cell_type": "markdown",
 962 |    "metadata": {},
 963 |    "source": [
 964 |     "## 5. Define dataloader"
 965 |    ]
 966 |   },
 967 |   {
 968 |    "cell_type": "code",
 969 |    "execution_count": 25,
 970 |    "metadata": {},
 971 |    "outputs": [],
 972 |    "source": [
 973 |     "train_batch_size = 256"
 974 |    ]
 975 |   },
 976 |   {
 977 |    "cell_type": "code",
 978 |    "execution_count": 26,
 979 |    "metadata": {},
 980 |    "outputs": [
 981 |     {
 982 |      "name": "stdout",
 983 |      "output_type": "stream",
 984 |      "text": [
 985 |       "8\n"
 986 |      ]
 987 |     }
 988 |    ],
 989 |    "source": [
 990 |     "cpu_count = os.cpu_count()\n",
 991 |     "print(cpu_count)"
 992 |    ]
 993 |   },
 994 |   {
 995 |    "cell_type": "code",
 996 |    "execution_count": 27,
 997 |    "metadata": {},
 998 |    "outputs": [
 999 |     {
1000 |      "name": "stdout",
1001 |      "output_type": "stream",
1002 |      "text": [
1003 |       "391\n"
1004 |      ]
1005 |     }
1006 |    ],
1007 |    "source": [
1008 |     "train_loader = DataLoader(trainset, batch_size=train_batch_size, shuffle=True, num_workers=cpu_count)\n",
1009 |     "print(len(train_loader))"
1010 |    ]
1011 |   },
1012 |   {
1013 |    "cell_type": "code",
1014 |    "execution_count": 28,
1015 |    "metadata": {},
1016 |    "outputs": [
1017 |     {
1018 |      "name": "stdout",
1019 |      "output_type": "stream",
1020 |      "text": [
1021 |       "torch.Size([256, 11, 27]) torch.Size([256, 11])\n"
1022 |      ]
1023 |     }
1024 |    ],
1025 |    "source": [
1026 |     "train_iter = iter(train_loader)\n",
1027 |     "X, Y = train_iter.next()\n",
1028 |     "print(X.size(), Y.size())"
1029 |    ]
1030 |   },
1031 |   {
1032 |    "cell_type": "markdown",
1033 |    "metadata": {},
1034 |    "source": [
1035 |     "## 6. Define model"
1036 |    ]
1037 |   },
1038 |   {
1039 |    "cell_type": "code",
1040 |    "execution_count": 29,
1041 |    "metadata": {},
1042 |    "outputs": [],
1043 |    "source": [
1044 |     "input_size = num_chars\n",
1045 |     "hidden_size = 54\n",
1046 |     "output_size = num_chars\n",
1047 |     "num_layers = 1"
1048 |    ]
1049 |   },
1050 |   {
1051 |    "cell_type": "code",
1052 |    "execution_count": 30,
1053 |    "metadata": {},
1054 |    "outputs": [],
1055 |    "source": [
1056 |     "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\"\n",
1057 |     "device = torch.device(device)"
1058 |    ]
1059 |   },
1060 |   {
1061 |    "cell_type": "code",
1062 |    "execution_count": 31,
1063 |    "metadata": {},
1064 |    "outputs": [],
1065 |    "source": [
1066 |     "class Model(nn.Module):\n",
1067 |     "    \n",
1068 |     "    def __init__(self, input_size, hidden_size, output_size, num_layers):\n",
1069 |     "        super(Model, self).__init__()\n",
1070 |     "        self.input_size = input_size\n",
1071 |     "        self.hidden_size = hidden_size\n",
1072 |     "        self.num_layers = num_layers\n",
1073 |     "        self.lstm1 = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=True)\n",
1074 |     "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
1075 |     "        self.fc3 = nn.Linear(output_size, output_size)\n",
1076 |     "        \n",
1077 |     "    def forward(self, X, states):\n",
1078 |     "        ht, ct = states\n",
1079 |     "        batch_size = X.size(0)\n",
1080 |     "        out, (ht, ct) = self.lstm1(X, (ht, ct))\n",
1081 |     "        out = F.relu(self.fc2(out))\n",
1082 |     "        out = self.fc3(out)\n",
1083 |     "        return out, (ht, ct) # out: Size([batch_size, max_length, num_chars])"
1084 |    ]
1085 |   },
1086 |   {
1087 |    "cell_type": "code",
1088 |    "execution_count": 32,
1089 |    "metadata": {},
1090 |    "outputs": [],
1091 |    "source": [
1092 |     "model = Model(input_size=input_size, hidden_size=hidden_size, output_size=output_size, num_layers=num_layers)\n",
1093 |     "model = nn.DataParallel(model)\n",
1094 |     "model = model.to(device)"
1095 |    ]
1096 |   },
1097 |   {
1098 |    "cell_type": "code",
1099 |    "execution_count": 33,
1100 |    "metadata": {},
1101 |    "outputs": [],
1102 |    "source": [
1103 |     "#list(model.parameters())"
1104 |    ]
1105 |   },
1106 |   {
1107 |    "cell_type": "code",
1108 |    "execution_count": 34,
1109 |    "metadata": {},
1110 |    "outputs": [],
1111 |    "source": [
1112 |     "ht = torch.zeros((num_layers, train_batch_size, hidden_size)).to(device)\n",
1113 |     "ct = torch.zeros((num_layers, train_batch_size, hidden_size)).to(device)\n",
1114 |     "writer.add_graph(model, (X, (ht, ct)))\n",
1115 |     "writer.close()"
1116 |    ]
1117 |   },
1118 |   {
1119 |    "cell_type": "code",
1120 |    "execution_count": 35,
1121 |    "metadata": {},
1122 |    "outputs": [],
1123 |    "source": [
1124 |     "#summary(model, input_size=(max_length, num_chars))"
1125 |    ]
1126 |   },
1127 |   {
1128 |    "cell_type": "markdown",
1129 |    "metadata": {},
1130 |    "source": [
1131 |     "## 7. Set optimizer"
1132 |    ]
1133 |   },
1134 |   {
1135 |    "cell_type": "code",
1136 |    "execution_count": 36,
1137 |    "metadata": {},
1138 |    "outputs": [
1139 |     {
1140 |      "name": "stdout",
1141 |      "output_type": "stream",
1142 |      "text": [
1143 |       "391\n"
1144 |      ]
1145 |     }
1146 |    ],
1147 |    "source": [
1148 |     "lr = 0.005\n",
1149 |     "step_size = len(train_loader) * 1\n",
1150 |     "gamma = 0.95\n",
1151 |     "print(step_size)"
1152 |    ]
1153 |   },
1154 |   {
1155 |    "cell_type": "code",
1156 |    "execution_count": 37,
1157 |    "metadata": {},
1158 |    "outputs": [],
1159 |    "source": [
1160 |     "criterion = nn.CrossEntropyLoss(reduction='mean')\n",
1161 |     "optimizer = optim.Adam(model.parameters(), lr=lr)\n",
1162 |     "lr_scheduler = optim.lr_scheduler.StepLR(optimizer=optimizer, step_size=step_size, gamma=gamma)"
1163 |    ]
1164 |   },
1165 |   {
1166 |    "cell_type": "markdown",
1167 |    "metadata": {},
1168 |    "source": [
1169 |     "## 8. Define sampler"
1170 |    ]
1171 |   },
1172 |   {
1173 |    "cell_type": "code",
1174 |    "execution_count": 38,
1175 |    "metadata": {},
1176 |    "outputs": [],
1177 |    "source": [
1178 |     "def generate_name(model, start='a', k=5):\n",
1179 |     "    \n",
1180 |     "    if len(start) >= max_length:\n",
1181 |     "        return name\n",
1182 |     "    \n",
1183 |     "    with torch.no_grad():\n",
1184 |     "        \n",
1185 |     "        ht = torch.zeros((num_layers, 1, hidden_size)).to(device)\n",
1186 |     "        ct = torch.zeros((num_layers, 1, hidden_size)).to(device)\n",
1187 |     "        length = 0\n",
1188 |     "        name = start\n",
1189 |     "        \n",
1190 |     "        for char in start:\n",
1191 |     "            X = torch.zeros((1, 1, num_chars)) # [batch_size, timestep, num_chars]\n",
1192 |     "            X[0, 0, char_to_id[char]] = 1\n",
1193 |     "            out, (ht, ct) = model(X, (ht, ct))\n",
1194 |     "            length += 1\n",
1195 |     "        vals, idxs = torch.topk(out[0], k) # 0 -> first eg in a batch\n",
1196 |     "        idx = np.random.choice(idxs.cpu().numpy()[0]) # 0 -> first...\n",
1197 |     "        char = id_to_char[idx]\n",
1198 |     "        vals, idxs = torch.topk(out[0], k) # 0 -> first eg in a batch\n",
1199 |     "        idx = np.random.choice(idxs.cpu().numpy()[0]) # 0 -> first...\n",
1200 |     "        char = id_to_char[idx]\n",
1201 |     "        \n",
1202 |     "        while char != \".\" and length <= max_length-1:\n",
1203 |     "            X = torch.zeros((1, 1, num_chars)) # [batch_size, timestep, num_chars]\n",
1204 |     "            X[0, 0, char_to_id[char]] = 1\n",
1205 |     "            out, (ht, ct) = model(X, (ht, ct))\n",
1206 |     "            vals, idxs = torch.topk(out[0], k) # 0 -> first eg in a batch\n",
1207 |     "            idx = np.random.choice(idxs.cpu().numpy()[0]) # 0 -> first...\n",
1208 |     "            char = id_to_char[idx]\n",
1209 |     "            length += 1\n",
1210 |     "            name += char\n",
1211 |     "    \n",
1212 |     "        if name[-1] != \".\":\n",
1213 |     "            name += \".\"\n",
1214 |     "    \n",
1215 |     "    return name"
1216 |    ]
1217 |   },
1218 |   {
1219 |    "cell_type": "code",
1220 |    "execution_count": 39,
1221 |    "metadata": {},
1222 |    "outputs": [],
1223 |    "source": [
1224 |     "def sampler(model, start='a', n=10, k=5, only_new=False):\n",
1225 |     "    \n",
1226 |     "    names = []\n",
1227 |     "    cnt = 0\n",
1228 |     "    while cnt <= n:\n",
1229 |     "        name = generate_name(model=model, start=start, k=k)\n",
1230 |     "        if only_new: \n",
1231 |     "            if name not in names_list and name not in names:\n",
1232 |     "                names.append(name)\n",
1233 |     "                cnt += 1\n",
1234 |     "        else:\n",
1235 |     "            if name not in names:\n",
1236 |     "                names.append(name)\n",
1237 |     "                cnt += 1\n",
1238 |     "    names = [name[:-1].title() for name in names]\n",
1239 |     "    \n",
1240 |     "    return names"
1241 |    ]
1242 |   },
1243 |   {
1244 |    "cell_type": "markdown",
1245 |    "metadata": {},
1246 |    "source": [
1247 |     "## 9. Train model"
1248 |    ]
1249 |   },
1250 |   {
1251 |    "cell_type": "code",
1252 |    "execution_count": 40,
1253 |    "metadata": {},
1254 |    "outputs": [],
1255 |    "source": [
1256 |     "epochs = 50\n",
1257 |     "print_every_n_epochs = epochs // 10"
1258 |    ]
1259 |   },
1260 |   {
1261 |    "cell_type": "code",
1262 |    "execution_count": null,
1263 |    "metadata": {},
1264 |    "outputs": [],
1265 |    "source": [
1266 |     "epoch_losses = []\n",
1267 |     "epoch_lrs = []\n",
1268 |     "iteration_losses = []\n",
1269 |     "iteration_lrs = []\n",
1270 |     "\n",
1271 |     "for epoch in tqdm(range(1, epochs+1), desc=\"Epochs\"):\n",
1272 |     "    epoch_loss = 0\n",
1273 |     "    epoch_lr = 0\n",
1274 |     "    \n",
1275 |     "    for i, (X, Y) in tqdm(enumerate(train_loader, 1), total=len(train_loader), desc=\"Epoch-{}\".format(epoch)):\n",
1276 |     "    #for i, (X, Y) in enumerate(train_loader, 1):\n",
1277 |     "        X, Y = X.to(device), Y.to(device)\n",
1278 |     "        \n",
1279 |     "        ht = torch.zeros((num_layers, X.size(0), hidden_size)).to(device)\n",
1280 |     "        ct = torch.zeros((num_layers, X.size(0), hidden_size)).to(device)\n",
1281 |     "\n",
1282 |     "        optimizer.zero_grad()\n",
1283 |     "        Y_pred_logits, (ht, ct) = model(X, (ht, ct))\n",
1284 |     "        Y_pred_logits = Y_pred_logits.transpose(1, 2) # Check Loss Doc: [N, d1, C] -> [N, C, d1]\n",
1285 |     "        loss = criterion(Y_pred_logits, Y.long())\n",
1286 |     "        loss.backward(retain_graph=True)\n",
1287 |     "        optimizer.step()\n",
1288 |     "        lr_scheduler.step()\n",
1289 |     "        \n",
1290 |     "        iteration_losses.append(loss.item())\n",
1291 |     "        iteration_lrs.append(lr_scheduler.get_lr()[0])\n",
1292 |     "        epoch_loss += loss.item()\n",
1293 |     "        epoch_lr += lr_scheduler.get_lr()[0]\n",
1294 |     "        \n",
1295 |     "    epoch_loss /= len(train_loader)\n",
1296 |     "    epoch_lr /= len(train_loader)\n",
1297 |     "    epoch_losses.append(epoch_loss)\n",
1298 |     "    epoch_lrs.append(epoch_lr)\n",
1299 |     "    \n",
1300 |     "    if epoch % print_every_n_epochs == 0:    \n",
1301 |     "        message = \"Epoch:{}    Loss:{}    LR:{}\".format(epoch, epoch_loss, epoch_lr)\n",
1302 |     "        print(message)\n",
1303 |     "        names = sampler(model, start='jo', n=10, k=10, only_new=False)\n",
1304 |     "        print(names)"
1305 |    ]
1306 |   },
1307 |   {
1308 |    "cell_type": "code",
1309 |    "execution_count": 42,
1310 |    "metadata": {},
1311 |    "outputs": [
1312 |     {
1313 |      "data": {
1314 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3YAAAHwCAYAAADq2/1hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxcdb3/8fdntixN0qRt0iRtutI2baEUCBRkq1QUEQU3Fr24FXFFr97rVa+/34Uf93oR9SqiIiBykXoFuSpaWZVCqbI2ldLSJd1oSJq2SdukabNNMvP9/TGTNlvXZOZkMq/n4zGPzDnn+z3nM2F4wDvfc75fc84JAAAAAJC6fF4XAAAAAAAYHIIdAAAAAKQ4gh0AAAAApDiCHQAAAACkOIIdAAAAAKQ4gh0AAAAApDiCHQAAAACkOIIdACBtmdl2M3uH13UAADBYBDsAAAAASHEEOwAA+jCzT5vZFjPbZ2ZLzaw0vt/M7IdmVm9mzWa21sxOjR+73MzWm9kBM9thZv/s7acAAKQTgh0AAD2Y2SWSbpN0taQSSdWSHo4ffqekiyTNlDQ63mZv/NgvJH3GOZcr6VRJzyaxbABAmgt4XQAAAMPMRyXd75z7uySZ2TclNZrZFEmdknIllUt61Tm3oUe/TklzzOx151yjpMakVg0ASGuM2AEA0FupYqN0kiTn3EHFRuUmOOeelfQTST+VVG9m95pZXrzpByVdLqnazJ43s/OSXDcAII0R7AAA6K1O0uTuDTMbJWmspB2S5Jy70zl3lqQ5it2S+bX4/pXOuSslFUn6g6RHklw3ACCNEewAAOkuaGaZ3S9JD0n6pJnNN7MMSf8p6RXn3HYzO9vMFphZUFKLpHZJUTMLmdlHzWy0c65TUrOkqGefCACQdgh2AIB094Skth6vhZL+r6TfSdopabqka+Nt8yT9XLHn56oVu0Xze/Fj10vabmbNkj6r2LN6AAAkhTnnvK4BAAAAADAIjNgBAAAAQIoj2AEAAABAiiPYAQAAAECKI9gBAAAAQIoj2AEAAABAigsk4qRmdr+kKyTVO+dOHeD4lZL+XbE1frok/aNz7m/HOu+4cePclClThrhaAAAAAEgNq1at2uOcK+y7PyHLHZjZRZIOSnrwCMEuR1KLc86Z2TxJjzjnyo913oqKCldZWTnk9QIAAABAKjCzVc65ir77E3IrpnNuhaR9Rzl+0B1OlKMksZgeAAAAAJwkz56xM7P3m9lGSY9L+pRXdQAAAABAqvMs2DnnHo3ffnmVYs/bDcjMbjSzSjOrbGhoSF6BAAAAAJAiPJ8VM37b5jQzG3eE4/c65yqccxWFhf2eEQQAAACAtJeQWTGPxcxOkbQ1PnnKmZIyJO31opbBiESdllfVa11ds+aW5mnhrCL5feZ1WQAAAADSTKKWO3hI0kJJ48ysVtLNkoKS5Jy7W9IHJX3MzDoltUm6xiVies4EikSdrv/FK1pd06S2cERZIb/ml+VryeIFhDsAAAAASZWQYOecu+4Yx2+XdHsirp0sy6vq9dpbjWrrjEqSWsMRra5p0vKqei2aPd7j6gAAAACkE8+fsUtV6+qa1R4Pdd3awhGtr2v2qCIAAAAA6Ypgd5LmluYpK+TvtS8r5Nec0jyPKgIAAACQrgh2J2nhrCLNL8uXP/44XXb8GbuFs4q8LQwAAABA2iHYnSS/z7Rk8QJdPq9EJun7Hz6diVMAAAAAeIJgNwh+n+k9p5XISZpYkEWoAwAAAOAJgt0glRfHnqnbuPOAx5UAAAAASFcEu0GaNCZbWUG/NuxiNkwAAAAA3iDYDZLPZ5pZnMuIHQAAAADPEOyGwOziXG3c1SznnNelAAAAAEhDBLshUF6cq8bWTtUf6PC6FAAAAABpiGA3BMpL4hOo7OJ2TAAAAADJR7AbAuXFuZKkjTuZQAUAAABA8hHshkB+dkglozMZsQMAAADgCYLdECkvztUGRuwAAAAAeIBgN0TKS/K0teGgwl1Rr0sBAAAAkGYIdkOkvDhXnRGnbXsOel0KAAAAgDRDsBsi5cXxmTFZqBwAAABAkiUk2JnZ/WZWb2ZvHOH4R81sjZmtNbMXzez0RNSRTNMKRynoN23YxXN2AAAAAJIrUSN2D0i67CjH35R0sXPuNEn/LuneBNWRNEG/T6cU5aqKmTEBAAAAJFlCgp1zboWkfUc5/qJzrjG++bKkiYmoI9lmF+dyKyYAAACApBsOz9gtlvSk10UMhfKSXO1qbldjS9jrUgAAAACkEU+DnZm9XbFg9/WjtLnRzCrNrLKhoSF5xZ2EQxOocDsmAAAAgCTyLNiZ2TxJ90m60jm390jtnHP3OucqnHMVhYWFySvwJJQX50qSNjKBCgAAAIAk8iTYmdkkSb+XdL1zbpMXNSRCYW6GxowK8ZwdAAAAgKQKJOKkZvaQpIWSxplZraSbJQUlyTl3t6R/kzRW0l1mJkldzrmKRNSSTGam8uJcRuwAAAAAJFVCgp1z7rpjHL9B0g2JuLbXyovz9OtXqxWJOvl95nU5AAAAANLAcJgVc0QpL8lVe2dUb+1r9boUAAAAAGmCYDfEZnfPjLmT2zEBAAAAJAfBbojNGJ8jn0kbWPIAAAAAQJIQ7IZYZtCvKeNGMWIHAAAAIGkIdgkwuziPRcoBAAAAJA3BLgHKi3P11r5WHezo8roUAAAAAGmAYJcA5SWxCVSqGLUDAAAAkAQEuwQoL86VRLADAAAAkBwEuwSYWJClnIyANu5iAhUAAAAAiUewSwAzU3lxrjbuZMQOAAAAQOIR7BKkvCRXG3Y1yznndSkAAAAARjiCXYLMKs7TgfYu1e1v97oUAAAAACMcwS5BZscnUGGhcgAAAACJRrBLkJndwY6ZMQEAAAAkGMEuQfIyg5pYkKUNjNgBAAAASDCCXQKVF+exlh0AAACAhCPYJdDsklxt29Oi9s6I16UAAAAAGMESEuzM7H4zqzezN45wvNzMXjKzDjP750TUMByUF+cpEnXaUn/Q61IAAAAAjGCJGrF7QNJlRzm+T9KXJH0/QdcfFmYxgQoAAACAJEhIsHPOrVAsvB3peL1zbqWkzkRcf7iYMjZbGQEfSx4AAAAASCiesUuggN+nmeNzGbEDAAAAkFDDPtiZ2Y1mVmlmlQ0NDV6Xc8LKi3O1cRcjdgAAAAASZ9gHO+fcvc65CudcRWFhodflnLDykjztORhWw4EOr0sBAAAAMEIN+2CX6mbHJ1BhPTsAAAAAiRJIxEnN7CFJCyWNM7NaSTdLCkqSc+5uMyuWVCkpT1LUzP5R0hzn3Ii7Z/HwzJjNumDGOI+rAQAAADASJSTYOeeuO8bxXZImJuLaw83YnAwV5WZow05G7AAAAAAkBrdiJsEsJlABAAAAkEAEuySYXZKnzbsPqisS9boUAAAAACMQwS4JyotzFY5E9eaeFq9LAQAAADACEeySoLw4T5K0gZkxAQAAACQAwS4JpheNUsBnquI5OwAAAAAJQLBLgoyAX9MLc7SRmTEBAAAAJADBLknKS3K1kVsxAQAAACQAwS5JZhXnakdTm/a3dXpdCgAAAIARhmCXJLPjE6hUMWoHAAAAYIgR7JKkvCRXklioHAAAAMCQI9glSXFepkZnBbWBCVQAAAAADDGCXZKYmcqLcxmxAwAAADDkCHZJNLskT5t2HVA06rwuBQAAAMAIQrBLovLiXLWEI6ptbPO6FAAAAAAjCMEuiWYVxyZQ2cDtmAAAAACGEMEuiWaOz5WZtJEJVAAAAAAMIYJdEo3KCGjymGwmUAEAAAAwpAh2SVZenKeNLFIOAAAAYAglJNiZ2f1mVm9mbxzhuJnZnWa2xczWmNmZiahjOCovydX2vS1qDXd5XQoAAACAESJRI3YPSLrsKMffLWlG/HWjpJ8lqI5hp7w4T85Jm3Yf9LoUAAAAACNEQoKdc26FpH1HaXKlpAddzMuS8s2sJBG1DDezS2IzY1bxnB0AAACAIeLVM3YTJNX02K6N7+vHzG40s0ozq2xoaEhKcYlUVpCt7JBfG5gZEwAAAMAQGfaTpzjn7nXOVTjnKgoLC70uZ9B8PtPM8bnMjAkAAABgyHgV7HZIKuuxPTG+Ly3MLsnVxl0H5JzzuhQAAAAAI4BXwW6ppI/FZ8c8V9J+59xOj2pJuvLiPDW1dmp3c4fXpQAAAAAYAQKJOKmZPSRpoaRxZlYr6WZJQUlyzt0t6QlJl0vaIqlV0icTUcdwVV4cm0Blw65mFY/O9LgaAAAAAKkuIcHOOXfdMY47SV9IxLVTQXlxniRp484DevusIo+rAQAAAJDqhv3kKSPR6OygSkdnsuQBAAAAgCFBsPNIeUmeNu5iyQMAAAAAg0ew88is4lxtqT+ocFfU61IAAAAApDiCnUfKi3PVFXXa2nDQ61IAAAAApDiCnUdml8QnUOE5OwAAAACDRLDzyKQx2Qr4TP/z8ltatmG3IlEWKwcAAABwcgh2HohEnT71wEpFnFNldaNueug1Xf+LVwh3AAAAAE4Kwc4Dy6vqtbqmSS6e41rDEa2uadLyqnpvCwMAAACQkgh2HlhX16y2cKTXvrZwROvreN4OAAAAwIkj2HlgbmmeskL+XvuyQn7NKc3zqCIAAAAAqYxg54GFs4o0vyxf2T3C3cSCLC2cVeRhVQAAAABSVcDrAtKR32dasniBllfVa92OZv3Pq9XymclnXlcGAAAAIBUxYucRv8+0aPZ4fekdM/RP75yljbsOaMXmPV6XBQAAACAFEeyGgavmT1BxXqZ+tnyL16UAAAAASEEEu2EgFPDphgun6uVt+/T3txq9LgcAAABAiiHYDRPXnjNJo7OCunv5Vq9LAQAAAJBiCHbDRE5GQB8/b7L+vH63ttQf8LocAAAAACkkIcHOzC4zsyoz22Jm3xjg+GQzW2Zma8xsuZlNTEQdqebjb5uizKBP9zy/zetSAAAAAKSQIQ92ZuaX9FNJ75Y0R9J1ZjanT7PvS3rQOTdP0q2SbhvqOlLR2JwMXVNRpj+s3qGd+9u8LgcAAABAikjEiN05krY457Y558KSHpZ0ZZ82cyQ9G3//3ADH09YNF05T1Em/+OubXpcCAAAAIEUkIthNkFTTY7s2vq+n1yV9IP7+/ZJyzWxsAmpJOWVjsvW+00v161ffUlNr2OtyAAAAAKQAryZP+WdJF5vZa5IulrRDUmSghmZ2o5lVmlllQ0NDMmv0zGcunqbWcEQPvlTtdSkAAAAAUkAigt0OSWU9tifG9x3inKtzzn3AOXeGpG/F9zUNdDLn3L3OuQrnXEVhYWECyh1+yovzdEl5kR54cbvawgPmXQAAAAA4JBHBbqWkGWY21cxCkq6VtLRnAzMbZ2bd1/6mpPsTUEdK+9zC6drXEtYjlTXHbgwAAAAgrQ15sHPOdUn6oqSnJW2Q9Ihzbp2Z3Wpm74s3Wyipysw2SRov6dtDXUeqO3vKGFVMLtC9K7apMxL1uhwAAAAAw1hCnrFzzj3hnJvpnJvunPt2fN+/OeeWxt//1jk3I97mBudcRyLqSHWfvXi6djS16bE1dV6XAgAAAGAY82ryFByHS8qLNHN8ju5evk3OOa/LAQAAADBMEeyGMZ/P9NmLp6tq9wE9V1XvdTkAAAAAhimC3TD33tNLNSE/Sz9bvtXrUgAAAAAMUwS7YS7o9+mGC6dq5fZGVW7f53U5AAAAAIYhgl0KuObsMhVkB3X384zaAQAAAOiPYJcCskMBfeJtU/XMhnpV7TrgdTkAAAAAhhmCXYr42HmTlRX0654VjNoBAAAA6I1glyIKRoV03TmTtHR1nWobW70uBwAAAMAwQrBLITdcOFWSdN9f3/S4EgAAAADDCcEuhZTmZ+mqMybo4ZVvaV9L2OtyAAAAAAwTBLsU89mLp6m9M6pfvrjd61IAAAAADBMEuxRzSlGuLp0zXr98abtaOrq8LgcAAADAMECwS0GfWzhdTa2duvWx9bpz2WYt27BbkajzuiwAAAAAHgl4XQBO3OkT85WXGdAjK2skSVkhv+aX5WvJ4gXy+8zj6gAAAAAkGyN2KWh5Vb06uqJykpyk1nBEq2uatLyq3uvSAAAAAHiAYJeC1tU1K9wV7bWvLRzR+rpmjyoCAAAA4CWCXQqaW5qnrJC/176MoE9zSvM8qggAAACAlxIS7MzsMjOrMrMtZvaNAY5PMrPnzOw1M1tjZpcnoo6RauGsIs0vy1d2yC+TZJK6Ik7TxuV4XRoAAAAAD5hzQzubopn5JW2SdKmkWkkrJV3nnFvfo829kl5zzv3MzOZIesI5N+VY566oqHCVlZVDWm+qikSdllfVa31dswpGhfS9pzdq7KgM/e5zb1PBqJDX5QEAAABIADNb5Zyr6Ls/ESN250ja4pzb5pwLS3pY0pV92jhJ3fcNjpZUl4A6RjS/z7Ro9njdtGiG/uHcybrv42ertqlNn36wUu2dEa/LAwAAAJBEiQh2EyTV9Niuje/r6RZJ/2BmtZKekHRTAupIK2dPGaMfXH26Kqsb9U//+7qirGsHAAAApA2vJk+5TtIDzrmJki6XtMTMBqzFzG40s0ozq2xoaEhqkanminml+tfLy/X4mp26/amNXpcDAAAAIEkSEex2SCrrsT0xvq+nxZIekSTn3EuSMiWNG+hkzrl7nXMVzrmKwsLCBJQ7snz6wmm6/tzJumfFNi15abvX5QAAAABIgkQEu5WSZpjZVDMLSbpW0tI+bd6StEiSzGy2YsGO4bghYGa6+b1z9I7ZRbp56To9s3631yUBAAAASLAhD3bOuS5JX5T0tKQNkh5xzq0zs1vN7H3xZv8k6dNm9rqkhyR9wg319JxpLOD36c7rztCpE0brpode05raJq9LAgAAAJBAQ77cQSKx3MGJqT/Qrg/c9aLaOyN69PPnq2xMttclAQAAABiEZC53gGGiKDdTD3zybHVGnD7x369qf2un1yUBAAAASACC3Qh3SlGu7r3+LNXsa9Onl1Sqo4s17gAAAICRhmCXBhZMG6vvfXieXn1zn772v2tY4w4AAAAYYQJeF4DkuHL+BO1oatN3n6rSxIIs/ctl5V6XBAAAAGCIEOzSyOcunq7axjbdtXyrDrR3qjA3U3NL87RwVpH8PvO6PAAAAAAniWCXRsxMN18xR0+u3aklL78lk5QV8mt+Wb6WLF5AuAMAAABSFM/YpZm/bdmjjq6oJMlJag1HtLqmScur6r0tDAAAAMBJI9ilmXV1zWoL954ZszUc0fObGjyqCAAAAMBgEezSzNzSPGWF/P32/+rlat3xzCaF46N5AAAAAFIHwS7NLJxVpPll+coO+WWSskN+nTOlQFfMK9Edz2zW+37yN72xY7/XZQIAAAA4AeZc6qxpVlFR4SorK70uI+VFok7Lq+q1vq5Zc3rMivmX9bv1r4+u1b6WsD6/cLq+eMkpygj0H90DAAAA4A0zW+Wcq+i3n2CHnppaw7r1sfX6/d93aNb4XH3vw/M0b2K+12UBAAAA0JGDHbdiopf87JB+cPV83f+JCjW1hfX+u17Ud5/aqPbOyLE7AwAAAPAEwQ4DuqR8vP78lYv1wTMn6K7lW3XFj/+m195q9LosAAAAAAMg2OGIRmcF9d0Pna4HPnm2Wjq69MGfvajbntiglo4uLduwW3cu26xlG3YrEk2d23kBAACAkYhn7HBcmts7ddsTG/TQqzXKDPokJ3V0RZUV8mt+Wb6WLF4gv8+8LhMAAAAY0XjGDoOSlxnUbR+Yp69eOkMdnVG1d0XlFFvcfHVNk5ZX1XtdIgAAAJC2CHY4Qf1H5VrDEf18xTbV7Gv1oB4AAAAACQl2ZnaZmVWZ2RYz+8YAx39oZqvjr01m1pSIOjD05pbmKSvUe207n0kvv7lPF373OX3k5y/rj6t3MIsmAAAAkESBoT6hmfkl/VTSpZJqJa00s6XOufXdbZxzX+nR/iZJZwx1HUiMhbOKNL8sX6trmtQWjhx6xu72D87To6/t0COVNfryw6uVlxnQVWdM0NUVZTp1wmivywYAAABGtCGfPMXMzpN0i3PuXfHtb0qSc+62I7R/UdLNzrm/HOvcTJ4yPESiTsur6rW+rllzSvO0cFbRoYlTolGnl7bt1W9W1uipdbsU7opqbmmerjm7TFeePkE5mQEtr6rXurpmze3TFwAAAMDRHWnylEQEuw9Jusw5d0N8+3pJC5xzXxyg7WRJL0ua6Jwb8N49M7tR0o2SNGnSpLOqq6uHtF4kzv7WTv3x9R36zcoaratrVtBvGp0V1IH2LoWZURMAAAA4YcN1VsxrJf32SKFOkpxz9zrnKpxzFYWFhUksDYM1Ojuoj503RY9/6UI9dtMFunDGOO05GFZHjxk1V1U36vG1O70uFQAAAEhpiQh2OySV9dieGN83kGslPZSAGjDMnDphtOaXFfSbU7OjK6ovP/yaPvizF3XHM5v097caWfAcAAAAOEFDPnmKpJWSZpjZVMUC3bWSPtK3kZmVSyqQ9FICasAw1D2jZmv48ABtRsCnS+eMV01jm360bLPueGaz8jIDumDGOF00o1AXzSxUaX7Wofbdz/fxjB4AAABw2JAHO+dcl5l9UdLTkvyS7nfOrTOzWyVVOueWxpteK+lhN9QP+WHYOtKMmj+69gz5fabGlrD+tmWP/rq5QSs27dETa3dJkqYXjtJFMwt1wfRx+vnftmlN7f5e/XlGDwAAAOluyCdPSSRmxUx9R5tRsyfnnDbXH9SKTQ1asXmPXtm2Vx1d0X7tsoI+/fi6M/SOOcXJKB8AAADwVNJmxUwkgl36au+M6FuPrtXv/t7/cc2MgE9nTMrX3NLRmluapzmleZpemKOgv/cjpNzGCQAAgFR3pGCXiGfsgCGXGfTr8tNK9OQbu3o9oxfym86dNlZNbZ361cvVh0b1QgGfyotzNackT3NL8zSrOE8/+EsVt3ECAABgRCLYIWUc6Rm9+z9xtvw+U1ckqjf3tGhdXbPW1e3X+p3NemrdLj28sqbfuVrDEVVWN+rBl7br2rMnKSvkP+b1GfEDAADAcMWtmEgpx/uMXjfnnOr2t+v2Jzdq6et1R2w3IT9L0wpHaXphjqYX5Wh64SidUpijwtwMmZkiUafrf/FKv1DJiB8AAACSiVsxMSL4faZFs8dr0ezxx9XezDQhP0tXzi/VMxt297qNMzPo0yfeNlXZIb+2NhzU1oaDeqSyplebnIyApheOUnbIr8rqRnVGYn8IaQ1HtLqmScur6o+rFkb7AAAAkEgEO6SFI93G+bV3zeoVsJxz2tXcrq31LYfC3raGFq2uaToU6rq1hiP6+m/XaF5ZvibkZ2liQZYmFGRpQn7sZ2EOo30AAABIDm7FRNo40ds4e1q2Ybdueui1XqN5AZ9pbmmeOrqi2tHUpgPtXb36hAI+TcjPUlbQr027D6grevjftcygT9/5wDxdOb9UZseugRE/AAAASCx3AAzK8Yy6Nbd3akdjW+zVFH81tmlVdaN2NbcPeN6g31SYk6HCvEwV5mSoKC9DRbkZKsrNVGFu7P3YnJC+9r9r9HrtyY34EQoBAABGDoIdMEgnO+I30GhfRsCn958xQfnZITUc6FD9gfb4zw7tawkf85x+n+nCGeM0b2K+xo4KacyoUOxnTkhjskMqGBVS0O8bkttACYYAAADDB8EO8MiJhqvOSFR7DnaovrlDDQc69PDKt/TMhvp+7TKDPnV0RXWkf4XzMgPKCvnVcKBDPe4CVdBv+sg5k3Te9HHKzw6qIDukguygRmcHlRHovezDYIMhoRAAAGBoEewADw31833ZIb9+fN0ZunhmoZraOrWvJay9B8Pa1xLWvtaw9h0Ma19Lh17atlebdh887jqzQ34VZIc0OiuoglFBdXRGtbqmqdfzgRkBnz6/cLounFmovMyAcjODyskIKDvk7/W8IKOFAAAAQ49gB6SowQSkI4XCW943V3NK8rS/rVONrWE1tnZqf/xnY2tY++M/39zTosbWzuOq0+8z5WQElBsPe13RqLbVH1Skz2jhNWeXqWLyGOVmBpSTEVBOZkB58XCYkxlQ0O8b9Ofu+bs72WBIqAQAAMMRwQ5IYSc74jfYcDRQMMwM+vSP75ipWeNz1dzeqYMdXTrQ3qUD7Z3xn7HXxp3Nqm1qO+HPmhn0KScjKL9P/W4jDfhMC2cVqbw4V1khv7Ljr6xQQNnB7vd+ZYcCygj49LXfvq51dc0n/NkJlQAAYLgi2AFpajC3gSZitPDfrzpVZ5Tl60B7V69Q2P2+++eq6n0D3kYa8vvUFY32CnwnYkx8spmsYCwEZnUHwh7bu5vb9dS6Xb3WLswI+HTTJafo/FPGKTsU6BEiY32G6jZUQiUAADgagh2AkzKcRgu7ny28pLxIHV1RtYUjau2MqC3cpdZwRK3hSGxfOKI/rt6hP6/f3e+8c0ryNHXcKLXG+7R3xvt1xvq2xbdPVFaPEcOoc9q1v71X+PT7TOdOHaMp40YpI+BXKOBTRsCnjKBPIb9PGUG/Mvw+ba4/oAdfqlZHV/RQ38ygT//nPXO0cFahMoP+2CvgUyB+22pPhEoAAEY2gh2ApPNqtFA6ejBcNHv8Ufs+s36XvvTw6n63oN709hmaXZrbL0QeCpbxcLi2tklbGlr6nTc3I6BQIDababgrqnAk2q/NiQj4LB70fMoI+JUR9KmzK6odTW39QuU5Uwo0acwoBQOmkN8f/xkLlcFA7Oe2PQf1yMraXnVlBHz64iWnaMHUsQoFfAr64/0CPgX9sVco3t9n0icfWJmWoZJACwBIFoIdgJTjVTBMVqiMRp3Ckag6uqLq6Ioo3BXV8qoG/cfj69Xe2Ttcfey8yZpRlKv2rtgoY3tnrE97Z/TQdntXRBvqmrVtT/9QmZcVUHYwoM7I4VAZjhx5uYyhVJiTobysgELxkcqQ3w4Fw+5gue9gh1Zub+w1A2vQZ3r3acWaVZynoN8U8MXaBn2moN+ngN8OhUufST/48yZtbTiojq6oMoI+zRqfq9s/OE+ZQf+htgGfxc8R6x/wmaJOg17WI11HSQm0AJB8SQ12ZnaZpB9J8ku6zzn3nQHaXC3pFklO0uvOuY8c67wEOwAnYrDBcCSHSklyzikSDwivwiMAACAASURBVJedXU7LNu7Wtx59Q22dvUcqv3TJDM2bmK/OeAjtjBx+xUKiU2ckqhWbGvTi1r39appdnKtphTmH+nYHy57v9xzoUHN71zE/XyL4TP2e2TRJRXkZKsgOxcOjHRqd7LXt96nhQGxpkV6h1G+6/NQSzSzOVcBn8vtiITIQD5f+eDjdsLNZD7y4vdettxkBn760aIYWTB0T7+eL/fQfPo/PYtuS9OWHXtMbPSYJOn1ivh781DkKBvrfqtvTUKxTmaqBdrD9uTZhHPBS0oKdmfklbZJ0qaRaSSslXeecW9+jzQxJj0i6xDnXaGZFzrn+KzD3QbADkCoIlTHHe/vrkfr+6Nr5unBGobqiTl3xUcauiFNXJBZIu6Kx7V+9Uq3fvFqjvv9Fe89pJVo0u0hdEafOeNtYKI2drzPq9PLWPXp1e2O/mmaNz9GUcaPUGe8TPhRq49vxcNrYEpv8Z7gxk/wWC4M9X93BsCsSVWNrZ6/fmUkqG5Ol/OyQfN19zeTzxW7p7blvb0tYa2ubei1p4veZ3jZtrMrGZg947e591Xtb9Pjanb0mKAr5TdeePUlzJ+TJ7/P1CsSHg21sv0m67ckN2lx/UB2dsRHa8uJc3faBeQr6D9fpM5Ov+zOYDr13kj73q1Vau2N/r+/6g586Z8BnV3tK5UCcymE8lQMt1+aPCEMtmcHuPEm3OOfeFd/+piQ5527r0ea7kjY55+47kXMT7ADg2AiVMYMNlcfT92j9f3TtfF00s1CRqIsHU6euaDS2HYnte2FLg/7j8Q39br296ZJTNG9iviLOKRJvGztP9ND5olGnv6zfrWUb+/9d9LxpY3X2lIJYv/g5IvER2u7X2h37ta6uuV/faYWjNGlMtiJRp2i8TzSqQ/279+1ubteeg+F+/Udl+JUVDCgSrzUSryEa1aBmtE0Wn/UOsb54KOwOp52RqJrbuvoF4pL8TOVlBg+181k8RPcItD6fqbGlQ+vrmvsF4rMmF6hkdKZ8Fguu1h1GzWQW265ratMLW/b0Gh0O+EzvmF2kqYU5sTrjYXag+jfXH9Sjf69VuFeg9umjCybp1Amj5ffFrnW4r+L9TWt37Nfdz2/tN7r85UUzdPbUMfJZd82x+rvr9sX/MBCNSt96dK027mpWe2dUmUGf5pSM1n9dfbqCAV+va/V8bz7JOekzSyq1pvbEg7iUvmE6Xa/dfQ6vQmkyJDPYfUjSZc65G+Lb10ta4Jz7Yo82f1BsVO98xW7XvMU599Sxzk2wA4DhLR1DJYH2xPo7Fwuk//ib3hMUZQX9+o+rTtW508fGw+zhEHv4Z2zU9X9X1ep3q2r7jdBeMa9E75xbrGjPUOqcIlEp6g7vW17VoOc3NfSr7dxpY1UxuSAeRHv37Q6oa2qb9MaO/oF4RlFshDca7R+iD/eXdja1qf5AR7/+BdlBjc4KKupi13Mu9rs6tC2ppaNrwFl7u0dDu6+bbnoFwR7BvDtYdkWjauno/3sryA4eWq4mFp4lkx0KqSapNdyl3c0dvYO8SRPzszQ6O9gviB/6qdg597d2alP9gV5/0PCZNLc0T0W5mYdCe3cfn+9w3/oDHVpV3djrn2n3qPjEMVm9wr8vvuTOoWDsM721r0XPrK/vd5v4ZacWa3phTo/PqsO/g/i1t9Qf1B9X7+g3qv6hs8pUXpJ76Pdz+Pd8uK+ZacPOZi15uVrhPn8I+MT5UzRvQn78WrFrq8/vfE1tk+5Zsa3fHxG++PZTdNbkAqnPP29fj3P4zBR1TrcsXaeq3Qd6jeh/+/2nxf54Ea8z1kuH3iu+PxJ1+vrv1mjjztgfIU4mWCbacAt2j0nqlHS1pImSVkg6zTnXNMD5bpR0oyRNmjTprOrq6iGtFwAwMngVKr28NoE2uaE0Va7tegRR53To/fKNDfr679b0eo42K+jXze+do/Omjz0UImOhVodCadQ5vfLmPv3X01Vq7/s/25ecojPKCg61cz2CaTQeTJ1zenzNTj22dme/z/SuueO1qHx8LPz2aN9dQ9Q5rdjUoBWb9/Tre960sTp76ph4AI6F8O73PQPy6ppGra7Z36//3NI8zS7JOxSinQ7/7K5j0+4DA66pOq1wlKaOHXX4M+rwtXv+Dmob27Sjqa1f//G5GSrMy1A0eoS+kva1hNXU2tmvb05GbC3VQ7+rPp+3+w8C4Ug0LYN+IpzIv+PJcKRgF0jAtXZIKuuxPTG+r6daSa845zolvWlmmyTNUOx5vF6cc/dKuleKjdgloF4AwAjg95kWzR5/Uv/hHUxfL6/t95mWLF5wUsFwMH29vvbCWUWaX5bfLxgunFWU8P6pcG2LT67T93/y3jOvRA+vfKtf/w9XlB3zdz+3dLSe21jfr+/nF55yXP/cQgGfnq2q7xdKr64oO+Z3f+q4UaqsbuzX94YLpw4qTH/10pknHaa/dfnsQV37Pz9w2qCePR7Mte+8dr4uKR/fI1BKTt2hMPb+uY31+tpv1/QbVb/tA6fpghnjeoXhvmHWOemFrQ269U/r1dbZe03Wf718thZMHdvvej3fv/rmPn3/z1W9blHPDPj05XfM1JmT8nvV2zcMO+f0x9V1Wrq6rt+I/nvnlejdp5Ucmg26+xyx97G+kvT0G7v0xBu7evVtC0e0vq552AS7I0nEiF1AsdssFykW6FZK+ohzbl2PNpcpNqHKx81snKTXJM13zvWfTq0HbsUEAABS6o6ypuu103V0mWun14h+siR7uYPLJd2h2PNz9zvnvm1mt0qqdM4tNTOT9F+SLpMUkfRt59zDxzovwQ4AACA1pWOg5dqp9UeEoeifDCxQDgAAAGDE8zKUJgPBDgAAAABS3JGC3bEX/wAAAAAADGsEOwAAAABIcQQ7AAAAAEhxBDsAAAAASHEpNXmKmTVIqva6jgGMk7TH6yKQFviuIVn4riFZ+K4hmfi+IVkS+V2b7Jwr7LszpYLdcGVmlQPNTAMMNb5rSBa+a0gWvmtIJr5vSBYvvmvcigkAAAAAKY5gBwAAAAApjmA3NO71ugCkDb5rSBa+a0gWvmtIJr5vSJakf9d4xg4AAAAAUhwjdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAAAACQ4gh2AAAAAJDiCHYAAAAAkOIIdgAA9GFm283sHX32LTSzqJkdNLMDZlZlZp/0qkYAAHoi2AEAcPzqnHM5kvIkfUXSz81slsc1AQBAsAMA4ES5mCck7ZM0z+t6AAAIeF0AAACpxsx8kq6QNE7SFo/LAQCAYAcAwAkoNbMmSVmK/Tf0q8651zyuCQAAbsUEAOAE1Dnn8hV7xu5OSZd4XA8AAJIIdgAAnDDnXIekr0s6zcyu8roeAAAIdgAADCxoZpndL/V5fME5F5b0X5L+zZPqAADowZxzXtcAAMCwYmbbJU3us/sFSVOccxN7tMuW9JakTzrn/pS8CgEA6I1gBwAAAAApjlsxAQAAACDFEewAAAAAIMUR7AAAAAAgxRHsAAAAACDFEewAAAAAIMUFjt1k+Bg3bpybMmWK12UAAAAAgCdWrVq1xzlX2Hd/SgW7KVOmqLKy0usyAAAAAMATZlY90H5uxQQAAACAFEewAwAAAIAUR7ADAAAAgBR3XMHOzC4zsyoz22Jm3xjgeIaZ/SZ+/BUzm9Lj2Dfj+6vM7F099m83s7VmttrMeHAOAAAAAE7SMSdPMTO/pJ9KulRSraSVZrbUObe+R7PFkhqdc6eY2bWSbpd0jZnNkXStpLmSSiU9Y2YznXOReL+3O+f2DOHnSapI1Gl5Vb3W1TVrbmmeFs4qkt9nXpcFAAAAIM0cz6yY50ja4pzbJklm9rCkKyX1DHZXSrol/v63kn5iZhbf/7BzrkPSm2a2JX6+l4amfO9Eok7X/+IVra5pUls4oqyQX/PL8rVk8QLCHQAAAICkOp5bMSdIqumxXRvfN2Ab51yXpP2Sxh6jr5P0ZzNbZWY3nnjp3lpeVa/VNU1qDUfkJLWGI1pd06TlVfVelwYAAAAgzXg5ecoFzrkzJb1b0hfM7KKBGpnZjWZWaWaVDQ0Nya3wKNbVNastHOm1ry0c0fq6Zo8qAgAAAJCujifY7ZBU1mN7YnzfgG3MLCBptKS9R+vrnOv+WS/pUcVu0ezHOXevc67COVdRWNhvgXXPzC3NU1bI32ufk3Qw3CXnnDdFAQAAAEhLxxPsVkqaYWZTzSyk2GQoS/u0WSrp4/H3H5L0rIulm6WSro3PmjlV0gxJr5rZKDPLlSQzGyXpnZLeGPzHSZ6Fs4o0vyxf2SG/TFJm0KdRIb/ueX6bFv+yUrWNrV6XCAAAACBNHHPyFOdcl5l9UdLTkvyS7nfOrTOzWyVVOueWSvqFpCXxyVH2KRb+FG/3iGITrXRJ+oJzLmJm4yU9GptfRQFJv3bOPZWAz5cwfp9pyeIFWl5Vr/V1zZpTmqcLThmnJS9X6wd/2aRLf7BC//TOmfrE26Yo4Ge5QAAAAACJY6l022BFRYWrrBz+S97VNrbq5j+u07KN9Tp1Qp5ue/88nTZxtNdlAQAAAEhxZrbKOVfRdz9DSQkwsSBb9328Qnd99Eztbu7QlT/9m27903q1dHR5XRoAAACAEYhglyBmpstPK9EzX71Y150zSfe/8Kbe+cMVWrZht9elAQAAABhhCHYJNjorqG+//zT97nPnaVSGX4t/WanP/88q7Wxq07INu3Xnss1atmG3ItHUuSUWAAAAwPDCM3ZJFO6K6ud/3aY7ntmkaNTJ7zN1RpyyQn7NL8vXksUL5PeZ12UCAAAAGKZ4xm4YCAV8+sLbT9H/e99cOUnhiJOT1BqOaHVNk5ZX1XtdIgAAAIAURLDzwJ6DYfUdKG0NR/T36iZvCgIAAACQ0gh2HphbmqeskL/f/l++tF2PrKxRlOftAAAAAJwAgp0HFs4q0vyyfGWH/DJJ2SG/Tp84WuXFOfqX363RB372otbW7ve6TAAAAAApgslTPBKJOi2vqtf6umbNKc3TwllF8pn06Gs79J9PbNTelg595JxJ+tq7Zik/O+R1uQAAAACGgSNNnkKwG4aa2zt1x18265cvbVdeZkD/clm5rqkok48ZMwEAAIC0xqyYKSQvM6h/e+8cPf6lCzSjKFff/P1avf+uF/R6DZOrAAAAAOiPYDeMlRfn6TefOVd3XDNfdfvbddVdL+ibv1+rxpawIlHHAucAAAAAJEkBrwvA0ZmZrjpjghbNLtKPntms/35xu55YW6exORnatb9dbeEIC5wDAAAAaY4RuxSRmxnU/7lijp788oUqzsvUtoYWtYYjLHAOAAAAgGCXamaOz9Xlp5X0298Wjmh9XbMHFQEAAADwGsEuBZ06YbSy+yxw7iRt3HVALR1d3hQFAAAAwDMEuxTUd4HzrKBf43JCenztTr39+8v1SGWNokymAgAAAKQN1rFLUQMtcL6mtkm3PrZer73VpFMn5On/vmeOFkwb63WpAAAAAIYIC5SnCeeclr5ep9uf3Ki6/e1696nF+ua7Z2vS2GyvSwMAAAAwSEcKdix3MMKYma6cP0Hvmlusn6/Ypp89v1XLNtTrk+dP0RcuOUV5mUGvSwQAAAAwxBixG+F2N7fre09X6Xd/r9WY7JC++s6Z+tCZE/W3LXu0rq5Zc+O3cbL+HQAAADD8cStmmltbu1///th6vbp9n7KDfkWcU7gryuLmAAAAQAo5UrBjVsw0cdrE0frNZ87V5y+errbOiDq6oixuDgAAAIwQBLs0YmbK7LP+nRQLdy9v2+tBRQAAAACGApOnpJm5pXnKCvnVGo702v/fL2yXz2f6/MWnaHQ2E6wAAAAAqYQRuzTTd3Hz7JBfZ00u0HtPL9G9K7bpwu8+q3ue36r2zsgxzwUAAABgeGDylDQ00OLmfp9pw85m3f7URi2valDJ6Ex95dKZ+uCZE5lUBQAAABgmmBUTx+3FrXt0+5Mb9Xrtfs0cn6OvX1auS8qLZEbAAwAAALzErJg4bm+bPk5/+ML5uuujZ6oz4rT4l5W65p6Xtaq6UVJsxG/Zht26c9lmLduwW5Fo6vxxAAAAABiJGLHDUXVGovrNyhrd8cxm7TnYoXfOKVLDgbCqdh9QWzjCOngAAABAEg1qxM7MLjOzKjPbYmbfGOB4hpn9Jn78FTOb0uPYN+P7q8zsXX36+c3sNTN77MQ/EpIh6PfpH86drOe/tlBfvXSmnt+0R6/VNKk1HGEdPAAAAGCYOGawMzO/pJ9KerekOZKuM7M5fZotltTonDtF0g8l3R7vO0fStZLmSrpM0l3x83X7sqQNg/0QSLxRGQF9adEMffL8Kf2OtYUjWl/XnPyiAAAAAEg6vhG7cyRtcc5tc86FJT0s6co+ba6U9Mv4+99KWmSxmTaulPSwc67DOfempC3x88nMJkp6j6T7Bv8xkCxnTxmj7D6LnDtJ63c2q7El7E1RAAAAQJo7nmA3QVJNj+3a+L4B2zjnuiTtlzT2GH3vkPQvkqInXDU803cdvMygT+NyQnryjV268LvP6Qd/rtL+tk6vywQAAADSSsCLi5rZFZLqnXOrzGzhMdreKOlGSZo0aVISqsPR+H2mJYsX9FsHb2vDQf3wL5t057Nb9MCL23XjRdP0ifOnKifDk68YAAAAkFaOZ8Ruh6SyHtsT4/sGbGNmAUmjJe09St/zJb3PzLYrdmvnJWb2q4Eu7py71zlX4ZyrKCwsPI5ykWh+n2nR7PG6adEMLZo9Xn6faeb4XP3sH87SYzddoHOmjtH3/7xJF97+rO55fqvawhGvSwYAAABGtGMudxAPapskLVIslK2U9BHn3Loebb4g6TTn3GfN7FpJH3DOXW1mcyX9WrHn6kolLZM0wzkX6dF3oaR/ds5dcaxiWe4gdbz2VqN+8JdN+uvmPRqXk6EvvH26rjtnkoJ+n5ZX1WtdXbPmxkf7WCYBAAAAOD5HWu7gmPfJOee6zOyLkp6W5Jd0v3NunZndKqnSObdU0i8kLTGzLZL2KTYTpuLtHpG0XlKXpC/0DHUYuc6YVKAlixdo5fZ9+v7TVfp/f1qve57fquyMgHbtb2cNPAAAAGAIsUA5Es45pxe37tW//fENbW1o6XUsO+TXj687Q4tmj/eoOgAAACB1DGqBcmAwzEznnzJO7zu9VH3H5VrDEa2p3e9JXQAAAMBIQbBD0pw6YbSy+qyBJ0kPvrRdD7zwpto7uUsXAAAAOBkEOyRN3zXwskN+zSnJ1fTCUbrlT+t1we3P6d4VW9XS0eV1qQAAAEBK4Rk7JFUk6vqtgef3mV7etlc/fnazXtiyVwXZQd1w4TRdf95k5WUGvS4ZAAAAGDaO9IwdwQ7DyqrqRv3k2c16rqpBeZkBfeL8qfrU+VOUnx3yujQAAADAcwQ7pJS1tfv1k+c26+l1uzUq5NfH3jZFiy+YqoLsEOvgAQAAIG0R7JCSNu5q1k+e3aLH1+5UyG8aMypDTa2dau9kHTwAAACkH5Y7QEoqL87TTz5ypp756sU6c3KBdu5vV1tnRE6xpRJW1zRpeVW912UCAAAAniLYISVML8zRedPGDbgO3nMbCXYAAABIbwQ7pIy5pXkDroP3q1fe0kd+/rJWbGpQKt1aDAAAAAwVgh1SxkDr4C2YOkbfePcsbW04qI/d/6ouv/Nv+uPqHeqKRL0uFwAAAEgaJk9BSjnSOngdXRH9cXWd7nl+q7Y2tGhiQZZuuGCqrj67TNmhgNdlAwAAAEOCWTGRFqJRp2Ub63X381u1qrpRBdlBfey8Kfr426ZodFaQpRIAAACQ0gh2SDuV2/fp7ue36pkN9coImAqyQ9rf1qn2zihLJQAAACAlsdwB0k7FlDG67+Nn689fuUhnTR6jXc0dauuMslQCAAAARhyCHUa8meNzde60sQMulfCH15hoBQAAAKmPYIe0MNBSCSbpT2t26uLvLdd9f92m5vZOb4oDAAAABolgh7Qw0FIJ500fq7s+coYm5GfpPx7foPP+c5luWbpO1XtbvC4XAAAAOCFMnoK0caSlEiRpbe1+3f/Cm/rT63WKOKd3zB6vxRdM1YKpY2TG5CoAAAAYHpgVEzgOu5vbteSlav3PK9VqbO3U3NI8fer8qbri9BIFfD6WSwAAAICnCHbACWgLR/SH1Tt0/9/e1Ob6gxqXE1JW0K89B8Nq74ywXAIAAAA8wXIHwAnICvl13TmT9OevXKQHP3WOivMyVdPYprbOCMslAAAAYNgh2AFHYWa6aGah3jm3eMDlEn67qlbtnRFPagMAAAC6EeyA43Ck5RKefGOX3vadZ3X7UxtVs6/Vm+IAAACQ9gh2wHE40nIJD37yHFVMLtA9z2/Vxd97Tjf8cqWe39SgaDR1nl0FAABA6mPyFOA4HW25hB1Nbfr1K9V6+NUa7W0Ja+q4Ufrogkn68FllGp0dPNSXGTUBAAAwGMyKCSRBR1dET67dpSUvV2tVdaMygz697/RSVe06oM31B9UWZkZNAAAAnLwjBbuAF8UAI1VGwK+rzpigq86YoDd27NevXq7W7/5eq87I4T+g9JxRc9Hs8R5WCwAAgJGCZ+yABDl1wmh954Pz9OkLp/U71hqO6K+bGzyoCgAAACMRwQ5IsLMmFyi7z4yakvTAi9X68N0v6vd/Z8kEAAAADA7BDkiwgWbUPHtKgb5+2Sw1HOjQVx95Xed8+xndsnSdNu5q9rpcAAAApKDjmjzFzC6T9CNJfkn3Oee+0+d4hqQHJZ0laa+ka5xz2+PHvilpsaSIpC855542s0xJKyRlKPac32+dczcfqw4mT0GqOtKMms45vbxtnx569S099cYuhSNRnTEpX9edPUlXnF6i7BCPwQIAAOCwk54V08z8kjZJulRSraSVkq5zzq3v0ebzkuY55z5rZtdKer9z7hozmyPpIUnnSCqV9P/bu/Mguc7y3uPfp3u6p3v2TZpNI40ka18sWbKwjTGOZRNjCA6B2HII4RLnusiFgOtybwJJ3YRLoEhIrsELpGKwg7HBBuwAxmA7siQvYGu1ZO37NiPNqtk1S8/0vPePPhrPKs3es/w+Vae6z9unT7+n6tT0PP287/O+AiwEOoFk51yTmQWA3wJfcM5tvVxfFNjJVFZzMcJ/vl3K09vPcqLqIqmJCdy5uoAN185mSX6alksQERERkRFVxVwHHHfOnfRO9AxwJ3Cw2zF3Al/xnj8LPGJm5rU/45xrA06Z2XFgnXPuLaDJOz7gbZNn3QWRMZCVHOQv3jePe2+cy84ztTy97Sw/21nKU1vPkpzop73D0R7t1HIJIiIiItLHYObYFQIl3fZLvbZ+j3HOdQD1QPbl3mtmfjPbA1QCG51z2/r7cDO7z8x2mtnOqipVEZSpz8y4tjiLB+5exfa/vZV71hXR3BYlEu3EEauouetMLZsPV8S7qyIiIiIyQcSteIpzLuqcWwXMAtaZ2fIBjnvUObfWObd2xowZ49tJkThLTwqQnx7u097W0ckXf/YO337lKCU1zXHomYiIiIhMJIMJ7M4BRd32Z3lt/R5jZglAOrEiKld8r3OuDtgC3D6UjotMF8sK0gj3Wi4hmOBjVkYSD246xvu+uYV7Ht3Kc7tKaY50xKmXIiIiIhJPgwnsdgALzGyumQWBDcDzvY55HviU9/zjwGYXq8ryPLDBzBLNbC6wANhuZjPMLAPAzMLECrMcHvnliEw9/S2XsHZOJr/6qxv57d/cwhdvW8j5+ha++LN3uPZrr/DXz77DjtM1XCqMFO10bDpUwUObjrHpUAXRTk1nFREREZlqBrvcwR3At4ktd/C4c+7rZvZVYKdz7nlv+YIngdVADbChW7GVvwP+HOgA7nfOvWhmK4EnvPP5gJ865756pX6oKqZMVwMtl3CJc44dp2t5dlcJv95bxsVIlOLsJP7omkJeO1rNobIGWiJRFV4RERERmeSGvdzBRKLATuTKLrZ18NL+cn62q4StJ2v6vJ4U9PPwPatZvyQ3Dr0TERERkZEYKLCLW/EUERkbyYkJfGzNLJ6573ruvXFun9ebI1F+s69MQzJFREREphAFdiJT2A3zs0nqVXgF4Lm3z3HdNzbxjy8cZP+5eiZT5l5ERERE+hrMAuUiMkldKryyp6Sua47dylnp/Nl1c/jlO+d58q0zPPbbU8yfkcxHVxdy56pCirKS4t1tERERERkizbETmeIuV3ilvrmd3+wv4+e7z7H9VGw+3to5mfzh6kI+tCKfzORg1/sPnG9gWT+FW0RERERk/Kh4iohcVmltM8+/c56fv32OY5VNBPzGTQtmUFrbQklts6pqioiIiEwACuxEZFCccxwsa+AXu8/xs50l1LX0XPRcVTVFRERE4megwE5z7ESkBzNjWUE6ywrSSU5M4MFXjtH955/mSJRvv3KUUMDPdfOylbkTERERmQAU2InIgFYUphMO+mmORLva/D7jaEUTn/j+NnJSgnxweT4fXpnPtcVZ+BTkiYiIiMSFAjsRGVB/VTVXFWXwvT9by+tHq3hhbxk/21XCk1vPkJuWyB0r8vmDqwtYXZSBmanwioiIiMg40Rw7Ebmsy1XVBLjY1sErhyp4YW8Zrx2pIhLtpDAjzB0r8th+qoZjlU0qvCIiIiIySlQ8RUTGXENrOxsPVPDC3vO8drSKzl5/XlR4RURERGRkBgrsfPHojIhMTWmhAB9bM4v/+PQ6PvP++X1eb45E+e6rx9l1ppbO3lGfiIiIiAyb5tiJyJhYMyeTpF6FV3wGe0rq+Ni/vUleWojfX5bL7y/PY11xFgl+/c4kIiIiMlwK7ERkTAxUeOU7n7iG145U8eL+Mn6ys4Qn3jpDVnKQ2ddn6QAAHthJREFUDyzN5fbledwwP4dgQizIU/EVERERkcHRHDsRGTNXKrzSHOnwgrxyNh+upKmtg9RQArcuyeUDS3N54q3T7C2tV/EVEREREY+Kp4jIhNbaHuV3x6t5cX85Gw9WUN/S3ucYFV8RERGR6W6gwE5DMUVkQggF/Kxfksv6Jbm0Rzv50nP7eO7t0h7HNEeiPLX1DIvyUpmVmRSnnoqIiIhMPKpWICITTsDv444VeSQF/T3aDdhypIob/3kLH3zwDR7YeJR9pfVMppEHIiIiImNBGTsRmZAGKr7y1TuXs/lwBa8crOSRzcd4aNMx8tND3Lokl1uX5nLdvCwSE2IBoYqviIiIyHShOXYiMmFdqfjKhaY2thypYuPBcl4/Wk1Le5SUxATev3AG6xfP5JmdJew/p+IrIiIiMnWoeIqITGmt7VHePFHNxoMVvHKokqrGtj7HqPiKiIiITHYDBXaaYyciU0Io4OeWxbl8449Wsu3L67lnXVGfY5ojUR7ZfJzXjlbR2h7t5ywiIiIik5Pm2InIlOPzGbcuyeWXe87THHk3gPMb7D9fz6ce30444Oe9V2Vz86KZ/N7imRRmhOPYYxEREZGRUWAnIlPSQMVXvvdna9l+uoYthyvZfLiSVw5VArA4L5WbF83klsUzuWZ2BmamwisiIiIyaWiOnYhMWVcqvuKc40RVE5sPV7LlcBU7TtfQ0elITfQTTPDT2NZBe0enCq+IiIjIhKHiKSIiV9DQ2s5vj1Xz1NYzvHniQo/XAj7j/tsW8t/fN49ggqYni4iISHwMFNhpKKaIiCctFOCOFfkcr2zirRMX6P6zV3un419ePsJ3txzn+vk5vH9hDjctnMGc7OS49VdERETkEgV2IiK9LCtIIxz09yi8Eg76+fQNxdS1tPP60SpeOVQBwJzsJG5aMIObFs7g+vnZpCTG/qxqcXQREREZTwrsRER6Gajwyhc/sAi/z3DOcar6Iq8freL1Y9U8u6uUJ7eeIcFnrJmTyY0LcnjlYAXHKpu0OLqIiIiMC82xExHpx5UKr3TX1hFl1+laXjtWxRtHqzlY1tDnmHDAxyN/co0WRxcREZERGVHxFDO7HXgQ8APfd879U6/XE4EfAmuAC8DdzrnT3mtfBu4FosDnnXMvm1mRd3wu4IBHnXMPXqkfCuxEZDL4xm8O8e+vn+zTnh6OzeF771XZ3DA/h6zkYBx6JyIiIpPZsIunmJkf+A5wG1AK7DCz551zB7sddi9Q65y7ysw2AP8M3G1mS4ENwDKgAHjFzBYCHcAXnXNvm1kqsMvMNvY6p4jIpLRubhZPbj3TY45e0G8UZyfzq3fO8/T2swAszU/jvVdl896rclg3N4ukoObniYiIyPAMZo7dOuC4c+4kgJk9A9wJdA/C7gS+4j1/FnjEzMxrf8Y51wacMrPjwDrn3FtAGYBzrtHMDgGFvc4pIjIpDTRH78l734Nzjr3n6nnzeDW/PV7NE2+e4XtvnCLgN1bPzuT6edlsPlzBiaqLmp8nIiIigzaYwK4QKOm2Xwq8Z6BjnHMdZlYPZHvtW3u9t7D7G82sGFgNbOvvw83sPuA+gNmzZw+iuyIi8eX3GU/e+54B5ugZ18zO5JrZmXzulgW0RKLsOF3D705U8+bxCzy46ViPczVHorx9tpZNhyr4wLK8+FyQiIiITHhxrYppZinAc8D9zrm+1QYA59yjwKMQm2M3jt0TERk2v89YvyT3isVSwkE/Ny2MLZcA8M2XDvPdV0/0OKa1vZPP/Xg3Ny7I4bp5WVw3L5ul+Wkk+LVQuoiIiMQMJrA7BxR125/ltfV3TKmZJQDpxIqoDPheMwsQC+p+5Jz7z2H1XkRkilkzJ5OkXmvoBRN8XD8vm9MXLrL5cCUAqYkJXDs3qyvQW1aQ3jVUU3P0REREpp/BBHY7gAVmNpdYULYB+JNexzwPfAp4C/g4sNk558zseeDHZvYAseIpC4Dt3vy7x4BDzrkHRudSREQmv4Hm5z3+367F7zMqG1rZeqqGrScvsPXkhR6B3rq5Waybm8Vv9pVpDT0REZFpZrDLHdwBfJvYcgePO+e+bmZfBXY65543sxDwJLG5cjXAhm7FVv4O+HNilTDvd869aGY3Am8A+4BO72P+1jn3m8v1Q8sdiMh0MJQ19HoHeierLvY5JjHBx4MbVnH78vyx7rqIiIiMsRGtYzdRKLATEbm8gdbQ8/uMa2ZncG1xFtfOzWLNnEzSQoE49FBERERGYtjr2ImIyOTR3xp6iQk+3r9wBhWNbTz6+km+++oJfAaL89JYNzeLtcWZrCvOYmZaSPPzREREJikFdiIiU8hAc/T+7U/X4PcZzZEO9pytY/vpGnacruEnO0r4wZunAZidFaalvZP65nbao52anyciIjKJaCimiMgUM5Q5eu3RTg6cb2Dn6Rp+vbeM3SV1PV73G9y5upC71xaxclYG4aB/PC5BREREBqA5diIiclkPbTrGtzYeZaBvhQSfsbQgjWtmZ7JmTmwryAj3OEZDOUVERMaW5tiJiMhlLStII9xrDb2koJ9vfHQFqeEEdp2pZdeZ2h7DN/PSQqyZk8k1czJZVZTBv758hHdK67TUgoiIyDhTYCciIsDA8/M+fHUBfp9xy+JcADqinRwub+wK9HadqeXX+8r6nK85EmX32Tq2HK7g1qV54305IiIi04qGYoqISJehzM/rrry+la/9+iAv7O0b4CUF/bz3qhxWFWWwuiiDlUUZpCT2/V1RwzhFRESuTEMxRUTkivw+Y/2SXNYvyR3S+/LSQ3x0dSGbD1f2GMoZ9BsrZ6VzvLKJjQcrADCDBTNTWFWUwaqi2BDO+TOS+fQPdvTJFmoYp4iIyOAosBMRkVEx0FDOS8FZXXOEd0rr2XO2jj0ltWw8WMFPd5YCEPT76OjspNMbRNIcibKnpI5Xj1QOOcgUERGZjjQUU0RERs1QhnI65zhb08yekjp+8OZpdp+t63NMcXYSf3B1ASsK01k5K4PctETMlMETEZHpS0MxRURkzA1lKKeZMSc7mTnZyaQkJvBXT+/uMYwzwWd0dDq+++oJol4qb0ZqIisL01kxK52rZ2WwvDCdGamJgOboiYjI9KbATkRE4u5ywzgvLaK+r7SOvefq2Vdaz+YjlVwacFKQHmJ5YRqHy5uoaGgl0tGpOXoiIjLtaCimiIhMCEMZxtnU1sGBc/XsO1fP3tJ6tp26QEVDW49j/AYfWpnPR64uZFlhGnlpIQ3jFBGRSW+goZgK7EREZNJ7aNMxvrXxKJf7RstODrK0II3lheksL0hnWUEas7OS8PlMwzhFRGTS0Bw7ERGZspYVpBEO+nvM0UsK+vmXj68kNy3EgfMN7D9Xz4HzDXzv9ZN0eHP2UhMTWJyfSmltCxeaIrRHNYxTREQmJwV2IiIy6Q00R+/25fn4fcba4qyuY9s6ohyraOoK9H53opqy+tau15sjUd46eYFPPraN9UtyWZKfytL8NDKSggN+vjJ+IiISbxqKKSIiU8JQ5uh1N9AwzqReGcD89BBL8tNYkp/qPaZRnJ0MwCcf26bF1UVEZFxoKKaIiExpQ1lqobuBhnE+fM9qVs7K4FBZQ7etkdePVnUN5QwH/OSlhyipae5q0+LqIiISDwrsRERkWhtoGOeljN+M1BnctHBG1/GXhnJeCvQ2HizvCuouaY5E+ccXDrL9VA0Lc1NZlJfKVTNTCAX8fT5fwzhFRGQ0aCimiIhMe8Mdxgmw6VBFn8XV/T4jPz1EZUMbkWhnV1txdhKL8lJZlJvWFez9/S/3axiniIgMmoZiioiIDGC4wzjh8ourO+c4feEih8sbOeJtB8438OL+cvr7XbU5EmXXmVqe3VXCXWuLtO6eiIgMmjJ2IiIiIzTUjF9zpINjFU18Z8tx/utgRb/HJAf9XDUzhQW5qSyYmcLC3FiGrzAjjK/buTWUU0RketEC5SIiIhNMf8M4ExN83LV2Fn6fj2OVjRytaKKqsa3r9aRLAd/MVObPTOZX75zndPVFWtu1Bp+IyHSgoZgiIiITzEDDOL/ykeU9ArO65gjHKps4VtHE0YpGjlc28caxKp57u7TH+ZojUbadquF//mQPty3L5aqZKRRnJ/dbtAWU7RMRmUqUsRMREYmjkRRu+eZLh/m3V0/0WYOvOzMoykziqpkpzJ+RzPwZKV0B3+ef2a3CLSIik4wydiIiIhPQSAq3rJmT2e8afP/6x1dTnJ3M8aomTlQ2caKqieOVTfz2eDWRjs5+z3WpcMvT289y19oiggm+K36+Mn4iIhOHMnYiIiKTVLTT8cnHtg066xbtdJyrbeFEVRM/ePMUrx2t7ve8fp8xOyuJuTnJXdu8GcnMy0khNy0RMxvyZ4uIyOhQxk5ERGSK8fuMJ+99z6CHcvp9xuzsJGZnJ9HpHDtO1/bI9oUCPj6xbg7hoJ9T1Rc5UdXEmyeqaW1/N8uXFPQzNyeZpKCfPSV1tEdjPxA3R6LsKanj1SOVw8o+iojIyChjJyIiMg0NNuPW2ekob2jlZNVFTlU3cbL6IqeqL7L7bB31Le19zpuXFuK6eVnMyY5l+uZkxzJ/GUnBPp+vYZwiIkOn5Q5ERESkh5EUbulvqYYEn7EgN4WGlg7O17f0WIQ9PRygODuJ4pxkirKSeHl/OSW1zbRpmQYRkSEZUWBnZrcDDwJ+4PvOuX/q9Xoi8ENgDXABuNs5d9p77cvAvUAU+Lxz7mWv/XHgw0Clc275YC5CgZ2IiMjEcKWMX2t7lNLaZk5VN3PmQizLd+ZCM6cvXKS0tqXP+czgmqIM1s7NYnZWEnOykpmdlURBRogEf99CLsr4ich0NezAzsz8wFHgNqAU2AHc45w72O2Y/wGsdM59xsw2AB91zt1tZkuBp4F1QAHwCrDQORc1s5uAJuCHCuxEREQmn+Fm/B7YeISHNx3vs0xDZlKApraOrnl7EMsCFmaGmZ2V1LUVZYb599dPcqyySYVbRGTaGUnxlHXAcefcSe9EzwB3Age7HXMn8BXv+bPAI2ZmXvszzrk24JSZHffO95Zz7nUzKx7e5YiIiEi8DXephqtnZQy4TMPNi2ZS3tDK2QvNnK2JZfnO1sS2X+8ro66577y+Swuzf+7Hb3PjghyKMpOYlRmmMDNMYoIWZxeR6WEwgV0hUNJtvxR4z0DHOOc6zKweyPbat/Z6b+GweysiIiKT3s2LZrKqKKPPMM5LwVVhRpjCjDDXz8/u8976lna++dJhfrTtbI/2aKfj5QPlvLi/vKvNDHJTQxRlhWPBnpftK8wI8/82HuVQWYMyfiIyZUz45Q7M7D7gPoDZs2fHuTciIiIyUkNdpqG79HCAWxbP5Oe7z/XJ+D24YRXLCtIpqWmmpLaFkppmSmtbKKltZuvJC5TtOUd/M1AuZfzu/8lu3rdgBrMyYtm+/PRwvwu1K9snIhPRYAK7c0BRt/1ZXlt/x5SaWQKQTqyIymDee1nOuUeBRyE2x24o7xUREZGJabjDOGHgjN8ti3Px+4yCjHCfoUUAkY5Ozte18PDmYzz3ds9/R6Kdjl+9U8av3inraruU8Sv0snyxYC/E09vPcrr6Iq3DrOipwFBExsJgiqckECuesp5YULYD+BPn3IFux3wWWNGteMofOefuMrNlwI95t3jKJmCBcy7qva8YeEHFU0RERGQoRnuphqSgn2/dtYrF+amcq22htK4l9ljbwrm6Zs7VtVBW10pHZ9//m8xgZWEGVxelU5ARjm3pIQoywsxMTexR1XOw6weKiAxk2MVTvDlznwNeJrbcwePOuQNm9lVgp3PueeAx4EmvOEoNsMF77wEz+ymxQisdwGe7BXVPAzcDOWZWCvyDc+6xUbhWERERmeLGIuN369JYxm9OdnK/74t2Ov7pxUN8/41TPSp6Ogfn6po5Wd1EY2tHn37mpYXI9wK99mgnO0/XEPEqfzZHouwpqePVI5WDuhZl+0RkIFqgXERERKad4Wb8Bsr2PXzPatYvyaWxtZ2y+lbO17Vwvs57rG/hfF0LZfWtlNa0EO3nf6/0cIDFeankp4fISw9TkBHyAsIweekhspODOFC2T0RGtNyBiIiIyJQy3Izf5Sp6AqSGAqSGAizMTe33/RsPlvP5p/fQ0v5uYJjgMxblpeIc7DpbS3l9WY+1/AACfiMtFKC2OcKl0aDNkSi7ztTyvTdO8qEV+eSmhfot9tKdMn4iU5cydiIiIiJDMJL5fYOZY9fZ6bhwMUJ5fStl9S2UN7Ryvq6VV49Ucri88bLnz04OkpsWIjctkbz0ELlpscxfbnqInORE/vGFA+w/P7xlHhQUikwMA2XsFNiJiIiIjKPRHAYaCvj4zPvnk58eory+jfKGViq6bdVNkcue02/w/kUzWDMni5mpicz0gsKZqSEykwKYWVefNQxUZGJQYCciIiIyiQ0nuIp0dFLZ2EpFQxv/8btTvLC3rM8xwQQfkY7Ovu1+HzNSE5mZlgjAvtL6HlVBQwEfX//DFdy5qqBH5c/L9V8ZP5GRU2AnIiIiMsmNxTIPD9+zmhvm51DV2EZFYyuVDW1UNLRS2dhGpfd4qKyBCxf7z/6ZxYaA5qQkxgLB1BAzUhPf3VISyUoO8ve/3M++c/UaBioyQgrsRERERKaxkQyn7C8oTEzw8cdrZ5GVnEhVY1tsa2qj2nseifbNAnbnN7hhfg4ri9LJSUns2makxoLE9HBsKKiGgYr0pMBOREREZJobbsZvqMGVc46Glg4qG1upamzjqa1n+M3+8j7HJQX9tHV0Eu1n4feA38hOTiSYYJTWttD9kGCCj8/ePJ/1S3LJ8TKCl6sIqoyfTCUK7ERERERk2MZqGOjvLZpJbXOE6qYI1U1tVDfFMn6X9nedruHUheYrfkZqKIGclESyk4NkpwTJTkkkJzlIRlKAZ3aUcOZCM5GOTkKBWFD61F9oGKhMTgrsRERERCQuRnsYaCjg4/5bF1KcncyFi23UNEW4cDEWCF5oilBzMRJrvxihn2QgEAss89JCZCYHyUoOkp0cJNN7zPKeZ4QDfO3XhzhUNrwlIi5duwJDGU0K7EREREQkbsZrGGjv9/7Ly4f599dO0vs/3lVF6czKTKLmYqTH1jFQJOgxgyV5acyfmUJWUoCMpHcDwaykWIYwywsOA37fiOYHKiiU/iiwExEREZFJaayGga5fktvjWOccDa0dXUHeD986zS/3nO9zzvz0EIkJPmqb26lvaR/wswN+oyPqegSVCT7j9uV5rCrKIDMpSGZygPRwkMykAJlJQdLCAfy+0Skao8BwalJgJyIiIiLTzmgPA+0dFHZEO6lraaeuOULNxXZqLkZiz5sjbDpUwa4zdX3Oa9Ang9j1mkFaKEBigo/qprYeQ0kDfuPja2axbm4WGeEg6UkB0sMBMsKxx+7rCY40MFRQOHEpsBMRERGRaSkew0Bh4MDwwQ2rWFecTW1zhLqW9thjc4S65nZqm2NB4rZTNRwpbxzSdaYmJnQFe53OcbSiqUfF0aDfx5/fOJcbr8oh3QsG08MBUkMJ+Lpdj7KFE5sCOxERERGRIRrJMNCxyBZ+46MrWFmUEQsEW9ppaGmnrtnbWiLUe8NDD5U1cL6+dVD9NIOUxISuQC/a6ThW0Ui0+xITfuNTNxRzw/wc0sIJpIUCpIUDpIUChAI+zEYvMFRQeHkK7ERERERExtlEyhaGA36+9MFFLM5Lo76lnYbWDupbYoFgg/dY39LO4SEEhRAbIpoW8rJ/4QDtHZ0cqWjslS00Pnl9MdfPyyY1lEBqKJYpTAsFSAkldF2TsoVXpsBORERERGQSmUjZwnDAz999aAnLCtJoaO2goaWdhtZ2Glo6vMf2rvajFY2UDSEwhFjGMDWUgBmU17f2mFuY4DM+cnUBVxdldB2Xciko7LafmOCfFoGhAjsRERERkWlkomUL/8+Hl7C8MJ3G1g4aW98NBGP7sbbdZ+s4XtU0rOsNJvhI9PtoauvoUZzG7zNuWpDDory0WBCYGNuSLwWF3Z6HA34+89SuEQWGY02BnYiIiIiIDMpEyhYmBf08cNfVXFucRVNbR49A8NJ+U1ssc/jWiQvsLa3vc95wwE9HZyft0aHHPgMtjxEvAwV2CfHojIiIiIiITFx+n7F+Se6wghm/z3jy3vcMKzC8edFMVhVl9AkKb1uah99nZKckXvb9V1qioq0jSlNrBxfbojS2tdPkBYWXtpf2l/PGseoe52yJRDl4vmHCBHYDUWAnIiIiIiKjariB4UiCQhg4MLx50UwAEhP8JKb4yU7p//15aSF2nantOYw06GdpQdqQriMeFNiJiIiIiMiEEa9sIVw5MJzINMdORERERETEM5L5heNBc+xERERERESuYCQZw3jyxbsDIiIiIiIiMjIK7ERERERERCY5BXYiIiIiIiKTnAI7ERERERGRSW5SVcU0syrgTLz70Y8coPqKR4mMnO41GS+612S86F6T8aT7TcbLWN5rc5xzM3o3TqrAbqIys539lRwVGW2612S86F6T8aJ7TcaT7jcZL/G41zQUU0REREREZJJTYCciIiIiIjLJKbAbHY/GuwMybehek/Gie03Gi+41GU+632S8jPu9pjl2IiIiIiIik5wydiIiIiIiIpOcArsRMLPbzeyImR03sy/Fuz8ytZjZ42ZWaWb7u7VlmdlGMzvmPWbGs48yNZhZkZltMbODZnbAzL7gtet+k1FlZiEz225m73j32v/12uea2Tbv+/QnZhaMd19lajAzv5ntNrMXvH3dazImzOy0me0zsz1mttNrG9fvUQV2w2RmfuA7wAeBpcA9ZrY0vr2SKeYHwO292r4EbHLOLQA2efsiI9UBfNE5txS4Dvis9/dM95uMtjbgFufc1cAq4HYzuw74Z+BbzrmrgFrg3jj2UaaWLwCHuu3rXpOx9HvOuVXdljkY1+9RBXbDtw447pw76ZyLAM8Ad8a5TzKFOOdeB2p6Nd8JPOE9fwL4w3HtlExJzrky59zb3vNGYv8EFaL7TUaZi2nydgPe5oBbgGe9dt1rMirMbBbwIeD73r6he03G17h+jyqwG75CoKTbfqnXJjKWcp1zZd7zciA3np2RqcfMioHVwDZ0v8kY8IbG7QEqgY3ACaDOOdfhHaLvUxkt3wb+Guj09rPRvSZjxwH/ZWa7zOw+r21cv0cTxvLkIjJ2nHPOzFTWVkaNmaUAzwH3O+caYj9ux+h+k9HinIsCq8wsA/g5sDjOXZIpyMw+DFQ653aZ2c3x7o9MCzc6586Z2Uxgo5kd7v7ieHyPKmM3fOeAom77s7w2kbFUYWb5AN5jZZz7I1OEmQWIBXU/cs79p9es+03GjHOuDtgCXA9kmNmlH5v1fSqj4b3AR8zsNLHpMrcAD6J7TcaIc+6c91hJ7EerdYzz96gCu+HbASzwqisFgQ3A83Huk0x9zwOf8p5/CvhlHPsiU4Q37+Qx4JBz7oFuL+l+k1FlZjO8TB1mFgZuIzancwvwce8w3WsyYs65LzvnZjnnion9j7bZOfcJdK/JGDCzZDNLvfQc+ACwn3H+HtUC5SNgZncQG7/tBx53zn09zl2SKcTMngZuBnKACuAfgF8APwVmA2eAu5xzvQusiAyJmd0IvAHs4925KH9LbJ6d7jcZNWa2klgBAT+xH5d/6pz7qpnNI5ZVyQJ2A3/qnGuLX09lKvGGYv4v59yHda/JWPDuq597uwnAj51zXzezbMbxe1SBnYiIiIiIyCSnoZgiIiIiIiKTnAI7ERERERGRSU6BnYiIiIiIyCSnwE5ERERERGSSU2AnIiIiIiIyySmwExGRacHMoma2p9v2pVE8d7GZ7R+t84mIiAxVQrw7ICIiMk5anHOr4t0JERGRsaCMnYiITGtmdtrMvmlm+8xsu5ld5bUXm9lmM9trZpvMbLbXnmtmPzezd7ztBu9UfjP7npkdMLP/MrOwd/znzeygd55n4nSZIiIyxSmwExGR6SLcayjm3d1eq3fOrQAeAb7ttT0MPOGcWwn8CHjIa38IeM05dzVwDXDAa18AfMc5twyoAz7mtX8JWO2d5zNjdXEiIjK9mXMu3n0QEREZc2bW5JxL6af9NHCLc+6kmQWAcudctplVA/nOuXavvcw5l2NmVcAs51xbt3MUAxudcwu8/b8BAs65r5nZS0AT8AvgF865pjG+VBERmYaUsRMREQE3wPOhaOv2PMq789g/BHyHWHZvh5lpfruIiIw6BXYiIiJwd7fHt7znbwIbvOefAN7wnm8C/hLAzPxmlj7QSc3MBxQ557YAfwOkA32yhiIiIiOlXw1FRGS6CJvZnm77LznnLi15kGlme4ll3e7x2v4K+A8z+99AFfBpr/0LwKNmdi+xzNxfAmUDfKYfeMoL/gx4yDlXN2pXJCIi4tEcOxERmda8OXZrnXPV8e6LiIjIcGkopoiIiIiIyCSnjJ2IiIiIiMgkp4ydiIiIiIjIJKfATkREREREZJJTYCciIiIiIjLJKbATERERERGZ5BTYiYiIiIiITHIK7ERERERERCa5/w8vLby/toDQbwAAAABJRU5ErkJggg==\n",
1315 |       "text/plain": [
1316 |        "<Figure size 1080x576 with 2 Axes>"
1317 |       ]
1318 |      },
1319 |      "metadata": {
1320 |       "needs_background": "light"
1321 |      },
1322 |      "output_type": "display_data"
1323 |     }
1324 |    ],
1325 |    "source": [
1326 |     "fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(15, 8))\n",
1327 |     "ax1.plot(epoch_losses, marker=\"o\", markersize=5)\n",
1328 |     "ax1.set_title(\"Loss\")\n",
1329 |     "ax2.plot(epoch_lrs, marker=\"o\", markersize=5)\n",
1330 |     "ax2.set_title(\"LR\")\n",
1331 |     "plt.xlabel(\"Epochs\")\n",
1332 |     "plt.show()"
1333 |    ]
1334 |   },
1335 |   {
1336 |    "cell_type": "code",
1337 |    "execution_count": 43,
1338 |    "metadata": {},
1339 |    "outputs": [
1340 |     {
1341 |      "data": {
1342 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3YAAAHwCAYAAADq2/1hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3xV9f3H8fcnGwgJK+wR9pBNQBAEHKiIdbeirYNaR6u1P7XDXVuraGutW6viqFattVZxoIiiggMIyB4S2TOB7J3c+/39kXCbQICE3JvDJa/n45GH9557zrmfm5NI3ue7zDknAAAAAED4ivC6AAAAAABA/RDsAAAAACDMEewAAAAAIMwR7AAAAAAgzBHsAAAAACDMEewAAAAAIMwR7AAAAAAgzBHsAACNlpltMrNTva4DAID6ItgBAAAAQJgj2AEAsB8zu8rM0sws08xmmlnHyu1mZn8zs3QzyzWzFWY2sPK1M81stZnlmdl2M/u1t58CANCYEOwAAKjCzE6WNF3SjyR1kLRZ0uuVL58mabykPpISK/fZW/naDEnXOOeaSxoo6dMGLBsA0MhFeV0AAABHmR9Let45t0SSzOxWSVlmliypTFJzSf0kLXTOralyXJmkAWa2zDmXJSmrQasGADRqtNgBAFBdR1W00kmSnHP5qmiV6+Sc+1TS45KekJRuZs+YWULlrhdIOlPSZjP73MzGNHDdAIBGjGAHAEB1OyR12/fEzJpJai1puyQ55x51zo2QNEAVXTJ/U7l9kXPuHEltJb0t6Y0GrhsA0IgR7AAAjV20mcXt+5L0mqRpZjbUzGIl3SdpgXNuk5mNNLPjzSxaUoGkYkl+M4sxsx+bWaJzrkxSriS/Z58IANDoEOwAAI3dB5KKqnxNlHSnpP9I2impp6SplfsmSHpWFePnNquii+ZfKl+7VNImM8uVdK0qxuoBANAgzDnndQ0AAAAAgHqgxQ4AAAAAwhzBDgAAAADCHMEOAAAAAMIcwQ4AAAAAwhzBDgAAAADCXJTXBdRFmzZtXHJystdlAAAAAIAnFi9evMc5l7T/9rAKdsnJyUpNTfW6DAAAAADwhJltrmk7XTEBAAAAIMwR7AAAAAAgzBHsAAAAACDMEewAAAAAIMwR7AAAAAAgzBHsAAAAACDMEezqYUd2kU5+8DN9uHKX16UAAAAAaMQIdvXg8ztt2FOg3OIyr0sBAAAA0IgR7OohMsIkSX6/87gSAAAAAI0Zwa4e9gU7nyPYAQAAAPAOwa4eIowWOwAAAADeI9jVQ6DFjmAHAAAAwEMEu3qItH1dMT0uBAAAAECjRrCrB6v87jnG2AEAAADwEMGuHgItdnTFBAAAAOAhgl09MCsmAAAAgKMBwa4emBUTAAAAwNGAYFcP/5sV0+NCAAAAADRqBLt6qMx1dMUEAAAA4CmCXT2YmSKMrpgAAAAAvEWwq6fICKPFDgAAAICnCHb1FGFGix0AAAAATxHs6ikywljHDgAAAICnQhLszCzOzBaa2TIzW2Vmf6hhn1gz+5eZpZnZAjNLDkUtoRZpdMUEAAAA4K1QtdiVSDrZOTdE0lBJZ5jZ6P32uVJSlnOul6S/SXogRLWElJlErgMAAADgpZAEO1chv/JpdOXX/vHnHEkvVT5+U9IpZpUrfgMAAAAAai1kY+zMLNLMlkpKl/Sxc27Bfrt0krRVkpxz5ZJyJLUOVT2hYmZyNNkBAAAA8FDIgp1zzuecGyqps6RRZjbwSM5jZlebWaqZpWZkZAS3yCCgjREAAACA10I+K6ZzLlvSXEln7PfSdkldJMnMoiQlStpbw/HPOOdSnHMpSUlJoS73iNBeBwAAAMBLoZoVM8nMWlQ+biJpkqS1++02U9LllY8vlPSpC8M+jSYmTwEAAADgragQnbeDpJfMLFIV4fEN59x7ZvZHSanOuZmSZkh62czSJGVKmhqiWkLKzORoswMAAADgoZAEO+fccknDath+V5XHxZJ+GIr3b0gMsQMAAADgtZCPsWsM6IoJAAAAwEsEu3oyY/IUAAAAAN4i2NUbnTEBAAAAeItgFwR0xQQAAADgJYJdPVUsUE6yAwAAAOAdgl09sY4dAAAAAK8R7OrJGGIHAAAAwGMEuyCgxQ4AAACAlwh29WQyOcbYAQAAAPAQwa6e6IoJAAAAwGsEuyCgKyYAAAAALxHs6snEYgcAAAAAvEWwqyczo8UOAAAAgKcIdgAAAAAQ5gh2QcCsmAAAAAC8RLCrJ2OQHQAAAACPEezqyYxcBwAAAMBbBLt6MrGQHQAAAABvEeyCwDEtJgAAAAAPEezqia6YAAAAALxGsKsnOmICAAAA8BrBLgjoiQkAAADASwS7ejIzumICAAAA8BTBrp5MTJ4CAAAAwFtBD3Zm1sXM5prZajNbZWa/qmGfiWaWY2ZLK7/uCnYdDYZBdgAAAAA8FhWCc5ZLutk5t8TMmktabGYfO+dW77ffPOfcWSF4/wZHex0AAAAALwW9xc45t9M5t6TycZ6kNZI6Bft9jhYmkewAAAAAeCqkY+zMLFnSMEkLanh5jJktM7NZZnbcIc5xtZmlmllqRkZGiCo9cmb0xQQAAADgrZAFOzOLl/QfSf/nnMvd7+Ulkro554ZIekzS2wc7j3PuGedcinMuJSkpKVTl1oujyQ4AAACAh0IS7MwsWhWh7p/Oubf2f905l+ucy698/IGkaDNrE4paQq1iVkyvqwAAAADQmIViVkyTNEPSGufcQwfZp33lfjKzUZV17A12LQ3BjGAHAAAAwFuhmBVzrKRLJa0ws6WV226T1FWSnHNPS7pQ0s/NrFxSkaSpLkwXgzPWOwAAAADgsaAHO+fcfB1mdTfn3OOSHg/2e3uFMXYAAAAAvBTSWTEbA7piAgAAAPAawQ4AAAAAwhzBLghosAMAAADgJYJdPZkZXTEBAAAAeIpgV08Vs8SQ7AAAAAB4h2BXT8ZqBwAAAAA8RrALArpiAgAAAPASwa6ezOiICQAAAMBbBLt6MpkcTXYAAAAAPESwqyfG2AEAAADwGsEuCGivAwAAAOAlgl09mZg8BQAAAIC3CHb1RV9MAAAAAB4j2AUBDXYAAAAAvESwq6eKrphEOwAAAADeIdjVEz0xAQAAAHiNYFdPTJ4CAAAAwGsEu3oyMzlG2QEAAADwEMGunmixAwAAAOA1gl09mRHsAAAAAHiLYFdPJrpiAgAAAPAWwa6+aLEDAAAA4DGCXT2ZWKAcAAAAgLeCHuzMrIuZzTWz1Wa2ysx+VcM+ZmaPmlmamS03s+HBrqOhGMkOAAAAgMeiQnDOckk3O+eWmFlzSYvN7GPn3Ooq+0yW1Lvy63hJT1X+N+xUjLHze10GAAAAgEYs6C12zrmdzrkllY/zJK2R1Gm/3c6R9A9X4RtJLcysQ7BraQjMigkAAADAayEdY2dmyZKGSVqw30udJG2t8nybDgx/+85xtZmlmllqRkZGKMqsFzOvKwAAAADQ2IUs2JlZvKT/SPo/51zukZ7HOfeMcy7FOZeSlJQUvAKDiAY7AAAAAF4KSbAzs2hVhLp/OufeqmGX7ZK6VHneuXJb2DGZHH0xAQAAAHgoFLNimqQZktY45x46yG4zJV1WOTvmaEk5zrmdwa6lIZjRYgcAAADAW6GYFXOspEslrTCzpZXbbpPUVZKcc09L+kDSmZLSJBVKmhaCOhpEem6J1u3O87oMAAAAAI1Y0IOdc26+KtbtPtQ+TtJ1wX5vL+wLdfkl5YqPDUVOBgAAAIBDC+msmI0J4+wAAAAAeIVgFyQRrHsAAAAAwCMEuyChvQ4AAACAVwh2QeKnKyYAAAAAjxDsgsTvJ9gBAAAA8AbBLkjIdQAAAAC8QrALEh/JDgAAAIBHCHZBQrADAAAA4BWCXZD4mDwFAAAAgEcIdkHi8xHsAAAAAHiDYBcktNgBAAAA8ArBLkh8fr/XJQAAAABopAh2QVLO5CkAAAAAPEKwq6dbJ/eTxKyYAAAAALxDsKunnknxkgh2AAAAALxDsKunyEiTRLADAAAA4B2CXT1FGsEOAAAAgLcIdvW0b5mD1TtzPa4EAAAAQGNFsKunVdtzJEl3vbPK40oAAAAANFYEu3qKjuRbCAAAAMBbpJJ6iouO9LoEAAAAAI0cwa6emsQQ7AAAAAB4i2BXT+cO7eR1CQAAAAAauZAEOzN73szSzWzlQV6faGY5Zra08uuuUNTREGKiyMYAAAAAvBUVovO+KOlxSf84xD7znHNnhej9AQAAAKDRCElzk3PuC0mZoTj30czPIuUAAAAAPOBlP8IxZrbMzGaZ2XEe1hE0izY1uiwLAAAA4CjgVbBbIqmbc26IpMckvX2wHc3sajNLNbPUjIyMBivwSMxZs9vrEgAAAAA0Qp4EO+dcrnMuv/LxB5KizazNQfZ9xjmX4pxLSUpKatA66+rZeRu9LgEAAABAI+RJsDOz9mZmlY9HVdax14taAAAAACDchWRWTDN7TdJESW3MbJuk30uKliTn3NOSLpT0czMrl1Qkaapz7piYeSS3uEwJcdFelwEAAACgEbFwylMpKSkuNTXV6zIO8LOXFmnOmvTA8033T/GwGgAAAADHKjNb7JxL2X87q2sHwc8n9vS6BAAAAACNGMEuCAZ1auF1CQAAAAAaMYJdEMREVf82lvn8HlUCAAAAoDEi2AXJe78cF3h8/6y1HlYCAAAAoLEh2AXJwE6Jgccz5m+Uzx8+k9IAAAAACG8EuxDZklnodQkAAAAAGgmCXRDdc+7AwOMrX1rkYSUAAAAAGhOCXRD9cETnwOMNGQUeVgIAAACgMSHYBVFcdGS152np+R5VAgAAAKAxIdiF0KOfrPe6BAAAAACNAMEuyDokxgUez1y2Q/e+v9rDagAAAAA0BgS7IHvpp6OqPX923kbd8x7hDgAAAEDoEOyCrHfb+AO2zZi/UQ9+tM6DagAAAAA0BgS7IDOzGrc/PjetgSsBAAAA0FgQ7EJg4/Qza9yefMv7KvP5G7gaAAAAAMc6gl0IHKzVTpJWbM9pwEoAAAAANAYEuxDZcF/NrXbnP/mV3kjd2sDVAAAAADiWEexCJCLC9OcLB9f42qwVOxu4GgAAAADHMoJdCP0opUuN4W7uugzG2gEAAAAIGoJdiLVPiKtx+81vLGvgSgAAAAAcqwh2IXZi7zZ64pLhB2yfuWyHkm95X845D6oCAAAAcCwh2IWYmWnK4A66dHS3Gl/3k+sAAAAA1BPBroHcc+5AtWgafcD2nrd9oORb3tdDs9d5UBUAAACAY0FIgp2ZPW9m6Wa28iCvm5k9amZpZrbczA7sq3gMuvm0vgd97dFP0/Ty15u0aU9BwxUEAAAA4JgQqha7FyWdcYjXJ0vqXfl1taSnQlTHUeXS0d10cr+2B339zndWaeKDnzVcQQAAAACOCSEJds65LyRlHmKXcyT9w1X4RlILM+sQilqONs9fMVLj+yQdcp/kW95XcZlPzjnlFpc1UGUAAAAAwpVXY+w6Sdpa5fm2ym2NwgtXjNScm8Yfcp9+d36oZ77YoMF3z9aunOIGqgwAAABAODrqJ08xs6vNLNXMUjMyMrwuJygiI0y92jY/7H7TZ62VJO3MKVJJuU/Jt7yvGfM3hro8AAAAAGHGq2C3XVKXKs87V247gHPuGedcinMuJSnp0F0Yw83cX0/Uu9ePO+x+5z35lbIKKrpkPvXZ96EuCwAAAECY8SrYzZR0WeXsmKMl5TjndnpUi2e6t2mmQZ0TlXbv5MPuO3r6J5KkPfkl+uK7DK3dlRvq8gAAAACEiVAtd/CapK8l9TWzbWZ2pZlda2bXVu7ygaQNktIkPSvpF6GoI1xERUZoVPdWtd7/sucX6oyH56mwtFyLNx9qjhoAAAAAjYE557yuodZSUlJcamqq12WEhN/vlFdSriF/mF3nYz+44UQN6JgQgqoAAAAAHE3MbLFzLmX/7Uf95CmNRUSEKbFJtDbcd2adj127K1fhFNABAAAABBctdkehtbtydf2r3yotPV+DOydq+bacWh97zYQeOm1Ae43o1jKEFQIAAADwwsFa7Ah2YeBHf/9aCzfWfSzd+cM76bQB7XRi7yQVlJTro9W7denobiGoEAAAAEBDOFiwi/KiGNTNLZP76fwnv6rzcW8t2a63llSsItGyabSyCst0Ut8kzV+/RyO7t1LPpPhglwoAAADAA4yxCwPDu7bUW784oV7nyCqsWAcvPa9Et7y1Qj94bH6N+y3dmq2sgtJ6vRcAAACAhkVXzDCSuilTz83bqA9X7QrK+S4f003Lt+eoXfM4/fKUXjquY6KSb3lfvdrGa85NE4LyHgAAAACChzF2x5CcwjK9u3yH7nh7ZVDP27ddc63bnSdJunZCT0VHmi4Y3lldWzVVud8pJipCT3/+vSb2TVK/9iyvAAAAADQ0gt0xZldOsUZP/8ST946KMKXdd6aWbMlSUnysurRq6kkdAAAAQGPD5CnHmPaJcXr1quM1uHMLxcdGafIj87RmZ26DvHe53+mZL77XfR+slSRNP3+QLh7VNfC63+/kJEVGmNbuytXizVn68fHMxgkAAACECpOnhLETerZRfGxFNj93aEdJUlx0hF658ng1iY7Ub07vG7L33hfqJOnWt1bonaXbtXRrtrILS9Xjtg/U87YPtHBjps54eJ5u/+9K3fbfFdqRXVTjuYrLfPL5K1qOnXN66atNyioo1awVO7U7t1jXv7pEhaXlSkvPD9nnAQAAAMIZXTGPEX6/U1pGvvq0a15t+47sIp1w/6ceVXWge84dqGFdWuisylk537x2jC58+mv175Cgk/slaWLftvrh018fcNyo7q20cGOm/vHTURrfJ+mQ77F5b4F+/soSPXbJMJZ0AAAAwDGFMXaN2Icrd6p1fGyNgSnctImP0VUn9tD5wzvrd/9ZrkWbMnXTpD6aNKCdmsZUtF4Ov+fjwP5zfz1Rv3tzuX40sovOHdpRUZGHb6RetSNHUx6drzevHaOU5FaB7eU+v5yk6IOco7Tcrxv/tVTXn9xL/TskqNznr9X7hZLP7/TZunSd3K+tzMzTWgAAAFB/BDsos6BUJmlYleDTmCTERenMQR30+qKtkqTPfzNR/1q0VU9+9r1m/epESdKz8zbo6+/3amdOsSRpzk0TVFharjKfXxc8VRGMn/rxcE0e1EGStHxbdqCVNOVPc5RfUi5JmnF5iq58KVV3njVArZvF6Im5aXryx8PVe78WVeecnJMiIkwvf7NZO7KLdHK/thqZ3Eqff5ehv85ep+evGKmmMZFaszNXI7q1Um35/U7Pf7lRf3p/jZ64ZLimDO5Qj+8egGPV+t15Sm7T7KA3rQAARxeCHQJKy/2a9LfPdcHwzvrFxJ668Y1lenfZDq/LCitNoiNVVOar83Ebp5+py55fqGvG99TKHTmatXKXlm3N1oLbTtHx9/1vltMXrhipaS8uOuD4F6eNVISZLnt+oSTpvvMGaWLfJOUVlyunqEyjurfSQx9/p0c/WS9Jmtg3SZ+ty9Ap/dpqxhUj9ewXG5TcppkmDWh3wLlzisr0+sItuurEHoqIqN66l1tcpne+3a6po7oe9o+/cp9fkhQVGaG5a9PVJCZSo5JbKT2vRC2aRmviXz7T9AsG6aS+bQ94/8Qm0Qecb2dOkWat2KWfjutebfvIe+coI69E714/ToM6Jx6yplBxzsnnd563zOLgVmzL0cBOCbRY76e4zKcHP1qnqaO66NSHvtAVJyTr7rOP87osAEAtEOxwSEu2ZGnqM9+otLzij/KbJ/XRXz/+rto+z1+Rop++yPc/FE7qm6S56zLqfZ4bTu6lRz9NO+x+153UU0/M/V6SdMUJyZoyuIN+9+ZybdhTIEm6//xBuuWtFbp4VBcVl/n132+3B/adPLC9sgpLdcbADnLOacOeAl38zDdKzyvRkjsnafg9HyuxSbTeuGaMTn/4i2rvGxVhKvc79Uhqph+O6KJ+7Ztr2ouL9JPRXfXKN1t0/vBOWro1W7N+daJioyJV7vNryqPztW53nr685WR1atEkcK7kW96XJI3p0VqvXT1axWU+FZb61LJptNLS89U6PlatmsXow5U7tSe/VL3bxuvxuWmat36PZt84/oDxqI99sl6Tjmt30DUa09LzVVharr35pRreraWe+ux7zVufoVU7cvXnCwbrvRU79djUYUpsGi2f3+mTNbsDAbouoeLbLVmaMX+jHp06LBCwt2UVqqDEp77tm6uo1KdtWYUHtP4eSkFJuc5/8iut252n5XefpoS4AwN0sDwyZ71GdGupcb3b1Ok455xmrdylU/u3U0xUcILyhyt36tpXluihHw3R+cM7B+WcXnh94RZ9+f1ePfSjITXeWCmuvMkUFx0pqaILdoQd+ufu759/r+mz1urkfm316dp09e+QEOi5IEnf7c7Tok2Z9ZrRuLjMp5IyvxKb1v7nLaeoTBf9/Ws9fskw9Wpb+5/xYMgqKNVfP16nO88aoNioyAZ9bwCoC4IdauXfqVt1x9srteoPp9c4niw9t1ij7vNm/Tw0HgM7JegPZw/UBU99Fdj2wAWDFBkRoV//e9khj+3Uoom2H2QG1qrW3ztZWQWlKvM7xURGaOS9cyRVTOjz19nfaeGmTJ1+XDs99KOh+nrDXk174cAW1MMxk5yTHpk6VO8u26lnLxuh7MIyLdmSpTcXb1PvtvFqlxin2/+7UuvvnayH53wXCNz7nDesUyBYb5x+psY9MDfw+Vo1i9GSOycpq6BUaRn5WrBhr64/uXfg2C/T9qi03F+t9ffjG8croUm02jaP1Z8/WqcTe7XRqO6tVFjmU4SZ5n2XocmDKkL7RX//Rhv25OuTmyYqdXOmTunfTkWlPs2Yv0HLtuVoRLeWSm7dVGcMrOjm65xT91s/kCRdlNJFD1w4WH6/09/mfKdLR3fTqPs+0cWjuuj2KQPknFPzuIoQXlzm05OfpemDFbt07YSe2plTpPG9k3TBiIowVlzm0yl//VwPXDA4EBi/3ZKloV1aKLOgVNFREUqIi9aM+RvVqlm0zhvWWVszC/XH91br49W7NahToh69eJiaxkQqKsLUOj72kNctq6BUy7Zlq1/7BDWJjjxkMFm8OUvvLN2uayb0VGRlkPp2S5a6tm6qknK/hnZuoUWbMvWvRVt146Q+tVr3c0d2kd5eul0/n9BTZha4ibHPpvunSJIKS8u1K6dYZzwyT36/07d3TVJxmT/ws7xvX5/fKXK/VvgnP0vTnz9cF3jer31zffh/4wPP973nvvdKzytW62axiowwZReWaktmoQZXfrYfPv21Pr15gnokxWtbVqHeWrJdfdpV3EhZuT1XN03qoxtO6a2aLN+WrZ5J8WpWOcPzO0u361evL9UPhnTUYxcPO2D/v85ep/jYKF0zoaeKy3yBMHs4ecVlKvM5tWoWow0Z+dqRXXzAzYdb31qu1xZu1fCuLXTHWQM0vGvLWp17f4s3Z6p5XPQBN4/S0vP1xXcZ+uN7q7Xw9lPUtnmcJGndrjz1aRd/0CD+r0VbdNqA9vp6w16N75Ok+Ngo7fvb7UhaorMLS3XBU1/p0YuH6biOR97bIS09Tz2TKupeuT1HZz02X+/fME4DOiTorSXbNWVwB8VFR6qgpFxNYyIDtZ735JfakV2kBbedesTvXVc/fu4bnT2koy4a2VUb9xSoXUJsYFz+/mYu26Hfv7NSC28/VdGRESrz+RVpdkBPlprMWb1b7RPjNLBT3b+vxWU+5RSVKSrClFNUph5HOPlbXnGZmkRH1tiTZOayHRrfu41aNI05onM3pLr8fofC8m3Zioywev2OhBLBDkG17x/9n4zuqvzicr29tKIrZ6+28SxLAHjorz8cojveXnlEXYWrGterjean7an1/k9cMlz5JWX63X9W1PqYU/u305w1uw/6epdWTdSiSYxWbM855HmaxUSqa+tmgbU8v771ZI2ZXrfZgFs3i9H08wepWWyUfvnat8osKA28FhsVoZLK3gyS9MK0kTqpb9tACDmUCJP8+/0ze+vkfhrdo7X6dWiupz77XvGxUfp0bbr+cPZx+u1/lmtrZpH25Jfo6Z+M0He78/TQfr0nFt1+qr7dkqWrX1582M/13GUp+tk/UnX+sE765Sm99WXaHt3x9soD9ktsEq0vfnuSEptE66WvNun3M1dJkv584WBtzSzUY5U9AR6/ZJge+yRN63bnBdYvfW3hFg3unKjCUt9B//+//O7TFGmmNTtz9drCrbpsTDclNonWxAc/U5v4WKXeUfFH/vRZa/T3zzdIku49b6C6t26mE3r9L4Dt+7dn5vVjdfbjX+q3Z/TVBcM7q3WzGH26Nl1Xv7xYD180VOcO66R56zMUYab3lu/Uawu3HFDTE5cM13WvLtGyu05TYtNo/ebfy/TvxdsCr08bm6xfn9ZX327J1vsrdmr2ql1674Zxah4XrV05xZq7Nl2PfrJehWU+vTRtlH4yY4Hm/+4kjXtgriTpjWvGqE+7eBWU+rRw417d+K/qN6W+vXOS5q5L101vLNPtZ/bXcR0TdEKvNvpw5S6V+/2aMqiDXvlms+58Z5XaJ8RpV27F2O/XrhqtaS8uVLdWzZSS3FL/XLBFV4/voVHJrfTfpds1oEOCkprH6su0PXpkakU4zsgr0Zjpn+jhqUOVV1yuW9+q+D19+ifDde0rS9S1VVNNGdxBvzql90H/mE7PK1bTmCjFx0YpdVOmLnz6a111Ynf175Cgfy7YosWbs3TTpD7q3yFBV/0jVddM6KErx3XXqHs/0S2T++naCT2rXcM5N41Xr7bNlZFXou8z8hUVYXrkk/W6YHhnje+TpFbNqoePjLwSFZaWq1vrZpKkjXsK9PHqXSoo8enGSX3k9ztt3FtQbTbs7dlFevaLDXrxq02SpFeuPF4/mbFAo5Jb6bWrR2vOmt3q0aaZynxORWXlen3h1sDPQN92zfX0pSN00oOfqVvrptq8tzDwu794c5YGdkrQ1sxCtUuI06C7Z1erdd8NkRnzN6p1sxjFx0bp261Z+s3p/fThyp1qHhetNTtzNaxrS32zYa+uGd9DP35ugRZszFR8bJTyS8oD59hfSblPRaW+QDhbvE2iTxoAACAASURBVDlL2YWluvKlVM25abxOfegLTRncQU9cMlxz16VrXK82io6M0Oa9BZrwl88kVcwxcPJfP9fM68dqQIf/9VJ57NM0nTu0kzLyi7VmZ55+Mrp6i312YanOe/Ir3XfeII3p2VqSlF9SrvjYKP3x3dVq2TRakwd1UHpucbXf26JSn8wq1jd+fdFWje3ZWvGxUWqbEKfiMp9e/GqTrjghOfCzd9Zj87Rye64+vXmCOrZoos/WZejaVxZr+vmD1KddvI7rmKhrXl6sO6b0P2zvlVv+s1z92jfXFWO767b/rtCrC7Yo9Y5T1aaGG3zpucV68atNumZCTw35w+xq1/JoQ7BDUKWl5yshruKX0ud3+u2by3XV+O7q1z5BZT6/et8+S785va/OH94p8AfWrF+dqJXbc/SbN5dLkk7o2Vpp6flKzyvx8qMAQJ3VtmUY9de3XXOt251Xp2OqBqHDiYww/SilS43h79LR3fTyN5vr9N7h7oLhnfWfJdsOv2MNBndO1PJtNd+IeeCCQerYookunbGwTud89rIU/er1b1VYWnGzau09Z6jfnR8e8ph948tD4dyhHQM3sw/mhpN7qVlslKbPWltt+4vTRuqKOvT+uPe8gerdtrk+XLlLq3bk6Mpx3fXCl5v09Ya9hz32zxcM1m//U/H31o9SOiunqEwfrTr4jbQHfzjkoD1i/nbREJ03rLNueO1bzayck+HuHwxQy2Yx+tXrS5UQF6Xc4vJqx1Qd8nEw5w/vpLeWbA88v/HUPsoqLA0E8YPp3TZe66vcRPrLhYP1x3dXKzoqQovvOFVX/WOx4mMjdeW4HvrB4xXLay37/WmBsCZJQ7u00K2T+2nBxkxFRVq1HgxVEexCiGAXnpJveb/aHdnl27I1sGNioFvDqh052pFdrF/8c7HKfE6//8EAnT+8s6Y+803gDnxtfXLzBJ3y18+D/hkAAADQuBDsQohgF542ZOSrfWLcQfuz77M7t1jf7c7Tib3/twD5/mNLXpg2UqOSW+nxuWl66rPv9flvJqpb62b6ePVuFZSU69xhnart/5PnFmh+2h51SIwLLGGwz+Vjuumlrw+8E9uvfXOt3VW3u8MAAAA4tqTdO/monPmaYIewlHzL+zqlX1s9dskwxUZFBiYA8PudcovLgjIAeFdOsXblFismMkJdWzeVSTru9x8FXl97zxn654Ituue91dWOm3PTeDWJidK0FxYquXUzzV5d0cXhvV+OU1Sk6YyH59Xq/bu2aqrbp/TX3TNXaWdOsXokNVPzuGgt25pdr8/VPC5Keft1jQAAAEDtXDuhp26Z3M/rMg5AsENYyiwoVXxsVNCmP6+P73bnqUl0pDbsKVBxmU+nH9e+2uvFZT45JzWJ+d/A8753zFJJuV8bp5+p+Wl7tGlPgUZ2b6XZq3brhlN6K7e4TPExUYqIMJWU+/Sn99boxkl9FB8bpdHTP6k2gcNtZ/bTfR9U9NdffMepmr16t259a4VeufJ45RSV6a+z1+m5y1O0aW+BUpJbKSEuutrMdqf/7Qut252nswZ30O1T+qtpdJRe/maTHpxdMTHDZWO66R81tGBKUlLzWGVUjoU847j2+nDVrlp9zw43PuW4jglataNu3W0BAAAawoQ+SXrpp6O8LuMABDvAAxl5JcotLqs2S1dtOedU7nf6ePVuLd2arZsm9dH63fmavXqXbj6tr6TDTwf8yJz1em7+Bq24+3Sl5xVrxvyN+u3p/QItn845/W3Oek3q367aIt/7WkQTm0Rra2aRurZuqj35JVq7M++AKcKH/XG2fnZiD23LKtJrC7dofJ8kdUyMU3xslG49s7963lYx/f2SOydp894CtWgao5Me/ExS9b7rxWW+wID4B384RIM7J+rLtD36y0frNKJbS/3h7OPUIyleT332vdonxuq8YZ21dleunp+/UW+kVgz0nzKog6aO6qIeSfEae//BZ0U8d2hHPTx1mHIKy7Rie45+/spi3T6lv07p305JzWO1K6dYlzz3jTZkVKzr17Z5rP507sADZiFcdPupmrlsh9LzigMz+e3z2MXDNGVQB/Wo/Pz7rPzD6Xp/+Q51adVUzWKi9Oy8DXpv+U4tuv1UfbNhr/zOVZtp8fWrR+v1hVsOO1i/Ji2bRiursCzwfES3lpo8sL0Sm0QHJjGqrwtHdNablbPIPXrxMN3w2rc6bUA7tY6P0WsLtx70uH1rFza0U/u31Zw16Q3+vgCA8PPqVcfrhJ51W5e1ITRosDOzMyQ9IilS0nPOufv3e/0KSX+RtG8qnMedc88d7rwEOyD8lJT75PdXb8n8Mm2PtmYWamrldOn7/PfbberbLkEDOta8SPjB/PTFRcovKdcb14wJbMspKtOyrdm67PmFuvHUPpo2LlkxkRG1XhfHOaetmUX6PiNfAzomqF1CnMp9fj0zb4MKS3w6pX9bDauyztX27CKNvf9TPXbxML2RulVP/2SEmsVG6Y3Urfr9O6tUVObTgz8cogtHHH6h7G1ZhZo+a63uP3+QmsdFVwu9UsUU9GN6tNaJf54b2Dbz+rFqnxinh2Z/p9cXbdXrV49W//YJio4yrd+dr77tm1f77NuyCmVWsSbZe8t36hcTe2rQ3bPVp128nrk0RTuyi9S/Q4L+Mnud7jprgB7/NE2Pz03T738wQGcMbK9f/3uZLhjeWecM7aQv0/bo/eU79cCFg2v8PMu3Zeu6V5fov78Yq5Q/Vayxtun+KTrj4S+0J780MLFSQUm5oiJNH67cpV05xXpu/ka9NG2UBnRM0FtLtummN5bpmUtHBAL2kC4ttGxrtqafP0ixUREqKPVpTI9WSs8t0asLt+i95Ts1ZXAHdWrRRL89ve8B4ySue3WJ3l++U5eP6aafT+yl615dopjICC3dmh1YLuLsIR11zzkD9f2efPVsE68FG/cG3r+mFumhXVqod9v4atPmSxVrGd7z3hr1TGqmBRsz9YMhHfXusv+F9U4tmmjm9WP17ZZstWkeq6FdWgRa3Mf2aq0v0w49C97SuyZp8eYsXflSqmIiI1Tq8ysqwvT61aN14dNfS5KuP6mXfjK6m6IjTSP+NKfa8f+6erQueuYbSQpMBX73zFVKbBKtRz5ZH9ivps88ZVAHvb9ipyQpOtJU5nOacXmKhnRpoZ+/sliLNmVJkq44IVlLtmRpQ0aB8ksO7CZ+Yu82mre+YomN/ZeYmNAnSfeeNzCwlMC+aeuD4dObJ+i6V78NTNa19K5JGvrHjw95TIum0cquctOktoZ2aaGldexm//BFQ/V///rfzZ5N90/Ry19v0p3vrKrz++/zt4uGaHDnFoHJxqaNTdYLX26qts+gTol64pLhGv+XuTWc4dAO1hvj7evG6twnvjzs8Sf1TdLcQ8xqefuZ/dWiae1uUF0wvLPG9mqtm9449BqodfHOdWN1Ti0+R209dvEwpW7K1KJNWVpdZdK4YM/uebDv/741V71w6+R+B8weejRp9JOnmFmkpO8kTZK0TdIiSRc751ZX2ecKSSnOuevrcm6CHYC6WrgxUyO6tTxggeaGNHdduqa9sEif3DzhiFpvpYqALEnfpxcEgu+89RlaszNXZw/ppPaJFYsd+/1Ou/OK1SGxSZ3fo8znV4TZQb9X6XnFSoqPPaJFkfcpLvOpuMx3RONj9+SXqHWzGI289xPtyS/Ryj+crjU7czUyuVWN7/PWku26eFSXg9ZbUFKuOWt265yhnWp8vSY+v9Px983RrZP764IRneXzO5X5/IHg/dH/jVff9s21M6dIv3z1Ww3r2kK/PaOfoitDZWm5X/kl5UqIi9K7y3foxn8tU6cWTfTxTeMPmGAq+Zb3NbpHK71+9Rj97ePv9OHKXXrn+rH6ZsNeje+dpN15xWrbPO6wP9tFpT7lFZepbUJcYNu+2Yj7tItX2+ZxahITqcWbs9StddMD1ndauT1Hry7cotU7cvX2dWMlSTmFZSr3+7WmshXfOacHPlynqSO7KLlNs8Cxry/colveWqEZl6folP7tAtuXbc3Wve+v0cJNmZKkjdPPlFR9we1/Ltis2/+7Ui9OG6mJfdtKkk7886famlmkZXedpsKycl341Nd67arRemXBZp01uIN6JMWruMynV77ZrKtO7KGMvBI99PF3+vHxXdUhsYmyCks1uHOivli/R+N7twm83/5rg+27rvtuhpT7/Cos8ymnsEzZhWUa0DEh0Bthxd2n6YMVOzUyuZX8TvpsXbomDWgXWDOsWUykCkp9gT8QMwtKA4vLD+nSIvCzUVzm0+7cYt361gp99f1effGbk5RdVKrOLZtq+D0VQfOOKf31sxN7SJIG3PWhCkt9WvWH0wMLvFe95g99vE5dWzfTmQPbq3V8rL7dkqXZq3frd2fUPF7o8+8ydPnzCxUdaXrnunHVbrDNmL9RTaIjddHILoqMMM1asVNdWzfVRyt36dFP0/TUj4frpH5tdfx9nyinqExL7pwUqFmSfnlyL00Z3EH92lecc1dOsVrHx6j37bMkST8b1113nDVAhaXlMpmaxERq054CpaXna1SPVnJ+Kbe4THPW7Na0sd0D369Lnv1Gd5w1QE2iIzX5kYpx7YvvOFVRERXf03+lbtHFo7qqeVy0pIrfqagIU7nf6ckfD1fnlk109uNf6s6zBugHgzsor6RcOUVluuU/y/Xmz09QUalP7RLiNHdduuQqrl37xDiN7dVGPr+TSYqIMN0/a62e/rxiav8Xrhipl7/ZrD+dO1Bm0pjpn+qaCT10fPdW6tqqqZ6bt1Gb9xbq6w17NW1ssiYP7KBR3Sv+H+ac00Mff6eLRnZR55ZNtX53nib97YvAz9ANJ/fSNRN6qmlMpLrf+kGgm2CZz6/n5m3UlEEdNP4vcwOTwE0a0E7PXnbA3/7avLdATWIi1bZ5xf8T9r2PJP34+K7amVOsZy4doV6V1yft3smBx1V9ecvJKiotV0GJT19v2KuhXVpo054C/XPBFp01uIOmz1qrcb3aKLFJdODmz77fiYenDpNJmtA3KfBzIElzfz1R3ds007Kt2Sop9+uzdeka17uNLnl2gYZ1baHLxyRXu9HxwrSRmtgnSWU+V23YTtWhKA98uFadWzbRS19t0ne78zXvtycpp6hMJeU+dWvdTCZpR3axPly1U/9csEXvXj9O/07dqkuO7xb4t/Vo05DBboyku51zp1c+v1WSnHPTq+xzhQh2AIBjxFdpe3TXzFV675fjat0qLEnrduWpU8smio89cNbgolKfoiIt8Id/OHLOacX2HA3u3KLG19ftylN8XJQ6tTjwRoRzTqt25Gpgp/91E9+yt1CzV+8KhBsvrdqRo5ZNY9Sxhtqlij8s94X2Mp9TYpPoWp03v6Rcm/YUVPvce/NLtCWzUEM6twgsFfT3z7/X9Flrgzpr39bMQiU1j63Tz3BVWQWlyisuV9fWTTVj/kaNSm6lAR0TGuTG2j8XbNaZAzuoZbOD3zTamVOkpjFRtb4WtVXu82vjnoIaF8suKvUpLjqi2k2LmhZTP5jt2UXqmBh3wA2q8sobcRFB+t7OX79HKcktq137qjf7duYU6dUFW3T9yb0UaXbYn7nScr9u/vcy/fq0PoGbKm2bx+rtpds1ZVDHaiGsLj93zjl9sqYi7C3cmKnxfZJq3G9rZqF25xYrpcqNv3KfX3vyS4/asFYXDRnsLpR0hnPuZ5XPL5V0fNUQVxnspkvKUEXr3o3OuYMPxqhEsAMAAOEgLT1PrZvFHjJoAMCROFiw8+o24LuSkp1zgyV9LOmlg+1oZlebWaqZpWZkBK+fMQAAQKj0atucUAegQYUi2G2X1KXK88763yQpkiTn3F7nXEnl0+ckjTjYyZxzzzjnUpxzKUlJNTe3AgAAAEBjFopgt0hSbzPrbmYxkqZKmll1BzPrUOXp2ZLWhKAOAAAAAGgUDhytXU/OuXIzu17SR6pY7uB559wqM/ujpFTn3ExJN5jZ2ZLKJWVKuiLYdQAAAABAY8EC5QAAAAAQJo62yVMAAAAAAEFCsAMAAACAMEewAwAAAIAwF1Zj7MwsQ9Jmr+uoQRtJe7wuopHjGniPa3B04Dp4j2vgPa6B97gGRweug/dCcQ26OecOWAcurILd0crMUmsawIiGwzXwHtfg6MB18B7XwHtcA+9xDY4OXAfvNeQ1oCsmAAAAAIQ5gh0AAAAAhDmCXXA843UB4BocBbgGRweug/e4Bt7jGniPa3B04Dp4r8GuAWPsAAAAACDM0WIHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAAAAAGGOYAcAAAAAYY5gBwAAAABhjmAHAMB+zGyTmZ2637aJZuY3s3wzyzOzdWY2zasaAQCoimAHAEDt7XDOxUtKkHSjpGfNrK/HNQEAQLADAKCuXIUPJGVKGux1PQAARHldAAAA4cbMIiSdJamNpDSPywEAgGAHAEAddDSzbElNVPFv6E3OuW89rgkAALpiAgBQBzuccy1UMcbuUUkne1wPAACSCHYAANSZc65E0u8kDTKzc72uBwAAgh0AADWLNrO4fV/ab/iCc65U0l8l3eVJdQAAVGHOOa9rAADgqGJmmyR122/zl5KSnXOdq+zXVNIWSdOcc+82XIUAAFRHsAMAAACAMEdXTAAAAAAIcwQ7AAAAAAhzBDsAAAAACHMEOwAAAAAIcwQ7AAAAAAhzUYff5ejRpk0bl5yc7HUZAAAAAOCJxYsX73HOJe2/PayCXXJyslJTU70uAwAAAAA8YWaba9pOV0wAAAAACHMEOwAAAAAIcwQ7AAAAAAhztQp2ZnaGma0zszQzu6WG12PN7F+Vry8ws+Qqr91auX2dmZ1eZfsmM1thZkvNjIFzAAAAAHCEDjt5iplFSnpC0iRJ2yQtMrOZzrnVVXa7UlKWc66XmU2V9ICki8xsgKSpko6T1FHSHDPr45zzVR53knNuTxA/DwAAAAA0OrVpsRslKc05t8E5VyrpdUnn7LfPOZJeqnz8pqRTzMwqt7/unCtxzm2UlFZ5PgAAAABAkNRmuYNOkrZWeb5N0vEH28c5V25mOZJaV27/Zr9jO1U+dpJmm5mT9Hfn3DN1L99be/JLdO/7a474eDPpqhN7qH+HhCBWBQAAAKCx8XIdu3HOue1m1lbSx2a21jn3xf47mdnVkq6WpK5duzZ0jYdUWu7X4s1ZR3z8lsxCdUxsQrADAAAAUC+1CXbbJXWp8rxz5baa9tlmZlGSEiXtPdSxzrl9/003s/+qoovmAcGusiXvGUlKSUlxtai3wXRs0URf/PakIz6+9+0fyOeOqo8EAAAAIAzVZozdIkm9zay7mcWoYjKUmfvtM1PS5ZWPL5T0qXPOVW6fWjlrZndJvSUtNLNmZtZcksysmaTTJK2s/8cJLxFm8vsJdgAAAADq57AtdpVj5q6X9JGkSEnPO+dWmdkfJaU652ZKmiHpZTNLk5SpivCnyv3ekLRaUrmk65xzPjNrJ+m/FfOrKErSq865D0Pw+Y5qkRGmcoIdAAAAgHqq1Rg759wHkj7Yb9tdVR4XS/rhQY69V9K9+23bIGlIXYs91kSayUewAwAAAFBPtVqgHKEREWHyM8YOAAAAQD0R7DwUFUGLHQAAAID6I9h5iBY7AAAAAMFAsPMQY+wAAAAABIOXC5Q3epERpneW7tDcdRlHdPwlo7rqxkl9glwVAAAAgHBDsPPQzaf10aJNWUd07Merd2vRpswgVwQAAAAgHBHsPHT+8M46f3jnIzr2+4x8unECAAAAkMQYu7AVaUy8AgAAAKACwS5MRUWaymmxAwAAACCCXdiKMJOfYAcAAABABLuwFRlh8tEVEwAAAIAIdmErMsLk83tdBQAAAICjAcEuTEXSFRMAAABAJZY7CFOREabt2UW6/tUlR3T8cR0T9fOJPYNcFQAAAAAvEOzC1Im922jNrlyt3plb52P35pfqkzXpBDsAAADgGEGwC1NTR3XV1FFdj+jY6bPW6IUvNwW3IAAAAACeYYxdIxQVwfg8AAAA4FhCsGuEIo2lEgAAAIBjCcGuEYqIMDknWu0AAACAYwTBrhGKijBJotUOAAAAOEYQ7BqhiH3BjhY7AAAA4JhAsGuEogh2AAAAwDGF5Q4aociIijx/2t++kFndj2/RNFqvXjVaCXHRQa4MAAAAwJEg2DVCp/Zvq+925anM76/zsduzirRgY6Z2ZBcpoT3BDgAAADgaEOwaoW6tm+mBCwcf0bEfrdqlBRszVe6jGycAAABwtGCMHepk3/g8PzNqAgAAAEcNgh3qZN+MmuVMvAIAAAAcNQh2qJNAix3BDgAAADhqEOxQJ5G02AEAAABHHYId6iTSaLEDAAAAjja1mhXTzM6Q9IikSEnPOefu3+/1WEn/kDRC0l5JFznnNlW+dqukKyX5JN3gnPuoynGRklIlbXfOnVXvT4OQi4qsCHavLdqq+Wl76nx8dGSELj8hWa2axQS7NAAAAKDROmywqwxfT0iaJGmbpEVmNtM5t7rKbldKynLO9TKzqZIekHSRmQ2QNFXScZI6SppjZn2cc77K434laY2khKB9IoRUpxZN1apZjD5cubPOxzpX0YWzU4sm+tHILiGoDgAAAGicatNiN0pSmnNugySZ2euSzpFUNdidI+nuysdvSnrczKxy++vOuRJJG80srfJ8X5tZZ0lTJN0r6aYgfBY0gPaJcVpy56QjOjY9t1ij7vvkiBZGBwAAAHBwtRlj10nS1irPt1Vuq3Ef51y5pBxJrQ9z7MOSfiuJv/IbiX0Tr/gYnwcAAAAElSeTp5jZWZLSnXOLa7Hv1WaWamapGRkZDVAdQiUqouLHrdxHsAMAAACCqTbBbrukqgOiOlduq3EfM4uSlKiKSVQOduxYSWeb2SZJr0s62cxeqenNnXPPOOdSnHMpSUlJtSgXR6vISFrsAAAAgFCoTbBbJKm3mXU3sxhVTIYyc799Zkq6vPLxhZI+dc65yu1TzSzWzLpL6i1poXPuVudcZ+dccuX5PnXO/SQInwdHsX1LJfgcwQ4AAAAIpsNOnuKcKzez6yV9pIrlDp53zq0ysz9KSnXOzZQ0Q9LLlZOjZKoirKlyvzdUMdFKuaTrqsyIiUaGMXYAAABAaJgLo9aTlJQUl5qa6nUZOEJ+v1OP2z7QoE6JGtgpsc7Hm0k/SumioV1ahKA6AAAA4OhnZoudcyn7b6/VAuVAMEREmE7o2Vrr0/O1K7e4zsfvyS+Rz+cIdgAAAMB+CHZoUK9eNfqIjx17/6cqpxsnAAAAcABPljsAjkRUpMnH4ub4//buPEyu6jzQ+Pt1VXcL7SuKViSQWARmcyN2BmPA4DiWgzfZnoBjMpgEbOPETwbsLMSJJ2Yc2+M42B5ibGOGxYSALRxiwAEMYRMSiB0ZIQRoA+370suZP+qKNI1a6rqq7urqfn/PU0/fOnW/W+fWqVtVX99zzpUkSdI7mNipZhTqwjN2kiRJ0m6Y2KlmFOvCGTUlSZKk3TCxU80o1NV5xk6SJEnaDSdPUc0o1gXbm1vZsLU5V/zgAcW3rqUnSZIk9SUmdqoZjcU6HnxpNUd99e5c8e85ZAw//sOZFa6VJEmSVH0mdqoZV37wcOa+sjZX7K3zl/L6um0VrpEkSZLUO5jYqWYcMWEYR0wYlit2wevreWbZhgrXSJIkSeodnDxF/UKxLmhu9Rp4kiRJ6ptM7NQvFLxUgiRJkvowEzv1C8WCl0qQJElS32Vip37Bi5tLkiSpL3PyFPULxUKwdstOmv7unlzxk0YO5NaLT/I6eJIkSeqVTOzUL3ysaRItrYm2VP5ZuxdWbOSJ19azdWcLQwbUd0PtJEmSpH1jYqd+4bBxQ/nbDx2RK/bHD73CE6+ttyunJEmSei3H2El7USyUDpPmVhM7SZIk9U4mdtJeFLNxdZ6xkyRJUm9lYiftxa4JU7zAuSRJknorx9hJe7HrjN2KDdtzzYq5X32BEYMaKl0tSZIk6S0mdtJeDGwoAPCx//tIrvgIuPPzp3LYuKGVrJYkSZL0FhM7aS9OP2R/vjP7aLY3t5Yd++qarXzv/pd5c9MODhvXDZWTJEmSMLGT9mpAfYFZR0/IFfvU6+v53v0v0+L4PEmSJHUjJ0+RulGxUBqT1+KMmpIkSepGJnZSNyrWlQ6xFq+BJ0mSpG5kYid1o/86Y2dXTEmSJHUfx9hJ3WjXpRJ+/NASfv3Cm2XHFwL+5D3TOHjskEpXTZIkSX2IiZ3UjcYOHcDMKSNZvXkHzy3bUFZsAl5ZvYUDxww2sZMkSdIemdhJ3WhAfYFbLj4xV2xKialX3OmMmpIkSdqrLo2xi4hzImJhRCyKiMt383hjRPwse/yxiJjS7rErsvKFEfG+rGxARMyNiKci4rmI+JtK7ZDUV0QE9YWg2Rk1JUmStBd7TewiogBcDZwLzAA+EREzOqx2IbAupTQN+DZwVRY7A5gNHA6cA3wv294O4IyU0lHA0cA5EXFCZXZJ6juKdXWesZMkSdJedeWM3UxgUUppcUppJ3AzMKvDOrOA67LlW4H3RkRk5TenlHaklF4BFgEzU8nmbP367OZpCamDYiFo9lIJkiRJ2ouujLGbALze7v5S4PjO1kkptUTEBmBUVv5oh9gJ8NaZwPnANODqlNJjeXZA6suKdcG8V9fyjbtezBV/5mFjOWbyiArXSpIkSb1N1SZPSSm1AkdHxHDg9og4IqX0bMf1IuIi4CKAyZMn93Atpeo6YsIwHnl5DS+u2FR2bEtbYuHKTfzwguO6oWaSJEnqTbqS2C0DJrW7PzEr2906SyOiCAwD1nQlNqW0rL2+1wAAGpJJREFUPiLuozQG7x2JXUrpGuAagKamJvukqV+5/sKOJ8e77ve/9xA7WhyfJ0mS1B90ZYzd48D0iJgaEQ2UJkOZ02GdOcAF2fJHgHtTSikrn53NmjkVmA7MjYgx2Zk6ImI/4CwgX18zSbtVX1dHsxOvSJIk9Qt7PWOXjZm7FLgLKAA/Sik9FxFfBeallOYA1wLXR8QiYC2l5I9svVuA54EW4JKUUmtEjAOuy8bZ1QG3pJR+2R07KPVXxUKw0zN2kiRJ/UKXxtillO4E7uxQ9lftlrcDH+0k9mvA1zqUPQ0cU25lJXVdsVDHlh0t1a6GJEmSekDVJk+R1L3q64LFq7bwR9fNyxV/zOThXPKeaRWulSRJkrqDiZ3UR501YywrNmxn+fptZceu3Lidx5esNbGTJEmqESZ2Uh81e+ZkZs/Md4mQv/vl89w097UK10iSJEndpSuzYkrqZ4qFOppbvbqIJElSrTCxk/QO9YWguc0ZNSVJkmqFXTElvUOxro6U4KU3NlFXF2XHDx1Qz5ghjd1QM0mSJO2OiZ2kdxg8oPTRcNa3H8gVX6gLHvvyexk92OROkiSpJ5jYSXqHjx83id8ZOoCWHN0xn3xtPT95eAnrt+40sZMkSeohJnaS3mFwY5HfPXJcrtjGYh0/eXiJk69IkiT1ICdPkVRR9YXSx0pzq5OvSJIk9RQTO0kVVXwrsfOMnSRJUk+xK6akimrIErs/uu5xGorl/++osVjge586liMmDKt01SRJkvosEztJFXXUpGFceMpUtuxoKTt2844Wfvn0Cl5cucnETpIkqQwmdpIqamBDkb/8wIxcsSs3bOeXT69wfJ4kSVKZHGMnqdeoL5Quhm5iJ0mSVB4TO0m9Rn02Jm9ni4mdJElSOeyKKanX2DXxyv/59Utc88DisuMj4H+ecyjnHTux0lWTJEnq1UzsJPUaA+oLXH7uoby6Zkuu+NueWMa8V9eZ2EmSpH7HxE5Sr3Lxfzsod+z9C1fRbDdOSZLUDznGTlKf0VCsc+IVSZLUL3nGTlKfUV+oY93WZhav2pwrfvzw/RhQX6hwrSRJkrqfiZ2kPmNwY5Hf/HYVZ3zzN7niT50+musvPL7CtZIkSep+JnaS+oxvfuwonl22IVfstf/5Cqs27ahwjSRJknqGiZ2kPuOgMYM5aMzgXLH3PP8Gzy/fWOEaSZIk9QwnT5EkStfQ2+nEK5IkqUZ5xk6SKM2ouXrzDi67+clc8VNHD+YLZ06vcK0kSZK6xsROkoATDxrFo4vX8OTr68uO3bCtmZ8vWM7Fpx9IY9FZNSVJUs8zsZMkYNbRE5h19IRcsf/8wGK+ducL7GxpM7GTJElV4Rg7SdpHDcXSR+nOFsfoSZKk6vCMnSTto12J3R1PLWf4wIay4wc1FnnvoftTVxeVrpokSeonupTYRcQ5wHeAAvDDlNLXOzzeCPwUeDewBvh4SmlJ9tgVwIVAK/D5lNJdETEpW38skIBrUkrfqcgeSVIPGzu0EYAr73g+9zZ+fsnJHD1peKWqJEmS+pm9JnYRUQCuBs4ClgKPR8SclFL7XzAXAutSStMiYjZwFfDxiJgBzAYOB8YDv46Ig4EW4M9SSk9ExBBgfkTc02GbklQTzjh0LA9dfkaurphPL13PF25ewObtLd1QM0mS1F905YzdTGBRSmkxQETcDMwC2idhs4Ars+VbgX+KiMjKb04p7QBeiYhFwMyU0iPACoCU0qaIeAGY0GGbklQzJgzfL1fchm3NAOxsba1kdSRJUj/TlcRuAvB6u/tLgeM7Wyel1BIRG4BRWfmjHWLfNu1cREwBjgEeK6PektQnNGbj8558bT2l/4eVp1gXHDdlJAPqnY1TkqT+rKqTp0TEYOBfgctSShs7Weci4CKAyZMn92DtJKn7jRxUmmzlu/cuyr2Nr846nPNPnFKhGkmSpFrUlcRuGTCp3f2JWdnu1lkaEUVgGKVJVDqNjYh6SkndDSml2zp78pTSNcA1AE1NTakL9ZWkmjF26ADu+9LprN+6s+zYtpT48PcfYcPW5m6omSRJqiVdSeweB6ZHxFRKSdls4JMd1pkDXAA8AnwEuDellCJiDnBjRHyL0uQp04G52fi7a4EXUkrfqsyuSFJtmjp6EDAoV2xdwM5Wr58nSVJ/t9fELhszdylwF6XLHfwopfRcRHwVmJdSmkMpSbs+mxxlLaXkj2y9WyhNitICXJJSao2IU4A/AJ6JiAXZU305pXRnpXdQkvqyxmKBJ15bx08fWZIr/tTpY7LEUpIk1bJIqXZ6NzY1NaV58+ZVuxqS1Gu879sPsPCNTbnjf/fIcVz9yWMrWCNJktSdImJ+SqmpY3lVJ0+RJO2bOz53Cpu25xtj9wfXzmXrDq+fJ0lSX2BiJ0k1rKFYx6jBjbliBzUW2JHjouqSJKn3MbGTpH6qsVjgqdfXM/uaR3LFz5w6ij896+AK10qSJOVRV+0KSJKqY9bR4zls/FDaEmXfFq/awo2PvVbtXZAkSRnP2ElSP/XRpkl8tGnS3lfcjb+54zlunbe0wjWSJEl5mdhJksrWWCywrbmVhxetzhU/cnADh/7O0ArXSpKk/svETpJUtlGDGmhpS3zyh4/lio+A+X9xFiMHNVS4ZpIk9U8mdpKksl1w0hSOmTyc1rbyr4X6n4tW8917F7FhW7OJnSRJFWJiJ0kqW0OxjqYpI3PFrt2yE4Dtza2VrJIkSf2aiZ0kqUft11AA4MPff5hCRNnxjfUFrvvMcRw+flilqyZJUs0ysZMk9ajjpozk0vdMY+vO8s/YbdzezK3zl/LSG5tN7CRJasfETpLUowY1FvnS+w7JFbt8/TZunb/UbpySJHVgYidJqhn71Ze6cX733kXcOLf8C6RHBF88czqnH7J/pasmSVJVmdhJkmrG8IH1/MEJB7B03dZc8f+5aDX3L1xlYidJ6nNM7CRJNSMi+NsPHZE7/vj/9Wu7cUqS+iQTO0lSv7FffYF/f3Ylzy7fkCv+g0eN56LTDqpwrSRJ2ncmdpKkfuPTJ03hwZdW54p9aul6/u3pFSZ2kqReycROktRvfPrkqXz65Km5Yv/4/83n5VWbK1wjSZIqw8ROkqQuGNhQ5LdvbOawv/xVrviDf2cIv7jk5ArXSpKkEhM7SZK64DOnTGHU4IZcsU++to7Hl6yjpbWNYqGuwjWTJMnETpKkLjl8/DAOHz8sV+wPH1zM40vWcf/CVQxsKJQdP2RAPe+amO+5JUn9g4mdJEndbPTgRgD+6Kfzcm/jni+exvSxQypVJUlSH2NiJ0lSN/vAkeOYPGogzS1tZcc+v2Ijf3PH86zevJPpY7uhcpKkPsHETpKkblYs1HHs5BG5YhvrS103r/rVi2+d+StHfSH4s7MPYdr+g3M9vySpNpjYSZLUix00ZhAnHjiKDduaWb5+W1mxbSnx4spNvPuAESZ2ktTHmdhJktSLDRlQz00XnZArtqW1jWlf+XdeXLmJhxfluzD7EROHMXRAfa5YSVLPMbGTJKmPKhbqGDmogVvnL+XW+UtzbeMTMyfx9+cdWeGaSZIqzcROkqQ+7BeXnFx2F85dLr/tGVZt2lHhGkmSuoOJnSRJfdikkQOZNHJgrtgxQxq5b+Eq3vXXd+WKP/3Q/fnuJ47JFStJKo+JnSRJ2q0vvHc6//HCm7liH355NY8tXlPhGkmSOtOlxC4izgG+AxSAH6aUvt7h8Ubgp8C7gTXAx1NKS7LHrgAuBFqBz6eU7srKfwR8AHgzpXRERfZGkiRVzMnTRnPytNG5Yv/ul8/z44eX8IlrHs0VP27YAL7x0aMo1EWueEnqb+r2tkJEFICrgXOBGcAnImJGh9UuBNallKYB3wauymJnALOBw4FzgO9l2wP4SVYmSZL6mDNnjKXpgBG0tqWybys3bue2J5c5vk+SytCVM3YzgUUppcUAEXEzMAt4vt06s4Ars+VbgX+KiMjKb04p7QBeiYhF2fYeSSk9EBFTKrETkiSpdznhwFH87LMn5oq946nlfO6mJ/ny7c8wqLH8USP71ddx+bmHMXJQQ67nl6Ra1JVPywnA6+3uLwWO72ydlFJLRGwARmXlj3aInZC7tpIkqc87cuIwZowbypLVW8qO3dnaxtJ123jPIftz7rvGdUPtJKl36vWTp0TERcBFAJMnT65ybSRJUnc7YNQg7vzCqblil67byilX3ccPfvMydzy9vOz4iODi0w7iXROH5Xp+SaqWriR2y4BJ7e5PzMp2t87SiCgCwyhNotKV2D1KKV0DXAPQ1NSUyomVJEn9y9ihAzh1+mhWbtjOS29sLjt+0arNjB82wMROUs3pSmL3ODA9IqZSSspmA5/ssM4c4ALgEeAjwL0ppRQRc4AbI+JbwHhgOjC3UpWXJElqr75Qx/UXdhwx0nUzv/Zrbn9yGY8vWZcr/rxjJ3D+iVNyP78k5bXXxC4bM3cpcBelyx38KKX0XER8FZiXUpoDXAtcn02OspZS8ke23i2UJlppAS5JKbUCRMRNwOnA6IhYCvx1Sunaiu+hJElSF/3hyVN5JOf1955dtoGfP7nMxE5SVURKtdO7sampKc2bN6/a1ZAkSXqHS258gn97egWR89J775owjDmXnlLZSknqcyJifkqpqWN5r588RZIkqRZccvo0Dho9KFfs3CVreXTxWu598Q2C8jPDofvV8+4DRuR6bkl9g4mdJElSBcwYP5QZ44fmir1l3us8ungtn/lJ/p5J933pdKbmTCwl1T4TO0mSpCr78LETmTFuKC1t5Q+ReXbZBv7i589yxW1PM2Jg+Rdlry/U8aWzD2HyqIFlx0rqPUzsJEmSqqxQFxwxId8lFg4YOZDbnljK2i07WbtlZ1mxbQkWvbmZKaMH8eFjJ+R6/okjBlKoyzmwUFLFOHmKJElSP9XaljjsL3/Fzta23Nv47GkHcsX7D6tgrSTtiZOnSJIk6W0KdcGN/+N4Xlu7NVf8N+/+Lb9YsJxX1+SLP2X6aP77CQfkipX0diZ2kiRJ/VjTlJE0TRmZK3b5+m3c8dQKXlm9pezYlRu3M/+1dYwfPiDXc+8/ZEDu7qtSX2RXTEmSJPW4b929kH+8d1Hu+GJdsOCvz2Zwo+cp1L901hXTxE6SJEk9bmdLGy+u3Eien6IPvrSKf7j7t9QXItd1//ZrKHDbn5zEQWMGl//kUpU5xk6SJEm9RkOxjiMnDs8VO3XMIFraEjtayp/0Zf3WZm6a+xqfu/FJRg0u//IQhbrgsjMP5uhJ+eoudRcTO0mSJNWUoQPquezMg3PF7mxpY/XmHazevIPNO1rKjn/q9fVs3NbMcVPzjUs867Cxucc0SntiYidJkqR+o6FYxz+f/45ebF326R/P5ZGX1/Dc8o1lx+5sbePeF97kgpOm5Hru6fsP5vgDR+WKVd/nGDtJkiSpB3zl9me44bHXcscPaSzy9Q8fmSt2xMB6Tpo2Ovdzq/dw8hRJkiSpitraEqu37MgV+4snl/O1O1/Yp+f/+/PexYiB5Y8rbCzWcfK00TQU6/bp+VUZJnaSJElSjWprS7yyZgutbeX/dn9l9RY+e/38fXr+Mw/bn987anzZcRHByQeNYtTgxn16fv0XEztJkiSpn3p1zRa27GgtO665tY1ZVz+0T889fGA9l713eq7YYw8YkXv21L7KxE6SJElS2dZt2cnarTtzxV5ywxO8uHLTPj3/hadMzRV3wKiBnH/ilH167t7IxE6SJElSj2ptS2zc1pwr9obHXuX7979MRPkXod91KYtjJg9ncGP5FwIY3Fjk6+cdybCB9WXHdjcvUC5JkiSpRxXqghGDyp+wBeDSM6Zz6Rn5unAuenMTX7n9WXa2tpV9vcLnlm9kZ0sbnzr+AE6ZXjsziZrYSZIkSepTpu0/hJ999sRcsUvXbeWHD77C2KG1NeGLiZ0kSZIkZSaOGMiVHzy82tUomxejkCRJkqQaZ2InSZIkSTXOxE6SJEmSapyJnSRJkiTVOBM7SZIkSapxJnaSJEmSVONM7CRJkiSpxkVKqdp16LKIWAW8Wu167MZoYHW1K9HP2QbVZxv0DrZD9dkG1WcbVJ9t0DvYDtXXHW1wQEppTMfCmkrsequImJdSaqp2Pfoz26D6bIPewXaoPtug+myD6rMNegfbofp6sg3siilJkiRJNc7ETpIkSZJqnIldZVxT7QrINugFbIPewXaoPtug+myD6rMNegfbofp6rA0cYydJkiRJNc4zdpIkSZJU40zs9kFEnBMRCyNiUURcXu369CURMSki7ouI5yPiuYj4QlZ+ZUQsi4gF2e397WKuyNpiYUS8r1257ZRTRCyJiGey13peVjYyIu6JiJeyvyOy8oiIf8xe56cj4th227kgW/+liLigWvtTiyLikHbv9wURsTEiLvNY6F4R8aOIeDMinm1XVrH3fkS8Ozu2FmWx0bN72Pt10gbfiIgXs9f59ogYnpVPiYht7Y6HH7SL2e1r3Vl76u06aYeKff5ExNSIeCwr/1lENPTc3tWGTtrgZ+1e/yURsSAr91joBtH579Le9b2QUvKW4wYUgJeBA4EG4ClgRrXr1VduwDjg2Gx5CPBbYAZwJfCl3aw/I2uDRmBq1jYF22mf22EJMLpD2f8GLs+WLweuypbfD/w7EMAJwGNZ+UhgcfZ3RLY8otr7Vou37P28EjjAY6HbX+vTgGOBZ9uVVey9D8zN1o0s9txq73Nvu3XSBmcDxWz5qnZtMKX9eh22s9vXurP29NaldqjY5w9wCzA7W/4B8MfV3ufedttdG3R4/JvAX2XLHgvd0wad/S7tVd8LnrHLbyawKKW0OKW0E7gZmFXlOvUZKaUVKaUnsuVNwAvAhD2EzAJuTintSCm9Aiyi1Ea2U+XNAq7Llq8DPtSu/Kep5FFgeESMA94H3JNSWptSWgfcA5zT05XuI94LvJxSenUP63gsVEBK6QFgbYfiirz3s8eGppQeTaVv85+225Yyu2uDlNLdKaWW7O6jwMQ9bWMvr3Vn7al2OjkWOlPW5092RuIM4NYs3nbYjT21QfYafgy4aU/b8FjYN3v4XdqrvhdM7PKbALze7v5S9px4KKeImAIcAzyWFV2andb+UbvuAp21h+20bxJwd0TMj4iLsrKxKaUV2fJKYGy2bBt0v9m8/cvbY6FnVeq9PyFb7liu8nyG0n+1d5kaEU9GxG8i4tSsbE+vdWftqa6pxOfPKGB9u2TdY6F8pwJvpJRealfmsdCNOvwu7VXfCyZ26tUiYjDwr8BlKaWNwPeBg4CjgRWUuh+o+5ySUjoWOBe4JCJOa/9g9l8lp9btAdm4kw8C/5IVeSxUke/96oqIrwAtwA1Z0QpgckrpGOBPgRsjYmhXt2d7ls3Pn97jE7z9H34eC91oN79L39IbXjsTu/yWAZPa3Z+YlalCIqKe0sFzQ0rpNoCU0hsppdaUUhvwz5S6d0Dn7WE77YOU0rLs75vA7ZRe7zeyLgO7una8ma1uG3Svc4EnUkpvgMdClVTqvb+Mt3chtC3KEBGfBj4AfCr7IUXW9W9Ntjyf0niug9nza91Ze2ovKvj5s4ZSF7Vih3J1Qfa6nQf8bFeZx0L32d3vUnrZ94KJXX6PA9Oz2ZwaKHWRmlPlOvUZWZ/xa4EXUkrfalc+rt1qvw/smiFqDjA7IhojYiowndIgVNspp4gYFBFDdi1TmrTgWUqv365ZnC4AfpEtzwHOz2aCOgHYkHVPuAs4OyJGZN11zs7KVJ63/VfWY6EqKvLezx7bGBEnZJ9157fblvYgIs4B/hz4YEppa7vyMRFRyJYPpPS+X7yX17qz9tReVOrzJ0vM7wM+ksXbDuU5E3gxpfRWFz6Phe7R2e9Setv3QjkzrXh7xww576c0K87LwFeqXZ++dANOoXQ6+2lgQXZ7P3A98ExWPgcY1y7mK1lbLKTdTEK2U+42OJDSzGVPAc/teu0ojYn4D+Al4NfAyKw8gKuz1/kZoKndtj5DaRD9IuAPq71vtXYDBlH6z/awdmUeC937mt9EqUtTM6WxDhdW8r0PNFH6Mfwy8E9AVHufe9utkzZYRGl8yq7vhR9k6344+5xaADwB/N7eXuvO2tNbl9qhYp8/2XfN3Kxt/wVorPY+97bb7togK/8JcHGHdT0WuqcNOvtd2qu+F3Y1qCRJkiSpRtkVU5IkSZJqnImdJEmSJNU4EztJkiRJqnEmdpIkSZJU40zsJEmSJKnGmdhJkvqUiNic/Z0SEZ+s8La/3OH+w5XcviRJeZnYSZL6qilAWYldRBT3ssrbEruU0kll1kmSpG5hYidJ6qu+DpwaEQsi4osRUYiIb0TE4xHxdER8FiAiTo+IByNiDvB8VvbziJgfEc9FxEVZ2deB/bLt3ZCV7To7GNm2n42IZyLi4+22fX9E3BoRL0bEDRERu7YXEc9ndfmHHn91JEl9yt7+MylJUq26HPhSSukDAFmCtiGldFxENAIPRcTd2brHAkeklF7J7n8mpbQ2IvYDHo+If00pXR4Rl6aUjt7Nc50HHA0cBYzOYh7IHjsGOBxYDjwEnBwRLwC/DxyaUkoRMbziey9J6lc8YydJ6i/OBs6PiAXAY8AoYHr22Nx2SR3A5yPiKeBRYFK79TpzCnBTSqk1pfQG8BvguHbbXppSagMWUOoiugHYDlwbEecBW/d57yRJ/ZqJnSSpvwjgcymlo7Pb1JTSrjN2W95aKeJ04EzgxJTSUcCTwIB9eN4d7ZZbgWJKqQWYCdwKfAD41T5sX5IkEztJUp+1CRjS7v5dwB9HRD1ARBwcEYN2EzcMWJdS2hoRhwIntHuseVd8Bw8CH8/G8Y0BTgPmdlaxiBgMDEsp3Ql8kVIXTkmScnOMnSSpr3oaaM26VP4E+A6lbpBPZBOYrAI+tJu4XwEXZ+PgFlLqjrnLNcDTEfFESulT7cpvB04EngIS8OcppZVZYrg7Q4BfRMQASmcS/zTfLkqSVBIppWrXQZIkSZK0D+yKKUmSJEk1zsROkiRJkmqciZ0kSZIk1TgTO0mSJEmqcSZ2kiRJklTjTOwkSZIkqcaZ2EmSJElSjTOxkyRJkqQa9/8BNC6JjJi0NM8AAAAASUVORK5CYII=\n",
1343 |       "text/plain": [
1344 |        "<Figure size 1080x576 with 2 Axes>"
1345 |       ]
1346 |      },
1347 |      "metadata": {
1348 |       "needs_background": "light"
1349 |      },
1350 |      "output_type": "display_data"
1351 |     }
1352 |    ],
1353 |    "source": [
1354 |     "fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(15, 8))\n",
1355 |     "ax1.plot(iteration_losses[::])\n",
1356 |     "ax1.set_title(\"Loss\")\n",
1357 |     "ax2.plot(iteration_lrs[::])\n",
1358 |     "ax2.set_title(\"LR\")\n",
1359 |     "plt.xlabel(\"Iterations\")\n",
1360 |     "plt.show()"
1361 |    ]
1362 |   },
1363 |   {
1364 |    "cell_type": "code",
1365 |    "execution_count": 44,
1366 |    "metadata": {},
1367 |    "outputs": [
1368 |     {
1369 |      "data": {
1370 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2oAAAD4CAYAAACdbRXeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxddZ3/8ffn3uz72rTN0nRf6EJXyl4BsUUWFUVQcAFFUHQcdRR/OsqM/nQY/c3o6CCDDIIbiLLIjrJToC0t3fc03ZsmTdLs+73f3x/3JqTNSpvk3CSv5+NxH9x7zvee87332xPyznc55pwTAAAAACBy+LyuAAAAAADgRAQ1AAAAAIgwBDUAAAAAiDAENQAAAACIMAQ1AAAAAIgwUV6dOCsryxUWFnp1egAAAADw1Lp168qdc9nd7fMsqBUWFmrt2rVenR4AAAAAPGVm+3vax9BHAAAAAIgwBDUAAAAAiDAENQAAAACIMAQ1AAAAAIgwBDUAAAAAiDAENQAAAACIMAQ1AAAAAIgwngW1stpmr04NAAAAABHNs6B2jKAGAAAAAN3yLKg557w6NQAAAABENOaoAQAAAECE8a5HTfSqAQAAAEB3+gxqZnafmZWZ2ZZeyiwzsw1mttXMXu3vyYPkNAAAAADooj89avdLWt7TTjNLk3SXpCudc2dI+lh/Tx4gqQEAAABAF30GNefca5IqeynyCUmPOucOhMuX9ffkBDUAAAAA6Gog5qhNk5RuZq+Y2Toz+1RPBc3sZjNba2ZrJaktGByA0wMAAADAyBI1QMdYKOliSfGS3jKzVc65XScXdM7dI+keSYodN9XRowYAAAAAXQ1EUDskqcI5Vy+p3sxekzRPUpegdrI2ghoAAAAAdDEQQx//Kuk8M4syswRJZ0na3p83BglqAAAAANBFnz1qZvagpGWSsszskKTvS4qWJOfc3c657Wb2nKRNkoKS7nXO9biUf2f0qAEAAABAV30GNefcdf0o8xNJP3mvJ2eOGgAAAAB0NRBDH08ZPWoAAAAA0JWnQS3A8vwAAAAA0IXHQc3LswMAAABAZPJ46CNJDQAAAABO5nGPGnPUAAAAAOBkLCYCAAAAABGGHjUAAAAAiDAENQAAAACIMAQ1AAAAAIgwzFEDAAAAgAjDDa8BAAAAIMJ426MWoEcNAAAAAE7maVALOoIaAAAAAJyMOWoAAAAAEGFY9REAAAAAIgxz1AAAAAAgwtCjBgAAAAARxtugxmIiAAAAANAFi4kAAAAAQITxNKg1twa8PD0AAAAARCRPg1pdc5uXpwcAAACAiORZUPOZqa6JoAYAAAAAJ/MwqNGjBgAAAADd8Syo+X2mWnrUAAAAAKALT4c+1tKjBgAAAABdeNqjVtfU6tXpAQAAACBiMfQRAAAAACKMt6s+MvQRAAAAALrwLqj5xPL8AAAAANCNPoOamd1nZmVmtqWPcovNrM3MPtqfE/vNVNfSpmDQ9beuAAAAADAq9KdH7X5Jy3srYGZ+SXdK+lt/T+z3mZyT6lvoVQMAAACAzvoMas651yRV9lHsy5IekVTW7xObSeKm1wAAAABwstOeo2ZmuZI+LOlX/Sh7s5mtNbO19XW1kpinBgAAAAAnG4jFRH4m6VvOuWBfBZ1z9zjnFjnnFqWlpkoSN70GAAAAgJNEDcAxFkl6yEJDGbMkXWZmbc65x3t7kz8cEWsauek1AAAAAHR22kHNOTex/bmZ3S/pqb5CmiRF+UJJrbyu5XSrAAAAAAAjSp9BzcwelLRMUpaZHZL0fUnRkuScu/uUTxzuUiutaTrVQwAAAADAiNRnUHPOXdffgznnPtPfsj6TUuKiVEZQAwAAAIATDMRiIqcsJyVOpTXNXlYBAAAAACKOp0FtTEqsymrpUQMAAACAzrztUUumRw0AAAAATuZxj1qcymqb5JzzshoAAAAAEFE8nqMWq9aA0/EG7qUGAAAAAO287VFLjpMk5qkBAAAAQCee96hJYp4aAAAAAHTi+fL8Eje9BgAAAIDOPA1q2cmhHrVjtfSoAQAAAEA7T4NaXLRfqfHR9KgBAAAAQCeeBjUpNE+NoAYAAAAA74qAoMZNrwEAAACgM8+D2pjkOOaoAQAAAEAn3ge1lFiV1TYpGHReVwUAAAAAIoLnQS0nOVatAafjDS1eVwUAAAAAIoL3Qa3jXmoMfwQAAAAAKRKCWmooqB2tafS4JgAAAAAQGTwPannp8ZKkg5UENQAAAACQIiCoZSfFKi7ap4OVDV5XBQAAAAAigudBzcyUn56gAwQ1AAAAAJAUAUFNkvIzEnTwOEMfAQAAAECKkKBWkJGgQ5UNco57qQEAAABARAS1vPR41Ta3qaqh1euqAAAAAIDnIiKo5WckSBLz1AAAAABAERLUpuUkS5J2Hq31uCYAAAAA4L2ICGoTMhKUHBulTYervK4KAAAAAHguIoKaz2eanZuqzYeqva4KAAAAAHguIoKaJM3NS9X2klq1tAW9rgoAAAAAeCpigtqcvFS1BILaVco8NQAAAACjW59BzczuM7MyM9vSw/5PmtkmM9tsZm+a2bxTqcjc3DRJ0iaGPwIAAAAY5frTo3a/pOW97N8r6ULn3BxJP5B0z6lUJD8jXqnx0drMgiIAAAAARrmovgo4514zs8Je9r/Z6eUqSXmnUhEz05zcVHrUAAAAAIx6Az1H7SZJz/a008xuNrO1Zrb22LFjXfbPyUvV1iM1qmtuG+BqAQAAAMDwMWBBzczep1BQ+1ZPZZxz9zjnFjnnFmVnZ3fZPyc3VZK0ak/FQFULAAAAAIadAQlqZjZX0r2SrnLOnXLKumjGGCXE+PXC9tKBqBYAAAAADEunHdTMrEDSo5JucM7tOp1jxUX7df7ULL2665icc6dbNQAAAAAYlvqzPP+Dkt6SNN3MDpnZTWZ2i5ndEi7yPUmZku4ysw1mtvZ0KnTJzByVVDdpy+Ga0zkMAAAAAAxb/Vn18bo+9n9O0ucGqkIXzRgjM+mF7aWak5c6UIcFAAAAgGFjoFd9PG2ZSbFaUJDOPDUAAAAAo1bEBTVJWjF7rLYeqdGu0lqvqwIAAAAAQy4ig9qH5+cq2m96bP1hr6sCAAAAAEMuIoNaZlKs5ualaVUx91MDAAAAMPpEZFCTpLMmZmjzoWrVN7d5XRUAAAAAGFIRG9QWT8xQW9Bp48Eqr6sCAAAAAEMqYoPawgnp8pm0am+l11UBAAAAgCEVsUEtJS5ac/LStHL3Ma+rAgAAAABDKmKDmiSdPyVL6w9WqaKu2euqAAAAAMCQieig9r4Z2XJOWs3wRwAAAACjSEQHtbl5aUqI8bNMPwAAAIBRJaKDWrTfp0WFGXprD0ENAAAAwOgR0UFNks6dnKndZXU6Wt3kdVUAAAAAYEhEfFBbNn2MJOnlnWUe1wQAAAAAhkbEB7VpOUnKTYvXyzsIagAAAABGh4gPamam86ZkafXeSgWDzuvqAAAAAMCgi/igJklnT85UdWOrth+t8boqAAAAADDohkVQO2tShiRpdTH3UwMAAAAw8g2LoDYuNV4FGQncTw0AAADAqDAsgpokLZ2UodV7K9UaCHpdFQAAAAAYVMMmqF06a6yqG1v16s5jXlcFAAAAAAbVsAlqF07PVkZijB5df8jrqgAAAADAoBo2QS3a79OV88brhe1lqm5o9bo6AAAAADBohk1Qk6SrF+SppS2opzYf8boqAAAAADBohlVQm52bomk5SfrzWoY/AgAAABi5hlVQMzN9fHGBNhys0tYj1V5XBwAAAAAGxbAKapJ09YJcxUf79bu39ntdFQAAAAAYFMMuqKUlxGjF7LF6enOJGlsCXlcHAAAAAAZcn0HNzO4zszIz29LDfjOz/zKzIjPbZGYLBr6aJ/roojzVNrXp5y/uHuxTAQAAAMCQ60+P2v2Slveyf4WkqeHHzZJ+dfrV6t3ZkzK1cEK67n29WAcqGgb7dAAAAAAwpPoMas651yRV9lLkKkm/dSGrJKWZ2biBqmB3zEw/+ehctQWd7nqlaDBPBQAAAABDbiDmqOVKOtjp9aHwtkE1KTtJ1yzK05Mbj6i+uW2wTwcAAAAAQ2ZIFxMxs5vNbK2ZrT127NhpH++aRfmqbwnoL+u4rxoAAACAkWMggtphSfmdXueFt3XhnLvHObfIObcoOzv7tE+8cEK6Fk5I12/e2Cvn3GkfDwAAAAAiwUAEtSckfSq8+uNSSdXOuZIBOG6fzEzXLy3QvooGPbvl6FCcEgAAAAAGXX+W539Q0luSppvZITO7ycxuMbNbwkWekVQsqUjSryV9cdBq240r5+Vqypgk3f3qnqE8LQAAAAAMmqi+Cjjnrutjv5P0pQGr0Xvk95luWDpB339iq17eWab3TR/jVVUAAAAAYEAM6WIig+XDC0KLTN63cq/HNQEAAACA0zciglpKXLRuXTZZK4vKVVRW53V1AAAAAOC0jIigJkmfOadQCdF+ffnB9arjvmoAAAAAhrERE9RyUuJ01/ULteNojX72911eVwcAAAAATtmICWqSdOG0bF27OF/3rtyrlbvLva4OAAAAAJySERXUJOmfL58lSbrhvtVqbgt4XBsAAAAAeO9GXFBLiInSP31gupyTfvUK91YDAAAAMPyMuKAmSV9cNlkfOCNHP3tht0prmryuDgAAAAC8JyMyqJmZvr1ipiTpyw+u97g2AAAAAPDejMigJkmFWYmaOS5Fa/ZWan9FvdfVAQAAAIB+G7FBTZLu/+xixfh9+vmLu72uCgAAAAD024gOajkpcfrU2RP06DuHtfFgldfVAQAAAIB+GdFBTZJuXTZZkvS9v25RIOg8rg0AAAAA9G3EB7XMpFj9xzXztPFQte5/c5/X1QEAAACAPo34oCZJl88dr9y0eP3gqW16bstRr6sDAAAAAL0aFUEtJsqn+z+7WJJ0+6Ob1NQa8LhGAAAAANCzURHUJGlqTrJ+e+MSVTW06l+e3CrnmK8GAAAAIDKNmqAmSRdMy9YXLpykB9cc1J/XHvK6OgAAAADQrVEV1CTpHy+ZJkn65iObVFHX7HFtAAAAAKCrURfU4qL9+tUnF0iSvvLQeo9rAwAAAABdjbqgJkkr5ozT+VOz9EZRhdbsrfS6OgAAAABwglEZ1CTp7usXKjctXp++b42C3AgbAAAAQAQZtUEtMTZKX71kqhpbA7rtwXe8rg4AAAAAdBi1QU2SProwT2kJ0Xpm81GtLq7wujoAAAAAIGmUBzUz09/+8QLlZ8Tr4/es0q9e2eN1lQAAAABgdAc1SRqTHKdHbjlHknTnczv00o5Sj2sEAAAAYLQb9UFNksakxOnN2y9SYWaCbrx/rR5++6DXVQIAAAAwihHUwsanxet3N50lKXQz7MLbn5ZzrAYJAAAAYOgR1DrJz0jQuu9e0vH696sPeFgbAAAAAKNVv4KamS03s51mVmRmt3ezv8DMXjaz9Wa2ycwuG/iqDo3MpFgV/+gy5WfE658f36InNh7xukoAAAAARpk+g5qZ+SX9t6QVkmZJus7MZp1U7LuSHnbOzZd0raS7BrqiQ8nnMz34+aWSpK88uF4/fX4nN8UGAAAAMGT606O2RFKRc67YOdci6SFJV51UxklKCT9PlTTsu6Hy0hP06BdDq0H+8uUi/ecLu1Re18y8NQAAAACDrj9BLVdS52UQD4W3dXaHpOvN7JCkZyR9ubsDmdnNZrbWzNYeO3bsFKo7tBYUpGvvjy/T8jPG6hcvFWnRD1/Qdb9eRe8aAAAAgEE1UIuJXCfpfudcnqTLJP3OzLoc2zl3j3NukXNuUXZ29gCdenCZmX527Zm6ZlGeJGlVcaXuXVnsca0AAAAAjGT9CWqHJeV3ep0X3tbZTZIeliTn3FuS4iRlDUQFI0FctF///tF52vvjyzRrXIp+9MwOvf8/XmUYJAAAAIBB0Z+g9rakqWY20cxiFFos5ImTyhyQdLEkmdlMhYJa5I9tfI/MTD/6yBxJ0u6yOt31yh6PawQAAABgJOozqDnn2iTdJul5SdsVWt1xq5n9q5ldGS72dUmfN7ONkh6U9Bk3QrubzsxP094fX6Zzp2TqJ8/v1Mf/5y395o29amwJeF01AAAAACOEeZWnFi1a5NauXevJuQdCRV2zFv7whY7XhZkJeu6rFygu2u9hrQAAAAAMF2a2zjm3qLt9A7WYyKiTmRSrdd+9RB86c7wunJatfRUNmvHPz+nbj25WWyDodfUAAAAADGP0qA0A55yu+/UqrSqulCTNy0vVtUsKdM2ifPl95nHtAAAAAESi3nrUCGoDqKapVY+9c1jff2Jrx7ZrFuXpzqvnyozABgAAAOBdDH0cIilx0fr0OYV64/aLOrY9vPaQJn77Gd37erGO17d4WDsAAAAAwwU9aoMoEHT6/G/X6qUdZR3bLpk5Rj/92DylJcR4WDMAAAAAXmPoo8d2Hq3Vjfe/rcNVjSdsf+rL52l2bqpHtQIAAADgJYY+emz62GS9cftF2vXDFfrIglxFhRcYufwXK/XIukNqauUebAAAAADeRY+aR4rK6nTJf7za8fqRW8/RwgnpHtYIAAAAwFCiRy0CTRmTpB0/WK7MxNBctU/8epV+8vwOVTe0dpRpDQTlnFNTa0CHjjd4VVUAAAAAQ4wetQhwoKJBF/zk5T7L3X39Qi2fPXYIagQAAABgsNGjFuEKMhP0yjeWKS66++ZYFB4Secvv1+ljd7+pV3aWdVsOAAAAwMhAj1qEcc7pjaIKLZ6Yrtgof8f2l3eW6bO/ebvj9Q1LJ+gbl06XTEqNj/aiqgAAAABOA8vzjxCtgaDe3lupbz6ySYeOv7vUv4XDmik0PDLgnObnpys+xt/zwQAAAAB4iqA2wgSDTp+4d5WKyupVXtfca9mCjAT97qYlmpCZOES1AwAAANAfBLVRoLSmSW8UlWtveb1+/XqxmlqD3Za765MLtL+iQedNyVJja0APrTmgf7t6rmKimK4IAAAADCWC2ijknNPru8tVUd+sf/zTxl7LJsb49cLXL9S41PgeyzS2BNTQ0qbMpNiBrioAAAAwKhHUIEl6atMR3fbH9R2v5+SmavPh6o7X+Rnx+oeLp2nqmCTNzUvVG0UV+u+Xi/RWcUVHmQ/Pz9UnzirQ7PGpzIEDAAAATgNBDb16dnOJbv3DO6f03hljk3XHlWd03EIgyu9TMOi051idJmYlKsrPkEoAAACgOwQ19KmhpU1/WXdIv3ypSHHRfh2obJAkXTFvvGaMTdYXl02WJL20o0xfe3ijqhtb+3XcsSlxqm1q1bdWzNCU7CQtnpihaMIbAAAAQFDD4Kisb1F9c5s+evebKq3pfvXJrKQYlde1dLvv9zedpTf2lGt+fpr+tq1UiwvTddmccUqMiZKZZGaqbmxVfLSfxU4AAAAw4hDUMGRaA8EuPWZv7anQjqM1+pcnt53SMWOjfLpkVo6+tGyKMhJjVFLdqAfe3KfHNxzRVy6aoq9dOn0gqg4AAAAMKYIaIkow6PTExiPaVVqrl3ce0/aSGn3gjBx9+pxC/fiZHScscCJJcdG+Hm830FluWrxuOm+iappatWZvpe648gxNy0lWIOgUCIb+nZfXNWtnaa3OnpSphpaASmuatKu0VudMzlJ2MitaAgAAYOgQ1DCsVTeGgtfnfxv69zI+NU5XzBuvi2aM0cfvWdXn++Oj/WpsDfRZbsnEDP3fD83WlDFJMjNJoVBZXtes13aX6/H1h7V2f6XOnZylez61SH7fu2V+/Ox2vbSjTDVNbTpW26w/fv4snTM5S3uO1SkrKVap8dGn8Q0AAABgJCKoYcQ7UtWox9Yf1rNbSvT+mWO1v6JeT28uUXNbUJmJMUpPjFFRWZ3SEqKVnRSr3WV1um5Jvl7ecUxHa5pO+bxnTczQtiM1qm1u67XcksIMfeXiqTpncqZKa5tU1dCqaTnJHWEPAAAAow9BDeiH7SU1+vR9a1RW23VhlCljkvTVS6bqkpk5mv3959UWPPG6+cIFk3RmfprqmttUUd+iv6w7pKKyOp0/NUvFx+p1uKqx13Pf/9nF+rdnd2jH0VpJUnJslObmh+5llxjj1w1nF2pBQZreOVCljMRo+cw0OzdVjS0BbTxUpfFp8apuCK3EedN5E+XzmfZX1Cs/PUEBFxr6GRftV1NrQLVNbQzzBAAAiAAENeA9cM7JzPT81qOaMiZJk7OTui0TdNK3H92k+QXpum5JQa/Hu+uVPfrJ8zsHs9onyEmJ7XElTklKS4jW1DFJunzueCXGRul/V+7V9pKajv13Xj1H8wvSNTk76YRev7Jwb+DY1Dit3F2uB97cp9V7K0849otfv7Db76w7gaBTY2tAGw5UadrYJI1JjnuPnxQAAGD4IqgBEexfntyqtfuO67aLpuiCqdk6XNWouGifxiTH6aUdZUqNj9bWI9WqaWzVvooG+X0mv88UF+3T0eomVda3qKapTVlJMVpVHApNaQnRqmro373u+uPM/DQ1tQY6evzei5yUWM0al6JfXb9QMX6fSmqa9P2/btUL20u7LR/lM31r+QzFx/j1kQW52l5So1XFldp2pEZPby7RFy6cpBvPnaiclHdDnQv3GpqZntlcojf3lCshJkpjU+J0w9kTFBvl65h32K60pklJsVFKjI16z58JAABgIBDUgFGkpLpR6Qkxiov268fPbNeKOeN0Zn6aGlsCMgvd7uBAZYMaWgK66f63FeX36ZYLJ+vD83MVF+2Tc9La/cf1n3/fpbeKKzqOOzs3RSlx0XpzT2jbnNxU3XTeRH1ofm5Hmd+8sfeUbsOQGONXfUtAMVE+tbT1vcLn6SjMTNC+ioaO10sKM3TrssmqaWrV/Px0/fatfXpwzQE5SQ0toUVofvqxebpi3jgdPt6opzeV6NevF6umKTQv8eTFaj55VoFmjktRU2tA247U6P2zcrRizrge61PV0KLnthzVb97Yp8+dP1EV9S367LmF2nW0TinxUSrISOgSMoNBJ18/5jcGgq7f8yCDQafapjYlx0X169gAAOD0EdQAnLKKumalJ8T0+5f39qGjZTVNqmlq0+7SWv1u1X69uadCOSmxOmdylnLT4vWl901RfIy/y/v/uuGwGlsCem7rUR2patS5U7J0vL5FK4vK9ctPLNC6/ce7HUYaF+1TfLRfxxta9ZWLpigpLko/emZHj/XMS49XVUOr6npYCGb5GWP10s6yjuCYGh+t6sbT66U8Z3JmR9D99NkT9MBb+/v1vsLMBB1vaNV5U7L09OaSE/Ytm56tcyZnqi3odOO5E/X67nK9va9SL24v1Z5j9R3l4qJ9+tr7p2liVpKOVDXq2iX5emFbmY7WNMkkPbz24Ak9ppOyEnXX9QuUHBetnORYNbcF9fLOMiXFRun8qdk6UtWosalxqqxv0Yvby3TVmeMV5Tf5zRQVvpdiQ0ub1uyt1B9WH9AFU7P0l3WHdM3ifH3yrAknfIZA0CnonKJ81iWUtnPO6bH1h3X7o5v1wTnjtLKoXLcvn6Er5o1XTJSv2/d0pzUQ1KHjjRqfFqdon2/YhVLnnEqqmzQ+Ld7rqgAABsBpBzUzWy7p55L8ku51zv1bN2WukXSHJCdpo3PuE70dk6AGYKi0BoJ6o6hc1Y2tuurM3BP2bThYpV+8uFuS9OKOMn1x2WR9c/mMjv2NLQH978piPb35qOqb2/TBuePU2BLQZ88t1ITMRLUFgory+9TYEtDtj25SZmKsJmQmqKisTgUZCfrflXv7XFn03CmZKq1p1kcX5mnzoWqV1TbpaE2TzhiXque2Hj2tzx7lsy6L3/Tk8rnj9NSmkr4L9kNSbFSPIbjdrHEpOlzV2G0AXjIxQ/npCXp8w2GNSY5VSXXv3+Ft75uiCZkJmpOXqn3lDUpLiNa6/cdVWd+ikupGrdl7XMcbWjruqdjuxnMn6uzJmTpjfIrGpcZpw8Eqff3hjfrw/Fy9vrtcxeV1KsxM1I6jtUpLiNbiwgy9b8YY/X7Vfn314qlaPDFDfjP5fCbnnEprmpUaH609x+q051id/uGhDZJCQfmWCyerICNBV52Z+55WfK1talVFXYuW/fSVjm1TxiSpqqFFD918tqaMeXdO6OriCt30wFrVNYd6R8elxunfrp6r4/UtOntyphJiTm+ob1VDi275/TplJsXq1gsna3Zu6mkdr/366W8v8elqaGk77e8AAAbSaQU1M/NL2iXp/ZIOSXpb0nXOuW2dykyV9LCki5xzx81sjHOurLfjEtQAjBYtbUH5THrnQJXion0dN1tfPnusYqO69ip2JxB0ag0ET5hv55zTq7uO6ecv7tb6A1WalJ2oqWOSdNGMMfrWI5v15u0XdfS8OOe0qrhSj60/pFXFlTpQ2aD3z8qRc6Eg+5OPzT1hMZd3DhzXN/68UcWdeuVuWDpBh6sa9dKOE3+8n5mfppgonxpa2rSrtK6jF3LG2GQ5Jy2emK7pOcl6q7hCK3eXdwwb7a/xqXE6Ut2khRPStWL2WF29IE93PrdD6w9UaWdp/+dNJsT4FeUz1TS1dRkCK0l+n3UJcoMtJsqnpZMy9dquY5Kk65YU6ME1BzRjbLIOH2/s89YfUqj3MyU+WscbWrT/pM90svSEaP3pC2drWk6yXt5Zph88uU3F5fUnlPnNZxdLTvrdqv2aPjZZSydl6gdPbVNRWV2fdVl+xlh9cmmBpo9N1pjkONU0terhtw9q/cEq/X1rqaL9oTm2MVF+ldd1XfAoymeamJWo5Lgo/fRj8zQ+LV53PrdD0X6f5uWl6Y9r9quorE7/de18SVJVY6veLCrXa7vLtbe8Xnnp8frkWRO0ZGKGzKT/eXWPnt9aKr/PFBvl6xjOLIVC+vVLC5SXntClV7Z9ZEDn19WNrbrjia16fMMRSdLVC/JUUt2or71/mhYVZqi5LaAon0/HG1r0o6e364p543XhtGwFnJPPrCOc1zW36UhVo/71yW2qbW5TdlKMVswep5njUjRzXHKvvcoHK0M92UHntGZvpd4oKtf/vFYsSbp0Vo4unzdeV8wd1+Mx+lLT1KoDFQ363l+36J0DVfrlJ+Zr5rgUFWYm9uuPC+1D77ubF9yusSWgivpm5aUnnFIdJWl3abMOiqAAAA/4SURBVK3SE2MUdE4JMVFKGoS5xg0tbYrx+zpGCESqQNDpaE2T4qJ8qqxv0aSTFgFD5DvdoHa2pDuccx8Iv/62JDnnftypzL9L2uWcu7e/lSKoAcDIdPIvub3ZVVrbsbpoc1tAptB/f/36Xj2+/rB+du2Zmp+f1uPxAkGn1cUVuu+NfdpwsErldc1aUJCm86Zk6fENR7S4MEOfOadQs8andPvLy9Yj1frqQxu0OxxCrl6Qp0feOaTb3jdFN543UcXH6hQb5VdDS5s2hXs7/76tVJfMzNGzW452ufWGz6RpOclKS4jW+VOztWRihhYXZsg5p+MNrXp60xHd/+a+E4al9se0nCTdumyyPjw/T1LoO/75i7v1sxd2n1DuqjPH6xuXTldja0A+k27743rFRfs1Pi1Oz2w+vd5ZSVo6KUMXTMtWQrRfd5zCfNR2XoTi3pw/NUuriiuUldR37+1gy8+I179ceYbOn5ot56T/89hm/WXdoVM+3jWL8vTs5qM9hv7vXT5LOSlxenLjkV578K9ZlKdl08fIZ6bvPr5Z5XUtkkKLV00bk6w1+yq7vGdCZoKaW4OqbGjpMv84xu9TVlKMPr64QB9ZkKu95fV6elOJXtxRpvK6ZmUnx2paTpIunpGjM8an6OWdx/T67mPaeqSmy3mk0IiA86dmaXFhhiZ1Wnl43f5KldY0a+OhKvnN9JlzCzv+KBUMOlU1turtfZUqyEjQ9pIafe3hjT1+B5OyE/X8Vy9QdDi8bTtSo3tfL9ZTm0qUmhCtY93c3keSctPiO35WFGYm6ODxRl00Y4xionz6wVWzFRPlU2KMX2am1kBQ3/jzRjW2BHToeKOi/aadpbWam5em8alx2nqkRkerm5SfkaBtJd1/F1JoPvojt56jaTnJfQ4PbwsE9cL2Mt33xl6t2Rv6Lv7pA9P1wTnjOnq62wJB/eKlIj2/9WjHMPnZuSny+3w6d3Kmbjh7gsamxKk14DrOFww6tQSCqm9u09GaJk3KSpKZwot/SSZTRX2zDh1vVFJslN7aU6Gr5o/vsgL0lsPVKsxKlKSOUH6gokFB5zq2n/x5zEyV9S369qOb9eKOUn3nspl66O2DSoqN0u7SWk3ITNSiwnTF+H06Z0qmCjISdM9rxfryRVOVn5GgptaAYqNC8/bNQn9gqW8OhP5QEv5c7X+QaP8jym/f2qd7V+7VtYvz9Z0PzlRyXHSP33lLW1CPrT+kaL9PVy/MP62g9lFJy51znwu/vkHSWc652zqVeVyhXrdzFRoeeYdz7rlujnWzpJslqaCgYOH+/f2bnwEAwEjUHmqdc1pZVK49ZXVaMWecfGZqbAmoILP3XofmtoB+/sJuXXrGWE3PSe523me7ptaA/vOFXXrgzX1qag0qIzFGd1+/UEsmZkgK/TK042itHl9/WCuLyhUf7dcnzyrQnmN1+s4HZ2rKmOQeP0P7/SeLyur08xd3a034th1z81L1hQsmqzArQTPHpqi0NrTaaly0v+OX3c7qm9t0vKFF247U6Et/fEdmpry0eF04PVtjkuP0iSUFOlrTpCc3Hgn94pedqOuXTpApFP6e3HhEcdF+1TW36ZnNJVp+xljNy0/TC9vL9OWLpija71NrIKiGloCe2HBY//zXrT1+X7FRPjWfFC7+8LmztHBCumqb2lTb1KoDlQ36w+oD+vu2E1exvXXZZAWCTr9ftf+EXjwp9MttXVObvnLxVM0cl6Kgc3p+y1Edq2vWkxtL+hwy3G5sSpyuW1KgJRMzVN3YoqPVTXp9d7le3NHrgKYeP1u7meNS9O0VMzQ+LU67Suv0xT+806/6tFs6KaNjBeL2+38Ohktn5ei13cfU1Nr1c7Qv8nQ6KyBnJcV0hNFI1/kPH5OyE08YCSGFFuzymen+GxcrKTZaBRkJenjtQX3/iZ7//bf74Ydm6+19lfpruCd5KGQmxigpLqrPEQLtZRcVpuuLy0Lz3p/aeET/9VLRENQyJC89XoeOd3+v3LEpcTp7cqbOnpSpc6ZkaufRWj245oBe2H7iNbr/zssHPag9JalV0jWS8iS9JmmOc67Hq5MeNQAARq730rPqtbrmNvktdNuTznUuq2lS0EmZSTHdBsv+CgSd9lXUKzctXnHRvQ933lVaq1+8VKQnN4Z+Mf7I/FxdceZ4vW/6mH6fr/13u3cOVKk0PEd2xeyxCjqd0LO8u7RWL+0o09v7KnXlmbk9DpusamjRquJKlVQ3KuhCvVc5KXFqbAl0fGd7y+tVmNl1ldq2QFB+n6muuU1JsVEd+9uHc9/2x/XaXVarWy6crOVnjFV6YkzHe0trmrTi56+rsr5FX3//NF08M0ezxqd0qV9bIKh7V+5VWU2znttSovkF6Xpt1zFNG5usbUdqtHhihr75gVBP8/X3rj4hpF48Y4zOGJ+iDYeqdcb4FF29IO+EeZ/tgkGnbz6y6YTezfOmZOlji/I0Ny9NafHRqmps1fi0OMX43/13dLCyQfUtbWppCyoQdHp7X6UumpGjpNgo/fDpbT3OC75mUZ5uuXCyMhNjVdMU6vW7bM44HW9o0bHaZk3LSe5xiKlzTuv2H9ej6w/rj6sPdHv8k80cl6LvfnCmlk7K1IaDx/W3baX6n1eLO/ZH+UyfOadQt100RWkJoTZqaQvKTPrdW/v1w6e36eRO8pgon1LionTG+FQFgk7HG1qUnhCjlUXlJ5T7yPxcZSbFKC89Qd9/YqvMpJPjyZjk2I4/CEmhFZb/sPpAr73zF0zL1ry8VJ03JUv7Kxu05XC13j8rR4sLM1RS3aStR6r14vYyPbb+sHLT4pWeGK34aL8aWgKqaWrVwcp3A1i039QaCJ1nQmZClxA5JzdV50zJ1I3nTtTft5Xqj6sP9NrjKUkXTstWdnKs/t81Zw760Me7Ja12zv0m/PpFSbc7597u6bgENQAAAEA6Xt+iZ7cc1ccX5w/KHLNV4TnCD7y1T+dOzlJb0Ok7H5ypid0MHWx3sLJBD7y5T0snZeq8qVl9/pHBK89sLtEfVu9Xblq8apva9OOPzOkIk4Opoq5ZyXHRPQ4trWtu0/99epveKKrQOZMzFeU3XTF3fHgO7bttfLpz1KIUGtZ4saTDCi0m8gnn3NZOZZYrtMDIp80sS9J6SWc65yq6O6ZEUAMAAAAwuvUW1Prsx3fOtUm6TdLzkrZLetg5t9XM/tXMrgwXe15ShZltk/SypH/qLaQBAAAAAHrGDa8BAAAAwAOn1aMGAAAAABhaBDUAAAAAiDAENQAAAACIMAQ1AAAAAIgwBDUAAAAAiDAENQAAAACIMJ4tz29mxyTt9+Tk6I8sSeVeVwL9QlsNL7TX8EFbDS+01/BBWw0vtNfgmuCcy+5uh2dBDZHNzNb2dE8HRBbaanihvYYP2mp4ob2GD9pqeKG9vMPQRwAAAACIMAQ1AAAAAIgwBDX05B6vK4B+o62GF9pr+KCthhfaa/igrYYX2ssjzFEDAAAAgAhDjxoAAAAARBiCGgAAAABEGILaKGFm+Wb2spltM7OtZvYP4e13mNlhM9sQflzW6T3fNrMiM9tpZh/otH15eFuRmd3uxecZ6cxsn5ltDrfJ2vC2DDP7u5ntDv83PbzdzOy/wu2xycwWdDrOp8Pld5vZp736PCOZmU3vdP1sMLMaM/sq11bkMLP7zKzMzLZ02jZg15OZLQxfr0Xh99rQfsKRo4e2+omZ7Qi3x2NmlhbeXmhmjZ2usbs7vafbNump3XFqemivAfvZZ2YTzWx1ePufzCxm6D7dyNJDW/2pUzvtM7MN4e1cW5HCOcdjFDwkjZO0IPw8WdIuSbMk3SHpG92UnyVpo6RYSRMl7ZHkDz/2SJokKSZcZpbXn2+kPSTtk5R10rZ/l3R7+Pntku4MP79M0rOSTNJSSavD2zMkFYf/mx5+nu71ZxvJj/D1cVTSBK6tyHlIukDSAklbOm0bsOtJ0ppwWQu/d4XXn3m4Pnpoq0slRYWf39mprQo7lzvpON22SU/tzmNA22vAfvZJeljSteHnd0u61evPPFwf3bXVSfv/n6TvhZ9zbUXIgx61UcI5V+Kceyf8vFbSdkm5vbzlKkkPOeeanXN7JRVJWhJ+FDnnip1zLZIeCpfF4LtK0gPh5w9I+lCn7b91IaskpZnZOEkfkPR351ylc+64pL9LWj7UlR5lLpa0xzm3v5cyXFtDzDn3mqTKkzYPyPUU3pfinFvlQr+h/LbTsfAedddWzrm/Oefawi9XScrr7Rh9tElP7Y5T0MO11ZP39LMv3FNzkaS/hN9Pe52G3toq/F1fI+nB3o7BtTX0CGqjkJkVSpovaXV4023hISX3deqqzpV0sNPbDoW39bQdA8tJ+puZrTOzm8PbcpxzJeHnRyXlhJ/TVpHjWp34Pzqurcg1UNdTbvj5ydsxOG5U6K/47Saa2Xoze9XMzg9v661Nemp3DKyB+NmXKamqU0jn2ho850sqdc7t7rSNaysCENRGGTNLkvSIpK8652ok/UrSZElnSipRqOsb3jvPObdA0gpJXzKzCzrvDP8li3trRJDw3IkrJf05vIlra5jgehoezOw7ktok/SG8qURSgXNuvqSvSfqjmaX093i0+6DhZ9/wc51O/CMj11aEIKiNImYWrVBI+4Nz7lFJcs6VOucCzrmgpF8rNARBkg5Lyu/09rzwtp62YwA55w6H/1sm6TGF2qU0POygffhBWbg4bRUZVkh6xzlXKnFtDQMDdT0d1olD8Wi3QWBmn5F0uaRPhn8JVHgIXUX4+TqF5jlNU+9t0lO7Y4AM4M++CoWGHkedtB0DKPz9fkTSn9q3cW1FDoLaKBEef/y/krY75/6j0/ZxnYp9WFL7akBPSLrWzGLNbKKkqQpNIH1b0tTwSkwxCg31emIoPsNoYWaJZpbc/lyhifRbFPqe21ea+7Skv4afPyHpUxayVFJ1ePjB85IuNbP08NCTS8PbMDhO+Isk11bEG5DrKbyvxsyWhn/OfqrTsTAAzGy5pG9KutI519Bpe7aZ+cPPJyl0LRX30SY9tTsGyED97AsH8pclfTT8ftprcFwiaYdzrmNII9dWBPF6NRMeQ/OQdJ5C3dCbJG0IPy6T9DtJm8Pbn5A0rtN7vqPQX1F2qtMqZuH37Qrv+47Xn22kPRRa+Wpj+LG1/TtWaLz+i5J2S3pBUkZ4u0n673B7bJa0qNOxblRownaRpM96/dlG6kNSokJ//U3ttI1rK0IeCgXoEkmtCs2puGkgrydJixT6ZXSPpF9KMq8/83B99NBWRQrNYWr/f9fd4bJXh39GbpD0jqQr+mqTntqdx4C214D97Av//3BN+N/AnyXFev2Zh+uju7YKb79f0i0nleXaipBH+5cLAAAAAIgQDH0EAAAAgAhDUAMAAACACENQAwAAAIAIQ1ADAAAAgAhDUAMAAACACENQAwAAAIAIQ1ADAAAAgAjz/wHrQzwqydElIQAAAABJRU5ErkJggg==\n",
1371 |       "text/plain": [
1372 |        "<Figure size 1080x288 with 1 Axes>"
1373 |       ]
1374 |      },
1375 |      "metadata": {
1376 |       "needs_background": "light"
1377 |      },
1378 |      "output_type": "display_data"
1379 |     }
1380 |    ],
1381 |    "source": [
1382 |     "window = 100\n",
1383 |     "plt.figure(figsize=(15, 4))\n",
1384 |     "pd.Series(iteration_losses).rolling(window=window).mean().iloc[window-1:].plot()\n",
1385 |     "plt.show()"
1386 |    ]
1387 |   },
1388 |   {
1389 |    "cell_type": "code",
1390 |    "execution_count": 45,
1391 |    "metadata": {},
1392 |    "outputs": [],
1393 |    "source": [
1394 |     "path = os.path.join(\"models\", \"language-words\", \"classifier.pth\")\n",
1395 |     "torch.save(model.state_dict(), path)"
1396 |    ]
1397 |   },
1398 |   {
1399 |    "cell_type": "markdown",
1400 |    "metadata": {},
1401 |    "source": [
1402 |     "## 10. Generate new baby names"
1403 |    ]
1404 |   },
1405 |   {
1406 |    "cell_type": "code",
1407 |    "execution_count": 47,
1408 |    "metadata": {},
1409 |    "outputs": [
1410 |     {
1411 |      "data": {
1412 |       "text/plain": [
1413 |        "<All keys matched successfully>"
1414 |       ]
1415 |      },
1416 |      "execution_count": 47,
1417 |      "metadata": {},
1418 |      "output_type": "execute_result"
1419 |     }
1420 |    ],
1421 |    "source": [
1422 |     "path = os.path.join(\"models\", \"language-words\", \"classifier.pth\")\n",
1423 |     "model = Model(input_size=num_chars, hidden_size=hidden_size, output_size=output_size, num_layers=num_layers)\n",
1424 |     "model = nn.DataParallel(model)\n",
1425 |     "model.load_state_dict(torch.load(path))"
1426 |    ]
1427 |   },
1428 |   {
1429 |    "cell_type": "code",
1430 |    "execution_count": 48,
1431 |    "metadata": {},
1432 |    "outputs": [
1433 |     {
1434 |      "name": "stdout",
1435 |      "output_type": "stream",
1436 |      "text": [
1437 |       "['Indi', 'Indiahwste', 'Indill', 'Indiaahiter', 'Indikol', 'Indina', 'Indiwa', 'Inditaneahn', 'Indinnaisha', 'Indilytofer', 'Indinta']\n"
1438 |      ]
1439 |     }
1440 |    ],
1441 |    "source": [
1442 |     "names = sampler(model, start='indi', n=10, k=5, only_new=True)\n",
1443 |     "print(names)"
1444 |    ]
1445 |   },
1446 |   {
1447 |    "cell_type": "code",
1448 |    "execution_count": 49,
1449 |    "metadata": {},
1450 |    "outputs": [
1451 |     {
1452 |      "name": "stdout",
1453 |      "output_type": "stream",
1454 |      "text": [
1455 |       "['Herbshw', 'Herbahwno', 'Herbet', 'Herbnollo', 'Herbahahaos', 'Herbhnahia', 'Herbhmopee', 'Herb', 'Herbney', 'Herbrinerah', 'Herbsty']\n"
1456 |      ]
1457 |     }
1458 |    ],
1459 |    "source": [
1460 |     "names = sampler(model, start='herb', n=10, k=5, only_new=False)\n",
1461 |     "print(names)"
1462 |    ]
1463 |   },
1464 |   {
1465 |    "cell_type": "code",
1466 |    "execution_count": 50,
1467 |    "metadata": {},
1468 |    "outputs": [
1469 |     {
1470 |      "name": "stdout",
1471 |      "output_type": "stream",
1472 |      "text": [
1473 |       "['Sul', 'Sumurio', 'Suianohonor', 'Suunn', 'Sumosgi', 'Su', 'Suphuroncio', 'Suieua', 'Sumoriusumi', 'Supyruco', 'Suresse']\n"
1474 |      ]
1475 |     }
1476 |    ],
1477 |    "source": [
1478 |     "names = sampler(model, start='su', n=10, k=5, only_new=True)\n",
1479 |     "print(names)"
1480 |    ]
1481 |   },
1482 |   {
1483 |    "cell_type": "code",
1484 |    "execution_count": 51,
1485 |    "metadata": {},
1486 |    "outputs": [
1487 |     {
1488 |      "name": "stdout",
1489 |      "output_type": "stream",
1490 |      "text": [
1491 |       "['Vistoapar', 'Visceranosg', 'Visni', 'Visti', 'Visan', 'Vissno', 'Vissia', 'Visshiashot', 'Visora', 'Visanell', 'Vistoo']\n"
1492 |      ]
1493 |     }
1494 |    ],
1495 |    "source": [
1496 |     "names = sampler(model, start='vis', n=10, k=5, only_new=True)\n",
1497 |     "print(names)"
1498 |    ]
1499 |   },
1500 |   {
1501 |    "cell_type": "code",
1502 |    "execution_count": 52,
1503 |    "metadata": {},
1504 |    "outputs": [
1505 |     {
1506 |      "name": "stdout",
1507 |      "output_type": "stream",
1508 |      "text": [
1509 |       "['Aleyanarano', 'Aishi', 'Aissond', 'Aexine', 'Atura', 'Alas', 'Athenele', 'Aicely', 'Athel', 'Aejuan', 'Adya']\n"
1510 |      ]
1511 |     }
1512 |    ],
1513 |    "source": [
1514 |     "names = sampler(model, start='a', n=10, k=3, only_new=True)\n",
1515 |     "print(names)"
1516 |    ]
1517 |   },
1518 |   {
1519 |    "cell_type": "code",
1520 |    "execution_count": 53,
1521 |    "metadata": {},
1522 |    "outputs": [
1523 |     {
1524 |      "name": "stdout",
1525 |      "output_type": "stream",
1526 |      "text": [
1527 |       "['Aaulosnikem', 'Aye', 'Ahaynzas', 'Adidr', 'Ane', 'Ahennofeeon', 'As', 'Aich', 'Asroctonait', 'Aaynemaednu', 'Afobiamairi']\n"
1528 |      ]
1529 |     }
1530 |    ],
1531 |    "source": [
1532 |     "names = sampler(model, start='a', n=10, k=8, only_new=True)\n",
1533 |     "print(names)"
1534 |    ]
1535 |   },
1536 |   {
1537 |    "cell_type": "code",
1538 |    "execution_count": 54,
1539 |    "metadata": {},
1540 |    "outputs": [
1541 |     {
1542 |      "name": "stdout",
1543 |      "output_type": "stream",
1544 |      "text": [
1545 |       "['Abckyeua', 'Amiyazcioto', 'Adwoykhurk', 'Ass', 'Anisklwuto', 'Aunnt', 'Ay', 'Atimliitefy', 'A', 'Achzesqahsp', 'Aopldookksy']\n"
1546 |      ]
1547 |     }
1548 |    ],
1549 |    "source": [
1550 |     "names = sampler(model, start='a', n=10, k=15, only_new=True)\n",
1551 |     "print(names)"
1552 |    ]
1553 |   },
1554 |   {
1555 |    "cell_type": "code",
1556 |    "execution_count": 55,
1557 |    "metadata": {},
1558 |    "outputs": [
1559 |     {
1560 |      "name": "stdout",
1561 |      "output_type": "stream",
1562 |      "text": [
1563 |       "['James', 'Jamel', 'Jame', 'Jamso', 'Jamshah', 'Jams', 'Jamela', 'Jamelahana', 'Jamsha', 'Jamesh', 'Jamson']\n"
1564 |      ]
1565 |     }
1566 |    ],
1567 |    "source": [
1568 |     "names = sampler(model, start='jam', n=10, k=2, only_new=False)\n",
1569 |     "print(names)"
1570 |    ]
1571 |   },
1572 |   {
1573 |    "cell_type": "code",
1574 |    "execution_count": null,
1575 |    "metadata": {},
1576 |    "outputs": [],
1577 |    "source": []
1578 |   }
1579 |  ],
1580 |  "metadata": {
1581 |   "kernelspec": {
1582 |    "display_name": "Python 3",
1583 |    "language": "python",
1584 |    "name": "python3"
1585 |   },
1586 |   "language_info": {
1587 |    "codemirror_mode": {
1588 |     "name": "ipython",
1589 |     "version": 3
1590 |    },
1591 |    "file_extension": ".py",
1592 |    "mimetype": "text/x-python",
1593 |    "name": "python",
1594 |    "nbconvert_exporter": "python",
1595 |    "pygments_lexer": "ipython3",
1596 |    "version": "3.6.9"
1597 |   }
1598 |  },
1599 |  "nbformat": 4,
1600 |  "nbformat_minor": 2
1601 | }
1602 | 


--------------------------------------------------------------------------------
/4-Neural-Style-Transfer.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/gokulkarthik/Deep-Learning-Projects.pytorch/2c2a20b605c9884c6841deb04b5aeb3251acadd7/4-Neural-Style-Transfer.pdf


--------------------------------------------------------------------------------
/5-Captcha-Text-Recognition-With-CRNN.ipynb:
--------------------------------------------------------------------------------
1 | {"cells":[{"metadata":{},"cell_type":"markdown","source":"\n**Project Repository:** https://github.com/GokulKarthik/deep-learning-projects-pytorch"},{"metadata":{},"cell_type":"markdown","source":"### References:\n[1] https://github.com/carnotaur/crnn-tutorial/"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import transforms\nfrom torchvision.models import resnet18\n\nimport string\nfrom tqdm.notebook import tqdm\nimport cv2\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score\nimport multiprocessing as mp","execution_count":1,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cpu_count = mp.cpu_count()\nprint(cpu_count)","execution_count":2,"outputs":[{"output_type":"stream","text":"2\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 1. Make train-test split"},{"metadata":{},"cell_type":"markdown","source":"**Data Link**: https://www.kaggle.com/shawon10/captcha-recognition"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"data_path = \"/kaggle/input/captcha-version-2-images/samples\"","execution_count":3,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_fns = os.listdir(data_path)\nprint(len(image_fns))\nprint(np.unique([len(image_fn.split(\".\")[0]) for image_fn in image_fns]))","execution_count":4,"outputs":[{"output_type":"stream","text":"1071\n[5 7]\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx, image_fn in enumerate(image_fns):\n    if len(image_fn.split(\".\")[0]) != 5:\n           print(idx, image_fn)","execution_count":5,"outputs":[{"output_type":"stream","text":"576 samples\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_fns.remove('samples')\nprint(len(image_fns))","execution_count":6,"outputs":[{"output_type":"stream","text":"1070\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_fns_train, image_fns_test = train_test_split(image_fns, random_state=0)\nprint(len(image_fns_train), len(image_fns_test))","execution_count":7,"outputs":[{"output_type":"stream","text":"802 268\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 2. Define character maps"},{"metadata":{"trusted":true},"cell_type":"code","source":"image_ns = [image_fn.split(\".\")[0] for image_fn in image_fns]\nimage_ns = \"\".join(image_ns)\nletters = sorted(list(set(list(image_ns))))\nprint(len(letters))\nprint(letters)","execution_count":8,"outputs":[{"output_type":"stream","text":"19\n['2', '3', '4', '5', '6', '7', '8', 'b', 'c', 'd', 'e', 'f', 'g', 'm', 'n', 'p', 'w', 'x', 'y']\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"vocabulary = [\"-\"] + letters\nprint(len(vocabulary))\nprint(vocabulary)\nidx2char = {k:v for k,v in enumerate(vocabulary, start=0)}\nprint(idx2char)\nchar2idx = {v:k for k,v in idx2char.items()}\nprint(char2idx)","execution_count":9,"outputs":[{"output_type":"stream","text":"20\n['-', '2', '3', '4', '5', '6', '7', '8', 'b', 'c', 'd', 'e', 'f', 'g', 'm', 'n', 'p', 'w', 'x', 'y']\n{0: '-', 1: '2', 2: '3', 3: '4', 4: '5', 5: '6', 6: '7', 7: '8', 8: 'b', 9: 'c', 10: 'd', 11: 'e', 12: 'f', 13: 'g', 14: 'm', 15: 'n', 16: 'p', 17: 'w', 18: 'x', 19: 'y'}\n{'-': 0, '2': 1, '3': 2, '4': 3, '5': 4, '6': 5, '7': 6, '8': 7, 'b': 8, 'c': 9, 'd': 10, 'e': 11, 'f': 12, 'g': 13, 'm': 14, 'n': 15, 'p': 16, 'w': 17, 'x': 18, 'y': 19}\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 3. Define data loader"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 16","execution_count":10,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CAPTCHADataset(Dataset):\n    \n    def __init__(self, data_dir, image_fns):\n        self.data_dir = data_dir\n        self.image_fns = image_fns\n        \n    def __len__(self):\n        return len(self.image_fns)\n    \n    def __getitem__(self, index):\n        image_fn = self.image_fns[index]\n        image_fp = os.path.join(self.data_dir, image_fn)\n        image = Image.open(image_fp).convert('RGB')\n        image = self.transform(image)\n        text = image_fn.split(\".\")[0]\n        return image, text\n    \n    def transform(self, image):\n        \n        transform_ops = transforms.Compose([\n            transforms.ToTensor(),\n            transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))\n        ])\n        return transform_ops(image)","execution_count":11,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trainset = CAPTCHADataset(data_path, image_fns_train) \ntestset = CAPTCHADataset(data_path, image_fns_test)\ntrain_loader = DataLoader(trainset, batch_size=batch_size, num_workers=cpu_count, shuffle=True)\ntest_loader = DataLoader(testset, batch_size=batch_size, num_workers=cpu_count, shuffle=False)\nprint(len(train_loader), len(test_loader))","execution_count":12,"outputs":[{"output_type":"stream","text":"51 17\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_batch, text_batch = iter(train_loader).next()\nprint(image_batch.size(), text_batch)","execution_count":13,"outputs":[{"output_type":"stream","text":"torch.Size([16, 3, 50, 200]) ('nbp3e', 'c86md', '6pfy4', 'e3cfe', '7gce6', 'f858x', 'mcc2x', 'bp2d4', '74853', 'bw44w', 'w6ny4', 'nn6w6', '7cgym', 'n3bm6', 'm3b5p', 'mxyxw')\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 4. Define model"},{"metadata":{"trusted":true},"cell_type":"code","source":"num_chars = len(char2idx)\nprint(num_chars)\nrnn_hidden_size = 256","execution_count":14,"outputs":[{"output_type":"stream","text":"20\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(device)","execution_count":15,"outputs":[{"output_type":"stream","text":"cuda\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"resnet = resnet18(pretrained=True)\n#print(resnet)","execution_count":16,"outputs":[{"output_type":"stream","text":"Downloading: \"https://download.pytorch.org/models/resnet18-5c106cde.pth\" to /root/.cache/torch/checkpoints/resnet18-5c106cde.pth\n","name":"stderr"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=46827520.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"a7f7a16e82514db6b874d54938f63733"}},"metadata":{}},{"output_type":"stream","text":"\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CRNN(nn.Module):\n    \n    def __init__(self, num_chars, rnn_hidden_size=256, dropout=0.1):\n        \n        super(CRNN, self).__init__()\n        self.num_chars = num_chars\n        self.rnn_hidden_size = rnn_hidden_size\n        self.dropout = dropout\n        \n        # CNN Part 1\n        resnet_modules = list(resnet.children())[:-3]\n        self.cnn_p1 = nn.Sequential(*resnet_modules)\n        \n        # CNN Part 2\n        self.cnn_p2 = nn.Sequential(\n            nn.Conv2d(256, 256, kernel_size=(3,6), stride=1, padding=1),\n            nn.BatchNorm2d(256),\n            nn.ReLU(inplace=True)\n        )\n        self.linear1 = nn.Linear(1024, 256)\n        \n        # RNN\n        self.rnn1 = nn.GRU(input_size=rnn_hidden_size, \n                            hidden_size=rnn_hidden_size,\n                            bidirectional=True, \n                            batch_first=True)\n        self.rnn2 = nn.GRU(input_size=rnn_hidden_size, \n                            hidden_size=rnn_hidden_size,\n                            bidirectional=True, \n                            batch_first=True)\n        self.linear2 = nn.Linear(self.rnn_hidden_size*2, num_chars)\n        \n        \n    def forward(self, batch):\n        \n        batch = self.cnn_p1(batch)\n        # print(batch.size()) # torch.Size([-1, 256, 4, 13])\n        \n        batch = self.cnn_p2(batch) # [batch_size, channels, height, width]\n        # print(batch.size())# torch.Size([-1, 256, 4, 10])\n        \n        batch = batch.permute(0, 3, 1, 2) # [batch_size, width, channels, height]\n        # print(batch.size()) # torch.Size([-1, 10, 256, 4])\n         \n        batch_size = batch.size(0)\n        T = batch.size(1)\n        batch = batch.view(batch_size, T, -1) # [batch_size, T==width, num_features==channels*height]\n        # print(batch.size()) # torch.Size([-1, 10, 1024])\n        \n        batch = self.linear1(batch)\n        # print(batch.size()) # torch.Size([-1, 10, 256])\n        \n        batch, hidden = self.rnn1(batch)\n        feature_size = batch.size(2)\n        batch = batch[:, :, :feature_size//2] + batch[:, :, feature_size//2:]\n        # print(batch.size()) # torch.Size([-1, 10, 256])\n        \n        batch, hidden = self.rnn2(batch)\n        # print(batch.size()) # torch.Size([-1, 10, 512])\n        \n        batch = self.linear2(batch)\n        # print(batch.size()) # torch.Size([-1, 10, 20])\n        \n        batch = batch.permute(1, 0, 2) # [T==10, batch_size, num_classes==num_features]\n        # print(batch.size()) # torch.Size([10, -1, 20])\n        \n        return batch","execution_count":17,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init(m):\n    classname = m.__class__.__name__\n    if type(m) in [nn.Linear, nn.Conv2d, nn.Conv1d]:\n        torch.nn.init.xavier_uniform_(m.weight)\n        if m.bias is not None:\n            m.bias.data.fill_(0.01)\n    elif classname.find('BatchNorm') != -1:\n        m.weight.data.normal_(1.0, 0.02)\n        m.bias.data.fill_(0)","execution_count":18,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"crnn = CRNN(num_chars, rnn_hidden_size=rnn_hidden_size)\ncrnn.apply(weights_init)\ncrnn = crnn.to(device)","execution_count":19,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#crnn","execution_count":20,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"text_batch_logits = crnn(image_batch.to(device))\nprint(text_batch)\nprint(text_batch_logits.shape)","execution_count":21,"outputs":[{"output_type":"stream","text":"('nbp3e', 'c86md', '6pfy4', 'e3cfe', '7gce6', 'f858x', 'mcc2x', 'bp2d4', '74853', 'bw44w', 'w6ny4', 'nn6w6', '7cgym', 'n3bm6', 'm3b5p', 'mxyxw')\ntorch.Size([10, 16, 20])\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## 5. Define loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = nn.CTCLoss(blank=0)","execution_count":22,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def encode_text_batch(text_batch):\n    \n    text_batch_targets_lens = [len(text) for text in text_batch]\n    text_batch_targets_lens = torch.IntTensor(text_batch_targets_lens)\n    \n    text_batch_concat = \"\".join(text_batch)\n    text_batch_targets = [char2idx[c] for c in text_batch_concat]\n    text_batch_targets = torch.IntTensor(text_batch_targets)\n    \n    return text_batch_targets, text_batch_targets_lens","execution_count":23,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def compute_loss(text_batch, text_batch_logits):\n    \"\"\"\n    text_batch: list of strings of length equal to batch size\n    text_batch_logits: Tensor of size([T, batch_size, num_classes])\n    \"\"\"\n    text_batch_logps = F.log_softmax(text_batch_logits, 2) # [T, batch_size, num_classes]  \n    text_batch_logps_lens = torch.full(size=(text_batch_logps.size(1),), \n                                       fill_value=text_batch_logps.size(0), \n                                       dtype=torch.int32).to(device) # [batch_size] \n    #print(text_batch_logps.shape)\n    #print(text_batch_logps_lens) \n    text_batch_targets, text_batch_targets_lens = encode_text_batch(text_batch)\n    #print(text_batch_targets)\n    #print(text_batch_targets_lens)\n    loss = criterion(text_batch_logps, text_batch_targets, text_batch_logps_lens, text_batch_targets_lens)\n\n    return loss","execution_count":24,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"compute_loss(text_batch, text_batch_logits)","execution_count":25,"outputs":[{"output_type":"execute_result","execution_count":25,"data":{"text/plain":"tensor(4.5599, device='cuda:0', grad_fn=<MeanBackward0>)"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 6. Train model"},{"metadata":{"trusted":true},"cell_type":"code","source":"num_epochs = 50\nlr = 0.001\nweight_decay = 1e-3\nclip_norm = 5","execution_count":26,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = optim.Adam(crnn.parameters(), lr=lr, weight_decay=weight_decay)\nlr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True, patience=5)","execution_count":27,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"crnn = CRNN(num_chars, rnn_hidden_size=rnn_hidden_size)\ncrnn.apply(weights_init)\ncrnn = crnn.to(device)","execution_count":28,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epoch_losses = []\niteration_losses = []\nnum_updates_epochs = []\nfor epoch in tqdm(range(1, num_epochs+1)):\n    epoch_loss_list = [] \n    num_updates_epoch = 0\n    for image_batch, text_batch in tqdm(train_loader, leave=False):\n        optimizer.zero_grad()\n        text_batch_logits = crnn(image_batch.to(device))\n        loss = compute_loss(text_batch, text_batch_logits)\n        iteration_loss = loss.item()\n\n        if np.isnan(iteration_loss) or np.isinf(iteration_loss):\n            continue\n          \n        num_updates_epoch += 1\n        iteration_losses.append(iteration_loss)\n        epoch_loss_list.append(iteration_loss)\n        loss.backward()\n        nn.utils.clip_grad_norm_(crnn.parameters(), clip_norm)\n        optimizer.step()\n\n    epoch_loss = np.mean(epoch_loss_list)\n    print(\"Epoch:{}    Loss:{}    NumUpdates:{}\".format(epoch, epoch_loss, num_updates_epoch))\n    epoch_losses.append(epoch_loss)\n    num_updates_epochs.append(num_updates_epoch)\n    lr_scheduler.step(epoch_loss)","execution_count":29,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=50.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e8ccee1f5b0f487986e859af7f504369"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:1    Loss:2.9900871351653455    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:2    Loss:2.419498644623102    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:3    Loss:1.9921862167470596    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:4    Loss:1.6599087832020778    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:5    Loss:1.4582493959688674    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:6    Loss:1.3014318709279977    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:7    Loss:1.2249731339660346    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:8    Loss:1.1271169396007763    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:9    Loss:1.0210105472919988    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:10    Loss:0.9753329952557882    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:11    Loss:0.9316047196294747    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:12    Loss:0.8916015344507554    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:13    Loss:0.8738840070425296    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:14    Loss:0.8947969651689717    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:15    Loss:0.8399395264831244    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:16    Loss:0.8126519462641548    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:17    Loss:0.8683857064621121    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:18    Loss:0.8311487120740554    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:19    Loss:0.8097569393176659    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:20    Loss:0.7513260782933703    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:21    Loss:0.736772402828815    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:22    Loss:0.7255906228925667    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:23    Loss:0.7293911389276093    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:24    Loss:0.7188699058457917    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:25    Loss:0.844322102911332    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:26    Loss:0.8478556462362701    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:27    Loss:0.7318797380316491    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:28    Loss:0.6731003663119148    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:29    Loss:0.6562695842163235    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:30    Loss:0.6465148189488579    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:31    Loss:0.6488102206996843    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:32    Loss:0.6641178142790701    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:33    Loss:0.8547234593653211    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:34    Loss:0.7804392471032984    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:35    Loss:0.6723369675524095    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:36    Loss:0.6460016451629937    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:37    Loss:0.6240485719605988    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:38    Loss:0.6367446721768847    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:39    Loss:0.6285087992163265    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:40    Loss:0.6388927207273596    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:41    Loss:0.7857545999919667    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:42    Loss:0.7721711233550427    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:43    Loss:0.6579085272901198    NumUpdates:51\nEpoch    43: reducing learning rate of group 0 to 1.0000e-04.\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:44    Loss:0.5907929399434257    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:45    Loss:0.567648145498014    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:46    Loss:0.5579055430842381    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:47    Loss:0.5511149986117494    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:48    Loss:0.5489027768957848    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:49    Loss:0.5459187615151498    NumUpdates:51\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":""}},"metadata":{}},{"output_type":"stream","text":"Epoch:50    Loss:0.545354514729743    NumUpdates:51\n\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n\nax1.plot(epoch_losses)\nax1.set_xlabel(\"Epochs\")\nax1.set_ylabel(\"Loss\")\n\nax2.plot(iteration_losses)\nax2.set_xlabel(\"Iterations\")\nax2.set_ylabel(\"Loss\")\n\nplt.show()","execution_count":30,"outputs":[{"output_type":"display_data","data":{"text/plain":"<Figure size 720x360 with 2 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAmQAAAFACAYAAAASxGABAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd8VGX2P/DPmfQC6UAkQGgiRaSEoiACNsSChbW76uryw7LququLa13dVVf367rYsbvWXQuigCwKSC+hdwk9EEhIQippk/P7YyYhZZJMwty5M3M/79drXt6588ydkwiXM085j6gqiIiIiMg8NrMDICIiIrI6JmREREREJmNCRkRERGQyJmREREREJmNCRkRERGQyJmREREREJmNCRkRERGQyJmREREREJmNCRkRERGSyYLMDaK3ExERNTU01Owwi8qK1a9ceU9Uks+PwBN7DiKzF3fuX3yVkqampSE9PNzsMIvIiEdlvdgyewnsYkbW4e//ikCURERGRyZiQEREREZmMCRkRERGRyZiQEREREZmMCRkRERGRyZiQEREREZnMsIRMRMJFZLWIbBSRrSLyFxdtwkTkCxHJEJFVIpJqVDxEREREvsrIHrJyAONV9SwAgwBMEJGRDdrcASBfVXsB+CeAvxsYDxFRq4hIkIisF5HvXbx2m4jkiMgG5+NOM2IkosBgWGFYVVUAxc6nIc6HNmg2CcBTzuMvAbwqIuJ8LxGR2e4HsB1A+yZe/0JV7/ViPEQUoAydQ+b8drkBQDaA+aq6qkGTzgAOAoCqVgEoAJDg4jpTRCRdRNJzcnLc+uyKqmr8Z81BbDlUcEo/AxFZk4ikALgUwDtmfP7SXcfw/abDZnw0EZnA0IRMVe2qOghACoDhIjKgQRNx9TYX15mhqmmqmpaU5N52diLAw19twk/bs1sdNxERgJcBPAygupk214jIJhH5UkS6ePLDP1tzAP+c/4snL0lEPswrqyxV9TiARQAmNHgpE0AXABCRYAAxAPI88ZkhQTa0Dw9GXkm5Jy5HRBYiIpcByFbVtc00+w5AqqoOBPAjgA+buV6re/kBF99OiShgGbnKMklEYp3HEQAuALCjQbNZAG51Hk8GsMCT88cSosOQW1LhqcsRkXWMAnCFiOwD8DmA8SLycd0GqpqrqjXf+N4GMLSpi7Wplx9gRkZkIUb2kCUDWCgimwCsgWMO2fci8rSIXOFs8y6ABBHJAPAggGmeDCA+KhR5TMiIqJVU9RFVTVHVVADXw/Fl8ea6bUQkuc7TK+CY/O8xIq5mdBBRoDJyleUmAINdnH+iznEZgF8ZFUN8VCgO5pUadXkishgReRpAuqrOAnCf88tlFRxTLW7z9Oexg4zIOgxLyHxBQlQoNh48bnYYROTHVHURHHNgG36hfATAI0Z9rjg+w6jLE5GPCeitk+KjQpFfWsGbGhH5HY5YEllLwCdklXZFYVmV2aEQEbUav0oSWUfAJ2QAOLGfiPwOO8iIrMUiCRlrkRGR/+FsCyLrCOiELCEqDACQW8weMiLyLyIC5aAlkWUEdEIWH80hSyLyTxyyJLKWgE7IEpxDlqzWT0T+iEOWRNYR0AlZeEgQIkODkM+EjIj8jTAhI7KSgE7IAG6fRET+SThoSWQpAZ+QJUSFcsiSiIiIfFrAJ2TsISMif8RK/UTWEvAJWRwTMiLyU9z2jcg6Aj4hcwxZsjAsEfkXAbdOIrKSgE/I4qPCUFZZjdIK7mdJRP6DQ5ZE1hLwCVltLTJW6yciP8MRSyLrCPiEjBuME5E/EnDrJCIrCfyErGb7pFImZETkPzhkSWQtAZ+Q1QxZ5nHIkoj8DIcsiawj4BMyDlkSkT8S4SpLIisJ+IQsOiwYIUHCav1E5Gc4ZklkJQGfkImIs1o/a5ERkX/hkCWRdQR8QgY4apFxyJKI/Akn9RNZiyUSMm4wTkRtISJBIrJeRL538VqYiHwhIhkiskpEUj0fAbvIiKzCEgkZNxgnoja6H8D2Jl67A0C+qvYC8E8Af/fkBws4ZElkJUzIiIhcEJEUAJcCeKeJJpMAfOg8/hLA+SKeG2jkkCWRtVgiIUuICkVRWRUqqqrNDoWI/MfLAB4G0NSNozOAgwCgqlUACgAkuGooIlNEJF1E0nNyctwOgB1kRNZhiYSsplp/Pqv1E5EbROQyANmqura5Zi7OucyhVHWGqqapalpSUpJ7MUCgHLMksgxLJGTcYJyIWmkUgCtEZB+AzwGMF5GPG7TJBNAFAEQkGEAMgDxPBcAhSyJrsURCFhfJav1E5D5VfURVU1Q1FcD1ABao6s0Nms0CcKvzeLKzjUe7tNg/RmQdwWYH4A0JziHLXBaHJaJTICJPA0hX1VkA3gXwbxHJgKNn7HqPfpYnL0ZEPs8SCVl8VBgA9pARUeup6iIAi5zHT9Q5XwbgV8Z+tpFXJyJfYokhy9iIENiECRkR+Q8RTuonshJLJGQ2myAuktX6iYiIyDdZIiEDHMVh85mQEZEfYf8YkXVYKiFjDxkR+QsRMCMjshDLJGQJ0dw+iYj8h3CdJZGlWCYhi4tkQkZE/oUdZETWYZmELCEqFPmlFbBX8xZHRL5PBFxlSWQhlknI4qNCoQoc536WROQHOGBJZC3WSciiWRyWiPwL+8eIrMMyCVntBuNMyIjID3BzcSJrMSwhE5EuIrJQRLaLyFYRud9Fm7EiUiAiG5yPJ1xdyxPio7jBOBH5F04hI7IOI/eyrALwB1VdJyLtAKwVkfmquq1BuyWqepmBcQA42UPGhIyI/IGIQDloSWQZhvWQqWqWqq5zHhcB2A6gs1Gf15I4JmRE5Ec4YklkLV6ZQyYiqQAGA1jl4uWzRWSjiMwVkf5NvH+KiKSLSHpOTk6bYggJsqFdeDATMiLyGxyyJLIOwxMyEYkG8BWAB1S1sMHL6wB0U9WzALwCYKara6jqDFVNU9W0pKSkNseSwO2TiMhfCFdZElmJoQmZiITAkYx9oqpfN3xdVQtVtdh5PAdAiIgkGhVPfFQo8krKjbo8EZHHcOskImsxcpWlAHgXwHZVfamJNp2c7SAiw53x5BoVU3xUGHKL2UNGRH6CXWRElmHkKstRAG4BsFlENjjP/RlAVwBQ1TcBTAZwl4hUATgB4Ho1cK+QhKhQbMo8btTliYg8RgRcZUlkIYYlZKq6FC0sFFLVVwG8alQMDcVHO/azVFUIqy4SERGRj7BMpX7A0UNWaVcUllWZHQoRUbMEXGVJZCWWSshYrZ+I/AU78YmsxVIJWVI7xwbjOUVcaUlEvo8dZETWYamELDkmHACQVXDC5EiIyJeJSLiIrHYWrd4qIn9x0eY2EcmpsxfvnR6NAQID1zgRkY8xcpWlz+kUEwEAOFJQZnIkROTjygGMV9ViZz3FpSIyV1VXNmj3harea0QAHLIkshZLJWTRYcGIDgvGkUImZETUNGf5nWLn0xDnw+vdVewfI7IOSw1ZAkCnmHD2kBFRi0QkyFlDMRvAfFV1tRfvNSKySUS+FJEuzVyr1fvxcpUlkbVYLiFLjglHFhMyImqBqtpVdRCAFADDRWRAgybfAUhV1YEAfgTwYTPXav1+vByzJLIUyyVkndqzh4yI3KeqxwEsAjChwflcVa1Zsv02gKFeDo2IAojlErLkmHBkF5Whyl5tdihE5KNEJElEYp3HEQAuALCjQZvkOk+vALDdozF48mJE5PMsNakfcKy0rFYgp7gcyc5Vl0REDSQD+FBEguD44vofVf1eRJ4GkK6qswDcJyJXAKgCkAfgNiMC4VZvRNZguYTsZC2yMiZkROSSqm4CMNjF+SfqHD8C4BGjYqjJwVQ5nYzICiw3ZNnJmZBxHhkR+TLhoCWRpVguIavbQ0ZE5OtY+YLIGiyXkMVEhCAs2IYj3D6JiHzYySFLpmREVmC5hExEkBwTjiOF3GCciHwXByyJrMVyCRlQU62fPWRE5PvYP0ZkDZZMyJJjIjiHjIh8Wt1VlkQU+CyZkHWKCcfRwjJUV/NOR0S+yWZzZGTVzMiILMGSCVlyTDgq7YrckgqzQyEicik0yHF7ruSuIkSWYMmErFN71iIjIt8W7Owhq7Szh4zICiyZkNVU6M/ixH4i8lEhwY7bM/fdJbIGSyZktdX6C9lDRkS+KcTmuD1XMCEjsgRLJmQJUaEICRKutCQin3Wi0g4AGP33hSZHQkTeYMmEzGYTdGgXjqNMyIjIRx0rZvFqIiuxZEIGOFZasoeMiHxVsM2yt2ciS7Ls3/hOMeGcQ0ZEPuuSMzuZHQIReZFlEzJHD9kJbtxLRD7p9I7tzA6BiLzIsglZp5gIlFVWo+BEpdmhEBE1i18ciQKfZROyZGfpC84jIyJfty2r0OwQiMhglk3IamuRMSEjIh815vQkAMBD/91kciREZDTrJmTt2UNGRL4tIsRxi+YG40SBz7IJWVK7MNgEOMLtk4jIRwU597OsqmZCRhToLJuQhQTZkNQujKUviMhndWrv2Hc3I7vY5EiIyGiWTcgAx0pLDlkSkSsiEi4iq0Vko4hsFZG/uGgTJiJfiEiGiKwSkVRPxvD7C3t78nJE5MMsnZAltw/npH4iako5gPGqehaAQQAmiMjIBm3uAJCvqr0A/BPA3z0ZQFRosCcvR0Q+zNIJWacYJmRE5Jo61IwVhjgfDSdzTQLwofP4SwDni4h4KgabzWOXIiIfZ+mELDkmHEXlVSgqY3FYImpMRIJEZAOAbADzVXVVgyadARwEAFWtAlAAIMHFdaaISLqIpOfk5BgdNhH5IUsnZDW1yI5yYj8RuaCqdlUdBCAFwHARGdCgiasurEZLIlV1hqqmqWpaUlJSm2LJKSpv0/uIyD9YOiFLjnGsYOLEfiJqjqoeB7AIwIQGL2UC6AIAIhIMIAZAnhExjH1xoRGXJSIfYemEjMVhiagpIpIkIrHO4wgAFwDY0aDZLAC3Oo8nA1igBm08WVJhN+KyROQjDEvIRKSLiCwUke3OJeP3u2gjIjLduWR8k4gMMSoeVzq0DwPA7ZOIyKVkAAtFZBOANXDMIfteRJ4WkSucbd4FkCAiGQAeBDDNpFiJyM8Zuaa6CsAfVHWdiLQDsFZE5qvqtjptLgHQ2/kYAeAN53+9IjwkCAlRoSwOS0SNqOomAINdnH+iznEZgF8ZGceXU8/G5DdXGPkRROQDDOshU9UsVV3nPC4CsB2OFUl1TQLwkXN5+UoAsSKSbFRMrrD0BRH5st4d25kdAhF5gVfmkDmrVw8G0OSScadMNE7aDF0ynhwTzjlkROSzYiJCzA6BiLzA8IRMRKIBfAXgAVUtbPiyi7cYsmS8KckxEcjML4VB83CJiDyGpS+IApehCZmIhMCRjH2iql+7aFK7ZNwpBcBhI2NqqFtCJIrKqnC8lMVhici37c7hJuNEgcrIVZYCxwqk7ar6UhPNZgH4tXO15UgABaqaZVRMrqQmRAEA9uWWePNjiYhaLSSIWykRBSoje8hGAbgFwHgR2eB8TBSRqSIy1dlmDoA9ADIAvA3gbgPjcSk1MRIAsD+31NsfTUTklmeurNkggAkZUaAyrOyFqi5FC3cPZwHFe4yKwR0pcZEQYQ8ZEfmumiLWT3y7BbPvO9fkaIjICJau1A84apGdFhPBHjIi8ln5JRUAgK2HCzHh5cUmR0NERrB8QgY4Jvazh4yIfNWJypPbJu04UmRiJERkFCZkALolROEAe8iIyEeN6pVodghEZDAmZABSEyKRW1KBwjKWviAi39OrQ7TZIRCRwZiQwTFkCYC9ZERERGQKJmRwDFkCXGlJRERE5mBChpM9ZFxpSUT+4Aj33yUKOEzIAESGBqNDuzDsO8YeMiLyTV/ddU7t8Z++2mRiJERkBCZkTqkJUewhIyKf1S+5fe1xpb3axEiIyAhMyJxYi4yIfFmQ7eTGJ1XVamIkRGQEtxIyEekpImHO47Eicp+IxBobmnelJkYhu6gcpRVVZodCRB4WCPew4DoJmZ0JGVHAcbeH7CsAdhHpBeBdAN0BfGpYVCbgxH6igOb39zAbEzKigOZuQlatqlUArgLwsqr+HkCycWF5X6qz9AUTMqKAFFD3sOxCrrIkCjTuJmSVInIDgFsBfO88F2JMSOboWttDxnlkRAEooO5hWUzIiAKOuwnZ7QDOBvA3Vd0rIt0BfGxcWN7XPjwE8VGh2MceMqJA1Kp7mIh0EZGFIrJdRLaKyP0u2owVkQIR2eB8PGFg/PUoRyyJAk6wO41UdRuA+wBAROIAtFPV540MzAzdEiLZQ0YUgNpwD6sC8AdVXSci7QCsFZH5zuvUtURVLzMmaiKyEndXWS4SkfYiEg9gI4D3ReQlY0PzPtYiIwpMrb2HqWqWqq5zHhcB2A6gs3eiJSIrcnfIMkZVCwFcDeB9VR0K4ALjwjJHt4RIHC44gbJKu9mhEJFntfkeJiKpAAYDWOXi5bNFZKOIzBWR/p4K1h1bDxd48+OIyGDuJmTBIpIM4FqcnBAbcFIToqAKZOazl4wowLTpHiYi0XCUzHjAmdDVtQ5AN1U9C8ArAGY2c50pIpIuIuk5OTmtj97pvvN71x5fOn1pm69DRL7H3YTsaQDzAOxW1TUi0gPALuPCMkdNLbJ9x5iQEQWYVt/DRCQEjmTsE1X9uuHrqlqoqsXO4zkAQkQk0dW1VHWGqqapalpSUlKbf4i6xWGJKLC4O6n/vwD+W+f5HgDXGBWUWWpqkXELJaLA0tp7mIgIHAVkt6uqy7lmItIJwFFVVREZDscX3FyPBt5AUIOErNJejZAg7oBHFAjcndSfIiLfiEi2iBwVka9EJMXo4LwtNjIE7cODObGfKMC04R42CsAtAMbXKWsxUUSmishUZ5vJALaIyEYA0wFcr2psQYpr07rUez5/21EjP46IvMitHjIA78OxzcivnM9vdp670IigzCIiSE2Mwv48JmREAaZV9zBVXQqg2fFBVX0VwKsejLFFSe3CsOUvF2PAk/MAcJNxokDibl93kqq+r6pVzscHANo+EcKHdY1nLTKiABQw97DosJPfo6vs1SZGQkSe5G5CdkxEbhaRIOfjZhg8V8IsqQlRyMw/gUre6IgCSUDew4rKqswOgYg8xN2E7DdwLBc/AiALjrkTtxsVlJm6JUTCXq04lH/C7FCIyHMC8h725KytZodARB7iVkKmqgdU9QpVTVLVDqp6JRwFFgNOaiJXWhIFGivdw4jIP53KeukHPRaFD6mpRcaVlkQBz2/vYS9fN6j2mIWsiQLDqSRkAVmhMCk6DJGhQdh7jD1kRAHOb+9hdSf2j/vHIvMCISKPOZWELCDXW4sI+iW3x4aDx80OhYiM5bf3sOE94muPK+2KIwVlJkZDRJ7QbEImIkUiUujiUQTgNC/F6HVpqfHYeriAm4wT+blAvYe1Dw/BsNS42ucbDuabGA0ReUKzCZmqtlPV9i4e7VTV3aKyfietWxwq7YqN7CUj8muBfA978vL+tcfG7g9ARN7ATdBcGNrN8c0zfT+/dRKRb4oIDao9ZsF+Iv/HhMyFuKhQ9EyKwlomZETkoyJCTiZk6r/T4YjIiQlZE9K6xWPt/nxU86snEfmgqNCTI668TRH5PyZkTRiaGoeCE5XYnVNsdihERI3ERIbUHisnkRH5PSZkTUjjPDIi8hPMx4j8HxOyJnRPjEJCVCjS9zEhIyLfxhI9RP6PCVkTRARDusVh7f48s0MhImrWtK83mx0CEZ0iJmTNSOsWh325pcgpKjc7FCKiRp64rF/t8WMzmZQR+TMmZM1Ic1bCZvkLIvJFneMiao8/XnnAxEiI6FQxIWvGgM4xCA22IX0fhy2JyPckRIU2OldQWsk5ZUR+yLCETETeE5FsEdnSxOtjRaRARDY4H08YFUtbhQUHYWDnGK60JCKflJYa3+jcWU//D9fNWGlCNER0KozsIfsAwIQW2ixR1UHOx9MGxtJmQ1PjuNE4EfkV7sNL5H8MS8hUdTEAvx/rS+sWz43GichnvXdbWu1xaUWViZEQ0akwew7Z2SKyUUTmikj/phqJyBQRSReR9JycHG/Gx43GiSxIRLqIyEIR2S4iW0XkfhdtRESmi0iGiGwSkSFmxGqvPnn80JebzAiBiDzAzIRsHYBuqnoWgFcAzGyqoarOUNU0VU1LSkryWoAAEB8Vih7caJzIaqoA/EFV+wIYCeAeEenXoM0lAHo7H1MAvOHdEB2q65Tp355VaEYIROQBpiVkqlqoqsXO4zkAQkQk0ax4mjOMG40TWYqqZqnqOudxEYDtADo3aDYJwEfqsBJArIgkezlU7mNJFCBMS8hEpJOIiPN4uDOWXLPiaQ43GieyLhFJBTAYwKoGL3UGcLDO80w0TtpqrmHYtIv2ESc3Gd+TU+LRaxOR9wQbdWER+QzAWACJIpIJ4EkAIQCgqm8CmAzgLhGpAnACwPXqo1/1RnR3LC1flnEMvTu2MzkaIvIWEYkG8BWAB1S14XiguHiLy3uYqs4AMAMA0tLSPHqfO6enTw4sEFErGZaQqeoNLbz+KoBXjfp8T+qWEIUeSVH4aUc2bhvV3exwiMgLRCQEjmTsE1X92kWTTABd6jxPAXDYG7ERUeAxe5Wl3zj/jA5YtScPxeVcVk4U6JzTKd4FsF1VX2qi2SwAv3authwJoEBVs7wWJBEFFCZkbhp/RkdU2KuxdNcxs0MhIuONAnALgPF1dhOZKCJTRWSqs80cAHsAZAB4G8DdJsWK343vZdZHE5GHGDZkGWjSUuPQLjwYC3YcxYQBncwOh4gMpKpL4XqOWN02CuAe70TUPOf6KCLyY+whc1NIkA1j+3TAgh05LH9BREREHsWErBXOP6MDjhWXY/OhArNDISI6yTcXqBNRKzAha4XzTk+CTYCfdmSbHQoRUS122hP5PyZkrRAXFYqh3eKwYMdRs0MhIqqlrsufEZEfYULWSuPP6IgthwpxpKDM7FCIiAAAt56dir7J7eudyyo4YVI0RNQWTMha6fy+HQAACzhsSUQ+okP7cHxz9zn1zl3z+nIAQGlFFV5bmIEqe7UZoRGRm5iQtVLvDtFIiYvgsCUR+ZQgW/3SF4edvfj/+nEXXpy3E1+vP2RGWETkJiZkrSQiOP+MDliacQxllXazwyEiAgAE21zXIiupcOwuUs77FZFPY0LWBuP7dkRZZTVW7M41OxQiIgCOL4uf/nZE4/PO+rZciUnk25iQtcGI7vGIDA3CTxy2JCIfck7PxHrPK+3VYBF/Iv/AhKwNwkOCMLpXIn7ang07v3YSkQ95/LJ+tcev/LSrdv8nZfFYIp/GhKyNrh6SgqyCMny8cr/ZoRAR1YoMDao9nr4go3afS353JPJtTMja6OL+HXFu70T8Y95OZBexJhkR+YaGc/vLq1jugsgfMCFrIxHBX67oj/Kqajw7e7vZ4RARAUBtj1iNz1YfAADW8ifycUzITkGPpGhMPa8HZm44jOW7j5kdDhERhnaLc3l+2+FCL0dCRK3BhOwU3T2uF7rER+CJb7eigkMDRGSynknRLs9/tS7Ty5EQUWswITtF4SFBeOry/sjILsa7S/eaHQ4REb7/3WizQyCiVmJC5gHn9+2Ii/p1xPSfdiEzv9TscIiIiMjPMCHzkCcud9T+eW7ODpMjISIiIn/DhMxDUuIi8ZvRqZizJQsZ2cVmh0NERER+hAmZB/1mVHeEBdvw5s+7zQ6FiIiI/AgTMg9KiA7D9cO6Yub6Q5xLRkSmaWqXpD99uQnj/rGIJTCIfBATMg+bMqYHAODtxXtMjoSIToWIvCci2SKypYnXx4pIgYhscD6e8HaMTTktNhwAcHrH+iUwvkg/iL3HSjBx+hIzwiKiZjAh87DTYiNw9ZDO+HzNQeQUlZsdDhG13QcAJrTQZomqDnI+nvZCTG5JiA7Drr9dgnkPjGnzNcoq7ZwPS+RFTMgMMPW8nqiwV+P9ZaxLRuSvVHUxgDyz42irkCBbo22UWuOP/92IC176GUVllR6MioiawoTMAD2SojHxzGT8e8V+FJzgzYwogJ0tIhtFZK6I9G+qkYhMEZF0EUnPycnxZnxN2nespNnXV+7JBQCUVXIHEiJvYEJmkLvH9kRReRU+Xrnf7FCIyBjrAHRT1bMAvAJgZlMNVXWGqqapalpSUpLXAgSARX8c6/L82H8s8mocRNQ8JmQG6X9aDMb1ScK7S/fiRIXd7HCIyMNUtVBVi53HcwCEiEiiyWE1kpoYZXYIROQGJmQGumdcL+SVVOA/6QfNDoWIPExEOolzkpaIDIfjfpprblStU1JeZXYIROTEhMxAaanxGNw1Fu8v24vq6iYKAxGRTxKRzwCsANBHRDJF5A4RmSoiU51NJgPYIiIbAUwHcL1qUxXAzPV/vzrL5fn+T87DtW+tAACoKu9TRCZiQmawO0Z3x77cUizYkW12KETUCqp6g6omq2qIqqao6ruq+qaqvul8/VVV7a+qZ6nqSFVdbnbMTblmaEqTr63e61hI+vqi3ejx5zkoZq8ZkSmYkBlsQv9OOC0mHO+xBAYRmej1m4Y0+dq6A/m1C5DySyoAAHZnb5mCvWZE3sCEzGDBQTb8+pxULN+di+1Z3K6EiMwx8cxkDOka6/K1q19fjqyCsnrn8ksdJXtmrj9keGxExITMK24Y1hURIUF4byl7yYjIPF/fParV78ktrjAgEiJqiAmZF8REhmDy0BR8u+EwjhVzOyUiMs+CP5xndghE5AITMi+5bVQqKuzV+GTlAbNDISIL65EUjYEpMWaHQUQNMCHzkp5J0RjXJwn/Xrkf5VUsFEtE5hmWGt/kax+t2Idnvt/m9rWq7NxaicgTmJB50R2je+BYcTm+25hldihEZGHNVUt7e8levFtnvuveZva8XH8gH70enYulu455MjwiSzIsIROR90QkW0S2NPG6iMh0EckQkU0i0vSa7AAxqlcCTu8YjXeX7oWP1o8kIgsIDhK32/5v29EmX1vlrGG2eJdvbJhO5M+M7CH7AMCEZl6/BEBv52MKgDcMjMUniAh+e24PbM8qxFOztjIpIyJTXDmos0euU3MLcz+9I6KmGJaQqepiAHnNNJkE4CN1WAkgVkSSjYrHV0wemoLfntsdH67Yj0dnbuHx4oZnAAAgAElEQVRWJUTkdf1Oa499z1/a6vcdzCtFbp2V4rVFY5mREZ2yYBM/uzOAurtuZzrPNZpgJSJT4OhFQ9euXb0SnFFEBH+e2BfBQTa8sWg37HbFc1efCZuNdzQi8q6IkCCcqGx5kVFFVTVCg20494WFEAH2PudI5k72kPH+RXSqzJzU7+pvsMvuIlWdoappqpqWlJRkcFjGExE8fHEf3De+F75IP4iHvtxUu00JEZG3XNS/o1vtPlqxr/bY1UwL8VI+VlZpR3ZRWcsNifyQmQlZJoAudZ6nADhsUixeJyJ48KI+ePDC0/HVukw8+J8NqOTycSLyouevHuhWu7/N2e7yfM08WG/1j9363moM/9tPXvo0Iu8yMyGbBeDXztWWIwEUqKrl6kHcd35v/GnCGfh2w2Hc9fE6lLkxfEBE5AkRoUG1xzeNaHo6iCqwak+uy/OA93rIalZ1EgUiI8tefAZgBYA+IpIpIneIyFQRmepsMgfAHgAZAN4GcLdRsfi6u8b2xDOT+uPH7Udx+/trUFxeZXZIRGQxf7vqzGYn+t/y7upG52pGLzmHjOjUGTapX1VvaOF1BXCPUZ/vb245OxXR4cH443834eZ3VuGD24chNjLU7LCIKMB9eucIhIUEtdiuwsWUipoessz8UsxcfwhXDvZMOQ0iK2Klfh9y1eAUvHHTEGw7XIjr3lqJ7EJOXiUiY53TKxFDu8XVPu/QLqzF96ROm42FO7Jry17M3HAYD3yxwbAYiayACZmPuah/J7x/+zAczC/FVa8vx84jRWaHREQWYnNzQtjtH6xBw8XhLHZN1HZMyHzQqF6J+GLK2ai0V2PyG8uxhNuSEJEvapCAsXoPUdsxIfNRZ6bEYOY9o9A5LgK3vb8Gn68+YHZIRGQB485wv9Zjhb1hQsaMjKitmJD5sNNiI/DfqWdjdK9ETPt6M56bu73ZshjZhWX414+7sGBH05sBExE15+lJA7Bs2ng8dmlfAMAzk/o32fbNn3fXe77h4HEOW/qJ2ZuysPHgcbPDoDqYkPm4duEhePfWNNw0oive+nkPhv3tRzz6zWZsrHPj23GkEH/870aM+vsC/PPHXzD13+uwZh/r9RCdChF5T0SyRWRLE6+LiEwXkQwR2SQiQ7wdoxFCgmzoHBuBO8/tgX3PX4pbzk51+72/enMFvt1gmfrebll/IB/XvbUCFVW+Vfj7nk/XYdJryzBz/SGzQyEnJmR+IDjIhr9eOQCf/nYELujbEV+ty8Sk15bh4pcX46Z3VmLCy0swe1MWbhjeFd86hzmnfJSO/bklZodO5M8+ADChmdcvAdDb+ZgC4A0vxGSKv145wO22u3OKDYzE/0z7ajNW7c1r9Hspr7L7RJLG1bG+gwmZnxARnNMzEf+8bhBWP3oBnr3qTESFBWPfsVI8dHEfrHhkPJ6eNABndYnFe7cNgwL4zQdrUHCissVr26sV9322Hre8uwo5ReXG/zBEfkBVFwNorqt5EoCP1GElgFgRSfZOdN41eWiK221fWZCB1Gmz8c6SPQCAuZuzkDptNnYcKTQqPJ9Ws2h11sbDWHcgv/Z8n8d+wLh/LPJqLLtzijF/m29Oaam0V+O7jYeROm02SiusWRydCZkfah8eghtHdMU3d4/Csmnjcc+4XvWKyHZPjMKbNw/FgbxS3P3J2hb3yHx+7nbM2ngYK/fk4vJXlmID5xU069DxE3hnyR5Uc0mZ1XUGcLDO80znuYATHhKEP1x4eqve89fZ2/HoN5vx3rK9AIAJLy8xIjS/8cai3bj69eX1zh06fsKrMZz/fz/jtx+lY6WLbbDM1v/JefjdZ+sBAJn53v29+AomZAFqZI8EPHvVmViWkYsnvt3a5ETbL9YcwNtL9uK2c1Lx7T2jERwkuPbNFfjPmoMu23uTquLdpXvx6Debcc8n63DDjJWY8PJiTHp1qVs9f0b5cPk+/HX2dnzuA78jMpWrgl0u/6KJyBQRSReR9Jwc/yxj87vze7f6PZ+sOoA1+072Ctlb8SXmSEEZUqfNxuJf/PP3VWOHD9aSvH7GSrNDaKTu8K1VV+syIQtgv0rrgrvH9sRnqw9g6sdrcTCvtN7rK/fk4rGZW3Bu70Q8dmlf9DutPb67dzSGd4/Hw19twuMzt5g6x2HNvnw88/02fL8pCzuOFKLSXo1OMeHYmFmAr9dlmhZXzSbLz8/djuwi7qZgYZkAutR5ngLA5Yx2VZ2hqmmqmpaU5H5ZCV8z9/5zcVZKTJvfvyzjWL3nq/fmId25AKm4vAqp02Zj7IsLsXZ/PpbvdrT9eOX+Rtdx9QXz89UHkDptNh78j+/MiSrhvsRtYtF8jAlZoPvjRX3w0MV9sPiXY7jgpZ/x0vxfcKLCjv25Jbjr47XoEh+JV28cguAgxx+FuKhQfHD7MPy/MT3w75X7MeaFhXh78R4UlXm/R+qrtZmIDA3C8mnj8dMfxuLLu87BB7cPx1kpMfh01QFTltcXl1dhy+FCXDowGWWV1Xjm++1ej4F8xiwAv3authwJoEBVs8wOykh9k9vj23tHY2SP+Da9/9fvrcazcxx/ZzLzS3HtWysw+c0VmPzGcrzlLKGxL7cU17yxHA/+ZyMA112Orv7qT/t6MwDg63WHcOZT83C8tKJNMXrCqj25KDhRWW/OGDWt4Zwxq/aQGba5OPkGm01wz7heuHpIZzw3Zwem/7QLX6YfRFhIEKoVeO/WYYiJCKn3nuAgGx6Z2Bfn9k7C64sy8Lc52zH9p124cWRX/GZUd3RsH2543Ccq7Ji9OQuXDEhGVFj9P6Y3juiKP321Gen78zEstW3/MLTV2v35sFcrrh/WBb07ROPlH3fhmiGdMbZPB6/GQcYTkc8AjAWQKCKZAJ4EEAIAqvomgDkAJgLIAFAK4HZzIvW+T+4ciZ5/ntOm985YvAczFu+pdy59fz7S97tOXlwlVg3/uW44F6uorAqDnp6PpX8ah5S4yDbF2ValFVW4bsZKDE+N54pTN2Tml2L03xfWO2fRfIw9ZFaRHBOB6TcMxhdTRqJ9RAgy80vxxs1DkJoY1eR7RvdOxKe/HYlZ947CmD5JeHvxHox5YSFW7zW+xtn/th1BcXkVrhnaeI705WedhnZhwfhslfd3L1i9NxfBNsHQbnG4a2xP9EiKwuPfbsGJiqYL9pJ/UtUbVDVZVUNUNUVV31XVN53JGJyrK+9R1Z6qeqaqppsds7cE2dzb79IT1uzLbzTU2bB3fNTzC1y+9/JXlqLKXo0Plu3Fc3O3Y87mLHy30dg6aZXO3Qs2HypAbknjZPKjFfsM/Xx/cyC3tNE5q/aQMSGzmBE9EjD7vnOx8pHzcU7PRLfeMzAlFq/dOASL/jgOnWMjcPcna5FVYOwqmC/XZqJzbARGdk9o9FpkaDCuHNwZ32/O8vqwxKo9eRjQOQaRocEICw7Cs1ediYN5J/Cvn3Z5NQ4is3VsH1Z7fOnAZCRGhzbT+tTc51x9V2PNvvzauZzN3QPySyvx0Jeb8NR32/DWz3tw9yfralfyGe1EE7uqPPHtVszeFNAj263iKrnfmFlgQiTmY0JmQUE2QUJ0WMsNG+iaEIm3bhmKExV2TP14HcqrjOkVOlJQhmUZx3D1kM6wNfFN/IbhXVFRVY2v1nmvynRZpR0bM49jRJ35MyN7JODatBS8vWQPtmeZW2dp7uYsvL4oA9sOF3L7GjLcyB6OL0v3jOuJ6dcPRvpjF+Khi/sAAJ65cgAuHei5kmy5JRXo8cjs2uc3vL0S181YCVXF9qzmVzF+Y3Al+p1HirB017GWG9Zxz6frDIrm1BzMK0VBqXfnC9fMX67r8ZkuN8cIeEzIqFV6d2yH/7v2LGw8eBxPzGy6nMap+Gb9IVQrcPWQpotR9jutPQZ1icWnq/Z7LflYdyAflXbFiO715609cklfxESE4C/fbfVKHK5U2qsx7evNeOGHnZg4fQnOfm4BHvl6E+ZvO8rkjAxhc1Y87ZEYXdvLcc+4Xtj97ETcMrIbXrtxiEeHNl1VzOj+SNvmsXnSxS8vxs3vrjI7DI8494WFuOjln736mcFeHP72dUzIqNUmDEjGPeN64ov0g/h0tWfncakqvlx7EEO7xaF7M/PbAMfk/t05JfXqHBlp9d48iABpDRYSxEWF4q7zemLlnjxsPWxOV/vqvXkoOFGJZ686Ey9MHojBXWPx3cYs/PajdLy7dK8pMVFgu3tsT3SNj8S4M+ovaKmbhE08s3W9ZL8+u1ur47jh7dbX1HrgcwOHLVv5/Se/pAI/eqF6/kvzf3Gr3dFC7+7W8sPWIy7PX/naMlPLG5mBCRm1yYMX9sF5pyfhqVlbsXa/5yb5b8wswO6cElzTTO9YjcsHnoZ24cH4dFXjOkVGWLUnD/2S26N9eEij165N64KIkCB8sGyfV2JpaN7WIwgPseGqwZ1xbVoXvHHzUKx7/EIMS43DRyv2c1cB8rjeHdth8cPjEB/V9NyxFycPrPd83gNjmr3mpEGdse7xCz0SX3NmbjjcqLj0npzi2u2eToW6yMjqzrdr6Lb3V+POj9KRW2xcIrQ7pxjTfXSe68Id2S7Pbzh4vLb0iVUwIaM2CbIJpl8/GMkxEbjurZW46Z2V+HD5Phw+xa1AvlqbidBgm1vzTyJCg3DV4M6Ys+UI8l2sZvKkiqpqrDuQjxEuFhkAQExkCK4e0hnfbjxs6I3Vlepqxf+2HsV5pychIjSo9nxosA23nJ2KA3mlWJLRujkuRJ4QHhKEK846rfZ5n07tmm3fLjy42QTPk95b6lh5WTOkf+1bK/HX2dux62jRKRV0dTVD4LFL++GD24e5bF8zgb3cwCLcLe1RPODJea3aRcGTrLqi0hUmZNRmMZEh+GzKSEwZ0wNHC8vx5KytOOf5Bbj8laV4cd4O/PxLDopbcWMrr7Jj1sbDuKhfx0a10Zpy44iayf3Gdm1vyjyO8qpqDO/edN2z285JRUVVtde3VNp0qABHCstwcf9OjV67uH9HJESF4hMX1c6JvOGf1w1CeIgNt52TCgB46dqzGrUJD7Ghc2wEeneIBgA8dXk/w+P610+78NbPe9D9kTl4bWEGjjm/SF34z8UY62LT722HC5GRXYwJLy/G/tySJq/rKsGIDA1qsVahNxOTTU9dhM6xEbXPi8urXO6I4A1V9uZ/7rJKu+kLpryFhWHplHSOjcDDE87AwxPOwO6cYszfdhTztx3Fmz/vwWsLd8MmQP/TYjCoSywq7dXILalAXkkF8ksqUK2Ks7rEYmi3OAzpGod9uSUoOFGJa4a2PFxZ44xO7TG0WxzeWbIXVw3u3KbVo+5Y5ay91lxC1rtjO5zbOxH/XrEfU8b0QIiL1UNG+GHLEQTbBOef0bHRa2HBQbh2WBfMWLwHWQUnkBwT4eIKRMYJsgl2PHNJ7fNLBiTXDkUtfmgc4qJC0K7BNIDbRnXH64t2I7uFnp26/nX9INz/uWPbpDduGoK7PnF/JeOL83bWe55TVI6PVjj2rP1q6jnoHBeBidNPbo5+3ouLsP7xC7H1cP1E4aX5v+CVBY2HBsc759ndMrIb/t1E4lNt4C51DafNR4YEYVDX2HoFdZ+cZc6ipMoWfvAzHv8BAPDmzUMxYUDjL53elldSgdF/X4BHL+2Lm0a0fs5jc5iQkcf0TIpGz/OiMfW8nigpr8L6A8exel8eVu/NxTfrDyEiNAgJUaGIjwpF39Paw25XrNidi283nCzU2KFdGM7t5V59tBpPXd4f17y5HL/7bD0++s1wl8uoT9WqvXk4vWN0i8Mpt52Tijs+TMcPW47g8jpDNUZRVfxv6xGM7JGAmEjXvYo3DOuKN3/ejc9XH8TvLzzd8JiImhMRGoQ9z07EsZJydGjX9K4fy6aNhypw0zsr3Vq4M2lQZ8RHhaJDu3D0SIrCJQM6Ye4W1xPG3fHEt44E5fJXl7p8ffAz8+s9f2n+Ly7naQ3vHg9xrkh95soBiI8KdVm3cMyLC/Hjg2PQq0Pzw7qeEBxka5SkuVJRVY0v12bi+mFdmixB1Faqih+3Z9fbL/msLrHYePC4y/a7jhb5REJWrYrSCrshQ7xMyMgQUWHBGN07EaN7N59cqSoOHT+Btfvzsf7AcYzsEd/qhOrMlBj87coBeOjLTXhh3k78eWLfUwm9kSp7Ndbuy2u2DEeNcX06oFtCJD5Yvs8rCVlGdjH2HCvB7aO7N9mma0IkxvROwudrDuB343sZkrAStYbNJs0mYwBqe5g//M1wbM8qcva878a5vROx/sBxl9Mhzu19cuP2N24eiie+3YLuiVH4y3fbPPsDuNDUpPmrBtffbeT+83s3WUh60qvLsOHJi7zSuz7m9CR830KB2jd/3o2X5v+CsGBbq0Yu3DF3yxHc3aAX8/mrz8SSXTl4ds6ORu3tPjLXrCaMmiTbk3hnJlOJCFLiIjFpUGc8dUV/TBjQtmKSv0rrgltGdsOMxXs8vjXK1sOFKKmw1ysI2xSbTXDr2alYuz8fmzJdf9PzpHnOJeMX9Ws8XFnXTSO64mhhOX5qYkUTka+KDA3G0G5xmHbJGdj97ER89Jvh2PKXi91679OTBuD2Ud2x7/lLMfW8ngZH6lrDjqXmeppKKux45adduOfTdTj7uZ8MjetXQ1OwvokVrZ+uOgBVrS2V0Zq5wO7KLixrdK57YhSmjOmJG0d0bfSarywUr1kEYkT1NCZkFDAev6wf0rrF4eEvN2HHEc9NAl3txvyxuianpSAqNAgfLN/nsRiaMm/rUQzuGtvihu/jz+iA5Jhw0ybuEnlCkE1qeyb+NOEM9EiMwsI/jsXqR89v8b0PX9wHm5+6CPuevxR3NNOj7GlJ7RrPa933/KXY9/ylLttPX5CB2ZuykFVQhtRps1Flr8Ynq/bjtYUZLos8L9nlWDz1+eoD9VZ47zxShI9X7kd5lb1eMtM13rHZuoggLioUF/Rt/GXuz99srnf/enLWVqROm42M7Ma7IpyosCN12mx8sKx19Q4b/iS3j0pFeIhjlXiIi6R10U7f+DJZ87u0sYeMqGmhwTa8ftMQtAsPxv/791qPlcJYtTcXPRKjWhxiqdE+PASTh6bg+41ZLS43PxWHjp/A5kMFLldXNhQcZMP1w7piya5jza4QI/IXd43tiQV/HIvubv7dtNmkdvFAzRZPAPDYpZ6d4lDXwJQYjGtmdaU7n93r0bl49JsteHHeTvw3/eRq8oU7s7FkVw5ueXc1Bjw5D9O+3owb316FA7mlKKu04+KXF+OxmVvQ57EfUGE/OU9r8cPj6l3/jZuHuPxcV8O8f3BRFyzPuZfoW4v3YNvhQtz18doWfyZX1w+qk+D8sc7/nxqbMgu8vnexKzV15ozYYIAJGQWUDu3D8cbNQ3D4+Amc8/wC3P/5eizcmY0qe9uWMJVV2rF6b57bvWM1bj0nFZXV1fj1e6uxYndumz67JfOcE5bdScgA4LphXRBkE4/vrkDkb8KCHf/03TW2J+48twcW/OE8Qz5n1r2jm51rdL6L3qnmPPzVJry/bC/W7s/H7e+vwS3vrq73+s6jRRjz4kI89OWmeueX7266DmFIkK3J3rqGKuyKiqpq/N//djaq1Xai0o6J05dg7pYjmLM5C/9JP4jffLCmXpsqezVSp83GnR+mN7p2Wmpc7XG78BDMue/cRm2ue2ul6TuPVNfOIfP8tcXf9rlLS0vT9PTG/zOJ6tpyqACfrT6A7zdloeBEJRKjQ3HZwNMwskc8BqbEIjkmvMkbZWlFFRbtzMGczVlYsCMbpRV2vHXLULcTnxpzN2fhme+34XBBGS7q1xF/ntgXqS1sB9Ua1721AvmlFfjf793/x2Tqv9di9b48LPvT+HpFZH2diKxV1TSz4/AE3sN801OztiIz/wSevXoAznluAc7plYjFv+Q0+5724cG4ZmgKusVH4ikXPUruJDpfr8v0WkX6xy/r1+Rw7cs//oKXf2y5mn+vDtHIyC7GiO7x2J1TjGPFzfdafXXX2ejQLhx5JRWY9NqyJtu5+l0tyziGm95pvE9o3bbHistxtLAM/U+LaTF2T8jML8Xovy/EC5MH4tq0Lm69x937F1dZUkAa0DkGf7vqTDxxeT8s2pmDb9YdwqerDtTOi0hqF4azUmLRLSESpRV2lFZUoaS8CoVlVdiUeRxlldVIiArFlYM747Izk3FOK0txAMAlZyZj3Bkd8M6SPXh90W5c+M+f8euzUzH1vJ4u55W0Rm5xOdbsy8O943q16n13ntsdP2w9gufmbsfTkwacUgxEgeSpK/rXHmc8OxGA48vZuv3HMbRbHP7xv524d1wv/Cf9IIJsgjvP7VHv/eEhQcgrrcALPzhqmv154hlufe7VQ1IQGRqMqW4O9Z2Ky5rZAeX+83vjhy1HsONI43lidWVkFwM4WZuxJde8scL9ABsY1cR9N3XabKx59AIktQvDxH8tQXZReaOErrpaPV6qA6izytLjV2ZCRgEuLDgIF/fvhIv7d0J5lR3bs4qw8eBxbDx4HBsyj2P57mOIDA1GdFgQIkODERUWhF8N7YKJZyZjePf4ehslt0V4SBDuHd8b16Z1wT/+txPvLduLj1fux7VpXTBlTA90cU6wBQB7tWLLoQIs352LkCBB3+T26NOpHRKdxW6rqxUZOcVYtz8f87cdRbUCF7Wy1y4tNR53jO6Od5fuxfl9O+K805NafhORRUWGBteW7nn8MsfuAf+vidWa1w93rAwc0T0BQ7rGtqosQkurpD3F1T68NUQEz159Jq5+fblXYnHXsmnjMer5BY3OX/PGcvzx4j61xYOn/7QLU8/ridBgGw7mleLcFxbipWvPcqtcUVOyi8pgE6m9BwMnEzIjJvUzISPLCAsOwqAusRjUJdbrn92hfThemHwW/t95PTHj5z34fM0BfLr6AC4fmIxh3eOxfHculmUcw/HSykbvTYwOQ9f4COzKLkZRmWPeRmxkCG4e2RX9T2vf6lgeurgPFv+Sg4f+uxHzHhiDOC/tHUhkBUO7xbXcqAGbTfDniWcgNiIUz83djnwX94G2+PmhsVh/4Dge+MKxg0F4SPPTxod0jcOuv12C3o/O9cjnu+OBC3rjnJ5Nj0B0jo3A7PtG49Lp9Qv0HsgrxX2fra99/tL8X2AT4N7xvWt78b7dcPiUErLhf3OUHqnb+1azxZURc8iYkBF5Uc+kaPx98kA8cGFvvLNkLz5bfQAzNxxGx/ZhuKBvR5zbOxGjeiVC1bFsfceRQuw8UoT9eaW4bOBpzm2mYtE9MarNhQnDQ4Lwz+sG4arXl+Gxb7fg1RsGG1LkkIjcN2WMo+dt8tAUvL98H575fhs+uH0YZq4/hPUHj+OaISm1dcHc1S0hqt52cu78PQ8JsiE8xIayymqMOT0JFVV2jO3TAc/PbVys1RMeuKDl3UPcnR/28o+7HD2Yzh9zx5FC/LT9KNJS4xvtj/y/rUdw6PgJnN0zAa8v3I3HLuvr1mrdmln37CEjChDJMRF4/LJ+uG98b+SWlLtMsJLahbW400FbDegcgwcuOB0vztuJi/p1xKRBnVt+ExEZzmYT/GZUKoalxmFgSmy9TcmHd4/HU7O2YuY9o5BVUIalGcdw84iuKK2w44EvNuCpK/rjme+24YetR/CPXzk2cQ92Trs4u0eC2zGse/xCVCsQHeZIEZbuanqVZl2jeiXAXq1YuScP3907Gte8sbxeyY2Gvpgy0u2YPrh9GCqqqjHl303PtauqVox49ifkOUseHS0sxx11VnR+eucIHCksc7mIoqS8Ck9fOaB20/X1B05u11VWacfjM7egfUQINji3duIqS3CFEpGnVNmrcd2MlfjlaBHmPTAGp8X67sbjXGVJ5J4qezVmbTyMKwd1rp3Uvv5APnp1iG60iXtr/GPeTmTml+KO0T1w7VsrcKLSXu/10GAbfvnrJY3elzptNgDHsF/qtNmY0L8T1uzLQ25JhdvlNhp6+MuN+E+dmmxmmH7DYFzh5vZ47t6/mJARWdj+3BJc8q8l6BofiUcm9sWY3ok+OXzJhIzItxw+fgIv/LADHduH49zeSejdMdrljiFFZZWoqKpGQnQYquzVsInUzsNq6766GdlFuOClxacU/6l69qozXW7x5Iq79y8WhiWysG4JUXjlhsEoPFGJW99bjateX46FO7NdbtFiNSIyQUR2ikiGiExz8fptIpIjIhucjzvNiJPIDKfFRuDl6wfjkYl9Mbp3YpPbt7ULD6mdxxYcZIPNJggOsrU5GQOAXh3aYe9zE9EzyXN1HVurso3FxpvDhIzI4s7v2xGLHhqHZ686EzlF5bj9/TW48vXl+GDZXqw7kI+yBkMTViAiQQBeA3AJgH4AbhCRfi6afqGqg5yPd7waJJGFiQh+fPA8jOuThOSYcDx0cR+sfOR8XNpMrTVPCg32fPrESf1EhNBgG24c0RWTh6bgq3WZePPn3bWVx4NtgjOS22FgSix6JkWjR2IUUhOjkBIXgZBT+Jbr44YDyFDVPQAgIp8DmASgcTl2IjKFiOD924fXO/fajUMQFrQBX68/BAB48+ah+G/6Qfy04+Tm5K/eOBg7sorw6sKMZq8fbBNUVbseLYg0YKcTJmREVCs02IYbhnfF9cO64EhhGTYeLMDGzOPYlHkc3288jMKyk/vXBdsEHduHIzosGJFhQYgKDUZkaBDahYegfUQw2oeHICYiBO3CgxEabINNBDYRBNkcS8bHndHBlxO6zgAO1nmeCWCEi3bXiMgYAL8A+L2qHnTRBiIyBcAUAOja1b15J0TUNs9e7ZjfdbSwHBf374gJAzrhWHE50v76IwDgsoGn4bKBaJSQLfjDeUhNiMKeYyXIL63A4C6x+HbDYZRW2jF5SArOfWFB7VZRNStQPYkJGRE1IiJIjolAckwEJgxw7AagqsgrqcC+3BLsySnBvtwSZBWUobTcjpKKKkxuZ8MAAAwFSURBVJRW2HGsuBxFZVUoPFGJogabDze0+amLfDkhc7WyoeFX5e8AfKaq5SIyFcCHAMa7upiqzgAwA3BM6vdkoERUX3hIENJS4+udS4wOw09/OA/hIfV7ts7pmYDsonJkZBejR1I0AMd+nTWuGXqysGz6Yxciv6QCHyzfV68ciacwISMit4gIEqLDkBAdhqHd4ltsb69WFJdVobCsEpX2alSrwl7tOF+tishQn779ZAKou3NwCoDDdRuoam6dp28D+LsX4iKiNuqZFF3v+aanLkJ4cBAUiiq7e9+T4qJC8fsLWy5m2xaG3hFFZAKAfwEIAvCOqj7f4PXbALwI4JDz1KucGEsUGIJsgpjIEMREtr32kYnWAOgtIt3huD9dD+DGug1EJFlVs5xPrwCw3bshEtGpqLu3pwEjkK1mWAh1VildCMe3zTUiMktVG06K/UJV7zUqDiKi1lLVKhG5F8A8OL5QvqeqW0XkaQDpqjoLwH0icgWAKgB5AG4zLWAi8ntG5oRcpUREfktV5wCY0+DcE3WOHwHwiLfjIqLAZOSMWlerlFxtmHeNiGwSkS9FpIuL1yEiU0QkXUTSc3JyjIiViIiIyDRGJmTurlJKVdWBAH6EY5VS4zepzlDVNFVNS0pK8nCYREREROYyMiFza5WSqpY7n74NYKiB8RARERH5JCMTstpVSiISCscqpVl1G4hI3T0OuEqJiIiILMmwSf1cpURERETkHkMrb3CVEhEREVHLfHbfEiIiIiKrYEJGREREZDJR9a99bkUkB8D+VrwlEcAxg8IxGmM3B2M3R3Oxd1PVgKh508p7mD/+//THmAH/jJsxe8epxuzW/cvvErLWEpF0VU0zO462YOzmYOzm8OfYjeKPvxN/jBnwz7gZs3d4K2YOWRIRERGZjAkZERERkcmskJDNMDuAU8DYzcHYzeHPsRvFH38n/hgz4J9xM2bv8ErMAT+HjIiIiMjXWaGHjIiIiMinMSEjIiIiMlnAJmQiMkFEdopIhohMMzuelojIeyKSLSJb6pyLF5H5IrLL+d84M2N0RUS6iMhCEdkuIltF5H7neX+IPVxEVovIRmfsf3Ge7y4iq5yxfyEioWbH2hQRCRKR9SLyvfO5P8W+T0Q2i8gGEUl3nvP5Pzfe4sv3sNb8vxOH6c6fY5OIDPFSjG7fU5uLUURudbbfJSK3mhDzUyJyyPm73iAiE+u89ogz5p0icnGd8177s9PafwN84XfdTMzm/q5VNeAecGxmvhtADwChADYC6Gd2XC3EPAbAEABb6px7AcA05/E0AH83O04XcScDGOI8bgfgFwD9/CR2ARDtPA4BsArASAD/AXC98/ybAO4yO9ZmfoYHAXwK4Hvnc3+KfR+AxAbnfP7PjZd+Nz59D2vN/zsAEwHMdf59GwlglZdidPue2lSMAOIB7HH+N855HOflmJ8C8EcXbfs5/1yEAeju/PMS5O0/O639N8AXftfNxGzq7zpQe8iGA8hQ1T2qWgHgcwCTTI6pWaq6GEBeg9OTAHzoPP4QwJVeDcoNqpqlquucx0UAtgPoDP+IXVW12Pk0xPlQAOMBfOk875OxA4CIpAC4FMA7zucCP4m9GT7/58ZL/O4ehqb/300C8JHz79tKALEikmx0MK28pzYV48UA5qtqnqrmA5gPYIKXY27KJOD/t3e3MXJVdRzHvz+2UBpaW0AgxCJbtTwEolWKUEFsDGlEGwLy0EoJBkiURq2QQCOS+EpjjU/4wpBQJRAopQpWSiSlSSkaodCmuF1aFK0piU1riyggoA1d/r44Z/Rmndnd2W7n3Fl+n+Rm7py5c/Z/z9w999xzHw4PRMT+iNgJ7CBtNx3ddkaxDyhe1kPE3EpHynq8NsjeA/yl8n4XQxd2XZ0QEXsgbUDA8YXjGZKkXuDDpJ6mrog9n/LrA/aRKoA/A69ExIG8SJ23nduBpcDb+f2xdE/skBq/6yRtkfSFnNYV200H1L0Oa+e3q9O6tBtjXWL/cj69d1flNH7tYh7hPqBWcQ+KGQqW9XhtkKlJmp/vcQhJmgw8BNwYEa+VjmekImIgImYB00lHO6c3W6yzUQ1P0nxgX0RsqSY3WbR2sVecFxEfAS4CviTpgtIB1Ujdf8t2fru6rwu0jrEOsd8BvB+YBewBvp/TaxVzG/uA2sTdJOaiZT1eG2S7gJMq76cDuwvFcjD2Nrr28+u+wvE0Jelw0ka9IiJ+kZO7IvaGiHgFeIJ0TcM0SRPyR3Xdds4DLpb0Iqmb/JOkHrNuiB2AiNidX/cBq0kN4q7abg6hWtdhbf52dVqXdmMsHntE7M0Hjm8Dy0llzRCxdTzmNvcBtYi7Wcyly3q8Nsg2AzOV7jg7AlgIrCkc02isARp3mnweeLhgLE3l65Z+Cvw+In5Q+agbYj9O0rQ8Pwm4kHQtwQbg8rxYLWOPiFsjYnpE9JK278cjYhFdEDuApKMkTWnMA/OAbXTBdtMhta3DRvHbrQGuyXfXnQu82jiVVUC7MT4GzJN0dD59NS+ndcyg6+0uJZV1I+aFkiZKmgHMBDbR4W1nFPuA4mXdKubiZX0wdyrUeSLdyfFH0jVBt5WOZwTxriR1kb5FanVfT7omaD3wp/x6TOk4m8R9PqmLth/oy9OnuyT2DwK/y7FvA76R09+X/9l2AD8HJpaOdZj1mMv/7rLsithznFvztL3xP9oN200Hy6iWdVi7vx3ptM6P83o8B8zuUJwjrlOHihG4Lv8/7QCuLRDzvTmmftLO/sTK8rflmF8ALiqx7bS7D6hDWQ8Rc9Gy9tBJZmZmZoWN11OWZmZmZl3DDTIzMzOzwtwgMzMzMyvMDTIzMzOzwtwgMzMzMyvMDTLrCEkDkvoq09fGMO9eSduGX9LMbGiSXs+vvZKuGuO8vz7o/VNjmb91Nz/2wjpC0usRMfkQ5d1Leg7XmYcifzN752jUVZLmAjdHxPw2vtsTEQPD5T0Wcdr44x4yK0rSi5K+I2lTnj6Q00+WtD4P8rpe0ntz+gmSVkvamqeP5ax6JC2XtF3SuvzkfSQtkfR8zueBQqtpZt1nGfDx3KN/k6QeSd+VtDnXJ18EkDRX0gZJ95MeKoqkXyoNvL5defB1ScuASTm/FTmt0RunnPc2Sc9JWlDJ+wlJD0r6g6QV+SnzSFpWqdu+1/HSsTHnHjLrCEkD5Moq+3ZErFIai3F5RHxL0jXAlRExX9IjwIMRcY+k64CLI+ISSauAjRFxu6QeYDJwNOnJzrMjok/Sz4A1EXGfpN3AjIjYL2lapDErzcyaatVDlhtWx0fENyVNBJ4ErgBOBn4FnBkRO/Oyx0TE3/OB4WbgExHx8uAessrfugy4AfgU8O78nXOAU0lDDp1BGiPxSeAW4HlgI3BaRITrtvHBPWTWKf+KiFmVaVXls5WV1zl5fg5wf56/lzTUBaRBtO8AiDQI7Ks5fWdE9OX5LUBvnu8HVki6GjgwlitkZu8o80hjMPYBz5CGBpqZP9vUaIxlSyRtBZ4mDT49k6GdD6zMddpe4NfA2ZW8d0Ua8LqPVLe9Bvwb+ImkzwJvHvTaWXFukFkdRIv5Vss0s78yPwBMyPOfIY2bdhawRdKEwV80MxsBAV+pHFTOiIh1+bM3/rtQ6lm7EJgTER8ijZd75AjybuX/6raIOAB8FHgIuARY29aaWC25QWZ1sKDyujHPPwUszPOLgN/m+fXAYkgX0Ep6V6tMJR0GnBQRG4ClwDTSKU4zs+H8E5hSef8YsFjS4QCSTpF0VJPvTQX+ERFvSjoNOLfy2VuN7w/yG2BBrtOOAy4ANrUKTNJkYGpEPArcCMxqZ8WsntxbYJ0yKXf1N6yNiMajLyZKeoZ0gPC5nLYEuEvSLcBLwLU5/avAnZKuJx0tLgb2tPibPcB9kqaSjkB/6OsszGyE+oED+dTj3cCPSKcLn80X1r9E6p0abC1wg6R+4AXSacuGO4F+Sc9GxKJK+mrSZRpbSWcDlkbEX3ODrpkpwMOSjiTVbTeNbhWtTnxRvxWVL+qfHRF/Kx2LmZlZKT5laWZmZlaYe8jMzMzMCnMPmZmZmVlhbpCZmZmZFeYGmZmZmVlhbpCZmZmZFeYGmZmZmVlh/wFiyMXE5hExWAAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{},"cell_type":"markdown","source":"## 7. Make predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_predictions(text_batch_logits):\n\n    text_batch_tokens = F.softmax(text_batch_logits, 2).argmax(2) # [T, batch_size]\n    text_batch_tokens = text_batch_tokens.numpy().T # [batch_size, T]\n\n    text_batch_tokens_new = []\n    for text_tokens in text_batch_tokens:\n        text = [idx2char[idx] for idx in text_tokens]\n        text = \"\".join(text)\n        text_batch_tokens_new.append(text)\n\n    return text_batch_tokens_new","execution_count":31,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results_train = pd.DataFrame(columns=['actual', 'prediction'])\ntrain_loader = DataLoader(trainset, batch_size=16, num_workers=1, shuffle=False)\nwith torch.no_grad():\n    for image_batch, text_batch in tqdm(train_loader, leave=True):\n        text_batch_logits = crnn(image_batch.to(device)) # [T, batch_size, num_classes==num_features]\n        text_batch_pred = decode_predictions(text_batch_logits.cpu())\n        #print(text_batch, text_batch_pred)\n        df = pd.DataFrame(columns=['actual', 'prediction'])\n        df['actual'] = text_batch\n        df['prediction'] = text_batch_pred\n        results_train = pd.concat([results_train, df])\nresults_train = results_train.reset_index(drop=True)","execution_count":32,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=51.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"2d3584a3b1ab4061bc72a0c11fca6bee"}},"metadata":{}},{"output_type":"stream","text":"\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"results_test = pd.DataFrame(columns=['actual', 'prediction'])\ntest_loader = DataLoader(testset, batch_size=16, num_workers=1, shuffle=False)\nwith torch.no_grad():\n    for image_batch, text_batch in tqdm(test_loader, leave=True):\n        text_batch_logits = crnn(image_batch.to(device)) # [T, batch_size, num_classes==num_features]\n        text_batch_pred = decode_predictions(text_batch_logits.cpu())\n        #print(text_batch, text_batch_pred)\n        df = pd.DataFrame(columns=['actual', 'prediction'])\n        df['actual'] = text_batch\n        df['prediction'] = text_batch_pred\n        results_test = pd.concat([results_test, df])\nresults_test = results_test.reset_index(drop=True)","execution_count":33,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=17.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"43caa22ac31b45858536eda827c13ed4"}},"metadata":{}},{"output_type":"stream","text":"\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(results_train.shape)\nresults_train.head()","execution_count":34,"outputs":[{"output_type":"stream","text":"(802, 2)\n","name":"stdout"},{"output_type":"execute_result","execution_count":34,"data":{"text/plain":"  actual  prediction\n0  pcm7f  ppccmm77ff\n1  377xx  337777x-xx\n2  6g45w  66gg4455ww\n3  mc35n  mmcc3355nn\n4  5mfff  55mmfff-ff","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>pcm7f</td>\n      <td>ppccmm77ff</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>377xx</td>\n      <td>337777x-xx</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6g45w</td>\n      <td>66gg4455ww</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>mc35n</td>\n      <td>mmcc3355nn</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5mfff</td>\n      <td>55mmfff-ff</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(results_test.shape)\nresults_test.head()","execution_count":35,"outputs":[{"output_type":"stream","text":"(268, 2)\n","name":"stdout"},{"output_type":"execute_result","execution_count":35,"data":{"text/plain":"  actual  prediction\n0  2fxgd  22ffxxggdd\n1  y5n6d  yy55nn66dd\n2  8gmnx  88ggmmnnxx\n3  wm47f  wwmmmm77ff\n4  dn26n  ddnn2266nn","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2fxgd</td>\n      <td>22ffxxggdd</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>y5n6d</td>\n      <td>yy55nn66dd</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>8gmnx</td>\n      <td>88ggmmnnxx</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>wm47f</td>\n      <td>wwmmmm77ff</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>dn26n</td>\n      <td>ddnn2266nn</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"def remove_duplicates(text):\n    if len(text) > 1:\n        letters = [text[0]] + [letter for idx, letter in enumerate(text[1:], start=1) if text[idx] != text[idx-1]]\n    elif len(text) == 1:\n        letters = [text[0]]\n    else:\n        return \"\"\n    return \"\".join(letters)\n\ndef correct_prediction(word):\n    parts = word.split(\"-\")\n    parts = [remove_duplicates(part) for part in parts]\n    corrected_word = \"\".join(parts)\n    return corrected_word","execution_count":36,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results_train['prediction_corrected'] = results_train['prediction'].apply(correct_prediction)\nresults_train.head()","execution_count":37,"outputs":[{"output_type":"execute_result","execution_count":37,"data":{"text/plain":"  actual  prediction prediction_corrected\n0  pcm7f  ppccmm77ff                pcm7f\n1  377xx  337777x-xx                 37xx\n2  6g45w  66gg4455ww                6g45w\n3  mc35n  mmcc3355nn                mc35n\n4  5mfff  55mmfff-ff                 5mff","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n      <th>prediction_corrected</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>pcm7f</td>\n      <td>ppccmm77ff</td>\n      <td>pcm7f</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>377xx</td>\n      <td>337777x-xx</td>\n      <td>37xx</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6g45w</td>\n      <td>66gg4455ww</td>\n      <td>6g45w</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>mc35n</td>\n      <td>mmcc3355nn</td>\n      <td>mc35n</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5mfff</td>\n      <td>55mmfff-ff</td>\n      <td>5mff</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"results_test['prediction_corrected'] = results_test['prediction'].apply(correct_prediction)\nresults_test.head()","execution_count":38,"outputs":[{"output_type":"execute_result","execution_count":38,"data":{"text/plain":"  actual  prediction prediction_corrected\n0  2fxgd  22ffxxggdd                2fxgd\n1  y5n6d  yy55nn66dd                y5n6d\n2  8gmnx  88ggmmnnxx                8gmnx\n3  wm47f  wwmmmm77ff                 wm7f\n4  dn26n  ddnn2266nn                dn26n","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n      <th>prediction_corrected</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2fxgd</td>\n      <td>22ffxxggdd</td>\n      <td>2fxgd</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>y5n6d</td>\n      <td>yy55nn66dd</td>\n      <td>y5n6d</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>8gmnx</td>\n      <td>88ggmmnnxx</td>\n      <td>8gmnx</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>wm47f</td>\n      <td>wwmmmm77ff</td>\n      <td>wm7f</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>dn26n</td>\n      <td>ddnn2266nn</td>\n      <td>dn26n</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 8. Evaluate the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"mistakes_df = results_test[results_test['actual'] != results_test['prediction_corrected']]\nmistakes_df","execution_count":39,"outputs":[{"output_type":"execute_result","execution_count":39,"data":{"text/plain":"    actual  prediction prediction_corrected\n3    wm47f  wwmmmm77ff                 wm7f\n42   pmg55  ppmmgg5555                 pmg5\n64   mddgb  mmddddggbb                 mdgb\n74   mmy5n  mmmmyy55nn                 my5n\n77   6n6gg  66nn66gggg                 6n6g\n78   664nf  666644nnff                 64nf\n80   x775w  xx777755ww                 x75w\n81   55y2m  5555yy22mm                 5y2m\n88   f2fge  ff22ggggee                 f2ge\n92   d666m  dd6---66mm                 d66m\n94   5p3mm  55pp33mmmm                 5p3m\n107  f22bn  ff2222bbnn                 f2bn\n115  7nnnx  77n---nnxx                 7nnx\n116  b5nmm  bb55nnmmmm                 b5nm\n121  y32yy  yy3322yyyy                 y32y\n137  8cccc  88c------c                  8cc\n149  ncyx8  nncccxxx88                 ncx8\n165  ddcdd  d-ddccdddd                 ddcd\n176  ddxpp  d-ddxxpppp                 ddxp\n178  6dd2y  66dddd22yy                 6d2y\n180  78eec  7788eeeecc                 78ec\n183  gxxpf  ggxxxxppff                 gxpf\n190  n6nn2  nn66nnnn22                 n6n2\n196  mxnw4  mmxxmmww44                mxmw4\n217  nnn57  n---nn5577                 nn57\n227  nnfx3  nnnnffxx33                 nfx3\n229  5nggg  55nng----g                 5ngg\n233  x4gg5  xx44gggg55                 x4g5\n254  wgnwp  wwggmmwwpp                wgmwp\n255  25w53  2255555533                  253\n256  22d5n  2222dd55nn                 2d5n\n260  mm3nn  mmmm33n-nn                 m3nn\n267  n4xx5  nn44xxxx55                 n4x5","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n      <th>prediction_corrected</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>wm47f</td>\n      <td>wwmmmm77ff</td>\n      <td>wm7f</td>\n    </tr>\n    <tr>\n      <th>42</th>\n      <td>pmg55</td>\n      <td>ppmmgg5555</td>\n      <td>pmg5</td>\n    </tr>\n    <tr>\n      <th>64</th>\n      <td>mddgb</td>\n      <td>mmddddggbb</td>\n      <td>mdgb</td>\n    </tr>\n    <tr>\n      <th>74</th>\n      <td>mmy5n</td>\n      <td>mmmmyy55nn</td>\n      <td>my5n</td>\n    </tr>\n    <tr>\n      <th>77</th>\n      <td>6n6gg</td>\n      <td>66nn66gggg</td>\n      <td>6n6g</td>\n    </tr>\n    <tr>\n      <th>78</th>\n      <td>664nf</td>\n      <td>666644nnff</td>\n      <td>64nf</td>\n    </tr>\n    <tr>\n      <th>80</th>\n      <td>x775w</td>\n      <td>xx777755ww</td>\n      <td>x75w</td>\n    </tr>\n    <tr>\n      <th>81</th>\n      <td>55y2m</td>\n      <td>5555yy22mm</td>\n      <td>5y2m</td>\n    </tr>\n    <tr>\n      <th>88</th>\n      <td>f2fge</td>\n      <td>ff22ggggee</td>\n      <td>f2ge</td>\n    </tr>\n    <tr>\n      <th>92</th>\n      <td>d666m</td>\n      <td>dd6---66mm</td>\n      <td>d66m</td>\n    </tr>\n    <tr>\n      <th>94</th>\n      <td>5p3mm</td>\n      <td>55pp33mmmm</td>\n      <td>5p3m</td>\n    </tr>\n    <tr>\n      <th>107</th>\n      <td>f22bn</td>\n      <td>ff2222bbnn</td>\n      <td>f2bn</td>\n    </tr>\n    <tr>\n      <th>115</th>\n      <td>7nnnx</td>\n      <td>77n---nnxx</td>\n      <td>7nnx</td>\n    </tr>\n    <tr>\n      <th>116</th>\n      <td>b5nmm</td>\n      <td>bb55nnmmmm</td>\n      <td>b5nm</td>\n    </tr>\n    <tr>\n      <th>121</th>\n      <td>y32yy</td>\n      <td>yy3322yyyy</td>\n      <td>y32y</td>\n    </tr>\n    <tr>\n      <th>137</th>\n      <td>8cccc</td>\n      <td>88c------c</td>\n      <td>8cc</td>\n    </tr>\n    <tr>\n      <th>149</th>\n      <td>ncyx8</td>\n      <td>nncccxxx88</td>\n      <td>ncx8</td>\n    </tr>\n    <tr>\n      <th>165</th>\n      <td>ddcdd</td>\n      <td>d-ddccdddd</td>\n      <td>ddcd</td>\n    </tr>\n    <tr>\n      <th>176</th>\n      <td>ddxpp</td>\n      <td>d-ddxxpppp</td>\n      <td>ddxp</td>\n    </tr>\n    <tr>\n      <th>178</th>\n      <td>6dd2y</td>\n      <td>66dddd22yy</td>\n      <td>6d2y</td>\n    </tr>\n    <tr>\n      <th>180</th>\n      <td>78eec</td>\n      <td>7788eeeecc</td>\n      <td>78ec</td>\n    </tr>\n    <tr>\n      <th>183</th>\n      <td>gxxpf</td>\n      <td>ggxxxxppff</td>\n      <td>gxpf</td>\n    </tr>\n    <tr>\n      <th>190</th>\n      <td>n6nn2</td>\n      <td>nn66nnnn22</td>\n      <td>n6n2</td>\n    </tr>\n    <tr>\n      <th>196</th>\n      <td>mxnw4</td>\n      <td>mmxxmmww44</td>\n      <td>mxmw4</td>\n    </tr>\n    <tr>\n      <th>217</th>\n      <td>nnn57</td>\n      <td>n---nn5577</td>\n      <td>nn57</td>\n    </tr>\n    <tr>\n      <th>227</th>\n      <td>nnfx3</td>\n      <td>nnnnffxx33</td>\n      <td>nfx3</td>\n    </tr>\n    <tr>\n      <th>229</th>\n      <td>5nggg</td>\n      <td>55nng----g</td>\n      <td>5ngg</td>\n    </tr>\n    <tr>\n      <th>233</th>\n      <td>x4gg5</td>\n      <td>xx44gggg55</td>\n      <td>x4g5</td>\n    </tr>\n    <tr>\n      <th>254</th>\n      <td>wgnwp</td>\n      <td>wwggmmwwpp</td>\n      <td>wgmwp</td>\n    </tr>\n    <tr>\n      <th>255</th>\n      <td>25w53</td>\n      <td>2255555533</td>\n      <td>253</td>\n    </tr>\n    <tr>\n      <th>256</th>\n      <td>22d5n</td>\n      <td>2222dd55nn</td>\n      <td>2d5n</td>\n    </tr>\n    <tr>\n      <th>260</th>\n      <td>mm3nn</td>\n      <td>mmmm33n-nn</td>\n      <td>m3nn</td>\n    </tr>\n    <tr>\n      <th>267</th>\n      <td>n4xx5</td>\n      <td>nn44xxxx55</td>\n      <td>n4x5</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(mistakes_df['prediction_corrected'].str.len().value_counts())","execution_count":40,"outputs":[{"output_type":"stream","text":"4    29\n5     2\n3     2\nName: prediction_corrected, dtype: int64\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = mistakes_df['prediction_corrected'].str.len() == 5\nmistakes_df[mask]","execution_count":41,"outputs":[{"output_type":"execute_result","execution_count":41,"data":{"text/plain":"    actual  prediction prediction_corrected\n196  mxnw4  mmxxmmww44                mxmw4\n254  wgnwp  wwggmmwwpp                wgmwp","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>actual</th>\n      <th>prediction</th>\n      <th>prediction_corrected</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>196</th>\n      <td>mxnw4</td>\n      <td>mmxxmmww44</td>\n      <td>mxmw4</td>\n    </tr>\n    <tr>\n      <th>254</th>\n      <td>wgnwp</td>\n      <td>wwggmmwwpp</td>\n      <td>wgmwp</td>\n    </tr>\n  </tbody>\n</table>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"mistake_image_fp = os.path.join(data_path, mistakes_df[mask]['actual'].values[0] + \".png\")\nprint(mistake_image_fp)\nmistake_image = Image.open(mistake_image_fp)\nplt.imshow(mistake_image)\nplt.show()","execution_count":42,"outputs":[{"output_type":"stream","text":"/kaggle/input/captcha-version-2-images/samples/mxnw4.png\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAB2CAYAAADY3GjsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJztfXmYXFWd9ntq6ep9SzqdTjohmyGbhEBIwMiisgoExAWQ8QME1EeZB2fcUEdndBhkPh/103EjDmpQlE0hBE1CWBNWTQIhQEjIRmfvTu9bdVdXn++P2++vfnVvJekkne5Ke97nyVOdqrv8znLPfX/rMdZaODg4ODic+AgNtQAODg4ODgMDt6A7ODg4DBO4Bd3BwcFhmMAt6A4ODg7DBG5Bd3BwcBgmcAu6g4ODwzCBW9AdHBwchgmOaUE3xlxsjNlkjNlijLl9oIRycHBwcDhymKNNLDLGhAFsBnABgF0A/g7gWmvtWwMnnoODg4NDfxE5hnPnAdhird0GAMaY+wFcAeCgC3pJSYkdPXp02nfGmGMQ4egxmPd19xpe9x7s+7t7nRj3OZ73Wrdu3QFrbcXhjjuWBX0sgJ3q/7sAzPcfZIz5DIDPAEBlZSUWLVrk/z3j34eC/7ijOS/TOf35brDu9Y/UFye6fP2VZyDu5frixJFvIO8VjUbf7c/9j8WGnqmFAfuNtXaRtXautXZuSUnJMdzOwcHBweFQOJYFfReAcer/1QD2HJs4Dg4ODg5Hi2NZ0P8O4D3GmInGmBwA1wB4bGDEcnBwcHA4Uhy1Dd1a22OMuRXACgBhAL+21r45YJI5ODg4OBwRjsUpCmvtXwH8dYBkcXBwcHA4BrhMUQcHB4dhAregOzg4OAwTuAXdwcHBYZjALegODg4OwwRuQXdwcHAYJnALuoODg8MwwTGFLQ4GhrpOxZFe93jeKxaLAQCSySQAoKenB729vQCAUCj4btaVNP0y8Dz9Wzgclmvxd17DWhs4zhiT9ju/o3xEJBKRc/zHayQSCeTm5qa1taurS47lNXp6ehCJBKeu/9q9vb2B++Tk5Ejb9Cfbxn7UfeFvD9upj/dfT1/Dfx7P5aeWM9M4ZtO87e7uBgAUFhbKMe3t7WnHFBQUSDv4W35+PpqbmwEAI0eOBAC0tLQExjEUCmWcG/2V73DnHMl5R3rd43mv/sIxdAcHB4dhgqxn6A4pJBKJtP+HQqEA29OMgEwnmUwKWyQjCofDab8DHvPlffzXC4fDwrr4aa2Vc/mZn5+PaDQauL9fdv17cXFxoH2dnZ3ymZOTkyZ7KBSSY7UmQbn0cQSPSyQSGVl4pj5jn/iPC4VCASbd09Mj/cfP4uLijBrJoTSJTAw9m0B2XV9fD8Bj7GTrHPf29nZpT0FBAQCv39mPZO2hUEg0MiKRSIgWwHF36D+ye/Y4ODg4OPQbjqGfQCDz04yxP7bxaDQqjJPMV9uNeT0yrJycHGFHZMLd3d3CnAjN2nmNjo4OOZe/aZl4j1AoJO3p6OgA4DFWPxvPzc2VvylvIpFAXl5e2rUztccYk3Zv3iMTaz6YpmOtFXs+5e3u7pb+5HE5OTkBptrU1BS4vmb3Wmvi77xHtmL//v0AIP1fXFws7aHs2k/R1dUFwOsTamK0pUciEZlfeizY30e7m9o/MtyCfgIhk4PFbyLQzkt+xuPxwKIdjUbld57LBTuRSARMAOFwWM7VC6w2GwDew8zjtDnGb/Kx1soCwMWhublZFgA+/LFYLOCg1QuBbqt2Mvq/44sik6kpmUwGFg99Xb0A8Vp+U0Fvby/i8TiA1EuzoKAgo3mF8vH++mWU7fA7rXt6etDa2gog1e8FBQXS35xT7e3tcq42jfnnXk5Ojlw7k5nO4dBwJhcHBweHYQLH0E8g+B1mfnMCj/GH1OXl5QWcl9o56GexkUgkECKZTCYDzDKT+UDLyOPa29vR0NAAANi9ezcAYPPmzdi+fTsA4K23vG1ou7q6MG3aNADAxRdfDACYNWsWioqKpB38JLsna45EImmOT/aPdgIDnrZyqHCzTJ86RA/w2Dbv4XfO6j5oaWkJOJJ5vr5eOBzOGOKYjcjPzwcANDY2AvDaXV5envZbR0eHOD7Z1mg0KiycTP1g7aaGkyk01eHQcAzdwcHBYZjAvQJPQGRy5ulQPb/dNlNIHZBirX6WZK1FS0sLgBRr144uOvtqamqwbds2ACln2e7du7Fv3z4AEAa+f//+gLMvHA6L7Nxrtr6+Xs559tlnAXi2WtrVKUthYSFOOukkAClb+6hRo1BR4W2KPmLECABARUUFRo0alXbcmDFjAu3W7fcnB2nNRGsj/Fs7gLV/AvCYql9D0KGe2tbP62VKYsomcOwZjhgKhdDW1gYAooVFo9GAYzw/Px+1tbUAIGPS0tKSxtb5SYbucORwDN3BwcFhmMAx9BMI/igOIMgydbQFP/Pz8+V32qHD4bDYNJkksnfvXgAe09q6dSuAFPPetm0bduzYIb8DXmSCtmHre2rk5+eLTZ4hipqJ8np5eXmYMmUKAGDy5MnSHspQV1cHwIuGefnllwEcPrTN/7tOZqFmUFFRgaqqKgDA6NGjAaRYfn5+PiZNmgQAKCsrk2P4u/Y1UJPQzNvP2oGUlqT7wJ8Ula3whxSuWbMGjz76KABgzx5vj/iZM2eKL0SHszJa6JZbbgHgtd/v29Hs3oUtHjmye/Y4pCFTjDTBvxOJhISR8fOll16SRZEmje3bt4tphGq0VnUzxZBnkieTKYfyMatw8uTJGD9+PIDUgjllyhRZtLk45uXliWmEaG9vl0WBTrdkMilt4yJRV1cnLyY67Orr69NeAoBnEqLDjsdv3rwZr7/+esY2RiKRjKGHbCNlqqioQHV1dVq7y8vLUVpamtbuyspKMQ3RmVhWViYmjGwHX9h///vfAQD3338//vpXbxdKml7+8pe/yPF8yQHA2LFjAQCf+MQnAHhmML5c+VJPJBJpL0SHI4MzuTg4ODgMEwwqQ9eJHvq7TH8fK+LxuLz9qe42NDSgsrISQMqZ9e677wLw2BKZG1lVKBQShqGdjjRVaJWZqqhWwekEojpPVqmvl0wmRfXXbJds+YUXXgAAvPLKK7jyyisBpEL/amtrsXnzZgApBlpbW4tNmzaJrEBmE82hKjHqc9iH8XhcwgdpFjn55JMxc+ZMAJB+raiowHvf+14AKaZcVFQkDJRaQXFxsYwB79/W1ib9rbURsmD2e1dXl4QSsj2TJk0KONNyc3OlX2hq6unpkX7hNVpbW7Fr1y4AKWffxo0bpT9piuL9a2pqZGz5W0FBAV599VUAqVolkUhExpT3zM3NFbMTEQ6Hha2zv6urq6VPOX/a29ulPRdddBEA4Oqrr5Z2s42RSEQYL7WFhoYGkUEn7nAM9Hdspz90ta6uTubbT3/6UwDe/NQsHEhn5ZznPT09Mm90CCnnmb4XzzmUdngw/KNXZ3UM3cHBwWGYwAym42HatGl20aJF6QIcp/rGI0aMkJA62i+NMThw4EDaObTfdnV1CTsiw9IMRtcR4Xdk1m1tbZJgkqlWCZlTR0dHoLJgIpEQRvLGG28AAP785z/jiSeekGsDHsvlOdrxqOuG+0H5NCPUJQKoibznPe8BAMyYMQOAZ+skO2R44EknnSSMlkyvs7NTrk0WnUgk5DiddJOpiiDP5fEdHR2iBZCdRSIRaa9OfOJ46Pv7mW9hYaH0C+9RV1cn9yAb37lzp9i/OT84Tg0NDfLdI488AgC48847xTavbcCUiffatGmT2Ol5/O7du+V6lHfXrl3YuXNn2nddXV3Cxgkdwsk5E41GxSZ9/vnnAwDe9773BUotGGNEVl3WgdfTNec1qwZS8/KFF17A0qVLAQCvvfaaHKPr1PMamcIvL7zwQgDA7373OznOX5lTJ4NlYuiHWy+GmqH3515Hw9Ajkchaa+3cw91/2DpFm5qapGPo9MvJyRHVlg8VoTdm4MTMzc2VRZGTq6OjQxYqOhqLi4sDzsF4PC6qMs/t7u6W+G6qtU8++aQ4kdauXSuy8CVAtLa2Zix6pQsiAZ7pY+rUqQBSL5dp06Zh3LhxAFLRIxUVFfKAZyo3y3NpJgqFQrI46ZouXBDYr9FoVM6hul9TUyPt5f3j8bj0LR/mWCyW9uIEvJchx4/y6axDDW3G4j1ojuD1ioqK5H40/+Tm5srfXETvuusuAMAZZ5yB973vfQCAiRMnAgAWLVqEyy67DEAqy/Wcc86Ra5AQTJ48GdOnTweQXr6X/aczUPni1i8qmqy42Dc2NuK5554DAIk4euONN7B48WIAqYWytLRUzF4LFiwAAFx++eUiP2Xp7OwUBy3nz759+6S/KRPNfn/4wx/SirsB6S8Zore3Ny2qh+32m8l0/H+mss8OR47DmlyMMeOMMc8YYzYaY940xtzW9325MWalMeadvs+y4y+ug4ODg8PB0B+G3gPgS9badcaYIgBrjTErAdwA4Clr7V3GmNsB3A7ga8dP1CNDR0eHsA+yCe0opYmATLihoUEcSYw3LioqErZCZl1UVCSmCrK+7du3C/ug02jFihWiYs6ZMwcAsGrVKtDkRKdaIpHIqJ76TRThcFgY7yWXXALAc0oyRppmBABiPqA5prW1Vf7W6rGuyaJ/AxCILx8xYoQwUGom+/btw/r16wGkHHennHKKXJes7/XXXxfmS/ZeX18vmgTHoKKiIsAAe3t7Zcx43a6urkBom1b9tQbh12qMMeJU5pgVFBSICeFvf/sbAAjDfe2117BkyRIAHlsnXnrpJQApp/qKFSvw/ve/HwBEG6qurhZZdB0TtoNzID8/X+YjzUtFRUUyfzmekUhEZKBJrK2tTWR+/PHHRRbKR0b/4x//GCeffDIAyOell14qf1NzstaKQ5jXIEN/++23A472kpISMV1Ro9AmF51lS41M1xry17Vx7PzYcFiGbq3da61d1/d3K4CNAMYCuALA4r7DFgO48ngJ6eDg4OBweByRDd0YMwHAHACvAKi01u4FvEXfGDPqIOd8BsBngBSrGAzk5eUFamsDKdujnxE9/PDDuP/++wF4LBPwwvNoD92yZQsAz7nlT2rJtBkxkLKJMyQOSDnidK0QMlCyyc7OTrFf8vizzjoLN954o/wNeL4BMibWx9i3b5/IxTZaa4Uh6nBAv+2cx8Tj8TQnJ+DZcR988EEAKefgvn37RJv55Cc/Kddlf/O6zc3NwooZ5vfaa6/htNNOA5BKOFm4cGGgbkoymZQ+4HiGQqG0rc0ATyPzbyat66vQ1/DSSy/JPSj79773PbGFU6siyx45cqRoJnRUjxo1Ck8++SSAlEN5yZIl0kaO57e+9a1Abe+8vLxAEpGuX6L9OGwvx6WsrAwTJkwAkJorW7dulf772Mc+BsBL3KFPg0k/Tz/9NDZs2AAA8vnoo4/KOJP5z549W7SOZ555BkBKkygpKZHnhdrVnDlzUFNTAwB48cUXRV6/hqnnhWbo7Jdsz5A9UdDvXjTGFAL4E4AvWmtb+uv1tdYuArAI8KJcjkbIo4FeZBlhEIvFMGbMGACpB4ITfvHixRJhQDWVn0C6Gs+HlE6mrq4ueSA54dva2sRcw4nc0tIix+n+owmHx7W3t4ssNPWwWBWQvrclMz+5sEybNi2j+q4ds/zk4sH2cCHSZgF+bt++XbIpeU8AOPfccwGkIizGjx8v1+NDOn78eHGorlmzBgDw/PPPS3kBmpImTpwof3MBHjlyZCDaIRQKyaKjF2x/GVfdV3xBzpw5E++88w6A1AJ91VVXicmI/cxFbMqUKXINzqO1a9eKmYgZkwAk3Z3HPfHEE/KC4HzLVBohGo3KS5h9V15enuZgZxvYp+vWrQMA/OIXv5CXD+O8586dK87YWbNmAQD+5V/+RV44fEF95zvfkb5Yvnw5AG8Rz1SaAPDMZYyEuuKKKwB4Tlc+QzTRsE1AuumMJi4NZ2IZWPQrDt0YE4W3mN9nrf1z39f7jTFVfb9XAag9PiI6ODg4OPQHh2XoxqOS9wDYaK39ofrpMQDXA7ir73PJcZHwKBEKhcRRSEYSCoWEMdCE8sADDwBIj9W+/PLLAQAf+MAHhHVpRs/wMF7jYCAr1CB75G/jxo2TrD8yu2XLlgmT1jHBVPPnzvXCUU899VSsXLkSAHD77bcD8Fji6aefDiDFfCsrKyWenE5UvSM9VXu9tycdWGTjDzzwQJrGAnimqTPPPBNAKp7fGCNmKrK0cePGYfXq1QAgTtRkMimmKH6uXr1acgbIkM877zz5XcfVP//88wBSzuCcnBzJWqW2UlhYKP1NM9mDDz6Ip59+Oq0vfv/734vMp556KgCIOWj58uWiTZE9A5mZNhno22+/DcBzkNM8RvNOOBwWBqxj5Kmd8bqJRCIQ4vrcc8+J1kdz3ksvvSSyz58/H4Cnhfi1mlgsJv1C0+fq1avx2GOPAYCEPj7zzDMil3/LuI6ODpkPLI4WDodlrKi1JJPJAMvXcfBatqPJBnU4OPpjclkA4FMANhhjXuv77hvwFvIHjTE3AagB8PHjI6KDg4ODQ39w2AXdWvs8gIMZzD80sOIMHKy1YrelE6qnpwc///nPAUDswWSM+fn5wjY/97nPAfAyJxnORna8ZMkSYdJkJLqGiM4U9TtlW1paAs7OnTt3SliYP8QNyJw1xjC12bNn45xzzgGQyuZ7++23hSFqdk8b90c/+lEAHgOlDDyODDQajUpook5goQOLWst5550nTFaHRbKN7J/x48cLi9ShoQxzIzZu3CgMkKFzEydOFHsxHb979uyRcSPDa2hokPA++jZKS0tFS6GN+sUXX5TxpdbQ2Ngofgzeg+OuQx+pweTl5cmY61o3v/3tbwGkWP7UqVPlXDLr0tLSQJhffX29MFn+1tbWJvOA7YrH41IvhvbqlpYWmedEQUGBtJf3b29vl/nFefnqq69ixYoVAFJzKplMppVYBlKMfsyYMTI+fB5WrVol982U7MVr6BotGs6GPrDIetfysaTy+neV2bNnj0Sy+HeNt9biy1/+MoBUZt7dd98tE5YPJIBAqjOQXvyJx1DNznQuF7hwOCwPCRfKyy+/XCIMuDjq2F5t6mGMOz+XL18uqr8+l9fjwzl27NiAg1i3h5EQVMV37twpD+QHP/hBAF48PL/TpiQuinzAGxoaAoWzGhsb5b7EihUrxNnGF9WKFStk4aMjtLm5WcaW6n5DQ4OYVRixcsopp8g4sz1XXXWVOAd53H/+53+Kg5YLJfvGWivHs62dnZ2BOH0glYPw4Q9/GIDnkOQc4DWeeuopKZ7Gl+aIESPEJMZFd/r06RIbznGcPHmyZBXzejoOnH2ii8DpHAPOPb6s1q1bJ+3Ve4TyHBKc66+/HoAXm0+zDefTqlWr5JmiTEDqxcR5XlxcLCSKcyE/Pz9Qu7+yslLk59gVFhZKP+vibewrPhdtbW0yDzOV4MhUlleXHPCbmHTggD8iTJdQ1jjaMgMDBVecy8HBwWGYIOsZ+tFC7wJPprhhwwZ56xNkErNnz8YNN9wAAIFwRyDFwI0xaY4rwp8d+d73vlccYgxt02Fd2hxC1nz11VcD8EwGrDVDc0NXV5ecQxW7trZWHGHnnXceAK8gFBl/psJdZKWrV6+WGiXMiiSTqa+vFzMQHZKxWEwckGTg1dXVafuWAp4JggxMm5x0sS0gvQwxEY1GxfE6e/ZsAB5zpEbCa+g9UjVj8oda7tixI1CQ6oUXXhBTC8P9rr32WmHLZHjsQ5qv9L0KCwuFNfMzFotJ2+i8rampEbbJDT5efvlliePXYaXsM2oG119/vYRh8jdd64bt0WGVWlvSBeZ4Llk4zVWPPPKItJPsfeLEiRLyeMEFFwBImZCKi4tFFmpQp59+ulz397//PQDvefMz05aWFvzxj3+U+wLAvHnzcNVVVwFIlQPWmpuO12c7OLa6zC/HrLKyUsxiunaOXxuPRqOBMtL5+flybZ2zwWv7czV0Mb9sgmPoDg4ODsMEw5ahNzY2ikOMYYv19fXCLumc0xXlDlVPgr91d3dn3PyBb26y7fnz50vmHm3ozz//fGB397y8PGFTDKObPn26JISQ1YfDYZGLxzc3N4vDlXVE3nrrLdnbMZP9jvbbnp4eYUBkqnSkvfXWW+Jk5b1mzZolrJmMvrS0NMDGc3NzhXny/ppp6dKtfkQiEekraibl5eVyTqZQQV2a2B+iuGfPHtEqyMgmTJggfXbzzTcD8MoVcyxpw//BD34AwOtXlmEmW9f2aF1fxt/epqYm0YhoI08mk4GyuLr/mOjT1NQkfc/7V1RUBCpKlpaWBuZrT09PoCztgQMHxPFJ7WvLli3CXukUnj59uvgAqGFSu4jFYmn71AJIY6ma2frHqqysTGzynIOrV69Oc6ryOGoGzD6eNm1aIEM2Ho8HKnPW1dWJXLp8sL8EdqatExOJRKAyp9bydRliwBvjTDWYhhrDdkEfPXq0DOqf/+zlQt13332BieZ/MDVKS0slppvRHDU1NeKYyhRnTqfR7Nmz5SHRJg9/caPOzk5ZYL773e8CAH71q1+JyUWXIfWbi/R3VPPPP/98WXj4AEciEWkfVdJ33nlHIiZ4f74I3nzzTUnRJ2bNmoVLL70UQMp529HRkWYOoDz+jZEbGhrS9gPlp1aB2RcsFsWHfvbs2fKy4sI2fvx4icPmy9oYIws5r6f3VyV6enrkevPmzQPgmRkYR//mm28CSM2L5ubmtPhzypkpYoOLA/t15syZIjNfkNFoVO5F05musc9rFBYWisP3lVdeAeA5o/1lA+LxuLyQKWdjY2NaXgDgvSAfeughACmzmy6hwCzTD33oQ5Ld6t/vs6ysTOY8XzZFRUUB04OOted8b2xsxEc+8hEAqeJyLS0tsqAzU3XVqlX405/+BABSaqKsrEyc/hdffDEAzxzEeagXWz7f2rSn9wDgGHCOkBBEo9HAom2MCexBkCknIZvgTC4ODg4OwwTDlqHX1tbKW5jOJV0+l29cvr3z8/PFHENVc+HChcIYqSYvXrw4zREGpJeiZQz05MmThR1qtkJmrhmrrpcCAF/96lcDpX8zaRA6rIpMYtasWRJWSGdiU1NTQC1PJBLC5JmRSAfogQMHxPRB5n/qqacKIyKr0cXB2NfxeDxQx2PDhg1i/mLNlL179wor1RoMmSVrxCxYsEAYNR2XEydOxL333gsgZZrJz88XJqvNOQzhpBOvoqJCfuf1qqqqRCNiSB81s5qaGrmX1uZ0iCf7hG2jTLNnz5YNJjhnWltbA87g3t7egOPuwIEDEkrJcWlqagqUCG5ubhbzHM0SyWQyzSkIeJoH5dK7dunNQABvTnFusu90WxkTzzkTj8cDfZGbmyvt1WWL2d86vv2f/umfAAAf//jHpT3UIJjF+tJLL0l2Lz/z8vJEM2GOxUUXXSTPLueWrkvEfrfWBp6HxsZGkYvPdU5OTmD3JO2Y1+WmswWOoTs4ODgMEwxbhq4TFOhk+f73vy9MhAyHYWIf+9jHhCUQRUVFwpj4lidjBVLMMicnR37nW7ukpETuT9aZk5MTCHlMJBK45pprAKRs2CtXrgzse9nS0hIIQ6yrq0uzFwMeU6VtmFmey5YtCzjiwuGw2Mn5SdZljBGGw9DGefPmBTJjw+GwfKdrz/irBG7atElkZ5nWxx57LKB1TJo0SY5buHAhAC9blw5FMjLtYKPs+/btk7HSjJ/9R2Z52223SbIPj6+trZUx+sAHPgAgxQR1lUu2VftOtLOcc4kMd+3atbjuuusApJydGzdulHnBuZKTkyMykwFu375drsP2NzY2SnIV+2737t0iA/03nZ2dMgZk9y0tLaKp6nLOPJfyfeUrX8FXvvIVAJDwRTr3r7nmGulvluwtKiqSviO0n4pyGGPE36G3dWRfkDFXVVWJnNRurLUyl1kh8/HHHxdtitnfP//5z2WOsN7RRRddJHOYGkx+fr7MV/qU9DaSHAudXcux10EN2ZjlOmwX9La2tkCp2mg0KgsfFywW4rr22mtlkPmwtLe3Bx60SCQig6ojLAidccd70NygJ7p2cPK4f/u3fwPgqcec/LxvJkcTnWZAesQNHzY6MWtrayX2Wm/czIXJXyApEolIxA0fjLFjx8q52rnERUln/7HPtOmFMf0soPX4449Le3jczJkzpTgXx2LSpEkiH4+rq6uTh51Ox/Xr10u8NL+rqqqSxYN99stf/hL//M//DCD1AhszZoyMAUsOMFrpjDPOEOdlpo24tdmIY8aF8KGHHpJx4Vi99dZb8rLWzjo9zoCXdUqzG00KW7duFecqx6Czs1PGRdeBp6mAC3UikZCxZMZmKBSSftFzk/3Ml8edd94JwMuopfnryiu9/WwWLFggL0h/diiQMl9oM4eeg3w2aa5qbW0N5DZUVFTIIk+T5s033yzjwUJtS5cuxcMPPwzAq/UOeDHvlIHRWZdccok4Zjkfm5ubpR0cs66ursCaoNuQjTXcncnFwcHBYZgg+14xA4RQKCQqK2O5NSsmi6DTU9eBIGvQtUbIWnTWo2YifqaaTCaFGZDJ6791bLVfjf3ud78rai8Zv46dZThkR0dHoABYa2urMAw6Mc866yxhaqxTox1Z/l3Yw+GwsEOyrng8nuZAZhv5nWbj7D+y4/LyctnNhjvdaHVVM0wyJoaGXnvttYGdexoaGiSskTHSp5xyimTLspRwQ0ODOEPJTqdOnSpheDy3t7dXvuM1qNFs3bo1bZzZJ7oWD0EnNPsYSMV8c0yqqqrEcUcnpXYQsz/feecdcdLTFNjU1CTOajLWgoICmY/UJHp7e0XToSMyHo+LSYbj09vbGwjj1bVP/GGysVhM2DA/Q6GQzEf2SSwWC5gvgNTmJqyZ9Pbbb4tMd911FwBvXtKEw3bV1dVJv9BRq00kZNsXX3wxfvWrX6X17ZIlS7Bs2TIAKYf3mjVr8P3vfx9A6hn/6Ec/KuGaDCoYMWJEYM9T/2e2wTF0BwcHh2GCQWfoh6pGNpBvPV0rhIwpFosF7KDabka2xfN0+Bvt5GVlZYFkJB3+RbusTrDRjlR/5Tcg5WQju58zZ04g6UY7EHlcU1OTMH6Vz4l9AAAaoUlEQVRtg/V/98EPflC0gPvuu09kZl/4a6BUVVWJzZflVRcsWCAVC5nkUVVVJc5W7U9gn5IxnnrqqVKJkNpSQUFBoMxuW1tbwPEbj8elv8lYN2zYILLQyVpbWytV/3QlRNqByQ7nzp0bsKsnEom0pBwgxYDHjRsnzI4sWzP2TI4x+kzOOussaS/PHTt2rIwfnZ3btm2TPqBm1NzcLA5D1gJaunSp2N9ZiXHkyJES3shz6+rqhHkzpHHdunVyPR5XX18fyKJMJBIBn4rOhvUnxhlj0nxIPN4/b/Py8iTRjWPX2dmZFiIIePPC/4zqJB72d0dHR9repATPpT9l/vz5uOOOOwBA9j595plnxMbO8fnNb36D3/zmN2nXmzt3rtR3ouZG30p5eXnGcsFHW21xoNbBYWty6e7uFlWQKhknMpCKKNGxu/50dP2w8rfS0tKAA0s/AFwQQqFQwPTA+/jBc7QDlqYHquz6HpxI9fX18jDxgbDWBpyN5eXl4iBl7PmuXbtkghM8fu/evWllSgEv+5HORpox4vF4mkkG8KIeWEaVqnNHR4cUEaNK3NXVJYsTF5pnn31WHIo8XjvTeP+5c+fKS4MxyyNHjpQICB1pxAWfL4Onn35aojbY78lkUuYGz+UCrM0n/nhrIHP2Lh3ADz/8sIw9X27xeBwf+pC3jQCjkV599VUxmVHdX758ubwEmRXb0dERiFfXm1jTHFNYWCjEgiaNG264QdrETdABBMoLrF27VhZcXZwOSN9PV+9u5Tfb6OJplFNn1+q9bCk75293d3egcNaRgP3Nfurq6pLrMArpuuuuk0J4PG79+vWSUc7SwH/729/kJfStb30r7TNb4UwuDg4ODsMEw5ah63KqdDaWlZUJSyD7oJOls7NTjtdqpb+YVGVlZeA7zSLphAqFQvKdZn9+hh4KhURb0EWBGA9N5xOAQMx5c3Oz3EPHNJMdaUcbmS9Zs950I9P1yV7J4FauXClMmoxx/vz54kyko7S9vV1igcnO1q9fLyFjZKIzZsyQUrlkjjfeeKOUYKWpZ/ny5bjssssApGLJe3t7ZUwZw3/DDTeIZsB6LPv375f+I6P/7Gc/Gyh1nEwmhXFSq6NGxP9r6HFk/8disUC55pKSkrQQPQD45je/Kb/TrHXllVdK3DudvW1tbWmFtQBvfMiGaQbau3evbEDBMZ0yZYqMFRn1aaedJmPFQIDW1tbAJjA5OTliCmN441//+lcAnrZI9q6zpAlqXFr2TCyb7c/NzZW2USNMJBKi9R1N8Ss+13zOc3Jy5NnQpgzOOc7buXPnYsaMGQCA//mf/wHgaS2ch9T0+IzU19dLe7MJjqE7ODg4DBMMW4aek5MjdkTNpPm2JlPUNmiyBF2/wc/Gy8vLA44hDbIkXeuBb/K8vLyMFRrJKnRZXtpXCV0HhtBahU5yIIvj/bu6ukRzIIvNJId2bvnbvXTpUtmE4Gtf+xoAjyWSAdMuHI/HhQnyGnPmzJE9KIkXX3xRHI88ftmyZYFkJ701GO28OsmJ1fcWLVok8jNJpba2VmzXtIWec845IotOkKIjk1mUZG579uzJuFEI4XcoazQ3N4vsZKU/+tGPJCSSGYy33nqrJO/ceOONIhvHj6w9JydH5i3R1NQkVQxpcy8vLxfNgs9AQUFBYIu1ESNGBGoFWWtFa6V89Gckk0m8/PLLAFKO2t/97nfSZ9QKdNlZPWfZp/yura0tsHXjqFGjxIHvryzZH3BOaX+Yv96RDq3VSYKUj8eNHj1aas3492PVVVCzCcN2QY9Go9L5Or5aO2SA1MKmS4BqZ6dfdczLyxPHi44R1ynggPcwc5Hg4JeWlgYWUj3hGf1QXl4uTjwdUeNf0OPxuKisfAiBYInepqYmKVxFR6g2udAcwfaEw+G08r5sIxcWOrLuvfdeydzji6KoqEgWVL5Y9u/fL6VQ6Xi68cYbJW6b/bNjxw7ccsstacddffXVsuCzHvu+ffvkJcjs2vLycvzhD38AkCoHEA6H0wqzAd7CzznAF21nZ6f0C+cB2z9hwgS5nl7Q/SaHTAt6KBSSe3BMrrvuOnz1q19Nk72rq0uiJ2hW6uzslH7mffPz82WcOX66mBZf6t3d3TKmbLeuka4dhpniq/1mEl43NzdXoj1oBrvpppvkpfmpT30KQOrFAqSnzPPl8oUvfAEA8IlPfEIctHo3Lr6MMjmcDwd/0T393OidndgX+vnid3wOotGozB//LmajRo3KGOUy1HAmFwcHB4dhgmHL0HUtFb0DuDa/8DvAYzD8W8cnE3o3I7IovskzYdeuXcLQidGjR4t6mslsQybILD8g5cTcv39/IP69t7dXwuooUywWk+Po0HzxxRclU5OMXl+bMbtkUO+++24gZK24uFgKJFEVvuOOO/DrX/9a7gF42ansN7KbvLw8iYdeuXIlAE8r8O/eBKRis6nav/HGG2LqoRbQ09MjWbBk/tXV1eIo5bgUFBRIO8jyP/e5z8neltQ+rLUBrUIzS/8mGUBq3MjUNRNk/8fjcbk/j9u6davs6XnrrbdKX9BByhj/7du3y331Tjq6fg/gOV65SQQ3YcnLy5P8AM3KyfQpS2lpqZiptKPSr6Xo0r/+zSx03R+Ot2b5lB1ImVzI8qurq4WNUxsZMWKEzNtMu1odDmT6bLcxJm2zC7aLfaHl43pBbSmZTErbdfG7bIZj6A4ODg7DBP1m6MaYMIA1AHZbay8zxkwEcD+AcgDrAHzKWps1RqVEIpHGvvkd2ZPepZ7/18X4Cf6tz6PdmJXv9DZXxLvvviuhTjp8kiF/OtOObIY2ZbJZIFW5b//+/QF7ZygUStuCDfDYJtkHk4iefPJJsX8TFRUVOPvsswGkEk3IllauXCllY/VWbPxd299pt2bSxvvf/35hPWQ3RUVF+OIXvwggxUozZSROmDABP/7xjwEAP/zhDwF44WK6TDHlJPPlmD344IPCXqnhsG+AVILNnXfeKdehzM3NzWIH5nygE3nkyJHSB9RoWltbA/tP6nHUtWkoHxPYdu/ejc9//vMAUslGZWVl4lymM7ytrU3q71Bb0BmqtKV/8pOflOQzhotWVlbK/Oa5erMGtv/AgQOB8rDWWtFweA06YisrKwOJQDpzms/Fyy+/LPNRh7P6s6m1Bqt9Npmyo/sLvQEF4I2FP7RXb0FH+ay1gSAKvXcs+5vX37t3b6BscDbgSBj6bQA2qv//N4AfWWvfA6ARwE0DKZiDg4ODw5GhXwzdGFMN4FIA/wXgX433+v0ggE/2HbIYwH8A+MVxkPGokEwm09Lh+Z2f7egd2gkd2eK3q/f09EgEChlPpuSJurq6wO/l5eVpTBpIj5ChDb23t1cYBKsTrlq1KpB63tvbK2xK2yzJ/FhxbufOnQF7/2mnnSa2TDIlJv/U19eLLEwkKSoqkj7SdlmmSTOd/sEHHxStgtfv6OgI1ENftmyZJFzxujt27JDxWLp0KQCvFjjbo22qjDRiAlZ7e7ucwxDFWCwWSPj63ve+h9tuuw1AauyNMQHbKKNTxo0bJyGZ9FNs2bJFGLXeUtAfYREKhYSt0/7f2dkpc4++i3379omfgLbsNWvWBDY81n9TQ2hsbBRNg36CWCwW2IhaR3ZwLmgbOmXKzc0N1GunhqRDODkXdegs/Rp63wFeV1cLpbYYj8cD/paurq6jYuYE76Ejc/z+Kp10qKuFkoXrshdsr78uelVVVVZGufTX5PL/AHwVAFOjRgBostZyhdkFYGx/LjQQxbn6UwAnEokEwp7C4XAgjpRmAR07y4HSCwLPy8vLk0VJX58LAs/dsGGD/M5JMGPGjDR1jtegLHSYalMEnWS9vb2BHYMKCgpkwdX7JXJBY+EhFqgCUuaI8ePHy07vNEewjRdeeKHIye9CoZBch6ppSUmJtI1O0RkzZsg5NOVoZzAdTp///Odx9913SzvYF8xi5P6qd9xxh4T56Th59hGdwo2NjWLWYKZod3e3jAEXuFtuuUXK1+oYfo4B+5F9PHnyZHGi8qUUDofFvMDwwB07dsjL99VXX5X2+PepvPDCC+UchmbOnz9fzGyM7962bZv0BUMq9f6dXHS185T9VFZWFsjg1eGNnNPd3d1phcwI9oG/QFw0Gg1sshKJROTlynLEmWCtDeydG41GA3t2dnV1ycKaaWE/3HrhrwOTSCQyrhf+7/S9dGapv3aOv4Df0SCT3ANVnOuwJhdjzGUAaq21a/XXGQ7NWEnHGPMZY8waY8yaTOnmDg4ODg4Dg/4w9AUAFhpjPgwgF0AxPMZeaoyJ9LH0agB7Mp1srV0EYBEATJs2Lfs24VMgi9IbWPj3kYxGoxJaRxQVFQVC27RDhW/c6dOn4+abbwaQYjMFBQWBanVaW+D9Y7FYWogV4DnnaEqg2v3UU09J7Q0meBQVFck9aDaYM2eOhJv5nXnl5eWyxRtZ8dKlS8VhR+fteeedJ22k+aS6ulqYOa/X0NAg12PdkUsvvVRYEfs2FotJSN8jjzwCwNtKTGc7Ap5DjnJxLHJzc2WXeN2f1EjIMkeNGiVMVZtI+LeuYgh4ZhFuhkAzVGtrq5Ssvfbaa+VeHFtqPDU1NaKtUHsoLy8XBs8MzzPOOEPCSdmPY8aMkTGlxlFSUiLhiJxve/bsES2Oc6GnpydQl0iXeD6aDEyCZkxeV2tBlDeRSKRt4wZ4c4xsXG+Bx7nH+Z6Tk5OVe3WeKDgsQ7fWft1aW22tnQDgGgBPW2uvA/AMgI/1HXY9gCXHTUoHBwcHh8PiWBKLvgbgfmPMHQBeBXDPwIg0dKBdTNvhyGbIjHp6esRmSDbX2toq7J7Me9asWXI9sr8pU6bgs5/9LICUHXrUqFFilyQD1FvW0XF2yy23SBKP3iSDzjam9j/66KNSiU8zerInbsV2yimnSDga2SGZVmFhobB3Ojtzc3Nx1VVXAUjZ5KdOnSpMmc63KVOmSPggtYGuri6xm/7gBz+Qz2984xsAIBsL3HHHHVJv+rrrrgPgVV3UtVl0fwKpui2VlZVi1yYr37BhgzjbaIfeuXOnhPfRVltQUJBW9wZIaS0lJSXSF7zXX/7yF6m58qUvfQmAFyK6ePFiACmH7pe//GVJfPrZz34mcnz6058GkKrgt3v3bhkX2tB/+tOfCqvXYZiUmQ7nNWvWyNZp2vmvE2sAjw1zrDJVSuwvdEVQXpd9R6cokL5puf87anjalq03Yedzk42bMGc7jqjHrLXPAni27+9tAOYd6vgTDX41Vat/emcVTkxGJsyZM0fMEVxU9MuAi7euBcLFVBf0p5pvrQ3sHblw4ULce++9AFIq82233SZx7Q899BAAL3qEsrId7e3tOP300wGkFoLy8vK0lxSQXoCI7WZb582bJzLzwe3o6JCIH71TOxc+lgUeO3asLIp8cG+66SZpL1+ClZWVuOkmL/qVETrNzc0S+6xjgfmws5/WrFkjTknmByxatEgWDDqKzz77bDHT6Fhlvvz8WYr5+fnSB3yhnHzyyZKVes899wSOY/vfeustiRE/99xzAXhjxwWYL97JkyfLIkeHaXFxscSrMxN03Lhx0l7WvDn33HNx5plnAkjNs/z8fGm3dvyybcdSWMq/AOtdwDjvqqur03IACP/ztX//fplT7Du9j68zvRw5XKaog4ODwzCBGcy34LRp0+z//u//pgtwnMIWD3fewUIdgcz7JfK30tJSYXMMDevs7BSWSZZSX18vrIP3KioqSmNRgMdAeW1tamGIHBlbKBQSVkan2pNPPhnYOm3Tpk2yLRtjxDUrZMheNBqVdlBmyqFDx6gyx2KxtL8BL1RQ153huWSRDBVsaWmR6+n9Wtkv7GOdO6C3OuN31CimTp0a2Mu0trZWnMFk4Pfcc49c+9vf/jYAT2ugFkUNYeTIkTJWdDbqMDZqRGSibW1t8p2eMzQ7cYwLCwslrJLjM3fuXHz9618HkApJveCCC2Q+sK1dXV0yB3iN5557DldccUVa31ZVVcm16RStrKwU04jebEPHmvvbSBzuGeE5OouUfcyxuPvuu0X7oBy5ubnSRoZcfuELX5B+1Fmceq/eI5UvE/qzXmT7vUKh0Fpr7dzD3d8xdAcHB4dhAsfQFfxJCTk5OYEEjVgsJkyM4WkdHR2B2h7JZDKwoQCQYry0Y8bjcWFlZHuZtspKJBJYtGgRAOAnP/kJAODf//3fJYuRDDgajQpjInvesWOH2CrJfHVSlN5VHfBYnZ+JlZaWyt+6f5h1SHaWKflkxIgREo7ImuZ79+6Vc3R4pz+ksKenJ007ArykGx7He5WUlEjtGn43ffp0YaXsi56enoBmoP0YBO9ZWFgovgD2U2VlpcwBvUUdr6ezgf3ZlnV1daJd8Pjzzz8/kLnc3NwsfUBNoqamRnwRHO+amho5h+Gf5eXlgflYUlIi3x0qw7m/zwhlLyoqkr/p70kkEuKEZv9EIhG5P+ei9lH5NyUHMldbzCbWPJj36i9Dd25kBU5+XWKXk4+dHI/H0wr0AN4k5wPJRaqwsFAWFp3FyUVBF9jXGxMA3uT2L2ytra244IILAEAyLFesWCEqOE0zyWRSVHUuOpMmTZKHSZeM1Rt58Du2lWqxzp71p0sXFxdLe3TaNB9OmoNaWlokQ5TXi0aj0kYuCNpEwUWsoaEhkCE7ceJEMY1oJybHiv0ZDoflhci+6O7uDsRS6xRvQu9qz0WUL/f6+nq5P+dMLBaTPuNLrqysTBZvmkgqKyulf3iNnp4eGWedxcloHMpSUlIi5+g+Yzv0Hpd6UxfAGzN/zPfRZCT6zTW9vb0yz/XLnWOvyxBTFm3eYr9TFl2eNxtT67MdzuTi4ODgMEzgGLqC3yna1tYmzFOzdzpAGb9dUFCQFkcLeKyLjFIXfiIT4fU6OzuFMWq138+mdFgcj9u8ebOo23SI5efnS0gdz929e3eayYjyaUaur5tMJoUJ6uxMvRUakL5rPNV9Y4xoC+yfnJwcUb2p1UQiEWGZ+hoMg+T9q6urpX8Y/97U1CT343U7OzuFFRKJRELYI8dRM1qyzI6ODvmbfaBr7lBmjllVVVXa9nWAp3GQwZOVtrS0BExDjY2NgfK9NTU1Yq4gKx05cqTcl+POfgVSmklvb2+ggFxbW5togNrExt/529HEofvLd0QiEelbzqOWlhbJ1aCc2rzCsQuFQvK8aO3Pv7evQ//hGLqDg4PDMMGwdYpmu5PjcPfKdA7H6jvf+Q4Ar94Js0e5WUVnZ6cwOn/I5UDKd7TnDfVYHQ/5BvNe/0h9ke3yDea9nFN0GIKqPB2MQCpenaqwzrRzcHD4x4IzuTg4ODgMEwyqycUYUwegHcCBwx2bBRiJ7JfzRJARcHIONJycA4sTQc6TrLUVhztoUBd0ADDGrOmPLWiocSLIeSLICDg5BxpOzoHFiSJnf+BMLg4ODg7DBG5Bd3BwcBgmGIoFfdEQ3PNocCLIeSLICDg5BxpOzoHFiSLnYTHoNnQHBwcHh+MDZ3JxcHBwGCYYtAXdGHOxMWaTMWaLMeb2wbrv4WCMGWeMecYYs9EY86Yx5ra+7//DGLPbGPNa378PZ4GsO4wxG/rkWdP3XbkxZqUx5p2+z7LDXec4y3iy6rPXjDEtxpgvZkN/GmN+bYypNca8ob7L2H/Gw0/65uvrxpjThlDG7xtj3u6T4xFjTGnf9xOMMZ2qT385GDIeQs6DjrEx5ut9fbnJGHPREMv5gJJxhzHmtb7vh6w/BwzW2uP+D0AYwFYAkwDkAFgPYMZg3LsfslUBOK3v7yIAmwHMAPAfAL481PL5ZN0BYKTvu/8L4Pa+v28H8N9DLadv3PcBOCkb+hPAOQBOA/DG4foPwIcBLANgAJwJ4JUhlPFCAJG+v/9byThBH5cFfZlxjPuep/UAYgAm9q0F4aGS0/f7DwB8e6j7c6D+DRZDnwdgi7V2m7W2G8D9AK4YpHsfEtbavdbadX1/twLYCGDs0Ep1RLgCwOK+vxcDuHIIZfHjQwC2WmvfHWpBAMBauwpAg+/rg/XfFQDutR5eBlBqjKkaChmttU9Ya1ka8WUA1cdbjsPhIH15MFwB4H5rbZe1djuALRikDeYPJafxiqZ8AsAfB0OWwcBgLehjAexU/9+FLFw0jTETAMwB8ErfV7f2qbm/HmpTRh8sgCeMMWuNMZ/p+67SWrsX8F5OAEYNmXRBXIP0hyXb+hM4eP9l65z9NDzNgZhojHnVGPOcMebsoRJKIdMYZ2tfng1gv7X2HfVdtvXnEWGwFvRMJceyKrzGGFMI4E8AvmitbQHwCwCTAZwKYC881WyoscBaexqASwB8wRhzzlALdDAYY3IALATwUN9X2difh0LWzVljzDcB9AC4r++rvQDGW2vnAPhXAH8wxhQPlXw4+BhnXV/24VqkE45s688jxmAt6LsAjFP/rwawZ5DufVgYY6LwFvP7rLV/BgBr7X5rbdJa2wvgVxgkFfFQsNbu6fusBfAIPJn20xTQ91k7dBKm4RIA66y1+4Hs7M8+HKz/smrOGmOuB3AZgOtsn8G3z4RR3/f3Wni26alDJeMhxjir+hIAjDERAFcBeIDfZVt/Hg0Ga0H/O4D3GGMm9jG3awA8Nkj3PiT67Gj3ANhorf2h+l7bSz8C4A3/uYMJY0yBMaaIf8NzlL0Brx+v7zvsegBLhkbCANLYT7b1p8LB+u8xAP+nL9rlTADNNM0MNowxFwP4GoCF1toO9X2FMSbc9/ckAO8BsG0oZOyT4WBj/BiAa4wxMWPMRHhy/m2w5fPhfABvW2t38Yts68+jwmB5X+FFDWyG99b75lB7g5Vc74en/r0O4LW+fx8G8DsAG/q+fwxA1RDLOQlepMB6AG+yDwGMAPAUgHf6PsuzoE/zAdQDKFHfDXl/wnvB7AWQgMcabzpY/8EzE/ysb75uADB3CGXcAs8Gzfn5y75jP9o3F9YDWAfg8iHuy4OOMYBv9vXlJgCXDKWcfd//FsDnfMcOWX8O1D+XKerg4OAwTOAyRR0cHByGCdyC7uDg4DBM4BZ0BwcHh2ECt6A7ODg4DBO4Bd3BwcFhmMAt6A4ODg7DBG5Bd3BwcBgmcAu6g4ODwzDB/wciOZFWCXtHLAAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_accuracy = accuracy_score(results_train['actual'], results_train['prediction_corrected'])\nprint(train_accuracy)\ntest_accuracy = accuracy_score(results_test['actual'], results_test['prediction_corrected'])\nprint(test_accuracy)","execution_count":43,"outputs":[{"output_type":"stream","text":"0.92643391521197\n0.8768656716417911\n","name":"stdout"}]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}
2 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2019 Gokul Karthik
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Deep Learning Projects using PyTorch
 2 | 
 3 | This space is to tune my practical deep learning skills.
 4 | 
 5 | 1. [Image] **Fashion MNIST Dataset** - Multi Class Image Classification using modified LeNet [[Link]](https://github.com/GokulKarthik/Deep-Learning-Projects-Pytorch/blob/master/1-Multi-Class-Image-Classification-Fashion-MNIST.ipynb)
 6 | 2. [Text] **Language Words Dataset** - Multi Class Word Classification using character level LSTM [[Link]](https://github.com/GokulKarthik/Deep-Learning-Projects-Pytorch/blob/master/2-Multi-Class-Word-Language-Classification.ipynb)
 7 | 3. [Text] **US Baby Names Dataset** - Name Generation using character level LSTM [[Link]](https://github.com/GokulKarthik/Deep-Learning-Projects-Pytorch/blob/master/3-Baby-Name-Generation.ipynb)
 8 | 4. [Image] Neural Style Transfer using pre trained VGG19 Net [[Link]](https://github.com/GokulKarthik/deep-learning/blob/master/4-neural-style-transfer.ipynb)
 9 | 5. [Image + Text] **CAPTCHA Dataset** - Text Recognition with CRNN using pre trained RESNET and Bi-GRU [[Link]]( https://github.com/GokulKarthik/Deep-Learning-Projects-Pytorch/blob/master/5-Captcha-Text-Recognition-With-CRNN.ipynb)
10 | 5. [Image] **Cityscape Image Pairs Dataset** - Image Segmentation with UNet [[Link]](https://github.com/GokulKarthik/Deep-Learning-Projects-Pytorch/blob/master/6-Image-Segmentation-with-UNet.ipynb)
11 | 


--------------------------------------------------------------------------------