├── 1_s4eFRgsGnHWmJWzgAe52Lg.png ├── 1_yqDjbLhneoEAyL8Lbwj43w.png ├── A3C.ipynb ├── CEM.ipynb ├── README.md ├── gym_into.ipynb └── qlearning.ipynb /1_s4eFRgsGnHWmJWzgAe52Lg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajit2704/Reinforcement-Learning-on-google-colab/1ae97f3f756dc6a273b592338e9d4e810bf9a3c4/1_s4eFRgsGnHWmJWzgAe52Lg.png -------------------------------------------------------------------------------- /1_yqDjbLhneoEAyL8Lbwj43w.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ajit2704/Reinforcement-Learning-on-google-colab/1ae97f3f756dc6a273b592338e9d4e810bf9a3c4/1_yqDjbLhneoEAyL8Lbwj43w.png -------------------------------------------------------------------------------- /A3C.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "A3C.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU" 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "view-in-github", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "\"Open" 26 | ] 27 | }, 28 | { 29 | "metadata": { 30 | "id": "wRXcuuh3qB6q", 31 | "colab_type": "text" 32 | }, 33 | "cell_type": "markdown", 34 | "source": [ 35 | "# Environment Setup" 36 | ] 37 | }, 38 | { 39 | "metadata": { 40 | "id": "jWdlrlePjq-g", 41 | "colab_type": "code", 42 | "colab": {} 43 | }, 44 | "cell_type": "code", 45 | "source": [ 46 | "!pip install gym" 47 | ], 48 | "execution_count": 0, 49 | "outputs": [] 50 | }, 51 | { 52 | "metadata": { 53 | "id": "xcOgo4xwj4PJ", 54 | "colab_type": "code", 55 | "colab": {} 56 | }, 57 | "cell_type": "code", 58 | "source": [ 59 | "!apt-get install python-opengl -y" 60 | ], 61 | "execution_count": 0, 62 | "outputs": [] 63 | }, 64 | { 65 | "metadata": { 66 | "id": "2aS0God8kBDE", 67 | "colab_type": "code", 68 | "colab": {} 69 | }, 70 | "cell_type": "code", 71 | "source": [ 72 | "!apt install xvfb -y" 73 | ], 74 | "execution_count": 0, 75 | "outputs": [] 76 | }, 77 | { 78 | "metadata": { 79 | "id": "WZuEBujtkKoj", 80 | "colab_type": "code", 81 | "colab": {} 82 | }, 83 | "cell_type": "code", 84 | "source": [ 85 | "!pip install pyvirtualdisplay\n", 86 | "!pip install piglet" 87 | ], 88 | "execution_count": 0, 89 | "outputs": [] 90 | }, 91 | { 92 | "metadata": { 93 | "id": "Sfq6FvJ9kW1G", 94 | "colab_type": "code", 95 | "outputId": "f5961c9a-1e86-4e71-f190-a8fe0cd403c6", 96 | "colab": { 97 | "base_uri": "https://localhost:8080/", 98 | "height": 54 99 | } 100 | }, 101 | "cell_type": "code", 102 | "source": [ 103 | "from pyvirtualdisplay import Display\n", 104 | "display = Display(visible=0, size=(1400, 900))\n", 105 | "display.start()" 106 | ], 107 | "execution_count": 0, 108 | "outputs": [ 109 | { 110 | "output_type": "execute_result", 111 | "data": { 112 | "text/plain": [ 113 | "" 114 | ] 115 | }, 116 | "metadata": { 117 | "tags": [] 118 | }, 119 | "execution_count": 6 120 | } 121 | ] 122 | }, 123 | { 124 | "metadata": { 125 | "id": "RW0KjeOVkc88", 126 | "colab_type": "code", 127 | "colab": {} 128 | }, 129 | "cell_type": "code", 130 | "source": [ 131 | "from __future__ import print_function, division\n", 132 | "from IPython.core import display\n", 133 | "import matplotlib.pyplot as plt\n", 134 | "%matplotlib inline\n", 135 | "import numpy as np\n", 136 | "\n", 137 | "#If you are running on a server, launch xvfb to record game videos\n", 138 | "#Please make sure you have xvfb installed\n", 139 | "import os\n", 140 | "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n", 141 | " !bash ../xvfb start\n", 142 | " %env DISPLAY=:1" 143 | ], 144 | "execution_count": 0, 145 | "outputs": [] 146 | }, 147 | { 148 | "metadata": { 149 | "id": "iTsbpF4In-3c", 150 | "colab_type": "text" 151 | }, 152 | "cell_type": "markdown", 153 | "source": [ 154 | "# Atari Environment\n", 155 | "\n", 156 | "---\n", 157 | "\n" 158 | ] 159 | }, 160 | { 161 | "metadata": { 162 | "id": "pJu7AeaZk7Dn", 163 | "colab_type": "code", 164 | "colab": {} 165 | }, 166 | "cell_type": "code", 167 | "source": [ 168 | "\"\"\"Auxilary files for those who wanted to solve breakout with CEM or policy gradient\"\"\"\n", 169 | "import numpy as np\n", 170 | "import gym\n", 171 | "from scipy.misc import imresize\n", 172 | "from gym.core import Wrapper\n", 173 | "from gym.spaces.box import Box\n", 174 | "\n", 175 | "class PreprocessAtari(Wrapper):\n", 176 | " def __init__(self, env, height=42, width=42, color=False, crop=lambda img: img, \n", 177 | " n_frames=4, dim_order='theano', reward_scale=1,):\n", 178 | " \"\"\"A gym wrapper that reshapes, crops and scales image into the desired shapes\"\"\"\n", 179 | " super(PreprocessAtari, self).__init__(env)\n", 180 | " assert dim_order in ('theano', 'tensorflow')\n", 181 | " self.img_size = (height, width)\n", 182 | " self.crop=crop\n", 183 | " self.color=color\n", 184 | " self.dim_order = dim_order\n", 185 | " self.reward_scale = reward_scale\n", 186 | " \n", 187 | " n_channels = (3 * n_frames) if color else n_frames\n", 188 | " obs_shape = [n_channels,height,width] if dim_order == 'theano' else [height,width,n_channels]\n", 189 | " self.observation_space = Box(0.0, 1.0, obs_shape)\n", 190 | " self.framebuffer = np.zeros(obs_shape, 'float32')\n", 191 | " \n", 192 | " def reset(self):\n", 193 | " \"\"\"resets breakout, returns initial frames\"\"\"\n", 194 | " self.framebuffer = np.zeros_like(self.framebuffer)\n", 195 | " self.update_buffer(self.env.reset())\n", 196 | " return self.framebuffer\n", 197 | " \n", 198 | " def step(self,action):\n", 199 | " \"\"\"plays breakout for 1 step, returns frame buffer\"\"\"\n", 200 | " new_img, reward, done, info = self.env.step(action)\n", 201 | " self.update_buffer(new_img)\n", 202 | " return self.framebuffer, reward * self.reward_scale, done, info\n", 203 | " \n", 204 | " ### image processing ###\n", 205 | " \n", 206 | " def update_buffer(self,img):\n", 207 | " img = self.preproc_image(img)\n", 208 | " offset = 3 if self.color else 1\n", 209 | " if self.dim_order == 'theano':\n", 210 | " axis = 0\n", 211 | " cropped_framebuffer = self.framebuffer[:-offset]\n", 212 | " else:\n", 213 | " axis = -1\n", 214 | " cropped_framebuffer = self.framebuffer[:,:,:-offset]\n", 215 | " self.framebuffer = np.concatenate([img, cropped_framebuffer], axis = axis)\n", 216 | "\n", 217 | " def preproc_image(self, img):\n", 218 | " \"\"\"what happens to the observation\"\"\"\n", 219 | " img = self.crop(img)\n", 220 | " img = imresize(img, self.img_size)\n", 221 | " if not self.color:\n", 222 | " img = img.mean(-1, keepdims=True)\n", 223 | " if self.dim_order == 'theano':\n", 224 | " img = img.transpose([2,0,1]) # [h, w, c] to [c, h, w]\n", 225 | " img = img.astype('float32') / 255.\n", 226 | " return img\n" 227 | ], 228 | "execution_count": 0, 229 | "outputs": [] 230 | }, 231 | { 232 | "metadata": { 233 | "id": "oR12anpKlcId", 234 | "colab_type": "code", 235 | "colab": {} 236 | }, 237 | "cell_type": "code", 238 | "source": [ 239 | "!apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig" 240 | ], 241 | "execution_count": 0, 242 | "outputs": [] 243 | }, 244 | { 245 | "metadata": { 246 | "id": "hl5urImpnSBQ", 247 | "colab_type": "code", 248 | "colab": {} 249 | }, 250 | "cell_type": "code", 251 | "source": [ 252 | "!pip install gym\n", 253 | "!pip install \"gym[atari]\"" 254 | ], 255 | "execution_count": 0, 256 | "outputs": [] 257 | }, 258 | { 259 | "metadata": { 260 | "id": "xMEwvBMNoRdt", 261 | "colab_type": "text" 262 | }, 263 | "cell_type": "markdown", 264 | "source": [ 265 | "# Deep Kung-Fu with advantage actor-critic\n", 266 | "\n", 267 | "---\n", 268 | "\n", 269 | "For starters, let's take a look at the game itself:\n", 270 | "\n", 271 | " * Image resized to 42x42 and grayscale to run faster\n", 272 | " * Rewards divided by 100 'cuz they are all divisible by 100\n", 273 | " * Agent sees last 4 frames of game to account for object velocity" 274 | ] 275 | }, 276 | { 277 | "metadata": { 278 | "id": "mRSoMY-Tkkmb", 279 | "colab_type": "code", 280 | "outputId": "39d30f04-0720-424d-f735-98b1ca581e8a", 281 | "colab": { 282 | "base_uri": "https://localhost:8080/", 283 | "height": 106 284 | } 285 | }, 286 | "cell_type": "code", 287 | "source": [ 288 | "import gym\n", 289 | "\n", 290 | "def make_env():\n", 291 | " env = gym.make(\"KungFuMasterDeterministic-v0\")\n", 292 | " env = PreprocessAtari(env, height=42, width=42,\n", 293 | " crop = lambda img: img[60:-30, 5:],\n", 294 | " dim_order = 'tensorflow',\n", 295 | " color=False, n_frames=4,\n", 296 | " reward_scale = 0.01)\n", 297 | " return env\n", 298 | "\n", 299 | "env = make_env()\n", 300 | "\n", 301 | "obs_shape = env.observation_space.shape\n", 302 | "n_actions = env.action_space.n\n", 303 | "\n", 304 | "print(\"Observation shape:\", obs_shape)\n", 305 | "print(\"Num actions:\", n_actions)\n", 306 | "print(\"Action names:\", env.env.env.get_action_meanings())" 307 | ], 308 | "execution_count": 0, 309 | "outputs": [ 310 | { 311 | "output_type": "stream", 312 | "text": [ 313 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 314 | "Observation shape: (42, 42, 4)\n", 315 | "Num actions: 14\n", 316 | "Action names: ['NOOP', 'UP', 'RIGHT', 'LEFT', 'DOWN', 'DOWNRIGHT', 'DOWNLEFT', 'RIGHTFIRE', 'LEFTFIRE', 'DOWNFIRE', 'UPRIGHTFIRE', 'UPLEFTFIRE', 'DOWNRIGHTFIRE', 'DOWNLEFTFIRE']\n" 317 | ], 318 | "name": "stdout" 319 | } 320 | ] 321 | }, 322 | { 323 | "metadata": { 324 | "id": "m7pd1KapnM0A", 325 | "colab_type": "code", 326 | "outputId": "f8d1707b-b384-4969-c8c3-fc90669ec6ae", 327 | "colab": { 328 | "base_uri": "https://localhost:8080/", 329 | "height": 499 330 | } 331 | }, 332 | "cell_type": "code", 333 | "source": [ 334 | "s = env.reset()\n", 335 | "for _ in range(100):\n", 336 | " s, _, _, _ = env.step(env.action_space.sample())\n", 337 | "\n", 338 | "plt.title('Game image')\n", 339 | "plt.imshow(env.render('rgb_array'))\n", 340 | "plt.show()\n", 341 | "\n", 342 | "plt.title('Agent observation (4-frame buffer)')\n", 343 | "plt.imshow(s.transpose([0,2,1]).reshape([42,-1]))\n", 344 | "plt.show()" 345 | ], 346 | "execution_count": 0, 347 | "outputs": [ 348 | { 349 | "output_type": "stream", 350 | "text": [ 351 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 352 | " if issubdtype(ts, int):\n", 353 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 354 | " elif issubdtype(type(size), float):\n" 355 | ], 356 | "name": "stderr" 357 | }, 358 | { 359 | "output_type": "display_data", 360 | "data": { 361 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMsAAAEHCAYAAAAAprJIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGARJREFUeJztnX+0FWW5xz+IeSvUABOOaF2vP85T\n3sPNC3rRjMtBLdDsukorbuYytZsthSAylbqYUFcMEynwdmFFYvgjFVaFImhBJVkCkugh7VFprX6A\nCv5CMUOgff+Y2Zw5m/3j3TOzz7yz9/NZa6+z9zvvnvk+c+a73/edd+aZPoVCAcMwarNf1gIMIy+Y\nWQzDETOLYThiZjEMR8wshuGImcUwHNk/awHNgIj0ASYAFwMHAG8BngKmqur6XtZyOHC/qnb05nZb\ngT42z5IcEbkWGA18TFWfFZG+wGeB64B2Vd2WqUAjFcwsCRGRgcBfgPep6tMlyw5S1dfC9wIsAA4h\naHmmquod4bIC8DngC0B/4ALgv4D3A08AH1HV3SJyCjAbGAC8AHxKVf9Qss0jgWdUdX8R+QxwFrAT\nGAkoMB34JnB0qGG+iOwHzAFOJ2gZfwVcpKq7wvX9KNR1P3AEsFhVF7roaSZszJKck4A/lRoFoGiU\nkG8B96rqe4GLgAUi8pbI8neq6lDgTmAJ8DWgHRgKjBKRg4B7gK+o6jHAt4G7HPSNAaYBxwLvBb5M\nYJyLgalhnY+GZR1hneHAJyO6H1DVfwJWEBiKBHpyi5klOQOAvd0sEekvIr8PX38RkSvCRWcD14fv\nfwW8FTgssp4fh3+7gE2q+pSq7gSeBoYQHMx/UdWfAoSt0jEi8u4a+p4oWdcDqron3M6QcF1LgBNU\ndZeq/g1YBxwVfn8kcEdY78fAlkh5HD25xQb4ydlGeNABqOorwHsAROR7wNvDRWOA/xaRQ4G/A33o\n+WNVbIX2ADsi5XuAvgTdoKNF5PeRZTuBQ4E/VdEXbd2i695T3H6oaY6IDAu1tRF0ryD4MXgpso7N\n4d+4enKLmSU5vwEGici/quqj5SqE3a27gU+o6n0i8g/AG3VuZwvwpKqekExuWf4H2AUMVdWdInJb\nZNmrwIGRz8XWsJF6vMS6YQkJxyVfBxaJyDEAIrKfiIwDPgE8A/QLX4+EX5sIvEnPg7AWa4DDRGRE\nuI2jRGRReNo6KYOArtAo7wNOiWhbSxAHInIW3a1oI/V4iZklBVR1JkG3ZXHYLfkDcCFwrqreGnbN\nZgKPisijwCaCMcq9ItLPcRtvAOcSdJeeJDhDdbeqpnE68wbg8+F6LwO+BHxWRD4OXAF8LIzrNIKW\ntNBgPV5ip46NmohIn6IJRGQd8A1V/UnGsnoda1mMqojI9cBN4fv3EJxa7tWrEnzBWhajKiJyGLAI\nOJLgDNq1qnpLpqIyInWziMiNBBN1BWCiqq5LdQOGkRGpdsNEZBRwrKqeTDBD/J00128YmVIoFFJ7\ntbe3T29vb/9s5PPv29vbD65UHyh0dXUVCFqh3L+aKZZWjqfS8Zr2pGQbPQd/28KyV8tV7urqoqOj\no2icpqCZYgGLJ0qjZ/CrTlANHTqUQqFAnz7NMY/VTLFA68ZTyVBpnzreQtCSFBkCPJvyNgwjE9I2\nywMEs7qEF+VtKblM3TBySyNOHV8H/DvB1auXqepjFTfep0+hmZr6ZooFWjeeQqFQtlKmk5JmFr9p\n1XgqmcUudzEMR7y+n2XixIlZS6ibPGquhsXTjbUshuGI1y1Lb3HJJZcAMG/evIrLopTWi9a55JJL\nqi6vtB3Df1q+ZSlnhnLL5s2bt/cgLzVHcXm570WXl/u+kR9a3iz2K2+40vJmqUZpa1GtS2atRfNj\nZjEMR2yAX4Nag/VondmzZ++zzGgerGWpgkvXyrpfrUPLtyzVTvvOmzev5qnjcnXKLS89s2bkj5Y3\nS60D1+XAjnbDytU3czQH1g0zDEfMLIbhSMt3w3xgwJQB+5S9POPlDJQY1bCWJWOKRnl5xst7X9Fy\nwx/MLIbhSOxumIjMJHj60/7ADOA/CB6v9mJY5XpVXZZYoWF4QiyziMhooENVTxaRQ4BHgVXAFFW9\nN02BhuELcVuWBwkecgPwCsGDevqmosgwPCVxwgoR+RxBd2wPQc6wA4CtwHhVfaHadzdu3Fjo6OhI\ntH3DaADpZ3cRkbOBrwAfAk4AXlTVDSJyFXCEqo6vqqhGdpe83f89e/ZsJk2alLWM1GjVeGbPnl32\ngEwywB8DfBUYq6rbgZWRxUuB78Zdt2H4SKxTxyLyDoJnup+lqi+FZUtEpPjs9E5gYyoKDcMT4rYs\nnwTeCdwlIsWym4E7ReSvBM9avzC5PMPwh1hmUdX5wPwyi1ry8WnNjN1a0E3LXRvWpf/Z4/NQuaOu\n5Wmsw2UbWVOataZ4T04rG8YudzHK0sqmqISZxahJtSSErYSZxaiJpXsKMLMYZWl1Y5Sj5Qb4tQbT\nLoPtpOvwcUBfyrx585g9e7adDYvQcmYx6qPVDRLFumGG4YiZxTAcMbMYhiNNP2YpHaCWzkJXWh79\nG6W0rLiuKVOe7vHXN2bMODZrCbmnJVqWtLJOlqZlNVqLljBLrTmD0uXl6rvUMZqbljBLaatQbnnp\n+9L6lsPYaAmzlFLpKV71fKdShn2jeUmcsCLRxu0efK9p1XhSvQdfRDqBu4HfhUVdwExgEUFKpGeB\n81V1Z5z1G4aPJOmG/VJVO8PXBGA6cJOqjgSeAS5KRaFheEKa8yydwOfD9/cAl+NRhpd65lPKlZf7\nTpQzVq/u8dc3lo8cmbWE3JPELMeJyFJgIDAN6Bfpdm0FDksqLm1qPUzV5WGrRusSa4AvIocDHwDu\nAo4Cfg4cqKoDw+XHAD9Q1fdXW49lpDQ8Jf2MlEVEZC1wIvB2VX1DREYBE1T13KqKevFsmOtlLqX1\nyyVuKDe/csbq1YxZv577hw9PTXOaxOmG2dmwnsRNsneeiFwevm8DBhPkDTsnrHIOsCLOuhtJracG\nx5l/MVqHuN2wg4Dbgf4EicCnETx24gfAW4E/Aheq6q6qG7d5Fq9p1XhSnWdR1deAj5RZ9ME46zOM\nPOD1DH7eaKZYoHXjKRQK6WbR7w2GfWNY1hLqJo+aq2HxdOO1WQYdMShrCXWTR83VsHi68dos+92V\ns4uiL8ih5mq0ajwXlC/22iwbjtiQtYS6yaPmalg83XhtlrZ3t2UtoW7yqLkaFk83TdTGGkZj8bpl\nyePgMo+aq2HxdGPzLCnSTLFA68ZTaZ7FumGG4YiZxTAc8XrMsmJY/maP86i5GhZPN9ayGIYjZhbD\ncMTMYhiOeD1mOf6+/M0e51FzNSyebqxlMQxH4t5WfDFwfqToBOARoB/welj2JVVdX3XjNSYlTx37\ncE0tU740AYAZN8ypWbdRFDWcfvpaTjtjTaYa0twPK5ePqCsen/4X5TS4xrNy+YhUbyteACwACDO5\nfAL4Z4L77jfGWWdc9ly5J3iT4VUZezVU/WnoJQ0+7Icm1ZDGmOVq4Dzghymsq276frNv8OaGLLZe\noiFDvNoPTaohkVlE5ETgz6r6nIgATBeRdwJPApNU9Y0UNFbFq18za1maWkOiCylFZB5wh6r+QkQ+\nCjyuqptE5LvAJlX9VrXvW0ZKw1MakrCiE5gAoKo/ipTfA3yy1peHDh1a9UpQlwH+FVsvBWDmoP+t\nWbdRFDWMWb8+swF+I/ZDvQN8n/4X5TTUMcAvWx7bLCIyBNihqm+KSB/gp8C5qvoKgYl6ZaDvVT85\nQ7zaD02qIUnLchhBtnxUtSAi84GVIvI6sBm4Jrm82njVT7YxS1NriG2WcA7ljMjnuwiy6vcqXv2a\nZYhX+6FJNXh9uYsLXv2aWcvS1Bpybxavfs0yxKv90KQacm8Wr37NrGVpag25N4tXv2YZ4tV+aFIN\nuTeLV79m1rI0tYbcm8WrX7MM8Wo/NKmG3JvFq18za1maWoPXSfaee+7MXlaUjMGDl/H88x/OWkZq\ntGo8gwcvsyR7hpEEr7thPx+Wr8cdjNucP83VaNV4xm0uX24ti2E4YmYxDEfMLIbhiNdjltG/PT5r\nCXWTR83VsHi6sZbFMBzxep7F5bZin6j3NlzfadV4KuUNs5bFMBxxGrOISAfwE+BGVZ0rIu8CFgF9\ngWeB81V1p4icB0wC/g7MD5PxGUZTULNlEZF+wBxgZaR4OnCTqo4EngEuCutdDZxOkLDiiyIyMHXF\nhpERLt2wncCZwJZIWSewNHx/D4FBRgDrVHV7mFzvIeCU9KQaRrbU7Iap6m5gd5hxskg/Vd0Zvt9K\nkOmlDdgWqVMsr0hXVxcQPEW2WaiUcyqvWDzdpDHPUulZyTWfoZxGkj2faNWzR3khaZK9uGfDdojI\n28L3hxN00bYQtC6UlBtGUxDXLD8DzgnfnwOsANYAJ4pIfxE5kGC8sjq5RMPwg5rdMBEZTnCT5pHA\nLhE5l+AREwtF5BLgj8AtqrpLRK4C7gcKwDRV3d4w5YbRy7gM8NcTnP0q5YNl6i4GFieXZRj+YTP4\nhuGImcUwHDGzGIYjZhbDcMTMYhiOmFkMwxEzi2E4YmYxDEfMLIbhiJnFMBwxsxiGI2YWw3DEzGIY\njphZDMMRM4thOGJmMQxHzCyG4UiSjJQ3A28BdgGfVtXnRGQXQb6wIqep6p60RRtGFrjcg18uI+U3\nCNKz3iUilwGTgSuA7ara2QihhpE1cTNSXgosCd9vAw5JWZdheIfzIydE5BrgBVWdGynrC6wCpqvq\nShHZQZDW9R+BJao6q9o6N27cWOjo6Iir3TAaRdmsj7EzUoZGWQSsUtViF+1y4FaCVEgPisiDqvpI\npXUkzUi5asVJznUbRVRDoVDwLoNjkn1kGSl7kuRs2M3A06o6rVigqv+nqjtU9XWCMc7QBOuvSvQg\nyAofNNRD3vT6RiyzhM9heVNVvxYpExG5XUT6iMj+BBkpf5eSzprYgbAvtk/SJW5GykHA30TkF2G1\nJ1T1UhH5M7CW4GFGS1V1bUNURyh2LbI8MHxOYO7D/mkWkmSkLFf3yqSC6qV4EGR5wHYfiH49OuPU\nsQ+bSVLE60d7V+PUsQ8z5/3T936e8OvsNfjI47eeuff9pFuvzlBJ/rHLXQzDkVybZcKvr+7xt1U1\nVKPYmlirkpzcdsMA2h97nAlkexBkpWHurIMBGD/5VYd632LuMJzqG5XJbcvS/tjjPf62koaiUUrf\nV6vnUt+oTq5bliLtjz3OU+/7l8w1BA9szh4zRGPInVmybEmiZKmj2JUqmqKWOUrrG/HIbTcMyLw1\nyVpDdPwxfvKrZT+XGsXGLPHJXctilKe01bBWJH1y3bIY5VuK0lamWl3DndyapdW7YFFKTTB31sHM\nnXVwD9OYUZKTW7NEB9hZHbQ+aKhG1DRGcnJrFqMnZojGk9sBvg+/5D5oiHLcccftc/X1qhUneX0L\nQZ6wlsUwHMmdWfZ8fxR7vj+qx+esdGStoRbWqqRLbrthAEdPHJC1BC80FDl17MM951dmPWFjmRSJ\nm5FyITAceDGscr2qLgvvzZ9EcFvxfFVd0ADN+3D0xAFs+vbLvbGpqhp8w4ySLnEzUgJMUdV7S+pd\nDfwb8CawTkR+pKovpah3rymKBsnqIC1uO0sNRu/i0rIUM1LWur9+BLBOVbcDiMhDBBle7kmksAq+\ntCh7NYzNVIrRYFwSVuwGdotI6aLxIjIZ2AqMB9oIUrkW2QocVm3dXV1dQJCcLjY+HKARDZUStGXB\nyuVprMOfeNIgSTxxB/iLgBdVdYOIXAVcA5SmjCifZjJC0oyU0e5PVi1MVMP8se2ZZnAsvXgy6ZjF\nMlL2JJZZIulaIcht/F1gMUHrUuRwwM5bZoQN7tMnbkbKJSJyVPixE9gIrAFOFJH+InIgwXhldSoq\ny+DDoNoHDUbvETcj5RzgThH5K7ADuFBV3wi7ZPcTZJubVhzsG43H7l9pPEkyUi4pLVDVxQTdsYbh\ny6+5LzqM3iN3l7uUI+vTx75oMBpL7i53KR6Upzy3GYCH2g7PTEfWGqKMn/yqF3mfmxmvzTJ52Ocr\nLltz31QgMM2IM7/eW5JqaHi0quZGs2pF8Dc9DdnGkz6u8TxatjS33bCsDOKbhnKcde2GrCU0Jc7P\nlGzIxvv0KVSblMwbzRQLtG48hUIh3WdK9gb3fuX4rCXUTR41V8Pi6Sa33TDD6G3MLIbhiJnFMByx\nAX6KNFMs0Lrx2AC/l8ij5mpYPN1YN8wwHPG6ZTnhC0OyllA3edRcDYunG6/N8vNh+ZqJHrc5f5qr\n0arxjNtcvty6YYbhiJnFMBwxsxiGI3EzUt4NHBouHkiQmOJaoAtYH5ZvU9WPp6zXMDIjVkbKqAlE\n5PvA97oXaWda4kb/Nn/n+POouRoWTzeJMlJKkHmvv6quFZEjY6uoQFvbfWmvsqEUCvnTXI1WjafS\nRS1JMlICTCRodYq0ichiYAhwk6reVm3dqWSk9IxmigUsniix51lE5ADgA6p6aVj0IjAVuBV4B7BW\nRFap6rOV1lErI+UdQ/I1ITZu82Z+eHj29+OnRavGM25z+YmWJJOSo4C1xQ+q+hpwc/jxBRF5BHgP\nUNEsaRHthxYnncqV9YaG0b89PnMN0e35pqE3dFTbXnFZHA1JTh2fCDxW/CAio0VkVvi+H3A88FSC\n9ddFueB9mH3ubQ0+7AdfNKStI25Gyo8RZMjfFKm6GrhARH4D9AVmqGqFCwcMI38kyUg5oaTebuAz\nqagyDA+xGXzDcKSpzBLtj2Y1XjEN/mhIe9teX6JfD77MNGetI+vt+64hiTav78G3eZZsadV4xm3e\nXPaA9NoseaOZYoHWjSeXCStgbt3fWHTyNADO/83X0hbjyFwPNKS5H+r/H6SvIRlFHQHx42mqAb5h\nNBIzi2E4YmYxDEe8HrO0DTkkk+8mIbrdrDRESaohjRh82A9FkmixlsUwHPG6ZTm0rb4nAs/65lQm\nX7kIgEW3TGXylb3/ZK5Ft8zKXEOa+6He/0EjNCQhquPTxI8HmqhluW3hdQwe3G/v58GD+3Hbwut6\nXUfWGnzYD75oKNVRLI+L3y3LoP51f6d0Z8RZR1KaSUMS7T7sh546xiTS4fUMfr2PqL594fQenz/1\nmatja4vDyuUjeP75D2eqAdLbDyuXj+C0M9ZkqiEpUR2DBy9zimfl8hH5u9wlb89zT3Jw+UirxlPJ\nLE0zZjGMRuOakXImMDKsPwNYBywiuH34WeB8Vd0pIucBk4C/A/NVdUFDVBtGBtRsWURkNNChqicD\nY4HZwHSCvGAjgWeAi8IkFVcDpxPchvxFERnYKOGG0du4dMMeBIrpWl8B+hGYYWlYdg+BQUYA61R1\nu6q+ATwEnJKqWsPIEJeEFXuA18OPFwP3AWNUdWdYtpUg00sbsC3y1WJ5RZoxI+XK5SOylpAqFk83\nzvMsInI2gVk+BDwdWVTpbpqad9nUykiZN5opFmjdeCr9eDudDRORMcBXgTNUdTuwQ0TeFi4+HNgS\nvtoiXyuWG0ZT4DLAfwdwPXCWqr4UFv8MOCd8fw6wAlgDnCgi/UXkQILxyur0JRtGNtSclBSRzwHX\n0DMV6wUEz2R5K/BH4EJVLWar/DJQAObUyqJv9+D7TavGU+kefK9n8PNGM8UCrRtPJbPYDL5hOGJm\nMQxHzCyG4YiZxTAcyXSAbxh5wloWw3DEzGIYjphZDMMRM4thOGJmMQxHzCyG4YiZxTAcySzJnojc\nCJxEcIXyRFVdl5WWOIhIJ3A38LuwqAuYSZlEHpkIdEREOoCfADeq6lwReRc5TkZSJp6FwHDgxbDK\n9aq6LE48mbQsIjIKODZMgnEx8J0sdKTAL1W1M3xNoEwij2zlVSdMMjIHWBkpzm0ykgrxAEyJ/J+W\nxY0nq27YacCPAVT1SWCAiByckZY06WTfRB4+sxM4k553tHaS32Qk5eIpR6x4suqGtQHrI5+3hWWv\nZiMnNseJyFJgIDAN6FcmkYe3qOpuYLeIRIvLxVB3MpIsqBAPwHgRmUygezwx4/FlgJ/HO4yeJjDI\n2QR3ji6g549PHmMqJXYyEo9YBFylqqcCGwju+i3FKZ6szFKa3GIIwWAyN6jqZlW9U1ULqroJeI6g\nO1mayCNvNFUyElVdqaobwo9LgaHEjCcrszwAnAsgIsOALar6WkZaYiEi54nI5eH7NmAwcDP7JvLI\nG02VjERElojIUeHHTmAjMePJ7BJ9EbkO+HeCU3eXqepjmQiJiYgcBNwO9AcOIOiSPQr8gJJEHpmJ\nrIGIDAduAI4EdgGbgfOAhSRMRpIFFeKZA1wF/BXYQRDP1jjx2P0shuGILwN8w/AeM4thOGJmMQxH\nzCyG4YiZxTAcMbMYhiNmFsNw5P8BZiHypT85V60AAAAASUVORK5CYII=\n", 362 | "text/plain": [ 363 | "" 364 | ] 365 | }, 366 | "metadata": { 367 | "tags": [] 368 | } 369 | }, 370 | { 371 | "output_type": "display_data", 372 | "data": { 373 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAACCCAYAAACJkUn8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAG9VJREFUeJztnXvcHWV177+5kgsSAiEvIUAhFRYQ\nILEQNIFw0VjCxXJQ1BaRq9oWFFt7ikcRS1tu1Vpi0J42UBFt+LSSmGrAAxU4GgQ1odwKhMXdtCSG\nGEjCJZDkzds/nmex553s6+z97pl5Wd9/9vWZWfObmedZs541a4b09fXhOI7jlIuheRvgOI7jtI53\n3o7jOCXEO2/HcZwS4p234zhOCfHO23Ecp4R45+04jlNCvPMuOCJyr4g83IX1fDJDm+dF5JiBsCcL\nEjg2vj9dRL7V4eVPF5GHRGR04ru9RGSjiJxbo8242OYpEdm9k/a0gogcLyJPZ2h3oYj8WkQuFZEp\nIvK0iDzUQvuhInKPiMxpdd1OfbzzLjAiciiwEVglIjMHcD3DgK8O1PK7yOnAsQCqukRVz+/UgkVk\nKPDPwB+r6ubET18HXq7T9HBgd1U9QFXXd8qeLvIh4FJVvRI4GlijqtObbayq24HzgX9KDnpO+wzP\n2wCnLucAtwBvAGcDP7cfROSLwJ8AvwJuBC5R1f1EZCdCRzwXGAksUNWrYpvngauBC4B9gJtV9c+A\nHwPjROQJ4CRVfS6xnqHAXxNOYoBfABep6mvx83tF5DpgAnCTqn5JRIYD/wDMBoYBjwDnquomETkN\nuAIYCzwNnKmqvxGRy4HJwDTg5mjnvqq6LtoxL+rwReA6YE7cvp8ROoe5wBeALSIyHvhP4CxVnSMi\nu0V7pgG90c6/icvti9p+DtgT+IqqXltlX5wBrFfV5D44OW7HT6r8HxHZF1gI9ERtjwHWxW04FzgE\nOAr4RlzOduBiVb1TRPYj7O9r4/4aEu28DJgO3GGDUy1Na9j0t8DvxXWdr6r3ici3gadV9Yr4n2/H\n5ewCzAQOFpGPEAaiXUTkYVWd1uy+VNV5IvIL4BOEfed0APe8C0r0hj8ILAZ+AJwsIiPjb1OBSwgn\nx2zgI4mmlxA6hcOAqcAZInJq4vdjCSfkEcBnRGRvQufXq6oHJTvuyEeAk+L/pwK7An+a+P0I4Mj4\neqGITANOBPYHDgIOAB4DZorIFOC7wB+o6hTg/xM6VeNk4GRVnRd/S9r9v4DvEbzr2cChwMFxvR9V\n1aXAEuDrcUBKchXwsqoKoQO9MBXumaqq7yJ0aldF7dOcEZcPgIiMIQySn67yXwBUdRWhw10VtbUO\ndYiqiqr2AguAr6rqQcA1KT0mAL+Odj8C/CthQD8cOFNEfrsJTZPsB9yvqgcCXwO+Wcv2aP8lwHKC\nY3AiYXD8eey4W9mXAN+n/3HqtIl33sXlRGCFqm5S1dcJ3t0H4m/HAj9R1TWq+gaQjO1+APh7VX0z\nesffIQwCxs2q2quqq4G1BA+8HqcQPNXXYmdzI/C7id8XxuW9CPyUMDCsIwwgpwNjVPUyVb2D4B3/\nRFUfjW3/Afi9RGf5y0QHt4jQmSIivwNsVdUHVHUxcKSqbo3bvgKY0sQ2/D2Aqr5E6EiS2/Dd+PoA\nMAqYWGUZR8V1GV8maPlsg3VX49bE++mEQQngHvpvy3DClReEK4kVqvqbGH5ZA+xFY02TvJFY1/eA\n6SIyKoP9NLHe5L4E+CXwbhEZknF9TgoPmxSXcwne9ob4eTgwnuCJjwdeSvz3hcT7XYFrReSq+Hkn\ngvdkbEy87yWENeqxB/1jui/Tv3Nbl1r2eFVdLiKfAT4D3CQiS4ELo23HxhBCso1N5CW36d+Av4ud\ni3ndiMgewHWxQ99OCHXMoz7VtmGvlA2oaq+IQHVNJgIvRhsOJXReR6X/JCJXEwYtCF53NZLb+THg\nYhF5R1xvsnPrTcTXe4FXk7/F/9fT9MXUetfHGDTApvg6voaNjWhlXxJtGcGOx66TEe+8C0iM2R4P\n7KaqW+J3w4H/jp3XJmDnRJNJifergb9V1aR31w5rqZyQxPdrE593S7x/68RU1UXAohhv/hbw58BT\nwJ2qekZ6JbHTfAtVfUlElgPvI3TeH48/XQlsBQ5T1TdFZGEL27CqxjY0Q7JT/QDhimVVtHsccLqI\nTFbVLxDCC7Zdx9daoIhMBq4H3q2qD4nIAcCTLdq1mhqaViHZUe8aX19ix0G8mQ695nrT+9IZGDxs\nUkx+H7jbOm4AVd0G3AH8AcGTPkFEJsQJynMSbX8AfEJEhonIEBH5kojMbbC+rcDQ6P2luRU4S0TG\nxAHkAuC2pK0xHWwiIRZ9j4icJyKXRbtfAp4A+qL9s2O8FBE5SkS+XseuRcAngZGqaumSE4H/jB33\nNEIGhA1kW6l0Sult+FRc5wRCGOm2Kv+rx4sEDx5VvVpVd1fVPVV1T0Is+rMxI6MV9gBeA56I2pqN\nO9dt1Z9WNB0jInZVcAYhDPMmIQQzLbafQpgX6OR6IWzrVmBDnf84LeCddzE5hxA2SLMEOFtVlwM3\nAQ8CdwNLCZ0jhEmoXxEmCZ8gTOr9rMH61sT/rBKRWanfFgE/Av4DeBT4L2B+4vcVhMHkfuBaVX2c\nMIAcEXObVxLi33+nqmsInfGS+P03CB1fLZYQJi1vSXz3NeCPYvuLgD8jDFYfjjr8kYgsSi3nS8D4\neIm/DLgmatgKy4EZLbZpxMMEbZ8kZJYsJWTz/LTZBbSo6ROEieMnCJPOF8Xvrwf2E5GnCFk+af3a\nXS/Au4HlibCN0yZDvJ53ORGRIaraF9+fAlwRMyacAUBEfh/4lKq+N29byoiI3EzovBvNTzhN4p53\nCYlx79+IyG/F2fuPkMgBdwaEW4BJItJp73vQE0MrswkevtMhvPMuIfHGlUuBuwiX3LsBl+dp02An\npkl+DPhHv1OweeJNXjcCn0jc2OV0gMxhExG5FngPIdb6WVVd0aCJ4ziO0yEyed4ichxwgKrOJGQf\nzG/QxHEcx+kgWcMm7yNmQ6jqSsJM/i4ds8pxHMepS9abdPYkpI4Z6+J3m6r9eevWrX3Dhw/noYdC\nJclt27ZlXO3AMXXqVB577LG8zahJ0r7p00NRt0cfDXcmF0HPMumXpChallW/NHnoOVi0q8aMGTNq\nlhPIFPMWkQXAbar6g/j5Z4QKZVXvDuvr6+sbMsRLGjiO47RIzY4zq+e9muBpG3sRbvSovvbYca9b\nF8pgbN9evDz9np4e1q5t9Y7p7pG0r6enByiWnmXSL/095K9lWfWr9j/orp6DRbtabWuRNeb974Tb\na63i22pVfSXjshzHcZwWyeR5xwLu/yEi9xEqu11U7/9btmxh5MiRPPtsqJ7Z29ubZbUDSk9PD888\n80zeZtQkad/48aFuUJH0LJN+SYqiZVn1S5OHnoNFu1pta5G5qqCq/p+sbR3HcZz26EpJ2M2bNzNy\n5EiWLl361mcAmyzNOpnZbvvkcmbNmsXixYvbXk479tRrn7Rv6tSpAIXSM2lfJ/dLO8tJtq+1f4ui\nZS37iqLlrFmzWLRoUcP2eejZyrmbh57V7Gu2/axZ6TpxFfz2eMdxnBLSFc/bRpeddtoJqOR+Wjxs\n5MiRmZa7devWfssfPry1zbGZcFuO2TF0aGtjmm1Hu9tjutj2DBvW/4Euo0aN6vd70fS09dpyRowY\nARRHT9MvSZG0TNqXPjbz1tJsqHVsGnnpWW3fJslbz7R9jc71ZnDP23Ecp4R0xfO2UcVmTl9//XWg\nEvfJMupAZfSz0avVUdTa2Sg4adKken9v2p6s25POiU1vj+lXVD332is8FtL0zGpH2p5O6Vlt5r5I\nWibtSx+beWsJ/c+PWtuTl571sjKS7fLSM21fo3O9GdzzdhzHKSG5PoDYRp92PausnqJ5A+3molr7\nTm2P0er2FE3Pdr2sTuuZpa1rWSGppx+b+Z7r4J634zhOKcnV884y2iSxUS/rctqZ6a1mR7vYdmTd\nHtezP+3o6VruyNChQ/3YLMCx+dYyOmKJ4ziO01UGRcw760x2UWPeWbcn7/UXNa6Ypeyxa7kj27dv\n92OzIOc6uOftOI5TSrriedtoZTVtLffTyBr3sdGr3fZGuzWBO2VPrfZmX1H1XLOmf0n3duOcndaz\n2v4tkpZJ+zqRjdCuPWnWrVvXsH1eejY6d/PWM21fJ/aHe96O4zglpKu1Tez+/nRFrSwxyWrLb3U5\n6YpeVo+h23Y0uxyzr6h6jhkzpt/nvOyotZxq+7dIWibtSx+bRdBy9OjRDZeTl56Nzt289axlXzu6\nuOftOI5TQrrqeVslryI8c7Ea7XreA03a8y6anu1UrOsG9TzvImhZluOvHnnpORi0axX3vB3HcUpI\nVzxvG303bNgA7DgDXRTWr1+ftwl1MfuKqmdZ9EtSJC3LqF+avPQcDNq1invejuM4JaSr9byt3vMb\nb7zRjdW2zN577523CXUx+4qqZ1n0S1IkLcuoX5q89BwM2rWKe96O4zglxDtvx3GcEuKdt+M4Tgnx\nzttxHKeEeOftOI5TQrzzdhzHKSHeeTuO45QQ77wdx3FKiHfejuM4JcQ7b8dxnBLS1O3xIvIVYHb8\n/9XACuC7wDBgDfBxVX1zoIx0HMdx+tPQ8xaRE4BDVXUmMBeYB/wV8E1VnQ08DZw/oFY6juM4/Wgm\nbLIM+HB8vwEYCxwP/DB+txSY03HLHMdxnJo0DJuoai/wWvx4AfAj4MREmORFYNLAmOc4juNUY0iz\nD74UkdOALwK/CzylqhPj9+8EvqOqs2q13b59e187j7h3HMd5mzKk1g/NTlieCFwKzFXVjSLyqoiM\nVtXNwGRgdb32mzdvZuzYsSxYsAAoTv3pJBdffDHz58/P24yaJO274IILAFi4cCFQDD3LpF+SomhZ\nVv3S5KHnYNGuVttaNDNhOQ74KnCqqr4Uv74T+FB8/yHg9kyWOY7jOJloxvP+KDAB+J6I2HfnADeI\nyB8CvwJuammlw8NqLZSyZcuWVpq/xahRowDYunUrAL29vZmWY5g99qRn8xqaDS11G9PRntRtOrSK\n6Wj7oShPo+8G9rRze22V9DGzefPmzhj2NsN0HDlyJFCMq8mi08yE5QJgQZWf3t95cxzHcZxm6Moz\nLNPMmhXmNkePHg3Aj3/8Y6B5j89G51NPPRWAxx9/vN9rVuy5e0cffTQAt98eokEbN25sa7kDxXHH\nHQfAq6++CsCyZcuA5q8Uxo4dC8App5wCwPLlywF4/vnnO2lmoRkzZgwAPT09QOvbblejU6dOBeDW\nW28F3HNslcMPPxyA/fffH4ClS5cCsG3bttxsKjqeAuI4jlNCcvG8jddffx1oPd5oT6h+882Qap41\n1pvGPP/XXgtp7RZTLirmYWf18mz7rH2nYt22P20/2VxEEecOzLasMW/bRjtmBiol1vaV7aPBNi9h\nutlVpG3vQHneg0FP97wdx3FKSC6upcVmLbvBYtjNYp7ikiVL+n3frKdso6yN8jvvvDMAzz77LAD3\n3nsvAPvss09Ly+02d999NwCvvPIKUMk6aZZNmzYBsHjxYqDi/Zg3mZVdd90VgBNOOKGfnRs2bGhr\nuQOBZYe88MILmdo/9thjADzwwANA68dyI+zYO+mkkwB48sknAVDVjq4nbx588EGgcjVt2TudZjDp\n6Z634zhOCcnFpTRP1+JZWWO2EyZMACqeZ7N53uZxf//73wfgmGOOAeCee+4B4L777gPgzDPPBOCA\nAw7IZN9As8suuwCV7W419m+e9h577AFUPPF243/veMc7+n3utDfaSSzWnTVWbRk7lrVise9OYZ5i\nOqd/sGD627FsV+MDlS8/mPR0z9txHKeEdNXztlG23XiWxWRt9ExnNTTCvKVp06YBIad0/vz5rFy5\nEqjEuu1/RcN0NK8h6x2WaR3N+2zX837uuecAePjhh/vZa95VkYqUZfW80xk11t4yqDqVWWMx4Ouv\nvx6o7Gtbvx2j7c5T5EVa/4G+U3Uw6Vmcs8hxHMdpmlxi3u3WIrFR0pbTqpdjo+pBBx0EVLJfLBb+\nrne9C4CJEydmsq9b2JxBu15ep/LkDbvT9aGHHgIqcwZHHHEEUEzP22LW5jk3286wuL5dxXRK07Vr\n1wJwww039FvvnnvuCcDpp58OVDJ8yoodw+Z5D1Se92DSszhnkeM4jtM0XfW8bXRNZ4e06onZaNxu\nzRGrrXLeeecBlVopBx98MFDc/G6LSVt2SNa7+2xm316z3mWYxvbvM888A8D48eM7uvxOYlq2mvFk\nx7Idg/baqasKs8s0NI/R6u7MnTsXqMwjlBXT8aWXQrVpuyru9LEyGPV0z9txHKeEdNW1NI/ZRj+r\nI23V/JrF4pJWAW7fffcFsmeHmF12l51ViCsq5q3YXWGmY6sxevPYTccpU6YAlSuSrFgev71arLuI\nVzIWm7armGY9Z2v36KOPApWryOnTpwPtZyuYHeYJ2vJs+WXyEOth2R+PPPIIUDlGbDs7xWDU0z1v\nx3GcEtJVVyh9V1PWuJbN7Nto2m587OWXXwYqcbeik86JzRpnTWdIdCpea97M+98fntexfv16oJi1\nmc2Dtjj9uHHjmmpnmtlVisVQbf6g3asXwzKirrzySqBYmTqdwDxgu3q0uYOsdY8aMZj0LK/ljuM4\nb2NyCUKaB2Z3UU2aNCnTciwv2zIFLFe3VSzWbbmeZbi7Cnasw221XlrFvM5OVXRbtWoVALfddhtQ\niaUnnoFaGGz+wOL/zXreRvquYdsnnfK87W5Vu/v3wAMPBIqpZRbsXEvf7Ws6dtrzHkx6uuftOI5T\nQrrqeduMvM3sW/5vq9iobMvLWhnM2pvXZdX1ik66HnlWHW27TYdOZYNY3PL+++8HKlknRfRuss6X\nmGb2atkKnX5akF0N2d2qdudfEbXMgs052DFtx8pAPXVpMOnpnrfjOE4J6YrnbTO6u+22GwAzZ84M\nK8+Y5WBejnnK7cbFLr/8cgDOOussoBKzLRrmJVh80OqQZ71T1XS0PPt2dTSvxp5Cb/FEm0vIu15E\ntfWnK0w2W5PEPEPLYTcP3o7pLB590j7LXimSls2s045BmzuwLJJamI627HS1xmZ1bGRb3noOxPLd\n83YcxykhXfG8bfS0bIii3c2UrsFc1Ni32WXejXnMlhObN7vvvjsAp512GlDJwLDMi7x1rbZ+09Lu\nTs1Ty6R9lpFVJC2bWWf6XO+Wno1sy1vPgVi+e96O4zglpKvZJhZPLIqnaJg9lgVTVNK6FU1P87zt\nNU3edtZbfxG0TK67iFq2ss5u69loPXnrORDLd8/bcRynhHTF87YcTpvxHajn07XLmjVr8jahLmZf\nUfUsi35JiqRlGfVLk5eeg0G7VnHP23Ecp4R0Nc+7p6cHqOQDF42sNVa6hdlXVD3Lol+SImlZRv3S\n5KXnYNCuVZrqvEVkNPAo8NfAXcB3gWHAGuDjqlqM3sNxHOdtQrOe95cAK3b9V8A3VfUWEbkKOB/4\nv80sJF1BrGgU1S4jbV/R9CyKHbWoZ18RtCyzfrX+261tGkzaNUvDmLeIHAQcAtwWvzoe+GF8vxSY\n03GrHMdxnLo043l/Dfg0cE78PDYRJnkRaBjMsXoPkydPBip3O9mdjel6Bs3W2Kj15G+rp9BsrQ/L\nwbS7oOxOy1brgw/09tizOouqp9lnelrNlaLoaXf9JbenSFqafkmKoiXA3nvv3XB78tKzmnbVyEvP\ntH2d6LuG1Cu9KCJnA/uq6hUicjnwPPAVVZ0Yf38n8B1VnVVvJX19fX3tPqrMcRznbUjNjrOR530K\nMEVETgX2Bt4EXhWR0aq6GZgMrG609m3btjFixAgefPBBoDL6LVu2DKjU47a61IccckijRQKVOyKt\nbrQNREceeSTQ/FNRVq5cydlnn828efOAyt1hs2fPBhrHq2wUte2xuglWm/iwww5ryo562zNnzhxW\nrFgBVJ4RaU8uL4KeM2bMeMs+e0qJPRO0CHp+/vOf55prrtlhe4qiZVK/JEXRcsaMGdx1110NtycP\nPZPnRiPy0LPavm32+JgxY0ZNW+p23qr6UXuf8LxnAR8C/jm+3l5vGY7jOE7nqRs2SZLovO8AvgOM\nAn4FnKeqzRVBdhzHcTpC05234ziOUxz89njHcZwS4p234zhOCfHO23Ecp4R45+04jlNCvPN2HMcp\nIV0pCSsi1wLvAfqAz6pqcxn1A4iIfAWYTdDgamAFBauWWORqjiLyMeASYBvwZeCRotgnIjsT0lnH\nAzsBfwn8mlBArQ94RFX/OAe7DgV+AFyrqt8QkX2oolnU9k+A7cACVf2nHO27ERgBbAXOUtVfF8W+\nxPcnArer6pD4uRD2icgI4CbgncArwBmq+nKn7Btwz1tEjgMOUNWZwAXA/IFeZyNE5ATg0GjTXGAe\nlWqJs4GnCdUS86ZaNcfc7ROR3YG/AI4BTgVOK5J9wLmAquoJwBnA1wn7+LOqejQwTkRO6qZBIjIW\nuI4wCBs7aBb/92VCwbfjgT8Vkd1ysu8KQudyHLAE+FzB7ENERgFfIAx+FMy+TwLrVPUo4F+B2Z20\nrxthk/cB/wagqiuB8SKySxfWW49lwIfj+w3AWApWLbHg1RznAHeq6iuqukZVP0Wx7PsNYE+aHU8Y\nAPdPXPHlYd+bwMn0LydxPDtq9m5ghapujCUo7gWOzsm+C4HF8f06gqZFsg/gi8A3AXvCb5Hs+wCw\nEEBVF6jqDztpXzc67z0JO95YF7/LDVXtVdXX4scLgB+RoVriAPM14HOJz0Wybz9gjIj8UETuEZH3\nUSD7VPVfgH1F5GnCQP2/gZcTf+m6faq6LZ6sSapplj5fumJrNftU9TVV7RWRYcBFwM1Fsk9EDgSm\nqeotia8LYx/hPDlJRH4iIv8SPeyO2ZfHhGVhyguKyGmEzvvTqZ9ytTFWc/y5qj5X4y95aziE4IV9\nkBCiuJH+NuWt31nAKlV9J/BeQh2eJHnrV41aNuWt5TBCXP5uVb2ryl/ytO9a+js41cjTviGE8N3x\nhLmrL9T4Tya60Xmvpr+nvRcxPpUncZLjUuAkVd1IrJYYf26qWuIAcgpwmoj8AvgEcBnFsm8tcF/0\nNp4hTMa8UiD7jibU4EFVHwZGAxMSv+dtn1Ftn6bPl7xtvRF4SlX/Mn4uhH0iMhk4CFgYz5NJIvLT\notgXWQv8NL6/A5hKB+3rRuf974RJI0Tkd4DVqvpKF9ZbExEZB3wVOFVVbULwTkKVRMi5WqKqflRV\nZ6jqe4AbCNkmhbGPsE/fKyJD4+TlzhTLvqcJsUVE5LcIg8tKETkm/v5BilENs5pmvwRmiMiuMWvm\naOCePIyLWRFbVPUvEl8Xwj5VfUFVf1tV3xPPkzVxYrUQ9kX+HyEhAuAIQDtpX1cKU4nINcCxhNSY\ni6I3lBsi8ingcuDJxNfnEDrKQlVLLGo1RxH5Q0LICUJWwgoKYl88Kb4F9BBSQS8jpAr+I8Fh+aWq\nNrrc7rRNRxDmMfYjpN29AHwM+DYpzUTkDODPCWmN16nqwpzsmwi8AWyKf3tcVS8skH0fNOdLRJ5X\n1f3i+6LYdyYh02kS8Cpwjqqu7ZR9XlXQcRynhPgdlo7jOCXEO2/HcZwS4p234zhOCfHO23Ecp4R4\n5+04jlNCvPN2HMcpId55O47jlBDvvB3HcUrI/wBKwUdPeQKDIgAAAABJRU5ErkJggg==\n", 374 | "text/plain": [ 375 | "" 376 | ] 377 | }, 378 | "metadata": { 379 | "tags": [] 380 | } 381 | } 382 | ] 383 | }, 384 | { 385 | "metadata": { 386 | "id": "ejdlsSzJovVP", 387 | "colab_type": "text" 388 | }, 389 | "cell_type": "markdown", 390 | "source": [ 391 | "# Build an agent\n", 392 | "\n", 393 | "---\n", 394 | "\n", 395 | "\n", 396 | "We now have to build an agent for actor-critic training - a convolutional neural network that converts states into action probabilities and state values .\n", 397 | "\n", 398 | "Your assignment here is to build and apply a neural network - with any framework you want.\n", 399 | "\n", 400 | "For starters, we want you to implement this architecture: \n", 401 | "\n", 402 | "After you get above 50 points, we encourage you to experiment with model architecture to score even better." 403 | ] 404 | }, 405 | { 406 | "metadata": { 407 | "id": "P0Qg7FRfkokL", 408 | "colab_type": "code", 409 | "colab": {} 410 | }, 411 | "cell_type": "code", 412 | "source": [ 413 | "import tensorflow as tf\n", 414 | "tf.reset_default_graph()\n", 415 | "sess = tf.InteractiveSession()" 416 | ], 417 | "execution_count": 0, 418 | "outputs": [] 419 | }, 420 | { 421 | "metadata": { 422 | "id": "tRb-25lKoAjy", 423 | "colab_type": "code", 424 | "outputId": "a510b615-7529-441b-bd6d-d45a9be619fa", 425 | "colab": { 426 | "base_uri": "https://localhost:8080/", 427 | "height": 34 428 | } 429 | }, 430 | "cell_type": "code", 431 | "source": [ 432 | "from keras.layers import Conv2D, Dense, Flatten, Input\n", 433 | "import keras\n", 434 | "from keras.models import Model, Sequential\n", 435 | "\n", 436 | "class Agent:\n", 437 | " def __init__(self, name, state_shape, n_actions, reuse=False):\n", 438 | " \"\"\"A simple actor-critic agent\"\"\"\n", 439 | " \n", 440 | " with tf.variable_scope(name, reuse=reuse):\n", 441 | " \n", 442 | " # Prepare neural network architecture\n", 443 | " ### Your code here: prepare any necessary layers, variables, etc.\n", 444 | "# self.network = Sequential()\n", 445 | " \n", 446 | "# # similar architecture as the dqn\n", 447 | "# self.network.add(Conv2D(32, (3, 3), strides=2, activation='relu', input_shape=state_shape))\n", 448 | "# self.network.add(Conv2D(32, (3, 3), strides=2, activation='relu'))\n", 449 | "# self.network.add(Conv2D(32, (3, 3), strides=2, activation='relu'))\n", 450 | "# self.network.add(Flatten())\n", 451 | "# self.network.add(Dense(128, activation='relu'))\n", 452 | " \n", 453 | "# # logits for n_actions and 1 unit for the state value\n", 454 | "# self.network.add(Dense(n_actions+1, activation='linear'))\n", 455 | "\n", 456 | "\n", 457 | " ####\n", 458 | " inputs = Input(shape=state_shape)\n", 459 | " x = Conv2D(32, (3, 3), strides=2, activation='relu')(inputs)\n", 460 | " x = Conv2D(32, (3, 3), strides=2, activation='relu')(x)\n", 461 | " x = Conv2D(32, (3, 3), strides=2, activation='relu')(x)\n", 462 | " x = Flatten()(x)\n", 463 | " x = Dense(128, activation='relu')(x)\n", 464 | " \n", 465 | " # two different output layers\n", 466 | " logits = Dense(n_actions, activation='linear')(x)\n", 467 | " state_value = Dense(1, activation='linear')(x)\n", 468 | " \n", 469 | " self.network = Model(inputs=inputs, outputs=[logits, state_value])\n", 470 | " \n", 471 | " \n", 472 | " # prepare a graph for agent step\n", 473 | " self.state_t = tf.placeholder('float32', [None,] + list(state_shape))\n", 474 | " self.agent_outputs = self.symbolic_step(self.state_t)\n", 475 | " \n", 476 | " def symbolic_step(self, state_t):\n", 477 | " \"\"\"Takes agent's previous step and observation, returns next state and whatever it needs to learn (tf tensors)\"\"\"\n", 478 | " \n", 479 | " # Apply neural network\n", 480 | " ### Your code here: apply agent's neural network to get policy logits and state values.\n", 481 | "# network_output = self.network(state_t)\n", 482 | "\n", 483 | "# logits = network_output[:,1:]\n", 484 | "# state_value = network_output[:,0]\n", 485 | " \n", 486 | " \n", 487 | " logits, state_value = self.network(state_t)\n", 488 | " state_value = state_value[:, 0]\n", 489 | " \n", 490 | " assert tf.is_numeric_tensor(state_value) and state_value.shape.ndims == 1, \\\n", 491 | " \"please return 1D tf tensor of state values [you got %s]\" % repr(state_value)\n", 492 | " assert tf.is_numeric_tensor(logits) and logits.shape.ndims == 2, \\\n", 493 | " \"please return 2d tf tensor of logits [you got %s]\" % repr(logits)\n", 494 | " # hint: if you triggered state_values assert with your shape being [None, 1], \n", 495 | " # just select [:, 0]-th element of state values as new state values\n", 496 | " \n", 497 | " return (logits, state_value)\n", 498 | " \n", 499 | " def step(self, state_t):\n", 500 | " \"\"\"Same as symbolic step except it operates on numpy arrays\"\"\"\n", 501 | " sess = tf.get_default_session()\n", 502 | " return sess.run(self.agent_outputs, {self.state_t: state_t})\n", 503 | " \n", 504 | " def sample_actions(self, agent_outputs):\n", 505 | " \"\"\"pick actions given numeric agent outputs (np arrays)\"\"\"\n", 506 | " logits, state_values = agent_outputs\n", 507 | " policy = np.exp(logits) / np.sum(np.exp(logits), axis=-1, keepdims=True)\n", 508 | " return np.array([np.random.choice(len(p), p=p) for p in policy])" 509 | ], 510 | "execution_count": 0, 511 | "outputs": [ 512 | { 513 | "output_type": "stream", 514 | "text": [ 515 | "Using TensorFlow backend.\n" 516 | ], 517 | "name": "stderr" 518 | } 519 | ] 520 | }, 521 | { 522 | "metadata": { 523 | "id": "bjtQetyEoHMD", 524 | "colab_type": "code", 525 | "colab": {} 526 | }, 527 | "cell_type": "code", 528 | "source": [ 529 | "agent = Agent(\"agent\", obs_shape, n_actions)\n", 530 | "sess.run(tf.global_variables_initializer())" 531 | ], 532 | "execution_count": 0, 533 | "outputs": [] 534 | }, 535 | { 536 | "metadata": { 537 | "id": "lRUs5NBuoNWB", 538 | "colab_type": "code", 539 | "outputId": "52863a41-e18d-49f2-cbf0-c9623e1ae7c4", 540 | "colab": { 541 | "base_uri": "https://localhost:8080/", 542 | "height": 211 543 | } 544 | }, 545 | "cell_type": "code", 546 | "source": [ 547 | "state = [env.reset()]\n", 548 | "logits, value = agent.step(state)\n", 549 | "print(\"action logits:\\n\", logits)\n", 550 | "print(\"state values:\\n\", value)" 551 | ], 552 | "execution_count": 0, 553 | "outputs": [ 554 | { 555 | "output_type": "stream", 556 | "text": [ 557 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 558 | " if issubdtype(ts, int):\n", 559 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 560 | " elif issubdtype(type(size), float):\n" 561 | ], 562 | "name": "stderr" 563 | }, 564 | { 565 | "output_type": "stream", 566 | "text": [ 567 | "action logits:\n", 568 | " [[-0.04011825 -0.03369963 -0.05471736 -0.01447331 -0.03145602 0.0269252\n", 569 | " 0.06747915 -0.00501616 -0.00702886 0.01228065 -0.05536278 0.02292078\n", 570 | " -0.05011498 0.01017556]]\n", 571 | "state values:\n", 572 | " [0.00383585]\n" 573 | ], 574 | "name": "stdout" 575 | } 576 | ] 577 | }, 578 | { 579 | "metadata": { 580 | "id": "7OqIoeqmo4n5", 581 | "colab_type": "text" 582 | }, 583 | "cell_type": "markdown", 584 | "source": [ 585 | "# Let's play!\n", 586 | "\n", 587 | "---\n", 588 | "\n", 589 | "\n", 590 | "Let's build a function that measures agent's average reward." 591 | ] 592 | }, 593 | { 594 | "metadata": { 595 | "id": "9AFcqPHKoQ_i", 596 | "colab_type": "code", 597 | "colab": {} 598 | }, 599 | "cell_type": "code", 600 | "source": [ 601 | "def evaluate(agent, env, n_games=1):\n", 602 | " \"\"\"Plays an a game from start till done, returns per-game rewards \"\"\"\n", 603 | "\n", 604 | " game_rewards = []\n", 605 | " for _ in range(n_games):\n", 606 | " state = env.reset()\n", 607 | " \n", 608 | " total_reward = 0\n", 609 | " while True:\n", 610 | " action = agent.sample_actions(agent.step([state]))[0]\n", 611 | " state, reward, done, info = env.step(action)\n", 612 | " total_reward += reward\n", 613 | " if done: break\n", 614 | " \n", 615 | " game_rewards.append(total_reward)\n", 616 | " return game_rewards\n" 617 | ], 618 | "execution_count": 0, 619 | "outputs": [] 620 | }, 621 | { 622 | "metadata": { 623 | "id": "PUPl5-sKoVpg", 624 | "colab_type": "code", 625 | "outputId": "5ee5f221-e9de-4787-c585-9870c035f799", 626 | "colab": { 627 | "base_uri": "https://localhost:8080/", 628 | "height": 263 629 | } 630 | }, 631 | "cell_type": "code", 632 | "source": [ 633 | "env_monitor = gym.wrappers.Monitor(env, directory=\"kungfu_videos\", force=True)\n", 634 | "rw = evaluate(agent, env_monitor, n_games=3,)\n", 635 | "env_monitor.close()\n", 636 | "print (rw)" 637 | ], 638 | "execution_count": 0, 639 | "outputs": [ 640 | { 641 | "output_type": "stream", 642 | "text": [ 643 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 644 | " if issubdtype(ts, int):\n", 645 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 646 | " elif issubdtype(type(size), float):\n", 647 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 648 | " if issubdtype(ts, int):\n", 649 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 650 | " elif issubdtype(type(size), float):\n", 651 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 652 | " if issubdtype(ts, int):\n", 653 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 654 | " elif issubdtype(type(size), float):\n" 655 | ], 656 | "name": "stderr" 657 | }, 658 | { 659 | "output_type": "stream", 660 | "text": [ 661 | "[3.0, 2.0, 9.0]\n" 662 | ], 663 | "name": "stdout" 664 | } 665 | ] 666 | }, 667 | { 668 | "metadata": { 669 | "id": "wcr9OW2IoZU1", 670 | "colab_type": "code", 671 | "outputId": "6c3f18f1-dfd3-4a15-af17-5fd73ac8fc81", 672 | "colab": { 673 | "resources": { 674 | "http://localhost:8080/kungfu_videos/openaigym.video.0.88.video000000.mp4": { 675 | "data": "CjwhRE9DVFlQRSBodG1sPgo8aHRtbCBsYW5nPWVuPgogIDxtZXRhIGNoYXJzZXQ9dXRmLTg+CiAgPG1ldGEgbmFtZT12aWV3cG9ydCBjb250ZW50PSJpbml0aWFsLXNjYWxlPTEsIG1pbmltdW0tc2NhbGU9MSwgd2lkdGg9ZGV2aWNlLXdpZHRoIj4KICA8dGl0bGU+RXJyb3IgNDA0IChOb3QgRm91bmQpISExPC90aXRsZT4KICA8c3R5bGU+CiAgICAqe21hcmdpbjowO3BhZGRpbmc6MH1odG1sLGNvZGV7Zm9udDoxNXB4LzIycHggYXJpYWwsc2Fucy1zZXJpZn1odG1se2JhY2tncm91bmQ6I2ZmZjtjb2xvcjojMjIyO3BhZGRpbmc6MTVweH1ib2R5e21hcmdpbjo3JSBhdXRvIDA7bWF4LXdpZHRoOjM5MHB4O21pbi1oZWlnaHQ6MTgwcHg7cGFkZGluZzozMHB4IDAgMTVweH0qID4gYm9keXtiYWNrZ3JvdW5kOnVybCgvL3d3dy5nb29nbGUuY29tL2ltYWdlcy9lcnJvcnMvcm9ib3QucG5nKSAxMDAlIDVweCBuby1yZXBlYXQ7cGFkZGluZy1yaWdodDoyMDVweH1we21hcmdpbjoxMXB4IDAgMjJweDtvdmVyZmxvdzpoaWRkZW59aW5ze2NvbG9yOiM3Nzc7dGV4dC1kZWNvcmF0aW9uOm5vbmV9YSBpbWd7Ym9yZGVyOjB9QG1lZGlhIHNjcmVlbiBhbmQgKG1heC13aWR0aDo3NzJweCl7Ym9keXtiYWNrZ3JvdW5kOm5vbmU7bWFyZ2luLXRvcDowO21heC13aWR0aDpub25lO3BhZGRpbmctcmlnaHQ6MH19I2xvZ297YmFja2dyb3VuZDp1cmwoLy93d3cuZ29vZ2xlLmNvbS9pbWFnZXMvbG9nb3MvZXJyb3JwYWdlL2Vycm9yX2xvZ28tMTUweDU0LnBuZykgbm8tcmVwZWF0O21hcmdpbi1sZWZ0Oi01cHh9QG1lZGlhIG9ubHkgc2NyZWVuIGFuZCAobWluLXJlc29sdXRpb246MTkyZHBpKXsjbG9nb3tiYWNrZ3JvdW5kOnVybCgvL3d3dy5nb29nbGUuY29tL2ltYWdlcy9sb2dvcy9lcnJvcnBhZ2UvZXJyb3JfbG9nby0xNTB4NTQtMngucG5nKSBuby1yZXBlYXQgMCUgMCUvMTAwJSAxMDAlOy1tb3otYm9yZGVyLWltYWdlOnVybCgvL3d3dy5nb29nbGUuY29tL2ltYWdlcy9sb2dvcy9lcnJvcnBhZ2UvZXJyb3JfbG9nby0xNTB4NTQtMngucG5nKSAwfX1AbWVkaWEgb25seSBzY3JlZW4gYW5kICgtd2Via2l0LW1pbi1kZXZpY2UtcGl4ZWwtcmF0aW86Mil7I2xvZ297YmFja2dyb3VuZDp1cmwoLy93d3cuZ29vZ2xlLmNvbS9pbWFnZXMvbG9nb3MvZXJyb3JwYWdlL2Vycm9yX2xvZ28tMTUweDU0LTJ4LnBuZykgbm8tcmVwZWF0Oy13ZWJraXQtYmFja2dyb3VuZC1zaXplOjEwMCUgMTAwJX19I2xvZ297ZGlzcGxheTppbmxpbmUtYmxvY2s7aGVpZ2h0OjU0cHg7d2lkdGg6MTUwcHh9CiAgPC9zdHlsZT4KICA8YSBocmVmPS8vd3d3Lmdvb2dsZS5jb20vPjxzcGFuIGlkPWxvZ28gYXJpYS1sYWJlbD1Hb29nbGU+PC9zcGFuPjwvYT4KICA8cD48Yj40MDQuPC9iPiA8aW5zPlRoYXTigJlzIGFuIGVycm9yLjwvaW5zPgogIDxwPiAgPGlucz5UaGF04oCZcyBhbGwgd2Uga25vdy48L2lucz4K", 676 | "ok": false, 677 | "headers": [ 678 | [ 679 | "content-length", 680 | "1449" 681 | ], 682 | [ 683 | "content-type", 684 | "text/html; charset=utf-8" 685 | ] 686 | ], 687 | "status": 404, 688 | "status_text": "Not Found" 689 | } 690 | }, 691 | "base_uri": "https://localhost:8080/", 692 | "height": 501 693 | } 694 | }, 695 | "cell_type": "code", 696 | "source": [ 697 | "from IPython.display import HTML\n", 698 | "import os\n", 699 | "\n", 700 | "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./kungfu_videos/\")))\n", 701 | "\n", 702 | "HTML(\"\"\"\n", 703 | "\n", 706 | "\"\"\".format(\"./kungfu_videos/\"+video_names[0])) #this may or may not be _last_ video. Try other indices\n" 707 | ], 708 | "execution_count": 0, 709 | "outputs": [ 710 | { 711 | "output_type": "execute_result", 712 | "data": { 713 | "text/html": [ 714 | "\n", 715 | "\n" 718 | ], 719 | "text/plain": [ 720 | "" 721 | ] 722 | }, 723 | "metadata": { 724 | "tags": [] 725 | }, 726 | "execution_count": 24 727 | } 728 | ] 729 | }, 730 | { 731 | "metadata": { 732 | "id": "8MmDLcuUpGMj", 733 | "colab_type": "text" 734 | }, 735 | "cell_type": "markdown", 736 | "source": [ 737 | "# Training on parallel games\n", 738 | "\n", 739 | "---\n", 740 | "\n", 741 | "\n", 742 | "\n", 743 | "![alt text](https://camo.githubusercontent.com/f894eba2ee745652f1fc95484cfa45bf26decb8e/68747470733a2f2f6769746875622e636f6d2f79616e646578646174617363686f6f6c2f50726163746963616c5f524c2f7261772f6d61737465722f7965745f616e6f746865725f7765656b2f5f7265736f757263652f2f656e765f706f6f6c2e706e67)\n", 744 | "\n", 745 | "To make actor-critic training more stable, we shall play several games in parallel. This means ya'll have to initialize several parallel gym envs, send agent's actions there and .reset() each env if it becomes terminated. To minimize learner brain damage, we've taken care of them for ya - just make sure you read it before you use it." 746 | ] 747 | }, 748 | { 749 | "metadata": { 750 | "id": "QX-uaGwCojPt", 751 | "colab_type": "code", 752 | "colab": {} 753 | }, 754 | "cell_type": "code", 755 | "source": [ 756 | "class EnvBatch:\n", 757 | " def __init__(self, n_envs = 10):\n", 758 | " \"\"\" Creates n_envs environments and babysits them for ya' \"\"\"\n", 759 | " self.envs = [make_env() for _ in range(n_envs)]\n", 760 | " \n", 761 | " def reset(self):\n", 762 | " \"\"\" Reset all games and return [n_envs, *obs_shape] observations \"\"\"\n", 763 | " return np.array([env.reset() for env in self.envs])\n", 764 | " \n", 765 | " def step(self, actions):\n", 766 | " \"\"\"\n", 767 | " Send a vector[batch_size] of actions into respective environments\n", 768 | " :returns: observations[n_envs, *obs_shape], rewards[n_envs], done[n_envs,], info[n_envs]\n", 769 | " \"\"\"\n", 770 | " results = [env.step(a) for env, a in zip(self.envs, actions)]\n", 771 | " new_obs, rewards, done, infos = map(np.array, zip(*results))\n", 772 | " \n", 773 | " # reset environments automatically\n", 774 | " for i in range(len(self.envs)):\n", 775 | " if done[i]:\n", 776 | " new_obs[i] = self.envs[i].reset()\n", 777 | " \n", 778 | " return new_obs, rewards, done, infos" 779 | ], 780 | "execution_count": 0, 781 | "outputs": [] 782 | }, 783 | { 784 | "metadata": { 785 | "id": "3JawlpBFo0Yb", 786 | "colab_type": "code", 787 | "outputId": "93048722-0185-4369-8110-c6b96bf39c1a", 788 | "colab": { 789 | "base_uri": "https://localhost:8080/", 790 | "height": 350 791 | } 792 | }, 793 | "cell_type": "code", 794 | "source": [ 795 | "env_batch = EnvBatch(10)\n", 796 | "\n", 797 | "batch_states = env_batch.reset()\n", 798 | "\n", 799 | "batch_actions = agent.sample_actions(agent.step(batch_states))\n", 800 | "\n", 801 | "batch_next_states, batch_rewards, batch_done, _ = env_batch.step(batch_actions)\n", 802 | "\n", 803 | "print(\"State shape:\", batch_states.shape)\n", 804 | "print(\"Actions:\", batch_actions[:3])\n", 805 | "print(\"Rewards:\", batch_rewards[:3])\n", 806 | "print(\"Done:\", batch_done[:3])\n" 807 | ], 808 | "execution_count": 0, 809 | "outputs": [ 810 | { 811 | "output_type": "stream", 812 | "text": [ 813 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 814 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 815 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 816 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 817 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 818 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 819 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 820 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 821 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 822 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 823 | "State shape: (10, 42, 42, 4)\n", 824 | "Actions: [13 11 8]\n", 825 | "Rewards: [0. 0. 0.]\n", 826 | "Done: [False False False]\n" 827 | ], 828 | "name": "stdout" 829 | }, 830 | { 831 | "output_type": "stream", 832 | "text": [ 833 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 834 | " if issubdtype(ts, int):\n", 835 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 836 | " elif issubdtype(type(size), float):\n" 837 | ], 838 | "name": "stderr" 839 | } 840 | ] 841 | }, 842 | { 843 | "metadata": { 844 | "id": "UwEPCL46pfHO", 845 | "colab_type": "text" 846 | }, 847 | "cell_type": "markdown", 848 | "source": [ 849 | "# Actor-critic\n", 850 | "\n", 851 | "---\n", 852 | "\n", 853 | "\n", 854 | "Here we define a loss functions and learning algorithms as usual." 855 | ] 856 | }, 857 | { 858 | "metadata": { 859 | "id": "0C6zD_V1o35d", 860 | "colab_type": "code", 861 | "colab": {} 862 | }, 863 | "cell_type": "code", 864 | "source": [ 865 | "# These placeholders mean exactly the same as in \"Let's try it out\" section above\n", 866 | "states_ph = tf.placeholder('float32', [None,] + list(obs_shape)) \n", 867 | "next_states_ph = tf.placeholder('float32', [None,] + list(obs_shape))\n", 868 | "actions_ph = tf.placeholder('int32', (None,))\n", 869 | "rewards_ph = tf.placeholder('float32', (None,))\n", 870 | "is_done_ph = tf.placeholder('float32', (None,))" 871 | ], 872 | "execution_count": 0, 873 | "outputs": [] 874 | }, 875 | { 876 | "metadata": { 877 | "id": "Ktpq6pJgo-Dl", 878 | "colab_type": "code", 879 | "colab": {} 880 | }, 881 | "cell_type": "code", 882 | "source": [ 883 | "# logits[n_envs, n_actions] and state_values[n_envs, n_actions]\n", 884 | "logits, state_values = agent.symbolic_step(states_ph)\n", 885 | "next_logits, next_state_values = agent.symbolic_step(next_states_ph)\n", 886 | "next_state_values = next_state_values * (1 - is_done_ph)\n", 887 | "\n", 888 | "# probabilities and log-probabilities for all actions\n", 889 | "probs = tf.nn.softmax(logits) # [n_envs, n_actions]\n", 890 | "logprobs = tf.nn.log_softmax(logits) # [n_envs, n_actions]\n", 891 | "\n", 892 | "# log-probabilities only for agent's chosen actions\n", 893 | "logp_actions = tf.reduce_sum(logprobs * tf.one_hot(actions_ph, n_actions), axis=-1) # [n_envs,]" 894 | ], 895 | "execution_count": 0, 896 | "outputs": [] 897 | }, 898 | { 899 | "metadata": { 900 | "id": "b-aNY8wRpBk9", 901 | "colab_type": "code", 902 | "colab": {} 903 | }, 904 | "cell_type": "code", 905 | "source": [ 906 | "\n", 907 | "\n", 908 | "# compute advantage using rewards_ph, state_values and next_state_values\n", 909 | "gamma = 0.99\n", 910 | "advantage = rewards_ph + gamma*next_state_values - state_values\n", 911 | "\n", 912 | "assert advantage.shape.ndims == 1, \"please compute advantage for each sample, vector of shape [n_envs,]\"\n", 913 | "\n", 914 | "# compute policy entropy given logits_seq. Mind the \"-\" sign!\n", 915 | "entropy = -tf.reduce_sum(probs * logprobs, 1, name=\"entropy\")\n", 916 | "\n", 917 | "assert entropy.shape.ndims == 1, \"please compute pointwise entropy vector of shape [n_envs,] \"\n", 918 | "\n", 919 | "\n", 920 | "\n", 921 | "actor_loss = - tf.reduce_mean(logp_actions * tf.stop_gradient(advantage)) - 0.001 * tf.reduce_mean(entropy)\n", 922 | "\n", 923 | "# compute target state values using temporal difference formula. Use rewards_ph and next_step_values\n", 924 | "target_state_values = rewards_ph+gamma*next_state_values\n", 925 | "\n", 926 | "critic_loss = tf.reduce_mean((state_values - tf.stop_gradient(target_state_values))**2 )\n", 927 | "\n", 928 | "train_step = tf.train.AdamOptimizer(1e-4).minimize(actor_loss + critic_loss)\n", 929 | "sess.run(tf.global_variables_initializer())" 930 | ], 931 | "execution_count": 0, 932 | "outputs": [] 933 | }, 934 | { 935 | "metadata": { 936 | "id": "Z9qCcr3cpEtw", 937 | "colab_type": "code", 938 | "outputId": "44c23e6f-e622-482a-b68c-cae6959aff63", 939 | "colab": { 940 | "base_uri": "https://localhost:8080/", 941 | "height": 34 942 | } 943 | }, 944 | "cell_type": "code", 945 | "source": [ 946 | "# Sanity checks to catch some errors. Specific to KungFuMaster in assignment's default setup.\n", 947 | "l_act, l_crit, adv, ent = sess.run([actor_loss, critic_loss, advantage, entropy], feed_dict = {\n", 948 | " states_ph: batch_states,\n", 949 | " actions_ph: batch_actions,\n", 950 | " next_states_ph: batch_states,\n", 951 | " rewards_ph: batch_rewards,\n", 952 | " is_done_ph: batch_done,\n", 953 | " })\n", 954 | "\n", 955 | "assert abs(l_act) < 100 and abs(l_crit) < 100, \"losses seem abnormally large\"\n", 956 | "assert 0 <= ent.mean() <= np.log(n_actions), \"impossible entropy value, double-check the formula pls\"\n", 957 | "if ent.mean() < np.log(n_actions) / 2: print(\"Entropy is too low for untrained agent\")\n", 958 | "print(\"You just might be fine!\")" 959 | ], 960 | "execution_count": 0, 961 | "outputs": [ 962 | { 963 | "output_type": "stream", 964 | "text": [ 965 | "You just might be fine!\n" 966 | ], 967 | "name": "stdout" 968 | } 969 | ] 970 | }, 971 | { 972 | "metadata": { 973 | "id": "KGMW7v42peBa", 974 | "colab_type": "code", 975 | "outputId": "3dbe5236-58fe-47c0-d854-ea0aebbe6ddc", 976 | "colab": { 977 | "base_uri": "https://localhost:8080/", 978 | "height": 124 979 | } 980 | }, 981 | "cell_type": "code", 982 | "source": [ 983 | "!pip install tqdm" 984 | ], 985 | "execution_count": 0, 986 | "outputs": [ 987 | { 988 | "output_type": "stream", 989 | "text": [ 990 | "Collecting tqdm\n", 991 | "\u001b[?25l Downloading https://files.pythonhosted.org/packages/93/24/6ab1df969db228aed36a648a8959d1027099ce45fad67532b9673d533318/tqdm-4.23.4-py2.py3-none-any.whl (42kB)\n", 992 | "\u001b[K 100% |████████████████████████████████| 51kB 4.9MB/s \n", 993 | "\u001b[?25hInstalling collected packages: tqdm\n", 994 | "Successfully installed tqdm-4.23.4\n" 995 | ], 996 | "name": "stdout" 997 | } 998 | ] 999 | }, 1000 | { 1001 | "metadata": { 1002 | "id": "jAFWsGKlpuIM", 1003 | "colab_type": "text" 1004 | }, 1005 | "cell_type": "markdown", 1006 | "source": [ 1007 | "# Train\n", 1008 | "\n", 1009 | "---\n", 1010 | "\n", 1011 | "\n", 1012 | "Just the usual - play a bit, compute loss, follow the graidents, repeat a few million times.\n", 1013 | "\n", 1014 | "![alt text](https://camo.githubusercontent.com/69cc1e7cffa2ebd5db67c39f7c7c2724aaf5594b/687474703a2f2f696d61676573362e66616e706f702e636f6d2f696d6167652f70686f746f732f33383930303030302f44616e69656c2d73616e2d747261696e696e672d7468652d6b61726174652d6b69642d33383934373336312d3439392d3238382e676966)\n", 1015 | "\n" 1016 | ] 1017 | }, 1018 | { 1019 | "metadata": { 1020 | "id": "hpqugPcIpIlq", 1021 | "colab_type": "code", 1022 | "outputId": "156a4bd1-d476-4cab-a163-c822b59d7a8d", 1023 | "colab": { 1024 | "base_uri": "https://localhost:8080/", 1025 | "height": 280 1026 | } 1027 | }, 1028 | "cell_type": "code", 1029 | "source": [ 1030 | "from IPython.display import clear_output\n", 1031 | "from tqdm import trange\n", 1032 | "from pandas import ewma\n", 1033 | "env_batch = EnvBatch(10)\n", 1034 | "batch_states = env_batch.reset()\n", 1035 | "\n", 1036 | "rewards_history = []\n", 1037 | "entropy_history = []" 1038 | ], 1039 | "execution_count": 0, 1040 | "outputs": [ 1041 | { 1042 | "output_type": "stream", 1043 | "text": [ 1044 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1045 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1046 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1047 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1048 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1049 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1050 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1051 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1052 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", 1053 | "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n" 1054 | ], 1055 | "name": "stdout" 1056 | }, 1057 | { 1058 | "output_type": "stream", 1059 | "text": [ 1060 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 1061 | " if issubdtype(ts, int):\n", 1062 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 1063 | " elif issubdtype(type(size), float):\n" 1064 | ], 1065 | "name": "stderr" 1066 | } 1067 | ] 1068 | }, 1069 | { 1070 | "metadata": { 1071 | "id": "C-uNJ4-TpNC_", 1072 | "colab_type": "code", 1073 | "outputId": "c4a0ec03-1336-411f-b81d-bc34821b7a71", 1074 | "colab": { 1075 | "base_uri": "https://localhost:8080/", 1076 | "height": 297 1077 | } 1078 | }, 1079 | "cell_type": "code", 1080 | "source": [ 1081 | "for i in trange(100000): \n", 1082 | " \n", 1083 | " batch_actions = agent.sample_actions(agent.step(batch_states))\n", 1084 | " batch_next_states, batch_rewards, batch_done, _ = env_batch.step(batch_actions)\n", 1085 | " \n", 1086 | " feed_dict = {\n", 1087 | " states_ph: batch_states,\n", 1088 | " actions_ph: batch_actions,\n", 1089 | " next_states_ph: batch_next_states,\n", 1090 | " rewards_ph: batch_rewards,\n", 1091 | " is_done_ph: batch_done,\n", 1092 | " }\n", 1093 | " batch_states = batch_next_states\n", 1094 | " \n", 1095 | " _, ent_t = sess.run([train_step, entropy], feed_dict)\n", 1096 | " entropy_history.append(np.mean(ent_t))\n", 1097 | "\n", 1098 | " if i % 500 == 0: \n", 1099 | " if i % 2500 == 0:\n", 1100 | " rewards_history.append(np.mean(evaluate(agent, env, n_games=3)))\n", 1101 | " if rewards_history[-1] >= 50:\n", 1102 | " print(\"Your agent has earned the yellow belt\")\n", 1103 | "\n", 1104 | " clear_output(True)\n", 1105 | " plt.figure(figsize=[8,4])\n", 1106 | " plt.subplot(1,2,1)\n", 1107 | " plt.plot(rewards_history, label='rewards')\n", 1108 | " plt.plot(ewma(np.array(rewards_history),span=10), marker='.', label='rewards ewma@10')\n", 1109 | " plt.title(\"Session rewards\"); plt.grid(); plt.legend()\n", 1110 | " \n", 1111 | " plt.subplot(1,2,2)\n", 1112 | " plt.plot(entropy_history, label='entropy')\n", 1113 | " plt.plot(ewma(np.array(entropy_history),span=1000), label='entropy ewma@1000')\n", 1114 | " plt.title(\"Policy entropy\"); plt.grid(); plt.legend() \n", 1115 | " plt.show()\n", 1116 | " \n", 1117 | " \n", 1118 | "\n" 1119 | ], 1120 | "execution_count": 0, 1121 | "outputs": [ 1122 | { 1123 | "output_type": "display_data", 1124 | "data": { 1125 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAEHCAYAAACDax4fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXecVNX5/993yhb6wtJEkaaHKlIs\niCgaTKyxm69R87OiRoyxxBKjsURj7A0VjT32rpFYEAEFkd7hAMLS2V1gG9um3d8f987MnbY7Ozs7\nM7tz3nlFZu49994zs3PP5z7Pec7zaLquo1AoFAqFIvOxpbsDCoVCoVAo4kOJtkKhUCgUrQQl2gqF\nQqFQtBKUaCsUCoVC0UpQoq1QKBQKRStBibZCoVAoFK0ER7o7kA0IIcYADwN9MB6U9gJ/kVL+mOTr\nvAF8IKX8IpnnTSdCiAOBbVJKLd19USgaQgihA78AHoz7vAK4XUr5XSPHvQZslFL+QwixDjheSlnc\n0v2N0o+ewFFSys9TfW1F/CjRbmGEEBrwBXCVlPJLc9s5wGdCiIOklDXJupaU8g/JOpdCoUiIiVLK\n7QBCiPHAF0IIIaUsjedgKeXgFu1dw5wATAKUaGcwSrRbnkKgNzDfv0FK+bEQYoFfsIUQk4GbgDzg\nJ+ByKWWtEOJ44AlzuwbcLaX8oIHts4B/Syn/I4SYCDwOtMN44r9OSrlICHEpcBpQCUzAsArOl1Ku\ntnbaPP5BYDvgllJeJIQ4E/gH0B7YCPweGAncK6U81jxuOlAmpbzIfL8CuBTIBZ41j/UBf5JSzhBC\n9APmAe8Bo6WUxwshLgf+bvbxLUuf+gBvmN9nLvCulPLOpv05FIrUIKWcK4TYCIwDPhdCnI/xu3YA\nOzEe5H+xHmNa6wdJKbcLIW4Drsa4R/8L/AXYAZwupVxktp8CTJJSnhV2nqHA8xj3Sj1wmXn/TwT+\nCcwCzsIYQy4FqjDuT4cQogNwO5H35USijyn3AP0wxroRGGPG2cD5wClSytPNPtmAXcBvpJTLEv5i\nsxw1p93y7AEWAt8LIa4QQvQHsDyNTwDuB06UUvbDuBnuN499FLhRSjkU+C3GjdDQdsxzdgA+AK43\nn9wfBt42bxqAU4HnpJSHAt8Df47R91HAC6ZgDwDeBC6UUg4wj3sB48YeLoRwCiHsQHdgiNmPLhiD\nxjLgReARsz8Pmcf6KQSWmQNDAfA0cLKUcgRwgKXdn4E55uceAQwQQvSO0XeFIhNwAvVCiL7AS8BZ\n5j3wJTAt1kFCiGOBKzEeiocDxwLnAO9jPCz7ORt4N+xYG/Ap8IZ5j1+D4dnzG2mjgPlSyiHAc8Df\npJRLMET7Qynl/5ntrPdlY2PKOea+g4FNwB1m+xOFEN3MNuMxHuiVYDcDJdotjJRSB04CPgFuADYJ\nIVabLnKAM4D3pJQ7zfcvYNwAACXAH4QQg6WUG6SUv29ku5+jgO1SyrlmHz7CuAH7mfvXSCkXm6+X\nAH1jdL9WSjnTfH0yMEtKucrSz98CLmA5xkAwElgH7DWt4vHAbCmlDzgcY8AB+AEYYLmO0/x+/H3f\nIKVca75/3dKuBPiNOaDVSykvlFLuitF3hSKtCCFOAXoBczHGgO+llBvN3f8GTrAIaTinAl9KKauk\nlC5gIvAx8A7wOyGETQjRFRiLMf1mZTDQA3gFDIsfKAWOMfdXSSk/M183dP+H35cNjSnfSyk3m68/\nBo6RUpZg3OvnmdvPxrDcFc1AucdTgJSyAsMt9ncz2ONS4F0hxEigC3C2EOLXZnMbkGO+vhz4GzBD\nCFEL3CGl/LCB7X66A2Vh3SjHuJHBsOb9eAF7jK7vs7zuAhxnBsr4qQC6YVjd4zBc9fMwrOvxwGjA\nH4RzEfAnIURH83rWwDKvlLLSfN01rH/Wz/GEeexzwAFCiKnAPeaDkUKRCcwSQvgD0Yow3MP7hRAh\n96SUssKMdymMcZ5CDBe6v70/9uUnIYQLOB44CPhaSlkddmwXDBf2WiGEf1snjHu1jPjvf+t92diY\nYh0ryoAC8/U7wGUYXoUzMYwURTNQot3CmNHP/fyR4mZU6L+EEBcAwzBuzNellLeEH2u2vR643hT1\nj4UQX8Xabjm0GOMG9fdBwxDDYoyn8ETYCcyQUp4XvkMI8T1wLcaT+b0YLu1TgDHAK6bV/RJGZOoy\nIcQhwPoY1ykDOlved/e/kFJ6MFzrDwkhDgX+B/wIfJvgZ1Iokk0gEC2MYowHWwDMaSAfxvRZNPZg\nEXS/i1lKuRfDHX4+cCChnig/O4HKaEFt5rx0IjQ0pkDow0dXgiL+CTBVCHEqUCOlXJPg9RUmyj3e\n8hwEfGou+wJACHEEhktqIUak5jnmkzhCiDOFELeZc8SzLHO2iwE3YI+x3We55gKglxDCP0j8H0Zw\nSFEzPsfXwARzbhshxJFCiKfMffMJzr2tMt8fC/SUUq7HEN5qYJ3pDpxsnqNDlOssMnaJQ8z3/8+/\nQwgxTQhxkvn2F2A3oKxsRWvgWwxPlX9a6BrgG/NBNBqfA78VQhSY98ynwG/MfW9juJqPAaZHOXYL\nsF0IcR6AEKJQCPGOEKJ9I310Y1jp0WhsTDlWCHGQ+fo8DLe438v4FYZ3TLnGk4AS7RZGSvkThkg9\nL4SQZjTpE8DvpJRbzACQBzHcamsxosg/k1K6Mea9vhNCrAFmYwR6VMTYXmO5ZjVwAfCs6c7+I/B/\nzXEjm3PHVwGfmP18FvMmlFLWY0S1FkkpfVLKcozo7nnm4csxBpf1GNHxX2AI++wo1ykFbsZw/a8C\npGX3C8AD5mdaY56rwTWwCkUmYFrfV2IEhK0DjsOIDI/Vfj7wCEYQ5xqMued3zH0rMXI9fC2lrI1y\nrI4hqlPMa80BvoviRg/nG4zAsYVRztnYmPIthkW9DcMg+Zfl8HeAg1GinRQ0VU9boVAoWhfm0spn\npZTRLO1U9+Ue4EAp5ZUx9h+J0dcjU9qxNoqytBUKhaIVIYykLf0w3M4ZjenavxtjGaciCSjRVigU\nilaCEOIVjKVcl5pLKTMWIcQojNiTnViSJCmaR1zucSHEcOAz4Akp5bOW7b8BvpJmXmghxEUYCTB8\nwItSypdbpNcKhUKhUGQhjVraZsThM4QF/Agh8jCy3uyytLsbI3ftROBGc/G/QqFQKBSKJBDPOu16\njOw8t4Vt/yswFSPCEYyMOQvN6GaEEHMxEmzErDhVWlqlouAUijjo3r1jxlc5U/ezQtE4zb2XG7W0\npZSe8GUFZmKLkVLKDyybe2GkyvNTgpEZS6FQKBQKRRJINCPaE8CfGmmT8ZaBQqFQKBStiSZHj5sp\nKQcDbwkh5gO9hRCzMSIEe1ma9sGSO1ehUCgUCkXzaLKlLaXcAQz0vxdCFJml2/KBf5vlGD0Y89mx\nSj4qFAqFQqFoIo2Ktpkz+zGMxfxuM5/tOVJKa1UXpJS1QojbMXJU68C9/qA0hUKhUCgUzSetaUxV\ntKlCER8qelyhaBu0ePS4QqFQKBSKzECJtkKhUCgUrQQl2gpFmtB1nS/mFbF5V2W6u5ISbvjsaf72\nxRvp7oZC0apRop1mNm3ayJQpk9PdDUUa2FdZzydzNvH90h3p7kqLU11fj7v9Dsrar2Lpjg3p7o7C\nwvffz0h3FxRNQIm2QpEm6txeABy2jI8xazbtc3PxlR4MwNby3WnujcKP2+3mvffeTnc3FE0g0Yxo\nbZrp079g/vx57NlTylFHjWP+/Llomo0JEyZy9tnncfXVl/H66++wZ08p55xzGp999jUFBQX8v/93\nIS+99Dr//Od9lJaWUFtby+WXT2b8+AlMmTKZAQOM5e0XX3wpd911O06nk0GDDg1c98knH2HdurV4\nvV7OPvs8Tj31jHR9BYoU4PYYop3jtKe5J6lhbJ/BLPUUsd+9P91dafN4vV4efvgBdu7cgcfj4cor\nr+HVV1/iiCOOYsmSRZSXl/Ovfz3BW2+9zi+/bOTRRx9i6NBhgXHv3nsf5Pvvv+O7774BYMKE47n4\n4kt54IF7yM/PZ8uWLVRUlPPXv97NjBnf0LdvX04//SwALr74fKZOfYnOnbuk8ytos2S0aL8/cyML\n15Uk9ZxHDO7BBScOarRdcfFu/v73f/DPf97Hc88ZFUavvfYKTjhhEu3bt6eqqooVK5YzcuQoVq9e\nybBhI+jSpQvV1fs58sijOeWU09mxYzt33XU748dPAGDAgIGcddZ5PPfcU/zqV7/mggsu5D//eY2N\nG9dTWVnBvHk/8v77n+HxeJg+PWadFUUbweU2yiE7Hdnh8MqzdQSgypMdc/iQvjHs22+/olu3Qu64\n427Ky8u54YZr6NixE+3bt+epp57n+eefYc6cmfz+95ewZs0qbrnldqZP/4Li4t288MIr7Nq1k//9\n7wteesmIQZg8+f9xwgmTAOOB4KmnnuPHH+fw6qv/5qqrruGZZ57g9NPPYvPmTRxwQB8l2C1IRot2\nOhkyZChr165m+/ZtXH/91QDU1FSze/dORo4cxZo1q1i5cjnnn38hq1evRNd9HH74aDp27MTatav5\n/POP0TQblZUVlnMOB6CoaHPgBhg1aizz58+jU6fOHHTQwdx++02ccMIkTj75tNR/aEVKcfkt7SwR\n7XytPQB7a8rT3JO2z6pVK1i+fCkrViwDoL6+nvx8DyNHjgKgR48eVFRE5r4aMmQomqaxYYNk2LAR\nOByGRIwYMZKNG9cDMHbskQAMH34YL7zwDAMGDGL//irKysr48cfZnHTSyan4iFlLRov2BScOissq\nbgkcDicOh5Nx48Zz6613huxzuVysWrWC7du3cv31NzJ9+ud4vR7Gjz+Ob7/9isrKSqZO/TeVlZVc\neeUlgeOcTuPr1nUdTbOZr32B/Y899jRSruPbb7/iq6++5IknpqbgkyrShdu0tLPFPb5wVTn6QBvb\ny0sbb9xGSNcY5nA4+cMfLg8R0ClTJmO3B39r0RJrORxO85UWst/tdgfGLJ9PtxxvxGOcdNLJzJ49\nk0WLFvKvfz2e5E+jsJIdj/gJIsQQlixZTF1dHbqu8+STj1JfX8fw4YexYsUycnJysNlsaJqGlJKh\nQ4dTXl5O794HYLPZmD17Jm63O+K8ffsezLp1awBYsmQRALt27eSDD95FiMFMmfLnqE/BiraFy2OK\ndpZY2iVldej1eWi5tY03VjSLoUOH8+OPswEoK9vHtGnRDQBNs+H1eiO2H3qoYNWqlXg8HjweD2vW\nrObQQwUAK1YsBWD16hX069cfgEmTfsP06V9QWNiNvLy8lvhICpOMtrTTTa9evbjgggu57rqrsNls\nHHfcRHJzjR9kXV0dY8YYbqL+/Qeydu1qnE4nEyeeyO2338SaNas47bTf0qNHD1599aWQ855//oXc\nddftzJnzPQMHHgJAYWF3Vq1aznfffYPT6eS0036b2g+rSDl+97jTkR2WNoDuzsOWX8Peyhq6dWrX\nYNuKahed2+ekqGdtixNPnMSSJQu55prL8Xq9XH75ZFauXB7RrrCwEI/Hzd/+dhvHHHNsYHvv3gfw\n29+ezfXXT8bn0znjjDPp1as3YHgab731zxQXF3P33fcD0LVrN/Lz2zFpknKNtzQq97hCkSa+W7yd\nt75dzzVnDuPIIT0bbNsSuceFEA8DEzAe3v8ppfzYsq8I2Ab4zbCLzAp/MWnsfr78oZnkDFqKvWsx\nfUrO4K//ZwRofvXzVob2K6Bvz44hbQF+O74fZ00Y0MRPpmgpHnjgHiZO/FUguNZPeXk5N998PS+9\n9Do2W3Z4jhKlufeysrQVijThDrjHU29pCyFOAIZLKccJIboBS4GPw5qdIqVM2vqsDvlO6t25AGyu\nKgImsGV3Fe9/vxGAv196BAf36sj6bcFAtc/nFinRznDmzJnFyy9P4/rrb1SCnQLUN6xQpImAe9yZ\nlttwDnC++bocaC+EaNGnB69Px1tRCICt0z627K6iqsYV2H/vawsB2LGnuiW7oWgGd955T4SVfdxx\nE3n99XcCUeWKlkVZ2gpFmnCnMRBNSukF/Op4BTDd3GblBSFEP+BH4A4pZbOms04bdzAfzjUMd0f3\nHezYv5t8PXI9rz86WaFQRKIsbYUiTdS7/eu00xeIJoQ4E0O0p4Ttuhu4CZgIDAfObe61hvXrCu48\ndJfhIt/nLgk8uFjZV1UX8v7JDyIDqBSKbEWJtkKRJgKWdnrc4wghfgPciTF3HbLGUEr5hpSyRErp\nAaYDI5p7PR3DgnYVDQNgU2l00c7PCXUArvhlb3MvrVC0GZR7XKFIE+lMYyqE6Aw8AkySUu6Lsu99\n4AwppQs4Hviwudf0+fXZZ3gW5Pa9lG7YGtFOa/v1UxSKhFGirVCkiUDBkPS4x38HFALvCyH822YC\nK6WUnwghpgPzhRC1GJHlzRbtvj07GC98xkOK3aGzozQy6Oyj2ZuaeylFE/j++xmBtMpthS1binj1\n1ZcoLS3BZrORm5vLFVdczZAhwwJtZs6cwT//eS/Tpr3KgAFG1rqFC3/mxRenYrPZGTduPJdeeiUA\nTz/9GKtXr0LTNG644WaGDBlGcfFu7r//bnw+H926FXLXXfeRk9PyeQWUaKeZTZs28vjjD/Pssy+m\nuysRlJQU8+qr/2br1iJsNhs2m53f//4SjjpqXKDN0qWLueuu27njjrsDUaUbNqznscceQtNg4MBD\nuOWWO9L1ETIaf0a0dFjaUsoXgZg/OinlU8BTybymw+5P3Wv826WjE1XvK734S3O2JdFetWoF06ZN\n5aabbqN/f2O5YGlpCffffzc333w7Bx/cj6VLFzN//txAcis/Tz31KI899gzdu/dgypTJHH/8iZSX\nl7F9+zamTXuVoqLN/POf9zFt2qu8/PI0zjnnAk48cRLTpk3lyy8/5+yzz2vxz9cmRLvOU8+u6mJ6\nt+9JniM33d1pE2zbtpUHH7yH6667keHDjenMqqoqHnjg7zidTkaPHsuOHdt57723GDFiZMixTz/9\nWOBp9J577uSnn+Yybtz4dHyMjMblD0RL05x22jAt7eKySMn2fyeKxIlWlnPMmCOYMmVyWkpzRuuP\nz+dj9uyZ3HLLHXzzzVf85z+v8sYb77Fnzx7uvfdODj98NBUV5Wzfvp2dO3dw1VXX8uWXn7N7904e\neeQpevbsxQMP3BNRAtnj8TB16lP861+P89lnH/Pww/+ga9du1NXVc/XVU/jww/e4+ebbEGIwo0aN\nYcqUyYHvbceO7XTs2ImePXsBMG7ceBYvXkB5eTkTJkwEoF+//lRVVVJdvZ+lSxcHDJLx4yfwzjtv\nZo5oCyGGA58BT0gpnxVCHAS8CjgBN3CxlHK3EOIi4M+AD3hRSvlyczr38cb/srRkZYNtdF2nwlWJ\nT/dh02x0zumE1sCk2KgeIzhn0Okx91trad9774PMmTOLGTO+yoh62tOmTWXFimX4fF7OOecChgwZ\nxhNPPMJjjz3NypXL+ctfbmD69Jn4fD4uu+z3XHjhJSxbtoTy8nI2b97E5MnXMmPG1xQVbebuu//B\nsGHDeeaZx1mzZjUul4uzzjqXM844y+zLo/z1r/ewatUKnn32CfLz8+natSvnn38hn376EaNHj6Vb\nt0IeeOARHnro/kAf3W43u3btDLihxo+fwKJFC5RoR8Ht8WG3adizLSGFbt6ftsggtDQmaGwR4hnD\nmkpjY1i0spyvv/4uQFpKc0brz7Rpr/Hii88BsHLlcgoKurJ//35WrlzG6NFj8fl8VFZW8vjjzzBt\n2lS++uq/PP74M7z00vPMnTuHk046OWoJ5IULf+boo49h/XpJcfFunn/+FebPn8djjz3EsGHD+fjj\n9wFo1659xPe2b99eunQpCLwvKChgx44dlJeXI8TgwPYuXQrYu3cvtbW1AXd4QUFX9u5NTcBko6It\nhGgPPAN8Z9n8DwxRfl8IcR1wkxDiXoxlIkcCLmChEOKT8CCXZOPVvfjMSlk+3YdX9+LQmudAsP5w\nZ836LiPqaS9fvpTi4t1MnfoSLpeLyy+/mJdffoPS0mJ0XWflyuUccohg8+ZNuN2ugGhu27aV5577\nN1988Sn/+c9rvPLKW/zvf18wY8bXDBp0CL16HcD1199EfX0dF1xwFmeccRZbthTRo0cPwJjvev75\nl9m6dQuXXnoht99+d6Bv0QoDVFSU07FjMB2l8WPe06y/R1vF5fFln5UNAUsbLVK0Pb7Ibcnmf/O3\nUO/2ttlMa9HKcvoLF6WjNGe0/tjtdnJycqirq6O4eBfHHTcxUO74uONOYPHihQwdaoxhhYWFAUOs\na9euVFRUxCyBvHLlcsaNG8/06V/wu99dBEDPnr0YONCYs/Z/nniI9QAZLfV3KtOBx/MJ6oFTgdss\n2/4I+BdTlgKjgaOAhf6lI0KIucB4IFR9msA5g05v8IkSDNf4w4ueobimhJ7tenDr2Oub7SL3/3Az\nqZ72ypXLWb16ZcCdo+s+9uzZw4ABg9i2bQtr1qzm7LPPZ9WqFdTX1zNq1Bh0XWfwYOOzdOtWyMCB\nh2C32yko6EZ19XJyc3OprKzgmmsux+FwUF5eBsCKFcsYNWosM2d+y9lnn4+mafTs2Yu+ffvhdDqb\n+MNvY6ZTEnG5vVlVLMSPf07bamnbOu1Fr89n1tIG05snhQ9m/QKQEtGOZwxLNtHKcvpJR2nOWP05\n7LDDWbRoAfn57Rg2bATz5v3I+vWSa6/9E4sXLwzpa3i/Y5VALisro3PnLhQXF9Onz4GAMZ717z/Q\nLIkc2wtbWNidffuC1nJpaQmFhYU4HI4QK3rPnj0UFhaSn9+O+vo6cnPzAm1TQaOP+VJKj5SyNmxb\ntZTSa6Y9vA54G+iFIeB+SoDeyexsNPIcudw69npuGTMlKYINwR+uv572s8++yLPPvsgbb7zH4YeP\nZtSoMaxevZLt27dy7LHHsXnzL6xcuZzRo8eG/JgefPDRkPPGU0/7sssms2HDem677cawY52cfvqZ\ngb689daH9OlzoNmXVdTX1zF69FhWr14Z6AvQ4A9/6dLFLFmyKHBOv6unrGwfXbp0oaQk/Ic/IHBs\nLLp0KQh5gt+zp5TCwu7xfO1Zh8vjy5qynCGYoq1p5u/I5iF38ELyRs5hf22wlK29607yj/wKe7ed\ncZ/6w1m/cN9rC7P6YTHespyQmtKcsfpz+OGj+eCDdxk6dBiDBh3KmjWryM3Nw+l00hixSiB37NiR\nPXtK6dGjB9u3b6OsbB9ffvk5BQUFzJ49k9Gjj4h5zt69D6C6uppdu3bi8XiYN+9HjjjiaI488mhm\nzTIczVKuo7CwkHbt2jN27JHMmmUUtpk9eyZHHXVMo/1OBgmPGKZgvwnMlFJ+F6VJylZb5jly6d+5\nb9KD0DKpnvbQocOZO/cHfD4f9fX1PPHEwwCMGjWar76aTp8+B9GlSxfKy8spLy8LBFM0REVFOT16\n9MThcPDjj7Pxen243W46duxEaWkJ3bv3YMeObdTW1vLhh+9RUNCVdevWcNBBfWOe0+FwcPDB/Vi+\n3HCFGT/mcTHbZzNuj48cZ/ZZ2oE5bdM9bu9SEtxF8CE2Z9AK49+BK2Keav6a3ZSU1QTeT5+/haLd\nVXizOBXqiSdOIj+/Hddcczm33nojhx12eMy21tKcVqylOa+77qqopTlfeukFLrvsKqDh0pyx+jNi\nxEiWL1/CsGGH4XA4qK2tZcSIw+L6jBMnnsi8eT9www3Xkp+fHyiBPG7ceL777hsuueQyXn55Gi+8\n8Cw333w7CxbMZ/16ya9/bfTvv//9lClTJrNx43oefPA+7r//bgBuueV27rnnTq677ipOPPEk+vY9\nmBEjRiLEEK655nKefPIRbrrJ+K6uuOJq/ve///LHP15JZWUlp5ySGo9KcyZ/XwU2SCnvNd/vxLC2\n/fQB5jfj/Gknk+ppjxgxklGjxnD11ZcBOmefbdR66Nu3H0VFmzjjjDMB40mzW7ducX2+sWOP4q23\nXmfKlMlMmHA8xxxzLI8++k8uueQynn76Me688x6efPJRvv32ayZP/iMvvfQcX375BTfccDMA8+b9\nyNtvv8HWrVuQci0ffvguTzwxlT/96WYeeeRBdN3H0KHDOeKIoxL7A7RxDPd4FlragTltHTRvQJwB\n6vT4i4Xs3FPNi58bD7+v3H5iyL4sNrRxOBzcfvtdEduty0rPPfd3gdf/+c8HUc9z7rkXcO65F0Rs\nP/bY46OW5ty/vyrqA3qs/rRr145Zs4ISMXVqcJy84oqro/bV+tofXAfw61+fEnj96acfMXv299xz\nzwM4HA7q6uo4+eTT6NEjWP729NPPCkS7Wzn88NFMm/ZqxPZrr70+YlthYSFPPvlcxPaWJu562kKI\ne4A9ZvT4RcAJUsorLfvzgZXAWMADLAGOCE+PaEXV085cpk59Cl3Xueqqa8jNzcPtdrNo0c/4fHrE\nDatoOrquc8W/vueQAztzx8VjGm3fEvW0k00897NRJ1sn/8iv8VZ2xb15OHkj5wT2H6afwc8L3YE2\nfqae+HDEudZvK+eht5YAQdH21+GedsvxMeMF/G3ChV7RONHqaVtLc2ZCpS+3280777zJggXz0TSN\n/Px2HHvscZx66hlNisdpKVq8nrYQYgzwGNAPcAshzgN6AHVCiFlmszVSyj8KIW4HvgZ04N6GBFuR\n2fzxj3/i008/4tZbb8Tn85GTk8sRRxzJb397Trq71ibweP15x7PLPT64bxfWbS03LGFNB4crZP/e\nmnKgPTgip5WaQhZ7x1uUO++8J2LbccdN5LjjJqa8L7FwOo3Atz/84fJ0d6VFaFS0pZSLMSr9NIqU\n8kOSkO5QkX40TePss89LSbKAbMSVxrKc6STPXwxEt6FpPjRTtL1VXbB3LKdobwnQP7AdQHc3HpgU\nTrrLe64p2kfHdjkc1KNDWvuhaHtk14ihUGQI6SwWkk56FOQbL3QNNB+a0xBnvdYQN1tBMQCa1dLW\nmi7AiUaP+3w6P63eHRLFngiPvruMv7+yoFnnUCiikV0jhkKRIbjSWywkbZw1oT+D+3YxgtE0PSDO\nvupOgGUZWIhoR0+4sqYodt6meCTbP0VhZe6qXbz0xRqe+yS5WcwUimShRFuhSANud3praaeLvBwH\nvz6iL+g2bO324+wrAdBrO6J47bEPAAAgAElEQVSbQg6EuMdjWdqfzy2KeZ14DO3Jj8yitDwkBQUl\nZcb7DdtVOI4iM8muEUOhyBCCc9rZZWlD9HrZujsH3ZWHlmMkWrS6xzWb3mR3d7zt124pa9J5FYp0\no0RboUgD/mpW2TanDaZoh0WH654c9Pp8Y45b8wb26z5D4X1603KSxyvx4c8Pi6SR1DGbk7MoMpvs\nGzEUigwgYGlnmXscjJUJmiXvuK5r4HWgu4zERVpOXcA9rruNLIfeRkR7X2VdyPu4DfMw1S7eVxO9\nnUKRIWTfiKFQZADuLA1Egyj5jT3OkK22TvuC7vGAaDdcZ/uW5+aFvI93yZeWumzLCkVSUKKtUKQB\nv6XtzFJL21MczF+vu41CNb4as6SrpoPdY+zzGPsaE22AB95cFHg9Y/G2OPsSVzOFImPIvhFDocgA\n/HPa2ZZcBQyhdG8V6C7Diu7aoT0Aen07Y7/NCzYfug661/BEeOOos/3LjsrA6zVFKsBM0TbJvhFD\nocgAsjp6HEC346s3Eq3kOQxr2nCTAw63Idw+W6AimC8OSzuhvihLW9HKUKKtUKQBd5YHooFpUQMO\nzRBr3WukONXsHiOhim4PiHZjgWgKRbaQfSOGQpEBBJd8ZaGl7bduzQhyh2Z8B7rF0iZgaRtDVDxz\n2gn1RQWiKVoZSrQVijSQrQVDIGhp+0XbrplFRHyGeGs2L5rNh+6zWNq+xEV7444K6l0xjlearWhl\nZN+IoVBkAME0ptlnaQcwq3fZbXbGHNo9YFWj+aJY2om5x+XWMh58czFPf7Qi6v49YWlMrdz07I9U\nVLti7lco0oESbYUiDQQLhmTfLVi+vx4Az+5+6D4bQzqMMlzmvnDRthuJV0g8EG3nnmogmK60uCw0\neUr5/tiiXL7fxYI1xQldV6FoKbJvxFAoMgB/IFo2pjHdtNNYmuXddwB1iyfRK683NpsGaEYmM5vP\nyDfuswWE3O3zNOkafq93eIqVv744P+T9kIML4juRQpEhZN+IoVBkAPX+ddrZ7B4H0G1GWlP/PLfP\nZkSPgzHHbc5z13ub5qb2i3V4OtPw945GHprWFpXxyDtLqXM17aFBoWgplGgrFGnAncWBaOHYNA1b\nwDS2B1OY+uxGMBrg9rrweH1U1Rj/vjNjQ1KuHW5IjzqkMOT9so17WLuljPnKTa7IEBzp7oBCkY24\nstg93rlDTsh7w8gOWtqBFKY+W8DS3l1exZsf/ERZVT2X/Ebw7aIYaUqddTj7rsNbNyauvlTXhVYb\ns9mUP1yR2WTfiKFQZABujxenwxZ0C2cRhw0MtWat34Gua2h2M+jM4h7/6IcNlFUZAWw/rdodekKH\ny/g/kCsW4ei2m8qui4iHf/93bcj7xWZpznCy76+kyFTisrSFEMOBz4AnpJTPCiEOAt4E7MAu4BIp\nZb0Q4iLgz4APeFFK+XIL9VuhaNW43L6sdY2HC6Bh3JqTzT7Ld6LbArnHsQWjx/0R4X7yR88EoHbB\nydja7Qeg3muU6tTjrtHZSJ+b8HCVrGsqFNFodNQQQrQHngG+s2y+D5gqpZwAbAQuN9vdDUwCJgI3\nCiG6Jr3HCkUbwOXxpj0ITQjxsBDiJyHEQiHEOWH7JgkhFpj772rJfoQIoh4cknSLpa1ZRLum3hIU\nZg91b/tqOhjHmiU9FYq2RjyP+vXAqcBOy7aJwOfm6y8whPooYKGUskJKWQvMBcYnr6sKRdvB5fGl\ndT5bCHECMFxKOQ44GXgyrMnTwLkY9/CvhRBDk3XtcKM15L3V0rbMaWOPvk47f4zFltB8lgQthrX7\n0ZxNzeytebqknEWhaD6NjhpSSo8pwlbaSynrzdclQG+gF2CdEPJvVygUYWSAe3wOcL75uhxoL4Sw\nAwghBgD7pJTbpJQ+YDrwq5bqSIilHSbguiniVks7JnZPIO0pmhHoFzN9aZM7GX9T5RxXtCTJGDVi\n/ZzVw6lCEQMjEC197nEppVdK6Z8cvgKYLqX0K1xKH8Ctmm1rH6yJbXWPE4doB9Z3QyCveTz85xvZ\n+LnVcKbIEBIV7f1CiHzzdR8M1/lOjJudsO0KhcKCz6fj8erkZkBZTiHEmRiiPaWBZi2qWLYYQV4n\nHt43UK4zuns8zKa1eQJirYXNdTfEzCU7Gm2ThUH+igwl0VFjBsZ8F+a/XwE/A0cIIboIITpgzIX9\n0PwuKhRti2AK07QHov0GuBM4RUpZYdmV0gdwq2i7dw4IvO6Un0fH3HYAOLrtinJgqDWt2b1Bi9ye\nxgxmyj+uaEEaXfIlhBgDPAb0A9xCiPOAi4DXhBBXA1uA16WUbiHE7cDXGD/be8MGAoVCAdRnQLEQ\nIURn4BFgkpRyn3WflLJICNFJCNEP2A6cjnHPtwhWK9ZX0Q0OMILHnDYnthAB1Akx+m1hwmzzBea+\nA1nVFFnDnOU7aZfrYOzgHunuSovSqGhLKRdjRIuHc1KUth8CHza/WwpF28VfltOZXvf474BC4H0h\nhH/bTGCllPIT4FrgHXP7e1LK9S3VkZDkKp5gtjSf19iuu3LRcurNyl/BIUsLd5nbvAELW7N7m1WD\nO7KP8bfVlandovh0PeqUymv/WwfAK7efmOoupRSVxlShSDHBspxpDUR7EXixgf1zgHEtce3wRCUe\nr49JYw/ip9XFIeurq2t9oGEINmAvKMG794DggYF0pxqaTUezeUIC1uq89TSVngX5FJfFrrGtSC+1\n9R6ue2IOJ4zqwyW/EY0f0AZJfySMQpFlZHuxkHAbyeP10b93J646YyhYLG2Pbri4PXvN6fWwCHLN\ndI8HhN7pCrGIn/govlSm8fU5sUg0/99akRx27zPqoX+/tPHgwbZKdo4aCkUacWWGezx9RCRXMTeY\nXmXX5mHorlx6OA2r2m9da+HBZX73uCn0fovcz+bishbrc0NYs5j6fMpVnkxUhljlHlcoUk4muMfT\nSbj+BTXbGJG9pQfhLT2I/L7tQQfdbVrfzrCa2qbl7d+vOeui7m8KSdeERsS+aHclHfKdFHbOb7ih\nQmGSpY/6CkX68JflzMlWSzsMf1BRuBWloVFR7QJzrXa4pe0PRNNjWNpxZVELJ4ZqL9+4p+nnioP7\nXlvErc//1CLnVrRN1KihUKSY4Jx2dlra4aHYMUXb3yyQFS18iVeopW3vZKxc012mZd6ErGiVNYYV\nHyvye8HakrjPZUXlZEkuKjJfibZCkXJcbkNs0lkwJJ1ElOa0+UU7dED2z3X7s6JpOaHub38gmjV4\nLeQKTbC0//z0jyxrIWtaoUgm2TlqKBRpJNvd4xFz2ua/4TZUoJ0p2vZOZTj7rsHW2UyLbg+1tAP4\nU5mGiXZjda6XbShtcL8iA1CGthJthSLVuN3ZHYgWGT0eo1lge7CBo9dWcsVi440tdE7bj3urMNsW\nhWz3LxdqiGREJ8d7jsYeIhSKaCjRVihSjCvr12lHV+28nPCHmGA7b1lkaspARjSPM3SHWVPb1r4q\nZHO9u2F3eUtoqJJlRbLJzlFDoUgj/iVfWTunHabZ5pQ2w/t3i7odorjAwRKIlhuy2VvePep173ut\n4WQrqRZYJehNR31nSrQVipTjT66S48xS93gY/oCzCDe5VbTDXOBau0rsXY3KX4HynX68TnzVndC9\nTf9+k2Nt61FfNtRMoYgXJdoKRYrJ+jSmYeqsBbY30C7MBZ43fB6avwSY1457x0CjmZnyVPc4Dfe5\n1rQ0oj41z6zIcFRGNIUixQTc48rSbhCrhkdY01Z0G54dh+At74Fe09HY5Bd5hxvC3Oexz5Na0VZr\njhNAfWXK0lYoUk3APZ6llnYEjUaPE1j21dAJ9OrOgSC0YBY1N1p+FdhdUY7z4RywIriEDNCTkCtc\nD/GOxz6fMuqbjnrQUaKtUKScbM+IFm9t6s7tgxZyk+enzSxqWm4teSPmkjt8XkQTe/cdOAp3BpeQ\nAaq+hyLTUaKtUKSYQMEQlVwl5H34UrADe7QPvNbr29MUdL9om1nUbLl1EW2cB8mIbcmoyqV036De\nlUDud0WjZOeooVCkEZfHh6aB3RanydnWiNPUtoq4Xtc00ca0zDW7O7jN+ppgyc/ANcz/hWPrUoKW\nW4Pb03QR+uD7X6it90Td15bd4wvWFnPt47P5afXukO1en4//ziuipLw2ofO25e8sXpRoKxQpxuX2\nkuOwR0RRZwsRnzrG9xC+2bVxJL7adnFdw29pW8t52jqUBxvYPDh6bjXa+teA6+ALCza3ddpL7qFL\nyBs5h007K+O6tpXvl+5g+vwtsXrZ5PO1FmYv2wnArKU7QrYvWFPCx3M28dB/Fkc7LCVs2lnJ3opI\nz0trQYm2QpFi3B5f1rrGgQjVLugQPbo7/KHGu683nl0D4ruG3z1uEW1rwREtN2jpWSPTw6PHcwcv\njNmfmIRp8f5ad/R2bZhYKVr930X5/miBgS2Pruv8441F/OX5yBiH1kIWjxwKRXpwuX1ZHTkeLn2B\nzHBxaGKDS7+s+EXbYREHawERzSIqvuDfInxO21sRzNKWqGMkO/0p0UmFc+mbBVtZsr7tFn9JaJ22\nEKID8AZQAOQC9wK7gecxnjNXSCmvTVYnFYq2hNvjpV2es/GGbZRmTQt4g9+be/sgfNWdozbTTSEO\nsbTtnqiv/QlYdCItbb2uHXTeC0C9rz6uLsa7LGlbSXVc7doSqZgSenfmRgBeuf3EiH219a0/OC7R\nx/1LASmlPAE4D3gKeBK4QUo5HugshDglOV1UKNoW9Z7strTj4dSjD4663Wppe3YOwlcRmme8Swdz\nftq0tG3tLfPQVkvbZhFtW3AiO8KraxH3Kk85CRFDqLaVVEXd3paIWCmQZreDJzxooRWS6MixB/D7\njQqAfUB/KaV/AugLYFIz+6ZQtEncbl9W5x2PWYrT8vq8iQOjN/I07Bzs16uT8cIX+f0GqoIRZnWb\nYh5t7tnarspd0eC1YxFLp9puGFps0h186XZnqWhLKd8F+gohNgJzgFuAMkuTEqB387unULQtPF4f\nPl3P2gpfzSXeOW09imiHWtq+iNdrivZFHmMR7eLK+EQ77mVJWaja6dLsvRV1fDGviJoYy+9aE4nO\naV8MbJVSniyEGAl8Alh/0Sr2QqGIgkph2szBwdtwLEBAFKJlULOKtt0btl0nsAzb5jEC1bzOEEt7\n026rXdIE1GgYwJYm1X78/WXs2lvD7r2tP44g0YIh44GvAaSUy4UQ+YD1buoD7Gxm3xSKNodbFQsJ\nMTCvP3dEEw+24do8rPFkK77Ih6JoLnEwhV7TQdcAnfyxMwDw7OmNlr/fcsrEl27V1Lmx2TTycixD\nbhaKebo+8q69NQD8tLo4TT1IHok+7m8EjgIQQhwMVAFrhRDHmvvPAb5qfvcUiraFy8w7npvFlrbV\nfTzqkGAgWbxGmLf0IHxVXaPu88+Z6tEqe/lF2+ZBcxqR4IGc5qaI2wqCg7qjcFew/CeAlphrVQOm\nPPkDf3x8TuiOLHSPN1e1Y63/joZ/KioW/hoArY1ELe1pwCtCiNnmOa7BWPI1TQhhA36WUs5IUh8V\nijaDX7Sz2dJuSQKaoNupW300ecPmB/bZOxrR33mH/YCWY4q2JwfNXmvMa3vBlhuZXlPXjQcK3Zao\naEdXquzU7NTZ2pMfmUXvbu144Kqjo/ellXo6EhJtKeV+4IIouyY0rzsKRdvG5TaLhWS1pd2CcmUd\niD2R89/2wu0BwQ60ya1F07zGOm2zHnfIKc1z6nFa2io/tgWLMm4r2c+7321I6eX9bvFotNa/U6KW\ntkKhSAC/Sy6bo8cd9pb77FbNjhZpruWFBiIF3Oh+17kWeyT3Jegez8a562jc++rCBt3VqSeT+hI/\nSrQVihQSLMuZfve4EGI48BnwhJTy2bB9RcA2wB+xdZGUMrT6Q4Lk5zqYfMZQDigMDyZrvrqFDMPR\nlod5ckLfmwFr9s578NR2CmRHCzmnKxctpx5v3KLdOsWgpckswVaWtkKhiAN3hiz5EkK0B54Bvmug\n2SnmVFjSOXpYr5Y4bSi6DV9te3w1HfGWHkju4EUhgWYQtMZtHcwVq1Es7boVE8gbMwMfiUWPx3wU\naa2q0UYor3bRo0t+urvRZLLXR6dQpIF6T8bMadcDp9LKl2YWdIxeIcxAo37leNy/jDRyiBMMRvPj\n3WPU1PbVdjCOiGJp43OAz96oe7y4rIbS8tpIOzuGardlyU7180gicRIzF29vgZ60PGkfORSKbMJv\naTsd6XWPSyk9UsrIUOlQXhBC/CiEeEgI0SpmZiM7aQM0dFd0iyow7+1ftx1madfL0cYLrwO7o+Fi\nE3dMm89tL/zUjL42zIpf9rB5V9NreqeTpn7GmjoPS9eXNlmEE3lGWL5xTwJHpR8l2gpFCvEv+WoF\n9bTvBm4CJgLDgXNb+oKJLMEJPyb24B3j5Ga6U2fvIrS8/RGi7S9IovvseEnvkq8nP1jB/a8vCrz3\neFvnOuOGeOy9pTzz8UrmNzUJSgKq7fW1Tl9Hxo8cCkVbwh89npNmS7sxpJRvSClLpJQeYDrQxNRl\nmYe3vHvkRkuO8pxBy6IEopmC63Xg0V2EU1pey+dzN4cIaLxGYnNcyNtK9jP5kVl8Ma8o8ZNkIJt3\nGZXP3vhaNum4eMuhWsm25CoKhSIB/Ou0nRlsaQshOgPvA2dIKV3A8cCH6e1VdJqSrEOPUiHMuizM\n1i5oaddvOBxfWS9LOztu3Y1P92HTgn87vzu8S4fYc+stkcRjmena/WTOJs44pl/yL5Bm6t3x1b32\n+nx8NGsTRw3t2eRrVFRHPoS1BpRoKxQpJJjGNL2WthBiDPAY0A9wCyHOAz4HNkspPxFCTAfmCyFq\ngaVkqmiHCWKD+mgRaN2nUbdsYmQJT797XA87k9nO7fOQaw9bNkb0sp4tioo8B2DhuhK+WrCVrxds\nTXdXUoYSbYUihfjXaac7uYqUcjHGfHWs/U8BT6WsQ0miISmzWtWenYPAE2kdaw5TfMNE23/spt17\nGXxAr7TXhW5t7K2oa5Hz1tUb91M2PcJkro9OoWiDuFtPIFrKSdR47NW1XXwNrZa2K7o72+av6qWH\n/X1MS/vR9xezenOUutsJ0FoDoeIh/JNlkyXc0qiRQ6FIIYE57fSv0844Eo0et9uCBzZ0CqulHS3F\nKYCWZ+aqDnePB6qBedi8uyry3JYnjvDlSrE+V1uM/m5pFqwtCXkfz2PPa/9bF7rB5sXefWvU7Het\nAeUeVyhSSHDJV2ZHj6eDRHKSN6lqlFWoo9TbBtCcphs3zNLWfcaxmt2LrYkPF7H6mAEJdlqcZM8i\nfL80NJPum3FEmc9ZHpo/yNl3LY4e2/EWlAKTktm9lKBEW6FIIcElX21/wE6Ef1x5VLMG+njntK0B\naO6tAmdfY/DXnMacth7L0rZ7Gp3PbrtO7ybQzEC5SWMOTFJHIrG1MzwlWm51Iy0zEzVyKBQpJFia\nU1na0TigsD29u4UXEmmAKPopDuoSvW1I9Hjw+/fs7k/dymNC24YHovnb27zYNA23xxd/AYwsTGOa\nKP6pjtycxu+PRDOa6f6SrTGmSDKd1tlrhaKV4vL4sNs0bE31sWY5uTl26l2Ra3fDv8UxojtjDu3O\n1Y/Ojmjrd3EDEUu9ItKchgeieYPucbfXx9WPzqJnQfAYXwNBZbELhsQ8pFGU4MNTH65IdxfSgrK0\nFYoU4nL71Hx2Alxx6hDsNo0Jh/UO3WFxVQ/q05mjh/aKnde9oTltrxPdui3GOm1sHqb/tAWA4rJg\n6vbPfiyyHNvQJ8k8WqJkZqJnTMnyc/9a/Fb63KxEW6FIIW6PV81nJ8DYwT146dYTorvOzcG3c/vI\npCdWQua0wy1pAL/bNMp+3WJpR8vW1aDwZbB73O3xceW/vufl/65pkfMnup69JZfBa3Yzh3wrjR5X\no4dCkUJcHp9a7tUMwgdzm0b86hdjTjuwLUS0Y1najafXzAQxjpeK6noA5q7aneaeGCSSQ7zJ+P+G\n9sQKwKQbNXooFCnE5fYq93gzaJYBZnV/R1nyZRVtPcLSNv5mWgIDfcxlac3xBbemJ4OEaEFT2/wb\nJvK3zAQSDkQTQlwE3Ap4MMr4rQDeBOzALuASKWV9MjqpULQV3B6fco83gSnnjAgJQIvmbh3Wvys7\n9lRzyIGdGzmb5dgoljYei3s9wtIOq7vdBJri6q2sdtGxnTNlaVKbtM69CST8TJGChxHNHrS0dV1v\ndSlpExo9hBDdgL8DxwKnA2cC9wFTpZQTgI3A5cnqpELRFtB1HZcS7SYx+tDujBserLYVTWPOmziQ\n234/ikljD2r0fN6qLvhqOkQ9UYjL3Bfd0o7LpZqgBb1s4x7+/MyP/LeNldtsCv5vrkVl1GZa2hrU\ne1tfpa9ER49JwAwpZZWUcpeUcjJG8YHPzf1f0BpTzSgULYg/sYpTuccTJnww1zQNh92G6FsQ1zI6\n17ojqV8zLuq+QN5xiJLGNBiIlizCpX3ZhlIgMutXS5KpRmZL9MteuJ2cwT+j2YLffL239TmDExXt\nfkA7IcTnQogfhBC/Atpb3OElQO+YRysUWYhLZUNrNkP6dQ15H2tsP/nIvtF36LbornHAV93J0i7Y\npm/PDiFLvhqjuR7eeI7P9CntrcWR+dnTTc6AVdg7lYVsq8si0daAbsA5wKXAq4TePxn6/KZQpA+3\nyjvebPoUtmfqjccx5tDuDbY7d+IAfj/pkCad2100DNcvI6hd/KuQ7cP7dwM0dK89Lku7Yn+oyzV+\nq7HtDJsut/FbT9RiLi2vbbxREqjztEzJ0JYkUdEuBuZJKT1Syl+AKqBKCOFPEdQH2BnzaIUiC8mU\nWtqtnfxcB2eM70f7PAcXnXRo1DZ2m41DDoyRzjQmGt69fcDrDNkayHbmdaDlND7IfzxnU8R5o5GS\nRCKtlJ9WF6fkOnWe7LG0vwFOFELYzKC0DsAM4Fxz/7nAV0non0LRZvBbH8o93nz69uzIM38+jsEH\nF7T4tfyJU7ScejSnC619eYPt3Z6mz3vXu7xs3FERd/vw8p+JsHlXJauLklMbPPMJ/b50jxGjUOtt\nfZZ2Qku+pJQ7hBAfAvPNTdcDC4E3hBBXA1uA15PTRYWibeC3tFWxkNSQrGCm/bXukPeOwp24q2Nb\n8eF6Gk83nv14BTv3mFWnUmSB3//6otRcKBMIX6pnTnPUtkL3eMLrtKWU04BpYZtPal53FIq2i9tv\naTuVpd2amGdmC9N9GppNR3c3ki41gWusLiqL2FZV42Jr8X6G9e8a5YjWQabM0muO0AcvX3kh9oJS\naj2pmTtPJmr0UChShD96XM1pt07cm0YAYTnMoxFmajfV4q+tNyLU//HGIh57bxlbdodGYifDNZ6J\nlFU1b37ZcdA68o/8ChxR1l6b27wV3XDvGIinuB8AtW4l2gqFIgaqlnZqsWa66terY8T+S34dPYgt\nnCMG9wBANwPUtASyojUF/8Ndabnhut1XGerC/ezHzS16/XSwYG0xN0+d26xzOHsXGf8eJCP2aU5D\ntH2VXfHsOKRVz2kr0VYoUkQwuYq67VJBr6755OfaOW3cwfTs2i5k3yW/PpQJIw9o9Bx2mxa0lP0W\ndiPLvuK1g+MujhFmqf+wYlfoeTLZ8vZ/eY14G5asL03aJX01xnp7rV0lWn4l9u7bcPYzqpj5H7z8\nKwRq3a1PtBOe01YoFE3DH4iWqyztlOB02Jl64/EAVNe5+XlNcBnRCaMPjOscD197DO/N3AAkXjTE\navH7fHowc1tz6oWkWKeralxsL61mSKLR+insr2aW3MwbPi9yp/ng5Z/iUJa2QqGIiZrTTh/t85yN\nN4pCQcdc/Mu0g0VDGhbtercXW4d95Az+OWJ+dYc/QrwZNGWO3OP1cffLPzNj0bZmXfP+1xfxyDtL\n2V6yv/HGFqpq4svtndSHkAbqZAfiEfzucTWnrVAoYhGY01bu8daFqSh6nPnHtxbvJ+fQJdg7leHo\nuSXEM2yPIz96OBpagy7whvRue+l+tpdW8/aMDY1eZ+7KXTH37akw59ebGCy2tXi/8btv4GPv3FPN\nwnUlTTpvg9i9xPxWPH7nsg3da2+VS77U6KFQpIjAnLZyj2cMXTvlRt1++jEHc9q4gwHL8N+ESl+a\nw2jj7PML+33BZCx2e1C9GhLb+Wt2N3h+67FllfVUVBsWbdHuyoSt6pe/XEtJo+lDm24S17q8DR72\nzneNP1A0Bc3miWlt69Zsd14Hu8rjT2iTKSjRVihSRKBgiLK0M4ZYaVDPOW4g5x4/ELC4bnWbsVa7\niXPa88tnBl7b4/Rtv/j5mrjP/5fn53HjMz8CcN9ri3h7xgb2VCTm9rXWLk8WyVirHU+wne4zr2T3\nouXVRG9kWa6nex249ewpzalQKJpIILmKsrTTwklx1NsG+MuFo0LeB4OvjOQqtg4VYHdHHmjBqjEu\nizDY7YkNuZGS1bCI+b06TaUxgU1VANxoS0GYR95Zym0v/NT4Qbr53dq82AuD5U3rVo4PLPHSXfnB\n9l5HfPXRMwwl2gpFiqgPpDFVt106yMuJfFiKJkK9wpaHnTC6T0Sb3GGNiIil/KemBa1X65x23Eu1\ntAbfNnJoE+3clkhhFnZOe48t2Ls37MK3Tlus3VIWmFNvELMGumb3otcY6/LdO/uj13akbskkahec\nHNrc60Cz+XD7Wpdwq9FDoWgB1m8r56Uv1lBpiZ5VpTnTS7So/cLOeQD07hYU6vBYMZumBepz+6oN\nMbDFcr/68Vr+xlpQnPdWNj3wSYOULZnSkpWw3XrOwH8McvqtJaf/aqbP35LcC5mijc0TWEuv13SK\n3d50lYeX55yzfCffLNjK/lo3lz80ky9/KkpuP5uJEm2FogWYtWwHP63ezWPvLqO6znCl+qPH1ZKv\n9BBN9/r27MitF47ijovHBLZFE64LThwEgGvTYQD46tpFtAm5ls+SAsMWdFXX1SfHqmspN3WL5QqP\n0t8PZ/0Ss3l1bQLfk1TA+AsAACAASURBVPlwpNm9gbiDhlLOBtZqh4n2a/9bx7szN/LMRysA+Gh2\neKnV9KJGD4WiBSjeZ1hi20r28/h7y6mt9wQtbYto17pr2VyxNS11fYUQw4UQvwghpkTZN0kIsUAI\n8ZMQ4q6Ud64FiOWOHnxwAR3y41vHrdd2RHflYsurMUt0xpg79lkt7WCbROa0W8D4jcli2fDSq6Q+\nK2ixg968vthz8lp+FfaeRZG98aeXtXmDa+m9DXi1/JZ2jAQrVTUNxy2kCyXaCkWS0XWd4n219Oza\njvEjerF5VyVPfrCcKrPEo9/Srqiv4NYf7uXRxc/y8KJnUircQoj2wDPAdzGaPA2cC4wHfi2EGJqq\nvrUUybJOtRzj75Q3bD7OAatiXMx6QPCNdclXUwhNedpyKv7JD8nPa65pWvQuOxITxbwRc8k5eB1a\ne+tyLR3NZn5HNm9gLX2IxyMMPYp73Ppgl8qHpaagRFuhSDL7a93U1Hvo3bUdl50yhCOH9GDD9gq2\n7K4ix2ELuF9X7FmLz7TUimtK2FVd3NBpk009cCqwM3yHEGIAsE9KuU1K6QOmA79KZedagngDv2xN\nSIDiKIz4+gwsQm1NxuKwBYdcX9wPERmqHgkRtKCbunQuHM1MouI4YCO2jmWh2/3nbqgiW8A9HnxY\nLrWsU9+1t5G4hTShRFuhSDLF+4wbv2fXfGw2jStPH8qoQwqB0Plsty9oafRs14Pe7XumrI9SSo+U\nMtZi3l6AtYJDCdC75XuVGTSWtcxllugMYPMQ4aq1iDa+4N88xNJO0PSPxwL8ZUclNXUt4N5trrfC\nZnmYMS3thAue2Lxo7apwHriR3CELLNs9ljnt2O5x3WNMidRYamrHepBKdAldS6BEW6FIMsVlxhN6\nzwIjWMlht3HNmcM5emjPkPWne2r3AXDx4PO5dez15DmiZ+fKANqEqdeYNtxw3mGceWx/8nMbrqPk\nq+wa8j5/7AycA5eHNtJ0dFOs/eIAofEM8UqVpjVd31+ZvpYpT/7A0g3Jq54VLw2KsDVTmbnWfU1R\nWYzGobz5TWjJTc3hjpr5TLPpwXX0DbnHPTkAVLuD+eDnrYqeyvWxd5fG1cdUoKp8KRRJJijawUQO\nToeNyb8dFtJut+kOH9XjsEwT7J0Y1rafPkRxo7c2/FIS6wlk5KBCRg4qbPw8vkhbx9FtN25rMLSm\no7tzjGvGylWeoIGpafFbfotl80V7R2mwSEjc5UTDCKwXt4isrV1V7PZR3AnfL9kRusHujulit3fe\nZzzoRPlbBTCtcP+c9r7KOv47L/oytPXbMyfdqbK0FYokE3SPN7wsaFd1MV3zCjJNsJFSFgGdhBD9\nhBAO4HTgm/T2KhmYgtNcv4Eez7CpG+uGffaQ5CqJSN7SDXsitq3bEp91Gs2VXufyNBidHc5dLy+I\nuW/X3uoIN3z4ZwyxvC3TBs6DmpdzXHO4G85o5nXQ0B/bH4hWXmM8ZDe1GEq6UJa2QpFkivfVkOOw\n0aVjbDGudtdQ6apiWLfBKexZECHEGOAxoB/gFkKcB3wObJZSfgJcC7xjNn9PSrk+LR1NInpAs5up\n2r7Gk+Nomo6u24y2OZYlRXrUlw0ya+kOfj/pkOC5m9D/aBbrHx+fE+IFSpQ6l4c7X/qZ9nkOnvnz\ncTHbhcS9W9as+8ysZU3DGuAX29I2LtDIw5Up2mu2lcBw2LyzMoH+pB4l2gpFEtF1neKyWnoU5GNr\nIGJod7WxHrZX+x6p6loIUsrFwMQG9s8BxqWsQymk2Ut5dBt1y44jd8TcsDKdFge8Zljaus8WIlRW\nAUs4AEsjbm9BrM9aXNb8OtK19cZnr67z4HJ72bC9giH9Chp8GsmxzP3r7pyY7TbviiGglu/S0Wsr\n7h0DY19Ma/j79VvaHjM3fFNWDaSTZom2ECIfWAXcj7He803ADuwCLpFStg5/g0KRJCqqXdS7vXG4\nxo3Si73bpS5iXJE8dFc7dHcumj24LChn0DJsnfZRt+RXAdE23OM6jj4bcPTcQp3ncMD4bTRn3Xi8\n1nbS05LG6PMbX0vmrdrNpacM5tgRvSOO8XfD1iE4N6w1UHSlpKyWjTsqGNSnc+iOsMAzZ5/YWdUa\nE21/kJpbN/rRHNHesruKHgX5jQYxJoPmzmn/Ddhnvr4PmCqlnABsBC5v5rkVilaHPxOaP3I8Fn5L\nu3cHJdqpIuAeT5aOhblf7V2LgxHNms+Y+zbbOPv8gubwUFxbzNINpWy3BHfFQ8i0cOA/jROuQwlb\n942watNeAIp2VUYEq8W8oqPhddolZVHWSdsanot3bQlON2mNnB+fDV3XAhnREtXsnXuqufe1hfzr\n7SWJnaCJJCzaQojBwFDgS3PTRIw5MYAvgEnN6plC0Qrxux0bmzP0J1Lp1S497vFsJNHI55jEmtu2\nu/EHounhbXSNZz5ayd0vL0i8N1r8s9rhlvY73zUv+KuBC8XcFetBIep8tObD0XsTOKOnFtWiLPEC\ncG8/hPo1R+GrKghs8+xpLLWAZkSQmylPF6xtOIVrLErMe35rcdMexBKlOZb2Y8BNlvftLe7wrErG\noFD4CVjacUSOF+R2Ic+Rl4puKcBi8iXH1I629AtMMdJ0dNM9bmXpL1ZhSHT5VPxu7/C4ihmLtid0\nzWjEEuPwzboeQ9OjiLatSynOg9aTe0iMddExLG3vvp749heErMt2m8VdrPzl/w7nsIHdOOe4AUbf\nzJra+yrrWBtnRH44SX8YbISERFsI8QfgJyllrES1rWNGX6FoIm6Pl3tfW8jHc6LPpQUs7QZEu8Zd\nS4WrMqUZ0BRBEnWP+8tzBohpaXtC5rStLFofTN6h66DlVWPvvjWOq4cKQ7yfIdmxVTqwvWQ/n/6w\nibkrg5+lstoV2N/w0UE0mw9PWC1rW56R6MTWoYKaOg/LNoYtd4thafu/Z70+H92Vi3tnf6LJ0JB+\nXfnz+SMZO9j0cHkdaHYvNXVNS6laW+/h4beXsLpoHy98trpJxzaXRGfNTwMGCCFOBw7EyGO8XwiR\nb6ZGbBPJGBSKcJZu2MOW3VWUVdZx1oQBEZZM8b4a8nLsdGoXu2rU7pr0Ro5nK40lV2mMQ/t24asF\nFoGNZWk7PKaoapHWuC000Uru8LloNh91NZ3Rq8OCrmKgafGLdiKBaG9/u57zJg6MWff97ldir9ue\nvWwnF510aMi2gEVuWtbeyq7gcWLvWhxRFjNQExt4e0YUV34MSztQglO3UbfshJj98+P0V1vzGe7x\not2NL/daW7SPIf2MbHhzV+5i3dZy1m1d1uhxySYhS1tK+Tsp5RFSyqOBf2NEj8/AqAqE+e9Xyemi\nQpE5/LDCsC4qa9zsKK0O2efTdUrKjepeDQ2Wgcjx9r1itlEknwG9OwFwxOAEH5Yi3L7RRU1zuMwG\nkZa2ZhHtOct3BpaDaTnR53BjEX/0eJNOC8CMxduZs7z5Npe9605yDlmC1+dFQ0NzmtZ4XbuYtaz1\nRhLX+Oe0ffVh00oNFQYxGd6/a8Q23eNEs+m88lXj1vIj7wYFOv5iL8knmRnR/g78PyHED0BX4PUk\nnluhSDt7KmpZs3kfDrPow+rN+0L2l1XW4/b4Gg1CC0SOK0s7pYwR3fnbH8byh5OTlNAmVvIOv2j7\nbJFtLKJtrdccK8DKj3WeeNfemias007Mr1BTn3gFrmVmBrecQSuwF5Swo8Z4APA/zOgeZ0Bk671h\nq4L1Rvprfn/e0gPDdjT+OfNyIh+y/PnHNWcTVye3UBR+PDR7UZmU8h7L25Oaez6FIlOZt3I3OnDG\nMf345IfNrNmyj5OPCs5z7i6Lb7lXIHJciXZK0TSNAQd0St4JY8xp2zqUA2AvKMVb+f/bO+84Oc7y\n8H9n+14/XdfppLuTTq96L5ZsVUsuYOPYBhtc6EkgjgOhGBIICSQhiSkGEkrAkARIfhhTDKa4S26y\nZVnVaqNinXS6qutt+8zvj9lebnelu9Ot/H4/H320N/Pu7DOzO/O8z/M+pSxWhlR1yNOkMsWTqSqO\nXtN+5LnMI8cfe/EMgYDOrcGArRCZ6KrvPHaYaLeEP+AHrJH+2X4rejAda8QTryzTfEDoOmlmvG8u\nxtb4Bt5TiQFnITYvm87hM710DyT3ZOheo2qhYvWge8a+b8Fw9SuKktLSPt02wOzpmS1zXCyy9rhE\nkgGarvPSG+3YrWa2rapjenk+J1r6Yxo3dIUjx9One5XYi3FaLr2UpOTykSp6PLqASIJiN6VQ2mks\n7YThGVrQR6I6aD35WktWn/H4ruasxkdjjuozPuwfMYrEBc9d18xhD8Tzh2KD8JQ0k5foYwS6a3G9\ndgOB3ukpx7/3hnlJ1+ZDEd+6L1hqONrSNvkj3pKE98W+P55//vHeMeUfD6TSlkgy4PjZProH3Kye\nX4nTbmFBfSlen8bp1sgDOpPIcZffTb9nQEaOXwGkcmmbHMbkzds8P1Fpp6qVbQpgKu3AueYJFFti\nUZF4FTHV03NMeZH7YtQfPJ/QhEUzh6/LnhMdsW+MuaZJFGPY0h4n1RVU2ootorTti3bhXPEcoKM4\nh1DskdiV1493EdA09p9IbOIyWUilLZFkQCgAbcMSo/zAgmAU6dGzkXXtTKqhdUjX+JVDOpd2wJpg\njadqcKEoGvYmI9DJseyFhP3f/+2RuPFZyDmuZLaWq3sjgWIj/hFjkhHl2g4VnVHiPQ9R19RcFhUM\np2jG3+E+2embtoxFvsMalDPiHg+LEJx0YfHiWPwyjqUvEjrv7/3mCDv2tXKq9fK16pRKWyJJw4jb\nx171AtXT8sK1kEVdCWaTwpEzEfdjR5+LfIeFAmfqdK/2cBCatLRzHd0ffPAHkisQPWDJ3NJOtdYd\nJL4957jXFB9ndF9EaY/6DCWYzD2esFwQZWlHW7+WGSewzT6EtU6NHCMNb7tqVsp9oRrhIfd4skA0\n54od4dc28Xr49fm4rJHJRiptiSQNu4924g9obFhSE35YOu0WGqcX0dwxyIjbR0DT6O53UZ2mElrI\n0pZKO/eIX8cMdNWh+2x431yC5/hq/D1xKXwBS4IbN97SDtdDH6vF5BQi46DpqGYdXcNGYF4y93hh\nQUj56lhqTmPKj+RLW+si3WDNRYZHSzEFj5uBe9yUgXYLKW1LRashwww16ThzcU/49Xikw10KUmlL\nJGl48WA7JkVh/aLYh/KC+mnourHe3TPgJqDpVGYaOS67e+U8ujcP9/6taH1VaINl+E4vw3N8VWR/\nwJxoEcYr51QWZxqmfhfJiNI+3dltLMKHlbYpfF2sNmOcqagHa93JGOUYjZIXV/wkI/f42BepqtQZ\nU/ZUyR/AOj1VkU+yDhacKKTSlkjG4FznEGc7h1gyu4ziAnvMvoWhde3mPjp6Q0Fo6SPHi21F5Fll\n5PiViO6N+l4DFghElkp0vyXRog4WE7FUZlkTfIq7x2Pc3MFI7LB7PGA2GnVAuFmHYo/t7637rUbt\ndnRATzjdTNzj6a5QIC5vy1I5djlZxTk5DUHSIZW2RDIGL8UFoEVTX1OIw2bmaHMvncEc7bHc426/\nmz5Pv3SNX8FEB2DpmgXdnYf/Qi3es/PCzSliSKV8lDSWt65jrjyHTezhYhuPXAwX+l3pB0FsL2uz\nUREtWSCarhjXw9YQG2inu50oim5cr2ReiAzc4yFFH4pDmVlVGLPfFzDkCXtHorR8YCA2vx7AsWhX\neJJxOZFKWyIZA7WlH5vVxOLZiTexxWxi3sxSOvtcHAvmw44ZOT4qg9CueKKU8N/fexWg4DuzmEBn\nfbA5Rfyadgp70OJLvj30PsBWfxRzcU+ClTqRPLozeaOcBKKUdviczdGBaCGlHUg6QQlNfhSLz+hR\nHk8Glvbm5bUAvGdbE/fduiimEBJAIBDM1XbnA2CJyi2PLiuruZ1Jt18upNKWSMage8BNRbETizn5\nrbKg3ujfe/C0Ed1bOUYJ01DkuEz3urLxnRP4L9RSVlgQsz1iaUcpNCXWStZGjfckpEKFxjuHUOyj\nsQFhWa6HTwYx52Xy8/Lh9sg5aRH3uG7yJ510hGqQK3lDSScwmbjHS4LLWXarmZWiMuEeDrnHda89\nIcDOc2xNZFx3beS80kymJoNLLmMqkVypjLp9uDx+ymakLksYytfWdSjOt4VTSZJxfsiYyU9zJDYu\nkFw5+Dsaku8IWAyXrSkQCYCKC24Kd6tKlgJmdeNY/LIxruPd4c2K2T/JHZ0zIMbS1hgY9mCLUtp6\ncL+u+CMNVoLoekQ5WirP4W+LLaUKZNQgJB2BQOjam2LWzD3HV4HfjmvfFszTOgh0zUIPWLDNOg7W\n5JXSJhNpaUskKQjVKy4vdqQcU1OWR2mhMaMfq1GI2+9hV7vR0vDRE4/h9mfZoECSc8RHeEcUcpSL\nPEq56V472pDhuUlmaVtrkkc2m8vak26/vASVcmjtObg2revEdD9z+bxgiV0yUBTwd9QDoI0UJ1ja\nuqaEA/guhfrqwqTbdXdwictvJ9AVzPUOV06bvKWIVEilLZGkIKK0UytjRVFYMMt40FaOEYR2oOsN\nvAFjlt7l6g6nfkneQgSVtmJzR0qVRlna3lNLIxZ4EqUdGIp4aPLzIo/uqbDOmkAo6CxYgAZTAHNh\nf7jPOJoJXQdN8aOYk7i/Q2va5sQ17XCu9iVSXZYffu1rrwfAc3IZujfxPtY8xjNgKlxrqbQlkhRk\nYmkDLGw0Hqa15flJ9w96h/jtm5H28lV5lTIY7S1AfCeokKXtWPgqjmUvYCroA0VHGy3Ac3Qt2vC0\nSHW1kHs8KkjL3rQ/cixLxFOjeyYuffDf/nffRb0vFHwWUr7mwr74EaCZUUwBzNNiJ7B6wByuNofF\nFz6WNjy+3bOiXeL+lnm4XrsBrS9Fj/ugpW0uvfyTbbmmLZGkoHvAcIWVpVHaa+ZXYVIUls4uT9jn\n1/w8/MZPGPAO8vb67cwvE9TkV+Gw2JMcSXIlocVHNwViy9vaF+w2xvmtaMOlwTdFanIr+f04Fr6K\n9/RiAj21Me/1KVFu2ksMRLM17cNcagRJ+rtm4GteFN6ntvRf3EGD1rOhtAeSl28NmIMpVMZ18nfM\nwlJ9Fu+pZeFrpZj96EFLOzBUGttBbRLRfUbfbZPDhXXmMXzn5qcce/7CMDMqClLuv1SkpS2RpKAn\nQ0vbpCismV+F3ZYY0fqLk49zeqCZFZVLuLFhGw3FM6XCzlWy9Mra41pCpqpRHrM+G7Xubak6C4Cl\n9lTCWzoH+o213eDYSyGksOEiirxEYW08iGPl04COYom1tEPWsr8zknala2ajv3jw/H1ts3G9dj3a\nQEWwapqCuaQbk8Oo9a0NJqZdAnzgbfMStl21IL0nK+OSrAB65LuzVJ8dc6jLM7G53NLSlkhS0D3g\nxm41j9kAJBVuv4cnm5/jxdZXqC2o4Z75d0z5Jg+S8SVeacdb2rrfaqzXRintkAtdMUe2J+sxrZtC\n+wOG4gO+9YtD4yh9Jug4lu0kMFSKv0VgKQ8GxFm8UZZ2cIIatJZjJi4BC1hHI527AhYiFU6U8Np1\nyH2ujRbiOzfXCE6LIlQ8JZqrFycWQ4pnom7Hib7PpdKWSFLQPeCmvNiR9U3o9nv459e+Tq+7DwWF\n9y14D3azbYKklOQKCZZ2yK0dVWAlei03hGLzoDiHYo9l9oWVdahK14FTwU5gVjfmkgsELswgk87b\nesAcOZaxJf37FA1TUTeKzYOlrAM9SpEqVq/h1vZb0ENu7lBaV/QExW/DZB5CsXrRA6b0EeEBK/6O\nJOlfSZg3qyTtmKwsbQwXuRJM+VJsrtiStUoAS3Uz/s5Z+HwTmzcv3eMSSRLCOdppXOPJaB1uo9dt\nBN7o6OGocclbG32kKObvsAUdXRUtbGn7Y7bb570WLrxiDIu0h1Ticrqdy3diaziCddaxzJpcxOc8\nZ+Bud65+CruIBKlFV3ZTLEYalx6wRM4nmdIOrhMrjtEEL0Q8uqbElC61mMeeVJgzaPHltGfXk9tz\nZF34tbXxDcLrJSY/ztVPY607iX3RLvzxEYjjjFTaEkkSMo0cT0bzYGRdUEaKS0JEpxJprkimga5F\nW58hyzQuzcnqi8npHglEdb2KyfuOKGlL1TmsM49nLefFVP2KroAWsrQJWND9QcUc6o2tRXkVQrnP\nJi19hbOAlWjr/zuf2JS1jPFsXDo9q/G610mg16hmaC7qxVzZAoCt8Y3IGJ8d01R1jwshHgQ2BI/x\nL8Ae4CeAGWgH7lVVVVaQkOQkmeRoJ2PUN8qTZ5/FbrLxwcX3MKe4QQaeXSHUVhiK1mY14fWNbcFW\nlCSf7LkPbERxjmAua8PkDFrLMZZ20OI0+7FEpUIFBspiy31GVREzOUaxL3kez6FNMdvBSFHynV0w\n9onFRZ9bqpvTvycOc1lUj2mrJ7hWr0Ss6WB+s66b+OpfrOdT39kFvqgloyRK23t6CbbZwXX6OOvf\nYjbxt/esxJzG4h6LmrLkKZpjofsi36t1+ikCXTNjUtYUU2DC26ZelKUthNgCLFJVdR1wA/AN4EvA\nt1VV3QCcAj44blJKJJPMxVraf2x+lhHfKDc0XMuisnlSYV9BVJbm8dW/WM9f3b4k7di/f/+apNt1\nb54RHe2PcgdHr+XqJvSACcUxEvM+xewHRQtHjMdbwyaHC6zuhJKgGaWDxQW6mQric6rjSNLgw5Qf\nWXO3zTKse8XqTWJpm5lW5ODt62aFLW1je6IqCvRELOFkwXhzZhTTUBO75PCvH1nHP3147djyXwK+\nc/MI9BqeM8XmxbnmiZj9pvxBzCn6FIwXF2tpvwC8FnzdD+QDm4GPBLc9DnwK+O6lCCeRXC4yzdGO\npnOki53nX6bcMY0tdRsmSrRxQwjxEHAVxuLcx1RV3RO1rxloAUJP6LtVVW2dbBmnGtOKHHT2pS9l\nmecY+9Eao7DiA7ACVkz2SOUt3W8xiowoOvhtYPOgWBOdmOZpHeiuuPzgJErbXHkOxeLF3zYH0FFM\nGoHBUrzH1+Bc82RSV7XiGMZS2YKvRYSDsdLhOydiJydEUsB0HXRPVOWxFGvv7sPrcSzahffMwow+\ns7JkgvvU6ya8p5YnKOvwbp8N0wSb2heltFVVDQChqeCHgD8A10e5w7uA9DH3EskUJdMc7Wh+der3\naLrGrU03YTVN7cQMIcQmoElV1XVCiPnAj4B1ccNuVFV1ePKlm9rUVV564YxYpR37kNf91ohliqHo\nFJvHcDf7LRAwG2vccdhmHTdKocZ8TuLv11Z/FAB/2+zIOrluAhR0vxVzYT/REeSW6aewzjByxTV3\nHtqwEZnt75iJr2UejpVPo5h0w0Ngjihff3et4TkIpbYRSQH7w6tnwRpR2qnWtPXRIlx7rhszsrxq\nWh5LZpexUlSkHJOK5U3l7D/ZnfX74vF3zcBSeR7F6kVjYvO0L8mOF0LcgqG0/zJul0xIleQ02eZo\nH+s5weGeYzSVNLK0PDOr4DJzLfAYgKqqx4BSIUTR2G+RABQ4rfzggc2XdIxopR1fz1qPiub2d8wK\nKz3F6jWU4FhlS4Nrv4EBoxCJksSVHR5a3op9/u7ghwZzwoPK1VwVKSASUtghWcMK2G8z5AlZz564\nmt3B89CirP+YSUTUmrY2NEbnuzSpYCZF4ePvWsqGJdkFll0Kgf7IBEEbLsbXvBDdGzqfKRo9LoS4\nHvgcxmx8ABgWQoR+TbVAW8o3SyRTnGxytANagJ+f/A0A72i8IVeKqFQDF6L+vhDcFs33hBAvCSH+\nVQiREyc1WYyVUpTJhQqt9QKJ/aSj0p/0gMVwiYc3KLH5wfGfHXSb+7vqCAyVgNWHoUQ0zNPaUPIj\nZUltjYcjZUHjrH1LefLHt2IOhCcGoclFSGlHB5P5zs0NK1vdE6WoY9zl0ZHkuVXHwH/BKCvrbZ6P\n5+g6QMF9YCuu167Hac0+4yQbLjYQrRj4CnCTqqq9wc3PALcHX98OJHf6SyRTnGxytE/1n+HLrz1E\n16ih/356/Be52nYzXtd8AfgERqzKIiL3tiQFjdMNR8WmZRlYfL7UHhzdH7G0tcFpEaUIMRHZAP7u\n6ZGCLBBRwpoZ/FYURcdU3I1zzVPY5hzCsfDVFB9qfP3+DqPMaPRnxhSFMfvDJUnD+d2BSCcv1+vb\n8Bxbjb+jkasXV8eOA0I/s2RVzDJl1bzKi37vxZDM26b1VeM+sIlA18y4PcrUXNMG7gTKgZ8LIULb\n3gc8LIT4c+As8D+XLp5EMvmkixx3+z2cHjjDnvb97OnaH7Ovc7SL9pFOGorjb+YpRxuxlvV0jFRN\nAFRV/XHotRDiD8Bi4BeTJl0O8rf3ruTU+YGEiOZkRLvHQ9HIkQ1RlrYnj8BQaaTetUkj0FODpcKI\nCdRHC3G/uQRL3XGsNc2RyG/NHLbm7WJvenmCStt3XmCpPoe5tAtL7Un8rU3GmnowPc1U2Is+avSh\nDh0/NGkw5Q2DZkEbMlzz04Nd76K9CiFau4eDn2uUE1WyqJ/+0Vsmd/npI7cs5Ks/O5CwPZXHwzwV\nlbaqqt8Hvp9k1/ZLE0ciufyMlaPt9nv44qsPMug1Ulym51fzzqabeeTEb+gc7cqlYipPAV8E/lMI\nsQJoU1V1CMKetJ8DN6uq6gU2IRV2Sm5YO5OrFhid3ubWpS+fCYBuxn3oGhT7qJECFr3LG1HoesCM\n3hf5PSlWT4wS1IMuaG2gHGqaI406NHNssFsawilkUS5ua+1p/K1NMRHoJrsb00zV+COorAPdtVgq\nWvG1xZYYvWpBNY/uOI2/rdHI/W4RfO7elQC4PMYxvcfWYp15HH9XXUZyOu3mcV1+apxelDYQLdu0\nz4leHZvaIa4SyWVgLEt7d/vesMIGeLe4ldklDTyw6n7aRzpzpu2mqqq7hBB7hRC7AA24TwjxfmBA\nVdVfB63rV4UQLmA/Ummn5KZ19WlTvJKhuwvQ3UaQ1vZVdbT1jHDkTG9c/rLRRMPXOhtr7WkjajzK\nfR6qMKa74wqF4tHcOwAAHWJJREFUBMxjrhPH1xtXzJFo9NBnBUeimP1orvxIMZjQnqDS1oam4dq3\nNVYuoDAv6DHQzbj3Gvbc7Di3uDZcGlwTTs+PPrs1o3HZcP0awyP2y+ffTDkmk5Ko0ciGIRLJJJMq\nR9sb8PFcywvhv6vyKqktMNYvHRZ7LrjEY1BV9bNxmw5G7fsm8M3JlSi3+Px7V3H+wvCYCntBfSlH\nm/uoKcujvWcUgGsW1/DSG+GVCG5YM5M7ts7B5fFz30MvxCrboNs6WpHrgcTCLLrXga6ZIkVINFNs\nAFsUum4oeSXfKIWqufPwvhkpGONvnRNR2hYvmP3orkIiWb7B40QfP8lnjWc5T6d9YlSVxWxi6Zzy\nMZV2tmvUE10bXCptiSSOVDnaTzQ/S7e7lw2161hbvTJnrGrJxNA4vSgcfJaK0OM+WrFvXDY9RmmH\nCCmmWLd28AjRUddRSjtSGU1B9zjDa8+6ZkltaQesRtpYUGl7Dm1MkNrX1oB1+hlM+YOGuzdgRhsu\njgS6QYJlnYBiuLNDrvBMueWaBhbWT+PLP42sxf/jh5JXmBsP0k0usi2VOtGBaLJhiCSnGRj28Mhz\nJzlyphct2157KUiWo9063M7T53ZSai/hT2a/jYbimVJhS9JiCZa0NEcpBi1NF6iQq1uL6goWLmjS\nNQMAX1tDcEzE3axHNSFBM6Vc01YsvvDx0slgyjMUux6w4Dm6Lpz/bTC2+lAga4UNhtKeMyPWjT6t\naOLSqJLp7PtuXRR+nW1g2eDoxHb1k0pbktP8+sU3efK1Fr72yAE+94PdPLWnhRF39l2KoonP0dZ0\njf87/ks0XePd4laprCUZc9f2uSysL+V9N87L/E2aBdf+zXiORaxL3evEtW9ruJGH//xc3G9cHVOU\nRBstijlGTMCa14avZa7x3q46AgPlY4oQKpQStqyDaVu+M4uCx0t/D4y1thuvCJfNSZTnSx+cOOs6\nmqrSPObUFrNt5YzwtiWzI5OTbJX2ONkOKZHucUnOMjjqZdfhTsqLHYi6EnYf6+Jnz57kV8+fZvPy\nWu7YOifrdbVwjnbUTP+F1ldoHjzHysqlLCqfP96nIbmCqShx8sl3Lx9zjJ6sglaS8qOx68ZKcJ05\ngjZUGneMyHhv80K0/ir87Q2EXO7es/PC6VvxaG5DaZtLuwwZQ+ldXieu124Y42xSE70uvWV5Lc/s\njbSwdSTpbT1jHMrFZoLJpPC3waj269bU0TvowWqJyJPtM8QyRRuGSCSXnecPtOEPaGxfVcf21XXc\neW0TLx5q47m9rTy1p4Xa8nw2ZNkzNz5yvGOki8dO/R6n2cE7575j3M9B8tZjooKLtaEyvM3z0cMW\nt4L3zcWYSzvR+ivD20IEOutTH8xnR9eVcJ/s8ahYds/2ueHX79oyJ0Zpp1symCzKi50JqZ7ZrlFb\nLqFdaCZI97gkJ/EHNJ7bdx6Hzcw1S4zeNAVOKzeuncXf3LMCu9XMoztPM+zKzlUenaPt9rn5t9e/\nhU/zYzVbsZlyq9SiZGqxoN6whCeyE1WgaxbacMTiDnTX4j25guzbQUQUNgTzwC8RJUrbWC2xqqdp\nRvI19rkzipk5SRZ3KrK1tKdkcRVJLLqu84vnT9Mz4KakwE5pofFvWpGDxpqiCY8mnMpous4vd56m\naUYJy5ou/cYPsed4FwPDXravqktIB5lW5OAd19Tz6I7T/PqFN7n3epHiKIlEW9q/O/Mk3oARVDLo\nHcqVSmeSKcon7ljGqMdP3gSlL4030SlkoXzyaL5239V88tsvZ3w8hzX2vLcsr2XH/tbw6wMnL7BS\nxJYo/czdK7IVe9zJ9vldWjixtcdz49czxXnjzV7++Oq5pPvu2DKHG9a+dR/0h9/s4Y+7z/HyG+08\nWL8emzV5C75s0HWdp/e0oADXrpqRdMz2VXW8dKidnftb2bC0hvrqzBpYhXK0vbZenm9+BRMKGnou\nVTqTTFFMJoUCp3XcshwmGvf+LdhmH8LXvCDp/tLC2GC09Yuq2XW4AwC7LfE+t1hild+914uw0jaZ\nlKRr/1Ol+c4Na2fyxO7kz/h44r0I4410j18iuq7z2ItGYv6n372Mz713Jffduoi7tjVhs5jYeaAV\nPUdu0ong6T0tAAyO+njh4Pg0fjvdOkhzxxDLmspTuhotZhP3bJ+LDvzkyRMZPyh7Btxg9vHH9l+j\no/Oni9/Hp1b+JQ+sul9GjUvGhamhhjIgYMV7YuWYXcVCfOtjG1jUEIlk/85fG7nf0fnV1aV5Ce/L\nFe7YMueS27GOF9LSvkQOnu6huWOIVfMqmV8f2xP2TPsgrxzp5ERLP2JmaYojXLm0XhjmSHMfs6oL\nae8Z4Y+7z7F5ee0lR1c+9boxEbhudfJ6xW6/x3Blz6hizfxKXjvWxY6DZ2lsMKUtiHJhwIWj8Qi9\nnj5umLWVJRXJrQyJ5GKZaOuxssRJV78r/cAk3LFlDj/fcSr9wCB/dfuSsAchel4cOsfaigK+98lN\n9A55KE8ywf7MXctxebPP5Z4Masvzae2OVIHLtpzpRCGV9iUQsrIV4Jar6xP2r11UzqvNKjsPnXtL\nKu2nXzeiQ9+xvp7j5/p5+vUWdh3uYGOWEd3RdA+42Kt2MbOyIKE5g67rnOp/kx+88RNG/KPYzXaa\nZjXh8PTxq65noddHgTWftzdcR7G9EItixaf5mFvSSJ4tD7ffQ0/eAZTSDmYX1/O2Btn/RjI1mFVd\nyNmOoZT7q6bl0dlrlEmdW1fCF96/ir/8xotpj2uzmPD6tfDfVy+uTqq01y2s4pUjnQnbo+NUkqau\nATarmeppya3sqfxc/OKH1hAITD0v6RWptAOaxqnzA8ysKkxZs1bXdc51DmMxK0wvz7+o2e/+k92c\n6xxmzfxKykqtnBk4R01+FSO+EY70qjze8gSOhS4Oug/RPzqHkrz89AfNEXoH3ew80Mb5rmHuvV4k\nrG8NjXp55UgHFSUOls4pp76miB37z/OHV85y9eLqi561PrevFV2H7avrUBQFt8/N3gsHOT/Uzhvd\nR+nz9IfHegIeDvceRomKfxv2jfDIiV8nHLfUVsKwfwQqfaDDXfPeidl06evvEkk6Mlm5mT29KKy0\n37V5No/uPB2z/58+vIbfvNTM73Y1c9umxjHvr2uW1PDSofbw6+f2tYb3FebZ2Li0hhcOxpZZtY9D\nLEquYVIUTJbM9MLKuRXsPXFhgiUyuOKUts8f4LuPHeHAqW7MJoX5s0pZPreCZXPKKcyzop7rZ9/J\nCxw42U3fkAcw3EnL55azvKmCuioHHa6utG5UTdf5zUtnUBTYuqacL736FQa8g5gUE5quxYxVHKN8\nbc/32NK4lnJHKU0ljTithqso5MrNhTrWuq5z9GwfO/a1sv/khfDDZnDUy2fuWhETgPH8gTZ8fo1t\nK+swmRRKC+1cs7iGnQfa2HOsi6sWVqf4lMhn/frFMxw6Fds2r713lKI8K6vnVbK34wA/Pv5z/JrR\njtBhtrOycimnB5rp9wxQ4Sznzxa/D4/fw0Ov/RcBywiax46/rRGTcwRLdSSwxBVw4dOC6WEKuPzu\ncbhiEklyNi+vZef+1pT7r1tdx1PBeBAg1oZNokfMJhO3bWzkto1Ge0yfP7nLuXF6Ee+/cV5Yad+1\nbS5bV8zg8w/vpqrUeCbVlidGigcyyKNWcme1ftxpmlEslTZkr9BcHj//8as3OHa2j4aaIgKaxuEz\nvRw+08tPnj6CvXAUz1AeaBby8nQWL4Fhemjr8PPMqTd55owP24zTYPWQby7gPXNvI8/uYNg3Qn3R\nTEodxZgUE26/h2eOHKbVd4Kq5QP8x/GnCejGTaLpGk0ljSwsm8cL53fR6+lH10z00skvT/42LKvD\n7CDfmseAdxC/5qfQWsCfLn4vDcUz8QZ8U06RHzrdwyPPnQx3KppVVcjWFbUcP9fHK0c6+clTKh+4\ncR6KoiTkUPe4+2gf7mTzqkpeONjO7185y9K5JXSMpp4cvXqkk9+9egpr/igWf2GwRSFYrAHEyn6+\nsu+btI10xLznI0s/SFNJQ9LfzYNbP8vJ7hZcA07O53t4s72HM+5ecAyjufKZPnod7WXP4WKAAqVU\nRopLJpT8FJ3Bbt3QwNOvn+e2jY3MqirkB787SmWJk01Lp7NjX2olH4/VYtTOj65TUFZk5/PvXQXA\np969jJ4BNyaT4Wn89l9vxGY1Jt1bVtTi9QdiOl/lO6yko6bccIHPiWu/KRlfpqzSdvs9fOHlBxkJ\nDFFkLeLvr/oUDmvq/Ldhl4+Hfn6QM+2DLG0qZvumIgptTtoHBjhw7ixvjL6CZvaQr5uxmW24NReh\nlRtTPcSXzRgJDPPwsR8nfE6+JQ93wENAD2BvggGgwlGOy+9i2DdCVV4FH1nyARwWOxtq19E+0smv\nnuziWO8J7HMPhI9TYM3D5XeHrcQh3zBf3/cd7CY7uqLjDXgpsRdz39IPU5NfiSfgvSyKvH/Yw/97\n5iR7jndhNimsW1jN1pW1NNYU4Ql4qZ7h4Xyvk5cOtVNfXcj6JRU8ffgo/e5BlizT+c8jD3OyP3Lz\nO1fZ6HZZ+cxLvyGg+ymyFfKhhXdTnV/JBVcvVc5yOgYG+emul3EsOYBi85BnyWP+tCaGvMOcHjjD\nYY8GHlhSvoDW4Q563L1U5VVSN0abTIfFzuLqOVANawRAI8OehTx75DgHWz0caXXBmdUozmHuuXbN\nlJksSa5Molfjoot33Hx1AzdfbTQDWbeomvISBzVl+THNa6ItWovZxPtuSF6H4DN3Lefvfvha1GdG\n3rcgLmg2ehnRYjbx9nX1DI36wtZ+JtkX9dVFfP69q5henrtR4um4aX09v9vVHLPt9k2NKVbzJ4Yp\nq7TbRzoZCRhrOIO+Qb6x9/u8b/GdSS2gjv5Bvvn4y3T2+Jm9zMUZx7N8+1BcJazgkoymBLBbrFQ7\nKmgejLhHt9VtZJqzjCebn2HAO4QVB/pAJf7iqNw8VyFua4CAJeJ6unf+HaytXplUqYaUx6YlDo78\nthenXoJL6acqr5IHVt2P2tLPfx77HopjBKcpn8UVgpP9p+nzGEX6+z0D/PNrX8NpdhDQA3g1H4XW\nAt4tbqMirxyP3830gprw58VbmEfO9PKbXSdYtMDGdYsXZqWINF3n+f2t/OL507g8AWbXFvGebY2M\nWrsY0lr4w5k2nm15EU/Ag3WWFWdhGT8/eZjfD/bg1lw4lsNJBeiPPW6BzclAYDjsmRj0DvHQ/u8l\nCjAn4gUc9Y+yt+tgwpDrZm2lJr/qoiczBfY8blmxgltWQHPHIM/ta6X1wjALZ1Wmf7NEMk7kO1M/\nhpNVCrNbI8tQ3//05pTvra0o4JN3LuOJ3Wc50tyXdWWvLctreWpPC+/aPJua8vwYd30q0rUqzXVu\n29jIzetn8edffT68bbKjyqes0q7Jr6Iqr5LO0S70gJmW0fN8efdDrJ++msbiegpthQR0P2pbJzs6\nnoNaD87p0KYAUUvK62vWMKNwOk+d3UG/Z4DKvAo+s+qvAHjw9X+nc7SLqrxKbmzYjsNiZ231irAS\nAPjy7m/S4+nGrhVja91IV78L+8JXMDlHKLeXs6xiMYqiJLXuQixrKqfA7sR7bB1/fXcDMwprOHZm\nkO8+dgxM6zHnj9A3lEfDdYu4c+2t/Oueb3DB1UOBNZ/G4nrODbXQ7zFa5A35hvnB4YgHwISJyrxy\nCmwFnB9qxR3wYDPZsHpLGHK7UaqGaO3Tef4lJ3eKW6gtrMHr91JTUI3VZMEb8PGrl4+x99xp8DlR\nMKErGj7djcfSg7VKYd5MC4pjhIeOPYqWZE7p03xQ0oEZcAevvaLA1TVr2VS3nh8e/t/wdX5g1f38\n4PEjHLU9jsk5QpGtkPrCmRzqORI+nn+gjHJbBYGi8wz5hpnmKOWjSz5Aoa2Ah/Z9L3yskKIejypl\n9dVFfPBtV/YDRzI12bYqeepiPF/56Ho6+0azKs27sGEamq5zpLmP69dk9jkhqqbl8fADW5JWBHsr\nF4yKbiYCxrNuMktxTFml7bDYeWDV/bQPd7Br3zA7TryBo0HlpbbdvNS2O3Zw6CwU2FC7nuO9Khdc\nPVTlVXJ7081BZbwywSJ7YNX9Ka3jEH+79mORMdvsjLh9nGxdwjC9rJo1OyPrzmI2sW5hNU+/3sJA\nVx4XWvv44e+PYbEo3H/bSvKdFr7+yEH++4/HGXXP4bOrPx4jl9vv4d/2fJMuVzfFtiLmTZvD7o59\nAGho9Hn66RjtCn+eV/PitXRhylcgWD/Yrbn4n2M/Sy1kksJiIYfcWS8Q1yL2pobtvNqxj25XD5V5\nFXxkyfvZeegMO/t+h8nuptRaxm1NN0W+x6jzuW2DQP2/flxKP+WFNWze3kSn64IxQXPnY21Zzac/\neDUOBxl9ZxJJrmJSlIwjs8uKHZQVO9h9NDH1aiwWN5bx3U9sSlqlLK18UQo7uuyo4yKOdaWiAJZg\nEO5kFHCbskobggq0ZBb1W3Q0v4WdJyw4FkTWaPwdMymxToOq0wz7h6jKq+RPZt8Is29Mq4xTbUsq\nQ9SYfIeVZbOrgbGjn+PZsKSGp19v4WfPnqRn0EOe3cLH71gaDtr47N0r+NojB/j5jlOMuH3ctrEx\nvAblsNj5zOqPxXgAmgfP0znaxTRbGdc43sWZjn4OWx5Ht42gu/O5ueIuNi+t4yv7/oPO0S40j4M8\nbzXuwuawTEVU0Dfkw1wY8WEvr1iC2WTi9c7I+vuHFt6NKJ3D1/Z9N2zlbqnbyJa6jTHX+Y615Wg7\nCulyd/HnG9amvPa15fl8+cPX8Ohzp3j5cAcP/vQQm1fciL/nPOfPKXz0HYsoKTDeezHfmUSSK6TK\nbR7zPRdh1l2Mwo7n3usF5SUOHt1xmpVzKy75eFcMisI1S2o43TrA9hQFn8aTcVfaQoiHgKswshQ+\npqrqnks9pqIo3L19Lm6/h32uI5ichmK6Zc6NXL+qAZ+WGG091R7sMyoLaKgp5Ez7EEV5Vj5x5zJm\nVkV62U4vz+dv7l7BVx85wO9fOcuOfa0pZm3NAOimFfjMA7SO5POIZmyz269hbpOZ91yzguoSw9Ub\n8lY8/VI/r6qdlKzow2MyIqQ79yylrMhBXsWrdLu7qcqr5J757wKgZagtrKAXlM1LajFD7HVWFIX3\nbJ0PpO85XZRn40M3LWD94hp+/KTKjr2dgJWrFlSxep5cU5Zc6Vy8SXY5qyLfuHYW16+e+ZZuggQw\nrchO76CRMrxmfiV2q5k/e8fCSflsZTzrYgshNgGfVlX1JiHEfOBHqqquSzX+woWhrD5c03T+97mj\n9HgucOf65dSU5lZqwfGzfTy1p4U7t86hKkWFoIFhD//zhMqFgfRlCB1WMzOrC2moLqJhehE10/JS\n3kw+f4B/+ek+mrv6EE0W1JN+SvONiUJBgSlBGU9m/rjPH+CPr57jbOcQH3z7/IzSS95qVFQUTvmn\nZLb381uZ8xeG+cIPX+Pu7XO5dmXypjepePmNdn74+2MA/OizWydCPEkaegfdHDjVzebltVkH+F3q\nvTzeSvtLwDlVVR8O/n0cWKOq6mCy8fImn1x6B9186b/3MDjqozjfxmfvXpFy8iCZWkilfeWhafpF\nWayvHOngB48fBaTSzkUu9V4eb/d4NbA36u8LwW1JlbZkcplW5OD+25fwxO5z/MmGBqmwJZLLyMW6\nmFeJCvadqGBblha65MpgogPRprx18FZjdm0x9922+HKLIZFILhKrxcx9t8p7+K3KeGeFtxEbVj0d\naE8xViKRSCQSSRaMt9J+CngngBBiBdCmqmrqfnISiUQikUgyZlyVtqqqu4C9QohdwLeA+8bz+BKJ\nRCKRvJUZ1+jxbJHRphJJZsjocYnkyuBS7+XJrXQukUgkEonkopFKWyKRSCSSHEEqbYlEIpFIcgSp\ntCUSiUQiyREuayCaRCKRSCSSzJGWtkQikUgkOYJU2hKJRCKR5AhSaUskEolEkiNIpS2RSCQSSY4g\nlbZEIpFIJDmCVNoSiUQikeQIUmlLJBKJRJIjWC63AGMhhHgIuArQgY+pqrrnMouUFiHEIuA3wEOq\nqv6HEKIO+Algxugtfq+qqp7LKWMqhBAPAhswfhf/AuwhB2QXQuQB/w1UAQ7gH4GD5IDsIYQQTuAw\nhuzPkkOyZ8Jk3cuZ/oaFEHcDHwc04Puqqv5QCGHF+B3NAgLAB1RVfVMIsRT4blD2Q6qqfvQi5Er7\n/U6mTMHPegDwA18ADl1OmYQQBcCPgVLADnwR6Eh2PCHEp4F3Bbd/UVXVPwghioH/A4qBYeAuVVV7\nhRDbgC8H5fyDqqr/mKE8GT3Hx+P6JDufsWSbspa2EGIT0KSq6jrgQxitPqc0Qoh84N8xbsoQXwK+\nrarqBuAU8MHLIVs6hBBbgEXB630D8A1yRHbgZuB1VVU3AXcAXyd3ZA/xeaA3+DrXZB+TybqXM/0N\nB+/TLwDbgM3AXwshpgF3Af2qql4D/DOG0id4nI+pqno1UCyEuPEixBvz+51MmYQQZcDfA9cANwG3\nXG6ZgPcDqqqqW4B3At9MdjwhRAPw7ijZvy6EMGMozp1BmX4FfCZ43G8BtwNXA9cJIRZkcH0yeo6P\nx/UZ43xSMmWVNnAt8BiAqqrHgFIhRNHlFSktHuBtQFvUts3Ab4OvH8f4gqciL2DM9gD6gXxyRHZV\nVR9RVfXB4J91wHlyRHYAIcQ8YAHw++CmzeSI7BkyWfdypr/htcAeVVUHVFV1AS9jPNSvBX4dHPsM\ncLUQwgY0RHkGsv4+Mvx+J1OmbcAzqqoOqararqrqn00BmbqBsuDrUowJTrLjbQH+qKqqV1XVC8BZ\njGsbLdPjwDYhRCPQq6pqi6qqGvCH4Lh0ZPocH4/rk+p8UjKVlXY1cCHq7wvBbVMWVVX9wS8vmvwo\n12YXUDPJYmWEqqoBVVVHgn9+COMHnhOyhxBC7MJwkX2c3JL9a8Anov7OJdkzYVLu5Sx+w/HyJGwP\nPuT14La+JGOzIZPvdzJlqgfyhBC/FUK8KIS49nLLpKrqz4CZQohTGJOvT6U4XlqZMhibTpZMn+Pj\ncX2ylnEqK+14Lqlx+BRhyp+DEOIWjAfeX8btmvKyq6q6HngH8FNi5Z2ysgsh3gu8oqrqmRRDpqzs\nl8CEntNF/Iaz2Z6V7Jfw/U6YTMHxZcBtGG7p/yKz+2Uir9M9wDlVVecAWzHu4fH+7PH63U3k9Ukr\n41RW2m3EzsanYwQA5BrDwSAUgFpiXS5TCiHE9cDngBtVVR0gR2QXQqwMBoqgquoBjCCkoVyQHXg7\ncIsQ4lXgw8DfkSPXPQsm7V7O8DccL0/C9mAwkRKUsyzJ2EzJ9PudTJk6gV1Bi/I0METy+2UyZboa\neBJAVdWDgBMoH+uzx9iebuzFMFHfWdYyTmWl/RRGQAJCiBVAm6qqQ5dXpIviGYxACIL/P3EZZUlJ\nMPryK8BNqqqGAmZyQnZgI/BJACFEFVBAjsiuquqdqqquVlX1KuBhjOjinJA9CyblXs7iN7wbWC2E\nKAlGLV8NvBiUM7QmfjOwQ1VVH3BcCHFNcPttZPF9ZPH9TppMwWNuFUKYgkFpqe6XyZTpFMYaMUKI\nWRgTiWNJjvcc8HYhhE0IMR1DyR2Nk+l24AlVVZuBIiFEvRDCghHo9VQWMkUzUdcn1fmkZEq35hRC\n/CvGA1kD7gvOwKYsQoiVGOtX9YAPaAXuxgj/d2AEGXwg+AVOKYQQfwb8A3AiavP7MB40U112J/BD\njCA0J0a6yOsYKSRTWvZohBD/ADRjWBw5JXs6JuNezuY3LIR4J/BpjPXGf1dV9X+DUbsPA00YwUjv\nV1W1JRhx/J8YRs5uVVWj16ezke8fGOP7nUyZhBB/jrGEAPBPGKlxl02moNL7EUbapgXDI9GR7HhC\niPsxnqs68HlVVZ8Nvv+nGNZsP3CPqqoDQoiNwL8FP+aXqqp+NQNZMn6Oj8f1SXY+Y8k3pZW2RCKR\nSCSSCFPZPS6RSCQSiSQKqbQlEolEIskRpNKWSCQSiSRHkEpbIpFIJJIcQSptiUQikUhyBKm0JRKJ\nRCLJEaTSlkgkEokkR/j/FM3d7OHmH8kAAAAASUVORK5CYII=\n", 1126 | "text/plain": [ 1127 | "" 1128 | ] 1129 | }, 1130 | "metadata": { 1131 | "tags": [] 1132 | } 1133 | }, 1134 | { 1135 | "output_type": "stream", 1136 | "text": [ 1137 | "100%|██████████| 100000/100000 [59:35<00:00, 27.97it/s]\n" 1138 | ], 1139 | "name": "stderr" 1140 | } 1141 | ] 1142 | }, 1143 | { 1144 | "metadata": { 1145 | "id": "XC3JUqqPpo6k", 1146 | "colab_type": "code", 1147 | "outputId": "40d34cd6-c8db-4d8c-a75f-140fa404f17d", 1148 | "colab": { 1149 | "base_uri": "https://localhost:8080/", 1150 | "height": 193 1151 | } 1152 | }, 1153 | "cell_type": "code", 1154 | "source": [ 1155 | "env_monitor = gym.wrappers.Monitor(env, directory=\"kungfu_videos\", force=True)\n", 1156 | "final_rewards = evaluate(agent, env_monitor, n_games=1,)\n", 1157 | "env_monitor.close()\n", 1158 | "print(\"Final mean reward:\", np.mean(final_rewards))\n", 1159 | "\n", 1160 | "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./kungfu_videos/\")))" 1161 | ], 1162 | "execution_count": 0, 1163 | "outputs": [ 1164 | { 1165 | "output_type": "stream", 1166 | "text": [ 1167 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 1168 | " if issubdtype(ts, int):\n", 1169 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 1170 | " elif issubdtype(type(size), float):\n", 1171 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n", 1172 | " if issubdtype(ts, int):\n", 1173 | "/usr/local/lib/python3.6/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", 1174 | " elif issubdtype(type(size), float):\n" 1175 | ], 1176 | "name": "stderr" 1177 | }, 1178 | { 1179 | "output_type": "stream", 1180 | "text": [ 1181 | "Final mean reward: 72.0\n" 1182 | ], 1183 | "name": "stdout" 1184 | } 1185 | ] 1186 | }, 1187 | { 1188 | "metadata": { 1189 | "id": "BVwb5QCk3YUb", 1190 | "colab_type": "code", 1191 | "colab": {} 1192 | }, 1193 | "cell_type": "code", 1194 | "source": [ 1195 | "" 1196 | ], 1197 | "execution_count": 0, 1198 | "outputs": [] 1199 | } 1200 | ] 1201 | } -------------------------------------------------------------------------------- /CEM.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "CEM.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "include_colab_link": true 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | }, 15 | "accelerator": "GPU" 16 | }, 17 | "cells": [ 18 | { 19 | "cell_type": "markdown", 20 | "metadata": { 21 | "id": "view-in-github", 22 | "colab_type": "text" 23 | }, 24 | "source": [ 25 | "\"Open" 26 | ] 27 | }, 28 | { 29 | "metadata": { 30 | "id": "2mZ4ltSLiQ6x", 31 | "colab_type": "text" 32 | }, 33 | "cell_type": "markdown", 34 | "source": [ 35 | "# Environment setup\n", 36 | "\n", 37 | "---\n", 38 | "\n" 39 | ] 40 | }, 41 | { 42 | "metadata": { 43 | "id": "kq7MIGh2iRmd", 44 | "colab_type": "code", 45 | "colab": {} 46 | }, 47 | "cell_type": "code", 48 | "source": [ 49 | "!apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig" 50 | ], 51 | "execution_count": 0, 52 | "outputs": [] 53 | }, 54 | { 55 | "metadata": { 56 | "id": "jxgRIYtyktVP", 57 | "colab_type": "code", 58 | "colab": {} 59 | }, 60 | "cell_type": "code", 61 | "source": [ 62 | "!pip install pyvirtualdisplay\n", 63 | "!pip install piglet" 64 | ], 65 | "execution_count": 0, 66 | "outputs": [] 67 | }, 68 | { 69 | "metadata": { 70 | "id": "kOY42GKDlizl", 71 | "colab_type": "code", 72 | "outputId": "2ecded0e-5b2b-485a-8f17-a9559ba4574c", 73 | "colab": { 74 | "base_uri": "https://localhost:8080/", 75 | "height": 55 76 | } 77 | }, 78 | "cell_type": "code", 79 | "source": [ 80 | "from pyvirtualdisplay import Display\n", 81 | "display = Display(visible=0, size=(1400, 900))\n", 82 | "display.start()" 83 | ], 84 | "execution_count": 0, 85 | "outputs": [ 86 | { 87 | "output_type": "execute_result", 88 | "data": { 89 | "text/plain": [ 90 | "" 91 | ] 92 | }, 93 | "metadata": { 94 | "tags": [] 95 | }, 96 | "execution_count": 3 97 | } 98 | ] 99 | }, 100 | { 101 | "metadata": { 102 | "id": "pR-Ho4ekl6Se", 103 | "colab_type": "code", 104 | "colab": {} 105 | }, 106 | "cell_type": "code", 107 | "source": [ 108 | "!pip install gym" 109 | ], 110 | "execution_count": 0, 111 | "outputs": [] 112 | }, 113 | { 114 | "metadata": { 115 | "id": "npqtnXaohXdw", 116 | "colab_type": "text" 117 | }, 118 | "cell_type": "markdown", 119 | "source": [ 120 | "# Crossentropy method\n", 121 | "\n", 122 | "---\n", 123 | "\n", 124 | "\n", 125 | "This notebook will teach you to solve reinforcement learning problems with crossentropy method. We'll follow-up by scaling everything up and using neural network policy." 126 | ] 127 | }, 128 | { 129 | "metadata": { 130 | "id": "G1bXoJ5alyEl", 131 | "colab_type": "code", 132 | "outputId": "19fb7229-e103-42d2-99d3-833208ab76db", 133 | "colab": { 134 | "base_uri": "https://localhost:8080/", 135 | "height": 164 136 | } 137 | }, 138 | "cell_type": "code", 139 | "source": [ 140 | "import gym\n", 141 | "import numpy as np, pandas as pd\n", 142 | "\n", 143 | "env = gym.make(\"Taxi-v2\")\n", 144 | "env.reset()\n", 145 | "env.render()" 146 | ], 147 | "execution_count": 0, 148 | "outputs": [ 149 | { 150 | "output_type": "stream", 151 | "text": [ 152 | "+---------+\n", 153 | "|\u001b[35mR\u001b[0m: | : :\u001b[43mG\u001b[0m|\n", 154 | "| : : : : |\n", 155 | "| : : : : |\n", 156 | "| | : | : |\n", 157 | "|Y| : |\u001b[34;1mB\u001b[0m: |\n", 158 | "+---------+\n", 159 | "\n" 160 | ], 161 | "name": "stdout" 162 | } 163 | ] 164 | }, 165 | { 166 | "metadata": { 167 | "id": "ZMV_2NM6l4Gg", 168 | "colab_type": "code", 169 | "outputId": "c4a344ce-db15-4966-f9d5-68e3002a6d56", 170 | "colab": { 171 | "base_uri": "https://localhost:8080/", 172 | "height": 35 173 | } 174 | }, 175 | "cell_type": "code", 176 | "source": [ 177 | "n_states = env.observation_space.n\n", 178 | "n_actions = env.action_space.n\n", 179 | "\n", 180 | "print(\"n_states=%i, n_actions=%i\"%(n_states, n_actions))" 181 | ], 182 | "execution_count": 0, 183 | "outputs": [ 184 | { 185 | "output_type": "stream", 186 | "text": [ 187 | "n_states=500, n_actions=6\n" 188 | ], 189 | "name": "stdout" 190 | } 191 | ] 192 | }, 193 | { 194 | "metadata": { 195 | "id": "1CgC3xnuhqSH", 196 | "colab_type": "text" 197 | }, 198 | "cell_type": "markdown", 199 | "source": [ 200 | "# Create stochastic policy\n", 201 | "\n", 202 | "---\n", 203 | "\n", 204 | "\n", 205 | "This time our policy should be a probability distribution.\n", 206 | "\n", 207 | "policy[s,a] = P(take action a | in state s)\n", 208 | "\n", 209 | "Since we still use integer state and action representations, you can use a 2-dimensional array to represent the policy.\n", 210 | "\n", 211 | "Please initialize policy uniformly, that is, probabililities of all actions should be equal" 212 | ] 213 | }, 214 | { 215 | "metadata": { 216 | "id": "AT6CtBAemETU", 217 | "colab_type": "code", 218 | "colab": {} 219 | }, 220 | "cell_type": "code", 221 | "source": [ 222 | "policy = np.ones((n_states, n_actions))/n_actions" 223 | ], 224 | "execution_count": 0, 225 | "outputs": [] 226 | }, 227 | { 228 | "metadata": { 229 | "id": "D6sjcJVamHg9", 230 | "colab_type": "code", 231 | "colab": {} 232 | }, 233 | "cell_type": "code", 234 | "source": [ 235 | "def generate_session(policy,t_max=10**4):\n", 236 | " \"\"\"\n", 237 | " Play game until end or for t_max ticks.\n", 238 | " :param policy: an array of shape [n_states,n_actions] with action probabilities\n", 239 | " :returns: list of states, list of actions and sum of rewards\n", 240 | " \"\"\"\n", 241 | " states,actions = [],[]\n", 242 | " total_reward = 0.\n", 243 | " \n", 244 | " s = env.reset()\n", 245 | " \n", 246 | " for t in range(t_max):\n", 247 | " \n", 248 | " a = np.random.choice(n_actions, 1, p=policy[s])[0]\n", 249 | " \n", 250 | " new_s, r, done, info = env.step(a)\n", 251 | " \n", 252 | " #Record state, action and add up reward to states,actions and total_reward accordingly. \n", 253 | " states.append(s)\n", 254 | " actions.append(a)\n", 255 | " total_reward += r\n", 256 | " \n", 257 | " s = new_s\n", 258 | " if done:\n", 259 | " break\n", 260 | " return states, actions, total_reward" 261 | ], 262 | "execution_count": 0, 263 | "outputs": [] 264 | }, 265 | { 266 | "metadata": { 267 | "id": "jth3-RCQmRrQ", 268 | "colab_type": "code", 269 | "colab": {} 270 | }, 271 | "cell_type": "code", 272 | "source": [ 273 | "s,a,r = generate_session(policy)" 274 | ], 275 | "execution_count": 0, 276 | "outputs": [] 277 | }, 278 | { 279 | "metadata": { 280 | "id": "FFbYwZZHmWQj", 281 | "colab_type": "code", 282 | "outputId": "fd04a047-3d38-4bb9-ee40-fce54b982c65", 283 | "colab": { 284 | "base_uri": "https://localhost:8080/", 285 | "height": 283 286 | } 287 | }, 288 | "cell_type": "code", 289 | "source": [ 290 | "import matplotlib.pyplot as plt\n", 291 | "%matplotlib inline\n", 292 | "\n", 293 | "sample_rewards = [generate_session(policy,t_max=1000)[-1] for _ in range(200)]\n", 294 | "\n", 295 | "plt.hist(sample_rewards,bins=20);\n", 296 | "plt.vlines([np.percentile(sample_rewards, 50)], [0], [100], label=\"50'th percentile\", color='green')\n", 297 | "plt.vlines([np.percentile(sample_rewards, 90)], [0], [100], label=\"90'th percentile\", color='red')\n", 298 | "plt.legend()" 299 | ], 300 | "execution_count": 0, 301 | "outputs": [ 302 | { 303 | "output_type": "execute_result", 304 | "data": { 305 | "text/plain": [ 306 | "" 307 | ] 308 | }, 309 | "metadata": { 310 | "tags": [] 311 | }, 312 | "execution_count": 11 313 | }, 314 | { 315 | "output_type": "display_data", 316 | "data": { 317 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD4CAYAAAATpHZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAFy9JREFUeJzt3Xt0VOW9xvFvQriblgAjN1EE5KeI\ndfWi6AEULahYuqx4qR7vIOAlPVgPVqqoCIJKFdRKLSq2FO/H2ipLFKUqYgWr2IU3/CmtiCagQVAR\naASS88fsYJRc55LsvDyftVzM7Jns/WTceWbnnZ1355SXlyMiImHIbewAIiKSOSp1EZGAqNRFRAKi\nUhcRCYhKXUQkIHmNufGSkk0ZPfWmoKANGzduyeQqs0ZZs0NZs0NZsyPVrIlEfk51jwV1pJ6X16yx\nI9SZsmaHsmaHsmZHNrIGVeoiIrs7lbqISEBU6iIiAVGpi4gERKUuIhIQlbqISEDqdJ66mfUDHgNm\nuvvtZtYdmAc0A9YCZ7l7qZmdAVwClAF3uvucLOUWEZEq1HqkbmZtgd8Cf6u0eDIwy90HAauAkdHz\nrgaGAIOBX5pZ+4wnFhGRatVl+KUUOB4orrRsMPB4dHs+ySLvD7zi7p+7+1bg78CAzEUVkRC99tqr\nDB8+hMLCMRQWjmHmzOkAfPzxOgoLx3DRRedz1VUT+Oqrr3jttVeZM2c2AM8/nzzOXLBgPrfffkuj\n5a/JunXrePvtNwG49dabKS4uYs6c2fz5zw9lbZu1Dr+4+3Zgu5lVXtzW3Uuj258AXYDOQEml51Qs\nr1ZBQZuM/0VVIpGf0fVlU3VZe9zSA4DVl6xuuDC1SCTyoUeP5J3VqxszSq1C2AfiKFtZ27VrQ//+\n/bntttu+sXzGjGmce+7ZDBs2jBkzZrB48UJ69epF27YtKS39nCVLnuWUU35Gfn4r2rRp8Y18cXld\nlyx5hi1btnDkkYdz3XWTAHjhhWfYY49WOzNmOmsm5n6pbg6CaucmqJDp+RkSiXxKSjZldJ3ZUlPW\nsrLklDhx+V4qsraPcm2ISa6qhLIPxE02s3722RZKS7ftsv6lS5dRWDiekpJNfP/7/XnggXkMGjSU\n447rysSJV7Ny5VtMnz6DTp06s2ZNEWPGXMDq1e8zZsxojjzy2J3rWbu2mKuumkD37nvz4Ydr2H//\nvowfP4H160u4/vopbN++jdzcXC6//Co6d+7MaaedSJ8++3Poof3p02d/br75RnJzc+jX72Auvngc\n77//b2bOnE5OTg5t2rThiism8eWXm5g6dRJdu3Zj1ar36NPHGDu2kFtvvY28vDzatm3Hgw/ex6WX\n/orNm0tp3vw/lJRs4t5772bp0pcpK9vBiBGnMnTocXV6zWp6I0i11L80s9bRMEs3kkMzxSSP1it0\nA5aluH4RaSQ/nNdvl2W5uTk7Dzbqa/lZb9b6nNWr3+fyy3/JF198wciRoznkkMPYunUrLVq0AKCg\noD2ffvoprVq1olWrVpx++lk8+ujDnHfeaBYsmE9xcRF33DGHoqIPmTx54jdKHWDVqneZOnU6e+7Z\nidGjz+G9997lkUce5LTTzuCQQ/qzdOmLzJ17N5dfPpHi4iKmTbuJnj17cdFF53PZZVfQu/d+TJly\nNevWreWWW37DZZddQffue/Poo//Ho48+zDHHDMN9JddeO42CgvaceOLxXHzxJQwbNpx27doxcOCR\nPPjgfd/ItGLFPykqKmLWrLv46quvGDnyTI44YjAtW7ZK6XWukGqpLwJOAu6N/n0KeBm428zaAdtJ\njqdfklY6EQle9+57c955ozn66KEUFxfxi1+M5aGH/vqN59R2LeUDDzyIZs2a0bHjnmzatOtvFN27\n702nTsljzr59D2TNmg94883XWbPmA+bOnUNZWRnt2hUA0KpVa3r27AXAmjUf0Lv3fgBcddVkAN5+\n+y1uvPE6ALZt28YBB/QFoFu37nTo0BGAjh0TbN78ZY2Z33hjBStWrKCwcEz0PZaxfv16unXbq8av\nq02tpW5mPwRuBnoA28zsZOAM4I9mNhb4AJjr7tvMbAKwECgHrnX3z9NKJyINrqoj62wOvyQSe/Lj\nHx8DQLdue9GhQwdKSj6hdes2lJb+h5YtW1FS8gkdO3asdh3NmtX82VzlN4XycsjJySEvrzlTpty4\ny3qbN/+6FnNzdz2XpFWrVvz2t7PJyfl6hHnt2uJdMtT2RtS8eXNOPvlkRoz47xqfV1+1nv3i7svd\nfbC793D3/aLbRe4+1N0HufuZ7r4teu4j7t7f3Q9z9/tqW7eIyNNPP8n9988D4NNP17NhwwYSiT35\n0Y8O5fnnnwVg8eJn6d//v3Z+TW5uLjt27KjzNoqKPmL9+vWUlZXx9ttv0qPHvvTt248lS54HYPny\nV3j66ad2+boePfblrbeSb3LXXz+Z1avfp3fv/Vi27CUAFi1ayKuv/qPa7daUs2/ffjz33HOUlZVR\nWlq686yfdDXqRTJERAYOPIJJkyby4ouL2bZtG+PHT6B58+aMGjWW6667mscee5TOnbswbNjwnV+z\nzz774v4Ot912M71796l1G3vvvQ933jmL99//Nwcd9D169uzFqFFjmDbtWhYtWkhOTg5XXHHNLl83\nbtx4brrpeiA5xNOjx76MGzee6dOnct99c2nRoiWTJl3H5s2bq9xuv34Hcd11k3YO7VR20EEH079/\nf8aOPQ8o58QTT6nT61WbnNp+RcimTF/5KJSzCSo+qKrLB0wNYefZLz9M5tqwPB65qhLKPhA3TTnr\n2rXFTJx4OXPmzGvEVFVL9XXdba58JCKyu1Opi0jQunTpGsuj9GxRqYuIBESlLiISEJW6iEhAVOoi\nIgFRqYtIoyorK+PGG6dywQUjKSwcwwcfrAY09W6qVOoi0qiWLFnM5s1f8vvf38Ovf301s2YlC3rO\nnNmMGHEqv/vd3ey1V3eeeOLxnV+zdm0xixYtbKzIdfbaa6+wcuVbAIwb97907dot69vUX5SKSKP6\n6KM1HHDAgUBy7pd169ayY8cO/vnP5Ywf/2sABgwYxAMPzGPYsOH07NmbqVOvYeXKt/jDH+6iU6fO\nrF9fwpVXXha7qXfvuedO8vLy6NSp886pdyubOXNmSlPv1kSlLiLfUPGXw9+Qm7NzPv36qu0vkHv2\n7M3DD9/PqaeeTlHRhxQXF/H5559p6t0UqdRFpFEdfvgA3nhjBYWFo+nVaz/22WffXWY41NS7dadS\nF5FvqOrIOpHIz+oVr8aMuWjn7VNPPYGCgvaaejdF+qBURBrVe++9y7Rp1wKwbNlL9OmzP7m5uZp6\nN0U6UheRRtWrV2/Ky8sZPfpsWrRoydVXTwHQ1Lsp0tS7jURT72ZHKPtA3DTlrJp6V0REmiyVuogE\nTVPviohIk6VSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpd\nRCQgKnURkYCo1EVEApLSRTLMbA/gT0AB0BK4FlgH3AGUA6+7+4WZCikiInWT6pH6uYC7+1HAycCt\nwC3AOHcfAHzXzIZlJqKIiNRVqqW+HugQ3S4ANgD7uvsr0bL5wJA0s4mISD2lNPzi7g+a2blmtopk\nqf8UmFXpKZ8AXWpbT0FBG/Lyar4KeH0lEvkZXV82VZc1NzenxscbQyKRDzHMVZW456tMWbNjd86a\n6pj6mcAadz/OzA4G/gJ8Xukp1V4/r7KNG7eksvlqNeXrKFZWVpa8dGtcvped1yiNcm2ISa6qhLIP\nxI2yZkca1yit9rFUh18GAAsB3H0F0BroWOnxbkBxiusWEZEUpVrqq4D+AGa2D7AJWGlmA6PHRwBP\npR9PRETqI6XhF2A2cI+ZLY7WcQHJUxpnm1ku8LK7L8pQRhERqaNUPyj9Eji1iocGpRdHRETSob8o\nFREJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo\n1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQg\nKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEApKX6hea\n2RnAr4DtwNXA68A8oBmwFjjL3UszEVJEROompSN1M+sAXAMMBIYDJwCTgVnuPghYBYzMVEgREamb\nVIdfhgCL3H2Tu6919zHAYODx6PH50XNERKQBpTr80gNoY2aPAwXAJKBtpeGWT4AuaacTEZF6SbXU\nc4AOwInAPsBz0bLKj9eqoKANeXnNUoxQtUQiP6Pry6bqsubm5tT4eGNIJPIhhrmqEvd8lSlrduzO\nWVMt9Y+Bl9x9O/AvM9sEbDez1u6+FegGFNe2ko0bt6S4+aolEvmUlGzK6DqzpaasZWXlALH5Xiqy\nto9ybYhJrqqEsg/EjbJmR6pZa3ojSHVM/WngaDPLjT403QNYBJwUPX4S8FSK6xYRkRSlVOruXgQ8\nAiwDngR+QfJsmHPMbAnQHpibqZAiIlI3KZ+n7u6zgdnfWjw0vTgiIpIO/UWpiEhAVOoiIgFRqYuI\nBESlLiISEJW6iEhAVOoiIgFRqYuIBESlLiISEJW6iEhAVOoiIgFRqYuIBESlLiISEJW6iEhAVOoi\nIgFRqYuIBESlLiISEJW6iEhAVOoiIgFRqYuIBCTla5RKakbe8Gytz1m/x9Yqn3vPhKOzkklEwqEj\ndRGRgKjURUQColIXEQmISl1EJCAqdRGRgKjURUQColIXEQmISl1EJCAqdRGRgKjURUQColIXEQmI\nSl1EJCAqdRGRgKQ1S6OZtQbeBKYAfwPmAc2AtcBZ7l6adkIREamzdI/UJwIbotuTgVnuPghYBYxM\nc90iIlJPKZe6me0P9AWeiBYNBh6Pbs8HhqSVTERE6i2d4ZebgULgnOh+20rDLZ8AXWpbQUFBG/Ly\nmqURYVeJRH5G1xcndbnARnXm33xCWttOJPIhN+fr2zEW93yVKWt27M5ZUyp1MzsbWOru75tZVU/J\nqct6Nm7cksrmq5VI5FNSsimj6wxFOq9LxevavqwcgA0xfo2b0j6grNmxO2St6Y0g1SP1nwA9zWw4\nsBdQCnxpZq3dfSvQDShOcd0iIpKilErd3X9ecdvMJgGrgf8CTgLujf59Kv14IiJSH5k8T/0a4Bwz\nWwK0B+ZmcN0iIlIHaZ2nDuDukyrdHZru+kREJHX6i1IRkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo\n1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQg\nKnURkYCo1EVEAqJSFxEJSNrXKN0djbzh2caOICJSJR2pi4gEREfqu4l0f7u4Z8LRGUoiItmkI3UR\nkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYDstn9RqvlbRCREKZe6\nmU0HBkXruB54BZgHNAPWAme5e2kmQoqISN2kNPxiZkcB/dz9cOA44BZgMjDL3QcBq4CRGUspIiJ1\nkuqY+gvAKdHtz4C2wGDg8WjZfGBIWslERKTeUhp+cfcdwObo7ihgAXBspeGWT4Auta2noKANeXnN\nUolQrUQiP6Prk6REIh9yc76+HWNxz1eZsmbH7pw1rQ9KzewEkqV+DPBepYdy6vL1GzduSWfzu0gk\n8ikp2ZTRdUpSSckm2peVA7Ahxq9xU9oHlDU7doesNb0RpHxKo5kdC1wJDHP3z4Evzax19HA3oDjV\ndYuISGpS/aD0u8BvgOHuviFavAg4Kbp9EvBU+vFERKQ+Uh1++TnQEXjYzCqWnQPcbWZjgQ+AuenH\nExGR+kj1g9I7gTureGhoenFERCQdmiZARCQgKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVE\nAqJSFxEJiEpdRCQgu+3l7KR+Rt7wLHd/vhWA8+t5KcB7JhydjUgiUgUdqYuIBKTJHqnrwtEiIrvS\nkbqISEBU6iIiAVGpi4gERKUuIhIQlbqISEBU6iIiAVGpi4gERKUuIhIQlbqISEBU6iIiAVGpi4gE\nRKUuIhIQlbqISEBU6iIiAVGpi4gERKUuIhIQlbqISEBU6iIiAWmyl7OTpiOdSw/qotUi9aMjdRGR\ngOhIXWJNR/ki9ZPxUjezmcBhQDkwzt1fyfQ2RESkahktdTM7EtjP3Q83swOAe4DDM7kNkYaQzm8I\n6UrnN4ym+ptNU8yd7j6SrdyZHlP/MfBXAHdfCRSY2XcyvA0REalGTnl5ecZWZmZ3Ak+4+2PR/SXA\nKHd/N2MbERGRamX77JecLK9fREQqyXSpFwOdK93vCqzN8DZERKQamS71p4GTAczsB0Cxu2/K8DZE\nRKQaGR1TBzCzG4AjgDLgYndfkdENiIhItTJe6iIi0ng0TYCISEBU6iIiAWmSc7+YWVtgLtAJ2Ayc\n6+7rzOxg4A6SUxS87u4XRs+/DDglWn6tuy9owKxdSf5lbUugGfBLd19uZkOAacAOYIG7T4me32jT\nLJjZlcDQ6G4u0Nnd+8Qxa7T98cCZwDbgInd/Jab7wLnAFOBf0aJn3H1qHLNWytwJeAc40d2fj2NW\nM9uTZA+0AloAl7r7y3HMGm0/D5gD9CLZvePd/cVM522qR+pjgH+5+yBgKjA5Wn4LyXIZAHzXzIaZ\n2b7AacBAYDgww8yaNWDWS4G/uPtRwIQoL8BtwEnAAOAYM+tbeZoFYFT0nAbj7lPdfbC7Dya5890V\n16xmdiDJ/68/AsaS/H8L8dwHAB6qeG3dvWIfiGtWgN8A/650P45ZzwTmRT9bV5B844xrVoCzgM3u\nPpDkz8yMbORtqqW+H/APAHdfAgw0sxbAvpWOFucDQ4CjgCfd/St3LwE+APo2YNb1QIfodgGw3sx6\nAhvc/UN3LwMWkJxiIRbTLERHFBcCt8c463DgYXff7u6vufs1Md4HdhHnrGZ2NLAJeCPOWd19hrvf\nH93tDnwU16yRe0ke5AGUAB2ykbdJDr+Q3NmOB/4cHTHuA3QENlZ6zidAF+BTki/gt5e/0TBRmQn8\nw8zOBr5D8p23cxWZepH8HpZXWl4SPfeLhom60whgobtvNbO4Zu0B7DCzp4DmJH9YSojnPgBwZKWs\n44GPiWHWqGSuAU4geQQJ8f3ZIto/5wP5wNHEOKu7byM5VAhwCXA/Wcgb+1I3s/OB87+1+Brge2b2\nIrCY5Df8bdVNUZC1qQuqyfokySPKqWY2HLgp+q8umRo66zXuvpDkr4Zj65mpobN2Ap4ChpEcFrqb\nZBHVJVNDZ30AmOTuT5jZ4cCfgGPrmKkx9te73P0zM6vuS+OStWJ/PcTMjgf+CJxbx0xZncKkprxm\ndjHwA+CnQKKOueqcN/al7u53k/yB/baFAGa2B8kf5hK+HuYA6EZy2oJiwKpY3iBZzexJYGJ09xng\nd+w6nUJFpq9ooGkWqntdow+h93L31dGiWGY1s2uBd9y9HHjRzHoQ033gW48vNbMEySOx2GU1s78D\nzcyskORvZIcCp8c065FmVuDuG919gZn9iRjsA9XljTKPIlnmP3P3bWaW8bxNckzdzI43s4oPRc4k\nOfa0DXjHzAZGy0eQPJJ7FviJmbWIzkTpBrzdgHFXAf2j24cA70WF+R0z6xGNXw8nOcVCHKZZOJjk\nWQ8AxDjrk0RHu2a2P/BhXPcBM/uVmZ0e3e4HlLh7aRyzuvsAdz/M3Q8DniB5VtGKOGaNcpwDYGYH\nEeN9IMrYE7gAGOHu/4GdQzIZzRv7I/VqPAdcbGbLgA0kjyQgOU4128xygZfdfRGAmd0FvEDy1KAL\now/8Gso0YI6ZnRrd/5/o3wtJ/loOyTMj3gXeNbPlZvYS0TQLDZizQhd2Hc6KXVZ3XxadJbA0WlSx\n/TjuA/cD88zsApI/c6NinLU6ccw6BZhrZiNInjJ8YYyzQnI4pgOwoNLQ1jGZzqtpAkREAtIkh19E\nRKRqKnURkYCo1EVEAqJSFxEJiEpdRCQgKnURkYCo1EVEAvL/Ggy8N+Bkb04AAAAASUVORK5CYII=\n", 318 | "text/plain": [ 319 | "" 320 | ] 321 | }, 322 | "metadata": { 323 | "tags": [] 324 | } 325 | } 326 | ] 327 | }, 328 | { 329 | "metadata": { 330 | "id": "ckHqupNuh1g1", 331 | "colab_type": "text" 332 | }, 333 | "cell_type": "markdown", 334 | "source": [ 335 | "# Play the game\n", 336 | "\n", 337 | "---\n", 338 | "\n", 339 | "\n", 340 | "Just like before, but we also record all states and actions we took." 341 | ] 342 | }, 343 | { 344 | "metadata": { 345 | "id": "PR_7INzima8a", 346 | "colab_type": "code", 347 | "colab": {} 348 | }, 349 | "cell_type": "code", 350 | "source": [ 351 | "def select_elites(states_batch,actions_batch,rewards_batch,percentile=50):\n", 352 | " \"\"\"\n", 353 | " Select states and actions from games that have rewards >= percentile\n", 354 | " :param states_batch: list of lists of states, states_batch[session_i][t]\n", 355 | " :param actions_batch: list of lists of actions, actions_batch[session_i][t]\n", 356 | " :param rewards_batch: list of rewards, rewards_batch[session_i][t]\n", 357 | " \n", 358 | " :returns: elite_states,elite_actions, both 1D lists of states and respective actions from elite sessions\n", 359 | " \n", 360 | " Please return elite states and actions in their original order \n", 361 | " [i.e. sorted by session number and timestep within session]\n", 362 | " \n", 363 | " If you're confused, see examples below. Please don't assume that states are integers (they'll get different later).\n", 364 | " \"\"\"\n", 365 | " \n", 366 | " reward_threshold = np.percentile(rewards_batch, percentile)\n", 367 | " \n", 368 | " elite_states = [s for i in range(len(states_batch)) if rewards_batch[i]>=reward_threshold for s in states_batch[i]]\n", 369 | " elite_actions = [a for i in range(len(actions_batch)) if rewards_batch[i]>=reward_threshold for a in actions_batch[i]]\n", 370 | " \n", 371 | " return elite_states,elite_actions" 372 | ], 373 | "execution_count": 0, 374 | "outputs": [] 375 | }, 376 | { 377 | "metadata": { 378 | "id": "EgJAEdXImgGa", 379 | "colab_type": "code", 380 | "colab": {} 381 | }, 382 | "cell_type": "code", 383 | "source": [ 384 | "from collections import defaultdict\n", 385 | "def update_policy(elite_states,elite_actions):\n", 386 | " \"\"\"\n", 387 | " Given old policy and a list of elite states/actions from select_elites,\n", 388 | " return new updated policy where each action probability is proportional to\n", 389 | " \n", 390 | " policy[s_i,a_i] ~ #[occurences of si and ai in elite states/actions]\n", 391 | " \n", 392 | " Don't forget to normalize policy to get valid probabilities and handle 0/0 case.\n", 393 | " In case you never visited a state, set probabilities for all actions to 1./n_actions\n", 394 | " \n", 395 | " :param elite_states: 1D list of states from elite sessions\n", 396 | " :param elite_actions: 1D list of actions from elite sessions\n", 397 | " \n", 398 | " \"\"\"\n", 399 | " \n", 400 | " new_policy = np.zeros([n_states,n_actions])\n", 401 | " \n", 402 | " #Don't forget to set 1/n_actions for all actions in unvisited states.\n", 403 | " state_action_dict = defaultdict(lambda: [0]*n_actions)\n", 404 | " for i in range(len(elite_states)):\n", 405 | " state_action_dict[elite_states[i]][elite_actions[i]]+=1\n", 406 | " for i in range(n_states):\n", 407 | " if i not in state_action_dict:\n", 408 | " new_policy[i] = [1/n_actions]*n_actions\n", 409 | " else:\n", 410 | " new_policy[i] = [p/sum(state_action_dict[i]) for p in state_action_dict[i]]\n", 411 | " \n", 412 | " \n", 413 | " return new_policy" 414 | ], 415 | "execution_count": 0, 416 | "outputs": [] 417 | }, 418 | { 419 | "metadata": { 420 | "id": "mXA9rz3viHzX", 421 | "colab_type": "text" 422 | }, 423 | "cell_type": "markdown", 424 | "source": [ 425 | "# Training loop\n", 426 | "\n", 427 | "---\n", 428 | "\n", 429 | "\n", 430 | "Generate sessions, select N best and fit to those." 431 | ] 432 | }, 433 | { 434 | "metadata": { 435 | "id": "UrtHpFtWmtEK", 436 | "colab_type": "code", 437 | "colab": {} 438 | }, 439 | "cell_type": "code", 440 | "source": [ 441 | "from IPython.display import clear_output\n", 442 | "\n", 443 | "def show_progress(batch_rewards, log, percentile, reward_range=[-990,+10]):\n", 444 | " \"\"\"\n", 445 | " A convenience function that displays training progress. \n", 446 | " No cool math here, just charts.\n", 447 | " \"\"\"\n", 448 | " \n", 449 | " mean_reward, threshold = np.mean(batch_rewards), np.percentile(batch_rewards, percentile)\n", 450 | " log.append([mean_reward,threshold])\n", 451 | "\n", 452 | " clear_output(True)\n", 453 | " print(\"mean reward = %.3f, threshold=%.3f\"%(mean_reward, threshold))\n", 454 | " plt.figure(figsize=[8,4])\n", 455 | " plt.subplot(1,2,1)\n", 456 | " plt.plot(list(zip(*log))[0], label='Mean rewards')\n", 457 | " plt.plot(list(zip(*log))[1], label='Reward thresholds')\n", 458 | " plt.legend()\n", 459 | " plt.grid()\n", 460 | " \n", 461 | " plt.subplot(1,2,2)\n", 462 | " plt.hist(batch_rewards,range=reward_range);\n", 463 | " plt.vlines([np.percentile(batch_rewards, percentile)], [0], [100], label=\"percentile\", color='red')\n", 464 | " plt.legend()\n", 465 | " plt.grid()\n", 466 | "\n", 467 | " plt.show()" 468 | ], 469 | "execution_count": 0, 470 | "outputs": [] 471 | }, 472 | { 473 | "metadata": { 474 | "id": "qkmNQga4m1PR", 475 | "colab_type": "code", 476 | "colab": {} 477 | }, 478 | "cell_type": "code", 479 | "source": [ 480 | "policy = np.ones([n_states, n_actions]) / n_actions" 481 | ], 482 | "execution_count": 0, 483 | "outputs": [] 484 | }, 485 | { 486 | "metadata": { 487 | "id": "LTJEVEovm3ya", 488 | "colab_type": "code", 489 | "outputId": "357a6bcb-e0a9-4608-bc63-cbf3250585c9", 490 | "colab": { 491 | "base_uri": "https://localhost:8080/", 492 | "height": 283 493 | } 494 | }, 495 | "cell_type": "code", 496 | "source": [ 497 | "n_sessions = 250 #sample this many sessions\n", 498 | "percentile = 50 #take this percent of session with highest rewards\n", 499 | "learning_rate = 0.5 #add this thing to all counts for stability\n", 500 | "\n", 501 | "log = []\n", 502 | "\n", 503 | "for i in range(100):\n", 504 | " \n", 505 | " sessions = [generate_session(policy) for _ in range(n_sessions)]\n", 506 | " \n", 507 | " batch_states,batch_actions,batch_rewards = zip(*sessions)\n", 508 | "\n", 509 | " elite_states, elite_actions = select_elites(batch_states,batch_actions,batch_rewards,percentile)\n", 510 | " \n", 511 | " new_policy = update_policy(elite_states,elite_actions)\n", 512 | " \n", 513 | " policy = learning_rate * new_policy + (1-learning_rate) * policy\n", 514 | " \n", 515 | " #display results on chart\n", 516 | " show_progress(batch_rewards, log, percentile)" 517 | ], 518 | "execution_count": 0, 519 | "outputs": [ 520 | { 521 | "output_type": "stream", 522 | "text": [ 523 | "mean reward = -80.132, threshold=5.000\n" 524 | ], 525 | "name": "stdout" 526 | }, 527 | { 528 | "output_type": "display_data", 529 | "data": { 530 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAD4CAYAAAAjBKUeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8leX5+PHPmdl7khDCvlnKEkVB\nBFfdrTjqqHVWq8Wf46sttbWuav1+WweCA8VVrVvrRKkICkpVhkIAuSGEkZCQvXPOyVm/P56TECAQ\nSELOSc71fr14mdzPfZ7nSvBwnXub/H4/QgghhAhN5mAHIIQQQogDk0QthBBChDBJ1EIIIUQIk0Qt\nhBBChDBJ1EIIIUQIswY7gPaUl9fLVHQhDkFaWpwp2DEcjLyXhTg0B3svS4taCCGECGGSqIUQQogQ\nJolaCCGECGEhOUYthOhZSqkxwAfAY1rreUqpt4G0wOVk4FvgISAPWB0oL9daX9TjwQoRZiRRCxHm\nlFIxwFzgi5aytglYKfUCsGDPJT29RwMUIsxJ17cQwgWcBRTve0EppYBErfX3PR6VEAKQFrUQYU9r\n7QE8Rk7ezy0Yre0WmUqpd4As4Emt9b96IEQhwpq0qIUQ7VJK2YGpWuulgaJK4G7gUuA84AGlVL9g\nxSdEuOixFrVS6jFgMuAHbtFar+ypZ/c2zV43uxtLqXBWEW2NYkjCQGwWGwAen4daVz1+fFhMFuLt\ncfjws612BxWOKgCirJGkRqUQb48j0hpBWVMFuxtLsZqtRFjsOL0unB4XAHaLjZTIZMwmExWOSlxe\nNwDx9liSIhOpb26gwlFJhaOKRncTyZGJJEUmEmOLxoSJJo+DGmctFc4qIi0RpEQlE2ePxWa2Utyw\nm3p3A2cNPI1oWxRen5ddjSVUOKoCz/dT66qn2lVDenQqQxMH4fA4qXHV0fb4VbvFRmpUMiZMlAdi\nqXRU4fF79vvdxdpiOD13BnH2WJq9brbUbEVX5VPvbjjg7zveHscZA08hyhpJpaMaXZ3f4d+Rx+em\n0llNXXM9ADG2aFTSUAB0dT6N7qb9XmPG3Pq7q3JW0+BubH3tgLj+RFkjaXI78Pp9WM0Wjk4dRaQ1\nssNYjqCTgNYub611PfBi4NsKpdQqYARQEoTYhAgbPZKolVInAcO01scrpUYCLwDH98SzQ1GVs5ot\n1QWUOSqocdbiw4fVZCE5Mol6dwPf7/4Bh8fRWt9mthJri8Xn91HXXI+fPUnMbDJjMZlx+/ZPWqFi\nV30Jl46YybN5/6SksfSIP2916Y8ckzGeb3evajdhtqfcUcl5g8/g0dVP0eg5tNfsa2nh15163YH8\ncvj5TOsf1LfJJGBtyzdKqRnAuVrr2wMT0MYBm4MVXG/0449ryM0dSFJSMrNn387DDz/KrFnXc/vt\nv2fw4KHBDk+EqJ5qUZ8CvA+gtf5JKZWklIrXWtf10PODotnbzLJd/2Vz9VYqHFU0e5vx+r2trbAD\nibfHMSn7eNKiU6l21rCleitNHicWk5nBCbkkRyZhMVvw+DxUOqpx+9wMSxxMdlwWJqDR3USFo4p6\ndwNOj5PkyESyYvvh9/txeV1EWiOJtERgwoTD66TCUQl+SIlKJtoahQ8/ta46ql01xNliSY1KJjUq\nJdASrKHGVUOT24EfP9HWKBIi4kmJTMbldVHpqKLB3USzt5mMmDS+272GteXreeC7R/D5fUxIP5pB\nCbnEWKMBiLXHkBiRQFF9MdvrCom1x5AUkYjFtGdUxuFxUuGsBCA1MiUQTzJ2i32/393q0rV8WPAZ\nXxQuI8YazSkDpjEqWZEWlYqpnQ36/H549ae3WFu+nk1Vm3F5mzlz4CmkRaUe9O/IYjKTHJVEgj0e\nk8lEpaOKTYGW+IikoaREJe/3Go/PQ5WzJtAzkUS8PQ6A2uZadtbtwuP3EGWNxGqyYjVbGJ0y8qAx\ndBel1ETgEWAg4FZKXQjMBPoBW9tUXQ5cqZT6L2AB/qa13tUjQfYRn3zyIZde+iuSkpJ5+OFHgx2O\n6MA1Dy/plvu8MPvkLr2+pxJ1JnvWXgKUB8r6ZKL2+rz8t2QlC7ctprbZ+BGjrVFEWaMwm8yMSRnJ\niORhZMdmkhSRhMVsxu01ulL9GP/QW8yW4P4QB9AvJuOg1wcl5O71/chkxdwfn2Vb7U4uHHYeM3Km\ntvu67Nh+HNdvYpfjOy13OippKIUNu5iYPo5Ia0SHr7lmzOU8vHIONa5azhp4KmcPPv2wn5scmcSw\npCEd1kuPTtuvLCUqicEJAw/7md1Fa70amN7OpZv3qecBruqBkI64hQs/4rvvVtDY2Eh5eRkXX3wZ\n/fvnMH/+k1itVtLTM/jDH/5MXt5a3njjVZqampg16za2by/gnXfexGQycckll3PKKafz1VdLeOON\nV7FYrCg1kptvvo2FCz9i3bofqampZufOHVx22RVkZGSyfPmXbNtWwF//+n9ce+3lfPJJ64o4mpoa\neeih+6ivr8fr9XLrrXcydOiwIP6WRKgI1qzvkD5IoCs2VW3hrc3vU9pUjt1s44zck5meM5U4e2yH\nr82ISe+BCHuW3WLjlvE3UOuqa7eVeSQMiO/PgPj+h1w/zh7LbRN+y/a6Qiamjz2CkYn2JE8c0633\nq1q9/pDqbdtWwAsv/IuGhgauuupSkpKSmDPnaeLjE3jqqTksXbqY1NQ0tm7N5/XX38PjcXPvvXfx\n8suv09zs5sEH7+H446fy8svP88wzL2K327n77tmsW/cjAFu35vPMMy9QVFTIPffcxUsvvcbQocO5\n/fbfk5mZuV88b731OscddwLnnvsLtm0rYM6cf/D440916+9G9E49laiLMVrQLbLoQxNQal111Lrq\n+KE8j893fInJZGJq9mTOGngqCRHxB3yd3+9nd1UTGcnRmNvrl+0jrGZrjyXpzkqNSiE1KiXYYYge\nNG7cBKxWK4mJicTExLBz5w7uuutOAJxOJwkJiaSmpjF06DDsdjv5+ZsZMGAgERGRRERE8vDDj7Jh\nw3pKS3dz++2zAGhsbGD37t0AjBlzNBaLhbS0dBobDzyZsUVe3jpqaqpZtGghAC6X8wj95KK36alE\n/R/gPmC+UmoCUByYQdqr+fw+Pi74D4t27BnHSI1M5poxl5Mbn9Ph6/+9vICPV+xgcFY8l506nMFZ\nB07qLfx+P3kFVSxeXci24jpu/MUYRg3ckwRdzV7MZrBZu7frfGdpPd/9VMr5Jw7GapFVfaL7HGoL\nuLv5fHsmZZpMZlJSUpk379m96qxZswqbzVhxYTZb8Pt9e1232Yzu7kcfnbdX+cKFH2Gx7HkPtl3F\ncCA2m5XbbruTMWOOPuyfRfRtPfIvrtZ6BbBaKbUCeAL4XU8890hyelzMX/cSi3YsITUymZNzTuT8\noWcz+9hbWpO0q9nLMx+s58FXVrF4VSGrdTmfrypk4/YqfthczscrdhAVYaGguI4H/7mKpT8cfF6O\n3+/nzSX5PP72WtYXVNHk8jD3vTx27DY+85TVOLjruW/5+xs/duvPWt/UzJx31vHptztZm1+x33VX\ns5eH/7WGz1cVdutzhTiSNmxYh9frpaamhqamRsxmM9u2FQDwzjtvkJ+/Za/6ubkD2blzB01NTbhc\nLm699SZycnLZvn0b1dXG0sjnn59PeXnZAZ9pNpvxer3tXhs1agzLln0JGN3yb7zxajf8lKIv6LEx\naq317J561pHW6G7iqbUvsL1uJyOTh3PN6MuIthmzmHeVN/Dtxq1kp8bwxZoitu4yJpO1/Lctm9XM\nHy6bQJPTw9MfrOeVRZriikZS4iMZmBnHiNyk1rotSfo/KwvJSo3hN+eMorzGwdPvr+cfb/zA9PHZ\nfLexlOp6FzX1LhwuD1ERXf/r9fn9PP/JT1TXG+uu8woqmaj2Hkv/Oq+EzYU1FJU1MO3oLCLsoTkR\nToi2MjOzuPvu2ezaVcj1199Ev37ZPPTQfdhsNlJT0zjvvJmsX7+utX5UVBTXXvtbbr31JgB++cvL\niIqK4pZb/oc77rgFu93GsGGK1NT9Jwy2GDduAn/+8x/4298e2e/ahRf+kgcfvJebbroOn8/Hrbfe\n0f0/tOiVTIfSJdPTysvrQy+oALfPwz9WzaOooZhjMyfwqxEXYTKZMZtMuJq93PPi95RV71kDPXlU\nBhdOH8KP+RU0u30kxNrZXFjDhm1VnD9tMMePNobud1c18cgbP1BZZyREE3DDz0dz7EhjlvXHK7bz\n3rICslJjuPPS8STEGEuTlq8r5o0v8nG4jHXUKfGRVNY5+cNl41EDkjgUm3ZU0+zxcfSQ/cdoP19Z\nyOtfbGH0oGR27K7HZjXzj5tOwBQYU/f6fPxx/rdU1BrjaVecPpwZE4yJXA6Xh7yCSiaNSG+tL7pX\nWlpcSP9iQ/W9vHDhRxQUbGXWrFuDHYoIYT25POtg72XZ6/swfbFzGUUNxUxMG0/p2uH8v0Xf0Oz2\nMvXofvh8fsqqHUw9uh/ZqTFERViZenQ/zCYTJ0/YMwu5JTm3lZkczT1XH8vmwhpczV5e/Vzz3Ecb\nqW9y4/X6eG9ZASnxEdxxybjWJA1w4tFZHDsig+83leL3Q6TdwjMfbKCgpI6c9DieeGctk0ZmcMrE\n9mdB/5hfwbx38/D5/Zw+KYeBmXF8nVfC0YNTGDcslXeXbSU2ysZ154zizS+28O3GUorKG8lJN2ax\nr9pUTkWtk2NUGj/mV/D5qiJOGp+N2WTiraX5fPVjMTFRNkYPDO3JZEIIEaokUR+GKmc1n23/gjh7\nLKaSMWzaUU5GcjRer4+vfjQOHuqXEs0Vpw/v1GSu2CgbE4Yb3WZJcRE8+tZa/vW5sfFTVISFWy4a\nS2Ls/uuCI+wWTjw6C4CKWqM1X1BcR0ykjc1FtWwuqsXt8XHGcQP2el3+rlqeeX89VouJpLhI/rNy\nzxjzxu3VfLRiO81uH1edMYKEGDtHDU7h242l5BVUkpMeS22Diw+/2YbJBBdOH4LdZmHF+t2sL6hk\nSHYC/11vzH7dXlIniVqElLPOOjfYIQhxyCRRH4b38j/B7XPzs/SzeOff5WQmR3P/tccC8Om3O1i5\nqYxrzx7VLTOuR+Qmcc/Vk9i8s5raxmYmDE+jf1rHa7FT4iOJj7axraQOZ6A7PD7GzltL8ykqb+CC\nk4aQFBeBq9nL/A/W4/H6ufmCoxiek8jbS/Ox2yxMGpHOm0vyyd9Vy9ghKRw3yuh+Hz04GROwWpeT\nGGvn7S+3UtvQzMkTsklPiub0STn8d/1u/vX5Zo4fnUmzx5ghW1jW8dIUIYQQ7ZNEfYh2N5byQ9k6\ncuNyyFsdhd/v4pJThrYuVTp3yiDOnTKoW5+ZnRpDdmrMYb3GZDIxqF88a7dWUl3vYnBWPL85dxRP\nvreeFet3s0qXcdUZI9hZ2kBlnYuzj89l7FBju8xfnzGi9T53XjqeH7aUc9TglNbx5fhoOwP7xbOt\npI4FH9dhMsHFM4bys2ONWe4DMuI4c3IuC7/dwYffbMduNWM2m9hZeuBE7ff7WbO5nKH9E/fq0hdC\nCGGQRH2Ivti5HICYesXqHTUcPSSFo4ccfD/oYBmUZSRqvx8mjUgnIymae6+exNd5Jby5ZAvPfrQR\nE5CaEMk5Jwxs9x42q7l1Iltbvzx5KN//VEp6YhQjcpMYkBG31/VfnDiIn3ZUsa2knsmjMyitcrSO\nu7c3G3zD9iqe/Pd6ZozP5oqftXseshBChDXZueIQ1DXX833pGmJMCaz+3kK/lGiuO2dUsMM6oLYb\npxwTWEplNpuYNjaLP11xDOlJUfiBK36miLAdXjf98JxEfnW64vRjB+yXpAGsFjM3/nwMU8Zkcs4J\nA8nJiMUPFJW336pevtbYoG5byZHd9r3B4ebZjzbw/U9H/vQuIYToTtKiPgTLilbg8Xlw7exPfEwE\nt188jtgoW7DDOqBB/eKxmE0MyoonJWHv84yzUmO49+pJVNQ6D2nMuzNSE6O4NvBBZkC6kcx3ljXQ\nPz0Wt8fX+rtrcLj5YUs5AEXljXh9Pizm7v/s2Oh088gbP7KjtJ7Vupyc9Fj6pRzekIIQQgSLtKg7\n0OBuZGnhN9hNkTSXZXHGsQP2S36hJibSxh8un8AN545u93qk3XrEkvS+BmQYzykoruWhV1Zzx1Pf\ntO5u9u2G3Xi8fuw2Mx6vj5LKA58Dnb+rlvyiWgCanB7+8vx3LPp+Z4fP9/v9zHlnHTtK6xmek4jb\n4+P5T37C6/N1+Nqe4PH6cHtCIxYhRGiSRN2B/+xYitPrxFw2HKvJxpSj9l8DHYqGZieExAeKrNQY\nLGYTK/J2U1jWQLPbx9x383jp059Y9H0hFrOJn00ylo3tLG1/+3e3x8ect9fy2Ns/4vb4WLu1gqLy\nRt75citFbWaUr99WyRPvrGvd/AVA76whv8iYvf77y8YzeXQGBcV1rV3uR4LH6zvkDwJPvLOOB15e\niS8ENx4SQoQGSdQHUe2s4auiFcRZ46nensGkEWnERcvM5MNhtZjplxKDH0hLjOR/fjmOqAgLy9aW\nUFnnZKJKY/QgY41129nhP22v4vXFW2h2e9mwrYpGpweHy8vG7VWtLXKvz89Ln23C5/PT7Pby4sJN\n/JhfwZrN5a33WRLYP/2s43Mxm0ycF5iZn7+rttM/09Zdtfz1n6v4y/Pf8dArq/nP9zupa2oGjCT9\nx/n/Zf6HGzu8j7PZw8bt1RSVN7K1C/EIIfo2GaM+iCWFy/H4PMTUjwa/hRnjD/2MY7HH8JwEiisa\nufbsUQzPSeR/f3sCFbUOrBYz6UlRuD0+TOxpUReWNTDn3XU0u30kx0e0HjoC8P1PpeQVVJESH8mQ\n7Hi+/6mM1xdvIT7G1rof+dr8CqYc1Y/qehc/bC6nf1osQ7MTAOPDgsVsorSq/W72XRWNNDrcDM9J\nbPe63+/n1c83s2N3PTGRVhyuJvJ31fKfVYX87frJ7Agse6usK6OovKF1iMHh8rBi/W6mje3Xus6+\noLiutSX9/cYyhvVv/5lCiPAmifog8msKMGNh28Y4hmTFMyS742Moxf4umjGUM44dQGpiFADRkVYG\nRO6ZMd6SsHeWNlDX1MzcQJK2W8188t8duD0+0hIj8Xj9fLuhFD9wwuhMzp06kMKyBr5YUwQYO7tF\n2Mys31aFx+tj+dpivD4/J0/Ibl0LbjEbz9pd1YTf799rD3K/38+89/KoqHHwj99NISHGTlWdk+hI\nK5F2462yWpezY3c9x45M57c/H0NdUzOv/mczqzaVsbmolu1tZq8v+n4n155tTKr77LudfLRiOy63\nl7Mm5wK0jrkDrNRlXHrqMMxmIx63x4fNKh1eQgjp+j4gt8/DrobdmF3x4DdzyanD5GCJToqwWVqT\n9IHkZMTR5PLw4D9XUVHr5LwpAzlzci4NDjcut5fjRmUwYVgaLSO5Y4elEB9t509XTGRk4JSxn08d\nxPjhaTibvazcVMbnqwqJtFuYPHrv9eAZSdE0Oj3UO9x7lReWNVBa1YTX52dFXgkVNQ7uevZbHntr\nLT6/H6/P2HPdbDJx/omDAWMTmGlj+wGQt7USXVgDQEp8BN9uKG1t5a/SxtGHS9fsaj0HeUuRUXei\nSqOusRm9sxowuv1nPb6sdQtWIUR4k0R9ALsaivH6vThr4jh+dAZDshKCHVKflhuYHV5e4+TUif35\n+dRBnHZMDjGRRkv22JEZHDPC2Ac9wm5B5RjJOTrSxm0Xj+Xeqydx8oRsxgV2WXtx4U80Oj2cP21w\na2u4RWaKcSTpvt3fKzftOUd42dpi3v96G80eH1uKalm6ZhdvLdnK7qomThzbj4zk6Na6KicRu9XM\n2q2V5BfVkpkczblTBuH1+Vn0/U52VTS2zmivrHOybmslPp+frcV19EuJbj2wZcmaXdQ2uHju4434\nfH6yDnNXOiFE3yRd3wewo87oTvU1JjDzrCFBjqbvGzUwmfeXb+OM4wYwc9pgTCYT0ZFWfnPuaIor\nGumfFovP52dY/wSGZCfs1S1stZhbN18ZnpNIVIQFh8vLsP4J7Z4alhlIsrsrm1rHhf1+P6s2lWG3\nmTlqUAqrN5dTWu2gX0o0tQ3NvL54Cz6/kTxnThu81/1sVgsjcpNYt7USADUgkeNHZ/DRN9v5YnUR\njYGW+xnHDeCz73ay5IcikuMjcDZ7GZqdgMpJpF9KNKs3l5O3rZJmt4+Z0waTm7n/hjJCiPDT6USt\nlDoJeBu4Rmv9caBsLPA04AfWaa1vDJTfCVwUKL9Pa72wq4EfaTvqjJOkEs1pIbHMqa8b1C+ep24/\nab9xWWOrVuOcbLPZxB9/NfGg97FazExU6azaVMY1Z43E3M5wRWuibtOiLixroLTawaQR6Zw8IZvV\ngZnjF80YSl1jMy99uon0pCjuuGRcuzP/jxqcsleitlktXHzyUJ5+fz3frN+NxWzinOMHkr+rlvUF\nVTicxhKyYf0TMZtN/OmKiby1NJ9la0tQOYmt49g9RSk1BvgAeExrPU8p9RIwEagMVPm71voTpdTl\nwK2AD3hWa/18jwYqRBjqVKJWSg0Bbge+2efS48AtWuuVSqnXlFJnApuAS4DjgQRguVJqkdba24W4\nj7iCmp34vRYGp2QFO5Sw0V2Tp379M8UlJw8lOrL93eP2TdR1Tc188PU2wNgbfXhOIsNzEom0Wxgb\n+JCQGBvBwH5xxB9ged6YwXuO8Wzplj9GpTG8fwKbi2oZPSiZ6Egrl5w8jKffX8/WYmPS2bD+xpBK\ndKSNq84cyZnH5ZIcH9E6qawnKKVigLnAF/tc+mPLh/A29f4CHAs0AyuVUv/WWlf1WLBChKHOtqhL\ngJlA66dppZQdGKS1Xhko+gg4FegHfKq1bgbKlVI7gFFAXqejPsKcHiflznJ8jUkMzpIlM72N1WJu\nPdWsPXHRNqIirOyuamLD9iqe+nceDpeX/mmxHDXEOC1s9uUT9npNS6v+QDKSosnNjMOEcZY4GCeZ\nXX66Yu6761rHoQdnxfO3GybzdV4JrmYv6Ul7T7JrO/bdg1zAWcAfOqh3HLBSa10LoJT6BpiC8V4X\nQhwhnUrUWusmAKX2Ou0oFahu830ZRpKuBMrbKQ/ZRF1Yb2yS4WtMYFA/WZLV15hMJjKTo9lZWs9L\nCzfR7PZx6SnDmDEh+6AJviOzL5uwX1lOeiz/d+MJe5VZLWamj8vu9HO6m9baA3j2eT8DzFJK3Y7x\nnp0FZNL+e1kIcQR1mKiVUtcB1+1TfI/WelEHLz1Q313Ir3HaUW9MJKMxkdx2TogSvV9mchTbSuqo\nrHNy5uQBnDYpp8v3bO8Yz17sFaBSa/2jUmo2cC+wYp86If9eFqIv6DBRa60XAAsO4V7lQNv+wWyg\nOPBHtVMesgrrjPDSIzP62j++IqBlnDopLoJzD3AmdzjTWrcdr/4QY5LoOxit6hbZwLc9GZcQ4ajb\n1lFrrd3AJqXU1EDRTOAzYAlwtlLKrpTKwnhzd7wRchDtqN2F32thSJpMJOurRg5MJsJm4VenD99v\nnbUApdS7SqmWdWjTgfXAd8AkpVSiUioWY3x6eZBCFCJsdHbW99nAncAIYKJS6v9prU/HWLYxXyll\nBr7TWi8O1H8OWIaxPOtGrXXInuvn8XmocFXgd8QxJFc2OemrhmYn8OTt09pdvhVulFITgUeAgYBb\nKXUhxizwN5VSTUADcLXW2hHoBl/EnqWWcpqIEEeYyR+Cx+uVl9cHLaii+mL+tvJxPGX9+eP0qxiY\nKZPJROhKS4sL6U8awXwvC9FV1zy8pFvu88Lskzusc7D3smwhuo9dDcY5xf6mOPqlyBaOQgghgksS\n9T5aEnW8OZUIm0wkE0IIEVySqPexs85YQ50Vm9lBTSGEEOLIk0S9j6KGEnyuKHJSk4IdihBCCCGJ\nuq1aVz0ObxP+pjiyZHxaCCFECJBE3UZxYHza1xRHdpokaiGEEMEnibqNkqZSAPyOWPolS6IWQggR\nfJKo2yhtMs4biLcmy9ahQgghQoIk6jZK6ssAyI5LD3IkQgghhEESdRu7G8vwuSLJSZWtQ4UQQoQG\nSdQBTo+LRm8DfmcMmSnRwQ5HCCGEACRRtypzGOPTfmcMaQlRQY5GCCGEMEiiDihrNBK1zxlDamJk\nkKMRQgghDJKoA1pmfJtcMSTHSaIWQggRGiRRB7Qk6gRrMmZzSJ8cKIQQIoxIog7Y3ViO32smLSY5\n2KEIIYQQraydeZFSygo8DwwJ3OMOrfXXSqmxwNOAH1intb4xUP9O4KJA+X1a64XdEXx38fv9lDnK\n8btkIpkQQojQ0tkW9RVAo9Z6KnAt8Gig/HHgFq31FCBBKXWmUmoQcAkwFTgHeFQpFVLbftW4anH7\n3PgcMaQmSqIWQggROjrVogZeBV4PfF0OpCil7MAgrfXKQPlHwKlAP+BTrXUzUK6U2gGMAvI6H3b3\nqnRWA+B3RZGWIBPJhBBChI5OJWqttRtwB769FXgNSAWq21Qrw0jSlRjJfN/ykEnUdc31APjdkdKi\nFkIIEVI6TNRKqeuA6/YpvkdrvUgp9TtgAnAukLZPnQNNnQ65KdV1rkCibo6QFrUQQoiQ0mGi1lov\nABbsW66UuhYjQf9Ca+1WSpUDKW2qZAPFgT+qnfKQ0dKitvgiiY+xBzkaIXqeUmoM8AHwmNZ6nlIq\nB3gRsGH0nv1Ka71bKeUGvmnz0lO01t6ej1iI8NGpyWRKqcHAb4GZWmsntHaHb1JKTQ1Umwl8BiwB\nzlZK2ZVSWRiJemOXI+9GLYk6KSoekynkGvxCHFFKqRhgLvBFm+K/As9qrU8C/g3cHiiv1VpPb/NH\nkrQQR1hnJ5Ndh9F6XqhUa2P5dIzx6vlKKTPwndZ6MYBS6jlgGcbyrBu11r4uRd3Nqh21AKTFJAY5\nEiGCwgWcBfyhTdlNgDPwdTnGEJcQIgg6O5nsLuCudi5tBE5sp/5cjE/sIanaWYffayEtPjbYoQjR\n47TWHsDT5kM3WutGgMBSyt8B9wcuRSqlXgNygXe11o8ihDiiZGcyoN7dgN8dQZLs8S1Eq0CSfgVY\norVu6Ra/A7geowftcqXUMcGKT4hw0dmu7z7D5/fh8DbidycQF20LdjhChJIXgS1a6/taCrTWz7R8\nrZT6AjgKWBWE2IQIG2GfqBvqwD0BAAAgAElEQVTdTfjxgzuC+GiZ8S0EgFLqcqBZa31PmzIF3ANc\nDliAKcA7wYlQiPAR9ol6z2YnkqhFeFJKTQQeAQYCbqXUhUA64FRKfRmotlFrfZNSqhD4HvABH2qt\nvw9CyEKElbBP1LWuOsBI1HEx0vUtwo/WejUw/RDr/qHjWkKI7hT2k8n2tKjtxEVJi1oIIURokUQd\nSNRmTyRRESF1qJcQQgghibolUUdbY2VXMiGEECFHEnXgQI5Ym2x2IoQQIvSEfaKuCSTqxIi4IEci\nhBBC7C/sE3Wtsw6/20Z8tOxKJoQQIvSEfaKud9cba6hlaZYQQogQFNaJ2u114/K5jDXUstmJEEKI\nEBTWibrJ4wDA77HJPt9CCCFCUlgnapfXZXzhtcr2oUIIIUJSWCdqZ0ui9lmk61sIIURI6tRe30qp\ndOBlIBKwA7drrb9TSo0Fngb8wDqt9Y2B+ncCFwXK79NaL+yO4LvK5WkGwO+1Ei9d30IIIUJQZ1vU\nvwJe0VrPAO4CHgiUPw7corWeAiQopc5USg0CLgGmAucAjwYOpA+6PV3f0qIWQggRmjrVotZaP9rm\n2xygSCllBwZprVcGyj8CTgX6AZ9qrZuBcqXUDmAUkNf5sLtHS6K2mGxE2EPis4MQQgixl04fc6mU\nysRIxnHAyUAqUN2mShlGkq4EytspD4FEbXR9R1ojghyJEEII0b4OE7VS6jrgun2K79FaLwImKaXO\nAl4CrtqnzoFOuAiZky+cHicA0TbZlUwIIURo6jBRa60XAAvalimlTlJKJWmtq7XWC5VS/8RoNae0\nqZYNFAf+qHbKg67BFUjUdknUQgghQlNnJ5PNBK4EUEodBRRqrd3AJqXU1DZ1PgOWAGcrpexKqSyM\nRL2xa2F3j3qnseFJQmR0kCMRQggh2tfZMeoHgJeVUjOBCODGQPmtwHyllBn4Tmu9GEAp9RywDGN5\n1o1aa1/Xwu4e9S4jUSdGS6IWQggRmjo767sCOLud8o3Aie2UzwXmduZZR1Jjs9H1nRQTE+RIhBBC\niPaF9c5kTW4jUafESqIWQggRmjq9PKsvcHpcYIK0uLhghyJEUCmlxgAfAI9precppXKAVwALUAJc\nobV2KaUuxxji8gHPaq2fD1rQQoSJsG5Ru7zN+H0mUuKigh2KEEGjlIrBGJr6ok3x/cCTWusTgXzg\nmkC9v2BsZDQduE0pldzD4QoRdsI6Ubv9zeCzEB8jG56IsOYCzmLvZZPTgQ8DX7fsMngcsFJrXau1\ndgDfAFN6ME4hwlJYd317/W5M2DCbQ2YPFiF6nNbaA3iUarvdATFa68Bm+K27CWbS/i6DQoh2LFjw\nG+OL2Vu7dJ+wbVH7/X58JjcW5NQsIToQ8rsMCtGXhW2ibnR6wOLFZpJELUQ7GpRSLZM32u4ymNmm\nTsjsMihEXxa2ibqqzoHJ7MNuluMthWjHYuCCwNcXYOwy+B3G/v6JSqlYjPHp5UGKT4iwEbZj1OV1\n9YCcnCWEUmoi8AgwEHArpS4ELgdeUkrdAOwAXtZau5VSs4FFGLsM3qe1rg1S2EKEjfBN1A0NAETJ\nyVkizGmtV2PM8t7Xae3UfQd450jHJITYI2y7vqsbGgGIkUQthBAihIVtoq5xNAEQGyGJWgghROgK\n20RdG0jU8VFycpYQQojQFbaJuuUs6rgI2T5UCCFE6ArbRN1yxGWEzPoWQggRwsIyUbs9XpqajRZ1\npEUStRBCiNDVpeVZSqkMYBNwvtb6S6XUWOBpjDWW67TWNwbq3QlcxJ61lwu7FnbXlNU4weIFIMIi\nG54IIYQIXV1tUf8dKGjz/ePALVrrKUCCUupMpdQg4BJgKnAO8KhSytLF53ZJWXUTmFsStbSohRBC\nhK5OJ2ql1MlAPZAX+N4ODNJarwxUaTkabwbwqda6WWtdjrHL0aguRd1FpVUOsHgA2ZlMCCFEaOtU\nog4k5XuAP7UpTgWq23wfskfjldU4MEnXtxBCiF6gwzFqpdR1wHX7FH8KPKe1rtnnDNu2QvZovNKq\nptYWtXR9CyGECGUdJmqt9QJgQdsypdQ3gEUpNQsYAhwLXAqktKnW9mg81U550JRVO7Bn+/EhiVoI\nIURo61TXt9Z6itZ6stZ6MvAJcJPWei2wSSk1NVBtJsbReEuAs5VSdqVUFkai3tgNsXeK2+Olqs6J\nze4DpOtbCCFEaOvu07NuBeYrpczAd1rrxQBKqeeAZRjLs27UWvu6+bmHrLzGiR+w2nz4zFYs5qBO\nQBdCCCEOqsuJWmt9VZuvNwIntlNnLjC3q8/qDqXVxh7fJotXur2FEEKEvLDbmay0ytiRzG/2SLe3\nEEKIkBd2ibqsxkjUHn+ztKiFEEKEvLBL1KVVTWDy4vK5iLfHBTscIYQQ4qDCLlGX1ziITzDmsiVE\nxAc5GiGEEOLgwipRe7w+qupcJCT7AUnUQgghQl9YJeqqehc+v5/YWGP7UOn6FkIIEeq6ex11SKsI\nTCSLiPaAT1rUQhyMUupa4Io2RccAq4AYoDFQ9j9a69U9HZsQ4SSsEnV5IFGbI1zggERJ1EIckNb6\neeB5AKXUScDFwGjgaq31+mDGJkQ4Cauu74paJwB+q/HfeLskaiEO0V+AB4IdhBDhKKwSdUuL2o2x\nO1mCjFEL0SGl1CSgUGu9O1B0v1JqmVJqvlIqKpixCREOwixRO7FaTDR6G4i2RmGz2IIdkhC9wXXA\nS4Gv5wB3aq2nAT7gd8EKSohwEXZj1CnxkdQ218v4tBCHbjpwM4DW+t9tyj8CfhmMgIQIJ2HTona4\nPDQ43KQk2nB4HCTI+LQQHQocTdugtW5WSpmUUouVUomBy9MBmVQmxBEWNom6ZSJZQqJsdiLEYegH\nlAForf3As8AXSqllQA7wZBBjEyIshE3Xd8tEsug4LzglUQtxKAJrpM9s8/1bwFvBi0iI8BM+LepA\norZHuQGk61sIIUSv0KkWtVLqKow1lVsDRZ9rrR9USo0Fngb8wDqt9Y2B+ncCFwXK79NaL+xq4Ier\nvMbo+jbbXQDER8jSLCGEEKGvK13fb2qt79in7HHgFq31SqXUa0qpM4FNwCXA8UACsFwptUhr7e3C\nsw9byznULZudyKxvIYQQvUG3dX0rpezAIK31ykDRR8CpwAzgU611s9a6HNgBjOqu5x6q0uomYqNs\nNHobANmVTAghRO/QlRb1SUqpzwAbcAdQClS3uV6GMWO0EihvpzyvC88+LB6vj4oaJ4Oz4qlz1QOy\nK5kQQojeocNErZS6DmNnorZeB+7VWn+ilDoe+Cfws33qmA5wywOVHzGVtU58fj8ZSVHsctUSY42W\nXcmEEEL0Ch0maq31AmDBQa7/VymVhtFyTmlzKRsoDvxR7ZT3mN1Vxt7eaUlRrHdWkxGd1pOPF0II\nITqtU2PUSqnfK6UuDXw9BijXWruATUqpqYFqM4HPgCXA2Uope2CXo2xgY9dDP3Sl1cZEssREE80+\nN0mRST35eCGEEKLTOjtG/RrwilLqt4F7XBsovxWYr5QyA99prRcDKKWeA5ZhLM+6UWvt61rYh6e0\n2mhRR0Q3A5AcmXiw6kIIIUTI6FSi1loXYczm3rd8I3BiO+VzgbmdeVZ3KAt0fZsjjaVZSZKohRBC\n9BJhsTNZabWDhFg7DZ46AJKl61sIIUQv0ecTtdvjpbLWSUZSNFVOY/VYUoS0qIUQQvQOfT5Rl9U4\n8QMZSVFUO2sAaVELIYToPfp+og6MT2cmR1PlqsFqshBnjwlyVEIIIcSh6fOJumVpVnqg6zsxMhGz\nqc//2EIIIfqIPp+xWpZmpSTaqG9uIFnGp4UQQvQifT5R765swgTYo43jLWV8WgghRG/S9xN1dRMp\nCZHUuY2lWbKGWgghRG/SpxO1w+WhtqGZjOToNjO+JVELIYToPfp0oi4LTCTLTDJmfIN0fQshhOhd\n+nSibjk1KyM5as9mJ9KiFkII0Yt09lCOXqG0zRrq1ZUlWE0WUqRFLcQhUUpNB94GNgSK8oD/A14B\nLEAJcEXg5DwhxBHSt1vUrUuz7Oxq2E12XBZWc5/+bCJEd/tKaz098Odm4H7gSa31iUA+cE1wwxOi\n7+vTibq0qgmrxYTDXI3X7yU3LifYIQnR200HPgx8/RFwavBCESI89Nnmpd/vZ3dVE+lJ0RQ2FAGQ\nG98/yFEJ0euMUkp9CCQD9wExbbq6y4B+QYtMiDDRZxN1XZMbh8vLiAFR7Kj7CYDceGlRC3EYtmAk\n57eAwcBS9v43wxSMoIQIN51O1EqpO4BfAW7gJq31SqXUWOBpwA+s01rfGKh7J3BRoPw+rfXCLkfe\ngbYTyTbVFxFhsZMRnXakHytEn6G13gW8Gfh2q1JqNzBJKRWltXYA2UBx0AIUIkx0aoxaKTUauAQ4\nBrgBOCdw6XHgFq31FCBBKXWmUmpQoO7UQL1HlVKWLkfegZalWclJVkobyxgQ118O4xDiMCilLg98\nIEcplQlkAC8CFwSqXAB8FqTwhAgbnW1RnwO8pbX2AGuANUopOzBIa70yUKdlokk/4FOtdTNQrpTa\nAYzCWOpxxBRXNAJgjqnDX+VngIxPC3G4PgReU0r9HLADNwI/AP9USt0A7ABeDmJ8QoSFzibqgYBX\nKfUZYANuB8qB6jZ1WiaaVAau7Vt+RBP1rkCidpgrAGTGtxCHSWtdD5zbzqXTejoWIcJZh4laKXUd\ncN0+xRkYXV5nAlOABcDP96lzoIkmPTIBpbiikaS4CHY0aAAGJ+T2xGOFEEKIbtVhotZaL8BIxK2U\nUvcBm7TWfuBrpdRAjFZzSptqLRNNigHVTvkR0+T0UF3vYtSgRLbUFJARnSZbhwohhOiVOju76lPg\nZwBKqRFAodbaDWxSSk0N1JmJ0epeApytlLIrpbIwEvXGroV9cCWVRrd3XEoTLm8zKmnokXycEEII\nccR0aoxaa/1tYEb3fwNFvwv891ZgvlLKDHyntV4MoJR6DliGsTzrRq21r4txH1TL+LQvphwcMFwS\ntRBCiF6q0+uotdb3APfsU7YROLGdunOBuZ191uFqmfFdwy5MmBieNKSnHi2EEEJ0qz65sLi4ohHM\nXkqcu+gfl0WMLTrYIQkhhBCd0icT9a6KRuLTG/D6vTI+LYQQolfrc4na4TJmfMemGDuTDUkYGNyA\nhBBCiC7oc4m6ZSKZNboBgOxYOdxHCCFE79XnTs/aVlIHgNtaix27rJ8WQgjRq/W5FnVBcR2YfNT5\nqugXkyEHcQghhOjV+lwW27qrluh4Fz6/j6yYzGCHI4QQQnRJn0rUdY3NVNQ6ycjyApAVkxHkiIQQ\nQoiu6VOJuqDYGJ+OSXQA0C9WWtRCCCF6tz6VqLcW1wLgj6gHkK5vIYQQvV6fStQFxXWYgHpfFTHW\naOLtccEOSQghhOiSPpOofT4/BSV1ZKRGUOGsJDMmA5OpR46+FkIIIY6YPpOo9c5qXM1esvv78eMn\nS8anhRBC9AF9JlEv/WEXABEZuwHkxCwhhBB9Qp9I1DUNLn7YUkF2eiSb6tcRb49jbOroYIclhBBC\ndFmfSNTL15Xg9fkZNLIOh8fJlKzjsJgtwQ5LCCGE6LJO7fWtlPoTcFrgWzOQqbUerpQ6FXgI8AIL\ntdYPBOo/BkwG/MAtWuuVXY48wO/3s3xtMRE2MyXmnzCbzEzJOra7bi+EEEIEVacStdb6QeBBAKXU\nlUB64NITwM+AXcBXSql3gTRgmNb6eKXUSOAF4PiuBt6iNrAb2Whlp6CxhKNTR8tBHEFSUlLMRRed\nxzPPvMiYMUe1ll933a8ZNGgwf/rTvcELrht8881yvvzyi17/cxwOpdT/ASdi/FvxN+A8YCJQGajy\nd631J0EKT4iw0KXTs5RSVuBGYIZSajBQpbUuDFxbCJyCkajfB9Ba/6SUSlJKxWut67oWuqGozDjO\nMjq5AbwyiSzYsrKyWbx4UWuiLioqpL6+W/6qRQ9TSs0AxgQ+ZKcAPwBLgD9qrT8ObnRChI+uHnM5\nE1iktXYopTKB8jbXyoAhQCqwuk15OZAJdE+iLjfOn/ZF1kAj5Mb3747b9npvLcln5aaybr3npBHp\nXHzy0IPWGT36KFat+g6v14vFYmHx4kVMmjQZl8sJwNq1PzB//pNYrVbS0zP4wx/+jMlk4sEH76W8\nvAyHw8E111zPlCknMmvW9UyadBxr1qyipqaG//3fx8jM3LPs7vnn51NcvIuSkmLmzp3PggXPsG7d\nj/h8XmbOvJiRI0fz2GN/55FHniAvby133nkLCxcuwefzcfXVl/HMMy9w331/xuFw4HQ6ue22Oxk1\nagyXXHI+kydPISkpialTT+Kvf/0L8fEJZGUZ/295PB7uv/9uKisraG5u5tprb2Dy5BO69XcdIpYB\n3we+rgFiAJn8IUQP63AymVLqOqXUt/v8+Vng8rXAiwd46YF2G+nWXUiKyo0WdZ2/HLPJTP/YrO68\nvThMVquVUaPGsGbNKgC+/noZxx8/pfX644//nYcffoQnnniG5ORkli5dTH19HcceO5l5857l/vv/\nxvPPz2+tHxMTw5w5TzN58gksW7Zkv+d5PG6eemoB69evo7R0N08++Rxz5jzDyy+/QFpaGuXlpfj9\nfvLy1jJsmGLbtgLy8zczcuRoKisrOeecXzB37nx++9tZ/OtfLwfu6WHy5BO48spreemlBVxzzfXM\nmfM0Fovxdtm6NZ/a2hqefPI5Hn10HnV1fbPHQGvt1Vo3Br69FliIMf9kllJqiVLqDaVUavAiFCI8\ndNii1lovABbsW66UigH6a623B4qKMVrKLbIDZc37lGcBJZ2Mdz9F5Q3YrFDq3E2/mAzsFnt33bpX\nu/jkoR22fo+UGTNOYfHiRaSkpJCWlkZUVBQAVVWVFBUVctdddwLgdDpJSEgkLi6en37awIcfvofJ\nZKaurrb1XmPHjgcgPT2d2tra/Z41cqSxDC8vby0bNuQxa9b1APj9PioqKhg8eCiFhTvYuHED559/\nEevXr8PlcjF+/ESSk1N4+eUFvP76K7jdbiIjI1vvO2qUcd/t2wsYM2YsAOPHT+Tbb1eQmzuQpqZG\nHnjgbqZNm8Gpp57e3b/CkKKU+jlGoj4dOAao1Fr/qJSaDdwLzApieEL0eV3p+h4LbGr5Rmu9XSkV\nr5QaCBQB5wCXY3R93wfMV0pNAIq11vVdeG4rr89HcUUT6Vleqn1ucuNyuuO2oouOOeY4Hn3076Sk\npDJ9+imt5VarjdTUNObNe3av+p9++jF1dXU8+eQC6urquO66K1qvWSx7elr9fv9+z7LZbK3/Peec\nn3PFFVfvdX38+Ils2LAel8vJhAnH8NRTc3A4HMyadStvvfUaqanp3H33A2zatJF58x7fK1bjmWA2\nG51APp8PgMjISObPf4m8vHV8+ulHfPPNcu66655O/a5CXaD37E/AGVrrWuCLNpc/BJ4OSmBChJGu\nrKPuhzEO3daNwOvAcuBNrfVmrfUKYLVSagXGrPDfdeGZeymtcuDx+ohLMXrnZHw6NNhsNsaNG88n\nn3zAlCnTWsvj4+MB2LatAIB33nmD/Pwt1NTU0K9fFmazma++WoLb7T7sZ44aNYZvvlmOz+fD5XLx\n2GP/B8D48RP47LOFZGfnkJiYSE1NDTU11WRkZFJbW0N2tvH/zFdfLcXj8ex33wEDctm06ScA1qwx\nplpovYnPP/+MsWPHcccdf2T79m2HHW9voJRKAP4OnKO1rgqUvRuYOAowHVgfpPCECBudblFrrd8F\n3t2nbBntLL3SWs/u7HMOpmV82hRdCx7IjZcWdaiYMeNUamqqiY2N3at89uy/8NBD92GzGa3r886b\nSUxMDLNn387Gjes5++zzSE9P58UXnzus5x111FjGj5/IDTdcDfg5//yLABgwYCDbtxdw7rk/ByAu\nLo6UlBQAzjjjbP7613tYunQxF1xwMYsX/4dPPvlwr/teeeW1PPTQfbz99utkZWXj8bjp1y+L+fOf\n5IMP3sNsNnPZZVfQR/0So0fsLaVUS9mLwJtKqSagAbj6AK8VQnQTU3vdicFWXl5/SEG9t6yAj1ds\nJ2fqD9R6Knlk2gOyI5kIK2lpcSF9RNyhvpeFCEXXPLz/BNbDsWDBbwAw52/tsO7B3su9egvRorIG\nsDRT1VxOTly2JGkhhBB9Tq9O1DvL6onNqMSHj6NSRwU7HCGEEKLb9dpEXVnrpKrORWSGscfKhPSj\ngxyREEII0f16baLeUlQDFjdNtt3kxGWTGpUS7JCEEEKIbtdrE/XmolosSaX48TEhTVrTQggh+qZe\nm6i3FNVgSykFYFz6UR3UFkIIIXqnXpmoGxxudlXUY46vIjMmg/Ro2W44FJSUFHPaadOYNet6Zs26\nnhtuuJr//d8H8Xq9PfL8goL81i1E21q6dDEAa9as4s9//n23PW/WrOspKMjvdN0DxSuEEG31ykSd\nv6sWU3QdfpOXoYmDgh2OaGPAgFzmzXuWefOeZf78F/F43Hz++WdBi8ftdvPmm68F7flCCNFVXT3m\nMii2FNZgiasGYGiCJOr2vJf/MT+U5XXrPcenH8XMoecc1mtGjRpDUVEhAO+++xaLF3+GyWTmxBOn\nc/75F3LDDVfz8suvU1FRzsyZZ/PBB4tISkriyisv5bnnXuZvf7u/3eMvBw82zh3/1a+u4u67Z2Oz\n2Rg6dPh+z3/iiUfZujWff/zjYU4++VSamhzcf//d5OdvZsaMU7n66t/sdb/f/nYWDz10H/X19Xi9\nXm699U6GDh3Gq6++xFdfLcVsNjNlyon8+tfXALBkyWLmzHmE2tpaHn74UTIzM3nqqTnk5a3F4/Fy\nwQUXc8YZZ7fGU1ZW2m68jz/+dzZt+gmv18v551/IWWede3h/OUKIPqtXJmpHsxdbQg0AQxIHBjcY\ncUAej4fly7/iF7+4gOLiXXz55Rc89dTzANx447XMmHEqMTEx1NfXs27dWsaOHc+GDXmMHn0UiYmJ\nNDY2cOyxkznzzHPYtauIu++ezZQpJwIwePAQfvGLC3nqqTmccsrpXHzxpbz66kvk52/eK4bLLruC\njRvXc8cds1mzZhXbtxfw2mvv4vP5uPji87j66t/sdb+XXlrAccedwLnn/oJt2wqYM+cfPP74U7zx\nxqu8//5nWCwW3n9/z865SUlJzJnzNM88M49ly5YwfPgICgq28vTTL+BwOLjyykuYNm16a/133nlj\nv3jr6mpZseJr3nrrAzweDwsXfnSE/2aE6Lu6uptYKOqVifqSk4ey/r/12C2JJEcmBTuckDRz6DmH\n3frtDjt37mgdd926NZ/LL/8106ZN54sv/kNRUSE333wDAE1NjezeXczYsePZuHE9eXlrueiiS9mw\nIQ+/38e4cRMOevzlyJFjANi+fRszZpwKwPjxx/DttysOGp9SI1qPs2y7fW7L/fLy1lFTU82iRQsB\ncLmcAEyffgq33noTp512Bqeffkbr644+ehwAaWlp1NbWsmnTRsaNmwBAVFQUAwcOprCwsLV+e/HG\nxyeQk5PL7Nm3M2PGqXu1wIUQolcm6qrmSho9TYxM2b+rUwRXyxg1wJ///HtycnIB49jI44+fwu9/\n/6e96jc3N7N+/TqKinZy8823sXDhh3i9HqZMmcbnn392wOMvbTbjf12/34/JZA587eswvrbHZrbV\ncj+bzcptt93JmDF7L/m7444/smPHdpYs+Zybb76BZ599eb/7GbGYaLt9vsfjbj0m82DxPvLIE62n\ncn322Sc89tiTHf4sQojw0Csnk22tMY4VlIlkoe2mm27hmWfm4nQ6UWoka9asxul04vf7efzxf+By\nORkz5mjWrfsRu92O2WzGZDKhtWbUqDGHdPylcQzlRsCY1b0vk8l8WLPOR40aw7JlXwLGcZxvvPEq\nDQ0NvPjic+TmDuTqq39DXFwCTU2N7b5+xIjR/PCDcRxmU1MTu3YV0b//gIPGW1JSzNtvv4FSI5g1\n61Zqa2v3v7EQImz1yhb1trqdAAyRiWQhLSsrm+nTT+Hll5/nhht+x8UXX8rvfvcbzGYz06ZNJyLC\n6IJ2Op1MnHgs8P/bu/8gK6s6juNvxdVBmSVyHUhqBBnnQ8ako5I6aAE6qEGZCP7InyOYIOav0WQm\nFY3BmjQxShsVnDJH05pIS/BnGir+IJpQCz9IZVmQLimEVsqC/XHO6vWyK+zlcp9n8fua2dl9zt7d\n+7lnnz3nPuc5z3lg4MBBLF36B5qamhg+fOQmb385fvyJXHbZVBYseIRBg/baKENLSwttbeu49NJL\nGDt2/CYzjxt3PDNmXMHZZ09kw4YNnH/+RfTq1YvVq1/nzDNPpWfPnRky5NM0N/fu8Of32WdfpMFM\nmXImbW1tTJp0Dj179vzAvC0tu/H880t4+OEHaGpqYvToL25eBYcQPhS65W0uf/vK71n2+nJO0Fi2\n365bDgqEUBdxm8v3q+dEolumjqzb7wqNU6bJZPW6zWVNR9SSdgduAXYCegAX2F4s6XDgKmA9MM/2\n9Pz4mcBBwDvAebYX1fK87Q7ouy8H9N13S35FCCF8oHo1+NHhhy1V6+HohcBc2yOAqcCMXD4LOBYY\nBoyStLekzwF72T4YmJAfE0IIIYTNUOs56lVA++2q+gCrJO0JvGb7ZQBJ84DDgN2AXwDYXiqpj6Rm\n2//esughhBC2FWUasi6bWjvqmcAzkk4FmoFDgH5Aa8VjXgUGAS3A4ory1vzY6KhDCKEAMazfvWyy\no5Y0EZhYVTwfuMv2DEljgGvyR6XOToyXevJLCCGEUCab7KhtzwZmV5ZJmg9cmjcfBG4AVpCOlNv1\nz2VvV5XvDqysPXIIIYQyiOHqxqh16Hs5cCBpSHso8KLtlyQ1SxoA/B0YA5xEGvq+ErhR0n7ACttr\ntzh5CKEQ9b6KY1sXw8xhS9XaUV8FzJF0XN4+N3+eDNyRv77T9jJgmaTFkhYCG4ApNacNIRSq8ioO\nSZ8kXaZ5cMGxQtim1Tlx49sAAAbJSURBVNRR214JfL6D8gV08E9re2otzxNCKJ3DiKs4QmioUq5M\nFkIoJ0k3AffavjtvPwZMyKNnIYStINbfDCFsibiKI4StLDrqEEJXVF/dEVdxhLCVRUcdQuiKB4Bx\nAHEVRwiNEeeoQwhdIulbwGfJV3HYXlJwpBC2adFRhxBCCCUWQ98hhBBCiUVHHUIIIZRYrSuTFaos\nSxhK+jZwKKkevwksAn4M9CDNhD3F9lsF5OoJPA9MBx4uSaaTgK8BbcDlwLNF5pLUC7iVdJvWnUjL\n3P4T+AFpv3rW9uQG5hkC3A3MtP19SZ+gg/rJ9Xg+6fzwTbbnNCpjLfJKZj8FzrD9q1y2Dx3Us6SL\ngfG5/Erb8yT1Bm4HegNvAF+2/Vqdsu1OWlltJ1I9X2B7saTDSasvrgfm2Z6eH9+QdkfSRcDJwDrg\nbNuLulJnWyNTfq6+wAvAMbYfLTKTpB2AOaQ7NO4AXGT78TLUU0XGuu0v3e6IunIJQ2ACMKugHCOA\nITnHkcB1wDeA620fSloP/YwispFumNLemBWeSdKuwDTS7VDHAEeXINfpgG2PIM1i/i7pb3ie7WFA\nb0lHNSKIpF2A75HeVLXbqH7y4y4HDgeGAxdI+mgjMtZC0iDgQuCJqm9tVM+SBgIn8N4+cq2kHqQ3\nJY/aPgT4OXBJHSNeCMzN+8BUYEYunwUcCwwDRknau1HtjqRPkerhAOAsUl1A1+psa7ka+HPFdpGZ\nTgHezPvFBODaEmR6V733l27XUVO1hCHQR1JzATkWkN6hAawGdiE1nvfksl+SGtSGkjQY2Bu4NxcV\nnik/50O219peafsrJci1Ctg1f92H9MZmYMW73kZmeou0JO+KirLhbFw/BwKLbK+x/V9SBzisQRlr\nsRIYC6xpL5C0Ix3X8whgvu23bbcCfyXtx4cBc6seWy/V+8AqSXsCr9l+2fYGYF7O0Kh2ZwzpFsJt\ntn9ne1oNdVZ3kkYCa4Hn8nbRmW4jvdECaAV2LUGmSnXdX7rj0Hc/0l272rXmsoauNWx7PfBm3pxA\n+oc+omL49lXgY43MlH0HOAc4LW/vUoJMA4CdJd1DahCvKDqX7Z9IOl3S8pzpC8D1FQ9pWCbbbUCb\npMrijuqnH2l/b3jGWtj+D0DV62oBXq/Ybn8N/6Lj11b5muv9emcCz0g6FWgmHXF1VMeDcu5GtDsD\ngPWS7gOaSJ1RK12rs+fqGSh3gNNII2HX5eKu/h3rmsn2OtKpAUijLrcXnalKXfup7thRVyt0CUNJ\nR5M66lHAixXfaniu3OA8afsvVY1ju6LqajvSkcsxwB7AI1VZiqirk4G/2T4yn9eaS8WRXxGZPkBn\nWUqTUdJEYGJV8TTb92/iR7vy2mp+vZ3km086ep0haQxwTf6oNV89MvUF7gOOIo2WzCZ1kEVmmg/c\nbHt1J+1KEZmm2b5f0hRgP9Ib7d0alakGW/Sc3bGjLs0ShpKOAL4OHGl7jaQ3JPXMw5L9ef9QZiOM\nBvbMjc7HSUOqRWcCeAVYmI8c/yRpLekIsshcw4D7AWwvyRPwmiq+X1Rdtevo71a97/cHnioiXDXb\ns0mdyqa08t5wM7z/tamT8n6kN1E1/006yidpPmk+B8CDwA10XMcrgLepc7vTSaYrgRdsvwM8LmkA\nXa+zemd6Augh6RzS6MJngBOLzJRzTSB10F+yvU5Sw+ppM9S1n+qO56hLsYRhno16NTCmYhbqQ6RJ\nKOTP9zUyk+3jbQ+1fRBpx55edKbsAWCkpO3zxLJeJci1nHTOF0l7kM6/LZV0SP7+2AIyVeqofp4G\nhkr6SJ61Pgx4rKB8NclDli90UM+/BkZL2jHPxu4P/JG077TPBan3fvLuPgAMBV60/RLQLGlAnlk8\nJmdoVLszHzgiP89g4OUa6qyubA+zfVBuV+4lzURfUmSmPJdgEjDW9v9yzkLrqUpd95dud0Rte6Gk\nxZIWkpcwLCjK8aRzIndVDAedBsyWdBZpwsKPCspWaRpwa5GZbP9D0s947+jvq6RL2YrMdSNwi6Tf\nkP4PJpEuz7pR0vbA07YfakQQSfuT5hYMANZJGgecBPywsn7yUcNU0khA+2Umazr5tYWTNBq4GBgM\n7C/pXNujSOcUN6pnSTeTJmm+A0y2vUHSLOA2pdtpriZdtlQvVwFzJB2Xt8/NnycDd+Sv78y38FzW\niHbH9lN5pvKTuaj9eTa7zrZGrk4UmWki6eh5XkX726V9aytkele9+6lYQjSEEEIose449B1CCCF8\naERHHUIIIZRYdNQhhBBCiUVHHUIIIZRYdNQhhBBCiUVHHUIIIZRYdNQhhBBCif0fIxZ3CSvvDBgA\nAAAASUVORK5CYII=\n", 531 | "text/plain": [ 532 | "" 533 | ] 534 | }, 535 | "metadata": { 536 | "tags": [] 537 | } 538 | } 539 | ] 540 | }, 541 | { 542 | "metadata": { 543 | "id": "OIJfD4Tbm9y-", 544 | "colab_type": "code", 545 | "colab": {} 546 | }, 547 | "cell_type": "code", 548 | "source": [ 549 | "" 550 | ], 551 | "execution_count": 0, 552 | "outputs": [] 553 | } 554 | ] 555 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Reinforcement Learning on google colab 2 | 3 | # Colab notebook links 4 | 5 | 1. [gym_intro](https://colab.research.google.com/drive/1AsFzJDEI9OtlmGHyuqr21iMLKy4OgAMg) 6 | 2. [crossentropy_method](https://colab.research.google.com/drive/1Nu7NtjIQG2MRB2TnQ63RW4E9n_2MNblR) 7 | 3. [qlearning](https://colab.research.google.com/drive/1nDO-XfgkAgABCe2jYNAFwmi4plr2xJoc) 8 | 4. [Actor-Critic](https://colab.research.google.com/drive/1-TiI1dNIQDukkQC1PXy7mhDT883j9ua9) 9 | 10 | # Guide to follow 11 | Google Colaboratory provides that 12GB GPU support with continuous 12 hr runtime. For RL it requires to render the environment visuals. Here is sort of a tutorial to get over that issue & continue free coding. 12 | 13 | Motive of this blog will be to use gym & gym[atari] on colab. For Deep Learning & other setup you may want to refer this article, it will also give you the basic understanding. 14 | 15 | Most of the requirements of python packages are already fulfilled on colab. To run gym, 1st you have to install prerequisites like xvbf,opengl & other python-dev packages: 16 | 17 | ``` 18 | !apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg- dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig 19 | ``` 20 | 21 | ![](https://github.com/ajit2704/Practical-Reinforcement-Learning-With-Colab/blob/master/1_yqDjbLhneoEAyL8Lbwj43w.png) 22 | 23 | Now for rendering environment I prefer to use pyvirtualdisplay, so to fulfill that: 24 | 25 | ``` 26 | !pip install pyvirtualdisplay 27 | !pip install piglet 28 | ``` 29 | 30 | ![](https://github.com/ajit2704/Practical-Reinforcement-Learning-With-Colab/blob/master/1_s4eFRgsGnHWmJWzgAe52Lg.png) 31 | 32 | To activate virtual display we need to run a script once for training an agent, as follows: 33 | 34 | ```python 35 | from pyvirtualdisplay import Display 36 | display = Display(visible=0, size=(1400, 900)) 37 | display.start() 38 | ``` 39 | 40 | Moving on to gym requirements, gym is already installed but not with atari game environments, to get that: 41 | 42 | ``` 43 | !pip install gym 44 | !pip install “gym[atari]" 45 | ``` 46 | 47 | # References 48 | 1. For gym, check the https://gym.openai.com/docs/ 49 | 2. For colab, checkout https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d 50 | --------------------------------------------------------------------------------