├── Dynamic-Programming ├── Dynamic_Programming.ipynb ├── algorithms.pdf ├── check_test.py ├── frozenlake.py └── plot_utils.py └── README.md /Dynamic-Programming/Dynamic_Programming.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Dynamic Programming\n", 8 | "\n", 9 | "In this notebook, you will write your own implementations of many classical dynamic programming algorithms. \n", 10 | "\n", 11 | "While we have provided some starter code, you are welcome to erase these hints and write your code from scratch.\n", 12 | "\n", 13 | "---\n", 14 | "\n", 15 | "### Part 0: Explore FrozenLakeEnv\n", 16 | "\n", 17 | "We begin by importing the necessary packages." 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "import numpy as np\n", 27 | "import copy\n", 28 | "\n", 29 | "import check_test\n", 30 | "from frozenlake import FrozenLakeEnv\n", 31 | "from plot_utils import plot_values" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "Use the code cell below to create an instance of the [FrozenLake](https://github.com/openai/gym/blob/master/gym/envs/toy_text/frozen_lake.py) environment." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 2, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "env = FrozenLakeEnv()" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "The agent moves through a $4 \\times 4$ gridworld, with states numbered as follows:\n", 55 | "```\n", 56 | "[[ 0 1 2 3]\n", 57 | " [ 4 5 6 7]\n", 58 | " [ 8 9 10 11]\n", 59 | " [12 13 14 15]]\n", 60 | "```\n", 61 | "and the agent has 4 potential actions:\n", 62 | "```\n", 63 | "LEFT = 0\n", 64 | "DOWN = 1\n", 65 | "RIGHT = 2\n", 66 | "UP = 3\n", 67 | "```\n", 68 | "\n", 69 | "Thus, $\\mathcal{S}^+ = \\{0, 1, \\ldots, 15\\}$, and $\\mathcal{A} = \\{0, 1, 2, 3\\}$. Verify this by running the code cell below." 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 3, 75 | "metadata": {}, 76 | "outputs": [ 77 | { 78 | "name": "stdout", 79 | "output_type": "stream", 80 | "text": [ 81 | "Discrete(16)\n", 82 | "Discrete(4)\n", 83 | "16\n", 84 | "4\n" 85 | ] 86 | } 87 | ], 88 | "source": [ 89 | "# print the state space and action space\n", 90 | "print(env.observation_space)\n", 91 | "print(env.action_space)\n", 92 | "\n", 93 | "# print the total number of states and actions\n", 94 | "print(env.nS)\n", 95 | "print(env.nA)" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "Dynamic programming assumes that the agent has full knowledge of the MDP. We have already amended the `frozenlake.py` file to make the one-step dynamics accessible to the agent. \n", 103 | "\n", 104 | "Execute the code cell below to return the one-step dynamics corresponding to a particular state and action. In particular, `env.P[1][0]` returns the the probability of each possible reward and next state, if the agent is in state 1 of the gridworld and decides to go left." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 4, 110 | "metadata": {}, 111 | "outputs": [ 112 | { 113 | "data": { 114 | "text/plain": [ 115 | "[(0.3333333333333333, 1, 0.0, False),\n", 116 | " (0.3333333333333333, 0, 0.0, False),\n", 117 | " (0.3333333333333333, 5, 0.0, True)]" 118 | ] 119 | }, 120 | "execution_count": 4, 121 | "metadata": {}, 122 | "output_type": "execute_result" 123 | } 124 | ], 125 | "source": [ 126 | "env.P[1][0]" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "Each entry takes the form \n", 134 | "```\n", 135 | "prob, next_state, reward, done\n", 136 | "```\n", 137 | "where: \n", 138 | "- `prob` details the conditional probability of the corresponding (`next_state`, `reward`) pair, and\n", 139 | "- `done` is `True` if the `next_state` is a terminal state, and otherwise `False`.\n", 140 | "\n", 141 | "Thus, we can interpret `env.P[1][0]` as follows:\n", 142 | "$$\n", 143 | "\\mathbb{P}(S_{t+1}=s',R_{t+1}=r|S_t=1,A_t=0) = \\begin{cases}\n", 144 | " \\frac{1}{3} \\text{ if } s'=1, r=0\\\\\n", 145 | " \\frac{1}{3} \\text{ if } s'=0, r=0\\\\\n", 146 | " \\frac{1}{3} \\text{ if } s'=5, r=0\\\\\n", 147 | " 0 \\text{ else}\n", 148 | " \\end{cases}\n", 149 | "$$\n", 150 | "\n", 151 | "To understand the value of `env.P[1][0]`, note that when you create a FrozenLake environment, it takes as an (optional) argument `is_slippery`, which defaults to `True`. \n", 152 | "\n", 153 | "To see this, change the first line in the notebook from `env = FrozenLakeEnv()` to `env = FrozenLakeEnv(is_slippery=False)`. Then, when you check `env.P[1][0]`, it should look like what you expect (i.e., `env.P[1][0] = [(1.0, 0, 0.0, False)]`).\n", 154 | "\n", 155 | "The default value for the `is_slippery` argument is `True`, and so `env = FrozenLakeEnv()` is equivalent to `env = FrozenLakeEnv(is_slippery=True)`. In the event that `is_slippery=True`, you see that this can result in the agent moving in a direction that it did not intend (where the idea is that the ground is *slippery*, and so the agent can slide to a location other than the one it wanted).\n", 156 | "\n", 157 | "Feel free to change the code cell above to explore how the environment behaves in response to other (state, action) pairs. \n", 158 | "\n", 159 | "Before proceeding to the next part, make sure that you set `is_slippery=True`, so that your implementations below will work with the slippery environment!" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "### Part 1: Iterative Policy Evaluation\n", 167 | "\n", 168 | "In this section, you will write your own implementation of iterative policy evaluation.\n", 169 | "\n", 170 | "Your algorithm should accept four arguments as **input**:\n", 171 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 172 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 173 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 174 | "- `theta`: This is a very small positive number that is used to decide if the estimate has sufficiently converged to the true value function (default value: `1e-8`).\n", 175 | "\n", 176 | "The algorithm returns as **output**:\n", 177 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s` under the input policy.\n", 178 | "\n", 179 | "Please complete the function in the code cell below." 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 5, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "def policy_evaluation(env, policy, gamma=1, theta=1e-8):\n", 189 | " V = np.zeros(env.nS)\n", 190 | " while True:\n", 191 | " delta = 0\n", 192 | " for s in range(env.nS):\n", 193 | " Vs = 0\n", 194 | " for a, action_prob in enumerate(policy[s]):\n", 195 | " for prob, next_state, reward, done in env.P[s][a]:\n", 196 | " Vs += action_prob * prob * (reward + gamma * V[next_state])\n", 197 | " delta = max(delta, np.abs(V[s]-Vs))\n", 198 | " V[s] = Vs\n", 199 | " if delta < theta:\n", 200 | " break\n", 201 | " return V" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "We will evaluate the equiprobable random policy $\\pi$, where $\\pi(a|s) = \\frac{1}{|\\mathcal{A}(s)|}$ for all $s\\in\\mathcal{S}$ and $a\\in\\mathcal{A}(s)$. \n", 209 | "\n", 210 | "Use the code cell below to specify this policy in the variable `random_policy`." 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": 6, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "random_policy = np.ones([env.nS, env.nA]) / env.nA" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "Run the next code cell to evaluate the equiprobable random policy and visualize the output. The state-value function has been reshaped to match the shape of the gridworld." 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 7, 232 | "metadata": {}, 233 | "outputs": [ 234 | { 235 | "data": { 236 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAFoCAYAAAD5IVjuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Hd8FNX+xvHP2ZZKCBBI6ARBARGRIF2xgqh47dhQbFeuV0WxYteLevVnRxFFL4iVq1iuigWlSBcQBEWxYUEJkNDSN7t7fn9MSFiyCQmSAfR5+9oXZubMOWe+M/vsZGfAWGsREZG65dnTExAR+StQ2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhK3sVY8xPxphj9vQ86oIx5lxjzEd7eh6yZyhs93HGmH7GmHnGmC3GmI3GmLnGmEPL1g0zxsypRV9tjDHWGOPbxbmMMsZ8GmN5mjEmaIzpvCv97g7GmIllc8jf7jWkDserVEtr7UvW2gF1Nabs3RS2+zBjTArwLjAGaAg0B+4CSvbQlF4A+hhjMndYfhawwlr75R6Y0/YesNYmb/eavIfnI38hCtt92/4A1tpXrLVha22RtfYja+1yY0xHYBzQu+wqbjOAMeYEY8xSY8xWY8yvxpg7t+tv21Xp5rJtepdtc5Ex5mtjzCZjzIfGmNaxJmOtXQNMB4busOp84PmyvvYzxkw3xuQaY3KMMS8ZY1Jj9Vd2NTp6u5+PMMas2e7nZsaYKcaYDcaY1caYq2pcuehxrDGmXaxxt41pjLnWGLPeGLPWGHPhdm0TjDEPGWN+LvvtYo4xJoEYtdzxNw1jTB9jzKKy7RYZY/pst26mMeZfZb+p5BljPjLGpO3K/sneQWG7b/sWCBtjnjfGDDLGNNi2wlr7NTAcmF92Fbct0Apwwi8VOAH4hzHm5LJ1h5f9mVq2zfyydTcDpwKNgdnAK9XM6Xm2C1tjzAFA1+22McB9QDOgI9ASuLO2O26M8QDvAF/gXNEfDVxtjBlY275qIAOoXzbOxcCT29X6QSAL6IPz28UNQIQYtdxh/g2B94DHgUbAw8B7xphG2zU7B7gQaAIEgOt2/66JWxS2+zBr7VagH2CB8cAGY8z/jDHp1Wwz01q7wlobsdYuxwnB/tUMcxlwn7X2a2ttCLgX6FrV1S3wJpC+3VXa+cD71toNZeN/b62dZq0tKVv28E7Gr8qhQGNr7d3W2qC19kecGpxVzTbXGWM2l71yajFWKXC3tbbUWjsVyAcOKAv8i4AR1trfyn67mGetrcnXOCcA31lrX7DWhqy1rwDfAIO3azPBWvuttbYI+C/Oh5bsoxS2+7iyEBxmrW0BdMa5Yny0qvbGmJ7GmBllv3pvwbn6re7X09bAY9tCCtiIc3Xa3Bhz83Y3m8aVzacQeA043xhjgHMp+wqhbPwmxphXjTG/GWO2Ai/uZPzq5tVsu/DcjHMFXuUHDfCgtTa17FWbMXPLPmi2KQSSceYdD/xQ28njHKefd1j2M87V8zbZMcaUfZTC9k/EWvsNMBEndMG54t3Ry8D/gJbW2vo43+uaatr/Cly2XUilWmsTyq7g7t3uZtPw7bZ5HjgTOBaoh3MTb5v7ysbpYq1NAc7bbvwdFQCJ2/2cscO8Vu8wr3rW2uOr6Ks6hdWMU50coBjYL8a6nf1zer/jfGBsrxXwWw3Hln2MwnYfZozpUHbjpkXZzy2Bs4EFZU3WAS2MMYHtNqsHbLTWFhtjeuB8L7jNBpzvG9tut2wcMMoYc2DZGPWNMWfsZGqzgc3AM8Cr1trgDuPn49w4ag5cX00/y4DjjTENjTEZwNXbrfsM2GqMubHsJpXXGNPZlD32VkvLgHPK+jiOGn6tYa2NAP8BHi67WectuxEWR+xabm8qsL8x5hxjjM84j6F1IvqDSf5EFLb7tjygJ7DQGFOAE7JfAteWrZ8OfAVkb/cd5eXA3caYPOB2nO8CgfKvAO4B5pb9at7LWvsmcD/watmv/V8Cg6qblHX+keRJOFduk3ZYfRfQDdiCc4PojWq6egHnBthPwEdA+aNa1towzvebXYHVOFeZz+LcyKqtEWV9bcb52uOtWmx7HbACWITzFcv9gCdWLbffyFqbC5yIc6xycW6snWitrc13ybIPMfrHw0VE6p6ubEVEXKCwFRFxgcJWRMQFClsRERcobEVEXFCrf0rPpKVZ07pNHU1FpOaMHqLZLWxVf51Easz+/BM2J2enlaxd2LZuQ9zcxbs+KwEgot8n/rBAcOdtZOeCgZ23keqV9u5eo3Z624uIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuMC1sA09PZaSDpkUp8ZT0ieLyJzZ1baPzJ5FSZ8sp33HtoTGj4teP+dTgqefRHHb5hQnGEIvTKzUR+ldt1FycAeKGyVR3LQBwUFHE5k/L7qfH38geOYpFLdsTHGTFILnnoldty7mnGxxMSU9DqY4wRBZsrh2BdgNwuPGEtw/k2BKPKW9alDDT2dR2iuLYEo8wQPaEn5mhxrO/pTSU08imNmcYJwhPGli5T7eeoPSEwYSbN6YYJwhMmtm7LEWfUbpoGMJNkwm2Kgepf37YHNyALCRiDNOu1bOXFo3JTTsPOxvv+1SHf6o0vFjKeicSX5aPIWHZRGeW30dw3NmUXhYFvlp8RQc1JbS56LrGHzwPgr7H0p+sxTy2zSm6IzBhFd+GdXGWkvJvXdS0L4Z+Y0TKBx0BOGvv4oeZ9nnFJ10LPktUslv1YjiK/+Ozc+PapNfz1R67TgfN+yJc9FaS+hfdxJs04xg/QRKjz2CyMqvKrUD571a2v1g55zd4b0aWbyI0uOOIZjegGCTVEoHHk1k0We1K8AucCVsw69NJnTdCLw33ExgwVI8PfsQPHkQ9pdfYraP/LSa4MnH4+nZh8CCpXivH0Vo5JWE35xS3sbm52M6dcb/4GOQkBCzH8/+B+B79EkCi1cQ+GQOpk0mwb8dVx6mtqCA0hMHgLUEpn5CYPpcCAYJnjYYG4lU6i9003WY5i3+eEF2Qfi1yYSvHYH3xpvxL1yK6dWH0ElV19CuXk3ob8djevXBv3Ap3htGEb7mSiLb1ZD8fMyBnfE9VHUNbUEBnt598D7wcJVzi3y2kNAJA/D0PwLf7AX45y/Be8114PeXt/EccRS+l/6Lf8UqfK9Owa7+kdCZp+xSLf6I0imTKblhBIFrbyZxzlK8PftQdNogIr9WfS4WnXY83p59SJyzlMDIUZRcdyWhtyvqGJ4zE/8ll5Pw8TwS3psOPh/Fg4/BbtxYMe4jD1A65iHiHhxDwqxFeBo3ofikY7F5ec44a3+n6KRjMG3akjh9IQlvfkDk668oHj6s0pzixown8fu15S/fORfs1hrtzJ46FyMPPUDk0YfwPTIG37xFmMZNCB1fUcOoOd54HcR4r9r8fEKDj8M0a4Z/5jz8s+ZjmjYldOLAmP3sVtbaGr9MtywbX2Rr/TLde1jvhZdEL9uvnfVed1PM9t6RN1izX7voZcMutqZHr5jtSUqyvmcm7HQeceu2WMD6//eBjS+y1v/OhxZjbNzvGyvaZG+2GGP9702L2tb/37es6djJBpautIANzFm0S7WIL7I2UFL7lzm0h/VcdEnUMvZrZz3X3xSzvefaGyz7tYteduHF1vTsFbM9SUnWO35CleP7f9tgAev7aEblufXqbT033lyr/fG9/rZzLLYU7VI9kvN27eXp3sP6LrgkapnZr531j7wpZnv/1c65uP0y3/kXW8+hvaocI2ltnsXjsfGT/+f8vDViTXqGDdw+uqLN+kJLcrKNe2ycTc6zNu6xpy0NG9mkzaHyNgkLllvAJi77rnwZYONfeG2X93/H175yLvqLI5aMDOu9a3TFss1ODb1PjIs+t15z3qv+Zc571TdvUcW6eYuc8+6bHyv6+ebHSu1qVY9uWbYm+VnnV7Y2GMQuXYLn6AFRyz3HDCCyYF7MbSIL5+M5Zsf2A7GfL8aWlu7yPMLPPQMpKXi6dHUWlpSAMRAfX9EwPh48HiLz5lRsu2YNpVf9A/+El6r81K1LNhjEfr4kRk0GYKuooY1Vw2MHYpfseg1jjrN+PXbBfExGU0qP7EewZTqlRx1GZPonVW+zcSORV1/C9OiJ2b72dcwGg0SWLsG3w7noO2oA4YWx6xj+bD6+o3Zof8xAIkurrqPNz4NIBJPawPn5p9XYddl4t+vHJCTg7XM44bLjZ4MlGL8f4/VWtIl3zrXw/Dlsr+SGEeS3TqOw/6GUPjcu5m9hdWWPnYurV0N2NuaY6BqafodHjWvXrCF01T/wPh/7vWr2PwAaNyY88TlsSQm2pITwf8ZDq1aYTgfWbC67qO6/RsjJgXAYk54etdg0SYd12bG3WZftrN++fXo6hEJOf7UQnvouxWnJlKTGExrzCIF3p5XPxdOjFyQnExp1PbagAFtQQOim6yAchuy1ANhwmOCF5+IbcS2eg7vWauzdpqyGxKiJzY5dQ5udXanmNNm1GlbHrv4RgPC/7sBzwUX43vkA0/cwQicOJLL8i6i2oZtvJNggidKmjbC//oLvzXd32zxqNNfcsnOxceVz0VZxLtpY52Jjp442N3YdgzeMwNOlK56evcv72DZOpXHXO+u8/Y/C5uYQfOjfTqBt2kTJHTc525ediwCBW+8mfuJkEt75GN9pZ1Fy87WUPnhvTUvwx+2hc7HKGm43rg2HCQ07F28171VTrx7+aTOJvDaZ0tRESlMTibw2Gf970zB1fCHl4tMIJvpHa52ryiqbx2gfa/lOePofSWDhMgIz5uEdcBzB887ErnVOXtO4Mf6XXiPy0fuUNK5HSXp97JbNmEO6QdkVRviBezF+P94RI2s1bp2IVRMXalitsqsqzyWX4R12EZ6uh+D7172YQ3sQ2eEmiHfk9fgXLsX33kfg9To3ybbNyU1/tI5UXceSm0YSnj+H+BenRF2l7mxcb8cDiXv6eUrHPkpBk0QK2mXgaZ3phMt2/QRuvA1vn354u3QlcNW1BG66g+Bj/1fd3taNPXUuVjNu5P57wefHc3XV71VbVETo7xfh6dUb3+wF+GbOxXQ9hNLT/4YtKKjdXGrJV6e9A6Slgddb6crBblhf6dOxXHpG5fbr14PPB40a1Wp4k5SE2a8d7NcOT89eRDq3JzzxWXyjbgPAe8wAvCt/cO6c+3yY1FSK22RgWmcCEJ7xCXbubErq+aP6Dfbvhef0IQQmvlSr+eySshru+JuAXb++8hVDGZORUflKY8Ou1bA6JqOp82fHTtHLO3TE7nDTyaSlQVoaZv/9MR06UrpfS+zcOZh+h+22+VQ710Zl5+L6yufijldM5dvEOhfL6mgaRtex5KZrCL3+KgnvzcCT2TaqDyi7OmvRMnrc7a6y/Weeg//Mc4isX4dJTAJjKH3iYTxtMqvcJ++hPWHrViLr1+Gp6v20O+2hc3H7GpqW29Vwu3EjMz7BzplNaVL0ezV0WC88ZwzB9/xLRF59GfvjD07Iln2ImUkvU5regMjbb+I957wazWdX1PmVrQkEMIdkEZk+LWp55JNpeHr1iT2pnr2JTP84uv30aZhu3TF+f8xtaiwSwZaUVJ5nWhomNZXwzOmwfj2eE08CwP/MBAKffeFcHS9chv+tqc7yCS/hH33/H5tLDZlAANMti8jHlWtoqqihiVXDj6dhsnZDDbfXpg00a4b9dlXUYvvdt5hWravebtv3jDGORV0xgQCeQ7II7XAuhqZPw9szdh29PXoTmvFxpfaeQ6LrWHLDCEL/fZmEd6fjOaBD9LhtMjHpGYS3G9cWFxOePxtvjOPnaZKOSU4mNGUyxMfjPfLYKvcpvHwZxMdj6qdW2WZ32mPnYmYmZGRgP4muoZ07u3xc3zMT8C3+At+iZc7rbee96n3+Jbz3lL1XCwudK2HPdtHn8TjL6vq7bzeeRvBPetXi91vf2PE2sHSl9V5+lSUpycZ985ONL7LWc85Q6zlnaMXd+q9/tCQmWu8/R9jA0pXWN3a8xe+3/pdfr3hqYEOeDSxYagMLlloSEqzvtrtsYMFSG7fq5/InD7w33mIDsxbYuFU/28DcxdZ7/oWWQMAGPvuivB/f0/+xgRnzbOCr763/Py9YGja03qtGVv0kwTer98jTCN4XnRp6nxpv/ctWWs8/nRr6v/3Jubt77lDrOXdo9B3WxETruWKE9S9bab1POTX0vfp6RZvcPOv7bKn1febU0Hv7Xdb32VLr/+7nijZrc502H82wgPU+Nd5p8/Pairn93yOWlBTre/m/1v/Vd9Z79z0Wn8/6Fi1z7gDPmme9jz5hfYuWWf+3P1nfB59Y07uPpXUb159GiJvo1DFuzHibuGil9f/DqWPiVz85TxqcPdT6zh5a3j5xhVNH/+UjbOKilTZujFPH+Bdfr3hi4dLLLfXq2fh3P7GJ368tfyWtzau463/Xv502L06xCQtXWN9pQ6zJaGqTft9a0ebBMTZh9hKb+PkqG3joCUtCgg088Fj5+vjJ/7Nxjz9jExausIlffO/MJSXF+v9xlatPI+ypc9F7j1ND3+Qp1vf5Cus5Y4ilaVPrz9kac57+VasrPWXg/+JrS1yc9Vw63PqXrbS+pV9azznnOXX84dc6fRrBlbCNL7LW9+iTllatLYGANYd0s4FpsyoeAzusvzWH9Y8OpI9mWtP1EKd96zbW9/hT0QH+ofPm3/HlOe8CJ2xzC6xn8MmWjKaWQMCS0dR6TjzJBmYtiH6k7NobLenpFr/fmnbtre/fD9m4wsheF7aBEmu9jz1paV1RQ9/HsyoO+OH9rTm8f/QjMNMqakjrNtY75qno9R9VUcOhF1SMOX5C7Da33hE9t3vut7RsaUlMtKb7odY3dVrFOJ8ttab/EZaGDcvn4rl0+C6f3H8kbJPzrI17+Elrys5FT9duNuH9WRWPhvXrbz39+ke1T3h/pvUcXHEuxj36VNT6WPUBrH/UHRWPem2NWP+oO6xJz3De7H0PtwkLV0Q/Unb2UEsDp0aezl1s3DOTotbHv/G+9XTpaklOdsKrU2cbuP9Rm7Sp1NWw3VPnor84Yj233mHJcGpoDjvc+j5fUeUcY4VtoMRa33sfWdOnr6V+fUtqqjX9j7C+mXN3uRY1DVtTmxsUnqzuNm6u+39z6s8mor8k/YcFgnt6Bn8OwcCensG+r7R3dyJLFu/0Tp/e9iIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICX61aWzC2jmbyFxIM7OkZ7PsabNrTM/hzyE/e0zP4EzA1a6YrWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBwlZExAUKWxERFyhsRURcoLAVEXGBa2EbenosxR0zKWoQT3GfLMJzZ1fbPjx7FsV9spz2ndoSGj8uev2cTyk5/SSK9mtOUaIh9MLEmP1EvvuWkrNOpahpKkWNEinu3Y3IN18DYDduJDjySoq7dqCoYQLF7VsSvOof2NzcqD6KO7ShKNFEvUpvu2nXi7GnjR0LmZkQHw9ZWTC7+mPBrFlOu/h4aNsWxo2rvv1ernDCWDYcmsm61vHkDsgiuKD6/Q/Om0XugCzWtY4np0dbCp+vev8LHruXdRmGraOuiFpurSX//+5kw8HNWNcmgY2nHEHom68qxpg7k3UZJuar+H+vlbfb0L1NpfV5o3Uu7gtcCdvQ65MpvX4EvutvJm7+Ujy9+hA8eRCRX3+J2T7y02qCpxyPp1cf4uYvxXfdKEqvvZLwW1MqGuXn4+nUmcCDj0FCQpX9lBzdF0+bTOKmTidu8Zf47xgNyckA2LW/Y3//Df/oB4hbtAL/f14kMvdTgsPOrtSXb9TtxP+4tvzlu/HWP16YPWHyZBgxAm6+GZYuhT59YNAg+CX2sWD1ajj+eKfd0qUwahRceSVMmRK7/V6u+K3J5N02gqQRN9No2lL83fuw+ZxBhNfE3v/wz6vZdO7x+Lv3odG0pSReNYq8W66k+N3K+x9csoDCF8fj69Sl0rrCJx6gcNxD1LtnDI3eX4QnrQmbhhxLJD8PAP+hfUhbvjbqlXjVKExSMoGjB0X1lTTy9qh2SdfoXNwXGGttjRt7unW38XMX13qQ4sN74unchcDY8RXLDmqP95TT8d99X6X2pbfeSPjtN4hf8V35suA/LiHy9VfEz5xfqX1R42T8Dz+Bb+iwqOXBYeeAMQQmvFTjuYY/mErwtBOJX7sZk5LizLVDG7zDr8B/9XU17qc6RbE/G9zRsyd06QLjK44F7dvD6afDfZWPBTfeCG+8Ad9VHAsuuQS++grmVz4Wbklft2vb5Q7qib9TF1Ieqtj/nN7tiTvxdOrdUnn/8/51IyVT3yBtfsX+bxl5CeFVX9HwvYr9j2zdwsZju5Hy0HjyH7obX4fOpNz3BOBc1eYc3IyEi64g+epbnGVFRWzo3ITkOx4k8fzLYs41p+8BBHr3J+XBZ8qXbejehsSLriDp8t1zLq5L3y3d7Jo/yblI9+7YxYvNzprV+ZWtDQaxS5fgPWZA9MBHDyCyYF7MbSIL5+M5Orq999iB2M8XY0tLazZuJEJ46juYDp0oOek4ilo1prjfoYRen1z9dnlbIS4OEhOjlocefZCiFo0o7tmV0vvvwQaDNZrHXiUYhCVLYEB0bRkwAObFPhbMn1+5/cCBsHgx1PBY7C1sMEho+RIC/aP3J9B/AKWLYu9/6ZL5ldrHHTGQ0i+iz8Wt1/2duBNPJ9DvqEp9hH9ZTWR9NnHb9WMSEvD3OrzKcYNzZxL+4VsSzvt7pXWFTz3I+o6NyD26K/mP6lzcV87Fuv8aIScHwmFoEv0RapqkY9dlx9zErsvG7NCeJukQCjn91cT69ZCfT+j/7sVzzADi3pmG98yzKb3wXMJT34097ubNhO6+De+Fl2J8vvLl3suvIvD8K8S9PwPf8CsIPfEIpSMur9k89ibbjkX6DrVNT4fs2MeC7OzY7WtzLPYSkY3O/nsaR++Pp3E6kQ2x9z+yPjtme0Ihpz+g8MXxhH/6nuQb/1VlH+Xb1XDcohefwXfgwfi7do9annjJVdR/6hUaTJlBwkVXUPjMI2y9SefivnAu+nbeZDcxO1xlW1t52c7ax1pelUgEAO+Jf8N/1UgAPAd3xX6+mNDTT+I9/sTo7gsKCJ4+GNOsOf57Hohat217AM9BXSAlhdKhQ7Cj78c0alSz+exN3D4We5vduP+h71eRf+/NNHx7NiYQ2C3jRjbmUjz1Derd+XCldUnDK85Ff6cueJJT2HLZEOrdej+ehjoX92Z1f2WblgZeL+xwFWs3rK989VrGpGdUvurdsB58PqhpuKWlgc+H6dApuu8DOmJ3uBli8/MJnuzchAhMeRcTH19t155Dezrb/fB9zeayt9h2LHa8cli/vvIVwzYZGbHb1+ZY7CU8DZ3933aluU0kZz2etNj772mSEbM9Ph+eBo0oXTwfuzGH3CM6s665j3XNfZTOn0XRxLGsa+7DlpTgaZLhbFfDcYtemwQeD/GnnbvTffJ3c87F8E86F/d2dR62JhDAHJJF+JNpUcsj06fh6dUn9qR69iYy4+OoZeFPpmG6dcf4/TUe15N1KPa7VVHL7fffYlq2rvg5L4/g346DcJjAm1MxZU8qVMd+scz5n6ZNazSXvUYg4Dw2My36WDBtmnOHN5beveHjjyu3794dangs9hYmEMDXJYvgp9H7H/x0Gv5DY++/P6s3wdkfV25/sHMuxg06mUYzVtDo42XlL9/B3Yk/+SwafbwMAgG8rTLxNMmIGtcWF1O6cHbMcYteepb4wWfiSam/030q/co5Fz1NdC7u7Vz5GsF31UhKLx5KqHsPPL37Enp2HHbt73gvGQ5A8JLzAQg8OwkA7yXDCY17guD1V+O7+DIi8+cSfnEigedfKe/T5udXXFlGIthffyHyxTJo2BBPy1bOuNfcQHDomYT6HIbniKOIzJpB+LVXCUx+y+kjL4+SwQMgb6uzrKAAW1Dg9NmwISYQILxwPvazBXgOPxLq1yeyZBGlN1yD54STysfZp4wcCUOHQo8e0Lev85zi77/DcOdYcL5zLJjkHAuGD4cnnoCrr4bLLoO5c2HiRHjllZjd7+2SLhvJliuH4jukB4FD+1I4aRyR7N9JPN/Z/y1XOPtf/wln/xPPH07hf54g77arSRh6GcFFcymaPJH6Tzn776mfiqd+atQYJjEJk9oQX8fO5csSL72agsfuwduuA762+5P/6GhMUjLxp54TtW1w4RzC366MegKhfN3i+ZQuWUCg75F46tWndNki8u64hriBJ+FtoXNxr2etrfHLHJJlEwrtLr38jzxpTavWlkDAmq7dbOCjWeXrPIf1t57D+ke1D3w405qDD3Hat25j/Y89Fb3+gxkWqPTynndB9LhPT7CmXXtLfLw1nQ+y/okv77QPwAY+mGETCq2Nm7vEmkN7WurXd/rY/wDru/kOG59TsMu1qFXR6+L15JOW1s6xoFs3y6xZFev693de27efOdNyiHMsaNPG8tRTe3wf0rN3/VUayGiuAAAePElEQVTvvietp4Wz/74u3WyDN2eVr/P37m/9vftHtW/wxkzrO8jZf0/LNrbe/U9V27+/d3+bcOE/o5Y1WRuxSdfeYT1NMixxcdbf63DbaMaKStvGn3G+9bbvGLPfhh8tsf5uPa1Jcc5Fb7sDbNK1d9gmPxbsci329HH8M5yLZGXZmjRz5TlbibZHn7P9k9jV52wl2h59zvbPYm95zlZERBS2IiKuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIu8NWmsTcCSQV1NZW/jk4r9/QM9n2tft7TM/hzePPUPT2Dvw5d2YqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLXAvbov+MZWNWJjkt4tl0dBal82dX27507iw2HZ1FTot4NnZvS9HEcdH9Pfckm/p3ITczhdzMFDYP6k3wo/eq7C9v5N/JaWwofPLBqOWRddnkXT6U3E4Z5LROYtMRB1P8+ktRbQofvofNx/clp3USOY1NLfd891n/2lhWnJTJ533i+fq8LPKWVl/DvCWz+Pq8LD7vE8+Kv7Vlw+vRNbThML89dVt5nytOyuS3sbdiQ6HyNuHCfH554EqWH9+Cz/sm8OWpB7DupUei+tnwxjOsuuxIlh2RypLuhpLff4qex+KZLOluYr42ffzaHyvKLvjxg7F8+I9M3j4rnhnXZ5Gzsuo6Fm9ay6JHzmHalR148wwvS8YMq7bvX2e/wpunGebde2LU8lVv3MeMGw7lnfNSeO/Cxsy/dzBbf/kyqo21lq8n38n7lzTj7bMTmH37EWz95auoNh8Ob8Obp5mo15cv3FS7AuxNxo6FzEyIj4esLJhd/TnNrFlOu/h4aNsWxo2rvv1exJWwLXlzMgW3jCDx6ptJnb4U/6F92HLWIMJrfonZPvzzaracczz+Q/uQOn0piSNGUTDqSkremVIx8WYtSLrtflI/+ZzUjxfj73cUWy84mdBXyyuP/7/XCS1dhCejWaV1eVecT/jbr0l54W0azFpB3Jnnk3/5UErnfVrexgZLCJx4Kgl/v3o3VGPXbPxoMr8+OIKMC2+m40tLSerSh++vGkQwO3YNS35bzfcjjiepSx86vrSUjGGj+OX/rmTTJxU1zH7+fja89iQtr3ucA1//hpbXPsaG154ke+J95W3WPDKSLXPfI/PuFzjwta9petEt/PbETeS+90J5m0hxISm9BtD073fGnEvSwX3o8sHaqFfGhaPwJCaT0mfQ7ilQDa2ZO5nl/xnB/qfezJEPLqXhAX2Yd88gCjdUcS6WlhBISWP/U26iYfue1fZdkP0jX066nkYdD6u0LufLmbQ97nL63zuPfndOx3h9zLnrGIJ5G8vbfPfWA3z/v4focvEYjrx/EXEpTZh797GUFuVF9dXhjNsZ9Oza8leH02/dhUrsBSZPhhEj4OabYelS6NMHBg2CX2IfC1avhuOPd9otXQqjRsGVV8KUKbHb72VcCduicQ8Td9Yw4odeim//jiT/ewye9KYUT3gqZvvi58fhSW9G8r/H4Nu/I/FDLyVuyAUUja24Ko0b9DcCxwzC27Yd3v32J+mWezDJ9ShdPD+qr/CvP1NwywjqPf0y+P2Vxir9bB7xF/0Tf1ZPvG3aknj5tXiat6R06WflbZJuupvEy6/Fd9Ahu6kitbfupYdJGzyMxqdcSkJmR1rdMAZ/WlM2vB67hhumjMPfuBmtbhhDQmZHGp9yKY1OvIB1L1bUsGD5POofNpjUwwcT16wNqf1Pov7hJ1Hw5cLyNvlfzKPR8UOp1/1I4pq1odGJ55N0UK+oNunnXE3TC0eR3LVfzLl4/AH8aRlRr02fTKHhwLPxJibvpgrVzPfvPEyrI4eReeylpLToyMGXjCE+tSmrP4xdx6QmbTj44sdpfdQw/MkNq+w3Eipl0aNn0+mce0hKb1tpfd/bP6T1UReS0qoz9VsfRPerXqBk6wZyv5kLOFe137/7KPufchPNe59GSqvOZF35PKGiPNbMfjmqL19CPeIbZJS/fAnu1nC3efhhGDYMLr0UOnaEMWOgaVN4KvaxYNw4aNbMadexo7PdBRfAgw/Gbr+XqfOwtcEgoS+WEDhiQNTywBEDKF00L+Y2pYvmV25/5EBCyxZjS0srjxEOU/Lmq9iCfPyH9qlYHgqRd9nZJIy8Fd/+HWOO5e/Zj5K3/0tkYy42EqHk/beJ5G4gcPgxtd3VOhMpDVL4zRJSekXXJKXXAPKXx65hwYr5ldrX7z2QgpWLsSGnhsld+5G3eAbFP30DQNGPK8lbPJ2UvseXb5PctR+bP32HYPavgBO+hauWkdLnuF3en7zFMyn55VvSTvn7LvexKyKlQTb/sIT0g6Pr0qTrAHJXxa5jTa18+RYSG7eh9ZEX1Kh9qDgPIhH8yQ0AKFy3mpLN2TTpWjE3b1wCjTodXmlu3/3vQd69oBHTr+3KqtfvIVIa/ENz3yOCQViyBAZEHwsGDIB5VRyL+fMrtx84EBYvhhi5sLfx1fUAkY05EA7jaZwetdzTJB376cext1mfjad/dNh5GqdDKITNzcFkNAUgtHIFmwf1hpJiTFIyKRPfxNfpoPJtCu+/A9OgEQkX/qPK+dV77r/kXXoWGw9IA58PAnHUe/oVfAd13dVd3u1Cm50a+hpG19DXMJ3ShbFrWJqbTb0ex1RqTzhEaHMO/rSmpF9wI+GCPL46oxN4vBAOkXHRLTQ54/LybVpe/zi/3DucFSe2Aq9zurS6fgyph0V/J1kbG958hoT9DyapU/dd7mNXlOTlYCNh4upH1zG+fjobNseuY02sW/YRa+ZO5qiHltV4m+XPjaB+Zlca7d8bgOLN2QCV5haXmk5x7m/lP7c9/ipSMw8hUK8Rm77/jK9evImC9avpdvmzuzz/PSLHOadJj95f0tPh4yqORXY2HHNM5fahkNNf06Z1M9fdpM7DtpzZ4caStZWX7az9Dsu97Q6gwYxlRLZuJvjOFPKuvID6b83E17EzpXNnUfLqRFJnVP8GKLz3VuzGHFKmfIynYRrB998i/5/n4/3fp/g6H1ybPax7tayhqaqGOMs3fTSZ3KmTyBz9Mgn7HUjhqmX8+tAI4pplknbyxQBsmDyG/C/mst/D/yPQtDX5n3/KmseuI9CsDfV34eo2tDmXzTPeoMU1D9d6291mh7pYLNtqUlslW3P4/IlhdL/6ZQJlV6k7s3zCSHK/mcPho+dgvN5q57bjMW5/0sjy/6/fpgu+hBQWPTyEA4feT1y9Rru0D3tUHeTC3qrOw9bTMA28XiLrs6OWRzasx+xwtVu+TZMMIut2aJ+zHnw+TMOKE8oEAs53toC/a3dCyxZRNO4R6j32HMG5M4isW8vGztt92oXDFN59I8VPP0rD5WsIr/6B4mfHkDpjWXmw+jofTOmC2RQ9O4Z6j+4dVwu+VKeGodzomoQ2rcffKHYN/Y0yKI3RHq8PX6pTwzWPX0/6edfRcOBZACS0O4jg2p/JnngfaSdfTKS4iN+eGEXbf79G6uGDAUhs34XCb5ex7sUHdylsc9+bBMZDo0Hn1nrbPyquXhrG46Vkc3RdSrasJy41dh13ZusvX1K8aS1z76q44rI2AsBbZ/g4+tGvqNf8gPJ1yydcw5o5r3LYXTNIyqj4bjc+NcOZy+ZsEtNa1nhu227aFaz9ft8K2zTnnCY7+liwfn3lq91tMjJit/f5oNHev+91/p2tCQTwHZxFcNa0qOXBWdOivl/dnv/Q3pTu8BVDcNY0fF27Y2Lc5CoXiUCwBICECy8nddZyUmcsK395MpqRMPwaUt74BABbVOhst+PVhcfr9LWX8PgDJHbIYuvC6BpuXTiN5C6xa5h0UG+27vAVw9aF00jq1B3jc2oYKS7EeHbYd6+3PCxsqNT5fneH+pg/UJ+ct5+l4bFn4k2uv0vb/xEef4DU/bJY/0V0Hdd/MY1GB8Su4840aHcoRz+ygqMeWlb+atr9JBp1PIyjHlpGUpPM8rbLnxvBmtkv0++u6dRr0SGqn8T0TOJSM6LmFg4Wk/v17Grntvkn5ze3+AZ796/QlQQCziNc06KPBdOmOU8bxNK7d+WvGKZNg+7dY9783tu48jVCwvCR5P1zKP5DeuDr2ZfiieOIZP9O/LDhAOT983wA6j05CYD4C4ZT9NwT5N9yNfEXXEZo4VxKXp1IvadfKe+z4O6bCBx7Ap7mLbH5eZRMeZnSuTNJedl51tbTuAmexk2iJ+L3Y5pk4GvnXGl423fAk9mO/BsuJ+muB/E0aETJ+29ROmsa9V54u3yz8JpfsJs2Ev71JwBCK5wT3JvZDpPszp3g9HNH8tPtQ0k6sAdJB/clZ8o4Sjf8TtppTg1X3+7UMPNup4aNTxvOhv8+wa8PXU3aqZdR8MVcct+ZSOY9FTVMPWww2c//m7jmmcS3PZDCVUtZ/9LDNDzB6cubnEJyt/78NuYmvAnJBJq2Ju/zWeROnUSLKx8o76c0J5vS3GxKfvkWgOIfVxLO20wgoxW++hV38POXzaH4x5W0vvmZui1WNdoNHsnix4fSoH0PGnboy08fjqN40+9kDnDquPhxZ9+7XzWpfJvNq53jHSrcijEeNq9ehscXIKVlJ3zxSaS06hw1hj8plUgkFLV82fh/8uusF+h141sEkhpQvMm5QvPFJ+NLSMYYQ7sTr2bVlHtIbt6Bes3255vXR+OLT6bFYecAkLtqPpu+XUBa5yPxJ9Zn0/eLWDHxGjIOPYnExq3qrmh1ZeRIGDoUevSAvn2dpw1+/x2GO8eC851jwaSyYzF8ODzxBFx9NVx2GcydCxMnwiuvxOx+b+NK2MadMoTIplwKHxlNZN1avB06U/+VqXhbtgao9Lytt3Um9V+eSv5t11A88Sk8Gc1Iuvdx4gafVt4msj6bvMvPI7I+G5NSH1+nLqS8+j6BowbWeF7G76f+K1Mp+NdNbD1vMLYgH29mO5Ifn0DcwMHl7Qr/fTslk58v/3nzUc4jYClvzSDQ94hdKUmtNRwwhNCWXNY+N5rSnLUk7NeZdo9NJa6pU8Mdn7eNa55Ju8em8uvD17Dh9afwN25Gy+sep8HRFTVsef0Yfh93G7/8+3JKN63Hn9aUtFMupeklt5e3aXvvq/z25ChW33Yuoa0bCWS0ptnwf9F4yBXlbTZMGcfa8XeV//z91ScA0PqOCaQNHlbR7s3xxGd2JLlr391am9po0XcIwbxcVr0+muJNa0lp1Zk+N08lsYlTx6Kcys94zrgu+pG/7MXvkNi4NQPH/VTjcVd/MBaAOXceHbW8w5l30HHInQC0P/kGwsEivhj/T0oLNtGgfU/63v4R/oR6AHj9cayZO5lv/nsX4VAJiWmtaXPMpbQ/+YYaz2OvMmQI5ObC6NGwdi107gxTp0Jr51hUet42M9NZf801zuNhzZrB44/DaadV7nsvZGz5TZOd83ftblM/XlyH0/lraP3znp7Bvq+VarhbvHnqnp7Bn0D37tjFi3d6h07/NoKIiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4QGErIuICha2IiAsUtiIiLlDYioi4wFebxiEf5KTV1VT+Ok59Y0/PYN/39GV7egZ/Dsbu6Rn8dejKVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFygsBURcYHCVkTEBQpbEREXKGxFRFywb4Xt2LGQmQnx8ZCVBbNnV99+1iynXXw8tG0L48a5M8868tXMsbxycybP/TOeN+7JYu13Ve9/4Za1fPLsOUy+vQPjh3uZOXFYpTbvPHQEz1xmKr1eu/PA8jar5k2M2SZUWlzeJhIJs+jt28rn9srNmSx661Yi4ZCzPlzKwik38vrdXfjPlUm8cH1TPnn2HPI3/rL7ilMLYxlLJpnEE08WWcxmJ+dRmTnMwYePznSOWv4ar9Gd7qSSShJJdKUrz/N8VJs88riaq2lNaxJIoA99WMSi8vWllHIjN9KFLiSRRFOacg7n8AvRNfqBHziFU2hMY1JI4UzOZB3rdrESe4G/0Hvat6cnUGOTJ8OIEc7B6dfP+XPQIFi5Elq1qtx+9Wo4/ni46CJ48UWYMwcuvxwaN4bTTnN//n/QD4smM2/yCPqdM5aMdv1YOXMs748ZxJl3riS5YeX9D5eWEJ+cRtfjbuKb2c/E7PPY4W8QCQUrtgmV8PrdB9E268yodr5AImeN/iF6mT++/P+/+OB+Vs58kiOGPU/D5geR+9tyZk68AK8/jm4n3EYoWEjOr59zyKBbaNSyK8GiLSx4/VqmPn4cp9+2HI/XvdNwMpMZwQjGMpZ+9GMsYxnEIFayklbEOI/KbGIT53M+R3M0v/Fb1LpGNOJWbqUDHfDj513e5WIupjGNOZ7jAbiES1jOcp7neVrQghd5kWM4hpWspDnNKaSQz/mcW7iFrnRlC1u4lms5juNYznJ8+CiggAEM4CAO4hM+wWC4jdsYzGAWsADPPnbt9Fd7Txtrbc0bd+9uWby4DqdTjZ49oUsXGD++Yln79nD66XDffZXb33gjvPEGfPddxbJLLoGvvoL58+t+vtX4e+zsq9ab9/WkUYsuHD60Yv9fva09bbudTo9TYuz/dj544kTik9M4YtjEatt9t/AlZk44n7Pv/Ynkhi0B58p27qtXcNHj+dX2H5fUiCMvrLiamzHhAkoKcjnuindjbrPp95W8dteBnH77cho2P6jaecXy9GW13gSAnvSkC10YT0Ud29Oe0zmd+6i6jqdyKgdzMBbL67zOl3xZ7Tjd6MZABnIf91FEEfWoxxSm8Df+Vt4miywGMYjRjI7Zx0pWciAHspzlHMRBfMRHHMdx5JJLAxoAsIUtNKABH/ERx3BMbUoBgKn523/3+7O8p7t3xy5ebHbWbN/4KAwGYckSGDAgevmAATBvXuxt5s+v3H7gQFi8GEpL62aedSQcCpLzyxJadIrenxYdB7Duhyr2fxd8M2c8LTsPKg/a8vGDRbw8qjUv3diCD544kZxflkatz2jXj99XzWBz9jeAE6S/r5pOy87HVzlWsHgrAIHEBrtt/jsTJMgSljCA6DoOYADzqLqOYxlLNtncyq07HcNi+YRPWMUqDudwAEKECBMmnviotgkkMIc5Vfa1FadG24K1hBIMJqqfeOLx4Km2n73SX/A9vW+EbU4OhMOQnh69PD0dsrNjb5OdHbt9KOT0tw8pzs/BRsIk1Iven4SUdAq3VrH/tbR53bes/XYWHfpdGrU8Nf0A+l/wHwZc/jZHXfIKXn88bz/Qly3rKq4uDh54I+17DeW/d3Zi/D/8vHbXgezf6wIOPOLymGOFQ0EWvH4trboMJrlBi90y/5rIIYcwYdKJrmM66WQTu44rWMFd3MVLvIQXb5V9b2ELySQTIMAJnMDjPM4gBgFQj3r0pjejGc1v/EaYMC/yIvOZz1rWxuwvSJBruZbBDKYFTo160Ytkkrme6yko++86riNMuMp+9lp/wff0vhG225gdrtStrbxsZ+1jLd9XxNyf3bMv38weT2L9prQ66ISo5en79Wb/3heQ1rIrTdsfxtGXTial8X58OWNMeZsfFk/muwWTOOrilznt1s854sJJrJw1lm/mPFdpnEg4xIz/nEewcDNHXDBht8y9tswONbPYSsvAuZI8i7N4kAfJJLPaPutRj2UsYxGLuId7GMlIPuGT8vUv8AIePLSgBXHE8TiPczZnxwzwECHO4zw2s5kJVNSoMY15jdd4n/epRz3qU5/NbKYb3ar9INir/YXe0/vGDbK0NPB6K3/irV9f+ZNum4yM2O19PmjUqG7mWUfik9MwHi9FO1zFFuWtJzGliv2vhXAoyLcLnqdDv0t3erPK4/HSuHV3tq6vuLJdOOV6uhx7He0OPQuAhs0PIj/3Z5Z9cB8d+l1c3i4SDvHJs2ez8bcVDL52JvHJ7h6HNNLw4q10Fbue9ZWudgHWspaVrOTCsv8AIkSwWHz4mMrU8q8kPHhoRzsAutKVr/mae7mXozkagP3Yj1nMooACtrKVpjRlCEMqhXiIEGdzNitYwUxm0ojoGg1gAD/wAznk4MNHKqlkkLHTD4O9zl/wPb1vXNkGAs7jHtOmRS+fNg369Im9Te/e8PHHldt37w5+f93Ms454fQHSWmWxZmX0/v/29TTS96ti/2vhp6VvUpyfQ4e+F++0rbWWjb8tJ6F+0/JloWAhxhN9ZWU8XqyNlP8cCZfy8fghbPxtOYOvnUFi/Yw/PO/aChAgiyymEV3HaUyjD5Xr2JzmrGAFy7b7bzjDaUc7lrEs5jbbRIhQQkml5dse69rEJj7kw6gbZqWUMoQhLGc5M5hBBlXXKI00UkllOtNZz3pO4qSalGDv8Rd8T+8bV7YAI0fC0KHQowf07es8X/f77zB8uLP+/POdPydNcv4cPhyeeAKuvhouuwzmzoWJE+GVV/bI9P+oLseMZMaEoTTJ7EH6fn35+tNxFGz5nY6HO/s/Y4Kz/0deOKl8m5xflwEQLNoKxkPOr8vwegM0aNYpqu9v5oyneYejSWncttK4S965iyZte1G/SXuCxVv5cvrj5K5ZTr9znipv07rLYL744N+kpGXSoOmB5Py6lBUfP0z7Xs6cIuEQ054+gw0/L+K4f74DGAq3OFcogYT6+AIJu69QOzGSkQxlKD3oQV/6Mo5x/M7vDMep4/k4c57EJPz4Kz1T24QmxBEXtfwe7qEnPWlLW0ooYSpTeYEXGEPFVy0f8iERInSgA9/zPddzPQdwQPkVc4gQZ3AGi1jEO7yDwZRfgdenPgk4NZrABDrQgSY0YT7zGcEIruEaDuCAuitaXfmLvaf3nbAdMgRyc2H0aFi7Fjp3hqlToXVrZ/0vOzwgn5nprL/mGnjqKWjWDB5/fJ94Hi+W/Q4dQnFBLp9PHU3hlrU0bNaZQVdMpV4jZ/9j/QWBN0YfEvXzL8vfIblRa86596fyZVs3/Mhvq6Zz9CWvxhy3pGgzs1/8O4Vbswkk1Cet5SGcdN2nNMnsUd6mz1ljWPz2bcx5+XLnq436TenQ71K6nXg7AAWb1vDzF287c7onK6r//hdM4IA+w2pdj101hCHkkstoRrOWtXSmM1OZSmucOu74lwhqIp98/sE/WMMaEkigAx2YxCTO5uzyNlvYwihGsYY1NKQhp3Ea93APfpwrsjWs4W2cGmURXaMJTGAYwwBYxSpGMYqNbKQNbbiFW7iGa3alFHveX+w9ve88Z/snsivP2Uq0XX3OVqLt0eds/yz+VM/Ziojs4xS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLhAYSsi4gKFrYiICxS2IiIuUNiKiLjAWGtr3tiYDcDPdTcdEZF9TmtrbeOdNapV2IqIyK7R1wgiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi5Q2IqIuEBhKyLiAoWtiIgLFLYiIi74f6OVqt302S9HAAAAAElFTkSuQmCC\n", 237 | "text/plain": [ 238 | "
" 239 | ] 240 | }, 241 | "metadata": {}, 242 | "output_type": "display_data" 243 | } 244 | ], 245 | "source": [ 246 | "# evaluate the policy \n", 247 | "V = policy_evaluation(env, random_policy)\n", 248 | "\n", 249 | "plot_values(V)" 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "metadata": {}, 255 | "source": [ 256 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 257 | "\n", 258 | "**Note:** In order to ensure accurate results, make sure that your `policy_evaluation` function satisfies the requirements outlined above (with four inputs, a single output, and with the default values of the input arguments unchanged)." 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": 8, 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "data": { 268 | "text/markdown": [ 269 | "**PASSED**" 270 | ], 271 | "text/plain": [ 272 | "" 273 | ] 274 | }, 275 | "metadata": {}, 276 | "output_type": "display_data" 277 | } 278 | ], 279 | "source": [ 280 | "check_test.run_check('policy_evaluation_check', policy_evaluation)" 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "metadata": {}, 286 | "source": [ 287 | "### Part 2: Obtain $q_\\pi$ from $v_\\pi$\n", 288 | "\n", 289 | "In this section, you will write a function that takes the state-value function estimate as input, along with some state $s\\in\\mathcal{S}$. It returns the **row in the action-value function** corresponding to the input state $s\\in\\mathcal{S}$. That is, your function should accept as input both $v_\\pi$ and $s$, and return $q_\\pi(s,a)$ for all $a\\in\\mathcal{A}(s)$.\n", 290 | "\n", 291 | "Your algorithm should accept four arguments as **input**:\n", 292 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 293 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 294 | "- `s`: This is an integer corresponding to a state in the environment. It should be a value between `0` and `(env.nS)-1`, inclusive.\n", 295 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 296 | "\n", 297 | "The algorithm returns as **output**:\n", 298 | "- `q`: This is a 1D numpy array with `q.shape[0]` equal to the number of actions (`env.nA`). `q[a]` contains the (estimated) value of state `s` and action `a`.\n", 299 | "\n", 300 | "Please complete the function in the code cell below." 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": 9, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "def q_from_v(env, V, s, gamma=1):\n", 310 | " q = np.zeros(env.nA)\n", 311 | " for a in range(env.nA):\n", 312 | " for prob, next_state, reward, done in env.P[s][a]:\n", 313 | " q[a] += prob * (reward + gamma * V[next_state])\n", 314 | " return q" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "Run the code cell below to print the action-value function corresponding to the above state-value function." 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 10, 327 | "metadata": {}, 328 | "outputs": [ 329 | { 330 | "name": "stdout", 331 | "output_type": "stream", 332 | "text": [ 333 | "Action-Value Function:\n", 334 | "[[0.0147094 0.01393978 0.01393978 0.01317015]\n", 335 | " [0.00852356 0.01163091 0.0108613 0.01550788]\n", 336 | " [0.02444514 0.02095298 0.02406033 0.01435346]\n", 337 | " [0.01047649 0.01047649 0.00698432 0.01396865]\n", 338 | " [0.02166487 0.01701828 0.01624865 0.01006281]\n", 339 | " [0. 0. 0. 0. ]\n", 340 | " [0.05433538 0.04735105 0.05433538 0.00698432]\n", 341 | " [0. 0. 0. 0. ]\n", 342 | " [0.01701828 0.04099204 0.03480619 0.04640826]\n", 343 | " [0.07020885 0.11755991 0.10595784 0.05895312]\n", 344 | " [0.18940421 0.17582037 0.16001424 0.04297382]\n", 345 | " [0. 0. 0. 0. ]\n", 346 | " [0. 0. 0. 0. ]\n", 347 | " [0.08799677 0.20503718 0.23442716 0.17582037]\n", 348 | " [0.25238823 0.53837051 0.52711478 0.43929118]\n", 349 | " [0. 0. 0. 0. ]]\n" 350 | ] 351 | } 352 | ], 353 | "source": [ 354 | "Q = np.zeros([env.nS, env.nA])\n", 355 | "for s in range(env.nS):\n", 356 | " Q[s] = q_from_v(env, V, s)\n", 357 | "print(\"Action-Value Function:\")\n", 358 | "print(Q)" 359 | ] 360 | }, 361 | { 362 | "cell_type": "markdown", 363 | "metadata": {}, 364 | "source": [ 365 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 366 | "\n", 367 | "**Note:** In order to ensure accurate results, make sure that the `q_from_v` function satisfies the requirements outlined above (with four inputs, a single output, and with the default values of the input arguments unchanged)." 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": 11, 373 | "metadata": {}, 374 | "outputs": [ 375 | { 376 | "data": { 377 | "text/markdown": [ 378 | "**PASSED**" 379 | ], 380 | "text/plain": [ 381 | "" 382 | ] 383 | }, 384 | "metadata": {}, 385 | "output_type": "display_data" 386 | } 387 | ], 388 | "source": [ 389 | "check_test.run_check('q_from_v_check', q_from_v)" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "### Part 3: Policy Improvement\n", 397 | "\n", 398 | "In this section, you will write your own implementation of policy improvement. \n", 399 | "\n", 400 | "Your algorithm should accept three arguments as **input**:\n", 401 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 402 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 403 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 404 | "\n", 405 | "The algorithm returns as **output**:\n", 406 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 407 | "\n", 408 | "Please complete the function in the code cell below. You are encouraged to use the `q_from_v` function you implemented above." 409 | ] 410 | }, 411 | { 412 | "cell_type": "code", 413 | "execution_count": 12, 414 | "metadata": {}, 415 | "outputs": [], 416 | "source": [ 417 | "def policy_improvement(env, V, gamma=1):\n", 418 | " policy = np.zeros([env.nS, env.nA]) / env.nA\n", 419 | " for s in range(env.nS):\n", 420 | " q = q_from_v(env, V, s, gamma)\n", 421 | " \n", 422 | " # OPTION 1: construct a deterministic policy \n", 423 | " # policy[s][np.argmax(q)] = 1\n", 424 | " \n", 425 | " # OPTION 2: construct a stochastic policy that puts equal probability on maximizing actions\n", 426 | " best_a = np.argwhere(q==np.max(q)).flatten()\n", 427 | " policy[s] = np.sum([np.eye(env.nA)[i] for i in best_a], axis=0)/len(best_a)\n", 428 | " \n", 429 | " return policy" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": {}, 435 | "source": [ 436 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 437 | "\n", 438 | "**Note:** In order to ensure accurate results, make sure that the `policy_improvement` function satisfies the requirements outlined above (with three inputs, a single output, and with the default values of the input arguments unchanged).\n", 439 | "\n", 440 | "Before moving on to the next part of the notebook, you are strongly encouraged to check out the solution in **Dynamic_Programming_Solution.ipynb**. There are many correct ways to approach this function!" 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "execution_count": 13, 446 | "metadata": {}, 447 | "outputs": [ 448 | { 449 | "data": { 450 | "text/markdown": [ 451 | "**PASSED**" 452 | ], 453 | "text/plain": [ 454 | "" 455 | ] 456 | }, 457 | "metadata": {}, 458 | "output_type": "display_data" 459 | } 460 | ], 461 | "source": [ 462 | "check_test.run_check('policy_improvement_check', policy_improvement)" 463 | ] 464 | }, 465 | { 466 | "cell_type": "markdown", 467 | "metadata": {}, 468 | "source": [ 469 | "### Part 4: Policy Iteration\n", 470 | "\n", 471 | "In this section, you will write your own implementation of policy iteration. The algorithm returns the optimal policy, along with its corresponding state-value function.\n", 472 | "\n", 473 | "Your algorithm should accept three arguments as **input**:\n", 474 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 475 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 476 | "- `theta`: This is a very small positive number that is used to decide if the policy evaluation step has sufficiently converged to the true value function (default value: `1e-8`).\n", 477 | "\n", 478 | "The algorithm returns as **output**:\n", 479 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 480 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 481 | "\n", 482 | "Please complete the function in the code cell below. You are strongly encouraged to use the `policy_evaluation` and `policy_improvement` functions you implemented above." 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": 14, 488 | "metadata": {}, 489 | "outputs": [], 490 | "source": [ 491 | "def policy_iteration(env, gamma=1, theta=1e-8):\n", 492 | " policy = np.ones([env.nS, env.nA]) / env.nA\n", 493 | " while True:\n", 494 | " V = policy_evaluation(env, policy, gamma, theta)\n", 495 | " new_policy = policy_improvement(env, V)\n", 496 | " \n", 497 | " # OPTION 1: stop if the policy is unchanged after an improvement step\n", 498 | " if (new_policy == policy).all():\n", 499 | " break;\n", 500 | " \n", 501 | " # OPTION 2: stop if the value function estimates for successive policies has converged\n", 502 | " # if np.max(abs(policy_evaluation(env, policy) - policy_evaluation(env, new_policy))) < theta*1e2:\n", 503 | " # break;\n", 504 | " \n", 505 | " policy = copy.copy(new_policy)\n", 506 | " return policy, V" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": {}, 512 | "source": [ 513 | "Run the next code cell to solve the MDP and visualize the output. The optimal state-value function has been reshaped to match the shape of the gridworld.\n", 514 | "\n", 515 | "**Compare the optimal state-value function to the state-value function from Part 1 of this notebook**. _Is the optimal state-value function consistently greater than or equal to the state-value function for the equiprobable random policy?_" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": 15, 521 | "metadata": {}, 522 | "outputs": [ 523 | { 524 | "name": "stdout", 525 | "output_type": "stream", 526 | "text": [ 527 | "\n", 528 | "Optimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\n", 529 | "[[1. 0. 0. 0. ]\n", 530 | " [0. 0. 0. 1. ]\n", 531 | " [0. 0. 0. 1. ]\n", 532 | " [0. 0. 0. 1. ]\n", 533 | " [1. 0. 0. 0. ]\n", 534 | " [0.25 0.25 0.25 0.25]\n", 535 | " [0.5 0. 0.5 0. ]\n", 536 | " [0.25 0.25 0.25 0.25]\n", 537 | " [0. 0. 0. 1. ]\n", 538 | " [0. 1. 0. 0. ]\n", 539 | " [1. 0. 0. 0. ]\n", 540 | " [0.25 0.25 0.25 0.25]\n", 541 | " [0.25 0.25 0.25 0.25]\n", 542 | " [0. 0. 1. 0. ]\n", 543 | " [0. 1. 0. 0. ]\n", 544 | " [0.25 0.25 0.25 0.25]] \n", 545 | "\n" 546 | ] 547 | }, 548 | { 549 | "data": { 550 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAFoCAYAAAD5IVjuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Hl4VNX9x/H3yUwme0IgGxGBIBKs7CAKiBRFEPeKFhURbOWHbd2gasGl1VZrrUC1KlBRQRRBq6hYRAUrVBYXEATFDRUQSCCBhGyEJJPz+2NCZMgEISQnIp+Xz3365Nzvvffc70w+uXPnFmOtRUREGlZYY09ARORYoLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtvKjYozZaIwZ0NjzaAjGmGHGmLcaex7SOBS2RzljzOnGmOXGmN3GmF3GmGXGmFOq1o00xiw9jH21NsZYY4y3jnMZb4z5X4jxJGNMmTGmQ132Wx+MMTOq5lC03zK0AY9Xo5fW2lnW2oENdUz5cVPYHsWMMfHAf4BHgKbAccA9wN5GmtIzQG9jTMYB45cD66y1nzTCnPb3d2tt7H7L8408HzmGKGyPbu0ArLWzrbV+a+0ea+1b1tq1xpiTgKlAr6qruHwAY8x5xpjVxpgCY8x3xpi799vfvqvS/KptelVt8ytjzGfGmDxjzJvGmFahJmOt3QL8Fxh+wKqrgaer9nWCMea/xpidxphcY8wsY0yTUPuruhq9d7+ff26M2bLfz+nGmJeMMTnGmG+NMTcecueCj2ONMW1DHXffMY0xvzfG7DDGZBljrtmvNsoYM9EYs6nq08VSY0wUIXp54CcNY0xvY8yHVdt9aIzpvd+6xcaYv1R9Uik0xrxljEmqy/nJj4PC9uj2JeA3xjxtjBlsjEnct8Ja+xlwHbCi6ipuX6AVEwi/JsB5wG+MMRdXrTuj6n+bVG2zomrd7cAlQDLwLjD7IHN6mv3C1hiTCXTZbxsD3A+kAycBxwN3H+6JG2PCgNeAjwlc0Z8F3GyMGXS4+zoEaUBC1XF+DTy2X68nAN2B3gQ+XdwGVBKilwfMvykwH/gn0AyYBMw3xjTbr+xK4BogBfABt9T/qYkrCtujmLW2ADgdsMA0IMcYM88Yk3qQbRZba9dZayuttWsJhGC/gxxmNHC/tfYza20F8FegS21Xt8DLQOp+V2lXAwustTlVx99grV1ord1bNTbpB45fm1OAZGvtn621Zdbabwj04PKDbHOLMSa/ask9jGOVA3+21pZba18HioDMqsD/FXCTtXZr1aeL5dbaQ7mNcx7wlbX2GWtthbV2NvA5cMF+NdOttV9aa/cALxD4oyVHKYXtUa4qBEdaa1sAHQhcMT5UW70x5lRjzDtVH713E7j6PdjH01bAw/tCCthF4Or0OGPM7ft92TS1aj4lwL+Bq40xBhhG1S2EquOnGGPmGGO2GmMKgGd/4PgHm1f6fuGZT+AKvNY/NMAEa22TquVwjrmz6g/NPiVALIF5RwJfH+7kCbxOmw4Y20Tg6nmf7BDHlKOUwvYnxFr7OTCDQOhC4Ir3QM8B84DjrbUJBO7rmoPUfweM3i+kmlhro6qu4P6635dN1+23zdPAL4GzgTgCX+Ltc3/VcTpZa+OBq/Y7/oGKgej9fk47YF7fHjCvOGvtubXs62BKDnKcg8kFSoETQqz7oX9ObxuBPxj7awlsPcRjy1FGYXsUM8a0r/ripkXVz8cDVwDvVZVsB1oYY3z7bRYH7LLWlhpjehK4L7hPDoH7jW32G5sKjDfGnFx1jARjzGU/MLV3gXzgcWCOtbbsgOMXEfji6Djg1oPsZw1wrjGmqTEmDbh5v3UfAAXGmD9UfUnlMcZ0MFWPvR2mNcCVVfs4h0O8rWGtrQSeAiZVfVnnqfoiLILQvdzf60A7Y8yVxhivCTyG9jOC/zDJT4jC9uhWCJwKvG+MKSYQsp8Av69a/1/gUyB7v3uUvwX+bIwpBP5I4F4gUH0L4D5gWdVH89OstS8DDwBzqj72fwIMPtikbOAfSZ5J4Mpt5gGr7wG6AbsJfEE09yC7eobAF2AbgbeA6ke1rLV+Avc3uwDfErjKfILAF1mH66aqfeUTuO3xymFsewuwDviQwC2WB4CwUL3cfyNr7U7gfAKv1U4CX6ydb609nHvJchQx+sfDRUQanq5sRUQcUNiKiDigsBURcUBhKyLigMJWRMSBw/qn9Jp6kmwLb+sGmoqIyNFnS8VGdvlza/s/5lQ7rLBt4W3NvPSVdZ+ViMhPzIXbehxSnW4jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDjgLGyfKZhM3y0ZZG6M5IJt3fmg9N2D1r9a9Bznbu3CSZuiOWVzGjfnXEVORXb1+tmF07gsqy9dNjel06YmXJHdnw9LlwbtY2bBY5yztRMdN8XTcVM8l2T14r8l84NqbskZScZGE7T8YttpQTXjckfRb8sJtN8URffNyYzafhEbyj47wo4cPvWwfqiPR049PHxOwvY/xc/z51038duE25mfvppuEb25ZvtgtlZsDlm/snQZY3OHMyR2BG+lf8rjKa+woXw9N+cOq655v3Qx58cMZVbq27yc/j5tvJmM2D6Ib8u/qq5J87ZgXOIDvJb+Ea+mr6RX5JmM3nExn5WtDTpen8gBfNAiq3qZnvp60PpOET14MGkGi9I/4+nUN7FYrto+gHJbXo9dOjj1sH6oj0dOPawbY6095OJOET3svPSVh32Qi7edSntfJ/6WNK16rP+WExkccym3Jd5fo/7x3RN4uuARlh2/qXrs34XTuXvXDXzaqijkMay19PyuOb9rcgcj42+odS5dNjfltsT7uTJuNBD4S5hXmcuTqf855PP5rGwt527rzKLjPueE8MxD3u5IqIf1Q308cuphsAu39WDt3pXmh+oa/Mq2zJbxSdkq+kYNDBo/PWogq0qXh9ymR0QfcvxZLCp5DWstu/y5vFY8h59HnVv7cShjry0lISwx5Hq/9fNa0RxKKovoFtE7aN2HpUvpsTmF/lvaMS53FLn+HbUep6SymBeLppPuaUkLb+ta6+qTelg/1Mcjpx7WnbdB9w7k+XPx4yfJkxo0nuxJZZl/UchtukX24uHk2YzJGUap3UMFFZweeTYTk56u9TgT8+4kJiyWAdEXBo1/XraOIVm92GtLiTaxTE15mfa+jtXr+0Wdw6CYSzjem8GWio1MzLuTYdlnMi99FREmorrumYLJ/C3vNkpsMW28mcxKeztofUNSD+uH+njk1MO6c/YFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CN94tih/Gf9NXMSVtCRng7fpdzGXsqS+rSijpTD+uH+njk1MPD1+BXtomeJDx4yPFnB43n+nfU+Ou4z+Td99M5oiejE24F4CQ6Ed0shl9m9+WWxPtI9x5fXTu94GEm5t3J9NQFdInoWWNfPuOjdXhbIHBjfO3eD3mq4B88kPRkyGOnetNJ87ZgY8VXQePxYQnEhyWQEX4iXSNOo8vmRBaUvMQlscMPvRl1pB7WD/XxyKmHddfgV7Y+46ODrztL9ywMGl+6ZyHdI3uH3KbUlhCGJ2jMU/Wz5fsv9J7YPYkJeXfwZOp8Tok8/ZDmU0kle+3eWtfv8ueyvWIryZ7mtdbYqv/KDrKf+qQe1g/18ciph3XX4Fe2ANcmjGVsznA6R/SkR0QfZhVOZYd/G1fGXQfA2JyrAZiUPBOAs6IuYPzOUTxbMIUzogaxw5/FX3bdTAdfN47ztgTgX7sfZGLeHUxKfpY23nbVz+xFhEURH5YAwAO7xtE/+jzSPcdTZAuZV/wc75Uu5qmUwEeK4soiHsq/m8HRQ0jxNGdLxUb+nj+eZp4UBkX/AoCN5Rt4o+Ql+kQOoKknmeyKLUzZ/Td8JoKzos530T71sB6pj+phY/XQSdieHzOUPP9OHs2/lxx/Fu18HXgq9XVaeFsBsO2A5/MujRtJkS1kZuGj3Jf3e+LCEugV2Z9xiX+vrnmm4DHKKeeGnKFB2w6JGcGE5BkA5PizGZNzFbn+bOLCEmjv68T01AX0ixoEBP66flG2jpeLZlJQmU+ypzm9IvvzaPILxIbFAeAzEbxXupgndk+koDKfJE8qPSPPYG7aCpK9aQ3VshrUw/qhPh459bBunDxnKyLyU/Wjec5WREQUtiIiTihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBzwNvYEjkUZ3zb2DI5+/xjT2DP4aRjzj8aewU9Aj0Mr05WtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOOAvbZwom03dLBpkbI7lgW3c+KH33oPWvFj3HuVu7cNKmaE7ZnMbNOVeRU5FdvX524TQuy+pLl81N6bSpCVdk9+fD0qVB+5hZ8BjnbO1Ex03xdNwUzyVZvfhvyfygmltyRpKx0QQtv9h2WlDNuNxR9NtyAu03RdF9czKjtl/EhrLPjrAjjWjyZMjIgMhI6N4d3j34a8GSJYG6yEho0wamTnUzzway9OPJ/OWpDG59JJKJz3Xn6621n/+G7xYz5iFTY9m+6/PqmhXrpvHPF/pyx5SmjJ/chMde7M83W4Pfi6Vlhby8+Gb+/GQrbnskioef783m7A9rPe7zi/6PMQ8Z3lk1IWh8+brHeezF/oyf3IQxDxl27d5Ytyb8WBxD70UnYfuf4uf5866b+G3C7cxPX023iN5cs30wWys2h6xfWbqMsbnDGRI7grfSP+XxlFfYUL6em3OHVde8X7qY82OGMiv1bV5Of5823kxGbB/Et+VfVdekeVswLvEBXkv/iFfTV9Ir8kxG77iYz8rWBh2vT+QAPmiRVb1MT309aH2niB48mDSDRemf8XTqm1gsV20fQLktr8cuOfL883DTTXD77bB6NfTuDYMHw+bQrwXffgvnnhuoW70axo+HG26Al15yO+96svqL53l5yU0MOOV2bhm2mtbNe/P4K4PJK6jl/Kv8Yfin3DMqq3pJbnJi9boNWxbTtd1QfjPkbcZc/j7JiZn86+VB5OR9/158fuG1fL7pTa4c+DS3Dl9HZsuBTJk7gPyirTWOtearF/lu+4ckxKTXWFdeXkJmy4Gcc9rddW/Cj8Ux9l50ErZP7J7EkNiRXBE3ira+k7in2SOkeJozq3BKyPqP9q4gzdOCXyeM4fjwDLpGnsaIuBtYs/f96pqHkmcxIv56To7oygnhmdzbbAoxJo4le96orhkYfRE/jx5M6/C2tAlvx62J9xETFsfqvSuCjhdhIkj2plUvTTxNg9ZfGTeanpF9aRHemg4R3fh94r1s929jc8U39dglRyZNgpEjYdQoOOkkeOQRaN4cpoR+LZg6FdLTA3UnnRTYbsQImDAhdP2P3OKPJtHzZyPp1XEUqU1PYkj/R4iPac6ytbWcf5XY6BTiY9Kql7AwT/W64YNn0bfL9bRI6UpK00wuO3MKEb44PtsUeC+WVexh7YaXOP/0v9H2+J+T3KQt5/S6m6QmbVl+wHF3FWzilcU3MXzwc4SFhdeYR79uNzOg53gyjju9HrrRyI6x92KDh22ZLeOTslX0jRoYNH561EBWlS4PuU2PiD7k+LNYVPIa1lp2+XN5rXgOP486t/bjUMZeW0pCWGLI9X7r57WiOZRUFtEtonfQug9Ll9Jjcwr9t7RjXO4ocv07aj1OSWUxLxZNJ93Tkhbe1rXW/SiVlcGqVTAw+LVg4EBYHvq1YMWKmvWDBsHKlVB+dF3ZV/jL2LJjFZktg88ns+VANmbVcv5VJj3Xgz8+3pzJL53FV9+9c9Bav7+M8opSoiMC78XKygoqrZ9wT2RQXbg3Kuh2g7+ygmcWXMHZPe8ktelJh3NqR59j8L3Y4GGb58/Fj58kT2rQeLInlRx/dshtukX24uHk2YzJGUa7TT66f5eMxTIx6elajzMx705iwmIZEH1h0PjnZes4eVMsmZsiuGPndUxNeZn2vo7V6/tFncPE5Jk8m/Y2dzSdyMd7P2BY9pnstXuD9vNMwWRO3hTLyZtjWVyygFlpbxNhIg63HY0rNxf8fkgNfi1ITYXs0K8F2dmh6ysqAvs7ihTvyaXS+omLDj6fuOhUCkpCn398THMuPXMK15z/Er86fy4piZlMeeksvt7yv1qP8/qKO4nwxdKhTeC9GOmLo3XzXrz1wb3kF22lstLPys+eZWPWCgpKsqq3e2PFn4iObEafzr+ph7P9kTsG34teVwcymKCfLbbG2D5fla3nnl03cn2TuzgjahA7/Fncv+tWbt85mknJM2vUTy94mNmF/+KZtEXEhcUHrWsTnsn89DUUVObzRslL3JI7gtlpi8n0dQDggtjLq2vb+zrS0ded07e04p2S+ZwTc0n1uotih3F61Nns8GcxbfcEfpdzGS+mLSMqLLrOPWk05oC+W1tz7IfqQ40fLcyhvxdTmmaS0jSz+ufW6b3YVbCRdz6awAktzqhRv2T1wyxf9y9+c8kiIiO+fy8OG/QMcxb+inueaEGY8dAipRvdMq9gy46PANiwZQkfrp/BLcPW1McZHj2Oofdig4dtoicJD54aV7G5/h01rnb3mbz7fjpH9GR0wq0AnEQnopvF8MvsvtySeB/p3uOra6cXPMzEvDuZnrqALhE9a+zLZ3y0Dm8LBL7oWrv3Q54q+AcPJD0Z8tip3nTSvC3YWPFV0Hh8WALxYQlkhJ9I14jT6LI5kQUlL3FJ7PBDb0ZjS0oCj6fmlcOOHTWvGPZJSwtd7/VCs2YNM88GEhOVRJjxUFgcfD5FJTtqXO0eTMu0U1n9xZwa40tWP8yC5XfyfxcvoFVa8HsxqckJXH/ZEvaWF1NaVkBCTHOenj+UpvEZAGz47h0KirP407Tm1dtUWj+vLf0DS1Y/xN3XbjmcU/3xOwbfiw1+G8FnfHTwdWfpnoVB40v3LKR7ZO+Q25TaEsLwBI15qn622OqxJ3ZPYkLeHTyZOp9TIg/tC4NKKmvcItjfLn8u2yu2kuxpXmuNrfqv7CD7+VHy+QKPzSwMfi1YuDDwDW8ovXrBokU163v0gPCaX+D8mHk9PlqkdOeLzcHn/+XmhbRuXsv5h7AtZw3xMcHvj8UfTeL15Xcw6qL5tDnIl1cR4TEkxDSnpDSPzze9SYcTLgKgT+ffcutVa7ll2JrqJSEmnX5dx/DbS94+jLM8ShyD70UntxGuTRjL2JzhdI7oSY+IPswqnMoO/zaujLsOgLE5VwNU3yI4K+oCxu8cxbMFU6pvI/xl18108HXjOG9LAP61+0Em5t3BpORnaeNtV/0MbkRYFPFhCQA8sGsc/aPPI91zPEW2kHnFz/Fe6WKeSgk8a1tcWcRD+XczOHoIKZ7mbKnYyN/zx9PMk8Kg6F8AsLF8A2+UvESfyAE09SSTXbGFKbv/hs9EcFbU+S7aV7/GjoXhw6FnT+jTJ/AN77ZtcF3gteDqwGvBzKrbNdddB48+CjffDKNHw7JlMGMGzJ7dKNM/Uj/vNpZZbw6nVVpPMtL7sGztVHYXb6N3p8D5z3ozcP7DBgXOf8lHD9E0vjVpzU6morKMVZ89y7qvX+Ga879/3Oi/Kx/k9eV3MOycZ0lObEdB1ZVzuDeKqIjAe/HzjW9ibSUpTduTm7+Bee/eSkpiJqf+7BoA4qJTiItOCZprWFg48TFpQbcxCoqzKSzOJifvSwCyd61nz958msS3JCYy+CmaH71j7L3oJGzPjxlKnn8nj+bfS44/i3a+DjyV+jotvK0A2HbA87aXxo2kyBYys/BR7sv7PXFhCfSK7M+4xL9X1zxT8BjllHNDztCgbYfEjGBC8gwAcvzZjMm5ilx/NnFhCbT3dWJ66gL6RQ0CAlfLX5St4+WimRRU5pPsaU6vyP48mvwCsWFxAPhMBO+VLuaJ3RMpqMwnyZNKz8gzmJu2gmRvWkO1rOEMHQo7d8K990JWFnToAK+/Dq0Cr0WNZxwzMgLrx4wJPJKTng7//CcMGeJ+7vWga+ZQikt38tb791JQkkXzZh34v4tep2l84PwPfN62orKMee/ewu6irYR7o0htdjKjLprPzzK+fzJm6ceP4a8sZ+brwe/FU04awZWDZgCwp2w385eNJ79oC9ERTel84hDO7X0fHs/hXZEtXzuVN9+/p/rnaa+eB8AVZ0+n58kjD2tfje4Yey8aa+0PV1XpFNHDzktf2YDTOTZkfNvYMzj6/WNMY8/gp2HMPxp7Bj8BPXpgV678wW/o9G8jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURccDb2BM4Fn2b0dgzOPrlN2nsGYgcHl3Ziog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigLOwfaZgMn23ZJC5MZILtnXng9J3D1r/atFznLu1CydtiuaUzWncnHMVORXZ1etnF07jsqy+dNnclE6bmnBFdn8+LF0atI+ZBY9xztZOdNwUT8dN8VyS1Yv/lswPqrklZyQZG03Q8ottpwXVjMsdRb8tJ9B+UxTdNyczavtFbCj77Ag7cvjUwyP3Qu5kzlufwakfR3LlF935qKj2Hv5x00i6rjE1ll5rY4LqyivLmJz1R85bn0HPjyMY/GlLnsv5Z8h9LsibTdc1hhu/OT9o/NxPW4c81g3fnFdds6rof9z0zYUM/PQ4uq4xzNs5o+6N+LGYPBkyMiAyErp3h3cP/p5myZJAXWQktGkDU6e6mWc9cBK2/yl+nj/vuonfJtzO/PTVdIvozTXbB7O1YnPI+pWlyxibO5whsSN4K/1THk95hQ3l67k5d1h1zfulizk/ZiizUt/m5fT3aePNZMT2QXxb/lV1TZq3BeMSH+C19I94NX0lvSLPZPSOi/msbG3Q8fpEDuCDFlnVy/TU14PWd4rowYNJM1iU/hlPp76JxXLV9gGU2/J67NLBqYdH7s2853lwy038OvV2ZmeuplNMb67/ZjBZZaF7eGuLh1l4clbQ0sLXhoFNfhlUN27TFSwvfIO7jn+cV9p/wd9b/5t2kZ1q7G/L3m94aNutdI3pW2Pds5kfBh1ndruPMBjO3u9YJZVFtI3swK3HPUykiTrCbvwIPP883HQT3H47rF4NvXvD4MGwOfTrwbffwrnnBupWr4bx4+GGG+Cll9zOu46MtfaQiztF9LDz0lce9kEu3nYq7X2d+FvStOqx/ltOZHDMpdyWeH+N+sd3T+DpgkdYdvym6rF/F07n7l038GmropDHsNbS87vm/K7JHYyMv6HWuXTZ3JTbEu/nyrjRQOCqLK8ylydT/3PI5/NZ2VrO3daZRcd9zgnhmYe83ZFQD4PlNznsTRj+5amcGNmJP7b8vocXrj+RAU0u5cb0mj080JqiZVyz4XSmn7iMLjG9AVhR8Ba3bbyMeT/7mkRvUq3blttyfvXV6VyW9FtWFr5Dvj+Xf7apvV9PZN/H0zse5K0O24gKi66xvvfaWMYd9ygXNhv5g/M+mK6rj2jzI3PqqdCpE0z7/vXgxBPh0kvh/hCvxx/+AHPnwlffXwxw7bXw6aewYkXDz7c2PXpgV640P1TW4Fe2ZbaMT8pW0TdqYND46VEDWVW6POQ2PSL6kOPPYlHJa1hr2eXP5bXiOfw86tzaj0MZe20pCWGJIdf7rZ/XiuZQUllEt4jeQes+LF1Kj80p9N/SjnG5o8j176j1OCWVxbxYNJ10T0taeFvXWlef1MMjV15Zxmclq+gVH9zDXvED+bg4dA8PNHfnNE6IPLk6aAHe2f0KP4s+hWd3TGLQpy24cP2JPLDlRkr8wX/QHsu6g3Rfay5sOuIHj2Ot5ZVdT3Je06tCBu1PQlkZrFoFA4NfDwYOhOW1vB4rVtSsHzQIVq6EcnefkOrK29AHyPPn4sdPkic1aDzZk8oy/6KQ23SL7MXDybMZkzOMUruHCio4PfJsJiY9XetxJubdSUxYLAOiLwwa/7xsHUOyerHXlhJtYpma8jLtfR2r1/eLOodBMZdwvDeDLRUbmZh3J8Oyz2Re+ioiTER13TMFk/lb3m2U2GLaeDOZlfZ20PqGpB4euX3/v1A8AAASvElEQVQ9bOoN7mFTbyo7K0L3cH+F/t0s3P1vbmj+16DxrWXfsKZ4Kb6wCCa0folCfz4PbL2BnPJtTMh4EQhc/b6Z9zzPZ645pLm+V7iQrWXf8oum1x7i2R2FcnPB74fU4NeD1FRYVMvrkZ0NAwbUrK+oCOyvefOGmWs9cfYFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CXQBfFDuM/6auZk7aEjPB2/C7nMvZUltSlFXWmHh65mv2qvYf7e33Xs1RaP+clDg8ar6QSg+GvrZ6jY8yp9I4fxLjjHuXt3S+xs3w7eRW5/HHzSP7S8mnivaE/MRxo7s5pnBx9CpnRXQ71tI5e5oDeW1tz7IfqQ43/CDX4lW2iJwkPHnL82UHjuf4dNa7U9pm8+346R/RkdMKtAJxEJ6KbxfDL7L7ckngf6d7jq2unFzzMxLw7mZ66gC4RPWvsy2d8tA5vCwS+pFm790OeKvgHDyQ9GfLYqd500rwt2FjxVdB4fFgC8WEJZISfSNeI0+iyOZEFJS9xSezwkPupT+rhkdvXw50VwT3cVbGjxtVuKHN3TuOsJkNI8DYNGk/yNicl/DjiPAnVYxmRJwGQXb6ZPf5iciuyuO7r76/IKqkEoMcaLy+2/5TWkd/fs95VvoPFBa8yvsVjh3+SR5OkJPB4Aler+9uxo+bV7j5paaHrvV5o1qxh5lmPGvzK1md8dPB1Z+mehUHjS/cspHtk75DblNoSwvAEjXmqfrZ8/4XeE7snMSHvDp5Mnc8pkacf0nwqqWSv3Vvr+l3+XLZXbCXZU/tHElv1X9lB9lOf1MMjFx7m46To7rxXGNzD9woX0jkmdA/3WVf8Pl+WfswlTUfVWNclpg855duC7tFu2vslAM3DW3Fy9Cn8O3MdczLXVC/94i+ka0xf5mSu4ThfRtD+Xt01HZ+JYFCTy+t6qkcHny/wCNfC4NeDhQsDTxuE0qtXzVsMCxdCjx4QHt4w86xHDX5lC3BtwljG5gync0RPekT0YVbhVHb4t3Fl3HUAjM25GoBJyTMBOCvqAsbvHMWzBVM4I2oQO/xZ/GXXzXTwdeM4b0sA/rX7QSbm3cGk5Gdp421X/fxoRFgU8WGBq4wHdo2jf/R5pHuOp8gWMq/4Od4rXcxTKYGPt8WVRTyUfzeDo4eQ4mnOloqN/D1/PM08KQyK/gUAG8s38EbJS/SJHEBTTzLZFVuYsvtv+EwEZ0UFPyupHv64e3hV8lju3Dyck6N70iWmDy/mTiWnfBuXJgV6eOemQA/vbTUzaLu5O6fRMuJEusf2q7HPwYlXMm37X/jT5mu4Lu1uCv35PLj1JgYkXErT8BQA2kZ1CNomztMEPxU1xq21vLzzCQY1uZwYT1yNY5X4i/hu74aq2kqyyjfzRcka4r1Nae5rWceuNKKxY2H4cOjZE/r0CTwzu20bXBd4Pbg68Hows+r1uO46ePRRuPlmGD0ali2DGTNg9uxGmf7hchK258cMJc+/k0fz7yXHn0U7XweeSn2dFt5WAGw74FnRS+NGUmQLmVn4KPfl/Z64sAR6RfZnXOLfq2ueKXiMcsq5IWdo0LZDYkYwIXkGADn+bMbkXEWuP5u4sATa+zoxPXUB/aIGAYErvS/K1vFy0UwKKvNJ9jSnV2R/Hk1+gdiwwJvdZyJ4r3QxT+yeSEFlPkmeVHpGnsHctBUke9MaqmU1qIdHblDiUHb7d/JE9r3kVmTRNrIDj7R5nXRfoIfZIZ63LfYX8mb+HP4v9Y+YEPcFoz2xTD1hEQ9svYGrvjyFOG8i/RMu5sbmfzvs+a0sWsx3ZRv4a7NZIdevL1nJqK/7V/88NftPTM3+ExckjuDPrWYc9vEa3dChsHMn3HsvZGVBhw7w+uvQKvB61HjeNiMjsH7MGJgyBdLT4Z//hCFD3M+9Dpw8ZytS3+rynK3U1KjP2f5U/FiesxUREYWtiIgTClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERB7yHU7yuI2SsbKipHDvyEht7Bke/1hsbewYih0dXtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOHB0he3kyZCRAZGR0L07vPvuweuXLAnURUZCmzYwdaqbeTaQJ/ZOpnNBBmn5kfy8sDvLKw5+/v8ue46+BV1Iz48mc3ca/1d8Fdsrs4Nqpu59mJ4F7WmeH8XJu1twS8nvKLJF1esnld7PmYWn0DI/nra7k7m86ALW+z8J2sd9e+6iZ0F7jsuPofXuRC4qOov3K5YH1Zxf+HMS803Q8qviy4+wI3UzmclkkEEkkXSnO+9y8D4+xmOcxElEEUUmmcxkZq21s5mNwXA+5weN/4//cSEXchzHYTDMYEaNbecyl0EMIplkDIbFLK5Rk002wxlOGmnEEENnOjOLWYd03j9Kx9LvtLX2kBe6dz/04vpe5syxeL2Wxx+3rF9vuf56S0yMZdOm0PXffGOJjg7UrV8f2M7rtbz4YuOdQ9WS1+Twlyej51gvXvtQ1OP2vbj1dpTvehtDjF0bvylk/YLYpTaMMHtf5CS7Ju4b+1bsCtvJ09We4T2zuubx6FnWh89OiZ5pP4771r4a87ZtGdbaXuX7VXXNmd6B9tGop+yyuHV2adxae174xTbFpNpv4ndW10yNfsa+ErPIro772i6P+8QO9/3axhFnv4jPrq7p4+lnh/musZ/HZ1UvGxPy69SLvCZ1b/4cAn18nMftetbb6wn0cRObQtZPZrKNIcY+x3P2a762s5ltY4m185hXo/ZrvrbHcZztS197HucFrZvPfDue8fbf/NtGEWWnM73G9jOZae/mbjuTmRaw7/BOjZqzOdt2p7t9j/fs13xtJzDBGoxdwpI69aNRfxd+Kr/T3bvbQyk7rH02atj27Gm59trgsbZtLePGha6/7bbA+v3Hfv1ry2mnNe4LY+sWLt09Pe3VvmuDxtqEtbU3R4wLWf/nyAdtC9MyaOzRqKdsDDHVP1/r+53t7TkjqOa2iD/a9mEn1zqP7xIKbRhh9rmYebXWbErYbQH7YswbQWF7re93dQ7X+grbnvS013Jt0Fhb2tpxjAtZ34te9mZuDhoby1jbhz5BY2WU2Z70tDOYYUcwokbY7r/EEBMybPctOeTUGrYxxNineCporCUt7YM8WKd+NOrvwk/ld/oQw/bouI1QVgarVsHAgcHjAwfC8uWht1mxomb9oEGwciWUlzfMPBtImS1jjX8V/b3B59PfO5APKkKf/6nePmy3WSwofw1rLTsrc5lbPoezw8+trjnNezrr/Gv4sOI9AL6r3MyCinlBNQcqsoVUUkkTk1jrXJ/e+zhxxNPR0yVo3dzyOZywO4leBSdz155bKLSFh3T+9aWMMlaxioEE93EgA1lO6D7uZS+RRAaNRRHFB3xAOd+/j+7gDlrTmhGMqP+J7+d0TucFXmAnO6mkkld5lRxyGMCABj1uvTsGf6ePjrDNzQW/H1JTg8dTUyE7O/Q22dmh6ysqAvs7iuy0ufjxkxwWfD4pYanssKHPv6e3F09Ez2Z08TBSdvtoW5CMxTI5+unqmiG+y7kr6q+cV3QGyfnhdCpoxc/COnJP5AO1zmX8npvo6OlCT0+voPE3yv9Di/xY0nZHMmXvP3g5diEp+833Ut+VPB49i3mx73BL5F3MK3+Jq4svqUs76iyXQB9TCe5jKqlkE7qPgxjEUzzFh3yIxbKSlTzBE5RTTi6B99FbvMXzPM9UGv7+4Qu8gMGQRBIRRDCMYcxmNl3o8sMb/5gcg7/TR0fY7mNM8M/W1hz7ofpQ40cJQ/C8LbbG2D6f+9czbs+N3BJ5F+/EreLFmDfYXpnNmJLR1TXLKpYwofQvTIiazOK4j3gmei5LKxZzf+mfQu7zjj1jea9iKTOjX8JjPEHr+nr787+4NbwZu5yzws/hmuJfkl2ZVb1+ZMT/cVb4IE72dGSI73Kein6exRWL+Ljio7q2o84Op493cRfncR696U044VzERdVXrx485JLLSEbyNE+TSOir/fp0J3eSSy6LWMRKVnIrt3I1V/MxHzf4sRvEMfQ77W3sCRySpCTweGr+xduxo+Zfun3S0kLXe73QrFnDzLOBNDNJePCw44AnCXIqd5BsQp//P0rvp5u3JzdG3gpAB08nok0M5xb15c7K+2gRdjz37rmTIb4ruDriWgBO9nSkmGJuKrmW2yL/iNd8//a4fc8Y5pbNYV7sO7T2tKlxvBgTQxtPW9rQllO8p9G94ESeKXuCWyPvCjm/rp4eePDwdeVXdKZbnfpyuJII9PHAq9gd7KhxtbtPFFE8xVP8i3+xne00pzmP8zhxxJFEEv/jf2SRFfQxvpJKALx4+ZRPySSzXub/NV/zCI+whjV0pjMAnenMu7zLIzzCEzxRL8dx4hj8nT46rmx9vsDjHgsXBo8vXAi9e4feplcvWLSoZn2PHhAe3jDzbCA+46OLpzuLK4LPf3HFQnp6Q5//HkrwEHz1ue9niz1ozb71+4wruYkXy57j1dj/0s7T/pDmXEkle+3eWtd/Wrku8JE+rPkh7a8++PDRne4sJLiPC1lIb2p5H1UJJ5wWtMCDhznM4XzOJ4wwTuEU1rGONfv9dyEX0pe+rGENGWTU2/xLKAEI+ZrtC/ijxrH4O304X7o1+qNf4eGWadMCj33ceGPgMZGNGwPrhw8PLPvq9z0mctNNgfpp0wLbN/ZjIrbuj36FE24fjppm34tbb0f7brQxxNiP4zfavCbWDg0fboeGD6+ufyxquvXitROiJtvVcV/bBbFLbVdPD9vZ06265g8Rf7JxxNknomfbNXHf2Lkxb9mMsBPsBeGXVNf82vdbG0ecfTXm7aDHtr5LKKx+8uD3EXfYhbHv2bXxm+w7sSvtMN811ofPvhv3sc1rYu1HcRvs+Mh77H9jP7Qfx31rn4+Zb9uFtbedPF1tbkKF80e/wgm305hm17Pe3kigjxvZaC3WDme4Hc7w6vov+MLOZKb9ki/t+7xvhzLUNqWp/ZZvaz1GqKcRCim0q1ltV7PaRhFl7+Eeu5rVQY+c7WSnXc1q+w7vWMBOY5pdzWqbRZa1BJ54aEtb25e+9n3etxvYUP3oV6hH0Q5ladTfhZ/K7/RP7tEvay2PPWZp1cri81m6dbMsWfL9un79Asv+9YsXW7p2DdS3bm2ZMqVx51+11PVxpwejHrPHm1bWh8929nSz/4ldEvRoVR9Pv6D6B6L+aduH/cxGEWVTTZodEn6F/ST+u+r1OQnldlzk3bZNWFsbSaRNNy3sr3y/sd/G76quAUIuf4j4k81rYu3WhGJ7XvjFNs00tz58Ns00t4O9F9qFse9V72Nd/Gbb23OGTTRNrQ+fzQg7wY723Rj0rK6rsLVY+xiP2VYE+tiNbkHPqPajn+1Hv+qf17PedqGLjSLKxhNvL+Ii+zmfH3T/ocJ2X4AeuIxgRHXNdKaHrPkTf6qu+ZIv7SVcYlNIsdFE2050sjOYUedeNPbvwk/id/oQw9ZYa2tc7dbG9OhhWbmy/i6rj1F5Df89yk9ek/zGnsFPgzn0X3+pTY8e2JUrf/AbuqPjnq2IyFFOYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBY6099GJjcoBNDTcdEZGjTitrbfIPFR1W2IqISN3oNoKIiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIA/8PmfP94gQGKz8AAAAASUVORK5CYII=\n", 551 | "text/plain": [ 552 | "
" 553 | ] 554 | }, 555 | "metadata": {}, 556 | "output_type": "display_data" 557 | } 558 | ], 559 | "source": [ 560 | "# obtain the optimal policy and optimal state-value function\n", 561 | "policy_pi, V_pi = policy_iteration(env)\n", 562 | "\n", 563 | "# print the optimal policy\n", 564 | "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n", 565 | "print(policy_pi,\"\\n\")\n", 566 | "\n", 567 | "plot_values(V_pi)" 568 | ] 569 | }, 570 | { 571 | "cell_type": "markdown", 572 | "metadata": {}, 573 | "source": [ 574 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 575 | "\n", 576 | "**Note:** In order to ensure accurate results, make sure that the `policy_iteration` function satisfies the requirements outlined above (with three inputs, two outputs, and with the default values of the input arguments unchanged)." 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": 16, 582 | "metadata": {}, 583 | "outputs": [ 584 | { 585 | "data": { 586 | "text/markdown": [ 587 | "**PASSED**" 588 | ], 589 | "text/plain": [ 590 | "" 591 | ] 592 | }, 593 | "metadata": {}, 594 | "output_type": "display_data" 595 | } 596 | ], 597 | "source": [ 598 | "check_test.run_check('policy_iteration_check', policy_iteration)" 599 | ] 600 | }, 601 | { 602 | "cell_type": "markdown", 603 | "metadata": {}, 604 | "source": [ 605 | "### Part 5: Truncated Policy Iteration\n", 606 | "\n", 607 | "In this section, you will write your own implementation of truncated policy iteration. \n", 608 | "\n", 609 | "You will begin by implementing truncated policy evaluation. Your algorithm should accept five arguments as **input**:\n", 610 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 611 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 612 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 613 | "- `max_it`: This is a positive integer that corresponds to the number of sweeps through the state space (default value: `1`).\n", 614 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 615 | "\n", 616 | "The algorithm returns as **output**:\n", 617 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 618 | "\n", 619 | "Please complete the function in the code cell below." 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": 17, 625 | "metadata": {}, 626 | "outputs": [], 627 | "source": [ 628 | "def truncated_policy_evaluation(env, policy, V, max_it=1, gamma=1):\n", 629 | " num_it=0\n", 630 | " while num_it < max_it:\n", 631 | " for s in range(env.nS):\n", 632 | " v = 0\n", 633 | " q = q_from_v(env, V, s, gamma)\n", 634 | " for a, action_prob in enumerate(policy[s]):\n", 635 | " v += action_prob * q[a]\n", 636 | " V[s] = v\n", 637 | " num_it += 1\n", 638 | " return V" 639 | ] 640 | }, 641 | { 642 | "cell_type": "markdown", 643 | "metadata": {}, 644 | "source": [ 645 | "Next, you will implement truncated policy iteration. Your algorithm should accept five arguments as **input**:\n", 646 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 647 | "- `max_it`: This is a positive integer that corresponds to the number of sweeps through the state space (default value: `1`).\n", 648 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 649 | "- `theta`: This is a very small positive number that is used for the stopping criterion (default value: `1e-8`).\n", 650 | "\n", 651 | "The algorithm returns as **output**:\n", 652 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 653 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`.\n", 654 | "\n", 655 | "Please complete the function in the code cell below." 656 | ] 657 | }, 658 | { 659 | "cell_type": "code", 660 | "execution_count": 18, 661 | "metadata": {}, 662 | "outputs": [], 663 | "source": [ 664 | "def truncated_policy_iteration(env, max_it=1, gamma=1, theta=1e-8):\n", 665 | " V = np.zeros(env.nS)\n", 666 | " policy = np.zeros([env.nS, env.nA]) / env.nA\n", 667 | " while True:\n", 668 | " policy = policy_improvement(env, V)\n", 669 | " old_V = copy.copy(V)\n", 670 | " V = truncated_policy_evaluation(env, policy, V, max_it, gamma)\n", 671 | " if max(abs(V-old_V)) < theta:\n", 672 | " break;\n", 673 | " return policy, V" 674 | ] 675 | }, 676 | { 677 | "cell_type": "markdown", 678 | "metadata": {}, 679 | "source": [ 680 | "Run the next code cell to solve the MDP and visualize the output. The state-value function has been reshaped to match the shape of the gridworld.\n", 681 | "\n", 682 | "Play with the value of the `max_it` argument. Do you always end with the optimal state-value function?" 683 | ] 684 | }, 685 | { 686 | "cell_type": "code", 687 | "execution_count": 19, 688 | "metadata": {}, 689 | "outputs": [ 690 | { 691 | "name": "stdout", 692 | "output_type": "stream", 693 | "text": [ 694 | "\n", 695 | "Optimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\n", 696 | "[[1. 0. 0. 0. ]\n", 697 | " [0. 0. 0. 1. ]\n", 698 | " [0. 0. 0. 1. ]\n", 699 | " [0. 0. 0. 1. ]\n", 700 | " [1. 0. 0. 0. ]\n", 701 | " [0.25 0.25 0.25 0.25]\n", 702 | " [0.5 0. 0.5 0. ]\n", 703 | " [0.25 0.25 0.25 0.25]\n", 704 | " [0. 0. 0. 1. ]\n", 705 | " [0. 1. 0. 0. ]\n", 706 | " [1. 0. 0. 0. ]\n", 707 | " [0.25 0.25 0.25 0.25]\n", 708 | " [0.25 0.25 0.25 0.25]\n", 709 | " [0. 0. 1. 0. ]\n", 710 | " [0. 1. 0. 0. ]\n", 711 | " [0.25 0.25 0.25 0.25]] \n", 712 | "\n" 713 | ] 714 | }, 715 | { 716 | "data": { 717 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAFoCAYAAAD5IVjuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Hl4VNX9x/H3yUwme0IgGxGBIBKs7CAKiBRFEPeKFhURbOWHbd2gasGl1VZrrUC1KlBRQRRBq6hYRAUrVBYXEATFDRUQSCCBhGyEJJPz+2NCZMgEISQnIp+Xz3365Nzvvffc70w+uXPnFmOtRUREGlZYY09ARORYoLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtvKjYozZaIwZ0NjzaAjGmGHGmLcaex7SOBS2RzljzOnGmOXGmN3GmF3GmGXGmFOq1o00xiw9jH21NsZYY4y3jnMZb4z5X4jxJGNMmTGmQ132Wx+MMTOq5lC03zK0AY9Xo5fW2lnW2oENdUz5cVPYHsWMMfHAf4BHgKbAccA9wN5GmtIzQG9jTMYB45cD66y1nzTCnPb3d2tt7H7L8408HzmGKGyPbu0ArLWzrbV+a+0ea+1b1tq1xpiTgKlAr6qruHwAY8x5xpjVxpgCY8x3xpi799vfvqvS/KptelVt8ytjzGfGmDxjzJvGmFahJmOt3QL8Fxh+wKqrgaer9nWCMea/xpidxphcY8wsY0yTUPuruhq9d7+ff26M2bLfz+nGmJeMMTnGmG+NMTcecueCj2ONMW1DHXffMY0xvzfG7DDGZBljrtmvNsoYM9EYs6nq08VSY0wUIXp54CcNY0xvY8yHVdt9aIzpvd+6xcaYv1R9Uik0xrxljEmqy/nJj4PC9uj2JeA3xjxtjBlsjEnct8Ja+xlwHbCi6ipuX6AVEwi/JsB5wG+MMRdXrTuj6n+bVG2zomrd7cAlQDLwLjD7IHN6mv3C1hiTCXTZbxsD3A+kAycBxwN3H+6JG2PCgNeAjwlc0Z8F3GyMGXS4+zoEaUBC1XF+DTy2X68nAN2B3gQ+XdwGVBKilwfMvykwH/gn0AyYBMw3xjTbr+xK4BogBfABt9T/qYkrCtujmLW2ADgdsMA0IMcYM88Yk3qQbRZba9dZayuttWsJhGC/gxxmNHC/tfYza20F8FegS21Xt8DLQOp+V2lXAwustTlVx99grV1ord1bNTbpB45fm1OAZGvtn621Zdbabwj04PKDbHOLMSa/ask9jGOVA3+21pZba18HioDMqsD/FXCTtXZr1aeL5dbaQ7mNcx7wlbX2GWtthbV2NvA5cMF+NdOttV9aa/cALxD4oyVHKYXtUa4qBEdaa1sAHQhcMT5UW70x5lRjzDtVH713E7j6PdjH01bAw/tCCthF4Or0OGPM7ft92TS1aj4lwL+Bq40xBhhG1S2EquOnGGPmGGO2GmMKgGd/4PgHm1f6fuGZT+AKvNY/NMAEa22TquVwjrmz6g/NPiVALIF5RwJfH+7kCbxOmw4Y20Tg6nmf7BDHlKOUwvYnxFr7OTCDQOhC4Ir3QM8B84DjrbUJBO7rmoPUfweM3i+kmlhro6qu4P6635dN1+23zdPAL4GzgTgCX+Ltc3/VcTpZa+OBq/Y7/oGKgej9fk47YF7fHjCvOGvtubXs62BKDnKcg8kFSoETQqz7oX9ObxuBPxj7awlsPcRjy1FGYXsUM8a0r/ripkXVz8cDVwDvVZVsB1oYY3z7bRYH7LLWlhpjehK4L7hPDoH7jW32G5sKjDfGnFx1jARjzGU/MLV3gXzgcWCOtbbsgOMXEfji6Djg1oPsZw1wrjGmqTEmDbh5v3UfAAXGmD9UfUnlMcZ0MFWPvR2mNcCVVfs4h0O8rWGtrQSeAiZVfVnnqfoiLILQvdzf60A7Y8yVxhivCTyG9jOC/zDJT4jC9uhWCJwKvG+MKSYQsp8Av69a/1/gUyB7v3uUvwX+bIwpBP5I4F4gUH0L4D5gWdVH89OstS8DDwBzqj72fwIMPtikbOAfSZ5J4Mpt5gGr7wG6AbsJfEE09yC7eobAF2AbgbeA6ke1rLV+Avc3uwDfErjKfILAF1mH66aqfeUTuO3xymFsewuwDviQwC2WB4CwUL3cfyNr7U7gfAKv1U4CX6ydb609nHvJchQx+sfDRUQanq5sRUQcUNiKiDigsBURcUBhKyLigMJWRMSBw/qn9Jp6kmwLb+sGmoqIyNFnS8VGdvlza/s/5lQ7rLBt4W3NvPSVdZ+ViMhPzIXbehxSnW4jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDjgLGyfKZhM3y0ZZG6M5IJt3fmg9N2D1r9a9Bznbu3CSZuiOWVzGjfnXEVORXb1+tmF07gsqy9dNjel06YmXJHdnw9LlwbtY2bBY5yztRMdN8XTcVM8l2T14r8l84NqbskZScZGE7T8YttpQTXjckfRb8sJtN8URffNyYzafhEbyj47wo4cPvWwfqiPR049PHxOwvY/xc/z51038duE25mfvppuEb25ZvtgtlZsDlm/snQZY3OHMyR2BG+lf8rjKa+woXw9N+cOq655v3Qx58cMZVbq27yc/j5tvJmM2D6Ib8u/qq5J87ZgXOIDvJb+Ea+mr6RX5JmM3nExn5WtDTpen8gBfNAiq3qZnvp60PpOET14MGkGi9I/4+nUN7FYrto+gHJbXo9dOjj1sH6oj0dOPawbY6095OJOET3svPSVh32Qi7edSntfJ/6WNK16rP+WExkccym3Jd5fo/7x3RN4uuARlh2/qXrs34XTuXvXDXzaqijkMay19PyuOb9rcgcj42+odS5dNjfltsT7uTJuNBD4S5hXmcuTqf855PP5rGwt527rzKLjPueE8MxD3u5IqIf1Q308cuphsAu39WDt3pXmh+oa/Mq2zJbxSdkq+kYNDBo/PWogq0qXh9ymR0QfcvxZLCp5DWstu/y5vFY8h59HnVv7cShjry0lISwx5Hq/9fNa0RxKKovoFtE7aN2HpUvpsTmF/lvaMS53FLn+HbUep6SymBeLppPuaUkLb+ta6+qTelg/1Mcjpx7WnbdB9w7k+XPx4yfJkxo0nuxJZZl/UchtukX24uHk2YzJGUap3UMFFZweeTYTk56u9TgT8+4kJiyWAdEXBo1/XraOIVm92GtLiTaxTE15mfa+jtXr+0Wdw6CYSzjem8GWio1MzLuTYdlnMi99FREmorrumYLJ/C3vNkpsMW28mcxKeztofUNSD+uH+njk1MO6c/YFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CN94tih/Gf9NXMSVtCRng7fpdzGXsqS+rSijpTD+uH+njk1MPD1+BXtomeJDx4yPFnB43n+nfU+Ou4z+Td99M5oiejE24F4CQ6Ed0shl9m9+WWxPtI9x5fXTu94GEm5t3J9NQFdInoWWNfPuOjdXhbIHBjfO3eD3mq4B88kPRkyGOnetNJ87ZgY8VXQePxYQnEhyWQEX4iXSNOo8vmRBaUvMQlscMPvRl1pB7WD/XxyKmHddfgV7Y+46ODrztL9ywMGl+6ZyHdI3uH3KbUlhCGJ2jMU/Wz5fsv9J7YPYkJeXfwZOp8Tok8/ZDmU0kle+3eWtfv8ueyvWIryZ7mtdbYqv/KDrKf+qQe1g/18ciph3XX4Fe2ANcmjGVsznA6R/SkR0QfZhVOZYd/G1fGXQfA2JyrAZiUPBOAs6IuYPzOUTxbMIUzogaxw5/FX3bdTAdfN47ztgTgX7sfZGLeHUxKfpY23nbVz+xFhEURH5YAwAO7xtE/+jzSPcdTZAuZV/wc75Uu5qmUwEeK4soiHsq/m8HRQ0jxNGdLxUb+nj+eZp4UBkX/AoCN5Rt4o+Ql+kQOoKknmeyKLUzZ/Td8JoKzos530T71sB6pj+phY/XQSdieHzOUPP9OHs2/lxx/Fu18HXgq9XVaeFsBsO2A5/MujRtJkS1kZuGj3Jf3e+LCEugV2Z9xiX+vrnmm4DHKKeeGnKFB2w6JGcGE5BkA5PizGZNzFbn+bOLCEmjv68T01AX0ixoEBP66flG2jpeLZlJQmU+ypzm9IvvzaPILxIbFAeAzEbxXupgndk+koDKfJE8qPSPPYG7aCpK9aQ3VshrUw/qhPh459bBunDxnKyLyU/Wjec5WREQUtiIiTihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBzwNvYEjkUZ3zb2DI5+/xjT2DP4aRjzj8aewU9Aj0Mr05WtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOOAvbZwom03dLBpkbI7lgW3c+KH33oPWvFj3HuVu7cNKmaE7ZnMbNOVeRU5FdvX524TQuy+pLl81N6bSpCVdk9+fD0qVB+5hZ8BjnbO1Ex03xdNwUzyVZvfhvyfygmltyRpKx0QQtv9h2WlDNuNxR9NtyAu03RdF9czKjtl/EhrLPjrAjjWjyZMjIgMhI6N4d3j34a8GSJYG6yEho0wamTnUzzway9OPJ/OWpDG59JJKJz3Xn6621n/+G7xYz5iFTY9m+6/PqmhXrpvHPF/pyx5SmjJ/chMde7M83W4Pfi6Vlhby8+Gb+/GQrbnskioef783m7A9rPe7zi/6PMQ8Z3lk1IWh8+brHeezF/oyf3IQxDxl27d5Ytyb8WBxD70UnYfuf4uf5866b+G3C7cxPX023iN5cs30wWys2h6xfWbqMsbnDGRI7grfSP+XxlFfYUL6em3OHVde8X7qY82OGMiv1bV5Of5823kxGbB/Et+VfVdekeVswLvEBXkv/iFfTV9Ir8kxG77iYz8rWBh2vT+QAPmiRVb1MT309aH2niB48mDSDRemf8XTqm1gsV20fQLktr8cuOfL883DTTXD77bB6NfTuDYMHw+bQrwXffgvnnhuoW70axo+HG26Al15yO+96svqL53l5yU0MOOV2bhm2mtbNe/P4K4PJK6jl/Kv8Yfin3DMqq3pJbnJi9boNWxbTtd1QfjPkbcZc/j7JiZn86+VB5OR9/158fuG1fL7pTa4c+DS3Dl9HZsuBTJk7gPyirTWOtearF/lu+4ckxKTXWFdeXkJmy4Gcc9rddW/Cj8Ux9l50ErZP7J7EkNiRXBE3ira+k7in2SOkeJozq3BKyPqP9q4gzdOCXyeM4fjwDLpGnsaIuBtYs/f96pqHkmcxIv56To7oygnhmdzbbAoxJo4le96orhkYfRE/jx5M6/C2tAlvx62J9xETFsfqvSuCjhdhIkj2plUvTTxNg9ZfGTeanpF9aRHemg4R3fh94r1s929jc8U39dglRyZNgpEjYdQoOOkkeOQRaN4cpoR+LZg6FdLTA3UnnRTYbsQImDAhdP2P3OKPJtHzZyPp1XEUqU1PYkj/R4iPac6ytbWcf5XY6BTiY9Kql7AwT/W64YNn0bfL9bRI6UpK00wuO3MKEb44PtsUeC+WVexh7YaXOP/0v9H2+J+T3KQt5/S6m6QmbVl+wHF3FWzilcU3MXzwc4SFhdeYR79uNzOg53gyjju9HrrRyI6x92KDh22ZLeOTslX0jRoYNH561EBWlS4PuU2PiD7k+LNYVPIa1lp2+XN5rXgOP486t/bjUMZeW0pCWGLI9X7r57WiOZRUFtEtonfQug9Ll9Jjcwr9t7RjXO4ocv07aj1OSWUxLxZNJ93Tkhbe1rXW/SiVlcGqVTAw+LVg4EBYHvq1YMWKmvWDBsHKlVB+dF3ZV/jL2LJjFZktg88ns+VANmbVcv5VJj3Xgz8+3pzJL53FV9+9c9Bav7+M8opSoiMC78XKygoqrZ9wT2RQXbg3Kuh2g7+ygmcWXMHZPe8ktelJh3NqR59j8L3Y4GGb58/Fj58kT2rQeLInlRx/dshtukX24uHk2YzJGUa7TT66f5eMxTIx6elajzMx705iwmIZEH1h0PjnZes4eVMsmZsiuGPndUxNeZn2vo7V6/tFncPE5Jk8m/Y2dzSdyMd7P2BY9pnstXuD9vNMwWRO3hTLyZtjWVyygFlpbxNhIg63HY0rNxf8fkgNfi1ITYXs0K8F2dmh6ysqAvs7ihTvyaXS+omLDj6fuOhUCkpCn398THMuPXMK15z/Er86fy4piZlMeeksvt7yv1qP8/qKO4nwxdKhTeC9GOmLo3XzXrz1wb3kF22lstLPys+eZWPWCgpKsqq3e2PFn4iObEafzr+ph7P9kTsG34teVwcymKCfLbbG2D5fla3nnl03cn2TuzgjahA7/Fncv+tWbt85mknJM2vUTy94mNmF/+KZtEXEhcUHrWsTnsn89DUUVObzRslL3JI7gtlpi8n0dQDggtjLq2vb+zrS0ded07e04p2S+ZwTc0n1uotih3F61Nns8GcxbfcEfpdzGS+mLSMqLLrOPWk05oC+W1tz7IfqQ40fLcyhvxdTmmaS0jSz+ufW6b3YVbCRdz6awAktzqhRv2T1wyxf9y9+c8kiIiO+fy8OG/QMcxb+inueaEGY8dAipRvdMq9gy46PANiwZQkfrp/BLcPW1McZHj2Oofdig4dtoicJD54aV7G5/h01rnb3mbz7fjpH9GR0wq0AnEQnopvF8MvsvtySeB/p3uOra6cXPMzEvDuZnrqALhE9a+zLZ3y0Dm8LBL7oWrv3Q54q+AcPJD0Z8tip3nTSvC3YWPFV0Hh8WALxYQlkhJ9I14jT6LI5kQUlL3FJ7PBDb0ZjS0oCj6fmlcOOHTWvGPZJSwtd7/VCs2YNM88GEhOVRJjxUFgcfD5FJTtqXO0eTMu0U1n9xZwa40tWP8yC5XfyfxcvoFVa8HsxqckJXH/ZEvaWF1NaVkBCTHOenj+UpvEZAGz47h0KirP407Tm1dtUWj+vLf0DS1Y/xN3XbjmcU/3xOwbfiw1+G8FnfHTwdWfpnoVB40v3LKR7ZO+Q25TaEsLwBI15qn622OqxJ3ZPYkLeHTyZOp9TIg/tC4NKKmvcItjfLn8u2yu2kuxpXmuNrfqv7CD7+VHy+QKPzSwMfi1YuDDwDW8ovXrBokU163v0gPCaX+D8mHk9PlqkdOeLzcHn/+XmhbRuXsv5h7AtZw3xMcHvj8UfTeL15Xcw6qL5tDnIl1cR4TEkxDSnpDSPzze9SYcTLgKgT+ffcutVa7ll2JrqJSEmnX5dx/DbS94+jLM8ShyD70UntxGuTRjL2JzhdI7oSY+IPswqnMoO/zaujLsOgLE5VwNU3yI4K+oCxu8cxbMFU6pvI/xl18108HXjOG9LAP61+0Em5t3BpORnaeNtV/0MbkRYFPFhCQA8sGsc/aPPI91zPEW2kHnFz/Fe6WKeSgk8a1tcWcRD+XczOHoIKZ7mbKnYyN/zx9PMk8Kg6F8AsLF8A2+UvESfyAE09SSTXbGFKbv/hs9EcFbU+S7aV7/GjoXhw6FnT+jTJ/AN77ZtcF3gteDqwGvBzKrbNdddB48+CjffDKNHw7JlMGMGzJ7dKNM/Uj/vNpZZbw6nVVpPMtL7sGztVHYXb6N3p8D5z3ozcP7DBgXOf8lHD9E0vjVpzU6morKMVZ89y7qvX+Ga879/3Oi/Kx/k9eV3MOycZ0lObEdB1ZVzuDeKqIjAe/HzjW9ibSUpTduTm7+Bee/eSkpiJqf+7BoA4qJTiItOCZprWFg48TFpQbcxCoqzKSzOJifvSwCyd61nz958msS3JCYy+CmaH71j7L3oJGzPjxlKnn8nj+bfS44/i3a+DjyV+jotvK0A2HbA87aXxo2kyBYys/BR7sv7PXFhCfSK7M+4xL9X1zxT8BjllHNDztCgbYfEjGBC8gwAcvzZjMm5ilx/NnFhCbT3dWJ66gL6RQ0CAlfLX5St4+WimRRU5pPsaU6vyP48mvwCsWFxAPhMBO+VLuaJ3RMpqMwnyZNKz8gzmJu2gmRvWkO1rOEMHQo7d8K990JWFnToAK+/Dq0Cr0WNZxwzMgLrx4wJPJKTng7//CcMGeJ+7vWga+ZQikt38tb791JQkkXzZh34v4tep2l84PwPfN62orKMee/ewu6irYR7o0htdjKjLprPzzK+fzJm6ceP4a8sZ+brwe/FU04awZWDZgCwp2w385eNJ79oC9ERTel84hDO7X0fHs/hXZEtXzuVN9+/p/rnaa+eB8AVZ0+n58kjD2tfje4Yey8aa+0PV1XpFNHDzktf2YDTOTZkfNvYMzj6/WNMY8/gp2HMPxp7Bj8BPXpgV678wW/o9G8jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURccDb2BM4Fn2b0dgzOPrlN2nsGYgcHl3Ziog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigLOwfaZgMn23ZJC5MZILtnXng9J3D1r/atFznLu1CydtiuaUzWncnHMVORXZ1etnF07jsqy+dNnclE6bmnBFdn8+LF0atI+ZBY9xztZOdNwUT8dN8VyS1Yv/lswPqrklZyQZG03Q8ottpwXVjMsdRb8tJ9B+UxTdNyczavtFbCj77Ag7cvjUwyP3Qu5kzlufwakfR3LlF935qKj2Hv5x00i6rjE1ll5rY4LqyivLmJz1R85bn0HPjyMY/GlLnsv5Z8h9LsibTdc1hhu/OT9o/NxPW4c81g3fnFdds6rof9z0zYUM/PQ4uq4xzNs5o+6N+LGYPBkyMiAyErp3h3cP/p5myZJAXWQktGkDU6e6mWc9cBK2/yl+nj/vuonfJtzO/PTVdIvozTXbB7O1YnPI+pWlyxibO5whsSN4K/1THk95hQ3l67k5d1h1zfulizk/ZiizUt/m5fT3aePNZMT2QXxb/lV1TZq3BeMSH+C19I94NX0lvSLPZPSOi/msbG3Q8fpEDuCDFlnVy/TU14PWd4rowYNJM1iU/hlPp76JxXLV9gGU2/J67NLBqYdH7s2853lwy038OvV2ZmeuplNMb67/ZjBZZaF7eGuLh1l4clbQ0sLXhoFNfhlUN27TFSwvfIO7jn+cV9p/wd9b/5t2kZ1q7G/L3m94aNutdI3pW2Pds5kfBh1ndruPMBjO3u9YJZVFtI3swK3HPUykiTrCbvwIPP883HQT3H47rF4NvXvD4MGwOfTrwbffwrnnBupWr4bx4+GGG+Cll9zOu46MtfaQiztF9LDz0lce9kEu3nYq7X2d+FvStOqx/ltOZHDMpdyWeH+N+sd3T+DpgkdYdvym6rF/F07n7l038GmropDHsNbS87vm/K7JHYyMv6HWuXTZ3JTbEu/nyrjRQOCqLK8ylydT/3PI5/NZ2VrO3daZRcd9zgnhmYe83ZFQD4PlNznsTRj+5amcGNmJP7b8vocXrj+RAU0u5cb0mj080JqiZVyz4XSmn7iMLjG9AVhR8Ba3bbyMeT/7mkRvUq3blttyfvXV6VyW9FtWFr5Dvj+Xf7apvV9PZN/H0zse5K0O24gKi66xvvfaWMYd9ygXNhv5g/M+mK6rj2jzI3PqqdCpE0z7/vXgxBPh0kvh/hCvxx/+AHPnwlffXwxw7bXw6aewYkXDz7c2PXpgV640P1TW4Fe2ZbaMT8pW0TdqYND46VEDWVW6POQ2PSL6kOPPYlHJa1hr2eXP5bXiOfw86tzaj0MZe20pCWGJIdf7rZ/XiuZQUllEt4jeQes+LF1Kj80p9N/SjnG5o8j176j1OCWVxbxYNJ10T0taeFvXWlef1MMjV15Zxmclq+gVH9zDXvED+bg4dA8PNHfnNE6IPLk6aAHe2f0KP4s+hWd3TGLQpy24cP2JPLDlRkr8wX/QHsu6g3Rfay5sOuIHj2Ot5ZVdT3Je06tCBu1PQlkZrFoFA4NfDwYOhOW1vB4rVtSsHzQIVq6EcnefkOrK29AHyPPn4sdPkic1aDzZk8oy/6KQ23SL7MXDybMZkzOMUruHCio4PfJsJiY9XetxJubdSUxYLAOiLwwa/7xsHUOyerHXlhJtYpma8jLtfR2r1/eLOodBMZdwvDeDLRUbmZh3J8Oyz2Re+ioiTER13TMFk/lb3m2U2GLaeDOZlfZ20PqGpB4euX3/v1A8AAASvElEQVQ9bOoN7mFTbyo7K0L3cH+F/t0s3P1vbmj+16DxrWXfsKZ4Kb6wCCa0folCfz4PbL2BnPJtTMh4EQhc/b6Z9zzPZ645pLm+V7iQrWXf8oum1x7i2R2FcnPB74fU4NeD1FRYVMvrkZ0NAwbUrK+oCOyvefOGmWs9cfYFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CXQBfFDuM/6auZk7aEjPB2/C7nMvZUltSlFXWmHh65mv2qvYf7e33Xs1RaP+clDg8ar6QSg+GvrZ6jY8yp9I4fxLjjHuXt3S+xs3w7eRW5/HHzSP7S8mnivaE/MRxo7s5pnBx9CpnRXQ71tI5e5oDeW1tz7IfqQ43/CDX4lW2iJwkPHnL82UHjuf4dNa7U9pm8+346R/RkdMKtAJxEJ6KbxfDL7L7ckngf6d7jq2unFzzMxLw7mZ66gC4RPWvsy2d8tA5vCwS+pFm790OeKvgHDyQ9GfLYqd500rwt2FjxVdB4fFgC8WEJZISfSNeI0+iyOZEFJS9xSezwkPupT+rhkdvXw50VwT3cVbGjxtVuKHN3TuOsJkNI8DYNGk/yNicl/DjiPAnVYxmRJwGQXb6ZPf5iciuyuO7r76/IKqkEoMcaLy+2/5TWkd/fs95VvoPFBa8yvsVjh3+SR5OkJPB4Aler+9uxo+bV7j5paaHrvV5o1qxh5lmPGvzK1md8dPB1Z+mehUHjS/cspHtk75DblNoSwvAEjXmqfrZ8/4XeE7snMSHvDp5Mnc8pkacf0nwqqWSv3Vvr+l3+XLZXbCXZU/tHElv1X9lB9lOf1MMjFx7m46To7rxXGNzD9woX0jkmdA/3WVf8Pl+WfswlTUfVWNclpg855duC7tFu2vslAM3DW3Fy9Cn8O3MdczLXVC/94i+ka0xf5mSu4ThfRtD+Xt01HZ+JYFCTy+t6qkcHny/wCNfC4NeDhQsDTxuE0qtXzVsMCxdCjx4QHt4w86xHDX5lC3BtwljG5gync0RPekT0YVbhVHb4t3Fl3HUAjM25GoBJyTMBOCvqAsbvHMWzBVM4I2oQO/xZ/GXXzXTwdeM4b0sA/rX7QSbm3cGk5Gdp421X/fxoRFgU8WGBq4wHdo2jf/R5pHuOp8gWMq/4Od4rXcxTKYGPt8WVRTyUfzeDo4eQ4mnOloqN/D1/PM08KQyK/gUAG8s38EbJS/SJHEBTTzLZFVuYsvtv+EwEZ0UFPyupHv64e3hV8lju3Dyck6N70iWmDy/mTiWnfBuXJgV6eOemQA/vbTUzaLu5O6fRMuJEusf2q7HPwYlXMm37X/jT5mu4Lu1uCv35PLj1JgYkXErT8BQA2kZ1CNomztMEPxU1xq21vLzzCQY1uZwYT1yNY5X4i/hu74aq2kqyyjfzRcka4r1Nae5rWceuNKKxY2H4cOjZE/r0CTwzu20bXBd4Pbg68Hows+r1uO46ePRRuPlmGD0ali2DGTNg9uxGmf7hchK258cMJc+/k0fz7yXHn0U7XweeSn2dFt5WAGw74FnRS+NGUmQLmVn4KPfl/Z64sAR6RfZnXOLfq2ueKXiMcsq5IWdo0LZDYkYwIXkGADn+bMbkXEWuP5u4sATa+zoxPXUB/aIGAYErvS/K1vFy0UwKKvNJ9jSnV2R/Hk1+gdiwwJvdZyJ4r3QxT+yeSEFlPkmeVHpGnsHctBUke9MaqmU1qIdHblDiUHb7d/JE9r3kVmTRNrIDj7R5nXRfoIfZIZ63LfYX8mb+HP4v9Y+YEPcFoz2xTD1hEQ9svYGrvjyFOG8i/RMu5sbmfzvs+a0sWsx3ZRv4a7NZIdevL1nJqK/7V/88NftPTM3+ExckjuDPrWYc9vEa3dChsHMn3HsvZGVBhw7w+uvQKvB61HjeNiMjsH7MGJgyBdLT4Z//hCFD3M+9Dpw8ZytS3+rynK3U1KjP2f5U/FiesxUREYWtiIgTClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERB7yHU7yuI2SsbKipHDvyEht7Bke/1hsbewYih0dXtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOHB0he3kyZCRAZGR0L07vPvuweuXLAnURUZCmzYwdaqbeTaQJ/ZOpnNBBmn5kfy8sDvLKw5+/v8ue46+BV1Iz48mc3ca/1d8Fdsrs4Nqpu59mJ4F7WmeH8XJu1twS8nvKLJF1esnld7PmYWn0DI/nra7k7m86ALW+z8J2sd9e+6iZ0F7jsuPofXuRC4qOov3K5YH1Zxf+HMS803Q8qviy4+wI3UzmclkkEEkkXSnO+9y8D4+xmOcxElEEUUmmcxkZq21s5mNwXA+5weN/4//cSEXchzHYTDMYEaNbecyl0EMIplkDIbFLK5Rk002wxlOGmnEEENnOjOLWYd03j9Kx9LvtLX2kBe6dz/04vpe5syxeL2Wxx+3rF9vuf56S0yMZdOm0PXffGOJjg7UrV8f2M7rtbz4YuOdQ9WS1+Twlyej51gvXvtQ1OP2vbj1dpTvehtDjF0bvylk/YLYpTaMMHtf5CS7Ju4b+1bsCtvJ09We4T2zuubx6FnWh89OiZ5pP4771r4a87ZtGdbaXuX7VXXNmd6B9tGop+yyuHV2adxae174xTbFpNpv4ndW10yNfsa+ErPIro772i6P+8QO9/3axhFnv4jPrq7p4+lnh/musZ/HZ1UvGxPy69SLvCZ1b/4cAn18nMftetbb6wn0cRObQtZPZrKNIcY+x3P2a762s5ltY4m185hXo/ZrvrbHcZztS197HucFrZvPfDue8fbf/NtGEWWnM73G9jOZae/mbjuTmRaw7/BOjZqzOdt2p7t9j/fs13xtJzDBGoxdwpI69aNRfxd+Kr/T3bvbQyk7rH02atj27Gm59trgsbZtLePGha6/7bbA+v3Hfv1ry2mnNe4LY+sWLt09Pe3VvmuDxtqEtbU3R4wLWf/nyAdtC9MyaOzRqKdsDDHVP1/r+53t7TkjqOa2iD/a9mEn1zqP7xIKbRhh9rmYebXWbErYbQH7YswbQWF7re93dQ7X+grbnvS013Jt0Fhb2tpxjAtZ34te9mZuDhoby1jbhz5BY2WU2Z70tDOYYUcwokbY7r/EEBMybPctOeTUGrYxxNineCporCUt7YM8WKd+NOrvwk/ld/oQw/bouI1QVgarVsHAgcHjAwfC8uWht1mxomb9oEGwciWUlzfMPBtImS1jjX8V/b3B59PfO5APKkKf/6nePmy3WSwofw1rLTsrc5lbPoezw8+trjnNezrr/Gv4sOI9AL6r3MyCinlBNQcqsoVUUkkTk1jrXJ/e+zhxxNPR0yVo3dzyOZywO4leBSdz155bKLSFh3T+9aWMMlaxioEE93EgA1lO6D7uZS+RRAaNRRHFB3xAOd+/j+7gDlrTmhGMqP+J7+d0TucFXmAnO6mkkld5lRxyGMCABj1uvTsGf6ePjrDNzQW/H1JTg8dTUyE7O/Q22dmh6ysqAvs7iuy0ufjxkxwWfD4pYanssKHPv6e3F09Ez2Z08TBSdvtoW5CMxTI5+unqmiG+y7kr6q+cV3QGyfnhdCpoxc/COnJP5AO1zmX8npvo6OlCT0+voPE3yv9Di/xY0nZHMmXvP3g5diEp+833Ut+VPB49i3mx73BL5F3MK3+Jq4svqUs76iyXQB9TCe5jKqlkE7qPgxjEUzzFh3yIxbKSlTzBE5RTTi6B99FbvMXzPM9UGv7+4Qu8gMGQRBIRRDCMYcxmNl3o8sMb/5gcg7/TR0fY7mNM8M/W1hz7ofpQ40cJQ/C8LbbG2D6f+9czbs+N3BJ5F+/EreLFmDfYXpnNmJLR1TXLKpYwofQvTIiazOK4j3gmei5LKxZzf+mfQu7zjj1jea9iKTOjX8JjPEHr+nr787+4NbwZu5yzws/hmuJfkl2ZVb1+ZMT/cVb4IE72dGSI73Kein6exRWL+Ljio7q2o84Op493cRfncR696U044VzERdVXrx485JLLSEbyNE+TSOir/fp0J3eSSy6LWMRKVnIrt3I1V/MxHzf4sRvEMfQ77W3sCRySpCTweGr+xduxo+Zfun3S0kLXe73QrFnDzLOBNDNJePCw44AnCXIqd5BsQp//P0rvp5u3JzdG3gpAB08nok0M5xb15c7K+2gRdjz37rmTIb4ruDriWgBO9nSkmGJuKrmW2yL/iNd8//a4fc8Y5pbNYV7sO7T2tKlxvBgTQxtPW9rQllO8p9G94ESeKXuCWyPvCjm/rp4eePDwdeVXdKZbnfpyuJII9PHAq9gd7KhxtbtPFFE8xVP8i3+xne00pzmP8zhxxJFEEv/jf2SRFfQxvpJKALx4+ZRPySSzXub/NV/zCI+whjV0pjMAnenMu7zLIzzCEzxRL8dx4hj8nT46rmx9vsDjHgsXBo8vXAi9e4feplcvWLSoZn2PHhAe3jDzbCA+46OLpzuLK4LPf3HFQnp6Q5//HkrwEHz1ue9niz1ozb71+4wruYkXy57j1dj/0s7T/pDmXEkle+3eWtd/Wrku8JE+rPkh7a8++PDRne4sJLiPC1lIb2p5H1UJJ5wWtMCDhznM4XzOJ4wwTuEU1rGONfv9dyEX0pe+rGENGWTU2/xLKAEI+ZrtC/ijxrH4O304X7o1+qNf4eGWadMCj33ceGPgMZGNGwPrhw8PLPvq9z0mctNNgfpp0wLbN/ZjIrbuj36FE24fjppm34tbb0f7brQxxNiP4zfavCbWDg0fboeGD6+ufyxquvXitROiJtvVcV/bBbFLbVdPD9vZ06265g8Rf7JxxNknomfbNXHf2Lkxb9mMsBPsBeGXVNf82vdbG0ecfTXm7aDHtr5LKKx+8uD3EXfYhbHv2bXxm+w7sSvtMN811ofPvhv3sc1rYu1HcRvs+Mh77H9jP7Qfx31rn4+Zb9uFtbedPF1tbkKF80e/wgm305hm17Pe3kigjxvZaC3WDme4Hc7w6vov+MLOZKb9ki/t+7xvhzLUNqWp/ZZvaz1GqKcRCim0q1ltV7PaRhFl7+Eeu5rVQY+c7WSnXc1q+w7vWMBOY5pdzWqbRZa1BJ54aEtb25e+9n3etxvYUP3oV6hH0Q5ladTfhZ/K7/RP7tEvay2PPWZp1cri81m6dbMsWfL9un79Asv+9YsXW7p2DdS3bm2ZMqVx51+11PVxpwejHrPHm1bWh8929nSz/4ldEvRoVR9Pv6D6B6L+aduH/cxGEWVTTZodEn6F/ST+u+r1OQnldlzk3bZNWFsbSaRNNy3sr3y/sd/G76quAUIuf4j4k81rYu3WhGJ7XvjFNs00tz58Ns00t4O9F9qFse9V72Nd/Gbb23OGTTRNrQ+fzQg7wY723Rj0rK6rsLVY+xiP2VYE+tiNbkHPqPajn+1Hv+qf17PedqGLjSLKxhNvL+Ii+zmfH3T/ocJ2X4AeuIxgRHXNdKaHrPkTf6qu+ZIv7SVcYlNIsdFE2050sjOYUedeNPbvwk/id/oQw9ZYa2tc7dbG9OhhWbmy/i6rj1F5Df89yk9ek/zGnsFPgzn0X3+pTY8e2JUrf/AbuqPjnq2IyFFOYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBY6099GJjcoBNDTcdEZGjTitrbfIPFR1W2IqISN3oNoKIiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIA/8PmfP94gQGKz8AAAAASUVORK5CYII=\n", 718 | "text/plain": [ 719 | "
" 720 | ] 721 | }, 722 | "metadata": {}, 723 | "output_type": "display_data" 724 | } 725 | ], 726 | "source": [ 727 | "policy_tpi, V_tpi = truncated_policy_iteration(env, max_it=2)\n", 728 | "\n", 729 | "# print the optimal policy\n", 730 | "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n", 731 | "print(policy_tpi,\"\\n\")\n", 732 | "\n", 733 | "# plot the optimal state-value function\n", 734 | "plot_values(V_tpi)" 735 | ] 736 | }, 737 | { 738 | "cell_type": "markdown", 739 | "metadata": {}, 740 | "source": [ 741 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 742 | "\n", 743 | "**Note:** In order to ensure accurate results, make sure that the `truncated_policy_iteration` function satisfies the requirements outlined above (with four inputs, two outputs, and with the default values of the input arguments unchanged)." 744 | ] 745 | }, 746 | { 747 | "cell_type": "code", 748 | "execution_count": 20, 749 | "metadata": {}, 750 | "outputs": [ 751 | { 752 | "data": { 753 | "text/markdown": [ 754 | "**PASSED**" 755 | ], 756 | "text/plain": [ 757 | "" 758 | ] 759 | }, 760 | "metadata": {}, 761 | "output_type": "display_data" 762 | } 763 | ], 764 | "source": [ 765 | "check_test.run_check('truncated_policy_iteration_check', truncated_policy_iteration)" 766 | ] 767 | }, 768 | { 769 | "cell_type": "markdown", 770 | "metadata": {}, 771 | "source": [ 772 | "### Part 6: Value Iteration\n", 773 | "\n", 774 | "In this section, you will write your own implementation of value iteration.\n", 775 | "\n", 776 | "Your algorithm should accept three arguments as input:\n", 777 | "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n", 778 | "- `gamma`: This is the discount rate. It must be a value between 0 and 1, inclusive (default value: `1`).\n", 779 | "- `theta`: This is a very small positive number that is used for the stopping criterion (default value: `1e-8`).\n", 780 | "\n", 781 | "The algorithm returns as **output**:\n", 782 | "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`). `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n", 783 | "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`). `V[s]` contains the estimated value of state `s`." 784 | ] 785 | }, 786 | { 787 | "cell_type": "code", 788 | "execution_count": 21, 789 | "metadata": {}, 790 | "outputs": [], 791 | "source": [ 792 | "def value_iteration(env, gamma=1, theta=1e-8):\n", 793 | " V = np.zeros(env.nS)\n", 794 | " while True:\n", 795 | " delta = 0\n", 796 | " for s in range(env.nS):\n", 797 | " v = V[s]\n", 798 | " V[s] = max(q_from_v(env, V, s, gamma))\n", 799 | " delta = max(delta,abs(V[s]-v))\n", 800 | " if delta < theta:\n", 801 | " break\n", 802 | " policy = policy_improvement(env, V, gamma)\n", 803 | " return policy, V" 804 | ] 805 | }, 806 | { 807 | "cell_type": "markdown", 808 | "metadata": {}, 809 | "source": [ 810 | "Use the next code cell to solve the MDP and visualize the output. The state-value function has been reshaped to match the shape of the gridworld." 811 | ] 812 | }, 813 | { 814 | "cell_type": "code", 815 | "execution_count": 22, 816 | "metadata": {}, 817 | "outputs": [ 818 | { 819 | "name": "stdout", 820 | "output_type": "stream", 821 | "text": [ 822 | "\n", 823 | "Optimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\n", 824 | "[[1. 0. 0. 0. ]\n", 825 | " [0. 0. 0. 1. ]\n", 826 | " [0. 0. 0. 1. ]\n", 827 | " [0. 0. 0. 1. ]\n", 828 | " [1. 0. 0. 0. ]\n", 829 | " [0.25 0.25 0.25 0.25]\n", 830 | " [0.5 0. 0.5 0. ]\n", 831 | " [0.25 0.25 0.25 0.25]\n", 832 | " [0. 0. 0. 1. ]\n", 833 | " [0. 1. 0. 0. ]\n", 834 | " [1. 0. 0. 0. ]\n", 835 | " [0.25 0.25 0.25 0.25]\n", 836 | " [0.25 0.25 0.25 0.25]\n", 837 | " [0. 0. 1. 0. ]\n", 838 | " [0. 1. 0. 0. ]\n", 839 | " [0.25 0.25 0.25 0.25]] \n", 840 | "\n" 841 | ] 842 | }, 843 | { 844 | "data": { 845 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAFoCAYAAAD5IVjuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Hl4VNX9x/H3yUwme0IgGxGBIBKs7CAKiBRFEPeKFhURbOWHbd2gasGl1VZrrUC1KlBRQRRBq6hYRAUrVBYXEATFDRUQSCCBhGyEJJPz+2NCZMgEISQnIp+Xz3365Nzvvffc70w+uXPnFmOtRUREGlZYY09ARORYoLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtvKjYozZaIwZ0NjzaAjGmGHGmLcaex7SOBS2RzljzOnGmOXGmN3GmF3GmGXGmFOq1o00xiw9jH21NsZYY4y3jnMZb4z5X4jxJGNMmTGmQ132Wx+MMTOq5lC03zK0AY9Xo5fW2lnW2oENdUz5cVPYHsWMMfHAf4BHgKbAccA9wN5GmtIzQG9jTMYB45cD66y1nzTCnPb3d2tt7H7L8408HzmGKGyPbu0ArLWzrbV+a+0ea+1b1tq1xpiTgKlAr6qruHwAY8x5xpjVxpgCY8x3xpi799vfvqvS/KptelVt8ytjzGfGmDxjzJvGmFahJmOt3QL8Fxh+wKqrgaer9nWCMea/xpidxphcY8wsY0yTUPuruhq9d7+ff26M2bLfz+nGmJeMMTnGmG+NMTcecueCj2ONMW1DHXffMY0xvzfG7DDGZBljrtmvNsoYM9EYs6nq08VSY0wUIXp54CcNY0xvY8yHVdt9aIzpvd+6xcaYv1R9Uik0xrxljEmqy/nJj4PC9uj2JeA3xjxtjBlsjEnct8Ja+xlwHbCi6ipuX6AVEwi/JsB5wG+MMRdXrTuj6n+bVG2zomrd7cAlQDLwLjD7IHN6mv3C1hiTCXTZbxsD3A+kAycBxwN3H+6JG2PCgNeAjwlc0Z8F3GyMGXS4+zoEaUBC1XF+DTy2X68nAN2B3gQ+XdwGVBKilwfMvykwH/gn0AyYBMw3xjTbr+xK4BogBfABt9T/qYkrCtujmLW2ADgdsMA0IMcYM88Yk3qQbRZba9dZayuttWsJhGC/gxxmNHC/tfYza20F8FegS21Xt8DLQOp+V2lXAwustTlVx99grV1ord1bNTbpB45fm1OAZGvtn621Zdbabwj04PKDbHOLMSa/ask9jGOVA3+21pZba18HioDMqsD/FXCTtXZr1aeL5dbaQ7mNcx7wlbX2GWtthbV2NvA5cMF+NdOttV9aa/cALxD4oyVHKYXtUa4qBEdaa1sAHQhcMT5UW70x5lRjzDtVH713E7j6PdjH01bAw/tCCthF4Or0OGPM7ft92TS1aj4lwL+Bq40xBhhG1S2EquOnGGPmGGO2GmMKgGd/4PgHm1f6fuGZT+AKvNY/NMAEa22TquVwjrmz6g/NPiVALIF5RwJfH+7kCbxOmw4Y20Tg6nmf7BDHlKOUwvYnxFr7OTCDQOhC4Ir3QM8B84DjrbUJBO7rmoPUfweM3i+kmlhro6qu4P6635dN1+23zdPAL4GzgTgCX+Ltc3/VcTpZa+OBq/Y7/oGKgej9fk47YF7fHjCvOGvtubXs62BKDnKcg8kFSoETQqz7oX9ObxuBPxj7awlsPcRjy1FGYXsUM8a0r/ripkXVz8cDVwDvVZVsB1oYY3z7bRYH7LLWlhpjehK4L7hPDoH7jW32G5sKjDfGnFx1jARjzGU/MLV3gXzgcWCOtbbsgOMXEfji6Djg1oPsZw1wrjGmqTEmDbh5v3UfAAXGmD9UfUnlMcZ0MFWPvR2mNcCVVfs4h0O8rWGtrQSeAiZVfVnnqfoiLILQvdzf60A7Y8yVxhivCTyG9jOC/zDJT4jC9uhWCJwKvG+MKSYQsp8Av69a/1/gUyB7v3uUvwX+bIwpBP5I4F4gUH0L4D5gWdVH89OstS8DDwBzqj72fwIMPtikbOAfSZ5J4Mpt5gGr7wG6AbsJfEE09yC7eobAF2AbgbeA6ke1rLV+Avc3uwDfErjKfILAF1mH66aqfeUTuO3xymFsewuwDviQwC2WB4CwUL3cfyNr7U7gfAKv1U4CX6ydb609nHvJchQx+sfDRUQanq5sRUQcUNiKiDigsBURcUBhKyLigMJWRMSBw/qn9Jp6kmwLb+sGmoqIyNFnS8VGdvlza/s/5lQ7rLBt4W3NvPSVdZ+ViMhPzIXbehxSnW4jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDjgLGyfKZhM3y0ZZG6M5IJt3fmg9N2D1r9a9Bznbu3CSZuiOWVzGjfnXEVORXb1+tmF07gsqy9dNjel06YmXJHdnw9LlwbtY2bBY5yztRMdN8XTcVM8l2T14r8l84NqbskZScZGE7T8YttpQTXjckfRb8sJtN8URffNyYzafhEbyj47wo4cPvWwfqiPR049PHxOwvY/xc/z51038duE25mfvppuEb25ZvtgtlZsDlm/snQZY3OHMyR2BG+lf8rjKa+woXw9N+cOq655v3Qx58cMZVbq27yc/j5tvJmM2D6Ib8u/qq5J87ZgXOIDvJb+Ea+mr6RX5JmM3nExn5WtDTpen8gBfNAiq3qZnvp60PpOET14MGkGi9I/4+nUN7FYrto+gHJbXo9dOjj1sH6oj0dOPawbY6095OJOET3svPSVh32Qi7edSntfJ/6WNK16rP+WExkccym3Jd5fo/7x3RN4uuARlh2/qXrs34XTuXvXDXzaqijkMay19PyuOb9rcgcj42+odS5dNjfltsT7uTJuNBD4S5hXmcuTqf855PP5rGwt527rzKLjPueE8MxD3u5IqIf1Q308cuphsAu39WDt3pXmh+oa/Mq2zJbxSdkq+kYNDBo/PWogq0qXh9ymR0QfcvxZLCp5DWstu/y5vFY8h59HnVv7cShjry0lISwx5Hq/9fNa0RxKKovoFtE7aN2HpUvpsTmF/lvaMS53FLn+HbUep6SymBeLppPuaUkLb+ta6+qTelg/1Mcjpx7WnbdB9w7k+XPx4yfJkxo0nuxJZZl/UchtukX24uHk2YzJGUap3UMFFZweeTYTk56u9TgT8+4kJiyWAdEXBo1/XraOIVm92GtLiTaxTE15mfa+jtXr+0Wdw6CYSzjem8GWio1MzLuTYdlnMi99FREmorrumYLJ/C3vNkpsMW28mcxKeztofUNSD+uH+njk1MO6c/YFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CN94tih/Gf9NXMSVtCRng7fpdzGXsqS+rSijpTD+uH+njk1MPD1+BXtomeJDx4yPFnB43n+nfU+Ou4z+Td99M5oiejE24F4CQ6Ed0shl9m9+WWxPtI9x5fXTu94GEm5t3J9NQFdInoWWNfPuOjdXhbIHBjfO3eD3mq4B88kPRkyGOnetNJ87ZgY8VXQePxYQnEhyWQEX4iXSNOo8vmRBaUvMQlscMPvRl1pB7WD/XxyKmHddfgV7Y+46ODrztL9ywMGl+6ZyHdI3uH3KbUlhCGJ2jMU/Wz5fsv9J7YPYkJeXfwZOp8Tok8/ZDmU0kle+3eWtfv8ueyvWIryZ7mtdbYqv/KDrKf+qQe1g/18ciph3XX4Fe2ANcmjGVsznA6R/SkR0QfZhVOZYd/G1fGXQfA2JyrAZiUPBOAs6IuYPzOUTxbMIUzogaxw5/FX3bdTAdfN47ztgTgX7sfZGLeHUxKfpY23nbVz+xFhEURH5YAwAO7xtE/+jzSPcdTZAuZV/wc75Uu5qmUwEeK4soiHsq/m8HRQ0jxNGdLxUb+nj+eZp4UBkX/AoCN5Rt4o+Ql+kQOoKknmeyKLUzZ/Td8JoKzos530T71sB6pj+phY/XQSdieHzOUPP9OHs2/lxx/Fu18HXgq9XVaeFsBsO2A5/MujRtJkS1kZuGj3Jf3e+LCEugV2Z9xiX+vrnmm4DHKKeeGnKFB2w6JGcGE5BkA5PizGZNzFbn+bOLCEmjv68T01AX0ixoEBP66flG2jpeLZlJQmU+ypzm9IvvzaPILxIbFAeAzEbxXupgndk+koDKfJE8qPSPPYG7aCpK9aQ3VshrUw/qhPh459bBunDxnKyLyU/Wjec5WREQUtiIiTihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBzwNvYEjkUZ3zb2DI5+/xjT2DP4aRjzj8aewU9Aj0Mr05WtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOOAvbZwom03dLBpkbI7lgW3c+KH33oPWvFj3HuVu7cNKmaE7ZnMbNOVeRU5FdvX524TQuy+pLl81N6bSpCVdk9+fD0qVB+5hZ8BjnbO1Ex03xdNwUzyVZvfhvyfygmltyRpKx0QQtv9h2WlDNuNxR9NtyAu03RdF9czKjtl/EhrLPjrAjjWjyZMjIgMhI6N4d3j34a8GSJYG6yEho0wamTnUzzway9OPJ/OWpDG59JJKJz3Xn6621n/+G7xYz5iFTY9m+6/PqmhXrpvHPF/pyx5SmjJ/chMde7M83W4Pfi6Vlhby8+Gb+/GQrbnskioef783m7A9rPe7zi/6PMQ8Z3lk1IWh8+brHeezF/oyf3IQxDxl27d5Ytyb8WBxD70UnYfuf4uf5866b+G3C7cxPX023iN5cs30wWys2h6xfWbqMsbnDGRI7grfSP+XxlFfYUL6em3OHVde8X7qY82OGMiv1bV5Of5823kxGbB/Et+VfVdekeVswLvEBXkv/iFfTV9Ir8kxG77iYz8rWBh2vT+QAPmiRVb1MT309aH2niB48mDSDRemf8XTqm1gsV20fQLktr8cuOfL883DTTXD77bB6NfTuDYMHw+bQrwXffgvnnhuoW70axo+HG26Al15yO+96svqL53l5yU0MOOV2bhm2mtbNe/P4K4PJK6jl/Kv8Yfin3DMqq3pJbnJi9boNWxbTtd1QfjPkbcZc/j7JiZn86+VB5OR9/158fuG1fL7pTa4c+DS3Dl9HZsuBTJk7gPyirTWOtearF/lu+4ckxKTXWFdeXkJmy4Gcc9rddW/Cj8Ux9l50ErZP7J7EkNiRXBE3ira+k7in2SOkeJozq3BKyPqP9q4gzdOCXyeM4fjwDLpGnsaIuBtYs/f96pqHkmcxIv56To7oygnhmdzbbAoxJo4le96orhkYfRE/jx5M6/C2tAlvx62J9xETFsfqvSuCjhdhIkj2plUvTTxNg9ZfGTeanpF9aRHemg4R3fh94r1s929jc8U39dglRyZNgpEjYdQoOOkkeOQRaN4cpoR+LZg6FdLTA3UnnRTYbsQImDAhdP2P3OKPJtHzZyPp1XEUqU1PYkj/R4iPac6ytbWcf5XY6BTiY9Kql7AwT/W64YNn0bfL9bRI6UpK00wuO3MKEb44PtsUeC+WVexh7YaXOP/0v9H2+J+T3KQt5/S6m6QmbVl+wHF3FWzilcU3MXzwc4SFhdeYR79uNzOg53gyjju9HrrRyI6x92KDh22ZLeOTslX0jRoYNH561EBWlS4PuU2PiD7k+LNYVPIa1lp2+XN5rXgOP486t/bjUMZeW0pCWGLI9X7r57WiOZRUFtEtonfQug9Ll9Jjcwr9t7RjXO4ocv07aj1OSWUxLxZNJ93Tkhbe1rXW/SiVlcGqVTAw+LVg4EBYHvq1YMWKmvWDBsHKlVB+dF3ZV/jL2LJjFZktg88ns+VANmbVcv5VJj3Xgz8+3pzJL53FV9+9c9Bav7+M8opSoiMC78XKygoqrZ9wT2RQXbg3Kuh2g7+ygmcWXMHZPe8ktelJh3NqR59j8L3Y4GGb58/Fj58kT2rQeLInlRx/dshtukX24uHk2YzJGUa7TT66f5eMxTIx6elajzMx705iwmIZEH1h0PjnZes4eVMsmZsiuGPndUxNeZn2vo7V6/tFncPE5Jk8m/Y2dzSdyMd7P2BY9pnstXuD9vNMwWRO3hTLyZtjWVyygFlpbxNhIg63HY0rNxf8fkgNfi1ITYXs0K8F2dmh6ysqAvs7ihTvyaXS+omLDj6fuOhUCkpCn398THMuPXMK15z/Er86fy4piZlMeeksvt7yv1qP8/qKO4nwxdKhTeC9GOmLo3XzXrz1wb3kF22lstLPys+eZWPWCgpKsqq3e2PFn4iObEafzr+ph7P9kTsG34teVwcymKCfLbbG2D5fla3nnl03cn2TuzgjahA7/Fncv+tWbt85mknJM2vUTy94mNmF/+KZtEXEhcUHrWsTnsn89DUUVObzRslL3JI7gtlpi8n0dQDggtjLq2vb+zrS0ded07e04p2S+ZwTc0n1uotih3F61Nns8GcxbfcEfpdzGS+mLSMqLLrOPWk05oC+W1tz7IfqQ40fLcyhvxdTmmaS0jSz+ufW6b3YVbCRdz6awAktzqhRv2T1wyxf9y9+c8kiIiO+fy8OG/QMcxb+inueaEGY8dAipRvdMq9gy46PANiwZQkfrp/BLcPW1McZHj2Oofdig4dtoicJD54aV7G5/h01rnb3mbz7fjpH9GR0wq0AnEQnopvF8MvsvtySeB/p3uOra6cXPMzEvDuZnrqALhE9a+zLZ3y0Dm8LBL7oWrv3Q54q+AcPJD0Z8tip3nTSvC3YWPFV0Hh8WALxYQlkhJ9I14jT6LI5kQUlL3FJ7PBDb0ZjS0oCj6fmlcOOHTWvGPZJSwtd7/VCs2YNM88GEhOVRJjxUFgcfD5FJTtqXO0eTMu0U1n9xZwa40tWP8yC5XfyfxcvoFVa8HsxqckJXH/ZEvaWF1NaVkBCTHOenj+UpvEZAGz47h0KirP407Tm1dtUWj+vLf0DS1Y/xN3XbjmcU/3xOwbfiw1+G8FnfHTwdWfpnoVB40v3LKR7ZO+Q25TaEsLwBI15qn622OqxJ3ZPYkLeHTyZOp9TIg/tC4NKKmvcItjfLn8u2yu2kuxpXmuNrfqv7CD7+VHy+QKPzSwMfi1YuDDwDW8ovXrBokU163v0gPCaX+D8mHk9PlqkdOeLzcHn/+XmhbRuXsv5h7AtZw3xMcHvj8UfTeL15Xcw6qL5tDnIl1cR4TEkxDSnpDSPzze9SYcTLgKgT+ffcutVa7ll2JrqJSEmnX5dx/DbS94+jLM8ShyD70UntxGuTRjL2JzhdI7oSY+IPswqnMoO/zaujLsOgLE5VwNU3yI4K+oCxu8cxbMFU6pvI/xl18108HXjOG9LAP61+0Em5t3BpORnaeNtV/0MbkRYFPFhCQA8sGsc/aPPI91zPEW2kHnFz/Fe6WKeSgk8a1tcWcRD+XczOHoIKZ7mbKnYyN/zx9PMk8Kg6F8AsLF8A2+UvESfyAE09SSTXbGFKbv/hs9EcFbU+S7aV7/GjoXhw6FnT+jTJ/AN77ZtcF3gteDqwGvBzKrbNdddB48+CjffDKNHw7JlMGMGzJ7dKNM/Uj/vNpZZbw6nVVpPMtL7sGztVHYXb6N3p8D5z3ozcP7DBgXOf8lHD9E0vjVpzU6morKMVZ89y7qvX+Ga879/3Oi/Kx/k9eV3MOycZ0lObEdB1ZVzuDeKqIjAe/HzjW9ibSUpTduTm7+Bee/eSkpiJqf+7BoA4qJTiItOCZprWFg48TFpQbcxCoqzKSzOJifvSwCyd61nz958msS3JCYy+CmaH71j7L3oJGzPjxlKnn8nj+bfS44/i3a+DjyV+jotvK0A2HbA87aXxo2kyBYys/BR7sv7PXFhCfSK7M+4xL9X1zxT8BjllHNDztCgbYfEjGBC8gwAcvzZjMm5ilx/NnFhCbT3dWJ66gL6RQ0CAlfLX5St4+WimRRU5pPsaU6vyP48mvwCsWFxAPhMBO+VLuaJ3RMpqMwnyZNKz8gzmJu2gmRvWkO1rOEMHQo7d8K990JWFnToAK+/Dq0Cr0WNZxwzMgLrx4wJPJKTng7//CcMGeJ+7vWga+ZQikt38tb791JQkkXzZh34v4tep2l84PwPfN62orKMee/ewu6irYR7o0htdjKjLprPzzK+fzJm6ceP4a8sZ+brwe/FU04awZWDZgCwp2w385eNJ79oC9ERTel84hDO7X0fHs/hXZEtXzuVN9+/p/rnaa+eB8AVZ0+n58kjD2tfje4Yey8aa+0PV1XpFNHDzktf2YDTOTZkfNvYMzj6/WNMY8/gp2HMPxp7Bj8BPXpgV678wW/o9G8jiIg4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURccDb2BM4Fn2b0dgzOPrlN2nsGYgcHl3Ziog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigLOwfaZgMn23ZJC5MZILtnXng9J3D1r/atFznLu1CydtiuaUzWncnHMVORXZ1etnF07jsqy+dNnclE6bmnBFdn8+LF0atI+ZBY9xztZOdNwUT8dN8VyS1Yv/lswPqrklZyQZG03Q8ottpwXVjMsdRb8tJ9B+UxTdNyczavtFbCj77Ag7cvjUwyP3Qu5kzlufwakfR3LlF935qKj2Hv5x00i6rjE1ll5rY4LqyivLmJz1R85bn0HPjyMY/GlLnsv5Z8h9LsibTdc1hhu/OT9o/NxPW4c81g3fnFdds6rof9z0zYUM/PQ4uq4xzNs5o+6N+LGYPBkyMiAyErp3h3cP/p5myZJAXWQktGkDU6e6mWc9cBK2/yl+nj/vuonfJtzO/PTVdIvozTXbB7O1YnPI+pWlyxibO5whsSN4K/1THk95hQ3l67k5d1h1zfulizk/ZiizUt/m5fT3aePNZMT2QXxb/lV1TZq3BeMSH+C19I94NX0lvSLPZPSOi/msbG3Q8fpEDuCDFlnVy/TU14PWd4rowYNJM1iU/hlPp76JxXLV9gGU2/J67NLBqYdH7s2853lwy038OvV2ZmeuplNMb67/ZjBZZaF7eGuLh1l4clbQ0sLXhoFNfhlUN27TFSwvfIO7jn+cV9p/wd9b/5t2kZ1q7G/L3m94aNutdI3pW2Pds5kfBh1ndruPMBjO3u9YJZVFtI3swK3HPUykiTrCbvwIPP883HQT3H47rF4NvXvD4MGwOfTrwbffwrnnBupWr4bx4+GGG+Cll9zOu46MtfaQiztF9LDz0lce9kEu3nYq7X2d+FvStOqx/ltOZHDMpdyWeH+N+sd3T+DpgkdYdvym6rF/F07n7l038GmropDHsNbS87vm/K7JHYyMv6HWuXTZ3JTbEu/nyrjRQOCqLK8ylydT/3PI5/NZ2VrO3daZRcd9zgnhmYe83ZFQD4PlNznsTRj+5amcGNmJP7b8vocXrj+RAU0u5cb0mj080JqiZVyz4XSmn7iMLjG9AVhR8Ba3bbyMeT/7mkRvUq3blttyfvXV6VyW9FtWFr5Dvj+Xf7apvV9PZN/H0zse5K0O24gKi66xvvfaWMYd9ygXNhv5g/M+mK6rj2jzI3PqqdCpE0z7/vXgxBPh0kvh/hCvxx/+AHPnwlffXwxw7bXw6aewYkXDz7c2PXpgV640P1TW4Fe2ZbaMT8pW0TdqYND46VEDWVW6POQ2PSL6kOPPYlHJa1hr2eXP5bXiOfw86tzaj0MZe20pCWGJIdf7rZ/XiuZQUllEt4jeQes+LF1Kj80p9N/SjnG5o8j176j1OCWVxbxYNJ10T0taeFvXWlef1MMjV15Zxmclq+gVH9zDXvED+bg4dA8PNHfnNE6IPLk6aAHe2f0KP4s+hWd3TGLQpy24cP2JPLDlRkr8wX/QHsu6g3Rfay5sOuIHj2Ot5ZVdT3Je06tCBu1PQlkZrFoFA4NfDwYOhOW1vB4rVtSsHzQIVq6EcnefkOrK29AHyPPn4sdPkic1aDzZk8oy/6KQ23SL7MXDybMZkzOMUruHCio4PfJsJiY9XetxJubdSUxYLAOiLwwa/7xsHUOyerHXlhJtYpma8jLtfR2r1/eLOodBMZdwvDeDLRUbmZh3J8Oyz2Re+ioiTER13TMFk/lb3m2U2GLaeDOZlfZ20PqGpB4euX3/v1A8AAASvElEQVQ9bOoN7mFTbyo7K0L3cH+F/t0s3P1vbmj+16DxrWXfsKZ4Kb6wCCa0folCfz4PbL2BnPJtTMh4EQhc/b6Z9zzPZ645pLm+V7iQrWXf8oum1x7i2R2FcnPB74fU4NeD1FRYVMvrkZ0NAwbUrK+oCOyvefOGmWs9cfYFmSH4Kttia4zt81XZeu7ZdSPXN7mLeemrmJH6Bjn+bG7fOTpk/fSCh5ld+C+mpMwlLiw+aF2b8Ezmp69hbvP3uCr+N9ySO4Ivyj6pXn9B7OWcHX0h7X0dGRB9ATNSF/BN+Re8c8CXQBfFDuM/6auZk7aEjPB2/C7nMvZUltSlFXWmHh65mv2qvYf7e33Xs1RaP+clDg8ar6QSg+GvrZ6jY8yp9I4fxLjjHuXt3S+xs3w7eRW5/HHzSP7S8mnivaE/MRxo7s5pnBx9CpnRXQ71tI5e5oDeW1tz7IfqQ43/CDX4lW2iJwkPHnL82UHjuf4dNa7U9pm8+346R/RkdMKtAJxEJ6KbxfDL7L7ckngf6d7jq2unFzzMxLw7mZ66gC4RPWvsy2d8tA5vCwS+pFm790OeKvgHDyQ9GfLYqd500rwt2FjxVdB4fFgC8WEJZISfSNeI0+iyOZEFJS9xSezwkPupT+rhkdvXw50VwT3cVbGjxtVuKHN3TuOsJkNI8DYNGk/yNicl/DjiPAnVYxmRJwGQXb6ZPf5iciuyuO7r76/IKqkEoMcaLy+2/5TWkd/fs95VvoPFBa8yvsVjh3+SR5OkJPB4Aler+9uxo+bV7j5paaHrvV5o1qxh5lmPGvzK1md8dPB1Z+mehUHjS/cspHtk75DblNoSwvAEjXmqfrZ8/4XeE7snMSHvDp5Mnc8pkacf0nwqqWSv3Vvr+l3+XLZXbCXZU/tHElv1X9lB9lOf1MMjFx7m46To7rxXGNzD9woX0jkmdA/3WVf8Pl+WfswlTUfVWNclpg855duC7tFu2vslAM3DW3Fy9Cn8O3MdczLXVC/94i+ka0xf5mSu4ThfRtD+Xt01HZ+JYFCTy+t6qkcHny/wCNfC4NeDhQsDTxuE0qtXzVsMCxdCjx4QHt4w86xHDX5lC3BtwljG5gync0RPekT0YVbhVHb4t3Fl3HUAjM25GoBJyTMBOCvqAsbvHMWzBVM4I2oQO/xZ/GXXzXTwdeM4b0sA/rX7QSbm3cGk5Gdp421X/fxoRFgU8WGBq4wHdo2jf/R5pHuOp8gWMq/4Od4rXcxTKYGPt8WVRTyUfzeDo4eQ4mnOloqN/D1/PM08KQyK/gUAG8s38EbJS/SJHEBTTzLZFVuYsvtv+EwEZ0UFPyupHv64e3hV8lju3Dyck6N70iWmDy/mTiWnfBuXJgV6eOemQA/vbTUzaLu5O6fRMuJEusf2q7HPwYlXMm37X/jT5mu4Lu1uCv35PLj1JgYkXErT8BQA2kZ1CNomztMEPxU1xq21vLzzCQY1uZwYT1yNY5X4i/hu74aq2kqyyjfzRcka4r1Nae5rWceuNKKxY2H4cOjZE/r0CTwzu20bXBd4Pbg68Hows+r1uO46ePRRuPlmGD0ali2DGTNg9uxGmf7hchK258cMJc+/k0fz7yXHn0U7XweeSn2dFt5WAGw74FnRS+NGUmQLmVn4KPfl/Z64sAR6RfZnXOLfq2ueKXiMcsq5IWdo0LZDYkYwIXkGADn+bMbkXEWuP5u4sATa+zoxPXUB/aIGAYErvS/K1vFy0UwKKvNJ9jSnV2R/Hk1+gdiwwJvdZyJ4r3QxT+yeSEFlPkmeVHpGnsHctBUke9MaqmU1qIdHblDiUHb7d/JE9r3kVmTRNrIDj7R5nXRfoIfZIZ63LfYX8mb+HP4v9Y+YEPcFoz2xTD1hEQ9svYGrvjyFOG8i/RMu5sbmfzvs+a0sWsx3ZRv4a7NZIdevL1nJqK/7V/88NftPTM3+ExckjuDPrWYc9vEa3dChsHMn3HsvZGVBhw7w+uvQKvB61HjeNiMjsH7MGJgyBdLT4Z//hCFD3M+9Dpw8ZytS3+rynK3U1KjP2f5U/FiesxUREYWtiIgTClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERB7yHU7yuI2SsbKipHDvyEht7Bke/1hsbewYih0dXtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOHB0he3kyZCRAZGR0L07vPvuweuXLAnURUZCmzYwdaqbeTaQJ/ZOpnNBBmn5kfy8sDvLKw5+/v8ue46+BV1Iz48mc3ca/1d8Fdsrs4Nqpu59mJ4F7WmeH8XJu1twS8nvKLJF1esnld7PmYWn0DI/nra7k7m86ALW+z8J2sd9e+6iZ0F7jsuPofXuRC4qOov3K5YH1Zxf+HMS803Q8qviy4+wI3UzmclkkEEkkXSnO+9y8D4+xmOcxElEEUUmmcxkZq21s5mNwXA+5weN/4//cSEXchzHYTDMYEaNbecyl0EMIplkDIbFLK5Rk002wxlOGmnEEENnOjOLWYd03j9Kx9LvtLX2kBe6dz/04vpe5syxeL2Wxx+3rF9vuf56S0yMZdOm0PXffGOJjg7UrV8f2M7rtbz4YuOdQ9WS1+Twlyej51gvXvtQ1OP2vbj1dpTvehtDjF0bvylk/YLYpTaMMHtf5CS7Ju4b+1bsCtvJ09We4T2zuubx6FnWh89OiZ5pP4771r4a87ZtGdbaXuX7VXXNmd6B9tGop+yyuHV2adxae174xTbFpNpv4ndW10yNfsa+ErPIro772i6P+8QO9/3axhFnv4jPrq7p4+lnh/musZ/HZ1UvGxPy69SLvCZ1b/4cAn18nMftetbb6wn0cRObQtZPZrKNIcY+x3P2a762s5ltY4m185hXo/ZrvrbHcZztS197HucFrZvPfDue8fbf/NtGEWWnM73G9jOZae/mbjuTmRaw7/BOjZqzOdt2p7t9j/fs13xtJzDBGoxdwpI69aNRfxd+Kr/T3bvbQyk7rH02atj27Gm59trgsbZtLePGha6/7bbA+v3Hfv1ry2mnNe4LY+sWLt09Pe3VvmuDxtqEtbU3R4wLWf/nyAdtC9MyaOzRqKdsDDHVP1/r+53t7TkjqOa2iD/a9mEn1zqP7xIKbRhh9rmYebXWbErYbQH7YswbQWF7re93dQ7X+grbnvS013Jt0Fhb2tpxjAtZ34te9mZuDhoby1jbhz5BY2WU2Z70tDOYYUcwokbY7r/EEBMybPctOeTUGrYxxNineCporCUt7YM8WKd+NOrvwk/ld/oQw/bouI1QVgarVsHAgcHjAwfC8uWht1mxomb9oEGwciWUlzfMPBtImS1jjX8V/b3B59PfO5APKkKf/6nePmy3WSwofw1rLTsrc5lbPoezw8+trjnNezrr/Gv4sOI9AL6r3MyCinlBNQcqsoVUUkkTk1jrXJ/e+zhxxNPR0yVo3dzyOZywO4leBSdz155bKLSFh3T+9aWMMlaxioEE93EgA1lO6D7uZS+RRAaNRRHFB3xAOd+/j+7gDlrTmhGMqP+J7+d0TucFXmAnO6mkkld5lRxyGMCABj1uvTsGf6ePjrDNzQW/H1JTg8dTUyE7O/Q22dmh6ysqAvs7iuy0ufjxkxwWfD4pYanssKHPv6e3F09Ez2Z08TBSdvtoW5CMxTI5+unqmiG+y7kr6q+cV3QGyfnhdCpoxc/COnJP5AO1zmX8npvo6OlCT0+voPE3yv9Di/xY0nZHMmXvP3g5diEp+833Ut+VPB49i3mx73BL5F3MK3+Jq4svqUs76iyXQB9TCe5jKqlkE7qPgxjEUzzFh3yIxbKSlTzBE5RTTi6B99FbvMXzPM9UGv7+4Qu8gMGQRBIRRDCMYcxmNl3o8sMb/5gcg7/TR0fY7mNM8M/W1hz7ofpQ40cJQ/C8LbbG2D6f+9czbs+N3BJ5F+/EreLFmDfYXpnNmJLR1TXLKpYwofQvTIiazOK4j3gmei5LKxZzf+mfQu7zjj1jea9iKTOjX8JjPEHr+nr787+4NbwZu5yzws/hmuJfkl2ZVb1+ZMT/cVb4IE72dGSI73Kein6exRWL+Ljio7q2o84Op493cRfncR696U044VzERdVXrx485JLLSEbyNE+TSOir/fp0J3eSSy6LWMRKVnIrt3I1V/MxHzf4sRvEMfQ77W3sCRySpCTweGr+xduxo+Zfun3S0kLXe73QrFnDzLOBNDNJePCw44AnCXIqd5BsQp//P0rvp5u3JzdG3gpAB08nok0M5xb15c7K+2gRdjz37rmTIb4ruDriWgBO9nSkmGJuKrmW2yL/iNd8//a4fc8Y5pbNYV7sO7T2tKlxvBgTQxtPW9rQllO8p9G94ESeKXuCWyPvCjm/rp4eePDwdeVXdKZbnfpyuJII9PHAq9gd7KhxtbtPFFE8xVP8i3+xne00pzmP8zhxxJFEEv/jf2SRFfQxvpJKALx4+ZRPySSzXub/NV/zCI+whjV0pjMAnenMu7zLIzzCEzxRL8dx4hj8nT46rmx9vsDjHgsXBo8vXAi9e4feplcvWLSoZn2PHhAe3jDzbCA+46OLpzuLK4LPf3HFQnp6Q5//HkrwEHz1ue9niz1ozb71+4wruYkXy57j1dj/0s7T/pDmXEkle+3eWtd/Wrku8JE+rPkh7a8++PDRne4sJLiPC1lIb2p5H1UJJ5wWtMCDhznM4XzOJ4wwTuEU1rGONfv9dyEX0pe+rGENGWTU2/xLKAEI+ZrtC/ijxrH4O304X7o1+qNf4eGWadMCj33ceGPgMZGNGwPrhw8PLPvq9z0mctNNgfpp0wLbN/ZjIrbuj36FE24fjppm34tbb0f7brQxxNiP4zfavCbWDg0fboeGD6+ufyxquvXitROiJtvVcV/bBbFLbVdPD9vZ06265g8Rf7JxxNknomfbNXHf2Lkxb9mMsBPsBeGXVNf82vdbG0ecfTXm7aDHtr5LKKx+8uD3EXfYhbHv2bXxm+w7sSvtMN811ofPvhv3sc1rYu1HcRvs+Mh77H9jP7Qfx31rn4+Zb9uFtbedPF1tbkKF80e/wgm305hm17Pe3kigjxvZaC3WDme4Hc7w6vov+MLOZKb9ki/t+7xvhzLUNqWp/ZZvaz1GqKcRCim0q1ltV7PaRhFl7+Eeu5rVQY+c7WSnXc1q+w7vWMBOY5pdzWqbRZa1BJ54aEtb25e+9n3etxvYUP3oV6hH0Q5ladTfhZ/K7/RP7tEvay2PPWZp1cri81m6dbMsWfL9un79Asv+9YsXW7p2DdS3bm2ZMqVx51+11PVxpwejHrPHm1bWh8929nSz/4ldEvRoVR9Pv6D6B6L+aduH/cxGEWVTTZodEn6F/ST+u+r1OQnldlzk3bZNWFsbSaRNNy3sr3y/sd/G76quAUIuf4j4k81rYu3WhGJ7XvjFNs00tz58Ns00t4O9F9qFse9V72Nd/Gbb23OGTTRNrQ+fzQg7wY723Rj0rK6rsLVY+xiP2VYE+tiNbkHPqPajn+1Hv+qf17PedqGLjSLKxhNvL+Ii+zmfH3T/ocJ2X4AeuIxgRHXNdKaHrPkTf6qu+ZIv7SVcYlNIsdFE2050sjOYUedeNPbvwk/id/oQw9ZYa2tc7dbG9OhhWbmy/i6rj1F5Df89yk9ek/zGnsFPgzn0X3+pTY8e2JUrf/AbuqPjnq2IyFFOYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBha2IiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIAwpbEREHFLYiIg4obEVEHFDYiog4oLAVEXFAYSsi4oDCVkTEAYWtiIgDClsREQcUtiIiDihsRUQcUNiKiDigsBURcUBhKyLigMJWRMQBY6099GJjcoBNDTcdEZGjTitrbfIPFR1W2IqISN3oNoKIiAMKWxERBxS2IiIOKGxFRBxQ2IqIOKCwFRFxQGErIuKAwlZExAGFrYiIA/8PmfP94gQGKz8AAAAASUVORK5CYII=\n", 846 | "text/plain": [ 847 | "
" 848 | ] 849 | }, 850 | "metadata": {}, 851 | "output_type": "display_data" 852 | } 853 | ], 854 | "source": [ 855 | "policy_vi, V_vi = value_iteration(env)\n", 856 | "\n", 857 | "# print the optimal policy\n", 858 | "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n", 859 | "print(policy_vi,\"\\n\")\n", 860 | "\n", 861 | "# plot the optimal state-value function\n", 862 | "plot_values(V_vi)" 863 | ] 864 | }, 865 | { 866 | "cell_type": "markdown", 867 | "metadata": {}, 868 | "source": [ 869 | "Run the code cell below to test your function. If the code cell returns **PASSED**, then you have implemented the function correctly! \n", 870 | "\n", 871 | "**Note:** In order to ensure accurate results, make sure that the `value_iteration` function satisfies the requirements outlined above (with three inputs, two outputs, and with the default values of the input arguments unchanged)." 872 | ] 873 | }, 874 | { 875 | "cell_type": "code", 876 | "execution_count": 23, 877 | "metadata": {}, 878 | "outputs": [ 879 | { 880 | "data": { 881 | "text/markdown": [ 882 | "**PASSED**" 883 | ], 884 | "text/plain": [ 885 | "" 886 | ] 887 | }, 888 | "metadata": {}, 889 | "output_type": "display_data" 890 | } 891 | ], 892 | "source": [ 893 | "check_test.run_check('value_iteration_check', value_iteration)" 894 | ] 895 | } 896 | ], 897 | "metadata": { 898 | "anaconda-cloud": {}, 899 | "kernelspec": { 900 | "display_name": "Python 3", 901 | "language": "python", 902 | "name": "python3" 903 | }, 904 | "language_info": { 905 | "codemirror_mode": { 906 | "name": "ipython", 907 | "version": 3 908 | }, 909 | "file_extension": ".py", 910 | "mimetype": "text/x-python", 911 | "name": "python", 912 | "nbconvert_exporter": "python", 913 | "pygments_lexer": "ipython3", 914 | "version": "3.6.8" 915 | } 916 | }, 917 | "nbformat": 4, 918 | "nbformat_minor": 2 919 | } 920 | -------------------------------------------------------------------------------- /Dynamic-Programming/algorithms.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/antonio-f/Dynamic-Programming/ca116016953dfff2cd4a8f1a7e9dd31ed660435e/Dynamic-Programming/algorithms.pdf -------------------------------------------------------------------------------- /Dynamic-Programming/check_test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | import copy 3 | from IPython.display import Markdown, display 4 | import numpy as np 5 | from frozenlake import FrozenLakeEnv 6 | 7 | def printmd(string): 8 | display(Markdown(string)) 9 | 10 | def policy_evaluation_soln(env, policy, gamma=1, theta=1e-8): 11 | V = np.zeros(env.nS) 12 | while True: 13 | delta = 0 14 | for s in range(env.nS): 15 | Vs = 0 16 | for a, action_prob in enumerate(policy[s]): 17 | for prob, next_state, reward, done in env.P[s][a]: 18 | Vs += action_prob * prob * (reward + gamma * V[next_state]) 19 | delta = max(delta, np.abs(V[s]-Vs)) 20 | V[s] = Vs 21 | if delta < theta: 22 | break 23 | return V 24 | 25 | def q_from_v_soln(env, V, s, gamma=1): 26 | q = np.zeros(env.nA) 27 | for a in range(env.nA): 28 | for prob, next_state, reward, done in env.P[s][a]: 29 | q[a] += prob * (reward + gamma * V[next_state]) 30 | return q 31 | 32 | def policy_improvement_soln(env, V, gamma=1): 33 | policy = np.zeros([env.nS, env.nA]) / env.nA 34 | for s in range(env.nS): 35 | q = q_from_v_soln(env, V, s, gamma) 36 | best_a = np.argwhere(q==np.max(q)).flatten() 37 | policy[s] = np.sum([np.eye(env.nA)[i] for i in best_a], axis=0)/len(best_a) 38 | return policy 39 | 40 | def policy_iteration_soln(env, gamma=1, theta=1e-8): 41 | policy = np.ones([env.nS, env.nA]) / env.nA 42 | while True: 43 | V = policy_evaluation_soln(env, policy, gamma, theta) 44 | new_policy = policy_improvement_soln(env, V) 45 | if (new_policy == policy).all(): 46 | break; 47 | policy = copy.copy(new_policy) 48 | return policy, V 49 | 50 | env = FrozenLakeEnv() 51 | random_policy = np.ones([env.nS, env.nA]) / env.nA 52 | 53 | class Tests(unittest.TestCase): 54 | 55 | def policy_evaluation_check(self, policy_evaluation): 56 | soln = policy_evaluation_soln(env, random_policy) 57 | to_check = policy_evaluation(env, random_policy) 58 | np.testing.assert_array_almost_equal(soln, to_check) 59 | 60 | def q_from_v_check(self, q_from_v): 61 | V = policy_evaluation_soln(env, random_policy) 62 | soln = np.zeros([env.nS, env.nA]) 63 | to_check = np.zeros([env.nS, env.nA]) 64 | for s in range(env.nS): 65 | soln[s] = q_from_v_soln(env, V, s) 66 | to_check[s] = q_from_v(env, V, s) 67 | np.testing.assert_array_almost_equal(soln, to_check) 68 | 69 | def policy_improvement_check(self, policy_improvement): 70 | V = policy_evaluation_soln(env, random_policy) 71 | new_policy = policy_improvement(env, V) 72 | new_V = policy_evaluation_soln(env, new_policy) 73 | self.assertTrue(np.all(new_V >= V)) 74 | 75 | def policy_iteration_check(self, policy_iteration): 76 | policy_soln, _ = policy_iteration_soln(env) 77 | policy_to_check, _ = policy_iteration(env) 78 | soln = policy_evaluation_soln(env, policy_soln) 79 | to_check = policy_evaluation_soln(env, policy_to_check) 80 | np.testing.assert_array_almost_equal(soln, to_check) 81 | 82 | def truncated_policy_iteration_check(self, truncated_policy_iteration): 83 | self.policy_iteration_check(truncated_policy_iteration) 84 | 85 | def value_iteration_check(self, value_iteration): 86 | self.policy_iteration_check(value_iteration) 87 | 88 | check = Tests() 89 | 90 | def run_check(check_name, func): 91 | try: 92 | getattr(check, check_name)(func) 93 | except check.failureException as e: 94 | printmd('**PLEASE TRY AGAIN**') 95 | return 96 | printmd('**PASSED**') -------------------------------------------------------------------------------- /Dynamic-Programming/frozenlake.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import sys 3 | from six import StringIO, b 4 | 5 | from gym import utils 6 | from gym.envs.toy_text import discrete 7 | 8 | LEFT = 0 9 | DOWN = 1 10 | RIGHT = 2 11 | UP = 3 12 | 13 | MAPS = { 14 | "4x4": [ 15 | "SFFF", 16 | "FHFH", 17 | "FFFH", 18 | "HFFG" 19 | ], 20 | "8x8": [ 21 | "SFFFFFFF", 22 | "FFFFFFFF", 23 | "FFFHFFFF", 24 | "FFFFFHFF", 25 | "FFFHFFFF", 26 | "FHHFFFHF", 27 | "FHFFHFHF", 28 | "FFFHFFFG" 29 | ], 30 | } 31 | 32 | class FrozenLakeEnv(discrete.DiscreteEnv): 33 | """ 34 | Winter is here. You and your friends were tossing around a frisbee at the park 35 | when you made a wild throw that left the frisbee out in the middle of the lake. 36 | The water is mostly frozen, but there are a few holes where the ice has melted. 37 | If you step into one of those holes, you'll fall into the freezing water. 38 | At this time, there's an international frisbee shortage, so it's absolutely imperative that 39 | you navigate across the lake and retrieve the disc. 40 | However, the ice is slippery, so you won't always move in the direction you intend. 41 | The surface is described using a grid like the following 42 | 43 | SFFF 44 | FHFH 45 | FFFH 46 | HFFG 47 | 48 | S : starting point, safe 49 | F : frozen surface, safe 50 | H : hole, fall to your doom 51 | G : goal, where the frisbee is located 52 | 53 | The episode ends when you reach the goal or fall in a hole. 54 | You receive a reward of 1 if you reach the goal, and zero otherwise. 55 | 56 | """ 57 | 58 | metadata = {'render.modes': ['human', 'ansi']} 59 | 60 | def __init__(self, desc=None, map_name="4x4",is_slippery=True): 61 | if desc is None and map_name is None: 62 | raise ValueError('Must provide either desc or map_name') 63 | elif desc is None: 64 | desc = MAPS[map_name] 65 | self.desc = desc = np.asarray(desc,dtype='c') 66 | self.nrow, self.ncol = nrow, ncol = desc.shape 67 | 68 | nA = 4 69 | nS = nrow * ncol 70 | 71 | isd = np.array(desc == b'S').astype('float64').ravel() 72 | isd /= isd.sum() 73 | 74 | P = {s : {a : [] for a in range(nA)} for s in range(nS)} 75 | 76 | def to_s(row, col): 77 | return row*ncol + col 78 | def inc(row, col, a): 79 | if a==0: # left 80 | col = max(col-1,0) 81 | elif a==1: # down 82 | row = min(row+1,nrow-1) 83 | elif a==2: # right 84 | col = min(col+1,ncol-1) 85 | elif a==3: # up 86 | row = max(row-1,0) 87 | return (row, col) 88 | 89 | for row in range(nrow): 90 | for col in range(ncol): 91 | s = to_s(row, col) 92 | for a in range(4): 93 | li = P[s][a] 94 | letter = desc[row, col] 95 | if letter in b'GH': 96 | li.append((1.0, s, 0, True)) 97 | else: 98 | if is_slippery: 99 | for b in [(a-1)%4, a, (a+1)%4]: 100 | newrow, newcol = inc(row, col, b) 101 | newstate = to_s(newrow, newcol) 102 | newletter = desc[newrow, newcol] 103 | done = bytes(newletter) in b'GH' 104 | rew = float(newletter == b'G') 105 | li.append((1.0/3.0, newstate, rew, done)) 106 | else: 107 | newrow, newcol = inc(row, col, a) 108 | newstate = to_s(newrow, newcol) 109 | newletter = desc[newrow, newcol] 110 | done = bytes(newletter) in b'GH' 111 | rew = float(newletter == b'G') 112 | li.append((1.0, newstate, rew, done)) 113 | 114 | # obtain one-step dynamics for dynamic programming setting 115 | self.P = P 116 | 117 | super(FrozenLakeEnv, self).__init__(nS, nA, P, isd) 118 | 119 | def _render(self, mode='human', close=False): 120 | if close: 121 | return 122 | outfile = StringIO() if mode == 'ansi' else sys.stdout 123 | 124 | row, col = self.s // self.ncol, self.s % self.ncol 125 | desc = self.desc.tolist() 126 | desc = [[c.decode('utf-8') for c in line] for line in desc] 127 | desc[row][col] = utils.colorize(desc[row][col], "red", highlight=True) 128 | if self.lastaction is not None: 129 | outfile.write(" ({})\n".format(["Left","Down","Right","Up"][self.lastaction])) 130 | else: 131 | outfile.write("\n") 132 | outfile.write("\n".join(''.join(line) for line in desc)+"\n") 133 | 134 | if mode != 'human': 135 | return outfile 136 | -------------------------------------------------------------------------------- /Dynamic-Programming/plot_utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import matplotlib.pyplot as plt 3 | 4 | def plot_values(V): 5 | # reshape value function 6 | V_sq = np.reshape(V, (4,4)) 7 | 8 | # plot the state-value function 9 | fig = plt.figure(figsize=(6, 6)) 10 | ax = fig.add_subplot(111) 11 | im = ax.imshow(V_sq, cmap='cool') 12 | for (j,i),label in np.ndenumerate(V_sq): 13 | ax.text(i, j, np.round(label, 5), ha='center', va='center', fontsize=14) 14 | plt.tick_params(bottom=False, left=False, labelbottom=False, labelleft=False) 15 | plt.title('State-Value Function') 16 | plt.show() 17 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Dynamic-Programming 2 | Algorithms for Policy Evaluation, Estimation of Action Values, Policy Improvement, Policy Iteration, Truncated Policy Evaluation, Truncated Policy Iteration, Value Iteration . From Udacity's Deep Reinforcement Learning Nanodegree program. 3 | --------------------------------------------------------------------------------