├── .github └── workflows │ └── pythonpackage.yml ├── .gitignore ├── .vscode └── settings.json ├── Final Presentation.ipynb ├── README.md ├── autodiff ├── .gitignore ├── __init__.py ├── autodiff │ ├── __init__.py │ ├── core.py │ ├── diff.py │ ├── global_vars.py │ ├── graph │ │ ├── __init__.py │ │ ├── manager.py │ │ ├── node.py │ │ └── tracer.py │ └── numpy_grad │ │ ├── .gitignore │ │ ├── __init__.py │ │ ├── vjps.py │ │ └── wrapper.py ├── nn │ ├── criterion.py │ ├── layer.py │ └── optimizer.py ├── tests │ ├── __init__.py │ ├── autodiff_test.py │ └── numpy_test.py └── utils │ ├── model_utils.py │ └── test_utils.py ├── examples ├── __init__.py ├── classification │ ├── Digits.png │ ├── Iris.png │ ├── __init__.py │ ├── digits.py │ └── iris.py ├── regression │ ├── Boston.png │ ├── __init__.py │ ├── boston.py │ ├── regression.png │ └── regression.py └── simple │ ├── __init__.py │ ├── poly.png │ └── poly_test.py ├── img ├── actions.png ├── backward.png └── forward.png ├── proposal.md └── requirements.txt /.github/workflows/pythonpackage.yml: -------------------------------------------------------------------------------- 1 | name: Python package 2 | 3 | on: [push] 4 | 5 | jobs: 6 | build: 7 | 8 | runs-on: ubuntu-latest 9 | strategy: 10 | max-parallel: 4 11 | matrix: 12 | python-version: [3.6, 3.7] 13 | 14 | steps: 15 | - uses: actions/checkout@v1 16 | - name: Set up Python ${{ matrix.python-version }} 17 | uses: actions/setup-python@v1 18 | with: 19 | python-version: ${{ matrix.python-version }} 20 | - name: Install dependencies 21 | run: | 22 | python -m pip install --upgrade pip 23 | pip install -r requirements.txt 24 | - name: Lint with flake8 25 | run: | 26 | pip install flake8 27 | # stop the build if there are Python syntax errors or undefined names 28 | flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics 29 | # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide 30 | flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics 31 | - name: Test with pytest 32 | run: | 33 | pip install pytest 34 | pytest autodiff/tests 35 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | env/ 2 | __pycache__/ 3 | debug*.txt 4 | .ipynb_checkpoints/ -------------------------------------------------------------------------------- /.vscode/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "python.pythonPath": "/home/chenyee/anaconda3/bin/python", 3 | "python.formatting.provider": "yapf" 4 | } -------------------------------------------------------------------------------- /Final Presentation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# A basic neural network library supported auto differentiation" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Outline\n", 15 | "\n", 16 | "- Motivation\n", 17 | "- Introduction\n", 18 | "- Implementation\n", 19 | "- Application\n", 20 | "- Software engineering\n", 21 | "- Misc\n", 22 | "\n", 23 | "---" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Motivation\n", 31 | "\n", 32 | "Most deep learning courses aim to teach math behind the network, architecture and their applications, but seldom course talk about how to implement and design the deep learning library.\n", 33 | "\n", 34 | "### Goal\n", 35 | "\n", 36 | "- Implement this kind of library\n", 37 | "- Learn how and why the priors (Tensorflow and PyTorch etc.) design their work.\n", 38 | "\n", 39 | "---" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## Introduction" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "### Computational Graph\n", 54 | "\n", 55 | "We can represent the computations by using computation graph.\n", 56 | "\n", 57 | "The node represent input and operation, and the edge represent the argument of the operation.\n", 58 | "\n", 59 | "For example, we have the computation like $(3-2) + 1$ (which will be visited again), the corresponding graph is\n", 60 | "\n", 61 | "![forward](img/forward.png)\n", 62 | "\n", 63 | "Note that the order of arguments matters (3 should be on top of 2 etc.), but unfortunately the plot toolkits is hard to control that." 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "### Back propagation\n", 71 | "\n", 72 | "The forward pass builds the graph, and the backward propagation calculate the gradient.\n", 73 | "\n", 74 | "Use the last example we used $(3 - 2) + 1$, but transform it into algebraic way $z = (a - b) + c$.\n", 75 | "\n", 76 | "Suppose we want to compute the gradient of `z` w.r.t. `c`, it's easily obtained by using the chain rule.\n", 77 | "\n", 78 | "$$\\frac{\\partial (a-b) + c}{\\partial c} = 1$$\n", 79 | "\n", 80 | "Now we want gradient w.r.t. `b`, again we use the chain rule.\n", 81 | "\n", 82 | "Before that, we decompose operation into different equation in order to introduce auto differentiation.\n", 83 | "\n", 84 | "$$\n", 85 | "\\begin{aligned}\n", 86 | "z &= (a - b) + c \\\\\n", 87 | "\\Rightarrow z &= d + c\n", 88 | "\\end{aligned}\n", 89 | "$$\n", 90 | "\n", 91 | "$$\n", 92 | "\\begin{aligned}\n", 93 | "\\frac{\\partial (a-b) + c}{\\partial b} &= \\frac{\\partial (d + c)}{\\partial d} \\times \\frac{\\partial d}{\\partial b} \\\\\n", 94 | " &= 1 \\times \\frac{\\partial (a - b)}{\\partial b} \\\\\n", 95 | " &= 1 \\times -1 \\\\\n", 96 | " &= -1\n", 97 | "\\end{aligned} \n", 98 | "$$" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "### Auto Differentiation\n", 106 | "\n", 107 | "We visit the last example again, all we need to know to calculate the gradient w.r.t. `d` is\n", 108 | "\n", 109 | "- Current operation node: add\n", 110 | "- Input argument: d and c\n", 111 | "- Previous gradient (upstream): 1 (I didn't say but it is actually 1 in the begining)\n", 112 | "\n", 113 | "$$\\frac{\\partial (d + c)}{\\partial d} = 1$$\n", 114 | "\n", 115 | "We know that `d` is the first argument of `add` operation, so after taking derivative w.r.t `d`, result becomes $1 \\times 1$ and we get 1, note that the first one is the upstream, and the second one is the derivative result.\n", 116 | "\n", 117 | "Now we get the gradient w.r.t. `d`, and because `d` is equal to `(a - b)`, so we pass the computed gradient (downstream) to their parent node (`a` and `b`, respectively), now we move forward (actually it's backward).\n", 118 | "\n", 119 | "Now we move ourselves to node `b` to compute the gradient w.r.t. `b`, and the current information are\n", 120 | "\n", 121 | "- Current operation node: sub\n", 122 | "- Input argument: a and b\n", 123 | "- Previous gradient (upstream): 1\n", 124 | "\n", 125 | "$$ \\frac{\\partial (a - b)}{\\partial b} = -1$$\n", 126 | "\n", 127 | "Because `b` is the second argument of `sub` operation, so the result is $1 \\times -1$ which is -1, and again, first 1 is the upstream, and the second -1 is the derivative result.\n", 128 | "\n", 129 | "The point I want to emphasize is that, it seems easy to get the derivative w.r.t. any variable without any decomposition because it's easy, but what if we want to compute the gradient of following eqation w.r.t. `z` directly in the following equation?\n", 130 | "\n", 131 | "$$y = \\frac{1}{1 + e^{-z}}$$ \n", 132 | "\n", 133 | "The philosophy of auto differntiation is that we don't take derivative directly, we decompose it into different primitive function and solve derivative of each decomposed function and combine together because we have chain rule got our back.\n", 134 | "\n", 135 | "So how does auto-diff solve the previous equation?\n", 136 | "\n", 137 | "It will decompose into\n", 138 | "\n", 139 | "- negative\n", 140 | "- exp\n", 141 | "- add\n", 142 | "- reciprocal\n", 143 | "\n", 144 | "and solve each function previously listed." 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "### VJP\n", 152 | "\n", 153 | "VJP stands for vector-jacobian product, this is usually the product between upstream and the derivative w.r.t one of the argument of operation. I will not elaborate here and it's highly recommended to take a look at [Automatic Differentiation, Toronto CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec10.pdf).\n", 154 | "\n", 155 | "In the much simple way, you can think that what's the computation we want to do to get the downstream w.r.t any argument in any operation node." 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "## Implementation" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "metadata": {}, 168 | "source": [ 169 | "### Forward Propagation\n", 170 | "\n", 171 | "Q: How do we obtain the computational graph?\n", 172 | "\n", 173 | "A: Trace every operation with the wrapped version by decorator." 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "#### Examples:\n", 181 | "\n", 182 | "That's say we want to print the numpy function name during the calculation.\n", 183 | "\n", 184 | "We can fully leverage the power of decorator." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 1, 190 | "metadata": {}, 191 | "outputs": [ 192 | { 193 | "name": "stdout", 194 | "output_type": "stream", 195 | "text": [ 196 | "Wrapped version of numpy function, and its name is add\n", 197 | "3\n", 198 | "Wrapped version of numpy function, and its name is subtract\n", 199 | "-1\n", 200 | "Wrapped version of numpy function, and its name is __getitem__\n", 201 | "[2]\n", 202 | "Wrapped version of numpy function, and its name is __add__\n", 203 | "[1 1]\n" 204 | ] 205 | } 206 | ], 207 | "source": [ 208 | "%matplotlib inline \n", 209 | "\n", 210 | "import numpy as np\n", 211 | "\n", 212 | "\n", 213 | "def get_name_decorator(func):\n", 214 | " def wrapped(*args, **kwargs):\n", 215 | " print(f\"Wrapped version of numpy function, and its name is {func.__name__}\")\n", 216 | " result = func(*args, **kwargs)\n", 217 | " return result\n", 218 | " return wrapped\n", 219 | " \n", 220 | "\n", 221 | "for function in [np.add, np.subtract, np.ndarray.__getitem__, np.ndarray.__add__]:\n", 222 | " globals()[function.__name__] = get_name_decorator(function)\n", 223 | "\n", 224 | "print(add(1, 2))\n", 225 | "print(subtract(1, 2))\n", 226 | "print(__getitem__(np.array([0,1,2]), [2]))\n", 227 | "print(__add__(np.array([0,1]), np.array([1,0])))" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "Now things have been little complicated. Due to the same reason, we can put the operator on the computational graph on the fly." 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": 2, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "class Node:\n", 244 | " def __init__(self):\n", 245 | " self.gradient = 0\n", 246 | "\n", 247 | "class OperationNode(Node):\n", 248 | " def __init__(self, func, args, kwargs, result):\n", 249 | " super().__init__()\n", 250 | " self.recipe = (func, args, kwargs, result, len(args))\n", 251 | "\n", 252 | "\n", 253 | "class VariableNode(Node):\n", 254 | " def __init__(self, var):\n", 255 | " super().__init__()\n", 256 | " self.var = var\n", 257 | "\n", 258 | "\n", 259 | "class PlaceholderNode(Node):\n", 260 | " def __init__(self, var):\n", 261 | " super().__init__()\n", 262 | " self.var = var\n", 263 | "\n", 264 | "\n", 265 | "class ConstantNode(Node):\n", 266 | " def __init__(self, constant):\n", 267 | " super().__init__()\n", 268 | " self.constant = constant\n", 269 | "\n", 270 | "def plot_graph(backward=False, backward_result={}):\n", 271 | " arrow_style = \"<|-\" if backward else \"-|>\" \n", 272 | " edge_labels = {}\n", 273 | " if backward:\n", 274 | " info = \"output\"\n", 275 | " node_index = add_node(node=ConstantNode(info), info=info)\n", 276 | " default_graph.add_edge(node_index - 1, node_index)\n", 277 | " for node, result in backward_result.items():\n", 278 | " for edge in default_graph.edges:\n", 279 | " head, tail = edge\n", 280 | " if head == node:\n", 281 | " edge_labels[tuple(edge)] = str(result)\n", 282 | " edge_labels[(node_index, node_index-1)] = \"1\"\n", 283 | " plt.figure(3,figsize=(10,10)) \n", 284 | " limits=plt.axis('off')\n", 285 | " labels = {i: default_graph.nodes[i]['info'] for i in default_graph.nodes}\n", 286 | " pos = nx.spring_layout(default_graph)\n", 287 | " nx.draw_networkx_nodes(default_graph, pos, node_size = 3000, alpha=0.8)\n", 288 | " nx.draw_networkx_labels(default_graph, pos, labels=labels, font_color=\"w\")\n", 289 | " nx.draw_networkx_edges(default_graph, pos, width=3, arrowstyle=arrow_style, arrowsize=15)\n", 290 | " if backward:\n", 291 | " nx.draw_networkx_edge_labels(default_graph,pos,edge_labels=edge_labels,font_size=30)\n", 292 | " plt.show() " 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 11, 298 | "metadata": {}, 299 | "outputs": [ 300 | { 301 | "name": "stdout", 302 | "output_type": "stream", 303 | "text": [ 304 | "Forward pass\n", 305 | "Result is 2\n", 306 | "Corresponding computational graph is \n" 307 | ] 308 | }, 309 | { 310 | "data": { 311 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAIuCAYAAAC7EdIKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZicVZn+8e9TVb2v2RMgUBBlkx1kHMZ9FNR2EHVwFGVxGVHHFdCxFPelHQd1GMefk3ENIC6MOoAl4zKuOCoKGCayKdAEQvak0+nu9FJVz++P83ZSaTpLd1fVW1V9f66rLlPn7a56yot03znnvM8xd0dERESkniXiLkBERESk3BR4REREpO4p8IiIiEjdU+ARERGRuqfAIyIiInVPgUdERETqngKPiIiI1D0FHhEREal7CjwiIiJS9xR4REREpO4p8IiIiEjdU+ARERGRuqfAIyIiInVPgUdERETqngKPiIiI1D0FHhEREal7CjwiIiJS9xR4REREpO4p8IiIiEjdU+ARERGRuqfAIyIiInVPgUdERETqngKPiIiI1D0FHhEREal7CjwiIiJS91JxFyAiItUlncm2A0cD3UAT0ACMA6NAP3B/X2/PYHwVikyfuXvcNYiISEzSmWwCOBk4ATgTOBVYBoxFX2LRw6MHQCOwHrgTuA1YA6zu6+0pVK5ykelR4BERmYPSmWwXcC5wKbAISBICzQhhJudAmoBmQhjKA5uBlcCNfb09A+WoWWQ2FHhEROaQdCZ7HHAJcB5hH+cYsKsEL91CmPkpAN8FvtrX23NvCV5XpCQUeERE5oB0JtsKXA5cRJiV2UmYmSm1JNBBCD7XAp/q6+0ZLsP7iEyLAo+ISJ1LZ7JnAFcDS4ABQhgptwTQCWwA3tbX23N7Bd5TZJ8UeERE6lQ6k20B3kmY1ckBcdxZ1U6Y9bkGuKqvt6cUy2ci06bAIyJSh9KZbCewCjgJ2EFlZnX2JQF0AXcBF2tTs8RBgUdEpM6kM9kFwPXACkLfnGrRDTwAXNDX27M17mJkblHgERGpI9HMzg1UX9iZMBF6ztdMj1SSjpYQEakT0Z6dVVRv2IFQ1wpgVVSvSEUo8IiI1I8rCHt2qjXsTOgn1HlF3IXI3KElLRGROhDdev51Qn+dWjjiIUHo1/Ny3bIulaDAIyJS46Kmgj8CFhDPrecz1Q5sAZ6r29Wl3LSkJSJS+y4nNBWspbADod6laGlLKkAzPCIiNSw6G+tGQniohaWsyRKEmZ5zdfaWlJNmeEREatslhJ/ltRh2INSdAC6OuxCpb5rhERGpUelMtgv4LeG083IcBFopScJp62eqN4+Ui2Z4RERq17mEsFDLYQdC/UnC5xEpC83wiIjUoHQmmwB+AcwjzPDUuhZgG/CMvt6eWl2ekyqmGR4Rkdp0MrCI+gg7ED7HYkJDQpGSU+AREalNJxCWgepJkvC5REpOgUdEpDadCdTbngQnfC6RkkvFXYCIiMzIqcBIuV78H565YtGLTjl04ZEL21p+fM/GbW/82h195XqvIiPAaRV4H5mDNMMjIlJj0plsO7AMGC3Xe2wYGBn//M8fWJ+9a/2Wcr3HFEaBZelMtq2C7ylzhGZ4RERqz9HAWDnf4Nt3rOsHOPmwrtYlDc2N5XyvScaAY4A7KvieMgdohkdEpPZ0V+qN3N0q9V5FKvb5ZO7QDI+ISO1pAsoeRHIDW5bu2Lp5aWdy3rgXCglLJCrRH8eASs4oyRyhGR4RkdrTQAUCT2F0aD5ALpdryPWvP7JCjWoVeKQsFHhERGrPOBW4JT3Z2rlp4s+F0eHu/MCm5eV+T8LnKuv+JJmbFHhERGrPKGUOPKmE0da9YGtLa9tQMmE0NSRhZOfi3M6ti8v5vijwSJloD4+ISO3pL/cbvPv5xx7yuqcetWzi+d+edRxX/ddv+NR//Wa5JRvGkq2d5ayh7J9P5h4dHioiUmOiPjyrge2VeD8vFGx826PH+Pho6I9j5g3zDrkv0dQ6VIa3mwec1NfbU47XljlMS1oiIjWmr7dnEFhPuFur7CyR8IZ5h/zZkqnQ6NDdcv3rn1AYHy31+zcB6xV2pBwUeEREatOdQHOl3sySqVxq3iF/skQiB+CFQiq3/bEnej5Xyq0RzajhoJSJAo+ISG26jQrcml4s0dA0mupe9mfMHMDzuabx7Y89wQuFUtVhhM8lUnIKPCIitWkNkK/0myaaWodSXYsfnHju46Ntuf71R5VoP2ie8LlESk6BR0SkNq0GNgMtlX7jZEtnf7JjwSMTz6MePYfN8mVbgE3AXbN8HZEpKfCIiNSgvt6eArCSmLoSp9rnbypuTJgfHliSG9w2mx49jcDK6HOJlJwCj4hI7bqJsAyUjOPNk52LH0k0te7umZPfuXV5ftfATA7+TBI+x00lK05kEgUeEZEa1dfbswO4EeiI4/3NjFT3sgetoWn3beS5HZuOKowOt03zpTqA7/b19gyUtkKRPRR4RERq21eBAjH9PC9Bj54Eof5V5apRBBR4RERqWl9vzz3AtUBnXDXMskdPJ3BtX2/PveWtUuY6BR4Rkdr3KWAj0B5XATPs0dMObACuqkiRMqcp8IiI1Li+3p5h4K2Ezb+x/VxPNLUOpTof16PnyH306EkQ6n1bX2/PrkrVKHOXAo+ISB3o6+25HbgG6IqzjmRrZ3+yfX5xj555++jR0wVcE9UtUnYKPCIi9eMqQuO+mdwaXjLJ9vmbki2P69GzqOhLugl1ailLKsZK1A5cRESqQDqT7QRuAFYA/Qf48rJxd3LbH1tRGB3eHb5S3UseSLZ0AjwAnK/b0KWSNMMjIlJHohBxASFUxDbTE/Xoeai4R09hZOio/K6BLcAFCjtSaZrhERGpQ9FMzyrgJGAHoddNxXk+lxrf9uixlmpqGtvUx9bvf2aLjw7/hbs/eODvFikdBR4RkTqVzmRbgCuAiwhHNwzGUEa758abd/z2PxcP/PqGFs+NAdwPnOXuW2OoR+YoBR4RkTqXzmRPB64GlgIDVGa2J0FoKrgBeNvDn3hhM/A/wEQH5luB57r7SAVqEdEeHhGRehfd+n02YYmrnbC3p1wHjiaBedH7rAKe29fbc7u7/wp4FTDxr+ynAqvMTL+HpCI0wyMiMoekM9ljgYuBlxD+0TsGlKLxXwvQSJg9+g6waqrjIszscva+Hf2f3f1dJXh/kf1S4BERmYOiTc3nApcCiwkzMw6MAKMH8RJNQDNghP1Bm4CVwE37uwPLzAz4V+DNRcNvdvfPzeBjiBw0BR4RkTksnckmCHdynQCcCZwGLCPM/EAINEYIQxO/MBqB9cDtwO+ANcBdfb09B7U3yMyShFmgc6OhAnCeu988288jsi8KPCIispd0JtsGHEPY69MYPcaiRz9wX19vz9C+X+HAzKwV+CkhZAEMA89099/N5nVF9kWBR0REYmFmi4HfAEdGQ5uAp7j7Q/FVJfVKu+NFRCQW7r4JeD6wLRpaDNxiZvPjq0rqlQKPiIjExt3vA17Eno3SxwD/ZWbN8VUl9UiBR0REYuXutxK6QU94GvAV9eiRUtJ/TCIiEjt3/xbwzqKhlwMfj6kcqUPatCwiIlUh6tHzb8Cbiobf5O6fj6kkqSMKPCIiUjXMLEXo0fM30VABeJG7fy++qqQeKPCIiEhVMbM24GfAGdHQMPAMd/99bEVJzVPgERGRqmNmSwg9etLR0EZCj56+uGqS2qZNyyIiUnXcfSOhR8/2aGgJoUfPvPiqklqmwCMiIlXJ3e8l9OiZONfrWEKPnqb4qpJapcAjIiJVy91/CVxcNPR01KNHZkD/wYiISFVz928A/1g09ArgozGVIzVKm5ZFRKTqRT16Pge8sWj4De6+MqaSpMYo8IiISE2IevT8F9ATDRWAv3H378dXldQKBR4REakZZtZO6NFzejQ0ROjRc3tsRUlNUOAREZGaYmZLCT16joiGNhB69DwcX1VS7bRpWUREaoq7byD06OmPhpaiHj1yAAo8IiJSc9z9Hvbu0XMc8B316JF9UeAREZGa5O6/AC4pGnom8OXoji6RvSjwiIhIzXL3rwOZoqELUI8emYI2LYuISE2LZnQ+D1xaNPx6d/9CTCVJFVLgERGRmhf16LkReEE0lCf06LklvqqkmijwiIhIXYh69PwcOC0aGgKe7u53xFeVVAsFHhERqRtmtozQo+fwaEg9egTQpmUREakj7r6e0KNnRzS0FPi+mXXHV5VUAwUeERGpK+5+N3AeMB4NHU/o0dMYX1USNwUeERGpO+7+M/bu0fMs4Evq0TN3KfCIiEhdcvfrgfcWDb0K+HBM5UjMtGlZRETqVjSjsxL4+6Lhv3f3L8ZUksREgUdEROpa1KPnZuB50VAe6HH3H8RXlVSaAo+IiNQ9M+sg9Og5NRoaBJ7m7n+IryqpJAUeERGZE8zsEEKPnuXR0HpCj5618VUllaJNyyIiMie4+2Ps3aNnGerRM2co8IiIyJzh7n8EXsKeHj1PAr6tHj31T4FHRETmFHf/CfCaoqFnA19Qj576psAjIiJzjrtfB1xZNHQR8KGYypEK0KZlERGZk6IZnf8AXlc0/Fp3/3JMJUkZKfCIiMicZWYNhB4950RDeeAF7v7D+KqSclDgERGROS3q0fML4JRoaCehR8/q+KqSUlPgERGROW+KHj3rCD16Ho2vKiklbVoWEZE5L+rR8wJgIBo6lNCjpyu+qqSUFHhEREQAd19D6NGTi4ZOBP4z2ucjNU6BR0REJOLu/wO8tmjoOcB/qEdP7VPgERERKeLu1wDvLxq6ZNJzqUHatCwiIjJJNKPzRfbuyPxqd/9qPBXJbCnwiIiITCHau/M94OxoKEfo0fOj+KqSmVLgERER2Qcz6wR+CZwUDe0Enurud8VXlcyEAo+IiMh+mNmhhB49h0VD6tFTg7RpWUREZD/cfR2P79GTjWZ/pEYo8IiIiByAu/8f8FL29Og5CfXoqSkKPCIiIgfB3X8M/H3R0HOBlerRUxsUeERERA5SdFv6B4uGXg1cGUsxMi3atCwiIjIN0YzOlwkNCSdc4u6r4qlIDoYCj4iIyDRFe3eyhGUtCHt7nhcdTSFVSIFHRERkBqK7tG4lHDIK4S6up0YbnKXKKPCIiIjMkJkdRujRc2g09CihR8+6+KqSqWjTsoiIyAxFzQd7CB2YITQnzJpZR3xVyVQUeERERGbB3VcDf8ueHj0nAzeoR091UeARERGZJXf/IfD6oqFzgM+rR0/1UOAREREpAXf/CvDhoqHXAu+JqRyZRJuWRURESiSa0fkqcFHR8IXufl08FckEBR4REZESMrNG4PvAX0dD44QePT+JrypR4BERESkxM+si9Og5IRraQejRsya+quY2BR4REZEyMLPlhB49h0RDjxB69DwWX1VzlzYti4iIlIG7P0Lo0TMYDS0HvqcePfFQ4BERESkTd/8DoUdPPho6FfiWmaXiq2puUuAREREpI3f/AXBp0dDzUI+eilPgERERKTN3/xLw0aKh1wGZmMqZk7RpWUREpAKiGZ1VwIVFw69y96/FVNKcosAjIiJSIVGPnv8GnhUNjQPnuPtP46tqblDgERERqSAz6yb06HlSNLQD+Ct3/2N8VdU/BR4REZEKM7PDCT16lkVDawk9etbHV1V906ZlERGRCnP3tezdo+dwQo+e9viqqm8KPCIiIjFw9zuBl7GnR89pwDfVo6c8FHhERERi4u63AG8sGnoB8Dn16Ck9BR4REZEYufsXgI8VDb0e+MeYyqlb2rQsIiISs2hG5xrgVUXDF7j712Mqqe4o8IiIiFSBqEfPD4BnRkNjwNnu/vPYiqojCjwiIiJVwszmEXr0HB8N9QNnufs98VVVHxR4REREqoiZHUHo0bM0GnqY0KNnQ3xV1T5tWhYREaki7v4woUfPUDR0BOrRM2sKPCIiIlXG3e9g7x49pwNfV4+emdOSloiISJUys9cDK4uGPg/8g5fgl3c6k20Hjga6gSaggXCY6Shh79D9fb09g/t+hdqiwCMiIlLFzOzjQKZo6B/d/ZPTeY10JpsATgZOAM4ETiWc4zU28TbRw6MHQCOwHrgTuA1YA6zu6+0pzOyTxEuBR0REpIqZWQK4FrigaPgV7v6NA31vOpPtAs4FLgUWAUlCoBkhzOQcSBPQTAhDeWAzYcbpxr7enoFpfIzYKfCIiIhUOTNrIvToeUY0NAY8x91/OdXXpzPZ44BLgPMI+3XHgF0lKKWFMPNTAL4LfLWvt+feErxu2SnwiIiI1ICoR8+vgOOioe2EHj27A0c6k20FLgcuIszK7GTPxudSSgIdhOBzLfCpvt6e4TK8T8ko8IiIiNQIM0sTevQsiYb6CD16NqYz2TOAq6NrA4QwUm4JoBPYALytr7fn9gq854wo8IiIiNQQMzsd+AXQCmANTbcf9pav3ZJobH4FkAPiuLOqnTDrcw1wVV9vTymWz0pKgUdERKTGmFkPcJM1tiQWvvAyGg85ZjjZNu8+M4vzDqoE0AXcBVxcbZua1XhQRESkxrh7tmHB8nctenGGhoVHkN+5tTW3Y+NhMU9iFAj7ik4EbkhnsgviLGYyzfCIiIjUmHQm2wnckB/ecWZux6buifFk+/xHUx0LNsZY2oRu4AHg/GqZ6dEMj4iISA1JZ7ItwCpgRaKl84FEU9u2iWv5wW2H5Yd3zIuvut36gRXAqqje2CnwiIiI1JYrgJOAfjMjNW9pX6KhafdG5dzA5iMLo0PthbGRlrHNDx83tqnvSYXx0aYY6uyP6rwihvd+HC1piYiI1Ijo1vOvE/rr7N6g7Plccnzro8d6frwZwCyRd9xwTwAkmtu3Nsxb1hdDyQlCv56Xx33LumZ4REREakDUVPBqQiPBve7GsmQq3zD/kD9ZIpkDcC8kJ8IOgI/t6ohpgqNAqPfquJe2FHhERERqw+WEpoJT99lJNoxZU+v2qS55Id/oubE4lrUg1LuUmJe2FHhERESqXHQ21oWEDspTyu/cckhh185F+7ruo8Od5ajtIA0AF6Yz2WPjKkCBR0REpPpdQvidvc/GgoWRwfn7e4HC2HBHiWuajgKh/ovjKkCBR0REpIqlM9ku4EWEjcr7lGzr3oDZPjfqFMZ2dcV8o9JO4MVRD6GKU+ARERGpbucSzqna76nnybZ5WxoXpe9Kts9bZ8nU2OO+wD3hY7vaylTjwcgTPse5cby5bksXERGpUulMNkE4KHQecNAHcro7hZHBzsLwjsWFsV1dE+MNC5bfnWhsjvNgzxZgG/CMvt6eip77larkm4mIiMi0nAws4gDLWZOZGcmWjoFkS8dAYXy0KT+4bWmisWVnzGEHQmhbTGhI+IdKvrECj4iISPU6gbAMNGOJhqbRxLxlD5eonlJIEj5XRQOP9vCIiIhUrzOBiuw9OWphW2Nfb8/pqYRNef3KnuMO+Y8LTz+yBG/lhM9VUQo8IiIi1etUYCTuIkpsBDit0m+qwCMiIlKF0plsO7AMGI27lhIbBZalM9mK3jGmPTwiIiLV6Wjg8beXT9M7zz5m6YtPPXRhd2tDw+ado2NX/8+f1n3nznX9STM+ct4Jh/WcuGzB8FiucO2vH95Q/H0rFrU1fvplpxz5xMXtrfesHxjs2zpcyuA1BhwD3FHC19wvBR4REZHq1F2KF3l469Do+St/fd/6HbvGzz99+byPv/jEI3/z0NY1f3PSId1Pe8LCrr/5t1vvHhrNFb508Rkrir/vsy8/7ai71vUPvmzlr+//y6MWtP2/V572xFv/vKW/FDVFSvL5DpaWtERERKpTEzD1DuJp+Nbtj25f179rvODwzd8/sn1d/67RM9Pz2573pKXzrvvtw5vWbhse3zo0lv/8zx/cPcNzxILWxmOWdrR9JHvPY6O5gv/s/s2D//vA1lKGHQMaS/h6B6QZHhERkerUQAkCz0VPOWLBxWellyzpbG4EaGlIJue3NaUWtjc1rOvftXvJbO3Wod1LVod2tzQMjuZyQ6O53c0BH+vfNba0q7lUIUWBR0RERAAYZ5a3pB+5oK3xfS88/ojXrvrd/b/689bBvDv/c9kzjjeDrUOj44d2t+wOHcvntzZN/Hl9/8h4e1Mq1daUSkyEnmVdLY1eujvknRLsT5oOLWmJiIhUp1FmGXjampIJd9i8c3Qc4NVnpRekF7S1APz3mg3bLzjz8MXL57c2zG9rTL7hGSuWTnzfQ1uHxu7fuHPovS847pDGZMKe/sSF7X/1hAWl3HNT8cCjGR4REZHqNOs9M2seGxi5/ra1G7916V8eV3D8ljXrt655bMcgwBdvfWjzkQvbm7NvfuqThsfy+Wt+3bfhtMPndUx871u+fueDn/m7U45c/f6zT7l7/Y7BW9Zs2NrRnJpV1+dJSrkn6IB0eKiIiEgVivrwrAa2x11LGcwDTurr7Rmq1BtqSUtERKQK9fX2DALrCXdr1ZMmYH0lww4o8IiIiFSzO4HmuIsosWYq2HBwggKPiIhI9bqNEtyaXmWM8LkqSpuWRUREqtcaID+bF3B38ju3HFIYGVyQaGrblupavK5Etc1UnvC5KkqBR0REpHqtBjYTNvnums43ujuFXQPz8oPbDvN8rhEgP7xjabJz4WNmibjuWGoBNgF3VfqNtaQlIiJSpfp6ewrASqbRldjdye8a6B7f8vDxuR2bjpoIO3tYnLdnNwIro89VUQo8IiIi1e0mwjLQAXvgFMZHm8a3rD0u179xhefGWyZft2Rq1Cy2LUFJwue4KY43V+ARERGpYn29PTuAG4GOA31tfsemIzw31rqv65Zqquit4JN0AN/t6+0ZiOPNFXhERESq31eBAgf6vZ1Mje7vsqUaR0pX0rQkCPWviun9FXhERESqXV9vzz3AtUDn/r4u1bVkbaKlY/O+rltD47Q2PpdQJ3BtX2/PvTG9vwKPiIhIjfgUsBFo39cXWCLhlmzY5yyPpZriCDztwAbgqhjeezcFHhERkRrQ19szDLyVsPl3yt/fhdHh1vzQ9kMnnlsiOb77z5bIW6pxv0teZZAg1Pu2vt6euGaXdhciIiIiNaCvt+d24Bqga/I1L+QTuR0bj8LdAKyhaahhcfr/ku3z1llD01Cyc+HaGO7Q6gKuieqOlRoPioiI1JargNOAE4F+CL13cjs2HuH5XBOE2ZyG7qUPmiU81bFwAx1siKHObkKDwViXsiZohkdERKSGREtDFwMPEEIFheEdCwojQ/MnvibZufBhSzWOxVQihLoeAC6OeylrggKPiIhIjYl62VwAPFDIjS/K7dxy+MS1RHP7lmRr1/b4qtsddi6Iq+fOVMw9zg7TIiIiMlOLX/LehdbY8qdU99LuwugwlkiONCw8/B5LJCp+dANhEqWLsIx1cTWFHVDgERERqVlm9hlLNb6948wX037iczzZPv/hREPT1hhKaSfcjXUNcFW1LGMVU+ARERGpQWbWA3xv4nnbCc/+5MIXXvZXwFJggNDZuNwShKaCGwi3nsd+N9a+aA+PiIhIjTGzQwjHTUy4aWjNT94NnE04vqGdsJfmgAeOzlASmBe9zyrgudUcdkAzPCIiIjXFzJLAj4BnRUPrgJPdffdSVjqTPZZwJ9dLCJMbY0AplplagEbC7NF3gFVxHhcxHQo8IiIiNcTM3gN8LHrqwLPc/edTfW06k+0EzgUuBRYTZmYcGAEOputyE9AMGJAHNgErgZuqbVPygSjwiIiI1AgzOwv4BXuWqj7i7u8/0PelM9kEcBJwAnAmoXHhMsLMD4RAY4QwNBEMGoH1wO3A74A1wF19vT1x3AE2awo8IiIiNcDMuoE/AEdEQ7cSZndyM3m9dCbbBhxD2OvTGD3Gokc/cF9fb8/QbOuuFgo8IiIiVc7CIVjfBM6PhvoJ+3bWxldVbdFdWiIiItXvdewJOwCvVdiZHs3wiIiIVDEzOx74PeEOKYB/d/c3xlhSTVLgERERqVJm1gL8lnAyOsAfgSe7e9V1Mq52WtISERGpXlexJ+yMAH+nsDMzCjwiIiJVyMzOA95UNPR2d/9jXPXUOi1piYiIVBkzWw6sJhzfAPBt4HzXL+0ZU+ARERGpItHRET8FnhYNrQVOcfft8VVV+7SkJSIiUl2uZE/YyQMXKOzMngKPiIhIlTCzpwPFR0V80N1/FVc99URLWiIiIlXAzOYT9u0cFg39DHiOu+djK6qOaIZHREQkZtHREV9iT9jZCrxKYad0FHhERETi90bgvKLnr3b3dXEVU4+0pCUiIhIjMzsR+B3QFA191t3fGmNJdUmBR0REJCZm1ko4J+u4aGg18BR3H4mvqvqkJS0REZH4/At7ws4w8HKFnfJQ4BEREYmBmZ0P/H3R0Fvc/d646ql3WtISERGpMDNLA38AuqKhbxAaDOqXcpko8IiIiFSQmTUAPwf+Mhp6CDjV3XfEV1X905KWiIhIZX2APWEnB7xCYaf8FHhEREQqxMyeDbynaOhKd/9tXPXMJVrSEhERqQAzW0S47XxZNPRj4Bx3L8RX1dyhGR4REZEyi46O+Ap7ws5m4CKFncpR4BERESm/twI9Rc8vdvf1cRUzF2lJS0REpIzM7FTgN0BjNPRpd788xpLmJAUeERGRMjGzduB24Oho6HbgLHcfi6+quUlLWiIiIuXzWfaEnUHCLegKOzFQ4BERESkDM7sAuKRo6E3u/qeYypnztKQlIiJSYma2ArgT6IiGrnX3i2Isac5T4BERESkhM2sEbgWeHA39GTjN3XfGV5VoSUtERKS0PsKesDMOvFxhJ34KPCIiIiViZmcD7yoaere73x5XPbKHlrRERERKwMyWEI6OWBIN3QK8UN2Uq4NmeERERGbJzBLAKvaEnQ3AJQo71UOBR0REZPYuA86J/uzAhe6+KcZ6ZBItaYmIiMyCmT0Z+F8gFQ39k7u/O8aSZAoKPCIiIjNkZp3AHcCKaOi3wNPcfTy+qmQqWtISERGZATMz4P+xJ+wMEI6OUNipQgo8IiIiM3MR8Mqi55e6+0NxFSP7pyUtERGRaTKzowlLWW3R0Jfd/bUxliQHoMAjIiIyDWbWBPwaODUaug843d2H4qtKDkRLWiIiItPzCfaEnTHC0REKO1VOgUKcP3MAACAASURBVEdEROQgmVkP8PaioSvc/Q9x1SMHT0taIiIiB8HMDiEcHbEwGroZeJHrF2lNUOARERE5ADNLAj8Enh0NPQac7O5b4qtKpkNLWiIiIgf2LvaEHQdeqbBTWxR4RERE9sPM/hL4SNHQR939ZzGVIzOkJS0REZF9MLNu4A/AEdHQr4BnunsuvqpkJjTDIyIiMoXo6Ij/YE/Y6ScsZSns1CAFHhERkam9Fji/6Pnr3P3huIqR2dGSloiIyCRmdjzwe6AlGlrp7m+IsSSZJQUeERGRImbWDNwGnBgN/RF4srvviq8qmS0taYmIiOztKvaEnRHC0REKOzVOgUdERCRiZucB/1A09A53XxNXPVI6WtISEREBzGw54eiIedHQd4C/1dER9UGBR0RE5rzo6IifAE+Phh4BTnH3bfFVJaWkJS0RERF4L3vCTgG4QGGnvijwiIjInGZmTwM+UDT0QXe/Na56pDy0pCUiInOWmc0nHB2xPBr6OfDX7p6PryopB83wiIjInBQdHfFF9oSdbcCrFHbqkwKPiIjMVW8AXlz0/NXu/mhcxUh5aUlLRETmHDM7Efgd0BQN/Zu7vyXGkqTMFHhERGROMbNWQtg5PhpaDTzF3Ufiq0rKTUtaIiIy13yGPWFnmHB0hMJOnVPgERGROcPMzgdeXzT0Vne/N656pHK0pCUiInOCmR1BWL7qioa+CbxCR0fMDQo8IiJS98wsReixc1Y01Ec4OmJHbEVJRWlJS0RE5oIPsifs5AkzOwo7c4gCj4iI1DUzexbwnqKhK939N3HVI/HQkpaIiNQtM1tI2LdzSDT0Y+Acdy/EV5XEQTM8IiJSl6KjI77CnrCzGbhIYWduUuAREZF69RbghUXPL3H39XEVI/HSkpaIiNQdMzsV+A3QGA19xt0vi7EkiZkCj4iI1BUzawduB46Ohu4AznL30fiqkrhpSUtEROrNv7In7AwRjo5Q2JnjFHhERKRumNkrgFcXDb3J3f8UVz1SPbSkJSIidcHMVgB3Ah3R0HXufmGMJUkVUeAREZGaZ2aNwK3Ak6OhPwOnufvO+KqSaqIlLRERqQcfYU/YGSccHaGwI7sp8IiISE0zs7OBdxUNZdz993HVI9VJS1oiIlKzzGwJ4eiIJdHQfwM96qYsk2mGR0REapKZJYBV7Ak7GwndlBV25HFScRcgIiL1LZ3JthP64nQDTUADYZ/NKNAP3N/X2zM4g5e+DDin6PmF7r5xluVKndKSloiIlEw6k00AJwMnAGcCpwLLgLHoSyx6ePSAcPzDesIt5bcBa4DVfb09+5ypMbMnA//Lnn+4/5O7v7ukH0bqigKPiIjMWjqT7QLOBS4FFgFJQqAZIczkHEgT0EwIQ3nCyeYrgRv7ensGir/QzDoJx0WsiIZuA57q7uOz/yRSrxR4RERkxtKZ7HHAJcB5hH2hY8CuErx0C2HmpwB8F/hqX2/PvWZmwLXAK6OvGwBOdfcHS/CeUscUeEREZNrSmWwrcDlwEWFWZidhZqbUkoTOyQXg2kc/e+H6/ND2LxRdv8Ddv16G95U6o8AjIiLTks5kzwCuJtwdNUAII+WW8Pz4gtH1fz50+4/+PTm28QGAr7j7ayrw3lIHFHhEROSgpDPZFuCdhFmdHDCTO6tmxL1g41seORZLtFoiyfB9v9pSGNl57I7ffHtrpWqQ2qbAIyIiB5TOZDsJPW9OAnZQmVmd3XI7Nh6WHx4I/XYSCW9YsHxdItX4e+DiyZuaRaaixoMiIrJf6Ux2AXADcCKwnQqHnfyunV27ww6QbJv3aCLVuDGq54aoPpH9UuAREZF9imZ2rifcAt5f6ff33HhDfmBTeuJ5orFlR7Jt3qboaX9U1/VRnSL7pMAjIiJTivbsrCKusONOrn/DkV4opAAskRxPdS/tC3em7zYRelZF9YpMSYFHRET25QrCnp2Khx2A/ODWpYXxkY6J56muJQ9ZMpWb4kv7CXVeUbHipOYo8IiIyONEt55fRNigXHGF0eG2/OD2QyeeJ9u61yea23bu51t2ABelM9nTy1+d1CIFHhER2UvUVPBqQiPBip887oV8Mrdj41ETz62haSjZsfCxA3xbgVDv1Vrakqko8IiIyGSXE5oKVqzPzoSwb2fjEZ7PNQKYJfIN3UsfnLRvZ18GgaVoaUumoMAjIiK7RWdjXUjooFxxheH+hYXRoXkTz5Odix62VOPY/r5nkgHgwnQme2zpq5NapsAjIiLFLiH8bqj4UlZhfKQ5t3Pr8onniZaOLcnWzu3TfRlC/ReXtDipeQo8IiICQDqT7QJeRDgItKLcC5br33AU7gkASzWMpDoXPzLDl9sJvFi9eaRYKu4CyiGdybYDRwPdQBPQAIwDo4TbF+/v6+2p+Nq0iEiVO5dwOnk5Tj3fr/yOTcs9Nx42G5t5qmvpA5ZIzHSWKU/4HOcC15WoRKlxNX+WVjqTTQAnAycAZwKnAsuAiTVfix4ePQAagfXAncBtwBpgdV9vT8WncEVEqkH0s/QXwDxgVyXfO79roDvXv3HFxPNUx4K1yfb5m2f5si3ANuAZ+tkuUMMzPNHU67nApcAiQpp3YIRw1sv+DAFdwPOA5xP+NbA5ncmuBG7UQXQiMgedTPhZWtHlrHB0xOb0xPNEU2t/om3ebMMOhNC2mNCQ8A8leD2pcTUXeKI7CC4BziPsQRpjZn9BR6PHhHnA+4Er05nsd4Gv9vX23Du7akVEasYJhH84Vky4BX39UV4oJAEskRxLdS2ZfHTEbCQJn0uBR2pn03I6k21NZ7LvA24CXgoMEzprlmrqdVf0esPA3wI3pTPZ90UNuERE6t2Z7Fn2r4j8zi3LCuOj7bdd9RqedvxyUt1LHrJkqpT7h5zwuURqI/BELc5/RLjNcCdh43G5NtXlo9cfjN7vh2pVLiJzwKmELQEVURgdas8P9R8y8byprXNroqmt1DeTjACnlfg1pUZVdeBJZ7It6Uz2/cA3gAWEIFKpzWeF6P0WAt+IZnvUrlxE6k50Z+sy9l7m30t+187O3M6tiz031jjb9/N8Lpnr33jkxHMz8+aO7q2zfd0pjALL0plsWxleW2pM1e7hifonrCJsONtBDE2wIoOEYHgJcFo6k71Ym5pFpM4czZ47Wx+nMD7anOvf8ESA/OC25Ymm1u3Jtu5N1tg6OHm/zRVnH7P05U9evri1MZncOjQ2/qGb7374pacdumDjwMjYB2+++zF35ynLkk/4zLsvaTz9si9hiUQ+lUoVTj18XtuVPccfPr+tseGXf9rSf9kNf3h4ZLxQiiW2MeAY4I4SvJbUsKoMPOlMdgFwPbCCA99xVQkFQh0nAjekM9kL+np7yvGvERGROHTv76LnxpqKnxdGh+cVRofnWapxONnatTHR2rndLOHHLe1o+rszli9+0ed+dc+6/l3jRy1sa0wm9k5EhaHti8ZHW9onnic7F/WZ2fIXnLBswYVfvu3+odFcYdWrz3ziu5937LIP3nz3gQ4MLcnnk7mh6pa0opmdibDTH3M5k/UT6rpeHTxFpB6YWXJ8+/r5ns81FsZHmgujw22FkcGO/K6B7vxQ/4L84LZFhdGhrqm+13NjrbmBzUeObXzw1Pzg9oX5gtOQNDv+kM7mhqTZg1uGxv60aXD3Mpm7W25w22ETz5MtnZuSLZ39AF//3dpNa7cNj28dGst//ucPrD/7+KXzS/URCb3XZI6rqhmeaI/MKqoz7EyYCD2ropmeijboEpG5zcwSQBvQsY9H+36uTfVo2f4/X6D7Wa/Bx2b448zd8kPbDr1/07zVn/zBfY+87dlPPOTT55/c8ru+bQPvu/GPe46H8IIZ5tHn8GTXokcnLq3r37V7SW3ttuHRBe2NpQopCjwCVFngAa4g7NmphmWs/ekn1HkF8JGYaxGRKmZhk0sLswslxY82wi/xkvFCfnc7+pmyhpadANfftnbb9bet3dbV0pD49PknH/H+Fx5/2NBoLt/ckExYIllIzT/k/sMPXbY4lUp1miV2v+Wh3S27Q8ny+a2NWwfHpnNC+v44+9mfJHNH1QSe6NbziwgblGvBDuCidCb7/b7entvjLkZESsfMmphdKJkcbira0G+anEJ+F5ZotGRDDrOCWSJPIpHHrIAl8gbkd+1cNNU3WzI1mmxfsC7Z2rn9uKUdTYd2tzT+8s9bBneN5X0kV/CEwd3rB4Yv/sv0kgVtjesbu5rHLn3Oic02KbO9/MmHL/r+mg39w6O5whufsWLZj+7ZWKp/+CrwCFAlgSdq7nc1oQdOrZx5UiDUe3U6k32ulrZE4mNmKUoXUDoIBw5Xs2FCT7J9PQYPcL34Mbzk5R89Bfga4didx/FCITE58FgimUu2zVuXaOveMnGnVlNDMnHFOccc9q+vOLU5X3Bf89jA4BU3rH5469Bo7qwVCzt/+a5nnbRhx8jojasf23LhXxyxtPj1blmzftt1rznz6AXtTQ23/mlL/z/9973rS/F/VKRat0hIBVXF4aFRB+WLqc3/KLuBVX29PVraEjlIB7EPZTqzJx2EJaNqNsrsQsle3+fuJW28GvXhWc0+thO4O+Nb1h7vubEWzDzR1Lo90dS+HdzwQtIL+ZSPj7ZZQ9NQqmPhhlLWVgLzgJP6enumDHMyd8QeeKKzsW4k/OWvldmdYgnCD91zdfaW1Kt97EOZbiiZPFbNcswulOz1cPfxCtc/belM9lbCocpTNh/03Fjj+NZHjvNCYb8rA6nORQ8l27q3laPGGWgCdvT19jw17kIkftWwpHUJITTUYtiBUHeCMEOVibkWkd2ifSizDSbFj6prY1HEKVE4iR6jHve/BivvTuB57Cvw5HMNBwo7AJTu4M9SaAZ+GncRUh1iDTzpTLYLeBEzO+28muwEXpzOZHvVhVlmKtqHMttQUk/7UKbzGJ6DAaXUbgOev6+L1tg8bMnUqOdzTfv8moamoURLZ7XM7kC4m+22uIuQ6hD3DM+5hLsXynUQaKXkCZ/jXOC6mGuRCon2oewrgMxkA21zZT/BtE21D2Vae0+K/1zqfSgya2vY389i94SlGnftK/BYIpFrmLfsz5OPmohZnvC5ROILPOlMNgFcSv3cLjgGXJrOZK/v6+2p1eW5uhbtQ2mlNOGkg7DptppN7EOZaSipuX0oMiurgc2ETb677zp1dwpD/QvzQ9sO3d+SVrJj4VpLNuQqUOfBagE2AXfFXYhUhzhneE4GFlH7y1kTdgGLCQ0J/xBzLXUhCihT7UOZaUBpp/r3ocwqlEx6zMV9KDJDfb09hXQmuxJ4P1HgKYwMduR2blnuufH93gWXaGrdnmjprLaGsY3ASv0DVCbEGXhOoLqbcc1EkvC55mzgMbMGZh9Mih9xL7seyBClCSfahyLV4CbgvYXx0Zb8wOZDC2O79jpDy5KpsWT7/EcLY7s6C7t2LoSwlJXqWrK2ypayJrZK3BR3IVI94vxlciaz62RejZzwuWpmH4+ZJQnBpFRN2/a5obFKjFCacLITGNI+FKknD3/ihYmF52UGGpccdUZhbNeeBGNWSLZ2r0+2z99oiYQnmtp2jo+PtlLINSW7ljxkyVQ1LWVB+Fl0g24ikWJxBp5TCb98yqIplbBPn3/y4Wek53d2NKdSj/WPjH76x/c/+v3/W1/OvwAjwGllfP397UOZ6aO1nPWWwDilCScTG2W1D0VkkugOwUuBD/X/8toFi1/6PrAEeIFEc/vWVMfCdZZq2P13x5KpXOOiI+6Jr+L9mmhzsiruQqS6xBJ4oq6eyyjjIaGpZMLWD4yMv+ILv7mvb+vQWM+Jy7qu+tuTVty7fuCPD24ZKtdG6VFgWTqTbZvo6rmffSgzfbRT4oMDS6xAacLJxEbZKXuCiEhpmNk5wKeB4wFyWx9lcM1P6Djl+SOJ5raHEo0tw/FWOG2dhO73agQre4lrhudoynx31tBorvDR7D2PTTy/+a71O97xnKNHTzt8XuuBAo+7G4V8IrRML4T/9UKS4j97IUGhkAzPPcHEWLKhYevNV91ln3jhxF6Wdqp/H8petwszw3ASPXZpH4pI9TOzY4FPAS+YdKlv6P9+/N7up73qH8xsQQylzUY7sAG4Ku5CpPrE9Yu4uxJv4u4URoc6yY01Lepoaji0u6Xljnse6B7fvq0bLyRw3xNe3JN4IeHuSdxnPIOSaARLNR5Vys8xhV2UJpxM7EPRXQwic0QUYj4AvIm9bxwZBD4KXJ0b2DKSzmTvA75B7XTCTxA+z9t0mLNMJa7A00QFlmXyO7cckh/qX5ZKJviXN5zHt269m3seXFvmf7EYJB73f+sYpQsog+5ebRsERaTKRXdQvhH4IKHXzgQHvgS8z913H/zZ19tzezqTvYZw/E+13XI+lS7gq329PbfHXYhUp7gCTwMVCDyezzWawWdffw7juTzvue7gj1QxS+QxKxD+N2+JxO4/Y4lCuJ7Ik9j95wKJRN6SDc2dZ5z7zyMP3XEze/ah1EtzRRGpMdE+whcQlq+OmXT5Z8A73H1frTSuItyIcSLQX64aS6Cb0GBQS1myT3EFnnEqcEt6Q+fCdZ85/9mty+a1pl752R9u8+bOfNISReElUbBEFFwsUbBEMvqz+Sx6ShRaVpzxJ3fvK+FHERGZNjN7EmFD8tmTLj0AXAHcuL89d329PbvSmezFwA3ACqoz9HQTPs/FWsqS/Ykr8IxSgcDzLxc8edkxSzoKL1v56zW51gUHccxvSTj1c1yGiNQgM1sEfIhwq3lxd/EB4CPAZw/2Dsi+3p6BdCZ7AXA91Rd6JsLOBeq5IwcSV+Ap+1+YIxe0NZ53yqGLxnIFv+09zzl5Yvyj37/74a/9dm25T/Otph8IIjJHmFkj8GbC8RDFXZILwH8AH3D3TdN93b7enq3pTPZ8Qm+bk4AdxLuROUH4fHcRZnYUduSALI47iKM+PKupjY1w0zUPOGmiD4+ISLlF+3TOJexhecKkyz8GLnP3/5vt+6Qz2RbCUthFhKMbBmf7mjPQTrgb6xrgKi1jycGKJfAApDPZWwkJvZ4ayzUBO/p6e54adyEiMjeY2UnAZ4BnT7p0P3A5kC11b6x0Jns6cDWwlLBMVonZngShqeAGwq3nuhtLpiXOk6PvBJpjfP9yaAbuiLsIEal/ZrbYzFYSfpYWh51+4B3Aie7+vXI0Ao3CxtmEJa52wl6ach0GnSTMnLdH7/dchR2ZiTg7AN8GPD/G9y8HI3wuEZGyMLMm4G3AlYRu7hPywL8DH3T3LeWuo6+3Zxj4SDqTvQG4GHgJ4R/RY4TmqLPVAjQSZo++hY6LkFmKM/CsIfwFrSd5wucSESmpaJ/Oi4F/BiZ3c/8BYZ/O3ZWuKwohmXQm20vYR3QpsJgwM+OEQ5UPZutCE2GW3Ag/SzcBK4GbtClZSiHOwLMa2EyYqqyHTWcthL+gd8VdiIjUFzM7lbBP5xmTLt1LCDq3VL6qvUWh5Lp0Jns94U6uE4AzCY0Ll7GnXYdFD2dPe5JGYD3wE+B3hH843tXX21MLR1pIjYht0zJAOpO9kHD75I7YiiidLuBDfb0918VdiIjUBzNbCnwMeDV7d6ffRjgPa6W7j8dR23SkM9k2QpfnbkK4aSQEoDHCnqP7dGerlFvcp3jfBLyXMPVZy8tbE/XfFHchIlL7zKyZsPH4PYTNuhNywOeAD7t7ufuJlUwUZnRDh8Qqzru06Ovt2QHcyN4b72pRB/BdrTOLyGxY8DLCUtXH2TvsfA84wd3fXkthR6RaxBp4Il8l7MKvhlpmIkGof1XchYhI7TKzM4BfAt8Ejii69EfgHHf/G3e/L5biROpA7CGjr7fnHuBaQkOpWtQJXKvbJUVkJszsUDNbRdis+1dFl7YCbwJOcfcfxlKcSB2JPfBEPgVsZO/p21rQTuj6eVXchYhIbTGzVjN7H6Ej8kVFl8YJPxOf4O6fd/dcLAWK1JmqCDxRA6u3Ejb/VkVNByFBqPdtOstFRA5WtE/nFYR9Oh8GWosu3wg8yd2vcHcdQixSQlUTLqJW4dew9wm/1awLuEYtzkXkYJnZU4D/Ba4Hlhddugv4a3c/z93/FEtxInWuagJP5CrCX/zuuAs5gG5CnVrKEpEDMrPlZnYd8GvgKUWXNgOvB05z95/EUpzIHBFr48GppDPZTuAGYAWhIVW16QYeAM7Xbegisj9m1ga8C3gnoRv7hDHgX4CPu3s9NF4VqXpVF3gA0pnsAsKUb7WFnomwc0Ffb8/WuIsRkepkZgnglcAngEMmXf428C53f7DihYnMYVUZeGD3TM8qwpksOwi9buKSIOzZuQu4WDM7IrIvZnYWYfbmyZMu3Qm8w91/XvmqRKTa9vDsFoWKCwiNCTuI75b19uj9v0qY2VHYEZHHMbMjzOwbwK/YO+xsAF4LPFlhRyQ+VTvDUyydyZ4OXA0sBQaozGxPgtBUcAPh1nPdjSUij2NmHcC7gcuBpqJLo4R+Op9w951x1CYie9RE4AFIZ7KthB8oFxLCyE7Kc+BokhB08oQO0Fepz46ITBbt07mYcObV0kmXvwX8o7v3VbouEZlazQSeCelM9ljCD5mXEILPGFCKQNICNBJmj74DrNJxESIyFTN7OvAZ4LRJl35P2Kdza+WrEpH9qbnAMyHa1HwucCmwmDAz48AIYSr5QJqAZsAIszmbgJXATdqnIyJTMbOjgE8CL5106TEgA1zn7nHeYCEi+1CzgWdCOpNNEO7kOgE4k/AvrmWEmR8IgcYIYWjiwzYC64HbCQf2rQHu6uvt0Q8qEXkcM+sE3gO8g/DzY8IIIQB90t2H4qhNRA5OzQeeqaQz2TbgGELfnMboMRY9+oH7+np79MNJRPbLzJLAa4CPEmaSi10PZNx9bcULE5Fpq8vAIyIyW2b2LEI/nZMmXfotYZ/OrytflYjMVNX24RERiYOZPcHMvgv8hL3DzqOE7slnKeyI1J5U3AWIiFQDM+sGrgTeCjQUXRoG/gm4yt2H46hNRGZPgUdE5jQzSwGvAz4CLJx0+RrgPe6+ruKFiUhJKfCIyJxlZs8l9NN50qRL/wu83d1/V/mqRKQctIdHROYcMzvGzG4GfsjeYedh4OXAUxV2ROqLZnhEZM4ws3nA+4E3s/fPvyHCERGfcXcdJSNShxR4RKTumVkDoSv7h4D5RZcc+Apwpbuvj6M2EakMBR4RqWtm9jzg08Bxky79gtBP547KVyUilabAIyJ1ycyOBz4FPG/SpYeAdwLfcXVeFZkzFHhEpK6Y2QLgg8AbCYcKT9gJfAy42t1HYihNRGKkwCMidcHMGoE3AR8gnKM3wYEvAu9z941x1CYi8VPgEZGaZmYG9BCWr46edPmnhH06qytemIhUFQUeEalZZnYCYUPycyddegC4HLhJ+3REBNR4UERqkJktMrPPA6vZO+wMAFcAT3L3GxV2RGSCZnhEpGaYWRPwFuB9QGfRpQKwEviAu2+OozYRqW4KPCJS9aJ9Oi8CrgJWTLr8I+Ayd19T8cJEpGYo8IhIVTOzkwkHfD5r0qX7gcuA72vpSkQORHt4RKQqmdkSM/sCcCd7h51+4O3Aie6eVdgRkYOhGR4RqSpm1gy8DXgv0FF0KQ98Hvigu2+NozYRqV0KPCJSFaJ9Oi8FPgkcOenyLcAV7n53xQsTkbqgwCMisTOz04B/AZ426dI9wOXufkvlqxKReqI9PCISGzNbZmZfBn7P3mFnG/Bm4GSFHREpBc3wiEjFmVkL4Q6rDNBWdCkH/BvwYXffHkdtIlKfFHhEpGKifTovI+zTOXzS5e8R9uncV/HCRKTuKfCISEWY2ZmEfjpnTbq0htA48EeVr0pE5grt4RGRsjKzw8zsGuC37B12tgBvBE5V2BGRctMMj4iUhZm1Au8E3gW0Fl0aB64GPubu/XHUJiJzjwKPiJSUmSWAVwCfAA6bdPm/gHe6+58rXpiIzGkKPCJSMmb2FEI/nb+YdGk18A53/2nlqxIR0R4eESkBMzvczK4Hfs3eYWcT8PfA6Qo7IhInzfCIyIyZWTthj847geaiS2OEO7I+7u4DcdQmIlJMgUdEpi3ap3Mh0Assm3T5P4F/dPcHK16YiMg+KPCIyLSY2VMJszdnTLp0J/B2d/9F5asSEdk/7eERkYNiZkea2beAX7J32NkAvBo4Q2FHRKqVZnhEZL/MrINw5tVlQFPRpVHgKuCf3H1nHLWJiBwsBR4RmZKZJYFLgI8BSyZd/iZhn87Dla5LRGQmFHhE5HHM7BmEfjqnTLr0O0I/nV9VvioRkZnTHh4R2c3MVpjZt4GfsXfYeQy4CHiKwo6I1CLN8IgIZtYFvBd4G9BYdGkX8Engn919KI7aRERKQYFHZA6L9um8FvgosGjS5a8BGXd/pOKFiYiUmAKPyBxlZn8NfBo4adKl3xD66fy28lWJiJSH9vCIzDFm9kQzuxH4MXuHnUeAVwJnKeyISL3RDI/IHGFm3cD7gLcADUWXhoFPAJ9y9+E4ahMRKTcFHpE6Z2Yp4PXAh4EFky6vAt7r7usqXpiISAUp8IjUMTM7m3Du1fGTLv2KsE/n95WvSkSk8rSHR6QOmdmxZvY94AfsHXYeBv4OeJrCjojMJZrhEakjZjYf+ADwJvb++z0IfBz4jLuPxFGbiEicFHhE6oCZNQBvAD4EzCu65MCXgSvdfUMctYmIVAMFHpEaZmYGPB/4FHDspMs/J5x7dWfFCxMRqTIKPCI1ysyeRAg650y69BBwBfBdd/eKFyYiUoW0aVmkxpjZQjP7HLCavcPOTuBdwHHu/h2FHRGRPTTDI1IjzKwR+AfCpuSuoksF4IvA+919Yxy1iYhUOwUekSoX7dN5IWH56omTLv+EsE/nrooXJiJSQxR4RKqYmZ1EOODzrydd+jNwOXCzlq5ERA5Me3hEqpCZLTaz/9/evcfWWd93HH//jm+x48TOHY9ADEVQuSP9HwAADpRJREFUaABxSze6rWMaUjtLrEVjatkKbKuENLEybi1mUNau5aw00NG13VIxNUDXamPiEs3tVtaqakc72FIuiQphsB5RIPf71bfz2x8/Z7PPQhLHPs9zzvH7Jfkfvs55vk+M5E9+z/f5/VYBzzEx7OwGbgbeFWNcY9iRpOPjCo9UQ0IIbcDHgDuBueNKZeBvgLtjjNvy6E2S6pmBR6oBY3M6HwBWAqdXlL8D3BJjXJ95Y5LUIAw8Us5CCBeQDvh8b0VpA+nx1bd9dCVJU+MMj5STEMJJIYQHgbVMDDs7gRuBc2OM3zLsSNLUucIjZSyEMAu4CbgD6BxXGgW+DHwqxrgjj94kqVEZeKSMjM3p/DZwL9BbUf4WcGuM8aWs+5KkmcDAI2UghHAR8JfAL1eUfgrcHGP8l+y7kqSZwxkeqYpCCL8QQlgN/CcTw8520jER5xt2JKn6XOGRqiCE0E7aCbkf6BhXGgG+CHwmxrgzj94kaSYy8EjTaGxO50PA54BTKsprgNtijK9k3pgkzXAGHmmahBDeTdpP55cqSutIB3x+N/uuJEngDI80ZSGEpSGER4B/Z2LY2QpcD1xg2JGkfLnCI52gEMJs4Dbg40D7uNIw6Y2sz8YYd+fRmyRpIgOPNEkhhAJwNfAXwMkV5ceAj8cYX8u8MUnS2zLwSJMQQriUtHpzSUXpedKczvczb0qSdEzO8EjHIYSwLITwTeBpJoadzcBHgYsNO5JUu1zhkY4ihNAJ3E7aU2fWuNIgcD9QjDHuzaM3SdLxM/BIRzA2p3MNcA/QU1H+B+ATMcZS1n1Jkk6MgUeqEEL4FdKczoUVpbWkOZ0fZt+VJGkqnOGRxoQQTgshPAr8gIlhZyNwHbDCsCNJ9ckVHs14IYS5wB3ATUDruNIh4PPAvTHGfXn0JkmaHgYezVghhCbg94HPAosryt8Ebo8xvp55Y5KkaWfg0YwUQriMdO7V+RWlZ0hzOj/OvitJUrU4w6MZJYRwRgjhceB7TAw7bwK/B1xq2JGkxuMKj2aEEEIXcCdwI9AyrnQAuBdYGWPcn0dvkqTqM/CooYUQmkk7IX8aWFRRfgS4I8b4RuaNSZIyZeBRwwoh/AZpTmd5RenHwJ/EGJ/NvitJUh6c4VHDCSGcGUJYAzzFxLDzOvAh4D2GHUmaWVzhUcMIIcwDPgncwMT/t/cDReD+GOPBPHqTJOXLwKO6Nzancz1pTmf+uFIEVgN3xhjfyqE1SVKNMPCoroUQ3gfcB5xTUfohaT+dtdl3JUmqNQYe1aUQwtmkoPP+ilIJuBV4LMYYs+5LklSbDDyqKyGEBcDdwB8BTeNKe0lHRDwQYzyUR2+SpNpl4FFdCCG0kELO3cC8caUIPAjcFWPcnEdvkqTaZ+BRTQshBOA3SY+vzqoof580p/N81n1JkuqLgUc1K4SwHLgfuLyi9BppTudJ53QkScfDjQdVc0IIi0IIXwFeYGLY2UMKOu+KMT5h2JEkHS9XeFQzQgitwB8DdwFd40pl4KvAJ2OMW/PoTZJU3ww8yt3YnM4VwErgjIryvwI3xxjXZd6YJKlhGHiUqxDC+aQDPi+rKL0C3AIM+OhKkjRVBp461ts/0AmcCXQDbUALMAwMAruAV0rFvn35dfj2QghLgD8HPgqEcaVdwKeAr8QYh/LoTZLUeIL/eK4Pvf0DBeB80unfK4ALgB7gcCgIY19x7AugFdgIPAc8C6wHXigV+8rZdT5RCKENuBG4E5gzrjQK/DXwZzHG7Xn0JklqXAaeGtfbP9BFmm+5HlhE2l04AodIKznH0gbMIoWhUWArsAp4slTs21ONno9kbE7nSuDzwGkV5X8Gbokx/jSrfiRJM4uBp0b19g+cDVwHfIC0fcAQcHAaPrqdtPJTBh4HVpeKfS9Pw+e+rRDChaQ5nV+tKL1ECjrfrub1JUky8NSY3v6BDtKw7jWkVZm9pJWZ6dZEeqRUBh4B7isV+w5M5wVCCD2k862uY+Kczg7SERGrYozD03lNSZKOxMBTQ3r7By4GHgCWkDbZy2LWpgDMBTYBN5aKfWun+oEhhHbgJuAOYPa40gjwJeDTMcadU72OJEnHy8BTA3r7B9qB20irOiNAHm9WdZJWfR4GVpaKfZN+fDY2p/M7wOeAZRXlfwJujTFumGqjkiRNloEnZ739A3OBh4DzgN1ks6rzdgqkHY5fBK6dzFBzCOES0pzOeypK60kbBz41bV1KkjRJBp4c9fYPLAC+AbyDtP9MregmHdB5danYd9RXxEMIJwP3kFanxttGOiLiwRjjSFW6lCTpOBl4cjK2svMotRd2Djsceq460kpPCKGDdJDnJ4COcaVh4IvAZ2KMtXhfkqQZyMCTg7GZnW8A51KbYeewbmAdaaXnIPzvnM6HSXM6Syu+/wngthjjq5l2KUnSMRTybmCGupU0s1PLYQdSf+eR+iWE8IvAj4C/Y2LYeQH49RjjBw07kqRa5FlaGRt79fwa0oByPdgdR0f+oP30iy4F3ldR2wL8KfC1GGM19gqSJGla+EgrQ2ObCj4FLCCfV88nJZbLhdF9208qDw8uKR/YXdjy93cRR4Yg7fr8BeCeGGNmx1NIknSifKSVrVtImwrWdNiJMTK6f9eC4a2l5aP7d/XEoYOFps75zFnxQYB/BM6OMd5u2JEk1QtXeDIydjbWk6Swk+deO0dVHjwwe2TP1lPjyND4N68ILW0HmrsW7yi0zLq82mdvSZI03Zzhyc51pBW1mgw7cWSodWTPtqXlwf3zxv/3UGgabuqc/2aho2t7CKEbuBboz6dLSZJOjCs8GejtH+gCniGddl5Tw72xPFoY3bu9Z/TgniXE+H8HfIYQmzq6NjV1zt8UCk2HQ1oT6bT1FZPZhVmSpLy5wpONK0hhoWbCToyR8oHdC0b37Tg5lkdbxtcKbbN3Ns9d+EZobh2q+GOjpPu4Avh6Vr1KkjRVrvBUWW//QAH4ATCPtMKTu/Kh/Z0je7ed8v/mdJpbDzTPXfR6oa1j/1H+eDuwA3hvqdhXk4/nJEmq5ApP9Z0PLAL25t1IeWSodXTP1qXlwQNHmtN5o9DRtSNtpHxUB4HFpA0Jn69Sq5IkTSsDT/UtJz0Gys1R5nTKTR1dm5s6F2wKhcJkVmuaSPdl4JEk1QUDT/WtAKr63PCrH7notIuXzZ8zq6XQtGP/0PDXflTa9Lf/9rNtaU5n18LRfTtPjuXRCT/rwqzZO5rnLHwjNLcOn8AlI+m+nOORJNUFA0/1XQAcquYF/up7r258ZfPe0uBIOZ7TM3fW1//w3Wf95NW3eGbdhsVxZLh9/PeGlrb9zXMW/vwYczrHcgi4cGpdS5KUHXdarqLe/oFOoAcYrOZ11r25+9DgSDkCjAwPtoyMjDQtah5cNj7shELTUHPX4p+1LDjl5SmGHUj309PbPzB7ip8jSVImXOGprjNJ505V3f1Xnbfs/ct7Fra3NrOutIXvvlhKhRDKTR3dm5o652+e5JzOsQwBZwE/mcbPlCSpKgw81dWdxUXKQwc7bvjS490hRi4+o4dL37mUoZFRCrM6tzfPWfhmaG45kTmd45HJ/UmSNFU+0qquNuCY73lP1ei+HT2xXG4ux8iz//UWSxd1D99w5WWbW+b1lKoYdgLQWqXPliRpWhl4qquFDAJPoW32TkKIoal5sLlr8X/P7e7effqS7mpf18AjSaobBp7qGqbKr6QvmdvW/Lu/dm5ccOqZL7QvOX193yVnjV5+9pL5T7+2rdobHUYymk+SJGmqnOGprkGqHHhihA+vOHXxnX3nLCsEwuY9g4Mrv7Ph5088/9aual4XA48kqY4YeKqr2qGDLXsHR37ry09vqPZ13kbV70+SpOngI63qeoXGnXNpBfIKWpIkTYqBp4pKxb59wEbS21qNpA3YWCr2TXUDQ0mSMmHgqb7ngFl5NzHNZuGGg5KkOmLgqb5nyeDV9IwF0n1JklQXDDzVtx4YzbuJaTZKui9JkuqCgaf6XgC2Au3H+sY60Q5sAV7MuxFJko6XgafKSsW+MrCKxnlbqxVYNXZfkiTVBQNPNtaQHgM15d3IFDWR7mNN3o1IkjQZBp4MlIp9u4EngTl59zJFc4DHS8W+PXk3IknSZBh4srMaKFO/f+cFUv8P5d2IJEmTVa+/fOtOqdj3EvAIMDfvXk7QXOCRUrHv5bwbkSRpsgw82boP2Ax05t3IJHUCm4CVeTciSdKJMPBkqFTsOwB8jDT8Wy9/9wVSvzeWin0H825GkqQTUS+/dBtGqdi3FngY6Mq7l+PUBTw81rckSXXJwJOPlaSN+7rzbuQYukl9+ihLklTXQowx7x5mpN7+gbnAo8A7gF05t3Mk3cBrwFW+hi5Jqneu8ORkLERcTQoVtbbSczjsXG3YkSQ1AgNPjkrFvu3AVcA6YB75/zwKY32sI63sbM+5H0mSpkXev2BnvHErPatJOxnn9cp659j1V+PKjiSpwTjDU0N6+wcuAh4ATgL2kHY2rrYCaVPBTaRXz30bS5LUcAw8Naa3f6ADuAX4CCmM7CUd2DndmkhBZ5S0A/RK99mRJDUqA0+N6u0feCdwLXAlKfgMAdMRSNqBVtLq0WPAQx4XIUlqdAaeGjf2+voVwPXAYtLKTAQOAYPH8RFtwCwgkFZztgCrgDXO6UiSZgoDT53o7R8oAOcBy4EVwIVAD2nlB1KgCaQwdPiH2gpsBNYC/wGsB14sFfuymA2SJKlmGHjqWG//wGzgLNK+Oa1jX0NjX7uADaVi3/78OpQkqTYYeCRJUsNzHx5JktTwDDySJKnhGXgkSVLDM/BIkqSGZ+CRJEkNz8AjSZIanoFHkiQ1PAOPJElqeAYeSZLU8Aw8kiSp4Rl4JElSwzPwSJKkhmfgkSRJDc/AI0mSGp6BR5IkNTwDjyRJangGHkmS1PAMPJIkqeEZeCRJUsMz8EiSpIZn4JEkSQ3PwCNJkhqegUeSJDU8A48kSWp4/wMwNTiGDvMzaQAAAABJRU5ErkJggg==\n", 312 | "text/plain": [ 313 | "
" 314 | ] 315 | }, 316 | "metadata": {}, 317 | "output_type": "display_data" 318 | } 319 | ], 320 | "source": [ 321 | "import warnings\n", 322 | "\n", 323 | "import numpy as np\n", 324 | "import networkx as nx\n", 325 | "import matplotlib.pyplot as plt\n", 326 | "import matplotlib.cbook\n", 327 | "\n", 328 | "warnings.filterwarnings(\"ignore\", category=matplotlib.cbook.mplDeprecation)\n", 329 | "warnings.filterwarnings(\"ignore\", category=RuntimeWarning)\n", 330 | "\n", 331 | "default_graph = nx.DiGraph()\n", 332 | "stack = []\n", 333 | "\n", 334 | "def add_node(node, info):\n", 335 | " node_index = len(default_graph.nodes) + 1\n", 336 | " default_graph.add_node(node_index, node=node, info=info)\n", 337 | " return node_index\n", 338 | "\n", 339 | "\n", 340 | "def constant(array):\n", 341 | " def wrapped(*args, **kwargs):\n", 342 | " node = ConstantNode(\"\".join(str(a) for a in args))\n", 343 | " node_index = add_node(node, *args)\n", 344 | " stack.append(node_index)\n", 345 | " value = array(*args, **kwargs)\n", 346 | " return value\n", 347 | " return wrapped\n", 348 | "\n", 349 | " \n", 350 | "def primitive(func):\n", 351 | " def wrapped(*args, **kwargs):\n", 352 | " result = func(*args, **kwargs)\n", 353 | " node = OperationNode(func, args, kwargs, result)\n", 354 | " node_index = add_node(node, func.__name__[:3])\n", 355 | " parents = stack[-len(args):]\n", 356 | " \n", 357 | " for parent in parents:\n", 358 | " default_graph.add_edge(parent, node_index)\n", 359 | " stack.pop()\n", 360 | " \n", 361 | " stack.append(node_index)\n", 362 | " return result\n", 363 | " return wrapped\n", 364 | " \n", 365 | "\n", 366 | "def wrapped_numpy_operator():\n", 367 | " for function in [np.add, np.subtract]:\n", 368 | " globals()[function.__name__] = primitive(function)\n", 369 | "\n", 370 | " globals()[\"const\"] = constant(np.array)\n", 371 | "\n", 372 | "\n", 373 | "wrapped_numpy_operator()\n", 374 | "result = add(const(1), subtract(const(3), const(2)))\n", 375 | "print(\"Forward pass\")\n", 376 | "print(f\"Result is {result}\")\n", 377 | "print(\"Corresponding computational graph is \")\n", 378 | "plot_graph()" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "metadata": {}, 384 | "source": [ 385 | "### Register VJP after loading\n", 386 | "\n", 387 | "- We need to look up the corresponding VJP of given operation while backtracking.\n", 388 | "- Register all VJP at loading time in `__init__.py`\n", 389 | "\n", 390 | "We can easily validate our result with numerical calculation (which can be used in test), which is\n", 391 | "$$f'(x) = \\lim_{\\epsilon\\to 0} \\frac{f(x + \\epsilon / 2) - f(x - \\epsilon / 2)}{\\epsilon}$$" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 4, 397 | "metadata": {}, 398 | "outputs": [ 399 | { 400 | "name": "stdout", 401 | "output_type": "stream", 402 | "text": [ 403 | "Upstream is 1\n", 404 | "Operator is negative, x is -6, result is 6\n", 405 | "Downstream is -1\n", 406 | "Numerical result is -0.9999999999976694\n" 407 | ] 408 | } 409 | ], 410 | "source": [ 411 | "from collections import defaultdict\n", 412 | "\n", 413 | "import numpy as np\n", 414 | "\n", 415 | "# Register the VJP in memory\n", 416 | "primitive_vhp = defaultdict(dict)\n", 417 | "\n", 418 | "def register_vjp(func, vhp_list):\n", 419 | " for i, downstream in enumerate(vhp_list):\n", 420 | " primitive_vhp[func.__name__][i] = downstream\n", 421 | "\n", 422 | "register_vjp(\n", 423 | " np.add,\n", 424 | " [\n", 425 | " lambda upstream, result, x, y: upstream, # w.r.t. x\n", 426 | " lambda upstream, result, x, y: upstream, # w.r.t. y\n", 427 | " ])\n", 428 | "\n", 429 | "register_vjp(\n", 430 | " np.subtract,\n", 431 | " [\n", 432 | " lambda upstream, result, x, y: upstream, # w.r.t. x\n", 433 | " lambda upstream, result, x, y: -upstream, # w.r.t. y\n", 434 | " ])\n", 435 | "\n", 436 | "register_vjp(\n", 437 | " np.negative,\n", 438 | " [\n", 439 | " lambda upstream, result, x: -upstream, # w.r.t. x\n", 440 | " ])\n", 441 | "\n", 442 | "# This is the numerical way to calculate the derivatives\n", 443 | "\n", 444 | "epsilon = 1e-4\n", 445 | "\n", 446 | "def numerical_vjp(func, arguments, wrt):\n", 447 | " args_pos = [args + epsilon/2 if i == wrt else args for i, args in enumerate(arguments)]\n", 448 | " func_pos = func(*args_pos)\n", 449 | " args_neg = [args - epsilon/2 if i == wrt else args for i, args in enumerate(arguments)]\n", 450 | " func_neg = func(*args_neg)\n", 451 | " return (func_pos - func_neg) / epsilon\n", 452 | " \n", 453 | "func = np.negative\n", 454 | "x = -6\n", 455 | "upstream = 1\n", 456 | "result = func(x)\n", 457 | "vjp = primitive_vhp[func.__name__][0]\n", 458 | "\n", 459 | "print(f\"Upstream is {upstream}\")\n", 460 | "print(f\"Operator is {func.__name__}, x is {x}, result is {result}\")\n", 461 | "print(f\"Downstream is {vjp(upstream, result, x)}\")\n", 462 | "print(f\"Numerical result is {numerical_vjp(func, [x], 0)}\")" 463 | ] 464 | }, 465 | { 466 | "cell_type": "markdown", 467 | "metadata": {}, 468 | "source": [ 469 | "### Backward Propagation" 470 | ] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "execution_count": 5, 475 | "metadata": {}, 476 | "outputs": [ 477 | { 478 | "name": "stdout", 479 | "output_type": "stream", 480 | "text": [ 481 | "Backward pass\n" 482 | ] 483 | }, 484 | { 485 | "data": { 486 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAIuCAYAAAC7EdIKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdebxcdX3/8dd3Zu6duy+52dkmRBLCvomCIK0sChdTFhFLZbFaKUWxVay9tVbUtpe22AoU+6PVyiZWERB0BBQqIpSC7LIGEiaE5Gbh5u7L3Fm+vz++Z5JJcvdZzszc9/PxmEdmzsyc85nk5t73/a7GWouIiIhIJQv4XYCIiIhIoSnwiIiISMVT4BEREZGKp8AjIiIiFU+BR0RERCqeAo+IiIhUPAUeERERqXgKPCIiIlLxFHhERESk4inwiIiISMVT4BEREZGKp8AjIiIiFU+BR0RERCqeAo+IiIhUPAUeERERqXgKPCIiIlLxFHhERESk4inwiIiISMVT4BEREZGKp8AjIiIiFU+BR0RERCqeAo+IiIhUPAUeERERqXgKPCIiIlLxFHhERESk4oX8LqDYIh3RBmAF0AKEgSogAcSBXmBNrLN90L8KRUREJN+MtdbvGgom0hENAIcDhwDHAkcCS4Ax7yXGu1nvBlANdAHPAk8CLwLPxzrb08WrXERERPKpIgNPpCPaDKwGLgUWAEFcoBnFteRMJQzU4MJQCtgG3AjcE+ts7y9EzSIiIlI4FRV4Ih3RVcAlwFm48UljwEgeTl2La/lJA3cDN8U621/Nw3lFRESkCCoi8EQ6onXAF4CLcK0yA7iWmXwLAo244HMr8M1YZ/twAa4jIiIieVT2gSfSET0GuBZYBPTjwkihBYAmYDPwuVhn+9NFuKaIiIjMUtkGnkhHtBb4Iq5VJwn4MbOqAdfqcwtwTayzPR/dZyIiIpJnZRl4Ih3RJuBm4DCgj+K06kwkADQDLwAXa1CziIhI6Sm7wBPpiLYBtwPLcevmlIoWYC1wQayzvdvvYkRERGSnsgo8XsvOHZRe2MnIhJ7z1NIjIiJSOspmawlvzM7NlG7YAVfXcuBmr14REREpAWUTeIArcWN2SjXsZPTi6rzS70JERETEKYsuLW/q+Q9w6+uUwxYPAdx6PR/TlHURERH/lXzg8RYV/CXQhj9Tz2erAXgHOFXT1UVERPxVDl1aX8AtKlhOYQdcvYtR15aIiIjvSrqFx9sb6x5ceCiHrqzdBXAtPau195aIiIh/Sr2F5xJcjeUYdsDVHQAu9rsQERGRuaxkW3giHdFm4AncbueF2Ai0WIK43daP1do8IiIi/ijlFp7VuLBQzmEHXP1B3OcRERERH5RkC0+kIxoAHgFacS085a4W2A6cFOtsL9fuORERkbJVqi08hwMLqIywA+5zLMQtSCgiIiJFVqqB5xBcN1AlCeI+l4iIiBRZqQaeY4HS62vLjcV9LhERESmyUg08RwKjhThxrLP96BULG8I+nG8UOCpf1xUREZHpK7nAE+mINgBLgLjfteRZHFgS6YjW+12IiIjIXFNygQdYAYxN9aKDlzbV/OTP3rfyxas+eMTDV/7ewX9w+NJmgJ/82ftWfvKEZfMzr7vkuEjbzz5zwkqAey9/n/vzMycc9MrXPnTk+cfs0/qBlQsbn/rrUw770gdXLn7+b087/ImOkw/9o/fsOy/z/pmeb4qyx4CV0/6bEBERkbwI+V3AOFqmekFV0Jj/vPCYd93z3MZ3Pnrj42vef8D8huv+8Mh3rdky8PJk71t9w2OvxTrbj179b4++vGbrYBzgAysXNrbWV1e11leHjv2HB184fnlb/b//0dEHPLO+Z+iVzQOTtjKNd758fD4RERHJr1Js4QkDZrIXHL98fn1NVTD4z794bfNYKm0ffHXrwONru3vPPXrvttle9BvRVzbFk2n7q9e2DT6+rrvv7CP3mjf1u2bMANUFOK+IiIhMohQDTxVTBJ7FTTVV2wbiY+mseVyb+kbHFjXWVM3mgoPxZHIontyxIGBX78jYwlmeawoKPCIiIj4oxcCTYIop6Zv7RxMLGsPVgaxYtKS5pnrLwGhiNJFK1VYFd3yuBY3hKYNLQzgUqg+HdrxncXNt9daB0QTAbM43Ccs0xieJiIhIfpVi4IkzReB5fG330GgilfrCqSsXVwWN+cDKhY3HL29rueuZjdtf3TwwcsqqRa111cHAioUN4bOO2Gt+9nu3D40lly3Ycxr5l89YtbQ6GDAnrVjQcPzytuafPLepB2C255uAAo+IiIgPSjHw9E71grFU2l5629NvnHDA/ObnvnLa4V/98EH7/s1PXnzz5a7+0Rt+9caWZCqdfurLpxz+Lx89Ytl9L3Vtz37vjY+s3dR59qGRF6/64BEf9WZV9QyNJfpHEsnffvmUw/75I4ct+/ufv7L+5a7+UYDZnC/XzyciIiL5VXKbh3rr8DwP9BTjeh9YubDxn849bNkx//DgC0W4XCtwWKyzfagI1xIRERFPybXwxDrbB4Eu3GytorPWkhoZaE6PDjbm+dRhoEthR0REpPhKLvB4ngVq/LhweqS/Ndm7+V2Jnq4VyYF3Fufx1DXAM3k8n4iIiExTqQaeJ5lianq+/M9rWwd2687acd3UYM9eqZH+fC0UaHCfS0RERIqsVAPPi0DKjwsHaht7AlU1g5nHyb6ty9JjI7V5OHUK97lERESkyEo18DwPbAPyETRmxJiADbUuWWuCIbdVhLWBZE/XATaVyGUbjlpgK1CMgdEiIiKym5IMPLHO9jRwIz6tSmyCoWSodckbxgRSADadqkr0dL3LptOz7WarBm70PpeIiIgUWUkGHs+9uG6goB8XD1TVjAabF67LPLaJeH2yd3NkFtP4g7jPcW8+6xMREZHpK9nAE+ts7wPuAfI9PXzagrWN/cHGtg2Zx+n40LzUQPeSGZ6mEbg71tnen9/qREREZLpKNvB4bgLS+FhnsL51a6C2cVvmcWqoZ2lqeNoztwK4+m8uSHEiIiIyLSUdeGKd7a8AtwJNftVgjCHUvGhDoKpmIHMs2b91WXpspG4ab28Cbo11tr9auApFRERkKiUdeDzfBLYADX4VYIwZb+bWu2wyMdnO6Q3AZuCaYtQoIiIiEyv5wBPrbB8GrsAN/vWtXhMMpUKtS3ebubXpXTadHq+mAK7ez8U620eKWqiIiIjsoeQDD0Css/1p4Bag2c86AlXh0VDLop0zt5JjdRPM3GoGbvHqFhEREZ+VReDxXINbuC9fWz3MSqCmoT/U2PZW5nE6PtSaGngne+ZWC65OdWWJiIiUiLIJPF7X0MXAWnwOPcGGedt2nbnVuzQ13NeKq2stcLG6skREREpH2QQeAG8tmwsogdATal60IVC9c+ZWOj68LDUysA24QGvuiIiIlBYzi5WDfRfpiDbh1rY5DOjDrXVTdDadCia6315lQtXhsa0xun/+r5ttfPgYa+1GP+oRERGR8ZVl4AGIdERrgSuBi3BbNwxO/o6CaLDJRE3fEz9e2P/4HbU2OQbwDHCitXbYh3pERERkHGUbeDIiHdGjgWuBxUA/xWntCeAWFdwMfG791We2AA+wc9+vHwPnW2u1WaiIiEgJKKsxPOPxpn6fhuviasCN7SnUhqNBoNW7zs3AqbHO9qettQ8Bn8l63UeArxaoBhEREZmhsm/hyRbpiB6Im8l1Di7MjQH5mC1VC1TjWo/uAm4eb7sIY8z17Bp8LrDW/iAP1xcREZEcVFTgyfAGNa8GLgUW4lpmLDAKxKdxijBQAxjc+KCtwI3AvZPNwDLGhIAorsUJ71onWWufmN0nERERkXyoyMCTEemIBnAzuQ4BjgWOApbgWn7ABRqDC0OZv4hqoAt4Gvgt8CLwQqyzfVrjcYwxLcD/ASu9Q5uBY621G3L9PCIiIjI7FR14xhPpiNbjwkgLLtxU4wLQGNALvBbrbB/K5RrGmHcBT+LG+wA8B5xgrc3pvCIiIjI7cy7wFIsx5veBXwAh79BdwHmauSUiIlJ8ZT9Lq1RZa38F/FnWoXOAr/tUjoiIyJymFp4CM8Z8C/hc1qGPW2u/71c9IiIic5ECT4F5M7d+CnzIOxQHft9a+7h/VYmIiMwtCjxFYIxpBh4HVnmHtgLvtta+5V9VIiIic4fG8BSBtbYP+DDQ7R1aCNxrjGnwryoREZG5Q4GnSKy1a3EDlxPeocOB24wx+jcQEREpMP2wLSJr7SPAZVmH/gD4e5/KERERmTM0hscHxphvAp/POnSxtfYWv+oRERGpdAo8PjDGBIF7gTO8Q2O4mVv/619VIiIilUuBxyfGmCbgf4GDvUPbcDO31vtXlYiISGXSGB6fWGv7cTO33vEOLQB+aoxp9K8qERGRyqTA4yNr7ZvA2eycuXUocLvX5SUiIiJ5osDjM2vto8Cnsw6dCXT6VI6IiEhFUuApAdbam4B/zjr0RWPMJf5UIyIiUnk0aLlEeN1Yd+PG9YDr5vqA1wIkIiIiOVDgKSHegOXHcGN5wA1oPtYb6yMiIiKzpC6tEmKtHcC18Gz1Ds3Hzdxq8q8qERGR8qfAU2K8dXjOxi1GCG6dnh9o5paIiMjsKfCUIG/F5T/JOnQG8E8+lSMiIlL2FHhKlLe31tVZhz5vjPmkX/WIiIiUMw1aLmHGmABwJ3CWdygBnGqt/bV/VYmIiJQfBZ4SZ4xpAB4FDvcOdQPvsdau9a8qERGR8qLAUwaMMfsCTwKLvEOvAMdZa/v8q0pERKR8aAxPGbDWvoXr1op7h1YBPzTGhApwuSBwOnAf8DLwP8AtwMXo60VERMqUWnjKiDHmj4Dbsg5da6398zydvgr4K+BzQDXQAJis54eAN4ALgd/l6ZoiIiJFocBTZowxfwd8OevQpdba/8jxtMuAn3p/1k3yujSulemfgK8B+uIREZGyoMBTZryZW3cA53iHksBp1tpfzfKUJwH3AvW47qzpGMZtdnrVLK8pIiJSVAo8ZcgYUw/8BjjSO9SDm7n1+gxP9SngOqB2FmUMA1cA353Fe0VERIpKgadMGWP2Bn4LLPYOvQa811rbO423B4FrgU8weRfWVEaADwGP5HAOERGRglPgKWPGmGOBXwM13qFfAO3W2uQkb2sC7gGOJbewk9GPa2lal4dziYiIFISmGZcxa+2TuFaajNOAf5nkLfsDzwPHkZ+wA27sz0NAc57OJyIikncKPGXOWvvfwNezDn3WGHPZOC89EXgG2AcI57GEILAE12qkHd1FRKQkKfBUhq/hZm5lXG+MOSXr8R8D9+NaYQoRSsLAu3HjgkREREqOxvBUCGNMHW7w8NHeod558+Yd193dfRluNla+urAmMwz8BZDrukAiIiJ5pcBTQYwxe+Fmbi1paGjg/vvvHz7++OMzYahYRoAzgIeLeE0REZFJKfBUGGPMMZFI5NGHHnoovGTJEmprZ7PEzk7WWr773e9SVVXFH/7hH1JdXT2dtw0AR+G2ohAREfFdITafFB9Za8NjY2OpQCBAKJTbP286neZb3/oWV111FYODg7z22mucf/75HH744VO9NTNz63BgOusCiYiIFJQGLVeWS4BfVldX1+UadgACgQC1tbWsWLECgGuvvZavfOUrPPzww1O+FViE259LoVpERHynLq3KEACuAS4lT4OTrbUYYxgbG+Oxxx7j61//Or/+9a+pqqqiubmZe+65h+OOO26q0wwDtwDjTZMXEREpGgWe8tcA3AUcj+tKyptM6AEYGhpi9erV/OY3vyGZTNLa2srzzz/P3nvvPdVphoEvAt/OZ20iIiIzoS6t8rYv8BxuUcG8hh1gR9gBqK+v57LLLmPx4sUEg0F6enq4/PLLp3OaOlzr0wfyXZ+IiMh0KfCUr+Nw20RE2LmXVkH09vby/e9/n29961u8/fbbpFIpjj/+eE4++WRGRkamc4pa4CfAAYWsU0REZCLq0ipPFwL/jyIsJrhp0ybuvPNOrr32WtatW0cgEOD000/nU5/6FGeeeSbB4LQXbk7jpqkfBsQLVrCIiMg4FHjKSwC4GricAoWd7HE7a9as4bbbbuOGG26gp6eHxsZGzj77bD75yU9y4okn7vH6aRgGbgD+shC1i4iITESBp3zUAHcCJ1GA8TrZrLU89dRTfOc73+G73/0u6XSapUuX8rGPfYxPfOITHHzwwTteN4OwkzEIzEetPCIiUkRaI6U81AEP4PbJym3p5CnE43EeeeQRvv3tb3PPPfcAsGrVKj7+8Y9z4YUX7piVNcuwA2CBs4H/zlfNIiIiU1ELT3n4D9y4nZwHJ2eCyniBpa+vj/vvv5/rrruOxx9/HIDjjjuOSy65hPPPP5+mpqZcL5/xJPCefJ1MRERkKgo8pe8Y3C7oObXsWGt56aWXWLduHaeffjpVVVWkUqkdg467urq48847ue6663jjjTcwxnDGGWfwx3/8x6xevZpgMJhLq87utgNt+TiRiIjIdKhLq7QFgdvIQ8vOfffdx6c+9Smam5sZGRnh/PPP3xF2Xn/99R2Dk7dv305DQ8OOwcnvf//7gZy6sMbTna8TiYiITIcCT2l7L7AUyDlphMNhNm/ezObNm7nppptYvHgxJ510Es888ww33ngj//Vf/0UqlWLJkiWcf/75fOITn+DQQw8F8h52ALbk82QiIiJTUeApbUtxg3xzdvLJJ3PddddxxRVX8MADD9DU1MQrr7zCQw89xJ133gnAgQceuGNw8j777AMUJOyMAP+bzxOKiIhMRWN4SttngH8ijzOzPvvZz3LDDTdQU1NDa2srXV1dALz3ve/dMTi5ubk5X5cbz3bc6tADhbyIiIhINrXwlLZa8rz9x/XXX8+aNWv45S9/ybZt2wA45ZRTuOKKKzj11FMJh8OFaNXJGAI+jcKOiIgUmfbSKm13k6curV1OevfdLF++fMeu56tWreL0008nHA6TTCYLFXZGgF/jdnYXEREpKgWe0vYG8GK+T1pXV8fPfvYzwuEwPT09PPnkkzvG8YRCIQrQzTkM/Aq34KD6UEVEpOgUeErfv+K2Y8irlStX7lhJ+YknnuB73/seDzzwAEC+W3iGgFuBPwDG8nliERGR6VLgKX13ADEgle8Tn3baaVx//fUAPPDAA3zve99j/fr1+bzEMG6j0D8Fkvk8sYiIyEwo8JS+BK51ZKQQJ7/88sv59Kc/DcABBxxATU3OaxwCpHEDk1cD387HCUVERHKhaenl40O43dLrCnHyu+++myOPPJJIJJLrqeLANuBkYE2uJxMREckHBZ7y8nng60C934VMYBh4DjgT6PG5FhERkR3UpVVe/hXXyjPsdyHjGAZ+APweCjsiIlJi1MJTfqqAR4EjgGqfa8kYAb4EXO93ISIiIuNR4ClPbcALwGJ8bKVLp9M2EAgMAecAv/SrDhERkamoS6s8deMGBfvWtTU6OsrmzZv52te+9nEUdkREpMSphae8nQb8hDxuLjodw8PD9oUXXjBnnHEGPT09G4BjrbWbi1mDiIjITKiFp7z9Avgr3GrGxTI8MDBwz/vf//7+np4egH2Au40xeVnAR0REpBAUeMrf9cAPKU731gjw14sWLTonkUh8DLfAIMB7gf80Bdp1VEREJFcKPOXP4rZueB636F+hrjGIG5x8LWCttfcBX8h6zcdxrU0iIiIlR2N4KkcrbubWUvIbZMfYOUj6lewnvBadG4E/yTp8rrX2rjxeX0REJGcKPJVlBfAU0Jin8w0DLwGn40LPHowx1cADuAUHM+85wVr7bJ5qEBERyZm6tCrLGly3Uz42Gh3Crep8AhOEHQBr7RjwEWCtd6gOuNcYsyQPNYiIiOSFAk/leRD4IrnN3BoBvgpcjOvSmpS1thu3f1afd2hv3Mytok6XFxERmYgCT2W6AbiVmYce673nI8A3vcfTe6O1rwIfZefMrfcA39XMLRERKQUKPJXrcuBeph96RoHNuKDy89lc0Fr7C+DPsw79IfDl2ZxLREQknzRoubIFgSuAv8NtOlo1zmtSuOns9wOfAPpzuaDXovNt3FT5jPOstT/O5bwiIiK5UOCZG/YC/gM4BdeSYwGDC0A/BP4NeDpfFzPGVOEC1Ae8QyPAidbavF1DRERkJhR45pYFwBJgEW7/rYco0LYUxph5wP8BB3iHNuL23NpUiOuJiIhMRoFHCsYYsxIXelq8Q08BJ1lrfdvlXURE5iYNWpaCsda+BpyHGycEcAzwPc3cEhGRYlPgkYKy1j6IGzid8VHgb30qR0RE5ih1aUlRGGP+DTdVPuNj1tof+lWPiIjMLQo8UhTGmBBwH26mGLjZYu+31v7Wv6pERGSuUOCRojHGtAKPAyu9Q13Au621G/2rSkRE5gKN4ZGisdb2AB8GerxDS3Abjdb5V5WIiMwFCjxSVNba13F7dSW9Q0cBNxtj9LUoIiIFox8yUnTW2v8BPpN16CPAVf5UIyIic4HG8IhvjDHXsuuU9QustT/wqx4REalcCjziG2/m1s+AD3qH4riVmJ/wryoREalECjziK2NMM27m1irv0Gbcnlsb/KtKREQqjcbwiK+stX24mVvbvUOLcTO36v2rSkREKo0Cj/jOWrsWOJedM7eOAG7VzC0REckX/UCRkmCtfRi4LOvQ2cA3/KlGREQqjcbwSEkxxvwL8BdZhz5urf2+X/WIiEhlUOCRkmKMCQL3Amd4h+LA71trH/evKhERKXcKPFJyjDFNuJlbB3mHtuJmbq33ryoRESlnGsMjJcda24+budXtHVqIm7nV4F9VIiJSzhR4pCRZa9cB5wAJ79BhwG2auSUiIrOhHx5Ssqy1jwB/mnXoD4C/96kcEREpYxrDIyXPGPPPwJVZhy621t7iVz0iIlJ+FHik5Hkzt34CnOkdGgM+YK19zL+qRESknCjwSFkwxjQC/wsc4h3ahpu5FfOtKBERKRsKPFI2jDER4ElggXfoReB4a+1ALueNdEQbgBVACxAGqnCDpeNAL7Am1tk+mMs1RETEXwo8UlaMMScADwHV3qGfAWdZa1PTeX+kIxoADse1FB0LHAkswXWTARjvZr0b3rW6gGdxgetF4PlYZ3s6188jIiLFocAjZccYczFwU9ahf7bW/uVk74l0RJuB1cCluBaiIC7QjOJacqYSBmpwYSiF61K7Ebgn1tneP8OPICIiRabAI2XJGHM18KWsQ39srf3e7q+LdERXAZcAZ+GWYRgDRvJQQi2u5ScN3A3cFOtsfzUP5xURkQJQ4JGy5C1AeBdubR5wY25Ottb+BiDSEa0DvgBchGuVGcC1zORbEGjEBZ9bgW/GOtuHC3AdERHJgQKPlC1vq4nHcKswg9uK4tj9/upn84BrgUVAPy6MFFoAaAI2A5+LdbY/XYRriojINCnwSFkzxuyHG0i80ISqaT7hgm1Nx54zYAKBBODHzKoGXKvPLcA1sc72fHSfiYhIjhR4pOwZY44z1XUPzz/zL6qrFkTApvtCrUvfMMb4VVIAaAZeAC7WoGYREf8p8EjZi3RE25J9Wx+2qeQh6fgQAMG6pi2h5kVv+1xaC7AWuCDW2d491YtFRKRwtHmolLVIR7QJuD3UvLDWBEObM8dTw/2LUkM9830sDdyihcuB2706RUTEJwo8UrYiHdFa4GZcqOgNNi3YGAjX9WaeTw5075seHWr0rUAnE3pu9uoVEREfKPBIObsSN0OrF8AYQ6hl8ZsmVO0GCltrkn2bl6cT8bCPNYKr7zB23fFdRESKSIFHylKkI3oMbo2dvuzjJhBMV7Uued0EgkkAm04Hkz1d77LpVNCPOrP0ARdFOqJH+1yHiMicpMAjZcdbVPBa3EKCe6yxY0LViVDL4jcwxgLYVKIm2dO1v88D9NO4eq9V15aISPEp8Eg5+gJuUcEJ19kJhOuGQk0LYpnH6bGRplTf1n2KUNtkBoHFqGtLRKToFHikrHh7Y12IW0F5UsG65u3B+pauzOPUSP/C1KDvM7f6gQsjHdEDfa5DRGROUeCRcnMJ7ut2WttFBBvnbwqE63oyj5ODvs/cSuPqv9jHGkRE5hwFHikbkY5oM26z0IHpvsfN3FoSM6Fqt6HnbjO30ol4ODXc12rTqWL+XxgAztbaPCIixaPAI+VkNW6fqhntem4CgXRV69I3TCCYgJ0zt1KD2xckujccnOzbun+yb8u+hSh4Ainc51hdxGuKiMxp2lpCykKkIxoAHgFagVltyJmOD9clejYdiLV7brJlTLp60fLnjDezqwhqge3ASbHO9mLs5i4iMqephUfKxeHAAmYZdgAC4brhUOP82LhPWhuwYyPFnC4+AizELUgoIiIFpsAj5eIQXDfQrNlkoio13Lt4oufTYyPFHswcxH0uEREpMAUeKRfHAjl1NyUH3tnLJhMTtuLYxGgDwP7z66tjne1HhwJ79nwB/E37qqX/ceHRy3KpJXNJ3OcSEZECC/ldgMg0HQmM5nICEwgkJ3veJkYbizymbRQ4qpgXlKIKAitw/8bvAQ7E7avWBawF3vRuMSZZRFNE8kOBR0pepCPaACwBeqZ67WSCTQvfJhBKpEf6F9pUsnr35206HbSJeC00zGgWWA7iwJJIR7Q+1tk+VKRrSnEcDkSBZlxLXgOQ3WQ4ivv3N0Cd93gTsA54CXiDnWFoPTmGfRFR4JHysAIYy/Ukf/nBAxeffeRe81vqqkJb+kYS//Dj/0397IlXagLG8DcfPYHzTziI4URq5e1Pvr0x+33LF9RX/8tHj1h2wMKGule6+gdj3cPxXGvJMgasBJ7J4znFX5cAN+Bm4o3fLwo13i2jAfd1vgL4IG5QewLXSlSLW7tpIy4IvcjO1qG1wFvk2N0rMhco8Eg5aMnHSdZ3D8XPu/Hx17r6RhLnHb1P6/WfOiXyu3eSa05e3rTvqUcsqzn1qtttunHxa9/75PG77Ll1/ceO2v+Fjb2DH73x8TXH7d9W/+0/OuqAR994pzcfNXny8vmkJKzGhZ26HM6RafXJ1uLdDgY+jAtESaDKe/4Z4O+AX6DwIzIuDVqWchBm4t+Up+1HT7/ds7F3JJG28MOnNvRs7B2Jv3fF0uBHTjgk8aNnt7z1TtWiZ3rijPz7r9dtzrxnv6m0aj0AACAASURBVLa66pWLG+u/EX1lUzyZtr96bevgo2u2DqRT6YC16Zxrwn2uPbrXpCwZ4J/ILexMRwCox3WX1Xm3E4AfA68AZ5GH/y8ilUYtPFIOqsjDN/CL3rtf28XHRxYtaqqpBqitCgbn1YdD8xvCVZv6RsaMcZd4q3toR5fVXi21VYPxZHIonkwDpPq37v1abGPz0nkNjG1ee5QJBBMEQ3ETrIqbYGjM/VkVN6GqMYJVO845CQWeynEasJeP12/AdY/eBvw1cJ2PtYiUHAUeKQcJcmymX9ZWX/2VMw/a75M3/3bNY290D6as5aHPn3SQMdA9FE/s1VK7I3TsM68unLnf1TuaaAiHQvXhUGAonkynE/H6vdoayczmsulUFelUlU3EG/a4qDHWBIJjJhgaIxDaEYR2BKJAKOGt7Jzz+CQpCV/FhQ6/1QNXA08C/+dzLSIlQ4FHykGcHANPfTgYsBa2DcQTAJ84PtIWaauvBbj/xc09Fxy778Kfv7i5dyieTP/pSct3LE74ZvfQ2JotA0NfPmPV0qvufWnj8Ucf9M5pR+7f8Ivn3szsej4xa41NJcM2lQwDey5qaIwNhOsTvQ9/72vm6jNXs3NWTmZA6narvV/KRYDSWmKgFvgpsC85rE4uUkkUeKQc5DxA+MVN/aO3P/nWlh9detyqtMXe92JX94ub+gYBvvPom9uWzW+oiX7mhIOHx1KpWx6PbT5q39YdAeWzP3h23b+ef8Sy5//2tCNe7uobfODlrdsam5qD1YuXx2wyUU0qEbbJRLVNJcI2lay26WSYVDJs06nJ/39Za0gnq5O9W45l/AUIB4wxMXZdr2XHfWttf65/L5I3y3GDiMNTvbCIqnEh7DG/CxEpBdo8VEqetw7P8+S4Dk+x2XQ6YFNj1TaZCNuUF4xci0816WTYptPBQE0DXf/1GWxiVjPdt7NnEMo8jllr9Zt98ZwO/AA3kLhUjABfAq73uxCRUqAWHil5sc72wUhHtAv3wySfa+AUlAkE0iZQM0pVzbiLxtl0qtYm4mM2Ef86EAGWebfM/fopLjHPux097vWN2cxurUJZ9zdYazV2KH+WMYvB59ZapjGwfbZqcbO3FHhEUOCR8vEs8CGKGHgK/MMIEwhWm3DdQ9bau/d4zl14PuMHocz9qX7ALvZu7x3nubQxZiMTdJcBG621xVpxuhIcgAsYM2KMYXR0lK6uLkZGRqiqqqKtrY158+blq66V+TqRSLlT4JFy8SSu26AoUkO9bcmBd/YLVNX0h+bt9UaBgo/Bfa49eIOVt3m33+7xRmMCuDAzURjal8l3lw8A+3i394/zfMIY8xbjh6E3gS0aUL2LVTN9w+joKJdddhldXV0YY+ju7mbdunVs374dYwyHHHIIV199NaefntOX/UAubxapJAo8Ui5eBIrW4pAeHWzFWpMeG2lOjw42B2sb+wpwmRTuc82YtTaN23tpE+MMSjXGhIC92TUIZQejpUy+tlEVbiDu8gmeH51kQPWbQM8cC0QT/T1NqK+vj3vuuYfe3l3H5FdVVZFIJNiyZQujo643NIfWxm2zeZNIJVLgkXLxPO6bdytFmGZrQtWjjI00gws/BQg8tcBW4IU8nxcAa20Sb/Ay8PDuzxtjwrhWoPHC0DJgwRSXqMHt/n3gBM8PGGMm6i5701pbaS0PS2b6hqGhIU488UQA6uvrSSaTDAwM8PLLL7NhwwaWLFnC4sWLpzjLpCxu5WURQYFHykSssz0d6YjeCPwtRQg8gZqGntRw3yIAOzbcYq013iKB+VIN3BjrbE/n8ZzTZq2NA697tz0YY+rZGYAyf2bfn2r/r0bgMO823vm7mbi7bH2ZzTBrYhbT0ffZZx+uv/56AoEA8+fPp6amhq6uLs477zw2bNjAvHnzaGnJaZu1YdzmoiKCAo+Ul3uBL+PGphS0e8tU1w6ZQDBh06kqm04HbXyowdQ05KtVIlP/vXk6X95Za4eAl7zbHowxLUwchpYx9X5Sbd7tmAnOv5mJu8s2WGsTM/g4hRbBhfA9F5ecRFVVFfvuu+8ux9588022bXO9UNmBZ5bdWUnc35eIoMAjZSTW2d4X6YjeA5xLHhYjnIwxhkC4vic10r8QIDU62BrIX+BpBO6IdbaX7cKB1tpe3My5Z3d/LmuG2XhBaBmwH9OfYXbcOM+ljTFvM3EL0aYizzBbBuTUUpcZo9PX10d3dzcAbW1tNDbOKEPtLoT7OxERFHik/NwEnI2bZVTQ7qBATcOOwGPjwy3W2rfyMFsrU/fNuZ6oVO02w2yPWWjeDLMlTNxCtA9TzzDb17udNM7zCWPMeiZeg2hrngdUR3BjmmYt83W1fft2+vrccLG2tjbq6nLaeL0G2JDLCaRyeQu6rsB1T4dxExUSuKU/eoE1sc72Qf8qzD8FHikrsc72VyId0VuBiyl0K0+4btAEAkmbTodsOlVlx0bqTbhuKMfTNgE3xzrbX81HjeXIm2G20bs9uvvz3gyzfZi4u2zpFJeoAt7l3cYzkjXDLPNndjCa6QyzleRhS4lUKsW2bdtIpVLU1NQwf/58AoHJt2ubQg+uW0vmuEhHNAAcDhyC28bmSNwvHZnFR413s+zct7DaW/D1WdwvLi8Cz/s17jAfFHikHH0TtwhhG1Cw30CMMZhwfa8dGZgPkB4daA3kFngagM3ANXkpsEJ5M8wyAeRXuz9vjKlh5wyzCHvONJtqhlktbt2cidbO6fdmmMXYMwy9aa3d/WtuxmvwjCcej7N161YAGhsbmT9/PpDTlHS17sxxkY5oM7AauBT3/yKICzSjTL1VzxBudfsP4dZASwHbvMkj95Rjl7wCj5SdWGf7cKQjegXw3xS4aytQ09CTzgSe+HCrtfbtWf7wCeC+2Xwu1tleTjOQSo61dhRY4932YIxpwAWfCOO3EE2131UT7rfhwyc4/ztkhaHe3t5Dm5tz30JrZGRkR+Bpamqira0t11OOOwNPKl+kI7oKuAQ4C/e9Z4zZLUIZZ9fV7VtxM2X/JtIRvRu4qZxaqxV4pCzFOtufjnREb8H9py7YpqKBcN2AMYGUtemgTSWrbWK0zlTXDs/iVM24bw5P57tG2ZXXAvMiEyzqaIxpZfItO6YaODPfux0DEA7nZ4P0oaEhtmzZAkBLS0uugSfFBDPspHJFOqJ1wBeAi3BdVAPkd0briHcLAh8BzvGGGHwz1tk+m++LRaXAI+XsGuAo4FAKNJ7HmIA14dpeOzrUBpAeGWgNzDzwtOAWGFRXVgmw1vbgQvJEM8wWMHF32S4zzObNm5fzfmuZLquBgYEdU9JbW1tpbW3N5bQjaEr6nBLpiB4DXAssAvop7KSOFO57bgA3nvKDkY7o50r9FzoFHilbsc72kUhH9GLgDtzS/gUJPYGaht50JvC4bq2NM/gh14Jb/O1idWWVPm+w8lbv9sTuz3szzJbihaGTTz75+FQq9UncQOlZyXwt9fb27rIGT6abbJaBKoUCz5wQ6YjWAl/EteokKfBkjt2kvevNB/7ba3W/plS/1+U0BUDEb97AuQtwoSKnZWknEgg39GFMGsCmEmGbiE93CnIm7FxQjgP8ZE/W2rS19m1r7aPW2lt/9KMfPVRXV5eXb+49PT309Lje2TyswVONAk/Fi3REm4Dbca0s/RRwEscUBnHdZ5cAt3t1lRwFHil7sc72buA84He4QXV5/bo2gYANVNft2EsrPTowVV9DwKvjd8B5Xn1SmZbhZn3lrLu7O29r8KRSqZpQKPQFY8xnjTEfNsYc4g3mlgoR6Yi24Vq3D8V10fo9XTzt1XEocIdXX0lR4JGKkNXScxNuJeO8fnMP1NTvGBidjg9NFngavOvfhFp25oIDyaE7KyOZTLJt2zastdTV1e2Ykj5bW7duNalU6i+A63BbmPwOt6HrNmPMk8aYHxpj/tEY86fGmA8aY1Z40/2lDGS17BSsKz8Hvbi6Sq6lR2N4pGJ4/cbfiHREf44bvLeYPA3eC9Q09NG/zWKtsclEbToRDweqwtnTNQO46cybcVPPS3rwnuTNRLvFT8lai7WWQCDA8PDwjhlajY2NLFiwYMdrZjOGZ/369RM9lZlh9u7xnjTGbGLiLTs2eGskiY+8MTs3U5phJyMTem6OdEQvKJUxPQo8UnG8Keun4aZnXogLIzlNzzSBYDpQXduXjg+3gLcIYVV4M256ZpN37psp4QF7UhD7zfaNxphdwsw777wDQHNzM/PmzQPc6svBYHDGoaexsfE3wH3sOttsP6ZujVrq3d43znOprD3Mxtuyo8tbRVsK60rgMAq4HEee9OLqvBL4hs+1AAo8UqG8NSG+EemI3oEb0HcOOxfgmlUgCYTrezOBxybG5nnnSQM/Yo5vFzFHGdwU4BlJJpM8+OCDbN++ncWLF7N48WK2b9/O22+/Dbg1eJYsWQJAKDSrb9HJgw8++OfW2qt3KdaYIDv3MBtvDaK9mXyYQxAXmvYDfm+c58e8PczGC0MxYFue9zCbc7yp5xcBfVO9tkT0ARdFOqI/L4VWb6OvP5kLvL7kzBLrC9l1ifX4JG/NCNt0ui7Zt2UZNm1SQ70Eahs/Vz1/35s0TmfOWgysY4aDlnt7ezn33HN54YUX2GuvvWhubiYQCPDSSy/xzjvvcMABB3DRRRdxxBFH0NDQQGtrK3vvvfeOVp9pGAD+BPjhTOoyxlTh9jCbaFPXJTM53ziGcMEnxvhbdpRq90xJ8BYV/CUF3lKnABqAd4BT/W79VuCROcXbRO8wdm6idxTT2EQP6AKe3v7gf/z+6Fu/OzSxNQbYP7fWXlvE8qW0HIfrNprRvhLr169n1apVjI6Ojvt8KBSisbGRlpYWWlpaCAaDfPrTn+bjH/84NTXTGlfcB3yQcdYRyoUxphbXuhNh/BaiXGfl9DLxDvcxa22uG/eWtUhH9CsUYdPkAmnBtYL72rWlwCNzXqQjWo/b8boFF26qcQFoDPfN5bVYZ/sQgDHm08CN3lsfsdaeVPyKpURcAPw/3Ky8abPW8uabb7J27VreeOMNXn/9dV577TXWrVvHhg0bGB7ecyHvL33pS1x11VXT3cZiFBdMts6krlwZYxqZOAwtY4Z/T+PYxsTdZeuttdNpqS1L3t5Y9+BadspxnFQA19Kz2s+ufwUekRkwxizCtfZkWoGWWGu3+FuV+OSvga/jukfzZmBggI0bN7Ju3bodYejMM8/kjDPOmO4pxoAadrZQ+s7bsqOVifcvi5DbekYW2MQE3WXA2+U8wyzSEf1H4FzKs3UnowW4I9bZ3uFXAQo8IjNkjPk18H7v4Z9aa2+c7PVSsb6Pa+UpNRuAff0uYia8QLSIiVuI9iW39Y5SuL+X8VqI3gQ2l+oMs0hHtBnXPTlCfjcCLbYgLtQe69e4R83SEpm5O9kZeM5lZxeXzC0r/C5gAjG/C5gpb/bWZu/2f7s/780wW8rELUTTmWEW8W6/P87z8awZZjH2DEbv+DjDbDWu/nIOO+DqD+I+z21+FKAWHpEZMsbsA7zlPUwCi6y1230sSfzRhZupVWr+E/i030UUkzGmmslnmOX675SZYTbuGkTW2oJME/cmWTyC6w6shPW9aoHtwEmxzvait6iphUdkhqy1G4wxT+JmeYWAD+MWHZS5pRT3phoDXvO7iGKz1o7hNupdO97zWTPMJlqDaKo5//XAwd5tvPP3MvGCjOtzmGF2OLAAt9RAJRjBLQtyGPBcsS+uwCMyO3fiAg+4bi0FnrlnWlOmimwU7ZK+B2vtCPCqd9uDMaaJnQEo82f2/almmLUAR3q38c6/lfHHDsVwgWhsvPfhls/I66D4EhDEfS4FHpEycSfwj97904wxjdbaSvktTKZnI+4HYqlR4Jkha20/8IJ324U3oHoeE4ehCG5W3GQWerf3jHd5Y8xGxhk7tPcV3z85UNtkZ7OfGkA6EQ8n+7YsMyaQDDbNfztQVbPL4k+X/97yBX9wxF7zl82vr33wlS3bL/v+M7FZXWhmLO6XxaKP41HgEZkFa+1aY8zzuCbnMHAGM1zZVsrek5Re4KmlDActlzJvsHK3d3tq9+eNMQGmnmE22c9agxt0vTdwQvYTie6NmJoea6xNmEAoTjAUN8GqMROqiptgVdwEq8YIhhITBaLUUM9im4jXWyDd/XZTsL51Y7Bh3pbM6zf3jyb+/ddru046YEFTuCow2aDvfBrFLfhadAo8IrN3Fy7wgOvWUuCZW36DG7+Vy/ox+Ran9DeVrCjedPYu7/b47s97M8z2YvIZZnskFlNVQ7ChlfTooLFQbVPJahLjdK0ZY00gFDfB0JgXiOImGBozwaq4TY7t/Nq01qQGt++djg+1hpoXvRmoCsfvfGZjL8DhezfXLaqqqc7172Ka4sCSSEe0PrOga7Eo8IjM3p3A17z7Zxhjar2xAjI3RIGrp3xVcb3kdwGyK2ttCjer8y3g17s/780w25fdglD1ov0PwaYPYqoxPNYam0rU2FRiWvuO2ES8PtG94eBgbdOWYNOCjbPtLsvRGG51+2eKeVEFHpHZexk3I2YlbhbHabjl32VueBP4JPBfQJ3PtYCbAXOf30XIzHgDlt/wbjtEOqIfAG6w6fSITY1V22QiTCpRbVPJsPX+JJ0M23R65oOarTWp4b7F6bGRpuoF+72Sn08yYy3FvqACj8gsWWutMeYuILNU+rko8Mw1PwR+D7gQF3r99m9+FyB5EwaMCQTSJlAzym4DjjNsOhW0ybFqm0qEbTJRjQtEYS8YTdrqY1MJv7pjDW7PwqIq1iAlkUp1Z9b91V7ztMwtlwOX4NaAGfSphjjwHdzAWqkMVYwztmd3JhBMBaprR4K1Tb2hxratoZZFG6ra9n4j2DR/w+RvNDZY37IpX8XOkAKPSBl6hp2zYpqBD/hXivgkDfwYOAD4GPAgbjXZMaDfu020zko+WNyK350FvIYUX4JcNoAdp6vLBAKpQG3jtqp5S1+rXrT8mZrmBZtrqgImYIwJGGNqqgImFCjKmB5LYf9PjEtdWiI5yOrW+rx36Bzgfh9LEv9Y3EDmqPd497VbDgRW4QaoLsLtLRTH/eJZy+y+H48C24DTcbOEpHLEySHwBMJ1AyZUPUw6GTbVtX2BmsbtgZr6fmMCO875V6cfuPRTJ+y/JPP41a+fPu87j67r+rvoK4Vu+fEl8GgvLZEcGWPeBzzqPdwGLPFmZohMZLzdwVfhBsDvC8zH/cBL4Gbp1LGzRX4E98OiHvg5bvyQL7tPS+FEOqJHAd/H7eNVaeqBP4p1tmuWlkiZeRz32/US3L43JwIP+1mQlLxJdwfHhZzM7uARdoagBtyS/M8Az1OZPwzFWYMb51KJ/8bV+LDnm1p4RPLAGHMD8Gfew+uttVf4WY+IlL9IR/RR3NjAuN+15FEY6It1tp8w5SvzTIOWRfIje7bWOd5y8yIiuXiWqffpKjc1FHnBwQx9UxbJj0fYOSV4L3bupC4iMltPMo2p6WXG4D5X0SnwiOSBtTbJrosOnutXLSJSMV7EzearJCnc5yo6BR6R/Nm9W6vSfjMTkeJ6Hjfzs5Q2qM1FLbAVeMGPiyvwiOTPQ+ycHrw/O3dSFxGZsVhnexq4ER9WJS6QauBG73MVnQKPSJ5Ya+PAz7IOqVtLRHJ1L64baOabhJaWIO5z3OtXAQo8Ivm1S7eWb1WISEWIdbb34cYHNvpdS44agbtjne2+LZKpwCOSX/fjVsIFOMgYc6CfxYhIRbgJt2dbuf7MDuDqv9nvIkQkT6y1w8B9WYfUrSUiOYl1tr8C3Ao0+V3LLDUBt8Y621/1swgFHpH8y+7WUuARkXz4JrAFt71IOWnAbaFyjd+FKPCI5N/P2LkT8JHGmGV+FiMi5S/W2T4MXIEb/FsuP7sDuHo/F+tsH5nqxcUoRkTyyFrbD/wy65AGL4tIzmKd7U8Dt+D21yoHzcAtXt2+U+ARKYy7su6rW0tE8uUa3MJ9LX4XMoUWXJ2+d2VlaLd0kQIwxrTh+tsza2fsba3d6GNJIlIhIh3RJuAOYDnQ63M542kB1gLn+TkNfXdq4REpAGttN/Bw1qGzfCpFRCqMFyIuwIWKUmvpyYSdC0op7IACj0ghqVtLRAoi1tneDZwH/A5oxf+f5wGvjt/hWna6fa5nD+rSEikQY8wSYCNgcItuLbbWbvO3KhGpJJGOaC1wJXARbuuGQR/KaMB1398CXFMKM7LGo8AjUkDGmEeB93kPP2Wt/a6f9YhIZYp0RI8GrgUW4zYxLsYGnQHcooKbcVPPS2I21kT8bgITqXRahFBECs4LG6fhtm9owI2lKdSGo0Fc91WDd71TSz3sgFp4RArKGBMB3vQeJoCF1tpSnFUhIhUi0hE9ELgYtwZYALcQaj66mWqBalzr0V3AzX5vFzETCjwiBWaMeQo42nt4obX2Nj/rEZG5wZu+vhq4FFiIa5mxwCgQn8YpwkANbhxiCtgK3AjcW2ozsKZDgUekwIwxHcA/eA/vttZq5WURKZpIRzQAHAYcAhwLHAUsYecWOMa7We8GriWnC3ga+C3wIvBCrLO9GGODCkKBR6TAjDErgNe8hyPAAmvtkI8licgcF+mI1gMrcWN9qr3bmHfrBV6LdbZX1PcpBR6RIjDGvAgc7D08z1r7Yz/rERGZazRLS6Q4smdrqUtLRKTI1MIjUgTGmMOA572HA7hurekMGhQRkTxQC49IcfwOeMO73wic6mMtIiJzjgKPSBFY15SqvbVERHyiLi2RIjHGHAs84T3cjttbK+FjSSIic4ZaeESK57fA2979ecBJPtYiIjKnKPCIFIm6tURE/KPAI1Jc2dPTzzbGFGpzPxERyaLAI1Jcj+H2owFYBBznYy0iInOGAo9IEVlrU8DdWYfUrSUiUgQKPCLFlz2O5xxjjPGtEhGROULT0kWKzBhTBWwBWr1D77bWPuVjSSIiFU8tPCJF5q29c2/WIXVriYgUmAKPiD+yZ2udq24tEZHCUpeWiA+MMTXANqDBO3SYtfZ3PpYkIlLR1MIj4gNr7SgQzTp0jl+1iIjMBQo8Iv7ZpVvLtypEROYAdWmJ+MQY04Dr1qrxDq2w1r7uY0kiIhVLLTwiPrHWDgIPZB1SK4+ISIEo8Ij4K7tbS+N4REQKRF1aIj4yxrTgurVC3qH9rLVv+ViSiEhFUguPiI+stb3AQ1mH1MojIlIACjwi/lO3lohIgalLS8RnxpiFQBfuFxALLLXWbva3KhGRyqIWHhGfWWu3Ar/xHhrgLB/LERGpSAo8IqVBixCKiBSQurRESoAxZi/gbe9hClhord3uY0kiIhVFLTwiJcBauxH4P+9hEFjtYzkiIhVHgUekdNyVdV/dWiIieaQuLZESYYzZH1jrPRwDFlhr+30sSUSkYqiFR6REWGvXAc95D6uBM3wsR0SkoijwiJQWzdYSESkAdWmJlBBjzEHAS97DYVy31rCPJYmIVAS18IiUEGvty8Cr3sM64IM+liMiUjEUeERKj/bWEhHJM3VpiZQYY8xRwNPewz7cIoRjPpYkIlL21MIjUnqeBWLe/WbgZP9KERGpDAo8IiXGumZXdWuJiOSRAo9IacoOPGcZY0K+VSIiUgEUeERK0xPAJu/+fOBEH2sRESl7CjwiJchamwbuzjqkRQhFRHKgwCNSurK7tc42xuj/q4jILOkbqEjp+g3Q7d1fCrzHx1pERMqaAo9IibLWJoGfZB1St5aIyCwp8IiUtl02EzXGGN8qEREpY1ppWaSEGWOqgW1Ak3foKGvtsz6WJCJSltTCI1LCvC0lfpp1SN1aIiKzoMAjUvp26dbyrQoRkTKmLi2REmeMqcN1a9V5hw6y1r7iY0kiImVHLTwiJc5aOwz8POuQWnlERGZIgUekPNyVdV+BR0RkhtSlJVIGjDFNuG6tau/QcmvtOh9LEhEpK2rhESkD1tp+4BdZh87xqxYRkXKkwCNSPtStJSIyS+rSEikTxpg2YAsQ9A7tY61928eSRETKhlp4RMqEtbYb+FXWobP9qkVEpNwo8IiUl+xFCDWOR0RkmtSlJVJGjDGLgU2AAdLAYmvtNn+rEhEpfWrhESkj1trNwGPewwBwlo/liIiUjZDfBYjIjN0JnODdPxf4z8wTkY5oA7ACaAHCQBWQAOJAL7Am1tk+WNRqRURKgLq0RMqMMWZfYD0Yqhftn1xw3lc/E2qYdwhwJLAEGMu81LtZ7wZu4cIu4FngSeBF4PlYZ3u6mJ9BRKTYFHhEykykI9rc/Yt/f7b+oJOWBeuaCda3bg2E6/qBUVxLzlTCQA0uDKVwKzjfCNwT62zvL1jhIiI+UuARKRORjugq4BLgrNRIf1tqsKfNJscIhOt6q+bttTaHU9fiWn7SwN3ATbHO9ldzr1hEpHQo8IiUuEhHtA74AnARrlVmIJ2IhxLvvHUIAMbY6oXLnjOBYK7dUkGgERd8bgW+GetsH87xnCIiJUGztERKWKQjegzwS+BiYAA38DgVqArHTahqBABrTXp0sDkPl0t55x/0rveLSEf06DycV0TEd2rhESlBkY5oLfBFXKtOEhdCdpHs37Y0NdS7BCAQru+pmrc037unN+BafW4Brol1to/k+fwiIkWjFh6REhPpiDYBt+NaWfoZJ+wABGoaezL302PDzdamTZ5LGcS1Kl0C3O7VJSJSlhR4REpIpCPaBtwBHAr04MbTjMtUhUdMMORmZVkbSI8OFSKQpL06DgXu8OoTESk7CjwiJSKrZWc5bizNpIwxBML1O1t5RgdbC1her1eXWnpEpCwp8IiUAG/Mzs1MM+xkBGoadgQeOzbcYq3Nd7dWtkzoudmrV0SkbCjwiJSGK4HDmEHYATDVtcMmEBwDsOl0MB0faixEcVl6cXVeWeDriIjklQKPiM+8qecXAX0zfa/XrbUjJKVHB1vyWdsE+oCLNGVdRMqJAo+Ij7xFuXf1FAAAIABJREFUBa/FrYEzq4UDA7VZ3Vrx4dYiLDWRxtV7rbq2RKRcKPCI+OsLwCImmHo+Haa6btAEgkkAm06F7NhwQ76Km8QgsBh1bYlImVDgEfGJtzfWhbi1dmbNGIMJ1+2crTVS0Nla2fqBCyMd0QOLdD0RkVlT4BHxzyW4/4O57oFFsKZh5zie+FCrtZZ0fLgunRityfXck0jj6r+4gNcQEckLbS0h4oNIR7QZeAIYwY2HyYm11iS2rjvcptNBABMIpDL3q1qXrgnU1A/keo0JBHG7rR8b62zPqaVKRKSQ1MIj4o/VuLCQc9hJj43UJnu79rPW7vj/nAk7AOnEaCEHFqdwn2N1Aa8hIpIzBR6RIot0RAPApcBYruey6VQgsX3jqvToUBsTLDpoQlU5X2cKY8Cl3ucSESlJ+gYlUnyHAwtw3Vm5cSFn0n5pE6oezfk6kxsBFuIWJBQRKUkKPCLFdwiuGyhnJhhKhRrnr5/0NaHqeOb+Ex0nH3rqQYsKsRpzEPe5RERKkgKPSPEdyxStMjMRrG/ZHmpaEBvvORMMxY0JFGNmgsV9LhGRkqTAI1J8RwJ57WYK1rd0h5r2bOkxgVAin9eZxChwVJGuJSIyYyG/CxCZSyId0QZgCdAz1WuncuVpKxd/7N37LKyrDga7h8YSX/vpy+s/fHDb0Jsbt9b/412PA/C+VXuZb//JKYcd8w8PvpB535H7tNT/zRmr9p1XX131m9ff6f38Hc+tH02kc20FigNLIh3R+lhn+1CO5xIRyTsFHpHiWsH/b+++w+OsrjyOf+8Udcmy5SJ3GVNMi0NZhxrSKEGsaYZgWGMgATawu5CEFBGShWUT0VNxAgnNVGMDNokoSwglIYAhphjcaHLBRZZtyerSzNz9487IYyHbkqZq5vd5nnmC35l53zvKaHTm3nPPicPurP3Li3O/cfj4kafe/sryTxvauvYaXpjj9RiTkzu2PScvH6AQwO/q75RFP/fkg0aXzbp78aqWjkDovgun7fOjk6aMvvZPy9bHOibc69oPWBKHc4mIxJWWtESSKy7dzIMhi99rzAFjSvL8XmM+rm/p/KCuuQOgoKh4u3/Y2BX+YWNX5BYUf6YY4MNvrKlbs7W1a0tLZ/B3L3204YQDyofFY0xhyejWLiLSbwp4RJIrF+i1Xk5/rKpr7rjp2ZVrr/jKPmOWXHP81LtnH77X2NJ8f+R+T25Biye3oNelpU8b2rpnmNZsbe0oK8rJiXU8YQaI17lEROJKS1oiyeUnDgEPwEOL12x9aPGarUPy/Z7bzpo68aenHDCupSMQzPN7u7/IjCzJ/czv+NjS/O6gZPywgpwtzZ3xKkyogEdE0pZmeESSq4s4bEnfv7w492tTRhbn+jymrTNo2wMhG7LWLtuwvfXoycOHlBXmeEcPyfPNPrJiVM/nnvMvE0aMH1bgLyvM8X77uMmjn1u+KeYE6jBLHPKTREQSQTM8IsnVQRwCnly/13PVifuN+/XMQ/KCIWvfW7+9+ar576ze0tIROGry8JK//eDLn9vY2N6x6J319bO+MLE8+rlPv7dh6wMXTdu3rCjX//cP6htufGbFhljHE6aAR0TSlrqliyRRRVXNocCDQMxbt4NtTUNCbduHG39ui7eobKMxfV8ps8GAF4832J/n9EEhcF5tdaV2aYlI2tEMj0hyrcLluQw44LHBgC/QWDc+1NHidld1tJZ680u24svp0+xKsK2pJNC4aW9jTNBbMmK1N7+kYaBj6SEHWBmnc4mIxJVyeESSqLa6shnYgNut1S/WWoItDcO66lcf2B3sAMbrb6cfFZVtZ1sx1hobCvkCDZsmBxrrxsZhpjcX2KCigyKSrhTwiCTfW0Bef55gA505ga2f7h3YvnmSDYW6Z2Y9eYVb/WXjVhpP3/tleQuH1hmvr3s2KNjaWB7Ysm5fGwzEMuObhwoOikgaU8AjknyL6ePWdGstweatIzrr1xwY6mwbEjluPN5OX2n5B/6hYz4xXl+gPxc3Pn+Xv2z8Mk9OfndRwlBXe3FX/ZoDQh2thf05V/Rpca9LRCQtKYdHJPneA4J7elCoqyMv0Lhpou3qKIo+7s0vqfOWDP/UeLyhgQ7AeH1B37CxHwSb6scEWxpGA9hQ0N+1bf1+vqJhaz2FQzf3M6E5iHtdIiJpSQGPSPK9A2wGhgJtPe+01ppg05byYGvDaKztjjqM19/uGzKydlcVlPvLGIOvZMR6489rCTbWTbI25MVaE2jaMsHT1V7oG1K+xng8fQmq8oE64N09PVBEJFW0pCWSZLXVlSHgDnqpShzqbCvoql+9f7Bl25juYMcY6y0s3eAfMWFZvIKdaN784kbf8PHLjC+nO/gKtbeUdW1ZMyXU1dGX5Ooc4I7w6xIRSUsKeERS40ncMpAXwIZCnkDjpnFdW9btbwNd+ZEHGV9Oq3/YuOW+khHrjel7YnJ/eXw5nf6y8cs9eUVbIsdsoCs/sGXd/sG2piG7eao3/DqeTNTYRETiQQGPSArUVlc2AouA4lB7c3FX/eoDgq3bd7SBMCbkLRq2zj98wnJPTt5nlr0SwXg81ldaXusrHr4aYyyAtSFvoGHj3oHtm8fsYut6MfBEbXXlZ7qyi4ikEwU8IinSvmbpY8HmbSO6Gjbta4OB7qUjjz+vyV82fpmvuGxTnCsh75ExBm/R0Hr/0DErjMe7Y+t6S8PowNZ1+/TYuu4BQsB9SR2kiMgAqLWESAoYY84Abi856hvlRQd/jVB7M8Z4gt7iYes8BaX1yQ50emODAV+gYcOkUGd7SeSY8fo6fUNGfeTJLWgFSoH7aqsrr0/dKEVE+kYzPCJJZIwpN8YsAB4DypsWLyTYsg1vYWmTf8SE972FQ9Mi2AEwXl/AN2zcB97C0u7mojYYyOnatn5KsK1prLV2I3BLCocoItJnCnhEksA4FwLLgTMjx22gY1Prir//wFcyYpPx+vdYmyfZIlvXfaXlHxrjCUaOhtqayuvmX7tl9Q2npHaAIiJ9pIBHJMGMMXsB/wfcjVsGirgb2H/7G4tuBuYCu9sNlVLe/OJGX9m45cbnb/PkFtD83vO0f/zP6cArxphJqR6fiMieKOARSRBjjNcY8x1gKfC1qLs+AY631n7TWrstfOwWXOG+UtKUx5/b4R8+YUOwZdvapsVPRA4fAvzTGHNyCocmIrJHCnhEEsAYczDwD+A2oCB8OBT+98HW2r9EP762urINmA18RPoGPaXGeD7MGT7hIBvovByIdGgfCvzZGHOtMUafKSKSlrRLSySOjDG5wNXhW/QW7veAb1prd9tgs6Kqpgx4CJgMNCRqnANQigvGzq2trtwCYIw5AlgAjI163DPAedbarckfoojIringEYkTY8yRwF3A/lGHu4DrgRuttZ29PrGHiqqaElxtm88BjbiZoVTx4HKL3gVm9ywwaIwZCTwMfCXqcC1wprV2SbIGKSKyJ5p+FomRMabIGPNr4BV2DnZeBT5vrb2+r8EOQDioOBe4F1fJuGi3T0icovD178XN7HymmrK1tg44Ebgh6nAF8A9jzEVJGKOISJ9ohkckBsaYE3GNQCdGHW4BqoA51tqYtppXVNUcBvwKKAe2k5zZHg9QAmwErqitrvxnX55kjDkNNzNVEnX4D8B/WWvb4z5KEZF+UMAjMgDGmDLgF8CsHnc9C1xqrV0dr2tVVNUUAN8LX8sDNOEadsabFxesBIH7gVvCydR9ZozZF3gcODDq8D9xS1xx+5mIiPSXAh6RfjCuDPLZwG+AEVF3bQWuBB6wCfqlqqiqmYLbyXUGLvDpBOLRWDQfyMHNHj2OaxexYqAnM8YU4mZ2ZkYd3gqca619NpaBiogMlAIekT4yxowD5gD/2uOuebhlm7pkjCOc1DwduBQYiZuZsUA70NGHU+QCeYDBzebU4ZblnoxX1/NwYPgfuG34kd1qFvhv4GfW2lQmYotIFlLAI7IH4doyFwM3sXN+yqfAt621f0rFuCqqajy4nVwHAdOAQ4HRuJkfcAGNwQUakV/0HGADbpnpDdx2+XdrqysTEoAYY44GHgXGRB2uAWZFFV0UEUk4BTwiuxHOSbkTOK7HXXcAP7TWNiZ/VLtWUVVTCOyHq5uTE751hm8NwMra6sqWZI7JGFMOPMLOP8OPcXk9bydzLCKSvRTwiPTCGOPDJQpfh1sCivgAuNha+1JKBjZIhX+e1cBVUYfbcQnec1MzKhHJJgp4RHowxhyCKyB4SNThIK7f1XXW2ngkCmclY8wM4B52ri30e+BKa21f8o9ERAZEAY8kiw8XQHwVl3DrBf6CK873Mq7GTEoZY/KBnwLfx40v4i3gW6ocHB/GmCm43WDRRRoXAzOstWtTMyoRyXQKeCTRcoH/xO3OsbicksgSURBXpK8TOAmXSJsSxpgvAn8E9ok63IEb923W2q5enygDYowpws2inR11uB6Y2bOxqohIPCjgkUT6Cq6A3RCgcA+PbQUuw1XqTRpjTAlwI/DvPe56GZersyqZ48km4a3rVwI3s2NGLQRcg+s9pq3rIhI3CngkEUpxhfnOxBW166tWXEuGXydiUD0ZY/4V+B07d/tuwi1p/UF/cJPDGHMsbut6edThJ4HZ1tp06hgvIoOYAh6Jt9NxSxX5uOJ2/dUGfBF4M56Dihbu8P1r4Bs97voTcJm1dl2iri29M8aMxgU9x0Qd/hA4w1q7NDWjEpFMom7pEi/lwJ9xS1hDGViwAy5QejyG5++ScWYBy9k52NkMnAOcqmAnNay1G3BLoL+MOrw38Lox5rzUjEpEMokCHomVAS7E1ac5nj3n6vRFGa7+TdwYYyYCTwFzgWFRd90P7G+tnZeoHljSN9baLmvtd3DBZ6Q4Yj7wgDHmN8aYnNSNTkQGOy1pSSwmAQ8AU4lPoBOtDTgSeCeWk4TbQlyOK3oXPcY1uKJ3z8RyfkkMY8yBwGO4qtERrwJnWWs/Tc2oRGQw0wyPDIQXV4X4PVwPp3gHO+CWtB5hR+PJfjPGHAD8HZevExmjxSVUH6RgJ31Za9/Hvbcejzp8JLDEGPPlFAwpF7gaeB74CDcD9QGucvTY3TxPRNKEZnikvw4CHsbN7iQi0InWAlyP2zbeZ+Gljx/itjdHL4MsxxUQ/EfcRigJFd66fhVwAzu+oIWAHwG3JGkZ8nBcQvUooKDHfW3hcf0DuAA3cygiaUgBj/RVLq4K8XfC/52s2cE2XEfwD/vyYGPMNFwBwYOjDgdwS1o/U/uCwSk8q/MIMDLq8OPAhdbaRFbp/gmuVEIeLl9tVwK4QpXfxuWFiUiaUcAjfXEE7o/NCD77DTfRgrgKzEfglqN6ZYwpBP4HV8guOhh7A/imtjYPfsaYscB83NJWxCrc1vX3E3DJK4Cf07/3fCvud+UyXAAkImlCOTyyO0W4wnx/BSaS/GAHXL7QgcC3dvUAY8xXgaXAd9nxnm7D5RkdqWAnM4STlb+Ey8GK2Be3db1nTaVYfR03K9jf93wBbpfZm8D4OI9JRGKgGR7ZlRNwU/PF9K9acqK04P64rY8cMMYMxXUwv6jHY58HLrHWfpy84UkyhWvz3MnOAckvgR/Eoe/ZJNzuwOIYzhHAvWfPxL0fRSTFNMMjPQ0D5gFP4PIl0iHYAZd8fA/hPApjzJnAMnYOdhrC/z5ewU5ms9Y+iFvmjM7tuhL4a7hq80DlA08T+2ymD9dD7k+45Pnd5f+ISBIo4JEIg+tc/TFwKqlZvtodP3D0+++//01jzGPAAnbuvfQYcIC19h4VEMwO4aXKw4FFUYePAd4yxnxxAKc0uLYoE9jRzDRW+bgdZc/gAiARSREtaQm4OiL34pJB47LV/MMPP2TEiBEMGTIEay1ud3HsGhoa7F577WW2bdsWObQRuNxa+/huniYZLFxc8gfAz9jxJS4YPvaLfgTAlwC3kZhyCx1APS43SDllIimgGZ7s5gEuBVYCxxGHD/qmpibOPvtsjjjiCF544QWAuAU7AHl5eeb222+P/PMu3KyOgp0sZq0NWWtvwOWd1YcPe4FbgXnGmL7k4vwLLgcoUbWlcoExwGuAeoOJpIACnuw1HvfheyvuQ94f6wmfeeYZJk+ezIIFC9i6dSvz589n5cqVsZ52J3l5eZx22mn29ttv/4G19lvW2m17fpZkA2vt88ChwOKow2fhdnHtv5unDgdqSHy+msEtFd8Zvqk3mEgSaUkrO52EqxybTwytGyK6urpYsGAB119/PStWrKCwsJCWlhZ8Ph833HADF198McXFxXFd2gI2AZPZ0WRSBABjTC7wC1wRwIhmXJHCBT0e7gVexLWxGFAA0tHRQW5ubn+f1opLuD4ZUG8wkSTQDE/2OQyX4FtMHIIdgBdffJHf/e53rFixAoD99tuP8ePHEwgEmDt3Lq+//joQ36UtoAS4KZ4nlMxgre2w1l4GzAbaw4eLgPnGmFuMMdHv+5/hZoUGFOx0dXVx+umnc8EFF/D888+zYcOGvj61ADgA14/uSwO5toj0k7VWt+y5jbTWbrZx9Oqrr9pJkyZZY4w1xtirrrrKbty40X7/+9+3fr/fGmPsRRddZD/++ON4Xjai1Vr7BZv6n6tuaXoDpuKafdqo24tAubX2FGtti43BL3/5S1tSUmKNMXbSpEn23HPPtQ899FB/T9Nqrf2RtdbE+/XrpptuO26a4ckuC3EzI3Hz+uuvU1tby4gRI/j5z3/OTTfdxKhRozj55JP54hfdzuBHH32UZ555hra2tnheGtyS3MMoF0J2wVr7Dm7r+p+jDh+37777vhMIBB4lxvILEyZM4PTTT2fkyJHU1tYyf/58zj//fKZPn8727X1u8ZWPq9XzBC65WUQSQDk82WMkrpNzXD5Qrd2Rj3PRRRcxbNgwrrzySsaNG9f9mDlz5nDjjTeydu1apk2bxs0338yxxx4bj8tHa8UlXv803ieWzBHeun418D/5+fnmnXfeYdKkSfh8sa/qbty4kQ8//JBbb72VRYsW4fV6CQaDlJaW8sILLzB16tS+nqoNt2X9Kyg3TSTuNMOTPb5AHJsZGmMIBoMAVFdXc91113UHO6FQCIDp06fz9a9/HYDFixczb9481q1bF68hRBQAV+HyIUR6Zd3W9f/1er1ff/DBBzvHjh0bc7AT+bJYXl7OMcccwxNPPMFVV11FUVERAA0NDSxYsIDOzs6+njIf+Bwu4VpE4kwBT/Y4ijjXGPF6XTHaUaNGUVhY2P0HwONxb6tx48YxY8YMjj76aAAefvhhnnvuuf78AeirXFyH6nhVx5UMFQgE9j711FODBQWxFxLvmYS/dOlS1q5dS2NjIx6Ph0suuYTzzz+fnJx+rbjm4er0fCnmAYrIThTwZI8ALmEzYaL/AESCny996UucfvrpjBo1im3btjF37lzeeuuteF/aA+wF/Ge8TywZ5QvAzR6PJ+71dtasWcM999zDvHnzADjppJOYNWsW++yzz0BOV4AL4NOlj51IRlDAkz3+gst3SQpjDNZafD4fp556KieeeCIAL730Eo899hibNm2K9yULcVuMJ8b7xJIRRuASl+MeRDQ2NvLEE09w1113AXDooYdy/vnnd89sDjBPMh+3w0xE4kQBT/Z4DTddnjSRGZ/JkyczY8YMDj/8cAAefPBBXnjhBQKBwE6Pj0MCfQ7wAOpMLTvzAU8S5x2K4OrwPPfcc8yZM4empibGjx/Peeedx/Tp04Gdk/v7yaDgXSSuFPBkjw5cyf2kbsuLBDFf/epXOfXUUxk6dCgbNmxg7ty5vPfee92PC4VC3X8YIknPA+ADDgFmxjRoyTQ34pKB416+YPHixfz+97/ngw8+oLCwkLPOOotzzjmHvLy8WIIdcF9OKuI3UhFRwJNdfoDb+po0kaWt/Px8zjjjDI4//njA9d164oknqKurA1yic1tbG4sWLeLVV1/9zOxPPxQCv0L5D+KcBvw7Mdbb6c3KlSu56667+Otf/wrAKaecwqxZsxg9ejQQc2XxIC7vTkTiRAFPdnkVl8sT921SuxP54N9///2ZMWMGBx98MAD3338/b7zxBgDLly/nlltu4Yc//CGXXHIJ7777biyXzAf+LbZRSwbYB7ifBAQ7dXV1PPLII8ydOxeA4447jgsuuKA/NXf64tF4nkwk28Wll5IMKpcCH5Dk6sSR6f0TTzyRt99+m08++YTa2lruuOMOli9fzptvvslf/vIXtm7dCsDq1as59NBDB3q5Qtw3+z/Eafgy+BQCzxDHYCfyHm5tbeXPf/4zd9xxB6FQiClTpjBr1ixOOOGEnR4XoxXA2lhPIiI7KODJPhuBK4BfE8e6PI2NjbS3tzNq1Khe7zfGEAqFKC4u5rzzzmPZsmUsXLiQmpoalixZwvr16wH4/Oc/z5w5czjiiCNiHdKxuMRPlRLPPgY3szOGOMxiRwKYyPLsyy+/zJw5c9i4cSPDhw/nnHPO4cwzz+y+Pw7BTgtwe6wnEZGdaUkrO90DvEsccgQCgQDLli3jpptu4uqrr95ljR1rbXdBwvLycoYPH86QIUPweDzdwc51113HkiVL4hHsgPujp+rL2ek/gROIYVfixo0bWbNmDbBzLs7bb7/NnXfeyZIlS/D5fJx++umcd955DBkyJF7BTgjXAubeWE8kIjvTDE92srgcl6XE8B6or6/npZde4u677+bpp58G4Mgjj2Tq1KndwU1E5A/BK6+8wsKFC3nxxRdpbGwE4IgjjuCPf/wjBxwQ1/jE4BI/Jbt8DriBGJLW165dy49//GO6urq4/PLLOeaYYwD45JNPuO+++1i4cCEAJ554IrNnz2by5MlAzEnKEe3ADJSwLBJ3Cniy18e4hpvX0c+lrY6ODpYvX84jjzzCXXfdxZYtWwC45ppr+Na3vtXrcxobG5k7dy41NTW8/PLLtLe34/F4uPXWW7niiitifCm98gCrEnFiSVt+4DFimNmx1vLoo4/ywAMP4PV6aW5uZtOmTRx99NE89dRT3cUFDz/8cGbPns1RRx0Vp6EDrjDovwPL4nlSEQmz1uqWvTevtfY9a23Q9lFTU5N9+OGH7VFHHWWNMdYYY4899li7fPny7scEg5893apVq2xlZWX3c04++WS7du3avl52IF61qf/56pbc23Rr7XYbo2AwaM8+++zu9+phhx1mZ82aZSdNmmSNMXbixIn2tttus+3t7dZaa0OhUKyXtNbaFmvtH2zqf4a66ZaxN+XwZLcgrkhfn7uoh0IhFixYwKuvvorH4+E3v/kNL7/8MlOmTCEYDBIKhT6znAUwZswYpk2bxrBhw7j77rupqanp7q6eAK24pGzJLhcDRbGexOPxMG/ePG699VYAlixZwsKFC6mtraWgoICZM2dy/vnnk5ubi7VxydsJAB8Bl8d6IhHZNWOtNrEIPwOupI9beOfNm8dzzz3Hf//3fzN+/HjAJS/7fLtfId20aRNFRUUUFsa1aXtPrcBc4NuJvIiknWJgM5Abz5O+9NJLzJgxgy1btuDz+Rg3bhxXXXUVl112GRC3LegNwMHAulhPJCK7poBHwP2RWEkfe/dEf8gHg0E8Hk+8EjZj1Qm8hduS3pXisUhyHQ/MB4bE+8SbN29mxowZ/O1vfwPgy1/+MjNnzmTGjBmUlpbGevo2oBJ4IdYTicjuea+99tpUj0FSLwi8DpyLS/zcreieV16vN12CnRZcg9RTw/8t2aUYuIg4z/AAFBYWcuGFF9LY2Mhrr71GbW0ttbW11NfXM3ny5FiCnhbgf3E1g0QkwZTDIwAYY9599tlnP2htbe3zc3rL1UmBIO5b8rXA14CtKR2NpMpqEtw/7bbbbuPBBx/E5/OxdOlS7rzzTt588026ugY0mdgO/A23hV5EkiAt/mJJahljvgYsPfPMMz/X3Nyc6uH0RwvwT1ztlVtwRdskOzXQj+T7gZo5cyZvv/0248aNY8qUKey33374/XucFO3JAvXAN1AlcJGkUQ5PFjPGDAVuBS6MHDvhhBNYtGhRKC8vL52D4S7cH7fvAn9EfzTE+S/g58SxZcquBINBli9fzkEHHTSQp7cC04D34zsqEdkdBTxZyLikmzOB3wLRza8agO+EQqGTjTHTSUA+RBy0AK/ggrT1KR6LpBcPLhftEMCb4rH0qqOjI9DZ2XlRcXGx8nZEkiydv8VLAhhjxgCP43a0RAc7C4D9rbX3GmO+jcsxSCcduIDsAuAkFOzIZ4Vwy0Tp9t4FoKWlhbvvvttXUlJyhTGmItXjEck2muHJEuFZnW/icl2it+5uAC6z1i7s8ZRvAHeRhOWBPmgFnsQVZlNSsuzJkcAzuCKEafGlLhAIhJYuXeqZNm0agUAA3Pv4XGvtsykemkjWSIsPA0ksY8zewPPAH9g52PkjcEAvwQ7Ao8CrpLaeTRuwCTgNVxFawY70xau4HJlPSZPZHq/Xu/3SSy+9JhCOdoBhwNPGmJ8YY/Q5LJIEmuHJYMYYH66C8v+w85bdj4GLrbV/3cMpxuIKEqZilqcNuAf4AaqrIwNTDDwMfInUzlS2AV8HXjLGHIVbTh4TdX8NMMtauy0VgxPJFvpmkaGMMVNx33RvZkewE8ItaR3ch2AH3Dfkq0huwNEK1AJfxi1hKdiRgWoC/hUX8LelaAwtuBpRLwFYa/8BHBr5d1gl8KYx5vNJH51IFtEMT4YxxuQB1wA/BKKbW70LfNNa+2Y/T5msnS8hXGLyLbjqs50JvJZkn+OAhbiZnn4XzhmgdtxS8r/So3RCePb158D3ezz+36219yVpfCJZRQFPBjHGHIPL05kSdbgT9w33JmvtQPNx9gHeIXGVbFtwy2wzUW0SSZwxwFO493OfGuXGIIRrBnoQbqapV8aYM4F72bnL+x3AFdbahBdSFMkmWtLKAMaYYmPMb3Gl6qODnVeAqdban8UQ7AB8gJt1iffyUgC3hHUNbgZJwY5Bz9QuAAAT80lEQVQk0npcMvNc3Psukdpw5RN2GewAWGsfA/4FWB51+FLgb8aYCYkbnkj20QzPIGeMORn4PTA+6nAzbknr99baeLVb8AHvAfsC8egW2gK8DfwbLmdHJJlm4nYp5hOf93O0Vly9qPl9fYIxpig8nm9EHd4CnGOt/UtcRyeSpTTDM0gZY4YbYx7A7fCIDnaeAg601s6JY7ADbjbmHGLf5tuJ+9Z7GXAsCnYkNR4GvoCb9Ynn0lErcDf9CHYArLXNuCDsO7jfNYAy4FljzNXaui4SO83wDDLhAoLnAL8GhkfdtQXXS+hhm9j/U3+K2yo+kG2+Lbgkzktw9XVEUm0I8Agu+I5163oXbnPAkcRQv8oYcyyuDlZ51OEngdnW2oaYRiiSxRTwDCLGmPHA73DbWKM9BFxprd2chGF4gOeAI+h74mc77pvvhbgPbpF0YoAfAT8htsT8rcCBwMaYB2TMaGAeLhCL+Ag4w1r7bqznF8lGmiYdBIwxnnB/q/fZOdhZB5xirT0vScEOuN0nJwEPsOfEzxAuefNRYC8U7Eh6skA1cArQyI4lpf5oBaYTh2AHwFq7Afgq8Iuow5OB14wx/xaPa4hkG83wpDljzH64rebH9rhrDlBlrd2e/FF1uyA8jk7ctlovLshpBvKAN4D/wCUniwwG43B5cJPp+wxmKzAb14A37owxZ+PygqKX3G4HvmutVb0qkT5SwJOmjDF+XFGynwK5UXetAr5lrf1bSgb2WcNxy1tH4oKypbjGjX/DdTcXGWxycGUY/gMXuO9qF1cHLi/tfNzmgYQxxhwAPA7sF3X4NeAsa+26RF5bJFMo4ElDxpjDcJ3Kp0YdDgI3Atdba9OiIaJIhjscV97hFFwScnH4f9twZRr+BHwbSEoPLGNMCW6m58yow5uBb1hrX0jGGEQGMwU8acQYU4Dru/M9ds6vWoJrC6GlIZHkK8C1h/gi8AmwGliBm81MqvAuze/ivvxEWr2EgCrg5gTv0BQZ1BTwpAljzJdxuTqTow6345a0fmGtHUgipYhkIGPMcbjNACOjDj8OXJjivD6RtJWxAU9FVU0RripwKS4Hxo+bju7A5Zasqq2ubE7dCB1jTClwE3Bxj7teBC621n6Y9EGJSNozxozFBT1HRR1ehdu6rjYtIj1kRMBTUVXjweW7HITrlXMIMJodHbdN+GbZ0bU4B9gAvAUsxrVNeKe2ujKe1Yl3yxhzKm6X05iow9uBq4A/anpaRHbHGJMD3IwrOhrRilsCfyQ1oxJJT4M64KmoqhmCq31xKTACt6ZtcUtBfSkXn8uOXRhBXALgHcCi2urKhE0LG2NGAb8Bzupx10Lgcmvt+kRdW0QyjzEm0hsseiv9r4Dvx9g4WCRjDMqAp6KqZn9cDZjTcMm9nbidE7HKx838hIAngHtrqytXxOG8QHfC4fm4YmJDo+6qAy4HHtOsjogMhDHmIFwezz5Rh18BztaXKJFBFvBUVNUU4HYwnY+blWnCzczEmxe3BTUE3A/cWltduaeqwrtljKnAzR6d0OOue4HvWWu3xnJ+ERFjzBDcZ8ppUYc34YKel1MyKJE0MWgCnoqqmsNxU7SjcHkuyci18QAluHLxV9RWV/6zvycwxnhxBcx+zs7TzbXApdba/4vDOEVEgO6Z5B/gPnMi5S2CuJpCt2kWWbJV2gc8FVU1+biKw+fjetykYmdVpG3CXOCW2urKPi2fGWMOxBUQ/ELUYYsL3H5irU35LjERyUzGmK/gOsGPiDq8ALjIWtuUmlGJpE5aBzwVVTUlwH3A53BN/ZK2g6oXHmAI8C4we3dJzeGdE1XAj3Hb4SPex7WFeC2RAxURATDGjMMFOdFfulbgtq4vT82oRFIjbbulV1TVlAHzgYNxpdtTGewQvv423Hjmh8f3GcaYI3CVka9lR7DTFf73oQp2RCRZwn22jsOVv4iYAiw2xvTcJSqS0dJyhic8szMfV3U4HRtQlgIfAWdFZnqMMUW4hoP/xc7NBl/H1cRQITARSRljzCzcxon8qMO3AT/S1nXJBmkX8IRzdh7CzaSkY7ATUYrrpXPu6htOORb3QVIRdX8rcDXwW2ttInaSiYj0izHmc7it69EtbF7GNSDdmKxxDJZK+JJZ0jHg+Qmuxk5SOhDHwoaCZc3v/mXb1md+c0iPu57D7cD6JBXjEhHZlXA7m7m4hqgRG4CzrLWvxPt6g7USvmSetAp4wlvPH8bV10nbN7a1llDb9qHB5q0TjC/XV7/oRjo3fQQuSPsOMFdbP0UkXRljPLiNFdezYwk+gGtr8+t4fH4N1kr4krnSJuAJFxV8DigjNVvP+8QGuvyB7XUTQh2tpQDGn0eobTt186993Ha2XWat3ZTqMYqI9IUx5gRcCkH0JoxHcI2LB/Q5PFgr4UvmS6eA5yfAbNI0b8daS6i1YXiwaes4a0PeyHHj8Xb5ho7e4snJn1NbXXl9KscoItJfxpiJuK3rh0cdfh+3dX1VX88zmCvhS3ZIi4An/I1gEW5mJ+2WskJdHbnBxrqJoa724ujjnvziel/JiHXG47W44oTT9Y1DRAYbY0weriDqJVGHm4DZ1ton9vT8wVoJX7JLutThuQA3lrQKdqy1BJq2jOrasvbA6GDHeH0d/qFjVvlLy1cbjzeIG7cHN0MlIjKoWGvbrbWXAhfhcmzAzaI8boy50Rjj6+15FVU1+RVVNT/FLYOV4Wbok/U5HgpfbzjwSEVVzU/Cu3xFepXyGZ5wYtvruDXetNm+Hepsyw801lXYQGd0/yu8BUM2eouHrzceT88fnBe3xjxNCXUiMlgZYw4BHgMmRR1+ATjHWlsXOTBYK+FL9kqHGZ7puGAh6cGODQU91obMTsdsyAQa68Z2bVl3QHSwY3w5bf6ycct9Q0Z+2kuwA278XtzrEREZlKy1b+HyeZ6KOvxlYIkx5kgYvJXwJbuldIYnXJ/hZWAo8cni77Ng2/bSQGPdXi7peMwqjz+3I9TRUhRo3DzRBrvyuh9ojPUWlq73FpVtMsbs6YeVD2wFjlO9CBEZzMJb16/BtcWJfDHs8paMqBr77btPMMYMmkr4IpD6GZ6puPoMSQ12bCjoDW7fPBFrjQ0GcgKNmyq6GjZO6Nq6fr/oYMf4c5v9ZePf9xUP39iHYAfc6xiJm+IVERm0rLUha+3/ACcTLgRrfDn+oV+75JZgS8ORNhRK12CiAVdJ+j7l9Ei0VAc8B+GWgZIquL1+jA2FupPwbFdHUaitaUT3A4wJ+YrL1vjLxq/0+HP7UiArmhf3ukREBj1r7TPAYcCS4mmnkzOigmBTfXHXljVTQl0duake3y404L54XpXqgUj66DXzPommsaOUeNzl+jzmtrOmTji8YlhJcZ7Pt76hvePmp5ZufvLFD0bu6jmenPxG35CRq40vZ6DN9CzudT0wwOeLiKQVa+0n4y6/7z+Nx/t0oHlrCYANdOUHtqzb3ztk5Cfe/OLGVI+xF43A+RVVNU9py7pA6md4DmHHFsi483k9ZsP29q6Zf3ht5UHXPvvWL59f9emt5xw2Ydzwkl4fb3w5rb5hYz+MIdgB93oOjeH5IiJppaKqpsBXMvxmb9HQDb7iYasJL/FbG/IGGjbuHWisG5vqHb+9COE2k/xKS1sCKQx4wt1yR9O3nioD0tIRCP1vzfL1H9e3dIYsLPrHspw1mxuZWtH7BI8NdBaE2puHxHjZDmB0RVVNYYznERFJF9/DFRVs9hYOrfcPG7vCeH2R5p8EWxvLA1vX7WODgVSvGvTUDJSjpS0htTM8+7KjW27CWWsp9bSN36t8KCs/3bLrx3W2FcXhcp3AfnE4j4hISoUr4c/CVVAGwJOT3+ovG7/Mk5PffSzU2V7SVb9m/1BHa0Fv50mh7cCsiqqaKakeiKRWKgOe0mRezO815vZLTjTz/76cDzds++wDjAl5/LnNnsLSzXG6ZFJfn4hIglxAL5XwjdcX9A0b+4G3sHRD5JgNBXO6tq2fEmzeNjyNlrhUCV+A1CYt57KjtkNCeQzcOevwSV6Pbf3xvH9YT15Ru8ef14zX12W8/k7j9XXi8QaNidtwDK6rr4jIoBWuhH8qrq/WZxhj8JWMWG/8eS3BxrpJ1oa8WGsCTfUTPV3tRb4ho1bvolBrsjUBp1dU1VSrNk/2SmXA4ycJAY8Bfjvz0IphhTm+mX94fYVn6DibhGktBTwikgn6VAnfm1/caPy5ywLbNuxtA535AKH25rKuQGeBb+joDz2+nM5ga+OwYNOWcSYnf7uvtLw2jl8w+yK6Er520GapVC5pdZHALekRv/jG5ydMGl6YN+uuxR+2dQWT9U3DksT8JBGReAtXwr+UPn6WeXw5nf6y8Ss8eUXdSZI20JkfqF97QKBpy6jA9s0VNhT0h9qby0Jt24cmaty70QlcGn5dkoVSOcPTQYIDnkllhTmnfX7siM5AyC6++mtTI8f/96llqx98fc3WBF5aAY+IDHaRSvi9Lmf1xng8IV9peW2opaE50LxlAtYaa0PeYPPWcdGPC7Y0jPbkl2xL8ixPdCX8t5N5YUkPqQx4Et6D5ZMtLZ0VVTWpKjiVjj1mRET6akCV8I0xeIuG1ht/bmugYeNkGwp+ZnnfBjrzQ+3NJd784mTn00Qq4SvgyUKpDHhW4fJcWlI4hkTJAVamehAiIjGIqRK+J7eg1eQWNNrotj1Rgi3bRt91yZfKDp84rDjP7/FubensuucftRvv+vsn9QMe8Z6pEn4WS1nAU1td2VxRVbMBGEICiw+mQC6woba6MhMDORHJHjFVwg91tOzco7AH29VR9Iun3/3k44ZgbUcgZA8YXZL3wDe/sN+76xpa36jd1jrQ6+6BKuFnsVQnb70F5O3xUYNLHrAk1YMQERmoeFTCt12de/xsf2fZqjEdgVC4TYW1gJ00vCiRDUlVCT+LpboM+GLg6ykeQ7wZ3OsSERmsYq6E7yko2eoJdBTYzrYhNhjotUyHDQZybzt76oSTDxpdluf3ej6oa2p9eumGRDcijVTC1xfTLJPqgOc99lDfYRAK4l6XiMhgFXOleOPxhvyl5WsAbDDgC3W1F9iu9gLb1VEY6mwrwVqP8eW0fffRd9Z8f/67a47eu6zo6L2HF7cHklI+RJXws1CqA553gM3AUNyWwcEuH6gD3k31QEREYhDXSvjG6wt4vUXbySvq3pVlrSWyLT1oLS9/UN98xqHjhl36xckjfvvCh3XxunZvw0GFYbNSSnN4aqsrQ8AdZM6bLwe4I/y6REQGq4RXwu+tBo/PY8zEYQWJzOEBBTxZK9VJywBPsqPs92AWKb/+ZKoHIiISo4RXwh9Vkus751/GDy3O9Xm8xnDSgeUlX9t/1LBXPqrvc6HDAVJh2CyV6iUtaqsrGyuqahYBZzK4i/UVA/PVmE5EMkDCK+FbCzOnTRh5TeUBEz0Gs2l7R8ct/7dy7cK31yf674ACniyV8oAn7F7gdNyM02BcDoqM+75UD0REJA4S/uWzrqkjcOrtr6SqQOtg/nItA5QOS1rUVlcuB+4HSlI9lgEqAe6vra5ckeqBiIjEQaQSfiZSJfwslRYBT9itwCagKNUD6aciYCNwS6oHIiISD7XVlc3ABtxurUyiSvhZLG0Cntrqylbgv3DJv2kzrj3w4MZ7RW11ZSZsqxcRiVAlfMkoaRVY1FZX/hOYi+uvNRgMAeaGxy0ikkkWk+Ct6SmgSvhZLK0CnrBbcIX70r0SZilunFrKEpFMpEr4klHSLuAJLw3NBj4ifYOeUtz4ZmspS0QyVKQSfn6qBxInqoSf5dIu4AEI17I5l/QMeiLBzrmquSMimUqV8CXTpGXAA1BbXbkFOAtYiuu1leqxesLjWAqcFR6fiEgmUyV8yRipDiJ2K2qm515cJeNUbVkvCl//XjSzIyJZora6shFYhPv8G8yKgSf02Z3d0qXS8i6Fc2Sur6iqeQr4FVAObCc5FZk9uKKCG3Fbz7UbS0Syzb2oEr5kgLSe4YkWDjZOwL1pi3C5NImaZvXilq+Kwtc7XsGOiGQjVcKXTGGsTWh/uISoqKqZgtvJdQYuaOsE4rFbKh+X2BYCHgfu0y+JiGS7iqqaAuA5oAxoTvFw+qMIqMd9adWO2iw3KAOeiIqqmhJgOnApMBI3M2OBdly33z3JxVXeNLiEtjrcroQntdYrIrJDRVXNYcAjQBODY2nLg8vdOUcz9AKDPOCJqKiq8QCfAw4CpgGHAqNxMz/gAhqDC4YiLzgH1yvmn8AbuGJU72rLoohI7yqqan4CXABsS/FQ+mIocG9tdeX1qR6IpIeMCHh6U1FVUwjsh8v1yQnfOsO3BmClGsiJiPRdRVVNPvAQcDDuczRdleJKiJyrpSyJyNiAR0RE4i+cSjAfmEx6Bj2R4rBnKTVBog2aXVoiIpJ6qoQvg5UCHhER6RdVwpfBKNVvUhERGYRUCV8GG+XwiIhITMJb1lUJX9KaAh4REYlZuDjh94BZuGCkCVffLN68uEAniKsAfYt2YklfKOAREZG4USV8SVcKeEREJO5UCV/SjQIeERFJGFXCl3ShgEdERJJKlfAlFRTwiIiISMZTHR4RERHJeAp4REREJOMp4BEREZGMp4BHREREMp4CHhEREcl4CnhEREQk4yngERERkYyngEdEREQyngIeERERyXgKeERERCTjKeARERGRjKeAR0RERDKeAh4RERHJeAp4REREJOMp4BEREZGMp4BHREREMp4CHhEREcl4CnhEREQk4yngERERkYyngEdEREQyngIeERERyXgKeERERCTjKeARERGRjKeAR0RERDLe/wO6Kn6IefVOQQAAAABJRU5ErkJggg==\n", 487 | "text/plain": [ 488 | "
" 489 | ] 490 | }, 491 | "metadata": {}, 492 | "output_type": "display_data" 493 | } 494 | ], 495 | "source": [ 496 | "import networkx as nx\n", 497 | "\n", 498 | "def backward_prop(upstream):\n", 499 | " default_graph.nodes[len(default_graph.nodes())]['node'].gradient = upstream\n", 500 | "\n", 501 | " gradient_dict = {}\n", 502 | " for node in reversed(list(nx.topological_sort(default_graph))):\n", 503 | " child_node = default_graph.nodes[node]['node']\n", 504 | " if isinstance(child_node, OperationNode):\n", 505 | " func, args, kwargs, result, arg_num = child_node.recipe\n", 506 | " upstream = child_node.gradient\n", 507 | "\n", 508 | " for i, parent in zip(range(arg_num), default_graph.predecessors(node)):\n", 509 | " vhp = primitive_vhp[func.__name__][i]\n", 510 | " downstream = vhp(upstream, result, *args, **kwargs)\n", 511 | " default_graph.nodes[parent]['node'].gradient += downstream\n", 512 | " else:\n", 513 | " gradient_dict[node] = child_node.gradient\n", 514 | "\n", 515 | " return gradient_dict\n", 516 | "\n", 517 | "print(\"Backward pass\")\n", 518 | "backward_result = backward_prop(np.ones_like(result))\n", 519 | "plot_graph(True, backward_result)" 520 | ] 521 | }, 522 | { 523 | "cell_type": "markdown", 524 | "metadata": {}, 525 | "source": [ 526 | "---\n", 527 | "## Application" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "### Simple demo\n", 535 | "\n", 536 | "We want to compute the derivatives of $x^3$, which is $3x^2$." 537 | ] 538 | }, 539 | { 540 | "cell_type": "code", 541 | "execution_count": 6, 542 | "metadata": {}, 543 | "outputs": [ 544 | { 545 | "data": { 546 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU1f3/8dcnCVkgQFhCgJCwhiUgIkQWd0Urbti6/MRapS5FrVq7Wq12b7+tta1Wba1UaRVRimgVV8SNurDvOwl7ICSBQELInjm/P+5gI4ZFMsmdZN7Px2MeM3PvzdyPkrzvmXPPPdecc4iISGSJ8rsAERFpegp/EZEIpPAXEYlACn8RkQik8BcRiUAxfhdwPDp37ux69erldxkiIs3KkiVL9jjnkutb1yzCv1evXixevNjvMkREmhUz23akder2ERGJQAp/EZEIpPAXEYlACn8RkQik8BcRiUAKfxGRCKTwFxGJQM1inL+ISCR6ZdlOHI6vDkvFzEL62Wr5i4iEobKqGn7zxlpmLMoNefCDwl9EJCz969Ot7Cmt4ocXDmiUz1f4i4iEmZKKap6cu5nzBnZhRM8OjbIPhb+ISJiZPHczxeXVfP+C/o22D4W/iEgYyS+p4KmPN3P5sO4MSW3faPtR+IuIhJFH3s2mNuD4wQWN09d/iMJfRCRM5BSUMmPxDq4b1ZP0Tq0bdV8KfxGRMPHQ7PUktIrmrvP6Nfq+FP4iImFgybZ9zF6Tz6Sz+tApMa7R99fg8DezeDNbaGYrzGyNmf0yuLy3mS0ws2wz+7eZxQaXxwXf5wTX92poDSIizZlzjgffWk/nxDhuPqN3k+wzFC3/SuA859zJwDBgnJmNBh4EHnbOZQD7gJuD298M7HPO9QMeDm4nIhKx3ltXwMKtRXz3/AzaxDXNrDsNDn/nKQ2+bRV8OOA8YGZw+TPAV4OvLw++J7h+rDXGtcsiIs1ATW2AB99eT+/Obbjm1LQm229I+vzNLNrMlgMFwBxgE7DfOVcT3CQXSA2+TgV2AATXFwOdQlGHiEhzM23BdrILSvnxuIG0im6607Ah2ZNzrtY5NwzoAYwEBtW3WfC5vla+O3yBmU0ys8VmtriwsDAUZYqIhJX9ZVU8/O5GTuvbiQsHpzTpvkN6mHHO7Qc+BEYDSWZ2qPOqB7Ar+DoXSAMIrm8PFNXzWZOdc1nOuazk5ORQlikiEhYeeTebkvJqfnZZZqPM3Hk0oRjtk2xmScHXCcD5wDrgA+Cq4GYTgVeDr2cF3xNc/75z7gstfxGRliw7/wBT52/j2pHpDOzarsn3H4rTyt2AZ8wsGu9gMsM597qZrQWmm9lvgGXA08HtnwammlkOXot/QghqEBFpNpxz/Or1tbSJjW7UyduOpsHh75xbCZxSz/LNeP3/hy+vAK5u6H5FRJqr99cX8FH2Hn56aWaTXNBVH13hKyLShKpqAvzmjXX0SW7DDWN6+laHwl9EpAn969MtbNlzkJ9ektmkQzsPp/AXEWkiu/aX88i72Zw/qAvnDuziay0KfxGRJvLL19YQcI6fXzbY71IU/iIiTeH99fnMXpPPd8ZmkNaxcefqPx4KfxGRRlZeVcvPXl1DRpdEbjmjj9/lAKEZ5y8iIkfx+AfZ5O4rZ/qk0cTGhEebOzyqEBFpoXIKDjD5v5u5Yngqo/uEzxyWCn8RkUbinOOBV1bTOjaGn1xc33yX/lH4i4g0kn8v2sH8zUX8eNxAOvt0Je+RKPxFRBpBXnE5v31jHWP6dGJCE96k5Xgp/EVEQsw5x09eXkVNwPH7K08iKir8blao8BcRCbFXlu/kgw2F/OjCAfTs1Mbvcuql8BcRCaGCAxX8YtZaRvTswMTTevldzhEp/EVEQujnr66hvLqWB68cSnQYdvccovAXEQmRN1bm8dbq3Xzv/P7065LodzlHpfAXEQmB/JIK7n9lFUN7tOdbZ/b2u5xjUviLiDRQIOD44YsrqKiu5eFrhhHj4zz9xyv8KxQRCXPPzNvKR9l7eOCSTPomh3d3zyEKfxGRBtiYf4DfvbWe8wZ24bpR6X6Xc9waHP5mlmZmH5jZOjNbY2Z3B5d3NLM5ZpYdfO4QXG5m9qiZ5ZjZSjMb3tAaRET8UFlTy93Tl9M2LoYHrxyKWfiO7jlcKFr+NcAPnHODgNHAHWaWCdwLvOecywDeC74HuAjICD4mAU+EoAYRkSb353c2si6vhAevHEpy2/Cau+dYGhz+zrk859zS4OsDwDogFbgceCa42TPAV4OvLweedZ75QJKZdWtoHSIiTenj7D1M/mgzXx+VzvmZKX6X86WFtM/fzHoBpwALgBTnXB54Bwjg0N2KU4EddX4sN7js8M+aZGaLzWxxYWFhKMsUEWmQ/JIK7p6+jH7JiTxwSXhN1Xy8Qhb+ZpYIvAR81zlXcrRN61nmvrDAucnOuSznXFZycnKoyhQRaZCa2gB3vbCMsqpa/nbdcFrHNs8bIoYk/M2sFV7wT3POvRxcnH+oOyf4XBBcngvUnd+0B7ArFHWIiDS2P8/ZyMItRfzfFUPISGnrdzknLBSjfQx4GljnnPtznVWzgInB1xOBV+ssvyE46mc0UHyoe0hEJJx9sKGAv324iQmnpvG1U3r4XU6DhOL7yunA9cAqM1seXPYT4PfADDO7GdgOXB1c9yZwMZADlAE3hqAGEZFGtWt/Od/793IGdm3LL8YP9rucBmtw+DvnPqb+fnyAsfVs74A7GrpfEZGmUllTyx3PL6W6JsDfrhtOfKtov0tqsOZ5pkJEpIk453jgP6tZtn0/T1w3nD7NZPqGY9H0DiIiR/GvT7fy4pJcvjM2g4tOajmXJCn8RUSO4JOcPfzmjXVckJnCd8dm+F1OSCn8RUTqsX1vGXc8v5S+yW14+JphYXkT9oZQ+IuIHKa0soZbnl2Ec/CPG7JIjGt5p0db3n+RiEgD1NQGuPP5pWwqPMgzN46kZ6c2fpfUKNTyFxEJcs7xwCur+XBDIb/56hDOyOjsd0mNRuEvIhL02Ps5TF+0g7vO68e1I5vPjVlORMsP/+KdflcgIs3AzCW5/HnORq4Ynsr3L+jvdzme2ho4kN8oH92yw79kFzw2AqZfB4Ub/a5GRMLUR9mF3PvSSs7o15nfXxEGd+RyDja8BU+cBjNu8N6HWMsO//j2cOYPYPNc+NtoeO1uOLDb76pEJIws37Gf26YuoV+XRP72jeHExvgci7mL4V+XwAsTwNXCmMaZDcdcIxxRQi0rK8stXrz4xD/g4B6Y+wdYPAWiW8Hob8Ppd0N8u9AVKSLNztpdJUyYPI+k1rG8eNsYUtrF+1fM3k3w3q9g7SvQJhnOuReGT/Qy6wSZ2RLnXFa96yIi/A8p2gzv/RrWvAytO8FZ90DWTRAT2/DPFpFmJaeglGuenEdsTBQzbh1DWsfW/hRyIB8++mOwcRoHp90Fp90JcQ2/V4DC/3A7l8Ccn8PWjyCpJ5xzHwz9fxDV/GfqE5Fj21FUxtV/n0dNwDHj1tH+TNZWVgSfPAILJkNtFQy/wWvtt+0asl0o/OvjHOS8C+/9Enavgs79vYNA5lchqmWfChGJZLuLK7j6yU8pKa/h37eOZmDXJu7+rSiB+U/AvMeh8gCcdLUX+p36hnxXRwv/yL3C1wwyLoC+Y2H9a/DB/8HMGyHlT3Du/TDgIm8bEWkxcveVcd1TC9h3sJppt4xq2uCvLoeF/4CPH4byIhh4qZc1KZlNV0MdkRv+h0RFQebl3j/E6pfgw9/B9Guh+3A4737v4KCDgEizt23vQb7+jwWUVFTz7M0jOTktqWl2XF0OS5+Fj/4Mpbu9TDnvAUgd3jT7PwKF/yFR0V6//+ArYMULMPdBeO5KSBvlnRjup4OASHOVU1DKdU/Np6omwAvfGs2Q1PaNv9OqMljyT/jkL1CaD+mnwVVToNfpjb/v46DwP1x0DAy/3jsQLH0WPn4Epl0J3U+Bs34EAy7WQUCkGVm/u4RvPLUAMKZPGsOArg0fRXNUlaWw+Gn49DE4WAi9zoQrn4beZzbufr+kyD3he7xqqrxvAh//GfZthZQhcNYPYdB4jQ4SCXNLtu3j5mcWER8TzbRvjaJvY47qqSiBRf+ATx/3+vT7nAtn3wM9T2u8fR7D0U74hmRYi5lNMbMCM1tdZ1lHM5tjZtnB5w7B5WZmj5pZjpmtNDN/O76OJSYWRkyEO5fA156Emkp48ZveFcMr/u3NvSEiYWf2mt18/R/zaZ/Qihm3jmm84D+4B97/LTxykneRVuoIuHkO3PCKr8F/LKEa0/gvYNxhy+4F3nPOZQDvBd8DXARkBB+TgCdCVEPjio6BkyfAHQvgqn9CVCv4zyR49BSY/3fvq56IhIVn523l9ueWMLBbO16+/TTSOzXCBVz7tsIbP4SHB8N//wA9T4dvvQ/fmAlpI0O/vxALWbePmfUCXnfODQm+3wCc45zLM7NuwIfOuQFm9mTw9QuHb3ekz/a12+dIAgHY+DZ8+ihsnwfxSXDqLTDqVkjs4nd1IhEpEHD8YfYG/j53E+cP6sJj1w4nITbE3bN5K72TuGv+AxYFJ18Dp90NyWEyE2gdfo3zTzkU6MEDwKFETAV21NkuN7jsc+FvZpPwvhmQnh6G82pHRcHAi73HjoXeL8NHf/JO8gy7FsbcBZ37+V2lSMQor6rlRzNX8PrKPL4xOp1fXDaYmOgQdW44B1vmwiePwqb3ILYtjPm2N09Yu+6h2UcT82O0T31DZb7w9cM5NxmYDF7Lv7GLapC0kTBhGuzJgXmPwfIXYMkzMPASGH2793VQI4REGs2OojJunbqEdbtLuPeigdx6Vp/QTMtcXQ6rXvSuyC1YC226wNifQdbNkNBE1wk0ksYM/3wz61an26cguDwXSKuzXQ9gVyPW0XQ694PL/uJdtbfgSW+41/rXIeUkrzvopKugVYLfVYq0KJ9u2sMd05ZSE3BM+eapnDsgBN2uJXmw6ClvsrXyIm+U3+V/hSFXQSsfZ/4MocacxGYWMDH4eiLwap3lNwRH/YwGio/W398sJXaBsT+F76+Dyx4FF4BZd8KfM73RALq7mEiDOef45ydbuP7phXRKjGPWnWc0PPh3LoGXboFHhnjduOljYOLrcNvHcMo3WkzwQ4hO+JrZC8A5QGcgH/g58AowA0gHtgNXO+eKzPsu9jje6KAy4Ebn3FHP5oblCd8vwznY+jEs+DtseBMwyBwPIyd5v1zqEhL5Ukoqqrnv5VW8sTKP8wel8PA1J9M2/gTnva8q86Z5XzzFC//Ytt6FniO/BR37hLbwJqZZPcPJvq3e5E5Lp0JlMSQPhBHf9IaRJnTwuzqRsLds+z7uemEZecUVfP+C/tx+dl+iok6gAVW40Qv8Fc9DRTF0HuDd32PY11vMjZ4U/uGo6iCsftmb+2PnEoiJh8Ff8375epyqbwMihwkEHE/+dzN/emcDKe3iefTaUxjR80s2mGqqvPNwi6d49/OIauV9C8+6qUUOzFD4h7u8ld5BYOUMqCqFLoMh60bvBLG+DYiwc385P565ko9z9nDxSV353RVDaZ/wJbp5CtbD8ue8q/IPFkBSOoy40evHb8HX5Sj8m4vKUlg902uV5K3wbuk28BIYdh30PVdzCUnEcc7x/MLt/O7N9QSc44FLMrl2ZNrxDeOsKPamaV82DXYuhqgY6D/Ouy9uv7ER8fek8G+Odi2H5c/DqhlQvg8Su3pXEg67DpIH+F2dSKPbUVTGvS+v5JOcvZzWtxMPXjn02PfZDQRg63+9wF83C2oqIHmQ18Ifeg0kJjdN8WFC4d+c1VTCxtnezKIbZ4Or9SaOOvla7xxBm85+VygSUtW1AZ6dt40/vbOBKDN+cvGgo7f2nYP8NV5DadVLUJILce29btNTrvNuzNTC+vKPl8K/pSgt8K42XDYNCtaARUOfc7xf8oGXtpgRChK55m3ayy9mrWFD/gHOGZDMb792EqlJR7gwct9WWDXTexSu8/4e+o31WvgDL9EFlSj8Wx7nvEvNV830zhHs3+6dH+j/Fe8KxP4X6hdfmpXdxRX89s11vLZiFz06JPCzSzO5IDPli639A/led86qF2HHAm9Z+hivAZT5VX0TPozCvyVzDnIXeweBNf/xbhcXm+id2Bp0GfQ7H+Ia8QYWIg1QUlHNP/67mac/3kJNwHH72X25/Zy+xLeqczJ2/w5Y95oX+tvnA86bbuGkq2DIld7IHamXwj9SBGq9K4lXz4T1b0DZXu/6gb7neQeC/uOgdUe/qxShorqWqfO28dcPc9hfVs2lQ7txz4UD/zfv/t5NXtivnQW7lnrLUoZ4d9DLHA9dBvlXfDOi8I9EtTWwY36wxfS6dxLMoqHXGcEDwYVqMUmTq6yp5aUlO3ns/Wzyiis4M6Mz91w4kJO6J3oXO258Gza87Z3TAu9kbeZ4L/Q79fW3+GZI4R/pnINdy4IHgtdgb7a3PHkQZFzgHQjSRkH0Cc6NInIMpZU1PL9gG099tIWCA5UMS0vivnO7M8ot90axZb/jfVO1aEgf7Z2wHXSZGigNpPCXz9uTHfyDmw3b5kGg2hsa1/dc70DQ7/wWfdWjNJ2Ckgqem7+NZ+Zto6S8kglpJUzqsY1e++dh2z6FQI13FXu/YCOk31hd1R5CCn85sooS7w5FG2dD9hwo3e0t75LpDSPtfbZ3E2oNI5Xj5Jxj4ZYips7fxorVaxhtK7kqKYfhtStoVbHX2yh5kBf2/cd5c1lF+3FfqZZP4S/HxznYvRI2vQ+bP/RGVtRUeF/FU0dAn7O9A0JqVoua11xCY9/BKuYsWsWGhe+QVrKUs2NW0/vQfZradPF+d/qe6z0301sfNjcKfzkx1RXeWOotc2HzXG/UhQt41xSkDvf6ZtPHeLex1Ff1iFS5ZysbFrzN/vX/JbVkGX3NC/ua6ASs52lE9zvPC/wumRF7la2fFP4SGuX7YdunsP1T71vBrmVeny14f9yHDgapI7ybYOiPvWWproDdq6jesYg96z8lPm8hHarzASihDbvbD6PdwLNJGXIu1m0YxMT6XLAo/KVxVJV5w/O2z4ft82DHQqg64K2Lbw/dT/GG6qUO957bddcBobkIBGBvjvfvu3Mx1dsXE12wmijnHezzXEdW2AAOpoyk5/ALGDZiDDEx6rcPN0cLf/1ryYmLbQ29z/Qe4F1kVrAWdi71uoh2LoVPH/3ft4PEFOg2DFIGBx9DoFM/nezzW2Wp9++2exXkr4bdq3H5a7DqgwAcJIHltX1Y4S5mS/xAkvufxphThjC2TydaRTfmbcClManlL42ruhx2r/7fwWD3Ktiz4X8HhOg4b4rqQweEzgOgcz9I6hkR8603qaqDXmu+cCPs2ej9O+xejSvajOHlQHl0IjnWi8UVqax1PVlt/enUM5PTM7pyZkZnMru1O7FbJoov1PIX/7RKgLRTvcchNVVe+OSv8Vqa+Wtg0wfetNWHRMdCh97QOcP7dnDokZQObbvqwHAk1eXeRH/7tsH+bV7Y79noXdtRvOOzzQJEsT+uG9nWi8WMYHlVD9YGerIvNoVT0jswIr0Dl/XqyK97d/z8PDvSYvjW8jezccBfgGjgKefc74+0rVr+EeLgXu/q4z3ZXmjtzfFeF232LkQ7JKoVtE/1DgTt073npDRol+p1LbVNgfiklnd+obYGDhbCgTw4sNt7LtkZDPrtXtiX5n/uR6qjE8iPTWerpbK2qivLy7uQHejONpdCTGw8A7u2ZVC3dgzs1o7h6UkMSGlLjLpyWoywa/mbWTTwV+ACIBdYZGaznHNr/ahHwkSbTt4jffTnl9fWQPF22LvZe96/3Zvpcf92yHn3fxem1RUd6x0IPnt08Sa1i0/yhqUmJAVfB5/j20Or1k1z/sE5r4VeXebds7mi2LtbW1mR91xeBGX7CJQXESjdizuwGyvNJ7q8EHOBz31UgGiKYpLJsxS2BYaQXXs2W2s7s8N1YYdLppAkOhJHesfW9EpvTUanNlwWDPz0jq3VhRPB/Or2GQnkOOc2A5jZdOByQOEvXxQd4w0d7din/vXVFV4LuGQXVcV5VO3Po7ZkdzA0C4jenU2r8k9pVVVMFIH6PyOoxlpRZXFURcVTHXyusjhqrRUOI2BRBIgiQHSd11FEUUuU+98j2tV8tizGVRMXqCDOlRPnKohzFURx9G/cB10c+2hLsWtDgUsi3w0in9MocB2C7zuQ7zpQGdeJTgmtSWkXT0q7eLq0jWNou3i+0i6enp1ak96pNe3iNWeTfJFf4Z8K7KjzPhcYVXcDM5sETAJIT9fkTpHMOUdxeTU795eza38FO/eVkVdSwd7SKooOVrH3YBV7SyspOlhFWVUboF/w8YVPIpFy2nOQ9uY9OtpBOkaX0T6qnDZWRWurJsEqSXCVxLsq4gKVJLgKWlGD4Yj+X/QT+1n0OwJEUUs0NRZNLdFUWTS1FkeAaGothsqYeCqiEqiKiqfSvOeqqNZURsVTHZNIVWwS1bHtqY3vQCCuAzFx8cTHRBPXKop28a1on9CK1ISYz163S2hFu/gYddHICfMr/Ov7rvm5ppBzbjIwGbw+/6YoSvxVWlnDpoJScgpKySn0nrfuOciu/eUcrKr93Lax0VF0SoylYxvv0adzm89et42PoU1sDG3iYkiMi6FNXHTwOYaEVtHExkTRKjqKmChTt4dELL/CPxdIq/O+BxyaBEQiQcGBClblFrMyt5jVO4tZm1dCXnHFZ+tjooyenVrTJzmRMzI6k5qUQGpSAt2TEkjtkECnNrFHvqG3iByTX+G/CMgws97ATmAC8HWfapFGFgg4sgtKWbi1iIVbili8teizoDeDfsmJjOrdkYyUtvRNTqRfl0R6dmqtC4hEGpEv4e+cqzGzO4HZeEM9pzjn1vhRizSOvOJy5m4oZO7GQuZt3sv+Mm+oZtd28ZzauyPD0pIY2qM9md3a0SZOl5uINDXf/uqcc28Cb/q1fwmtQMCxPHc/s9fs5sP1hWzI9+b46dY+ngsGpTCqTydG9upIWscEddeIhAE1ueSEBQKOZTv28cbK3by1Oo+84gpaRRun9urIfcMHcs6ALvRPSVTYi4Qhhb98adv2HmTmklxeWpLLruIKYqOjOKt/MveMG8DYQSkaVy7SDCj85bhUVNfyxso8Xlyyg/mbizCDszKSuWfcQMYO6kJbBb5Is6Lwl6PKKy5n6rxtPL9wO/vLqunZqTU/unAAVwxPpVv7BL/LE5ETpPCXei3bvo8pn2zlzVV5OOe4IDOFiaf1YkyfTurDF2kBFP7yOQs27+XR97P5JGcvbeNjuOn0XtwwphdpHVv7XZqIhJDCX3DOMW/TXv7yXjYLthSR3DaOBy4ZxLUj0zUGX6SF0l92hFuxYz//9+Y6FmwpIqVdHL+4LJMJI9N1Aw+RFk7hH6F2FJXxx3c28OryXXROjOWX4wdzzalpCn2RCKHwjzAHKqp5/P0c/vnJVqKi4K7z+nHr2X1JVPeOSETRX3yEcM7x1urd/PK1NRQcqOTK4T34wVf6a7imSIRS+EeA7XvL+Omrq5m7sZDB3dsx+fosTk5L8rssEfGRwr8Fq6kNMPmjzfzl3WxiooyfXZrJDWN66u5PIqLwb6k2F5by/RkrWL5jP+MGd+UX4wfTtX2832WJSJhQ+LcwgYBj6vxt/O6tdcTFRPPotacw/uTufpclImFG4d+C5BWX88MXV/BJzl7OGZDMg1cOJaWdWvsi8kUK/xZi7sZCvvfv5VRU1/K7K05iwqlpmoNHRI5I4d/M1QYcj7y7kcc/yGFASlv+et1w+iYn+l2WiIQ5hX8zVlBSwXemL2P+5iKuyUrjF+MHkxCrK3RF5NgU/s3U0u37uHXqEg5UVPPHq0/mqhE9/C5JRJqRBg34NrOrzWyNmQXMLOuwdfeZWY6ZbTCzC+ssHxdclmNm9zZk/5HqP8tymTB5Pgmtonn1jjMU/CLypTW05b8auAJ4su5CM8sEJgCDge7Au2bWP7j6r8AFQC6wyMxmOefWNrCOiBAIOB56ZwNPfLiJ0X068sR1I+jQJtbvskSkGWpQ+Dvn1gH1jSq5HJjunKsEtphZDjAyuC7HObc5+HPTg9sq/I/hYGUN3/33cuaszefaken8cvxgYmN0pa6InJjG6vNPBebXeZ8bXAaw47Dlo+r7ADObBEwCSE9Pb4QSm489pZV8858LWburhJ9flsk3T+ulYZwi0iDHDH8zexfoWs+q+51zrx7px+pZ5qj/HIOr7wOcc5OByQBZWVn1bhMJdhSVcf3TC9hdUsFTE7M4b2CK3yWJSAtwzPB3zp1/Ap+bC6TVed8D2BV8faTlcpi1u0qY+M+FVNUEmHbLaEb07OB3SSLSQjRWp/EsYIKZxZlZbyADWAgsAjLMrLeZxeKdFJ7VSDU0aws27+WaJ+cRE2XMvG2Mgl9EQqpBff5m9jXgMSAZeMPMljvnLnTOrTGzGXgncmuAO5xztcGfuROYDUQDU5xzaxr0X9ACfbChgFunLiGtQwJTbx5F9yTdcEVEQsucC//u9KysLLd48WK/y2gSH6z3gj8jJZGpN4+io4ZyisgJMrMlzrms+tbpCt8w8t66fG5/bikDurZl6s0jSWqt4BeRxqGB4mHi3bX53PbcEgZ2a8tzN49S8ItIo1L4h4E5a/O5fdoSMru1Y+rNo2jfupXfJYlIC6duH599nL2HO6YtJbN7e6bePJJ28Qp+EWl8avn7aNn2fUyaupg+yW149kYFv4g0HYW/TzbmH+DGfy2ic2Icz940Ul09ItKkFP4+ODRlQ2x0FM/dPIouus+uiDQx9fk3scIDlVz/9AIqqgPMuHUM6Z1a+12SiEQgtfybUHlVLbc8u5jdJRVM+eapDOja1u+SRCRCqeXfRAIBx/dnLGdl7n6e/MYIzdUjIr5Sy7+JPDh7PW+t3s39Fw/iK4PrmyFbRKTpKPybwAsLt/Pk3M1cNyqdm8/o7Xc5IiIK/8b2UXYhD7yymrP7J/PL8YN1By4RCQsK/0a0ubCUb09bSkaXRB7/+inEROt/t4iEB6VRIymtrOHWqUuIiTL+cUMWbXX1roiEEY32aQTOOe6ZuYJNhaU8e9Mo0m29eJAAAArzSURBVDpqLL+IhBe1/BvB3+du5s1Vu7n3ooGckdHZ73JERL5A4R9i/91YyEOz13Pp0G5868w+fpcjIlIvhX8I7Sgq4zvTl5HRpS1/uGqoRvaISNhS+IdIVU2AO55fSm3A8eT1I2gdq9MpIhK+GhT+ZvaQma03s5Vm9h8zS6qz7j4zyzGzDWZ2YZ3l44LLcszs3obsP5w8+PZ6VuYW89BVJ9Orcxu/yxEROaqGtvznAEOcc0OBjcB9AGaWCUwABgPjgL+ZWbSZRQN/BS4CMoFrg9s2a++ty+fpj7dww5iejBuiqRtEJPw1KPydc+8452qCb+cDPYKvLwemO+cqnXNbgBxgZPCR45zb7JyrAqYHt2228orL+eGLK8js1o6fXDzI73JERI5LKPv8bwLeCr5OBXbUWZcbXHak5V9gZpPMbLGZLS4sLAxhmaFTUxvg7unLqawJ8NjXTyG+VbTfJYmIHJdjnpU0s3eB+voy7nfOvRrc5n6gBph26Mfq2d5R/8HG1bdf59xkYDJAVlZWvdv47dH3c1i4pYg/XX0yfZMT/S5HROS4HTP8nXPnH229mU0ELgXGOucOhXQukFZnsx7AruDrIy1vVhZtLeKx97O5YngqV47ocewfEBEJIw0d7TMO+DEw3jlXVmfVLGCCmcWZWW8gA1gILAIyzKy3mcXinRSe1ZAa/HCwsoYfzFhBjw4J/OryIX6XIyLypTV0MPrjQBwwJ3hB03zn3G3OuTVmNgNYi9cddIdzrhbAzO4EZgPRwBTn3JoG1tDkfvvmOnbsK+Pfk8aQGKfx/CLS/DQouZxz/Y6y7rfAb+tZ/ibwZkP266cPNhTw/ILt3HpWH0b27uh3OSIiJ0RX+H4J+w5W8eOZKxmQ0pbvXdDf73JERE6Y+iy+hJ++upp9ZVX888ZTNaxTRJo1tfyP02srdvH6yjy+e35/Bndv73c5IiINovA/DntLK/nZq6sZlpbErWdpmmYRaf4U/sfh16+vpbSyhj9cNVT34RWRFkFJdgwfbCjgleW7+PY5/eif0tbvckREQkLhfxSllTXc//IqMrok8u1z+/pdjohIyGi0z1H8cfYG8koqmHnbacTFaHSPiLQcavkfwdLt+3hm3lZuGN2TET07+F2OiEhIKfzrUVUT4N6XVtKtXTw/GjfQ73JEREJO3T71+MdHm9mYX8qUb2Zp7h4RaZHU8j/Mzv3lPPZ+NuMGd+W8gSl+lyMi0igU/of59WtrAfjpZc3+1sIiIkek8K9j7sZC3l6zm7vOyyA1KcHvckREGo3CP6iyppZfzFpD785tuOXM3n6XIyLSqHQ2M+ipj7awZc9BnrlppMb0i0iLp5Y/kLuvjMfez+aiIV05u3+y3+WIiDQ6hT/exG2G8cClOskrIpEh4sP/0017mL0mnzvO7auTvCISMRoU/mb2azNbaWbLzewdM+seXG5m9qiZ5QTXD6/zMxPNLDv4mNjQ/4CGqA04fvP6OlKTErjlTM3TLyKRo6Et/4ecc0Odc8OA14GfBZdfBGQEH5OAJwDMrCPwc2AUMBL4uZn5NnHOy0tzWZtXwj3jBui2jCISURoU/s65kjpv2wAu+Ppy4FnnmQ8kmVk34EJgjnOuyDm3D5gDjGtIDSeqrKqGh2ZvYFhaEuNP7u5HCSIivmnwUE8z+y1wA1AMnBtcnArsqLNZbnDZkZY3uSfnbqbgQCVPfGM4ZuZHCSIivjlmy9/M3jWz1fU8Lgdwzt3vnEsDpgF3Hvqxej7KHWV5ffudZGaLzWxxYWHh8f3XHKfdxRU8+d9NXDK0GyN6dgzpZ4uINAfHbPk7584/zs96HngDr08/F0irs64HsCu4/JzDln94hP1OBiYDZGVl1XuAOFEPzd5AwMG9mq5ZRCJUQ0f7ZNR5Ox5YH3w9C7ghOOpnNFDsnMsDZgNfMbMOwRO9XwkuazKrcot5aWkuN53em7SOrZty1yIiYaOhff6/N7MBQADYBtwWXP4mcDGQA5QBNwI454rM7NfAouB2v3LOFTWwhi/lwbfX07FNrO7JKyIRrUHh75y78gjLHXDHEdZNAaY0ZL8n6pOcPXycs4efXppJu/hWfpQgIhIWIuYKX+ccf5i9ge7t47luVLrf5YiI+Cpiwn/2mnxW7NjPdy/orwu6RCTiRUT41wYcf3xnA32T23DFKb5cViAiElYiIvxfXppLTkEpP/zKAGKiI+I/WUTkqFp8ElbW1PLIu9kM7dGecUO6+l2OiEhYaPHhP23+dnbuL+eeCwdqGgcRkaAWHf6llTX89YMcTu/XiTMyOvtdjohI2GjR9/Atq6zh1F4due0cXdAlIlJXiw7/Lu3i+fv1I/wuQ0Qk7LTobh8REamfwl9EJAIp/EVEIpDCX0QkAin8RUQikMJfRCQCKfxFRCKQwl9EJAKZd9Ot8GZmhXi3iQwnnYE9fhfxJTSneptTrdC86m1OtULzqjcca+3pnEuub0WzCP9wZGaLnXNZftdxvJpTvc2pVmhe9TanWqF51ducagV1+4iIRCSFv4hIBFL4n7jJfhfwJTWneptTrdC86m1OtULzqrc51ao+fxGRSKSWv4hIBFL4i4hEIIV/A5nZXWa2wczWmNkf/K7neJjZD83MmVnY3tvSzB4ys/VmttLM/mNmSX7XdDgzGxf8t88xs3v9rudozCzNzD4ws3XB39W7/a7pWMws2syWmdnrftdyLGaWZGYzg7+z68xsjN81HYvCvwHM7FzgcmCoc24w8EefSzomM0sDLgC2+13LMcwBhjjnhgIbgft8rudzzCwa+CtwEZAJXGtmmf5WdVQ1wA+cc4OA0cAdYV4vwN3AOr+LOE5/Ad52zg0ETqYZ1K3wb5jbgd875yoBnHMFPtdzPB4G7gHC+ky/c+4d51xN8O18oIef9dRjJJDjnNvsnKsCpuM1BMKScy7PObc0+PoAXjil+lvVkZlZD+AS4Cm/azkWM2sHnAU8DeCcq3LO7fe3qmNT+DdMf+BMM1tgZnPN7FS/CzoaMxsP7HTOrfC7li/pJuAtv4s4TCqwo877XMI4TOsys17AKcACfys5qkfwGikBvws5Dn2AQuCfwW6qp8ysjd9FHUuLvoF7KJjZu0DXelbdj/f/rwPe1+hTgRlm1sf5OH72GPX+BPhK01Z0ZEer1Tn3anCb+/G6LKY1ZW3HwepZFtbfpgDMLBF4Cfiuc67E73rqY2aXAgXOuSVmdo7f9RyHGGA4cJdzboGZ/QW4F/ipv2UdncL/GJxz5x9pnZndDrwcDPuFZhbAm9ypsKnqO9yR6jWzk4DewAozA68bZamZjXTO7W7CEj9ztP+3AGY2EbgUGOvnAfUIcoG0Ou97ALt8quW4mFkrvOCf5px72e96juJ0YLyZXQzEA+3M7Dnn3Dd8rutIcoFc59yhb1Iz8cI/rKnbp2FeAc4DMLP+QCzhN6sfAM65Vc65Ls65Xs65Xni/sMP9Cv5jMbNxwI+B8c65Mr/rqcciIMPMeptZLDABmOVzTUdk3hH/aWCdc+7PftdzNM65+5xzPYK/pxOA98M4+An+De0wswHBRWOBtT6WdFzU8m+YKcAUM1sNVAETw7CF2lw9DsQBc4LfVOY7527zt6T/cc7VmNmdwGwgGpjinFvjc1lHczpwPbDKzJYHl/3EOfemjzW1JHcB04INgc3AjT7Xc0ya3kFEJAKp20dEJAIp/EVEIpDCX0QkAin8RUQikMJfRCQCKfxFRCKQwl9EJAL9fyf65QCffokUAAAAAElFTkSuQmCC\n", 547 | "text/plain": [ 548 | "
" 549 | ] 550 | }, 551 | "metadata": { 552 | "needs_background": "light" 553 | }, 554 | "output_type": "display_data" 555 | }, 556 | { 557 | "data": { 558 | "text/plain": [ 559 | "
" 560 | ] 561 | }, 562 | "metadata": {}, 563 | "output_type": "display_data" 564 | } 565 | ], 566 | "source": [ 567 | "%run examples/simple/poly_test.py" 568 | ] 569 | }, 570 | { 571 | "cell_type": "markdown", 572 | "metadata": {}, 573 | "source": [ 574 | "Blue line is $x^3$, and the orange line is $3x^2$." 575 | ] 576 | }, 577 | { 578 | "cell_type": "markdown", 579 | "metadata": {}, 580 | "source": [ 581 | "### Boston house dataset – linear regression with multiple variables\n", 582 | "\n", 583 | "Description: 13 numerical attributes, and our target is predicting house price.\n", 584 | "\n", 585 | "Criterion: Mean Square Error" 586 | ] 587 | }, 588 | { 589 | "cell_type": "code", 590 | "execution_count": 7, 591 | "metadata": {}, 592 | "outputs": [ 593 | { 594 | "name": "stderr", 595 | "output_type": "stream", 596 | "text": [ 597 | "100%|██████████| 200/200 [00:00<00:00, 1747.03it/s]\n" 598 | ] 599 | }, 600 | { 601 | "data": { 602 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD4CAYAAAAZ1BptAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAc2klEQVR4nO3de5BcZ53e8e/T3dOju+TL6IIkW2ItG7xODEJlKwtL1aLFlr27yNldUqY2sUJUUYU1CYSkwISqmECoMtnKElwBb3mxgrzLIhwWygplECoBYbPri8YgX2VbYwHWoNvYsm6WdZmZX/7ot0c9rdM9PdJ099j9fKqm+vSv33POqzOtfuY97+luRQRmZmZZcu3ugJmZTV4OCTMzq8khYWZmNTkkzMysJoeEmZnVVGh3BybapZdeGkuWLGl3N8zM3lAef/zxlyOip7r+pguJJUuW0Nvb2+5umJm9oUj6VVbdp5vMzKwmh4SZmdXUUEhImiPp25Kek7RT0j+RdLGkrZJ2pduLUltJultSn6QnJS2v2M7a1H6XpLUV9XdJeiqtc7ckpXrmPszMrDUaHUl8GfhBRLwNuBbYCdwBbIuIZcC2dB/gJmBZ+lkP3AOlF3zgTuB64DrgzooX/XtS2/J6q1O91j7MzKwFxgwJSbOA9wL3AUTE6Yg4DKwBNqZmG4Fb0vIa4P4oeQSYI2kBcCOwNSIORcSrwFZgdXpsVkQ8HKUPkrq/altZ+zAzsxZoZCTxVmAA+F+Sfi7pa5KmA/MiYh9Aup2b2i8E9lSs359q9er9GXXq7GMUSesl9UrqHRgYaOCfZGZmjWgkJArAcuCeiHgn8Br1T/sooxbnUW9YRNwbESsiYkVPzzmX+ZqZ2XlqJCT6gf6IeDTd/zal0DiQThWRbg9WtF9csf4iYO8Y9UUZdersY8J99+f9/PUjmZcJm5l1rDFDIiL2A3skXZVKq4Bngc1A+QqltcCDaXkzcFu6ymklcCSdKtoC3CDpojRhfQOwJT12TNLKdFXTbVXbytrHhPs/T+xj0/aXmrV5M7M3pEbfcf1vgW9IKgK7gQ9TCpgHJK0DXgI+mNo+BNwM9AEnUlsi4pCkzwPbU7vPRcShtPwR4OvAVOD76Qfgrhr7mHDdhRynzgw3a/NmZm9IDYVEROwAVmQ8tCqjbQC319jOBmBDRr0XuCaj/krWPpqhu5Dj1KBDwsyskt9xnUzpynNqcKjd3TAzm1QcEolHEmZm53JIJN1dec9JmJlVcUgkpZHEEKUpFTMzA4fEiO5CjuGAwWGHhJlZmUMi6S7kATwvYWZWwSGRdHeVDsWpM77CycyszCGRdBdSSHgkYWY2wiGRlE83nfRIwsxshEMi8UjCzOxcDolkZE7CIWFmNsIhkYxc3eTTTWZmIxwSiU83mZmdyyGR+H0SZmbnckgkZ+ckfLrJzKzMIZGMnG7yh/yZmY1wSCRTuny6ycysmkMiOTtx7dNNZmZlDonEE9dmZudySCRFz0mYmZ3DIZHkc6IrL59uMjOr4JCo0F3I+3STmVkFh0SF8leYmplZiUOiQnch5zkJM7MKDYWEpF9KekrSDkm9qXaxpK2SdqXbi1Jdku6W1CfpSUnLK7azNrXfJWltRf1daft9aV3V20ezdHf5dJOZWaXxjCR+JyLeEREr0v07gG0RsQzYlu4D3AQsSz/rgXug9IIP3AlcD1wH3Fnxon9Palteb/UY+2iK7kLOXzpkZlbhQk43rQE2puWNwC0V9fuj5BFgjqQFwI3A1og4FBGvAluB1emxWRHxcEQEcH/VtrL20RSlOQmPJMzMyhoNiQB+KOlxSetTbV5E7ANIt3NTfSGwp2Ld/lSrV+/PqNfbR1OUrm7ySMLMrKzQYLt3R8ReSXOBrZKeq9NWGbU4j3rDUnCtB7jsssvGs+oo3V05jp8aPO/1zczebBoaSUTE3nR7EPgupTmFA+lUEen2YGreDyyuWH0RsHeM+qKMOnX2Ud2/eyNiRUSs6OnpaeSflMlXN5mZjTZmSEiaLmlmeRm4AXga2AyUr1BaCzyYljcDt6WrnFYCR9Kpoi3ADZIuShPWNwBb0mPHJK1MVzXdVrWtrH00hU83mZmN1sjppnnAd9NVqQXgbyLiB5K2Aw9IWge8BHwwtX8IuBnoA04AHwaIiEOSPg9sT+0+FxGH0vJHgK8DU4Hvpx+Au2rsoyk8cW1mNtqYIRERu4FrM+qvAKsy6gHcXmNbG4ANGfVe4JpG99Esfp+Emdlofsd1hdKchE83mZmVOSQqdHf5dJOZWSWHRIXyp8CWzpiZmZlDokL5K0xPD3k0YWYGDolRzn7PtUPCzAwcEqN0d6XvufYb6szMAIfEKGdHEr7CycwMHBKj+HSTmdloDokK3QWfbjIzq+SQqNDdVTocJ326ycwMcEiMMnK6ySMJMzPAITHKyOkmjyTMzACHxCjlkcRJjyTMzACHxCjlkDjjd1ybmQEOiVGK5Y/l8CWwZmaAQ2KUoj+7ycxsFIdEha68RxJmZpUcEhWKnpMwMxvFIVGhmPfHcpiZVXJIVCj6dJOZ2SgOiQq5nCjk5IlrM7PEIVGlWMhxxiMJMzPAIXGOYiHnkYSZWeKQqFLM5zwnYWaWOCSqdDkkzMxGNBwSkvKSfi7pe+n+UkmPStol6VuSiqnene73pceXVGzj06n+vKQbK+qrU61P0h0V9cx9NFO3TzeZmY0Yz0jiY8DOivtfBL4UEcuAV4F1qb4OeDUirgC+lNoh6WrgVuA3gdXAV1Pw5IGvADcBVwMfSm3r7aNpigWPJMzMyhoKCUmLgN8DvpbuC3gf8O3UZCNwS1pek+6THl+V2q8BNkXEqYj4BdAHXJd++iJid0ScBjYBa8bYR9N44trM7KxGRxL/A/gkUH71vAQ4HBGD6X4/sDAtLwT2AKTHj6T2I/WqdWrV6+1jFEnrJfVK6h0YGGjwn5TNcxJmZmeNGRKSfh84GBGPV5YzmsYYj01U/dxixL0RsSIiVvT09GQ1aVgxn/NnN5mZJYUG2rwb+ICkm4EpwCxKI4s5kgrpL/1FwN7Uvh9YDPRLKgCzgUMV9bLKdbLqL9fZR9MUCzlOnBgcu6GZWQcYcyQREZ+OiEURsYTSxPOPIuJPgB8Df5yarQUeTMub033S4z+KiEj1W9PVT0uBZcBjwHZgWbqSqZj2sTmtU2sfTVMs5PwBf2ZmyYW8T+JTwCck9VGaP7gv1e8DLkn1TwB3AETEM8ADwLPAD4DbI2IojRI+CmyhdPXUA6ltvX00TTHviWszs7JGTjeNiIifAD9Jy7spXZlU3eYk8MEa638B+EJG/SHgoYx65j6aqVjwnISZWZnfcV3FH8thZnaWQ6KK30xnZnaWQ6KK3ydhZnaWQ6JKaU4i8+0YZmYdxyFRpfyxHKUrcM3MOptDokp3IX3Pta9wMjNzSFTrypc+DcTzEmZmDolzFPOlQ+J5CTMzh8Q5ioU84JGEmRk4JM5RLM9JOCTMzBwS1UbmJIaG2twTM7P2c0hUGbm6adBzEmZmDokqRV8Ca2Y2wiFRpZj3xLWZWZlDoorfJ2FmdpZDokr5dJO/U8LMzCFxjnJI+CtMzcwcEufwZzeZmZ3lkKjSlfeb6czMyhwSVTwnYWZ2lkOiStEjCTOzEQ6JKv7sJjOzsxwSVUbmJHy6yczMIVHNp5vMzM4aMyQkTZH0mKQnJD0j6b+k+lJJj0raJelbkoqp3p3u96XHl1Rs69Op/rykGyvqq1OtT9IdFfXMfTRTLie68vJIwsyMxkYSp4D3RcS1wDuA1ZJWAl8EvhQRy4BXgXWp/Trg1Yi4AvhSaoekq4Fbgd8EVgNflZSXlAe+AtwEXA18KLWlzj6aqpjPeSRhZkYDIRElx9PdrvQTwPuAb6f6RuCWtLwm3Sc9vkqSUn1TRJyKiF8AfcB16acvInZHxGlgE7AmrVNrH03VVXBImJlBg3MS6S/+HcBBYCvwInA4IgZTk35gYVpeCOwBSI8fAS6prFetU6t+SZ19VPdvvaReSb0DAwON/JPqKuZzfp+EmRkNhkREDEXEO4BFlP7yf3tWs3SrGo9NVD2rf/dGxIqIWNHT05PVZFyKHkmYmQHjvLopIg4DPwFWAnMkFdJDi4C9abkfWAyQHp8NHKqsV61Tq/5ynX00VbGQ45RHEmZmDV3d1CNpTlqeCvwusBP4MfDHqdla4MG0vDndJz3+o4iIVL81Xf20FFgGPAZsB5alK5mKlCa3N6d1au2jqTxxbWZWUhi7CQuAjekqpBzwQER8T9KzwCZJ/xX4OXBfan8f8FeS+iiNIG4FiIhnJD0APAsMArdHxBCApI8CW4A8sCEinknb+lSNfTRVseA5CTMzaCAkIuJJ4J0Z9d2U5ieq6yeBD9bY1heAL2TUHwIeanQfzeaRhJlZid9xncET12ZmJQ6JDF35nN9xbWaGQyKTRxJmZiUOiQzFgkcSZmbgkMg0rSvPydND7e6GmVnbOSQyTCvmec0hYWbmkMgyrbvAidODYzc0M3uTc0hkmF7Mc2YoPHltZh3PIZFhWrH0HkOPJsys0zkkMkzvzgN4XsLMOp5DIsPISOKURxJm1tkcEhk8kjAzK3FIZPBIwsysxCGRYXoKCY8kzKzTOSQyTEunm3x1k5l1OodEhukjl8B6JGFmnc0hkaE8knjNcxJm1uEcEhmmdZVPN3kkYWadzSGRoZDP0V3I8ZrnJMyswzkkaphWzHPilEcSZtbZHBI1TCsWPJIws47nkKhherdHEmZmDokaPJIwM3NI1DS9O++rm8ys4zkkaphWLPh9EmbW8cYMCUmLJf1Y0k5Jz0j6WKpfLGmrpF3p9qJUl6S7JfVJelLS8optrU3td0laW1F/l6Sn0jp3S1K9fbTC9KJHEmZmjYwkBoH/EBFvB1YCt0u6GrgD2BYRy4Bt6T7ATcCy9LMeuAdKL/jAncD1wHXAnRUv+vektuX1Vqd6rX00nb/n2sysgZCIiH0R8bO0fAzYCSwE1gAbU7ONwC1peQ1wf5Q8AsyRtAC4EdgaEYci4lVgK7A6PTYrIh6OiADur9pW1j6aziMJM7NxzklIWgK8E3gUmBcR+6AUJMDc1GwhsKditf5Uq1fvz6hTZx/V/VovqVdS78DAwHj+STVNKxY4cXqI4eGYkO2Zmb0RNRwSkmYAfwt8PCKO1muaUYvzqDcsIu6NiBURsaKnp2c8q9ZU/na61894NGFmnauhkJDURSkgvhER30nlA+lUEen2YKr3A4srVl8E7B2jviijXm8fTTdt5IuHPC9hZp2rkaubBNwH7IyIP694aDNQvkJpLfBgRf22dJXTSuBIOlW0BbhB0kVpwvoGYEt67JiklWlft1VtK2sfTVceSfhd12bWyQoNtHk38C+ApyTtSLX/BNwFPCBpHfAS8MH02EPAzUAfcAL4MEBEHJL0eWB7ave5iDiUlj8CfB2YCnw//VBnH03nkYSZWQMhERH/j+x5A4BVGe0DuL3GtjYAGzLqvcA1GfVXsvbRCv52OjMzv+O6pqlFfzudmZlDooaROQmPJMysgzkkaiifbjp+0iMJM+tcDokaemZ2AzBw/FSbe2Jm1j4OiRqmdOWZM62LfUdeb3dXzMzaxiFRx/xZU9h/xCMJM+tcDok65s+ewv6jHkmYWedySNSxYLZHEmbW2RwSdcybNYWXj5/i9OBwu7tiZtYWDok65s+aAsDBYyfb3BMzs/ZwSNQxf3YpJA4cdUiYWWdySNRRDol9RxwSZtaZHBJ1LJg1FYD9Dgkz61AOiTpmTS0wpSvnkDCzjuWQqEMSC2ZPZb/nJMysQzkkxjBvVrcnrs2sYzkkxrBg9lT2HnZImFlnckiM4fJLprH3yOucPOPvlTCzzuOQGMNV82YSAX0Hj7e7K2ZmLeeQGMOV82cC8Pz+Y23uiZlZ6zkkxnD5xdMoFnI8f8AhYWadxyExhkI+xxU9MzySMLOO5JBowNvmz+QFjyTMrAM5JBpw5fyZ7DtykiOvn2l3V8zMWmrMkJC0QdJBSU9X1C6WtFXSrnR7UapL0t2S+iQ9KWl5xTprU/tdktZW1N8l6am0zt2SVG8f7XDVvNLk9S6PJsyswzQykvg6sLqqdgewLSKWAdvSfYCbgGXpZz1wD5Re8IE7geuB64A7K17070lty+utHmMfLTdyhZNDwsw6zJghERE/BQ5VldcAG9PyRuCWivr9UfIIMEfSAuBGYGtEHIqIV4GtwOr02KyIeDgiAri/altZ+2i5t8yewozugievzazjnO+cxLyI2AeQbuem+kJgT0W7/lSrV+/PqNfbxzkkrZfUK6l3YGDgPP9JtUniynm+wsnMOs9ET1wroxbnUR+XiLg3IlZExIqenp7xrt6Qq+bP4oUDxygNeMzMOsP5hsSBdKqIdHsw1fuBxRXtFgF7x6gvyqjX20dbXDVvBq+eOMPA8VPt7IaZWUudb0hsBspXKK0FHqyo35aucloJHEmnirYAN0i6KE1Y3wBsSY8dk7QyXdV0W9W2svbRFuXJ6xf2+zOczKxzNHIJ7DeBh4GrJPVLWgfcBbxf0i7g/ek+wEPAbqAP+EvgTwEi4hDweWB7+vlcqgF8BPhaWudF4PupXmsfbVG+DPa5/Ufb2Q0zs5YqjNUgIj5U46FVGW0DuL3GdjYAGzLqvcA1GfVXsvbRLpfM6ObSGUW/89rMOorfcT0OV82fyfMHfLrJzDqHQ2Icrpw3kxf2H2No2Fc4mVlncEiMwzVvmc3rZ4bYPeDRhJl1BofEOFy7eDYAT/QfaXNPzMxawyExDksvncH0Yp6n+g+3uytmZi3hkBiHfE5cs3C2RxJm1jEcEuN07eI5PLvvKKcHh9vdFTOzpnNIjNM/Wjib04PDfr+EmXUEh8Q4XbtoDgAPv/hKm3tiZtZ8DolxWnzxVJZfNoe7fvAc3/lZ/9grmJm9gTkkxkkS96+7nuuWXMwnv/0kh1473e4umZk1jUPiPMzoLvDJ1VcxOBz83a6J/5IjM7PJwiFxnv7xojnMmdbFT194ud1dMTNrGofEecrnxHuuuJSf7hrwt9WZ2ZuWQ+ICvPfKHgaOneLZfUc5eWao3d0xM5twDokL8N5lpe/TXvM//553fX4rB46ebHOPzMwmlkPiAsyfPYV/t2oZf7R8Ea+dHmLTY3va3SUzswk15jfTWX2feP+VAOw98jrffOwlbv+d36CQd/aa2ZuDX80myD9feTn7j55k23MH290VM7MJ45CYIKveNpe3zJ7CV3/yoq92MrM3DYfEBCnkc/z791/JE3sO870n97W7O2ZmE8IhMYH+cPki3r5gFl/8wXP+uA4ze1NwSEygfE589g+u5uDRU9z85b/jwR2/5vAJh4WZvXE5JCbY9W+9hO/86W8xtZjnY5t2cN0XtrHxH37peQoze0PSm+3Fa8WKFdHb29vubjA4NMwT/Yf5yo9f5EfPHWTZ3BnMndXNh39rKavePhdJ7e6imdkISY9HxIpz6pM9JCStBr4M5IGvRcRd9dpPlpAoGx4ONvz9L3hk9yu8cOA4Lx06wfxZUwiCqV15rpg7k3XvWcpV82cyo7tAseDBnZm13hsyJCTlgReA9wP9wHbgQxHxbK11JltIVDozNMw3HvkVT/76CF25HK+fGeIfXnyZl4+X5i0KOXHF3BksumgqPTO7uXRGN7OmdDGtO8+M7gLTiwWmdxeY0V0gnxP5nJgzrYtpxTz5nMipVMtL5HIeqZhZ42qFxGR/x/V1QF9E7AaQtAlYA9QMicmsK5/jX7576ajayTNDbHlmP4dPnOHgsZPs3HeMXx8+yY49R3jltVOcb4bnBIVcjkJeI4FS3lb5DwNJ5FSacB9ZlsY8FVb9cOV9ofptRz3WWJBNxB8yF7qFifhbKi6wFxP595x09ndVWi79Psbzp0Wj3Wn099f49hps1+AWG97eBP89PZ7ndaMtv/mvV7Lk0unn16EaJntILAQqPxCpH7i+upGk9cB6gMsuu6w1PZsgU7ryrHnHwszHhoaDE6cHOXF6iOOnBnnt1CDHTw1y/OQgwxEMDgevnjjDydNDDEUwNBwMD8fI8pmhYGh4mDNDwXDEyAtB9T6GIxgORtatVP08Puc/XmQupnWjVtNx/4ebiCmcC93ERMwjXfAWJmKAGGd/FxGl32hU1Rr9tzbanUYPXePbm9j+Ndqw0Rid6H9vo9ucVsyPY4uNmewhkXVYznl5iYh7gXuhdLqp2Z1qlXxOzJzSxcwpXcxrd2fMrCNN9lnSfmBxxf1FwN429cXMrONM9pDYDiyTtFRSEbgV2NzmPpmZdYxJfbopIgYlfRTYQukS2A0R8Uybu2Vm1jEmdUgARMRDwEPt7oeZWSea7KebzMysjRwSZmZWk0PCzMxqckiYmVlNk/qzm86HpAHgV+e5+qXAyxPYnYkyWfsFk7dv7tf4uF/jN1n7dr79ujwieqqLb7qQuBCSerM+4KrdJmu/YPL2zf0aH/dr/CZr3ya6Xz7dZGZmNTkkzMysJofEaPe2uwM1TNZ+weTtm/s1Pu7X+E3Wvk1ovzwnYWZmNXkkYWZmNTkkzMysJodEImm1pOcl9Um6o439WCzpx5J2SnpG0sdS/bOSfi1pR/q5uQ19+6Wkp9L+e1PtYklbJe1Ktxe1uE9XVRyTHZKOSvp4u46XpA2SDkp6uqKWeYxUcnd6zj0paXmL+/Vnkp5L+/6upDmpvkTS6xXH7i9a3K+avztJn07H63lJN7a4X9+q6NMvJe1I9VYer1qvD817jkVEx/9Q+hjyF4G3AkXgCeDqNvVlAbA8Lc8EXgCuBj4L/Mc2H6dfApdW1f4bcEdavgP4Ypt/j/uBy9t1vID3AsuBp8c6RsDNwPcpfQPjSuDRFvfrBqCQlr9Y0a8lle3acLwyf3fp/8ETQDewNP2fzbeqX1WP/3fgP7fheNV6fWjac8wjiZLrgL6I2B0Rp4FNwJp2dCQi9kXEz9LyMWAnpe/6nqzWABvT8kbgljb2ZRXwYkSc7zvuL1hE/BQ4VFWudYzWAPdHySPAHEkLWtWviPhhRAymu49Q+ubHlqpxvGpZA2yKiFMR8Qugj9L/3Zb2S6Uv2P5nwDebse966rw+NO055pAoWQjsqbjfzyR4YZa0BHgn8GgqfTQNGTe0+rROEsAPJT0uaX2qzYuIfVB6AgNz29CvslsZ/R+33cerrNYxmkzPu39F6S/OsqWSfi7p/0r67Tb0J+t3N1mO128DByJiV0Wt5cer6vWhac8xh0SJMmptvTZY0gzgb4GPR8RR4B7gN4B3APsoDXdb7d0RsRy4Cbhd0nvb0IdMKn297QeA/51Kk+F4jWVSPO8kfQYYBL6RSvuAyyLincAngL+RNKuFXar1u5sUxwv4EKP/GGn58cp4fajZNKM2rmPmkCjpBxZX3F8E7G1TX5DURekJ8I2I+A5ARByIiKGIGAb+kiYNs+uJiL3p9iDw3dSHA+Xha7o92Op+JTcBP4uIA6mPbT9eFWodo7Y/7yStBX4f+JNIJ7HT6ZxX0vLjlM79X9mqPtX53U2G41UA/hD4VrnW6uOV9fpAE59jDomS7cAySUvTX6S3Apvb0ZF0vvM+YGdE/HlFvfI84j8Fnq5et8n9mi5pZnmZ0qTn05SO09rUbC3wYCv7VWHUX3ftPl5Vah2jzcBt6QqUlcCR8imDVpC0GvgU8IGIOFFR75GUT8tvBZYBu1vYr1q/u83ArZK6JS1N/XqsVf1Kfhd4LiL6y4VWHq9arw808znWihn5N8IPpasAXqD0V8Bn2tiP91AaDj4J7Eg/NwN/BTyV6puBBS3u11spXVnyBPBM+RgBlwDbgF3p9uI2HLNpwCvA7IpaW44XpaDaB5yh9FfculrHiNKpgK+k59xTwIoW96uP0vnq8vPsL1LbP0q/4yeAnwF/0OJ+1fzdAZ9Jx+t54KZW9ivVvw78m6q2rTxetV4fmvYc88dymJlZTT7dZGZmNTkkzMysJoeEmZnV5JAwM7OaHBJmZlaTQ8LMzGpySJiZWU3/H0AZC2WXvCz7AAAAAElFTkSuQmCC\n", 603 | "text/plain": [ 604 | "
" 605 | ] 606 | }, 607 | "metadata": { 608 | "needs_background": "light" 609 | }, 610 | "output_type": "display_data" 611 | } 612 | ], 613 | "source": [ 614 | "%run examples/regression/boston.py" 615 | ] 616 | }, 617 | { 618 | "cell_type": "markdown", 619 | "metadata": {}, 620 | "source": [ 621 | "### Iris dataset – classification \n", 622 | "\n", 623 | "Description: 4 attributes, predict one of class over 3 classes.\n", 624 | "\n", 625 | "Criterion: Cross Entropy" 626 | ] 627 | }, 628 | { 629 | "cell_type": "code", 630 | "execution_count": 8, 631 | "metadata": {}, 632 | "outputs": [ 633 | { 634 | "name": "stderr", 635 | "output_type": "stream", 636 | "text": [ 637 | "100%|██████████| 300/300 [00:00<00:00, 1527.80it/s]\n" 638 | ] 639 | }, 640 | { 641 | "data": { 642 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXzcVb3/8ddnJluzNU2bdEmXdKV7oaRlU5C99CIgooDKot7LVbk/8bqiXr1y9eKKVxRFlssionBBEGSvQKFshXSle5ouaZs2nSTNnkwyM+f3x0zSrE2KTeeb5P18PPLIZOabmc+Z7+SdM2fO93zNOYeIiHiXL94FiIjIkSmoRUQ8TkEtIuJxCmoREY9TUIuIeFxCf9zpqFGjXH5+fn/ctYjIoLRq1apy51xOd7f1S1Dn5+dTWFjYH3ctIjIomdnunm7T0IeIiMcpqEVEPE5BLSLicQpqERGPU1CLiHicglpExOMU1CIiHuepoP71y0W8ti0Q7zJERDzFU0F95/Ji3ihSUIuItOepoPb7jIjOYyAi0oGngtoMwkpqEZEOPBXU0R61glpEpD1vBbWZetQiIp14Kqh9GqMWEenCW0FtEFFSi4h04Kmg9psR1hi1iEgHngpqn8/UoxYR6cRTQa1ZHyIiXXkqqH1mhJXTIiIdeCyo9WGiiEhnngpqv0/zqEVEOvNUUPtMY9QiIp0pqEVEPK7XoDazE8xsbbuvGjP7Sn8Uo6EPEZGuEnrbwDm3FTgRwMz8wD7gyf4oxufTrA8Rkc6OdujjXKDYObe7X4oxcBr6EBHp4GiD+irgz93dYGY3mFmhmRUGAh/sLC1aPU9EpKs+B7WZJQGXAI91d7tz7m7nXIFzriAnJ+eDFaMxahGRLo6mR30RsNo5V9ZfxfjN0MiHiEhHRxPUV9PDsMex4vOh1fNERDrpU1CbWSpwPvBEvxajMWoRkS56nZ4H4JxrAEb2cy1aPU9EpBueOjLRryMTRUS68FRQmxnhSLyrEBHxFk8Ftd+nZU5FRDrzWFDrnIkiIp15Kqi1ep6ISFfeC2oNfYiIdOCpoNbQh4hIV54K6miPOt5ViIh4i6eC2u9DY9QiIp14Kqh1CLmISFfeCmodQi4i0oWnglonDhAR6cpbQe0zlNMiIh15KqjNdAi5iEhnngpqv2ketYhIZ94Kap0zUUSkC08Ftc+ncyaKiHTmraA2nTNRRKQzTwW1pueJiHTlqaD2+QzQzA8RkfY8FdR+iwW1hj9ERNp4Kqhbe9QapxYROaxPQW1mWWb2uJltMbPNZnZavxTT2qPWUqciIm0S+rjd7cALzrkrzCwJSO2PYvyxfxvqUYuIHNZrUJtZJnAmcD2Ac64ZaO6PYnwaoxYR6aIvQx9TgABwv5mtMbN7zSyt80ZmdoOZFZpZYSAQ+GDFmGZ9iIh01pegTgAWAnc6504C6oGbO2/knLvbOVfgnCvIycn5QMX4Wz9MVFCLiLTpS1DvBfY651bGfn6caHAf+2I060NEpIteg9o5dwDYY2YnxK46F9jUL8VY62P2x72LiAxMfZ318f+Ah2MzPnYAn+2PYloPeNHQh4jIYX0KaufcWqCgn2s5PPShoBYRaeOpIxNbe9Qa+hAROcxTQe3TAS8iIl14K6g1Ri0i0oWngrp1HrWOTBQROcxbQa1DyEVEuvBUUJuGPkREuvBUULcNfWiZUxGRNh4L6uh3zfoQETnMU0GtZU5FRLryZlBrjFpEpI2nglrLnIqIdOWpoG474EVDHyIibTwV1K09auW0iMhhngrq1vWoNfQhInKYt4JaZ3gREenCU0Ht16wPEZEuvBXUbYsyxbkQEREP8VRQm8aoRUS68FRQa5lTEZGuvBXUWj1PRKQLTwW1Tz1qEZEuvBXUWpRJRKSLhL5sZGa7gFogDISccwX9UczhoY/+uHcRkYGpT0Edc7ZzrrzfKuHwWcg1j1pE5DBPDX1o1oeISFd9DWoHvGRmq8zshu42MLMbzKzQzAoDgcAHK0ar54mIdNHXoD7DObcQuAi40czO7LyBc+5u51yBc64gJyfngxWjQ8hFRLroU1A750pj3w8CTwKL+6MYnThARKSrXoPazNLMLKP1MnABsKFfiokdQq6cFhE5rC+zPkYDT1p0WCIB+JNz7oX+KEYHvIiIdNVrUDvndgALjkMtOoRcRKQbHp2eF+dCREQ8xFNB3brM6U9f2MK9K3bEtxgREY/wVFC3Dn0A/OjZzXGsRETEO7wV1D7rfSMRkSHGU0Ft7XrUWamJcaxERMQ7PBXU7Y1MS4p3CSIinuDZoE5J9Me7BBERT/BsUDc2h+NdgoiIJ3g2qOubQ/EuQUTEEzwb1A1B9ahFRMCDQb3lh0v4wllTqW8O4bTmh4iI94I6JdFP5rAEIg6CIZ08UUTEc0ENkJYUXSuqPqhxahERTwZ1alJ0al6DZn6IiHgzqNOSYz1qzfwQEfFmULf2qOs180NExJtB3dqjblCPWkTEm0GtHrWIyGGeDOrWWR/qUYuIeDSoU5NjPWrN+hAR8WZQt/WoNY9aRMSbQT0sUT1qEZFWfQ5qM/Ob2Roze6Y/CwLw+YzUJL961CIiHF2P+ibguJ1xNjUpQT1qERH6GNRmNh74J+De/i3nsJREH8GQglpEpK896l8B3wSO23J2yQk+gi1aPU9EpNegNrOLgYPOuVW9bHeDmRWaWWEgEPiHC0tO8KtHLSJC33rUZwCXmNku4BHgHDP7Y+eNnHN3O+cKnHMFOTk5/3Bh0aEP9ahFRHoNaufct51z451z+cBVwCvOuc/0d2HJCX6aWtSjFhHx5DxqgGT1qEVEAEg4mo2dc8uB5f1SSScpCX59mCgigsd71E36MFFExMNBrel5IiKAh4M6JVHT80REwMNBnZzgo0k9ahERLwd1tEftnIt3KSIiceXZoE5J9BFxEIooqEVkaPNsUCcnRNek1lxqERnqvBvUidHSdHSiiAx1ng3qFPWoRUQADwd1a486qB61iAxx3g3qhNahD/WoRWRo825QJ7YOfahHLSJDm3eDOtaj1hi1iAx1ng3qlFiPWrM+RGSo82xQq0ctIhLl4aDW9DwREfBwUKfogBcREcDDQa0etYhIlHeDWge8iIgAHg5qHUIuIhLl2aBO9Btm6lGLiHg2qM2M5AQfG0trCNQG412OiEjceDaoIbrOx8tbDvLNx9fFuxQRkbjpNajNLMXM3jWzdWa20cxuOR6FtVdWox61iAxdfelRB4FznHMLgBOBJWZ2av+WFXXf9QWMHZ5Cgt+Ox8OJiHhSr0HtoupiPybGvo7LiQzPmTma06aOpLK++Xg8nIiIJ/VpjNrM/Ga2FjgILHPOrexmmxvMrNDMCgOBwDErcERqEocU1CIyhPUpqJ1zYefcicB4YLGZze1mm7udcwXOuYKcnJxjVmB2WhL1zWEdSi4iQ9ZRzfpwzlUBy4El/VJNN0akJgFQ1dByvB5SRMRT+jLrI8fMsmKXhwHnAVv6u7BW2WmJABqnFpEhK6EP24wFHjQzP9Fg/z/n3DP9W9ZhrT3qQw0KahEZmnoNaufceuCk41BLt7LTokGtHrWIDFWePjIRIEs9ahEZ4gZAUGuMWkSGNs8HdaLfR2ZKguZSi8iQ5fmgBhiVkcyBmqZ4lyEiEhcDIqjn5w1ndUkVzh2XI9dFRDxlQAR1QX42gdogJZUN8S5FROS4GxBBvSg/G4D3dh2KcyUiIsffgAjq6bnpDB+WSOGuyniXIiJy3A2IoPb5jBPGZFAcqOt9YxGRQWZABDXApOxUdldojFpEhp6BE9QjUzlYG6ShORTvUkREjqsBE9QTR6YBaOaHiAw5AyaoJ2WnAmj4Q0SGnAET1PmxHvULGw6ws7w+ztWIiBw/Ayaoh8cWZ3pyzT6+8siaOFcjInL8DJigBjh3Zi4A5XVaoElEho4BFdT3XFvA1Ysn6kS3IjKkDKig9vmM3IxkKuqbaQlH4l2OiMhxMaCCGiAnIxmACg1/iMgQMeCCOjcW1AdrtT61iAwNAy+oM1MA+My9K7n5L+vjXI2ISP8bcEHdOvRR0xTinR0Vca5GRKT/9RrUZjbBzF41s81mttHMbjoehfVkVHpS2+W9hxoJ6UNFERnk+tKjDgFfc87NAk4FbjSz2f1bVs+SE/yHC4s49ldrrFpEBrdeg9o5t985tzp2uRbYDOT1d2F9pbU/RGSwO6oxajPLB04CVnZz2w1mVmhmhYFA4NhU14OfXzGfby2ZCcDuSq37ISKDW5+D2szSgb8AX3HO1XS+3Tl3t3OuwDlXkJOTcyxr7OITBRO44cwpJPl9lKhHLSKDXJ+C2swSiYb0w865J/q3pL7x+4zx2cM09CEig15fZn0Y8L/AZufcL/u/pL6bNTaTwt2VmvkhIoNaX3rUZwDXAOeY2drY19J+rqtPLp43lvK6Zlbu1NnJRWTwSuhtA+fcG4Adh1qO2tkzc0lL8vP02lLOmDYq3uWIiPSLAXdkYnspiX4umDOG5zfspzmk4Q8RGZwGdFADXLJgHDVNIV7f1r9TAkVE4mXAB/UZ00aRlZrIX1bvjXcpIiL9YsAHdVKCj0+fMpHnNxzgD2/vwjkX75JERI6pAR/UAP9+3gzOnJHD95/ayDcf19KnIjK4DIqgTvD7uO+6Aj5ZMJ4n1uyjuqGl7bbiQB3LNpXFsToRkX/MoAhqiIb1JwsmEI44Xi86/MHib1/Zzo1/Wq1zLIrIgDVoghrgpIkjyEpN5NUtB9uuKzpYR3MowvaDdXGsTETkgxtUQe33GWfNyOH1onKcc0QijuJANKA3lXZcR6o5FGFfVWM8yhQROSqDKqgBFk/OprwuyDs7Klm2uYyG5jAAG2NB7ZyjORThD2/v4rzbXqO2qeUI9yYiEn+9HkI+0CzKzwbg+vvfJRg7WjEpwcfG0moA7ntzF3e9Vsyiydk0toRZu6eKD0/PoTkUISlh0P3fEpFBYNAl07ScdIYPS2wLaYDzZuWyfm81FXVBlm06wMHaIK9tjX7g+EZROdfe9y4n/2gZVQ3NH/hxwxHN35b+F444dld442QZ1Y0t1AdDR9wmFI7Q1BI+ThUNXoMuqH0+Y1H+CFKTDp9b8avnz6A5HOHnL25lTUkVAHWxF9hdr+/g9W0BaptCbbcBHKpv5jcvF3V4kTU0h2hsDvPwyt08s7607bbSqkbm/+BFXthwoEs9rQFeuKuSQG3w2Df4CHaW1/P714r1h3KU2k/vdM7xp5UlnjlBxcMrd3Puba91+/lKKBzhpY0HiMRecz11Hspqmo74mthT2UAwdOTXjHOOK+96m0/du/KInZTvPbWBS+94s8OBaKFwpMNzPFBEIi5uHbJBN/QB8P2L51BeH2RSdirldc1My83gU4sn8tA7u7vd/vrT83nond389tXtfO2xdeRmJHPerNHc8ep2VmwvZ2d5PSdNyGLZ5jLysoax91D0jyQnI5kzp+cwIjWR+uYw96zYQV7WMGaNzeClTWVU1Dfzixe3cs7MXP66dh/jhg/j51fM59QpI/H5ogsSfuGhVcwYk8FXz59xxDZt2FfN3kMNnDNzNEkJPn776nZW7qzkgesXtd0XwNPrSnn+/f3ccukc/u1Pq9lYWsPz7+/nsS+cfsShneZQhBVFAWaOzWRPZQOJfuOkCSM63Ldzjrd3VDB/fBbpydGXTkVdkB89u5mPLxzPGdNGEl2+vKOmljBm0Tbc9tI25uUN59tLZ3VbR3ldkAPVTcwam4nf1/dFG6sbWthzqIG5ecPb2tPXoax9VY08+NYuLjsxj5REH0tuX8HnzpjMzRfNZEVROd958n1mjsngjk8tZGNpNadNHUluRkrb7wdDYVbtOsQvXtrKh6bndNiXz67fT3KCj/Nmj2677tbnNrMjUM+91xV0qOPR90pYuaOSX3xiQYfnvb0XNx4gFHG8URTgykUTO9z2f4V7+c6T73PvtQVkpyfx+Qfe41dXncRZMw6fcWlneT3n//I1zODa0/L5xoUnUNsUYmRaEj6fcccrRdy2bBtXLZrArR+bx+0vF7E4P5vTp43COcfdr+8gFHFMz01ny4FaAL785zWcPTOXfYcaGZbk48pFE9m4r5pxWcP465pSGlvCPL2ulGBLhPNmj+bW5zbz6paDvPGtc2iJRMhMSWRneT1vbi/n06dM7PY11F444thUWsPKnRVcuWgCqUkJtIQj3P/mLq5ePIGs1CSci574emtZLeOGD8MMJo1M7XBy7O7UBUNEnCMzJRGIvubNjEjEcd397+IcPPT5xV1qDIUjNLSE237vWLP+OOS6oKDAFRYWHvP7/UfUBUPM/c8XAZiSk8aOQD3fWTqTzftr+fHl8/j4nW+xsbSGjOQEaoMhkhJ8bSvyjR2ewv7qJj40bRTr9lbxvYtnk5c1jAff2sVr2wIEQxF8Bq3/bC+eP5Zn1u8HYFiin8aWMNlpSfjMKK8LMm54CilJfs6cnsMDb+0C4MHPLeat4nKeXlvK1Jx0vrN0FrPGZvC39fupqAty+8tFVDW0kOg3PrpgHCuKygnUBrnz0wu5aN5YIhHHCxsPcNMja2gJu7b6L1+YxxOr9/HbTy3khDEZ3Lm8mFMmZ/Oh6aN4eOVuLpo7lkMNzdyzYmeXha2uOXUS/3XpHADeKq6gvC7ITY+sZeaYDCrqm/nyudO5/42d7CivJys1kQSfUTApm69fOIOpOenc8cp2po9O59bntrC/urFDXf916Rze2l7BqIwkfnTZPCAaeB/9zRtsK6tjwfjhfHTBOB55bw8TRgzjgjljONTQjM+Mh97ezcdPHk9mSgJz84bz+9eKebu4gmAowpfPnc7KHRWs21vFTz8+n+VbA2wsraY+GObXV5/IyLRkblu2ja+dP4NJI1P548oSfvLcZuqbwyT5fczNy2R17J3VFSePZ2NpDXsqG9regQEsmTOG3316ISWVDeRkJPPxO99iy4FafBadefT8TR9myqh0fvPKdv7n79vwGdx1TQFnzcihsTnM6T95mfrmMNefnk9jc5gfXjaX9/dVc+VdbxOKOH728fk0hyMs3xogKcH4r0vnUnywjtnjMln4w2W0hB2XLBjHRxeM494VO/jZFfMpqwny389uYt3eaq5ePJE9lQ28sb2cSSNT+daSmSycOIIxw1P4+YtbuHN5MZedmMcTa/ZRMGkEa/dUcfnCPM6dNZp/fWgVORnJVDU086WPTOP2l4sYnZnMtNx0qhtb2LDv8OypjOQEFk4awds7KjqsXpmbkczB2mCHv4lWKYk+mlqi284am8mu8npuuWQOty3bSllNkFsumcMFc0azbk8Vf1u/n5U7Knn8C6dRFwwxZ1wmxYE6PnnXO1TWR4cpp+Wms/dQA7PGZrKmpIovnzudUyZn883H13d51zEyLYkbz55G5rBEMlIS+N3yYq44eTzXnDqJ17cFeH9fNXcuLyYYCnPGtFEUldVx+tSR/MfFs3nwrV38ctk2AO69toBzZuby1Lp9PPLuHpIT/VTWBymrCfLiV84kOy2pr7HUgZmtcs4VdHvbUAlqiB6luG5PFWv3VPGHt3ez+nvntz2p3/vrBh56ZzffXHICj763h90VDXzqlImcMXUU588eTVVjM7kZKW3/YVs98OZOfvC3TXzxI1Mp3FXJvkONlFY3keg3/vC5U5iWm87XH1vHVYsmcNYJOSzbVMYz6/ezrayW3RUN0f/02alU1DVTGwzxoWmj2Ly/hrpgiFljM1m7JxoawxL9/PCyuawoCvDU2lIg+iFpSoKPpfPGcrA2yCtbDjJjdDo3XzSTlzcfZG7ecD5ZMIEzf/YqE7KH4fcZbxVX4BxkpSZS1e7tZ4LP+NoFJ5DgM6aNTuf1bQHuf3MX/zR/LKFwhBc3Ro/uHJ2ZTHldMwk+a/sc4NaPzeO/n93E2KxhlFQ00ByOMDcvs8Mf9dWLJzJzTAaXnZjHZx94ty0MAX5z9UlMyE7ljle28/fNZVx32iQefDv67mfBhCy2Hqhp++P2WXR529bZPABjMlO4cM5oNu+v5d1dlYzOTAagrCZIRkoCp0zOZsuBWppawmTEem8zRqdz5aKJ/PCZTXx4+ii+ceEJfP+pjazdU8WSOWMYPiyRZ9/fT10wxK0fm8e4rBQCtUFWlxzikff2MG74MPZVNZKXFf3+7YtmcvbMXC7+9Rs0hyOMG55CaXUTl5+UR3Ggjo2lNYxMT6I+GO4Q+gCnTonWl5GSQKLPx47y6Bj0hOxh7KlsJCMlgdqmEEvnjeG59w8waWQquysa8PuMcMSRnOBr2xftL18wezSvbDlIKJaWs8dmcqCmiXl5w3nwc4u5c3kxP31hC+nJCdQFQyT6jem5Gfzu0ws575evEYo4Jo9KY2d5PQk+IzcjmVOnjuSfPzSF37xSxMmTRvDPH56Cc46n15USCjt+uWwblfXN/Pjyefz65SIAcjOTWbmzkl9csYBXth7kYE0TuyoaCLQL8xGpieSPSusw/JiW5Kcl7EhL9nOooYXLF+aRNSyJh97ZxU8un09tUws/+Nsm8kemsqsi+i4wNSmB2qYWpuSkc+1pk5iem8GBmkYiEXhyzT7e2F7e4blPT07gaxfM4Ja/bQJgXt5wTpmczbLNZdQHQ1Q1tJA3InrKv1OnZFNWEyQUiZA/Mo0VReVMzUmjvK6ZppYw4YjjowvG8T9XnthTBB2RgrqT4kAdK7YFuP6MyW3XvVFUzvef3sDjXzide1bs4M7lxdx//SLOnpl7xPuKRBx/W1/K+bNHk5qUwFNr93HTI2tZMmcMv7/m5B5/b8uBGpb8agWL87O55dI5XHrHm4zLSuHFfz+TuqYQ3/rLevZVNfGJk8czfXQ6qUl+Tp6UTTAU5uyfL+dQQwt//OfFPPDWbl7aeICWcIT/+KfZXHvaJBL8Hd/y3/VaMT9+fgsA37t4NtsP1vLnd/fw3x+bi9+MidmpzB6XSVbq4Z6Ac47fLS/mtpe24jPjM6dO4v191XxryUym56ZTUtnAZb97k6sWTeDHl8+nurGF9OQEDtQ08VjhHn719yLm5mVyqL6FM6aN5GdXLGi771A4wrJNZWSnJXHr81vYvL8GXDRkrjs9n69feAK3/72IzftruP3qE9lZXs+2sjr+tHI3a0qq+PtXz2J4aiJvFJXz2tYA3146k6zUJGqbWni7uIKzTshh8/5a7nhlO99ZOpMpOekUldXy9cfWsa2sjn/58GR+t7yYUMSxKH8Ej95wGj6fUVEX5CfPb+ELH5nK1Jx0nHNt7wJaHaxt4sM/fZVR6cmcP3s0D7y1iymj0vj7V8/C5zOejf0TfqxwDxOyU3no86fQFApz48OrqaxvpqSigWFJfpbMHcPqkkNcuiCPu1fsICM5gQc+u5iy2iaee38/l56Yx4Lxw/nyI2t5Zn0pE7Oj4XzalJFcc9okvvTwai6eP5bTp47iJ89v5trT8jlQ08TMMRn86NnNzByTwV9vPIPaphD7qxt5c3sFr245yOqSQ9x1zcmcO2t0W8CeMnkkjxXuobqxhetOz2dCdirv7aqkrKaJj5yQy59XljA1N41zZh4evulJ6/j2tNwMmlrCBFsiHKhpoqSygfPbDf/8+PnNPPxOCX/6l1MoKqvjgjmjCUccf12zD5/PmJc3nNnjMvnp81u5782dnDQxizUlVfgMzpyRwwOfXQxEh7zSkv28XhSgoq6Zbzy+npljMnj8i6e3Dc+1f02/XVxBRkoiO8rryByWyOcfeI+Ig0X5I/jN1QvJzUhuG3baWV7P2b9YDsDtV53I0nlj2bCvmqvveYdgKMItl8zhM6dM4lBDM7VNIZ5Ys49Xtxzk0X89ldSkox9VPlJQ45w75l8nn3yyG8gOVDe6W5/b5IIt4aP+3cbmkPviHwvdmpJDvW776LslbvXuSuecc2tKDrmSivo+PUbhrgr33PrSDvVu3Ffd4/ahcMQ9+m6J++6T611TS8iFw5E+P1ZNY7Nragl1e9uOQJ1rCXX/HK0pOeQq6oIu2BJ2kUikx/uvrAu6q+9+233i92+5Q/XBI9bS2Bzqc909aa1lTckh96U/rnI7A3VHfR87A3WuprHZRSIRd8/rxe69nRVdtgmFIy4U7trunYE6t/VAjYtEIi4cuz0SifT4HDW1hNzWAzWurKbRPVa4p+35bm73vLf/3aaWkHv4nd2uprG52/s70r44noItYReobep1u7qmFvfXNXtdSyjsPv/Ae27St55xj75b0uN93vFKkdtT2ffXyKtbytyLG/a7xubuX+NffXSt++6T6ztct3p3pVuxLdDt4/f099AXQKHrIVOHZI9aRAaespom7lxezDcuPIG05ME3D+JIPerB11oRGZRGZ6bwg0vmxLuMuBh086hFRAYbBbWIiMf1GtRmdp+ZHTSzDcejIBER6agvPeoHgCX9XIeIiPSg16B2zr0OVB6HWkREpBvHbIzazG4ws0IzKwwEAr3/goiI9MkxC2rn3N3OuQLnXEFOTk7vvyAiIn2iWR8iIh7XLwe8rFq1qtzMul9TtHejgPJetxoY1BbvGSztALXFqz5oWyb1dEOvh5Cb2Z+Bj8QevAz4T+fc/36AIvrEzAp7OoxyoFFbvGewtAPUFq/qj7b02qN2zl19LB9QRESOjsaoRUQ8zotBfXe8CziG1BbvGSztALXFq455W/plmVMRETl2vNijFhGRdhTUIiIe55mgNrMlZrbVzLab2c3xrudomdkuM3vfzNaaWWHsumwzW2ZmRbHvI+JdZ3e6WyGxp9ot6tex/bTezBbGr/KuemjLD8xsX2zfrDWzpe1u+3asLVvN7ML4VN09M5tgZq+a2WYz22hmN8WuH3D75ghtGXD7xsxSzOxdM1sXa8stsesnm9nK2H551MySYtcnx37eHrs9/6gftKdzdB3PL8APFANTgCRgHTA73nUdZRt2AaM6Xfcz4ObY5ZuBn8a7zh5qPxNYCGzorXZgKfA8YMCpwMp419+HtvwA+Ho3286OvdaSgcmx16A/3m1oV99YYGHscgawLVbzgNs3R2jLgNs3sec3PXY5EVgZe77/D7gqdv3vgS/GLn8J+H3s8lXAo0f7mF7pUS8GtjvndjjnmoFHgEvjXNOxcOiF9N8AAALfSURBVCnwYOzyg8BlcaylR677FRJ7qv1S4A8u6h0gy8zGHp9Ke9dDW3pyKfCIcy7onNsJbCf6WvQE59x+59zq2OVaYDOQxwDcN0doS088u29iz29d7MfE2JcDzgEej13feb+07q/HgXPNzI7mMb0S1HnAnnY/7+XIO9GLHPCSma0ysxti1412zu2H6AsVyI1bdUevp9oH6r76t9hwwH3thqAGTFtib5dPItp7G9D7plNbYADuGzPzm9la4CCwjGiPv8o5F4pt0r7etrbEbq8GRh7N43klqLv77zLQ5g2e4ZxbCFwE3GhmZ8a7oH4yEPfVncBU4ERgP3Bb7PoB0RYzSwf+AnzFOVdzpE27uc5T7emmLQNy3zjnws65E4HxRHv6s7rbLPb9H26LV4J6LzCh3c/jgdI41fKBOOdKY98PAk8S3XllrW89Y98Pxq/Co9ZT7QNuXznnymJ/WBHgHg6/hfZ8W8wskWiwPeyceyJ29YDcN921ZSDvGwDnXBWwnOgYdZaZtS7L0b7etrbEbh/OUZ6MxStB/R4wPfapaRLRAfen41xTn5lZmplltF4GLgA2EG3DdbHNrgOeik+FH0hPtT8NXBubYXAqUN36NtyrOo3TfozovoFoW66KfSo/GZgOvHu86+tJbBzzf4HNzrlftrtpwO2bntoyEPeNmeWYWVbs8jDgPKJj7q8CV8Q267xfWvfXFcArLvbJYp/F+xPUdp+kLiX6SXAx8N1413OUtU8h+gn1OmBja/1Ex6FeBopi37PjXWsP9f+Z6NvOFqL//T/fU+1E38b9Nraf3gcK4l1/H9ryUKzW9bE/mrHttv9urC1bgYviXX+ntnyI6Fvk9cDa2NfSgbhvjtCWAbdvgPnAmljNG4Dvx66fQvSfyXbgMSA5dn1K7OftsdunHO1j6hByERGP88rQh4iI9EBBLSLicQpqERGPU1CLiHicglpExOMU1CIiHqegFhHxuP8PK8muqlKdx5cAAAAASUVORK5CYII=\n", 643 | "text/plain": [ 644 | "
" 645 | ] 646 | }, 647 | "metadata": { 648 | "needs_background": "light" 649 | }, 650 | "output_type": "display_data" 651 | } 652 | ], 653 | "source": [ 654 | "%run examples/classification/iris.py" 655 | ] 656 | }, 657 | { 658 | "cell_type": "markdown", 659 | "metadata": {}, 660 | "source": [ 661 | "### Hand-written digits dataset – classification\n", 662 | "\n", 663 | "Description: 8 * 8 image attribute, predict which digit (0-9) is\n", 664 | "\n", 665 | "Criterion: Cross Entropy" 666 | ] 667 | }, 668 | { 669 | "cell_type": "code", 670 | "execution_count": 9, 671 | "metadata": {}, 672 | "outputs": [ 673 | { 674 | "name": "stderr", 675 | "output_type": "stream", 676 | "text": [ 677 | "100%|██████████| 300/300 [00:00<00:00, 1615.85it/s]\n" 678 | ] 679 | }, 680 | { 681 | "data": { 682 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3gc5bn38e+96pKLJEuyZVuy3CsuWDamg43pxTkBAgkckpDwnnBIOElIAoecECA5SSihBEIJnYAhMb3buDdsy93YKpZVLFldVu/S8/4xs03FslUsz8n9uS5dWo12d57Z2f3NM/c8syPGGJRSSjmPa6AboJRSqmc0wJVSyqE0wJVSyqE0wJVSyqE0wJVSyqECT+bMYmJiTFJS0smcpVJKOd727dtLjTGx7aef1ABPSkoiJSXlZM5SKaUcT0RyOpuuJRSllHIoDXCllHIoDXCllHIoDXCllHIoDXCllHIoDXCllHIoDXCllHIoRwT4ygNF/HXNwYFuhlJKnVIcEeBr0kp4YX3WQDdDKaVOKY4IcJdAm154Qiml/DgiwEWEtjYNcKWU8uWQAAftgCullD9HBLhLBM1vpZTy55AA1xq4Ukq155AAFw1wpZRqxxEBjoAew1RKKX+OCHCXCFoEV0opfw4JcK2BK6VUew4JcK2BK6VUe90GuIi8JCLFIrKvk//dJSJGRGL6p3me+WgNXCml2jmeHvgrwKXtJ4pIArAYyO3jNnUg9m+jvXCllPLoNsCNMeuA8k7+9RjwS07C4UWXiN2W/p6TUko5R49q4CJyNZBvjNl9HPe9TURSRCSlpKSkJ7PDZXfBtQ6ulFJeJxzgIhIO3Av85njub4x53hiTbIxJjo2NPdHZAeCyE1zr4Eop5dWTHvh4YCywW0SygdHADhEZ0ZcN64z2wJVSyivwRB9gjNkLxLn/tkM82RhT2oft8uOugSullPI6nmGES4HNwGQRyRORW/u/Wf60Bq6UUh112wM3xtzYzf+T+qw1XXD3wLUGrpRSXo44E1O0B66UUh04JMB1HLhSSrXniAB318D1TEyllPJySIBrDVwppdpzRIBrDVwppTpySIBrDVwppdpzRIBrDVwppTpySIBrDVwppdpzRIC7T6TXGrhSSnk5IsA93wc+wO1QSqlTiSMC3DMKRWsoSinl4YgA1yvyKKVUR44IcB0HrpRSHTkiwL2jUDTAlVLKzREB7u6Ba3wrpZSXIwLcWwPXCFdKKTdHBbgOQlFKKS9HBLgexFRKqY4cEeDe70IZ2HYopdSpxBEBLjoKRSmlOnBEgOuJPEop1ZEjAly/zEoppTrqNsBF5CURKRaRfT7THhaRVBHZIyLviUhkvzbSbqXmt1JKeR1PD/wV4NJ201YAM4wxM4F04J4+bpcfrYErpVRH3Qa4MWYdUN5u2nJjTIv951fA6H5om4eOA1dKqY76ogb+feCzrv4pIreJSIqIpJSUlPRoBu4auJ6JqZRSXr0KcBG5F2gB3ujqPsaY540xycaY5NjY2B7NRy/ooJRSHQX29IEicgtwJbDI9HPX2KUXdFBKqQ56FOAicinwK+B8Y0xd3zap0/kBWgNXSilfxzOMcCmwGZgsInkicivwFDAYWCEiu0Tk2f5spOfrZLUGrpRSHt32wI0xN3Yy+cV+aEuXtAaulFIdOeJMTJd+G6FSSnXgiADXGrhSSnXkkAC3fmsPXCmlvBwR4C69KKZSSnXgkAC3fmsPXCmlvBwS4FoDV0qp9hwR4G7aA1dKKS9HBLj3ijwa4Eop5eaMANcLOiilVAfOCHCtgSulVAcOCXDrt9bAlVLKyxEB7r6kgwa4Ukp5OSLA3T1wpZRSXg4JcO2BK6VUe84K8LYBbohSSp1CHBHg+mVWSinVkaMCXONbKaW8HBHgeiamUkp15KgA1xN5lFLKyxEBrjVwpZTqyFEBrvmtlFJejghwrYErpVRH3Qa4iLwkIsUiss9nWrSIrBCRDPt3VL82UmvgSinVwfH0wF8BLm037W5gpTFmIrDS/rvfuM+k1xq4Ukp5dRvgxph1QHm7ydcAr9q3XwWW9HG7/HhLKP05F6WUcpae1sCHG2MKAOzfcV3dUURuE5EUEUkpKSnp0czEbqX2wJVSyqvfD2IaY543xiQbY5JjY2N79BzaA1dKqY56GuBFIhIPYP8u7rsmdaQ1cKWU6qinAf4hcIt9+xbgg75pTud0FIpSSnV0PMMIlwKbgckikicitwJ/BBaLSAaw2P6733i/zEoTXCml3AK7u4Mx5sYu/rWoj9vSJa2BK6VURw45E9P63aY1FKWU8nBEgIvWwJVSqgNHBLhLa+BKKdWBIwJce+BKKdWRIwIcrF64fhuhUkp5OSbARURP5FFKKR+OCXCrBz7QrVBKqVOHYwLc6oEPdCuUUurU4ZgA1xq4Ukr5c0yAC1oDV0opX44JcK2BK6WUPwcFuNbAlVLKl2MCXES/D1wppXw5KMBFD2IqpZQPxwS4S9BvQlFKKR8OCnAdhaKUUr4cE+B6Io9SSvlzUIDriTxKKeXLMQGu48CVUsqfgwJca+BKKeXLYQE+0K1QSqlTh2MCHPREHqWU8tWrABeRn4rI1yKyT0SWikhoXzWsPZdLa+BKKeWrxwEuIqOAnwDJxpgZQABwQ181rD2XnomplFJ+eltCCQTCRCQQCAeO9L5JndMauFJK+etxgBtj8oFHgFygAKg0xixvfz8RuU1EUkQkpaSkpMcN1S+zUkopf70poUQB1wBjgZFAhIjc1P5+xpjnjTHJxpjk2NjYHjdU0Bq4Ukr56k0J5SIgyxhTYoxpBt4FzuqbZnXkEsHo11kppZRHbwI8F1ggIuEiIsAi4EDfNKsjlwhtbf317Eop5Ty9qYFvAZYBO4C99nM930ft6kBr4Eop5S+wNw82xtwH3NdHbTkm/TZCpZTy55gzMV0CekkHpZTyclCAaw9cKaV8OSjAtQaulFK+HBPgaA9cKaX8OCbAXXpFHqWU8uOgABc9E1MppXw4KMC1Bq6UUr4cE+CCXlJNKaV8OSfA9aLGSinlxzEBrjVwpZTy55wAd2kNXCmlfDkmwLUGrpRS/pwT4IKeyKOUUj4cE+DWBR2UUkq5OSjA9UxMpZTy5aAA1xq4Ukr5ckyAi6CXVFNKKR8OCnCtgSullC/HBLjWwJVSyp+DAlxr4Eop5csxAa7jwJVSyp+DAly0hKKUUj56FeAiEikiy0QkVUQOiMiZfdWw9vTLrJRSyl9gLx//BPC5MeZaEQkGwvugTZ3SCzoopZS/Hge4iAwBzgO+C2CMaQKa+qZZncwPrYErpZSv3pRQxgElwMsislNEXhCRiPZ3EpHbRCRFRFJKSkp63lARjI4EV0opj94EeCBwOvCMMWYOUAvc3f5OxpjnjTHJxpjk2NjYHs9MRPRMTKWU8tGbAM8D8owxW+y/l2EFer/QE3mUUspfjwPcGFMIHBaRyfakRcD+PmlVJ3QcuFJK+evtKJQfA2/YI1AOAd/rfZM6pzVwpZTy16sAN8bsApL7qC3HJCLaA1dKKR+OORNTa+BKKeXPMQGuNXCllPLnmAB36XehKKWUH0cFuPbAlVLKy1EB3qoJrpRSHo4J8OBAF00teiqmUkq5OSvAW9u0Dq6UUjbHBHhIoNXUplbthSulFDgwwBu1jKKUUoCDAjzY3QPXAFdKKcBJAR6gAa6UUr6cE+DaA1dKKT+OCfCQwABAa+BKKeXmmADXHrhSSvlzXoC3tg5wS5RS6tTgmADXYYRKKeXPMQEerAGulFJ+nBPgOoxQKaX8OCbAQ/QgplJK+XFQgFvDCDXAlVLK4pgA1xq4Ukr5c1yAN7XoMEKllII+CHARCRCRnSLycV80qCvB+nWySinlpy964HcCB/rgeY7JMw68WQNcKaWglwEuIqOBK4AX+qY5XQt0CSLaA1dKKbfe9sAfB34JdJmqInKbiKSISEpJSUmPZyQiBAfodTGVUsqtxwEuIlcCxcaY7ce6nzHmeWNMsjEmOTY2tqezA6wyio5CUUopS2964GcDV4tINvAWsFBE/t4nrepCcGCABrhSStl6HODGmHuMMaONMUnADcAqY8xNfdayToQEaglFKaXcHDMOHKyhhHoQUymlLIF98STGmDXAmr54rmOxeuB6Io9SSoEDe+BaA1dKKYuzAlyHESqllIezAryPD2KW1zb12XMppdTJ5qgA78tx4AeLq0n+3Qq255T3yfMdS25ZHdmltf0+H6XUvxZHBXhve+CHy+s8t/flV9FmIK2wpi+adkz//d5efvnOnn6fz6ngcHkdB4urB7oZ3Vr857W8tjn7pMyrpbWNo6fQ3t7P3t7FHz7t968vUieBwwI8oMfDCNeml3DuQ6vZl18JwKESK7gLK+v7rH1dOVJZT/7R459PZV0ze/Iq+rFF/ee+D7/mp2/vHuhmHFNNYwsZxTVszer/vS+AN7fmcv7Dq0+J4zfGGN7dmc9z6w6x+7Az32M98ftP9vP4l+knbX6V9c38/pP9VDc09+t8HBXgvTmR58v9RQDsst+0mXZJ40hlQ9807hhKqxspqW7EGHNc97/+uc1c/dRGWtuO7/4nizGGh79I5aPdR7q8T3ZpLQUnYaPYG8VV1jrPKavr5p5940BBFVUNLQN+zKWoqsHv/f78+kOd3m9bdjnr0jt+b9HKA0X8eXkaAGmF1fwj5XCP23Iy39vNrW28sSWX93fmn7R5rjxQxN/WZ/H2tp6/RsfDUQFuDSP0Hwd+oKCK3328v0M4ltY08uflaTTbPfZ1GdYbMr3I2r3PKrECvLCHAf7Kxix2H67gzyvS+eLrQgCO1jbR0OzfvqaWNqoaWmhqbWPjwTK2ZXff60uz21hW29ijtvWHe9/by00vbuHp1Zn893t7O72PMYb8inrKaptoOcVOuGpsaWXCf3/KP7YdpqjKel2zy2qPe6PaG3n23ldpzcCuz2uf3cTtb+zw/J1Z3Hn58Lcffs39H33dYfqbW3J5avVBahpbuOTxdfxy2R7aehDEe/IqmHDvp9z6yjYq646/h1pY2dCj99WevErqmlrJKa+jrqnlhB/vK7+intc3Z3f7vkkttD7Db27N7df3mLMCPMBFfVOr39Z72fY8XtiQRUm7D8fn+wp5ctVBnl59kKS7P/H0tlILq2lrM2TZPfCe9BYr6pq4/+P9vLwxiydXZvD/Xt/Or9/fy5wHV3Dve/v8eqG+IfyTt3byy2XHroX7vsGKKr2PzTtax5+Xp5FVWktRVQNPfJnh2Tj1hcaWVi59fB1Lt+Z2+N/2nHLe2JLL5swyAIaEBnW4z+7DFXx5oJjGljaM6XyET2NLKxV1/tM/3VvA1U9t6NNl6cyOnApa2gxPrT5IcbW10a5uaOHocQbI1qxythwq69G83QFe1kUPvKCynmXb847rg15W08jrX+VQ5bNr3tZmePDj/RwoqOryceW1TRwur/eUTeYlRZFbXueZ51/XHORv6w5R09jCgYIqDpfX8+qmbG5+cQubMksBq2PRZmBn7lHP8/q2o6qhmdWpxd0ux/qMUoyBlanF/PQfu45rI1Db2MKFj6zhtc053d63vc12+42BjKLeHfN6dk0m//PB12T77L0dLK7httdSqG30fnYPFFThEjhUUssnewt6Nc9jcVSAzx0TRW1TK0+uzPBMS7O3dMVV/gGeX2F9aJ5efRCAUZFhXDg5lrTCagqrGqhvbiUiOICCyoYT3kJuySrHGNhw0PuB/vtXVvCt2F/ITS9u4cdv7qSltY2Sam+7ymubyC6r7dBL97Unr9Jzu6jKu3fw2uYcnlx1kKv+soE/fZ7KY1+m814nu4TpRdU96hWtTy8ltbCaR5endeilPLv2EJHhQWz/9WJuv2A8BZX1nsDNKq3lSEU9P166k9vf8H4xZXF1x97mPe/uZfYDK/yW/8sDRezJq/Ssx67kltVx4/NfccPzm/lyfxELH13jOShdWd/Mj/6+nbyj3g9VelG130bBHULjYiP81skrm7LJPY5Syi+X7e5wILq8tonn1mby7o68Lh/X1mY878XS6kYamltJ8dkL+/pIJWf+YRV3/XM3W7qpyR8qqeGCh9fwP+/v84TtHW/u4MPdR3hxQxZvbum48XVLLfQP97MnxFDX1Mqq1GLSi6p5aUM2r2zKZmfuUdqM9b37T6zMYH1GKT9+cyc1jS2eDdG7O7zvO9+N0gMf7ed7r2zjEbvM0pXdhysYFxPBfVdNY1VqMRszS6lvOvYZ1lmltdQ3t7LhYOkx79eeMYZVqcVEhVudju7eZ0u35vJfb+3sML20ppFVqUWsSi0GYHuOdyO2OrWY5fuL2HiwlE0HS5n9wHLWZ5Ry9ayRzEqI5J539vZbWdFRAX7lzHguP20Ez6zN9ISUe1fl7W2HOf/h1Z7wcb/ZmlsNl0wfzsa7F3LB5Dgq65t5Y4u1Fb9gchx1Ta1UNXgD63h6gl/ZPTHfXeKrZ43koWtnUtVgvdFTco4y7/df8p0Xtvg91hhriw3wyZ4CT03x6yOVvLczjx0+vZtCnwB3l15qGlvYa4f8W1tzPQdlATZklHLxY+tY8teNnpCsb2qlqaWNh79I9Wvv0dombnh+M2vtWufHe44QEuiitKaJpVu9dbvWNsO69BKWzB5FVEQwY2MiaDOQf7SeJ1dmsOjRNVz2xHpyy+tobvVuONrvEYH3g7/CPh4BcKDAWn++H4jOvLjhENtzjrIjt4IfvJbCoZJaNtof5uVfF/LZvkJWp1nLkpJdzsWPrePljVne18a+b31Tq9+G8cmVGTz0RSoAGUXVHLHDtqiqwbN7n1tWR3ZZHTlldeSUeYeD/vTtXfzhs1TufmevXyngs70F3PziFv6ZcpjSmkbPcZuy2kaufXYT1z672bOH8sJ6bxu3dRPgD3y8H4A5iZEs257HQ5+n8vGeAn73iTV9S1YZb2/L9Rtt5eYbXOHBAcxKiATgh6+l8L2Xt1Fa00h+Rb2nHAjWBioiOICy2iZW7PdO9+04lNc20dTSRn5FPe/vzGdoWBBPr86krJP1X17bxMNfpLLxYCmzEiK5LjkBEXhr62Fm3v+F34atPfce9PacoyzbntdhGbNKaz17dxsySj2l0vd35bMjt4L/umgSIYEu3tqWe8xjEa9uyuaD3Uf8etNgdQS//0qKZ2PsO/w4p9x6T2zKLOOR5WlU2O+F00ZH8uQNs6lpauGtrf1TC3dUgIsI85OiaWppo7yuibKaRk8ofbArn5yyOjZnlvGXlRnk+qzgxdNGAHB6YhQAT6/OZF5SFJfOsKa76+ClNY3Mvn+535u4vUMlNay2t8Jum+9ZyJM3zuGs8cMA79WDjtY1U93Qsebm/jAt3ZrL02syqWpo5rEVGfxq2V4OFFQzYkgoLvEebKtvamVffiVXzIwHIMPeAOzIreDKv2zwjKZwh/yevEre3ZHPwkfWMPU3n/PXNQd5enWmX6D98bNUvjpUzn0f7OOuf+7m072FLJk9ijmJkby9zVu3yy2vo7GljekjhwAwZlgEYI3qefzLdGYnRFJZ37EMUdJJD3zEkFAAz8Gv5tY2z5BD3w1XelE133t5K7e/sZ2G5lZqGlt4d0c+V8yM59Zzxnru5w7blQes9eE+rvHyxmz7eWo8r5+7dFBS3UhxdaOnLWB98A4W13D1Uxu54JE1PPjxfs7435X87B+7rGXN8B7Qu/LJDTy1KoOiqgbWZ5SwcEocTa1tfLzXOrDb0NzKbz/6mq8OlfGLZXtY43MwcF16Kfvyq+zbJfzg1W18ureAmxeMYfLwwWzNLveEnDsY3fbmVbImrYQfL5rAD88dR0FlA69tzsElUFrT5FneX72zl/s/+poPduX7HW9JK6xGxLo9OiqMsfZ6bDPevVWwSpIxg4I9f39nwRgAlm6x1tnF04b7rdPHVqQz6defcefSnYjAXZdMBqz3qG+pBez3++pMaptamTV6KINCApkYN4hP9hbQ3Go8JTqwjh39atkeZt2/nOVfF5Jtbzgr65u565+7edDemLmnXfnkei59fD1phdXcsXQH97y7l4bmVv7waSqzEyK5acEYpo0cwo7cCi5+bJ1nmRtbWnl2bSY/e3sXr27KJrWwGmOsjuGuwxX8bd0hz3sIwCUwLX6IX4fDvXF5ZVM2O3IrGBxqfcXUpOGDGDMsgnMmxPDOjrwe7Rl3x1EBDhBnf/CKqxo9B/sATy/64S/SeHRFOrsPVzB/bDSLpw1nsf2mO230UH7/jRkMDgnkrosnMzLSei53EOzIOUptUysf7T5Ca5vh/o++JrWwije35HLhI2t4fXM21zy1kcNH61kyeyQAEcEBnjAYHRXO+NgIzp8Uyzdmj/Jrd3CA96V29w4yiqtpbTOsTy9lS1aZfaCzlAlxg4gZFMLy/UXc/c4e3tuZT3OrYcnsUZ6LO18xM54Xb0kmOiKY59dlArAvv9LTlqVbczlk1/lf3GAF9/s7j/DShiwyiqp5O+UwsxMiyS6r48PdR1gyZyQ/v2QS181NIL2ohm/8dRMvbcgizd71njxiMABjhoUD1nDB4EAXz92czM0LxvDvZ44h0CWEBQUA8MSXGfzO50NW19RCYVUDwYEu1meU8s+Uw/zps1SaWw2hQS625xz1bDQe/zKdteklfLq3kKdXH+TcP62iurGFmxaM4c5FE3nihtmMGBJKVlkdjS2trLcDNqu0huKqBj7bZ9Uc3WWcrNJa2gzEDQ6xAryqkdFRYfxs8SS+lZxAeW0TFz+2luBAFxdPG+55vVbaG+r16SWMigwDoLqxhUeWp/P+znzaDNx7xVQmxA3iPXvvYunWXIqqGnnsW7MJDnDx+0+s8dYi+O3+v7kl13PM4FvzEpg3NoodOUd5bEU6T6/O5OqnNjD9vs/ZkGE9ZlVqMSJw7dwEFk8bzn+cP57//cZp3LloEoAnNMAK9Dvf2sV1z272HNNILawmeUwUwQEuRkeFMyoqjACXeB4TFGB1Ohqa2/jRBRMItP93xWnxDAoJZGt2OaFBLp769ums+8WFrPr5+YC18QNIyTnKA9fMYMHYaAAeXZ7Gvz2zyW/j8PEeby14/lirszPb3hMA2OuzN/nh7iO8nXKY5tY2XlifRU5ZLUEB3vauOFDE5swytmaV88L6Q9Q2tVLd0Mwvlu2moq6Z7fZrWVzdyC8vmUyAS3js+tk8/q3ZNDa38tO3rY3zve/t44+fpbIqrZj7PvQeuN1fUMVvPtjH7z89QHZpLZX1zcQPDeWdH53FZTNGkF5U49nr8h3NNCshks33LOLpb5/O2eNjALh27mjyjtbzVVbPjqEcS598G+HJFDc4BIDi6gbW2LvMvmdopvrsKp49PoY7L5ro9/jvnDGGG+YlEuASz8kV6UXVXDglzvMGWp9Rypq0Yl7emM227HL25VcR6BJ++9F+WtsMy/7jTIYNCuH9XUcYGxuBiPeN9eYPFxAS6CIyPJgbz0jkumc3A1avJ6+innExEaQWVlNZ3+wZDfHM2oOennp5bRNjYyKorG9mb34lqYXVvLXtMMEBLuYnRXsePyluMIumDufmBWN4YmUG2aW17Mmv5NwJMWw4WMre/EoCXMLEuEGe1yS/op4HPt7PnETrQ/M/V07jYHE1pydGMXG4FdBXzornD58eYH9BFbsOVzBmWDgiMCFuEACxg0I8y3rbueOIHRzCg0tmALA7r5KW1jZyy+rIr6jnhQ1ZlNY0UlHfzF0XWz2zB6+ZzkOfp/ELn4O5N8xL5JVN2Tzw8X5uPWcsn+8r5Ptnj+WdHXn8ZdVBIsODeOu2BcwdY+1BXTN7FO/syCertIY1aSXUNrUSMyiYQ6W1bMoso83A+NgIzygL9wHrM8YN46PdR8gtr2NWwlB+smgiRVUNvJ1ymDYDD187k4unj+BHF1Ty1tbDnoOFO3KPct6kWK44LZ5XNmWzPqOUZ9ZmMmv0UMbHDuLfTh/FQ5+nkV5UzV/XZHLmuGFcOXMkq1NLeMeuj4+LiSCzpJboiGDKa5vYk19BeHAAH95xDhPiBnF2eQx//yqX17/KYdLwQRwsriE8OJDff3qAT358DqvTipk5OpLoCKt3fPdlUwDrgOJjX6bzzdNHszmzjLSianb7nEPw3NpMIsOC2HW4gjsunMC8pGgmjxhMUICLhKgwBocGUVrTSNyQUGobWyiqauCGeQm8vjmb3PI6Jo8YzNT4wWzLPsotZyYRHOgicVi432iwsTER/OaqaVw4Oc7TU92WbfVQDxypYlhEMK9vzuFAQRW/vmIql84YwegoqyMwKyGSf6RYr9G+/EqaW9tobTP8bd0hpowYzNWzR/LQ52kcKq1h1uhIzho/jLlJ0dz+9+3c+LevPG1IHhNFYnQ47/qUd55bd4jTEyM5094zToqJICkmgtKaRn73yQG2HCrjo91H+PYZidx31TT+4/XtlNc1k11ay6ubsj2lzo92H6G8tomEqHDmJEZ5lnFH7lHOmRhDfkU9Ny8Yw8jIMG4+cwyDQgI9e8sAl0wfwU8WTvDsvfYlBwa41cN8bu0hNh8q49q5o0ktrPLsmvoaHRXW6XO4ex5REcGMigxj3xHrsXvyKhGxdsl+Z/ec9uVXERzo4peXTOZ3nxxgQtwg5o6JoqXNEOgSktqtlOE+u+ZT44d4bsdHhhISFMDEuEHsyD3qKR1Ehgd1aPvYmAjPLuPiacO5cmY8M0dHMjQ8iPGxViCPjbXme8P8BJ5YmcGLG7IoqW7ktNFDKahsoLi6kXExEZw5fhiphdVcMn04idHh/G19FjtzKwhwCdNHDvGEotuQ0CBW/Ox8wkMC+O5LW9mRW0H80FDCg623issljIoMo6CynjsW+m8cH/rmTJpb27jyLxs8097fZZUWBoVYjz9tVCQPLpnBezvzqWtqYc/hSv778qm0tLXx8sZsymqaMMD3zxlLVUMz/0jJ4+YFY1gwbpjfvMbFRLA9u5y3tuYyfEgI3zx9NM+uzWRteglDQgO5cuZInliZQX1TK1ml1gdxflIUH+0+Qn5FvWevbPiQUK5PHs20+CFcPN0qqU0fOZSFU6zRHqsOFFNa08SchEgWTR3OtJFDOPMPq6ioa+bORdZe1pLZo3j4izR++JAhpJIAAA8ZSURBVFoKJdWNPP3t0wF44JrpzBw9lNAgF5/tKySzpJYLJ8fxzo48GprbmBY/xLNhvGT6CO69fCof7M7nme/MZdigYL48UMxPlu5kyV83sje/kjsX+b/eADNHR/Lds5K4cX4iv716Oo8uT+Mvq6wD97+4ZDIPf5HGf765g9kJkdyxcAKh9h4SwJ++OZOIkEAaW1oJCQygpLqRljZDREgg42MHERTgIjQogIeunUVueR3nT/JeEjEkMIDBIYFUN7ZwemIUF06OAyAsOIBYe08HrJEr6zNKeHVzDnGDQ7hq1ki/z8h5E2OJHxrKWeOtMsPEez/znHH9lxvncMa4aJ74MoPSmibOnxTHz+yOwOf/dR47D1cwOCSQA4VVXDApjj15Fby7Mx+XWL3e1jb41aWT/TpYANfNTeCR5Wn85K2dNLa0cd3c0YQEBvDy9+bT1ma4/Mn1pBZWEzMohFGRoXxon/cwPtZaV7MTIwlwCdtzjjIuNoLWNsNpo4Zy/byEDusHIDQowNPuvua8AB9i9QA3HyojbnAID31zJj98LYV9eEPQJVZtb1QXAe5r2sghfH2kksaWVvbmV3Lp9BFszzlKVmkt85Oi2ZpdzuUzRnDTgjH8MyWP756dhIgQFCD84pLJnoNBnXGHFsBvrpxOc2sbn+8r5JO9Bey3D949d9Nc3tyaS4BLWJ1azNG6ZsbGRvDcOuv//37mGM6d6P3gjLeD213DjB8aRvKYKN60d5VnJ0RyqKSWzYfKmBI/hOQx0by8MZt5SdH84NxxVNW38HbKYSYNH+z3YfY1Yqj1Afv5xZP5zgtbqGl3QOejH59DcKDLU85xc5dZ3P7t9FFkltRSVd/s2X0eGxPBtJFDuPy0eIwx1De3Ehzo4vYLJvD3r3L5cPcR5o6JYmRkGDfOT2RffhU3nzmmQxvHxkRQ29TK6rQSfrxwAuNirYOr7+3MZ/G04Uyy9yiySms5VFpL/NBQEn02tr5tfejaWR2ef0q89f+3tlmvq3s9xw8NIzE6nPyKeq6aZZXRRkaGsWDsMDYfKuOGeQnMt8sIESGB3HJWEuA98DcnMZK16dZGwff96XIJPzxvHD88b5xn2lUz42lobuXF9VnMHB3JknZlObA6I7+9errnb3fIDA0L4vtnj+XljdkkRIfx0i3zOqzvM9ptFH09sGSG50D42JgIxsZ07D1GDwqmurGFxOhwv+mJ0eGeAP9w1xEyiqu5aUEiD14zo0OYJkSHs/meRWw5VMY7O/IYFhHM5afFc9mMEZw1wSpB3H3ZFO7/aD/DfGrzCdHhJNjzvXCKtfEICXJ5XoPO1qnb0PAg7rhwAn9ekc742Ai/Mo7LJXxjziieWZvJ67fOZ+WBIh5Zns6gkECSk6z1Gh4cyLT4IaTklHs6QO7S4snmuAAPDQpgcGgg1Q0tTB4xGJdLPHXxpGHhZJfVcX1yAh/sOuLp3RzL9JFDWLG/iGm/+YLWNsM5E2P44zdn8uneAi6dPoKP9xzhwilxhAYF8MVPz/N77P87f3y3z//iLcm4XOIJjP0FVZ6RHaFBLuYlRXs+SN98ZpO1VY+J4M5Fk7jvw32cMdb/Q3bRtOHsPFzBxOHeZbtiZjwpOUe5Pnk0sxMiPQfspowYzLmTYrhsxgjPAdt5Y6N5O+Uws0YP7bbtZ40fxg/OGevZBXVz78Z35a6LJ7Enr5JHr7M+RLsOV/DAx/uJDAsiLNgbIiLi6dmPjAxjavwQDhRUcdFUq3c8JzGKT+88t9N5jLM3ZEEBws1njqGgwjuy5MxxwxgfZ/3/+XWZ7MitYFxshF/5Z5H9oe/KiCGhDA0L4qtD5QS6hCkjvHtT3z0rieLqRmJ8nu8P/3YaGcU1XDS18+d1Hyg/bdRQ4gaHWgEeeewOhohwfXIC1yd33rPrjDvAp8YPJiw4gFV3nU9EcKBfvft4dNc2sN4HOWV1JET73zchKoztOUdxidUDdx9zah/evuYlRfPodbNYPH14h/MMvntWEkNCgzh/8rEvij4uJoKRQ0M9gxWO5Y6FE7l+XgIukQ7tuu28cdx23jhExHO8qqaxhWE+7/u5Y6J4Y0sOP1m6k/DgAE+H4WRzXICDVQevbmhhYpz1orkP3C2eNpy/rc/ijoUTeHDJDIICuj9GO32kFWStbYaHrp3JVTNHEhYcwI3zEwG4+cykXrV10VT/o/bu3sq69BKm2Bsgt/GxEezJq2BUZBjfPiORb5+R2OH5Zo6O5PVbz/Cb9u0zEokZFMIl00cgIky2w2bGqKEMCQ3imZvmeu67YFw0gS7x9CaORUT49ZXTjn9hbe1LK3MSo3jv9rO7fdziqXEcKKjylDeO5ezxMTz+rdlcOCWOoWFBxESEcN9V02htM1yXPJrQoAAumhrHx3sKaGkzJI+J8uy9gfdgeFdEhGtmj+S1zTmMi43w29v4vs9IGDd3fbUr/3nhBH6xbA+TRwxm+JAQ9hd0XeLrjXGxEYh4y3ednXTVV4ZFWK9nZz1wsHrumSW1XD8vgcjwY2/0XS7hm3NHd/o/ka7/1/5+y350FoNCjy/W3OXYzp7HLcFn2Xw7LkvmjGJffiWJw8K5/YIJRHXTqekvDg3wUDJLaplk90IvnTGCo3VN/GzxZL8DJMdj/thozhw3jLsumdyhHtwf3G/uxpY25rTrKfzoggksnDKcwOPY8PgKCQzw7M6DFdJv/uCMDj1nsEbKrL7rguPqYZ1st50/nnljo49rz8nlEpbMGeX39/fO9g/WF26Zx7r0Ev79pa3MHxtNtB0i3zs76bjac//V01k4Jc6vZttT1yUncJ3dk3Y/38h+WAcRIYE88525zErofg+rt9w90vYBvnDqcPYXVHHTgjH8/B+7/YZ+9re+fk0Tuwjw2QmRLPvRWX06r55wZoDbPSn3yInJIwZ76oBzx3Tfs/Q1NCyIpbct6NsGHsPwIaHWlYVa2zyjQdy6qjWeKBHx1A87kxA9MPW67gwKCfSr9/eF8ybFsvu+ixkSGoiIkPH7yzxD5LojIlww+dillp5wB3h/bUTd5bL+Ni42gtjBIcQODvGbPjshkhdumQfA9v9ZfFLa0l+GRQQTFhRAfXNrt6XDgeC4ceDgHUroWwd2igCXeHad5yT0f49fWRtp925xUIDrmLXYk2HayCEMCgk8ZsnFCb5/zli+/Nn5A/569icR8dT4T8UAd2QP/OYFSUwbOaRf63v9KXFYOJX1zR0O/qh/DRdPG07Kry/qchSQUwQFuBga5sg+4AlJiAonvajGbxTMqaLHAS4iCcBrwAigDXjeGPNEXzXsWBKHhZM4QMN2+sLPF0+mvK7p/3TPRXVNRBwf3v9K3CXHqG4OxA6E3vTAW4CfG2N2iMhgYLuIrDDG7O/ugf/qTjuOIXxKqVPDDfMTiB8aekpudHsc4MaYAqDAvl0tIgeAUYAGuFLq/4wpI4b4nQdwKumTApaIJAFzgC2d/O82EUkRkZSSko6XaVJKKdUzvQ5wERkEvAP8lzGmwxeSGGOeN8YkG2OSY2P7doiYUkr9K+tVgItIEFZ4v2GMebdvmqSUUup49DjAxRpC8SJwwBjz575rklJKqePRmx742cDNwEIR2WX/XN5H7VJKKdWN3oxC2QDoQGallBog//dPo1JKqf+jNMCVUsqhxH0h2ZMyM5ESIKeHD48BSru9lzPospyadFlOTbosMMYY02Ec9kkN8N4QkRRjTPJAt6Mv6LKcmnRZTk26LF3TEopSSjmUBrhSSjmUkwL8+YFuQB/SZTk16bKcmnRZuuCYGrhSSil/TuqBK6WU8qEBrpRSDuWIABeRS0UkTUQOisjdA92eEyUi2SKy1/6+mBR7WrSIrBCRDPv3KXmFYxF5SUSKRWSfz7RO2y6WJ+31tEdETh+4lvvrYjl+KyL5nX2Xj4jcYy9HmohcMjCt7pyIJIjIahE5ICJfi8id9nQnrpeulsVx60ZEQkVkq4jstpflfnv6WBHZYq+Xt0Uk2J4eYv990P5/0gnP1BhzSv8AAUAmMA4IBnYD0wa6XSe4DNlATLtpDwF327fvBv400O3sou3nAacD+7prO3A58BnWd+QsALYMdPu7WY7fAnd1ct9p9vssBBhrv/8CBnoZfNoXD5xu3x4MpNttduJ66WpZHLdu7Nd3kH07COsCNwuAfwA32NOfBX5k374deNa+fQPw9onO0wk98PnAQWPMIWNME/AWcM0At6kvXAO8at9+FVgygG3pkjFmHVDebnJXbb8GeM1YvgIiRST+5LT02LpYjq5cA7xljGk0xmQBB7Heh6cEY0yBMWaHfbsacF/O0Inrpatl6copu27s17fG/jPI/jHAQmCZPb39enGvr2XAIjnBK507IcBHAYd9/s7j2Cv4VGSA5SKyXURus6cNN9Z1RbF/xw1Y605cV2134rq6wy4rvORTxnLMcrS7nKGj10snl2Z03LoRkQAR2QUUAyuw9hAqjDEt9l182+tZFvv/lcCwE5mfEwK8sy2S08Y+nm2MOR24DPhPETlvoBvUT5y2rp4BxgOzsS7Q/ag93RHL0d3lDH3v2sm0U2p5OlkWR64bY0yrMWY2MBprz2BqZ3ezf/d6WZwQ4HlAgs/fo4EjA9SWHjHGHLF/FwPvYa3YIvdurP27eOBaeMK6aruj1pUxpsj+wLUBf8O7K37KL0cXlzN05HrpbFmcvG4AjDEVwBqsGnikiLivveDbXs+y2P8fyvGX+QBnBPg2YKJ9JDcYq9j/4QC36biJSISIDHbfBi4G9mEtwy323W4BPhiYFvZIV23/EPh3e9TDAqDSvUt/KmpXB/4G1noBazlusEcJjAUmAltPdvu6YtdJO7ucoePWS1fL4sR1IyKxIhJp3w4DLsKq6a8GrrXv1n69uNfXtcAqYx/RPG4DfeT2OI/uXo51dDoTuHeg23OCbR+HddR8N/C1u/1Yta6VQIb9O3qg29pF+5di7cI2Y/UYbu2q7Vi7hE/b62kvkDzQ7e9mOV6327nH/jDF+9z/Xns50oDLBrr97ZblHKxd7T3ALvvncoeul66WxXHrBpgJ7LTbvA/4jT19HNZG5iDwTyDEnh5q/33Q/v+4E52nnkqvlFIO5YQSilJKqU5ogCullENpgCullENpgCullENpgCullENpgCullENpgCullEP9fxcmTy2pjuOSAAAAAElFTkSuQmCC\n", 683 | "text/plain": [ 684 | "
" 685 | ] 686 | }, 687 | "metadata": { 688 | "needs_background": "light" 689 | }, 690 | "output_type": "display_data" 691 | } 692 | ], 693 | "source": [ 694 | "%run examples/classification/digits.py" 695 | ] 696 | }, 697 | { 698 | "cell_type": "markdown", 699 | "metadata": {}, 700 | "source": [ 701 | "---\n", 702 | "## Software engineering" 703 | ] 704 | }, 705 | { 706 | "cell_type": "markdown", 707 | "metadata": {}, 708 | "source": [ 709 | "### Unit Test\n", 710 | "\n", 711 | "Reference the test case from google/jax. So far, We have already covered the test for value and vjp.\n", 712 | "\n", 713 | "We test different shape including scalar, vector and matrix." 714 | ] 715 | }, 716 | { 717 | "cell_type": "code", 718 | "execution_count": 10, 719 | "metadata": {}, 720 | "outputs": [ 721 | { 722 | "name": "stdout", 723 | "output_type": "stream", 724 | "text": [ 725 | "\u001b[1m============================= test session starts ==============================\u001b[0m\n", 726 | "platform linux -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0\n", 727 | "rootdir: /home/chenyee/AutoDiff-from-scratch\n", 728 | "plugins: arraydiff-0.3, remotedata-0.3.2, openfiles-0.4.0, doctestplus-0.4.0, forked-1.1.3, xdist-1.30.0\n", 729 | "collected 78 items \u001b[0m\n", 730 | "\n", 731 | "autodiff/tests/autodiff_test.py \u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[36m [ 50%]\u001b[0m\n", 732 | "autodiff/tests/numpy_test.py \u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[32m.\u001b[0m\u001b[36m [100%]\u001b[0m\n", 733 | "\n", 734 | "\u001b[32m\u001b[1m============================== 78 passed in 0.35s ==============================\u001b[0m\n" 735 | ] 736 | } 737 | ], 738 | "source": [ 739 | "! pytest autodiff/tests" 740 | ] 741 | }, 742 | { 743 | "cell_type": "markdown", 744 | "metadata": {}, 745 | "source": [ 746 | "### CI/CD\n", 747 | "\n", 748 | "- Use Github Actions\n", 749 | "- Also testing on container provided by TA (on `dev` branch)\n", 750 | "\n", 751 | "![actions](img/actions.png)" 752 | ] 753 | }, 754 | { 755 | "cell_type": "markdown", 756 | "metadata": {}, 757 | "source": [ 758 | "## Misc" 759 | ] 760 | }, 761 | { 762 | "cell_type": "markdown", 763 | "metadata": {}, 764 | "source": [ 765 | "### C++ building\n", 766 | "\n", 767 | "- Try to build C++ VJP on `dev` branch, but improved efficiency is questionable.\n", 768 | "- There is another potential bottleneck worth to improve, which is the graph building (use networkx currently).\n", 769 | "\n", 770 | "```c++\n", 771 | "typedef py::array_t tensor;\n", 772 | "\n", 773 | "tensor negative_vjp(tensor upstream, tensor result, tensor x)\n", 774 | "{\n", 775 | " return -upstream;\n", 776 | "}\n", 777 | "\n", 778 | "tensor reciprocal_vjp(tensor upstream, tensor result, tensor x)\n", 779 | "{\n", 780 | " py::object np_pow = py::module::import(\"numpy\").attr(\"power\");\n", 781 | " return -upstream / np_pow(x, 2);\n", 782 | "}\n", 783 | "```" 784 | ] 785 | }, 786 | { 787 | "cell_type": "markdown", 788 | "metadata": {}, 789 | "source": [ 790 | "### Open Source contributions\n", 791 | "\n", 792 | "Find the redundant calculations of the derivatives of power function.\n", 793 | "1. [PyTorch](https://github.com/pytorch/pytorch/pull/28651)\n", 794 | "2. [JAX](https://github.com/google/jax/pull/157)\n", 795 | "3. [Autograd](https://github.com/HIPS/autograd/pull/541)" 796 | ] 797 | } 798 | ], 799 | "metadata": { 800 | "kernelspec": { 801 | "display_name": "Python 3", 802 | "language": "python", 803 | "name": "python3" 804 | }, 805 | "language_info": { 806 | "codemirror_mode": { 807 | "name": "ipython", 808 | "version": 3 809 | }, 810 | "file_extension": ".py", 811 | "mimetype": "text/x-python", 812 | "name": "python", 813 | "nbconvert_exporter": "python", 814 | "pygments_lexer": "ipython3", 815 | "version": "3.7.4" 816 | } 817 | }, 818 | "nbformat": 4, 819 | "nbformat_minor": 2 820 | } 821 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Basic neural network library supported auto-differentiation 2 | 3 | ## Introduction 4 | 5 | This is the very simple neural network library that supporting auto-differtiation. After referenced works from priors (see *Reference*), I provide the implementation which is simple enough that can tackle the real-world problem (regression and classification). 6 | 7 | ## Documents 8 | 9 | ### Forward-prop and backward-prop 10 | 11 | Source: 12 | 13 | - `autodiff/autodiff/core.py` 14 | - `autodiff/autodiff/diff.py` 15 | 16 | We init the graph in the forward prop and call the wrapped operation (See *Wrapped operation and VJP*). I am too lazy to use the DiGraph in networkx as the computational graph but only use litte features (build graph and topological sort). 17 | 18 | Note that unlike `autograd` and `jax`, I can only calculate the first order derivative of the given function. 19 | 20 | ### Wrapped operation and VJP 21 | 22 | Source: 23 | 24 | - `autodiff/autodiff/numpy_grad/wrapper.py` 25 | - `autodiff/autodiff/numpy_grad/vjps.py` 26 | 27 | In the forward-prop, we construct the computational graph (See *Computational graph*) with the wrapped numpy function in the wrapper module. 28 | 29 | In the backward-prop, we need to reference the VJP form of the given operation, we register the VJPs in the very begining. I only support little function but it's enough to do the more higher implementation (See *High level usage*). 30 | 31 | ### Computational graph 32 | 33 | Source: 34 | 35 | - `autodiff/autodiff/graph/tracer.py` (wrapped operation and graph building) 36 | - `autodiff/autodiff/graph/node.py` (nodes classes) 37 | - `autodiff/autodiff/graph/manager.py` (Context manager for graph) 38 | 39 | Notice that we don't have to worry about the how the node has been built (order of execution), because the python interpreter know it, for example 40 | 41 | ```python 42 | ad.add(ad.Variable(x), ad.Variable(y) 43 | ``` 44 | 45 | The interpreter know that we need to construct the Variable of x and then Variable of y first, after that we do the operation of add. 46 | 47 | By fully leverage this feature, we can push variable nodes into stack and pop them from stack in the primitive operation and connect them with an edge. 48 | 49 | Note that in the current implementation, I use `id(x)` to identify different nodes in the graph (which is bad but simple). Due to small integer may have the same address which can cause program no longer work as expected, there are lots of TODO in the future. 50 | 51 | ### High level usage 52 | 53 | Source: 54 | 55 | - `autodiff/nn/layer.py` (Module and Layer classes) 56 | - `autodiff/nn/criterion.py` (Criterion for different problems) 57 | - `autodiff/nn/optimizer.py` (Optimizers) 58 | 59 | In the high-level implementation, Fully imitate the syntax from pytorch but it only support the multi-layer perceptron with little weird syntax in the current time, as shown below. 60 | 61 | ```python 62 | class SimpleModel(Module): 63 | def __init__(self, num_features, num_classes): 64 | super().__init__() 65 | self.linear1 = Linear(num_features, 5) 66 | self.linear2 = Linear(5, num_classes) 67 | 68 | def forward(self, x): 69 | x = self.linear1(x, bridge=False) 70 | x = self.linear2(x) 71 | return x 72 | ``` 73 | 74 | This model want to solve the classification problem with two layers, notice that in the `forward` method, first layer MUST assign bridge to `False` to generate a new `Placeholder` for x. 75 | 76 | ### Example 77 | 78 | Source: 79 | 80 | - `examples\*` 81 | 82 | We cover the real-world problem including regression and classification. I use MSE criterior for regression and cross-entropy loss for classification. 83 | 84 | ### Test 85 | 86 | Source: 87 | 88 | - `autodiff\tests\*` 89 | 90 | So far, we cover the test for get value from function and it's derivatives. 91 | 92 | ## Reference 93 | 94 | ### Implementation 95 | 96 | - Core idea from [Autograd](https://github.com/HIPS/autograd) 97 | - Test procedure and test cases from [JAX](https://github.com/google/jax) 98 | - Variable, Placeholder and Constant syntax from [TensorFlow](https://github.com/tensorflow/tensorflow) 99 | - Layer syntax and cross entropy from [PyTorch](https://github.com/pytorch/pytorch) 100 | 101 | 102 | ### Source code 103 | 104 | - [Autograd](https://github.com/HIPS/autograd) 105 | - [Autodidact](https://github.com/mattjj/autodidact) 106 | - [PyTorch](https://github.com/pytorch/pytorch) 107 | - [TensorFlow](https://github.com/tensorflow/tensorflow) 108 | - [Caffe](https://github.com/BVLC/caffe) 109 | - [Caffe2](https://github.com/pytorch/pytorch/tree/master/caffe2) 110 | - [JAX](https://github.com/google/jax) 111 | 112 | ### Lecture 113 | 114 | - [Backpropagation, Toronto CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec06.pdf) 115 | - [Automatic Differentiation, Toronto CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec10.pdf) 116 | - [Backpropagation, Stanford CS224N](https://www.youtube.com/watch?v=yLYHDSv-288&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z&index=5&t=2177s) 117 | - [Introduction to Neural Networks, Stanford CS231N](https://www.youtube.com/watch?v=d14TUNcbn1k&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=4) 118 | - [Backpropagation: Find Partial Derivatives, MIT 18.065](https://www.youtube.com/watch?v=lZrIPRnoGQQ&list=PLUl4u3cNGP63oMNUHXqIUcrkS2PivhN3k&index=30&t=0s) 119 | 120 | ### Documents 121 | 122 | - [Jax official](https://jax.readthedocs.io/en/latest/index.html) 123 | - [Phd Thesis by Dougal Maclaurin (one of Autograd author)](https://dougalmaclaurin.com/phd-thesis.pdf) 124 | -------------------------------------------------------------------------------- /autodiff/.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ -------------------------------------------------------------------------------- /autodiff/__init__.py: -------------------------------------------------------------------------------- 1 | from . import autodiff # register the VHP of primitive function 2 | import autodiff.autodiff.numpy_grad.wrapper as np # export the primitive function 3 | #TODO need refactor, may use __all__? 4 | from autodiff.autodiff.core import forward_prop, backward_prop, zero_grad 5 | from autodiff.autodiff.global_vars import set_forwarded, set_parameters 6 | 7 | # export the gradient-related function 8 | from autodiff.autodiff.diff import * 9 | globals().update(np.__dict__) 10 | 11 | for func in [ 12 | set_parameters, 13 | set_forwarded, 14 | forward_prop, 15 | backward_prop, 16 | zero_grad, 17 | ]: 18 | globals()[func.__name__] = func 19 | -------------------------------------------------------------------------------- /autodiff/autodiff/__init__.py: -------------------------------------------------------------------------------- 1 | from . import numpy_grad, graph 2 | -------------------------------------------------------------------------------- /autodiff/autodiff/core.py: -------------------------------------------------------------------------------- 1 | from collections import defaultdict 2 | 3 | import networkx as nx 4 | 5 | from .graph.node import OperationNode, VariableNode, PlaceholderNode 6 | from .global_vars import register_graph, pop_graph 7 | 8 | primitive_vhp = defaultdict(dict) 9 | 10 | 11 | def is_wrt(node): 12 | return type(node) in [VariableNode, PlaceholderNode] 13 | 14 | 15 | def forward_prop(func, provided_graph=None): 16 | def forward_wrap(*args, **kwargs): 17 | graph = nx.DiGraph() if provided_graph is None else provided_graph 18 | register_graph(graph) 19 | return func(*args, **kwargs) 20 | 21 | return forward_wrap 22 | 23 | 24 | def backward_prop(upstream): 25 | graph = pop_graph() 26 | graph.nodes[len(graph.nodes())]['node'].gradient = upstream 27 | # print("Set gradient to ", upstream, len(graph.nodes())) 28 | gradient_dict = {} 29 | for node in reversed(list(nx.topological_sort(graph))): 30 | child_node: OperationNode = graph.nodes[node]['node'] 31 | if isinstance(child_node, OperationNode): 32 | func, args, kwargs, result, arg_num = child_node.recipe 33 | upstream = child_node.gradient 34 | # print(func.__name__, node, upstream, args, arg_num) 35 | 36 | for i, parent in zip(range(arg_num), graph.predecessors(node)): 37 | vhp = primitive_vhp[func.__name__][i] 38 | downstream = vhp(upstream, result, *args, **kwargs) 39 | # print(i, "downstream size is", downstream.shape) 40 | graph.nodes[parent]['node'].gradient += downstream 41 | 42 | elif is_wrt(child_node): 43 | gradient_dict[child_node.var] = child_node.gradient 44 | 45 | return gradient_dict 46 | 47 | 48 | def zero_grad(graph): 49 | for node_index in graph.nodes: 50 | graph.nodes[node_index]['node'].gradient = 0 51 | 52 | 53 | def register_vjp(func, vhp_list): 54 | for i, downstream in enumerate(vhp_list): 55 | primitive_vhp[func.__name__][i] = downstream 56 | -------------------------------------------------------------------------------- /autodiff/autodiff/diff.py: -------------------------------------------------------------------------------- 1 | import numpy as onp 2 | 3 | from .core import forward_prop, backward_prop 4 | 5 | 6 | def value(func): 7 | def valueWrapped(*args, **kwargs): 8 | forward_func = forward_prop(func) 9 | return forward_func(*args, **kwargs) 10 | 11 | return valueWrapped 12 | 13 | 14 | def grad(func, wrt=None, upstream=None): 15 | def gradVal(*args, **kwargs): 16 | forward_func = forward_prop(func) 17 | end_value = forward_func(*args, **kwargs) 18 | g = onp.ones_like(end_value) if upstream is None else upstream 19 | grad = backward_prop(g) 20 | return grad if wrt is None else grad[wrt] 21 | 22 | return gradVal 23 | 24 | 25 | def value_and_grad(func, wrt=None): 26 | def gradVal(*args, **kwargs): 27 | forward_func = forward_prop(func) 28 | end_value = forward_func(*args, **kwargs) 29 | grad = backward_prop(onp.ones_like(end_value)) 30 | return (end_value, grad) if wrt is None else (end_value, grad[wrt]) 31 | 32 | return gradVal 33 | -------------------------------------------------------------------------------- /autodiff/autodiff/global_vars.py: -------------------------------------------------------------------------------- 1 | from collections import defaultdict 2 | 3 | 4 | class GraphInfo: 5 | def __init__(self): 6 | self.stack = [] 7 | self.vars = dict() 8 | self.places = dict() 9 | self.forwarded = False 10 | 11 | 12 | class var: 13 | pass 14 | 15 | 16 | global_vars = var() 17 | global_vars._graph_stack = [] 18 | global_vars._graph_info_dict = defaultdict(GraphInfo) 19 | 20 | 21 | def register_graph(graph): 22 | global_vars._graph_stack.append(graph) 23 | 24 | 25 | def pop_graph(): 26 | return global_vars._graph_stack.pop() 27 | 28 | 29 | def set_forwarded(graph): 30 | global_vars._graph_info_dict[graph].forwarded = True 31 | 32 | 33 | def set_parameters(parameters, graph): 34 | for array_id, node_index in global_vars._graph_info_dict[graph].vars.items( 35 | ): 36 | parameters.append({ 37 | "array_id": array_id, 38 | "variables": graph.nodes[node_index]['node'].content 39 | }) 40 | 41 | 42 | def get_graph_info(graph): 43 | return global_vars._graph_info_dict.get(graph, GraphInfo()) 44 | 45 | 46 | def update_graph_info(graph, graph_info): 47 | global_vars._graph_info_dict[graph] = graph_info 48 | -------------------------------------------------------------------------------- /autodiff/autodiff/graph/__init__.py: -------------------------------------------------------------------------------- 1 | from . import tracer -------------------------------------------------------------------------------- /autodiff/autodiff/graph/manager.py: -------------------------------------------------------------------------------- 1 | import networkx as nx 2 | 3 | from ..global_vars import get_graph_info, update_graph_info, register_graph, pop_graph 4 | 5 | 6 | class GraphManager: 7 | def __init__(self): 8 | pass 9 | 10 | def __enter__(self): 11 | self.graph: nx.DiGraph = pop_graph() 12 | self.graph_info = get_graph_info(self.graph) 13 | return self.graph, self.graph_info 14 | 15 | def __exit__(self, *args): 16 | register_graph(self.graph) 17 | update_graph_info(self.graph, self.graph_info) 18 | 19 | 20 | def add_node(graph, node): 21 | node_index = len(graph.nodes()) + 1 22 | graph.add_node(node_index, node=node) 23 | return node_index -------------------------------------------------------------------------------- /autodiff/autodiff/graph/node.py: -------------------------------------------------------------------------------- 1 | only_take_lhs_grad = [ 2 | "reshape", 3 | "__getitem__", 4 | ] 5 | 6 | class Node: 7 | def __init__(self): 8 | self.gradient = 0 9 | 10 | 11 | class OperationNode(Node): 12 | def __init__(self, func, args, kwargs, result): 13 | super().__init__() 14 | self.recipe = (func, args, kwargs, result, 15 | len(args) if func.__name__ not in only_take_lhs_grad else 1) 16 | 17 | 18 | class VariableNode(Node): 19 | def __init__(self, var_id, content): 20 | super().__init__() 21 | self.var = var_id 22 | self.content = content 23 | 24 | 25 | class PlaceholderNode(Node): 26 | def __init__(self, var): 27 | super().__init__() 28 | self.var = var 29 | 30 | 31 | class ConstantNode(Node): 32 | def __init__(self, constant): 33 | super().__init__() 34 | self.constant = constant -------------------------------------------------------------------------------- /autodiff/autodiff/graph/tracer.py: -------------------------------------------------------------------------------- 1 | from functools import wraps 2 | 3 | import networkx as nx 4 | 5 | from .node import ConstantNode, OperationNode, VariableNode, PlaceholderNode 6 | from .manager import GraphManager, add_node 7 | 8 | #TODO refactor this part 9 | def constant(array): 10 | def const_wrapped(content): 11 | with GraphManager() as (graph, info): 12 | if not info.forwarded: 13 | node = ConstantNode(content) 14 | node_index = add_node(graph, node) 15 | info.stack.append(node_index) 16 | # print('const', node_index, content) 17 | return content if isinstance(content, tuple) else array(content) 18 | 19 | return const_wrapped 20 | 21 | 22 | def variable(array): 23 | def var_wrapped(content): 24 | with GraphManager() as (graph, info): 25 | if not info.forwarded: 26 | var_id = id(content) 27 | if var_id not in info.vars: 28 | node = VariableNode(var_id, content) 29 | node_index = add_node(graph, node) 30 | info.vars[var_id] = node_index 31 | else: 32 | node_index = info.vars[var_id] 33 | info.stack.append(node_index) 34 | # print('var', node_index, content.shape) 35 | return array(content) 36 | 37 | return var_wrapped 38 | 39 | 40 | def placeholder(array): 41 | def place_wrapped(content): 42 | with GraphManager() as (graph, info): 43 | if not info.forwarded: 44 | place_id = id(content) 45 | if place_id not in info.places: 46 | node = PlaceholderNode(place_id) 47 | node_index = add_node(graph, node) 48 | info.places[place_id] = node_index 49 | else: 50 | node_index = info.places[place_id] 51 | info.stack.append(node_index) 52 | # print('place', node_index, content.shape) 53 | return array(content) 54 | 55 | return place_wrapped 56 | 57 | 58 | def primitive(func): 59 | @wraps(func) 60 | def func_wrapped(*args, **kwargs): 61 | result = func(*args, **kwargs) 62 | with GraphManager() as (graph, info): 63 | if not info.forwarded: 64 | node = OperationNode(func, args, kwargs, result) 65 | node_index = add_node(graph, node) 66 | 67 | parents = info.stack[-len(args):] 68 | for parent in parents: 69 | graph.add_edge(parent, node_index) 70 | info.stack.pop() 71 | 72 | # print('fun', node_index, func.__name__) 73 | 74 | info.stack.append(node_index) 75 | return result 76 | 77 | return func_wrapped 78 | -------------------------------------------------------------------------------- /autodiff/autodiff/numpy_grad/.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/ -------------------------------------------------------------------------------- /autodiff/autodiff/numpy_grad/__init__.py: -------------------------------------------------------------------------------- 1 | from . import vjps -------------------------------------------------------------------------------- /autodiff/autodiff/numpy_grad/vjps.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import numpy as onp 3 | 4 | from ..numpy_grad import wrapper as wnp 5 | from ..core import register_vjp 6 | 7 | # print(__name__) 8 | """ 9 | Unary operation 10 | """ 11 | register_vjp( 12 | wnp.negative, 13 | [ 14 | lambda upstream, result, x: -upstream, # w.r.t. x 15 | ]) 16 | 17 | register_vjp( 18 | wnp.reciprocal, 19 | [ 20 | lambda upstream, result, x: upstream * (-1 / x**2), # w.r.t. x 21 | ]) 22 | 23 | register_vjp( 24 | wnp.exp, 25 | [ 26 | lambda upstream, result, x: upstream * result, # w.r.t. x 27 | ]) 28 | 29 | register_vjp( 30 | wnp.log, 31 | [ 32 | lambda upstream, result, x: upstream / x, # w.r.t. x 33 | ]) 34 | 35 | register_vjp( 36 | wnp.sin, 37 | [ 38 | lambda upstream, result, x: upstream * onp.cos(x), # w.r.t. x 39 | ]) 40 | 41 | register_vjp( 42 | wnp.cos, 43 | [ 44 | lambda upstream, result, x: upstream * -onp.sin(x), # w.r.t. x 45 | ]) 46 | """ 47 | Binary operation 48 | """ 49 | register_vjp( 50 | wnp.add, 51 | [ 52 | lambda upstream, result, x, y: unbroadcast(x, upstream), # w.r.t. x 53 | lambda upstream, result, x, y: unbroadcast(y, upstream), # w.r.t. y 54 | ]) 55 | 56 | register_vjp( 57 | wnp.subtract, 58 | [ 59 | lambda upstream, result, x, y: unbroadcast(x, upstream, other=y 60 | ), # w.r.t. x 61 | lambda upstream, result, x, y: unbroadcast(y, -upstream, other=x 62 | ), # w.r.t. y 63 | ]) 64 | 65 | register_vjp( 66 | wnp.multiply, 67 | [ 68 | lambda upstream, result, x, y: unbroadcast(x, upstream * y 69 | ), # w.r.t. x 70 | lambda upstream, result, x, y: unbroadcast(y, upstream * x 71 | ), # w.r.t. y 72 | ]) 73 | 74 | register_vjp( 75 | wnp.true_divide, 76 | [ 77 | lambda upstream, result, x, y: unbroadcast(x, upstream / y 78 | ), # w.r.t. x 79 | lambda upstream, result, x, y: unbroadcast(y, upstream * 80 | (-x / y**2)), # w.r.t. y 81 | ]) 82 | 83 | register_vjp( 84 | wnp.maximum, 85 | [ 86 | lambda upstream, result, x, y: upstream * balanced_eq(x, result, y 87 | ), # w.r.t. x 88 | lambda upstream, result, x, y: upstream * balanced_eq(y, result, x 89 | ), # w.r.t. y 90 | ]) 91 | 92 | register_vjp( 93 | wnp.minimum, 94 | [ 95 | lambda upstream, result, x, y: upstream * balanced_eq(x, result, y 96 | ), # w.r.t. x 97 | lambda upstream, result, x, y: upstream * balanced_eq(y, result, x 98 | ), # w.r.t. y 99 | ]) 100 | 101 | register_vjp( 102 | wnp.power, 103 | [ 104 | lambda upstream, result, x, y: unbroadcast( 105 | x, upstream * (y * x**(y - 1))), # w.r.t. x 106 | lambda upstream, result, x, y: unbroadcast( 107 | y, upstream * (result * onp.log(replace_zero(x, 1.)))), # w.r.t. y 108 | ]) 109 | 110 | 111 | # shamelessly taken from autograd 112 | def replace_zero(x, val): 113 | return onp.where(x, x, val) 114 | 115 | 116 | def unbroadcast(target, g, broadcast_idx=0, other=None): 117 | """ 118 | Let downstream have the same shape as target 119 | """ 120 | # if onp.ndim(g) == 2: 121 | # g = g.diagonal()[:,None] 122 | # print("-" * 10) 123 | # print(onp.ndim(g) > onp.ndim(target), g.shape, target.shape) 124 | # print(other) 125 | # print(target) 126 | # print("Before", g) 127 | # print(g, target) 128 | while onp.ndim(g) > onp.ndim(target): 129 | g = onp.sum(g, axis=broadcast_idx) 130 | # print("Sum", g) 131 | 132 | if onp.ndim(g) == onp.ndim(target): 133 | for axis, size in enumerate(onp.shape(target)): 134 | if size == 1: 135 | g = onp.sum(g, axis=axis, keepdims=True) 136 | # print("After", g) 137 | # print("-" * 10) 138 | return g 139 | 140 | 141 | def balanced_eq(x, z, y): 142 | return (x == z) / (1.0 + (x == y)) 143 | 144 | 145 | """ 146 | Matrix calculation 147 | """ 148 | 149 | 150 | def dot_vjp_first(upstream, result, x, y): 151 | # print("first: ", upstream, result.shape, x.shape, y.shape) 152 | 153 | if not (onp.ndim(x) == onp.ndim(y) == 2): 154 | raise NotImplementedError("Only care about MM or MV product!") 155 | 156 | # Take the derivative of output respect to x (input) 157 | downstream = onp.dot(upstream, y.T) 158 | 159 | assert downstream.shape == x.shape 160 | 161 | return downstream 162 | 163 | 164 | def dot_vjp_second(upstream, result, x, y): 165 | # print("second: ", upstream, result.shape, x.shape, y.shape) 166 | 167 | if not (onp.ndim(x) == onp.ndim(y) == 2): 168 | raise NotImplementedError("Only care about MM or MV product!") 169 | 170 | # Take the derivative of output respect to y (weight) 171 | downstream = onp.dot(x.T, upstream) 172 | 173 | assert downstream.shape == y.shape 174 | 175 | return downstream 176 | 177 | 178 | register_vjp( 179 | wnp.dot, 180 | [ 181 | dot_vjp_first, # w.r.t. x 182 | dot_vjp_second, # w.r.t. y 183 | ]) 184 | 185 | # Special operator 186 | 187 | register_vjp(wnp.reshape, [ 188 | lambda upstream, result, x, shape, order=None: onp.reshape( 189 | upstream, onp.shape(x), order=order) 190 | ]) 191 | 192 | 193 | def sum_vjp(upstream, result, x, axis=1, keepdims=False, dtype=None): 194 | shape = onp.shape(x) 195 | 196 | if not shape: 197 | return upstream 198 | 199 | # print(result, x, axis) 200 | axis = list(axis) if isinstance(axis, tuple) else axis 201 | new_shape = onp.array(shape) 202 | new_shape[axis] = 1 203 | # print(onp.reshape(upstream, new_shape)) 204 | return onp.reshape(upstream, new_shape) + onp.zeros(shape, dtype=dtype) 205 | 206 | 207 | register_vjp(wnp.sum, [sum_vjp]) 208 | 209 | 210 | def getitem_vjp(upstream, result, x, index): 211 | # print(x, index, upstream) 212 | onp.add.at(x, index, upstream) 213 | return x 214 | 215 | register_vjp(wnp.__getitem__, [getitem_vjp]) 216 | 217 | -------------------------------------------------------------------------------- /autodiff/autodiff/numpy_grad/wrapper.py: -------------------------------------------------------------------------------- 1 | import types 2 | 3 | import numpy as _np 4 | 5 | from ..graph.tracer import primitive, constant, variable, placeholder 6 | 7 | nograd_functions = [ 8 | _np.ndim, _np.shape, _np.iscomplexobj, _np.result_type, _np.zeros_like, 9 | _np.ones_like, _np.floor, _np.ceil, _np.round, _np.rint, _np.around, 10 | _np.fix, _np.trunc, _np.all, _np.any, _np.argmax, _np.argmin, 11 | _np.argpartition, _np.argsort, _np.argwhere, _np.nonzero, _np.flatnonzero, 12 | _np.count_nonzero, _np.searchsorted, _np.sign, _np.ndim, _np.shape, 13 | _np.floor_divide, _np.logical_and, _np.logical_or, _np.logical_not, 14 | _np.logical_xor, _np.isfinite, _np.isinf, _np.isnan, _np.isneginf, 15 | _np.isposinf, _np.allclose, _np.isclose, _np.array_equal, _np.array_equiv, 16 | _np.greater, _np.greater_equal, _np.less, _np.less_equal, _np.equal, 17 | _np.not_equal, _np.iscomplexobj, _np.iscomplex, _np.size, _np.isscalar, 18 | _np.isreal, _np.zeros_like, _np.ones_like, _np.result_type 19 | ] 20 | 21 | excluded_function = [ 22 | _np.linspace, 23 | _np.arange, 24 | ] 25 | 26 | 27 | def wrap_func(numpy, local): 28 | # Wrap numpy primitive function 29 | function_types = {_np.ufunc, types.FunctionType, types.BuiltinFunctionType} 30 | for name, func in numpy.items(): 31 | if func in nograd_functions or func in excluded_function: 32 | local[name] = func 33 | elif type(func) in function_types: 34 | local[name] = primitive(func) 35 | elif isinstance(func, type) and _np.issubdtype(func, _np.integer): 36 | local[name] = func 37 | 38 | # Wrap numpy array member function 39 | for func in [_np.ndarray.__getitem__, _np.ndarray.__len__, _np.ndarray.__contains__]: 40 | local[func.__name__] = primitive(func) 41 | 42 | 43 | wrap_func(_np.__dict__, globals()) 44 | globals()['Constant'] = constant(_np.array) 45 | globals()['Variable'] = variable(_np.array) 46 | globals()['Placeholder'] = placeholder(_np.array) 47 | -------------------------------------------------------------------------------- /autodiff/nn/criterion.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | 3 | import numpy as _np 4 | 5 | import autodiff as ad 6 | from autodiff import value_and_grad, value, grad 7 | 8 | 9 | class Criterion: 10 | def __init__(self): 11 | pass 12 | 13 | def __call__(self, true, predicted): 14 | value, loss_grad = value_and_grad(self.loss_func, 15 | id(predicted))(feed_dict={ 16 | 'predicted_y': predicted, 17 | 'true_y': true 18 | }) 19 | return _np.average(value), loss_grad 20 | 21 | def loss_func(self): 22 | pass 23 | 24 | 25 | class MSE(Criterion): 26 | def __init__(self): 27 | super().__init__() 28 | 29 | def loss_func(self, feed_dict={}): 30 | diff = ad.subtract(ad.Placeholder(feed_dict['predicted_y']), 31 | ad.Placeholder(feed_dict['true_y'])) 32 | return ad.multiply(ad.power(diff, ad.Constant(2)), ad.Constant(1 / 2)) 33 | 34 | 35 | class CrossEntropy(Criterion): 36 | def __init__(self): 37 | super().__init__() 38 | 39 | def loss_func(self, feed_dict={}): 40 | batch_size = feed_dict["predicted_y"].shape[0] 41 | return ad.add( 42 | ad.negative( 43 | ad.__getitem__( 44 | ad.Placeholder(feed_dict["predicted_y"]), 45 | ad.Constant( 46 | tuple([range(batch_size), 47 | feed_dict["true_y"].ravel()])))), 48 | ad.log( 49 | ad.sum(ad.exp(ad.Placeholder(feed_dict["predicted_y"])), 50 | axis=1))) 51 | -------------------------------------------------------------------------------- /autodiff/nn/layer.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import numpy as _np 3 | import networkx as nx 4 | 5 | import autodiff as ad 6 | 7 | 8 | class Module: 9 | def __init__(self): 10 | self._graph = nx.DiGraph() 11 | self._parameters = [] 12 | 13 | def __call__(self, *args): 14 | result = ad.forward_prop(self.forward, provided_graph=self._graph)(*args) 15 | # only set parameters at the first forward pass 16 | if not self._parameters: 17 | ad.set_parameters(self._parameters, self._graph) 18 | ad.set_forwarded(self._graph) 19 | 20 | return result 21 | 22 | def forward(self, *args): 23 | return 24 | 25 | def zero_grad(self): 26 | ad.zero_grad(self._graph) 27 | 28 | def backward(self, upstream): 29 | return ad.backward_prop(upstream) 30 | 31 | def parameters(self): 32 | return self._parameters 33 | 34 | 35 | class Layer: 36 | def __init__(self): 37 | pass 38 | 39 | def __call__(self): 40 | pass 41 | 42 | 43 | class Linear(Layer): 44 | def __init__(self, in_features, out_features): 45 | super().__init__() 46 | self.weight = _np.random.random((in_features, out_features)) 47 | 48 | def __call__(self, features, bridge=True): 49 | features = features if bridge else ad.Placeholder(features) 50 | return ad.dot(features, ad.Variable(self.weight)) 51 | -------------------------------------------------------------------------------- /autodiff/nn/optimizer.py: -------------------------------------------------------------------------------- 1 | import numpy as _np 2 | 3 | import autodiff as ad 4 | from autodiff import value_and_grad, value, grad 5 | 6 | 7 | class Optimizer: 8 | def __init__(self, lr, parameters): 9 | self.lr = lr 10 | self.parameters = parameters 11 | 12 | def step(self): 13 | pass 14 | 15 | 16 | class GradientDescent(Optimizer): 17 | def __init__(self, lr, parameter): 18 | super().__init__(lr, parameter) 19 | 20 | def step(self, model_grad): 21 | for i, para_info in reversed(list(enumerate(self.parameters))): 22 | self.parameters[i]["variables"] -= self.lr * model_grad[para_info["array_id"]] 23 | -------------------------------------------------------------------------------- /autodiff/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/autodiff/tests/__init__.py -------------------------------------------------------------------------------- /autodiff/tests/autodiff_test.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import unittest 3 | 4 | from absl.testing import parameterized 5 | 6 | import autodiff as ad 7 | from autodiff.autodiff.core import primitive_vhp 8 | import autodiff.utils.test_utils as adu 9 | 10 | 11 | class TestJVPMethods(adu.AutoDiffTestCase): 12 | @parameterized.named_parameters([{ 13 | "testcase_name": 14 | adu.format_test_name(record.name, shape), 15 | "ad_func": 16 | record.ad_func, 17 | "np_func": 18 | record.np_func, 19 | "rng": 20 | record.rng, 21 | "shape": 22 | shape, 23 | "nargs": 24 | record.nargs 25 | } for shape in adu.tested_shapes for record in adu.OP_RECORDS]) 26 | def test_op(self, ad_func, np_func, rng, shape, nargs): 27 | args = [rng(shape) for _ in range(nargs)] 28 | adu.check_vjp(np_func, ad_func, args) 29 | 30 | 31 | if __name__ == "__main__": 32 | unittest.main() -------------------------------------------------------------------------------- /autodiff/tests/numpy_test.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import unittest 3 | import functools 4 | import itertools 5 | from collections import namedtuple 6 | 7 | import numpy as np 8 | import numpy.random as npr 9 | from absl.testing import parameterized 10 | 11 | import autodiff as ad 12 | from autodiff.autodiff.core import primitive_vhp 13 | import autodiff.utils.test_utils as adu 14 | 15 | 16 | class TestNumpyMethods(adu.AutoDiffTestCase): 17 | @parameterized.named_parameters([{ 18 | "testcase_name": 19 | adu.format_test_name(record.name, shape), 20 | "ad_func": 21 | record.ad_func, 22 | "np_func": 23 | record.np_func, 24 | "rng": 25 | record.rng, 26 | "shape": 27 | shape, 28 | "nargs": 29 | record.nargs 30 | } for shape in adu.tested_shapes for record in adu.OP_RECORDS]) 31 | def test_op(self, ad_func, np_func, rng, shape, nargs): 32 | args = [rng(shape) for _ in range(nargs)] 33 | adu.check_value(np_func, ad_func, args) 34 | 35 | 36 | if __name__ == "__main__": 37 | unittest.main() -------------------------------------------------------------------------------- /autodiff/utils/model_utils.py: -------------------------------------------------------------------------------- 1 | import random 2 | 3 | from tqdm import trange 4 | 5 | class Dataset: 6 | def __init__(self, X, Y): 7 | self.X = X 8 | self.Y = Y 9 | 10 | def __len__(self): 11 | return len(self.X) 12 | 13 | def __getitem__(self, idx): 14 | return self.X[idx], self.Y[idx] 15 | 16 | 17 | class DataLoader: 18 | def __init__(self, dataset: Dataset): 19 | self.dataset = dataset 20 | 21 | def next_batch(self, num): 22 | candidate = list(range(len(self.dataset))) 23 | sample_index = random.sample(candidate, num) 24 | return self.dataset[sample_index] 25 | 26 | 27 | def train_procedure(epoch): 28 | for i in trange(epoch): 29 | yield i -------------------------------------------------------------------------------- /autodiff/utils/test_utils.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import unittest 3 | from functools import partial 4 | from collections import namedtuple 5 | 6 | from absl.testing import parameterized 7 | import numpy as np 8 | import numpy.random as npr 9 | 10 | import autodiff as ad 11 | from autodiff import value_and_grad, value, grad 12 | """ 13 | This numerical VJP checking is greatly referenced from google/jax. 14 | """ 15 | """ 16 | For random generator 17 | """ 18 | 19 | 20 | def _rand_type(rand, shape, dtype=np.float64, scale=1., post=lambda x: x): 21 | r = lambda: np.asarray(scale * rand(*shape), dtype=dtype) 22 | vals = r() 23 | return post(vals) 24 | 25 | 26 | def rand_default(): 27 | rand = npr.RandomState(0).randn 28 | return partial(_rand_type, rand, scale=3) 29 | 30 | 31 | def rand_positive(): 32 | post = lambda x: np.where(x < 0, -x, x) 33 | rand = npr.RandomState(0).randn 34 | return partial(_rand_type, rand, scale=2, post=post) 35 | 36 | 37 | def rand_small(): 38 | rand = npr.RandomState(0).randn 39 | return partial(_rand_type, rand, scale=1e-3) 40 | 41 | 42 | def rand_not_small(): 43 | post = lambda x: x + np.where(x > 0, 10.0, -10.0) 44 | rand = npr.RandomState(0).randn 45 | return partial(_rand_type, rand, scale=3., post=post) 46 | 47 | 48 | """ 49 | Format string 50 | """ 51 | 52 | 53 | def format_test_name(func, shape): 54 | return f"{func}_{shape}" 55 | 56 | 57 | """ 58 | For grad check 59 | """ 60 | epsilon = 1e-4 61 | 62 | add = lambda wrt, argument: [ 63 | np.add(arg, epsilon / 2) if i == wrt else arg 64 | for i, arg in enumerate(argument) 65 | ] 66 | sub = lambda wrt, argument: [ 67 | np.subtract(arg, epsilon / 2) if i == wrt else arg 68 | for i, arg in enumerate(argument) 69 | ] 70 | 71 | 72 | def numerical_jvp(func, arguments, wrt): 73 | func_pos = func(*add(wrt, arguments)) 74 | func_neg = func(*sub(wrt, arguments)) 75 | return (func_pos - func_neg) / epsilon 76 | 77 | 78 | default_tol = 1e-2 79 | 80 | 81 | def check_vjp(func, func_vjp, args): 82 | for i in range(len(args)): 83 | out = grad(func_vjp, wrt=id(args[i]))(*args) 84 | expected = numerical_jvp(func, args, i) 85 | np.testing.assert_allclose(out, 86 | expected, 87 | atol=default_tol, 88 | rtol=default_tol) 89 | 90 | 91 | """ 92 | For value check 93 | """ 94 | 95 | 96 | def check_value(func, func_value, args): 97 | out = value(func_value)(*args) 98 | expected = func_value(*args) 99 | np.testing.assert_allclose(out, expected) 100 | 101 | 102 | def func_helper(func): 103 | def wrapped(*args, **kwargs): 104 | arg_list = (ad.Variable(arg) for arg in args) 105 | return ad.__dict__[func](*arg_list) 106 | 107 | return wrapped 108 | 109 | 110 | class AutoDiffTestCase(parameterized.TestCase): 111 | def assert_all_close(self): 112 | pass 113 | 114 | """ 115 | Test cases 116 | """ 117 | 118 | tested_shapes = [(), (3, ), (3, 2)] 119 | record = namedtuple("TestRecord", 120 | ["name", "np_func", "ad_func", "nargs", "rng"]) 121 | 122 | 123 | def record_factory(name, nargs, rng): 124 | np_func = np.__dict__[name] 125 | ad_func = func_helper(name) 126 | return record(name, np_func, ad_func, nargs, rng) 127 | 128 | 129 | OP_RECORDS = [ 130 | record_factory("negative", 1, rand_default()), 131 | record_factory("reciprocal", 1, rand_positive()), 132 | record_factory("exp", 1, rand_small()), 133 | record_factory("log", 1, rand_positive()), 134 | record_factory("sin", 1, rand_default()), 135 | record_factory("cos", 1, rand_default()), 136 | record_factory("add", 2, rand_default()), 137 | record_factory("subtract", 2, rand_default()), 138 | record_factory("multiply", 2, rand_default()), 139 | record_factory("true_divide", 2, rand_not_small()), 140 | record_factory("maximum", 2, rand_default()), 141 | record_factory("minimum", 2, rand_default()), 142 | record_factory("power", 2, rand_positive()), 143 | ] 144 | -------------------------------------------------------------------------------- /examples/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/__init__.py -------------------------------------------------------------------------------- /examples/classification/Digits.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/classification/Digits.png -------------------------------------------------------------------------------- /examples/classification/Iris.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/classification/Iris.png -------------------------------------------------------------------------------- /examples/classification/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/classification/__init__.py -------------------------------------------------------------------------------- /examples/classification/digits.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import warnings 3 | 4 | from sklearn.datasets import load_digits 5 | import matplotlib.pyplot as plt 6 | import matplotlib.cbook 7 | 8 | import autodiff as ad 9 | from autodiff.utils.model_utils import Dataset, DataLoader, train_procedure 10 | from autodiff.nn.optimizer import GradientDescent 11 | from autodiff.nn.criterion import CrossEntropy 12 | from autodiff.nn.layer import Module, Linear 13 | 14 | warnings.filterwarnings("ignore", category=matplotlib.cbook.mplDeprecation) 15 | warnings.filterwarnings("ignore", category=RuntimeWarning) 16 | 17 | 18 | class SimpleModel(Module): 19 | def __init__(self, num_features, num_classes): 20 | super().__init__() 21 | self.linear1 = Linear(num_features, 40) 22 | self.linear2 = Linear(40, num_classes) 23 | 24 | def forward(self, x): 25 | x = self.linear1(x, bridge=False) 26 | x = self.linear2(x) 27 | return x 28 | 29 | 30 | X, Y = load_digits(return_X_y=True) 31 | X = X / 16 32 | Y = Y.reshape(-1, 1) 33 | data_size, num_features = X.shape 34 | num_classes = 10 35 | 36 | model = SimpleModel(num_features, num_classes) 37 | dataset = Dataset(X, Y) 38 | dataloader = DataLoader(dataset) 39 | 40 | lr = 1e-5 41 | opt = GradientDescent(lr, model.parameters()) 42 | loss_func = CrossEntropy() 43 | epoch = 300 44 | batch_size = 32 45 | loss_list = [] 46 | for _ in train_procedure(epoch): 47 | x, y = dataloader.next_batch(batch_size) 48 | predicted_y = model(x) 49 | v, loss_grad = loss_func(y, predicted_y) 50 | model_grad = model.backward(loss_grad) 51 | opt.step(model_grad) 52 | model.zero_grad() 53 | loss_list.append(v) 54 | 55 | plt.plot(range(epoch), loss_list) 56 | plt.savefig('Digits.png') 57 | plt.show() -------------------------------------------------------------------------------- /examples/classification/iris.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import warnings 3 | 4 | from sklearn.datasets import load_iris 5 | import matplotlib.pyplot as plt 6 | import matplotlib.cbook 7 | 8 | import autodiff as ad 9 | from autodiff.utils.model_utils import Dataset, DataLoader, train_procedure 10 | from autodiff.nn.optimizer import GradientDescent 11 | from autodiff.nn.criterion import CrossEntropy 12 | from autodiff.nn.layer import Module, Linear 13 | 14 | warnings.filterwarnings("ignore", category=matplotlib.cbook.mplDeprecation) 15 | warnings.filterwarnings("ignore", category=RuntimeWarning) 16 | 17 | 18 | class SimpleModel(Module): 19 | def __init__(self, num_features, num_classes): 20 | super().__init__() 21 | self.linear1 = Linear(num_features, 5) 22 | self.linear2 = Linear(5, num_classes) 23 | 24 | def forward(self, x): 25 | x = self.linear1(x, bridge=False) 26 | x = self.linear2(x) 27 | return x 28 | 29 | 30 | X, Y = load_iris(return_X_y=True) 31 | Y = Y.reshape(-1, 1) 32 | data_size, num_features = X.shape 33 | num_classes = 3 34 | 35 | model = SimpleModel(num_features, num_classes) 36 | dataset = Dataset(X, Y) 37 | dataloader = DataLoader(dataset) 38 | 39 | lr = 1e-5 40 | opt = GradientDescent(lr, model.parameters()) 41 | loss_func = CrossEntropy() 42 | epoch = 300 43 | batch_size = 32 44 | loss_list = [] 45 | for i in train_procedure(epoch): 46 | x, y = dataloader.next_batch(batch_size) 47 | predicted_y = model(x) 48 | v, loss_grad = loss_func(y, predicted_y) 49 | model_grad = model.backward(loss_grad) 50 | opt.step(model_grad) 51 | model.zero_grad() 52 | loss_list.append(v) 53 | 54 | plt.plot(range(epoch), loss_list) 55 | plt.savefig('Iris.png') 56 | plt.show() -------------------------------------------------------------------------------- /examples/regression/Boston.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/regression/Boston.png -------------------------------------------------------------------------------- /examples/regression/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/regression/__init__.py -------------------------------------------------------------------------------- /examples/regression/boston.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import warnings 3 | 4 | from sklearn.datasets import load_boston 5 | import matplotlib.pyplot as plt 6 | import matplotlib.cbook 7 | 8 | import autodiff as ad 9 | from autodiff.utils.model_utils import Dataset, DataLoader, train_procedure 10 | from autodiff.nn.optimizer import GradientDescent 11 | from autodiff.nn.criterion import MSE 12 | from autodiff.nn.layer import Module, Linear 13 | 14 | warnings.filterwarnings("ignore", category=matplotlib.cbook.mplDeprecation) 15 | warnings.filterwarnings("ignore", category=RuntimeWarning) 16 | 17 | 18 | class SimpleModel(Module): 19 | def __init__(self, num_features): 20 | super().__init__() 21 | self.linear1 = Linear(num_features, 5) 22 | self.linear2 = Linear(5, 1) 23 | 24 | def forward(self, x): 25 | x = self.linear1(x, bridge=False) 26 | x = self.linear2(x) 27 | return x 28 | 29 | 30 | X, Y = load_boston(return_X_y=True) 31 | Y = Y.reshape(-1, 1) 32 | data_size, num_features = X.shape 33 | 34 | model = SimpleModel(num_features) 35 | dataset = Dataset(X, Y) 36 | dataloader = DataLoader(dataset) 37 | 38 | lr = 1e-8 39 | opt = GradientDescent(lr, model.parameters()) 40 | loss_func = MSE() 41 | epoch = 200 42 | batch_size = 16 43 | loss_list = [] 44 | for i in train_procedure(epoch): 45 | x, y = dataloader.next_batch(batch_size) 46 | predicted_y = model(x) 47 | v, loss_grad = loss_func(y, predicted_y) 48 | model_grad = model.backward(loss_grad) 49 | opt.step(model_grad) 50 | model.zero_grad() 51 | loss_list.append(v) 52 | 53 | plt.plot(range(epoch), loss_list) 54 | plt.savefig('Boston.png') 55 | plt.show() -------------------------------------------------------------------------------- /examples/regression/regression.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/regression/regression.png -------------------------------------------------------------------------------- /examples/regression/regression.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import warnings 3 | 4 | import matplotlib.pyplot as plt 5 | import matplotlib.cbook 6 | import numpy as _np 7 | 8 | import autodiff as ad 9 | from autodiff.utils.model_utils import Dataset, DataLoader, train_procedure 10 | from autodiff.nn.optimizer import GradientDescent 11 | from autodiff.nn.criterion import MSE 12 | from autodiff.nn.layer import Module, Linear 13 | 14 | warnings.filterwarnings("ignore", category=matplotlib.cbook.mplDeprecation) 15 | warnings.filterwarnings("ignore", category=RuntimeWarning) 16 | 17 | 18 | class SimpleModel(Module): 19 | def __init__(self, num_features): 20 | super().__init__() 21 | self.linear = Linear(num_features, 1) 22 | 23 | def forward(self, x): 24 | return self.linear(x, bridge=False) 25 | 26 | data_size = 300 27 | X = 2 * _np.random.rand(data_size, 1) 28 | aug_X = _np.c_[_np.ones(data_size), X] 29 | 30 | # True parameters 31 | theta = _np.array([[5], [10]]) 32 | Y = aug_X @ theta + _np.random.randn(data_size, 1) 33 | print("training shape", X.shape, Y.shape) 34 | 35 | model = SimpleModel(2) 36 | dataset = Dataset(aug_X, Y) 37 | dataloader = DataLoader(dataset) 38 | 39 | lr = 0.001 40 | opt = GradientDescent(lr, model.parameters()) 41 | loss_func = MSE() 42 | epoch = 300 43 | for i in train_procedure(epoch): 44 | x, y = dataloader.next_batch(10) 45 | predicted_y = model(x) 46 | v, loss_grad = loss_func(y, predicted_y) 47 | model_grad = model.backward(loss_grad) 48 | opt.step(model_grad) 49 | model.zero_grad() 50 | 51 | predict = model(aug_X) 52 | 53 | plt.scatter( 54 | X, 55 | Y, 56 | ) 57 | plt.plot(X, predict, 'r') 58 | # plt.axis('off') 59 | plt.savefig('regression.png') 60 | plt.show() 61 | -------------------------------------------------------------------------------- /examples/simple/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/simple/__init__.py -------------------------------------------------------------------------------- /examples/simple/poly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/examples/simple/poly.png -------------------------------------------------------------------------------- /examples/simple/poly_test.py: -------------------------------------------------------------------------------- 1 | #pylint: disable=no-member 2 | import warnings 3 | 4 | import networkx as nx 5 | import matplotlib.pyplot as plt 6 | import matplotlib.cbook 7 | 8 | import autodiff as ad 9 | from autodiff import value_and_grad, value, grad 10 | 11 | warnings.filterwarnings("ignore", category=matplotlib.cbook.mplDeprecation) 12 | warnings.filterwarnings("ignore", category=RuntimeWarning) 13 | 14 | 15 | def tanh(x): 16 | return ad.divide( 17 | ad.subtract(ad.Constant(1), ad.exp(ad.negative(ad.Variable(x)))), 18 | ad.add(ad.Constant(1), ad.exp(ad.negative(ad.Variable(x))))) 19 | 20 | 21 | def test(x, y, z, w): 22 | return ad.multiply( 23 | ad.add(ad.multiply(ad.Variable(x), ad.Variable(y)), 24 | ad.maximum(ad.Variable(z), ad.Variable(w))), ad.Constant(2)) 25 | 26 | 27 | def test2(x, y, z): 28 | return ad.multiply(ad.add(ad.Variable(x), ad.Variable(y)), 29 | ad.maximum(ad.Variable(y), ad.Variable(z))) 30 | 31 | 32 | def test3(x): 33 | return ad.power(ad.Variable(x), ad.Constant(3)) 34 | 35 | 36 | def test4(x): 37 | return ad.multiply(ad.power(ad.Variable(x), ad.Constant(2)), 38 | ad.Constant(1 / 2)) 39 | 40 | 41 | def power_demo(): 42 | x_list = ad.linspace(-7, 7, 200) 43 | y_list, dy_list = value_and_grad(test3, id(x_list))(x=x_list) 44 | 45 | plt.plot(x_list, y_list, x_list, dy_list) 46 | # plt.axis('off') 47 | plt.savefig('poly.png') 48 | plt.show() 49 | 50 | 51 | if __name__ == "__main__": 52 | power_demo() -------------------------------------------------------------------------------- /img/actions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/img/actions.png -------------------------------------------------------------------------------- /img/backward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/img/backward.png -------------------------------------------------------------------------------- /img/forward.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/titaneric/AutoDiff-from-scratch/993d70ccb83122889a04f93d3e884f93f7e1fb84/img/forward.png -------------------------------------------------------------------------------- /proposal.md: -------------------------------------------------------------------------------- 1 | # Basic neural network library supported auto-differentiation 2 | 3 | ## Motivation 4 | 5 | Most deep learning courses aim to teach math behind the network, architecture and their applications. 6 | 7 | However, seldom course talk about how to implement and design the deep learning library and start everything from scratch. 8 | 9 | Wish to implement this kind of library and learn how and why the priors (TensorFlow and PyTorch etc.) design their work during the development of final project. 10 | 11 | ## Target 12 | 13 | Based on [Autograd](https://github.com/HIPS/autograd) project, build a similar library that user simply define the function, and this lib can automatically calculate this differentiation form of given function. 14 | 15 | Build the computational graph when function is called, calculate the backward propogation with respect to variable or placeholder (in tensorflow term). 16 | 17 | Provide a benchmark compared to TensorFlow and PyTorch. 18 | 19 | If time allowed, provide a simple multi-layer perceptron (neural network) interface, criterion, optimizer, datasets and dataloader like the priors. 20 | 21 | ## Project Link 22 | 23 | [AutoDiff-from-scratch](https://github.com/titaneric/AutoDiff-from-scratch) 24 | 25 | ## TODO 26 | 27 | - benchmark 28 | - documents 29 | 30 | ## Reference 31 | 32 | ### Source code 33 | 34 | - [Autograd](https://github.com/HIPS/autograd) 35 | - [Autodidact](https://github.com/mattjj/autodidact) 36 | - [PyTorch](https://github.com/pytorch/pytorch) 37 | - [TensorFlow](https://github.com/tensorflow/tensorflow) 38 | - [Caffe](https://github.com/BVLC/caffe) 39 | - [Caffe2](https://github.com/pytorch/pytorch/tree/master/caffe2) 40 | 41 | ### Lecture 42 | 43 | - [Backpropagation, Toronto CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec06.pdf) 44 | - [Automatic Differentiation, Toronto CSC321](http://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/slides/lec10.pdf) 45 | - [Backpropagation, Stanford CS224N](https://www.youtube.com/watch?v=yLYHDSv-288&list=PLoROMvodv4rOhcuXMZkNm7j3fVwBBY42z&index=5&t=2177s) 46 | - [Introduction to Neural Networks, Stanford CS231N](https://www.youtube.com/watch?v=d14TUNcbn1k&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=4) 47 | - [Backpropagation: Find Partial Derivatives, MIT 18.065](https://www.youtube.com/watch?v=lZrIPRnoGQQ&list=PLUl4u3cNGP63oMNUHXqIUcrkS2PivhN3k&index=30&t=0s) 48 | 49 | ### Documents 50 | 51 | - [Jax official](https://jax.readthedocs.io/en/latest/index.html) 52 | - [Phd Thesis by Dougal Maclaurin (one of Autograd author)](https://dougalmaclaurin.com/phd-thesis.pdf) 53 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | networkx==2.3 2 | numpy==1.17.2 3 | absl-py==0.8.1 4 | --------------------------------------------------------------------------------