├── readme_pics ├── jax_logo.png ├── going_deeper.jpg ├── tpu_jit_speed.PNG └── api_onion_structure.jpg ├── .gitignore ├── LICENCE ├── README.md ├── Tutorial_3_JAX_Neural_Network_from_Scratch_Colab.ipynb └── Tutorial_4_Flax_Zero2Hero_Colab.ipynb /readme_pics/jax_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gordicaleksa/get-started-with-JAX/HEAD/readme_pics/jax_logo.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # PyCharm IDE 2 | .idea 3 | __pycache__ 4 | 5 | # Jupyter notebook checkpoints 6 | .ipynb_checkpoints 7 | 8 | -------------------------------------------------------------------------------- /readme_pics/going_deeper.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gordicaleksa/get-started-with-JAX/HEAD/readme_pics/going_deeper.jpg -------------------------------------------------------------------------------- /readme_pics/tpu_jit_speed.PNG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gordicaleksa/get-started-with-JAX/HEAD/readme_pics/tpu_jit_speed.PNG -------------------------------------------------------------------------------- /readme_pics/api_onion_structure.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gordicaleksa/get-started-with-JAX/HEAD/readme_pics/api_onion_structure.jpg -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Aleksa Gordić 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Get started with JAX! :computer: :zap: 2 | 3 | The goal of this repo is to make it easier to get started with [JAX](https://github.com/google/jax), [Flax](https://github.com/google/flax), and [Haiku](https://github.com/deepmind/dm-haiku)! 4 | 5 | `JAX` ecosystem is becoming an increasingly popular alternative to `PyTorch` and `TensorFlow`. :sunglasses: 6 | 7 |
8 |
9 | 10 |

11 | 12 |

13 | 14 |
15 |
16 | 17 | *Note: I'm only going to recommend content that I've personally analyzed and found useful here. 18 | If you want a comprehensive list check out the [awesome-jax repo](https://github.com/n2cholas/awesome-jax).* 19 | 20 | ## Table of Contents 21 | * [Machine Learning with JAX](#my-machine-learning-with-jax-tutorials) 22 | + [Tutorial #1: From Zero to Hero](#tutorial-1-from-zero-to-hero) 23 | + [Tutorial #2: From Hero to Hero Pro+](#tutorial-2-from-hero-to-heropro) 24 | + [Tutorial #3: Coding a Neural Network from Scratch in Pure JAX](#tutorial-3-building-a-neural-network-from-scratch) 25 | + [Tutorial #4: Flax From Zero to Hero](#tutorial-4-machine-learning-with-flax---from-zero-to-hero) 26 | + [Tutorial #5: Haiku From Zero to Hero (coming soon)](#tutorial-5-coming-up-machine-learning-with-haiku---from-zero-to-hero) 27 | * [Other useful JAX resources](#other-useful-content) 28 | 29 | ## My Machine Learning with JAX Tutorials 30 | 31 | *Tip on how to use notebooks: just open the notebook directly in Google Colab 32 | (you'll see a button on top of the Jupyter file which will direct you to Colab). 33 | This way you can avoid having to setup the Python env! (This was especially convenient for me since I'm on Windows which is still not supported)* 34 | 35 | ### Tutorial #1: From Zero to Hero 36 | 37 | In this video, we start from the basics and then gradually dig into the nitty-gritty details 38 | of `jit`, `grad`, `vmap`, and various other idiosyncrasies of JAX. 39 | 40 | [YouTube Video (Tutorial #1)](https://youtu.be/SstuvS-tVc0)
41 | [Accompanying Jupyter Notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_1_JAX_Zero2Hero_Colab.ipynb)
42 | 43 |

44 | 46 |

47 | 48 | ### Tutorial #2: From Hero to HeroPro+ 49 | 50 | In this video, we learn all additional components needed to train ML models (such as NNs) on multiple machines! 51 | We'll train a simple MLP model and we'll even train an ML model on 8 TPU cores! 52 | 53 | [YouTube Video (Tutorial #2)](https://www.youtube.com/watch?v=CQQaifxuFcs)
54 | [Accompanying Jupyter Notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_2_JAX_HeroPro%2B_Colab.ipynb)
55 | 56 |

57 | 59 |

60 | 61 | ### Tutorial #3: Building a Neural Network from Scratch 62 | 63 | Watch me code a Neural Network from scratch! :partying_face: In this 3rd video of the JAX tutorials series. 64 | 65 | In this video, I build an [MLP](https://en.wikipedia.org/wiki/Multilayer_perceptron) and train it as a classifier on MNIST 66 | using PyTorch's data loader (although it's trivial to use a more complex dataset) - all this in "pure" JAX (no Flax/Haiku/Optax). 67 | 68 | I then do an additional analysis: 69 | * Visualize MLP's learned weights 70 | * Visualize embeddings of a batch of images using t-SNE 71 | * Finally, I analyze whether we have too many dead ReLU neurons in our network 72 | 73 | [YouTube Video (Tutorial #3)](https://www.youtube.com/watch?v=6_PqUPxRmjY)
74 | [Accompanying Jupyter Notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_3_JAX_Neural_Network_from_Scratch_Colab.ipynb) (Note: I'll soon refactor it but I'll link the original)
75 | 76 |

77 | 79 |

80 | 81 | --- 82 | 83 | ### Tutorial #4: Machine Learning with Flax - From Zero to Hero 84 | 85 | In this video, I cover everything you need to know to get started with [Flax](https://github.com/google/flax)! 86 | 87 | We cover `init`, `apply`, `TrainState`, etc. and other idiosyncrasies like the usage of `mutable` and `rngs` keywords. 88 | 89 | [YouTube Video (Tutorial #4)](https://www.youtube.com/watch?v=5eUSmJvK8WA)
90 | [Accompanying Jupyter Notebook](https://github.com/gordicaleksa/get-started-with-JAX/blob/main/Tutorial_4_Flax_Zero2Hero_Colab.ipynb)
91 | 92 |

93 | 95 |

96 | 97 | --- 98 | 99 | ### Tutorial #5 (coming up): Machine Learning with Haiku - From Zero to Hero 100 | 101 | todo 102 | 103 | ## Other useful content 104 | 105 | Aside from the [official docs](https://jax.readthedocs.io/) here are some resources that helped me. 106 | 107 | ### Videos 108 | 109 | * [Introduction to JAX](https://www.youtube.com/watch?v=0mVmRHMaOJ4&ab_channel=GoogleCloudTech) (gives a very high-level overview) 110 | * [JAX: Accelerated Machine Learning Research | SciPy 2020 | VanderPlas](https://www.youtube.com/watch?v=z-WSrQDXkuM&ab_channel=Enthought) (many more details) 111 | * [NeurIPS 2020: JAX Ecosystem Meetup](https://www.youtube.com/watch?v=iDxJxIyzSiM&t=1s&ab_channel=DeepMind) (DeepMind team about the ecosystem of libs around JAX) 112 | * [Introduction to JAX for Machine Learning and More](https://www.youtube.com/watch?v=QkmKfzxbCLQ&ab_channel=UWaterlooDataScience) (nice, hands-on workshop) 113 | * [Day 1 Talks: JAX, Flax & Transformers | HuggingFace](https://www.youtube.com/watch?v=fuAyUQcVzTY&ab_channel=HuggingFace) (all 4 talks are good) 114 | * [Day 2 Talks: JAX, Flax & Transformers | HuggingFace](https://www.youtube.com/watch?v=__eG63ZP_5g&ab_channel=HuggingFace) (only the first 2 talks are relevant) 115 | 116 | ### Blogs 117 | 118 | * [Using JAX to accelerate our research | DeepMind](https://deepmind.com/blog/article/using-jax-to-accelerate-our-research) (similar info as the NeuroIPS 2020 video) 119 | * [You don't know JAX | Colin Raffel](https://colinraffel.com/blog/you-don-t-know-jax.html) 120 | 121 | ## Acknowledgements 122 | 123 | * The notebooks were heavily inspired by the official [JAX](https://jax.readthedocs.io/), [Flax](https://flax.readthedocs.io/en/latest/), and [Haiku](https://dm-haiku.readthedocs.io/en/latest/) docs. 124 | 125 | ## Citation 126 | 127 | If you find this content useful, please cite the following: 128 | 129 | ``` 130 | @misc{Gordic2021GetStartedWithJAX, 131 | author = {Gordić, Aleksa}, 132 | title = {Get started with JAX}, 133 | year = {2021}, 134 | publisher = {GitHub}, 135 | journal = {GitHub repository}, 136 | howpublished = {\url{https://github.com/gordicaleksa/get-started-with-JAX}}, 137 | } 138 | ``` 139 | 140 | ## Connect With Me 141 | 142 | If you'd love to have some more AI-related content in your life :nerd_face:, consider: 143 | * Subscribing to my YouTube channel [The AI Epiphany](https://www.youtube.com/c/TheAiEpiphany) :bell: 144 | * Follow me on [LinkedIn](https://www.linkedin.com/in/aleksagordic/) and [Twitter](https://twitter.com/gordic_aleksa) :bulb: 145 | * Follow me on [Medium](https://gordicaleksa.medium.com/) :books: :heart: 146 | * Join the [Discord](https://discord.gg/peBrCpheKE) community! :family: 147 | 148 | ## Licence 149 | 150 | [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/gordicaleksa/get-started-with-JAX/blob/master/LICENCE) -------------------------------------------------------------------------------- /Tutorial_3_JAX_Neural_Network_from_Scratch_Colab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "Tutorial 3: JAX - Building a Neural Network from Scratch.ipynb", 7 | "provenance": [], 8 | "collapsed_sections": [], 9 | "authorship_tag": "ABX9TyMpTL6XC+tcxSqZ2FePUhlZ", 10 | "include_colab_link": true 11 | }, 12 | "kernelspec": { 13 | "name": "python3", 14 | "display_name": "Python 3" 15 | }, 16 | "language_info": { 17 | "name": "python" 18 | } 19 | }, 20 | "cells": [ 21 | { 22 | "cell_type": "markdown", 23 | "metadata": { 24 | "id": "view-in-github", 25 | "colab_type": "text" 26 | }, 27 | "source": [ 28 | " $\"Open$ " 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": { 34 | "id": "XZuyP-M3KPUR" 35 | }, 36 | "source": [ 37 | "# MLP training on MNIST" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "metadata": { 43 | "id": "8-SzJ0NTKRP1" 44 | }, 45 | "source": [ 46 | "import numpy as np\n", 47 | "import jax.numpy as jnp\n", 48 | "from jax.scipy.special import logsumexp\n", 49 | "import jax\n", 50 | "from jax import jit, vmap, pmap, grad, value_and_grad\n", 51 | "\n", 52 | "from torchvision.datasets import MNIST\n", 53 | "from torch.utils.data import DataLoader" 54 | ], 55 | "execution_count": 1, 56 | "outputs": [] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "metadata": { 61 | "colab": { 62 | "base_uri": "https://localhost:8080/" 63 | }, 64 | "id": "G4NrxSVjKt8f", 65 | "outputId": "6bb8bef6-3098-4fd5-8ffe-62f4b0b1aa79" 66 | }, 67 | "source": [ 68 | "seed = 0\n", 69 | "mnist_img_size = (28, 28)\n", 70 | "\n", 71 | "def init_MLP(layer_widths, parent_key, scale=0.01):\n", 72 | "\n", 73 | " params = []\n", 74 | " keys = jax.random.split(parent_key, num=len(layer_widths)-1)\n", 75 | "\n", 76 | " for in_width, out_width, key in zip(layer_widths[:-1], layer_widths[1:], keys):\n", 77 | " weight_key, bias_key = jax.random.split(key)\n", 78 | " params.append([\n", 79 | " scale*jax.random.normal(weight_key, shape=(out_width, in_width)),\n", 80 | " scale*jax.random.normal(bias_key, shape=(out_width,))\n", 81 | " ]\n", 82 | " )\n", 83 | "\n", 84 | " return params\n", 85 | "\n", 86 | "# test\n", 87 | "key = jax.random.PRNGKey(seed)\n", 88 | "MLP_params = init_MLP([784, 512, 256, 10], key)\n", 89 | "print(jax.tree_map(lambda x: x.shape, MLP_params))" 90 | ], 91 | "execution_count": 4, 92 | "outputs": [ 93 | { 94 | "output_type": "stream", 95 | "name": "stdout", 96 | "text": [ 97 | "[[(512, 784), (512,)], [(256, 512), (256,)], [(10, 256), (10,)]]\n" 98 | ] 99 | } 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "metadata": { 105 | "colab": { 106 | "base_uri": "https://localhost:8080/" 107 | }, 108 | "id": "U_z7eLxINv9x", 109 | "outputId": "e9909f9f-6778-4977-91f1-f5b14dd9ecd4" 110 | }, 111 | "source": [ 112 | "def MLP_predict(params, x):\n", 113 | " hidden_layers = params[:-1]\n", 114 | "\n", 115 | " activation = x\n", 116 | " for w, b in hidden_layers:\n", 117 | " activation = jax.nn.relu(jnp.dot(w, activation) + b)\n", 118 | "\n", 119 | " w_last, b_last = params[-1]\n", 120 | " logits = jnp.dot(w_last, activation) + b_last\n", 121 | "\n", 122 | " # log(exp(o1)) - log(sum(exp(o1), exp(o2), ..., exp(o10)))\n", 123 | " # log( exp(o1) / sum(...) )\n", 124 | " return logits - logsumexp(logits)\n", 125 | "\n", 126 | "# tests\n", 127 | "\n", 128 | "# test single example\n", 129 | "\n", 130 | "dummy_img_flat = np.random.randn(np.prod(mnist_img_size))\n", 131 | "print(dummy_img_flat.shape)\n", 132 | "\n", 133 | "prediction = MLP_predict(MLP_params, dummy_img_flat)\n", 134 | "print(prediction.shape)\n", 135 | "\n", 136 | "# test batched function\n", 137 | "batched_MLP_predict = vmap(MLP_predict, in_axes=(None, 0))\n", 138 | "\n", 139 | "dummy_imgs_flat = np.random.randn(16, np.prod(mnist_img_size))\n", 140 | "print(dummy_imgs_flat.shape)\n", 141 | "predictions = batched_MLP_predict(MLP_params, dummy_imgs_flat)\n", 142 | "print(predictions.shape)" 143 | ], 144 | "execution_count": 5, 145 | "outputs": [ 146 | { 147 | "output_type": "stream", 148 | "name": "stdout", 149 | "text": [ 150 | "(784,)\n", 151 | "(10,)\n", 152 | "(16, 784)\n", 153 | "(16, 10)\n" 154 | ] 155 | } 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "metadata": { 161 | "colab": { 162 | "base_uri": "https://localhost:8080/" 163 | }, 164 | "id": "5pPM1dZ4QyYe", 165 | "outputId": "3317666b-e167-46b7-8cf4-b8592adc065a" 166 | }, 167 | "source": [ 168 | "def custom_transform(x):\n", 169 | " return np.ravel(np.array(x, dtype=np.float32))\n", 170 | "\n", 171 | "def custom_collate_fn(batch):\n", 172 | " transposed_data = list(zip(*batch))\n", 173 | "\n", 174 | " labels = np.array(transposed_data[1])\n", 175 | " imgs = np.stack(transposed_data[0])\n", 176 | "\n", 177 | " return imgs, labels\n", 178 | "\n", 179 | "batch_size = 128\n", 180 | "train_dataset = MNIST(root='train_mnist', train=True, download=True, transform=custom_transform)\n", 181 | "test_dataset = MNIST(root='test_mnist', train=False, download=True, transform=custom_transform)\n", 182 | "\n", 183 | "train_loader = DataLoader(train_dataset, batch_size, shuffle=True, collate_fn=custom_collate_fn, drop_last=True)\n", 184 | "test_loader = DataLoader(test_dataset, batch_size, shuffle=False, collate_fn=custom_collate_fn, drop_last=True)\n", 185 | "\n", 186 | "# test\n", 187 | "batch_data = next(iter(train_loader))\n", 188 | "imgs = batch_data[0]\n", 189 | "lbls = batch_data[1]\n", 190 | "print(imgs.shape, imgs[0].dtype, lbls.shape, lbls[0].dtype)\n", 191 | "\n", 192 | "# optimization - loading the whole dataset into memory\n", 193 | "train_images = jnp.array(train_dataset.data).reshape(len(train_dataset), -1)\n", 194 | "train_lbls = jnp.array(train_dataset.targets)\n", 195 | "\n", 196 | "test_images = jnp.array(test_dataset.data).reshape(len(test_dataset), -1)\n", 197 | "test_lbls = jnp.array(test_dataset.targets)" 198 | ], 199 | "execution_count": null, 200 | "outputs": [ 201 | { 202 | "output_type": "stream", 203 | "name": "stdout", 204 | "text": [ 205 | "(128, 784) float32 (128,) int64\n" 206 | ] 207 | } 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "metadata": { 213 | "id": "YQEYcSNzVeim" 214 | }, 215 | "source": [ 216 | "num_epochs = 5\n", 217 | "\n", 218 | "def loss_fn(params, imgs, gt_lbls):\n", 219 | " predictions = batched_MLP_predict(params, imgs)\n", 220 | "\n", 221 | " return -jnp.mean(predictions * gt_lbls)\n", 222 | "\n", 223 | "def accuracy(params, dataset_imgs, dataset_lbls):\n", 224 | " pred_classes = jnp.argmax(batched_MLP_predict(params, dataset_imgs), axis=1)\n", 225 | " return jnp.mean(dataset_lbls == pred_classes)\n", 226 | "\n", 227 | "@jit\n", 228 | "def update(params, imgs, gt_lbls, lr=0.01):\n", 229 | " loss, grads = value_and_grad(loss_fn)(params, imgs, gt_lbls)\n", 230 | "\n", 231 | " return loss, jax.tree_multimap(lambda p, g: p - lr*g, params, grads)\n", 232 | "\n", 233 | "# Create a MLP\n", 234 | "MLP_params = init_MLP([np.prod(mnist_img_size), 512, 256, len(MNIST.classes)], key)\n", 235 | "\n", 236 | "for epoch in range(num_epochs):\n", 237 | "\n", 238 | " for cnt, (imgs, lbls) in enumerate(train_loader):\n", 239 | "\n", 240 | " gt_labels = jax.nn.one_hot(lbls, len(MNIST.classes))\n", 241 | " \n", 242 | " loss, MLP_params = update(MLP_params, imgs, gt_labels)\n", 243 | " \n", 244 | " if cnt % 50 == 0:\n", 245 | " print(loss)\n", 246 | "\n", 247 | " print(f'Epoch {epoch}, train acc = {accuracy(MLP_params, train_images, train_lbls)} test acc = {accuracy(MLP_params, test_images, test_lbls)}')\n" 248 | ], 249 | "execution_count": null, 250 | "outputs": [] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "metadata": { 255 | "colab": { 256 | "base_uri": "https://localhost:8080/", 257 | "height": 316 258 | }, 259 | "id": "YmdBRBvU1wuA", 260 | "outputId": "efcfa75e-d0bb-4f16-9fb2-e85e82a53bcf" 261 | }, 262 | "source": [ 263 | "imgs, lbls = next(iter(test_loader))\n", 264 | "img = imgs[0].reshape(mnist_img_size)\n", 265 | "gt_lbl = lbls[0]\n", 266 | "print(img.shape)\n", 267 | "\n", 268 | "import matplotlib.pyplot as plt\n", 269 | "\n", 270 | "pred = jnp.argmax(MLP_predict(MLP_params, np.ravel(img)))\n", 271 | "print('pred', pred)\n", 272 | "print('gt', gt_lbl)\n", 273 | "\n", 274 | "plt.imshow(img); plt.show()" 275 | ], 276 | "execution_count": null, 277 | "outputs": [ 278 | { 279 | "output_type": "stream", 280 | "name": "stdout", 281 | "text": [ 282 | "(28, 28)\n", 283 | "pred 7\n", 284 | "gt 7\n" 285 | ] 286 | }, 287 | { 288 | "output_type": "display_data", 289 | "data": { 290 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAANiklEQVR4nO3df4wc9XnH8c8n/kV8QGtDcF3j4ISQqE4aSHWBRNDKESUFImSiJBRLtVyJ5lALElRRW0QVBalVSlEIok0aySluHESgaQBhJTSNa6W1UKljg4yxgdaEmsau8QFOaxPAP/DTP24cHXD7vWNndmft5/2SVrs7z87Oo/F9PLMzO/t1RAjA8e9tbTcAoD8IO5AEYQeSIOxAEoQdSGJ6Pxc207PiBA31c5FAKq/qZzoYBzxRrVbYbV8s6XZJ0yT9bUTcXHr9CRrSeb6wziIBFGyIdR1rXe/G254m6auSLpG0WNIy24u7fT8AvVXnM/u5kp6OiGci4qCkeyQtbaYtAE2rE/YFkn4y7vnOatrr2B6xvcn2pkM6UGNxAOro+dH4iFgZEcMRMTxDs3q9OAAd1An7LkkLxz0/vZoGYADVCftGSWfZfpftmZKulLSmmbYANK3rU28Rcdj2tZL+SWOn3lZFxLbGOgPQqFrn2SPiQUkPNtQLgB7i67JAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJGoN2Wx7h6T9kl6TdDgihptoCkDzaoW98rGIeKGB9wHQQ+zGA0nUDXtI+oHtR2yPTPQC2yO2N9nedEgHai4OQLfq7sZfEBG7bJ8maa3tpyJi/fgXRMRKSSsl6WTPjZrLA9ClWlv2iNhV3Y9Kul/SuU00BaB5XYfd9pDtk44+lvRxSVubagxAs+rsxs+TdL/to+/zrYj4fiNdAWhc12GPiGcknd1gLwB6iFNvQBKEHUiCsANJEHYgCcIOJNHEhTApvPjZj3asvXP508V5nxqdV6wfPDCjWF9wd7k+e+dLHWtHNj9RnBd5sGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4zz5Ff/xH3+pY+9TQT8szn1lz4UvK5R2HX+5Yu/35j9Vc+LHrR6NndKwN3foLxXmnr3uk6XZax5YdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRP8GaTnZc+M8X9i35TXpZ58+r2PthQ+W/8+c82R5Hf/0V1ysz/zg/xbrt3zgvo61i97+SnHe7718YrH+idmdr5Wv65U4WKxvODBUrC854VDXy37P964u1t87srHr927ThlinfbF3wj8otuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATXs0/R0Hc2FGr13vvkerPrr39pScfan5+/qLzsfy3/5v0tS97TRUdTM/2VI8X60Jbdxfop6+8t1n91Zuff25+9o/xb/MejSbfstlfZHrW9ddy0ubbX2t5e3c/pbZsA6prKbvw3JF38hmk3SFoXEWdJWlc9BzDAJg17RKyXtPcNk5dKWl09Xi3p8ob7AtCwbj+zz4uIox+onpPUcTAz2yOSRiTpBM3ucnEA6qp9ND7GrqTpeKVHRKyMiOGIGJ6hWXUXB6BL3YZ9j+35klTdjzbXEoBe6DbsayStqB6vkPRAM+0A6JVJP7Pbvltjv1x+qu2dkr4g6WZJ37Z9laRnJV3RyyZRdvi5PR1rQ/d2rknSa5O899B3Xuyio2bs+b2PFuvvn1n+8/3S3vd1rC36u2eK8x4uVo9Nk4Y9IpZ1KB2bv0IBJMXXZYEkCDuQBGEHkiDsQBKEHUiCS1zRmulnLCzWv3LjV4r1GZ5WrP/D7b/ZsXbK7oeL8x6P2LIDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKcZ0drnvrDBcX6h2eVh7LedrA8HPXcJ15+yz0dz9iyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASnGdHTx34xIc71h799G2TzF0eQej3r7uuWH/7v/1okvfPhS07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBeXb01H9f0nl7cqLL59GX/ddFxfrs7z9WrEexms+kW3bbq2yP2t46btpNtnfZ3lzdLu1tmwDqmspu/DckXTzB9Nsi4pzq9mCzbQFo2qRhj4j1kvb2oRcAPVTnAN21trdUu/lzOr3I9ojtTbY3HdKBGosDUEe3Yf+apDMlnSNpt6RbO70wIlZGxHBEDM+Y5MIGAL3TVdgjYk9EvBYRRyR9XdK5zbYFoGldhd32/HFPPylpa6fXAhgMk55nt323pCWSTrW9U9IXJC2xfY7GTmXukHR1D3vEAHvbSScV68t//aGOtX1HXi3OO/rFdxfrsw5sLNbxepOGPSKWTTD5jh70AqCH+LoskARhB5Ig7EAShB1IgrADSXCJK2rZftP7i/Xvnvo3HWtLt3+qOO+sBzm11iS27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOfZUfR/v/ORYn3Lb/9Vsf7jw4c61l76y9OL887S7mIdbw1bdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgvPsyU1f8MvF+vWf//tifZbLf0JXPra8Y+0d/8j16v3Elh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuA8+3HO08v/xGd/d2ex/pkTXyzW79p/WrE+7/OdtydHinOiaZNu2W0vtP1D20/Y3mb7umr6XNtrbW+v7uf0vl0A3ZrKbvxhSZ+LiMWSPiLpGtuLJd0gaV1EnCVpXfUcwICaNOwRsTsiHq0e75f0pKQFkpZKWl29bLWky3vVJID63tJndtuLJH1I0gZJ8yLi6I+EPSdpXod5RiSNSNIJmt1tnwBqmvLReNsnSrpX0vURsW98LSJCUkw0X0SsjIjhiBieoVm1mgXQvSmF3fYMjQX9roi4r5q8x/b8qj5f0mhvWgTQhEl3421b0h2SnoyIL48rrZG0QtLN1f0DPekQ9Zz9vmL5z067s9bbf/WLnynWf/Gxh2u9P5ozlc/s50taLulx25uraTdqLOTftn2VpGclXdGbFgE0YdKwR8RDktyhfGGz7QDoFb4uCyRB2IEkCDuQBGEHkiDsQBJc4nocmLb4vR1rI/fU+/rD4lXXFOuL7vz3Wu+P/mHLDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJcJ79OPDUH3T+Yd/LZu/rWJuK0//lYPkFMeEPFGEAsWUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQ4z34MePWyc4v1dZfdWqgy5BbGsGUHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSSmMj77QknflDRPUkhaGRG3275J0mclPV+99MaIeLBXjWb2P+dPK9bfOb37c+l37T+tWJ+xr3w9O1ezHzum8qWaw5I+FxGP2j5J0iO211a12yLiS71rD0BTpjI++25Ju6vH+20/KWlBrxsD0Ky39Jnd9iJJH5K0oZp0re0ttlfZnvC3kWyP2N5ke9MhHajVLIDuTTnstk+UdK+k6yNin6SvSTpT0jka2/JP+AXtiFgZEcMRMTxDsxpoGUA3phR22zM0FvS7IuI+SYqIPRHxWkQckfR1SeWrNQC0atKw27akOyQ9GRFfHjd9/riXfVLS1ubbA9CUqRyNP1/SckmP295cTbtR0jLb52js7MsOSVf3pEPU8hcvLi7WH/6tRcV67H68wW7QpqkcjX9IkicocU4dOIbwDTogCcIOJEHYgSQIO5AEYQeSIOxAEo4+Drl7sufGeb6wb8sDstkQ67Qv9k50qpwtO5AFYQeSIOxAEoQdSIKwA0kQdiAJwg4k0dfz7Lafl/TsuEmnSnqhbw28NYPa26D2JdFbt5rs7YyIeMdEhb6G/U0LtzdFxHBrDRQMam+D2pdEb93qV2/sxgNJEHYgibbDvrLl5ZcMam+D2pdEb93qS2+tfmYH0D9tb9kB9AlhB5JoJey2L7b9H7aftn1DGz10YnuH7cdtb7a9qeVeVtketb113LS5ttfa3l7dTzjGXku93WR7V7XuNtu+tKXeFtr+oe0nbG+zfV01vdV1V+irL+ut75/ZbU+T9J+SLpK0U9JGScsi4om+NtKB7R2ShiOi9S9g2P4NSS9J+mZEfKCadoukvRFxc/Uf5ZyI+JMB6e0mSS+1PYx3NVrR/PHDjEu6XNLvqsV1V+jrCvVhvbWxZT9X0tMR8UxEHJR0j6SlLfQx8CJivaS9b5i8VNLq6vFqjf2x9F2H3gZCROyOiEerx/slHR1mvNV1V+irL9oI+wJJPxn3fKcGa7z3kPQD24/YHmm7mQnMi4jd1ePnJM1rs5kJTDqMdz+9YZjxgVl33Qx/XhcH6N7sgoj4NUmXSLqm2l0dSDH2GWyQzp1OaRjvfplgmPGfa3PddTv8eV1thH2XpIXjnp9eTRsIEbGruh+VdL8GbyjqPUdH0K3uR1vu5+cGaRjviYYZ1wCsuzaHP28j7BslnWX7XbZnSrpS0poW+ngT20PVgRPZHpL0cQ3eUNRrJK2oHq+Q9ECLvbzOoAzj3WmYcbW87lof/jwi+n6TdKnGjsj/WNKfttFDh77eLemx6rat7d4k3a2x3bpDGju2cZWkUyStk7Rd0j9LmjtAvd0p6XFJWzQWrPkt9XaBxnbRt0jaXN0ubXvdFfrqy3rj67JAEhygA5Ig7EAShB1IgrADSRB2IAnCDiRB2IEk/h9BCfQTVPflJQAAAABJRU5ErkJggg==\n", 291 | "text/plain": [ 292 | "