├── README.md ├── fun_to_jax_ir.png ├── jax-utils ├── README.md ├── jax_utils │ ├── set_conda_env_vars_for_jax_gpu.py │ └── test_jax_installation.py └── setup.py ├── tutorial.ipynb └── tutorial_with_solutions.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # JAX tutorial 2 | 3 | 4 | Open in Colab 5 | 6 | 7 | The tutorial notebook can be opened and run interactively on [Google Colab](https://colab.research.google.com/) from the badge above. The corresponding notebook with solutions to the exercises can also be run on Google Colab from [this link](https://colab.research.google.com/github/pierreglaser/jax-tutorial/blob/main/tutorial_with_solutions.ipynb). Alternatively follow the instructions below to set up a local Python environment to run the notebook from. 8 | 9 | ## Installation Instructions 10 | 11 | 12 | ### Requirements: 13 | 14 | - A UNIX-Compliant distribution 15 | - A `conda`-based package manager 16 | - (Optional) for GPU support: CUDA driver libraries `>= 11.6`. 17 | 18 | ### Jax Installation (CPU) 19 | 20 | To use a CPU-only powered jax, create a `conda` virtual environment containing `python` and `jax`: 21 | ```bash 22 | conda create -n jax-tutorial python=3.9 && conda activate jax-tutorial 23 | conda install -c conda-forge numpy scipy jax flax numpyro 24 | ``` 25 | 26 | ### Jax Installation (GPU) 27 | 28 | In all cases, you will need to install a GPU-able version of jax. 29 | 30 | ```bash 31 | # Installs the wheel compatible with CUDA 11 and cuDNN 8.2 or newer. 32 | # Note: wheels only available on linux. 33 | pip install --upgrade "jax[cuda]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html 34 | ``` 35 | 36 | A fully-functionning version of jax (i.e which includes working working (sparse) linear algebra and deep network primitives) on GPU requires `cudatoolkit` libraries, `cudnn`, as well as `nvcc` (a CUDA compiler). 37 | In most cases, these libraries should already be present in your system. Alas for research staff working on compute clusters with only user privileges, they often reside in a non-standard locations. 38 | 39 | #### If CUDA-related utilities are available in standard locations 40 | You should be all set. Congrats for living such a luxurious life. 41 | 42 | #### If using properly configured modulefiles (case of the Sainsbury Wellcome Center Compute Cluster). 43 | Some compute environments (like the SWC compute cluster) use modulefiles to integrate specific libraries and executables with your current shell session, removing the need for environment variables plumbing when the said libraries/executables are present in non-standard locations. 44 | 45 | If you're a SWC staff researcher working on the SWC compute cluster, you can load the cuda/11.6 modulefile by executing: 46 | 47 | ```bash 48 | module load cuda/11.6 49 | ``` 50 | 51 | and *voila*. 52 | 53 | 54 | #### If CUDA-related utilities are available in a non-standard locations 55 | If none of the two cases above apply, for instance in the case of user (conda) installed CUDA-libraries, or incomplete module files, you will need to point to `jax` yourself the place where such libraries can be found. 56 | To do so, locate the root directory containing the cuda utilities, say, `/path/to/cuda`, and run: 57 | 58 | ```bash 59 | export XLA_FLAGS=--xla_gpu_cuda_data_dir=/path/to/cuda/dir; 60 | export LD_LIBRARY_PATH=/path/to/cuda/dir/lib64; # YMMV: might be lib and not lib64 61 | ``` 62 | 63 | 64 | ### Testing your installation 65 | 66 | To test that your jax environment is properly setup, a convenience script is provided as part of this tutorial. From the root directory of this repository run: 67 | ```bash 68 | python -m pip install ./jax-utils 69 | # if on CPU: 70 | python -m jax_utils.test_jax_installation 71 | # if on GPU: 72 | python -m jax_utils.test_jax_installation --gpu 73 | ``` 74 | 75 | This script will test a subset of jax features relying on different libraries and will loudly error out if some piece of software is missing. 76 | 77 | 78 | ### Installing jupyter-related utilities 79 | 80 | To execute jupyter notebooks that will use the previously setup `jax-tutorial` environment as the execution environment, either install `jupyterlab` directly in this environment: 81 | 82 | ```bash 83 | conda install jupyterlab 84 | ``` 85 | 86 | or install `ipykernel` and register your kernel to your external jupyterlab installation: 87 | 88 | ```bash 89 | conda install ipykernel 90 | python -m ipykernel install --prefix=path/to/miniforge/installation/envs/ --name="jax-tutorial"; 91 | conda deactivate && conda activate 92 | ``` 93 | 94 | If you're using a GPU-powered jax, jupyterlab, and you're feeling fancy, install the jupyterlab extension `jupyterlab_nvdashboard`, which will dynamically display 95 | valuable metrics such as GPU memory usage or GPU volatle utilisation: 96 | 97 | ```bash 98 | pip install jupyterlab_nvdashboard 99 | ``` 100 | 101 | At this point, you should bee all set. To execute the notebooks `tutorial.ipynb`, simply make sure you are in the root directory of this tutuorial's repository, and execute: 102 | 103 | ```bash 104 | jupyter lab 105 | ``` -------------------------------------------------------------------------------- /fun_to_jax_ir.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pierreglaser/jax-tutorial/071c923c082e8dbd7aea54b220052283dc3dbe9d/fun_to_jax_ir.png -------------------------------------------------------------------------------- /jax-utils/README.md: -------------------------------------------------------------------------------- 1 | ## A set of `jax`-related utilities. 2 | 3 | 4 | Right now, this repository only contains a small function that sets up the right 5 | environment variables to get `jax` running against a `cuda` toolkit installed using 6 | `conda` 7 | -------------------------------------------------------------------------------- /jax-utils/jax_utils/set_conda_env_vars_for_jax_gpu.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | from pathlib import Path 4 | 5 | 6 | def set_environment_for_jax(): 7 | try: 8 | __import__("jax") 9 | except ModuleNotFoundError: 10 | return 11 | 12 | if os.environ.get("CUDA_VISIBLE_DEVICES") is None: 13 | return 14 | 15 | # XXX: this script has been written with condainstallation in mind -- 16 | # I should maybe check whether this python belongs to a conda env. 17 | 18 | conda_env_bin_dir = Path(sys.executable).parent 19 | conda_env_dir = conda_env_bin_dir.parent 20 | conda_env_lib_dir = conda_env_dir / "lib" 21 | 22 | # TODO(piereglaser): expose cuda toolkit to jax 23 | os.environ["PATH"] = f"{os.environ['PATH']}:{conda_env_bin_dir}" 24 | 25 | if "XLA_FLAGS" not in os.environ: 26 | print("setting XLA_FLAGS") 27 | os.environ["XLA_FLAGS"] = f"--xla_gpu_cuda_data_dir={conda_env_dir}" 28 | 29 | if "LD_LIBRARY_PATH" not in os.environ: 30 | print("setting LD_LIBRARY_PATH") 31 | os.environ["LD_LIBRARY_PATH"] = f"{conda_env_lib_dir}" 32 | 33 | # Don't prealloacte 90% of GPU memory as it recently led to memory leaks 34 | os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false" 35 | # os.environ["XLA_PYTHON_CLIENT_MEM_FRACTION"] = ".5" 36 | os.environ["XLA_PYTHON_CLIENT_ALLOCATOR"] = "platform" 37 | -------------------------------------------------------------------------------- /jax-utils/jax_utils/test_jax_installation.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | import argparse 4 | from typing import Literal 5 | 6 | 7 | def show_environment_variables(): 8 | # Make sure jax knows where to look for cuda runtime libraries 9 | print(f"XLA_FLAGS={os.environ.get('XLA_FLAGS')}") 10 | print(f"LD_LIBRARY_PATH={os.environ.get('LD_LIBRARY_PATH')}") 11 | print(f"PATH={os.environ.get('PATH')}") 12 | print(f"CUDA_VISIBLE_DEVICES={os.environ.get('CUDA_VISIBLE_DEVICES')}") 13 | print(f"XLA_PYTHON_CLIENT_PREALLOCATE={os.environ.get('XLA_PYTHON_CLIENT_PREALLOCATE')}") 14 | print(f"XLA_PYTHON_CLIENT_ALLOCATOR={os.environ.get('XLA_PYTHON_CLIENT_ALLOCATOR')}") 15 | 16 | 17 | def _test_jax_installation_unsafe(device: Literal['cpu', 'gpu']): 18 | try: 19 | import jax 20 | import jax.numpy as jnp 21 | from jax import random 22 | except ModuleNotFoundError: 23 | raise ValueError("jax is not installed") 24 | 25 | assert device in ['cpu', 'gpu'], f"device must be 'cpu' or 'gpu', got {device}" 26 | 27 | if device == 'gpu': 28 | from jaxlib.xla_extension import GpuDevice 29 | device_cls = GpuDevice 30 | else: 31 | from jaxlib.xla_extension import Device 32 | device_cls = Device 33 | 34 | if device == "gpu": 35 | # Check access to cuda compiler 36 | print("test access to a cuda compiler...", end="") 37 | try: 38 | subprocess.check_output(["which", "ptxas"]) 39 | # os.system("ptxas --version") 40 | except subprocess.CalledProcessError as e: 41 | raise ValueError("No cuda compiler found in $PATH") from e 42 | print(" OK.") 43 | 44 | # tell which device jax uses 45 | 46 | print(f"checking if jax can detect a {device} device...", end="") 47 | assert any(isinstance(d, device_cls) for d in jax.local_devices()) 48 | print(" OK.") 49 | 50 | # create a simple jax array 51 | print(f"testing array creation on {device}...", end="") 52 | key = random.PRNGKey(0) 53 | x = random.normal(key, (10,)) 54 | print(" OK.") 55 | 56 | # Use specialized cuda lib such as linear algebra solvers 57 | print( 58 | "testing use of specialized cuda libraries such as linear algebra solvers...", 59 | end="", 60 | ) 61 | A = jnp.array([[0, 1], [1, 1], [1, 1], [2, 1]]) 62 | _, _ = jnp.linalg.qr(A) 63 | 64 | A = jnp.eye(10) 65 | _, _ = jnp.linalg.eigh(A) 66 | 67 | print(" OK.") 68 | 69 | # Use cudnn primitives such as convolutions 70 | # (cudnn has to be installed separately) 71 | print("testing use of cudnn primitives...", end="") 72 | key = random.PRNGKey(0) 73 | x = jnp.linspace(0, 10, 500) 74 | y = jnp.sin(x) + 0.2 * random.normal(key, shape=(500,)) 75 | 76 | window = jnp.ones(10) / 10 77 | _ = jnp.convolve(y, window, mode="same") 78 | print(" OK.") 79 | 80 | print("Test done, everything seems well installed.") 81 | 82 | 83 | def test_jax_installation(device, verbose=False): 84 | try: 85 | _test_jax_installation_unsafe(device=device) 86 | except Exception as e: 87 | print('\n') 88 | print('\n') 89 | print('##################################################################') 90 | print('# #') 91 | print('# #') 92 | print('# ERROR WHILE TESTING JAX INSTALLATION #') 93 | print('# #') 94 | print('# #') 95 | print('##################################################################') 96 | print("An error occured during the test.") 97 | if not verbose: 98 | print("Run python -m jax_utils.test_jax_installation --verbose to get more information.") 99 | else: 100 | print("Here are the environment variables:") 101 | show_environment_variables() 102 | print("Here is the error message:") 103 | print(e) 104 | 105 | if __name__ == "__main__": 106 | 107 | parser = argparse.ArgumentParser() 108 | parser.add_argument('--verbose', action='store_true') 109 | parser.add_argument('--gpu', action='store_true') 110 | args = parser.parse_args() 111 | test_jax_installation(device="gpu" if args.gpu else "cpu", verbose=args.verbose) 112 | -------------------------------------------------------------------------------- /jax-utils/setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | try: 4 | from setuptools import setup 5 | except ImportError: 6 | from distutils.core import setup 7 | 8 | description = ("Some sanity-checking-scripts to ensure that jax is properly installed") 9 | 10 | dist = setup( 11 | name="jax-utils", 12 | version="0.0.0dev0", 13 | description=description, 14 | author="Pierre Glaser", 15 | author_email="pierreglaser@msn.com", 16 | license="BSD 3-Clause License", 17 | packages=["jax_utils"], 18 | install_requires=["jax"], 19 | classifiers=[ 20 | "Development Status :: 4 - Beta", 21 | "License :: OSI Approved :: BSD License", 22 | "Operating System :: POSIX", 23 | "Programming Language :: Python :: 3.8", 24 | "Programming Language :: Python :: Implementation :: CPython", 25 | ], 26 | python_requires=">=3.8", 27 | ) 28 | -------------------------------------------------------------------------------- /tutorial_with_solutions.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "79b3ba15-2375-4efd-97de-1c5fa6e6d2d4", 6 | "metadata": {}, 7 | "source": [ 8 | "# Tutorial: JAX\n", 9 | "\n", 10 | "The cell below will install the additional required dependencies using `pip` if running the notebook on [Google Colab](https://colab.research.google.com/) - it should do nothing if running locally." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "id": "cc17c4af-1dbb-421a-b9c8-353c835c8698", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "try:\n", 21 | " import google.colab\n", 22 | " # Running on Google Colab therefore install additional dependencies\n", 23 | " !pip install flax numpyro\n", 24 | "except ImportError:\n", 25 | " # Assume dependencies installed if not running on Colab\n", 26 | " pass" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "id": "a61a4ff0-537e-42cc-bdfd-2a76ca8de292", 32 | "metadata": { 33 | "slideshow": { 34 | "slide_type": "slide" 35 | }, 36 | "tags": [] 37 | }, 38 | "source": [ 39 | "## JAX: Intro\n", 40 | "\n", 41 | "At a high level, jax is an extensible system for composable function transformations based on trace-specializing functional python code. These transformations include:\n", 42 | "\n", 43 | "- Just In Time Compilation\n", 44 | "- Automatic Differentiation\n", 45 | "- Automatic Vectorization\n", 46 | "- Single Program Multiple Device (SMPD) transformations.\n", 47 | "- And more...\n", 48 | "\n" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "id": "89bfe947-f766-4771-90cf-711f3d086e04", 54 | "metadata": { 55 | "slideshow": { 56 | "slide_type": "subslide" 57 | }, 58 | "tags": [] 59 | }, 60 | "source": [ 61 | "The functions on which jax can operate must:\n", 62 | "\n", 63 | "- take a (collections of) tensor-like inputs and return (collections of) tensor-like outputs, which are instances of the `DeviceArray` class (similar to np.ndarry instances)\n", 64 | "- manipulate these tensors using only a set of (closed) primitives exposed in the `jax` libraries, for the most part in the `jax.numpy` and `jax.scipy` modules, and which often have a direct equivalent in `numpy` or `scipy`\n", 65 | "- be functionally pure: running the same function with the same inputs should yield the same outputs. Functional programming should feel natural when the code logic is the direct translation of mathemtatical operations, a case which is very frequent in Machine Learning.\n", 66 | "\n", 67 | "\n", 68 | "We will start with a few very examples, and expand on that." 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "id": "4a294589-a6ed-40f5-a7a4-242e237d7dc6", 74 | "metadata": { 75 | "slideshow": { 76 | "slide_type": "slide" 77 | }, 78 | "tags": [] 79 | }, 80 | "source": [ 81 | "## Starting simple... The gaussian kernel" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": 1, 87 | "id": "ae21ec89-b9a1-455d-953c-72532653acf5", 88 | "metadata": { 89 | "slideshow": { 90 | "slide_type": "skip" 91 | }, 92 | "tags": [] 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "import numpy as np\n", 97 | "\n", 98 | "import jax\n", 99 | "import jax.numpy as jnp\n", 100 | "from jax import jit, grad, vmap" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "id": "664690cf-3eea-4772-ac4e-ee45260b3f21", 106 | "metadata": { 107 | "slideshow": { 108 | "slide_type": "subslide" 109 | }, 110 | "tags": [] 111 | }, 112 | "source": [ 113 | "The `gaussian_kernel` function is a jax implementation of the famous gaussian kernel:\n", 114 | "$$ k(x, y) = e^{-\\frac{\\|x - y\\|^2}{\\sigma^2}}$$\n", 115 | "\n", 116 | "A function often used when implementing kernel-based methods, such as [Kernel Regression](https://en.wikipedia.org/wiki/Kernel_regression)" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": 2, 122 | "id": "8eb17e5b-ad06-4919-b10b-290f6d743463", 123 | "metadata": { 124 | "tags": [] 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "def gaussian_kernel(x, y):\n", 129 | " sigma = 2\n", 130 | " z = x - y\n", 131 | " return jnp.exp(-jnp.sum(jnp.square(z)) / sigma ** 2)" 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "id": "9b1ea2ff-218d-4b81-ae5e-cc0c30785073", 137 | "metadata": {}, 138 | "source": [] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 3, 143 | "id": "60ef0206-4aef-4b64-b1a0-a168213bcaf9", 144 | "metadata": { 145 | "slideshow": { 146 | "slide_type": "fragment" 147 | }, 148 | "tags": [] 149 | }, 150 | "outputs": [ 151 | { 152 | "data": { 153 | "text/plain": [ 154 | "DeviceArray(0.60653067, dtype=float32)" 155 | ] 156 | }, 157 | "execution_count": 3, 158 | "metadata": {}, 159 | "output_type": "execute_result" 160 | } 161 | ], 162 | "source": [ 163 | "x_input = jnp.ones((2,)) # [0., 0.]\n", 164 | "y_input = jnp.zeros((2,)) # [1., 1.]\n", 165 | "retval = gaussian_kernel(x_input, y_input)\n", 166 | "retval" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "id": "922cde14-db48-4379-8731-82c92fd08a5e", 172 | "metadata": { 173 | "slideshow": { 174 | "slide_type": "subslide" 175 | }, 176 | "tags": [] 177 | }, 178 | "source": [ 179 | "The gaussian kernel relies on the following jax primitives: `jnp.sum`, `jnp.square`, and `jnp.exp`, which are the jax-analogues of `np.sum`, `np.square` and `np.exp`. Almost all `numpy` primitives are present in `jax.numpy`, including traditional linear algebra operations like `jnp.dot` or `jnp.matmul`.\n", 180 | "\n", 181 | "This function satisfies all the constraints enumerated in the introductory paragraph of this section: it takes as input 2 jax arrays, return a scalar (0-th order jax array), and is pure (same input->same output). We can thus apply jax transformations on it!" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "id": "55861338-3dd4-40ef-9586-388d927184ca", 187 | "metadata": { 188 | "slideshow": { 189 | "slide_type": "slide" 190 | }, 191 | "tags": [] 192 | }, 193 | "source": [ 194 | "### Jax Transformations" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "id": "ade6e9ca-f328-4430-ba90-5b9bfd483b68", 200 | "metadata": { 201 | "slideshow": { 202 | "slide_type": "subslide" 203 | }, 204 | "tags": [] 205 | }, 206 | "source": [ 207 | "### JIT compilation" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 4, 213 | "id": "6d7ac895-e6b3-4bff-b5e9-8306b3470447", 214 | "metadata": { 215 | "tags": [] 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "# just-in-time compilation\n", 220 | "jitted_gaussian_kernel = jit(gaussian_kernel)\n", 221 | "assert gaussian_kernel(x_input, y_input) == gaussian_kernel(x_input, y_input)" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "id": "eb30fc49-3860-4c99-a1a7-3b375b9595a7", 227 | "metadata": { 228 | "tags": [] 229 | }, 230 | "source": [ 231 | "The JIT-compiled version of `gaussian_kernel`, `jitted_gaussian_kernel` executes the same end-to-end mathematical operations, makes additional software and harware optimizations to speed up computations." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": 5, 237 | "id": "97aa17d7-d822-470a-a01a-c2edffefdf0f", 238 | "metadata": {}, 239 | "outputs": [ 240 | { 241 | "name": "stdout", 242 | "output_type": "stream", 243 | "text": [ 244 | "14.5 µs ± 195 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n" 245 | ] 246 | } 247 | ], 248 | "source": [ 249 | "%timeit gaussian_kernel(x_input, y_input).block_until_ready()" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": 6, 255 | "id": "c2ffc3ea-b12f-417d-89b5-2d8cea507000", 256 | "metadata": { 257 | "tags": [] 258 | }, 259 | "outputs": [ 260 | { 261 | "name": "stdout", 262 | "output_type": "stream", 263 | "text": [ 264 | "2.32 µs ± 63.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n" 265 | ] 266 | } 267 | ], 268 | "source": [ 269 | "%timeit jitted_gaussian_kernel(x_input, y_input).block_until_ready()" 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "id": "75d37c83-691e-453e-8f25-eef4b9c6ec36", 275 | "metadata": {}, 276 | "source": [ 277 | "The speedup here is minimal, due to the fact that the function `gaussian_kernel` is quite simple: there is not much optimizations to perform.\n", 278 | "We will see other use cases where the speedup yieled by jit-compilation of such functions can become significant." 279 | ] 280 | }, 281 | { 282 | "cell_type": "markdown", 283 | "id": "1c509077-fe69-4863-889f-34bcf897226f", 284 | "metadata": { 285 | "slideshow": { 286 | "slide_type": "subslide" 287 | }, 288 | "tags": [] 289 | }, 290 | "source": [ 291 | "### Automatic differentiation\n", 292 | "\n", 293 | "Jax can compute derivatives of functions with respect to one or several arguments:" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 7, 299 | "id": "e3573043-703f-4c33-885a-07b17ac1a39a", 300 | "metadata": {}, 301 | "outputs": [ 302 | { 303 | "data": { 304 | "text/plain": [ 305 | "DeviceArray([-0.30326533, -0.30326533], dtype=float32)" 306 | ] 307 | }, 308 | "execution_count": 7, 309 | "metadata": {}, 310 | "output_type": "execute_result" 311 | } 312 | ], 313 | "source": [ 314 | "# automatic differentiation\n", 315 | "d_dx_gaussian_kernel=grad(gaussian_kernel, argnums=0) # compute the partial derivative of gaussian_kernel w.r.t x\n", 316 | "d_dx_gaussian_kernel(x_input, y_input) # returns a 2-d vector" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "id": "b72448b9-d6ae-410f-b4e5-90b7f3274951", 322 | "metadata": { 323 | "slideshow": { 324 | "slide_type": "subslide" 325 | }, 326 | "tags": [] 327 | }, 328 | "source": [ 329 | "### Automatic vectorization\n", 330 | "\n", 331 | "Last but not least, jax can also \"automatically vectorize\" a function. The vectorized function:\n", 332 | "- takes as input a \"batch\" input tensors, stacked in a new dimension\n", 333 | "- applies the original function to all input tensors in the batch" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": 8, 339 | "id": "b230cf65-3a6f-41a5-8fde-09a93e141706", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "vmapped_gaussian_kernel = vmap(gaussian_kernel, in_axes=(0, None))" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 9, 349 | "id": "9c8e506e-3891-471f-868b-27019f70d364", 350 | "metadata": {}, 351 | "outputs": [ 352 | { 353 | "data": { 354 | "text/plain": [ 355 | "DeviceArray([1. , 0.60653067], dtype=float32)" 356 | ] 357 | }, 358 | "execution_count": 9, 359 | "metadata": {}, 360 | "output_type": "execute_result" 361 | } 362 | ], 363 | "source": [ 364 | "batch_of_x_inputs = jnp.stack((jnp.zeros((2,)), jnp.ones((2,))))\n", 365 | "vmapped_gaussian_kernel(batch_of_x_inputs, y_input)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "id": "1159d42e-59f8-4492-a344-62ff98833536", 371 | "metadata": { 372 | "slideshow": { 373 | "slide_type": "subslide" 374 | }, 375 | "tags": [] 376 | }, 377 | "source": [ 378 | "### Arbitrary transformation composition\n", 379 | "\n", 380 | "Importantly, all these transformations can be composed together (!):" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": 10, 386 | "id": "231a5524-346c-45c8-af0b-ca82db740d39", 387 | "metadata": {}, 388 | "outputs": [ 389 | { 390 | "data": { 391 | "text/plain": [ 392 | "DeviceArray([[-0. , -0. ],\n", 393 | " [-0.30326533, -0.30326533]], dtype=float32)" 394 | ] 395 | }, 396 | "execution_count": 10, 397 | "metadata": {}, 398 | "output_type": "execute_result" 399 | } 400 | ], 401 | "source": [ 402 | "jit(vmap(grad(gaussian_kernel), in_axes=(0, None)))(batch_of_x_inputs, y_input)" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "id": "b1c86aad-47cb-4f49-a442-6a57b8dc05ac", 408 | "metadata": { 409 | "slideshow": { 410 | "slide_type": "subslide" 411 | }, 412 | "tags": [] 413 | }, 414 | "source": [ 415 | "With these basic building blocks in mind, we can start implementing interestiting mathematical and statistical objects. As you will see, using the function transformations provided by \n", 416 | "jax will often benefit you in one or multiple of the following ways:\n", 417 | "\n", 418 | "- reduce code complexity, increase code readability\n", 419 | "- increase code efficiency" 420 | ] 421 | }, 422 | { 423 | "cell_type": "markdown", 424 | "id": "41fb74c3-e527-43e0-b044-f9dcd81bf322", 425 | "metadata": {}, 426 | "source": [ 427 | "### Exercise 1: Computing the (Gaussian Kernel) gram matrix between two datasets" 428 | ] 429 | }, 430 | { 431 | "cell_type": "markdown", 432 | "id": "3acd4131-e729-47ed-8519-97a88604fe28", 433 | "metadata": { 434 | "slideshow": { 435 | "slide_type": "subslide" 436 | }, 437 | "tags": [] 438 | }, 439 | "source": [ 440 | "As a first nontrivial composition exercise, try to implement a function that computes the **gram matrix** between two datasets $X = \\{x_i\\}_{i=1}^{n}$ and $X = \\{y_i\\}_{i=1}^{m}$, for a given kernel $k$. This gram matrix is given by:\n", 441 | " \n", 442 | " \n", 443 | "$$\n", 444 | "M = \\begin{vmatrix}\n", 445 | "k(x_1, y_1) & \\dots & k(x_1, y_m)\\\\\n", 446 | "\\vdots & \\ddots & \\vdots \\\\\n", 447 | "k(x_n, y_1) & \\dots & k(x_n, y_m)\\\\\n", 448 | "\\end{vmatrix}\n", 449 | "$$\n" 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": 11, 455 | "id": "e53ccace-db4f-415b-ba88-c3a5d0336bd2", 456 | "metadata": { 457 | "slideshow": { 458 | "slide_type": "subslide" 459 | }, 460 | "tags": [] 461 | }, 462 | "outputs": [], 463 | "source": [ 464 | "import numpy as np\n", 465 | "random_state = np.random.RandomState(42)\n", 466 | "X = random_state.randn(100, 2)\n", 467 | "Y = random_state.randn(200, 2)" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "id": "7294df9d-f253-442c-be38-260dfc291871", 473 | "metadata": { 474 | "slideshow": { 475 | "slide_type": "subslide" 476 | }, 477 | "tags": [] 478 | }, 479 | "source": [ 480 | "#### numpy-style implementations" 481 | ] 482 | }, 483 | { 484 | "cell_type": "markdown", 485 | "id": "fb8ecf36-55a8-4fe0-8e71-7e15ebb2de4b", 486 | "metadata": {}, 487 | "source": [ 488 | "A naive implementation would consists in two nested for-loops that iterates over X and Y to compute each element of the matrix:" 489 | ] 490 | }, 491 | { 492 | "cell_type": "code", 493 | "execution_count": 12, 494 | "id": "ee4d029d-c36b-477c-9cbc-3092e41bfbf7", 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "def compute_gram_matrix_naively(X, Y):\n", 499 | " gram_matrix = np.empty((X.shape[0], Y.shape[0]))\n", 500 | " for i in range(X.shape[0]):\n", 501 | " for j in range(Y.shape[0]):\n", 502 | " kij = gaussian_kernel(X[i], Y[j]) # note the interoperability between jax and numpy arrays within **untransformed** jax funtions\n", 503 | " gram_matrix[i, j] = kij\n", 504 | " return gram_matrix" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": 13, 510 | "id": "aa26bae6-6835-4e55-8291-7ca3319f20b8", 511 | "metadata": {}, 512 | "outputs": [], 513 | "source": [ 514 | "M_naive = compute_gram_matrix_naively(X, Y)" 515 | ] 516 | }, 517 | { 518 | "cell_type": "markdown", 519 | "id": "09b880d6-021a-465d-9b8b-63d9be6620d1", 520 | "metadata": { 521 | "slideshow": { 522 | "slide_type": "subslide" 523 | }, 524 | "tags": [] 525 | }, 526 | "source": [ 527 | "A more efficient implementation consists in leveraging numpy's broadcasting abilities to compute the entries of the gram matrix in a vectorized manner.\n", 528 | "Note that this requires plumbing axis values in reduction operations happening during the kernel computation:" 529 | ] 530 | }, 531 | { 532 | "cell_type": "code", 533 | "execution_count": 14, 534 | "id": "437676b6-b054-427c-ba86-6ba0a0babea4", 535 | "metadata": {}, 536 | "outputs": [], 537 | "source": [ 538 | "def gaussian_kernel_explicit_reduction_axes(x, y):\n", 539 | " sigma = 2\n", 540 | " z = x - y\n", 541 | " # we add a axis=-1 to prevent numpy from summing over all axes when giving gaussian_kernel a stack of tensors.\n", 542 | " # return jnp.exp(-jnp.sum(jnp.square(z)) / sigma ** 2)\n", 543 | " return jnp.exp(-jnp.sum(jnp.square(z), axis=-1) / sigma ** 2)" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": 15, 549 | "id": "bf3a5a6c-f1c4-4ca0-bf00-f96da0f71368", 550 | "metadata": {}, 551 | "outputs": [], 552 | "source": [ 553 | "def compute_gram_matrix_bcast_semantics(X, Y):\n", 554 | " return gaussian_kernel_explicit_reduction_axes(X[:, None, :], Y[None, :, :])" 555 | ] 556 | }, 557 | { 558 | "cell_type": "code", 559 | "execution_count": 16, 560 | "id": "a752c3d7-3f9a-4521-9214-b5aab5fec8c6", 561 | "metadata": {}, 562 | "outputs": [], 563 | "source": [ 564 | "M_bcast = compute_gram_matrix_bcast_semantics(X, Y)\n", 565 | "assert np.allclose(M_naive, M_bcast)" 566 | ] 567 | }, 568 | { 569 | "cell_type": "markdown", 570 | "id": "bb590f0f-1855-4e88-9deb-d29bafd77c56", 571 | "metadata": { 572 | "slideshow": { 573 | "slide_type": "subslide" 574 | }, 575 | "tags": [] 576 | }, 577 | "source": [ 578 | "Benchmarking these two functions highlights the dramatically higher efficiency of the latter method:" 579 | ] 580 | }, 581 | { 582 | "cell_type": "code", 583 | "execution_count": 17, 584 | "id": "d89c1d1f-3acb-4b26-a24b-3097a4e58192", 585 | "metadata": {}, 586 | "outputs": [ 587 | { 588 | "name": "stdout", 589 | "output_type": "stream", 590 | "text": [ 591 | "292 ms ± 4.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 592 | ] 593 | } 594 | ], 595 | "source": [ 596 | "%timeit M_naive = compute_gram_matrix_naively(X, Y)" 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "execution_count": 18, 602 | "id": "5a2475e2-c16b-4da4-885b-e33618a72b81", 603 | "metadata": {}, 604 | "outputs": [ 605 | { 606 | "name": "stdout", 607 | "output_type": "stream", 608 | "text": [ 609 | "216 µs ± 3.17 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)\n" 610 | ] 611 | } 612 | ], 613 | "source": [ 614 | "%timeit M_bcast = compute_gram_matrix_bcast_semantics(X, Y)" 615 | ] 616 | }, 617 | { 618 | "cell_type": "markdown", 619 | "id": "cd9cb735-f547-462b-8f82-692e6685a2d2", 620 | "metadata": { 621 | "slideshow": { 622 | "slide_type": "subslide" 623 | }, 624 | "tags": [] 625 | }, 626 | "source": [ 627 | "However, the latter method required:\n", 628 | "\n", 629 | "- rewriting `gaussian_kernel` to account for the case when it is given batched inputs. What if `gaussian_kernel` is replaced by a much more complex function? What is it is replaced by a third-party function that you are not familiar with?\n", 630 | "- relying on broadcasting semantics in `compute_gram_matrix_bcast_semantics`: althought in that case, the use of such semantics was quite simple, its use in complex code base can quickly become error-prone." 631 | ] 632 | }, 633 | { 634 | "cell_type": "markdown", 635 | "id": "cd07bd73-cc67-472c-a68c-47df3f789915", 636 | "metadata": { 637 | "slideshow": { 638 | "slide_type": "subslide" 639 | }, 640 | "tags": [] 641 | }, 642 | "source": [ 643 | "#### Jax-implementation" 644 | ] 645 | }, 646 | { 647 | "cell_type": "code", 648 | "execution_count": 19, 649 | "id": "f700875e-b672-4c39-b67b-e355b01d2864", 650 | "metadata": {}, 651 | "outputs": [], 652 | "source": [ 653 | "# Exercise: combine of vmap and gaussian_kernel within compute_gram_matrix_using_vmap to compute the gram matrix between X and Y\n", 654 | "def compute_gram_matrix_using_vmap(X, Y):\n", 655 | " # raise NotImplemented\n", 656 | " return vmap(vmap(gaussian_kernel, in_axes=(None, 0)), in_axes=(0, None))(X, Y)" 657 | ] 658 | }, 659 | { 660 | "cell_type": "code", 661 | "execution_count": 20, 662 | "id": "a143ddf1-8a5e-4ea0-a519-2cfc777f2587", 663 | "metadata": {}, 664 | "outputs": [], 665 | "source": [ 666 | "M_vmap = compute_gram_matrix_using_vmap(X, Y)\n", 667 | "assert np.allclose(M_vmap, M_bcast)" 668 | ] 669 | }, 670 | { 671 | "cell_type": "code", 672 | "execution_count": 21, 673 | "id": "5acaaa09-a7a6-454e-b9e8-0f0912d8c213", 674 | "metadata": {}, 675 | "outputs": [], 676 | "source": [ 677 | "jitted_gaussian_kernel = jit(gaussian_kernel)\n", 678 | "_ = jitted_gaussian_kernel(jnp.ones((2,)), jnp.zeros((2,))).block_until_ready()" 679 | ] 680 | }, 681 | { 682 | "cell_type": "markdown", 683 | "id": "7a056ab3-6fca-4c36-8c3b-d39d3973d393", 684 | "metadata": {}, 685 | "source": [ 686 | "### Exercise 2: A jax-implementation of the Kernelized Stein Discrepancy.\n", 687 | "\n", 688 | "\n", 689 | "Further composition of `vmap`, `jit` and `grad` happens when computing the infamous `Stein` kernel, which is used to evaluate whether samples $\\{x_i\\}_{i=1}^{n}$ are distributed according to a test density $q(x) = p(x) / Z$ known up to a normalizing constant Z. Given some given kernel $k$, Writing $s(x) = \\nabla \\log p(x)$, the stein kernel writes:\n", 690 | "\n", 691 | "$$\n", 692 | "k_{\\textrm{stein}}(x, y) = s(x)^\\top s(y)k(x, y) + s(x)^\\top \\nabla_yk(x, y) + \\nabla_x k(x, y)^\\top s(y) + \\textrm{div}_x(∇_yk(x, y))\n", 693 | "$$\n", 694 | "\n", 695 | "which can be used to compute a measure of discrepancy between the samples and the density $p$, called the Kernelized Stein Discrepancy (or KSD):\n", 696 | "\n", 697 | "$$\n", 698 | "\\text{KSD}(p, x) = \\sqrt{ \\frac{1}{N^2} \\sum_{i=1}^n \\sum_{j=1}^n k_{\\textrm{stein}}(x_i, y_j)}\n", 699 | "$$" 700 | ] 701 | }, 702 | { 703 | "cell_type": "markdown", 704 | "id": "3516c328-d001-495b-b758-f9372697db83", 705 | "metadata": {}, 706 | "source": [ 707 | "As an exercise, compute KSD between $p$ and $X$ and using the gaussian kernel as the base kernel, and $p$ being standard normal distributio\n", 708 | "We will place ourselves under the null hypothesis, meaning that $X$ will be sampled from $p$." 709 | ] 710 | }, 711 | { 712 | "cell_type": "code", 713 | "execution_count": 22, 714 | "id": "1ad2457f-0879-4640-91c2-380aa6028c28", 715 | "metadata": {}, 716 | "outputs": [], 717 | "source": [ 718 | "def p(x):\n", 719 | " return jnp.exp(-0.5 * jnp.sum(jnp.square(x))) # unnormalized standard normal density\n", 720 | "\n", 721 | "s = ... # compute the score function" 722 | ] 723 | }, 724 | { 725 | "cell_type": "code", 726 | "execution_count": 23, 727 | "id": "72df2649-5067-482c-a116-00001b1cb4f7", 728 | "metadata": {}, 729 | "outputs": [], 730 | "source": [ 731 | "# Solution\n", 732 | "s = grad(lambda x: jnp.log(p(x)))" 733 | ] 734 | }, 735 | { 736 | "cell_type": "code", 737 | "execution_count": 24, 738 | "id": "21ffa12f-8032-45c1-98ec-87cc329cf19b", 739 | "metadata": {}, 740 | "outputs": [], 741 | "source": [ 742 | "# You will need the jacobian function transformation of jax, which computes the jacobian of a vector-valued function\n", 743 | "from jax import jacobian" 744 | ] 745 | }, 746 | { 747 | "cell_type": "code", 748 | "execution_count": 25, 749 | "id": "0760266f-9f58-4f21-8299-eb71f19ec634", 750 | "metadata": {}, 751 | "outputs": [], 752 | "source": [ 753 | "def stein_kernel(x, y):\n", 754 | " k = gaussian_kernel\n", 755 | " term_1 = jnp.dot(s(x), s(y)) * k(x, y)\n", 756 | " term_2 = jnp.dot(s(x), grad(k, argnums=1)(x, y))\n", 757 | " term_3 = jnp.dot(grad(k, argnums=0)(x, y), s(y))\n", 758 | " \n", 759 | " # Solution\n", 760 | " term_4 = jnp.trace(jacobian(grad(k, argnums=1), argnums=0)(x, y))\n", 761 | " \n", 762 | " return term_1 + term_2 + term_3 + term_4" 763 | ] 764 | }, 765 | { 766 | "cell_type": "code", 767 | "execution_count": 26, 768 | "id": "c29f9790-7b8b-4c6e-a490-8d3ecfecd25b", 769 | "metadata": {}, 770 | "outputs": [], 771 | "source": [ 772 | "def compute_gram_matrix_stein_kernel(X):\n", 773 | " return vmap(vmap(stein_kernel, in_axes=(None, 0)), in_axes=(0, None))(X, X)" 774 | ] 775 | }, 776 | { 777 | "cell_type": "code", 778 | "execution_count": 27, 779 | "id": "0e3b506e-754c-4e7b-98d4-3202e1d46567", 780 | "metadata": {}, 781 | "outputs": [], 782 | "source": [ 783 | "def ksd(X):\n", 784 | " N = X.shape[0]\n", 785 | " return jnp.sqrt(jnp.sum(compute_gram_matrix_stein_kernel(X)) / (N **2))" 786 | ] 787 | }, 788 | { 789 | "cell_type": "code", 790 | "execution_count": 28, 791 | "id": "b732e0ce-12fd-47c3-bba1-6ef2616a3d21", 792 | "metadata": {}, 793 | "outputs": [], 794 | "source": [ 795 | "from jax import random\n", 796 | "key = random.PRNGKey(0)\n", 797 | "X = random.normal(key, (5000,2))" 798 | ] 799 | }, 800 | { 801 | "cell_type": "markdown", 802 | "id": "ce3fd341-7705-4570-9ca0-2f27607630ae", 803 | "metadata": {}, 804 | "source": [ 805 | "Under the null hypothesis KSD(X, p) should tend to 0 at a $1/\\sqrt{N}$ rate. Given that X is actually sampled from a unit gaussian, we should recover this regime if we implemented\n", 806 | "the KSD properly. The following cells will plot the relationship between $1/\\sqrt{N}$ and $\\text{KSD}(X, p)$ for different sample size $N$ for X." 807 | ] 808 | }, 809 | { 810 | "cell_type": "code", 811 | "execution_count": 29, 812 | "id": "73272e70-12d4-46f4-a460-6d4dab6fe0e4", 813 | "metadata": {}, 814 | "outputs": [], 815 | "source": [ 816 | "sample_sizes = (50, 100, 200, 300, 500, 1000, 2000, 5000)\n", 817 | "random_state = np.random.RandomState(40)\n", 818 | "\n", 819 | "ksd_vals = []\n", 820 | "for N in sample_sizes:\n", 821 | " ksd_vals_this_iter = []\n", 822 | " for rs in range(5):\n", 823 | " X = random_state.randn(N, 2)\n", 824 | " ksd_vals_this_iter.append(ksd(jnp.array(X)))\n", 825 | " ksd_vals.append(jnp.mean(jnp.array(ksd_vals_this_iter)))" 826 | ] 827 | }, 828 | { 829 | "cell_type": "code", 830 | "execution_count": 30, 831 | "id": "6b46607b-7aae-4690-a619-76215260b7a6", 832 | "metadata": {}, 833 | "outputs": [ 834 | { 835 | "data": { 836 | "text/plain": [ 837 | "" 838 | ] 839 | }, 840 | "execution_count": 30, 841 | "metadata": {}, 842 | "output_type": "execute_result" 843 | }, 844 | { 845 | "data": { 846 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAABl3UlEQVR4nO3dd3yN9///8cfJToggNkFixxZ701pVNapiFjVq1eqgA61WVZWqWTWqVqQtLW21pZTaKhXUqiBmgiBDQsY51+8PX/l9UqMJOTkZz/vtltvNuc41XtdbnPP0vq73+zIZhmEgIiIikoPY2boAERERkYymACQiIiI5jgKQiIiI5DgKQCIiIpLjKACJiIhIjqMAJCIiIjmOApCIiIjkOA62LiAzslgsXL58GXd3d0wmk63LERERkVQwDIOYmBiKFSuGnd2j+3gUgB7g8uXLeHl52boMEREReQwXLlygRIkSj1xHAegB3N3dgbsNmCdPHhtXIyIiIqkRHR2Nl5dX8vf4oygAPcC9y1558uRRABIREcliUnP7im6CFhERkRxHAUhERERyHAUgERERyXF0D9ATMJvNJCYm2roMeQQnJ6f/HAopIiI5jwLQYzAMg/DwcCIjI21divwHOzs7vL29cXJysnUpIiKSiSgAPYZ74adQoUK4ublpssRM6t6ElmFhYZQsWVJ/TyIikkwBKI3MZnNy+PH09LR1OfIfChYsyOXLl0lKSsLR0dHW5YiISCahmyPS6N49P25ubjauRFLj3qUvs9ls40pERCQzUQB6TLqckjXo70lERB5EAUhERERyHAUgERERyXEUgERERCTHUQDKQfr160enTp1SLPv2229xcXHh448/JjY2lnHjxuHj44OLiwsFCxakefPm/Pjjj8nrN2/eHJPJhMlkwtnZmeLFi9OhQwfWrVuXwWcjIiLy+BSAcrDFixfTq1cv5s6dyxtvvMGQIUP4/vvvmTt3LidOnOCXX37h+eef5/r16ym2GzRoEGFhYYSEhLB27Vp8fX3p3r07gwcPttGZiIhIVmCxWJg2bRqff/65rUvRPEDpwTAMbifaZpi1q6P9Y410+vjjj5k4cSKrV6/m+eefB+CHH37gs88+45lnngGgdOnS+Pn53betm5sbRYoUAcDLy4v69etTsWJFXnrpJbp168bTTz/9BGckIiLZ0ZUrV3jxxRfZtGkTTk5OtG7dGh8fH5vVowCUDm4nmvGd+KtNjn1schvcnNL21zh+/HjmzZvHjz/+mCKsFClShI0bN9KlSxfc3d3TtM++ffvy6quvsm7dOgUgERFJYcuWLfTu3Zvw8HBcXV2ZPXs23t7eNq1Jl8BymJ9//plp06axfv36+4LKF198we7du/H09KROnTqMGTOGXbt2pWq/dnZ2lC9fntDQUCtULSIiWVFSUhLvvPMOrVq1Ijw8nMqVK/Pnn38ycOBAm8/Tph6gdODqaM+xyW1sduy0qFatGhEREUycOJE6deqk6Olp2rQpZ86cYe/evezatYutW7fy2Wef8d577zFhwoT/3LdhGDb/hRYRkczBbDbTqlUrtm3bBsDgwYP59NNPM82TFNQDlA5MJhNuTg42+Ulr4ChevDjbt28nLCyMtm3bEhMTk+J9R0dHmjRpwvjx49m0aROTJ0/m/fffJyEh4ZH7NZvNnDp1yuZdmiIikjnY29vTunVr8uTJw5o1a1i4cGGmCT+gAJQjlSxZku3bt3P16lVat25NdHT0Q9f19fUlKSmJO3fuPHKfX331FTdv3ky+oVpERHKe+Ph4Lly4kPx63Lhx/P333/j7+9uwqgfTJbAcqkSJEmzbto0WLVrQunVrfv31Vzp27EiPHj2oXbs2np6eHDt2jLfeeosWLVqQJ0+e5G3j4uIIDw8nKSmJS5cusW7dOj799FOGDh1KixYtbHhWIiJiKyEhIfj7+xMfH8/+/ftxc3PDzs4OLy8vW5f2QOoBysHuXQ6LjIykVatWtGnThq+++orWrVtTqVIlXnnlFdq0acPXX3+dYrtFixZRtGhRypQpQ+fOnTl27BiBgYHMnz/fRmciIiK2tHr1amrWrMlff/1FeHg4J0+etHVJ/8lkGIZh6yIym+joaDw8PIiKikrR8wFw584dzp49i7e3Ny4uLjaqUFJLf18iItYTGxvLyJEjWbp0KXB3MM2qVasoUaKETep51Pf3v6kHSERERNLsyJEj1KlTh6VLl2IymZg4cSJbtmyxWfhJK90DJCIiImn2+uuvc/z4cYoWLcqqVavSdA/opcjbWCwGXvltNypMPUAiIiKSZosXL6ZHjx4EBwenOvycux7L+LWHaT79d6b+fNzKFT6aeoBERETkP+3fv58tW7bw5ptvAndHE69evTpV24ZcvcX830NYf+gyZsvdW4+jbieSkGTBycE2fTEKQCIiIvJQFouFTz/9lPHjx5OUlETVqlV59tlnU7XtifBo5m4N4acjYdwbctWsfEFeaVmW2qXzW7Hq/6YAJCIiIg8UERFB37592bhxIwBdu3alcePG/7nd35eimL3lFJuOXUle9nSlwrzSsizVvfJaq9w0UQASERGR+2zfvp2ePXty+fJlnJ2dmTVrFi+//PIjH8H01/mbzNlyit9PXgPAZIJ2VYowokU5fIs9elh6RlMAEhERkRQ++eQTxo0bh8VioWLFigQGBlKtWrWHrr/vzHXmbA1hZ0gEAHYmeK56MYa3KEu5wu4P3c6WNAosB2nevDmjR49Ofl26dGlmzZpls3r+S3h4OK1atSJXrlzkzZsXuPvg2e+//96mdYmIZHelS5fGYrHQr18/Dhw48MDwYxgGO05do9vne/D/Yi87QyJwsDPxgl8JtrzanFnda2ba8APqAcrR/vzzT3LlymXrMh7q008/JSwsjODgYDw8PAAICwsjX758AISGhuLt7c3BgwepUaOGDSsVEcn6bt68mfz52rVrV/bt20fdunXvW88wDH4/eZXZW0IIvhAJgJO9HV1rl2BoszI2ndsnLRSAcrCCBQvaugQAEhMTcXR0vG/56dOn8fPzo1y5csnLihQpkpGliYhkewkJCbz11lusWLGCgwcPUqxYMYD7wo/FYrDp2BXm/n6Kvy9FA+DsYEePuiV5uZkPRT1cM7z2J6FLYDnYvy+BmUwmFi9eTOfOnXFzc6NcuXJs2LAhxTbHjh3jmWeeIXfu3BQuXJg+ffoQERGR/P4vv/xC48aNyZs3L56enjz77LOcPn06+f3Q0FBMJhNff/01zZs3x8XFhZUrVz6wtrVr17J8+XJMJhP9+vVLrvHeJTBvb28Aatasiclkonnz5unTMCIiOcSZM2do0qQJM2bM4OrVq/d95gOYLQY/HLpMu892MGRlEH9fisbNyZ7BTX3YMa4F7z5XOcuFH1AASlexsbEP/blz506q1719+3aq1rWG9957j27dunH48GGeeeYZevXqxY0bN4C7l5+aNWtGjRo1OHDgAL/88gtXrlyhW7duKWodO3Ysf/75J1u2bMHOzo7OnTtjsVhSHGfcuHGMHDmS48eP06ZNm/vq+PPPP2nbti3dunUjLCyMzz777L519u/fD8Bvv/1GWFgY69atS8+mEBHJ1r755htq1qzJ/v37yZcvH99//z1DhgxJfj/JbGFt0EVafbqdVwIOcvJKDLmdHRjeogw7x7XkrWcqUcg96z5kWpfA0lHu3Lkf+t4zzzzDTz/9lPy6UKFCxMXFPXDdZs2asW3btuTXpUuXTtHLco9xb1apdNSvXz969OgBwIcffsicOXPYv38/bdu2ZcGCBdSqVYsPP/wwef2lS5fi5eXFP//8Q/ny5Xn++edT7G/JkiUUKlSIY8eOUaVKleTlo0ePpkuXLg+to2DBgjg7O+Pq6vrQy173LuF5enrq0piISCrdvn2bMWPGsHDhQgAaNmxIQEAAJUuWBCAhycK6vy4yf9tpzt+4+z2Vx8WBlxp707+hNx5u99+ykBUpAEkK/3unf65cuXB3d+fq1asABAUF8fvvvz8w6J0+fZry5ctz+vRpJkyYwN69e4mIiEju+Tl//nyKAFS7dm0rn4mIiDzI1KlTWbhwISaTiTfffJN3330XR0dH7iSa+ebABT7ffoZLkXevROTP5cTAJt70qV8Kd5fsEXzuUQBKR7du3Xroe/b29ile3wsVD2Jnl/LKZGho6BPVlRb/vhnZZDIlhxiLxUKHDh2YNm3afdsVLVoUgA4dOuDl5cWiRYsoVqwYFouFKlWqkJCQkGL9zDz6TEQkOxs3bhw7duzgrbfeolWrVtxOMPPVjjN88ccZrsbEA1DQ3ZmXm/rQs15J3JyyZ1TInmdlI2n5UrfWutZUq1Yt1q5dS+nSpXFwuP9X5/r16xw/fpyFCxfSpEkTAHbu3Gm1epycnAAwm81WO4aISFYXExPDwoULGTt2LHZ2duTKlYutW7cSm2BmwbbTLN5xhuuxd/+TWtTDhSHNyuBfxwsXR/v/2HPWpgAkqTZ8+HAWLVpEjx49eP311ylQoAAhISGsWbOGRYsWkS9fPjw9Pfniiy8oWrQo58+fZ/z48Varp1ChQri6uvLLL79QokQJXFxckucLEhGRu7cudO/enZCQEAzD4PXXXyfqdiJf7Q5l6a6zRMYlAuCV35VhzcvSpVZxnB2yd/C5R6PAJNWKFSvGrl27MJvNtGnThipVqjBq1Cg8PDyws7PDzs6ONWvWEBQURJUqVRgzZgzTp0+3Wj0ODg7Mnj2bhQsXUqxYMTp27Gi1Y4mIZCWGYTB79mwaNGhASEgIXl5eVKlZh09+PUnjj7Yyc/M/RMYl4lMgF5+8UJ2trzanR92SOSb8AJgMawwlyuKio6Px8PAgKiqKPHlSPrztzp07nD17Fm9vb1xcsu7wv5xCf18iktPcuHGDl156ifXr1wPQtn0Havd5k7VHo4hLuHvLQPnCuRneoizPViuGvd3DH26a1Tzq+/vfdAlMREQkm9i3bx8vvPACFy5cwMnJiaf6vcrpgo05fvDufG6+RfMw8qmytPYtgl02Cj6PQwFIREQkm7C3tyc8PJz8RUuRq92rHMvnA0kG1b3yMrJlWVpWLITJlLODzz0KQCIiIlnYvecphkbE8nWoIwW6TMChaEXsnN2oUzofr7QsR5NyBRR8/kUBSEREJIvatGkT/QcMpPmI6eyJzI3FAKfStWhU1pNXWpajvo+nrUvMtBSAHpPuHc8a9PckItlRYmIiw8aOY/HcTwH4fulsCnZ+i+YVCvJKy3L4lcpn4wozPwWgNLo3U3JcXByurlnv6bc5zb0ZqP89E7eISFb1694jvNinF1dDjgCQu0Y7Xhj+FqPbVqFaiby2LS4LUQBKI3t7e/LmzZv8KAs3NzddV82kLBYL165dw83N7YEzV4uIZCVB527y2vRF/LF4Mpb4WEzObrR6eSKzxg+lUtFHD/mW++lb4THce/L4o57nJZmDnZ0dJUuWVEgVkSxr75nrzNl6it82b+bq1xMBKFymCgEBq2lRp6qNq8u6FIAeg8lkomjRohQqVIjExERblyOP4OTkdN/DZUVEMjvDMNhxKoK5W0PYH3p3Dp/cPjVxq96AVo3rMnfmx8nPQ5THowD0BOzt7XVviYiIpBvDMNh64ipztoYQfCGSuJO7yVOuNt0blmVIszIUeb+tLumnE7WiiIiIjVksBpuOhTNnawhHL0djSbhN1JaFRB/+jVYv9ueDTkttXWK2owAkIiJiI2aLwY+HLzPv9xD+uXILALub54ndOJ3oi2exs7OjQpnSGIahexnTmQKQiIhIBks0W1gffJn5v4dwJiIWgNxO9lS4uZefvvqI+Ph4ihUrxurVq2nWrJmNq82eFIBEREQySEKShbV/XWT+thAu3LgNQF43R7pX92TPsims+/47ANq3b8+yZcsoUKCALcvN1hSARERErOxOopnAPy/w+fbThEXdAcAzlxODmvrQu34pIq+F88mAP3B0dOSjjz5izJgxuuRlZQpAIiIiVhKXkMTqfedZ+McZrsXEA1DI3ZmXm5WhRx0v3Jzvfg3nLlGCgIAA8ubNS506dWxZco6hACQiIpLOYu4ksmLvORbvOMuN2LuP5Cnm4cLQ5mV4obYX0Tev83ynDrz88st06tQJgFatWtmw4pxHAUhERCSdRMUl8uXus3y5K5So23cnyi2Z343hLcrQuWYJnBzs2Lp1K7169SI8PJzDhw/Trl07nJ2dbVx5zqMAJCIi8oRuxCawZOcZlu8+R0x8EgA+BXMxokVZnqteDAd7O5KSkpgwYRJTpkzBMAx8fX0JDAxU+LERBSAREZHHdDXmDot3nGXl3nPEJZgBqFDYnREty/JM1aLY2929kfnixYv07NmTHTt2ADBgwABmz56Nm5ubzWrP6RSARERE0ig86g6fbz9NwP7zxCdZAKhSPA+vtCxHq0qFsbP7/yO4rl69So0aNbh+/Tru7u4sXLiQHj162Kp0+T8KQCIiIql04UYcC7af5tsDF0kw3w0+NUvmZWTLcjSvUPCBQ9cLFSpEt27d2L9/P2vWrKFs2bIZXbY8gAKQiIjIfwiNiGXe7yF8d/ASSRYDgLre+RnZshyNynreF3xOnz6Nm5sbRYsWBWDmzJmYTCbd75OJKACJiIg8RMjVGOZuDWHDocv8X+6hSbkCjGhRlno+ng/cZs2aNQwePJjatWuzefNm7O3tcXFxycCqJTUUgERERP7l2OVo5v5+ip//Dsf4v+DTsmIhRrQsS62S+R64TVxcHKNGjWLx4sUAJCYmEh0dTb58D15fbMvO1gXMnz8fb29vXFxc8PPzS75D/kHWrVtHq1atKFiwIHny5KFBgwb8+uuv9623du1afH19cXZ2xtfXl++++86apyAiItnE4YuRDPzqAM/M3sHGI3fDT5vKhfnxlcYs7VfnoeHn6NGj1K1bl8WLF2MymXjnnXf4/fffFX4yMZsGoMDAQEaPHs3bb7/NwYMHadKkCe3ateP8+fMPXP+PP/6gVatWbNy4kaCgIFq0aEGHDh04ePBg8jp79uzB39+fPn36cOjQIfr06UO3bt3Yt29fRp2WiIhkMUHnbtB36X6em7uL345fwWSCZ6sV5ZfRTVjYpzZVins8cDvDMFi0aBF16tTh6NGjFClShN9++433338fBwddZMnMTIZxr3Mv49WrV49atWqxYMGC5GWVKlWiU6dOTJ06NVX7qFy5Mv7+/kycOBEAf39/oqOj+fnnn5PXadu2Lfny5SMgICBV+4yOjsbDw4OoqCjy5MmThjMSEZGswjAM9py5zpwtIew5cx0AezsTHWsUY3iLspQpmPs/93Hnzh1q1arF8ePHad26NStWrKBQoULWLl0eIi3f3zaLpwkJCQQFBTF+/PgUy1u3bs3u3btTtQ+LxUJMTAz58+dPXrZnzx7GjBmTYr02bdowa9ash+4nPj6e+Pj45NfR0dGpOr6IiGQ9hmHwx6kI5mw5xYFzNwFwtDfxfK0SDG1ehlKeuVK9LxcXFwIDA/n555957bXXsLOz+Z0lkko2C0ARERGYzWYKFy6cYnnhwoUJDw9P1T5mzJhBbGws3bp1S14WHh6e5n1OnTqV9957Lw3Vi4hIVmMYBluOX2XO1lMcuhgFgJODHf61vRjSvAzF87qmah+zZs3CZDIxevRoAKpWrUrVqlWtWbpYgc0vUP577gTDMB44kdS/BQQE8O6777J+/fr7uhvTus8333yTsWPHJr+Ojo7Gy8srNeWLiEgmZ7EY/HI0nDlbQzgedreH38XRjl71SjG4qQ+F86RuiHpERAT9+/fnxx9/xMHBgbZt21KxYkVrli5WZLMAVKBAAezt7e/rmbl69ep9PTj/FhgYyIABA/jmm294+umnU7xXpEiRNO/T2dlZk1OJiGQzZovBj4cvM3drCKeu3gIgl5M9fRqUZmATbwrkTv3n/h9//EHPnj25dOkSzs7OzJw5kwoVKlirdMkANrtY6eTkhJ+fH5s3b06xfPPmzTRs2PCh2wUEBNCvXz9Wr15N+/bt73u/QYMG9+1z06ZNj9yniIhkH4lmC98cuMDTM7czak0wp67ewt3FgZEty7JzXEvGt6uY6vBjNpuZPHkyLVq04NKlS1SoUIG9e/cybNiwVF2tkMzLppfAxo4dS58+fahduzYNGjTgiy++4Pz58wwZMgS4e2nq0qVLLF++HLgbfl588UU+++wz6tevn9zT4+rqiofH3SGKo0aNomnTpkybNo2OHTuyfv16fvvtN3bu3GmbkxQRkQwRn2RmbdAl5m8L4eLN2wDkdXNkYGNvXmxYmjwujmnan2EYtG/fPnm+ub59+zJ37lxy5/7v0WGS+dk0APn7+3P9+nUmT55MWFgYVapUYePGjZQqVQqAsLCwFHMCLVy4kKSkJIYPH87w4cOTl/ft25dly5YB0LBhQ9asWcM777zDhAkTKFOmDIGBgdSrVy9Dz01ERDLGnUQza/afZ+EfZwiLugNAgdxODGriQ+/6pcjl/HhfdSaTibZt27Jz504WLFhAnz590rNssTGbzgOUWWkeIBGRzC8uIYlVe8/zxY4zXIu5O5VJ4TzOvNy0DD3qlsTVyT7N+0xMTCQsLIySJUsCd3uBLly4kPxaMrcsMQ+QiIjI44i5k8jyPedYsvMsN2ITACie15WhzcvwQu0SODukPfgAhIaG0r17d27cuEFQUBDu7u6YTCaFn2xKAUhERLKEqLhElu46y5e7zhJ9JwmAUp5uDG9els61iuNo//jjetauXcuAAQOIiooib968HDt2TLdOZHMKQCIikqldvxXPkp1nWb7nHLfi7wafMgVzMaJlWTpUK4bDEwSfO3fuMHbs2ORHMtWvX5+AgABKly6dHqVLJqYAJCIimdLV6Dss2nGGlXvPczvRDEDFIu680rIcbasUwd7uyYahnzhxAn9/fw4fPgzAuHHjeP/993F0TNtoMcmaFIBERCRTuRx5m4XbTxPw5wUSkiwAVC3uwSsty/J0pcLYPWHwuefNN9/k8OHDFCxYkBUrVtCmTZt02a9kDQpAIiKSKVy4Ecf8baf5NugCiea7A5RrlczLK0+Vo3n5guk+8eDnn3+Oi4sLM2fOpGjRoum6b8n8FIBERMSmzly7xfxtp/nu4CXMlrvBp75Pfka2LEeDMp7pFnwOHjzIDz/8wMSJE4G7D8oOCAhIl31L1qMAJCIiNvHPlRjmbg3hx8OX+b/cQ5NyBRj5VDnqlM6fbscxDIN58+bx6quvkpCQgK+vL127dk23/UvWpAAkIiIZ6ujlKOZuDeHnv///g6ufrlSIES3LUcMrb7oe6+bNmwwYMIDvvvsOgOeee44WLVqk6zEka1IAEhGRDHHoQiRztp7it+NXk5e1q1KEES3LUrmYR7ofb/fu3fTo0YPz58/j6OjI9OnTGTlypB5iKoACkIiIWNmB0BvM3hrCH/9cA8DOBM9WK8aIlmUpX9jdKsecO3cuo0ePxmw2U7ZsWdasWYOfn59VjiVZkwKQiIikO8Mw2HP6OrO3nmLvmRsA2NuZ6FyzOMOal8GnoHWfqO7t7Y3ZbKZnz54sWLBAz3WU+ygAiYhIujEMg+3/XGPO1hCCzt0EwNHeRFc/L4Y1L4NXfjerHfvmzZvky5cPgPbt27Nv3z7q1KmjS17yQApAIiLyxAzD4LfjV5mz9RSHL0YB4ORgR486XrzcrAzF8rpa7dhJSUlMmjSJhQsXEhQURKlSpQCoW7eu1Y4pWZ8CkIiIPDaLxeDnv8OZs/UUJ8JjAHB1tKd3/ZIMauJDoTwuVj3++fPn6dmzJ7t27QJg3bp1jBkzxqrHlOxBAUhERNIsyWzhx8NhzP09hJCrtwDI7ezAiw1KMaCxN565na1ew/r16+nfvz83b94kT548fPHFF/j7+1v9uJI9KACJiEiqJZotfHfwEvN/DyH0ehwAeVwc6N/Im/6NSpPXzcnqNcTHx/P6668zZ84cAOrUqcOaNWvw8fGx+rEl+1AAEhGRVFn310VmbPqHS5G3Acify4kBjb15sUEp3F0y7gnqM2fOTA4/r776Kh9++CFOTtYPXpK9KACJiMh/WrDtNNN+OQFAgdzOvNzUh171S+LmlPFfI2PGjGHLli2MGTOG9u3bZ/jxJXtQABIRkUda9MeZ5PAzrHkZRj5VDhdH+ww7fmxsLAsWLGDMmDHY29vj4uLC5s2bNbxdnogCkIiIPNTSnWeZsvE4AKOfLsfop8tn6PGPHDmCv78/x48f5/bt20yYMAFA4UeemJ2tCxARkcxp+Z5QJv94DIBXWpZl1FPlMuzYhmGwcOFC6taty/HjxylatChNmjTJsONL9qceIBERuc/KveeYuP4oAEObl2Fsq/IZ1usSGRnJ4MGD+eabbwBo164dX331FQULFsyQ40vOoB4gERFJYc3+87zz/d8ADG7qwxttKmRY+AkKCqJmzZp88803ODg4MH36dH788UeFH0l36gESEZFkXx+4wJvfHQHgpUbevNmuYobeb+Pk5ER4eDje3t6sWbNGj7MQq1EAEhER4O48P+PWHsYwoF/D0kx4tlKGhJ/ExEQcHe/OI1S1alU2bNhA3bp18fDwsPqxJefSJTAREWF98CVe++YQhgG965dkUgffDAk/27Zto1y5cuzduzd5WatWrRR+xOoUgEREcrgfDl1mTGAwFgN61C3J5OeqWD38mM1m3n33XZ566inOnTvH5MmTrXo8kX/TJTARkRxs45EwRv9f+OlWuwRTOlXBzs664efSpUv06tWL7du3A9C/f//kR1uIZBQFIBGRHOqXv8MZGXAQs8Xg+Vol+KhLNauHn40bN9K3b18iIiLInTs3n3/+Ob169bLqMUUeRAFIRCQH2nzsCiNW/0WSxaBTjWJ83NX64WfHjh3Jz+6qWbMmgYGBlCuXcZMrivwvBSARkRxm64krDFsVRJLFoEP1YnzyQnXsrRx+ABo3bkyHDh3w9vbm448/xtnZ2erHFHkYBSARkRxk+z/XGLLiLxLNBu2rFuXTbtVxsLfeeJj169fTsmVL3N3dMZlMrFu3DgcHffWI7WkUmIhIDrHzVASDlh8gwWyhbeUizOpew2rh5/bt27z88st06tSJoUOHYhgGgMKPZBr6TRQRyQF2h0Qw4Ks/SUiy0Mq3MLN71MTRSuHn2LFj+Pv78/fff2MymShVqhSGYegJ7pKpKACJiGRze89cZ8BXB4hPsvBUxULM61kLJ4f0Dz+GYfDll18yYsQIbt++TeHChVm5ciVPP/10uh9L5EkpAImIZGP7z97gpWV/cjvRTPMKBZnf2zrhJyYmhiFDhrB69Wrg7mzOK1asoHDhwul+LJH0oHuARESyqaBzN+j/5X7iEsw0KVeAz3v74exgb5VjxcXFsWXLFuzt7Zk6dSq//PKLwo9kauoBEhHJhg6ev0nfpX8Sm2CmYRlPFr1YGxfH9A0//3tfT+HChQkMDMTBwYFGjRql63FErEE9QCIi2cyhC5G8uGQ/t+KTqO+TnyV966R7+Ll+/TqdOnUiICAgeVmzZs0UfiTLUAASEclG/r4URZ8l+4iJT6Ju6bvhx9UpfcPPzp07qVGjBhs2bGDkyJHExcWl6/5FMoICkIhINnH0chS9Fu8j+k4SfqXysbR/HXI5p9+dDmazmSlTptC8eXMuXrxIuXLl2LRpE25ubul2DJGMonuARESygRPh0fRevI+o24nU8MrLsv51yJ2O4ScsLIw+ffqwZcsWAHr37s38+fNxd3dPt2OIZCQFIBGRLO6fKzH0WrSPm3GJVC/hwfIBdXF3cUy3/d+8eZOaNWty5coV3NzcmD9/Pi+++KImNpQsTZfARESysJCrMfRctJfrsQlULe7B8gH1yJOO4QcgX7589OrVi2rVqhEUFETfvn0VfiTLMxn3HtAiyaKjo/Hw8CAqKoo8efLYuhwRkQc6fe0W3b/Yy7WYeHyL5mH1oHrkdXNKl32fO3cOOzs7vLy8AEhISMBsNuPq6pou+xexhrR8f6sHSEQkCwqNiKXnorvhp2IRd1YNTL/w891331GjRg26d+9OYmIiAE5OTgo/kq0oAImIZDHnrsfSY9FerkTHU75wblYNrEe+XE8efu7cucOIESPo0qULkZGRJCUlcfPmzXSoWCTzUQASEclCLtyIo8cXewmLukPZQrlZNbA+nrmdn3i/J0+epH79+sybNw+A119/nZ07d1KoUKEn3rdIZqRRYCIiWcTFm3H0WLSXy1F38CmYi9WD6lHQ/cnDz/Llyxk2bBixsbEUKFCA5cuX065du3SoWCTzUgASEckCLkfepueifVy8eRvvArkIGFSfQu4uT7zfxMREZs6cSWxsLC1atGDlypUUK1YsHSoWydwUgEREMrnwqDv0XLSX8zfiKJnfjdWD6lE4z5OHHwBHR0cCAwP59ttvGT9+PPb21nlavEhmo2HwD6Bh8CKSWVyNvkP3L/ZyJiKWEvlcCXy5AcXzPv5oLMMwWLBgATExMYwbNy4dKxWxvbR8f6sHSEQkk7oWE0+PRXfDT/G8rgQMqv9E4ScyMpKBAweydu1a7OzsaNu2LdWrV0/HikWyDgUgEZFMKOJWPD0X7eX0tViKebgQMKg+Xvkf/6Gje/fupXv37pw7dw5HR0emTZtGtWrV0rFikaxFw+BFRDKZG7EJ9F68j1NXb1EkjwurB9WnpOfjhR+LxcLHH39MkyZNOHfuHD4+PuzevZsxY8bocRaSo6kHSEQkE4mMS6DX4n2cCI+hkLszqwfVo3SBXI+1L8MweOGFF1i3bh0A/v7+LFy4EA8Pj/QsWSRLUg+QiEgmERWXSK/F+zgeFk2B3M6sHlQfn4K5H3t/JpOJ1q1b4+rqyqJFiwgICFD4Efk/GgX2ABoFJiIZLep2In2W7OPwxSg8czmxZnB9yhV2T/N+kpKSuHjxIqVLlwbu9gKdP3+eUqVKpXPFIpmPHoYqIpKFRN9J5MWl+zl8MYr8uZxYPejxws/Fixdp2bIlzZs3T36Gl8lkUvgReQAFIBERG7oVn0S/pfs5dCGSvG6OrBxQjwpF0h5+fvjhB6pXr86OHTu4ceMGR44csUK1ItmHApCIiI3ExifR/8v9/HU+kjwuDqwcUA/fYmm77B4fH8+YMWN47rnnuHHjBn5+fvz11180bdrUSlWLZA8KQCIiNhCXkET/ZX/yZ+hN3F0cWDWwPlWKp+0G5ZCQEBo1asSsWbMAGD16NLt27aJs2bJWqFgke9EweBGRDHY7wcyAZQfYf/YG7s4OrBhQj6ol0j46a+LEiQQFBZE/f36WLVtGhw4drFCtSPakACQikoHuJJoZtPwAe85cJ7ezA18NqEsNr7yPta85c+ZgsViYPn06Xl5e6VuoSDanS2AiIhnkTqKZl1cEsTMkAjcne5b1r0OtkvlSvf3ff//NhAkTuDd7iaenJ2vWrFH4EXkM6gESEckA8Ulmhq36i+3/XMPV0Z4v+9Whdun8qdrWMAwWL17MyJEjuXPnDhUqVKB3795Wrlgke7N5D9D8+fPx9vbGxcUFPz8/duzY8dB1w8LC6NmzJxUqVMDOzo7Ro0fft86yZcswmUz3/dy5c8eKZyEi8nAJSRaGr/qLrSeu4uJox9J+dajn45mqbaOioujRoweDBw/mzp07tGnThtatW1u5YpHsz6YBKDAwkNGjR/P2229z8OBBmjRpQrt27Th//vwD14+Pj6dgwYK8/fbbVK9e/aH7zZMnD2FhYSl+XFxcrHUaIiIPlWi2MGL1X/x2/CrODnYs6VuHBmVSF37+/PNPatWqRWBgIA4ODkybNo2NGzdSqFAhK1ctkv3ZNADNnDmTAQMGMHDgQCpVqsSsWbPw8vJiwYIFD1y/dOnSfPbZZ7z44ouPfJ6NyWSiSJEiKX5ERDJaotnCyICDbDp2BScHOxa9WJtGZQukattFixbRqFEjzpw5Q6lSpdixYwdvvPEGdnY277gXyRZs9i8pISGBoKCg+7pyW7duze7du59o37du3aJUqVKUKFGCZ599loMHDz5y/fj4eKKjo1P8iIg8iSSzhTGBwfz8dzhO9nYs7O1H0/IFU729t7c3SUlJPP/88wQHB1O/fn0rViuS89gsAEVERGA2mylcuHCK5YULFyY8PPyx91uxYkWWLVvGhg0bCAgIwMXFhUaNGnHq1KmHbjN16lQ8PDySfzSiQkSehNli8Oo3h/jxcBiO9ibm96pFi4r/fdnq3vO7AJ5++mn27t3LN998Q968ea1YrUjOlOZRYPHx8ezfv5/Q0FDi4uIoWLAgNWvWxNvb+7EKMJlMKV4bhnHfsrSoX79+iv8pNWrUiFq1ajFnzhxmz579wG3efPNNxo4dm/w6OjpaIUhEHovZYvD6N4dYH3wZBzsT83rW4mnfwo/exmxmypQpfPrppxw4cIAyZcoAULdu3YwoWSRHSnUA2r17N3PmzOH7778nISGBvHnz4urqyo0bN4iPj8fHx4fBgwczZMgQ3N3/+0F+BQoUwN7e/r7enqtXr97XK/Qk7OzsqFOnziN7gJydnXF2dk63Y4pIzmSxGIxfe5h1By9hb2dibs+atK786HsQL1++TO/evfn999+Bu4ND3nrrrYwoVyRHS9UlsI4dO9K1a1eKFy/Or7/+SkxMDNevX+fixYvExcVx6tQp3nnnHbZs2UL58uXZvHnzf+7TyckJPz+/+9bdvHkzDRs2fLyzeQDDMAgODqZo0aLptk8RkX+zWAze+u4I3wRdxN7OxOzuNWlb5dGfO7/88gs1atTg999/J1euXCxfvlzhRySDpKoHqHXr1nzzzTc4OTk98H0fHx98fHzo27cvR48e5fLly6k6+NixY+nTpw+1a9emQYMGfPHFF5w/f54hQ4YAdy9NXbp0ieXLlydvExwcDNy90fnatWsEBwfj5OSEr68vAO+99x7169enXLlyREdHM3v2bIKDg5k3b16qahIRSSvDMJiw/m/W/HkBOxPM7Fad9tUeHn4SEhJ45513mD59OgDVq1cnMDCQChUqZFTJIjleqgLQ8OHDU73DypUrU7ly5VSt6+/vz/Xr15k8eTJhYWFUqVKFjRs3UqpUKeDuxIf/nhOoZs2ayX8OCgpi9erVlCpVitDQUAAiIyMZPHgw4eHheHh4ULNmTf744w9dSxcRqzAMg3c3HGXVvvOYTDCjW3U61ij+yG0WLFiQHH5GjBjB9OnTNVeZSAYzGfceKpNGBw4c4Pjx45hMJipWrEjt2rXTuzabiY6OxsPDg6ioKPLkyWPrckQkkzIMg8k/HuPLXaGYTDC9a3W6+pX4z+0SEhJ47rnnePnll+ncuXMGVCqSM6Tl+zvNo8AuXrxIjx492LVrV/LQzMjISBo2bEhAQIBGT4lIjmAYBlN+Os6Xu0IB+KhL1YeGn9u3bzN37lxGjx6No6MjTk5O/Pzzz0804lVEnkya5wF66aWXSExM5Pjx49y4cYMbN25w/PhxDMNgwIAB1qhRRCRTMQyDj345weKdZwH4sHNV/OuUfOC6J06coH79+rzxxhtMmjQpebnCj4htpbkHaMeOHezevTvFzXoVKlRgzpw5NGrUKF2LExHJbAzD4JNNJ1m4/QwA73esTM9694cfwzD46quvGD58OHFxcRQqVIjmzZtncLUi8jBpDkAlS5YkMTHxvuVJSUkUL/7oG/9ERLK6Wb+dYt7vpwF4t4MvfRqUvm+dmJgYhg0bxsqVKwF46qmnWLlypZ5LKJKJpPkS2Mcff8wrr7zCgQMHuHf/9IEDBxg1ahSffPJJuhcoIpJZzN5yis+23J1U9Z32lejX6P4Z8I8cOYKfnx8rV67E3t6eKVOm8Ouvvyr8iGQyaR4Fli9fPuLi4khKSsLB4W4H0r0/58qVK8W6N27cSL9KM5BGgYnIv837PYTpv54E4K1nKjK4aZkHrnfixAn8/PzInz8/AQEBNG7cOCPLFMnRrDoKbNasWY9bl4hIlrRw++nk8PNG2wr3hZ/ExEQcHR2Buw9k3rBhAzVq1MDT0zPDaxWR1HnseYCyM/UAicg9i3ec4YOfjgPwaqvyvPJUuRTv7969m969e/Pll1/SrFkzW5QoIv8nLd/fab4HSEQkp1i262xy+Bn1VLkU4cdisTB16lSaNm3K2bNneffdd21UpYg8jjRfAhMRyQlW7Anl3R+OATCiRVlGP/3/w8+VK1fo06dP8sOce/bsyeeff26TOkXk8agHSETkX1bvO8+E9UcBGNKsDK+2Lp88ceFvv/1G9erV2bx5M25ubixdupSVK1fi7u5uy5JFJI3UAyQi8j8C/zzPW98dAWBQE2/Gta2QHH7+/PNPWrdujWEYVKlSha+//ppKlSrZslwReUwKQCIi/+ebAxcYv+5u+OnfqDRvPVMpxSMrateuTdeuXcmfPz+ffvoprq6utipVRJ5Qul4Ce+mll1ixYkV67lJEJEN8d/Aib6w9jGFA3walmPisLyaTiZ9++onIyEjg7vO7Vq9ezeeff67wI5LFpWsAOnPmDBMnTqR69erpuVsREataH3yJV78+hGFA7/olefe5yiQkJDBy5EieffZZBg4cmDzz/b0JYEUka0vXf8nbtm0D4OTJk+m5WxERq/nx8GXGBAZjMaBHXS8mP1eFU6dO0b17dw4ePAhA6dKlMZvNCj8i2Ui6/mu+Nxvq/z4pXkQks/r5SBij1twNPy/4lWBKp6qsXr2KoUOHcuvWLTw9Pfnqq69o3769rUsVkXSW6ktgL774ItHR0Q99/8CBA9SsWTNdihIRsbZNR8N5JeAgZotBl1rFmdC2DAMGvESfPn24desWzZo149ChQwo/ItlUqgPQ33//ja+vL7/++muK5YmJibz11ls0bNhQD/0TkSxhy/ErDF/9F0kWg441ijG9a3USE+L57bffsLOzY9KkSWzZsoXixYvbulQRsZJUXwLbv38/kydPpkOHDvTv358ZM2Zw4sQJ+vbtS2xsLD/99BOtWrWyZq0iIk/s95NXGbryLxLNBu2rFuGTrtWwtzPh6elJYGAgCQkJNG/e3NZlioiVpflhqEFBQfTt25cbN24QERFBv379mDFjRraaBVUPQxXJnrb/c41Byw+QkGThKZ9cRG+eR/v27enXr5+tSxORdJCW7+803wTt7OyMo6MjUVFRODk50ahRo2wVfkQke9p5KoLB/xd+arpEsHXqCEJDQ/ntt9/o0qWL/rMjksOk+h4gwzCYOnUqtWvXpkaNGly+fJmPP/6YESNG0LFjR65evWrNOkVEHtvu0xEMXP4ndxKTKBy6iR+nDCQ0NBRvb29+/fVXhR+RHCjVAahBgwbMmTOHb775hi+//BIPDw+GDRvGoUOHiIyMxNfXl8DAQGvWKiKSZvvOXGfAsgPERt3E/PNH7A+cTVJSEi+88AIHDx6kbt26ti5RRGwg1QGodOnS/P3333To0CHFch8fH7Zt28bbb7/NgAED0r1AEZHHdSD0Bv2X/Uls7C1urBzDpSO7cXFxYeHChQQGBuLh4WHrEkXERtJ8E/SjnDp1inLlyqXX7mxGN0GLZH1B527y4pJ9xCaYaVKuAMVD1rNh/fcEBgZStWpVW5cnIlaQlu/vdA1A2YUCkEjWFnwhku4zfyLm9h2a+VVmSd86ONoZxMfHkytXLluXJyJWkpbv71RdAmvbti27d+/+z/ViYmKYNm0a8+bNS12lIiLp7MjFKDqNm82phcO4/fMnzOteFVcnexwcHBR+RCRZqobBv/DCC3Tr1g13d3eee+45ateuTbFixXBxceHmzZscO3aMnTt3snHjRp599lmmT59u7bpFRO5z8Ow1WvcaQsSedQCU8izD7Zgo8rkr+IhISqm+BJaQkMC3335LYGAgO3bsIDIy8u4OTCZ8fX1p06YNgwYNyhYPQtUlMJGs59c9h+j8QjduX/oHgKHDR/DpjE9wdna2cWUiklEy5B6gqKgobt++jaenJ46Ojo9VaGalACSStXy6cBmvjRqOJT4ORzd3li/7ku4vPG/rskQkg1l1Juh7PDw8NIRURGzuxOVI3nn/IyzxceTzqcofP39HlfJlbF2WiGRyaQ5AW7duZd26dYSGhmIymfD29qZr1640bdrUGvWJiDxUyNVb9F56gLzPvk7xczvZEzgfzzyuti5LRLKANF0CGzJkCF988QX58uWjfPnyGIbBqVOniIyMZNiwYcyZM8eatWYYXQITybwMw2Dp0qUcOXmG3R4tuBoTj2/RPKweVI+8bk62Lk9EbMgql8C+++47vvzyS5YuXUrfvn0xmUwAWCwWli1bxtChQ2nVqhXPPffck1UvIvIQ0dHRDBkyhICAAACK9PGkeq3arByo8CMiaZPqAPTll18yduxY+vXrl2K5nZ0dL730EidPnmTJkiUKQCJiFUFBQXTv3p2QkBCwsyNvkxepUr0mqwbWI38uhR8RSZtUPwvsr7/+onPnzg99//nnnycoKChdihIRuccwDD777DMaNGhASEgITnkLUaTnNGp16MvqwQ3xzK1h7iKSdqnuAYqIiKB48eIPfb948eJcv349XYoSEbmnd+/erF69GgDPyo1xfXoE5UoUJmBQfQq6K/yIyONJdQ9QQkICTk4P72Z2cHAgISEhXYoSEbmnTZs2ODk5UbbTSHK1H0eZ4oVYPag+hfK42Lo0EcnC0jQMfsKECbi5uT3wvbi4uHQpSERyNrPZzPnz5/H29gagTadu1DrpRJjZnVL53QgYXJ8iHgo/IvJkUh2AmjZtysmTJ/9zHRGRxxUWFkbv3r05ceIEhw4dwuKUmx6L9hJmdqdEPlcCBtenqIfm+RGRJ5fqALRt2zYrliEiOd2vv/7Kiy++yNWrV3Fzc+P3XftYeDo3Z67FUjyvKwGD6lM8r8KPiKSPVN8D9DBJSUncunUrPWoRkRwoMTGR8ePH07ZtW65evUrVqlX5bftuFp3JTcjVWxT1cCFgUH288j/48ruIyONIdQDauHEjK1asSLFsypQp5M6dm7x589K6dWtu3ryZ7gWKSPYVGhpK06ZNmTZtGgBDhw7l5607eG9nFP9cuUXhPM4EDKpPSU+FHxFJX6kOQJ988gnR0dHJr3fv3s3EiROZMGECX3/9NRcuXOD999+3SpEikj29//777N27Fw8PD7799ls+/GQWg1Yd5kR4DAXd74af0gVy2bpMEcmGUn0P0N9//82MGTOSX3/77be0atWKt99+GwAXFxdGjRrFzJkz079KEcmWZs6cSWxsLFOnTsWSuyAvfL6HU1dvUSD33fDjUzC3rUsUkWwq1T1AMTExeHp6Jr/euXMnLVu2TH5duXJlLl++nL7ViUi2cvLkSd566y3uPYPZw8ODNWvWcOKWC8/N2cmpq7co5O7M6kH1KFtI4UdErCfVAahYsWIcP34cgFu3bnHo0CEaNWqU/P7169cfOkeQiMjy5cvx8/Nj6tSpLF68GIAks4UpPx1j6Kq/iE0wU887Pz+ObEz5wu42rlZEsrtUXwLr2rUro0eP5q233mLjxo0UKVKE+vXrJ79/4MABKlSoYJUiRSTrunXrFsOHD2f58uUAtGjRgvbt23M15g4jVh9k/9kbAAxu6sMbbSrgYP/Eg1NFRP5TqgPQpEmTuHz5MiNHjqRIkSKsXLkSe3v75PcDAgLo0KGDVYoUkazp0KFD+Pv7c/LkSezs7Hj33Xd56623+OtCFMNm7+RaTDy5nR2Y3rUa7aoWtXW5IpKDmIx7F+PTQXR0NHny5Emv3dlMdHQ0Hh4eREVFZYvzEbGFlStXMnDgQOLj4ylevDgBAQE0btyYJTvPMvXnE5gtBuUK5ebzPn6U0c3OIpIO0vL9naZh8P910NatW6d2dyKSzZUuXZqkpCQ6dOjAoUOHqFm3ASMCDvLBT8cxWwyeq16M74c3UvgREZtI9SWwCRMm4OnpSf/+/e9779atW7Rp0ybFPEEikvPcvHmTfPnyAdC4cWN2795NnTp1OH3tFi/P3cnpa7E42Jl4p30l+jYsjclksnHFIpJTpboHaMWKFQwbNozvv/8+xfJbt27RunVrbty4we+//57e9YlIFmCxWJg2bRre3t7Jo0UB6taty09Hwug4dxenr8VSOI8zgS/Xp18jb4UfEbGpVAegrl27MmfOHHr27JkcdG7dukXbtm2JiIhg27ZtFC5c2GqFikjmdPXqVZ555hnGjx9PVFQUq1atAiDRbGHyD8cYsfogsQlm6vvk58dXmuBXKr+NKxYRScMlMICBAwdy48YNOnXqxPr165kwYQLh4eFs376dokU1gkMkp9m6dSu9evUiPDwcV1dXZs+ezYABA7gafYfhq//iz9C7zwcc0qwMr7UuryHuIpJppCkAAbzxxhvcvHmTp556itKlS7N9+3aKFy9ujdpEJJNKSkrivffeY8qUKRiGga+vL19//TWVK1dm35nrDF99kIhb8bg7O/BJt+q0qVzE1iWLiKSQ6gDUpUuXFK8dHR0pUKAAI0eOTLF83bp16VOZiGRaS5cu5YMPPgDu9gx/9tlnuLq6suiPM3z0y90h7hUKu/N5Hz+89TBTEcmEUh2APDw8Urzu0aNHuhcjIlnDSy+9xPr16+nTpw/du3cn5k4iw1b9xc9/hwPQuWZxpnSugptTmjuZRUQyRLpOhJhdaCJEkZTi4+OZM2cOr7zyCs7OzgAYhoHJZOKfKzEMWRnEmWuxONqbmPisL73rl9IoLxHJcGn5/tZ/z0TkkUJCQvD39+evv/7i0qVLfPrppwCYTCY2HLrMuG8PczvRTFEPF+b1qkWtkvlsXLGIyH9TABKRhwoICODll18mJiYGT09PnnrqKQASkix8uPE4y3aHAtCorCezu9fEM7ezDasVEUk9BSARuU9cXBwjR45kyZIlADRt2pRVq1ZRokQJwqPuDnEPOnd3iPvwFmUY26oC9na65CUiWYcCkIikcOLECZ5//nmOHTuGyWRiwoQJTJgwAQcHB3afjmBkwEEibiXg7uLAzG41aOWrCVBFJOtRABKRFJycnLh48SJFixZl1apVtGjRAsMw+Hz7aT7+5QQWAyoWcefz3n6U1hB3EcmiFIBEhMTERBwdHQHw8fFh/fr1+Pr6UqhQIaLvJPLa14fYdOwKAF1qFWdKp6q4OtnbsmQRkSeieelFcrg///yTSpUqsWnTpuRlzZs3p1ChQpwIj6bj3F1sOnYFJ3s7pnSuwowXqiv8iEiWZ/MANH/+fLy9vXFxccHPz48dO3Y8dN2wsDB69uxJhQoVsLOzY/To0Q9cb+3atfj6+uLs7Iyvry/fffedlaoXybosFgszZsygYcOGnD59mgkTJvC/04J9f/ASneft5mxELMU8XPhmSAN61dP8PiKSPdg0AAUGBjJ69GjefvttDh48SJMmTWjXrh3nz59/4Prx8fEULFiQt99+m+rVqz9wnT179uDv70+fPn04dOgQffr0oVu3buzbt8+apyKSpURERNChQwdee+01kpKS6NKlC7/88gsmk4mEJAsT1//N6MBgbieaaVKuAD+ObEJ1r7y2LltEJN3YdCboevXqUatWLRYsWJC8rFKlSnTq1ImpU6c+ctvmzZtTo0YNZs2alWK5v78/0dHR/Pzzz8nL2rZtS758+QgICHjgvuLj44mPj09+HR0djZeXl2aClmxp+/bt9OzZk8uXL+Ps7Mynn37KkCFDMJlMhEXdZtiqvzh4PhKAkS3LMurp8hriLiJZQlpmgrZZD1BCQgJBQUG0bt06xfLWrVuze/fux97vnj177ttnmzZtHrnPqVOn4uHhkfzj5eX12McXycyOHDlCy5YtuXz5MhUqVGDfvn0MHToUk8nErpAI2s/eycHzkeRxcWBpv9qMba35fUQke7LZKLCIiAjMZjOFC6ecQ6Rw4cKEh4c/9n7Dw8PTvM8333yTsWPHJr++1wMkkt1UrVqV3r17Y2dnx9y5c8mVKxcWi8GC7aeZsekkFgN8i+bh895+lPR0s3W5IiJWY/Nh8P++ofLeAxYzcp/Ozs7JD3gUyW5+/fVX/Pz8KFCgAABLlizBweHuP/2o24m8+vUhfjt+d4j7C34leL9TFVwcNcpLRLI3m10CK1CgAPb29vf1zFy9evW+Hpy0KFKkSLrvUyQrSkhI4PXXX6dt27b069cveYTXvfBzPCya5+bu5LfjV3BysOOjLlWZ/kJ1hR8RyRFsFoCcnJzw8/Nj8+bNKZZv3ryZhg0bPvZ+GzRocN8+N23a9ET7FMlqzp49S5MmTfjkk08A8Pb2JikpKfn9tUEX6Tx/F+eux1E8ryvfDmlA97olbVWuiEiGs+klsLFjx9KnTx9q165NgwYN+OKLLzh//jxDhgwB7t6bc+nSJZYvX568TXBwMAC3bt3i2rVrBAcH4+TkhK+vLwCjRo2iadOmTJs2jY4dO7J+/Xp+++03du7cmeHnJ2IL3377LQMHDiQqKoq8efOydOlSOnfuDEB8kpnJPxxj1b67U000K1+QWf41yJfLyZYli4hkOJsGIH9/f65fv87kyZMJCwujSpUqbNy4kVKlSgF3Jz7895xANWvWTP5zUFAQq1evplSpUoSGhgLQsGFD1qxZwzvvvMOECRMoU6YMgYGB1KtXL8POS8QWbt++zdixY/n888+Bu72hAQEByf+eLkXeHeJ+6EIkJhOMbFmOUU+Vw06jvEQkB7LpPECZVVrmERDJLKKioqhZsyahoaGMHz+e9957L/n5XjtOXWNkwEFuxiXi4erIrO41aFGhkI0rFhFJX2n5/rb5KDAReXz3/v9iMpnw8PAgMDCQyMhIWrVqBYDFYjB/WwgzNv+DYUDV4h7M71ULr/wa4i4iOZsCkEgWFRMTw9ChQ2nUqBFDhw4FoE6dOsnvR8UlMvbrYLacuApA9zpevPtcZY3yEhFBl8AeSJfAJLM7ePAg/v7+nDp1ily5cnHu3Dk8PT2T3//7UhTDVv3F+RtxODnY8UHHKnSro8k9RSR70yUwkWzKMAzmzp3La6+9RkJCAl5eXgQEBCSHH4vFYPHOM0z/9SSJZgOv/K4s6OVHleIeNq5cRCRzUQASySJu3LjBSy+9xPr16wF47rnn+PLLL8mfPz8AYVG3efXrQ+w+fR2AVr6F+aRrdTzcHG1Ws4hIZqUAJJIF3L59m9q1a3P27FmcnJyYPn06r7zySvIjXjYeCePNdUeIup2Iq6M9kzr44l/H64kfKyMikl0pAIlkAa6urgwYMIBly5YRGBhIrVq1ALgVn8S7G47ybdBFAKqV8GCWfw18Cua2ZbkiIpmeboJ+AN0ELZlBeHg4t27domzZsgCYzWbi4uJwd3cHIOjcTcYEBnP+Rhx2JhjWvCyjni6Ho73NnnAjImJTuglaJIvbvHkzvXv3plChQuzfvx9XV1fs7e1xd3cnyWxhztYQ5v4egtliUDyvK5/616Cud35bly0ikmXov4oimUhiYiJvvfUWbdq04erVq5hMJq5du5b8/vnrcXRbuIfPtpzCbDHoVKMYP49uovAjIpJG6gESySTOnTtHz5492b17NwBDhgxh5syZuLq6YhgG3wZd5N0NR4lNMOPu7MAHnavQsUZxG1ctIpI1KQCJZALff/89/fv3JzIykjx58rB48WJeeOEFACLjEnj7u7/56UgYAHVL52emf3VK5NPjLEREHpcCkIiNWSwWZs6cSWRkJHXq1GHNmjX4+PgAsDskgrFfHyI8+g4OdibGtCrPkGZlsNcT3EVEnogCkIiN2dnZsWrVKhYuXMjEiRNxcnIiPsnMjE3/sGjHGQwDfArkYlb3GlQrkdfW5YqIZAsaBv8AGgYv1rZy5UqOHz/OlClT7nsv5GoMIwOCORYWDUCPuiWZ8Gwl3Jz0/xURkUfRMHiRTCo2NpYRI0awbNkyAFq3bk2zZs2Au8/5WrH3HFN+Ok58koX8uZz4qEtVWlcuYsOKRUSyJwUgkQxy+PBh/P39OXHiBHZ2dkycOJHGjRsDcC0mnje+PcTvJ+8OeW9aviCfdK1GoTwutixZRCTbUgASsTLDMFi4cCGjR48mPj6eYsWKsXr16uSeny3Hr/DGt4e5HpuAk4Mdb7arSN8GpbHTjc4iIlajACRiZYMGDWLJkiUAPPPMMyxbtoyCBQtyO8HMlI3HWLn3PAAVi7jzWfeaVCjibstyRURyBM0ELWJlbdq0wdHRkRkzZvDDDz9QsGBB/r4UxbNzdiSHn4GNvfl+eCOFHxGRDKIeIJF0ZrFYOHfuHN7e3gC88MIL1K1bl1KlSmG2GHy+/TQzNp0k0WxQyN2ZGd2q06RcQRtXLSKSsygAiaSja9eu0bdvX4KCgjh06BBFitwdwVWqVCkuR95m7NfB7D1zA4C2lYswtUtV8uVysmXJIiI5ki6BiaST33//nerVq/Pzzz8THR3NgQMHkt/74dBl2s76g71nbuDmZM/Hz1djQe9aCj8iIjaiHiCRJ5SUlMT777/P+++/j2EYVKpUicDAQKpWrUrMnUQmbTjKur8uAVDdKy+z/GvgXSCXjasWEcnZFIBEnsDFixfp1asXf/zxBwADBgzgs88+I1euXASdu8HowGAu3LiNnQlGtCjLK0+Vw9FeHa8iIramACTyBD766CP++OMPcufOzRdffEGPHj1IMluYufkf5m49hcWAEvlc+dS/BnVK57d1uSIi8n8UgESewEcffcT169d5//33KVu2LKERsYwODCb4QiQAXWoW592Olcnj4mjbQkVEJAX1xYukwenTp3njjTewWCwA5M6dm4CAAMqUKcPXf17gmdk7CL4QibuLA3N61GSmfw2FHxGRTEg9QCKpFBgYyKBBg4iJiaF48eKMGjUKgJuxCbz13RF+/jscgHre+ZnpX4PieV1tWa6IiDyCApDIf4iLi2P06NEsWrQIgMaNG9OlSxcAdp6K4NVvgrkSHY+DnYlXW1dgcFMf7PUcLxGRTE0BSOQRjh07Rrdu3Th69Cgmk4m3336bSZMmYcbElJ+OsWjHWQB8CubiM/+aVC3hYeOKRUQkNRSARB7i66+/pl+/fty+fZsiRYqwcuVKnnrqKf65EsOoNcEcD4sGoFe9krzT3hdXJ3sbVywiIqmlACTyEN7e3iQmJtK6dWuWL19OoUKFWLbrLFN/PkF8kgXPXE5Me74aT/sWtnWpIiKSRgpAIv/j5s2b5MuXD4A6deqwa9cuateuTURsAv2X/cm2k9cAaF6hIB93rUYhdxdblisiIo9Jw+BFAMMwmDVrFqVKlSI4ODh5ed26ddl64hptZ+1g28lrODvY8d5zlfmyXx2FHxGRLEw9QJLjXb9+nX79+vHjjz8CsGLFCmrUqEHU7UQ++vk4AfsvAFCpaB4+616D8oXdbVmuiIikAwUgydF27NhBz549uXjxIk5OTsycOZNhw4bxy9/hTFz/N1dj4gEY3NSHV1uXx9lBNzqLiGQHCkCSI5nNZqZOncqkSZOwWCyUL1+ewMBAivpUZOjKv/jl6N1JDX0K5GJql6rU8/G0ccUiIpKedA+Q5EirV69mwoQJWCwW+vTpw4EDBziekJ+nZ27nl6PhONiZGNGiLBtHNVH4ERHJhtQDJDlSz549Wbt2LV26dKFxuy4MXH2EfWdvAFC9hAcfPV+NSkXz2LhKERGxFvUASY6QmJjIjBkzuH37NgD29vZ8/e1aYrwa0vazHew7ewNXR3smPOvLumGNFH5ERLI59QBJthcaGkqPHj3Yu3cv//zzDwsXLuTwxUje+PYwJ8JjAGhSrgAfdq6KV343G1crIiIZQQFIsrV169YxYMAAIiMj8fDwoFnLp/jgx2Ms3XUWiwH53ByZ2MGXTjWKYzLpAaYiIjmFApBkS3fu3OHVV19l/vz5ANSvX5/RU+Yy588oLt68+wDTjjWKMfFZXzxzO9uyVBERsQEFIMl2QkJC6Nq1K4cOHQJg1NjXMPy6MW7T3aHtxfO68kHnKrSoUMiWZYqIiA0pAEm24+zszIULFyhYsCDD3v2UH64X4vrhq5hM0K9haV5rXYFczvrVFxHJyfQtINlCYmIijo6OAHh5ebFoRSCBJxNYdt4AEqhQ2J2Pnq9KzZL5bFuoiIhkChoGL1lecHAwVatWZcOGDZgtBst2neWdvUnsu2LgZG/Hq63K88MrjRV+REQkmXqAJMsyDIP58+fz6quvEh8fzxtvvs3S8/kIvhgNQJ3S+ZjapRplC+W2caUiIpLZKABJlnTz5k0GDBjAd999B4BvvRbcbjCY4IvR5HZ2YFy7ivSqWxI7Ow1tFxGR+ykASZazZ88eunfvzvnz53FwdMT7mcHcqtAWk8nE05UK836nyhT1cLV1mSIikokpAEmW8s8//9C0aVOSkpLIV8QL59ZjSShajoK5nZncsTLtqhTRhIYiIvKfFIAkSylfvjxtuvRk74kLuLUcip2zG/61vXjrmUp4uDnaujwREckiFIAk09u6dSu+vr7Y58rHuz8c5Uip58lV2o7SBXIxtXNVGpYtYOsSRUQki1EAkkwrKSmJd999lw8//JDKtRtiavc20fEWHBwcGNTEh9FPl8PF0d7WZYqISBakACSZ0oULF+jRowe7du0CIDQhN/lj46laypOPulSjSnEPG1coIiJZmQKQZDobNmygX79+3Lx5EzsnV/K3fQXPas0Z26o8LzXyxsFe83eKiMiTUQCSTCM+Pp433niD2bNnA+BUpBwFnnuD5nWq8GHnqpTyzGXjCkVEJLtQAJJMIzr2DqvX/QCAe+2OlGo7iIkdq9HVr4SGtouISLpSABKbMwyDPaev8+Z3R3BsNYaC0RF069KRSR0qU9Dd2dbliYhINqQAJDYTGxvLkGEjuEh+zhZtCUCpcr6837EKT/sWtnF1IiKSnSkAiU0cOXKE9p2e58KZU2DvSPEhNXipVU1eb1MBdxdNaCgiItalACQZyjAMPpk9jzdffw1zYjz2ufNTrc8EvnitPbVL57d1eSIikkMoAEmGuXkzkrYv9Gb/lp8AcPPxY9zUOYzrUhdnB01oKCIiGUcBSDLEsYvXqV/bj5gr58DOnsodX+bbeR9SsagmNBQRkYynACRWFXMnkbm/h/DlzlAcfJ/GMf5Hxn+8gEkDOmFvp6HtIiJiGzafUnf+/Pl4e3vj4uKCn58fO3bseOT627dvx8/PDxcXF3x8fPj8889TvL9s2TJMJtN9P3fu3LHmaci/WCwGizcF02DcShZuP0OC2ULHPoM5fOgQkwd1VvgRERGbsmkACgwMZPTo0bz99tscPHiQJk2a0K5dO86fP//A9c+ePcszzzxDkyZNOHjwIG+99RYjR45k7dq1KdbLkycPYWFhKX5cXFwy4pQEOBB6g0aj5zLk+ac4uXICXu52LO1Xm6X96lCxpIa3i4iI7ZkMwzBsdfB69epRq1YtFixYkLysUqVKdOrUialTp963/rhx49iwYQPHjx9PXjZkyBAOHTrEnj17gLs9QKNHjyYyMvKx64qOjsbDw4OoqCjy5Mnz2PvJaS5H3ubDn46ycsFMonYHgmGhcEkfft/0C5UqlLN1eSIiks2l5fvbZj1ACQkJBAUF0bp16xTLW7duze7dux+4zZ49e+5bv02bNhw4cIDExMTkZbdu3aJUqVKUKFGCZ599loMHDz6ylvj4eKKjo1P8SOrdTjAz67d/aDLpWxa/2Z+oXQFgWOjeqw+njx1W+BERkUzHZgEoIiICs9lM4cIpL4kULlyY8PDwB24THh7+wPWTkpKIiIgAoGLFiixbtowNGzYQEBCAi4sLjRo14tSpUw+tZerUqXh4eCT/eHl5PeHZ5QyGYbDh0GWemrGNqQtXE7poBPEX/sYtV25WrFhBwMrl5MqlB5iKiEjmY/NRYP9+yKVhGI988OWD1v/f5fXr16d+/frJ7zdq1IhatWoxZ86c5KeM/9ubb77J2LFjk19HR0crBP2HIxejeO+Hoxw4dxPDMIgP/hHL7Whq1qzJmjVrKF++vK1LFBEReSibBaACBQpgb29/X2/P1atX7+vluadIkSIPXN/BwQFPT88HbmNnZ0edOnUe2QPk7OyMs7Meupka12Li+eTXk3wddAHDABdHO4Y1L8tzI9azcP5c3n33XbWliIhkejYLQE5OTvj5+bF582Y6d+6cvHzz5s107Njxgds0aNCAH374IcWyTZs2Ubt2bRwdH/z8KMMwCA4OpmrVqulXfA4Un2Rm2a5Q5mwN4VZ8ErEndlLSfIkfViykWF5XgAfeuC4iIpIZ2fQS2NixY+nTpw+1a9emQYMGfPHFF5w/f54hQ4YAdy9NXbp0ieXLlwN3R3zNnTuXsWPHMmjQIPbs2cOSJUsICAhI3ud7771H/fr1KVeuHNHR0cyePZvg4GDmzZtnk3PM6gzDYMvxq3zw0zFCr8dhSYzHbv9yInauJwI4sq8Xxdq0sXWZIiIiaWLTAOTv78/169eZPHkyYWFhVKlShY0bN1KqVCkAwsLCUswJ5O3tzcaNGxkzZgzz5s2jWLFizJ49m+effz55ncjISAYPHkx4eDgeHh7UrFmTP/74g7p162b4+WV1/1yJ4f0fj7Hj1N0bzHPfDidm4yecCzmByWTizTffpGXLljauUkREJO1sOg9QZpXT5wGKjEvg083/sHLfecwWA0c7E9VvH+TnLz4kLi6OwoULs2LFClq1amXrUkVERJKl5fvb5qPAJPNIMltYvf88Mzf/Q2Tc3XmVWvsWJmHHYr5avBCAVq1asXz5cooUKWLLUkVERJ6IApAAsPNUBJN/PMo/V24BUKGwOxM7+NKobAF+yBfGqmVLmDx5MuPGjcPOzuaPkBMREXkiCkA53LnrsXzw03E2H7sCQF43R8Y+XY76Bc2UL1sAgA4dOhASEpJ8b5aIiEhWpwCUQ92KT2Lu1hCW7jxLgtmCvZ2JPvVL0beWJ2NGDGHMjh0EBwcnTwip8CMiItmJAlAOY7EYfPvXRT7+5SQRt+IBaFKuABOf9eXKqUM0a9iOCxcu4OTkxP79+zUjtoiIZEsKQDlI0LkbvLvhGEcuRQFQ2tONCc/60qycJ9OmTWPSpEmYzWbKli1LYGAgtWrVsnHFIiIi1qEAlANcjrzNtF9OsD74MgC5nR0Y+VRZ+jYszc2Ia7Rt25YtW7YA0KtXLxYsWIC7u7stSxYREbEqBaBs7vDFSHot3kfMnSRMJujm58VrbSpQ0P3u87pmzJjBli1bcHNzY+7cufTr1++RD6MVERHJDhSAsrFjl6Pps2Q/MXeSqFbCgw87V6VKcY8U60yePJmLFy8yceJEKlWqZKNKRUREMpYmdMmmTl2JofeSfUTdTqRmybysHlSfKsU9OHfuHK+++ipmsxkAV1dXAgICFH5ERCRHUQ9QNnTm2i16Lt7HjdgEqhb3YFn/uuR2duD777+nf//+REZGUrBgQcaPH2/rUkVERGxCPUDZzPnrcfRctI9rMfFULOLOigF1cTaZeeWVV+jcuTORkZHUrVsXf39/W5cqIiJiMwpA2cilyNv0WLSX8Og7lCuUm1UD63H1YigNGjRg7ty5ALz22mvs2LEDb29vG1crIiJiO7oElk1cib5Dz0V7uRR5G+8CuVg1sB67tv5Kz549iY2NpUCBAixfvpx27drZulQRERGbUwDKBq7FxNNz0V7OXY/DK78rqwfVo1AeF0qXLk1SUhLNmzdn1apVFCtWzNalioiIZAoKQFncjdgEei/ex+lrsRTzcGF+1woU9XAFoFq1auzYsYNatWphb29v40pFREQyD90DlIVF3U6kz5J9nLwSQ8HcTrS1P0yjGpXYv39/8jp16tRR+BEREfkXBaAsKuZOIn2X7ufo5Wjy2ifguW8ek8aNJSYmhmXLltm6PBERkUxNl8CyoLiEJF5a9ifBFyJxvB7CtV9mcujieRwdHfnoo48YM2aMrUsUERHJ1BSAspg7iWYGfnWA/WevE//Xei5t+4qkpCR8fHxYs2YNderUsXWJIiIimZ4ugWUh8UlmBq8IYvfp61hO7+XKb0tISkrC39+fv/76S+FHREQkldQDlEUkJFkYvuov/vjnGq6O9iz7eCwz7E7RqlUrBg4cqCe4i4iIpIECUBaQZLYwcvWfrFu+GE+/dizp25R6Pp4EBgYq+IiIiDwGBaBMzmwxGPz5JlZPe534C39Txz2ahmW7ACj8iIiIPCYFoEzMYjHo9vZcvv/sHSy3o3HLlZt+/p1sXZaIiEiWpwCUScXHx9P0hYHs/2ElAGV9q/Hz+rWULVvWxpWJiIhkfRoFlgmdPXsWnyp+yeGnQ8+B/P3XfoUfERGRdKIAlMkYhsHi3RcJv3wJOxd33pixhA2rFuHs7Gzr0kRERLINXQLLJBITE3F0dOSzLadYdSSaQl3eZlzXRozu2MDWpYmIiGQ7CkCZwNGjR/H396fe84PZEl8GgPcHd2FgEx8bVyYiIpI96RKYDRmGwaJFi6hTpw5Hjx5l+dzpGBYzb7StoPAjIiJiReoBspHo6GgGDx5MYGAgAC6la1Lg2bGMblWRYc11s7OIiIg1KQDZwIEDB/D39+fMmTPY2duTp3Ef8tTrwrAW5Rj9dDlblyciIpLtKQBlsHPnztGoUSMSEhIoWLQE9k+PwrlYJV5q5M0bbSpodmcREZEMoACUwUqVKsWwYcPYd/gkl6r1xeScm971SzLh2UoKPyIiIhlEN0HbQJv+r3Gl7nBMzrl5wa8Ek5+rovAjIiKSgRSAMti2k1cZGXgYswEdaxTjo+erYWen8CMiIpKRFIAy0O7TEby8IogEs4V2VYow44Xq2Cv8iIiIZDjdA5SBPHM54+7iQBOvvHzWvSYO9sqfIiIitqAAlIEqFHHnu2GNKJTHGScHhR8RERFbUQDKYF753WxdgoiISI6nbggRERHJcRSAREREJMdRABIREZEcRwFIREREchwFIBEREclxFIBEREQkx1EAEhERkRxHAUhERERyHAUgERERyXEUgERERCTHUQASERGRHEcBSERERHIcBSARERHJcfQ0+AcwDAOA6OhoG1ciIiIiqXXve/ve9/ijKAA9QExMDABeXl42rkRERETSKiYmBg8Pj0euYzJSE5NyGIvFwuXLl3F3d8dkMtm6nP8UHR2Nl5cXFy5cIE+ePLYuJ9NQuzyc2ubh1DYPp7Z5OLXNw2Vk2xiGQUxMDMWKFcPO7tF3+agH6AHs7OwoUaKErctIszx58ugf3gOoXR5ObfNwapuHU9s8nNrm4TKqbf6r5+ce3QQtIiIiOY4CkIiIiOQ4CkDZgLOzM5MmTcLZ2dnWpWQqapeHU9s8nNrm4dQ2D6e2ebjM2ja6CVpERERyHPUAiYiISI6jACQiIiI5jgKQiIiI5DgKQCIiIpLjKABlQvPnz8fb2xsXFxf8/PzYsWPHI9ffvn07fn5+uLi44OPjw+eff57i/UWLFtGkSRPy5ctHvnz5ePrpp9m/f781T8Fq0rtt/teaNWswmUx06tQpnavOGNZom8jISIYPH07RokVxcXGhUqVKbNy40VqnYDXWaJtZs2ZRoUIFXF1d8fLyYsyYMdy5c8dap2AVaWmXsLAwevbsSYUKFbCzs2P06NEPXG/t2rX4+vri7OyMr68v3333nZWqt670bpuc+jmc2t+bezL0c9iQTGXNmjWGo6OjsWjRIuPYsWPGqFGjjFy5chnnzp174Ppnzpwx3NzcjFGjRhnHjh0zFi1aZDg6Ohrffvtt8jo9e/Y05s2bZxw8eNA4fvy40b9/f8PDw8O4ePFiRp1WurBG29wTGhpqFC9e3GjSpInRsWNHK59J+rNG28THxxu1a9c2nnnmGWPnzp1GaGiosWPHDiM4ODijTitdWKNtVq5caTg7OxurVq0yzp49a/z6669G0aJFjdGjR2fUaT2xtLbL2bNnjZEjRxpfffWVUaNGDWPUqFH3rbN7927D3t7e+PDDD43jx48bH374oeHg4GDs3bvXymeTvqzRNjn1czg1bXNPRn8OKwBlMnXr1jWGDBmSYlnFihWN8ePHP3D9N954w6hYsWKKZS+//LJRv379hx4jKSnJcHd3N7766qsnLzgDWattkpKSjEaNGhmLFy82+vbtmyUDkDXaZsGCBYaPj4+RkJCQ/gVnIGu0zfDhw42WLVumWGfs2LFG48aN06lq60tru/yvZs2aPfCLrFu3bkbbtm1TLGvTpo3RvXv3J6o1o1mjbf4tp3wO/69HtY0tPod1CSwTSUhIICgoiNatW6dY3rp1a3bv3v3Abfbs2XPf+m3atOHAgQMkJiY+cJu4uDgSExPJnz9/+hSeAazZNpMnT6ZgwYIMGDAg/QvPANZqmw0bNtCgQQOGDx9O4cKFqVKlCh9++CFms9k6J2IF1mqbxo0bExQUlHwJ48yZM2zcuJH27dtb4SzS3+O0S2o8rO2eZJ8ZzVpt82855XM4tWzxOayHoWYiERERmM1mChcunGJ54cKFCQ8Pf+A24eHhD1w/KSmJiIgIihYtet8248ePp3jx4jz99NPpV7yVWattdu3axZIlSwgODrZW6VZnrbY5c+YMW7dupVevXmzcuJFTp04xfPhwkpKSmDhxotXOJz1Zq226d+/OtWvXaNy4MYZhkJSUxNChQxk/frzVziU9PU67pMbD2u5J9pnRrNU2/5ZTPodTw1afwwpAmZDJZErx2jCM+5b91/oPWg7w8ccfExAQwLZt23BxcUmHajNWerZNTEwMvXv3ZtGiRRQoUCD9i81g6f17Y7FYKFSoEF988QX29vb4+flx+fJlpk+fnmUC0D3p3Tbbtm1jypQpzJ8/n3r16hESEsKoUaMoWrQoEyZMSOfqrSet7WKrfdqCNc8jp30OP4otP4cVgDKRAgUKYG9vf1+Svnr16n2J+54iRYo8cH0HBwc8PT1TLP/kk0/48MMP+e2336hWrVr6Fm9l1mibo0ePEhoaSocOHZLft1gsADg4OHDy5EnKlCmTzmeS/qz1e1O0aFEcHR2xt7dPXqdSpUqEh4eTkJCAk5NTOp9J+rNW20yYMIE+ffowcOBAAKpWrUpsbCyDBw/m7bffxs4uc99d8DjtkhoPa7sn2WdGs1bb3JPTPof/y+nTp232OZy5/5XmME5OTvj5+bF58+YUyzdv3kzDhg0fuE2DBg3uW3/Tpk3Url0bR0fH5GXTp0/n/fff55dffqF27drpX7yVWaNtKlasyJEjRwgODk7+ee6552jRogXBwcF4eXlZ7XzSk7V+bxo1akRISEjyhxHAP//8Q9GiRbNE+AHrtU1cXNx9Icfe3h7j7sCSdDwD63icdkmNh7Xdk+wzo1mrbSBnfg7/F5t+Dlv9NmtJk3tDDJcsWWIcO3bMGD16tJErVy4jNDTUMAzDGD9+vNGnT5/k9e8N2R0zZoxx7NgxY8mSJfcN2Z02bZrh5ORkfPvtt0ZYWFjyT0xMTIaf35OwRtv8W1YdBWaNtjl//ryRO3duY8SIEcbJkyeNH3/80ShUqJDxwQcfZPj5PQlrtM2kSZMMd3d3IyAgwDhz5oyxadMmo0yZMka3bt0y/PweV1rbxTAM4+DBg8bBgwcNPz8/o2fPnsbBgweNo0ePJr+/a9cuw97e3vjoo4+M48ePGx999FGWHgafnm2TUz+HDeO/2+bfMupzWAEoE5o3b55RqlQpw8nJyahVq5axffv25Pf69u1rNGvWLMX627ZtM2rWrGk4OTkZpUuXNhYsWJDi/VKlShnAfT+TJk3KgLNJX+ndNv+WVQOQYVinbXbv3m3Uq1fPcHZ2Nnx8fIwpU6YYSUlJ1j6VdJfebZOYmGi8++67RpkyZQwXFxfDy8vLGDZsmHHz5s0MOJv0k9Z2edDnSKlSpVKs88033xgVKlQwHB0djYoVKxpr167NgDNJf+ndNjn5czg1vzf/K6M+h03/V5yIiIhIjqF7gERERCTHUQASERGRHEcBSERERHIcBSARERHJcRSAREREJMdRABIREZEcRwFIREREchwFIBEREclxFIBEJEcLDQ2ldOnSbN261daliEgGUgASkSzpjz/+oEOHDhQrVgyTycT333//wPWaN2/O559//tD9rFixgrlz59KyZUu2bduGyWSiSpUqmM3mFOvlzZuXZcuWpeMZiIgtKQCJSJYUGxtL9erVmTt37kPXuXHjBrt376ZDhw4PXWfTpk20bds2xbLTp0+zfPnydKtVRDIfB1sXICLyONq1a0e7du0euc5PP/1E9erVKV68+APf37t3L7Vr18bBIeVH4SuvvMKkSZPo0aMHLi4u6VaziGQe6gESkWxrw4YNdOzYMfl1WFhYiveXL19O375979tu9OjRJCUlPbJ3SUSyNgUgEcmW4uPj+fXXX+nYsSMWi4Xhw4dTsmRJLl26BEBCQgInTpygRo0a923r5ubGpEmTmDp1KlFRURlcuYhkBAUgEcmWtm7diqenJ1WrVsXOzo558+ZRv359Vq9eDdy9PPbss88+dPsBAwZQoEABpk2bllEli0gGUgASkWzp35e/APr06cPKlSsBWL16NT179nzo9g4ODnzwwQd89tlnXL582aq1ikjGUwASkWzHMAx++OEHnnvuuRTLu3XrxsmTJ9m2bRtms5kiRYo8cj8vvPAClStX5r333rNmuSJiAxoFJiJZ0q1btwgJCUl+ffbsWYKDg8mfPz9Xr14lNjaWpk2bptgmb968PPPMM7z44ot88sknqTrORx99RJs2bdK1dhGxPQUgEcmSDhw4QIsWLZJfjx07FoC+ffvi5eVF+/bt7xveDtC7d29+//33+3qHHqZly5a0bNmSTZs2pU/hIpIpmAzDMGxdhIhIeqpWrRrvvPMO3bp1u++9hIQElixZwtChQ21QmYhkFgpAIpKtJCQkMHXqVMaOHYu7u7utyxGRTEoBSERERHIcjQITERGRHEcBSERERHIcBSARERHJcRSAREREJMdRABIREZEcRwFIREREchwFIBEREclxFIBEREQkx1EAEhERkRzn/wGBT/cVWtMc5QAAAABJRU5ErkJggg==\n", 847 | "text/plain": [ 848 | "
" 849 | ] 850 | }, 851 | "metadata": {}, 852 | "output_type": "display_data" 853 | } 854 | ], 855 | "source": [ 856 | "import matplotlib.pyplot as plt\n", 857 | "plt.plot(1 / jnp.sqrt(jnp.array(sample_sizes)), ksd_vals, label=\"KSD\")\n", 858 | "\n", 859 | "coef = np.polyfit(1 / jnp.sqrt(jnp.array(sample_sizes)),ksd_vals,1)\n", 860 | "poly1d_fn = np.poly1d(coef) \n", 861 | "# poly1d_fn is now a function which takes in x and returns an estimate for y\n", 862 | "\n", 863 | "plt.plot(1 / jnp.sqrt(jnp.array(sample_sizes)), poly1d_fn(jnp.array(1 / jnp.sqrt(jnp.array(sample_sizes)))), '--k', label=\"linear fit\")\n", 864 | "plt.xlabel(\"1/√N\")\n", 865 | "plt.ylabel(\"KSD(X, p)\")\n", 866 | "plt.legend()" 867 | ] 868 | }, 869 | { 870 | "cell_type": "markdown", 871 | "id": "1e11843b-d02a-4a33-b822-c0d7369ee2bb", 872 | "metadata": {}, 873 | "source": [ 874 | "## Randomness in JAX" 875 | ] 876 | }, 877 | { 878 | "cell_type": "markdown", 879 | "id": "65142534-1c18-4e40-af17-d611499a1da3", 880 | "metadata": {}, 881 | "source": [ 882 | "### Description " 883 | ] 884 | }, 885 | { 886 | "cell_type": "markdown", 887 | "id": "9a9a720f-6f12-48ff-aeeb-c3c2c9a9ccbb", 888 | "metadata": {}, 889 | "source": [ 890 | "- So far, we have conveniently evaded the notion of randomness in Jax by delegating draws from probability distributions to numpy.\n", 891 | "- Random variables can be drawn from jax. However, the handling of randomness is an example of the functionalization constraint that jax imposes.\n", 892 | "- his constrain prevents numpy-like random variables drawing, since drawing a random variable in numpy incurs a side effect by modifying numpy's random state:" 893 | ] 894 | }, 895 | { 896 | "cell_type": "code", 897 | "execution_count": 31, 898 | "id": "93f314aa-5ec6-4f67-9e70-e1215e7ff5d6", 899 | "metadata": {}, 900 | "outputs": [], 901 | "source": [ 902 | "def draw_one_random_variable():\n", 903 | " return np.random.randn()" 904 | ] 905 | }, 906 | { 907 | "cell_type": "code", 908 | "execution_count": 32, 909 | "id": "36bdb6a9-e92c-474d-b846-8f97d462d1fa", 910 | "metadata": {}, 911 | "outputs": [ 912 | { 913 | "name": "stdout", 914 | "output_type": "stream", 915 | "text": [ 916 | "-0.07537296806232933 1.6193415953720864\n" 917 | ] 918 | } 919 | ], 920 | "source": [ 921 | "print(draw_one_random_variable(), draw_one_random_variable()) # two different outputs for the same input (Φ): " 922 | ] 923 | }, 924 | { 925 | "cell_type": "markdown", 926 | "id": "8c2781b0-d006-4f43-b51f-682286dd4dac", 927 | "metadata": {}, 928 | "source": [ 929 | "`RandomState` Based randomness generation is also forbidden in functional programming frameworks, since it modifies its input:" 930 | ] 931 | }, 932 | { 933 | "cell_type": "code", 934 | "execution_count": 33, 935 | "id": "90a08a0c-b2c7-44f5-a9a3-70e4dbe72bbe", 936 | "metadata": {}, 937 | "outputs": [], 938 | "source": [ 939 | "def draw_one_random_variable_RandomState(rs):\n", 940 | " return rs.randn()" 941 | ] 942 | }, 943 | { 944 | "cell_type": "code", 945 | "execution_count": 34, 946 | "id": "a1655007-e9ce-40ed-a57e-a49c2de6c981", 947 | "metadata": {}, 948 | "outputs": [ 949 | { 950 | "data": { 951 | "text/plain": [ 952 | "False" 953 | ] 954 | }, 955 | "execution_count": 34, 956 | "metadata": {}, 957 | "output_type": "execute_result" 958 | } 959 | ], 960 | "source": [ 961 | "rs = np.random.RandomState(42)\n", 962 | "orig_state = rs.get_state()\n", 963 | "\n", 964 | "draw_one_random_variable_RandomState(rs)\n", 965 | "np.allclose(rs.get_state()[1], orig_state[1])" 966 | ] 967 | }, 968 | { 969 | "cell_type": "markdown", 970 | "id": "bcb152e1-01a9-4689-9bcd-64eef2d61a6c", 971 | "metadata": {}, 972 | "source": [ 973 | "Instead, `jax` generates random variables in a functionally pure manner by:\n", 974 | "- requiring a one-off `PRNGKey`, a `RandomState` equivalent characterizing some pseudorandomness state.\n", 975 | "- not alterating this key when generating a random variable\n", 976 | "\n" 977 | ] 978 | }, 979 | { 980 | "cell_type": "markdown", 981 | "id": "c7033363-a038-487e-bf3d-b9775c409a17", 982 | "metadata": {}, 983 | "source": [ 984 | "- The consequence of this pure randomness generation is that new keys must be repeteadly generated to \"create\" new randomness.\n", 985 | "- Generating new keys from old ones is known in the computational pseudorandomness literrature as splitting, for which a utility function is provided in jax." 986 | ] 987 | }, 988 | { 989 | "cell_type": "code", 990 | "execution_count": 35, 991 | "id": "09d05b35-7018-4530-9b44-e66f31e08f50", 992 | "metadata": {}, 993 | "outputs": [], 994 | "source": [ 995 | "# key \n", 996 | "# | \n", 997 | "# | random.split(key)\n", 998 | "# | \n", 999 | "# ---------------------------\n", 1000 | "# | |\n", 1001 | "# | |\n", 1002 | "# | |\n", 1003 | "# subkey1 subkey2\n", 1004 | "# | \n", 1005 | "# random.split(subkey1) | | random.split(subkey2)\n", 1006 | "# | | \n", 1007 | "# -------------- --------------\n", 1008 | "# | | | |\n", 1009 | "# | | | |\n", 1010 | "# | | | |\n", 1011 | "# subkey1.1 subkey1.2 subkey2.1 subkey2.2\n", 1012 | "#" 1013 | ] 1014 | }, 1015 | { 1016 | "cell_type": "code", 1017 | "execution_count": 36, 1018 | "id": "bb7037e4-253e-47e2-b794-d4b6cb7b96ab", 1019 | "metadata": {}, 1020 | "outputs": [], 1021 | "source": [ 1022 | "from jax import random\n", 1023 | "key = random.PRNGKey(42)\n", 1024 | "\n", 1025 | "key, subkey = random.split(key)\n", 1026 | "# provide a one-shot \"RandomState\"-like subkey to generate a random variable.\n", 1027 | "x1 = random.normal(subkey)\n", 1028 | "\n", 1029 | "# From now on, subkey cannot be used since it was used to generate randomness in random.normal\n", 1030 | "x1_same = random.normal(subkey)\n", 1031 | "assert x1_same == x1\n", 1032 | "\n", 1033 | "# You're left with an \"untouched\" new key that can be split to generate new randomness\n", 1034 | "key, subkey = random.split(subkey)\n", 1035 | "\n", 1036 | "x2 = random.normal(subkey)\n", 1037 | "assert x2 != x1" 1038 | ] 1039 | }, 1040 | { 1041 | "cell_type": "markdown", 1042 | "id": "42a1af8c-eec9-4e90-8a78-37651b0b6600", 1043 | "metadata": {}, 1044 | "source": [ 1045 | "### Exercise 3: re-plot the KSD values obtained from your previous exercise, but by generating random jax arrays natively this time!\n" 1046 | ] 1047 | }, 1048 | { 1049 | "cell_type": "code", 1050 | "execution_count": 37, 1051 | "id": "61430432-a145-4e2f-a6ab-301c13f4ddf5", 1052 | "metadata": {}, 1053 | "outputs": [], 1054 | "source": [ 1055 | "from jax import random\n", 1056 | "\n", 1057 | "sample_sizes = (50, 100, 200, 300, 500, 1000, 2000, 5000)\n", 1058 | "\n", 1059 | "key = random.PRNGKey(0)\n", 1060 | "\n", 1061 | "ksd_vals = []\n", 1062 | "for N in sample_sizes:\n", 1063 | " ksd_vals_this_iter = []\n", 1064 | " for rs in range(5):\n", 1065 | " key, subkey = random.split(key)\n", 1066 | " X = jax.random.normal(key, (N, 2))\n", 1067 | " ksd_vals_this_iter.append(jit(ksd)(jnp.array(X)))\n", 1068 | " ksd_vals.append(jnp.mean(jnp.array(ksd_vals_this_iter)))" 1069 | ] 1070 | }, 1071 | { 1072 | "cell_type": "code", 1073 | "execution_count": 38, 1074 | "id": "94d5ec3d-aab8-446b-a63b-58e21972779b", 1075 | "metadata": {}, 1076 | "outputs": [ 1077 | { 1078 | "data": { 1079 | "text/plain": [ 1080 | "" 1081 | ] 1082 | }, 1083 | "execution_count": 38, 1084 | "metadata": {}, 1085 | "output_type": "execute_result" 1086 | }, 1087 | { 1088 | "data": { 1089 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAA9hAAAPYQGoP6dpAABk0ElEQVR4nO3dd3RU1d7G8e+kE0iBBEILCaD0ngChKSBdmqIEkN4E8dJsKCgqCiKidARE6QGkCCgKKNJ7TJAmIqGT0AKppM55/+Ca90YCJJhkUp7PWrOWc2bPmd/ZwMzjPufsbTIMw0BEREQkH7GydAEiIiIi2U0BSERERPIdBSARERHJdxSAREREJN9RABIREZF8RwFIRERE8h0FIBEREcl3bCxdQE5kNpu5evUqTk5OmEwmS5cjIiIi6WAYBlFRUZQsWRIrq4eP8SgApeHq1at4enpaugwRERF5DJcuXaJ06dIPbaMAlAYnJyfgXgc6OztbuBoRERFJj8jISDw9PVN+xx9GASgNf5/2cnZ2VgASERHJZdJz+YoughYREZF8RwFIRERE8h0FIBEREcl3dA3Qv5CcnExiYqKly5CHsLOze+StkCIikv8oAD0GwzAICwvjzp07li5FHsHKyoqyZctiZ2dn6VJERCQHUQB6DH+Hn2LFiuHo6KjJEnOovye0DA0NpUyZMvpzEhGRFApAGZScnJwSftzc3CxdjjxC0aJFuXr1KklJSdja2lq6HBERySF0cUQG/X3Nj6Ojo4UrkfT4+9RXcnKyhSsREZGcRAHoMel0Su6gPycREUmLxQPQnDlzKFu2LA4ODvj4+LB79+4Htl23bh0tW7akaNGiODs706BBA7Zs2ZKqzaJFizCZTPc94uLisvpQREREJJewaABatWoVI0eOZOzYsQQFBdGkSRPatm3LxYsX02y/a9cuWrZsyebNmwkMDKRZs2Z06NCBoKCgVO2cnZ0JDQ1N9XBwcMiOQxIREZFcwKIXQX/++ecMGDCAgQMHAjBt2jS2bNnC3LlzmTRp0n3tp02blur5xIkT2bBhA5s2baJ27dop200mE8WLF8/S2kVERCT3stgIUEJCAoGBgbRq1SrV9latWrFv37507cNsNhMVFUWRIkVSbY+OjsbLy4vSpUvTvn37+0aI/ik+Pp7IyMhUj7yob9++dO7cOdW2NWvW4ODgwKeffkpMTAxvvfUW5cqVw8HBgaJFi9K0aVO+//77lPZNmzZNOa1ob29PqVKl6NChA+vWrcvmoxERkdzqyPlwbsckWLQGiwWgmzdvkpycjIeHR6rtHh4ehIWFpWsfU6dOJSYmhq5du6Zsq1SpEosWLWLjxo0EBATg4OBAo0aNOHPmzAP3M2nSJFxcXFIenp6ej3dQucxXX33FSy+9xKxZs3jzzTcZMmQI3333HbNmzeKPP/7gp59+okuXLty6dSvV+wYNGkRoaCh//fUXa9eupUqVKnTr1o3Bgwdb6EhERCQ3MAyDhXvO4T//ACNWBZNsNixWi8XnAfrnXTqGYaTrzp2AgADef/99NmzYQLFixVK2+/n54efnl/K8UaNG1KlTh5kzZzJjxow09/X2228zevTolOeRkZEZCkGGYXA30TK3WRewtX6sO50+/fRT3nvvPVasWEGXLl0A2LRpE9OnT6ddu3YAeHt74+Pjc997HR0dU04xenp64ufnR6VKlejfvz9du3alRYsW/+KIREQkL4pNSOKttcfYdPQqAIUdbUlMNmNtZW2ReiwWgNzd3bG2tr5vtOf69ev3jQr906pVqxgwYADffvvtI39sraysqFu37kNHgOzt7bG3t09/8f9wNzGZKu9teXTDLHDyw9Y42mXsj3HMmDHMnj2b77//PlX/FS9enM2bN/P888/j5OSUoX326dOH1157jXXr1ikAiYhIKuduxjBkaSCnr0VhY2Vi3LOV6dPQ26JTlVjsFJidnR0+Pj5s27Yt1fZt27bRsGHDB74vICCAvn37smLFCp599tlHfo5hGAQHB1OiRIl/XXNe8OOPPzJ58mQ2bNhwX1CZP38++/btw83Njbp16zJq1Cj27t2brv1aWVlRoUIFzp8/nwVVi4hIbrXt5DU6ztzD6WtRFHWyJ2CwH30blbX4PG0WPQU2evRoevXqha+vLw0aNGD+/PlcvHiRIUOGAPdOTV25coUlS5YA98JP7969mT59On5+fimjRwUKFMDFxQWADz74AD8/P5588kkiIyOZMWMGwcHBzJ49O8uOo4CtNSc/bJ1l+3/UZ2dEjRo1uHnzJu+99x5169ZNNdLz1FNPERISwoEDB9i7dy/bt29n+vTpfPDBB7z77ruP3Hd6T1+KiEjel2w2+GLbn8z69S8A6noXZnaPOhh3I7hw4QJeXl4Wrc+i8wD5+/szbdo0PvzwQ2rVqsWuXbvYvHlzSqeEhoammhNo3rx5JCUlMWzYMEqUKJHyGDFiREqbO3fuMHjwYCpXrkyrVq24cuUKu3btol69ell2HCaTCUc7G4s8Mho4SpUqxc6dOwkNDaVNmzZERUWlet3W1pYmTZowZswYtm7dyocffsiECRNISHj41frJycmcOXOGsmXLZrj/REQkb7kdk0Dfbw6lhJ9+jbxZMciPowd3U7NmTV588cVH/q5kNYtfBP3KK6/wyiuvpPnaokWLUj3fsWPHI/f3xRdf8MUXX2RCZXlXmTJl2LlzJ82aNaNVq1Zs2bIFZ2fnNNtWqVKFpKQk4uLiUtbVSsvixYu5fft2ygXVIiKSPx2/EsHLSwO5cucuBWyt+aRLddpVLcZ748YyefJkDMOgWLFi3Lhxg1KlSlmsTosHILGM0qVLs2PHjlQhqFOnTnTv3h1fX1/c3Nw4efIk77zzDs2aNUsVkGJjYwkLCyMpKYkrV66wbt06vvjiC4YOHUqzZs0seFQiImJJq49cYtx3x0lIMuPl5si8Xj4UiL/N008/zf79+wEYOnQoU6dOpUCBAhatVQEoH/v7dFizZs1o2bIlzz33HIsXL+add94hNjaWkiVL0r59e957771U71uwYAELFizAzs4ONzc3fHx8WLVqFc8995yFjkRERCwpPimZ9zeeJODQvctWWlQuxtSutfjlx00MGDCAO3fu4OLiwldffcULL7xg4WrvMRmGYblZiHKoyMhIXFxciIiIuO/UUFxcHOfOnUtZwFVyNv15iYhkrat37jJ0+W8cvXQHkwlGt6jAsGZPYBhm/Pz8OHLkCH5+fgQEBODt7Z2ltTzs9/ufNAIkIiIij2XfXzf5T0AQt2IScClgy/RutWha8e/Jia1ZuXIlixYt4r333sPW1taitf6TRe8CExERkdzHMAzm7TxLz4UHuRWTQNWSzmx6tRHn9v3ARx99lNKufPnyTJgwIceFH9AIkIiIiGRAdHwSb645yuZj9+bie8GnNG82L8Oo4YNZsWIFJpOJVq1aZen0M5lBAUhERETS5a/r0by89Ahnb8Rga21ifIeqVLS+TsP6dTl79izW1tZMmDABX19fS5f6SApAIiIi8kg/Hgvl9W+PEpOQTHFnB2a/VJtd6xbT/623SExMpEyZMgQEBDx0OaucRAFIREREHigp2cyUraeZtzMEAL9yRZjZvQ7/GdSHVatWAfDcc8+xcOFCChcubMlSM0QXQYuIiEiabkXH0/vrQynhZ/BT5Vg2oD5Fnexp3bo19vb2zJ49m7Vr1+aq8AMaARIREZE0BF+6w9BlgYRGxOFoZ80nz1WjmksCNtb3xk769u1L8+bNLb6o6ePSCFA+0rRpU0aOHJny3Nvbm2nTplmsnkcJCwujZcuWFCxYEFdXV+DewrPfffedResSEcnLDMNgxcGLdP1yP6ERcZRzL8i857yZOvIlnnrqKW7dugXc+z7OreEHNAKUrx0+fJiCBQtauowH+uKLLwgNDSU4OBgXFxcAQkNDU4ZZz58/T9myZQkKCqJWrVoWrFREJG+IS0zmvQ3HWX3kMgCtq3rwjOMlOrfoQnh4OIUKFeL333/PE+s+KgDlY0WLFrV0CQAkJiamOUnW2bNn8fHx4cknn0zZVrx48ewsTUQk37gUHssry3/j2JUIrEwwsllZQjbPx3/6dADq1KnDypUrU30n52Y6BZaP/fMUmMlk4quvvuK5557D0dGRJ598ko0bN6Z6z8mTJ2nXrh2FChXCw8ODXr16cfPmzZTXf/rpJxo3boyrqytubm60b9+es2fPprx+/vx5TCYTq1evpmnTpjg4OLBs2bI0a1u7di1LlizBZDLRt2/flBr/PgVWtmxZAGrXro3JZKJp06aZ0zEiIvnMrj9v0GHWHo5diaCwoy0fN3dn8du9mP7f8DNy5Ej27duXZ8IPKABlqpiYmAc+4uLi0t327t276WqbFT744AO6du3K77//Trt27XjppZcIDw8H7p1+evrpp6lVqxZHjhzhp59+4tq1a3Tt2jVVraNHj+bw4cP88ssvWFlZ8dxzz2E2m1N9zltvvcXw4cM5deoUrVu3vq+Ow4cP06ZNG7p27UpoaGjKP8L/dejQIQB+/vlnQkNDWbduXWZ2hYhInmc2G8z+9S/6fHOIO7GJ1CjtwvfDm/DTsjn89ttvFClShI0bN/LFF19gb29v6XIzlU6BZaJChQo98LV27drxww8/pDwvVqwYsbGxabZ9+umn2bFjR8pzb2/vVKMsfzMM4/GLfYC+ffvSvXt3ACZOnMjMmTM5dOgQbdq0Ye7cudSpU4eJEyemtP/666/x9PTkzz//pEKFCnTp0iXV/hYuXEixYsU4efIk1apVS9k+cuRInn/++QfWUbRoUezt7SlQoMADT3v9fQrPzc1Np8ZERDIoMi6R11YfZdvJawB0q+vJ+x2r4mBrzfTp00lOTuaTTz6hdOnSFq40a2gESFKpUaNGyn8XLFgQJycnrl+/DkBgYCC//vorhQoVSnlUqlQJIOU019mzZ+nRowflypXD2dk55TTVxYsXU31ObpgmXUQkrzodFkWnWXvZdvIadjZWDK1ug3FoOfY292KBq6sry5Yty7PhBzQClKmio6Mf+Jq1tXWq53+HirRYWaXOpefPn/9XdWXEPy9GNplMKaevzGYzHTp0YPLkyfe9r0SJEgB06NABT09PFixYQMmSJTGbzVSrVo2EhIRU7XPy3WciInnZpqNXeXPN79xNTKakiwPNjWDGDxhLfHw8lSpVol+/fpYuMVsoAGWijPyoZ1XbrFSnTh3Wrl2Lt7c3Njb3/9W5desWp06dYt68eTRp0gSAPXv2ZFk9dnZ2ACQnJ2fZZ4iI5BWJyWY++fEPFu45B0DdEnbE/zqbjzesB+DZZ5+lQ4cOliwxW+kUmKTbsGHDCA8Pp3v37hw6dIiQkBC2bt1K//79SU5OpnDhwri5uTF//nz++usvtm/fzujRo7OsnmLFilGgQIGUi7EjIiKy7LNERHKzG1HxvPTVwZTw86xHJIe/GMSmDeuxtbXl888/Z9OmTbi7u1u40uyjACTpVrJkSfbu3UtycjKtW7emWrVqjBgxAhcXF6ysrLCysmLlypUEBgZSrVo1Ro0axZQpU7KsHhsbG2bMmMG8efMoWbIknTp1yrLPEhHJrQIvhNN+5m4OnQunkL0NHeyOM++NXly4cIHy5cuzb98+Ro0ahclksnSp2cpkZMWtRLlcZGQkLi4uRERE4OzsnOq1uLg4zp07R9myZXFwcLBQhZJe+vMSkfzKMAyWHrjAhO9Pkphs8GSxQnzZy4erfwTRtGlT/P39+fLLL+/7ncvNHvb7/U+6BkhERCSPuZuQzNj1x1gXdAWAZmXsmTWgEQXtbShftAm//fYbNWrUyHejPv9Lp8BERETykIu3Ynl+7j7WBV3Bykim8qVNrHnrOS6f//9Z+WvWrJmvww8oAImIiOQZv/5xnfYzd3MqNBKnxHAct07gpxXziI6OZsOGDZYuL0fRKTAREZFczmw2mLH9DNN/OYNhQLFbwZxePYXIyAhcXFxYuHDhfTP153cKQI9J147nDvpzEpG8LiI2kVGrg9n+x3XMifEUPbGSw1u+BcDPz4+AgAC8vb0tW2QOpFNgGfT3TMkPWsdLcpa/Z6D+50zcIiJ5wanQSDrM2sP2P65jb2PFM/zOb/8NP2PGjGHXrl0KPw+gEaAMsra2xtXVNWUpC0dHx3x/IVlOZTabuXHjBo6OjmnOXC0ikputD7rM2+uOEZdopnThAnzZ04eKxVoQce53hgwZQqtWrSxdYo6mX4XH8PfK4w9bz0tyBisrK8qUKaOQKiJ5RkKSmY9/OMni/Rcwx8dS+OxPrFsynWKuhQBYt26dhSvMHRSAHoPJZKJEiRIUK1aMxMRES5cjD2FnZ3ff4rIiIrnVtcg4Xln+G4EXbhMfeobEbV9wKfQin35UmM8++8zS5eUqCkD/grW1ta4tERGRbHEw5BbDVgRxIyqOhKPfc/OXr0lKSsTLy0t3eD0GBSAREZEczDAMvt57nombT5EQfYf4X2Zx4+R+AJ5//nm++uorChcubOEqcx8FIBERkRwqNiGJMWuPsfHoVeKvnibq+0+IuX0De3t7vvjiC4YMGaJrHB+TLo4QERHJgc7djOG52fvYePQqNlYmRravg1VSHJUqVeLQoUMMHTpU4edf0AiQiIhIDvPzyWuMWhVMRFQUHm6uzHmpDnW9i/CU5xZq1KhBwYIFLV1irqcRIBERkRwi2WwwdetpBi45wrUTe7m+YBBvVkugrncRABo0aKDwk0kUgERERHKA2zEJ9Ft0mBlbTxH+ywJurJ1AfPQdFi+YY+nS8iSdAhMREbGw41ciGLIskHNn/+LWpinEh/0FwKhRo5g0aZKFq8ubFIBEREQsaPWRS4z77ji3f9/O7S2zSU64i5ubG4sWLaJ9+/aWLi/PUgASERGxgPikZD7YdJIVBy8Sd/kENzfdm8n5qaeeYvny5ZQuXdrCFeZtCkAiIiLZLDTiLkOX/UbwpTuYTPB2384EWp2gfPlyjBs3TqsMZAMFIBERkWy07+xNXl3+Gxf2/0DxGk2Y1e8pmlUshvHMIs3rk40UgERERLKBYRjM3xXCxO+OcHPzDGL/3Edt8xmaVri3jpfCT/ZSABIREcli0fFJvLnmKOt+/JWbmz4lOfIGtra2tG/bytKl5VsKQCIiIlnor+vRvLzkML9tWsSd3cvAMFO+fHlWrlyJr6+vpcvLtxSAREREsshPx0MZuWgXF9Z8QtyFYAB69OjB3LlzcXZ2tmxx+ZwCkIiISCZLSjYzZetp5u0MwZwE1tHXcHR0ZNasWfTt21fX++QACkAiIiKZ6FZ0PMOWHWb/uTuYTCZeblmdFj024OxUiMqVK1u6PPkvBSAREZFMEnzpDv1n/sCJ5RMoUqsVCyaNoX2NkpYuS9KgACQiIpIJAg5dZPTkeVz7YTpGfAzJSXd45snJli5LHkCrwYuIiPwLcYnJjF5+iMEvDyFs3USM+Bjq1qvPoYMHKFCggKXLkwdQABIREXlMl2/H0ub9AGaN8ic6+EcA3nrrLfbu2Y23t7dli5OH0ikwERGRx7D7zA1eWbiTE18Mxki4S2G3oqxcsYxWrTS5YW6gACQiIpIBZrPB3J1nmbr1NGYKUKFlD4pFh7B65QqKFy9u6fIknRSARERE0ikyLpHekwM4fDkWWzdPutX15N3351DAzgYrK11VkpsoAImIiKTDH6ERtB/8Nmc3L8DO3ZMFa36id5MKli5LHpPiqoiIyCMs+/UYvk+15Oz3c8GcxFM+1elcU6e7cjMFIBERkQdITDbTb+I39O3YlJi/DmNlY8eUaTPZ+sN3Wssrl9MpMBERkTSE3o6hRa/hnPzhG8CgqGc5ftqwljq1a1m6NMkEGgESERH5h8ALt+k0aw9ng/cDBq2e6865U78r/OQhGgESERH5L8MwWLr/PBN+OEVisoFvv/fpUiqGUUP6Wbo0yWQWHwGaM2cOZcuWxcHBAR8fH3bv3v3AtuvWraNly5YULVoUZ2dnGjRowJYtW+5rt3btWqpUqYK9vT1VqlRh/fr1WXkIIiKSB9yJisXn2V68OnI0ickGz1YvwU/jnlf4yaMsGoBWrVrFyJEjGTt2LEFBQTRp0oS2bdty8eLFNNvv2rWLli1bsnnzZgIDA2nWrBkdOnQgKCgopc3+/fvx9/enV69eHD16lF69etG1a1cOHjyYXYclIiK5zM5Dv+NVpTZBPy4n6vAG+lexZVaP2hSy14mSvMpkGIZhqQ+vX78+derUYe7cuSnbKleuTOfOnZk0aVK69lG1alX8/f157733APD39ycyMpIff/wxpU2bNm0oXLgwAQEBae4jPj6e+Pj4lOeRkZF4enoSERGhq/xFRPK4sZ/N4ZOxr2NOuIu1ozOTps3ljUE9LF2WPIbIyEhcXFzS9fttsRGghIQEAgMD71szpVWrVuzbty9d+zCbzURFRVGkSJGUbfv3779vn61bt37oPidNmoSLi0vKw9PTMwNHIiIiuVFUVDT1W3dh4hvDMCfcxe2JWhw8fEThJ5+wWAC6efMmycnJeHh4pNru4eFBWFhYuvYxdepUYmJi6Nq1a8q2sLCwDO/z7bffJiIiIuVx6dKlDByJiIjkNndiEihXy49DW9eByYom/kO5ePwgPlWetHRpkk0sfnLTZDKlem4Yxn3b0hIQEMD777/Phg0bKFas2L/ap729Pfb29hmoWkREcqtToZEMWRYI1Z7F5kYY46bMZfzLL1q6LMlmFgtA7u7uWFtb3zcyc/369ftGcP5p1apVDBgwgG+//ZYWLVqkeq148eKPtU8REcnbbt++zdebD7DgD4hLNFOxYWs+nzKcehVKWbo0sQCLnQKzs7PDx8eHbdu2pdq+bds2GjZs+MD3BQQE0LdvX1asWMGzzz573+sNGjS4b59bt2596D5FRCRv27lrD2UrVuXNwT2IuX2LpyoUZdOrjRV+8jGLngIbPXo0vXr1wtfXlwYNGjB//nwuXrzIkCFDgHvX5ly5coUlS5YA98JP7969mT59On5+fikjPQUKFMDFxQWAESNG8NRTTzF58mQ6derEhg0b+Pnnn9mzZ49lDlJERCzGbDbz7gcfM+mjDzDMydi4lqB7DVcm9a2LtdWjL7eQPMywsNmzZxteXl6GnZ2dUadOHWPnzp0pr/Xp08d4+umnU54//fTTBnDfo0+fPqn2+e233xoVK1Y0bG1tjUqVKhlr167NUE0REREGYERERPybQxMREQsKDQ016jX6/98Nl2pNje8OnrF0WZKFMvL7bdF5gHKqjMwjICIiOc+WLVvo2v0lIm/fwmRrT5UuI9k4YxzlihaydGmShTLy+23xu8BEREQyU2xCEiM/nkXk7VvYFvWm65ufMX94Jxzt9JMn/09/G0REJM84dzOGIUsDifHpTWGzExM/eJeXm1dK1/Qqkr8oAImISK63Zs0avlyyirBaA4hOSMbDvTBrls6iXtkij36z5EsKQCIikmvdvXuXUaNGMW/ePADcDC+atnueOS/VoZizg4Wrk5xMAUhERHKlkydP8sKLXTl18gRgwtnvBYb268m7HWtgZ2Oxae4kl1AAEhGRXMUwDL755huGDXuVuLi7WBV0pWSn15nxeh+eq13a0uVJLqEAJCIiucqbb77JZ599BoCDd21q9RrL10NbUrmEpi2R9FMAEhGRXCM+KZkb7rUw2djj0qg7z/Uewhfd6uDiaGvp0iSXUQASEZEczWw2c+zYMYp5V2Dost8Ivu1K6aELea1jPf7T/AmstKSFPAYFIBERybFu3LhB3759+fnnXyg3cDp3nUrj7GDD9H4taVaxmKXLk1xMAUhERHKkHTt28NJLL3H16lVM1rbcvBxC3eZV+LKnD2XcHC1dnuRyCkAiIpKjJCUlMWHCBCZMmIBhGNgUKU3RTm/RrU1jPu5cnQJ21pYuUfIABSAREckxLl++TI8ePdi9ezcABau3xKP1EN5/vg49/by0pIVkGgUgERHJMZYvX87u3buxsitAkdbDKOfXmjkv+eDjVdjSpUkeowAkIiI5QlKymeSq7XHy2YuTTwca16nGrB51KOpkb+nSJA/SXOEiImIxZ86coVevXly+cYfeXx9i/p7zFGnxMkM7NGLZwPoKP5JlNAIkIiIWsWzZMoYOHUp0dDS/XIjHrmEfHO2smdylBh1qlrR0eZLHKQCJiEi2io6O5tVXX2Xx4sUAFChTHavqz1LWvSDzevlQwcPJwhVKfqAAJCIi2ebo0aP4+/tz+vRpTCYrnBt1x6VBV1pXK8lnXWvi7KAlLSR7KACJiEi2WL9+Pd27dyc+Ph4Hl6K4tBuNo1d1XmtVkaFPl9eSFpKtFIBERCRb+Pj4YOdQAHuvWji3Ho67uzszutemyZNFLV2a5EMKQCIikmUuXLiAl5cXhmGw6WwCLt0+xcqlBDVKuzK3Zx1KF9aSFmIZug1eREQyXXJyMhMnTuSJJ55g9brveHlpIFO2nMbatSTd6pbh2yENFH7EojQCJCIimSosLIyePXvyyy+/ADD68yXYNB6AnbUVH3SqSvd6ZSxcoYgCkIiIZKItW7bQu3dvrl+/jr1DAYq0HIJ15eaUdHFgTk8fanm6WrpEEUABSEREMkFiYiLjxo3j008/BcDDuwLWz4zC1t2ThuXdmNm9Nm6FNKuz5BwKQCIi8q/9/PPPKeGn/NPPk+DzEla29gx5ujyvt6qAjbUuOZWcRQFIRET+tbZt29Jz0DAORLuRWKYeTnbWTO1akzbVSli6NJE0KZKLiEiG3b17lzfeeIOwsDAMw2Dp/vMcKNqOxDL1KF+0IBtebazwIzmaRoBERCRDTp48ib+/P8ePH+fo779Ttf8nrA+6CkC76sX59IWaFLLXz4vkbPobKiIi6WIYBl9//TX/+c9/uHv3LkWLFeNO+dasD7qKlQnGtK3EoCblMJm0pIXkfApAIiLySBEREQwZMoSVK1cC4NuoKTENhnDduhBuBe2Y2aM2Dcu7W7hKkfRTABIRkYc6ffo07dq1IyQkBGtra9r2HcUx96cAK2p6uvJlzzqUcClg6TJFMkQBSEREHqpUqVJYW1vjWcaLGr3e5VhScQB61C/D+A5VsLextnCFIhmnACQiIve5ffs2rq6umEwmChUqxMxFK/lg22WO37XGzsaKjzpXo6uvp6XLFHlsug1eRERS+fXXX6latSrTp08HYOuJMEb/dJ2rd60p5VqAdUMbKvxIrqcRIBERASApKYkJEyYwYcIEDMNg8eLFGFVaM317CAB+5Yow5yUfihS0s3ClIv+eApCIiHD58mV69OjB7t27AejTtx82jfqnhJ8+DbwY174KtlrSQvIIBSARkXxu06ZN9O3bl/DwcJycnJg4dQbfx5Tjj78isLU2MaFTNbrVK2PpMkUylQKQiEg+dunSJbp06UJiYiK+vr68MXkOk/bc5nZsFO6F7JnXqw4+XkUsXaZIplMAEhHJxzw9PZk0aRKXL1+mWuehvL3lL5LNBtVLuTCvlw8lXTW/j+RNOpkrIpLPLFu2jN9//z3l+asjRmLXqB8TfjxDstmgc62SfDukgcKP5GkaARIRySeio6N59dVXWbx4MZUqVeLIkSPEmK0ZsjSQ3y7e0Xpekq8oAImI5APBwcH4+/vz559/YmVlRffu3fnzZhxDlwcTFhmHk4MNM7vXpmnFYpYuVSRbKACJiORhhmEwZ84cXnvtNeLj4ylVqhTLly8n3Lk83RYcIj7JTPmiBVnQ25dyRQtZulyRbKNrgERE8qjo6Gi6dOnCq6++Snx8PO3btyfwtyD2xhRj1KqjxCeZaV6pGOuHNVL4kXxHI0AiInlUgQIFiIiIwNbWlilTptBn4FCGrwpm1583ABjWrDyjW1bE2krX+0j+owAkIpKHmM1mEhMTsbe3x9rammXLlnH16lVcPCvQec5ezt+KxcHWiikv1KRDzZKWLlfEYnQKTEQkjwgLC6N169aMGjUqZVuJEiW441iazrP3cf5WLKVcC7B2aEOFH8n3FIBERPKALVu2ULNmTX7++WcWL17MpUuXMAyD2b/+xcAlR4iOT6Je2SJsfLURVUu6WLpcEYtTABIRycUSExN56623aNOmDdevX6dGjRoEBgbi5lGCVwOCmLLlNIYBPf3KsGxAfdwK2Vu6ZJEcQdcAiYjkUufOnaN79+4cPHgQgFdeeYXPPvuMW3EGL8zdz8nQSGysTHzQqSov1feycLUiOYsCkIhILpSUlESLFi0ICQnB1dWVhQsX8vzzz3Mg5BavLP+N8JgE3AraMbenD/XKajFTkX/SKTARkVzIxsaGzz//nIYNGxIcHMzzzz/P0gMX6PnVQcJjEqhWypmN/2ms8CPyACbDMAxLF5HTREZG4uLiQkREBM7OzpYuR0QEgBMnTnDt2jWaN2+ess1sNpNkhvc3nWDFwYsAdKxZksldalDAztpSpYpYREZ+v3UKTEQkhzMMg4ULFzJ8+HAcHR0JDg6mdOnSANyKSeSV5YEcPn8bkwnebF2JIU9rMVORR1EAEhHJwSIiInj55ZdZtWoVAE2aNMHW1haA41ciGLzkCFcj4nCyt2FG99o0q6TFTEXSI8MBKD4+nkOHDnH+/HliY2MpWrQotWvXpmzZsllRn4hIvnX48GG6detGSEgINjY2fPzxx7z++utYWVmxIfgKb675nfgkM+XcC7Kgjy/ltZ6XSLqlOwDt27ePmTNn8t1335GQkICrqysFChQgPDyc+Ph4ypUrx+DBgxkyZAhOTk5ZWbOISJ5mGAaff/45Y8aMISkpCW9vbwICAvDz8yPZbPDJj3/w5c6zADSrWJRp3WrjUsDWwlWL5C7pugusU6dOvPDCC5QqVYotW7YQFRXFrVu3uHz5MrGxsZw5c4Zx48bxyy+/UKFCBbZt25bVdYuI5Fkmk4nTp0+TlJREly5dCAoKws/Pj8i4RAYuPpwSfoY2Lc9Xfeoq/Ig8hnTdBTZ79mwGDRqEnZ3dI3d44sQJrl69SsuWLTOlQEvQXWAiYgnJyclYW9+7cys2Npb169fTo0cPTCYTZ29EM2jJEUJuxGBvY8WnL9SgU61SFq5YJGfJyO+3boNPgwKQiGSnpKQkPvzwQw4dOsTmzZuxsko9OP/rH9cZHhBEVHwSJVwcmN/Ll+qltZ6XyD9l5Pf7sSdCPHLkCEuXLmXZsmUcOXLkcXfDnDlzKFu2LA4ODvj4+LB79+4Htg0NDaVHjx5UrFgRKysrRo4ceV+bRYsWYTKZ7nvExcU9do0iIlnl0qVLNGvWjAkTJrBlyxZ+/PHHlNcMw2DujrP0X3yYqPgk6noXZuOrjRV+RDJBhu8Cu3z5Mt27d2fv3r24uroCcOfOHRo2bEhAQACenp7p3teqVasYOXIkc+bMoVGjRsybN4+2bdty8uRJypQpc1/7+Ph4ihYtytixY/niiy8euF9nZ2dOnz6dapuDg0O66xIRyQ4bNmygX79+3L59GycnJ+bNm8ezzz4LwN2EZN5a+zsbj14FoHu9MnzQsSp2NprAXyQzZPhfUv/+/UlMTOTUqVOEh4cTHh7OqVOnMAyDAQMGZGhfn3/+OQMGDGDgwIFUrlyZadOm4enpydy5c9Ns7+3tzfTp0+nduzcuLg/+PyCTyUTx4sVTPR4mPj6eyMjIVA8RkawSHx/P8OHD6dy5M7dv38bX15egoCC6d+8OwJU7d3lx3j42Hr2KjZWJjzpXY9Lz1RV+RDJRhv817d69m7lz51KxYsWUbRUrVmTmzJkPPX31TwkJCQQGBtKqVatU21u1asW+ffsyWlYq0dHReHl5Ubp0adq3b09QUNBD20+aNAkXF5eUR0ZGsUREMqp3797MnDkTgNGjR7N3717Kly8PwOHz4XSatYfjVyIpUtCOZQPr09NPK7mLZLYMB6AyZcqQmJh43/akpCRKlUr/HQk3b94kOTkZDw+PVNs9PDwICwvLaFkpKlWqxKJFi9i4cSMBAQE4ODjQqFEjzpw588D3vP3220RERKQ8Ll269NifLyLyKG+//TalSpXi+++/Z+rUqSl32K44eJEeCw5wMzqByiWc2fhqI/zKuVm4WpG8KcPXAH366af85z//Yfbs2fj4+GAymThy5AgjRozgs88+y3AB/1yvxjCMf7WGjZ+fH35+finPGzVqRJ06dZg5cyYzZsxI8z329vbY29s/9meKiDxMdHQ0e/fupXXr1gDUqlWLs2fPpnzvJCSZ+fD7Eyw7cG8x02drlGDKCzVwtNNqRSJZJcP/uvr27UtsbCz169fHxube25OSkrCxsaF///70798/pW14ePgD9+Pu7o61tfV9oz3Xr1+/b1To37CysqJu3boPHQESEckqR48exd/fn5CQEPbt24evry9ASvi5FR3P0OW/cehcOCYTvN6qIq80La/FTEWyWIYD0LRp0zLlg+3s7PDx8WHbtm0899xzKdu3bdtGp06dMuUz4N6IUnBwMNWrV8+0fYqIPIphGMyZM4fXXnuN+Ph4SpUqRUJCQqo2J65GMHhJIFfu3KWQvQ3Tu9XimcqZ9z+AIvJgGQ5Affr0ybQPHz16NL169cLX15cGDRowf/58Ll68yJAhQ4B758mvXLnCkiVLUt4THBwM3BtSvnHjBsHBwdjZ2VGlShUAPvjgA/z8/HjyySeJjIxkxowZBAcHM3v27EyrW0TkYcLDwxkwYADfffcdAO3bt+ebb77B3d09pc33v1/l9W+PEpdoxtvNka/6+PJEMa2jKJJdLHqC2d/fn1u3bvHhhx8SGhpKtWrV2Lx5M15e9+54CA0N5eLFi6neU7t27ZT/DgwMZMWKFXh5eXH+/Hng3pxEgwcPJiwsDBcXF2rXrs2uXbuoV69eth2XiORfe/fupUePHly8eBFbW1umTJnC8OHDU05pmc0GU7edZvav99bzeqpCUWZ2q42Lo9bzEslOWgojDVoKQ0Qe19SpU3n99dd54oknWLlyJT4+PimvRcYlMmplML/8cR2Al58qx5ttKmFtpet9RDJDRn6/dYuBiMi/9L93r44aNQqAwYMH4+T0/6e0zt2MYeDiw5y9EYOdjRWTu1TnudqlLVKviPyLtcBERAR++uknnn76aWJiYoB7d56+9tprqcLPzj9v0GnWHs7eiKG4swNrhjRQ+BGxMAUgEZHHkJCQwJtvvknbtm3ZvXs3n3766X1tDMNg/q6z9PvmEJFxSfh4FWbjfxpRo7Rr9hcsIqlk6imw/v3706xZM3r16pWZuxURyVFCQkLo3r07hw4dAmDYsGG8/fbbqdrEJSYzZu3vfBd8bzFTf19PPuxcFXsb62yvV0Tul6kBKCQkhF9//ZXPPvuMo0ePZuauRURyhG+//ZaBAwcSGRmJq6srX3/9daq5zABCI+4yeEkgx65EYG1l4r32VejdwEuTG4rkIJkagHbs2AHA6dOnM3O3IiI5wsyZMxk+fDgADRs2TJmG438dOR/OkGW/cTM6nsKOtsx+qQ4Ny7untTsRsaBMvQbo70VS/3eleBGRvKJLly54eHjwzjvvsHPnzvvCz8pDF+m+4AA3o+OpVNyJja82VvgRyaHSHYB69+5NZGTkA18/cuRIqkkKRURyO8Mw2LNnT8rzkiVLcvr0aT7++OOUtRABEpPNjN9wnDHrjpGYbNC2WnHWDm2IZxFHS5QtIumQ7gB0/PhxqlSpwpYtW1JtT0xM5J133qFhw4Y0btw40wsUEbGEiIgIunfvTpMmTVizZk3KdhcXl1TtwmMS6LXwIIv3XwBgdMsKzHmpDgXtNc2aSE6W7n+hhw4d4sMPP6RDhw7069ePqVOn8scff9CnTx9iYmL44YcfaNmyZVbWKiKSLQ4fPky3bt0ICQnBxsaG0NDQNNudvBrJ4KVHuHz7LgXtrPnCvxatqhbP5mpF5HFkeCmMwMBA+vTpQ3h4ODdv3qRv375MnTo11aRfuZ2WwhDJn8xmM1988QVjxowhKSkJb29vAgIC8PPzu6/t5mOhvLb6KHcTk/Fyc2RBb18qeOSd70GR3ChLl8Kwt7fH1taWiIgI7OzsaNSoUZ4KPyKSP924cYO+ffuyefNmAF544QUWLFiAq6trqnZms8G0n/9kxva/AGjypDszu9fG1dEuu0sWkX8h3dcAGYbBpEmT8PX1pVatWly9epVPP/2UV199lU6dOnH9+vWsrFNEJEsdPHiQzZs34+DgwJdffsnq1avvCz9RcYkMXhqYEn4GNi7LN33rKvyI5ELpPgXm5+fHxYsXmTdvHh06dEjZHhISQr9+/Thx4gSzZ8/G398/y4rNLjoFJpI/ffbZZ7Ru3Zrq1avf99r5mzEMWnKEM9ejsbOxYtJz1enio/W8RHKSjPx+p3sEyNvbm+PHj6cKPwDlypVjx44djB07lgEDBjxexSIi2ezSpUt07tyZy5cvp2x7/fXX0ww/u/68QcdZezhzPZpiTvasGuyn8COSy2X4IuiHOXPmDE8++WRm7c5iNAIkkrdt3LiRfv36ER4ezrPPPsv333+fZjvDMFi45xwTN5/CbEAtT1fm9fLBw9khmysWkfTI0ougHyYvhB8Rybvi4+N58803mTFjBgC+vr5Mnz49zbZxicm8s/4Y6367AsALPqX5qHM1HGy1mKlIXpCuU2Bt2rRh3759j2wXFRXF5MmTmT179r8uTEQkM/355580aNAgJfyMHj2avXv3Ur58+fvahkXE4T//AOt+u4K1lYnxHaow5YUaCj8ieUi6RoBefPFFunbtipOTEx07dsTX15eSJUvi4ODA7du3OXnyJHv27GHz5s20b9+eKVOmZHXdIiLptn//flq2bElMTAxubm4sXryYZ599Ns22v128zZClgVyPiselgC1zXqpDoye0npdIXpPua4ASEhJYs2YNq1atYvfu3dy5c+feDkwmqlSpQuvWrRk0aFCeWAhV1wCJ5C13796lXr16uLm5sXz5ckqVKpVmu9VHLjFu/XESks1U8CjEgt6+eLkVzOZqReRxZeT3+7Evgo6IiODu3bu4ublha2v7WIXmVApAIrnf6dOnefLJJ7GyunemPzQ0lGLFimFtff9prKRkMx9vPsU3e88D0LqqB1O71qKQ1vMSyVWy5Db4f3JxcaF48eJ5LvyISO5mGAazZs2iZs2aTJ48OWV7iRIl0gw/t2MS6P31oZTwM7LFk8x9yUfhRySPy/C/8O3bt7Nu3TrOnz+PyWSibNmyvPDCCzz11FNZUZ+ISLqFh4czYMAAvvvuOwCOHDmCYRiYTKY0258Oi2LQkiNcDI/F0c6az7vWok01LWYqkh9kaARoyJAhtGjRgoCAAG7dusWNGzdYvnw5zZo14z//+U9W1Sgi8kh79uyhVq1afPfdd9jZ2TF9+nTWrFnzwPDz0/Ewnpuzl4vhsXgWKcC6Vxoq/IjkI+keAVq/fj3ffPMNX3/9NX369En5UjGbzSxatIihQ4fSsmVLOnbsmGXFioj8U3JyMpMmTWL8+PGYzWaeeOIJVq1aRZ06ddJsbzYbzNh+hmk/nwGgYXk3ZveoQ+GCWs9LJD9J90XQHTt2pGrVqkyaNCnN19966y3++OMPNmzYkKkFWoIughbJPU6fPk3NmjWJj4+nZ8+ezJkzBycnpzTbRscn8drqYLacuAZAv0bejG1XGRvrx74cUkRykCyZCfq3335j3LhxD3y9S5cuPP/88+mvUkQkE1SsWJGZM2diZ2dHnz59Htju4q1YBi05wulrUdhZW/HRc9Xo6uuZjZWKSE6S7gB08+bNB86dAVCqVClu3bqVKUWJiDxIQkIC7733Hl26dKFu3boADBo06KHv2fvXTYat+I07sYkUdbJnXi8f6pQpnB3likgOle4AlJCQgJ3dg8+R29jYkJCQkClFiYikJSQkhO7du3Po0CHWrFnDiRMnsLe3f2B7wzBYtO88H/1wimSzQU1PV+b19KG4ixYzFcnvMnQb/Lvvvoujo2Oar8XGxmZKQSIiafn2228ZOHAgkZGRuLq68umnnz40/MQnJTNu/XG+DbwMwPO1SzHx+epaz0tEgAwEoKeeeorTp08/so2ISGaKjY1l1KhRzJ8/H4AGDRoQEBCAl5fXA99zPTKOl5cFEnTxDlYmeKddZQY0LvvAW+JFJP9JdwDasWNHFpYhInK/Gzdu0KxZM06cOIHJZOLtt9/m/ffff+gM9MGX7vDy0iNci7y3mOmsHrVp8mTRbKxaRHKDfz3Xe1JSEnFxcRQqVCgz6hERSeHm5kaZMmW4efMmy5Yto0WLFg9tvzbwMm+vP0ZCkpkni91bzNTbXYuZisj90j35xebNm1m6dGmqbR9//DGFChXC1dWVVq1acfv27UwvUETyl4iICKKjowGwsrJiyZIlHD169KHhJynZzEffn+S1b4+SkGSmZRUP1g9rpPAjIg+U7gD02WefERkZmfJ83759vPfee7z77rusXr2aS5cuMWHChCwpUkTyh8OHD1OnTp1US+u4u7vj4eHxwPfciU2g36LDfLXnHADDmz/BvJ5azFREHi7d3xDHjx9n6tSpKc/XrFlDy5YtGTt2LAAODg6MGDGCzz//PPOrFJE8zWw288UXXzBmzBiSkpIwm83cunULNze3h77vz2v3FjO9cCuWArbWTO1ak3bVS2RT1SKSm6V7BCgqKirVl9GePXto3rx5yvOqVaty9erVzK1ORPK8Gzdu0L59e15//XWSkpJ44YUXCAoKemT42XoijOdm7+XCrVhKFy7A2qENFX5EJN3SHYBKlizJqVOnAIiOjubo0aM0atQo5fVbt249cI4gEZG0bN++nZo1a/Ljjz/i4ODAl19+yerVq3F1dX3gewzDYOYvZxi8NJCYhGT8yhVh46uNqVJS6/aJSPql+xTYCy+8wMiRI3nnnXfYvHkzxYsXx8/PL+X1I0eOULFixSwpUkTynrt37/LSSy8RFhZG5cqVWbVqFdWrV3/oe2Lik3hjzVE2HwsDoE8DL8a1r4KtFjMVkQxKdwAaP348V69eZfjw4RQvXpxly5Zhbf3/M6oGBATQoUOHLClSRPKeAgUKsGjRItasWcO0adMoWPDhd2xdCr+3mOkfYVHYWpuY0Kka3eqVyaZqRSSvMRmGYWTWziIjIx+5/HxuEBkZiYuLCxEREXnieERyig0bNmAYBp07d87Q+/advcmw5b9xOzYR90L2fNmzDr7eRbKmSBHJtTLy+52h2+Af9aGtWrVK7+5EJB+Jj49n+PDhdO7cmb59+3LhwoV0vc8wDBbvO0+vhYe4HZtI9VIubHy1kcKPiPxr6T4F9u677+Lm5ka/fv3uey06OprWrVunmidIRATgzz//pFu3bgQFBQEwcOBASpR49N1a8UnJjN9wgpWHLwHQuVZJPulSQ4uZikimSHcAWrp0Kb169aJw4cKphq+jo6Np1aoV4eHh7Nq1KytqFJFcaunSpQwdOpSYmBjc3d1ZvHgx7dq1e+T7rkfFMXTZbwReuI2VCca0rcSgJuW0mKmIZJoM3QV2584devTowQ8//ECzZs2Ijo6mTZs23Lx5k507dz50tlYRyT/MZjP9+/dn8eLFADRt2pRly5ZRqlSpR77398t3GLwkkLDIOJwcbJjZvTZNKxbL6pJFJJ/J0FzxAwcOJDw8nM6dO7NhwwbeffddwsLC2LlzZ7qGtEUkf7CyssLFxQUrKyvef/993nnnnVR3jaYlOj6JlYcuMmXLaeKTzJQvWpAFvX0pV1QLLYtI5svwYjlvvvkmt2/f5plnnsHb25udO3em6//qRCRvMwyDqKiolDsvPv30U3r06EH9+vUf+r5L4bEs2nee1YcvERWfBEDzSsWY1q0Wzg62WV63iORP6Q5Azz//fKrntra2uLu7M3z48FTb161blzmViUiuER4eTv/+/QkPD2f79u3Y2Nhgb2//wPBjGAYHQsL5Zu85fj51DfN/J+Mo616Q/o3L0qNeGaytdL2PiGSddAcgFxeXVM+7d++e6cWISO6zZ88eevTowaVLl7Czs+Pw4cM0aNAgzbZxiclsDL7KN/vOcyr0/+8abfKkO/0bleXpCkWxUvARkWyQ7gD0zTffZGUdIpLLJCcnM2nSJMaPH4/ZbOaJJ55g1apV1KlT57621yLjWHbgAssPXiQ8JgGAArbWPF+nFH0bevOkh1N2ly8i+VyGrwESEQkNDaVnz55s374dgJdeeom5c+fi5JQ6yARfusM3e8/xw++hJP33PFdJFwd6N/SmW11PXB3tsr12ERFQABKRx/B3+HF0dGTOnDn07t07ZY6exGQzPx0P45u95/jt4p2U99T1Lky/RmVpVcUDGy1eKiIWpgAkIhk2c+ZMBg4cyDfffEPFihUBuB2TQMDhiyzdf4HQiDgAbK1NdKhRkn6NylK9tMvDdikikq0UgETkkUJCQti7dy+9evUCoEqVKuzduxeTycTpsCgW7TvH+qArxCWaAXAvZMdL9b14ya8MxZwcLFm6iEiaFIBE5KG+/fZbBg4cSExMDOXKlaNRo0aYzQa/nr7O13vPsfevWyltq5Z0pl+jsnSoWQJ7G63ZJSI5lwKQiKQpNjaWUaNGMX/+fAAaNWpEkWIl+GbvORbvO8/5W7EAWJmgddXi9GtUlrrehbVel4jkCgpAInKfEydO4O/vz4kTJzCZTAwb9QZFmrzEi8vOEP3f2ZqdHWzoVq8MvRt4Ubqwo4UrFhHJGAUgEUll0aJFvPLKK9y9exe3osWo128831t5YRy4BED5ogXp26gsXeqUwtFOXyEikjvp20tEUomIir4XfirVo0CL/3DSVBgMeLpCUfo18uapJzVbs4jkfgpAIkJCQgLhd80sPXCe5eFPUrTzOxSo4IejnS0v+JSmT0NvniimVdlFJO9QABLJx8xmM6+9+xGLvvkG126fYtjdu5bnyfrP0LehN13reuJSQCuyi0jeowAkkg8lJptZsfMYb/7nZa6fPAiA6fefafFiX/o38qZFZc3WLCJ5m8W/4ebMmUPZsmVxcHDAx8eH3bt3P7BtaGgoPXr0oGLFilhZWTFy5Mg0261du5YqVapgb29PlSpVWL9+fRZVL5K7hMckMGv7GWoMnsqATs25fvIgJhs7Wg4ex85Fk1n9cgPaVCuh8CMieZ5Fv+VWrVrFyJEjGTt2LEFBQTRp0oS2bdty8eLFNNvHx8dTtGhRxo4dS82aNdNss3//fvz9/enVqxdHjx6lV69edO3alYMHD2bloYjkaH+ERfLWmt/x+3gr48a9yx9fv0VydDjFvZ5gx579bJ03geqlXS1dpohItjEZhmFY6sPr169PnTp1mDt3bsq2ypUr07lzZyZNmvTQ9zZt2pRatWoxbdq0VNv9/f2JjIzkxx9/TNnWpk0bChcuTEBAQLrqioyMxMXFhYiICJydndN/QCI5SLLZYPsf1/lm7zn2nb03W/PtnYuJPPAtAP37D2DmzBk4OmoOHxHJGzLy+22xEaCEhAQCAwNp1apVqu2tWrVi3759j73f/fv337fP1q1bP3Sf8fHxREZGpnqI5FZms8Hqw5do9tkOBi05wr6zt7C2MvFs9RKsmfURlSpVIiAggIULv1L4EZF8y2IXQd+8eZPk5GQ8PDxSbffw8CAsLOyx9xsWFpbhfU6aNIkPPvjgsT9TJKf4IyySseuPE3jhNgBOtgZV4k/x+ZhXKeVaAIDjx49jba11ukQkf7P4lY7/XDfIMIx/vZZQRvf59ttvExERkfK4dOnSv/p8kewWm5DEpM2neHbGHgIv3MbRzpoB1e2x/v49Vk95k+2b1qS0VfgREbHgCJC7uzvW1tb3jcxcv379vhGcjChevHiG92lvb4+9vf1jf6aIJW07eY33N57gyp27ALSpWpwqMcG88/IIYmJicHd3x93d3cJViojkLBYbAbKzs8PHx4dt27al2r5t2zYaNmz42Ptt0KDBffvcunXrv9qnSE505c5dBi05wqAlR7hy5y6lXAsw64XK3P15BiOGDiQmJoZmzZpx9OhR2rZta+lyRURyFItOhDh69Gh69eqFr68vDRo0YP78+Vy8eJEhQ4YA905NXblyhSVLlqS8Jzg4GIDo6Ghu3LhBcHAwdnZ2VKlSBYARI0bw1FNPMXnyZDp16sSGDRv4+eef2bNnT7Yfn0hWSEw28/Wec0z7+Qx3E5OxsTIx6KlyPO0WQ+8ebThz5gxWVlaMHz+esWPH6pSXiEgaLBqA/P39uXXrFh9++CGhoaFUq1aNzZs34+XlBdyb+PCfcwLVrl075b8DAwNZsWIFXl5enD9/HoCGDRuycuVKxo0bx7vvvkv58uVZtWoV9evXz7bjEskqR86HM3b9cU5fiwKgnncRPnquGhU8nNi6dStnzpyhdOnSrFixgiZNmli4WhGRnMui8wDlVJoHSHKa2zEJTP7pD1YevneBfmFHW95pV5nna5dMNcKzbNky2rZti5ubm6VKFRGxmFwxD5CIPJphGKwJvMwzn+9MCT/+vp5sf60pJeIuULt2bc6dO5fSvmfPngo/IiLpoAAkkkOduRaF//wDvP7tUcJjEqjgUYhvhzRg4nNVmf3Fpzz99NMcO3aMsWPHWrpUEZFcR6vBi+QwdxOSmbn9DPN3hZBkNnCwtWJkiwoMaFyWm9ev0apVK7Zv3w7cG/GZM2eOhSsWEcl9FIBEcpBf/7jOuxuOc/n2vTl9nqlUjPc7VsWziCM//fQTvXv35saNGzg6OjJnzhz69Olj4YpFRHInBSCRHCA04i4fbjrJj8fvTeJZwsWB9ztWpVUVD0wmE5s2baJjx44A1KxZk1WrVlGxYkVLliwikqspAIlYUFKymcX7L/D51tPEJCRjbWWifyNvRraoQEH7///n2apVK+rUqUPDhg2ZMmUKDg4OFqxaRCT3UwASsZCgi7cZu/44J0MjAahTxpWPn6tO5RL3bt3csmULzzzzDDY2Ntjb27Nnzx4KFChgyZJFRPIM3QUmks0i7iYy7rtjPD93HydDI3EpYMuk56uzZkhDKpdwJjY2lsGDB9OmTRs+/PDDlPcp/IiIZB6NAIlkE8Mw2BB8lY9+OMnN6AQAnq9TinfaVca90L3FeI8fP46/vz8nT57EZDJZslwRkTxNAUgkG5y9Ec273x1n39lbAJQvWpCPOlenQfl7kxYahsFXX33F8OHDiYuLo3jx4ixbtoxnnnnGkmWLiORZCkAiWSguMZk5O87y5Y6zJCSbsbexYvgzTzKoSTnsbO6dgY6IiGDw4MGsXr0agNatW7NkyRKKFStmydJFRPI0BSCRLLLrzxu8u+E4F27FAvB0haJM6FSNMm6OqdpdvXqV77//HhsbGyZOnMhrr72GlZUuzxMRyUoKQCKZ7HpkHB9+f5Lvfw8FwMPZnvEdqtK2WvE0r+upXLky33zzDV5eXtSvXz+7yxURyZcUgEQySbLZYNmBC3y25TRR8UlYmaBPQ29Gt6yAk4NtSrvr168zcOBA3nzzTRo3bgxA165dLVW2iEi+pAAkkgmOXY7gnfXHOHYlAoCapV34+LnqVCvlkqrd9u3b6dmzJ6GhoZw+fZqTJ09ibW1tiZJFRPI1BSCRfyEyLpGpW06z9MAFzAY42dvwZpuK9KjvhbXV/5/uSkpK4oMPPuDjjz/GMAyqVKnCqlWrFH5ERCxEAUjkMRiGwfe/h/Lh9ye5ERUPQKdaJRn7bGWKOaVepuLSpUv06NGDPXv2ADBo0CCmTZuGo6PjffsVEZHsoQAkkkHnb8bw7obj7D5zE4Cy7gWZ0KkajZ90v6/tuXPn8PHx4fbt2zg7OzN//nz8/f2zu2QREfkHBSCRdIpPSmbezhBm/foXCUlm7KyteKVZeYY8XR4H27RPZXl7e9OsWTMuXbrEypUrKVeuXDZXLSIiaVEAEkmHfX/dZNx3xwm5GQNA4yfcmdC5GmXdC97X9s8//8TDwwMXFxdMJhOLFi3C3t4eOzu77C5bREQeQLOtiTzEzeh4Rq0KpsdXBwm5GYN7IXtmdK/N0gH10gw/S5YsoU6dOgwePBjDMABwcnJS+BERyWE0AiTyABdvxdJt/n6uRsRhMkEvPy9ea1URlwK297WNiopi2LBhLF26FIAbN24QGxtLwYL3hyQREbE8BSCRNPxv+CnnXpAv/GtR09M1zbZBQUH4+/tz5swZrKys+OCDD3j77bd1i7uISA6mACTyD5fCY+m+4MC98FO0ICsH+VHM2eG+doZhMHPmTN544w0SEhIoXbo0K1asoEmTJhaoWkREMkLXAIn8j0vhsXSbf4Ard+5Szv3B4QfureI+efJkEhIS6NixI8HBwQo/IiK5hEaARP7r8u17Iz9X7tylrHtBAgY/OPwAuLq6snz5co4dO8arr76a5kKnIiKSM5mMv29VkRSRkZG4uLgQERGBs7OzpcuRbHDlzl385+3n8u3/hp9BfhR3SR1+kpOT+eSTT/Dy8qJnz54WqlRERB4kI7/fGgGSfO/Knbt0m38v/Hi7OaYZfkJDQ+nZsyfbt2+nYMGCPPPMM5QoUcJCFYuIyL+la4AkX7t65y7d5x/gUvhdvNwcCRh8f/j58ccfqVmzZkr4mTNnjsKPiEgupwAk+VZoxF26LzjAxfBYyhS5N/JTwqVAyusJCQm88cYbtGvXjhs3blCzZk0CAwPp3bu3BasWEZHMoFNgki+FRcTRbf4BLtyKxbNIAQIG+1HSNXX4eeqppzh48CAAr776KlOmTMHB4cEXRYuISO6hESDJd+6Fn/1cuBVL6cIFCBjkR6n/CT8AdnZ2NG3alMKFC7N+/Xpmzpyp8CMikofoLrA06C6wvOta5L2Rn3M3YyhduAArB/tRurAjALGxsdy5c4eSJUsCkJiYyLVr1yhdurQlSxYRkXTKyO+3RoAk37gWGUf3/4afUq73Rn7+Dj/Hjx+nbt26PPfccyQkJABga2ur8CMikkcpAEm+cD0yju4LDhDy3/CzcrAfnkUcMQyDBQsWULduXU6ePMnFixcJCQmxdLkiIpLFFIAkz7se9d/wcyOGki4OBAy6F34iIiLo1q0bgwcPJi4ujtatWxMcHEylSpUsXbKIiGQxBSDJ025ExdNjwUHO3oihhIsDKwc3oIybI4cOHaJ27dqsXr0aGxsbPv30UzZv3oyHh4elSxYRkWyg2+Alz7oXfg7w1/Voijs7sHKwH2Xc7p32GjFiBOfOncPb25uVK1dSv359S5crIiLZSCNAkifdjL4Xfs78T/jxcisIgMlkYunSpfTp04egoCCFHxGRfEgBSPKcW/8Tfjyc7QkY7MfZoweYOnVqSpsnnniCRYsW4erqarlCRUTEYnQKTPKUe+HnIH9ei6aYkz1L+/mycNokJk6cCEC9evVo0qSJhasUERFLUwCSPCM8JoGXvjrI6WtRFHOyZ2q7UvR/sT179+4FYNCgQfj4+Fi4ShERyQkUgCRPCI9JoMeCA/wRFkVRJ3v6lbpOx+YvcufOHZydnZk/fz7+/v6WLlNERHIIBSDJ9W7/d+Tnj7Ao3AvZUyv0e14ZNx2AunXrsnLlSsqVK2fhKkVEJCfRRdCSq/0dfk6FRuJeyI6Vg+vToE4NAF5//XX27Nmj8CMiIvfRCJDkWndiE+i58CAnQyNxMcUSMOgpnijmRPl+/ahduza1a9e2dIkiIpJDKQBJrhQRm0jPhQc5dv4aMdvnEX/jFG5vPAPcm+dH4UdERB5Gp8Ak1/k7/Pz2WxDXl4wk/OjPXA8LZfv27ZYuTUREcgmNAEmuEnE3kZ4LD7B343Lu/Po1RnIipUuXZsWKFZrfR0RE0k0BSHKNiLuJdJuxlZ1fTeDuXwcB6NixI19//TVubm4Wrk5ERHITnQKTXCEyLpHeXx9iz8rZ3P3rILZ2dsyYMYPvvvtO4UdERDJMI0CS40XGJdJ74SGOXrqDV5sBODndZfa0qdSpU8fSpYmISC6lACQ52p8hF3hu1CSiK7XH1dGO5QMbU21SF0uXJSIiuZwCkORYazdsokfP3iRE36GU4cDyuR9QrZSLpcsSEZE8QNcASY6TkJDA8FGjeaFzRxKi7+BQvDxfvtFL4UdERDKNRoAkRwkJCaGrvz+BR44A4FavIz8tn4/vEx4WrkxERPISjQBJjrFhwwZq165N4JEjWDkUooz/eLavXazwIyIimU4jQJJj2Ds6ERkVjX2pKni9MIbVr3WgRmlXS5clIiJ5kAKQWFRMTAwFCxYkNiGJr0MKUMx/AkWfqMWywQ2p6elq6fJERCSP0ikwsQjDMJg/fz5ly5Yl+NgJ+i86zKFz4RSt4MPSQQ2opfAjIiJZSCNAku3u3LnD4MGD+fbbbwHwHzWBeN9eFLK3YXH/etQuU9jCFYqISF6nACTZ6uDBg3Tr1o3z589jY2NDtc5DCC/XikJ21izuXxcfL4UfERHJehY/BTZnzhzKli2Lg4MDPj4+7N69+6Htd+7ciY+PDw4ODpQrV44vv/wy1euLFi3CZDLd94iLi8vKw5BHMJvNTJkyhcaNG3P+/Hm8vcvS/M0vuV2+DYXsbVncvx4+XkUsXaaIiOQTFg1Aq1atYuTIkYwdO5agoCCaNGlC27ZtuXjxYprtz507R7t27WjSpAlBQUG88847DB8+nLVr16Zq5+zsTGhoaKqHg4NDdhySPMCiRYt48803SUpKossLL1J31AJOJxfH0c6aRf3r4eut8CMiItnHZBiGYakPr1+/PnXq1GHu3Lkp2ypXrkznzp2ZNGnSfe3feustNm7cyKlTp1K2DRkyhKNHj7J//37g3g/tyJEjuXPnzmPXFRkZiYuLCxERETg7Oz/2fuT/JSYm0rZtW7q82JW91jXY89ete+GnXz3qlVX4ERGRfy8jv98WGwFKSEggMDCQVq1apdreqlUr9u3bl+Z79u/ff1/71q1bc+TIERITE1O2RUdH4+XlRenSpWnfvj1BQUEPrSU+Pp7IyMhUD/l3kpKSmD17NgkJCQDY2tqyafNP7LOpmRJ+vulbV+FHREQswmIB6ObNmyQnJ+PhkXqWXw8PD8LCwtJ8T1hYWJrtk5KSuHnzJgCVKlVi0aJFbNy4kYCAABwcHGjUqBFnzpx5YC2TJk3CxcUl5eHp6fkvjy5/u3jxIk2bNuXVV19l7NixAMQlJvPyst/YfeYmBWyt+bpvXeqXc7NwpSIikl9Z/CJok8mU6rlhGPdte1T7/93u5+dHz549qVmzJk2aNGH16tVUqFCBmTNnPnCfb7/9NhERESmPS5cuPe7h5HvfffcdtWrVYu/evTg7O+Pr60tsQhJDlgWy688bONha8XXfuvgp/IiIiAVZ7DZ4d3d3rK2t7xvtuX79+n2jPH8rXrx4mu1tbGxwc0v7B9XKyoq6des+dATI3t4ee3v7DB6B/K+4uDjeeOMNZs2aBUDdunVZuXIlyQWL0mnWXs5cj04JPw3KK/yIiIhlWWwEyM7ODh8fH7Zt25Zq+7Zt22jYsGGa72nQoMF97bdu3Yqvry+2trZpvscwDIKDgylRokTmFC73OXPmDH5+finh5/XXX2fPnj0cjbCj43/DTzEne5YOqE/D8u4WrlZERMTCEyGOHj2aXr164evrS4MGDZg/fz4XL15kyJAhwL1TU1euXGHJkiXAvTu+Zs2axejRoxk0aBD79+9n4cKFBAQEpOzzgw8+wM/PjyeffJLIyEhmzJhBcHAws2fPtsgx5gcmk4mQkBDc3d1ZsmQJzVq0YvymEwQcuncqsdETbkzzr01RJ42yiYhIzmDRAOTv78+tW7f48MMPCQ0NpVq1amzevBkvLy8AQkNDU80JVLZsWTZv3syoUaOYPXs2JUuWZMaMGXTp0iWlzd/LLISFheHi4kLt2rXZtWsX9erVy/bjy8uSkpKwsbn31+eJJ55g3bp1VKlShXg7F56bs49ToZGYTDC8+ZMMf+ZJrK0efF2XiIhIdrPoPEA5leYBerigoCC6d+/OnDlzaN68ecr2H34P5a21vxMdn4RbQTumdatFkyeLWrBSERHJT3LFPECS+xiGwYwZM/Dz8+P06dO88847GIZBfFIy4zccZ9iK34iOT6KedxE2j2ii8CMiIjmWFkOVdLl16xb9+/dn48aNAHTu3JmFCxdy+fZdhq34jd8vRwAwtGl5XmtZARtrZWsREcm5FIDkkXbv3k2PHj24fPkydnZ2TJ06lWHDhvHzqeu8tvowkXFJuBSw5Qv/mjSvlPYUBiIiIjmJApA81O+//07Tpk0xm81UqFCBVatWUbV6DSb9+Afzd4UAUMvTlVk9alO6sKOFqxUREUkfBSB5qOrVq9O9e3esra2ZPXs2UcnWdJt/gMALtwHo36gsY9pWws5Gp7xERCT3UACS+2zZsgVfX1/c3NwwmUx888032NrasuP0dUatCuZ2bCJO9jZMebEGbappgkkREcl99L/tkiIhIYHXX3+dNm3a0L9///9fZ83Kms+2nKbfosPcjk2kaklnvh/eWOFHRERyLY0ACQBnz56lW7duHDlyBAAvLy+SkpK4fTeZ4SuDOBASDkBPvzKMe7YKDrbWlixXRETkX1EAElatWsWgQYOIioqicOHCfPPNN3Tq1Il9Z28yPCCYm9HxONpZM+n56nSqVcrS5YqIiPxrCkD5WGxsLCNGjOCrr74CoFGjRqxYsYLSpT2Z+csZvvj5T8wGVPRwYvZLdXiiWCELVywiIpI5dA1QPhYfH8+2bdswmUyMGzeOHTt2ULCIB30XHWbqtnvh50Wf0nw3rJHCj4iI5CkaAcpnUi5sNpkoXLgwq1atIiYmhubNm3PkfDivrggiLDIOB1srPuxUja6+nhauWEREJPMpAOUjd+7cYfDgwbRo0YLBgwcDUL9+fQzDYP6us0z+6TTJZoNyRQsy56U6VCquhWBFRCRvUgDKJw4ePEi3bt04f/48W7ZsoWvXrri6unInNoHXvz3Kz6euA9CxZkkmPl+dQvb6qyEiInmXfuXyOLPZzGeffcbYsWNJSkqibNmyrFy5EldXV4Iv3WHY8t+4cucudtZWjO9YhR71ymAymSxdtoiISJZSAMrDrl27Rp8+fdiyZQsAXbt2Zf78+Tg7O7No7zk+3nyKxGSDMkUcmfNSHaqVcrFwxSIiItlDASiPio6OxsfHhytXruDg4MCMGTMYOHAgUfFJDFvxG5uPhQHQpmpxPn2xBs4OthauWEREJPsoAOVRhQoVYvDgwaxevfreCu5Vq3LiagTDlv/G+Vux2FiZeKddZfo18tYpLxERyXdMxt/3RUuKyMhIXFxciIiIwNk599wJdfHiRRISEnjiiScASE5OJj4+ngIFCrDy8CXGbzxBQpKZUq4FmNmjNnXKFLZwxSIiIpknI7/fGgHKI9avX0///v3x9vZm//79ODg4YGAi8EoMyw+c4qcT9055Na9UjKkv1qRwQTsLVywiImI5CkC5XFxcHK+//jqzZ88GwM7Ojl9/D2F/qJnNx0K5GZ0AgLWViTdaV2Rwk3JYWemUl4iI5G8KQLnY6dOn8ff35+jRowA0fr4fCbW6MnTduZQ2hR1taVe9BN3rldFdXiIiIv+lAJRLLV68mFdeGUZsbAy2BV0p3G4Ul8r5QEwyTvY2tKpanI61StKwvBu21lryTURE5H8pAOUyF27FsOG3S4x//zNiY2Nw8KqBW/vXKVTYnRaVPehQsyRPVyiKg621pUsVERHJsRSAcoHQiLv88Hsom45e5ejlCAAKthmN9enddOz1Mp1ql6ZFZQ8KavkKERGRdNEvZg51KzqezcfD2BR8lYPnbhEVuInkmNu4Ne1Doyfc6VCzBq2r9MTFURMYioiIZJQCUA4ScTeRLSfC2HT0KvvO3iLZbJB8N5Jbm6dx969DAHw7aTitmta3cKUiIiK5mwKQhcUmJPHzqetsOnqVnadvkJBsTnmtVNw5TgZ8zN0bYdjZ2TF16lRaPt3IgtWKiIjkDQpAFhCflMyO0zfYdPQqv5y6zt3E5JTXKngUon01D87/sozpMydhNpupUKECq1atolatWpYrWkREJA9RAMpGf16LYv6uELacCCMqLillu5ebIx1qlKRDzZJULO5E586d2bBhAwC9e/dm9uzZFCpUyFJli4iI5DkKQNko4m4iawIvA1Dc2YH2NUrQsVZJqpdySbUgqb+/Pz///DNz5syhd+/elipXREQkz9JiqGnIqsVQzWaDyVv+4JlKHvh6FU5ZkiIhIYGQkBAqVaqU0jYsLIzixYtn2meLiIjkdRn5/dYUwdnIysrE220rU69skZTwExISQuPGjWnWrBnXrl1LaavwIyIiknUUgCzo7wubDx8+THx8PGfOnLF0SSIiIvmCApAFxMbGMmjQILp160ZUVBSNGzfm6NGjNG7c2NKliYiI5AsKQNns2LFj+Pr68tVXX2EymRg3bhy//vornp6eli5NREQk39BdYNls48aNnDp1iuLFi7N8+XKaN29u6ZJERETyHQWgbDZmzBju3r3L8OHDKVasmKXLERERyZd0G3wasuo2eBEREck6ug1eRERE5CEUgERERCTfUQASERGRfEcBSERERPIdBSARERHJdxSAREREJN9RABIREZF8RwFIRERE8h0FIBEREcl3FIBEREQk31EAEhERkXxHAUhERETyHQUgERERyXcUgERERCTfsbF0ATmRYRgAREZGWrgSERERSa+/f7f//h1/GAWgNERFRQHg6elp4UpEREQko6KionBxcXloG5ORnpiUz5jNZq5evYqTkxMmk8nS5TxSZGQknp6eXLp0CWdnZ0uXk2OoXx5MffNg6psHU988mPrmwbKzbwzDICoqipIlS2Jl9fCrfDQClAYrKytKly5t6TIyzNnZWf/w0qB+eTD1zYOpbx5MffNg6psHy66+edTIz990EbSIiIjkOwpAIiIiku8oAOUB9vb2jB8/Hnt7e0uXkqOoXx5MffNg6psHU988mPrmwXJq3+giaBEREcl3NAIkIiIi+Y4CkIiIiOQ7CkAiIiKS7ygAiYiISL6jAJQDzZkzh7Jly+Lg4ICPjw+7d+9+aPudO3fi4+ODg4MD5cqV48svv0z1+oIFC2jSpAmFCxemcOHCtGjRgkOHDmXlIWSZzO6b/7Vy5UpMJhOdO3fO5KqzR1b0zZ07dxg2bBglSpTAwcGBypUrs3nz5qw6hCyTFX0zbdo0KlasSIECBfD09GTUqFHExcVl1SFkiYz0S2hoKD169KBixYpYWVkxcuTINNutXbuWKlWqYG9vT5UqVVi/fn0WVZ+1Mrtv8uv3cHr/3vwtW7+HDclRVq5cadja2hoLFiwwTp48aYwYMcIoWLCgceHChTTbh4SEGI6OjsaIESOMkydPGgsWLDBsbW2NNWvWpLTp0aOHMXv2bCMoKMg4deqU0a9fP8PFxcW4fPlydh1WpsiKvvnb+fPnjVKlShlNmjQxOnXqlMVHkvmyom/i4+MNX19fo127dsaePXuM8+fPG7t37zaCg4Oz67AyRVb0zbJlywx7e3tj+fLlxrlz54wtW7YYJUqUMEaOHJldh/WvZbRfzp07ZwwfPtxYvHixUatWLWPEiBH3tdm3b59hbW1tTJw40Th16pQxceJEw8bGxjhw4EAWH03myoq+ya/fw+npm79l9/ewAlAOU69ePWPIkCGptlWqVMkYM2ZMmu3ffPNNo1KlSqm2vfzyy4afn98DPyMpKclwcnIyFi9e/O8LzkZZ1TdJSUlGo0aNjK+++sro06dPrgxAWdE3c+fONcqVK2ckJCRkfsHZKCv6ZtiwYUbz5s1TtRk9erTRuHHjTKo662W0X/7X008/neYPWdeuXY02bdqk2ta6dWujW7du/6rW7JYVffNP+eV7+H89rG8s8T2sU2A5SEJCAoGBgbRq1SrV9latWrFv374037N///772rdu3ZojR46QmJiY5ntiY2NJTEykSJEimVN4NsjKvvnwww8pWrQoAwYMyPzCs0FW9c3GjRtp0KABw4YNw8PDg2rVqjFx4kSSk5Oz5kCyQFb1TePGjQkMDEw5hRESEsLmzZt59tlns+AoMt/j9Et6PKjv/s0+s1tW9c0/5Zfv4fSyxPewFkPNQW7evElycjIeHh6ptnt4eBAWFpbme8LCwtJsn5SUxM2bNylRosR97xkzZgylSpWiRYsWmVd8Fsuqvtm7dy8LFy4kODg4q0rPclnVNyEhIWzfvp2XXnqJzZs3c+bMGYYNG0ZSUhLvvfdelh1PZsqqvunWrRs3btygcePGGIZBUlISQ4cOZcyYMVl2LJnpcfolPR7Ud/9mn9ktq/rmn/LL93B6WOp7WAEoBzKZTKmeG4Zx37ZHtU9rO8Cnn35KQEAAO3bswMHBIROqzV6Z2TdRUVH07NmTBQsW4O7unvnFZrPM/ntjNpspVqwY8+fPx9raGh8fH65evcqUKVNyTQD6W2b3zY4dO/j444+ZM2cO9evX56+//mLEiBGUKFGCd999N5OrzzoZ7RdL7dMSsvI48tv38MNY8ntYASgHcXd3x9ra+r4kff369fsS99+KFy+eZnsbGxvc3NxSbf/ss8+YOHEiP//8MzVq1Mjc4rNYVvTNiRMnOH/+PB06dEh53Ww2A2BjY8Pp06cpX758Jh9J5suqvzclSpTA1tYWa2vrlDaVK1cmLCyMhIQE7OzsMvlIMl9W9c27775Lr169GDhwIADVq1cnJiaGwYMHM3bsWKyscvbVBY/TL+nxoL77N/vMblnVN3/Lb9/Dj3L27FmLfQ/n7H+l+YydnR0+Pj5s27Yt1fZt27bRsGHDNN/ToEGD+9pv3boVX19fbG1tU7ZNmTKFCRMm8NNPP+Hr65v5xWexrOibSpUqcezYMYKDg1MeHTt2pFmzZgQHB+Pp6Zllx5OZsurvTaNGjfjrr79SvowA/vzzT0qUKJErwg9kXd/ExsbeF3Ksra0x7t1YkolHkDUep1/S40F992/2md2yqm8gf34PP4pFv4ez/DJryZC/bzFcuHChcfLkSWPkyJFGwYIFjfPnzxuGYRhjxowxevXqldL+71t2R40aZZw8edJYuHDhfbfsTp482bCzszPWrFljhIaGpjyioqKy/fj+jazom3/KrXeBZUXfXLx40ShUqJDx6quvGqdPnza+//57o1ixYsZHH32U7cf3b2RF34wfP95wcnIyAgICjJCQEGPr1q1G+fLlja5du2b78T2ujPaLYRhGUFCQERQUZPj4+Bg9evQwgoKCjBMnTqS8vnfvXsPa2tr45JNPjFOnThmffPJJrr4NPjP7Jr9+DxvGo/vmn7Lre1gBKAeaPXu24eXlZdjZ2Rl16tQxdu7cmfJanz59jKeffjpV+x07dhi1a9c27OzsDG9vb2Pu3LmpXvfy8jKA+x7jx4/PhqPJXJndN/+UWwOQYWRN3+zbt8+oX7++YW9vb5QrV874+OOPjaSkpKw+lEyX2X2TmJhovP/++0b58uUNBwcHw9PT03jllVeM27dvZ8PRZJ6M9kta3yNeXl6p2nz77bdGxYoVDVtbW6NSpUrG2rVrs+FIMl9m901+/h5Oz9+b/5Vd38Om/xYnIiIikm/oGiARERHJdxSAREREJN9RABIREZF8RwFIRERE8h0FIBEREcl3FIBEREQk31EAEhERkXxHAUhERETyHQUgEcnXzp8/j7e3N9u3b7d0KSKSjRSARCRX2rVrFx06dKBkyZKYTCa+++67NNs1bdqUL7/88oH7Wbp0KbNmzaJ58+bs2LEDk8lEtWrVSE5OTtXO1dWVRYsWZeIRiIglKQCJSK4UExNDzZo1mTVr1gPbhIeHs2/fPjp06PDANlu3bqVNmzaptp09e5YlS5ZkWq0ikvPYWLoAEZHH0bZtW9q2bfvQNj/88AM1a9akVKlSab5+4MABfH19sbFJ/VX4n//8h/Hjx9O9e3ccHBwyrWYRyTk0AiQiedbGjRvp1KlTyvPQ0NBUry9ZsoQ+ffrc976RI0eSlJT00NElEcndFIBEJE+Kj49ny5YtdOrUCbPZzLBhwyhTpgxXrlwBICEhgT/++INatWrd915HR0fGjx/PpEmTiIiIyObKRSQ7KACJSJ60fft23NzcqF69OlZWVsyePRs/Pz9WrFgB3Ds91r59+we+f8CAAbi7uzN58uTsKllEspECkIjkSf88/QXQq1cvli1bBsCKFSvo0aPHA99vY2PDRx99xPTp07l69WqW1ioi2U8BSETyHMMw2LRpEx07dky1vWvXrpw+fZodO3aQnJxM8eLFH7qfF198kapVq/LBBx9kZbkiYgG6C0xEcqXo6Gj++uuvlOfnzp0jODiYIkWKcP36dWJiYnjqqadSvcfV1ZV27drRu3dvPvvss3R9zieffELr1q0ztXYRsTwFIBHJlY4cOUKzZs1Sno8ePRqAPn364OnpybPPPnvf7e0APXv25Ndff71vdOhBmjdvTvPmzdm6dWvmFC4iOYLJMAzD0kWIiGSmGjVqMG7cOLp27XrfawkJCSxcuJChQ4daoDIRySkUgEQkT0lISGDSpEmMHj0aJycnS5cjIjmUApCIiIjkO7oLTERERPIdBSARERHJdxSAREREJN9RABIREZF8RwFIRERE8h0FIBEREcl3FIBEREQk31EAEhERkXxHAUhERETynf8D7PyIE+srizsAAAAASUVORK5CYII=\n", 1090 | "text/plain": [ 1091 | "
" 1092 | ] 1093 | }, 1094 | "metadata": {}, 1095 | "output_type": "display_data" 1096 | } 1097 | ], 1098 | "source": [ 1099 | "import matplotlib.pyplot as plt\n", 1100 | "plt.plot(1 / jnp.sqrt(jnp.array(sample_sizes)), ksd_vals, label=\"KSD\")\n", 1101 | "\n", 1102 | "coef = np.polyfit(1 / jnp.sqrt(jnp.array(sample_sizes)),ksd_vals,1)\n", 1103 | "poly1d_fn = np.poly1d(coef) \n", 1104 | "# poly1d_fn is now a function which takes in x and returns an estimate for y\n", 1105 | "\n", 1106 | "plt.plot(1 / jnp.sqrt(jnp.array(sample_sizes)), poly1d_fn(jnp.array(1 / jnp.sqrt(jnp.array(sample_sizes)))), '--k', label=\"linear fit\")\n", 1107 | "plt.xlabel(\"1/√N\")\n", 1108 | "plt.ylabel(\"KSD(X, p)\")\n", 1109 | "plt.legend()" 1110 | ] 1111 | }, 1112 | { 1113 | "cell_type": "markdown", 1114 | "id": "8aafd0bb-3c90-423a-b969-4ace884b7acf", 1115 | "metadata": {}, 1116 | "source": [ 1117 | "## Control-Flows in JAX\n", 1118 | "\n", 1119 | "Understanding control flows in `jax` requires some additional digging of jax internals:" 1120 | ] 1121 | }, 1122 | { 1123 | "cell_type": "markdown", 1124 | "id": "4534d45e-92f9-472d-a93f-8790dd79c717", 1125 | "metadata": {}, 1126 | "source": [ 1127 | "### A primer on jax internals: static shapes, functional purity" 1128 | ] 1129 | }, 1130 | { 1131 | "cell_type": "markdown", 1132 | "id": "ad38c9fe-1f83-41de-be22-945cddfc1dda", 1133 | "metadata": {}, 1134 | "source": [ 1135 | "\n", 1136 | "- Jax works by transforming a functionally pure function into a directed acyclic computation graph\n", 1137 | "- The interpreted nature of python makes the creation of this graph a challenge: such computation graphs are typically created ahead of time, during the compilation process when working with compiled languages.\n", 1138 | "- In intrepreted languages, the structure of the computational graph will be \"discovered\" at runtime, when executing the said program." 1139 | ] 1140 | }, 1141 | { 1142 | "cell_type": "markdown", 1143 | "id": "3551379f-4939-4459-b35c-cec5e31c201f", 1144 | "metadata": {}, 1145 | "source": [ 1146 | "- To build this graph, jax relies on the amazing polymorphism capabilities of Python by substituting input to jax functions with special jax constructs called \"tracers\".\n", 1147 | "- These tracers progressively record the set of instructions applied on them, gathering the necessary informations to create a computational graph, called a \"Jax Intermediate Representation\", or JAX IR.\n", 1148 | "- This graph can then be traversed to perform computation graph transformations by defining appropritate transformation translation rules, leading to `vmap`, `jit`, `grad` etc.\n", 1149 | "\n", 1150 | "\n", 1151 | "Jax IRs can be described using a 1st order ANF called a `jaxpr`. Example is given below (credits to the Jax Team for the slides)" 1152 | ] 1153 | }, 1154 | { 1155 | "cell_type": "markdown", 1156 | "id": "e0700bcf-6928-4298-b329-a501c130b159", 1157 | "metadata": {}, 1158 | "source": [ 1159 | "![title](./fun_to_jax_ir.png)" 1160 | ] 1161 | }, 1162 | { 1163 | "cell_type": "markdown", 1164 | "id": "4f26f401-37e9-4c90-8ddb-4016b07d7507", 1165 | "metadata": {}, 1166 | "source": [ 1167 | "### Constructing jax-friendly control flows " 1168 | ] 1169 | }, 1170 | { 1171 | "cell_type": "markdown", 1172 | "id": "abe45ea2-fc0a-4171-97d9-f4a371fcab71", 1173 | "metadata": {}, 1174 | "source": [ 1175 | "Additionally, as of now, `jaxpr`s are specialized to specific input and output shapes: thus, these shapes are required to be static, and cannot change depending on the value of the input. Thus, the following function:" 1176 | ] 1177 | }, 1178 | { 1179 | "cell_type": "code", 1180 | "execution_count": 39, 1181 | "id": "03e7e1b7-f98b-4a58-a283-b8c6a2b67b32", 1182 | "metadata": {}, 1183 | "outputs": [], 1184 | "source": [ 1185 | "def f(x):\n", 1186 | " if x > 0:\n", 1187 | " return jnp.ones((2,))\n", 1188 | " else:\n", 1189 | " return jnp.ones((3,))" 1190 | ] 1191 | }, 1192 | { 1193 | "cell_type": "markdown", 1194 | "id": "649b55df-9635-413c-836e-ebb561ee0067", 1195 | "metadata": {}, 1196 | "source": [ 1197 | "will not yield a valid `jaxpr`, as the shape of the output depends on dynamic values (namely the values on the input).\n", 1198 | "Note however that because shapes are treated as static this:" 1199 | ] 1200 | }, 1201 | { 1202 | "cell_type": "code", 1203 | "execution_count": 40, 1204 | "id": "e39493c8-5931-4508-9718-50afaf57d5f3", 1205 | "metadata": {}, 1206 | "outputs": [], 1207 | "source": [ 1208 | "def f(x):\n", 1209 | " if x.shape[0] == 2:\n", 1210 | " return jnp.ones((2,))\n", 1211 | " elif x.shape[0] == 3:\n", 1212 | " return jnp.ones((3,))\n", 1213 | " else:\n", 1214 | " raise ValueError" 1215 | ] 1216 | }, 1217 | { 1218 | "cell_type": "markdown", 1219 | "id": "4ad67505-d14e-46c4-8ad6-5124d3a4e28b", 1220 | "metadata": {}, 1221 | "source": [ 1222 | "will yield a valid `jaxpr`, and can thus be subject to transformations.\n" 1223 | ] 1224 | }, 1225 | { 1226 | "cell_type": "markdown", 1227 | "id": "9488908a-ba58-49bf-a932-16fc49c6c7d7", 1228 | "metadata": {}, 1229 | "source": [ 1230 | "#### Jax Conditionals\n", 1231 | "\n", 1232 | "Note however, that even a modified version of the first function that would return outputs with identical shapes on both branches:" 1233 | ] 1234 | }, 1235 | { 1236 | "cell_type": "code", 1237 | "execution_count": 41, 1238 | "id": "a259c020-0222-499b-9e15-7dbe8c4ffc73", 1239 | "metadata": {}, 1240 | "outputs": [], 1241 | "source": [ 1242 | "def f(x):\n", 1243 | " if x > 0:\n", 1244 | " return jnp.ones((2,))\n", 1245 | " else:\n", 1246 | " return jnp.zeros((2,))" 1247 | ] 1248 | }, 1249 | { 1250 | "cell_type": "markdown", 1251 | "id": "4973897c-0388-49e9-8ea0-bb89683dad5d", 1252 | "metadata": {}, 1253 | "source": [ 1254 | "is fundamentally incompatible with a tracing-based mechanism: within a single function call, a tracer will visit only one branch,\n", 1255 | "leading to an incomplete construction of the funciton's computational graph.\n", 1256 | "\n", 1257 | "To solve this problem, jax exposes special control-flow primitives to be used in-lieu of Python control flows. Here is a jax-compatible rewrite of `f`,\n", 1258 | "using `jax.lax.cond`, the native jax if-statement primitive:" 1259 | ] 1260 | }, 1261 | { 1262 | "cell_type": "code", 1263 | "execution_count": 42, 1264 | "id": "0678e724-7cc2-4bdd-ac0e-5b230b0b7d15", 1265 | "metadata": {}, 1266 | "outputs": [], 1267 | "source": [ 1268 | "import jax.lax\n", 1269 | "\n", 1270 | "def f(x):\n", 1271 | " return jax.lax.cond(x > 0, lambda: jnp.ones((2,)), lambda: jnp.zeros((2,)))" 1272 | ] 1273 | }, 1274 | { 1275 | "cell_type": "code", 1276 | "execution_count": 43, 1277 | "id": "23fde93a-338d-45c6-930b-106b4834c171", 1278 | "metadata": {}, 1279 | "outputs": [ 1280 | { 1281 | "name": "stdout", 1282 | "output_type": "stream", 1283 | "text": [ 1284 | "f(-1.)=DeviceArray([0., 0.], dtype=float32)\n", 1285 | "f(2.)=DeviceArray([1., 1.], dtype=float32)\n" 1286 | ] 1287 | } 1288 | ], 1289 | "source": [ 1290 | "print(f\"{f(-1.)=}\")\n", 1291 | "print(f\"{f(2.)=}\")" 1292 | ] 1293 | }, 1294 | { 1295 | "cell_type": "markdown", 1296 | "id": "db9f0aed-2a14-4da5-9b1e-3fd5d4a76bc3", 1297 | "metadata": {}, 1298 | "source": [ 1299 | "#### Jax for-loops" 1300 | ] 1301 | }, 1302 | { 1303 | "cell_type": "markdown", 1304 | "id": "8aada819-eabe-4c3d-8f26-ef12fcc6048e", 1305 | "metadata": {}, 1306 | "source": [ 1307 | "for `for-loop` primitives, something different is at stake. Indeed, consider the following, (jax-valid) function" 1308 | ] 1309 | }, 1310 | { 1311 | "cell_type": "code", 1312 | "execution_count": 44, 1313 | "id": "8ca4099f-802d-4830-8e2a-90e72f5adf52", 1314 | "metadata": {}, 1315 | "outputs": [], 1316 | "source": [ 1317 | "def f(x):\n", 1318 | " for i in range(100):\n", 1319 | " x = x+1\n", 1320 | " return x" 1321 | ] 1322 | }, 1323 | { 1324 | "cell_type": "markdown", 1325 | "id": "dd13ca3b-7119-4268-b8c5-727f90815b19", 1326 | "metadata": {}, 1327 | "source": [ 1328 | "Unlike for the if-statement case, a jax-tracer will completely visit `f`'s computational graph. However, the tracer will not be aware of the loop-structure:\n", 1329 | "\n", 1330 | "- Looking at the resulting computational graph, the loop will appear \"unrolled\", e.g. each iteration yielding it's independent sequence of operations.\n", 1331 | "- Statically unrolling python for-loops can become costly when the number of iterations increases, increasing the compilation time." 1332 | ] 1333 | }, 1334 | { 1335 | "cell_type": "markdown", 1336 | "id": "cdd8878b-8bc5-4411-9391-954ccf784a74", 1337 | "metadata": {}, 1338 | "source": [ 1339 | "- As you may have figured it, jax exposes a for-loop special primitive, that lowers the entire loop to a single HLO For node.\n", 1340 | "- Using `jax.lax.cond` the user tells much more about the structure of the for loop than using a dynamic Pythonic for loop.\n", 1341 | "- In particular, all outputs of `body_fun` must keep the same shape across iterations" 1342 | ] 1343 | }, 1344 | { 1345 | "cell_type": "code", 1346 | "execution_count": 45, 1347 | "id": "72a2b700-4cdc-428b-91b1-ac3bdb4063fb", 1348 | "metadata": {}, 1349 | "outputs": [], 1350 | "source": [ 1351 | "import jax.lax\n", 1352 | "\n", 1353 | "def f_fori_loop(x):\n", 1354 | " def body_fun(i, x):\n", 1355 | " return x+1\n", 1356 | " return jax.lax.fori_loop(0, 100, body_fun, x)" 1357 | ] 1358 | }, 1359 | { 1360 | "cell_type": "code", 1361 | "execution_count": 46, 1362 | "id": "984c76be-96c0-42c7-838f-4e0478f59c93", 1363 | "metadata": {}, 1364 | "outputs": [], 1365 | "source": [ 1366 | "assert f(1) == f_fori_loop(1)" 1367 | ] 1368 | }, 1369 | { 1370 | "cell_type": "markdown", 1371 | "id": "16e2ce1a-3534-4f12-9db4-7a4991f9a7d8", 1372 | "metadata": {}, 1373 | "source": [ 1374 | "Note that this `fori_loop` does not capture intermediates iterates, but only the final iterate. Instead, use `jax.lax.scan` to capture all intermediate iterates." 1375 | ] 1376 | }, 1377 | { 1378 | "cell_type": "markdown", 1379 | "id": "cda11e2e-3f8c-4de4-a25d-a5c4d76a0a2f", 1380 | "metadata": {}, 1381 | "source": [ 1382 | "### Exercise 4: implementing a Monte-Carlo Markov Chain Algorithm\n", 1383 | "\n", 1384 | "\n", 1385 | "- A Monte-Carlo Markov Chain algorithm is an algorithm that can be used to approximately sample from an unnormalized probabilty distribution p.\n", 1386 | "- MCMC algorithm srepeateadly generates iterates $x_i$ by drawing them from a **Markov Kernel** $k(x_{i-1}, \\cdot)$, a distributution parametrized by $x_{i-1}$ and which admits $p$ as an invariant distribution:\n", 1387 | "\n", 1388 | "$$\n", 1389 | "\\int k(x, y) p(x) = p(y)\n", 1390 | "$$\n", 1391 | "\n", 1392 | "A famous family of kernels are Metropolis Hasting kernels, which, given some input $x$ (typically, $x_{i-1}$ during a loop), are parametrized by a proposal distribution $q(\\cdot | x)$ and are described by the following probabilistic program:\n", 1393 | "\n", 1394 | "\n", 1395 | "**How to sample from $k(\\cdot, x)$**\n", 1396 | "\n", 1397 | "- generate $x' \\sim q(\\cdot|x)$\n", 1398 | "- $\\alpha = \\frac{p(x')q(x | x')}{p(x)q(x'|x)}$\n", 1399 | "- with prob. $\\min(\\alpha, 1)$, return x'. Else, return x." 1400 | ] 1401 | }, 1402 | { 1403 | "cell_type": "markdown", 1404 | "id": "6f6f72cc-b615-43d2-88f9-9ae146c0283a", 1405 | "metadata": {}, 1406 | "source": [ 1407 | "The goal of this exercise is to write an efficient implementation of a MCMC algorithm with for some target density $p(x)$ given below, random walk MH kernel, which is a MH kernel characetrized by the proposal distribution $q(x, y) = \\mathcal N(x-y, \\sigma^2 I_2)$. \n", 1408 | "\n", 1409 | "Importantly, this proposal is symmetric $q(x|x')=q(x'|x)$. Thus, the computation of $\\alpha$ reduces to $p(x')/p(x)$.\n", 1410 | " \n", 1411 | "\n", 1412 | "To complete this algorithm you will have to:\n", 1413 | "- implement the random walk kernel\n", 1414 | "- create the for-loop representing the MCMC algorith, using a jax.lax.scan primitive" 1415 | ] 1416 | }, 1417 | { 1418 | "cell_type": "code", 1419 | "execution_count": 47, 1420 | "id": "f5bf9063-affe-4c2c-8e80-09949d87e2d7", 1421 | "metadata": {}, 1422 | "outputs": [], 1423 | "source": [ 1424 | "def p(x):\n", 1425 | " \n", 1426 | " # the unnormalized density of interest\n", 1427 | " sigma = 1.\n", 1428 | " return jnp.exp(-0.5/sigma**2 * jnp.sum(jnp.square(x - jnp.ones((2,)))))" 1429 | ] 1430 | }, 1431 | { 1432 | "cell_type": "code", 1433 | "execution_count": 48, 1434 | "id": "6d878b9d-068e-46d9-a74b-cbff7c7c19df", 1435 | "metadata": {}, 1436 | "outputs": [], 1437 | "source": [ 1438 | "# the proposal distribution q(\\cdot | x)\n", 1439 | "def q(x, y):\n", 1440 | " return jnp.exp(-0.5/sigma**2 * jnp.sum(jnp.square(x - y)))\n", 1441 | "\n", 1442 | "def random_walk_kernel(x, key, sigma=1):\n", 1443 | " # 1. generate x' from q(x, \\cdot). Hint: use a shifted and scaled sample from random.normal.\n", 1444 | " # 2. compute \\alpha\n", 1445 | " # 3. return x or x'.\n", 1446 | " raise NotImplemented" 1447 | ] 1448 | }, 1449 | { 1450 | "cell_type": "markdown", 1451 | "id": "f75e6c74-b459-46a2-9310-3378b3151c40", 1452 | "metadata": {}, 1453 | "source": [ 1454 | " Careful! there should not be native if statements! try running a jitted version of `random_walk_kernel`\n", 1455 | " \n" 1456 | ] 1457 | }, 1458 | { 1459 | "cell_type": "code", 1460 | "execution_count": 49, 1461 | "id": "f51cba4e-f50b-4853-ae61-b6ac4c6e2904", 1462 | "metadata": {}, 1463 | "outputs": [ 1464 | { 1465 | "ename": "TypeError", 1466 | "evalue": "exceptions must derive from BaseException", 1467 | "output_type": "error", 1468 | "traceback": [ 1469 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 1470 | "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", 1471 | "Input \u001b[0;32mIn [49]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 1\u001b[0m x0 \u001b[38;5;241m=\u001b[39m jnp\u001b[38;5;241m.\u001b[39mzeros((\u001b[38;5;241m2\u001b[39m,))\n\u001b[1;32m 2\u001b[0m key \u001b[38;5;241m=\u001b[39m random\u001b[38;5;241m.\u001b[39mPRNGKey(\u001b[38;5;241m0\u001b[39m)\n\u001b[0;32m----> 4\u001b[0m x1 \u001b[38;5;241m=\u001b[39m \u001b[43mrandom_walk_kernel\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx0\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkey\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5\u001b[0m x1_jitted \u001b[38;5;241m=\u001b[39m jit(random_walk_kernel)(x0, key)\n\u001b[1;32m 7\u001b[0m \u001b[38;5;28;01massert\u001b[39;00m jnp\u001b[38;5;241m.\u001b[39mallclose(x1, x1_jitted)\n", 1472 | "Input \u001b[0;32mIn [48]\u001b[0m, in \u001b[0;36mrandom_walk_kernel\u001b[0;34m(x, key, sigma)\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrandom_walk_kernel\u001b[39m(x, key, sigma\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m 6\u001b[0m \u001b[38;5;66;03m# 1. generate x' from q(x, \\cdot). Hint: use a shifted and scaled sample from random.normal.\u001b[39;00m\n\u001b[1;32m 7\u001b[0m \u001b[38;5;66;03m# 2. compute \\alpha\u001b[39;00m\n\u001b[1;32m 8\u001b[0m \u001b[38;5;66;03m# 3. return x or x'.\u001b[39;00m\n\u001b[0;32m----> 9\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mNotImplemented\u001b[39m\n", 1473 | "\u001b[0;31mTypeError\u001b[0m: exceptions must derive from BaseException" 1474 | ] 1475 | } 1476 | ], 1477 | "source": [ 1478 | "x0 = jnp.zeros((2,))\n", 1479 | "key = random.PRNGKey(0)\n", 1480 | "\n", 1481 | "x1 = random_walk_kernel(x0, key)\n", 1482 | "x1_jitted = jit(random_walk_kernel)(x0, key)\n", 1483 | "\n", 1484 | "assert jnp.allclose(x1, x1_jitted)" 1485 | ] 1486 | }, 1487 | { 1488 | "cell_type": "markdown", 1489 | "id": "43495098-33bd-4b44-bd97-a21042aafb64", 1490 | "metadata": {}, 1491 | "source": [ 1492 | "For the next step, implement a full MCMC algorithm by iteratively sampling from this kernel.\n", 1493 | "Below is a for-loop equivalent:" 1494 | ] 1495 | }, 1496 | { 1497 | "cell_type": "code", 1498 | "execution_count": null, 1499 | "id": "945d3e58-266e-4f9d-acfd-bd2b89c27071", 1500 | "metadata": {}, 1501 | "outputs": [], 1502 | "source": [ 1503 | "def rw_mcmc(num_steps, x0, key):\n", 1504 | " x = x0\n", 1505 | " iterates = [x]\n", 1506 | " for i in range(steps):\n", 1507 | " key, subkey = random.split(key)\n", 1508 | " x = random_walk_kernel(x, subkey)\n", 1509 | " iterates.append(x)\n", 1510 | " return iterates" 1511 | ] 1512 | }, 1513 | { 1514 | "cell_type": "markdown", 1515 | "id": "009f8a91-7032-412f-a75d-b38ab7a684ce", 1516 | "metadata": {}, 1517 | "source": [ 1518 | "Fill the following code:" 1519 | ] 1520 | }, 1521 | { 1522 | "cell_type": "code", 1523 | "execution_count": null, 1524 | "id": "87aeeb01-a985-4e26-9e72-09bed56ccbf2", 1525 | "metadata": {}, 1526 | "outputs": [], 1527 | "source": [ 1528 | "def rw_mcmc_jax_control_flow(num_steps, x0, key):\n", 1529 | " x = x0\n", 1530 | " iterates = [x]\n", 1531 | " \n", 1532 | " def step_fn(carry, input_):\n", 1533 | " output = ... # do a MH kernel step\n", 1534 | " return output, output\n", 1535 | " \n", 1536 | " # call jax.lax.scan using step_fn as its step function.\n", 1537 | " _, iterates = jax.lax.scan(...)\n", 1538 | " return iterates" 1539 | ] 1540 | }, 1541 | { 1542 | "cell_type": "markdown", 1543 | "id": "3f963ff5-81b2-47d1-9144-90492a3d5560", 1544 | "metadata": {}, 1545 | "source": [ 1546 | "You will notice how much faster the version of `rw_mcmc` using jax.lax control flow is as opposed to its Python counterpart!" 1547 | ] 1548 | }, 1549 | { 1550 | "cell_type": "markdown", 1551 | "id": "0d96037f-e74f-402c-84d1-465a485315ac", 1552 | "metadata": {}, 1553 | "source": [ 1554 | "## Extending the flexibility of jax functions with PyTrees\n", 1555 | "\n", 1556 | "\n", 1557 | "### A Primer on PyTrees \n", 1558 | "\n", 1559 | "The aforementioned fact that valid `jax` functions could only take `jax` arrays as inputs and return `jax` arrays is a slight reduction. Actually, jax functions can take as inputs arbitrary nested collections JAX arrays. These nested collections are called PyTrees.\n", 1560 | "\n", 1561 | "- Collections that defines valid PyTrees by defaults are tuples, namedtuple, lists, dictionaries. PyTrees are characetrized by the structure or the nested collections (where the cardinality of each collection is a static attribute of the PyTree structure, i.e cannot change across instances of this pytree structure). Such collections have a tree structure (hence the name), and the leaves are the array.\n", 1562 | "- The structure of the PyTree is called a PyTreeDef" 1563 | ] 1564 | }, 1565 | { 1566 | "cell_type": "code", 1567 | "execution_count": null, 1568 | "id": "38782b7d-6956-4905-8f24-014f73681128", 1569 | "metadata": {}, 1570 | "outputs": [], 1571 | "source": [ 1572 | "# Example\n", 1573 | "some_pytree = {'a': jnp.ones((2,)), 'b': jnp.ones((2,))}" 1574 | ] 1575 | }, 1576 | { 1577 | "cell_type": "markdown", 1578 | "id": "6becb5a7-3abc-4dc0-9335-ec7aa2b72f0c", 1579 | "metadata": {}, 1580 | "source": [ 1581 | "The principles applying to jax arrays mostly generalize to PyTrees. For instances, shapes must be known statically. This function for example is invalid jax code" 1582 | ] 1583 | }, 1584 | { 1585 | "cell_type": "code", 1586 | "execution_count": null, 1587 | "id": "c8ca3b35-d4cd-4918-9df4-76d2b93b60bd", 1588 | "metadata": {}, 1589 | "outputs": [], 1590 | "source": [ 1591 | "def f():\n", 1592 | " return jax.lax.cond(True, lambda: [jnp.ones((2,)), jnp.ones((3,))], lambda: [jnp.ones((3,)), jnp.ones((3,))])" 1593 | ] 1594 | }, 1595 | { 1596 | "cell_type": "markdown", 1597 | "id": "0d64ce41-986e-4e50-adb8-57ca62423d59", 1598 | "metadata": {}, 1599 | "source": [ 1600 | "Pytrees can be differentiated of vmapped against:" 1601 | ] 1602 | }, 1603 | { 1604 | "cell_type": "code", 1605 | "execution_count": null, 1606 | "id": "23648821-a677-4c8c-8e48-f37c1da6f75e", 1607 | "metadata": {}, 1608 | "outputs": [], 1609 | "source": [ 1610 | "def f(tree):\n", 1611 | " return 2 * tree['a']['c'] * tree['b']" 1612 | ] 1613 | }, 1614 | { 1615 | "cell_type": "code", 1616 | "execution_count": null, 1617 | "id": "bccf93c7-9fc3-4b4b-a449-d12461f60e04", 1618 | "metadata": {}, 1619 | "outputs": [], 1620 | "source": [ 1621 | "grad_f = grad(f)\n", 1622 | "grad_f({'a': {'c': 10.}, 'b': -2.})" 1623 | ] 1624 | }, 1625 | { 1626 | "cell_type": "code", 1627 | "execution_count": null, 1628 | "id": "bcc247d1-6d55-4769-b83d-153fa642c36f", 1629 | "metadata": {}, 1630 | "outputs": [], 1631 | "source": [ 1632 | "vmapped_f = vmap(f)\n", 1633 | "vmapped_f({'a': {'c': 10. * jnp.ones((10,))}, 'b': -2. * jnp.arange(10)})" 1634 | ] 1635 | }, 1636 | { 1637 | "cell_type": "markdown", 1638 | "id": "da0e5217-c07d-4dc0-8f4d-b7d494b9680e", 1639 | "metadata": {}, 1640 | "source": [ 1641 | "All such operations can be done only on part of the PyTree structure of each argument:" 1642 | ] 1643 | }, 1644 | { 1645 | "cell_type": "code", 1646 | "execution_count": null, 1647 | "id": "0d9446ba-ba53-4832-b7a8-165221182cac", 1648 | "metadata": {}, 1649 | "outputs": [], 1650 | "source": [ 1651 | "vmapped_f_partial = vmap(f, in_axes=({'a': {'c': 0}, 'b': None}, )) # a bit error-prone: this is currently a jax usability weak spot.\n", 1652 | "vmapped_f_partial({'a': {'c': 10. * jnp.ones((10,))}, 'b': -2.})" 1653 | ] 1654 | }, 1655 | { 1656 | "cell_type": "markdown", 1657 | "id": "927a6a98-60d7-4ca1-b786-991a66f2b18e", 1658 | "metadata": {}, 1659 | "source": [ 1660 | " pytrees are useful to represent the internal state (e.g. the weights, the batch norm statistics, etc) of a neural network, as often, neural network have a nested/hierechical/modular structure." 1661 | ] 1662 | }, 1663 | { 1664 | "cell_type": "markdown", 1665 | "id": "3aab100e-83a9-4fc3-ac47-6b7126ed7ce0", 1666 | "metadata": {}, 1667 | "source": [ 1668 | "### Convenients PyTrees\n", 1669 | "\n", 1670 | "Two important remarks on PyTree are in order:\n", 1671 | "\n", 1672 | "- NamedTuple are PyTree. Using namedtuples often prevent extenive kwargs plumbing in functions, by gathering all \"configuration\" arguments (as opposed to actual variables) to a function within a `Config` NamedTuple Umbrella (note that all attributes of the named tuple must be tracable!)." 1673 | ] 1674 | }, 1675 | { 1676 | "cell_type": "code", 1677 | "execution_count": null, 1678 | "id": "8036103b-2eb7-4c82-835b-287849917fde", 1679 | "metadata": {}, 1680 | "outputs": [], 1681 | "source": [ 1682 | "from typing import NamedTuple\n", 1683 | "class MyAlgorithmConfig(NamedTuple):\n", 1684 | " learning_rate: int = 0.01\n", 1685 | " regularization_val: int = 0.001\n", 1686 | " # careful! if the training loop was implememted as a jax.lax.scan loop,\n", 1687 | " # num_iter will have to be static, as it impacts the output shape of the scan loop\n", 1688 | " # num_iter = 100\n", 1689 | " ..." 1690 | ] 1691 | }, 1692 | { 1693 | "cell_type": "code", 1694 | "execution_count": null, 1695 | "id": "110ec796-b937-4f85-908a-888ae62e0ae8", 1696 | "metadata": {}, 1697 | "outputs": [], 1698 | "source": [ 1699 | "def train_my_network(X, y, my_algorithm_config: MyAlgorithmConfig,\n", 1700 | " # num_iter = 100\n", 1701 | " ):\n", 1702 | " ...\n", 1703 | " \n", 1704 | "# instead of\n", 1705 | "def train_my_network(X, y, learning_rate, regularization_val, \n", 1706 | " # num_iter = 100\n", 1707 | " ):\n", 1708 | " ..." 1709 | ] 1710 | }, 1711 | { 1712 | "cell_type": "markdown", 1713 | "id": "a1d21c30-1722-47e8-afa9-b703799e0d1f", 1714 | "metadata": {}, 1715 | "source": [ 1716 | "- Additional User-Defined Collection Types can be registered as PyTreeNodes, using the jax.tree_util.register_pytree_node_class\n", 1717 | "- However, `flax`, a jax-powered neural network library, exposes a `struct.PyTreeNode` helper allowing the extremly useful operation:\n", 1718 | " - automatically register dataclass-like classes...\n", 1719 | " - ...while manually specifying which arguments are static and which are traceable! Thanks to this, `num_iter` in the previous example can be safely ignored:\n", 1720 | " " 1721 | ] 1722 | }, 1723 | { 1724 | "cell_type": "code", 1725 | "execution_count": null, 1726 | "id": "fda057f5-afaf-4a2b-b737-cc3b9f8095c2", 1727 | "metadata": {}, 1728 | "outputs": [], 1729 | "source": [ 1730 | "from flax import struct\n", 1731 | "\n", 1732 | "class MyAlgorithmConfig(struct.PyTreeNode):\n", 1733 | " learning_rate: int = 0.01\n", 1734 | " regularization_val: int = 0.001\n", 1735 | " # num_iter can be marked as static!\n", 1736 | " num_iter: int = struct.field(pytree_node=False, default=100)" 1737 | ] 1738 | }, 1739 | { 1740 | "cell_type": "markdown", 1741 | "id": "6d550e43-baa4-4e97-9eb5-e17c3c943ef3", 1742 | "metadata": {}, 1743 | "source": [ 1744 | "## A final End-to-end example. Mixture of Gaussians:" 1745 | ] 1746 | }, 1747 | { 1748 | "cell_type": "markdown", 1749 | "id": "8d682d61-4542-427b-b6e3-aad6e186e17b", 1750 | "metadata": {}, 1751 | "source": [ 1752 | "### What This Example shows" 1753 | ] 1754 | }, 1755 | { 1756 | "cell_type": "markdown", 1757 | "id": "3bcf1ebc-122d-4266-bb07-08e1b783abab", 1758 | "metadata": {}, 1759 | "source": [ 1760 | "This final example is an end-to-end macro-example showcasing the capabilities of jax on a end-to-end algorithm with a few subtlelties: a mixture of gaussian algorithms. In particular, this examples demostrates:\n", 1761 | "\n", 1762 | "- the functional nature of jax through the use of functional slice setting semantics (x = x.at[i].set(y))\n", 1763 | "- the use of user-defined PyTrees, both as input and output algorithm\n", 1764 | "- top-level use of jax.vmap to seamlessly vectorize the algorithm over random initializations" 1765 | ] 1766 | }, 1767 | { 1768 | "cell_type": "markdown", 1769 | "id": "524f1124-831e-46d0-ab3f-3d1a617c1fb7", 1770 | "metadata": {}, 1771 | "source": [ 1772 | "### Show Me the Code" 1773 | ] 1774 | }, 1775 | { 1776 | "cell_type": "code", 1777 | "execution_count": null, 1778 | "id": "4c45c09b-fbfe-4956-9e06-71ef7ddd9d93", 1779 | "metadata": {}, 1780 | "outputs": [], 1781 | "source": [ 1782 | "from typing import NamedTuple\n", 1783 | "\n", 1784 | "import jax\n", 1785 | "import jax.numpy as jnp\n", 1786 | "from flax import struct\n", 1787 | "from jax import random\n", 1788 | "from jax._src.api import vmap\n", 1789 | "from jax.nn import log_softmax, logsumexp\n", 1790 | "from jax.tree_util import tree_map\n", 1791 | "from jax.flatten_util import ravel_pytree # type: ignore\n", 1792 | "from numpyro import distributions as np_distributions\n", 1793 | "import numpy as np\n", 1794 | "\n", 1795 | "from typing import Any\n", 1796 | "Array, Numeric, PRNGKeyArray = Any, Any, Any" 1797 | ] 1798 | }, 1799 | { 1800 | "cell_type": "code", 1801 | "execution_count": null, 1802 | "id": "12fdf9e5-cb5a-44d6-a55c-b55b077be951", 1803 | "metadata": {}, 1804 | "outputs": [], 1805 | "source": [ 1806 | "class MOGDistribution(np_distributions.Distribution):\n", 1807 | " def __init__(self, cluster_means: Array, cluster_covs: Array, cluster_props: Array):\n", 1808 | " assert len(cluster_means.shape) == 2\n", 1809 | " self._num_clusters, self._num_dims = cluster_means.shape\n", 1810 | " self.cluster_means = cluster_means\n", 1811 | "\n", 1812 | " assert len(cluster_props.shape) == 1\n", 1813 | " self.cluster_props = cluster_props\n", 1814 | "\n", 1815 | " self.cluster_covs = cluster_covs\n", 1816 | "\n", 1817 | " self._dists = np_distributions.MultivariateNormal(\n", 1818 | " cluster_means, covariance_matrix=cluster_covs\n", 1819 | " )\n", 1820 | "\n", 1821 | " super(MOGDistribution, self).__init__(\n", 1822 | " batch_shape=(), event_shape=(self._num_dims,)\n", 1823 | " )\n", 1824 | "\n", 1825 | " def log_prob(self, x: Array):\n", 1826 | " assert len(x.shape) == 1\n", 1827 | " return logsumexp(self._dists.log_prob(x), b=self.cluster_props)\n", 1828 | "\n", 1829 | " def _sample_from_cluster_idx(self, key, idx):\n", 1830 | " return tree_map(lambda d: d[idx], self._dists).sample(key)\n", 1831 | "\n", 1832 | " def sample(self, key: PRNGKeyArray, sample_shape: tuple = ()) -> Array:\n", 1833 | " if sample_shape == tuple():\n", 1834 | " sample_shape = (1,)\n", 1835 | "\n", 1836 | " key, key_latent = random.split(key)\n", 1837 | "\n", 1838 | " mn = np_distributions.Categorical(probs=self.cluster_props)\n", 1839 | " idxs = mn.sample(key_latent, sample_shape=sample_shape)\n", 1840 | " keys_observed = random.split(key, num=sample_shape[0])\n", 1841 | " return vmap(self._sample_from_cluster_idx, in_axes=(0, 0))( # type: ignore\n", 1842 | " keys_observed, idxs\n", 1843 | " )" 1844 | ] 1845 | }, 1846 | { 1847 | "cell_type": "code", 1848 | "execution_count": null, 1849 | "id": "ed3896a3-f636-40b5-96ce-6c9d1a771af2", 1850 | "metadata": {}, 1851 | "outputs": [], 1852 | "source": [ 1853 | "class MOGTrainingConfig(struct.PyTreeNode):\n", 1854 | " num_clusters: int = struct.field(pytree_node=False)\n", 1855 | " max_iter: int = struct.field(pytree_node=False)\n", 1856 | " num_inits: int = struct.field(pytree_node=False)\n", 1857 | " min_std: float = struct.field(pytree_node=True, default=0.01)\n", 1858 | " max_train_samples: int = struct.field(pytree_node=False, default=1000)\n", 1859 | " cov_reg_param: int = struct.field(pytree_node=False, default=1e-6)" 1860 | ] 1861 | }, 1862 | { 1863 | "cell_type": "code", 1864 | "execution_count": null, 1865 | "id": "ec1d8153-fd45-4cfa-a9f0-2f8633c7e257", 1866 | "metadata": {}, 1867 | "outputs": [], 1868 | "source": [ 1869 | "def _kmeans_plus_plus_init(data: Array, num_clusters: int, key: PRNGKeyArray) -> Array:\n", 1870 | " num_points, num_dim = data.shape\n", 1871 | " clusters = jnp.empty((num_clusters, num_dim))\n", 1872 | "\n", 1873 | " init_cluster_data_idx = random.choice(key, a=num_points)\n", 1874 | " init_cluster_center = data[init_cluster_data_idx, :]\n", 1875 | "\n", 1876 | " this_cluster_center = init_cluster_center\n", 1877 | " clusters = clusters.at[0, :].set(this_cluster_center)\n", 1878 | "\n", 1879 | " all_sq_dists = jnp.inf * jnp.ones((num_points, num_clusters))\n", 1880 | "\n", 1881 | " for i in range(num_clusters - 1):\n", 1882 | " sq_dists = jnp.sum(jnp.square(data - this_cluster_center), axis=1)\n", 1883 | "\n", 1884 | " all_sq_dists = all_sq_dists.at[:, i].set(sq_dists)\n", 1885 | " min_sq_dists = jnp.min(all_sq_dists, axis=1)\n", 1886 | "\n", 1887 | " key, subkey = random.split(key)\n", 1888 | " next_cluster_idx = random.categorical(\n", 1889 | " subkey, logits=jnp.log(min_sq_dists + 1e-15)\n", 1890 | " )\n", 1891 | " next_cluster_center = data[next_cluster_idx]\n", 1892 | "\n", 1893 | " this_cluster_center = next_cluster_center\n", 1894 | " clusters = clusters.at[i + 1, :].set(this_cluster_center)\n", 1895 | "\n", 1896 | " return clusters" 1897 | ] 1898 | }, 1899 | { 1900 | "cell_type": "code", 1901 | "execution_count": null, 1902 | "id": "9f7c926c-9a55-4361-8fde-30a342a82055", 1903 | "metadata": {}, 1904 | "outputs": [], 1905 | "source": [ 1906 | "class MOGResult(NamedTuple):\n", 1907 | " # don't return a MOGDistribution because numpyro Distributions objects are not\n", 1908 | " # vmap-able\n", 1909 | " min_std: float\n", 1910 | " cluster_init: Array\n", 1911 | " cluster_means: Array\n", 1912 | " cluster_covs: Array\n", 1913 | " cluster_props: Array\n", 1914 | " log_probs: Array\n", 1915 | " final_log_prob: Numeric\n", 1916 | " converged: bool\n", 1917 | " num_iter_convergence: Numeric\n", 1918 | "\n", 1919 | " def to_dist(self) -> MOGDistribution:\n", 1920 | " return MOGDistribution(\n", 1921 | " self.cluster_means, self.cluster_covs, self.cluster_props\n", 1922 | " )" 1923 | ] 1924 | }, 1925 | { 1926 | "cell_type": "code", 1927 | "execution_count": null, 1928 | "id": "3d41eeda-49b6-4dec-8bb4-ab01d787dc88", 1929 | "metadata": {}, 1930 | "outputs": [], 1931 | "source": [ 1932 | "def _fit_one_mog(\n", 1933 | " data: Array, num_clusters: int, min_std: float, max_iter: int,\n", 1934 | " max_train_samples: int, cov_reg_param: float, key: PRNGKeyArray\n", 1935 | ") -> MOGResult:\n", 1936 | " num_points, num_dims = data.shape\n", 1937 | "\n", 1938 | " assert len(data.shape) == 2\n", 1939 | " # TODO: kmeans++?\n", 1940 | "\n", 1941 | " if data.shape[0] > max_train_samples:\n", 1942 | " key, subkey = random.split(key)\n", 1943 | " data = random.permutation(subkey, data, axis=0)\n", 1944 | " data = data[:max_train_samples]\n", 1945 | "\n", 1946 | " # init_cluster_data_idx = random.choice(key, a=num_points, shape=(num_clusters,))\n", 1947 | "\n", 1948 | " # cluster_means = data[init_cluster_data_idx]\n", 1949 | " key, key_init = random.split(key)\n", 1950 | " init_cluster_means = _kmeans_plus_plus_init(data, num_clusters, key_init)\n", 1951 | " init_cluster_covs = jnp.stack(\n", 1952 | " [jnp.eye(num_dims) for _ in range(num_clusters)], axis=0\n", 1953 | " )\n", 1954 | "\n", 1955 | " log_cluster_props = -np.log(num_clusters) * jnp.ones((num_clusters,))\n", 1956 | "\n", 1957 | " log_prob = prev_log_prob = -jnp.inf\n", 1958 | "\n", 1959 | " iter_no = 0\n", 1960 | " assert max_iter > 0\n", 1961 | "\n", 1962 | " converged = False\n", 1963 | " num_iter_convergence = 0\n", 1964 | "\n", 1965 | " log_probs = jnp.empty((max_iter,))\n", 1966 | "\n", 1967 | " cluster_means = init_cluster_means\n", 1968 | " cluster_covs = init_cluster_covs\n", 1969 | "\n", 1970 | "\n", 1971 | " for iter_no in range(max_iter):\n", 1972 | " dists = np_distributions.MultivariateNormal(\n", 1973 | " cluster_means,\n", 1974 | " covariance_matrix=cluster_covs,\n", 1975 | " )\n", 1976 | " log_joint = dists.log_prob(data[:, None, :]) + log_cluster_props[None, :]\n", 1977 | "\n", 1978 | " log_prob = logsumexp(log_joint, axis=1).mean()\n", 1979 | " log_probs = log_probs.at[iter_no].set(log_prob)\n", 1980 | "\n", 1981 | " # assert log_prob - prev_log_prob > -1e-6\n", 1982 | " converged = jnp.abs(log_prob - prev_log_prob) < 1e-4\n", 1983 | " num_iter_convergence += 1 - converged\n", 1984 | "\n", 1985 | " # E-step: compute posterior p(z=k|x) (num_points, num_clusters)\n", 1986 | " log_resps = log_softmax(log_joint, axis=1)\n", 1987 | "\n", 1988 | " # M-step\n", 1989 | " # data: (num_points, dim)\n", 1990 | " # normalized_data_weights: (num_points,num_clusters)\n", 1991 | " normalized_data_weights = jax.nn.softmax(log_resps, axis=0)\n", 1992 | " cluster_means = jnp.sum(data[:, None, :] * normalized_data_weights[:, :, None], axis=0)\n", 1993 | "\n", 1994 | " log_cluster_props = jax.nn.log_softmax(jax.nn.logsumexp(log_resps, axis=0))\n", 1995 | "\n", 1996 | " def _compute_cov(mean, weights):\n", 1997 | " return (\n", 1998 | " (data - mean).T @ jnp.diag(weights) @ (data - mean) + cov_reg_param * jnp.eye(num_dims)\n", 1999 | " )\n", 2000 | "\n", 2001 | " cluster_covs = vmap(_compute_cov, in_axes=(0, 1))(cluster_means, normalized_data_weights) # type: ignore\n", 2002 | "\n", 2003 | " prev_log_prob = log_prob\n", 2004 | "\n", 2005 | " def smooth_cov(cov_mat):\n", 2006 | " from jax.numpy.linalg import eigh\n", 2007 | " eigvals, eigvecs = eigh(cov_mat)\n", 2008 | " return eigvecs @ jnp.diag(jnp.clip(jnp.real(eigvals), a_min=min_std ** 2)) @ eigvecs.T\n", 2009 | "\n", 2010 | " cluster_covs = vmap(smooth_cov)(cluster_covs)\n", 2011 | " return MOGResult(\n", 2012 | " min_std,\n", 2013 | " init_cluster_means,\n", 2014 | " cluster_means,\n", 2015 | " cluster_covs,\n", 2016 | " jax.nn.softmax(log_cluster_props),\n", 2017 | " log_probs,\n", 2018 | " log_prob,\n", 2019 | " converged,\n", 2020 | " num_iter_convergence,\n", 2021 | " )" 2022 | ] 2023 | }, 2024 | { 2025 | "cell_type": "code", 2026 | "execution_count": null, 2027 | "id": "2a4cb854-b195-4aa4-b3f5-6beddf8bde9e", 2028 | "metadata": {}, 2029 | "outputs": [], 2030 | "source": [ 2031 | "def fit_mog(data: Array, config: MOGTrainingConfig, key: PRNGKeyArray) -> MOGResult:\n", 2032 | " keys = random.split(key, num=config.num_inits)\n", 2033 | " vmapped_fit = vmap(_fit_one_mog, in_axes=(None, None, None, None, None, None, 0)) # type: ignore\n", 2034 | " rets = vmapped_fit(data, config.num_clusters, config.min_std, config.max_iter,\n", 2035 | " config.max_train_samples, config.cov_reg_param, keys)\n", 2036 | "\n", 2037 | " print(rets.final_log_prob)\n", 2038 | " best_fit_idx = jnp.argmax(rets.final_log_prob)\n", 2039 | " return tree_map(lambda l: l[best_fit_idx], rets)\n" 2040 | ] 2041 | }, 2042 | { 2043 | "cell_type": "code", 2044 | "execution_count": null, 2045 | "id": "bdd8aac9-31fe-48fd-bb8e-77437b83ef14", 2046 | "metadata": {}, 2047 | "outputs": [], 2048 | "source": [ 2049 | "data = jnp.concatenate([\n", 2050 | " np_distributions.Normal(0, 0.25).sample(random.PRNGKey(0), (1000, 2)),\n", 2051 | " np_distributions.Normal(1, 0.25).sample(random.PRNGKey(1), (1000, 2)),\n", 2052 | " np_distributions.Normal(2, 0.25).sample(random.PRNGKey(2), (1000, 2))\n", 2053 | " ], axis=0)\n", 2054 | "\n", 2055 | "# fix the algorithm by increasing the number of clusters\n", 2056 | "ret = fit_mog(data, MOGTrainingConfig(2, 100, 3, 0.01), random.PRNGKey(1))" 2057 | ] 2058 | }, 2059 | { 2060 | "cell_type": "code", 2061 | "execution_count": null, 2062 | "id": "6abf29a7-4fe2-49f8-9bc2-8c3877d013d9", 2063 | "metadata": {}, 2064 | "outputs": [], 2065 | "source": [ 2066 | "simulated_data = ret.to_dist().sample(random.PRNGKey(3), sample_shape=(len(data),))" 2067 | ] 2068 | }, 2069 | { 2070 | "cell_type": "code", 2071 | "execution_count": null, 2072 | "id": "537f7b68-85bf-4659-9a90-db1c000859ef", 2073 | "metadata": {}, 2074 | "outputs": [], 2075 | "source": [ 2076 | "import matplotlib.pyplot as plt\n", 2077 | "f, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 5))\n", 2078 | "ax1.scatter(data[:, 0], data[:, 1], s=3)\n", 2079 | "ax1.set_title('training data')\n", 2080 | "\n", 2081 | "ax2.scatter(simulated_data[:, 0], simulated_data[:, 1], s=3)\n", 2082 | "ax2.set_title('MoG-simulated data')" 2083 | ] 2084 | }, 2085 | { 2086 | "cell_type": "markdown", 2087 | "id": "f46bfced-631c-49a9-8c02-49ed03660f24", 2088 | "metadata": {}, 2089 | "source": [ 2090 | "#### Jax and Functional purity\n", 2091 | "\n", 2092 | "As mentioned above, operates only on functionally pure programs (no side effect).\n", 2093 | "This \"no side effect\" policy, applied to the nested structure of computer programs, imply that variables cannot be modified.\n", 2094 | "\n", 2095 | "- Imposing Such constraints constituted a natural first step for building a framework as complex as jax, being ensuring a Smaller Surface Area for Compilation.\n", 2096 | "- Functionally pure programs imply very handy property called \"referential transparency\". A refenrentially transparent program can be extensively optimized by compilers.\n", 2097 | "\n", 2098 | "The functionaly purity requirement imposed by jax imposes python-program rethinking, especially at the control-flow levels. In addition to yield static shapes, branching and iterations need to be \"purified\"" 2099 | ] 2100 | } 2101 | ], 2102 | "metadata": { 2103 | "kernelspec": { 2104 | "display_name": "Python 3 (ipykernel)", 2105 | "language": "python", 2106 | "name": "python3" 2107 | }, 2108 | "language_info": { 2109 | "codemirror_mode": { 2110 | "name": "ipython", 2111 | "version": 3 2112 | }, 2113 | "file_extension": ".py", 2114 | "mimetype": "text/x-python", 2115 | "name": "python", 2116 | "nbconvert_exporter": "python", 2117 | "pygments_lexer": "ipython3", 2118 | "version": "3.9.13" 2119 | } 2120 | }, 2121 | "nbformat": 4, 2122 | "nbformat_minor": 5 2123 | } 2124 | --------------------------------------------------------------------------------