├── README.md ├── environment.yml ├── environment_windows.yml ├── exercises ├── 01_first_optimization_with_scipy_optimize.ipynb ├── 02_convert_previous_example_to_estimagic.ipynb ├── 03_play_with_algorithm_and_algo_options.ipynb ├── 04_benchmarking.ipynb ├── 05_bounds_and_constraints.ipynb ├── 06_scaling.ipynb ├── 07_automatic_differentiation.ipynb ├── 08_jaxopt.ipynb └── solutions │ ├── 01_first_optimization_with_scipy_optimize.ipynb │ ├── 02_convert_previous_example_to_estimagic.ipynb │ ├── 03_play_with_algorithm_and_algo_options.ipynb │ ├── 04_benchmarking.ipynb │ ├── 05_bounds_and_constraints.ipynb │ ├── 06_scaling.ipynb │ ├── 07_automatic_differentiation.ipynb │ └── 08_jaxopt.ipynb ├── slides.pdf └── test_installation.py /README.md: -------------------------------------------------------------------------------- 1 | # Tutorial - SciPy 2022 2 | 3 | ## Practical Numerical Optimization with SciPy, Estimagic and JAXopt 4 | 5 | > by Janos Gabler & Tim Mensinger 6 | 7 | ## Contents 8 | 9 | 1. [Installation](#installation) 10 | 1. [Slides](#slides) 11 | 1. [Exercises](#exercises) 12 | 1. [Troubleshooting](#troubleshooting) 13 | 14 | > **Warning** Please pull the repo and update your conda environment before the tutorial 15 | > to make sure that the most recent versions are installed. 16 | 17 | ## Installation 18 | 19 | 1. Install [miniconda](https://docs.conda.io/en/latest/miniconda.html) 20 | 21 | 1. Clone the repository 22 | 23 | ```console 24 | $ git clone https://github.com/OpenSourceEconomics/scipy-estimagic.git 25 | $ cd scipy-estimagic 26 | ``` 27 | 28 | 1. Install and activate the environment 29 | 30 | ```console 31 | $ conda env create -f environment.yml 32 | $ conda activate scipy-estimagic 33 | ``` 34 | 35 | > **Note** You have to repeat the activation step each time after closing your 36 | > terminal. 37 | 38 | 1. Test your installation 39 | 40 | ```console 41 | $ python test_installation.py 42 | ``` 43 | 44 | 1. Update the environment 45 | 46 | > **Note** This step is only necessary if you have installed the environment a long 47 | > time ago and want to make sure that you're using the most recent versions. 48 | 49 | ```consolse 50 | $ cd scipy-estimagic 51 | $ conda activate scipy-estimagic 52 | $ conda env update -f environment.yml 53 | ``` 54 | 55 | or use a completely fresh install: 56 | 57 | ```console 58 | $ cd scipy-estimagic 59 | $ conda deactivate scipy-estimagic 60 | $ conda env remove --name scipy-estimagic 61 | $ conda env create -f environment.yml 62 | ``` 63 | 64 | ## Slides 65 | 66 | You can download the slides by clicking 67 | [here](https://github.com/OpenSourceEconomics/scipy-estimagic/raw/main/slides.pdf), or 68 | you can view them directly on GitHub 69 | [here](https://github.com/OpenSourceEconomics/scipy-estimagic/blob/main/slides.pdf). 70 | 71 | ## Exercises 72 | 73 | You find the exercise notebooks in the folder 74 | [exercises](https://github.com/OpenSourceEconomics/scipy-estimagic/tree/main/exercises), 75 | and the corresponding solutions in the subfolder 76 | [exercises/solutions](https://github.com/OpenSourceEconomics/scipy-estimagic/tree/main/exercises/solutions). 77 | 78 | ## Troubleshooting 79 | 80 | If you have questions, problems with the installation or any other part of the 81 | repository, please 82 | [open an issue](https://github.com/OpenSourceEconomics/scipy-estimagic/issues). 83 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: scipy-estimagic 2 | channels: 3 | - conda-forge 4 | - nodefaults 5 | 6 | dependencies: 7 | - python ==3.9 8 | - pip 9 | 10 | # Dependencies 11 | - pandas 12 | - numpy 13 | - scipy 14 | - jupyterlab 15 | - ipython 16 | - seaborn 17 | - jax 18 | - estimagic >=0.4.0 19 | 20 | # Algorithms 21 | - nlopt 22 | - pygmo 23 | 24 | 25 | - pip: 26 | - black 27 | - blackcellmagic 28 | - kaleido 29 | 30 | # Pip Algorithms 31 | - jaxopt 32 | - Py-BOBYQA 33 | - DFO-LS 34 | - fides==0.7.4 35 | -------------------------------------------------------------------------------- /environment_windows.yml: -------------------------------------------------------------------------------- 1 | name: scipy-estimagic 2 | channels: 3 | - conda-forge 4 | - nodefaults 5 | 6 | dependencies: 7 | - python ==3.9 8 | - pip 9 | 10 | # Dependencies 11 | - pandas 12 | - numpy 13 | - scipy 14 | - jupyterlab 15 | - ipython 16 | - seaborn 17 | - estimagic >=0.4.0 18 | 19 | # Algorithms 20 | - nlopt 21 | - pygmo 22 | 23 | 24 | - pip: 25 | - black 26 | - blackcellmagic 27 | - kaleido 28 | 29 | # Pip Algorithms 30 | - Py-BOBYQA 31 | - DFO-LS 32 | - fides==0.7.4 33 | -------------------------------------------------------------------------------- /exercises/01_first_optimization_with_scipy_optimize.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "fe2f76da", 6 | "metadata": {}, 7 | "source": [ 8 | "# First Optimization\n", 9 | "\n", 10 | "In this exercise you will code a criterion function for a numerical optimization from scratch and optimize it with `scipy.optimize`.\n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- [scipy Documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html)\n", 15 | "\n", 16 | "\n", 17 | "## The criterion function\n", 18 | "\n", 19 | "**Notation**:\n", 20 | "\n", 21 | "- $a$ is a scalar floating point number.\n", 22 | "- $b$ is a vector of length 3.\n", 23 | "- $C$ is a 2 by 2 matrix.\n", 24 | "- $I_n$ is a n-dimesional identity matrix. (Use `np.eye`)\n", 25 | "- $|| . ||^2$ is the squared vector or matrix norm. (Use `np.sum((x - y)**2)`, see below for more information)\n", 26 | "\n", 27 | "**Criterion Function**:\n", 28 | "\n", 29 | "$f(a, b, C) = (a - \\pi)^2 + ||b - \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top||^2 + ||C - I_2||^2$\n", 30 | "\n", 31 | "**Optimization problem**:\n", 32 | "\n", 33 | "$\\min_{a, b, C} f(a, b, C)$\n", 34 | "\n", 35 | "**Solution**:\n", 36 | "\n", 37 | "$\n", 38 | "a = \\pi \\\\\n", 39 | "b = \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top \\\\\n", 40 | "C = I_2\n", 41 | "$\n", 42 | "\n", 43 | "\n", 44 | "*Information on norm*:\n", 45 | "The vector norm is the euclidean norm, the matrix norm is the Frobenius norm." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 1, 51 | "id": "0157c5df", 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "import numpy as np\n", 56 | "from scipy.optimize import minimize" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "id": "d1511e95", 62 | "metadata": {}, 63 | "source": [ 64 | "## Task 1: Define the criterion function\n", 65 | "\n", 66 | "Note that for scipy.optimize, x has to be a 1-dimensional numpy array. Thus the first step in your criterion function is to parse the vector `x` into `a`, `b` and `c`." 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 2, 72 | "id": "6901c5c3", 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "def f(x):\n", 77 | " pass" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "id": "0584bd40", 83 | "metadata": {}, 84 | "source": [ 85 | "## Task 2: Set up start parameters\n", 86 | "\n", 87 | "For simplicity, set all start parameters to 0. " 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "id": "1eb12c3f", 93 | "metadata": {}, 94 | "source": [ 95 | "## Task 3: Minimize the criterion function with scipy" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "id": "85cc9c98", 102 | "metadata": {}, 103 | "outputs": [], 104 | "source": [] 105 | } 106 | ], 107 | "metadata": { 108 | "kernelspec": { 109 | "display_name": "Python 3 (ipykernel)", 110 | "language": "python", 111 | "name": "python3" 112 | }, 113 | "language_info": { 114 | "codemirror_mode": { 115 | "name": "ipython", 116 | "version": 3 117 | }, 118 | "file_extension": ".py", 119 | "mimetype": "text/x-python", 120 | "name": "python", 121 | "nbconvert_exporter": "python", 122 | "pygments_lexer": "ipython3", 123 | "version": "3.9.0" 124 | } 125 | }, 126 | "nbformat": 4, 127 | "nbformat_minor": 5 128 | } 129 | -------------------------------------------------------------------------------- /exercises/02_convert_previous_example_to_estimagic.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "a74d1232", 6 | "metadata": {}, 7 | "source": [ 8 | "# Switch to estimagic\n", 9 | "\n", 10 | "In this exercise you will use estimagic to solve the previous exercise.\n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- [Optimization tutorial](https://estimagic.readthedocs.io/en/stable/getting_started/first_optimization_with_estimagic.html)" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "id": "d3be0dbb", 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "import estimagic as em\n", 25 | "import numpy as np" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "id": "3a22a25c", 31 | "metadata": {}, 32 | "source": [ 33 | "## Task 1: Simple switch\n", 34 | "\n", 35 | "- Copy your criterion function and start parameters from the previous example without any changes. \n", 36 | "- Use `em.minimize` to solve the optimization problem. For reasons that will be mentioned later, estimagic has no default algorithm. You can set `algorithm=\"scipy_neldermead\"` for now and will learn how to pick algorithms later. \n", 37 | "- Look at `res.params` and `res.criterion` to make sure the optimization is succeeded." 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "id": "c7dfc656", 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "id": "e0a97d47", 51 | "metadata": {}, 52 | "source": [ 53 | "## Task 2: More flexible `params`\n", 54 | "\n", 55 | "In estimagic, parameters can be arbitrary [pytrees](https://jax.readthedocs.io/en/latest/pytrees.html). In the current example, a dictionary with the keys `\"a\"`, `\"b\"` and `\"c\"` would be a good choice. \n", 56 | "\n", 57 | "- Adjust your function such that it takes a dictionary of parameters instead of a flat numpy array.\n", 58 | "- Re-run the optimization with `em.minimize` with the start parameters given below\n", 59 | "- Look at `res.params` and `res.criterion` to make sure that the optimization worked. " 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "id": "189078bd", 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "start_params = {\n", 70 | " \"a\": 0,\n", 71 | " \"b\": np.zeros(3),\n", 72 | " \"c\": np.zeros((2, 2))\n", 73 | "}\n", 74 | "\n", 75 | "def new_f(x):\n", 76 | " pass" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "id": "9e8065fa", 82 | "metadata": {}, 83 | "source": [ 84 | "## Task 3: Plotting\n", 85 | "\n", 86 | "Use `em.criterion_plot` and `em.params_plot` to visualize the convergence of your optimization. Use the `selector` argument of `em.params_plot` to visualize only a subset of the parameters. Example: `selector=lambda params: params[\"b\"]`." 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "id": "b17eeed3", 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [] 96 | } 97 | ], 98 | "metadata": { 99 | "kernelspec": { 100 | "display_name": "Python 3 (ipykernel)", 101 | "language": "python", 102 | "name": "python3" 103 | }, 104 | "language_info": { 105 | "codemirror_mode": { 106 | "name": "ipython", 107 | "version": 3 108 | }, 109 | "file_extension": ".py", 110 | "mimetype": "text/x-python", 111 | "name": "python", 112 | "nbconvert_exporter": "python", 113 | "pygments_lexer": "ipython3", 114 | "version": "3.9.0" 115 | } 116 | }, 117 | "nbformat": 4, 118 | "nbformat_minor": 5 119 | } 120 | -------------------------------------------------------------------------------- /exercises/03_play_with_algorithm_and_algo_options.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "e8d9a9ff", 6 | "metadata": {}, 7 | "source": [ 8 | "# Picking Algorithms\n", 9 | "\n", 10 | "In this exercise you get code snippets for two optimizations that fail silently. You need to fix them by selecting a different optimization algorithm. To make it a bit easier, we provide the criterion value at the optimum.\n", 11 | "\n", 12 | "If you find one that works, continue to search for one that works better. \n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [List of algorithms](https://estimagic.readthedocs.io/en/stable/algorithms.html)\n", 17 | "- [Documentation of algo_options](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_algorithm_and_algo_options.html)" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "id": "4ab37132", 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np\n", 28 | "import estimagic as em" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "id": "bd6afe6b", 34 | "metadata": {}, 35 | "source": [ 36 | "## Problem 1\n", 37 | "\n", 38 | "This is a modified version of the `powell_singular` function from the More and Wild benchmark set. " 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 2, 44 | "id": "e1da93b6", 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "def powell_steps(x):\n", 49 | " x = x.round(4)\n", 50 | " fvec = np.zeros(4)\n", 51 | " fvec[0] = x[0] + 10 * x[1]\n", 52 | " fvec[1] = np.sqrt(5.0) * (x[2] - x[3])\n", 53 | " fvec[2] = (x[1] - 2 * x[2]) ** 2\n", 54 | " fvec[3] = np.sqrt(10.0) * (x[0] - x[3]) ** 2\n", 55 | " out = {\"root_contributions\": fvec, \"value\": fvec @ fvec}\n", 56 | " return out\n", 57 | "\n", 58 | "powell_start = np.array([3, -1, 0, 1])" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 3, 64 | "id": "618a29b6", 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "# optimal criterion value\n", 69 | "powell_criterion = 0" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "id": "f58deebd", 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": [ 81 | "Minimize with 4 free parameters terminated successfully after 1 criterion evaluations, 1 derivative evaluations and 0 iterations.\n", 82 | "\n", 83 | "The value of criterion improved from 215.00000000000003 to 215.00000000000003.\n", 84 | "\n", 85 | "The scipy_lbfgsb algorithm reported: CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL" 86 | ] 87 | }, 88 | "execution_count": 4, 89 | "metadata": {}, 90 | "output_type": "execute_result" 91 | } 92 | ], 93 | "source": [ 94 | "res_powell = em.minimize(\n", 95 | " criterion=powell_steps,\n", 96 | " params=powell_start,\n", 97 | " algorithm=\"scipy_lbfgsb\",\n", 98 | ")\n", 99 | "\n", 100 | "res_powell" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "id": "78accb9f", 106 | "metadata": {}, 107 | "source": [ 108 | "---\n", 109 | "The optimization thinks it terminated \"successfully\" but when you look at `res.params` you see that it actually got stuck at the start values." 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 5, 115 | "id": "b341cf09", 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "data": { 120 | "text/plain": [ 121 | "array([ 3., -1., 0., 1.])" 122 | ] 123 | }, 124 | "execution_count": 5, 125 | "metadata": {}, 126 | "output_type": "execute_result" 127 | } 128 | ], 129 | "source": [ 130 | "res_powell.params" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "id": "7dab7866", 136 | "metadata": {}, 137 | "source": [ 138 | "## Task 1:\n", 139 | "\n", 140 | "- Try to understand why the algorithm `scipy_lbfgsb` got stuck at the start parameters.\n", 141 | "- Find an algorithm that converges to the optimum." 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "id": "3548183c", 147 | "metadata": {}, 148 | "source": [ 149 | "## Problem 2\n", 150 | "\n", 151 | "This is the `bratu_2d` problem from the Cartis and Roberts benchmark set. " 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 6, 157 | "id": "740089a8", 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "def bratu(x):\n", 162 | " alpha = 4.\n", 163 | " x = x.reshape((int(np.sqrt(len(x))), int(np.sqrt(len(x)))))\n", 164 | " p = x.shape[0] + 2\n", 165 | " h = 1 / (p - 1)\n", 166 | " c = h**2 * alpha\n", 167 | " xvec = np.zeros((x.shape[0] + 2, x.shape[1] + 2))\n", 168 | " xvec[1 : x.shape[0] + 1, 1 : x.shape[1] + 1] = x\n", 169 | " fvec = np.zeros_like(x)\n", 170 | " for i in range(2, p):\n", 171 | " for j in range(2, p):\n", 172 | " fvec[i - 2, j - 2] = (\n", 173 | " 4 * xvec[i - 1, j - 1]\n", 174 | " - xvec[i, j - 1]\n", 175 | " - xvec[i - 2, j - 1]\n", 176 | " - xvec[i - 1, j]\n", 177 | " - xvec[i - 1, j - 2]\n", 178 | " - c * np.exp(xvec[i - 1, j - 1])\n", 179 | " )\n", 180 | " fvec = fvec.flatten()\n", 181 | " out = {\"root_contributions\": fvec, \"value\": fvec @ fvec}\n", 182 | " return out\n", 183 | "\n", 184 | "bratu_start = np.zeros(64)" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 7, 190 | "id": "aaeab5da", 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "# optimal criterion value \n", 195 | "bratu_criterion = 0" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 8, 201 | "id": "0feec888", 202 | "metadata": {}, 203 | "outputs": [ 204 | { 205 | "data": { 206 | "text/plain": [ 207 | "Minimize with 64 free parameters terminated unsuccessfully after 5000 criterion evaluations and 4697 iterations.\n", 208 | "\n", 209 | "The value of criterion improved from 0.1560737692424935 to 0.14373729795343512.\n", 210 | "\n", 211 | "The scipy_neldermead algorithm reported: Maximum number of function evaluations has been exceeded.\n", 212 | "\n", 213 | "Independent of the convergence criteria used by scipy_neldermead, the strength of convergence can be assessed by the following criteria:\n", 214 | "\n", 215 | " one_step five_steps \n", 216 | "relative_criterion_change 0.0001292 0.0007854 \n", 217 | "relative_params_change 0.02281 0.0461 \n", 218 | "absolute_criterion_change 1.857e-05 0.0001129 \n", 219 | "absolute_params_change 0.002281 0.00461 \n", 220 | "\n", 221 | "(***: change <= 1e-10, **: change <= 1e-8, *: change <= 1e-5. Change refers to a change between accepted steps. The first column only considers the last step. The second column considers the last five steps.)" 222 | ] 223 | }, 224 | "execution_count": 8, 225 | "metadata": {}, 226 | "output_type": "execute_result" 227 | } 228 | ], 229 | "source": [ 230 | "res_bratu = em.minimize(\n", 231 | " criterion=bratu,\n", 232 | " params=bratu_start,\n", 233 | " algorithm=\"scipy_neldermead\",\n", 234 | " algo_options={\"stopping.max_criterion_evaluations\": 5000},\n", 235 | ")\n", 236 | "\n", 237 | "res_bratu" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "id": "8d1152d0", 243 | "metadata": {}, 244 | "source": [ 245 | "---\n", 246 | "The optimization will terminate because it reaches the maximum number of criterion evaluations. Of course, you could increase that number even further, but you can do better! Also: One million evaluations would not even be enough!\n", 247 | "\n", 248 | "## Task 2:\n", 249 | "\n", 250 | "- Try to understand why `scipy_neldermead` does not converge on this problem.\n", 251 | "- Find an algorithm that converges to the optimum." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "id": "aa785805", 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [] 261 | } 262 | ], 263 | "metadata": { 264 | "kernelspec": { 265 | "display_name": "Python 3 (ipykernel)", 266 | "language": "python", 267 | "name": "python3" 268 | }, 269 | "language_info": { 270 | "codemirror_mode": { 271 | "name": "ipython", 272 | "version": 3 273 | }, 274 | "file_extension": ".py", 275 | "mimetype": "text/x-python", 276 | "name": "python", 277 | "nbconvert_exporter": "python", 278 | "pygments_lexer": "ipython3", 279 | "version": "3.9.0" 280 | } 281 | }, 282 | "nbformat": 4, 283 | "nbformat_minor": 5 284 | } 285 | -------------------------------------------------------------------------------- /exercises/04_benchmarking.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "2415cd79", 6 | "metadata": {}, 7 | "source": [ 8 | "# Benchmarking\n", 9 | "\n", 10 | "In this exercise you will compare a few optimizers on estimagic's built-in benchmark suites. \n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- [Benchmarking tutorial](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_benchmark_optimization_algorithms.html)\n", 15 | "- [API reference](https://estimagic.readthedocs.io/en/stable/reference_guides/index.html#benchmarks)" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "id": "36c0ae43", 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import estimagic as em" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "id": "8352040f", 32 | "metadata": {}, 33 | "outputs": [], 34 | "source": [ 35 | "problems = em.get_benchmark_problems(name=\"example\")" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 3, 41 | "id": "833d1ab8", 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "optimizers = [\n", 46 | " \"scipy_neldermead\",\n", 47 | " \"nlopt_lbfgsb\",\n", 48 | " \"fides\",\n", 49 | " \"nag_dfols\",\n", 50 | "]" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "id": "fad97ed1", 56 | "metadata": {}, 57 | "source": [ 58 | "## Task 1:\n", 59 | "\n", 60 | "- Use `em.run_benchmark` to run the benchmark.\n", 61 | "- Use `em.profile_plot` to visualize the results." 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "id": "d343639f", 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "id": "3c628e12", 75 | "metadata": {}, 76 | "source": [ 77 | "## Task 2: \n", 78 | "\n", 79 | "- Switch to the `estimagic` problem set instead of the `example` problem set. Choose a suitable value for `n_cores` to parallelize on your laptop. Most laptops have at least 2 cores. You won't get any benefit from choosing more cores than you have. \n", 80 | "- Compare `scipy_neldermead` against `nlopt_neldermead`. Spoiler: They are very different!\n", 81 | "- Use `em.convergence_plot` to look at individual problems." 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "id": "313d82a4", 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [] 91 | } 92 | ], 93 | "metadata": { 94 | "kernelspec": { 95 | "display_name": "Python 3 (ipykernel)", 96 | "language": "python", 97 | "name": "python3" 98 | }, 99 | "language_info": { 100 | "codemirror_mode": { 101 | "name": "ipython", 102 | "version": 3 103 | }, 104 | "file_extension": ".py", 105 | "mimetype": "text/x-python", 106 | "name": "python", 107 | "nbconvert_exporter": "python", 108 | "pygments_lexer": "ipython3", 109 | "version": "3.9.0" 110 | } 111 | }, 112 | "nbformat": 4, 113 | "nbformat_minor": 5 114 | } 115 | -------------------------------------------------------------------------------- /exercises/05_bounds_and_constraints.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "634e78d2", 6 | "metadata": {}, 7 | "source": [ 8 | "## Bounds and constraints\n", 9 | "\n", 10 | "In this exercise you learn how to use bounds and simple constraints. \n", 11 | "\n", 12 | "Note that we will just scratch the surface of the topic. Look at the resources for more information. \n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [How to specify bounds](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_bounds.html)\n", 17 | "- [How to use constraints](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_constraints.html)\n", 18 | "- [Background: How constraints are implemented](https://estimagic.readthedocs.io/en/stable/explanations/optimization/implementation_of_constraints.html)\n", 19 | "\n", 20 | "## Example\n", 21 | "\n", 22 | "We reproduce the example from previous exercises for convenience" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "id": "d312ee51", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import estimagic as em \n", 33 | "import numpy as np\n", 34 | "\n", 35 | "def criterion(x):\n", 36 | " first = (x[\"a\"] - np.pi) ** 2\n", 37 | " second = np.sum((x[\"b\"] - np.arange(3)) ** 2)\n", 38 | " third = np.sum((x[\"c\"] - np.eye(2)) ** 2)\n", 39 | " return first + second + third\n", 40 | " \n", 41 | " \n", 42 | "start_params = {\n", 43 | " \"a\": 1,\n", 44 | " \"b\": np.ones(3),\n", 45 | " \"c\": np.ones((2, 2))\n", 46 | "}" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 2, 52 | "id": "c3462ee1", 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "{'a': 3.141592653589793,\n", 59 | " 'b': array([1.55630437e-16, 1.00000000e+00, 2.00000000e+00]),\n", 60 | " 'c': array([[ 1.00000000e+00, -4.50891722e-18],\n", 61 | " [-4.51479888e-18, 1.00000000e+00]])}" 62 | ] 63 | }, 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "output_type": "execute_result" 67 | } 68 | ], 69 | "source": [ 70 | "res = em.minimize(\n", 71 | " criterion=criterion,\n", 72 | " params=start_params,\n", 73 | " algorithm=\"nlopt_bobyqa\",\n", 74 | ")\n", 75 | "\n", 76 | "res.params" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "id": "eb87df22", 82 | "metadata": {}, 83 | "source": [ 84 | "## Task 1: Bounds\n", 85 | "\n", 86 | "Repeat the optimization with an upper bounds of 2.0 on `a` and a lower bound of 0.5 for all entries in `b`." 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "id": "d7d30d6d", 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "id": "f9c2124d", 100 | "metadata": {}, 101 | "source": [ 102 | "## Task 2: Fixing parameters\n", 103 | "\n", 104 | "Remove the bounds but now fix the parameter `a` as well as the top right entry in `c` (i.e. `x[\"c\"][0, 1]`) at their start value." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "id": "f59b5c6a", 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "id": "80ac3084", 118 | "metadata": {}, 119 | "source": [ 120 | "## Optional: Play around with more constraints\n", 121 | "\n", 122 | "Look at the [documentation](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_constraints.html) and impose the constraint that the parameters in `\"c\"` sum to 1. " 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "id": "efcc13ab", 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [] 132 | } 133 | ], 134 | "metadata": { 135 | "kernelspec": { 136 | "display_name": "Python 3 (ipykernel)", 137 | "language": "python", 138 | "name": "python3" 139 | }, 140 | "language_info": { 141 | "codemirror_mode": { 142 | "name": "ipython", 143 | "version": 3 144 | }, 145 | "file_extension": ".py", 146 | "mimetype": "text/x-python", 147 | "name": "python", 148 | "nbconvert_exporter": "python", 149 | "pygments_lexer": "ipython3", 150 | "version": "3.9.0" 151 | } 152 | }, 153 | "nbformat": 4, 154 | "nbformat_minor": 5 155 | } 156 | -------------------------------------------------------------------------------- /exercises/06_scaling.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "ccde9e5c", 6 | "metadata": {}, 7 | "source": [ 8 | "# Scaling\n", 9 | "\n", 10 | "In this exercise you will use estimagic's scaling capabilities to solve a badly scaled optimization problem. \n", 11 | "\n", 12 | "## Resources:\n", 13 | "\n", 14 | "- [`slice_plot` example](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_visualize_an_optimization_problem.html)\n", 15 | "- [how to use scaling](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_scale_optimization_problems.html)" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "id": "1baf0d2b", 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import numpy as np\n", 26 | "import estimagic as em" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "id": "6ef94deb", 32 | "metadata": {}, 33 | "source": [ 34 | "## Get a badly scaled problem from a benchmark set" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 2, 40 | "id": "e233409e", 41 | "metadata": {}, 42 | "outputs": [ 43 | { 44 | "data": { 45 | "text/plain": [ 46 | "0" 47 | ] 48 | }, 49 | "execution_count": 2, 50 | "metadata": {}, 51 | "output_type": "execute_result" 52 | } 53 | ], 54 | "source": [ 55 | "problem = em.get_benchmark_problems(\n", 56 | " \"more_wild\", \n", 57 | " scaling=True,\n", 58 | " scaling_options={\"min_scale\": 1, \"max_scale\": 100},\n", 59 | ")[\"chebyquad_6\"]\n", 60 | "\n", 61 | "criterion = problem[\"inputs\"][\"criterion\"]\n", 62 | "start_params = problem[\"inputs\"][\"params\"]\n", 63 | "solution = problem[\"solution\"][\"value\"]\n", 64 | "solution" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "id": "2f104a0b", 70 | "metadata": {}, 71 | "source": [ 72 | "## Task 1: Plotting \n", 73 | "\n", 74 | "- Use `em.slice_plot` to to visualize `criterion`. Set the lower bounds as `start_params - 1` and the `upper_bounds` as `start_params + 1` and the center at `start_params`. \n", 75 | "- Repeat the plot, but this time use `lower_bounds = start_params - np.abs(0.1 * start_params)` and upper bounds analogously. \n", 76 | "- After looking at the plots, do you think that rescaling the problem with the `start_params` method will help?" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "id": "35839334", 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "id": "f6667e1b", 90 | "metadata": {}, 91 | "source": [ 92 | "## Task 2: Optimization\n", 93 | "\n", 94 | "- Extend the code below by doing an optimization with `scaling=True` (store your results in the dictionary).\n", 95 | "- Compare the scaled and unscaled optimization in the criterion_plot." 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 3, 101 | "id": "5a7e5846", 102 | "metadata": {}, 103 | "outputs": [ 104 | { 105 | "data": { 106 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAH0CAYAAADfWf7fAAAgAElEQVR4XuzdB3gVxcLG8TehhBIILfSidOkGhFAVsIANFKyf9aoXEQQVRcQGqKCAqCgIoiAXENELiCBVkF5DaAKhSicQWiiBAEm+Z9abmJB2dPckOef893v45CY7szO/nYQ3k9lZv4SEhARxIIAAAggggAACCCDgpQJ+BF4vvbN0CwEEEEAAAQQQQMASIPAyEBBAAAEEEEAAAQS8WoDA69W3l84hgAACCCCAAAIIEHgZAwgggAACCCCAAAJeLUDg9erbS+cQQAABBBBAAAEECLyMAQQQQAABBBBAAAGvFiDwevXtpXMIIIAAAggggAACBF7GAAIIIIAAAggggIBXCxB4vfr20jkEEEAAAQQQQAABAi9jAAEEEEAAAQQQQMCrBQi8Xn176RwCCCCAAAIIIIAAgZcxgAACCCCAAAIIIODVAgRer769dA4BBBBAAAEEEECAwMsYQAABBBBAAAEEEPBqAQKvV99eOocAAggggAACCCBA4GUMIIAAAggggAACCHi1AIHXq28vnUMAAQQQQAABBBAg8DIGEEAAAQQQQAABBLxagMDr1beXziGAAAIIIIAAAggQeBkDCCCAAAIIIIAAAl4tQOD16ttL5xBAAAEEEEAAAQQIvIwBBBBAAAEEEEAAAa8WIPB69e2lcwgggAACCCCAAAIEXsYAAggggAACCCCAgFcLEHi9+vbSOQQQQAABBBBAAAECL2MAAQQQQAABBBBAwKsFCLxefXvpHAIIIIAAAggggACBlzGAAAIIIIAAAggg4NUCBF6vvr10DgEEEEAAAQQQQIDAyxhAAAEEEEAAAQQQ8GoBAq9X3146hwACCCCAAAIIIEDgZQwggAACCCCAAAIIeLUAgderby+dQwABBBBAAAEEECDwMgYQQAABBBBAAAEEvFqAwOvVt5fOIYAAAggggAACCBB4GQMIIIAAAggggAACXi1A4PXq20vnEEAAAQQQQAABBAi8jAEEEEAAAQQQQAABrxYg8Hr17aVzCCCAAAIIIIAAAgRexgACCCCAAAIIIICAVwsQeL369tI5BBBAAAEEEEAAAQIvYwABBBBAAAEEEEDAqwUIvF59e+kcAggggAACCCCAAIGXMYAAAggggAACCCDg1QIEXq++vXQOAQQQQAABBBBAgMDLGEAAAQQQQAABBBDwagECr1ffXjqHAAIIIIAAAgggQOBlDCCAAAIIIIAAAgh4tQCB16tvL51DAAEEEEAAAQQQIPAyBhBAAAEEEEAAAQS8WoDA69W3l84hgAACCCCAAAIIEHgZAwgggAACCCCAAAJeLUDg9erbS+cQQAABBBBAAAEECLyMAQQQQAABBBBAAAGvFiDwevXtpXMIIIAAAggggAACBF7GAAIIIIAAAggggIBXCxB4vfr20jkEEEAAAQQQQAABAi9jAAEEEEAAAQQQQMCrBQi8Xn176RwCCCCAAAIIIIAAgZcxgAACCCCAAAIIIODVAgRer769dA4BBBBAAAEEEECAwMsYQAABBBBAAAEEEPBqAQKvV99eOocAAggggAACCCBA4GUMIIAAAggggAACCHi1AIHXq28vnUMAAQQQQAABBBAg8DIGEEAAAQQQQAABBLxagMDr1beXziGAAAIIIIAAAggQeBkDCCCAAAIIIIAAAl4tQOD16ttL5xBAAAEEEEAAAQQIvIwBBBBAAAEEEEAAAa8WIPB69e2lcwgggAACCCCAAAIEXsYAAggggAACCCCAgFcLEHi9+vbSOQQQQAABBBBAAAECL2MAAQQQQAABBBBAwKsFCLxefXvpHAIIIIAAAggggACBlzGAAAIIIIAAAggg4NUCBF6vvr10DgEEEEAAAQQQQIDAyxhAAAEEEEAAAQQQ8GoBAq9X3146hwACCCCAAAIIIEDgZQwggAACCCCAAAIIeLUAgderby+dQwABBBBAAAEEECDwMgYQQAABBBBAAAEEvFqAwOvVt5fOIYAAAggggAACCBB4GQMIIIAAAggggAACXi1A4PXq20vnEEAAAQQQQAABBAi8jAEEEEAAAQQQQAABrxYg8Hr17aVzCCCAAAIIIIAAAgRexgACCCCAAAIIIICAVwsQeL369tI5BBBAAAEEEEAAAQIvYwABBBBAAAEEEEDAqwUIvF59e+kcAggggAACCCCAAIGXMYAAAggggAACCCDg1QIEXq++vXQOAQQQQAABBBBAgMDLGEAAAQQQQAABBBDwagECr1ffXjqHAAIIIIAAAgggQOBlDCCAAAIIIIAAAgh4tQCB16tvL51DAAEEEEAAAQQQIPAyBhBAAAEEEEAAAQS8WoDA69W3l84hgAACCCCAAAIIEHhtjIFq1app165dNmqgKAIIIIAAAggggIC7BQi8NoQJvDbwKIoAAggggAACCGSRAIHXBjSB1wYeRRFAAAEEEEAAgSwSIPDagCbw2sCjKAIIIIAAAgggkEUCBF4b0AReG3gURQABBBBAAAEEskiAwGsDmsBrA4+iCCCAAAIIIIBAFgkQeG1AE3ht4FEUAQQQQAABBBDIIgECrw1oAq8NPIoigAACCCCAAAJZJEDgtQFN4LWBR1EEEEAAAQQQQCCLBAi8NqAJvDbwKIoAAggggAACCGSRAIHXBjSB1wYeRRFAAAEEEEAAgSwSIPDagCbw2sCjKAIIIIAAAgggkEUCBF4b0AReG3gURQABBBBAAAEEskiAwGsDmsBrA4+iCCCAAAIIIIBAFgkQeG1AE3ht4FEUAQQQQAABBBDIIgECrw1oE3jbjO1vo4aURWsEldIr9do6Vh8VIYAAAggggAACCEgEXhujwOnAWy2opF6td6uNFlEUAQQQQAABBBBA4FoBAq+NMeHUkoY9Z09o8Kb5qlK4hHrXv91GiyiKAAIIIIAAAgggQOB1cAw4FXj/OHdSH26cp+sLFVefBnc42EKqQgABBBBAAAEEEGCG18YYcCrw7j9/SgM3zFWlwGLqe2M7Gy2iKAIIIIAAAggggAAzvA6OAacC78ELp/V++BxVKFhUb4W0d7CFVIUAAggggAACCCDADK+NMeBU4D184YwGhM9WuQJBeqfhXTZaRFEEEEAAAQQQQAABj57hTUhI0NmzZxUUFJThnXTlvMuXLytXrlzWn+RHfHy8dY1ChQql+ty1F3Uq8B6NOat+62epTIHC6tfwbkYpAggggAACCCCAgIMCHjPDu2TJEvXs2dPqenBwsEaNGqUqVaqkonDlvAMHDqht27YaOHCgHnjgAauOU6dOqV+/flq6dKmKFSumbt26qVOnThlSOxV4j108p3fCZqpU/kIa0OgeB28vVSGAAAIIIIAAAgh4ROC9dOmSQkNDNWLECDVv3lyTJk3S1KlTNW3atBR30JXz4uLi9Nhjj+nYsWPq2rVrUuC9//771axZM7300kvKnTu3SyPDqcAbdem83lr3s4LzBer9m+516dqchAACCCCAAAIIIOCagEcE3hUrVlizrwsWLLB6ZYJt3bp1tXjxYpUrVy6pp66cN27cOO3cuVP+/v5q0KCBFXjXrl2rPn36aP78+S6HXXNRpwLvydgL6rt2hooHFNTAxh1cu3OchQACCCCAAAJ/S8D8hrdixYp/qwwne4eARwTeGTNmaM6cOdYyhsSjTZs2Gjp0qEJCQpI+ltl5u3fv1tNPP61Zs2bpww8/tMqawPvtt99q27ZtVoDdvn27tdzh1ltvVUBAQIZ32anAezo2Rn3W/qQiefProyb3ecfIohcIIIAAAgjkIIELFy5YE12rVq1SiRIlclDLaEpWCHhE4J04caLCw8M1bNiwJJMOHTqoV69eatWqVdLHMjqvadOm6ty5s7VkoXXr1nrjjTeSAu/gwYM1ZswYvfzyy6patarMLLCZQe7bt29S3Y888kiq+xEWFqZdu3bZvk/Rly+p95ppCsqbT4Ob3G+7PipAAAEEEEAAgdQCZjljqVKloPFBAY8IvDNnzrRmZUePHp1ihtcEYPPTWuKR0XlmucPGjRs1aNAg6/QBAwYkLWn4/PPPdf78eeshNnNEREToySef1MqVK5N2aoiKiko1PMyaXycC77krsXp19VQVyhOgoaEZPyjng2OULiOAAAIIIJAkYHZZMpNejz/+uL788kvlyZPHesjcPGxuDrNMcciQIda/+bVq1bKWLJpJL1OuXbt2mj59up544gm99957qlevnlVm69atev311/XTTz9p3759evPNN62JtkaNGlm54brrruMOeLiARwReM5Pau3dvLVq0yOKOiYlR/fr1U63hzeg8M7NrBv+1h/miqV27thVcEwOvWeNz7733Wl80efPmTfcWO7Wk4cLVy3pl1X9VIHdefdK0s4cPKZqPAAIIIOBtArMObMmWLt1dsW6q68bGxqpOnTrWQ+yvvvqqrl69ai1P/PXXX1WpUiWZ5Yt79+61JrXMw+3Lly+X+Q1wYrn169fr448/tpYtJv4m1/ym12xJ+v777+uhhx6yHmI3D7OPHTtW0dHRKX7DnC0QXNS2gEcEXjOYzcA2P2WZtbtmAJr1uuZPZGSkNfv77LPPWoM+vfOulUq+pCFxm7Kff/7Z2ups+PDhOnnyZNJscHrKTgXei3FX9NLKH5U/Vx592uzPbdI4EEAAAQQQyCkCXZZ9ly1NGd3y0XQD79dff62bb77Z+vwLL7xg/d2E1cTD/ObWhF8Ths3zOWaXJhOUTeA1k1ymjPlNrjlMwDW/7TX/rjdu3NgKz2bm2OQDM5O8ZcsW5cuXL1sMuKgzAh4ReE1Xzeyt+XWF+QnN7JM7cuRI1axZU/PmzVP37t21YcMGBQYGpnteWoG3YcOG1rpec5hfcfTv39/6u1kXbPb8TWuf3+T1OBV4Y+OvqseKH5TXP5c+b/7XF6szt5haEEAAAQQQsCeQE2d4kz98ZmZqTSYwSxXMQ+7mYfQjR45YM75r1qxJFXhNXjAB2cz0mhdQmfBr6jO/STZblprneZIfX331lSpUqGAPkdLZKuAxgTdRyfxqIfmb1sxb1U6fPm2F4OTHtee5omzesnbx4kUVLFjQldMd25bsakK8ui3/Xrn9/DWixcMuXZuTEEAAAQQQ8EWB5EsTChcubBEkBt777rtPLVu2lAmoZqbWzOyaJYrXzvCacmZG1+QHs/d+gQIFrIfaTeg1oZkZXe8bWR4XeHPSLXBqhjcuIUEvLJ8sf/npy5apd4PISX2mLQgggAACCGSnQEaB1+zCZJY+mlldswTBPJszZcqUNAPvnj179Oijfy6ZMC+0MrO6586dswJzly5drODr5+dnPcRmHn7j8GwBAq+N++dU4DVNSFwfldZ6JRtNpCgCCCCAAAJeJZBR4DUh1SxPNMsUzWHerGoeWDPLIpOv4U2cGTYPppmPm2eCEo/El1EdPHjQ+pB50+uECRO8ytAXO0PgtXHXnQy8zy/7TgmSRrV8VH422kRRBBBAAAEEfF3ALE80a3Mz2mkpMyOza4Opw9VljpnVx+ezV4DAa8PfycBrljSYpQ0jWzysXH7+NlpFUQQQQAABBBBAAIHkAgReG+PBycDbfcUUXYmP0xfNH1Ie/1w2WkVRBBBAAAEEEEAAAQKvQ2PAycBrtiUz25MNb/agAnLldqiFVIMAAggggAACCCDADK+NMeBk4DUvnjAvoDAvnjAvoOBAAAEEEEAAAQQQcEaAwGvD0cnAa14tbF4xPKxpZxXMnf7rjG00l6IIIIAAAggggIBPChB4bdx2JwPvq6un6tyVWA0NvV+F8vD6Qhu3haIIIIAAAggggEAKAQKvjQHhZODtvWa6oi9f1OAm9ykob34braIoAggggAACCCCAQHIBAq+N8eBk4O2z9iedjo3RoMYdVSyggI1WURQBBBBAAAEEEECAwOvQGHAy8PZdO0MnYy9oYOMOKh5Q0KEWUg0CCCCAAAIIIIAAM7w2xoCTgfetdT8r6tJ5vX/TvQrOF2ijVRRFAAEEEEAAAQQQYIbXoTHgZOB9J2ymjl08p/6N7lbp/IUdaiHVIIAAAggggAACCDDDa2MMOBl4+63/RUdjotWv4V0qUyDIRqsoigACCCCAAAIIIMAMr0NjwMnAOyB8tg5fOKO3Q+5U+YJFHGoh1SCAAAIIIIAAAggww2tjDDgZeD/YMEcHzp/Wmze2V8XAojZaRVEEEEAAAQQQQAABZngdGgNOBt6BG+Zq//lT6ntjO1UKLOZQC6kGAQQQQAABBBBAgBleG2PAycD74cZ5+uPcSb3e4HZVLlTCRqsoigACCCCAAAIIIMAMr0NjwMnAO2TTAu0+G6XX6t+mqoWDHWoh1SCAAAIIIIAAAggww2tjDDgZeIdu/lW7oo+rV71bVT2opI1WURQBBBBAAAEEEECAGV6HxoCTgfeTLQsVceaYXq7bVjWLlHKohVSDAAIIIIAAAgggwAyvjTHgZOD9bMsibTsTqZ5126hWkdI2WkVRBBBAAAEEEEAAAWZ4HRoDTgbe4b//pq2nj+rFOreoTtGyDrWQahBAAAEEEEAAAQSY4bUxBpwMvCO2LtHmU4fVrfbNqlesnI1WURQBBBBAAAEEEECAGV6HxoCTgffLbUu18eQhda3VSg2Kl3eohVSDAAIIIIAAAgggwAyvjTHgZOAdvX2Zwk8cVJcbWiqkRAUbraIoAggggAACCCCAADO8Do0BJwPvmIjlCos6oOdqtlCj4IoOtZBqEEAAAQQQQAABBJjhtTEGnAy830Ss0Nqo/XqmRjM1LnmdjVZRFAEEEEAAAQQQQIAZXofGgJOBd9yOVVp9/A89XaOpQkte71ALqQYBBBBAAAEEEECAGV4bY8DJwDt+52qtPLZXT1YPVbNSlW20iqIIIIAAAggggAACzPA6NAacDLwTdq3R8sg9erxaE7UoXcWhFlINAggggAACCCCAADO8NsaAk4F30u61Wnp0tx6tepNuLlPNRqsoigACCCCAAAIIIMAMr0NjwMnAO3lPmBYf2alHqjTSLWWrO9RCqkEAAQQQQAABBBBghtfGGHAy8E7Zs16LjuzQQ1Uaqk3ZGjZaRVEEEEAAAQQQQAABZngdGgNOBt4f94br18MR6lw5RLeVq+lQC6kGAQQQQAABBBBAgBleG2PAycA79Y8Nmn9ouzpdf6NuL3+DjVZRFAEEEEAAAQQQQIAZXofGgJOBd/q+TZp7cKs6Xldf7SvUdqiFVIMAAggggAACCCDADK+NMeBk4P15/2b9cuB33Vupnu6qWMdGqyiKAAIIIIAAAgggwAyvQ2PAycA768AWzdy/RfdUqqu7K9Z1qIVUgwACCCCAAAIIIMAMr40x4GTgNbO7Zpb3zop11KFSPRutoigCCCCAAAIIIIAAM7wOjQEnA69Zv2vW8barUFv3XVffoRZSDQIIIIAAAggggAAzvDbGgJOB1+zQYHZqMDs0mJ0aOBBAAAEEEEAAAQScEfCowJuQkKCzZ88qKCgow967ct7ly5eVK1cu68+1R2xsrPLmzSs/P78Mr+Nk4DV78Jq9eG8tV1MPVA5x5u5SCwIIIIAAAggggIA8JvAuWbJEPXv2tG5ZcHCwRo0apSpVqqS6ha6cd+DAAbVt21YDBw7UAw88kKKOOXPmqEePHpo3b54qV66cZYHXvGXNvG3NvGXNvG2NAwEEEEAAAQQQQMAZAY8IvJcuXVJoaKhGjBih5s2ba9KkSZo6daqmTZuWQsGV8+Li4vTYY4/p2LFj6tq1a4rAe/z4cT388MM6deqUVXdWBt7FR3Zq8p4w3VKmuh6p2siZu0stCCCAAAIIIIAAAp4xw7tixQr169dPCxYssG6ZCbZ169bV4sWLVa5cuaTb6Mp548aN086dO+Xv768GDRqkCLxdunRR+/bt9fHHH2v8+PFZGniXHt2tSbvXqlWZqvq/qo0ZmggggAACCCCAAAIOCXjEDO+MGTNklhqYZQyJR5s2bTR06FCFhPy13jWz83bv3q2nn35as2bN0ocffmiVTVzSYGaM586dqzFjxqhJkyaaPHlylgbe5ZF7NGHXGrUoXUWPV2vi0O2lGgQQQAABBBBAAAGPCLwTJ05UeHi4hg0blnTHOnTooF69eqlVq1ZJH8vovKZNm6pz58566aWX1Lp1a73xxhtJgffw4cO6//77ZQJz6dKl0wy87777bqrR8t1332nXrl2OjKKVx/Zq/M7Valaqsp6sHupInVSCAAIIIIAAAgggIM9Y0jBz5kxrVnb06NEpZnhNADbLEhKPjM4zyx02btyoQYMGWacPGDDAKtupUyd1797dWhtsArE57rrrLo0cOVK1atVS/vz5rY+Z8tceTz31lGOBd/XxPzRuxyqFlrxeT9doythEAAEEEEAAAQQQcEjAI2Z4w8LC1Lt3by1atMjqdkxMjOrXr59qDW9G55mZXRN4rz3uvPNOzZ49O03ODz74QA8++GC61E5uS7b2+D59s2OlGgdX0jM1mzt0e6kGAQQQQAABBBBAwCMC79WrV60ZWDM7a9bujh071lp+YP5ERkZas7/PPvusMjrv2ludfEnDtZ/LjjW8YVEHNCZiuRoFV9RzNVswMhFAAAEEEEAAAQQcEvCIwGv6amZvu3XrJvNSiGLFillLDmrWrGntl2uWJGzYsEGBgYHpnpdW4G3YsGHSMobknzeBd8qUKbruuusyZHZyhjf8xEGN3r5MN5aooOdvaOnQ7aUaBBBAAAEEEEAAAY8JvIm3Kjo6OsWb1sxb1U6fPm2F4OTHtee541Y7GXg3njykL7ctVYPi5dW11l8P4rmj3dSJAAIIIIAAAgj4koDHBd6cdHOcDLybTx3WiK1LVK9YOXWrfXNO6iZtQQABBBBAAAEEPFqAwGvj9jkZeH8/fUSf/75YtYuWUY86rW20iqIIIIAAAggggAACyQUIvDbGg5OBd9uZSH22ZZFqFSmtnnXb2GgVRRFAAAEEEEAAAQQIvA6NAScDb8SZY/pky0LVCCqlV+q1daiFVIMAAggggAACCCDADK+NMeBk4N0VfVxDN/+qakEl9Wq9W220iqIIIIAAAggggAACzPA6NAacDLy7z0ZpyKYFqlo4WK/Vv82hFlINAggggAACCCCAADO8NsaAk4F377kT+mjjfF1fqLj6NLjDRqsoigACCCCAAAIIIMAMr0NjwMnAu//8KQ3cMFeVAoup743tHGoh1SCAAAIIIIAAAggww2tjDDgZeA+cP60PNsxRhYJF9VZIexutoigCCCCAAAIIIIAAM7wOjQEnA+/hC2c0IHy2yhUsondC7nSohVSDAAIIIIAAAgggwAyvjTHgZOA9GhOtfut/UZkChdWv4d02WkVRBBBAAAEEEEAAAWZ4HRoDTgbeYxfP6Z2wmSqVv5AGNLrHoRZSDQIIIIAAAggggAAzvDbGgJOBN+rSeb217mcF5wvU+zfda6NVFEUAAQQQQAABBBBghtehMeBk4D0Ze0F9185Q8YCCGti4g0MtpBoEEEAAAQQQQAABZnhtjAEnA+/p2Bj1WfuTigYU0IeNO9poFUURQAABBBBAAAEEmOF1aAw4GXijL19U7zXTFZQ3nwY3ud+hFlINAggggAACCCCAADO8NsaAk4H33JVYvbp6qgrlCdDQ0E42WkVRBBBAAAEEEEAAAWZ4HRoDTgbeC1cv65VV/1XB3Hk1rGlnh1pINQgggAACCCCAAALM8NoYA04G3otxV/TSyh+VP1cefdrsARutoigCCCCAAAIIIIAAM7wOjQEnA29s/FX1WPGDAvxza3jzBx1qIdUggAACCCCAAAIIMMNrYww4GXivxMep+4opyu3nrxEtHrbRKooigAACCCCAAAIIMMPr0BhwMvDGJSToheWTlcvPTyNbPOJQC6kGAQQQQAABBBBAgBleG2PAycBrmtFl2XfykzSq5aM2WkVRBBBAAAEEEEAAAWZ4HRoD7gi8pmmjCbwO3SGqQQABBBBAAAEEJGZ4bYwCpwOvWdJgljaYJQ1maQMHAggggAACCCCAgH0BlwPvyZMntX//fpUoUUIVK1a0f2UvqMHpwNtt+fe6mhCvL5o/pDz+ubxAiC4ggAACCCCAAALZL+BS4B04cKDGjRtntfbFF19Ujx49NG/ePA0dOlQLFizI/l5kUwucDrxmWzKzPZnZlsxsT8aBAAIIIIAAAgggYF8g08C7ceNGPfXUU3rzzTcVHh6uMmXKWIF33759uu2227R06VLrY754OB14zYsnzAsozIsnzAsoOBBAAAEEEEAAAQTsC2QaeF9//XUFBgbq7bff1meffSY/Pz8r8F66dEl169bVrFmzVKNGDfst8cAanA68L6/6r2KuXrZeLWxeMcyBAAIIIIAAAgggYF8g08BrljNERkZq+PDhKQKvWcrwwgsvKCwsTEFBQfZb4oE1OB14X109VeeuxGpoaCcVyhPggSI0GQEEEEAAAQQQyHkCmQbe9evX6+GHH9bTTz9tBd+8efOqevXqmjBhgjXDO3LkyJzXqyxqkdOBt/eaaYq+fEmDm9ynoLz5s6gXXAYBBBBAAAEEEPBugUwDr+n+3Llz9cEHH1iBN/Ho0KGDta63aNGi3i2UQe+cDrx91v6k07Ex+rBxRxUNKOCzrnQcAQQQQAABBBBwUsClwGsumJCQoKioKJ09e1Zly5ZVgQIEMqcDb9+1M3Qy9oIGNu6g4gEFnbzP1IUAAggggAACCPisgMuB12eFsnCG9611Pyvq0nm9f9O9Cs4XCDkCCCCAAAIIIICAAwKZBt6JEydqxowZ6V7K7M9rdnHwxcPpGd53wmbq2MVzGtDoHpXKX8gXSekzAggggAACCCDguECmgXfNmjXavHlz0oWvXLmiQ4cO6ccff9Rjjz2mvn37Kk8e39wz1unA22/9LB2NOat+De9SmQK+ufOF4yOcChFAAAEEEEDA5wUyDbzpCc2ZM0f9+vXTqlWr5O/v75OQTgfeAeGzdfjCGb0Tcr2qXlMAACAASURBVKfKFSzik6Z0GgEEEEAAAQQQcFrgHwfeixcvql69elq0aJEqVKjgdLs8oj6nA+/74XN08MJpvRXSXhUK+u7uFx5x82kkAggggAACCHiMwD8OvFu3blXHjh1lXj1csKBv7ijgdOAduGGu9p8/pb43tlOlwGIeM4hoKAIIIIAAAgggkJMFMg285oG1JUuWpOjDqVOntGLFCt122228eGLXLsfu74cb5+mPcyfVp8Edur5QccfqpSIEEEAAAQQQQMCXBTINvOalEytXrkxhlC9fPt1888266aabrDev+erh9Azv4E3ztefsCb1W/zZVLRzsq6z0GwEEEEAAAQQQcFQg08Dr6NW8rDKnA+/Qzb9qV/RxvVrvVlULKullWnQHAQQQQAABBBDIHoE0A++ePXsUHR3tUovMg2u5c+d26VxvO8npwDts80LtiD6ml+u2Vc0ipbyNi/4ggAACCCCAAALZIpBm4H3++ee1cOFClxoUFhamoKCs2TPWvN7YvNo4s+u5ct7ly5eVK1cu60/iERsbKz8/P5eXaTgdeD/bskjbzkSqZ902qlWktEv+nIQAAggggAACCCCQsUCagdeEwbi4OJfs8ufP79J5dk8yD8717NnTqiY4OFijRo1SlSpVUlXrynkHDhxQ27ZtNXDgQD3wwAOKiIjQ+++/r99//92q7/bbb1f//v2VWd+cDrzDf/9NW08fVY86rVW7aBm7ZJRHAAEEEEAAAQQQkOQRa3gvXbqk0NBQjRgxQs2bN9ekSZM0depUTZs2LcVNdOU8E+TNG+KOHTumrl27WoF306ZN1tvj7rzzTl24cEHPPPOMHnzwQXXq1CnDQeJ04P1i6xJtOXVY3WrfrHrFyjFAEUAAAQQQQAABBBwQyDTwmlcJz58/X+bNakeOHEl1yf/85z8KDAx0oCnpV2G2QDNvdVuwYIF1kgm2devW1eLFi1Wu3F/B0JXzxo0bp507d1pvh2vQoIEVeK89hg8frnPnzunNN9/M0sD75bal2njykLrWaqUGxcu71ZTKEUAAAQQQQAABXxHINPAuW7ZM//rXv6wZT7MUwKyfNWHRBGCzxMCs93X3Q2tmL2BzPbOMIfFo06aNhg4dqpCQkKSPZXbe7t279fTTT2vWrFn68MMPrbLXBl6z/te8UKN79+7WPsMZHU7P8I7avkwbThxUlxtaKqSEb769zle+8OgnAggggAACCGSdQKaB16xzNUffvn31+eefW7O5JjT+9ttv6tOnj7VHb/IHv9zR9IkTJyo8PFzDhg1Lqr5Dhw7q1auXWrVqlfSxjM5r2rSpOnfurJdeekmtW7fWG2+8kWbgHT16tEzINzPBefLkSarbLKO49jCzzrscfPHEmIjlCos6oOdqtlCj4IruoKROBBBAAAEEEEDA5wQyDbzvvPOOChUqpNdee02TJ0/W+vXrrZlV82Bb7dq19euvv6pSpUpuhZs5c6Y1K2vCaOJhZnhNADbLEhKPjM4zyx3Ma5AHDRpknT5gwICkJQ2mf+b45ZdfrJnfn376ScWLp3zTmQnA1x7mhwEnA+83ESu0Nmq/nqnZXI2D3Wvq1htG5QgggAACCCCAQA4SyDTwfv/99xo/fry1pMDMsprlDWPGjLHW0Zq/m48lBkZ39ctsfda7d28tWrTIukRMTIzq16+fag1vRueZmV0TeK89zEyxCfBmdwdzDbMmuUaNGi51xeklDWN3rNKa43/o6RpNFVryepfawEkIIIAAAggggAACGQtkGnjNCyhWr16tO+64Q2Z9q1nOYGZLzfHQQw9Z23m5+7h69aq1O4OZnTUzu2PHjpVZr2v+REZGWrO/zz77rDI679o2Jl/SsG7dOj333HP65ptvrFnrxMO8Qjmjw+nAO37naq08tldPVg9Vs1KV3c1K/QgggAACCCCAgE8IZBp4z58/n2IXBrOUwcyUlihRQpUrZ10oM7O33bp1k3k5RLFixTRy5EjVrFlT8+bNsx4w27Bhg9XO9M5LK/A2bNjQWtdrwq7Z8eHaY+vWrRm+hMLpwDth1xotj9yjx6s1UYvSqfcY9okRSScRQAABBBBAAAGHBTINvEOGDLF+3f/II4+offv2VtjMzsPMOCd/05qZdT59+nSqdl17njva7HTgnbR7rZYe3a3/q9pYrcpUdUeTqRMBBBBAAAEEEPA5gUwD7549ezRlyhT98MMP1ksZTOg1s6LNmjVz+3ZkOf1uOB14J+8O0+KjO/VI1Ua6pUz1nN592ocAAggggAACCHiEQKaBN7EX5gUUZgsys27W7IZgZnoffvhha5lB3rx5PaKzTjfS6cA7Zc96LTqyQw9Vaag2ZV17cM7pPlEfAggggAACCCDgbQIuB97kHTfrZM1DX/v27bPWzCZfYuBtQBn1x+nA++PecP16OEIPVA7RreVq+hIlfUUAAQQQQAABBNwm4HLgNa8VNvvUmqUNJuiaXRPM0gazxMHdL55wW+9tVux04J36xwbNP7Rdna6/UbeXv8Fm6yiOAAIIIIAAAgggYAQyDbxr1qzRF198YW1NVrp0aevhNbN3bbly5Xxe0OnAO33fJs09uFX3XVdf7Sr8tT2az0MDgAACCCCAAAII2BDINPCa1wkfOnRInTp1UqNGjeTv72/jct5V1OnAO2P/Zs0+8Ls6VKqnOyvW8S4seoMAAggggAACCGSTQKaBN5va5RGXdTrwzty/RbMObNE9lerq7op1PcKARiKAAAIIIIAAAjldgMBr4w45HXh/OfC7ft6/WXdVrKN7K9Wz0TKKIoAAAggggAACCCQKEHhtjAWnA++cg1v1075N1vpds46XAwEEEEAAAQQQQMC+AIHXhqHTgdfs0GB2ajA7NJidGjgQQAABBBBAAAEE7Au4HHgjIiK0e/fuVFe8/fbbefGE/ftg1bDgcIT+uzdct5Wrqc6VQxyqlWoQQAABBBBAAAHfFsg08B47dszaiuzgwYMqWLCgAgICUogtWLBAhQsX9klFp2d4Fx7eoR/2rrfesmbetsaBAAIIIIAAAgggYF8g08A7efJkTZo0SaNGjVL58uXtX9GLanA68C4+slOT94TplrLV9UiVRl4kRVcQQAABBBBAAIHsE8g08L766quqVKmSXnzxxexrZQ69stOBd8nRXfpu9zq1KlNV/1e1cQ7tNc1CAAEEEEAAAQQ8SyDTwPv111/LLFuYMmWKZ/UsC1rrdOBdHrlHE3atUYvSVfR4tSZZ0AMugQACCCCAAAIIeL9ApoH3+PHjuueee9SxY0eFhoYqKCgohUq9evWUO3du75dKo4dOB96Vx/Zq/M7Valaqsp6sHuqTpnQaAQQQQAABBBBwWiDTwDtkyBB99dVX6V43LCwsVQh2upE5tT6nA++qY3/o252rFFryej1do2lO7TbtQgABBBBAAAEEPEog08B75coVXb16Nd1O5c+f36M67GRjnQ68a4/v0zc7Vqpxyev0TI1mTjaVuhBAAAEEEEAAAZ8VyDTwJpeJi4uTCcD58uXzWbDkHXc68IZF7deYiBVqFFxRz9VsgTECCCCAAAIIIICAAwIuBd5NmzbJLG1Ys2aNdclixYpZe/N26dJFzPDucuA2/FlF+ImDGr19mUJKVFCXG1o6Vi8VIYAAAggggAACviyQaeDdu3ev7rjjDjVv3lw333yzSpcurXXr1umXX35RmzZtNGjQIJ/1c3qGd+PJQ/py21I1KF5eXWu18llXOo4AAggggAACCDgpkGng/eSTTzR79mzNmzdP/v7+SdeeP3++unXrpg0bNigwMNDJNnlMXU4H3s2nDmvE1iWqV6ycutW+2WMcaCgCCCCAAAIIIJCTBTINvK+88orKli0r8wKK5EdMTIzq16+vuXPnqkqVKjm5j25rm9OB9/fTR/T574tVp2hZvVjnFre1m4oRQAABBBBAAAFfEsg08H7++ef64Ycf9Ntvv6XYb/enn35Sv379ZLYlYx9eZ4bMttNH9dnvv6lWkdLqWbeNM5VSCwIIIIAAAggg4OMCmQbeyMhItWvXTsHBwWrdurVKlSqltWvXatGiRTKzv127dvVZQqdneCPOHNMnWxaqZpFSerluW591peMIIIAAAggggICTApkGXnOxffv26YsvvlB4eLguXLigBg0a6M4771SHDh2cbIvH1eV04N0ZfVwfb/5V1YJK6tV6t3qcBw1GAAEEEEAAAQRyooBLgTcnNjwntMnpwLv7bJSGbFqgqoWD9Vr923JCF2kDAggggAACCCDg8QJpBt7Y2FitX79ejRo1UlRUlI4dO5ZuR+vVq8caXoeGwd5zJ/TRxvmqXKiEXm9wu0O1Ug0CCCCAAAIIIODbAmkG3tWrV+vxxx/Xjz/+qAULFuirr75KV8k8tBYUFOSTik7P8O47d1KDNs5TpcBi6ntjO580pdMIIIAAAggggIDTAukuaThz5oyKFClivUr46tWr6V6XN60596a1A+dP64MNc1QxsKjevLG90/ea+hBAAAEEEEAAAZ8UyHQN75QpU1SyZElrhwaOlAJOz/AeunBG74XPVrmCRfROyJ1wI4AAAggggAACCDggkGng7du3rwICAvTuu+86cDnvqsLpwHs0Jlr91v+iMgWC1K/hXd6FRW8QQAABBBBAAIFsEsg08JoXTvz73//W8uXLrT14Of4ScDrwRl48q3fDZql0/sLq3+huqBFAAAEEEEAAAQQcEMg08O7atUsDBgyQeZCtffv2ql27dorLPvXUU9YMsC8eTgfe45fO6e11MxWcL1Dv33SvL5LSZwQQQAABBBBAwHGBTAPvxIkTNXPmzHQv/M033ygwMNDxhnlChU4H3pOxF9R37QwVDyiogY19+6UennD/aSMCCCCAAAIIeIZApoHXM7qRPa10OvCeio3RG2t/UtGAAvqwccfs6RRXRQABBBBAAAEEvEzApcBrtiZbvHix9u7dK/OiiaZNm+r06dOKjo7Wdddd52UkrnfH6cAbffmieq+ZrqC8+TW4yX2uN4QzEUAAAQQQQAABBNIVyDTwmj14O3XqpG3btlmVvPjii+rRo4fMw2wvv/yy1q1bpzx58vgksdOB99yVS3p19TQVypNPQ0Pv90lTOo0AAggggAACCDgtkGngnTNnjgYOHKjx48dba3n9/PyswGteN9yiRQstXLhQFStWdLpdHlGf04H3/NVY9Vo1VQVz59Wwpp09woBGIoAAAggggAACOV0g08D70ksvqUKFCurVq5c+++yzpMBr3sR20003afbs2TLBzxcPpwPvxbgremnlj8qfK48+bfaAL5LSZwQQQAABBBBAwHGBTAPvp59+qhUrVujHH39MEXgnTZqkfv36acuWLcqXL5/jDfOECp0OvLFxV9Vj5Q8K8M+t4c0f9AQC2ogAAggggAACCOR4gUwD76FDh3T33XerfPnyyps3r7Xnbv78+bVs2TK98MIL1jpeXz2cDrxX4uPUfcUU5fHPpS+aP+SrrPQbAQQQQAABBBBwVCDTwGuutn//fo0aNUphYWGKioqyljA8+OCDuv/++5UrVy5HG+RJlTkdeOMS4vXC8u+Vy89fI1s87EkUtBUBBBBAAAEEEMixAi4F3vRaf+HCBRUsWDBHdM7sJnHp0qVMX4KRkJCgs2fPKigoKN12x8bGWrPZ5gG9jA6nA2+CEvT8sskyVx3V8tEc4UojEEAAAQQQQAABTxfINPAOGTJEwcHBMq8QvvZo0KCBZs2aZS13yM7jiy++sNYXm/BtHqT75JNP0gy+S5YsUc+ePa2mmj6ZWesqVaqkaLrZlcLsQjFv3jxVrlw5SwOvuViXZd9Z1xxN4M3OIcW1EUAAAQQQQMCLBP5x4D148KDatGmjpUuXqkyZMtlGsnnzZj333HOaMWOGFWLNbhKJu0okb5SZ/Q0NDdWIESPUvHlzmYfupk6dqmnTpiWddvz4cT388MM6deqU9fHsCLxdl01WvBI0ssUjypXJDHO2oXNhBBBAAAEEEEDAgwTSDbzh4eF65513ZB5aMw+qmTCZeMTFxWn37t0KCQnRlClTsrW7ZmbXvPHNtNUc69evl9lKzTxUl/wwO02YXSUWLFhgfdgE4Lp161pvkCtXrpz1sS5duqh9+/b6+OOPrX2HsyPwdlv+va4mxGtEi4eV288/W225OAIIIIAAAggg4A0C6Qbe+Ph4bd26VR999JG1PKBZs2ZJ/TVrW8361YYNG2b7W9b69u2rmjVr6oknnrDal/hCjIiIiBQP1JkZYLNcwSxjSDzMDPXQoUOt4G5me+fOnasxY8aoSZMmmjx5crYE3hdXTNHl+Dh93vwh5fX33QcCveGLiz4ggAACCCCAQM4QyHRJw86dO3X+/HkrFObEo3v37mrVqpW1a4Q5zANpJohv2LAhxTreiRMnysxaDxs2LKkbHTp0sJZAmHW8ZscJE4pLly6dZuA19V17mGvu2rXLUZaeK3/Qpbir1osnzAsoOBBAAAEEEEAAAQTsCaQZeM0uBWZpQKNGjaxtyMysaXpHvXr1lDt3bnutsFH6rbfesgLr008/bdUSGRmpli1baseOHfL3/2tJgHktsnnAbvTo0UlXS5zhNQ+5mXW9nTv/+Trfu+66SyNHjlStWrWsPYfN0bt371StnD59uuOB9+VV/1XM1cv6pGlnFcid14YMRRFAAAEEEEAAAQSMQJqBd/Xq1Xr88cett6uZNa9fffVVulpmb96MtvhyN/OXX36po0ePasCAAdalTNtfe+21VGt4TTtNaF20aJF1XkxMjOrXr69ffvnFCrhpHR988EHSzHFan3d6WzJzjV6rp+r8lVgNDe2kQnkC3M1H/QgggAACCCCAgNcLpLuk4cyZMypSpIiuXLkis8dtekfiDGh2SZmH58zMrJm9NcsRzJZi119/vRV6zWyv+fizzz5r9cHM4g4aNMjaXWLs2LHWEgbz59ojO9fw9l4zTdGXL2lwk/sVlNc3X9mcXWOJ6yKAAAIIIICAdwpkuobX7MJQsmRJtW7dOscK/Oc//0lam2vWGpslCmbW2eyla9b4Jq7nNbO83bp1k1myUaxYMWvZgnngLa3Aa/p93XXXZdhnd8zwvr5mus5cvqgPG3dU0YACOdachiGAAAIIIIAAAp4ikGngNbsgmG3J3n333Rzdp7TetGbeqnb69Gkr3CY/zDZmTizDcEfg7bt2hk7GXtDAxh1UPCBnvMUuR994GocAAggggAACCGQikGng/e233/Tvf/9by5cvV6lSpQBNJuCOwPvWup8Vdem83r/pXgXnC8QbAQQQQAABBBBAwKZApoHXbLtlHggzD4OZlzLUrl07xSXNK4fNDLAvHu4IvG+HzdTxi+c0oNE9KpW/kC+y0mcEEEAAAQQQQMBRgUwDr9m/1mzpld7xzTffpNjv1tHW5fDK3BF4+62fpaMxZ9Wv4d0qU6BwDhegeQgggAACCCCAQM4XyDTw5vwuZF8L3RF4B6z/RYdjovVOyJ0qV7BI9nWOKyOAAAIIIIAAAl4ikG7g/fTTT62dGcxetWkdhw4d0saNG3X33Xd7CcXf74Y7Au/74XN08MJpvRXSXhUKFv37jaIEAggggAACCCCAQAqBdAOv2Yu2f//+ateuXZpkmzdvVqdOnWRePezn5+eTrO4IvAM3zNX+86fU98Z2qhSYcncJn0Sm0wgggAACCCCAgE2BNAOveXNZq1atrLeVmZc5pHWcPXtWDRs2zPAcm23L8cXdEXg/3DhPf5w7qT4N7tD1hYrneAMaiAACCCCAAAII5HSBNAPvsWPH1KJFC61du1ZFi6b9a3Wzl22jRo2sB9rSenlDTu+4E+1zR+AdvGm+9pw9od71b1eVwiWcaCZ1IIAAAggggAACPi2Q7pIG8/pd83pesxVZWsfChQv1/PPPyyxtyO7XC2fXHXRH4B26+Vftij6uV+vdqmpBJbOra1wXAQQQQAABBBDwGoF0A+/gwYP13Xffafz48akeXNuxY4eeeOIJa0mDeT2vrx7uCLzDNi/UjuhjeqVeW9UI4kUfvjq26DcCCCCAAAIIOCeQbuA1r+p98803NW3aNIWGhuqGG26wrrpnzx4tXbpUFSpU0A8//KASJXz31+7uCLyfblmk7Wci9VLdNrqhSNrrp527/dSEAAIIIIAAAgh4v0CG+/DGx8dr3rx52rBhg8LCwnT58mVVrlxZZgeHBx54QHnz5vV+oQx66I7AO/z337T19FH1qNNatYuW8WlfOo8AAggggAACCDghwIsnbCi6I/B+sXWJtpw6rO61b1bdYuVstI6iCCCAAAIIIIAAAkaAwGtjHLgj8I7ctlSbTh5S11qt1KB4eRutoygCCCCAAAIIIIAAgdfmGHBH4B21fZk2nDio529oqRtLVLDZQoojgAACCCCAAAIIMMNrYwy4I/B+tX251p84oH/f0EINS1S00TqKIoAAAggggAACCDDDa3MMuCPwfh2xQuui9uuZms3VOLiSzRZSHAEEEEAAAQQQQIAZXhtjwB2Bd+yOVVpz/A/9q0ZTNSl5vY3WURQBBBBAAAEEEECAGV6bY8Adgffbnau16thePVk9VM1KVbbZQoojgAACCCCAAAIIMMNrYwy4I/BO2LVGyyP36PFqTdSidBUbraMoAggggAACCCCAADO8NseAOwLvxF1rtSxytx6r1lgtS1e12UKKI4AAAggggAACCDDDa2MMuCPwfrd7nZYc3aVHqjbSLWWq22gdRRFAAAEEEEAAAQSY4bU5BtwReKfsWa9FR3booSoN1aZsDZstpDgCCCCAAAIIIIAAM7w2xoA7Au8Pe8O18HCEHqgcolvL1bTROooigAACCCCAAAIIMMNrcwy4I/BO/WOD5h/ark7X36jby99gs4UURwABBBBAAAEEEGCG18YYcEfgnb5vo+Ye3Kb7rmugdhVq2WgdRRFAAAEEEEAAAQSY4bU5BtwReH/at0lzDm5Vh0r1dGfFOjZbSHEEEEAAAQQQQAABZnhtjAF3BN6Z+7do1oEtuqdSXd1dsa6N1lEUAQQQQAABBBBAgBlem2PAHYH3lwO/6+f9m3VXxTq6t1I9my2kOAIIIIAAAggggAAzvDbGgDsCr1nOYJY1tK9QWx2vq2+jdRRFAAEEEEAAAQQQYIbX5hhwR+Cdd2ibpv2x0dqhwezUwIEAAggggAACCCBgT4AZXht+7gi8Cw5H6L97w3VbuZrqXDnERusoigACCCCAAAIIIMAMr80x4I7Au/DwDv2wd73alquhBys3tNlCiiOAAAIIIIAAAggww2tjDLgj8P52ZKe+3xOmW8pW1yNVGtloHUURQAABBBBAAAEEmOG1OQbcEXiXHN2l73av081lqunRqjfZbCHFEUAAAQQQQAABBJjhtTEG3BF4l0Xu1sRda9WidBU9Xq2JjdZRFAEEEEAAAQQQQIAZXptjwB2Bd+WxvRq/c7WalaqsJ6uH2mwhxRFAAAEEEEAAAQSY4bUxBtwReFcd+0Pf7lylpqWu11PVm9poHUURQAABBBBAAAEEmOG1OQbcEXjXHN+nsTtWqnHJ6/RMjWY2W0hxBBBAAAEEEEAAAWZ4bYwBdwTesKj9GhOxQo2CK+m5ms1ttI6iCCCAAAIIIIAAAszw2hwD7gi8608c0FfblyukRAV1uaGlzRZSHAEEEEAAAQQQQIAZXhtjwB2Bd+PJQ/py21I1KF5eXWu1stE6iiKAAAIIIIAAAggww2tzDLgj8G4+dVgjti5RvWLl1K32zTZbSHEEEEAAAQQQQAABZnhtjAF3BN4tp47oi62LVadoWb1Y5xYbraMoAggggAACCCCAgMfN8CYkJOjs2bMKCgrK8O5ldN7Vq1d1+fJlFShQIFUd8fHx1ufy5cvn0uhwR+DddvqoPvv9N9UqWkY967R2qR2chAACCCCAAAIIIJC+gMfM8C5ZskQ9e/a0ehIcHKxRo0apSpUqqXqW3nn79u3Te++9p/Xr11tl6tSpo9dff11169a1/vdHH32kWbNmWYG3du3aev/991W2bNkMx447Am/EmUh9smWRahYppZfrtmXsIoAAAggggAACCNgU8IjAe+nSJYWGhmrEiBFq3ry5Jk2apKlTp2ratGkpup/ReVFRUVqzZo3atWsnPz8/jRw5Uhs3btQ333xj/bdLly5auHChNfNrwm7u3LnVt2/fLA+8O6OP6+PNv6p6UEn1qnerzdtLcQQQQAABBBBAAAGPCLwrVqxQv379tGDBAuuOmWBrZmYXL16scuXKJd1FV88zBaZPn26FXTOrm1hu3rx58vf3t4L10aNHreCb0eGOGd7dZ6M0ZNMCVS0crNfq38YIRQABBBBAAAEEELAp4BGBd8aMGZozZ461jCHxaNOmjYYOHaqQkJCkj2V2ngnKO3futGZ0v/32W/Xo0UMdO3ZUbGysnn32WZm1vw0bNtTMmTP19ddfq3Llykl1m7LXHiZ079q1y+YtSFl879kT+mjTfFUuVEKvN7jd0bqpDAEEEEAAAQQQ8EUBjwi8EydOVHh4uIYNG5Z0jzp06KBevXqpVau/9qrN7Ly9e/daIXfHjh1q0qSJ3nrrLdWsWdOq0yyR6NOnj/V3U/cHH3yggICApOvdc889qcZHRESE44F337mTGrRxnq4rVFxvNLjDF8ckfUYAAQQQQAABBBwV8IjAa2ZczdKD0aNHJ3XezPCaANygQYOkj7l6npnRnTx5sr788ktrOcPSpUut5Qvjxo2zdmh45513lD9/fn366acZYrtjScOB86f0wYa5qhhYVG/e2N7Rm01lCCCAAAIIIICALwp4ROANCwtT7969tWjRIusexcTEqH79+qnW8Lp6nqnj3Llz1nIIs1TCBN0yZcqoe/fuVv1mmcKdd96p7du3Ww+vpXe4I/AeunBG74XPVvmCRfR2yJ2+OCbpMwIIIIAAAggg4KiARwRes3eu2Z1hGAjH3AAAIABJREFU0KBBMjO7Y8eOlVmva/5ERkZas79mDW5G55m1u2b/3lKlSunixYuaMGGChgwZos2bN2v+/Pn68ccf9dVXX1nLGMaPH2/t6JB8RjktdXcE3iMx0eq//heVLRCkdxve5ejNpjIEEEAAAQQQQMAXBTwi8JobY2Zvu3XrZj1gVqxYMWtbMbP+1uysYGZmN2zYoMDAwHTPM8sd3n77bSvQmjrM2t/HHntMjRs3tnZ96N+/f9IMcr169fTyyy+rVq1aGY4JdwTeyItn9W7YLJXOX1j9G93ti2OSPiOAAAIIIIAAAo4KeEzgTex1dHR0ijetmZ0VTp8+bYXg5Me155nPxcXFWW9qK1SoUJpLFcyb1kz4TestbFk1w3v80jm9vW6mSuYrpPduSv2gnKN3n8oQQAABBBBAAAEfEPC4wJuT7ok7ZnhPXDqvN9f9rOIBBTWwcYec1F3aggACCCCAAAIIeKQAgdfGbXNH4D0VG6M31v6kYgEFNKhxRxutoygCCCCAAAIIIICAESDw2hgH7gi8Zy5f1OtrpqtI3vz6qMl9NlpHUQQQQAABBBBAAAECr80x4I7Ae/bKJb22epoK5cmnoaH322whxRFAAAEEEEAAAQSY4bUxBtwReM9fjVWvVVMVmDtAHzftZKN1FEUAAQQQQAABBBBghtfmGHBH4I25elkvr/qv8ufKo0+bPWCzhRRHAAEEEEAAAQQQYIbXxhhwR+CNjbuqHit/UECu3Bre7EEbraMoAggggAACCCCAADO8NseAOwLv5fg4vbhiivL459IXzR+y2UKKI4AAAggggAACCDDDa2MMuCPwxiXE64Xl3yuXn79GtnjYRusoigACCCCAAAIIIMAMr80x4I7Am6AEPb9ssvzkp1EtH7HZQoojgAACCCCAAAIIMMNrYwy4I/Ca5nRZ9p3VqtEtH7XROooigAACCCCAAAIIMMNrcwy4K/B2XTZZ8UrQly0fkb/8bLaS4ggggAACCCCAgG8LMMNr4/67K/B2W/69ribEa0SLh5Xbz99GCymKAAIIIIAAAgggQOC1MQbcFXjNLg1mt4bPmz+kvP65bLSQoggggAACCCCAAAIEXhtjwF2Bt+fKH3Qp7qo+a/ag8uXKbaOFFEUAAQQQQAABBBAg8NoYA+4KvOZNa+aNa5807awCufPaaCFFEUAAAQQQQAABBAi8NsaAuwJvr9VTdf5KrD4O7aTAPAE2WkhRBBBAAAEEEEAAAQKvjTHgrsD72uppOnvlkgY3uV9BefPZaCFFEUAAAQQQQAABBAi8NsaAuwLv62um68zli/qoyX0qkje/jRZSFAEEEEAAAQQQQIDAa2MMuCvwvrF2hk7FXtCgxh1ULKCgjRZSFAEEEEAAAQQQQIDAa2MMuCvwvrnuZ524dF7v33SvgvMF2mghRRFAAAEEEEAAAQQIvDbGgLsC79thM3X84jm91+gelcxfyEYLKYoAAggggAACCCBA4LUxBtwVeN8Nm6XIi2fVr+HdKlOgsI0WUhQBBBBAAAEEEECAwGtjDLgr8A5Y/4sOx0TrnYZ3qVyBIBstpCgCCCCAAAIIIIAAgdfGGHBX4H0vfI4OXTitt0Paq3zBojZaSFEEEEAAAQQQQAABAq+NMeCuwPvBhrk6cP6U+t7YTpUCi9loIUURQAABBBBAAAEECLw2xoC7Au+HG+fpj3Mn1afBHbq+UHEbLaQoAggggAACCCCAAIHXxhhwV+D9aNN87T17Qr3r364qhUvYaCFFEUAAAQQQQAABBAi8NsaAuwLv0M2/alf0cb1a71ZVCyppo4UURQABBBBAAAEEECDw2hgD7gq8wzYv1I7oY3qlXlvVCCplo4UURQABBBBAAAEEECDw2hgD7gq8n2xZpIgzkXqpbhvdUKS0jRZSFAEEEEAAAQQQQIDAa2MMuCvwDv/9N209fVQ96rRW7aJlbLSQoggggAACCCCAAAIEXhtjwF2B94uti7Xl1BF1r32z6hYrZ6OFFEUAAQQQQAABBBAg8NoYA+4KvCO3LdWmk4f0Qq1Wql+8vI0WUhQBBBBAAAEEEECAwGtjDLgr8I7atkwbTh7U87Va6sbiFWy0kKIIIIAAAggggAACBF4bY8Bdgfer7cu1/sQB/fuGFmpYoqKNFlIUAQQQQAABBBBAgMBrYwy4K/B+HbFC66L269mazXVTcCUbLaQoAggggAACCCCAAIHXxhhwV+Adu2Ol1hzfp3/VaKomJa+30UKKIoAAAggggAACCBB4bYwBdwXeb3eu1qpje/VU9VA1LVXZRgspigACCCCAAAIIIEDgtTEG3BV4/7NzjVYc26PHqzVRi9JVbLSQoggggAACCCCAAAIEXhtjwF2Bd+KutVoWuVuPVWuslqWr2mghRRFAAAEEEEAAAQQIvDbGgLsC73e712nJ0V16tOpNurlMNRstpCgCCCCAAAIIIIAAgdfGGHBX4P1+T5h+O7JTD1VpqDZla9hoIUURQAABBBBAAAEEPCrwJiQk6OzZswoKCsrwzmV03tWrV3X58mUVKFAgzTri4+OtaxQqVEi5cuXK8DruCrw/7A3XwsMRerByiNqWq8koRQABBBBAAAEEELAh4DGBd8mSJerZs6fV1eDgYI0aNUpVqqR+oCu98/bt26f33ntP69evt+qoU6eOXn/9ddWtW9f636dOnVK/fv20dOlSFStWTN26dVOnTp2yJfD+948NWnBouzpdf6NuL3+DjdtLUQQQQAABBBBAAAGPCLyXLl1SaGioRowYoebNm2vSpEmaOnWqpk2bluIOZnReVFSU1qxZo3bt2snPz08jR47Uxo0b9c0331h13H///WrWrJleeukl5c6d26WR4a4Z3un7Nmnuwa3WA2s3leTFEy7djL9xUtkCRVQoT8DfKMGpCCCAAAIIIODJAh4ReFesWGHNvi5YsMCyNsHWzMwuXrxY5cqVS/J39TxTYPr06VbYnTVrltauXas+ffpo/vz5LoddU4e7Au+M/Zs1+8DvnjyucnzbyxUIUo0ipVWjSCnlz50nqb35cuVRpcBiOb79NBABBBBAAAEEXBfwiMA7Y8YMzZkzx1rGkHi0adNGQ4cOVUhISNLHMjvPBOWdO3daM7vffvutevTooY4dO1p/37ZtmxVgt2/frrZt2+rWW29VQEDGs4DuCrwrj+2V+cPhvMDV+Dj9ce5khhWXzF9IzUtVtl76EZQ3v/ONoEYEEEAAAQQQyFIBjwi8EydOVHh4uIYNG5aE06FDB/Xq1UutWrVK+lhm5+3du9cKuTt27FCTJk301ltvqWbNmho8eLDGjBmjl19+WVWrVtW4ceOsGeS+ffsm1f2vf/0r1Y1ZtmyZdu3alaU3jIvZF7gSH6c9Z6MUceaY/jh3QnEJCUmVXrgSqyMx0Un/27z4o0jetB9wtN+S9GsoU6CwGgWznMWdxtSNAAIIIOA7Ah4ReGfOnGktPRg9enTSnTEzvCYAN2jQIOljrp4XGxuryZMn68svv5RZBmEC7/nz5zVw4ECrroiICD355JNauXJl0k4NBw4cSDUqzEwwgdf7vlh2n43SosM7tP5E6nuelb2tWaSUHqvWRMH5ArPyslwLAQQQQAABrxPwiMAbFham3r17a9GiRdYNiImJUf369VOt4XX1PFPHuXPnrOUQZqlE4kxtYuA14fbee++11vbmzZs33ZvuriUNXjfKPLRDJ2MvKOrieSWY/0sw/1+KT4j/33+tj8pMDid+Pl6yzou3zk35+aSPJZg6Ej+XWGfq/736+B86cem8JdehUj1VCQrOFkUe8MsWdi6KAAIIIOCwgEcEXrN3rtmdYdCgQTIzu2PHjpVZr2v+REZGWrO/zz77rDI6z6zdNfv3lipVShcvXtSECRM0ZMgQbd68WWYHBzNb+/PPP1tbnQ0fPlwnT560rpfRQeB1eDRSXQqBn/Zt0pyDW7NdJfEBv5ASFVQtqGS2t4cGIIAAAggg8HcFPCLwmk6Z2VuzN65ZjmD2yTXbipn1t/PmzVP37t21YcMGBQYGpnueWe7w9ttvWw+imTrM2t/HHntMjRs3tszMrg39+/e3/m4+Z/b8TWuf3+TABN6/O9w4/+8KHI05q5/2bdSFq5f/blHb57vygN/fuUge/1zK5ecnfz9/5bL++P3vv/7yN3/3T+fj5lz//52TSdncfv4pmlQ8X0GZP+48/GT6ZP7/n//98+/6879+/jItMlsh+pszkj6f/NzkZf7spylvvDgQQAABBJwR8JjAm9jd6OjoFG9aM79CPn36tBWCkx/Xnmc+FxcXl/QWtbT22jVvWTOzvwULuvYPJIHXmUFILTlX4HJ8nMya5h3mAb+zJ2Qe+ItLiLce9DP/NUs8Uvw9/s+Pmz+mLAcCCCCAgDMCo1s+6kxFPlqLxwXenHSfCLw56W7QlpwuEBt/VXHxfwbkP4PyX2HZCs3xfwXo1J+PV1x8WuWS1WfK/29ddXIL8zGzbjrF+mvzv/+3NjtxTXWK//7v89bHzLrr/9VhrdFOVi7p76nqMuu5/2pP8rrTLP+/dd1/tjNBVxPMinAOBBBA4C8BAq+90UDgteFH4LWBR1EEEEAAAQQQQCCLBAi8NqAJvDbwKIoAAggggAACCGSRAIHXBjSB1wYeRRFAAAEEEEAAgSwSIPDagCbw2sCjKAIIIIAAAgggkEUCBF4b0AReG3gURQABBBBAAAEEskiAwGsDmsBrA4+iCCCAAAIIIIBAFgkQeG1AE3ht4FEUAQQQQAABBBDIIgECrw1oAq8NPIoigAACCCCAAAJZJEDgtQFN4LWBR1EEEEAAAQQQQCCLBAi8NqAJvDbwKIoAAggggAACCGSRAIHXBjSB1wYeRRFAAAEEEEAAgSwSIPDagCbw2sCjKAIIIIAAAgggkEUCBF4b0AReG3gURQABBBBAAAEEskiAwGsDmsBrA4+iCCCAAAIIIIBAFgkQeG1AE3ht4FEUAQQQQAABBBDIIgECrw1oE3g5EEAAgb8jEBoaqtWrV/+dIpyLAAIIaNeuXSjYECDw2sCbMGGCEhIS9MQTT9ioxTeKYuX6fcYKK9cFXD+TcYWV6wKun8m4wsp1gew9k8Brw58vdNfxsMLKdQHXz2RcYeW6gOtnMq6wcl3A9TMZV65bueNMAq8NVQav63hYYeW6gOtnMq6wcl3A9TMZV1i5LuD6mYwr163ccSaB14Yqg9d1PKywcl3A9TMZV1i5LuD6mYwrrFwXcP1MxpXrVu44k8BrQ5XB6zoeVli5LuD6mYwrrFwXcP1MxhVWrgu4fibjynUrd5xJ4HWHKnUigAACCCCAAAII5BgBAm+OuRU0BAEEEEAAAQQQQMAdAgRed6hSJwIIIIAAAggggECOESDw5phbQUMQQAABBBBAAAEE3CFA4LWhGhMTo9y5cytv3rw2avHeoufPn1dgYGCqDpqXdZw9e1ZBQUHe2/l/0LOrV69aLzLJkydPUmms0oY0Y8scyccXVqmtLl68qPz58/M1+De/HjP63s73/ZSY5muxYMGC8vPzS6WMVeqBFx8fL/O9/trcgNXf/CL9B6cTeP8BmvlH5LXXXtPy5cut0o899ph69eqV5hf8P6jeo4uYL+RvvvlGX375pdWPSpUqqUePHmrbtq31v5csWaKePXtafw8ODtaoUaNUpUoVj+6zE42/fPmy7r//ftWsWVNDhw7FKh3UX375RR9//LGuXLlihd05c+ZglYbVxo0bNWjQIO3bt88KI48//riefvpprP5nZcbPJ598ojFjxmjHjh3y9/e3PpPR93Zf/b6fntWCBQs0ePBgRUVFqVChQnr00UfVtWvXTB2d+H6ZU+tIzyp5e/v162d931qzZo1PW2XHPSTw/gP1sWPH6rfffrOCnfkmeN9996l///5q2bLlP6jNu4qYn17HjRunu+++W6VKldLcuXP1wQcfWF4mDIeGhmrEiBFq3ry5Jk2apKlTp2ratGnehfAPevPpp5/q559/VkhIiBV4L126hNU1juaHpbfeesv6uqtevXrSZ7FKPeA6dOigBx54wPph/NChQ2rdurX1A7r5rYqvfw2eO3dOzz77rIoUKaJFixalCLwZfW/3xe/7GVnNnDnTmtCoV69e0hj79ddfrY9hlXJcJX6FrlixQm+//bYuXLiQFHh90eof/BPpSBEC7z9gfOihh6zZknbt2lmlzSzlwYMHrWDHkVLAhJG6detq9erVioiIkPnp1swMmCPxc4sXL1a5cuV8ls7Mxr3xxht66qmntG7dOivwmm+MWKUcEia8mRBnwlzyA6vUXzrme9QTTzyhu+66y/rHtUGDBjJfZ2bG19fHlfmhfN68eWrWrJkaNWqUIvBm9L3dF7/vZ2R17agzPs8884xuv/12YZVyXBmr6Ohoa3Ls3XffVe/evZMCry9aZdc/9gTefyBvZnJHjx6tWrVqWaXNryd+/PFH66dajpQCZmbX/NrLGM2YMcP6r/kBIfFo06aNFfDMzKYvHuY3BPfee6+GDBlihREzC2c8sEo9GqpVq6bhw4dr1apV1q9QO3XqpMqVK2OVxheO+bp7+eWX9dxzz2n+/PnWrK75oYpx9ReWCSDXBt6Mvrf78vf9tKySDzuzrMH8AGEmNooXL279ttNX/41Mz8qE3Bo1aui2226zfnBPXNLgy1ZZ/W8+gfcfiJt/eM2v6hPXnpp/XD7//HN+NX+N5fHjx61QYma+W7VqpYkTJyo8PFzDhg1LOtPM1pn1z+bzvngMHDjQekjNrAn/6aefkgIvVilHQ+I/IuYHo3//+9/auXOnNY5M+DVfi4yrlF4nTpywfmNgljOYGV6zfKhx48Z8DSZjSiuYZPS93Ze/72cUeM1SNbN210wAmR+yzIFVyhles9TD/AAwefJkHTlyJEXg9WWrrP43n8D7D8TNT2Tmoaw6depYpWfNmmWFXWZ4/8I0/8iaB2XuuOMOdenSxfqEWfNlrMwXfvIZXhNczK9cfe1Yu3atXnzxRf33v/+1nqSfPXu2zMcGDBhgLWnA6q8RcerUKTVp0sRaC16+fHnrE927d7d+UDJ2WKX86jEPQJp1uyaImCVE5sFRY2R+UMDqT6v0ZnjT+97uy9/3Mwq8ZonM4cOHrd/c5cqVy7LF6q/Aa753md9kGp+qVataP4SaH0ZNCDbryM3XKXkia/71J/D+A2cT5My6G/NgljnM7G5kZCRreP9nGRsba4XcihUrWg/zJW5XExYWZq1dMg+KmMNsw1K/fn1rbaEvruE1T4mPHDky1QgsVqyY9WAfVn/RmC3HzINqyQOv+RW9+ZhZI47VX1ZmBunmm2/W77//roCAAOsTJvDWrl1bDRs2xOp/VGmFuIy+t/vy9/30Aq952Hbp0qUaP368tcwo8cDqr8BrfMza5rQOs8TP/BtJnvgHQewfFCHw/gO077//XtOnT7e+yE1oM7+WN7Ny5ic1Xz/Mtixm1tLsMfjhhx8mbfdj9is2h9mdwWyXZH7iNTPiZk2h+cOhFEsazK8JsUo5Kvr27Wt9wPwDYWaUOnbsaO1sUbZsWaySUcXFxemWW25J+p5kZpjat29vrRM36ywZV39ipRXiMvre7svf99Oy+vrrrzVlyhRNmDDBmqk0h9nezXzvxyr1Q2uJX6IHDhxIsaTBl62y+t99Au8/EDd7pprtkczDIOZX92bHhj59+iSFu39QpdcU2bx5s7Vu99rDzDCZIGxmebt16yYzC2xmMs0Mp9l7liNl4DUeWKUcFSa4vfLKK9ZyjwoVKli7EJhfDWKV+qtn4cKF1m8JzA8GZh/ee+65J2l9JeMq/cCb0fd2X/6+n95652tHnnkI0KxTxcr1wOvLVln97z6B14a4ecLerFniTWt/H9F8A+VNa665YZXSKaM3O2GV0sq80bBw4cJpDjSs0v/6y+h7O9/3Xfu+Zc7CCivXBdx/JoHX/cZcAQEEEEAAAQQQQCAbBQi82YjPpRFAAAEEEEAAAQTcL0Dgdb8xV0AAAQQQQAABBBDIRgECbzbic2kEEEAAAQQQQAAB9wsQeN1vzBUQQAABBBBAAAEEslGAwJuN+FwaAQQQQAABBBBAwP0CBF73G3MFBBBAAAEEEEAAgWwUIPBmIz6XRgABBBBAAAEEEHC/AIHX/cZcAQEEEEAAAQQQQCAbBQi82YjPpRFAAAEEEEAAAQTcL0Dgdb8xV0AAAQQQQAABBBDIRgECbzbic2kEEEAAAQQQQAAB9wsQeN1vzBUQQAABBBBAAAEEslGAwJuN+FwaAQQQQAABBBBAwP0CBF73G3MFBBBAAAEEEEAAgWwUIPBmIz6XRgABBBBAAAEEEHC/AIHX/cZcAQEEEEAAAQQQQCAbBQi82YjPpRHIyQIHDx5UYGCgihYtmqKZly9fVmRkpCpWrJhlzT969KimTJmiX3/9VSEhIRowYICta0dERCguLk61a9e2Vc+1hd1Vr6ONTKeyr776SrGxsXrxxRfddrlly5apevXqKlWqlNuuQcUIIIBAWgIEXsYFAgikKVCtWjWVLl1ac+fOVcGCBZPO2bRpkzp37iwT7nLlypUleq+//rr27dun559/Xvny5VPTpk1tXdfUFx0drVGjRln1bNy40QrwxYoVc7ReW5VlceF33nlHMTExGjp0qCNXPnDggC5dumQF3MSjSZMm6tatm5544glHrkElCCCAgKsCBF5XpTgPAR8TMIHXHE899ZTefPPNbAu8Zia2YcOGGjx4sG6//XZH7oIJu6bexIDbpk0bK+iZ2WM7x7X12qkrq8s6HXj79eunMmXKqEuXLkldOXz4sEqUKKGAgICs7h7XQwABHxcg8Pr4AKD7CKQnYAJvjx49NHz4cGs5QWIYTGuGd8aMGRo7dqz279+vypUrq2vXrrrttttcwjVLJEaMGKHZs2crKirKus4bb7whc/3t27dr4MCBWr16tRVO/7+9cw3VKfsD8CIU5VLyBTVTplxyrSGNzKXxYcS45k6T6wepMWKaptGkoSEZRCSExi2aoZDQjEsZjduUW7mMT1Lklks+ENOzar1tr3fOe87++3/Y5ll1Osd597rsZ231rN/+rXWaNWsWNm3aFFq3bv1K27SxYsWKGI0mEvzee++FNWvWhL///jv+bsCAAWHJkiXh0qVL4dy5c2HdunUx+sg4J0+eHM6ePVtqf9asWaF///5xLD/88EM4duxY/GzUqFFhypQpMap9+PDhGtudM2dOHN/t27fDwoULw/Hjx+O/P/3000B0uXnz5vHfjLlJkybh1q1bYf/+/TF6PWjQoPDFF1/Ee61U4E+bsHnnnXfCzJkzwyeffBJ27doVtm7dGrZv316KvL948SKMGTMmjBgxIrbLXB44cCAyItqKjPbt2zd2kxVeuBFN37lzZ2jRokX8/PHjx2Ho0KFh9erVke/NmzfDsmXLwokTJ8KjR48Ci4YZM2bE+Z83b17YvHlzfDPQqlWr0K9fv3jfkyZNCiNHjgyfffZZbJO5WLx4cTh//ny8btiwYWHq1KmhQYMG8fN0/e7du+Mz0LFjxzB48OAwevTo+Pnz58/jeJiP69evx4URYxw4cGCtnj0vkoAE/jsEFN7/zlx7pxKoEwGEE9FFopCsPXv2RCErF959+/ZF6UKsEDr+jXwhwEmmaur4+++/DwjztGnTokht2LAhnD59OgoO4oN8IZuILzm3XNOoUaNXmkxtjB8/Pnz44Yfh8uXLUYqQu++++y5GFPkMIfrggw9Kcoc4Il20/+OPP4ZOnTrFqCSyiTTxnXHxep7+EWDa3bt3b43tEi1GxhBnCvKIlK9duza0bNky7NixI9SrVy+OY9u2beHjjz+O7d67dy8sWLAgCu9XX331GjZymbk/xBCBJacZsYQ54oz4btmyJfTq1SvWhSPz8ttvv8WUDUS0ffv2Mc2AeUX0f//999eEF5mmfeaA8VKIXr///vsB+WQeWBAsXbo0zjGcWFB06dIl9nHt2rWYC9y7d+8o23zetm3bKMUsMvjdjRs34nipP27cuDgPLGay98715JIzdyygTp06FVauXFl6tn7++efYLyJPH3/99VccW7r/Oj3wXiwBCbzVBBTet3p6vTkJ5CeA8CJTfEc8kI7Zs2e/JrxE3Nq0aRNWrVpV6mzChAnxZ4SkppIkCvFLdYi8Ik4I3/Tp08PTp09D165dX4kyZ9tMbXz77bdh4sSJr3SHmNIOAso4U8lGMyu1z+YqoosIMxFLCtFuJO3XX3+NwlutXaKOyDIyj0hTELaxY8dG2ezZs2cUXq7jK0U1v/7663Dnzp0odeUFuUOWiRhzPRFcIuJpbvjOeNOmPuSTXGukurwQjSfymu4xy6Q2wlve3i+//BIXBORDU1hE8NxkUxqywktkFzlHytO98wwh0chv48aNY30km3QWysuXL6NEMze0S4ScNwMszP4tIp7/f4A1JSCBt4mAwvs2zab3IoE3SCAJL6++ie7yqp/ILbmvadMaUUoihllhZQhsBuMryc+/DYtX2UQrs1LIteQNk0bw008/VRXeixcvhiFDhsSxde7cuaLwnjlz5hUhqia8GzdujJFWIq+pEGkmAss9JeGtqV0i1cuXLw9ckzb3cQoCY5w/f34UQsZB2kPaPEdfpGIQdUXiyguRYqKuyHIqSDRRX6KcRF/JnT158mT8mEgnEW7mi8Lvjx49GqOmDx8+jOKMqLKgqKvwErE+ePBgjBKT3kD0mZSRq1ev1kp4EVa4ZBdKsCLSTXoHkXyElw1uPA+pILvdunULX375ZTwthAUE84Ls83N5ussb/C9hUxKQQIEJKLwFnjyHLoH/J4Gs8NIPuZVIDbLGK2kih5QOHTqUBC6NB2Ekv7Oa8F64cCHmXJJn265du9LtIHakLSBx1SK8CBfyeOjQofDuu+++JrxEHf/8889Xfl9NeJFOosJ81a9fv1SXSCRpCgifL93oAAAGNklEQVRvtXaT8PKanYUBhTQHIpbk+PIav9JGsZqEF9m7e/duzCXOFk7TQILJs+3Ro0eUZmSS65KUc+wYUVXEsHv37qV0jdoKL2Lep0+fUkoDY2GxQU4uAkoaAxHouggvOb4salJJixf4spDKRoQrCS+/YxHB80OqA7nJ69evjwsAiwQkIIEsAYXX50ECEqhIoFx4ieJ99NFHMZcSuUzHkpEjSkpD9jgrhBVB4vV/TSWlL5Afi0RnpTC9pq8mvET5yAMlBzdFMlOftRHTNIZs7isCRQ4qkW2EvrzUpl0km3vIpkUkoWPTHCzrKrykKsA0mwZQPjZSIkh1QHiJxqd5QR7JX2bBQiGKTJS1kvAir4h99v5T9JUoMtF3pBJZ/fzzz2N7ixYtipsBs8LLvLCRLZWswJKOQMoMEeu0IEiRdRYxTZs2rZXwprZZTJBy8+DBg0A7FglIQAIKr8+ABCRQlUC58FIBKeSVOSUJL8L0zTffBDaOkRPK62gElg1hw4cPj/moiAgb24gsVhI0oqDs4qdPZAURSqJYTXhpj2gjr+lJuyDCyaY1NqgdOXKkaiSW+mxQ4/U/90FBFElnYDxIYToxgggvElcb4eVMW/KG2SzGvT979ixGrNkAh1ATwa6r8MIcwSSqSoSYCClpIURD0zFyf/zxR9z4xWdIZdo4SD400VAEmBMNmB8i8JWEl7ETtaUfcpcR4BSxR3jpC16ccEFfyD1R6ydPnpSElwUI6RZEusnH5X6zwsscwZ28axYGLAaIQGdzdqtFeDkhghMvSH+4f/9+PN0D0WdzoEUCEpCAwuszIAEJVCWA1GSjnkkEyZMkAoewpNf9CAbRPXIpKYgnG7aQjyQ25MQiUOUFCZs7d27MB0WYiB4izJz4QEkR2OzRaOVtcKwXuaoILoVX/GzuIipZLfWA65PAMv60+e3KlSvx/OFsWkb6rDbCS7tsckOiU0oFm/HYgIWgUSoJL6kHnKpQKYeXOpzMwGY0ItsUxJY66WQCIp2kHlDS5jZ+5j5YlJBnS+FnNsaRZ8y4ysfCPZJbzPxRuA8kOeVbc2QZ8gwz5gw2LGxShBc5ZxFDf4graRZ8R7xTJJ4j3xBjhJqCzDOudGxb+fVck83hZTMekWWeGwrPDNLLcW0WCUhAAlkCpjT4PEhAAm+MAPLDua3ZvFcaR874c7Lp1XWlDhE1clDTua95BoUcI9BJmOrSBlFdooTIW3b8RJgRKtps2LBhXZosXUsbpBlk/2JdroYylTidgjYZVznvmtrmlT/pArX9K3nMKX1Uup7+GUf5n5/O9k99/kR1+VFy2Wtog2PV8vDluSGvmXnLU/9/nQfrS0ACxSCg8BZjnhylBCQgAQlIQAISkEBOAgpvTnBWk4AEJCABCUhAAhIoBgGFtxjz5CglIAEJSEACEpCABHISUHhzgrOaBCQgAQlIQAISkEAxCCi8xZgnRykBCUhAAhKQgAQkkJOAwpsTnNUkIAEJSEACEpCABIpBQOEtxjw5SglIQAISkIAEJCCBnAQU3pzgrCYBCUhAAhKQgAQkUAwCCm8x5slRSkACEpCABCQgAQnkJKDw5gRnNQlIQAISkIAEJCCBYhBQeIsxT45SAhKQgAQkIAEJSCAnAYU3JzirSUACEpCABCQgAQkUg4DCW4x5cpQSkIAEJCABCUhAAjkJKLw5wVlNAhKQgAQkIAEJSKAYBBTeYsyTo5SABCQgAQlIQAISyElA4c0JzmoSkIAEJCABCUhAAsUgoPAWY54cpQQkIAEJSEACEpBATgIKb05wVpOABCQgAQlIQAISKAYBhbcY8+QoJSABCUhAAhKQgARyElB4c4KzmgQkIAEJSEACEpBAMQgovMWYJ0cpAQlIQAISkIAEJJCTgMKbE5zVJCABCUhAAhKQgASKQUDhLcY8OUoJSEACEpCABCQggZwEFN6c4KwmAQlIQAISkIAEJFAMAgpvMebJUUpAAhKQgAQkIAEJ5CSg8OYEZzUJSEACEpCABCQggWIQUHiLMU+OUgISkIAEJCABCUggJwGFNyc4q0lAAhKQgAQkIAEJFIOAwluMeXKUEpCABCQgAQlIQAI5CSi8OcFZTQISkIAEJCABCUigGAT+ASmoRtc2eF6tAAAAAElFTkSuQmCC" 107 | }, 108 | "metadata": {}, 109 | "output_type": "display_data" 110 | } 111 | ], 112 | "source": [ 113 | "algorithm = \"nag_pybobyqa\"\n", 114 | "\n", 115 | "res_naive = em.minimize(\n", 116 | " criterion,\n", 117 | " start_params,\n", 118 | " algorithm=algorithm,\n", 119 | ")\n", 120 | "\n", 121 | "\n", 122 | "results = {\"naive\": res_naive}\n", 123 | "\n", 124 | "fig = em.criterion_plot(results, monotone=True)\n", 125 | "fig.show(renderer=\"png\")" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "id": "61e1f9da", 131 | "metadata": {}, 132 | "source": [ 133 | "## Optional tasks\n", 134 | "\n", 135 | "- Try to find an optimizer that works well for a badly scaled problem.\n", 136 | "- Run a benchmark on the the 'example' benchmark set, using the `scaling_options` as above and compare multiple optimizers." 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "id": "e2bd0912", 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [] 146 | } 147 | ], 148 | "metadata": { 149 | "kernelspec": { 150 | "display_name": "Python 3 (ipykernel)", 151 | "language": "python", 152 | "name": "python3" 153 | }, 154 | "language_info": { 155 | "codemirror_mode": { 156 | "name": "ipython", 157 | "version": 3 158 | }, 159 | "file_extension": ".py", 160 | "mimetype": "text/x-python", 161 | "name": "python", 162 | "nbconvert_exporter": "python", 163 | "pygments_lexer": "ipython3", 164 | "version": "3.9.0" 165 | } 166 | }, 167 | "nbformat": 4, 168 | "nbformat_minor": 5 169 | } 170 | -------------------------------------------------------------------------------- /exercises/07_automatic_differentiation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "483a1998", 6 | "metadata": {}, 7 | "source": [ 8 | "# Automatic differentiation\n", 9 | "\n", 10 | "In this exercise you will use automatic differentiation in JAX and estimagic to solve the previous problem.\n", 11 | "\n", 12 | "> Note. Because JAX cannot (yet) be installed on Windows there will be extra exercises for Windows users.\n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- https://jax.readthedocs.io/en/latest/jax.numpy.html\n", 17 | "- https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "id": "62dc1afa", 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import jax \n", 28 | "import jax.numpy as jnp\n", 29 | "import estimagic as em\n", 30 | "\n", 31 | "jax.config.update(\"jax_enable_x64\", True)" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "id": "3395cd5c", 37 | "metadata": {}, 38 | "source": [ 39 | "## Task 1: Switch to JAX\n", 40 | "\n", 41 | "- Use the code from exercise 2, task 2, and convert the criterion function and the start parameters to JAX. Look at the [`jax.numpy` documentation](https://jax.readthedocs.io/en/latest/jax.numpy.html) and slides if you have any questions.\n", 42 | "\n", 43 | "---\n", 44 | "\n", 45 | "## Task 1 (Windows): Copy functions\n", 46 | "\n", 47 | "- Copy the criterion function and start parameters from exericse 2, task 2, here." 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "id": "bde7053b", 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "id": "bdd454fa", 61 | "metadata": {}, 62 | "source": [ 63 | "## Task 2: Gradient\n", 64 | "\n", 65 | "- Compute the gradient of the criterion (the whole function). Look at the [`autodiff_cookbook` documentation](https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html) and slides if you have any questions.\n", 66 | "- Measure the runtime of a jitted and unjitted version of the gradient (using `%timeit`.)\n", 67 | "\n", 68 | "---\n", 69 | "\n", 70 | "## Task 2 (Windows): Gradient\n", 71 | "\n", 72 | "The analytical gradient of the function is given by:\n", 73 | "\n", 74 | "- $\\partial_a f(a, b, C) = 2 (a - \\pi)$\n", 75 | "- $\\partial_b f(a, b, C) = 2 (b - \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top)$\n", 76 | "- $\\partial_C f(a, b, C) = 2 (C - I_2)$\n", 77 | "\n", 78 | "---\n", 79 | "\n", 80 | "- Implement the analytical gradient\n", 81 | " - return the gradient in the form of `{\"a\": ..., \"b\": ..., \"C\": ...}`" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "id": "f29aa744", 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "id": "85959d9b", 95 | "metadata": {}, 96 | "source": [ 97 | "## Task 3 (all systems): Minimize\n", 98 | "\n", 99 | "- Use estimagic to minimize the criterion\n", 100 | " - pass the gradient function you computed above to the minimize call.\n", 101 | " - use the `\"scipy_lbfgsb\"` algorithm or other gradient based optimizers." 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "id": "e37e2dc2", 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [] 111 | } 112 | ], 113 | "metadata": { 114 | "kernelspec": { 115 | "display_name": "Python 3 (ipykernel)", 116 | "language": "python", 117 | "name": "python3" 118 | }, 119 | "language_info": { 120 | "codemirror_mode": { 121 | "name": "ipython", 122 | "version": 3 123 | }, 124 | "file_extension": ".py", 125 | "mimetype": "text/x-python", 126 | "name": "python", 127 | "nbconvert_exporter": "python", 128 | "pygments_lexer": "ipython3", 129 | "version": "3.9.0" 130 | } 131 | }, 132 | "nbformat": 4, 133 | "nbformat_minor": 5 134 | } 135 | -------------------------------------------------------------------------------- /exercises/08_jaxopt.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "0d6f8ff1", 6 | "metadata": {}, 7 | "source": [ 8 | "# Using JAXopt\n", 9 | "\n", 10 | "In this exercise you will use JAXopt to solve a *batched* version of the first and second exercise.\n", 11 | "\n", 12 | "> Note. You cannot tackle these exercises on Windows.\n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [JAX documentation](https://jax.readthedocs.io/en/latest/notebooks/quickstart.html)\n", 17 | "- [jax.numpy documentation](https://jax.readthedocs.io/en/latest/jax.numpy.html)\n", 18 | "- [JAXopt documentation](https://jaxopt.github.io/stable/unconstrained.html)\n", 19 | "\n", 20 | "---\n", 21 | "\n", 22 | "Lets start by defining the criterion function from the solutions of the second exercise, but this time we use jax.numpy instead of numpy, and we parametrize the function." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "id": "8cb07e7a", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import jax\n", 33 | "import jax.numpy as jnp\n", 34 | "\n", 35 | "jax.config.update(\"jax_enable_x64\", True)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 2, 41 | "id": "72460b9f", 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "x0 = {\"a\": jnp.array(0.0), \"b\": jnp.zeros(3), \"c\": jnp.zeros((2, 2))}\n", 46 | "b0 = jnp.arange(3, dtype=\"float64\")\n", 47 | "\n", 48 | "\n", 49 | "def f(x, b0):\n", 50 | " value = (\n", 51 | " (x[\"a\"] - jnp.pi) ** 2\n", 52 | " + jnp.sum((x[\"b\"] - b0) ** 2)\n", 53 | " + jnp.sum((x[\"c\"] - jnp.eye(2)) ** 2)\n", 54 | " )\n", 55 | " return value" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "id": "dcca3dea", 61 | "metadata": {}, 62 | "source": [ 63 | "## Task 1: Optimize using `JAXopt`\n", 64 | "\n", 65 | "- Create a solver instance of the class `LBFGS` for your problem. You need to make sure to\n", 66 | " - pass the function `f`,\n", 67 | " - set `tol=1e-6` for increased accuracy.\n", 68 | "- Run the optimization using `solver.run`. You need to make sure to\n", 69 | " - pass the initial parameters,\n", 70 | " - pass additional arguments of `f`.\n", 71 | "- Look at the output of the results. How do you access the parameters?" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 3, 77 | "id": "4fabb6b9", 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "from jaxopt import LBFGS\n", 82 | "\n", 83 | "solver = ..." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "id": "964ef052", 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "id": "74e17e85", 97 | "metadata": {}, 98 | "source": [ 99 | "## Task 2: Batched optimization\n", 100 | "\n", 101 | "Now you will optimize `f` not only for a single value of `b`, but for many.\n", 102 | "\n", 103 | "- Write a wrapper for `solver.run` that takes starting values for `x` and a single vector-valued parameter `b0`.\n", 104 | "- Use `vmap` and `jit` to create a vectorized and jitted version of this wrapper that allows for array-valued `b0`.\n", 105 | "- Execute your vectorized function on `b_arr` (this should perform 500 optimizations.)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 4, 111 | "id": "76906f44", 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "from jax import jit\n", 116 | "from jax import vmap\n", 117 | "\n", 118 | "\n", 119 | "b_arr = jnp.arange(1500, dtype=\"float64\").reshape(500, 3)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "id": "95009bda", 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "id": "42a7f3ed", 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "id": "591b5d9c", 141 | "metadata": {}, 142 | "source": [ 143 | "## Optional: Speed comparison\n", 144 | "\n", 145 | "Lets compare the speed of `jaxopt`'s batch optimization to a loop with scipy's `minimize`.\n", 146 | "\n", 147 | "- Finish the loop in `batch_solve_scipy` using `method=\"L-BFGS-B\"` in scipy's `minimize`.\n", 148 | "- Time the functions (Use `%timeit func()` in a notebook cell.)" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 5, 154 | "id": "9035fa70", 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "import numpy as np\n", 159 | "from scipy.optimize import minimize\n", 160 | "\n", 161 | "\n", 162 | "x0_numpy = np.zeros(8)\n", 163 | "b_arr_numpy = np.array(b_arr)\n", 164 | "\n", 165 | "\n", 166 | "def f_numpy(x, b0):\n", 167 | " a = x[0]\n", 168 | " b = x[1:4]\n", 169 | " c = x[4:].reshape(2, 2)\n", 170 | "\n", 171 | " value = (a - np.pi) ** 2 + np.sum((b - b0) ** 2) + np.sum((c - np.eye(2)) ** 2)\n", 172 | " return value" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 6, 178 | "id": "f2f61cb7", 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "def batch_solve_scipy(x0, b_arr):\n", 183 | "\n", 184 | " results = []\n", 185 | " for b in b_arr:\n", 186 | "\n", 187 | " res = ...\n", 188 | "\n", 189 | " b = res.x[1:4]\n", 190 | " results.append(b)\n", 191 | "\n", 192 | " return np.stack(results)" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "id": "46c77b7d", 198 | "metadata": {}, 199 | "source": [ 200 | "### Timing" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": 7, 206 | "id": "a1b6391c", 207 | "metadata": {}, 208 | "outputs": [ 209 | { 210 | "name": "stdout", 211 | "output_type": "stream", 212 | "text": [ 213 | "10.3 ns ± 0.0976 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)\n" 214 | ] 215 | } 216 | ], 217 | "source": [ 218 | "%timeit ..." 219 | ] 220 | } 221 | ], 222 | "metadata": { 223 | "kernelspec": { 224 | "display_name": "Python 3 (ipykernel)", 225 | "language": "python", 226 | "name": "python3" 227 | }, 228 | "language_info": { 229 | "codemirror_mode": { 230 | "name": "ipython", 231 | "version": 3 232 | }, 233 | "file_extension": ".py", 234 | "mimetype": "text/x-python", 235 | "name": "python", 236 | "nbconvert_exporter": "python", 237 | "pygments_lexer": "ipython3", 238 | "version": "3.9.0" 239 | } 240 | }, 241 | "nbformat": 4, 242 | "nbformat_minor": 5 243 | } 244 | -------------------------------------------------------------------------------- /exercises/solutions/01_first_optimization_with_scipy_optimize.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "bc1add0f", 6 | "metadata": {}, 7 | "source": [ 8 | "# First Optimization\n", 9 | "\n", 10 | "In this exercise you will code a criterion function for a numerical optimization from scratch and optimize it with `scipy.optimize`.\n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- [scipy Documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html)\n", 15 | "\n", 16 | "\n", 17 | "## The criterion function\n", 18 | "\n", 19 | "**Notation**:\n", 20 | "\n", 21 | "- $a$ is a scalar floating point number.\n", 22 | "- $b$ is a vector of length 3.\n", 23 | "- $C$ is 2 by 2 matrix.\n", 24 | "- $I_n$ is a n-dimesional identity matrix. (Use `np.eye`)\n", 25 | "- $|| . ||^2$ is the squared vector or matrix norm. (Use `np.sum((x - y)**2)`, see below for more information)\n", 26 | "\n", 27 | "**Criterion Function**:\n", 28 | "\n", 29 | "$f(a, b, C) = (a - \\pi)^2 + ||b - \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top||^2 + ||C - I_2||^2$\n", 30 | "\n", 31 | "**Optimization problem**:\n", 32 | "\n", 33 | "$\\min_{a, b, C} f(a, b, C)$\n", 34 | "\n", 35 | "**Solution**:\n", 36 | "\n", 37 | "$\n", 38 | "a = \\pi \\\\\n", 39 | "b = \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top \\\\\n", 40 | "C = I_2\n", 41 | "$\n", 42 | "\n", 43 | "\n", 44 | "*Information on norm*:\n", 45 | "The vector norm is the euclidean norm, the matrix norm is the Frobenius norm." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 1, 51 | "id": "386890ff", 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "import numpy as np\n", 56 | "from scipy.optimize import minimize" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "id": "68043abd", 62 | "metadata": {}, 63 | "source": [ 64 | "## Task 1: Define the criterion function\n", 65 | "\n", 66 | "Note that for scipy.optimize, x has to be a 1-dimensional numpy array. Thus the first step in your criterion function is to parse the vector `x` into `a`, `b` and `C`.\n", 67 | "\n", 68 | "## Solution 1:" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 2, 74 | "id": "e31fbcbb", 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "def f(x):\n", 79 | " a = x[0]\n", 80 | " b = x[1:4]\n", 81 | " c = x[4:].reshape(2, 2)\n", 82 | "\n", 83 | " value = (\n", 84 | " (a - np.pi) ** 2\n", 85 | " + np.sum((b - np.arange(3)) ** 2)\n", 86 | " + np.sum((c - np.eye(2)) ** 2)\n", 87 | " )\n", 88 | " return value" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "id": "b7a849ae", 94 | "metadata": {}, 95 | "source": [ 96 | "## Task 2: Set up start parameters\n", 97 | "\n", 98 | "For simplicity, set all start parameters to 0.\n", 99 | "\n", 100 | "## Solution 2:" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 3, 106 | "id": "79413d4b", 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "x0 = np.zeros(8)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "id": "5e4569a9", 116 | "metadata": {}, 117 | "source": [ 118 | "## Task 3: Minimize the criterion function with scipy\n", 119 | "\n", 120 | "## Solution 3:" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": 4, 126 | "id": "678cede7", 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "res = minimize(fun=f, x0=x0)" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": 5, 136 | "id": "4371783f", 137 | "metadata": {}, 138 | "outputs": [ 139 | { 140 | "data": { 141 | "text/plain": [ 142 | "array([ 3.14159243e+00, 4.94960928e-07, 1.00000038e+00, 2.00000027e+00,\n", 143 | " 9.99999867e-01, -1.94503768e-08, -1.94503768e-08, 9.99999867e-01])" 144 | ] 145 | }, 146 | "execution_count": 5, 147 | "metadata": {}, 148 | "output_type": "execute_result" 149 | } 150 | ], 151 | "source": [ 152 | "res.x" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 6, 158 | "id": "358feaa3", 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "a = res.x[0]\n", 163 | "b = res.x[1:4]\n", 164 | "c = res.x[4:].reshape(2, 2)" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": 7, 170 | "id": "3105c23a", 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "from numpy.testing import assert_array_almost_equal\n", 175 | "\n", 176 | "assert_array_almost_equal(a, np.pi)\n", 177 | "assert_array_almost_equal(b, np.arange(3))\n", 178 | "assert_array_almost_equal(c, np.eye(2))" 179 | ] 180 | } 181 | ], 182 | "metadata": { 183 | "kernelspec": { 184 | "display_name": "Python 3 (ipykernel)", 185 | "language": "python", 186 | "name": "python3" 187 | }, 188 | "language_info": { 189 | "codemirror_mode": { 190 | "name": "ipython", 191 | "version": 3 192 | }, 193 | "file_extension": ".py", 194 | "mimetype": "text/x-python", 195 | "name": "python", 196 | "nbconvert_exporter": "python", 197 | "pygments_lexer": "ipython3", 198 | "version": "3.9.0" 199 | } 200 | }, 201 | "nbformat": 4, 202 | "nbformat_minor": 5 203 | } 204 | -------------------------------------------------------------------------------- /exercises/solutions/02_convert_previous_example_to_estimagic.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "f33371be", 6 | "metadata": {}, 7 | "source": [ 8 | "# Switch to estimagic\n", 9 | "\n", 10 | "In this exercise you will use estimagic to solve the previous exercise.\n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- [Optimization tutorial](https://estimagic.readthedocs.io/en/stable/getting_started/first_optimization_with_estimagic.html)" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "id": "8ba99ec7", 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "import estimagic as em\n", 25 | "import numpy as np" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "id": "e06cb065", 31 | "metadata": {}, 32 | "source": [ 33 | "## Task 1: Simple switch\n", 34 | "\n", 35 | "- Copy your criterion function and start parameters from the previous example without any changes. \n", 36 | "- Use `em.minimize` to solve the optimization problem. For reasons that will be mentioned later, estimagic has no default algorithm. You can set `algorithm=\"scipy_neldermead\"` for now and will learn how to pick algorithms later. \n", 37 | "- Look at `res.params` and `res.criterion` to make sure the optimization is succeeded.\n", 38 | "\n", 39 | "## Solution 1:" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 2, 45 | "id": "b7a1e266", 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "def f(x):\n", 50 | " a = x[0]\n", 51 | " b = x[1:4]\n", 52 | " c = x[4:].reshape(2, 2)\n", 53 | "\n", 54 | " value = (\n", 55 | " (a - np.pi) ** 2\n", 56 | " + np.sum((b - np.arange(3)) ** 2)\n", 57 | " + np.sum((c - np.eye(2)) ** 2)\n", 58 | " )\n", 59 | " return value\n", 60 | "\n", 61 | "x0 = np.zeros(8)" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 3, 67 | "id": "b13a7827", 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "res = em.minimize(\n", 72 | " criterion=f,\n", 73 | " params=x0,\n", 74 | " algorithm=\"scipy_neldermead\",\n", 75 | ")" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 4, 81 | "id": "e75f860e", 82 | "metadata": {}, 83 | "outputs": [ 84 | { 85 | "data": { 86 | "text/plain": [ 87 | "array([ 3.14159266e+00, -1.68302929e-09, 1.00000000e+00, 2.00000000e+00,\n", 88 | " 1.00000000e+00, 7.12514518e-09, 7.33457472e-11, 1.00000000e+00])" 89 | ] 90 | }, 91 | "execution_count": 4, 92 | "metadata": {}, 93 | "output_type": "execute_result" 94 | } 95 | ], 96 | "source": [ 97 | "res.params" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 5, 103 | "id": "f0321ce2", 104 | "metadata": {}, 105 | "outputs": [ 106 | { 107 | "data": { 108 | "text/plain": [ 109 | "1.1388734391083942e-16" 110 | ] 111 | }, 112 | "execution_count": 5, 113 | "metadata": {}, 114 | "output_type": "execute_result" 115 | } 116 | ], 117 | "source": [ 118 | "res.criterion" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "id": "31d2b24c", 124 | "metadata": {}, 125 | "source": [ 126 | "## Task 2: More flexible `params`\n", 127 | "\n", 128 | "In estimagic, parameters can be arbitrary [pytrees](https://jax.readthedocs.io/en/latest/pytrees.html). In the current example, a dictionary with the keys `\"a\"`, `\"b\"` and `\"c\"` would be a good choice. \n", 129 | "\n", 130 | "- Adjust your function such that it takes a dictionary of parameters instead of a flat numpy array.\n", 131 | "- Re-run the optimization with `em.minimize` with the start parameters given below.\n", 132 | "- Look at `res.params` and `res.criterion` to make sure that the optimization worked.\n", 133 | "\n", 134 | "## Solution 2:" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 6, 140 | "id": "bdf77720", 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "start_params = {\"a\": 0, \"b\": np.zeros(3), \"c\": np.zeros((2, 2))}\n", 145 | "\n", 146 | "\n", 147 | "def new_f(x):\n", 148 | " value = (\n", 149 | " (x[\"a\"] - np.pi) ** 2\n", 150 | " + np.sum((x[\"b\"] - np.arange(3)) ** 2)\n", 151 | " + np.sum((x[\"c\"] - np.eye(2)) ** 2)\n", 152 | " )\n", 153 | " return value" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": 7, 159 | "id": "1dc0dea3", 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "res = em.minimize(\n", 164 | " criterion=new_f,\n", 165 | " params=start_params,\n", 166 | " algorithm=\"scipy_neldermead\",\n", 167 | ")" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 8, 173 | "id": "4286a366", 174 | "metadata": {}, 175 | "outputs": [ 176 | { 177 | "data": { 178 | "text/plain": [ 179 | "{'a': 3.141592659470512,\n", 180 | " 'b': array([-1.68302929e-09, 1.00000000e+00, 2.00000000e+00]),\n", 181 | " 'c': array([[1.00000000e+00, 7.12514518e-09],\n", 182 | " [7.33457472e-11, 1.00000000e+00]])}" 183 | ] 184 | }, 185 | "execution_count": 8, 186 | "metadata": {}, 187 | "output_type": "execute_result" 188 | } 189 | ], 190 | "source": [ 191 | "res.params" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 9, 197 | "id": "ea9eb053", 198 | "metadata": {}, 199 | "outputs": [ 200 | { 201 | "data": { 202 | "text/plain": [ 203 | "1.1388734391083942e-16" 204 | ] 205 | }, 206 | "execution_count": 9, 207 | "metadata": {}, 208 | "output_type": "execute_result" 209 | } 210 | ], 211 | "source": [ 212 | "res.criterion" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": 10, 218 | "id": "8abdc1a1", 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "from numpy.testing import assert_array_almost_equal\n", 223 | "\n", 224 | "assert_array_almost_equal(res.params[\"a\"], np.pi)\n", 225 | "assert_array_almost_equal(res.params[\"b\"], np.arange(3))\n", 226 | "assert_array_almost_equal(res.params[\"c\"], np.eye(2))" 227 | ] 228 | }, 229 | { 230 | "cell_type": "markdown", 231 | "id": "f2a1890b", 232 | "metadata": {}, 233 | "source": [ 234 | "## Task 3: Plotting\n", 235 | "\n", 236 | "Use `em.criterion_plot` and `em.params_plot` to visualize the convergence of your optimization. Use the `selector` argument of `em.params_plot` to visualize only a subset of the parameters. Example: `selector=lambda params: params[\"b\"]`.\n", 237 | "\n", 238 | "## Solution 3:" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": 11, 244 | "id": "94711955", 245 | "metadata": {}, 246 | "outputs": [ 247 | { 248 | "data": { 249 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAH0CAYAAADfWf7fAAAgAElEQVR4Xu3dCXhV1bn/8ZdBIAQbIUBQpgqVUC2BggOgYAWt0guiFXH4K0Vb5wlBHKiigsJVuQyKOFtRHICrLU8QUAYVZBQChoqEABUJQkSEUMIY5P+8q57cJGQ6e52zc9Y+3/08Pio5a++1Pu8+4XfWWXvvaseOHTsmbAgggAACCCCAAAIIBFSgGoE3oJVlWAgggAACCCCAAAJGgMDLiYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLycAwgggAACCCCAAAKBFiDwBrq8DA4BBBBAAAEEEECAwMs5gAACCCCAAAIIIBBoAQJvoMvL4BBAAAEEEEAAAQQIvJwDCCCAAAIIIIAAAoEWIPAGurwMDgEEEEAAAQQQQIDAyzmAAAIIIIAAAgggEGgBAm+gy8vgEEAAAQQQQAABBAi8nAMIIIAAAggggAACgRYg8Aa6vAwOAQQQQAABBBBAgMDLOYAAAggggAACCCAQaAECb6DLy+AQQAABBBBAAAEECLwW58Bpp50m2dnZFnugKQIIIIAAAggggEC0BQi8FsIEXgs8miKAAAIIIIAAAj4JEHgtoAm8Fng0RQABBBBAAAEEfBIg8FpAE3gt8GiKAAIIIIAAAgj4JEDgtYAm8Frg0RQBBBBAAAEEEPBJgMBrAU3gtcCjKQIIIIAAAggg4JMAgdcCmsBrgUdTBBBAAAEEEEDAJwECrwU0gdcCj6YIIIAAAggggIBPAgReC2gCrwUeTRFAAAEEEEAAAZ8ECLwW0AReCzyaIoAAAggggAACPgkQeC2gCbwWeDRFAAEEEEAAAQR8EiDwWkATeC3waIoAAggggAACCPgkQOC1gCbwWuDRFAEEEEAAAQQQ8EmAwGsBTeC1wKMpAggggAACCCDgkwCB1wKawGuBR1MEEEAAAQQQQMAnAQKvBbQG3h6vPy7NEutL3ZonSN2ataR5vfpmj/pn+t/JtRMtjkBTBBBAAAEEEEAAAVsBAq+FYCjwlrWLNkmNZUjahRZHoCkCCCCAAAIIIICArQCB10IwtKRha/5u2V9wWHYdzDf/6P9/uSvH7LnHKalyVetOFkehKQIIIIAAAggggICNAIHXQq+8NbxZebkyNnO+2fvgtJ6SmpRicSSaIoAAAggggAACCHgVIPB6lRORii5am7pplSz4LktY2mCBTFMEEEAAAQQQQMBSgMBrAVhR4NVlDvcu/V9zhHFd+pmL2tgQQAABBBBAAAEE/BUg8Fp4VxR4dddjMudJdt730rtFO+nTsp3F0WiKAAIIIIAAAggg4EWAwOtF7ec2lQm887etl2mbM6RLyqkysE0Xi6PRFAEEEEAAAQQQQMCLAIHXi1oYgVfv2PBExmxzP95RZ/e1OBpNEUAAAQQQQAABBLwIEHi9qIURePWltyx6x7Tgbg0W2DRFAAEEEEAAAQQ8ChB4PcJps8osadDXjcyYJTn5e+RPbTpL15RWFkekKQIIIIAAAghEW+DYsWOyd+9eSUpKivah2L9PAgReC+jKBt70LWtl5rdruXDNwpqmCCCAAAII+CHw2WefyT333GMO1ahRI3nxxReldevWfhyaY0RRgMBrgVvZwBu6cI2nrllg0xQBBBBAAIEoCxw8eFA6d+4szz//vJx77rny9ttvy/vvvy8ffPBBlI/M7qMtQOC1EK5s4A09de20pMZyX9qFFkekKQIIIIAAAghES2Dx4sXy2GOPydy5c80hNAC3a9dOPv30U2natGm0Dst+fRAg8FogE3gt8GiKAAIIIICAiFnyVxWb3h+/5DZjxgyZPXu2WcYQ2nr06CFjxoyRjh07VkU3OWaEBAi8FpCVDbx6iNCdGl7qdq3FEWmKAAIIIIBAsARCfz/6ParS/j6eMmWKZGRkyNixYwu707dvXxkyZIh0797d7y5yvAgKEHgtMAm8Fng0RQABBBBAIMZmeNPT02XmzJny0ksvFZvh1QDcoUMH6uWwAIHXonjhBN7Qrcke7thLmifWtzgqTRFAAAEEEEAgGgIrV66U+++/XxYsWGB2v3//fmnfvj1reKOB7fM+CbwW4OEE3jGZ8yQ773sePmHhTVMEEEAAAQSiKVBQUGDuzjB69GjRtbuvv/666Lpe/YfNbQECr0X9wgm8b2xYKktz/yX9W3WUnk3bWhyVpggggAACCCAQLQGd5b3jjjvk0KFD0qBBA5k0aZK0bcvf29Hy9mu/BF4L6XACLw+fsICmKQIIIIAAAj4L5OXl8aQ1n82jeTgCr4VuOIF3Se5mmbxhmXRJOVUGtulicVSaIoAAAggggAACCIQjQOANR6vEa8MJvDx8wgKapggggAACCCCAgIUAgdcCj8BrgUdTBBBAAAEEEEDAJwECrwU0gdcCj6YIIIAAAggggIBPAgReC2gvgbdZYn15pGMvi6PSFAEEEEAAAQQQQCAcAQJvOFolXhtO4N2av1ueyJht9sDjhS3QaYoAAggggAACCIQpQOANE6zoy8MJvNou9LxwnrZmgU5TBBBAAAEEEEAgTIG4C7z6nOzBgwfL1KlTpWPHjoYr9OjAonbjxo2T3r17l8vpNfAOTuspqUkpYZaKlyOAAAIIIIAAAgh4EYirwDtixAhZsWKF7Ny50zw5pVOnTsZs+/btctlll8nChQsLDWvWrCk1atSIaOAdmTFLcvL38HhhL2cqbRBAAAEEEEAAAY8CcRV4FyxYIJ07d5ZrrrlGhg8fXhh4v/76axk6dKjMnDkzLMZwZ3jHZM6T7LzvCbxhKfNiBBBAAAEEEEDATiCuAm+ISpcqPP7444WBd+nSpTJo0CDp16+fNGzYUHr27CktWrSoUJbAWyERL0AAAQQQQAABBKpcgMArIrm5uWZNb4MGDWTdunUya9YsGT16tPTq9X+3D5swYcJxxZo4caJkZ2dXuojM8FaaihcigAACCCCAAAIREyDwlkI5ZcoUmTNnjui/Q5te7FZy04vfvATe207vLh2Sm0WsiOwIAQQQQAABBBBAoGwBAm8pNosWLZK77rpL1qxZU+6543VJQ+8W7aRPy3aclwgggAACCCCAAAI+CBB4RWT9+vWSkpIi9evXN8sb9IK2hIQEGT9+fEQD79RNq2TBd1nSJeVUGdimiw/l5RAIIIAAAggggAACcRl4+/btK48++mjhfXh1ucKzzz4re/fulVq1asmll14qN954oyQnJ0c08KZvWSszv10rpyU1lvvSLuTsQwABBBBAAAEEEPBBIC4Db1mu+/btk8TERKlWrVql6MNd0kDgrRQrL0IAAQQQQAABBCIqQOC14Aw38C7J3SyTNyxjhtfCnKYIIIAAAggggEC4AgTecMWKvD7cwJuVlytjM+cTeC3MaYoAAggggAACCIQrQOANV4zAayFGUwQQQAABBBBAwH8BAq+FOTO8Fng0RQABBBBAAAEEfBIg8FpAE3gt8GiKAAIIIIAAAgj4JEDgtYAm8Frg0RQBBBBAAAEEEPBJgMBrAe018DZLrC+PdOxlcWSaIoAAAggggAACCFRWgMBbWalSXuc18OquXup2rcWRaYoAAggggAACCCBQWQECb2WlIhB4dRe3LHrH7GnU2X0luXaixdFpigACCCCAAAIIIFAZAQJvZZTKeE24M7xFA+/gtJ6SmpRicXSaIoAAAggggAACCFRGgMBbGSUCr4USTRFAAAEEEEAAgaoVIPBa+DPDa4FHUwQQQAABBBBAwCcBAq8FtJfA+9CKGfLjoXxhSYMFPE0RQAABBBBAAIEwBAi8YWCVfKmXwDsmc55k531P4LVwpykCCCCAAAIIIBCOAIE3HK0SryXwWuDRFAEEEEAAAQQQ8EmAwGsBTeC1wKMpAggggAACCCDgkwCB1wKawGuBR1MEEEAAAQQQQMAnAQKvBTSB1wKPpggggAACCCCAgE8CBF4LaAKvBR5NEUAAAQQQQAABnwQIvBbQXgLvGxuWytLcf0n/Vh2lZ9O2FkenKQIIIIAAAggggEBlBAi8lVEq4zVeAm/6lrUy89u10rtFO+nTsp3F0WmKAAIIIIAAAgggUBkBAm9llAi8Fko0RQABBBBAAAEEqlaAwGvhbzPD2z65mdx+eneLo9MUAQQQQAABBBBAoDICBN7KKEVhhve0pMZyX9qFFkenKQIIIIAAAggggEBlBAi8lVGKYOBdkrtZJm9YJgReC3iaIoAAAggggAACYQgQeMPAKvlSL0sasvJyZWzmfAKvhTtNEUAAAQQQQACBcAQIvOFolXgtgdcCj6YIIIAAAggggIBPAgReC2gCrwUeTRFAAAEEEEAAAZ8ECLwW0AReCzyaIoAAAggggAACPgkQeC2gCbwWeDRFAAEEEEAAAQR8EiDwWkATeC3waIoAAggggAACCPgkQOC1gLYJvAk1TpDxXa+0ODpNEUAAAQQQQAABBCojQOCtjFIZr7EJvLrLl7pda3F0miKAAAIIIIAAAghURoDAWxklAq+FEk0RQAABBBBAAIGqFSDwWvgzw2uBR1MEEEAAAQQQQMAnAQKvBTSB1wKPpggggAACCCCAgE8CBF4LaAKvBR5NEUAAAQQQQAABnwQIvBbQBF4LPJoigAACCCCAAAI+CRB4LaAJvBZ4NEUAAQQQQAABBHwSIPBaQHsJvPsLDsvIjNny46F8GXV2X0munWjRA5oigAACCCCAAAIIVCRA4K1IqJyfewm8ursxmfMkO+97GZzWU1KTUix6QFMEEEAAAQQQQACBigQIvBUJEXgthGiKAAIIIIAAAghUvQCB16IGzPBa4NEUAQQQQAABBBDwSYDAawFN4LXAoykCCCCAAAIIIOCTAIHXAprAa4FHUwQQQAABBBBAwCcBAq8FNIHXAo+mCCCAAAIIIICATwIEXgtoAq8FHk0RQAABBBBAAAGfBAi8FtAEXgs8miKAAAIIIIAAAj4JEHgtoAm8Fng0RQABBBBAAAEEfBIg8FpA2wbe/q06Ss+mbS16QFMEEEAAAQQQQACBigQIvBUJlfNz28Dbu0U76dOynUUPaIoAAggggAACCCBQkQCBtyIhAq+FEE0RQAABBBBAAIGqFyDwWtSAGV4LPJoigAACCCCAAAI+CRB4LaAJvBZ4NEUAAQQQQAABBHwSIPBaQBN4LfBoigACCCCAAAII+CRA4LWAJvBa4NEUAQQQQAABBBDwSYDAawFN4LXAoykCCCCAAAIIIOCTAIHXAprAa4FHUwQQQAABBBBAwCcBAq8FtNfAO3/bepm2OUO6pJwqA9t0segBTRFAAAEEEEAAAQQqEvA18O7atUu2bNkiDRs2lBYtWlTUt5j/udfAm5WXK2Mz58tpSY3lvrQLY36cdBABBBBAAAEEEHBZwLfAO2rUKPnb3/5mrO666y65++675aOPPpIxY8bI3LlznTQk8DpZNjqNAAIIIIAAAnEm4EvgXbNmjQwcOFD++te/SkZGhpx88skm8H7zzTdy0UUXycKFC82fubYReF2rGP1FAAEEEEAAgXgU8CXwPvDAA1KvXj155JFHZMKECVKtWjUTeA8ePCjt2rWTmTNnSmpqqnP+BF7nSkaHEUAAAQQQQCAOBXwJvLqcYceOHfLss88WC7y6lOH222+XlStXSlJSknP8BF7nSkaHEUAAAQQQQCAOBXwJvKtWrZKrr75abrjhBhN8a9WqJW3atJG33nrLzPBOmjTJSXoCr5Nlo9MIIIAAAgggEGcCvgReNZ0zZ448+eSTJvCGtr59+5p1vfXr13eSncDrZNnoNAIIIIAAAgjEmYBvgVddjx07Jjt37pS9e/fKKaecInXr1nWam8DrdPnoPAIIIIAAAgjEiYCvgTdopgTeoFWU8SCAAAIIIIBAEAV8CbxTpkyRGTNmlOmn9+fVuzi4thF4XasY/UUAAQQQQACBeBTwJfAuX75cMjMzC32PHDkiOTk5Mn36dLnuuutk2LBhcsIJJzjnbxt4E2qcIOO7XuncuOkwAggggAACCCDgkoAvgbcskNmzZ8tjjz0mS5culerVq7vkZvpqG3h1Hy91u9a5cdNhBBBAAAEEEEDAJYEqDbwHDhyQtLQ0WbBggTRv3twlNwKvc9WiwwgggAACCCAQrwJVGni/+uorueyyy0QfPZyYmOhcDZjhda5kdBgBBBBAAAEE4lDAl8CrF6x99tlnxXh//PFHWbx4sVx00UVx++AJBWFJQxy+6xgyAggggAACCPgq4Evg1YdOLFmypNjA6tSpI+eff76cddZZ5slrsbAVFBTIwYMHK33HCGZ4Y6Fq9AEBBBBAAAEEEChfwJfAG0tFSE9Pl8GDB8vUqVOlY8eOhV2bOHGiTJgwwSyt0BA+bty4CoMvgTeWKktfEEAAAQQQQACB0gWiFng3bdokeXl5lXLXC9dq1qxZqdfavGjEiBGyYsUK87S3SZMmSadOnczu9JZpN910k7lXcKNGjWTIkCHmIjr9d3kbgdemGrRFAAEEEEAAAQT8EYha4L311ltl/vz5lRrFypUrJSkpqVKvtXmR3g2ic+fOcs0118jw4cMLA6/O7Go41z/TbdWqVTJo0CBZtGhRVALv/oLDcu/S/zX7Zg2vTUVpiwACCCCAAAIIVCwQtcB7+PBhOXr0aMU9EJGEhIRKvS5SL+rdu7c8/vjjhYFXH3zRtm1bGTBggDlEbm6unHfeebJ+/XqpUaNGmYf1OsOrOxy0ZLocOHpExnXpJ3VrxsYa5kj5sh8EEEAAAQQQQCCWBKIWeGNpkCX7UjLw3nnnndK9e3fp37+/eenevXtNGF69enXhOt5//vOfxw3p8ssvl+zsbE9DHZM5T7LzvpdmifWlTVJjuar1f5ZXsCGAAAIIIIAAAghEVsCXwKuPEv74449Fn6z23XffHTeCN998s8ILxCI57JKB9+GHH5bWrVvLDTfcYA6zY8cO6datm2RlZRU+AU6XOJTcPvzwQ+vAq/vU0PtIx16RHCL7QgABBBBAAAEEEPhZwJfAq2thb7zxRjOD+u2335r1uvooYQ3A99xzj+h6Xz8uWgtVvWTgfeGFF2T79u2iF7XptmzZMhk6dGjU1vDqMUIzvKE+sZaX9yQCCCCAAAIIIBAdAV8C76hRo0zvda3sc889Z2ZzdTb1k08+kQcffNDco7e8tbKRHnrJwLtx40bp16+fzJw5U5o0aSJ33323nHrqqSb0lrfZrOEl8Ea6quwPAQQQQAABBBAoXcCXwKt3PzjxxBNNgHz33XfNXRDGjBkjemHbGWecIfPmzZOWLVv6VqO+ffvKo48+Wuw+vLqsYuzYsaYPen9evQ9vRXeOIPD6VjIOhAACCCCAAAIIeBbwJfC+9957MnnyZLOEISMjwyxveOWVV8xTzfS/9c80EFf15teT1nSczPBWdbU5PgIIIIAAAgjEi4AvgVfvcavrYi+++GI5duyYWc6wePFiY3zVVVfJE0884aR3JGd4B6f1lNSkFCcd6DQCCCCAAAIIIBDLAr4E3n379hW7C4MuZVizZo00bNhQWrVqFcs+5faNwOts6eg4AggggAACCMSRgC+B95lnnpHPPvvMPOGsV69e0qBBg0AQRzLw6q3JhqT15CEUgTgzGAQCCCCAAAIIxJKAL4F306ZNMnXqVJk2bZrk5+eb0Kt3RejatauvtyOLNLxN4H1jw1JZmvuvYl3qknKqDGzTJdLdZH8IIIAAAggggEBcC/gSeEPC+gAKvQXZjBkzJD093cz0Xn311XLHHXdIrVruPV7XJvBm5eXK2Mz5xU6+5NqJ0rxefdm6b7d0bfKfpR5tklIkoeYJ0jyxflyfqAweAQQQQAABBBDwKuBr4C3ayZUrV8pDDz0k33zzjeh/V3QLMK8DjGY7m8Cr/UrfstZ0T0Nu1p5cWZq7WQ4cPVJmlxNqnCB9WqZJh4bNRMMxGwIIIIAAAggggEDFAr4GXn2ssD6OV5c2aNA999xzzdIGXeLg54MnKmap3CtsA2/Jo4QCcNE/15ngAwWHJSd/T+Efa9jt37qTdEhuVrmO8ioEEEAAAQQQQCCOBXwJvMuXL5eJEyeaW5Ppk8z04jV9+EPTpk2dpo904K0IY82uHEnfkmnCr872Dml/IUsdKkLj5wgggAACCCAQ9wK+BF59nHBOTo5cccUVcuaZZ0r16tUDAe934FW0/QWHZWTGbPnxUL7onR1+27CZWfOra311aUS4W5ukxuE24fUIIIAAAggggIBTAr4EXqdEwuhsVQRe7d7W/N0y6auFJvRW1RZuUE6uU08a1im+7jihRq3jQjoX6FVVRTkuAggggAACwRUg8FrUtqoCr3Z516F8s7whuXY9MwINwTr7G+6Ws293uRfKhbu/aL2+tICtM9p1axa/u4eub9ZwXXQjREerKuwXAQQQQAABNwQIvBZ1qsrAa9HtiDTVi+nC2TRY7y8ofgeKXYf2yQ8Hi89Sl7xAL5xjhPPank1TpX+rTuE04bUIIIAAAggg4KgAgdeicPEceC3YPDUtLWCXFqJLm+nedTC/1OUfXVJaycA2nT31h0YIIIAAAggg4I6Ar4F3/fr1snHjxuN0fv/738fdgyfcOUWC19OiT7n7U5vO0jXlPw/5YEMAAQQQQACBYAr4Enhzc3PNrci2bt0qiYmJUrt27WKac+fOlV/84hfOCTPD61zJCjscCr16e7dRZ/c9bi2wuyOj5wgggAACCCBQUsCXwPvuu+/K22+/LS+++KI0axachyUQeN1+Q43JnCfZed9Lj1NS5arWrOd1u5r0HgEEEEAAgbIFfAm89913n7Rs2VLuuuuuQNWCwOt2OfVBHi+sW2ge4sEsr9u1pPcIIIAAAgiUJ+BL4H311VdFly1MnTo1UNUg8LpfzlsWvWMGwVpe92vJCBBAAAEEEChLwJfA+/3330ufPn3ksssuk86dO0tSUlKx/qSlpUnNmjWdqxKB17mSHdfh9C1rZea3a6V9cjO5/fTu7g+IESCAAAIIIIDAcQK+BN5nnnlGXn755TL5V65ceVwIdqFWBF4XqlR+H/UBHsNWzDDLGi5s1ta8uHeLdu4PjBEggAACCCCAQKGAL4H3yJEjUlBQUCZ7QkKCkyUh8DpZtuM6PTJjluTk7yn884c79pLmifWDMThGgQACCCCAAALiS+At6ucKWLAAACAASURBVHz06FHRAFynTh3n+Qm8zpfQDGDqplWy4LuswsE0S6wvDeskmv+/jWUOwSgyo0AAAQQQiGsB3wLvl19+Kbq0Yfny5Qa8QYMG5t68t9xyizDDG9fnYEwMfknuZpm8YdlxfdGnsXVJOdXM+NatWSsm+konEEAAAQQQQCA8AV8C7+bNm+Xiiy+Wc889V84//3xp0qSJfPHFF/Lhhx9Kjx49ZPTo0eH1OkZezQxvjBQiQt3Q9bzpWzIloUYt2ZCXW2yZQ3LtRGlerz4zvhGyZjcIIIAAAgj4KeBL4B03bpzMmjVLPvroI6levXrh+D7++GO54447ZPXq1VKvXj0/xx2RYxF4I8IYkzvZX3BYluZuFr2Lw4GjRwr7qDO+/Vt1ZLY3JqtGpxBAAAEEEChdwJfAO3jwYDnllFNEH0BRdNu/f7+0b99e5syZI61bt3auRgRe50oWdoe35u+WNT/kmHbzt6034ZfQGzYjDRBAAAEEEKhSAV8C73PPPSfTpk2TTz75pNj9dv/xj3/IY489JnpbMu7DW6XnAQevhEDRdb46y9uz6X9uY8aGAAIIIIAAArEt4Evg3bFjh1xyySXSqFEjueCCCyQlJUVWrFghCxYsEJ39ve2222JbqYzeMcPrZNmsOh0KvXoh28A2Xaz2RWMEEEAAAQQQ8EfAl8CrQ/nmm29k4sSJkpGRIfn5+dKhQwf5wx/+IH379vVnpFE4CoE3Cqgxvktd4vBExmzTS2Z5Y7xYdA8BBBBAAIGfBXwLvEEUJ/AGsaoVj+mWRe+YF+n9en/bsJls3bebuzdUzMYrEEAAAQQQqDKBqAXeQ4cOyapVq+TMM8+UnTt3Sm5ubpmDTEtLYw1vlZ0CHDhcgTW7cuSFdQuLNXup27Xh7obXI4AAAggggIBPAlELvMuWLZPrr79epk+fLnPnzpWXX365zCHpRWtJSUk+DTlyh2GGN3KWru1p0rqF8uWu/9y9QbfBaT0lNSnFtWHQXwQQQAABBOJCIGqBV/X27NkjJ510knmUcEFBQZmgPGktLs61QA2y5JPZCLyBKi+DQQABBBAImEBUA2/IaurUqdK4cWNzh4YgbczwBqma4Y1FH0xx79L/LWzUJqmx9G7ZjkcQh8fIqxFAAAEEEPBFwJfAO2zYMKldu7Y8+uijvgzKr4MQeP2Sjs3jPLRihvx4KL9Y50IXsvVu0S42O02vEEAAAQQQiEMBXwKvPnDi5ptvls8//9zcgzcoG4E3KJX0Pg69gG3qplXHBd/k2omSXCfxuB2nnnT8+Z9Qo5Y0r1f/uNcm1DzBzBizIYAAAggggICdgC+BNzs7W0aMGCF6IVuvXr3kjDPOKNbrgQMHmhlg1zYCr2sVi05/dx3Kl6w9ueb2ZPrfRS9mi8YRdfmEBucfDuZLalJjaZ/cTOrWrBWNQ7FPBBBAAAEEAiHgS+CdMmWKpKenlwn22muvSb169ZwDJfA6VzJfOpy+Za2ZsdUZ2pLbhj3fH/dnuh5YH2hRcjtQcFhy8vdU2GedTdbHHLc5qTEzwhVq8QIEEEAAgXgU8CXwBhWWwBvUysbuuDQcb8j73swob8jLLRaIE2qcYIL2bad3Z8Y3dktIzxBAAAEEqkDAt8Crtyb79NNPZfPmzaIPmujSpYvs3r1b8vLy5Je//GUVDN3+kARee0P2YCegAXj+tiyZv229HDh6xOxMlzwMSbvQbse0RgABBBBAIEACvgRevQfvFVdcIevWrTN0d911l9x9992iF7Pde++98sUXX8gJJxz/9W+sOxN4Y71C8dM/XTus4fd/vpxngm+H5GZmtlfvGqH/zYYAAggggEA8C/gSeGfPni2jRo2SyZMnm7W81apVM4FXHzd83nnnyfz586VFixbO1YHA61zJAt9hXQscCr2hweoa31Fn9w382BkgAggggAACZQn4EngHDRokzZs3lyFDhsiECRMKA68+ie2ss86SWbNmiYZH1zYCr2sVi4/+ZuXlmmUORe8WMbBNZ2lzUopo+GVDAAEEEEAg3gR8Cbzjx4+XxYsXy/Tp04sF3rffflsee+wxWbt2rdSpU8c5ewKvcyWLmw7rEod5OetF7xMcejiGXtTWp2U7c0cHNgQQQAABBOJJwJfAm5OTI71795ZmzZpJrVq1zD13ExISZNGiRXL77bebdbwubgReF6sWX33W4Dty1azCC9p0Te8jHXvFFwKjRQABBBCIewFfAq8qb9myRV588UVZuXKl7Ny50yxh6N+/v/zxj3+UGjVqOFkIAq+TZYu7To/JnCfZef93/98ep6TKVa07xZ0DA0YAAQQQiF8B3wJvWcT5+fmSmOjmukICb/y+cVwcedHgO65LP+7V62IR6TMCCCCAgCcBXwLvM888I40aNRJ9hHDJrUOHDjJz5kyz3MG1jcDrWsXobyj06q3K+rfuxEVsnBIIIIAAAnEhUKWBd+vWrdKjRw9ZuHChnHzyyc6BE3idK1ncd1jv4DA2c36hg96ujDs3xP1pAQACCCAQeIGoBt6MjAwZPny46EVreqGazvKGtqNHj8rGjRulY8eOMnXqVCehCbxOli3uOz110ypZ8F2WcdA7N9x2RndJTUqJexcAEEAAAQSCKxDVwPvTTz/JV199JU899ZTUq1dPunbtWiipD5/QwNipUycnn7KmAyHwBveNEeSR6RPZ0resLRZ6Q09l42K2IFeesSGAAALxKxDVwBti3bBhg+zbt8/M5gZpI/AGqZrxN5ZbFr1TbNDM9sbfOcCIEUAAgXgRiFrgPXTokKxatUrOPPNMcxsyfYxwWVtaWprUrFnTOXMCr3Mlo8NFBEZmzJKc/D3Hmei9enVrWCdRdDY49SR/ljvoWuLkOvUiXqPkOomsU464KjtEAAEE3BKIWuBdtmyZXH/99ebpanPnzpWXX365TBm9N29SUpJbcixpcK5edLi4gK7l1aUMuw7my5LczYVPZMOpcgL/Cejh3VJRA71+kGiTlCJtkhpX7kC8CgEEEEDAWiBqgVd7tmfPHjnppJPkyJEjUlBQUGZn9alrLm7M8LpYNfpcmsD8besloWYt8yMNwDqzW/fn//dLbNehffLDwfyIHy5n3+7CJ81FfOcWO9QlJF2btJL+rXgIiAUjTRFAAIFKCUQ18IZ6oHdhaNy4sVxwwQWV6pQrLyLwulIp+olA5AW25u82HwzC2TR8a6hfsyuncEZd74n8pzadff+AEU6/eS0CCCDguoAvgXfYsGHmtmSPPvqo617F+k/gDVQ5GQwCvgroPZFf+GqhmX3W5REdGjaT3i3aEXx9rQIHQwCBeBHwJfB+8skncvPNN8vnn38uKSn+XADjRwEJvH4ocwwEgiuw61C+TPrqs8KLB3WZw5D2F0rzny8cDO7IGRkCCCDgr4AvgTc7O1tGjBgheiFbr1695Iwzzig2Sn3ksM4Au7YReF2rGP1FIPYEdFnE/G1ZhRcOaujVO2PoBYX5Rw4L90aOvZrRIwQQcE/Al8A7ZcoUSU9PL1PntddeMw+mcG0j8LpWMfqLQOwK6JpgfSDIl7tyCjupSx30SXhb9+2WrimtYrfz9AwBBBCIcQFfAm+MG3juHoHXMx0NEUCgDIE3NiyVNT/kFLuzhAbfUWf3xQwBBBBAwKNAVAPv+PHjzZ0Z2rdvX2r3cnJyZM2aNdK7d2+P3a/aZgTeqvXn6AgEVSB0B4jQRW06zoc79mJtb1ALzrgQQCDqAlENvOecc448/vjjcskll5Q6kMzMTLniiitEHz1crVq1qA820gcg8EZalP0hgEBRgaLLHHSW90+pnSU1KTgX/lJtBBBAwC+BqAXe7du3S/fu3WXRokXSpEmTUsezd+9e6dSpU7mv8QvCy3EIvF7UaIMAAuEIaOh9I2upuZMDSxvCkeO1CCCAwP8JRC3w5ubmynnnnScrVqyQ+vXrl2qel5cnZ555prmgrW3bts7VhcDrXMnoMALOCgxaMt2s69WHVHABm7NlpOMIIFBFAlELvDqeHj16yNChQ82tyErb5s+fL7feeqvo0gYXHy9M4K2is5bDIhCHAnoHh5nfrpU2SY1lSNqFcSjAkBFAAAHvAlENvE8//bS88847Mnny5OMuXMvKypIBAwaYJQ2TJk3yPoIqbEngrUJ8Do1AnAno/XqHrZhhZnm5gC3Ois9wEUDAWiCqgbegoED++te/ygcffCCdO3eWX//616bDmzZtkoULF0rz5s1l2rRp0rBhQ+uBVMUOCLxVoc4xEYhfgambVsmC77KY5Y3fU4CRI4CAR4GoBl7t008//SQfffSRrF69WlauXCmHDx+WVq1aid7B4corr5RatWp57HrVNyPwVn0N6AEC8SSgjyIeuWqWmeXVpQ39W3fiVmXxdAIwVgQQ8CwQ9cDruWcONCTwOlAkuohAwATGZM6T7Lzvzaj6t+ooPZu6d8FvwErCcBBAwAEBAq9FkQi8Fng0RQABTwL6JLaluf8ybXu3aCd9WrbztB8aIYAAAvEkQOC1qDaB1wKPpggg4EkgdLcGbdwl5VQZ2KaLp/3QCAEEEIgnAQKvRbUJvBZ4NEUAAU8Cuo53zQ9bZdrmDGZ5PQnSCAEE4lGAwGtRdQKvBR5NEUDAs0BWXq6MzZxP4PUsSEMEEIg3AQKvRcUJvBZ4NEUAASuBWxa9Y9qzrMGKkcYIIBAnAgRei0ITeC3waIoAAlYCD62YIT8eypfTkhrLfTx5zcqSxgggEHwBAq+I7N+//7gnwY0bN0569+5d7hlA4A3+G4QRIhCrAqGL1wi8sVoh+oUAArEkQOAVke3bt8tll11mnv4W2mrWrCk1atQg8MbS2UpfEECgmEBoWcNL3a5FBgEEEECgHAECr4h8/fXXMnToUJk5c2ZYJwszvGFx8WIEEIiwAIE3wqDsDgEEAitA4BWRpUuXyqBBg6Rfv37SsGFD6dmzp7Ro0aLCohN4KyTiBQggEEWB0Drehzv24hHDUXRm1wgg4L4AgVdEcnNzZerUqdKgQQNZt26dzJo1S0aPHi29evUqrPB11113XLWXL18u2dnZ7p8FjAABBJwUCD1muFlifRmY2pnQ62QV6TQCCPghQOAtRXnKlCkyZ84c0X+Hth07dhz3ym7duhF4/ThLOQYCCJQqEAq8+sPBaT0lNSkFKQQQQACBUgQIvKWgLFq0SO666y5Zs2ZNuScNSxp4TyGAQFUKFH3M8J/adJauKa2qsjscGwEEEIhZAQKviKxfv15SUlKkfv36ZnnD8OHDJSEhQcaPH0/gjdlTl44hgEDRwNu7RTvp07IdKAgggAACzPCWfg6kp6fLs88+K3v37pVatWrJpZdeKjfeeKMkJycTeHnbIIBAzAosyd0skzcsM/3rcUqqXNW6U8z2lY4hgAACVSnADG8R/X379kliYqJUq1atUjVhSUOlmHgRAghESSArL1fGZs43e+cBFFFCZrcIIBAIAQKvRRkJvBZ4NEUAAWsBAq81ITtAAIE4ESDwWhSawGuBR1MEEIiIQOjhEwk1TpDxXa+MyD7ZCQIIIBA0AQKvRUUJvBZ4NEUAgYgIzN+2XqZtzjD74hHDESFlJwggEEABAq9FUQm8Fng0RQCBiAkMWjJdDhw9ItyaLGKk7AgBBAImQOC1KCiB1wKPpgggEDGB0AMokmsnStcmrURvUcaGAAIIIPB/AgRei7OBwGuBR1MEEIiYQNEnrulOuUVZxGjZEQIIBESAwGtRSAKvBR5NEUAgYgJTN62SBd9lFe6PW5RFjJYdIYBAQAQIvBaFJPBa4NEUAQQiJlD09mS6UwJvxGjZEQIIBESAwGtRSAKvBR5NEUAgogKhC9dCO+VRwxHlZWcIIOC4AIHXooAEXgs8miKAQEQFSq7jJfBGlJedIYCA4wIEXosCEngt8GiKAAIRFViSu1nSt6yVHw/lF+53cFpPSU1Kiehx2BkCCCDgogCB16JqBF4LPJoigEDEBUrO8hJ4I07MDhFAwFEBAq9F4Qi8Fng0RQCBiAuUDLxdUlrJwDadI34cdogAAgi4JkDgtagYgdcCj6YIIBBxgZKBVw8wrks/qVuzVsSPxQ4RQAABlwQIvBbVIvBa4NEUAQQiLlBa4NWD3HZ6d+mQ3Czix2OHCCCAgCsCBF6LShF4LfBoigACERdYsytHmterL/qI4ZIPo+jZNFX6t+oU8WOyQwQQQMAFAQKvRZUIvBZ4NEUAgagKzN+2XqZtzih2jDZJjSX1pBQTiPcXHDHhuFniSSx5iGol2DkCCMSCAIHXogoEXgs8miKAgC8CeruyyRuW+XKsSB1EA3lyncRI7S6i+9H10PpBIVa3Zon1w/oAk1DzBGmeGLvjiVVn+uWeAIHXomYEXgs8miKAgG8C+wsOy9b83ZKzb7fM25ZlwqT+d0LNWnKg4LAcOHrEt75woNgU0Nl/DfJtklJY7x2bJaJXlgIEXgtAAq8FHk0RQMB3AQ2+LtyxYdehfPnh4D7ffSpzwAMFR2Trvt2VeWmVvCYrLzes4+46mF/sYSUJNU6QDg2bi675ZuY3LEpeHOMCBF6LAhF4LfBoigACCCAQEwKhbwDmb8uSL3flmD5p8O3fupN0TWkVE32kEwjYChB4LQQJvBZ4NEUAAQQQiDmBNzYslaW5/yrsl97Zg4sbY65MdMiDAIHXA1qoCYHXAo+mCCCAAAIxKaAXOupdPnLy9xT2r8cpqXJVa25rF5MFo1OVEiDwVoqp9BcReC3waIoAAgggELMCusxB7+scusOHLnF4pNMfzC3t2BBwUYDAa1E1Aq8FHk0RQAABBGJeQGd6V+/Kkey870VveXZh01Q5JsLa3pivHB0sKUDgtTgnCLwWeDRFAAEEEHBCoOS9nE9Laiz3pV3oRN/pJAIhAQKvxblA4LXAoykCCCCAgDMCk9YtLLyDA4HXmbLR0SICBF6L04HAa4FHUwQQQAABZwT0/r5jM+eb/uo63lFn93Wm73QUARUg8FqcBwReCzyaIoAAAgg4J3DLondMn8d16efEQ0ycA6bDURMg8FrQEngt8GiKAAIIIOCcwEMrZpgns912enceQexc9eK7wwRei/oTeC3waIoAAggg4JxA+pa1MvPbtdK7RTvp07Kdc/2nw/ErQOC1qD2B1wKPpggggAACzgnovXlfWLdQuHDNudLFfYcJvBanAIHXAo+mCCCAAALOCWzN3y1PZMwu7PfgtJ6SmpTi3DjocPwJEHgtak7gtcCjKQIIIICAkwKhC9e08wReJ0sYl50m8FqUncBrgUdTBBBAAAEnBcZkzjNPXtONtbxOljAuO03gtSg7gdcCj6YIIIAAAk4KvLFhqSzN/ReB18nqxW+nCbwWtSfwWuDRFAEEEEDASYH529bLtM0ZBF4nqxe/nSbwWtSewGuBR1MEEEAAAScFdh3Kl/QtmWaWt31yM7n99O5OjoNOx5cAgdei3gReCzyaIoAAAgg4KxB61DC3J3O2hHHXcQKvRckJvBZ4NEUAAQQQcFaAwOts6eK24wRei9ITeC3waIoAAggg4KyALmsYtmKGJNdOlFFn93V2HHQ8fgQIvBa1JvBa4NEUAQQQQMBpgdD9eF/qdq3T46Dz8SFA4LWoM4HXAo+mCCCAAAJOCxB4nS5f3HWewGtRcgKvBR5NEUAAAQScFhi0ZLocOHpExnXpJ3Vr1nJ6LHQ++AIEXosaE3gt8GiKAAIIIOC0QOiJazxe2Okyxk3nCbwWpSbwWuDRFAEEEEDAaQECr9Pli7vOE3gtSk7gtcCjKQIIIICA0wKhwHvb6d2lQ3Izp8dC54MvQOC1qDGB1wKPpggggAACTgukb1krM79dK71btJM+Lds5PRY6H3wBAq9FjQm8Fng0RQABBBBwWiAUeHnamtNljJvOE3gtSk3gtcCjKQIIIICA0wIEXqfLF3edJ/BalJzAa4FHUwQQQAABpwXmb1sv0zZnCDO8TpcxbjpP4LUoNYHXAo+mCCCAAAJOC2Tl5crYzPkEXqerGD+dJ/Ba1JrAa4FHUwQQQAABpwV2HcqXYStmSHLtROnapJV0SWll/psNgVgUIPBaVIXAa4FHUwQQQAAB5wVCjxfWgfAACufLGegBEHgtykvgtcCjKQIIIICA8wIEXudLGDcDIPBalJrAa4FHUwQQQAAB5wVCD5/QgXA/XufLGegBEHgtykvgtcCjKQIIIICA8wKhC9cIvM6XMvADIPBalJjAa4FHUwQQQAAB5wUIvM6XMG4GQOC1KDWB1wKPpggggAACzgsUDbztk5vJ7ad3d35MDCCYAgRei7oSeC3waIoAAggg4LxA0cCrtyTr3bKddE1p5fy4GEDwBAi8FjUl8Frg0RQBBBBAwHmB/QWH5d6l/1s4Dp665nxJAzsAAq9FaQm8Fng0RQABBBAIhEDRW5MReANR0kAOgsBrUVYCrwUeTRFAAAEEAiFA4A1EGQM/CAKvRYkJvBZ4NEUAAQQQCIRA+pa1MvPbtWYszPAGoqSBHASB16KsBF4LPJoigAACCARGgFnewJQysAMh8FqUlsBrgUdTBBBAAIHACIzMmCU5+XuY5Q1MRYM3EAKvRU0JvBZ4NEUAAQQQCIxA0UcMhwY1OK2npCalBGaMDMRtAQKvRf0IvBZ4NEUAAQQQCIxAaYG3WWJ9eaRjr8CMkYG4LUDgtagfgdcCj6YIIIAAAoERWJK7WSZvWHbceF7qdm1gxshA3BYg8FrUj8BrgUdTBBBAAIFACRS9cC00sHFd+kndmrUCNU4G46YAgdeibgReCzyaIoAAAggESqC0wMs63kCV2OnBEHgtykfgtcCjKQIIIIBAoARKW9ZA4A1UiZ0eDIHXonwEXgs8miKAAAIIBE6gtFleHSTBN3Cldm5ABF6LkhF4LfBoigACCCAQOAECb+BKGpgBEXiLlLKgoEAOHjwo9erVq1SBCbyVYuJFCCCAAAJxIlD0ARRFh8wMb5ycADE8TALvz8WZOHGiTJgwQRITE+Wss86ScePGVRh8CbwxfGbTNQQQQACBKhFgLW+VsHPQCgQIvCKSmZkpN910k8yYMUMaNWokQ4YMkebNm5t/l7cReHl/IYAAAgggUFwgKy9XxmbOL/aH/Vt1lJ5N20KFQJUJEHhFzMxuXl6eDB8+3BRi1apVMmjQIFm0aBGBt8pOTQ6MAAIIIOCiQGmBNzSO5NqJctsZ3aV5Yn0Xh0afHRYg8IrIsGHDpG3btjJgwABTytzcXDnvvPNk/fr1UqNGjTLLywyvw2c+XUcAAQQQiJrA/oLDMm3zKtm6b7fk5O8pdhwNvV2btJI2SSnmz5PrJIr+GRsC0RQg8IrInXfeKd27d5f+/fsb671790qnTp1k9erVhet433rrrePqMGLECMnOzo5mfdg3AggggAACTgs8tGKG/Hgo3+kxxELneUyzXRUIvCLy8MMPS+vWreWGG24wmjt27JBu3bpJVlaWVK9e3fzZm2++eZz0yJEjCbx25x+tEUAAAQTiQCB9y1rRpQ6pSSmyZtdWSfj5ccO7DubLgYLDcuDokThQsBsigdfOj8ArIi+88IJs375ddMZWt2XLlsnQoUNZw2t3btEaAQQQQAABBBCICQECr4hs3LhR+vXrJzNnzpQmTZrI3XffLaeeeqoJveVtrOGNiXOYTiCAAAIIIIAAAuUKEHh/5tElC2PHjjX/17FjR3Mf3qSkJAIvbyAEEEAAAQQQQMBxAQJvkQLypDXHz2a6jwACCCCAAAIIlCJA4LU4LVjSYIFHUwQQQAABBBBAwCcBAq8FNIHXAo+mCCCAAAIIIICATwIEXgtoAq8FHk0RQAABBBBAAAGfBAi8FtAEXgs8miKAAAIIIIAAAj4JEHgtoAm8Fng0RQABBBBAAAEEfBIg8FpAE3gt8GiKAAIIIIAAAgj4JEDgtYAm8Frg0RQBBBBAAAEEEPBJgMBrAU3gtcCjKQIIIIAAAggg4JMAgdcCmsBrgUdTBBBAAAEEEEDAJwECrwU0gdcCj6YIIIAAAggggIBPAgReC2gCrwUeTRFAAAEEEEAAAZ8ECLwW0Bp42RBAAIFwBDp37izLli0LpwmvRQABBCQ7OxsFCwECrwXeW2+9JceOHZMBAwZY7IWmrgn06dNHpk2bJgkJCa51nf56FJg5c6Zs3LhRBg0a5HEPNHNR4MYbb5THHntMWrRo4WL36bMHgeXLl0t6ero88cQTHlrTJJYFCLwW1SHwWuA53JTA63DxPHadwOsRzvFmBF7HC+ih+wReD2iONCHwWhSKwGuB53BTAq/DxfPYdQKvRzjHmxF4HS+gh+4TeD2gOdKEwGtRKAKvBZ7DTQm8DhfPY9cJvB7hHG9G4HW8gB66T+D1gOZIEwKvRaEIvBZ4Djcl8DpcPI9dJ/B6hHO8GYHX8QJ66D6B1wOaI00IvI4Uim4igAACCCCAAAIIeBMg8HpzoxUCCCCAAAIIIICAIwIEXkcKRTcRQAABBBBAAAEEvAkQeL250QoBBBBAAAEEEEDAEQECr0Wh9u/fLzVr1pRatWpZ7IWmsS5QUFAgBw8elHr16h3X1fLOAc6PWK+syL59+yQxMVGqVat2XGfz8vLkxBNPlOrVqxf7mT5sZu/evZKUlHRcm/J+Fvsa8dFDfT8fPnxY6tate9yAy3uve/09EB+qbozyp59+Eq1jyb+zy3qv66i8/B5wQyP+ekng9VDzAwcOyNChQ+Xzzz83ra+77joZMmRIqX9petg9TXwWOHLkiIwbN05eeeUVycrKKhZwJk6cKBMmTDCh6KyzzjKv0+Bb3jnA+eFzAT0cbu7cufL000/Lzp07Tai99tpr5bbbbjN7+vbbb+WWW26R7du3m/9/6qmn5OKLLzb//dlnn8k999xj/rtRo0by4osvSuvWrSv8mYcu0iTCAt98842MHDlSVq1aZfb8m9/8Rh544AFp166d+f+y3uvl/Yz3eoSLFOXdvvsH8gAAFaxJREFU6VPzZs+eLXonhore615/D0R5COzeQoDA6wHv9ddfl08++URee+01E3wuv/xyefzxx6Vbt24e9kaTqhT497//LX/5y1/kpJNOkgULFhQLvJmZmXLTTTfJjBkzTLjRDzXNmzc3/y7vHOD8qMqKVu7Y+ujQli1bSlpamuTk5MgFF1wg8+bNM3+mNdcQdPfdd8u6deukb9++snLlSqldu7Z07txZnn/+eTn33HPl7bfflvfff18++OAD8w1AWT+rXI94VbQF9MONBp1LLrnETE5MmjRJ1qxZY36Pl/de9/p7INrjYf/hCSxevFgeeeQRyc/PLwy8Zb3X9dsbL78HwusRr/ZbgMDrQfyqq66SG264wfzi1E1nebZu3SpPPvmkh73RpCoF9Cuujz76SLp27SpnnnlmscCrM7v6ddbw4cNNF3VmaNCgQbJo0SIp7xzg/KjKino7ttbsz3/+s5x33nnSvn17WbJkifmQo9vAgQPNh9qGDRuKzhDp7LBuGnI1GH/66aeis4dl/axp06beOkWrqAr8/e9/N2FX77Fc3nvd6++BqHaenYcloL/H9T386KOPyv33328Cry45K+u9ftFFF3n6PcB7Payy+P5iAq8Hcp3Jfemll+T00083rfUrkunTp5tZPzY3BfQXYsnAO2zYMGnbtq0MGDDADCo3N9cEovXr18vvfve7Ms8Bzg+3zgGd+dMPPMuWLTN/Cfbo0UOys7MLB6FB9uSTT5YmTZqY97p+wA1t+toxY8aYD7xl/axjx45ugQS4t/ohZcOGDWZm94033jCz+JdddpmU917XWUEvvwcCzOjc0DTkpqamigbZK6+80gRefc+W9V7/wx/+4On3AO/12D41CLwe6nPaaafJnDlzCtfu6fKG5557zny1yeamQGmB984775Tu3btL//79zaD0QqVOnTrJ6tWr5be//W2Z5wDnhzvngF7Aomt39cPrvffeaz7MXH311SYQhTZd66sXrmngzcjIkLFjxxb+TJc76BIXXe9X1s/0HGKLDYHNmzebkKtr9c855xx5+OGHTZgt773+4IMPevo9EBsjphe6VEknqN5991357rvvCgNvee/13r17e/o9wHs9ts83Aq+H+ugM3gsvvGAuetBNvxLTsMsMrwfMGGlSWuDVvwz1giRdvqLbjh07zDpt/cvy/PPPL/Mc4PyIkaJWohs6e7tt2zYza1ujRg3zF6LWtugMry5padasmZnl1fe6/uVZdIZXA7DOFpX1sw4dOlSiJ7zET4FDhw6ZAKS/x3Vtp54HZb3Xtf5efg/4OR6OVbrAjz/+aGZq9f39q1/9yqzX1yVKGoL125yePXuW+l7XwOvl9wDv9dg+Ewm8Hupz/fXXmzWc+qbQTWd3NQyxhtcDZow0KS3w6l+GeqX+iBEjTC/1K2+9O4eu4S3vHOD8iJGiVtCN8ePHy8KFC2Xy5MnmTg26aRDSD7L65xpwdbvmmmvMbI+uz9OvRvXiRt1CawB1Da+eJ2X9jHV9sXk+6AWr+hW0LkXRddllvde9/h6IzVHHV6/0faxr80vb9GJk/YamtPe6Xp/j5fcA7/XYPr8IvB7q895774le8KB/Uepfevqm0VCkV3qzuSlQWuDduHGj9OvXz8zc6dfZ+lXoqaeeakJveecA50fsnwOvvvqqTJ06Vd566y1zhw7ddNmC3p9TL0w85ZRTTJ2//PJL8xXoihUrTCjWuzOMHj3azBrpNzr6l6b+o0sjyvpZ7GvERw917a5efZ+SkmLurqO1f+aZZ8wdGnSWv6z3utffA/Gh6tYodelRaA2v9rys93r9+vU9/R5wSyP+ekvg9VBzvWm5ft398ccfm1uc6Ffeus6r5A3qPeyaJlUkUFrg1a68+eabhWs2dTZI78Orf2mWdw5wflRREcM4rK6zLrnpRYv6NbdenKjrenW9p25PPPFE4bc5enuyO+64w8wEN2jQwNzaSteA6lbez8LoGi+NkoDeik4vQNPby2n9dL2l3kP97LPPNkcs673u9fdAlIbBbi0ESgbe8t7rXn8PWHSPplEWIPBaAOssga7740lrFogONC3vCUvlnQOcHw4Ut5wu6lfe+sCR0j7I6gek0p60prsr72dui7jf+6NHj5qLT3W2Xp+SWXLz+qQ13utunxvlvde9/h5wWySYvSfwBrOujAoBBBBAAAEEEEDgZwECL6cCAggggAACCCCAQKAFCLyBLi+DQwABBBBAAAEEECDwcg4ggAACCCCAAAIIBFqAwBvo8jI4BBBAAAEEEEAAAQIv5wACCCCAAAIIIIBAoAUIvIEuL4NDAAEEEEAAAQQQIPByDiCAAAIIIIAAAggEWoDAG+jyMjgEEEAAAQQQQAABAi/nAAIIIIAAAggggECgBQi8gS4vg0MAAQQQQAABBBAg8HIOIIAAAggggAACCARagMAb6PIyOAQQQAABBBBAAAECL+cAAggggAACCCCAQKAFCLyBLi+DQwABBBBAAAEEECDwcg4ggAACCCCAAAIIBFqAwBvo8jI4BBBAAAEEEEAAAQIv5wACCJQqsHXrVqlXr57Ur1+/2M8PHz4sO3bskBYtWvgmt337dpk6darMmzdPOnbsKCNGjLA69vr16+Xo0aNyxhlnWO2nZONo7TeinSxjZy+//LIcOnRI7rrrrqgdbtGiRdKmTRtJSUmJ2jHYMQIIIFCaAIGX8wIBBEoVOO2006RJkyYyZ84cSUxMLHzNl19+Kf369RMNdzVq1PBF74EHHpBvvvlGbr31VqlTp4506dLF6ri6v7y8PHnxxRfNftasWWMCfIMGDSK6X6ud+dx4+PDhsn//fhkzZkxEjvztt9/KwYMHTcANbeecc47ccccdMmDAgIgcg50ggAAClRUg8FZWitchEGcCGnh1GzhwoPz1r3+tssCrM7GdOnWSp59+Wn7/+99HpAoadnW/oYDbo0cPE/R09thmK7lfm3353TbSgfexxx6Tk08+WW655ZbCoWzbtk0aNmwotWvX9nt4HA8BBOJcgMAb5ycAw0egLAENvHfffbc8++yzZjlBKAyWNsM7Y8YMef3112XLli3SqlUrue222+Siiy6qFK4ukXj++edl1qxZsnPnTnOchx56SPT4X3/9tYwaNUqWLVtmwukvfvELmTx5spxyyinF9q37eO6558xstM4E/+pXv5KXXnpJNm3aZP7sv/7rv+R//ud/ZN26dZKZmSmvvvqqmX3Ufv75z3+WjIyMwv0PHjxYevXqZfoycuRIWbhwofnZVVddJX/5y1/MrPYnn3xS7n6HDh1q+vf999/Lf//3f8vixYvN//fs2VN0djkpKcn8v/a5bt26kpubK7Nnzzaz15deeqn86U9/MmMtbVN/3afatGzZUgYNGiQXXHCB/P3vf5d33nlH3nvvvcKZ959++kmuueYaufLKK81+tZYfffSRMdLZVg2j3bp1M4cpGnjVTWfTp0+fLieddJL5+b59++Tyyy+XF154wfh+9913Mn78eFm6dKn8+9//Fv3QcOedd5r6P/744zJlyhTzzUCjRo3kwgsvNOO+8cYbpX///nLJJZeYfWotnnnmGVm7dq153R//+Ee56aabpGbNmubnodf/4x//MOfAr3/9a+nbt69cffXV5ucFBQWmP1qPzZs3mw9G2sfevXtX6tzjRQggED8CBN74qTUjRSAsAQ2cGnQ1RGnISk9PN4GsZOD98MMPTejSYKWBTv9fw5cG4FCYKu/Ajz76qGhgvvnmm02Q+tvf/iYrV640AUeDj4YvDZsafHXNrb6mVq1axXYZ2sd1110n3bt3l6ysLBOKNNw9/PDDZkZRf6aBqGvXroXhToOjhi7d/+jRo+X00083s5IaNjU06b+1X/r1vB5fA7Dud+bMmeXuV2eLNYxpcNZNw6OG8ldeeUWSk5Nl2rRpUq1aNdOPd999V373u9+Z/f7444/y5JNPmsB77733Hsema5l1fBoMNcDqmmYNlmquwVmD79tvvy1nn322aauOWpf58+ebJRsaRFNTU80yA62rBv0FCxYcF3g1TOv+tQbaX9109vrMM88UDZ9aB/1AMG7cOFNjddIPFO3atTPH2Lhxo1kL3LlzZxO29efNmjUzoVg/ZOif5eTkmP5q+//3//6fqYN+mCk6dn29riXX2ukHqC+++EImTpxYeG699dZb5rga5PUYq1evNn0LjT+sE54XI4BAoAUIvIEuL4NDwLuABl4NU/pvDR4aOu67777jAq/OuDVt2lQmTZpUeLDrr7/e/LcGkvK2UIjS4BdqozOvGpw08N1+++1y4MABSUtLKzbLXHSfoX0MGzZMbrjhhmKH02Cq+9EAqv0MbUVnM0vbv15cpbOLGph1xlI3ne3WkPbBBx+YwFvRfnXWUcOyhnkN0rppYLv22mtN2DzrrLNM4NXX6T+hWc37779ffvjhBxPqSm4a7jQs64yxvl5ncHVGPFQb/bf2N3RRn4ZPXWutobrkprPxOvMaGmNRk8oE3pL7e//9980HAl0PrZt+iNDzpuiShqKBV2d2NZxrKA+NXc8hDdEafhMSEkx7Ddm6nEW3Y8eOmRCttdH96gy5fjOgH8zKmhH3/g6gJQIIBEmAwBukajIWBCIoEAq8+tW3zu7qV/06c6trX0MXrekspc4YFg2s2gW9GEz/CYWfsrqlX2XrbGXRUKiv1XXDuoxg7NixFQber776Si677DLTt9/85jelBt5Vq1YVC0QVBd433njDzLTqzGto05lmnYHVMYUCb3n71ZnqCRMmiL4mdHGf3gVB+/jEE0+YQKj90GUPoYvn9Fi6FENnXTXEldx0plhnXTUshzYN0Trrq7OcOvuqa2dXrFhhfqwznTrDrfXSTf/8s88+M7Ome/fuNcFZg6p+oAg38OqM9ccff2xmiXV5g84+65KR7OzsSgVeDazqUvSDklrpTLcu79CZfA28eoGbng+hTcNu+/bt5Z577jF3C9EPEFoXDfv63yWXu0TwLcGuEEDAYQECr8PFo+sIRFOgaODV4+jaSg01Gtb0K2mdOdStbdu2hQEu1B8NjLq+s6LA+89//tOsudR1tq1bty4cjgY7XbagIa6iGV4NXBoe586dK7/85S+PC7w667h8+fJif15R4NXQqbPC+k/16tUL2+pMpC5T0MBb0X5DgVe/ZtcPBrrpMgedsdQ1vvo1fmkXipUXeDXs7dq1y6wlLrrp3TQ0BOs629/+9rcmNGuY1NeFQrnedkxnVTUYdujQoXC5RmUDrwbzc889t3BJg/ZFP2zomlwNoLqMQWegwwm8usZXP9SEttCHF/XVD1JFZ4RLC7z6Z/ohQs8fXeqga5Nfe+018wGADQEEECgqQODlfEAAgVIFSgZencU7//zzzVpKDZeh25LpGlFd0lD0dlYaWDUg6df/5W2h5Qu6PlZDdNFQGPqavqLAq7N8ug5U1+CGZjJDx6xMMA31oejaVw1QugZVZ7Y10JfcKrNfDdk6hqLLIkKBTi+aU8twA68uVVDTossASvZNl0ToUgcNvDobH6qLhkddv6wfWHTTWWSdZS0t8Gp41WBfdPyh2VedRdbZdw2VGlb79Olj9vfUU0+ZiwGLBl6ti17IFtqKBlhdjqBLZnTGOvSBIDSzrh9iTjzxxEoF3tC+9cOELrnZs2eP6H7YEEAAAQIv5wACCFQoUDLwagMNhfqVuW6hwKuB6cEHHxS9cEzXhOrX0Rpg9YKwK664wqxH1SCiF7bpzGJpAU1nQfUqfj2mhhUNQqGgWFHg1f3pbKN+Ta/LLnSGUy9a0wvUPv300wpnYrW9XqCmX//rOHTToKjLGbQ/GgpDd4zQGV4NcZUJvHpPW103rBeL6diPHDliZqz1AjgN1DqDHW7gVXMNmDqrqjPEOkOqy0J0NjR0G7klS5aYC7/0ZxoqQxcO6nponQ3VAKx3NND66Ax8aYFX+66ztnocXbusATg0Y6+BV4+lXnqHCz2Whnudtc7Pzy8MvPoBRJdb6Ey3rsfV8RYNvFojddd11/rBQD8M6Ax00TW7Fc3w6h0i9I4Xuvxh9+7d5u4eGvT14kA2BBBAgMDLOYAAAhUKaKgpOusZCoK6TlJn4DSwhL7u14Chs3u6llI3DZ56wZaGj1Cw0TWxGqBKbhrCHnnkEbMeVAOTzh5qYNY7PugWmoEtemu0kvvQ23rpWlUNuLrpV/x6cZfOSla09EBfHwqw2v/QxW8bNmww9x8uuiwj9LPKBF7dr17kpiE6tKRCL8bTC7A0oOlWWuDVpQd6V4XS1vBqG70zg16MpjPbummw1TahOxPoTKcuPdAtdHGb/reOQz+U6Dpb3fS/9cI4XWes/SrZFx2jri3W+umm49CQHFpvrbcs0/CsZloztdEPNqEZXg3n+iFGj6fBVZdZ6L81eIdm4vWWbxqMNVDrpmFe+xW6bVvJ1+triq7h1YvxdGZZzxvd9JzR0Ku3a2NDAAEEigqwpIHzAQEEIiag4Ufv21p03avuXMOZPk429NV1aQfUoKZrUEP3ffXSKQ3HGqBDgSmcfeisrs4Sangr2n+dYdZApfs84YQTwtll4Wt1H7rMoOgT6zztqEgjvTuF7lP7VdK7vH3rV/66XKCyT8nTmuoxSnu9Hl/7UfLx00WPr+31EdUlbyVX9DW6D72tmhdfPW90XbPWzUt72zrQHgEE3BAg8LpRJ3qJAAIIIIAAAggg4FGAwOsRjmYIIIAAAggggAACbggQeN2oE71EAAEEEEAAAQQQ8ChA4PUIRzMEEEAAAQQQQAABNwQIvG7UiV4igAACCCCAAAIIeBQg8HqEoxkCCCCAAAIIIICAGwIEXjfqRC8RQAABBBBAAAEEPAoQeD3C0QwBBBBAAAEEEEDADQECrxt1opcIIIAAAggggAACHgUIvB7haIYAAggggAACCCDghgCB14060UsEEEAAAQQQQAABjwIEXo9wNEMAAQQQQAABBBBwQ4DA60ad6CUCCCCAAAIIIICARwECr0c4miGAAAIIIIAAAgi4IUDgdaNO9BIBBBBAAAEEEEDAowCB1yMczRBAAAEEEEAAAQTcECDwulEneokAAggggAACCCDgUYDA6xGOZggggAACCCCAAAJuCBB43agTvUQAAQQQQAABBBDwKEDg9QhHMwQQQAABBBBAAAE3BAi8btSJXiKAAAIIIIAAAgh4FCDweoSjGQIIIIAAAggggIAbAgReN+pELxFAAAEEEEAAAQQ8ChB4PcLRDAEEEEAAAQQQQMANAQKvG3WilwgggAACCCCAAAIeBQi8HuFohgACCCCAAAIIIOCGAIHXjTrRSwQQQAABBBBAAAGPAgRej3A0QwABBBBAAAEEEHBDgMDrRp3oJQIIIIAAAggggIBHAQKvRziaIYAAAggggAACCLgh8P8BdFvWUEM/hgwAAAAASUVORK5CYII=" 250 | }, 251 | "metadata": {}, 252 | "output_type": "display_data" 253 | } 254 | ], 255 | "source": [ 256 | "fig = em.criterion_plot(res)\n", 257 | "fig.show(renderer=\"png\")" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 12, 263 | "id": "f64f1f6e", 264 | "metadata": {}, 265 | "outputs": [ 266 | { 267 | "data": { 268 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAArwAAAH0CAYAAADfWf7fAAAgAElEQVR4XuydB5gUVfb2D3lAZkAEyYqgRFFXAUXQNQCm3QUUEAykXV0liYoSVMSV6OoqKLDqp0RFRQQEJGMA/mQXUWDAQE4i6MwQhjjf897mDtU9Haq7uqurat77PDwD0zf+zh3mrVPnnlsgJycnR1hIgARIgARIgARIgARIwKMEClDwetSyXBYJkAAJkAAJkAAJkIAiQMHLjUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDByz1AAiRAAiRAAiRAAiTgaQIUvJ42LxdHAiRAAiRAAiRAAiRAwcs9QAIkQAIkQAIkQAIk4GkCFLyeNi8XRwIkQAIkQAIkQAIkQMHLPUACJEACJEACJEACJOBpAhS8njYvF0cCJEACJEACJEACJEDBa2EPXHHFFfLjjz9a6IFNSYAESIAESIAESIAEEk2AgtcCYQpeC/DYlARIgARIgARIgARsIkDBawE0Ba8FeGxKAiRAAiRAAiRAAjYRoOC1AJqC1wI8NiUBEiABEiABEiABmwhQ8AYBfeLECSlatKgUKFAgrBkoeG3apRyGBEiABEiABEiABCwQoOANgDd37lzp1auXzJ8/X6pXr07Ba2FzsSkJkAAJkAAJkAAJOIEABa/BCr/++qu0b99eDh8+LJ999hkFrxN2KOdAAiRAAiRAAiRAAhYJUPAaAP7zn/+Uu+66S1577TWZMGECBa/FzcXmJEACJEACJEACJOAEAhS856wwbdo0mTdvnrz77rty/fXXy5QpUyh4nbBDOQcSIAESIAESIAESsEiAgldE9uzZI/fee6/MnDlTKlSoEFTwLliwIA/q7t278+IJixuQzUmABEiABEiABEgg0QTyveA9e/asdOrUSZo0aSJt2rRRvO+55x4ZM2aM1K1bV4oXL66+98orr+SxBbzBvGkt0VuU/ZMACZAACZAACZCANQL5XvAeOXJE/vSnPwWlOGTIEGnXrl1IwkxLZm3zsTUJkAAJkAAJkAAJ2EEg3wveYJAZw2vH1uMYJEACJEACJEACJGAPAQreIJwheD/++GOpVq1aWCvQw2vPJuUoJEACJEACJEACJGCFAAWvBXoUvBbgsSkJkAAJkAAJkAAJ2ESAgtcCaApeC/DYlARIgARIgARIgARsIkDBawE0Ba8FeGxKAiRAAiRAAiRAAjYRoOC1AJqC1wI8NiUBEiABEiABEiABmwhQ8FoATcFrAR6bkgAJkAAJkAAJkIBNBCh4LYCm4LUAj01JgARIgARIgARIwCYCFLwWQFPwWoDHpiRAAiRAAiRAAiRgEwEKXgugKXgtwGNTEiABEiABEiABErCJAAWvBdAUvBbgsSkJkAAJkAAJkAAJ2ESAgtcCaApeC/DYlARIgARIgARIgARsIkDBawE0Ba8FeGxKAiRAAiRAAiRAAjYRoOC1AJqC1wI8NiUBEiABEiABEiABmwhQ8FoATcFrAR6bkgAJkAAJkAAJkIBNBCh4LYCm4LUAj01JgARIgARIgARIwCYCFLwWQFPwWoDHpiRAAiRAAiSQYAIffvihjBs3Tg4ePCi33XabPP3001K5cuUEj8runUiAgteCVSh4LcBjUxIgARIgAc8ReGPRj0lZU+9mVwQdd/ny5VK4cGGpVq2avPzyy1KnTh3p3r17UubIQZNLgILXAn8KXgvw2JQESIAESMBzBKr1m5OUNW0ffk/YcTMyMuTzzz+XxYsXy/jx45MyRw6aXAIUvBb4U/BagMemJEACJEACniPgNA/vm2++KV988YXiXKhQISldurRMnjzZc9y5oMgEKHgjMwpZg4LXAjw2JQESIAESIIEEEvj222+la9eusmTJEilTpowSuvPmzaPgTSBzJ3dNwWvBOhS8FuCxKQmQAAmQAAkkkMDMmTPl/fffl88++0x+++03efTRRyU1NZWCN4HMndw1Ba8F61DwWoDHpiRAAiRAAiSQQAJHjhyRRx55RDZv3qxG+fvf/y6rV6+WSZMmJXBUdu1UAhS8FixDwWsBHpuSAAmQAAmQgA0EsrKy5IILLpCCBQvaMBqHcCoBCl4LlqHgtQCPTUmABEiABEiABEjAJgIUvBZAU/BagMemJEACJEACJEACJGATAQpeC6ApeC3AY1MSIAESIAESIAESsIkABa8F0BS8FuCxKQmQAAmQAAmQAAnYRICC1wJoCl4L8NiUBEiABEiABEiABGwiQMFrATQFrwV4bEoCJEACJEACJEACNhGg4LUAmoLXAjw2JQESIAESIAESIAGbCFDwWgBNwWsBHpuSAAmQAAmQAAmQgE0EKHgtgKbgtQCPTUmABEiABEgggQROnz4t7733nnTu3FmKFSsW1Uhom52dLSVLloyqHSs7lwAFrwXbUPBagMemJEACJEACJJBAAidOnJArr7xS1q1bJ2lpaaZHeuutt2TkyJHqdraGDRvK66+/TuFrmp5zK1LwWrANBa8FeGxKAiRAAiRAAgkkEIvg3bBhgzzyyCMyc+ZMKVeunDz99NNStWpV9ZXF3QQoeC3Yj4LXAjw2JQESIAESIIEEEtCC98knn5QpU6bIyZMnVXjDY489JgUKFAg6Mjy7GRkZMnDgQPU5vMO9e/eWpUuXJnCm7NoOAhS8FihT8FqAx6YkQAIkQALeI/DV8OSs6ZZ+ecbVgvfmm29WHtrdu3fLs88+Kx9++KHUrVs36DwHDBggtWvXlo4dO6rPDxw4IE2bNpX09HQpVKhQctbGUeNCgILXAkYKXgvw2JQESIAESMB7BAaVSs6aBmWEFLyff/651KlTR30OQXv55ZdL165dg86zR48eAoHcrl079XlmZqZcd9118r///Y9xvMmxbNxGpeC1gJKC1wI8NiUBEiABEvAeAQd6eFesWCFly5ZVrCdOnCgrV66UMWPGBGX//PPPS40aNaRLly7q8/3798tNN90kW7ZskYIFC3rPXvloRRS8FoxNwWsBHpuSAAmQAAmQQAIJ6JCG+fPnS/Xq1dVIL730khw7dkxGjBgRdOSxY8fKvn375F//+pf6HOL4mWeeCRnD+8aiHxO4Av+ueze7wraxvDgQBa8Fq1LwWoDHpiRAAiRAAiSQQAJa8D7wwAPSp08fFY8Lz+1TTz0lrVu3DjryTz/9JG3atJHZs2dLhQoVpFevXnLZZZcp0RusVOs3J4Er8O96+/B7bBvLiwNR8FqwKgWvBXhsSgIkQAIkQAIJJADBi3jcO++8Ux1UQ7n33ntl8ODBUqRIkZAjI+zhP//5j/r82muvVXl4S5UKHpv8+sKtCVyBf9dPNq9p21heHIiC12DV48ePq1OYRYsWNWVrCl5TmFiJBEiABEiABJJKALem5eTkSPHixdU88Pv+zJkzfnMqXLiwpKSkqO/xprWkmishg1PwiqjYnKFDh6q4HRQ8Eb788sshn+i0JSh4E7In2SkJkAAJkICDCWzcmykvz94kHz16g4NnGX5q3bp1k23btvlVaty4cW7+XdcujBMPSYCCV0TWr18vp06dUlcI4qmve/fugo2P21bCFQpe/mSRAAmQAAnkNwIrfjkkHd5ZKYwpzW+Wd/d6KXiD2K9v377Ku4t8fRS87t7gnD0JkAAJkEB8CVDwxpcne7OHAAXvOc6HDx+Wn3/+WaUgmTx5sowfPz43UTWq7Nq1K49FbrvtNvnxR/tSktizJTgKCZAACZAACYQmQMHL3eFGAhS856yGFCS4Q3v79u3y4IMPCuJ7Lr744lyb6iTURiMvW7aMgteNu55zJgESIAESiJkABW/M6NgwiQQoeAPgZ2RkqMTUOKE5atSosKZhDG8Sdy6HJgESIAESSAoBCt6kYOegFglQ8AYB+PXXX8sTTzyhDrOFKxS8Fncfm5MACZAACbiOAAWv60zGCYsIBa+IfPvtt1K7dm0pUaKEIJZ30KBBcuTIEXn//fcpePljQgIkQAIkQAIGAhS83A5uJEDBKyJDhgxRh9TKlCmjLp249dZb5dFHH5UqVapQ8LpxV3POJEACJEACCSPgFsGL0MT33ntPOnfuLMWKFUsYD3bsDgIUvOfsdPLkSeXVheg1WxjSYJYU65EACZAACXiFgFsEL64WvvLKK2XdunWSlpZmGj/y8uM64XfffVe2bNkiBQsWNN2WFZ1LgILXgm0oeC3AY1MSIAESIAFXEvCy4M3KypJ//OMfUrp0aVmyZAkFryt3aPBJU/BaMCYFrwV4bEoCJEACJOA6Art+Py4LN+2Xf83a5Pib1rSH98knn5QpU6YI3uQivOGxxx6TAgUKBGV/9uxZmT9/vtx4443SoEEDCl7X7dDQE6bgtWBMCl4L8NiUBEiABEjAdQReX7hVRi72XbgU7Grhsd+NTcqaHr/68TzjasF78803y9NPPy27d++WZ599Vj788EOpW7du2HkiRSkFb1JMmbBBKXgtoKXgtQCPTUmABEiABFxHIJLgrT+hflLW9H2n70MK3s8//zz35tQBAwbI5ZdfLl27dqXgTYqlkjcoBa8F9hS8FuCxKQmQAAmQgOsIRBK8TvTwrlixQsqWLatYT5w4UVauXCljxoyh4HXd7rM2YQpeC/woeC3AY1MSIAESIAHXEbj/7RWyatthNe9gIQ1OWpAOaUBMbvXq1dXUcJPqsWPHZMSIERS8TjKWDXOh4LUAmYLXAjw2JQESIAEScA0BHFabtm63fLpul+z+/birBO8DDzwgffr0kQMHDkiXLl3kqaeektatW1Pwumb3xWeiFLwWOFLwWoDHpo4gkHn8lGzalyWpKYWlXiXzeSodMXlOggRIwDYCOhWZcUA3eHhxYO3OO+9UB9VQ7r33Xhk8eLAUKVKEgte23eOMgSh4LdiBgtcCPA81nbput/J83HddFWl7Xfjb+Zy2bP1L7PrLysjH/2zstOlxPlESgBduz+/HpfKFxaXqhcXFzXszyqWzeoIJuFHwGpFkZ2dLTk6OFC9eXH37+PHjcubMGT9qhQsXlpSUlASTZPfJIkDBa4E8Ba8FeB5qqg9xPHH7FfJk85quWhkFr6vMFXGygXvR+O82DaqqBzOIYbc9mEVcOCsklAAenFb9ckg+Xbfbbxyne3jDQenWrZts27bNr0rjxo1l4MCBCWXJzpNHgILXAnsKXgvwPNTUDYJXx9+lFS8sdSuWkjcWbVUWeKLZFdLhnZVCD683NmTgXnz6k+9k2re7BQ9jN9S4iLb2hpnjtopIIU34fwNvCowH1YyDu1nwxg0iO3INAQpeC6ai4LUAz0NN3SB4g72OVIL39itUEnkKXm9sSKPAxdsGLVRg57qV0uSfk9bR1t4wdVxWof/vCvXzj8/xwISiD6rpgetUTJW5T9wcl3mwExKwgwAFrwXKFLwW4Hmoqf6lUeXC4vJEs5qOfF0cSvDiFx1SDOHrq+2u8Yv/9JCJErYUeMjSioc//JKwwQM6bv/OStm0N0Mys0+rB5lAwYvqfLixyxrOH0fHd6/85ZCabNcml0mLehWkbsVU9W8cZkVGhsAwBnx237VVpHfzmsr7y0ICbiFAwWvBUhS8FuB5qKkxEXub66rIDdUvEnx1UgkleCHS4blJSymshBv+jrkP/Etdxwi5ZHDEq1yI2ZGLtqqHmGemfqfCP8Yt3y5gBj7vL98uGcdPSlb2aXm17dWCNqXOcUzGnKv1m5M7bKDg7dKkmqSlFPETvCt/8eVSvaF6mWRMl2MmicDGvZlqX0Po4uEosED4ZmafCip0dV2GMiTJeBzWEgEKXgv4KHgtwPNQU/0a2bikKY/eII2rX+SYVb40a6MSa2bLwL/WlSoXlpD3l21TQs9JazG7hmjrQbAu3LRfMo+fzo1xDtUHHhCCiQV8v26lUkoUQ2DCY6YzJkQ7n2jrhxO88ODjQQweXj1H7dmb0+smpqSLFrYL6+P/gIWbDuQJTYhlKRS8sVBjm2QToOC1YAEKXgvwPNQ02IEOpwneUIdOzJoBYgll9+/HZFnf2wQe44UbD0iXppdF9VpTp80yO26wepv2ZShRaqVosaf70KEAkfpEvmJ4dKMp2ouOr1MebRwVL7PjBHrwAz28Rg9+YJ9O26tm18x6/gTwRgIFb2qwH1b9cliur15GCVxk5wi1x2PZ0xS83H1uJEDBa8FqFLwW4HmoqdMFr9HzFw/sdSumyaZ9mfHoypF9aG8oshos2LhfzRHebgjWO+pVULltIS5wCAwecISAwAMOMf+vWRtzr10NtjijZxjtEC4RSxzke8u2qVhxHT8cKHgx1+Z1yyuhE8wTbZxbi7rlpWvT6ip20ynxyI7cGA6fFEKr4MGFDY3iVj9wGaePh5xHJ64V/CwjXd3uw8eU9z9YMQpitEPJD298HG5uTi8GAhS8MUDTTSh4LcDzSFNj/K5xSYiD693sCkcIiHgIXhxSgXcXB9yMJVrvUOXSxZVwtFIgNBEuYKVAzAaWlT8jpvGUvPjXela6VoIYwrbP1O+UoAhkFig8ohUPENt3j1qqRPe/216tRHiwGG19INHsYhDG0uba8yIa7dAvBY5ZgqHr6XjpcD3hzQX2NUJg8BUPQhv3ZaqHLdgabxZUbO3aXSrFHB62uja9THYdPqYOmy7YdCBk98io8Grba1R7CN3vB92Rp+4jE9cqwRxYjPvIbZ7d06dPy3vvvSedO3eWYsWKRWXIEydOSIECBaRo0aJRtWNl5xKg4LVgGwpeC/A80BRiYOSiH9Xhj2BFH1zD5xCMybqUworgDUxXhAMv+IWJE9oIc4jFO+kB00e1BJ3JAUIYHmBjSATEP0RrNKLXKG5hg48evSEugheLglBHgVdYe6Pxtc11VdX3tQc41J6PCkySKpsNX0nS9CwNi59XvH3p3aym/L3pZaqv+Rv3q/2lbRcuswg+g8hGbm4U/L8FUe3Wa8chWq+88kpZt26dpKWZuzo9PT1dXT38ww8/KAYtWrSQl156KfeGNksGYuOkEqDgtYCfgtcCPA80jeYgGIQNYl+NJfAXDzw0eL0MbxA8MXidjjoQH74E8Zmy4cUWQb3GmMvmfVnKo4hfbLjmWItRK4IXp/sDPZ5OSsXltm2EUASkeoKtdIn2hr5Aby68bvHw8LqNpZvmCyFqpuBtAN6C7PnjuF91eGjh9cX/AfqBSYtb/Bs/pzgseUfd8n4PJmbGDFYH+6nPJ9+pcJ1kPajHOndju1gE73fffSe7d++Wu+++W44ePSp///vfpV27dnLffffFY0rsI4kEKHgtwKfgtQDPA01jOQiGXyCb9maq8AAIU32SHyIoMLF7MEQQxHXOvSaHqEUcJwSoysFqiKtFrJ1+FXrPqKVR0XbzK8yoFprEysZQGO2lNTudeApeHZKC/YL9h7AOPJzBg4v9g30aLIQkWEiI2fknu169cw+FyZ5HsPH1w2Soh0p8f/6mA0rY4v8PPOhi/yCsJREFb3Tc6t0FDy14n3zySZkyZYqcPHlShTc89thjKlzBTBk1apRkZWXJc889Z6Y66ziYAAWvBeNQ8FqA54GmwQRvtHGToTDAy9OiXnklQiBAkJVA33ikXzVrz2C4SyUQeqBfT5pFjlfsENIsiSMQaLNoUoMFtv3o0cYqHzBuUTOWcHvRKHSjCadIHBH27BUCv701OilLKduje55xteC9+eab5emnn1ae22effVY+/PBDqVu3bsR55uTkSKtWraRHjx7SvHnziPVZwdkEKHgt2IeC1wI8FzWFlwOvDQNPsQcTvPqq3sDl4ZUkRCvEKzxmeP34xsKtkla8sPLQwKOGr/hjjLcz9hOY71cLXhyOCnYbEtoaswJA5KBESqvFNFWJ35zYU8aMDmavdg4VnhJs3wU7UIiYzDYNqqi9h9fntHXibZ3fRthcu05SllwnfXNIwfv5559LnTq+eQ0YMEAuv/xy6dq1a8R5vv3227J06VIZN26cFCli7aBsxMFYIeEEKHgtIKbgtQDPRU21sA0UB8EEL7yjuJULBSIGrx116p/AA17G15Zm4mIDPXsQxxDQocRuIGIdRxguawDqDPxrPVe/xnTR1hItYMMJXmNO1VDe+lAPWoEs3n74uoS9/nYTd841cQSc6OFdsWKFlC1bVi164sSJsnLlShkzZkxYCHPmzJHhw4fLjBkz5KKLnHOJUOIs5/2eKXgt2JiC1wI8BzbV4iMw9Y4WthCzVS8skXtzVqj8u5+u3a2EaLwPe8Ar2P6dFUE9tAiBgADGYbdgqYWAO1RqMaMp3JZ2yIHbKKop3TXym9zDhnjQCHYrm473Rfx3qIcbM4I32sNxUS2ElUnAgQR0SMP8+fOlevXqaobIuHDs2DEZMWJEyBl//fXXKvQB4rhWrVoOXBmnFAsBCt5YqJ1rQ8FrAZ4Dm4YSvE1HLFGxtDomUguHqwbNz5PU345XxMGuMtZzwuUHyNMZLIk85qZfZQc7CQ6TUPDauzEDM30YRanO3fresl9CPsTo2VLw2ms3juYOAlrwPvDAA9KnTx85cOCAdOnSRZ566ilp3bp10EWsWbNGHnnkEZW/t1698zm5U1JS3LFozjIkAQpeC5uDgtcCPAc2DSZ4jWEEiMNFOiktSoLFU9oheIMdUgscN3BuL/ylrjqIhhPeuFWpSpkSuaEX9PAmdzMG2kofGtTfD/VwYpx1OO+vURTH+61DcslxdBIITwCCFwfW7rzzTnVQDeXee+9VeXZDxeRC7H711Vd5Ot64cSMvoXD5hqPgtWBACl4L8BzYNJLgNQqHxiHYTlkAACAASURBVDXKqvCCwLK0720Jv4zBKHghhnAI7t2ODfymEiiKAwVxqMwO9PDavzHrD5rvF6aC0BS8TQh11WusM2RIQ6zk2M4LBLKzswVZF4oX9930ePz4cTlz5ozf0goXLiz05HrB2sHXQMFrwbYUvBbgOayp8dWyMUUUbsfSh9CMgjeUGLFDMOKAGy4w0JkdgnntKHgdtsHCTCdYLDhiwM3kZY5mlRS80dBiXa8T6Natm2zbts1vmY0bN5aBAwd6fen5dn0UvBZMT8FrAZ7DmhpFh9EbarwgQE9ZpxELtgQ7BK8ZdIjl7fPJelUV88U1o/pqUXyPHl4zFO2pE8sFJtHme0b9Ng2qMr+yPSblKCRAAg4kQMFrwSgUvBbgOaypPpiGaUUSvMGmbkfsbjyRUfDGk6a1voIdQgzXI2LJuzatHjQGO1Q7pzyIWSPF1iRAAiQQOwEK3tjZCQWvBXgOa2o8OGQUr2a9b3bE7sYTmRa8+kY3XISBw2w81BRPyub6CvYWIVxLeGujuUHP7KUW5mbLWiRAAiTgTgIUvBbsRsFrAZ6DmiIm9qqXFuTOSOfb3f37MXlj0VZTsZRu86BpwUsxlPyNaEbw4sEEcb24NKR53fLStellpq+MdtveTL5FOAMSIAEvEqDgtWBVCl4L8BzUNPD1Pm5G27Qv0/QM3XgYCCJ/475MSUspwlvVTFs6MRXnb9wv/5y0LmzneDBBLDYOS2K/3VDjIgrexJiDvZIACXiUAAWvBcNS8FqA56CmoeJZzUzRbbG7ZtbEOvYSwMMH9mCg6IWwxWFD/WCyYOP+XMGL7yNTB25e2/PH8bATpofXXntyNBIgAWcSoOC1YBcKXgvwHNQUwuHl2ZtimhHFREzY2CgIAWMcOUIYlve7za+WDn0wvlEwE2POPcrtRgIkQAIiFLwWdkE8BC/SR8HDU69SmoWZsKkVAmZiKOHJhZ0ys0/7nY6nmLBCnm2NBLTgxV4LFmqCnNC4NtqYXiyS4GWMNvcYCZAACfgIUPBa2AlWBS8PDlmAH6emeOB4edZGWbDpgIS6wjVQNAS7kS1O02E3+ZhALPsqnODV6ctwpTQLCSSawLFVq2VHp04Rh7l04kQpmFpSUurUiViXFUggngQoeA00ce92gQIFTN+XHS/Bi9PXy/r6v76Mp5HZV2gCRu9uqGT+FLzcQXYQwAMwSuPqF5keLpzgZXy5aYysGAcCZgUvhipWu7ZUGDBAUmrXkoJpfLsZB/zswgQBCl4RSU9Pl8GDB8sPP/ygkLVo0UJeeuml3Du3Q3G0KniN19by1biJ3ZqAKkbBi3RkbyzcmucQUKDgjUWYJGDq7JIEhIKXm8ApBKIRvHrOl06YICWub+SUJXAeHidAwSsi3333nezevVvuvvtuOXr0qPz973+Xdu3ayX333RfW/FYFr/GXFQVvcn7SHpm4VhZuOqAGhw2MNsEr4RZ1K0iLehUYY50c83DUCARemrVRxi3fHrQWPbzcPnYSiEXwluvRQ4pUqiSn9u6VUq1bSaHUVDVlen3ttFz+GctWwXvo0CHZsWOHlC1bVi655BLHUh41apRkZWXJc889R8HrWCvFZ2KBDx3w+K785ZDKeYrUT2nFi8RnIPZCAgkgEO7AJQVvAoCzy5AEYhG8xs4qDR0qhyZOlIs6dpRS97YmaRKIOwHbBO/QoUNl3LhxagE9e/aUXr16yfz58+XVV1+VhQsXxn1hsXaYk5MjrVq1kh49ekjz5s0peGMF6ZJ2WvC+8Je6SuCykICbCODQ5aMT18jmfVl5pk3B6yZLun+uVgWvJlCiUSPJ3rw591BbkcqV5dSePepP6u23S6F8HPNbtkd392+UJK7AFsG7fv166dy5s/KYfvvtt1KxYkUleLdv365E5TfffKO+54Ty9ttvy9KlS5U4L1LkvHcvWHjDhg0b5Mcff4x52gxpiBld3Brqk/EbXmxBb27cqLIjOwmECmug4LXTChwrXoKXJEMTqJO+mXgsELBF8Pbt21dKliwpL7zwgowcOVJlQoDgzc7Olvr168vs2bOlVq1aFpYRn6Zz5syR4cOHy4wZM+Sii/xPSmdkZOQZpEGDBhS88UGftF5iSQWVtMlyYBIIQsAY1nDNVWukVPHCck3Jdipfb9ULi5MZCdhCIB6Ct2DJ8+nKUurUFnh3//hsuvqKNGZnszKlYGr+zepQrmcPW2zp1UFsEbwIZ9i/f78gNtYoeBHK0K1bN1m7dq2UKlUqqYy//vprefbZZ2XixImmxbfVQ2vGm5V4aM1+8zNLhv3MOWL8CWjBizzSmRWfUAN83+n7+A/EHkkgDIFfWrWWE+nplhiV799PypjI5WtpEDbOtwRsEbzr1q2T9u3bS5cuXZTwLVq0qNSsWVMmTZqkPLxjxoxJqgHWrFkjjzzyiLz33ntSr1693LmkpKSEnRcFb1LNZnlwHVLC26gso2QHSSSgBS/28aaURyl4k2iL/Dz05trRXySBV/TaM1ysVi2pOma08uaykEAiCNgieDHxefPmyZAhQ5Tg1aVly5YqrvfCCy9MxNpM9wmx+9VXX+Wpv3HjxrCXUFDwmkbsyIpXDZqvrgqm4HWkeTgpkwQoeE2CYrWEEjAreHUaMkwGgvdsZqZkb06XgmmpvH0toRZi57YJXqBGBoSDBw9KZmamVKpUSUqUKOFqC1Dwutp8okNK7ru2irzW7mp3L4azz7cEtOC9usZR+aXoy/Tw5tudkNyFmxW8JRo2lGNr1qjJ8hBWcm2W30a3VfB6DW48BW/vZjXlvuuq8JBJgjdJ5vFTKhuDMX56Tq+beLFEgrmz+8QR0IK3fo3fZHvRVyl4E4eaPYchEI3gReoxFB7C4payk4Atgnfy5Mkyc+bMkOtCCjBkcXBbsSJ4cT1th3dW+i2ZaYQSvwPAfeSiH9XlErrwwGDiuXOExBGg4E0cW/ZsjgBy5P50ezNTleHhvXTSRFN1WYkE4knAFsG7atUqQc5aXU6dOqWu8p06dao89NBDMmDAAL+ct/FcYCL7ouBNJN3E9P3esm3y8uxNqnNcHfxq22vo3U0MavZqEwEKXptAc5iQBKJJSUbBy42ULAK2CN5Qi5s7d64MGjRIVqxYIQULFkwWg5jH9aLghQf0nxPXqsNcXvM4I5zhX7M3yafrdiub87BazFufDR1EQKfXMxPSsGb/Gll7YK00KN9AGlZo6KBVcCrJJjB89XDZ+vtWebbhs1K7TO2opkPBGxUuVk4SgaQK3uPHj8tVV10lS5YskapVqyYJQezDxlvwpqUUljlP3JzUOF5jEvsbql8kuHK3XqU00bGv+IqCOFg3lY17M2XBxv0ycrHvZrwnbr9C6lZKkzvqVXDTMjhXEshDAD+TG/dlyu7jP8i/1voS04fKwztm/RgZ+91Yefzqx6XbNd1IkwRyCXSZ10U9DL1/x/thH4a6zu+q2qCeLpEEL7y65Xr0kB2dOgk9vNx0ySKQVMGLtF+tWrUSXD18wQUXJItBzOPGW/BiIsn2qj4yca0s3HTAj0nXJpepmNcW9Sqor7t/PyYD/1pP0lJCi97M7FNKJF9f/aKkCni9EOM1zk7gHPOmY0MSCEEA3lstRiB4tbj9W42/yZCmQ1QrCl5un1AEtOCN9DBUf0L9PA9VZgRv1dFvMf0Yt19SCdgieHFgDTeZGcvhw4dl+fLl0rx586RfPBGrBRIheOtWTJMvnrjJ1JS0iHv74esseyp3/X5c9vx+XPpMXS+7fz8uqSmFJSv7tKl5RKqENcFbjCwU8BYHFow97VyYQe9mV+R+jO9DNAdrE2lMfA6vLq5WXbDpgLyxaKtaFwq8u082r2mmC9YhAdcQCBS8vZb0ki93fanCF8bdOU7wyhr/3ntkLz28rrGqPRPF3vhg8wd+gyHkBQ9LrS5v5fd9LXiNnuBIGRoqDR0qpe5tbc9iOAoJhCBgi+DFpRP/93//5zcF3GL25z//WRo2bBj2cgcnWy4RghfrNZMmC0Kw6YglKtYWAq5xjbICr2qLuuUVMojFTXsz/IQwvgcBiK8LN+2XTXszlZcW4QkQhMaytO9tqj3q6DAAfI6DXpv3ZamqiIFF8XlzT0uVC4vnMdeqbYf9vtfmuirSpcllfiLWmLHio0cby9S1u2Tat7sFQhne5Lc7NpDG1S8SHDiDlxmlVErh3HUGE8SI0/3XrI2Kj7EgROPvTS9z8rbi3EggJgJGwQuRm3UqS7Yc3qL6gse3zaw2uf+O5MWLaQJs5EoCd0y7Qz0EBSv6YQmfaQ+wrtfy8pbSskZLFf4QSfAy364rt4bnJm2L4PUctXMLSpTghdD76NEbwsbJGmNtEfurhR3aQoDCmwkBqgUtxKsugSJQfx9itsqFJeS1tlfnGVvHCeJz/B1jQISaKfM37lehEOOWb1fVMUeERNxQvYxfPtxwfWEt2kMbWA/rr1upVO63sVa9RniqEZKBthDPH/+zsZkpsw4JuI5AoOBFPKYuU/86VdrOapv7b4gViN7KJXmNq+sMHacJpx9Oly2/b5Hnlz0fssdwgheN9OfhBC/FbpwMxm4sE0iY4P35558lI+O8yAo3UxxcK1zY57FzU0mU4AUD3P7VtoHvIB+EYWAJFmtrlh1EIEIMIADhra1curgM/GtdJWATeRgNc4bHeM8fvtCCcCIW4hsiFYJd1ze7Pl2PN6hFS4z13Upgz5E96jDazJ98+c4hRIyCN/DfqGOM7XXrujnv6AmMWDNC7ZOsk743dboE2yO1ytSSfo36yeZDm+WVNa/kGQxt3qk/OGwOXgre6G3EFokhkDDB+9hjj8nixYtNzXrt2rVSqtR5D52pRg6olEjBa1wewgAg/p5odoUSpSqUQXlZz7+u1yJWX6jwTscG6pV+i7oVVDYCfB9eUHhXjSEA8L7amakA877/nRW5YRHGdeoQCYQtGMMOEPKA9cNz2+a6qkqoY+3oS3mxDRkjIKh1RgmGLjjgh4RTsIWA0bsbTPCmFk0NKnAQ28uSfwgs3rlYen/ZO8+CSxYpqbJ2BBO1qFzzwpoqZVlggeB95cfr5LfRo0NCpODNP/vL6StNmOA9efKknDlzxtT6ixfPG/tpqmGSK1kRvMYLEMwuA8IXHs/3l2/L00THpkLszd90QNpeVyVX+Jnt3656ENn/nLQudzh4mBEi8f2gO+yaAschAU8RiCR4gy22UslKuWENkzdPllur3prngFLmyUxJK5r3oKmn4Hl8MdqGeAOALB1awOIrRCxE6+Cmg1Ucr87yYRZJKMFbrFYtObHFFz9OwWuWJuslmkDCBG+iJ+6E/q0IXmMMbqxrcevFCToeWK/bbCxwrJzYjgS8TiAWwRvIBK+vIW7h7aucWlmW7FyiYnzhHcZlBCiM+U3+TkLsLS6GgLcWIvXhug+rSeH7KPhM15nx0wx5YfkLeSa9vMPyPA8yEMY45Ah7T9o0ST7/+fOIi0Xdmb89kMfDq/PuooMS1zeK2A8rkIAdBGwRvLhKeMGCBYKb1fbuzXsadOLEiVKyZEk71hvXMRIteOG11dfgBpt4PNKRxRUIOyMBEkgKAQgU4+voYPGYSZkYBw1KAEIx1G1mEKuIr0UdY8H3kBEBn+OzwMwK+B689ij4LFgdhCYgJtfMLXs6/VgkE36w9TYpMm2BXzVeLhGJGj9PBgFbBO/SpUula9eu0q5dO9m5c6eK18VVwhDATzzxhCDeN78dWovk4YX39tV21+Re2oBUYp+u3aX2CGJWcegs1vy0ydhoHJMESCBxBPSFEnqEUIIX3tsjp44kbiLs2XEEHqzzoPICQyjffsntpufX4tMWsu/ovoj1p3xxuRT6zuddLlKpkpzau5e3qUWkxgrJIGCL4B06dKha24ABA+TNN99U3twuXbrIl19+Kf369VM5egsVKpSM9VsaM5Ee3u3D77E0NzYmARLIPwQCBW+olb9x6xvKu9fm8zZ+YgYC+bZLblNhDA/VfUhGrB6hwhdQF/9WD9qM5Y3bhtJe3MAO8X3YAeyNIQZGby3qwJv/cpOXlRcXnmLYH2Et+Dtshs9hU3w+6rZRMc3b7J56b2YlSd20U41Rtnt3Fd5AD29MyNkowQRsEbwDBw6U1NRUeeaZZ2TKlCmybt06efXVVwUH2+rVqyeLFi2SSy+9NMFLjX/3iRC8OMCFDAv03sbfXuyRBLxKwKw40bdjQRwhlRnEUbC4XIgtilzn7hbYL1xYAkJcdGxvrKvA/sC+ihTLO2jyGam7K0dwUK1czx6yu0dPCt5YobNdQgnYIng/+ugjmTBhggph+Pbbb1V4w7vvvivZ2dnq7/geBLHbSiIEL6+9ddsu4HxJIPkEtOB98QNfZpyXHgz+xsx4HWzyZ80ZOJ2AmQcpLXjp1XW6NTk/WwQvLqBYuXKl3HHHHZKTk6PCGZYvX67o33///TJ48GBXWiLeghfe3VfbXW36BjNXQuOkSYAE4k5AC5NPhvlyc7frn/ciH4rduGP3fIdGwVvxgooqDCYwDlzvOQpez28H1y/QFsF75MgRvywMCGVYv369lC1bVqpXr+5aiPEWvG5NM+ZaA3LiJOARAhAmcxeMkVfe93l4KXg9YtgkL0OnNYPYXdDGl4kh0OurBe+/RzZW6etCZZ9I8lI4PAmILYL33//+t3z99dfSoUMHueuuu6RMmbxX5brRFhS8brQa50wC3iMAEfLN52PkxQ8peL1n3eStSOd3xgE4fSsfvoc/uMgCxfhWgW8RkmcrjhyZgC2C9+eff5aPP/5YPvnkEzl69KgSvW3atJEbb7zRlenINFYK3sgbjDVIgAQSTwA5U+vtyAkreL/v9H3iJ8IRPEUAGSGQsQOXkiB/r7G89lBdufRXkXo7c3LfKujLSyB8WUjAaQRsEbx60biAAinIZs6cKbNmzVKe3vbt20v37t2laNGiTmMTcT5WBO8jE9fKwk0HcseoUzFVXm17DbMzRKTOCiRAAoEEKHi5J+JJIOOz6fLHjBlSulUrKXVv66BdT2teV2VnQDmYJtK9+/m4cXp642kN9hUvArYKXuOk165dK/3795ft27cL/o7LKNxWrAje+99eIau2Hc5dMuN33WZ9zpcEnEMglODVB40wU3p4nWMvp8/k4JtvqXy6yKuLVGPBilHwbqpaQAY9dD4zCAWv0y2cP+dnq+DFtcJz5sxRoQ0Quk2aNFGhDQhxyG8XT1Dw5s8fOK6aBBJBIJjgNcZW4mT9igdWJGJo9ulBAlrwlun4sKQ2ay4H33pLrfLSiRNyV0vB60HDe3xJtgjeVatWyVtvvaVSk1WoUEEdXmvZsqVUrlzZ1Xjj6eFl/l1XbwVOngSSSiBQ8A56sJAMOpeTFxP7qHtdeanntKTOkYM7m8CxVavl2Jo1ciYzU7IWL5ZTe/ZIwdRUOZuVlTvxOumbKXidbUbOLgwBWwQvrhPevXu33HfffdKgQQMpWLCgJ4wST8E75dEbmH/XE7uCiyAB+wkECt7AGVDw2m8Tt42ovbrh5h1K8K65vID8uy1DGtxm8/w2X1sEr1ehUvB61bJcFwm4iwAFr7vs5cTZ7urWXY4sWRJ2alrwBorjqU0KytSbzzuydAyvTl12a9VbmZ/XiUbPZ3Oi4LVgcApeC/DYlARIIC4EMk9mSpMpTfzSkgV2fOmECVLi+kZxGY+deI/A2cxM2d6xk5xIT4+r4MWDGIoxj6/36HFFbiFAwWvBUvEQvAxlsGAANiUBElCXAHSd39WU4M3evFnOZh2RlNq1pGBaGumRgIrV3dt/gBxbvToijUpDh6p0ZWiDP7qE8vBS8EZEygo2ErBF8B46dEhdLVysWDEbl5b4oSh4E8+YI5AACYQnAMH76rtdpNqvOdJ54dmglbWHd8fDHdXBJHp8uas0Ab0njESK1aolJ7ZsyQMJacqQriywBApefQEF9iZKpZKVpGGFhjK4yWCCJ4GkEbBF8D711FNStWpVefLJJ5O20EQMTMGbCKrskwRIIBoCEBUlb+kYtkngaXsK3mgIe7fu4QkT5MCw4X4LhKhFCSZsSzRqFNQT/NIDhWTjpQUigmIu6IiIWCGBBGwRvKNHj5bFixfLZ599lsCl2N81Ba/9zDkiCZCAPwEzgjeQGQUvdxEI7O3XXzJmzMiFUb5/PynTqZMgRdmOTp2kSKVKUqp1a8mYPl1O7d0bEhoFL/eTGwjYInh/++036dy5s9x4440qNVmtWrXcwCbiHCl4IyJiBRIggQQTiEXwQsSUbt1ailSuJEVcng89wXhd3T3ibDOmz5BidWrLBQ0b+sVtB4pdPASl1Kmt6mjBW6JhQ7l00kQJFvZgBEPB6+ptkm8mb4vgHTdunIwcOVKOHj0aFGx+vlqYh9byzc8aF0oCCSGwbv1cKdH+qZj6xivqCxo1EggbZHE4tnqNZG/eJIXSSkmp1q1Un4cnTpQyHcOHTMQ0eD5shGwI2el5Y2PNoED4QWqzZlIwLVUOT5ioHlQq9O+nBCr6DTyECNF6cPTo3BCEYrVrS7ke3VUfWtBi3IIlS0rV0aP9snioeW5OV2Ol1Kmj+t/S6Hp6eM0YinUcS8AWwYtrhPcYTnQG0rj++uulcOHCjoUUamL08LrOZJwwCXiOwP/mTpKUJ4fGfV0QVIj9NaaqwveS6REOzA4Q90W7rEPYBwI2a9EiuWLxolzxe3jiJDk0YYLfLWlK3Kb6BKzOyIDDaeV69lB9RCqhvLwny5WSp9oelV9LRepBpG+jvipFWe0ytSNXZg0SiDMBWwSvcc6nT59W/3SjwA1kT8Eb593I7kiABKImkCjBG/VE2MAUAXjTYykQ+7j2F4IVwjXUJRHGA4oQtIUQorDGly0hsFw2/TPVn5kSSvAe+WqiSotntjAnr1lSrBdvArYJ3nnz5qmwhp9++kl69uwpvXr1ktWrVwu+P3DgwHivy5b+rArejVv3yLibL5T6NSuZ/k/HloVxEBIgAVcQwKUTX858U2q+MDku88UhJQgmeP2yFi1WnkC8Ck9r1iypnl29uCJVkuthjgvkGDtBWMHJPXtyf1cgpzLsE5hlAd3DjvDElx/QP7c+whj+mD4995BamY4PS/kBA0zPhoLXNCpWdCgBWwQvwhluueUWadmypRw4cEAaNmyoBO/GjRulVatWsmrVKilTpoxDEYWeVqyCd+PeTOnwzgqptmuLjFg+VsXP4WAACwmQAAlEQwAH1l57p4u8+OGZaJqp/3MqDR8mp3bvkTNZmeoSgaOrVkvl4cNUPzoeFKLKrAcwqgmwctwIwEa46hflok6d1Fd9+CxwEB2bG8ute/ESvMjJ2+ryVlLxgorqKwsJ2EXAFsE7bNgw2bdvn4waNUp5eQsUKKAE74kTJ+TKK6+UL774QiAek1lOnTolr7/+urz77ruyZcsWKVjw/L3goeYVq+Bd8csh6fDOSnll6Ripf+gXCt5kGp5jk4CLCaxb9qkcGPCCXPZrdIuok745ugasne8JHBg6VBAbHFiiDWnQ7RnakO+3lO0AbBG8AwYMULesvfjii36C99tvv5X7779fli5dKhUqVLB98XrArKws+cc//iGlS5eWJUuWUPAmzRIcmARIwCwBeHf3Ll1gOpwBr7mRS5VvlMwSZj0jAXiRg11GQcHLfeIWArYI3lmzZskLL7wgr732mkDkFilSRIU4vPHGGyrEYe7cuUnldfbsWZk/f77KE9ygQQMK3qRag4OTAAmYIdBlXhc5vnqN6XCGKm+9Kbt79KTgNQOXdfIQiLfgxQDBvLx4kMM1xCjph9OZ0YF7MW4EbBG8OTk5MnToUBk/frzfxOHVRQhB7drOSFGSkZFBwRu3rcWOSIAEEkkgWsHLMIZEWsP7fePQ297+/f1uXEMO38zZY6LK0hBIqlaZWpJWNE19G/G9S3YuUSIXonfSpknqK/5tFL/GNt4nf36FeEBgiZ2ALYJXT2/Hjh2yadMmyczMlCpVqsh1110nKSkpsc8+zi3DCd7nn38+z2gff/yx/Pjjj1HPQsfwzp3RR7XlK8aoEbIBCeR7AhS8+X4L2A4g8HY2/O769d89TQnekkVKypFTR2yfs5cG/L7T915aju1rsUXwrlixQkqUKCFXX311ngXiENujjz4qxYsXt33xgQOGE7wrV67MM7+HH344ZsH7yOgvZeqcF1SfOhk47jHnieikbwNOgAQcT2DPkT3SdlZbeXjaH3LLDzkh5wsP3NkjPpFBD6/jzer4CRpvaNPOGi14IWhRQonax69+XMZ+NzbiGmteWFOyTmapPy0vb6m8v9jvqUVT1d/h6cVn+bGMu3Ncflx23NZsi+D997//LeXKlZPOnTv7TfzMmTPKy/vJJ59IzZo147aoWDuyM6Rh2JAPVEoyY6k0dKiUurd1rNNnuyQQwFWs6j/+RrElk0/ClDmkBwiMWT9GiYdBk89I3V2hBS88cPrSAQpeDxg+yUtACrQDQ4eptGc6hy8E6IjVI0SHGYQSte/f8X6uJxgpySBgt/6+Va3owToPSr9G/XJXh/zSiOW9/ZLbk7xiDu8lAgkVvIcOHZL169fL1KlT5cILL5RmhusLcVBs2bJlMnPmTHUBRdGiRZPONdmCFwCiufkm6cA4Adlc23dLEcUEN4OdBMwK3rLdu+eerOcetdNC+XMsvS+Drd4oeJd3WK6qbDm8RX3Vh9TyJzWu2i4CCRW827dvF+TgRaqvYAUZETp27Ch33XWXXesNO44TBC9uNao+Y7opHhmfTVcHCHCjTurtt6lk8fhearPbcxPHm+qIlWImQMEbMzo2tEBAC4vRY85IuYzQHt5LJ0yQHecuI6DgtQCcTU0RCCd4IXL14TRTnbESCcSZQEIFr57ruHHj1GUTgSENcV6L7d1ZuXji476vyD+//zzonFNvv11dCQkhG6pkLVoke/sPkLNZvlgmtDmTlaWumkRM8AWNGql4YFxFiVuU4so3vgAAIABJREFUCqWmKhFc1NAnPsO/ccWkvlnJdoguH5CC1+UGdOn0tbD4ZNjpPCvAWQB93SzfGLnUwC6ddjjBywNXLjWqh6Zti+BF+AL+wIOKTA1ly5aVSy65xPUYrQje2U++JA9tWRiSAUSrUcxC/CJOFEIWt93gTnQI2XiVEo0aSbnu3SWWKyfjNQc39kPB60aruX/O4QQvvLq68OfZ/bZ20wooeN1krfw3V1sEL7AiDy88vSg9e/ZUVwvjsodXX31VFi4MLfycbJJECl4z68YJ7NRmzZTwxZ8zmZlSplMnKZSWKkdXrVZd6KwPEMrZ6ZvlTOb5061nszIla9Fiv7yKpVq3lkrDhpoZnnVEGMPLXWA7AaQj+23DGnlm2tmg4QwQvBS6tpuFA4oIBS+3gZMJ2CJ4cXAN4QzPPfecummtYsWKSvAixrd58+byzTffqO+5rSRS8OIa0DKdOsofn01XohVi9oghFhqfI+wBghfFGLYQDcezmZmyf+gwFQqBMZDCCDcy6X6j6Ss/1nWDhxc2zk7fIgVTSzLtnQc2aaT8uxS8HjCyS5dAwetSw+WTadsiePv27SslS5ZU1wsj7y7ieSF4s7OzpX79+jJ79mypVauW65DHKnhfX7hVfnvrrZAhDcVq1ZLqM2f48YBo0SUR8bYQzIcnTFDhEii1Vq9KWFyvSm0zbLiKK0aoRpEqlSWldh1XpPZCHkqkeYJ41HGS4OXEA0Gw6U+3+x6IUHjBiev+iwk64Run3CiX/JgZ8kphCl5v2NmNq6DgdaPV8s+cbRG8CGfYv3+/jBo1yk/wIpShW7dusnbtWilVqpTrqMcqeO9/e4Vc/sWUPIIXIQoVBgxIWi5eCNFtre/1swMOwCBMItYCgYjXqxDsWqgjk8TeAQPydFmmY0f1PRzAM76SxQG9s1lHpFTrVqamgbGOrl6tvNT4Ow7z6QOAOvQj1gs+djzcMTevqXEyly9eFPaQoamJx7HSgaFDJWvxEr84bwreOAJOYlf1J9SXe9aclU6LzgadBQVvEo2Tz4dG7lz8mfHTDNl3dJ8fDR5ay+ebwwHLt0Xwrlu3Ttq3by9dunRRwhc5d3HRxKRJk5SHd8yYMQ5AEf0U4i14nSBIdnXr7hc6ASqI6y3d2nchBkIfIDyRIQLzLdezh+DyBXg8ISYhJCEuM6b7PNS4ex3tIaYRR4y2u3v0jHjgzpdq7XYlXE+kp6u+cLDOWDAewj7OZmZJwbRU9RVhGVmLF6v+jQf/IKaz09PVPHAYEHOq0L9f1F7sUILXaSIj2DydsL+i/yljCyMBJOR/8I0bQ8bvoq7T9iItmP8ITNo0SZbsXKLy6+ocu8y1m//2gdNWbIvgxaLnzZsnQ4YMUYJXl5YtW6q4XlxK4cZiRfBe8/n70urnZX7LdoIgCXwNHm+7QMhiDMQgI19w4KG5eI9nvFo1sG+kY9O3BZkZF2IZQl8LcGMbp4mMX1q1zjNPJ+wvM5xZJzQBeM9ee6eLXzjDpqoF5Oa/dVMhSYjBd9pepD1JgARIwAkEbBO8WGxOTo4cPHhQMjMzpVKlSlKiRAknMIh5DlYE730TB0v9Q7/kju0kMfJLy1ZyYovvBpzAAqGKyy6MBTHHELH4ZRsoMEu1aiUqJOHIkdwmqI/QDR22gM/h9dUFY8BTC48uPMqoCw9u9uZ0daVlodQ0Nd7+oUOlUFqamg/44bOC5z6DOMVnVUe/JYcmTFRzQFH1kdVi717lAUbYQ7ke3U2FIwQTkXrOThMZ+jBdoP0Qa2wML4l587NhUghA8I59s7Py8Ooy99Y0eWrsKtFefaftxaSA4qAkQAIkEEDAVsGLsY8fP57HCMWLF3elYbwqeBFjqwUmhCPCFJDqDN5ZnTEiXAysFlT6K/qAyIS4REwvDqkFXqpx8M231B5A2AM+1wXCN9RY4Q7yBYo6vQ70G+jFxvWrCM0IVtAPYmHPZGWqO+RDFSeJjFAx0nruuM0PDwLhLjZx5Q9kPpg0BO/c5zpL2+XnBe/yO6rIP0YuFMRt46EQ2VtijVHPBwi5RBIggXxKwBbBe/ToUXVYbe7cuX4hDZp5fjy0FujhtXo4LJ/u35iXDYH92+jRqn3J226TqmN8fw8sEOj6atZwgzlJ8EL46GwboeYcTuTHDJUNE0LAmNVk3wudQgrehAzOTkmABEjAIwRsEbyzZs2SwYMHq1RkderUkYIFC/rhu/LKK6Vw4cKuQxoPDy9e+asDWs1up1fG5h2gX/uDP7IsBCuB4RahpgjBe/CcgL504vmbrmxekhou1MG6wLk4SaQng5NbxjR67H+5o46krkz3u3BCe3jdsh7OkwRIgASSQcAWwdunTx8pX768PPPMM8lYY8LGjIfgpehImHkidmwUhqHy6Bo9weE6hB21JziZOXl3dOyUm4kiEgA8bFUaHjpMI1J7fh5/Agi3QYYTxLGXure1Cr/BQUnEsqPggFrdXTl+A6e3ulpaD/8o/pNhjyRAAiTgIQK2CN7//ve/smbNGnnvvfc8hE6Egtfd5sSrf9xkhwN1l03/LKiHHWIjY/r0iAtF3KSO8U2W4A2WYQOiFinlcMAv8CCikw5KRgScTyroEBrYBg8jSLFnjB0/WkzkghP+MH7rcLvc9KIvBp6FBEiABEggOAFbBC8yM9x///1y/fXXS5MmTeTiiy/2m821116bb0Ma6OFN7o+m9vLiIFf1Gf7CFmmejLepYaYQIirP8Az/m/CMq7DTpnr+iMnVMcmYi86EoQ8IBgtzCBfKkVyr5K/REbKArCHIUX1q9x5TMeMgpDOiMB47f+0Xx6923N0iO5aLtP9ApPZfHD9dTjD/ELBF8O7cuVMefPDBoAfWgDo/HVrb9ftxuWnEEnll6RiVlsxOcZR/trX5lRpDFrRnVguQk7t35xG22vOGCzpCpW4r16OHEsbG2+LMz8h8zWA34+nWgQfxQsX16jRl2elb1OUhPN1vnn+8ahrTiaFPM4ckUY//d8TLAuwnrgS04P1zP5Fb+8e1a3ZGAlYI2CJ4kaFh9uzZ8p///Edq1aolBQoU8JtzkSJFrKwhaW1jCWlY8cshGTbkAxmxfKyad6hX6UlbVD4bOJjg1QJE39QG8XpszRpFRse9RjoYluhwgXBiF/MM9PptHthH5JM5eawLwWt8jX7ppIn5bAckf7k/3d5Mxeoi73Sh1FQVxmCmUPCaocQ6thHYvlRkvMGjS8FrG3oOZI6ALYL32WefVRdN9O7d29ysXFIrHoI3WfGeLkGc8GkaBa++ulhfP6wHh7DAFccIGdBCMlfwXlBC5OixPPNEuED16Z8JPKcptWvJmaysuOW9HbN+jJRcuVEavrEkJJ/ABynkby15S8c89bFmzA9pzBIt0hNuTJcOEOqSkEjLoeCNRIif20qAgtdW3BwsegK2CN6PP/5Yxo8fr/LweqlQ8LrfmmayMAQTFlrwHqhZVspv/c0UiEpDh6qT9ygQ1YcnTJTU5s0k9fbb87RX1y9X9l3AoS7u2LtX1XviyyfkwNLFflfLBjYOFK57juyRL3d9KcV7DVVVA0/5aw82Ba8pM8a9UqyCF6n0eHlI3M3BDmMlECh4qzUVqXaTSLUmvq8sJJBkArYI3vT0dGnfvr3KaoCDa6mpqX7L7ty5sxQrVizJKKIf3qrgxaGTWmt9r8pZkkcA8bhHloT2lgYTFlrwBksTFW4l8Kji9rgT6em51SBadPgEvqprmrOypFTr1pJZppjkvHcu5VTFi2VO5YNS7VeRejtz5GCaCE7tVzt4fkTkad3ZtbncWvVWqV2mtvoAHuGx3/lCaFA+GXbab4r68BMFb3L2YKyCl2+HkmMvjhqCQKDg1dUY2sAt4xACtgjeyZMnCy6fCFWQrqxkyZIOQWJ+GlYFLwWGedaJrBnOy1um48NSfsCAoMNDSP725mi/a14TOc/Avpc0Kyf/bfi7+vagyWeU53bsPYXky6sKyONXPy4tL28pn//8uSCcAX90Qd1qB3KkxEn/HrkfY7cevPCF0tL8vPLw4CO1WOA114GjmBG8mQ/dJbu/nqdsjAcUHC5kvHXs9mLLBBCg4E0AVHYZTwK2CN54TthJfVHwOskasc/FeJNVYC/hvGhd5nWRy6auDip4pzbx3SZYLiNHLs4Q2V5eZE3NgtJsYyFp8t1JWXN5AVlbq6D8ecNZmdOogBwrVkDq7shRfR0rKvJrKZGvry4ol+7Pkc2XFpRt5X19PTPtrO9SgtatZUqN/fLuwZlqnD8fv0R++3WHGudoik/wNqzQULrO7xoUjBbIgR/iUJ4UKCClW7VKeJaJ2C3mrJYQtLt69My98ANeehTtpcfnCGW5oGFDKZiW5jd5M9dAo8FLDxSStkvPKsHLNGTOsj9nc44ABS+3gsMJ2Cp4T58+LdnZ2XmQuNG7i0XEInjfW7ZNZo6bpbI0FL3uT/L7f56SkkVK5r5+dvh+8eT0dJaCwMUhl231maHz7QYKXoQ3TL3JJ3Q3XlpAKl5QUfYd3efXLcQvCgRtqHLrhhzlpcW+SC2aqvpAiEL64XQZddsov72C72WdzJJaZWrJE0uekLUH1qpuUf/hug9HLXj/qFNZSm/e43eALZKH0pObImBR8OCe2JwuZ7IypUynTupTXDuN2Gq8IYC4jVRwOQkeJIyiN5h3F552XBbyx/TpudlBIHjvXn1WGv5EwRuJMz9PEgEK3iSB57BmCdgieDMzM+W1116TmTNnytGjR/PMLT/l4X194VZZPXm6DFw1Xg5ed5l0b7FLKpWsJPPvm2/WZqyXAAKhhEe418YQvBCYbb85qzyzELyDHiqUO7v373hfeVlxaGzmTzOVKN1yeIv6N4QsBOneI3vV3xFvi+9DwFYuWTlXxKYVTZPMk5mCr2bK4p2LpfeXvVWf2FcYL1gJ5eHVN3nBS1m6dStJqV1H9g8bJrXW+K62zS8FIv/knj1KcKrb+LKyVGw14q1TateW7PR09W9jwQMSPkddlenjyJE8uHDwEML31J69Ku/xttb35qlzslNrKduzu9oHePuwfN0Mea38t+oh6Y1b35DbL8l7yDG/2IXrdDABCl4HG4dTAwFbBC9y8I4YMUL69Okj//d//yflypWTihUrqswNnTp1koceesiV1ojFw/v0J9/J9SMHqEsn5t1aSt6/wfcA8H2n713JwCuT3tKgYa5A0TGSKXVqh4zfxbpDCd4G5RsooYsYWogWu0v9CfVDDom5QaR3m3VGbvkhx/TUcHAP+WEhgHEDHWJIcTMYvhfotTTdaZwq+tLI5RWX6L5Eo4amRsmYPkN5VHGgEGuDmD22OrLIR3hJarPbVYiJ8dIOeH8Rw4uCfYS0b2YLwmEgeLtd002Grx6uMmzgwQhFP0SZ7Yv1SMA2AhS8tqHmQLERsEXwDhgwQAncnj17ypgxY9TFE48//rhs2LBB7rvvPvnhhx/yTZaG+99eIfdNHKwEL36xTb3Z9woc3jj8gsOraLPevNhMzlaBBOBB3flwJyn0nS9zwgX/fU0uueXusKC02EUl7eHV9lzeYXlSbdji0xZ5Qin0Yl5u8rLy/i5/qVfSDtslewfqfMt6HhC3gd7aYHPE7XUQwQhpQP1yPXuYSgsGz/2h4a9IkU8XmFq68eAhDhvqMBU05oOxKYSsZDeBQWFitJilwW5rcLwQBGwRvEOHDlWhDEOGDJFp06bJggUL5O233xbE9NapU0fmzZsnNWrUcJ2RYvHwGgXvhGYFZU5Dn+DVBa+98XobcZl45Y2ScSJD/bvV5a1cx8jpEx6xZoR67V9v0kq5e63P44l4ScTg3nbJbQKBaHwA0eJj9P9Gn4+X3ZAjT/96rbx28bcq9jbZogQiKfCwmo4HRgww9tLc5zp7RvAi5jVU0TfkmdmH6sDeuYLQhEJpqVIoNS338J4KV8jMzHPwLFTfOr4auZPBvOHWs9J91tk82THUz3jVC+WNmzJVV/rgIR5M0E4X2HDFAyvMLIV1SMBeAhS89vLmaDERsEXwzp8/XwYPHixLly6V7du3S/PmzeXJJ5+UU6dOybhx42TNmjXixuuFYxG8Vw2aL88vHKU8vFpYRbKc/sWHV+SDmwyOVN2Vn0/ePFn9csda8YsdwhKZBlBwaAsiDp4yFP1AELhQeMlRMk9kKrEaKpwAQmTv0b2yZOcSFVuLor20RsGLvz9U9yF5qM5DuX3pcAEdGqDnkGyRG8hCz7PmhTVl6+9bBfMdd+c4VQ0sX3unS+4hqMC2yBKhU5ataFpGdlxwXAodOSHtOrwkNeo0VjGteE2PV/huu/hAeXN3+8fe6tjcEtc3itvPDt4a3DntTj/Bis4fXF5Yrt6c7Zc7Gd/fVSNNnm6X98Y+44SS/eYgbnDYkfcIUPB6z6YeXJEtgvfkyZPy+++/S/ny5RXCUaNGyZtvvillypRRcb1t27Z1JdpYBG+1fnPklaVjIgpeiL4jp/LGJerY0JY1WrqSWbBJz/hphryw/IW4rwfechQ8JOhLGCCsR6weEXQsfSFD4IPI32r8TcVRog+dz9bofXu24bO53vi4LyLGDnVYA+aO+E+w6NeoX67g1R7gwEsoUOGrKwvkxvca30Low3Woow/eaQ8kHi6iOVwXy7KwhkSG++BBCD9fsDMeFALHwuf4LNQ69feRn3nJriV+BwZhB7TFQ+vWnd/Knm+Xy4WvTpZymSLB3vQE4+O0h6pYbMg2HiVAwetRw3prWbYI3kOHDqmLJYy3qZ04cUIKFSokhQsXdi3RaAXvil8OSYd3VsrcGX3Umrs8WUjlTNUFaazgmUToAn454sAKfgHjFy9EIS4R0AWxvng9nYiCceEV1LlcEzGG7hPCCa98EVYQLI0XvocCYQUPLnigbqD4gdjA9yHKIE6NxSjUjBcwBK5Li79j/+9leevIHL/YyVAMML9P//ZpQoVYLPx7LemlmPVt1Ddkyjuwz2zQLE/3OocwbnMb37ygbC/vS5EW7AEslrm5pQ32GPYbOELQT9o0Se1B/Dwi3h4lJydHiVu9R3U9YygCPgv0zmK/ftvmHnUtdaQ3Pdhj6C+qcIb1H4j8sUvkmg4ipS91C3LO060EKHjdarl8NW9bBO9TTz0lVatWVWEMXipWBW+7/v5i3/jaORgniDUtRoP9Eo0HW/wibjurbe6pcAhwLTThucIpcfzyxZ9wAkh7yNAfxDMEE9ps+X2LOpgHYYD16BhazB0plyZvmqxEK8bFeBCTKNF49jAm+oYAhkgJnCc4I1QBDxX69Dvq39r+TTUWLptAH/AEGx8ygvF1u9ctWDo2ffgONkQBI4gtMELaM6TF0h7eaOxidX+Ge1ix2rduj/VhTcaDYrH0jT2GPazj8IP1sXPd17I8fYGMPDLT78FX18XPGvahfjsRVcaPcXeL7Fgu0nm2SLWbYlkC25CAeQIUvOZZsWbSCNgieEePHi2LFy+Wzz77LGkLTcTAsQjeYUM+UJdO6Jg9vOrEL7JoUljpDAE4UBWPg2zwHuOXM4QNBGIkkRcPljonLfqCsILHLBH5RfEaGsJZ573VlzQEE2pa/Onb1YId/gpcu9sF795+/SVjhv/lGhv+WkcGX/mj8vBrT2Y8bO62PvSVzPgK4anfLuhYcuOlIvpNhA5bMLuXg+0x/DwMaTrE2mU0WoBQ8Lpt27lzvhS87rRbPpu1LYL3t99+k86dO8uNN96o0pDVquWLrXR7iVbwGi+d+LHehfLc37JiyqupY14hGj/9q88DGmuBRwvxs4GvYI2vsPVNcKG8XvBmGQvEAcSz9qxqTyE8vYHlwToP5saWxrqGeLXDjWso+vAShA3EcrgHALcLXtwS9tvo0WrdyD+MyxJ4dW3oHYW9jaLfDODnBqIVbwxi8XYHCt54PcQKBW+8/ltgP2YIUPCaocQ6SSZgi+BFJoaRI0cGvWUN688vN609MnGtXDxtojy0ZWHupRNT/zo1ak8Ofuk2mdJEbR144PBvHGJD6AF+gcLzhMwCKPoXtBab+rUwPJ8qW8G5hPbGfQivM17FwiuqDzyF+mUe7UElHQeJeaDvRMUhx/PnKtxFDl4SvJWGDlWXKBivvo0nR/YVnIDeX3ERu/9tKrLfcIkNPbzcdnYQCCd4L23iC6/B1y5f2DEbjkECQQnYIniRimxPwDWcxtlcf/31rjy8Fq2HFzl4m346RprtWpt7MjtWwfTcsufyeB4D83aa3fM6phWiFqI5qlhBs4O4uJ5R8Gpvt+bk9lf+Rg/vpRMm5Hq3XWwu101dHzDUaeMsLSBQeFDwWsLJxgEEsv8Q2f+DSEqaSIWrRHA4cv2HItuXhUZFwctt5BACtgheh6w17tOIVvA2HbFE+k0bIjUy96qT2cUbNczNjRrL5Bp/2DjPgSx9wYAxvhB967ADnd1Af6XAjUweYQ0obhe3wVaa8dl02TtggPqIgjfyXnB8DQpex5vI1RPU1wdrb+2Xw0S+Hh5+SVrwppQSuXOYyDUPuhoBJ+9eArYJ3v3798uiRYsEXwNLjx49JCUlxXUUoxG8G/dmSv8XxqsDayg9uhWWG679mzqcEmvRXl4cmMHhNRw80ye6ETKQ6Lylsc6b7ZxF4MDQoZK9OV3KD+ivLpNgcTEBCl4XG88FU9cCV4vYwCmXqiqSscv/u8a6vGbYBUb27hRtEbwbN26UVq1aSYUKFdSNasi/e/DgQRXT26ZNGxk0aJBfjl634I5G8CIH78d9X5F/fv+5rLu5goxo8ptYvbBAnyKHlzYe2Rrcwp3zJAESCEGAgpdbI5EEIgneYEK49CUif+z0zeqaB3xe3pTSiZwl+yaBoARsEbxvvfWWbN26Vd2w9sYbb8jFF1+sblf7f//v/8k333wjU6ZMcaV5ohG8L83aKFVfe1Ea798oU+67SKbXzJBYDqy5EhQnTQIkYA+BYIeHIDh6Gw6y2TMTjuJFArEI3kAOdw4XqVBfpMKVFL5e3CMOXpMtgrdv375yySWXSPfu3WX8+PGyY8cOefHFFyUjI0MaNGggy5Yty712OJmsTp8+LdnZ2epWODMlGsGL+N3BH78g5Y8dlme7FpIT1SvJgjYLzAzDOiRAAiQQmcD+DSL/DXHJxKCMyO1ZgwQiEZj+mMh3U3wZF5B5IbCE+n6wfnmgMhJtfh5nArYI3rfffluWLl0qkydPli+//FKGDh0q06ZNE1wvjNy8+AzhDsks8EIjddoFF1wgDRs2lNdffz2i8DUreBG/e/9rC2TqnBfUEnHDmpPyzyaTO8cmARKIEwF9oChYdxS8cYKcz7vRN/iFwkDBm883iLOXb4vg3blzpxK7AwYMkJMnT8pNN92kxC4KROPUqVOTSmnDhg3yyCOPyMyZM6VcuXLy9NNPq6uQ8TVcMSt427+zUlp8+KoKZ1hzeQH5d9tCDGdIqsU5OAl4kAAFrweN6rAlxVPwIrThq2Ei2Rm8AtthZvbqdGwRvIHwkJN3+vTpKozhrrvuiuhJTTR8eHYRXjFw4EA11Lp166R3797K82xV8BpvV0NfCGfo3X5kQq7RTTQn9k8CJOBgAuEEL6bNE/IONp5LphZPwWtccmB4A/L98mCbSzaFe6Zpi+BFNgZkZnBq6jF4nmvXri0dO3ZUljtw4IA0bdpU0tPT1bxDFe3h3bXzV1m7bIaqdujAYUnZtl2OnD4smcdPSb0te6T64Sz12dQmBWVb20aWcu+6Z2txpiRAArYSiCR4qzUV6TxHBLG+2Zk8NGSrcZI42Ffn8uQiDy4OMEqO79BY6Uujn1S4G9XQWzQhDcbRMTfM6ZZ+IulzfFkdWo0RwdwhfK/p4PMEY/4UwtHbjS0UgYQKXghHeEpxdTDKHXfcIUOGDJFSpUo5Cj/yAN98883Srl07Na/MzEy57rrr5H//+1+u9/mDDz7IM2ekU/u8UOGIazlWVGROw4Ky7K7K6ipdnSs3YkNWIAESIAGzBNJni3wUIak/0kLhViwICh4aMkvWvfVmPO67CS2wQDje8LjIDd3Mry3coUjdS6yC1/ws8ndNxuJbsn9CBW+fPn1UWADiY8+cOSPvv/++CmGAUHRSef7556VGjRrSpUsXNS1cjoE44y1btkjBggXV98aNG5dnyjh8B8ELQXuoXE7u5/srn5WyZ86of/9etZBUvfRqKd76WWlYoaGTls25kAAJeImAmVuvjOuFB423XnlpB/ivZfw94a/8hUcVcbQo8P5HKpHCGdD++sdEVv03Uk/8PFYCFLyxklPtEip4r7nmGunfv7/cf//9arBVq1bJQw895CckLc0+To3Hjh0r+/btk3/961+qx5UrV8ozzzxjPoZX3S8ekOdy2zLfq8MtX/hm2f4Dkdp/idOM2Q0JkAAJGAhEK3bRFF4+CF6EQtS+R6TCVSIpab6vxvLHDv/X34yvtH/rBTIPZgNtJ3w2o5svNAAFsdsouAI42E1o+AyhLqWrhg9zGH6JL6wgVIEYixRWY2yLuWAPVrtJ5LKmvr63nTs3c9cIEbyxwL8RwvCnB2ILwbDfUhzRwQQSJniPHTsmV199tcyZM0dq1qypEOhQgRUrVkjZsmUdg+Wnn35SN77Nnj1bpUfr1auXXHbZZUr0hiumsjSsHCMyr7+vG4pex9icEyEBTxGIRfAGA6BjKfEZwh70DVnxhGXGmxjP8UL1BQ+nU+JB8aABLhCA25efd6BAQGKeCEu4pb8I7AOb4PcKwlPwcAKhiDofPeD7no6Bxbpxq5kOW4AQRjgLHC86n66RDfp4bJkIPMO6rTHWO1L8LgQvnDzTHxc58ENoC5a/0vc5D1Hascs5hoFAwgSvvlQCXt0yZcqoIc+ePSu1atWSxYsXq4sonFQmTpwo//nPf9SUrr32WpWHN1KssSnBiw7n9vW95rm6g0hrvu5xkt05FxLwBIEpHc6/TfLEgrgIywQQXgBPabACIQ1hmrHL/9N+O0SGnzvMpuP4PA83AAAgAElEQVRxdax3OMEbKKzxxhN/tLPHOIr2OMOrC+8uCwnYRCDhghfrwGUOuiBjQ+D3cL1wWlqaTUsOPUzCblrDq6Y3zr0mxH8oTvEqJJ04J0ACJBAXAmbiK0MNVCxN5ESmCL7Cywgv4K39RVaM8XkP7xruC32AhxDF6v9fEFtOKPCU/r4zuTNRGTPgGf3eZwNjgT1QYBOExukQAF0H7eAp1fbT3zfrWDG+fdRtIW7Hnwu9gzfZmCM3lODFeNgjgfsiVHgDD0smd8/l49ETJnhPnTol8+fPN4W2RYsWUrRoUVN1nVTJtIcXk9a/kHhQxEkm5FxIwBsEYhG8WkD96UFfDK8Ws4idxGvveAlcbxBO7CrgFEHRqbfwd4hgHXYBm0D4BhOVaAN7oQ88rJg9iAixvXmOyPoPznt6EVIRGMYSycNr9O4aKWE+//vQv398TsGb2L3E3kMSSJjgzQ/MoxK8+E8FBwlq3S3SYUp+wMM1kgAJ2EUgFsHLGEq7rOPscfC7CTHggeENetaIC4aoRmywfkjS3mUzAhZ9w9u7Y7mvRwpeZ+8HD8+OgteCcaMSvPr1Dp6gewdkdLAwBzYlARIggdw3SGZR4OBQ427mvYFm+2U9dxLQ50wizR5xvV2+8HmTx93jE8lmBawO33DSYcFI6+XnniJAwWvBnFEJXoyjY6DM/gdhYW5sSgIkkI8IROvh5f9B+WhzmFgqvLyI2Q6XXQHdaMGLv1PAmgDLKk4iQMFrwRpRC179S4m/bCxQZ1MSIIE8BKIRvPz/hxsoGAHj4epQhIyClxRJwGUEKHgtGCxqwatTB4UK8rcwFzYlARLIxwTMCl7E7TKJfz7eKBGWHinXLnPJc++4mAAFrwXjRS14dXJ4PiVboM6mJEACUXl4dT5VNOLVpNw84QgEHjAz1g2X15dUScAFBCh4LRgpasGrD65R8FqgzqYkkA8JQIhsmSNyw+PBD5qF8/BS8ObDDWNhyaHy5/JhyQJUNnUCAQpeC1ag4LUAj01JgATME9DhUKFSiWnBq8Wt8TICCl7znFnTRwB5fz960J8GBS93h8sJUPBaMGDUghenWv97k+8+9N4brN9YZGHubEoCJOBwArj4Yf8PIilpInP7+fKYIq0hLq8JvJI1UPAaRS7qIwc4CkWLw43ukOkF8/Jy7zjEOJxGrAQoeGMlJyJRC171C6eUb0SelLZAnk1JwOMEcGI+/QuRef18qaBw7Sxu3kLRXl7UWT9FpHRVkf994BPEWugGenX1/zsULR7fOHFenlH4cu/EGS67s5sABa8F4jEJ3rFNfLkOKXgtkGdTEvA4AX3ANdgyteA1HoJFvXCC1+O4uLwEEaDgTRBYdpsMAhS8FqjHJHiZi9cCcTYlgXxCIB6CV4c93No/n0DjMhNCgG8HEoKVndpPgILXAnMKXgvw2JQESEBEC1udHxchCojxR7z/+g+DEzLr4cUVsCwkYJUABa9VgmzvEAIUvBYMEZPgNf6Co+fFAn02jTuB8ff4uuw8R2T7Mt/fqzWN+zDs0EBA/39Q626RjF3n43TDQQoUvMa6xhheCl5utXgQoOCNB0X24QACFLwWjGBJ8DIXrwXybJoQAvoXG0SuFrw8qJIQ1KpTHDpD9gSwRvaFP3aaG4uC1xwn1iIBEiABAwEKXgvbISbBy8snLBBn04QQwJ7cvlzkq2F5u7+lv0i1JnnTYCVkIknsFCEEEJwIJ0DsK8Qo/p5SOjGTCpXc38xoEMe9vz8fDmFsAzGMtVS4SoRvkMzQZB0SIIF8QoCC14KhKXgtwGNT5xAId0AKs7zmAZE7hyVO/CWbxFfDz4t9iFysd+VYXzhHdobvZjMIVAjhClf6hPC8/j6v7C39REpfen4FyJ1rRiQHS+wfDQfk8dbeYWM7euSjoci6JEAC+YgABa8FY1sSvBXqizx2Lk7SwhzYlAQsEcDr9JVjRNLnhO+m/Qcitf/iq4M2EIPwJKJA+JkReZYmGqSxvpghVL86LCPY55i7zmtrNpQgWD/4OcYfddDse59ADnX9r7F9pIeMSKyu7iDy3ZS8tSh4I5Hj5yRAAvmUAAWvBcNbErwYl7+cLNBnU8sEdIq8aDqCsIOo81Ipf6Xv9T+EO8rmc+IfYQ0QwxCyet34isNl4QoOoHUIIkbRBt5klJwcka/P/T0WlqHswP9TYqHJNiRAAvmAAAWvBSNT8FqAx6bJJxCL4E3+rP1ngMOfoUrg9bvGehXPeWXxvXD1IHpRtMiH0IQ3fN8GkYpXiWxbKrLlC98NZxDHWgzDA45sF4FFHwyMhqMWses/OH9FcKj2FLzRkGVdEiCBfESAgteCsSl4LcBj0+QSQDjA+L+YS4OlZ4rX6HX+cl784VW+DmeAMDTGsiZ3dfaOruN2ESaBK35X/dc3fu17zoeKQIiaEazGmZeqKtK4m8gN3XzfNXPQjYLXXttzNBIgAdcQoOC1YKqYBC9+OY5t6vME4eBJfhUJFrizqUUCEE641CDUxQaBogv/xn7lddjmwAfznONnffrjvut/jQWx0fAaz+3nu3LcWAJDIyh4zfFnLRIgARIIQoCC18K2iEnwYjxeL2yBugObwrOHU/uIBb1rhAMnGDAlswemsJ7WY30prljME0AGhhVj8orbwB7A9/FzAjiYSNb5dnU7Cl7zNmBNEiABEgggQMFrYUtQ8FqA56WmbsutbFbw8nKU2HcpHoLg0Q302uoeEa7Q4cPzDxMId/h9p8ifHvDVQDwwwkWMb4AiCV7aK3Z7sSUJkIDnCVDwWjAxBa8FeF5pavTMuUVwmBG8iNf904Pev3Ai0fsQIUzDDXl69SG7u4ZH7zlHnDRsZ0xHhvzIiBuGsA6XHSLR62T/JEACJOBwAhS8FgxEwWsBnleaekXwthrjy62L3LXa08j48vjs0rl9zx9ksxoHHejlRX8QwYgNDgyBiM/s2QsJkAAJeIIABa8FM1oWvPDO6BPYFubBpkkk4EbBGxgv6hbPdBLNbHnoYVVFTmRaP6gKL+8bhphqCl7LpmEHJEAC+YMABa8FO1sWvPTIWKDvkKYUvA4xhAumYfba4UhLMebypeCNRIufkwAJkIAiQMFrYSNQ8FqA55WmFLxesaR71vH6lb40cfqBWe9BPkC7x4acKQmQgO0EKHgtIKfgtQDPK03/2/T85Q3Ip9pvp/NXFhjSwMsKnG8z4wwDBa4+hEjB6y47crYkQAK2EqDgtYCbgtcCPK80Dbwq1g3ikYLX3buPHl1324+zJwESSAoBCl4L2GMWvCvH+C4qQOqn1ueuIbUwDzZNEoHx9/iyGhiLGw6AacGEiw/gle7yRZIActiYCOBii33fi1zWlGnjYgLIRiRAAvmRAAWvBavHLHjddlGBBUaebhro3cVi3SR4rabI8rRxuTgSIAESIAEvEaDgtWBNCl4L8LzQNJjgxe1Y1zwock0H/1uynLJeYx5XCl6nWIXzIAESIAESSDABCl4LgCl4LcBze9PAG7QC13PNA75wgWo3+b6WruovgPFauvZfzFOAUJUCIhUQhlDafDtjTeRw/Wq4yPoPfd91Q7xxbCtlKxIgARIgARLwI0DBa2FDUPBagOf2poE3XkVaD24xg8Ddv8FXE6IT36tQ3ydgIUZveFwk/QuRW/qdE6U5PoH6R4jMD6i/cqx58crDapGsxM9JgARIgAQ8SoCC14JhKXgtwHN702gFbzTrhUc4OyOaFiKPLRWpYLiBK1hrCt7omLI2CZAACZCAZwhQ8FowJQXv/2/vXMBtKvM//kO5dKPbk27UVEOFKEWkUqNRMTGVojIJU2RKMxO6Uxq6THRzGbmUkS5PqSdCNSJddHVJRUx/ojDNVEiR4v983906tn323uec9e69z177fN7n8eDs9b7rXZ/3Xft81m/91rs84EW9almFt+ahsSOWzK5bbKYVElQU5VWEV5HfpQmrJaiOcoL15+TeZm+NMFs4OfZg3Mo3diaoFAq9qjpZuoOix9+uMtOqEkGJwsN1UZ8j9B8CEIAABPKGAML7y1Bs3brVhg0bZmPGjLGlS5da5cqVSxwkb+GNyosKSiRRATdQKsGMASUfuMS227SdRVSyrNzexCLp1TJnElzJbefHi9dTeoMeilMbn0w1ezvJsnZ9F+3IF1Z7CybtyNsN9jlgZfhc4JKPmi0gAAEIQAACeUUA4TWzjRs3Wo8ePaxWrVo2a9as3AmvpgIPDuXVCVHqziRboSFZ5Wy+/UqCPLlL7DWziSVIcUjWT0WOr1tc6kNlQwhAAAIQgEDUCSC8ZrZt2zabOXOmtWjRwpo2bYrwRn1W56L/JQlvtb1iD6Q10RJll2SvR5JevVo2MR1CLzVR+kOyQjpD9saDliEAAQhAIC8JILxxw7J+/XqENy+naR52Kl54lbagFRj0Br0tG2KdzWZkNxmOYQ12jvQqLSLV6g7K9W3eOw+h0iUIQAACEIBAdghUGOGdNGmSDRw4sBjFcePGWatWsXzKdMI7f/78YnU7depky5YtK/vIxD/wREpD2fnlQ4144Q0iptP778ipzbXwal1frezwf3NTR3bFTZHfs4eSv5sPc4g+QAACEIBAzghUGOHdvn27S11ILFWqVCn6UTrh7devX7G6U6ZM8RdeHh7K2WTP2I4SXzoRnyLgVkT4PPlDaRnrQJqGtP/haZYny7WI5+KY2QcEIAABCECgBAIVRnhLMxNyltIgKRl/buwWNK93Lc3Q5Nc2iqY+EZeXq6hpxySrJZRXr0e2jC19lqwgvOU1KuwXAhCAAATKkQDCGwc/Z8KrfQYvAUB4y3H6h9z1lKt2ThvIN4lMFuXN1UN0IZFSDQIQgAAEIJBNAggvwpvN+VWYbSe+sSzfhFfUE1eRYGWGwpyLHBUEIAABCJSKAMJbKkzJNwr94gkivB7U86BqlIRX6RZBTnHrG/IAHl2AAAQgAAEI5J4AwuvBHOH1gBflqlESXlJmojzT6DsEIAABCGSIAMLrARLh9YAXxapanWHt4tgrhdd+uOMI8jGlQS+jUDm5F0uQRXGu0WcIQAACEMgoAYTXAyfC6wEvalX1INjsoWYLHs9v0Y0aV/oLAQhAAAIQyAEBhNcDMsLrAS/fqrr1c1eZVdcrgZOsYxv/shD1Pd+WIss3nvQHAhCAAAQgkEcEEF6PwUB4PeDlW1WlAMwZapZqNYNE4c3HNIZ8Y0p/IAABCEAAAnlCAOH1GIiMCG+wfx4u8hiJDFRNJ7wrXjfTyybmjdyxI4Q3A9BpAgIQgAAEIJAbAgivB2eE1wNevlVNJbyJL5kI+t1hhFnjuLet5dvx0B8IQAACEIAABIoIILwekwHh9YCXb1UDsQ1SGpTTq4iuHlLbvL54b4nI59sI0h8IQAACEIBASgIIr8fkyKjw8hCUx0hkoGqwtm6tOma1G8aWHdMLG1IVhDcD0GkCAhCAAAQgkBsCCK8H54wKL69+9RiJDFRNfJlESU0ivCUR4nMIQAACEIBA3hBAeD2GAuH1gJdvVRHefBsR+gMBCEAAAhDIGAGE1wOll/BO72/29qgde69eM7a269l3efSIqqEJDK2TPFc3VYNEeEOjpiIEIAABCEAg1wQQXg/iXsIbrAoQv3/SGjxGw7PqwJrpG6imF1I0NFv5Rmw7hNcTONUhAAEIQAACuSOA8HqwRng94OVb1XTC2+yqHZH3YLsBK82q18q3o6A/EIAABCAAAQgkIYDwekyLjAuvIogXTzKrVdejV1QNRSCZ8Cqqu2WDWfxLJoLtBiZZqizUjqkEAQhAAAIQgEC2CSC8HoQzLrzqC2/w8hiRkFUTXxusZiS7HUeaKbdaS5UFFyFKRVFpfUPInVENAhCAAAQgAIFcE0B4PYhnRXglV4ry1m7k0TOqlolAMuEln7pMCNkYAhCAAAQgkM8EEF6P0cmK8Ab94Za5x8iUsWogvAc0MGtyidmMG8wQ3jJCZHMIQAACEIBA/hJAeD3GJqvCe9gpZh1GxG6lr3jdbPO3sVfd6u1f+qNb7fq8fjuPI6CqI7BkqtkTl5jVOyeWqjB9QGxFBpaIY4JAAAIQgAAECoIAwusxjFkVXvWr7VCzGQNS91DpD40vMVOEUuLbuHNsW60esHZRbF3Zw1p5HGEWq0rg82WVg+BBNPKnszjgNA0BCEAAAhAoPwIIrwf7rAtvYt+CVQNS9VlRX/1RdFJRYRX920mwft5ox+ezh5i1HRKLFqvkMlIsGZ8gQe8S26+i2eVV4vN3Ed7yGgX2CwEIQAACEMgqAYTXA6+X8Aa30cuyfwlq895meg2uRHX9qrLUTr2tZPjix81qN8h+1FWC+VzvHaKtXpXnSxwQ3szMIVqBAAQgAAEI5DEBhNdjcLyEN9nKAPF9iY/mSghVFK0N0gDmjYg9XBUUPXC1bnEsavvJVLOTe8f+VtGDWBLkb5T/u9JsybTY+rKJxS2/Vcfs8mnJqajPSpGQrGu7sqwkofxj1Qsiz4l7kHQP+CXa7DEmZaqq45k9NNYnPaSm/N18TQEp04GxMQQgAAEIQAAC8QQQXo/5kDXhDdaALSnNQJFevepWkqv0gMSc2FR5svq5pFeSqYe1kpWr5ppt3hDLA1b6g7aVGCr9YO2HOwQ8qCuhlgSrBOkT2o+2DR60C7ateahZr9djPx8Vl2Nc/9zY/tROh5E7eiVJdw/v/SLcyfqrNAm3Zu4vL+3QvnVBIC7/97rZ4acUl9n41zuTzuBxJlAVAhCAAAQgkN8EEF6P8cma8Gr1BT2MlouyYFJsL9qfxHL4L+v/Sh4ln5ksiqIqgnr0uTuiw8MaJE/NCPJ6FdVWdFgSLEHW/9WvQIzVP/07yEXW/4NtgtUsguNQm/EyHS+85ZlWkUnGtAUBCEAAAhCAQDECCK/HpMia8JbnGrzxEhiwOa5zLCIcpDxIICXEh7eK/a1UCRVFUQMBXfNLFLhGTbMf1psd2DD5g3GKzH4yzWzOUI+RCFG1ea+YJOu4VPou4pXOITBSBQIQgAAEIBAFAgivxyhlRXjLO9KoVICRp8SirsoLPnto8VSATC8pFp/PrHQOPZgX5AkHkWZJtvoiOZY8Bz/X3xLXNYtiectKaZBEz58US8FQHUV8tY3SKyT0iQ/7NbuKNXc9zgOqQgACEIAABPKdAMLrMUJZEd58iDTG5wZLPrNdlEox/3EzRYMztb9UUi4ZHn/ujof2lE/ccSQPq2V7jGkfAhCAAAQgUI4EEF4P+BkX3vKO7gYsJIVKX8iXF0N4jFHKqor0qpzcq7CPMxvsaBMCEIAABCAQMQIIr8eAFazwejChKgQgAAEIQAACEMg3Agivx4ggvB7wqAoBCEAAAhCAAARyRADh9QDtJbza78CaO/audWCbdGGlAI/xoCoEIAABCEAAAhBIRgDh9ZgXGRXe8lyKzIMBVSEAAQhAAAIQgEC+E0B4PUYI4fWAR1UIQAACEIAABCCQIwIIrwdohNcDHlUhAAEIQAACEIBAjgggvB6gvYV3yKE71oMlpcFjJKgKAQhAAAIQgAAEUhNAeD1mh7fwBi94UB8QXo+RoCoEIAABCEAAAhBAeLMyBxDerGClUQhAAAIQgAAEIJBRAkR4PXAivB7wqAoBCEAAAhCAAARyRADhjQO9ZcsWq1SpklWtWrVU+BHeUmFiIwhAAAIQgAAEIFCuBBBeM1uyZIkNHjzYFi9e7AbjrLPOskGDBlmNGjXSDo638L46xGzO0Ng+yOEt1xOBnUMAAhCAAAQgULgEEF4zW7hwoa1evdrOOecc27Rpk3Xv3t06depk559/fnaFd8VcswntzKrtZXbDqsKdZRwZBCAAAQhAAAIQKEcCCG8S+A888IBt3LjRbrrpptwIb92WZt1eLMdpwK4hAAEIQAACEIBA4RJAeBPGdvv27dahQwfr06ePtWnTpujTzZs3F5sFDRs2tGXLlhXu7ODIIAABCEAAAhCAQAEQqDDCO2nSJBs4cGCxIRs3bpy1atWq6OejR4+2uXPn2vjx423XXXct+nn79u2L1VXuL8JbAGcBhwABCEAAAhCAQEETqDDCq8jttm3big1mlSpVin42bdo0Gzp0qD333HO27777ljjw3g+tlbgHNoAABCAAAQhAAAIQ8CVQYYS3JFBz5syxfv362WOPPWb16tUraXP3OcJbKkxsBAEIQAACEIAABMqVAMJrZu+++6717NnTxo4da8cee2zRgFSvXj3t4CC85Tp32TkEIAABCEAAAhAoFQGE18zJ7uzZs4sB++ijj9K+hALhLdUcYyMIQAACEIAABCBQrgQQXg/8CK8HPKpCAAIQgAAEIACBHBFAeD1AI7we8KgKAQhAAAIQgAAEckQA4fUAjfB6wKMqBCAAAQhAAAIQyBEBhNcDNMLrAY+qEIAABCAAAQhAIEcEEF4P0AivBzyqQgACEIAABCAAgRwRQHg9QCO8HvCoCgEIQAACEIAABHJEAOH1AI3wesCjKgQgAAEIQAACEMgRAYTXAzTC6wGPqhCAAAQgAAEIQCBHBBBeD9AIrwc8qkIAAhCAAAQgAIEcEUB4PUAjvB7wqAoBCEAAAhCAAARyRADh9QAt4aVAAAIQKAuB5s2b27x588pShW0hAAEI2LJly6DgQQDh9YA3ceJE2759u3Xt2tWjFapGjUD79u3tqaeesho1akSt6/Q3JIGpU6fa8uXLrW/fviFboFoUCVxxxRU2cOBAq1OnThS7T59DEHj77bfthRdesMGDB4eoTZV8JoDweowOwusBL8JVEd4ID17IriO8IcFFvBrCG/EBDNF9hDcEtIhUQXg9Bgrh9YAX4aoIb4QHL2TXEd6Q4CJeDeGN+ACG6D7CGwJaRKogvB4DhfB6wItwVYQ3woMXsusIb0hwEa+G8EZ8AEN0H+ENAS0iVRBej4FCeD3gRbgqwhvhwQvZdYQ3JLiIV0N4Iz6AIbqP8IaAFpEqCG9EBopuQgACEIAABCAAAQiEI4DwhuNGLQhAAAIQgAAEIACBiBBAeCMyUHQTAhCAAAQgAAEIQCAcAYQ3HDdqQQACEIAABCAAAQhEhADC6zFQ33//ve2yyy5WtWpVj1aomu8EfvrpJ9u8ebPtsccexbqabg4wP/J9ZM2+++4723333a1SpUrFOrt+/Xrbc889rXLlyjt9ppfNbNiwwWrWrFmsTrrP8p9Gxeihzucff/zRdtttt2IHnO5cD/s9UDGoRuMot23bZhrHxN/Zqc51HVWY74Fo0Kh4vUR4Q4z5Dz/8YNdff729/vrrrvall15qf/nLX5L+0gzRPFVyTGDr1q02bNgwGzNmjC1dunQnwXnooYfs/vvvd1J04oknuu0kvunmAPMjxwMYYncvv/yy3X333fbVV185qe3SpYv16tXLtfT555/blVdeaWvWrHH/v+uuu+y3v/2t+/ecOXPs2muvdf/ef//9bdSoUXbEEUeU+FmILlIlwwRWrFhhd9xxh73//vuu5QYNGlj//v2tYcOG7v+pzvV0n3GuZ3iQstyc3po3ffp000oMJZ3rYb8HsnwINO9BAOENAW/cuHH26quv2tixY534dOzY0QYNGmStWrUK0RpVypPAxo0brUePHlarVi2bNWvWTsK7aNEi69mzpz3//PNObnRRc+ihh7q/080B5kd5jmjp9q1Xh9atW9caNWpkq1evttatW9srr7zifqYxlwRdc8019vHHH9t5551n7733nlWrVs2aN29uDz/8sLVs2dImTZpkzzzzjD377LPuDkCqz0rXI7bKNgFd3Eh02rZt64ITI0aMsAULFrjv8XTnetjvgWwfD+2XjcAbb7xht9xyi23atKlIeFOd67p7E+Z7oGw9YutcE0B4QxC/6KKLrFu3bu6LU0VRnlWrVtmdd94ZojWqlCcB3eKaOXOmtWjRwpo2bbqT8Cqyq9tZt956q+uiIkN9+/a1uXPnWro5wPwozxENt2+NWffu3e2UU06x4447zt588013kaNy+eWXu4va/fbbzxQhUnRYRZIrMZ49e7Ypepjqs4MPPjhcp6iVVQJTpkxxsqs1ltOd62G/B7LaeRovEwF9j+scvu2226xfv35OeJVylupcb9OmTajvAc71Mg1LzjdGeEMgVyR39OjRdswxx7jaukXy9NNPu6gfJZoE9IWYKLw33nij1a9f37p27eoOat26dU6IlixZYqeffnrKOcD8iNYcUORPFzzz5s1zvwTPOOMMW7ZsWdFBSGQPPPBAq127tjvXdYEbFG177733ugveVJ8df/zx0QJSwL3VRcqnn37qIrsTJkxwUfwOHTpYunNdUcEw3wMFjDFyhybJrVevnklkL7zwQie8OmdTnevnnHNOqO8BzvX8nhoIb4jxOeqoo2zGjBlFuXtKb3jwwQfdrU1KNAkkE94+ffrYqaeeap06dXIHpQeVTjjhBJs/f741adIk5RxgfkRnDugBFuXu6uL1uuuucxczF198sROioCjXVw+uSXg/+OADu++++4o+U7qDUlyU75fqM80hSn4Q+Oyzz5zkKle/WbNmdvPNNzuZTXeuDxgwINT3QH4cMb1QqpICVJMnT7Yvv/yySHjTnevt2rUL9T3AuZ7f8w3hDTE+iuCNHDnSPfSgoltikl0ivCFg5kmVZMKrX4Z6IEnpKypr1651edr6ZXnaaaelnAPMjzwZ1FJ0Q9HbL774wkVtq1Sp4n4hamzjI7xKaTnkkENclFfnun55xkd4JcCKFqX6rHHjxqXoCZvkksCWLVucAOl7XLmdmgepznWNf5jvgVweD/tKTuDrr792kVqd30ceeaTL11eKkiRYd3POPPPMpOe6hDfM9wDnen7PRIQ3xPhcdtllLodTJ4WKoruSIXJ4Q8DMkyrJhFe/DPWk/u233+56qVveWp1DObzp5gDzI08GtYRuDB8+3F577TV79NFH3UoNKhIhXcjq5xJclc6dO7toj/LzdGtUD+x5PgQAABDtSURBVDeqBDmAyuHVPEn1GXl9+Tkf9MCqbkErFUV52anO9bDfA/l51BWrVzqPlZufrOhhZN2hSXau6/mcMN8DnOv5Pb8Q3hDj88QTT5geeNAvSv3S00kjKdKT3pRoEkgmvMuXL7cLLrjARe50O1u3Qg8//HAnvenmAPMj/+fAI488Yk8++aRNnDjRrdChorQFrc+pBxMPOuggN84LFy50t0DfeecdJ8VanWHIkCEuaqQ7OvqlqT9KjUj1Wf7TqBg9VO6unr4/4IAD3Oo6Gvt77rnHrdCgKH+qcz3s90DFoBqto1TqUZDDq56nOtf33nvvUN8D0aJR8XqL8IYYcy1artvdL730klviRLe8leeVuEB9iKapUk4EkgmvuvLYY48V5WwqGqR1ePVLM90cYH6U0yCWYbfKs04semhRt7n1cKLyepXvqTJ48OCiuzlanuzqq692keB99tnHLW2lHFCVdJ+VoWtsmiUCWopOD6BpeTmNn/IttYb6SSed5PaY6lwP+z2QpcOgWQ8CicKb7lwP+z3g0T2qZpkAwusBWFEC5f3xpjUPiBGomu4NS+nmAPMjAoObpou65a0XjiS7kNUFUrI3ram5dJ9Fm0j0e//zzz+7h08VrddbMhNL2Detca5He26kO9fDfg9Em0hh9h7hLcxx5aggAAEIQAACEIAABH4hgPAyFSAAAQhAAAIQgAAECpoAwlvQw8vBQQACEIAABCAAAQggvMwBCEAAAhCAAAQgAIGCJoDwFvTwcnAQgAAEIAABCEAAAggvcwACEIAABCAAAQhAoKAJILwFPbwcHAQgAAEIQAACEIAAwsscgAAEIAABCEAAAhAoaAIIb0EPLwcHAQhAAAIQgAAEIIDwMgcgAAEIQAACEIAABAqaAMJb0MPLwUEAAhCAAAQgAAEIILzMAQhAAAIQgAAEIACBgiaA8Bb08HJwEIAABCAAAQhAAAIIL3MAAhCAAAQgAAEIQKCgCSC8BT28HBwEIAABCEAAAhCAAMLLHIAABCAAAQhAAAIQKGgCCG9BDy8HBwEIQAACEIAABCCA8DIHIACBpARWrVple+yxh+299947ff7jjz/a2rVrrU6dOjkjt2bNGnvyySftlVdeseOPP95uv/12r30vWbLEfv75Zzv22GO92kmsnK12M9rJFI394x//sC1bttif/vSnrO1u7ty59utf/9oOOOCArO2DhiEAAQgkI4DwMi8gAIGkBI466iirXbu2zZgxw3bfffeibRYuXGgXXHCBSe6qVKmSE3r9+/e3FStW2FVXXWXVq1e3k08+2Wu/am/9+vU2atQo186CBQucwO+zzz4ZbdersRxXvvXWW+3777+3e++9NyN7/vzzz23z5s1OcIPSrFkzu/rqq61r164Z2QeNQAACECgtAYS3tKTYDgIVjICEV+Xyyy+3m266qdyEV5HYE044we6++24766yzMjIKkl21GwjuGWec4URP0WOfktiuT1u5rptp4R04cKAdeOCBduWVVxYdyhdffGH77befVatWLdeHx/4gAIEKTgDhreATgMOHQCoCEt5rrrnGHnjgAZdOEMhgsgjv888/b+PGjbOVK1far371K+vVq5e1adOmVHCVIvHwww/biy++aF999ZXbzw033GDa/yeffGJ/+9vfbN68eU5O99prL3v00UftoIMO2qlttfHggw+6aLQiwUceeaSNHj3a/v3vf7ufnXvuufb3v//dPv74Y1u0aJE98sgjLvqofnbv3t0++OCDovb//Oc/29lnn+36cscdd9hrr73mPrvooousR48eLqr96quvpm33+uuvd/37z3/+Y0OHDrU33njD/f/MM880RZdr1qzp/q8+77bbbrZu3TqbPn26i17/7ne/sz/84Q/uWJMV8VebYlO3bl3r27evtW7d2qZMmWKPP/64PfHEE0WR923btlnnzp3twgsvdO1qLGfOnOkYKdoqGW3VqpXbTbzwipui6U8//bTVqlXLff7dd99Zx44dbeTIkY7vl19+acOHD7e33nrLNm7caLpo6NOnjxv/QYMG2T//+U93Z2D//fe33/zmN+64r7jiCuvUqZO1bdvWtamxuOeee+zDDz902/3+97+3nj172i677OI+D7Z/7rnn3Bw4+uij7bzzzrOLL77Yff7TTz+5/mg8PvvsM3dhpD62a9euVHOPjSAAgYpDAOGtOGPNkUKgTAQknBJdSZQk64UXXnBClii806ZNc9IlsZLQ6f+SLwlwIFPpdnzbbbeZhPmPf/yjE6nx48fbe++95wRH4iP5kmxKfJVzq22qVq26U5NBG5deeqmdeuqptnTpUidFkrubb77ZRRT1mYSoRYsWRXIncZR0qf0hQ4bYMccc46KSkk1Jk/5Wv3R7XvuXAKvdqVOnpm1X0WLJmMRZRfIoKR8zZoztu+++9tRTT1mlSpVcPyZPnmynn366a/frr7+2O++80wnvddddVwybcpl1fBJDCaxymiWWYi5xlvhOmjTJTjrpJFdXHDUu//rXv1zKhkS0Xr16Ls1A4yrRnzVrVjHhlUyrfY2B+qui6HXTpk1N8qlx0AXBsGHD3BiLky4oGjZs6PaxfPlylwvcvHlzJ9v6/JBDDnFSrIsM/Wz16tWuv6p/ySWXuHHQxUz8sWt75ZJr7HQB9e6779pDDz1UNLcmTpzo9iuR1z7mz5/v+hYcf5kmPBtDAAIFTQDhLejh5eAgEJ6AhFcypb8lHpKOv/71r8WEVxG3gw8+2EaMGFG0s8suu8z9W0KSrgQSJfEL6ijyKnGS8PXu3dt++OEHa9So0U5R5vg2gzZuvPFG69at2067k5iqHQmo+hmU+Ghmsvb1cJWiixJmRSxVFO2WpD377LNOeEtqV1FHybJkXiKtImHr0qWLk80TTzzRCa+2058gqtmvXz/773//66QusUjuJMuKGGt7RXAVEQ/GRn+rv8FDfZJP5VpLqhOLovGKvAbHGM+kNMKb2N4zzzzjLgiUD62iiwjNm/iUhnjhVWRXci4pD45dc0gSLfmtUaOGqy/JVjqLyvbt251Ea2zUriLkujOgC7NUEfHwZwA1IQCBQiKA8BbSaHIsEMgggUB4detb0V3d6lfkVrmvwUNrilIqYhgvrOqCHgbTn0B+UnVLt7IVrYyXQm2rvGGlEdx3330lCu9HH31kHTp0cH1r0KBBUuF9//33dxKikoR3woQJLtKqyGtQFGlWBFbHFAhvunYVqb7//vtN2wQP92kVBPVx8ODBTgjVD6U9BA/PaV9KxVDUVRKXWBQpVtRVshwUSbSivopyKvqq3Nl33nnHfaxIpyLcGi8V/XzOnDkuarphwwYnzhJVXVCUVXgVsX7ppZdclFjpDYo+K2Vk2bJlpRJeCau4xF8oiZUi3UrvUCRfwqsH3DQfgiLZPe644+zaa691q4XoAkLjItnXvxPTXTJ4StAUBCAQYQIIb4QHj65DIJsE4oVX+1FupaRGsqZb0oocqtSvX79I4IL+SBiV31mS8C5evNjlXCrP9ogjjig6HImd0hYkcSVFeCVckseXX37ZDjvssGLCq6jj22+/vdPPSxJeSaeiwvpTuXLlorqKRCpNQcJbUruB8Oo2uy4MVJTmoIilcnx1Gz/Zg2LphFey97///c/lEscXraYhCVaebZMmTZw0Sya1XSDlWnZMUVWJYePGjYvSNUorvBLzli1bFqU0qC+62FBOrgRUaQyKQJdFeJXjq4uaoAQXL+KrC6n4iHAy4dXPdBGh+aNUB+Umjx071l0AUCAAAQjEE0B4mQ8QgEBSAonCqyjeaaed5nIpJZfBsmTKEVVKQ/xyVhJWCZJu/6crQfqC8mMl0fFSGNymL0l4FeVTHqhycINIZrDP0ohp0If43FcJlHJQFdmW0CeW0rQrydYxxKdFBEKnh+bEsqzCq1QFMY1PA0jsm1IilOog4VU0PhgXyaPyl3XBoqIosqKsyYRX8iqxjz/+IPqqKLKi75JKyWr79u1de3fddZd7GDBeeDUuepAtKPECq3QEpcwoYh1cEASRdV3E7LnnnqUS3qBtXUwo5ebbb781tUOBAAQggPAyByAAgRIJJAqvKkgKdctcJRBeCdOAAQNMD44pJ1S3oyWweiDs/PPPd/moEhE92KbIYjJBUxRUT/Frn5IViVAgiiUJr9pTtFG36ZV2oQinHlrTA2qzZ88uMRKr+npATbf/dRwqEkWlM6g/ksJgxQhFeCVxpRFerWmrvGE9LKZj37p1q4tY6wE4CbUi2GUVXjGXYCqqqgixIqRKC1E0NFhG7s0333QPfukzSWXw4KDyoRUNlQBrRQONjyLwyYRXfVfUVvtR7rIEOIjYS3i1L/HSChfal+ReUetNmzYVCa8uQJRuoUi38nF1vPHCqzESd+Vd68JAFwOKQMfn7JYU4dUKEVrxQukP33zzjVvdQ6KvhwMpEIAABBBe5gAEIFAiAUlNfNQzEEHlSSoCJ2EJbvdLMBTdUy6lisRTD2xJPgKxUU6sBCqxSMJuueUWlw8qYVL0UMKsFR9Ugghs/NJoiW1oWS/lqkpwVXSLXw93KSpZUuqBtg8EVv0PHn779NNP3frD8WkZwWelEV61q4fcJNFBSoUextMDWBI0lWTCq9QDraqQLIdXdbQygx5GU2RbRWKrOsHKBIp0KvVAJXi4Tf/WceiiRHm2Kvq3HoxTnrH6ldgXHaNyizV+KjoOSXKQb60lyyTPYqYxExtd2AQRXsm5LmK0P4mr0iz0t8Q7iMRryTeJsYRaRTKvfgXLtiVur23ic3j1MJ4iy5o3Kpozkl4t10aBAAQgEE+AlAbmAwQgkDECkh+t2xqf96rGJWd6nWxw6zrZDiVqykEN1n0N0ynJsQQ6EKaytKGorqKEkrf4/ivCLKFSm7vuumtZmizaVm0ozSD+jXWhGoqrpNUp1Kb6lcg7Xdu65a90gdK+JU9jqn0k2177Vz8SXz8dv3/V1yuqE5eSi99GbWhZtTB8NW+U16xxC1PfdxyoDwEIRIMAwhuNcaKXEIAABCAAAQhAAAIhCSC8IcFRDQIQgAAEIAABCEAgGgQQ3miME72EAAQgAAEIQAACEAhJAOENCY5qEIAABCAAAQhAAALRIIDwRmOc6CUEIAABCEAAAhCAQEgCCG9IcFSDAAQgAAEIQAACEIgGAYQ3GuNELyEAAQhAAAIQgAAEQhJAeEOCoxoEIAABCEAAAhCAQDQIILzRGCd6CQEIQAACEIAABCAQkgDCGxIc1SAAAQhAAAIQgAAEokEA4Y3GONFLCEAAAhCAAAQgAIGQBBDekOCoBgEIQAACEIAABCAQDQIIbzTGiV5CAAIQgAAEIAABCIQkgPCGBEc1CEAAAhCAAAQgAIFoEEB4ozFO9BICEIAABCAAAQhAICQBhDckOKpBAAIQgAAEIAABCESDAMIbjXGilxCAAAQgAAEIQAACIQkgvCHBUQ0CEIAABCAAAQhAIBoEEN5ojBO9hAAEIAABCEAAAhAISQDhDQmOahCAAAQgAAEIQAAC0SCA8EZjnOglBCAAAQhAAAIQgEBIAghvSHBUgwAEIAABCEAAAhCIBgGENxrjRC8hAAEIQAACEIAABEISQHhDgqMaBCAAAQhAAAIQgEA0CCC80RgnegkBCEAAAhCAAAQgEJIAwhsSHNUgAAEIQAACEIAABKJBAOGNxjjRSwhAAAIQgAAEIACBkAQQ3pDgqAYBCEAAAhCAAAQgEA0CCG80xoleQgACEIAABCAAAQiEJIDwhgRHNQhAAAIQgAAEIACBaBD4f1gMkCHFkUbLAAAAAElFTkSuQmCC" 269 | }, 270 | "metadata": {}, 271 | "output_type": "display_data" 272 | } 273 | ], 274 | "source": [ 275 | "fig = em.params_plot(res, selector=lambda params: (params[\"a\"], params[\"b\"]))\n", 276 | "fig.show(renderer=\"png\")" 277 | ] 278 | } 279 | ], 280 | "metadata": { 281 | "kernelspec": { 282 | "display_name": "Python 3 (ipykernel)", 283 | "language": "python", 284 | "name": "python3" 285 | }, 286 | "language_info": { 287 | "codemirror_mode": { 288 | "name": "ipython", 289 | "version": 3 290 | }, 291 | "file_extension": ".py", 292 | "mimetype": "text/x-python", 293 | "name": "python", 294 | "nbconvert_exporter": "python", 295 | "pygments_lexer": "ipython3", 296 | "version": "3.9.0" 297 | } 298 | }, 299 | "nbformat": 4, 300 | "nbformat_minor": 5 301 | } 302 | -------------------------------------------------------------------------------- /exercises/solutions/03_play_with_algorithm_and_algo_options.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "57dee0b6", 6 | "metadata": {}, 7 | "source": [ 8 | "# Picking Algorithms\n", 9 | "\n", 10 | "In this exercise you get code snippets for two optimizations that fail silently. You need to fix them by selecting a different optimization algorithm. To make it a bit easier, we provide the criterion value at the optimum.\n", 11 | "\n", 12 | "If you find one that works, continue to search for one that works better. \n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [List of algorithms](https://estimagic.readthedocs.io/en/stable/algorithms.html)\n", 17 | "- [Documentation of algo_options](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_algorithm_and_algo_options.html)" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "id": "e1fcb3fa", 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np\n", 28 | "import estimagic as em" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "id": "7b50a918", 34 | "metadata": {}, 35 | "source": [ 36 | "## Problem 1\n", 37 | "\n", 38 | "This is a modified version of the `powell_singular` function from the More and Wild benchmark set. " 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 2, 44 | "id": "edced7fe", 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "def powell_steps(x):\n", 49 | " x = x.round(4)\n", 50 | " fvec = np.zeros(4)\n", 51 | " fvec[0] = x[0] + 10 * x[1]\n", 52 | " fvec[1] = np.sqrt(5.0) * (x[2] - x[3])\n", 53 | " fvec[2] = (x[1] - 2 * x[2]) ** 2\n", 54 | " fvec[3] = np.sqrt(10.0) * (x[0] - x[3]) ** 2\n", 55 | " out = {\"root_contributions\": fvec, \"value\": np.dot(fvec, fvec)}\n", 56 | " return out\n", 57 | "\n", 58 | "powell_start = np.array([3, -1, 0, 1])" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 3, 64 | "id": "c92ebf6b", 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "# optimal criterion value\n", 69 | "powell_criterion = 0" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "id": "c1ba89c1", 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": [ 81 | "Minimize with 4 free parameters terminated successfully after 1 criterion evaluations, 1 derivative evaluations and 0 iterations.\n", 82 | "\n", 83 | "The value of criterion improved from 215.00000000000003 to 215.00000000000003.\n", 84 | "\n", 85 | "The scipy_lbfgsb algorithm reported: CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL" 86 | ] 87 | }, 88 | "execution_count": 4, 89 | "metadata": {}, 90 | "output_type": "execute_result" 91 | } 92 | ], 93 | "source": [ 94 | "res_powell = em.minimize(\n", 95 | " criterion=powell_steps,\n", 96 | " params=powell_start,\n", 97 | " algorithm=\"scipy_lbfgsb\",\n", 98 | ")\n", 99 | "\n", 100 | "res_powell" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "id": "0ca9f8b4", 106 | "metadata": {}, 107 | "source": [ 108 | "---\n", 109 | "The optimization thinks it terminated \"successfully\" but when you look at `res.params` you see that it actually got stuck at the start values." 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 5, 115 | "id": "0517af36", 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "data": { 120 | "text/plain": [ 121 | "array([ 3., -1., 0., 1.])" 122 | ] 123 | }, 124 | "execution_count": 5, 125 | "metadata": {}, 126 | "output_type": "execute_result" 127 | } 128 | ], 129 | "source": [ 130 | "res_powell.params" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "id": "91c451d2", 136 | "metadata": {}, 137 | "source": [ 138 | "## Task 1:\n", 139 | "\n", 140 | "- Try to understand why the algorithm `scipy_lbfgsb` got stuck at the start parameters.\n", 141 | "- Find an algorithm that converges to the optimum." 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "id": "85f7a03e", 147 | "metadata": {}, 148 | "source": [ 149 | "## Solution 1:\n", 150 | "\n", 151 | "- The algorithm uses gradient information, however, because `powell_steps` rounds the input, the gradient is zero. Hence, it thinks it is at an optimum.\n", 152 | "- Use a gradient-free optimizer that utilizes the least-squares structure. (`nag_dfols`)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 6, 158 | "id": "82863097", 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "res_powell = em.minimize(\n", 163 | " criterion=powell_steps,\n", 164 | " params=powell_start,\n", 165 | " algorithm=\"nag_dfols\",\n", 166 | ")" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": 7, 172 | "id": "ad3df589", 173 | "metadata": {}, 174 | "outputs": [ 175 | { 176 | "data": { 177 | "text/plain": [ 178 | "array([ 0.00476221, -0.00054761, 0.0022716 , 0.00230015])" 179 | ] 180 | }, 181 | "execution_count": 7, 182 | "metadata": {}, 183 | "output_type": "execute_result" 184 | } 185 | ], 186 | "source": [ 187 | "res_powell.params" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": 8, 193 | "id": "a3c09b53", 194 | "metadata": {}, 195 | "outputs": [ 196 | { 197 | "data": { 198 | "text/plain": [ 199 | "4.1067145100000216e-08" 200 | ] 201 | }, 202 | "execution_count": 8, 203 | "metadata": {}, 204 | "output_type": "execute_result" 205 | } 206 | ], 207 | "source": [ 208 | "res_powell.criterion" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "id": "e861c83e", 214 | "metadata": {}, 215 | "source": [ 216 | "## Problem 2\n", 217 | "\n", 218 | "This is the `bratu_2d` problem from the Cartis and Roberts benchmark set. " 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": 9, 224 | "id": "291166fe", 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "def bratu(x):\n", 229 | " alpha = 4.\n", 230 | " x = x.reshape((int(np.sqrt(len(x))), int(np.sqrt(len(x)))))\n", 231 | " p = x.shape[0] + 2\n", 232 | " h = 1 / (p - 1)\n", 233 | " c = alpha * h**2\n", 234 | " xvec = np.zeros((x.shape[0] + 2, x.shape[1] + 2))\n", 235 | " xvec[1 : x.shape[0] + 1, 1 : x.shape[1] + 1] = x\n", 236 | " fvec = np.zeros_like(x)\n", 237 | " for i in range(2, p):\n", 238 | " for j in range(2, p):\n", 239 | " fvec[i - 2, j - 2] = (\n", 240 | " 4 * xvec[i - 1, j - 1]\n", 241 | " - xvec[i, j - 1]\n", 242 | " - xvec[i - 2, j - 1]\n", 243 | " - xvec[i - 1, j]\n", 244 | " - xvec[i - 1, j - 2]\n", 245 | " - c * np.exp(xvec[i - 1, j - 1])\n", 246 | " )\n", 247 | " fvec = fvec.flatten()\n", 248 | " out = {\"root_contributions\": fvec, \"value\": np.dot(fvec, fvec)}\n", 249 | " return out\n", 250 | "\n", 251 | "bratu_start = np.zeros(64)" 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": 10, 257 | "id": "9b6fd517", 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "# optimal criterion value \n", 262 | "bratu_criterion = 0" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": 11, 268 | "id": "8413eda0", 269 | "metadata": {}, 270 | "outputs": [ 271 | { 272 | "data": { 273 | "text/plain": [ 274 | "Minimize with 64 free parameters terminated unsuccessfully after 5000 criterion evaluations and 4697 iterations.\n", 275 | "\n", 276 | "The value of criterion improved from 0.1560737692424935 to 0.14373729795343512.\n", 277 | "\n", 278 | "The scipy_neldermead algorithm reported: Maximum number of function evaluations has been exceeded.\n", 279 | "\n", 280 | "Independent of the convergence criteria used by scipy_neldermead, the strength of convergence can be assessed by the following criteria:\n", 281 | "\n", 282 | " one_step five_steps \n", 283 | "relative_criterion_change 0.0001292 0.0007854 \n", 284 | "relative_params_change 0.02281 0.0461 \n", 285 | "absolute_criterion_change 1.857e-05 0.0001129 \n", 286 | "absolute_params_change 0.002281 0.00461 \n", 287 | "\n", 288 | "(***: change <= 1e-10, **: change <= 1e-8, *: change <= 1e-5. Change refers to a change between accepted steps. The first column only considers the last step. The second column considers the last five steps.)" 289 | ] 290 | }, 291 | "execution_count": 11, 292 | "metadata": {}, 293 | "output_type": "execute_result" 294 | } 295 | ], 296 | "source": [ 297 | "res_bratu = em.minimize(\n", 298 | " criterion=bratu,\n", 299 | " params=bratu_start,\n", 300 | " algorithm=\"scipy_neldermead\",\n", 301 | " algo_options={\"stopping.max_criterion_evaluations\": 5000},\n", 302 | ")\n", 303 | "\n", 304 | "res_bratu" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "id": "d7cbc4d9", 310 | "metadata": {}, 311 | "source": [ 312 | "---\n", 313 | "The optimization will terminate because it reaches the maximum number of criterion evaluations. Of course, you could increase that number even further, but you can do better! Also: One million evaluations would not even be enough!\n", 314 | "\n", 315 | "## Task 2:\n", 316 | "\n", 317 | "- Try to understand why `scipy_neldermead` does not converge on this problem.\n", 318 | "- Find an algorithm that converges to the optimum." 319 | ] 320 | }, 321 | { 322 | "cell_type": "markdown", 323 | "id": "0fc83307", 324 | "metadata": {}, 325 | "source": [ 326 | "## Solution 2:" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "id": "8b10c9ab", 332 | "metadata": {}, 333 | "source": [ 334 | "- The algorithm `scipy_neldermead` does not use gradient information.\n", 335 | "- Use an algorithm that utilizes the gradient information. (`scipy_lbfgsb`)" 336 | ] 337 | }, 338 | { 339 | "cell_type": "code", 340 | "execution_count": 12, 341 | "id": "fa3e9928", 342 | "metadata": {}, 343 | "outputs": [ 344 | { 345 | "data": { 346 | "text/plain": [ 347 | "Minimize with 64 free parameters terminated successfully after 41 criterion evaluations, 41 derivative evaluations and 36 iterations.\n", 348 | "\n", 349 | "The value of criterion improved from 0.1560737692424935 to 1.7033175395306e-10.\n", 350 | "\n", 351 | "The scipy_lbfgsb algorithm reported: CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH\n", 352 | "\n", 353 | "Independent of the convergence criteria used by scipy_lbfgsb, the strength of convergence can be assessed by the following criteria:\n", 354 | "\n", 355 | " one_step five_steps \n", 356 | "relative_criterion_change 2.448e-09** 1.652e-07* \n", 357 | "relative_params_change 6.455e-05 0.002428 \n", 358 | "absolute_criterion_change 2.448e-10** 1.652e-08* \n", 359 | "absolute_params_change 1.675e-05 0.000642 \n", 360 | "\n", 361 | "(***: change <= 1e-10, **: change <= 1e-8, *: change <= 1e-5. Change refers to a change between accepted steps. The first column only considers the last step. The second column considers the last five steps.)" 362 | ] 363 | }, 364 | "execution_count": 12, 365 | "metadata": {}, 366 | "output_type": "execute_result" 367 | } 368 | ], 369 | "source": [ 370 | "res_bratu = em.minimize(\n", 371 | " criterion=bratu,\n", 372 | " params=bratu_start,\n", 373 | " algorithm=\"scipy_lbfgsb\",\n", 374 | ")\n", 375 | "\n", 376 | "res_bratu" 377 | ] 378 | } 379 | ], 380 | "metadata": { 381 | "kernelspec": { 382 | "display_name": "Python 3 (ipykernel)", 383 | "language": "python", 384 | "name": "python3" 385 | }, 386 | "language_info": { 387 | "codemirror_mode": { 388 | "name": "ipython", 389 | "version": 3 390 | }, 391 | "file_extension": ".py", 392 | "mimetype": "text/x-python", 393 | "name": "python", 394 | "nbconvert_exporter": "python", 395 | "pygments_lexer": "ipython3", 396 | "version": "3.9.0" 397 | } 398 | }, 399 | "nbformat": 4, 400 | "nbformat_minor": 5 401 | } 402 | -------------------------------------------------------------------------------- /exercises/solutions/05_bounds_and_constraints.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "c889506e", 6 | "metadata": {}, 7 | "source": [ 8 | "## Bounds and constraints\n", 9 | "\n", 10 | "In this exercise you learn how to use bounds and simple constraints. \n", 11 | "\n", 12 | "Note that we will just scratch the surface of the topic. Look at the resources for more information. \n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [How to specify bounds](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_bounds.html)\n", 17 | "- [How to use constraints](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_constraints.html)\n", 18 | "- [Background: How constraints are implemented](https://estimagic.readthedocs.io/en/stable/explanations/optimization/implementation_of_constraints.html)\n", 19 | "\n", 20 | "## Example\n", 21 | "\n", 22 | "We reproduce the example from previous exercises for convenience." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "id": "774762d1", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import estimagic as em \n", 33 | "import numpy as np\n", 34 | "\n", 35 | "def criterion(x):\n", 36 | " first = (x[\"a\"] - np.pi) ** 2\n", 37 | " second = np.sum((x[\"b\"] - np.arange(3)) ** 2)\n", 38 | " third = np.sum((x[\"c\"] - np.eye(2)) ** 2)\n", 39 | " return first + second + third\n", 40 | " \n", 41 | " \n", 42 | "start_params = {\n", 43 | " \"a\": 1,\n", 44 | " \"b\": np.ones(3),\n", 45 | " \"c\": np.ones((2, 2))\n", 46 | "}" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 2, 52 | "id": "13550ef6", 53 | "metadata": {}, 54 | "outputs": [ 55 | { 56 | "data": { 57 | "text/plain": [ 58 | "{'a': 3.141592653589793,\n", 59 | " 'b': array([1.55630437e-16, 1.00000000e+00, 2.00000000e+00]),\n", 60 | " 'c': array([[ 1.00000000e+00, -4.50891722e-18],\n", 61 | " [-4.51479888e-18, 1.00000000e+00]])}" 62 | ] 63 | }, 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "output_type": "execute_result" 67 | } 68 | ], 69 | "source": [ 70 | "res = em.minimize(\n", 71 | " criterion=criterion,\n", 72 | " params=start_params,\n", 73 | " algorithm=\"nlopt_bobyqa\",\n", 74 | ")\n", 75 | "\n", 76 | "res.params" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "id": "3618be54", 82 | "metadata": {}, 83 | "source": [ 84 | "## Task 1: Bounds\n", 85 | "\n", 86 | "Repeat the optimization with an upper bounds of 2.0 on `a` and a lower bound of 0.5 for all entries in `b`.\n", 87 | "\n", 88 | "## Solution 1:" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": 3, 94 | "id": "37694bd8", 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "data": { 99 | "text/plain": [ 100 | "{'a': 2.0,\n", 101 | " 'b': array([0.5, 1. , 2. ]),\n", 102 | " 'c': array([[1., 0.],\n", 103 | " [0., 1.]])}" 104 | ] 105 | }, 106 | "execution_count": 3, 107 | "metadata": {}, 108 | "output_type": "execute_result" 109 | } 110 | ], 111 | "source": [ 112 | "res = em.minimize(\n", 113 | " criterion=criterion,\n", 114 | " params=start_params,\n", 115 | " algorithm=\"nlopt_bobyqa\",\n", 116 | " lower_bounds={\"b\": 0.5 * np.ones_like(start_params[\"b\"])},\n", 117 | " upper_bounds={\"a\": 2.0},\n", 118 | ")\n", 119 | "\n", 120 | "res.params" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "id": "5140ac87", 126 | "metadata": {}, 127 | "source": [ 128 | "## Task 2: Fixing parameters\n", 129 | "\n", 130 | "Remove the bounds but now fix the parameter `a` as well as the top right entry in `c` (i.e. `x[\"c\"][0, 1]`) at their start value.\n", 131 | "\n", 132 | "## Solution 2:" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": 4, 138 | "id": "bf40cf97", 139 | "metadata": {}, 140 | "outputs": [ 141 | { 142 | "data": { 143 | "text/plain": [ 144 | "{'a': 1.0,\n", 145 | " 'b': array([0., 1., 2.]),\n", 146 | " 'c': array([[1., 1.],\n", 147 | " [0., 1.]])}" 148 | ] 149 | }, 150 | "execution_count": 4, 151 | "metadata": {}, 152 | "output_type": "execute_result" 153 | } 154 | ], 155 | "source": [ 156 | "res = em.minimize(\n", 157 | " criterion=criterion,\n", 158 | " params=start_params,\n", 159 | " algorithm=\"nlopt_bobyqa\",\n", 160 | " constraints=[\n", 161 | " {\"type\": \"fixed\", \"selector\": lambda x: x[\"a\"]},\n", 162 | " {\"type\": \"fixed\", \"selector\": lambda x: x[\"c\"][0, 1]},\n", 163 | " ],\n", 164 | ")\n", 165 | "\n", 166 | "res.params" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "id": "ddf2cc89", 172 | "metadata": {}, 173 | "source": [ 174 | "## Optional: Play around with more constraints\n", 175 | "\n", 176 | "Look at the [documentation](https://estimagic.readthedocs.io/en/stable/how_to_guides/optimization/how_to_specify_constraints.html) and impose the constraint that the parameters in `\"c\"` sum to 1.\n", 177 | "\n", 178 | "## Solution:" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 5, 184 | "id": "c184fe11", 185 | "metadata": {}, 186 | "outputs": [ 187 | { 188 | "name": "stderr", 189 | "output_type": "stream", 190 | "text": [ 191 | "/Users/Tim_Mensinger/miniconda3/envs/scipy-test/lib/python3.9/site-packages/pandas/core/frame.py:6254: FutureWarning: In a future version, the Index constructor will not infer numeric dtypes when passed object-dtype sequences (matching Series behavior)\n", 192 | " diff = Index(subset).difference(self.columns)\n" 193 | ] 194 | }, 195 | { 196 | "data": { 197 | "text/plain": [ 198 | "{'a': 3.1415928282824384,\n", 199 | " 'b': array([-7.47836576e-09, 9.99999785e-01, 2.00000000e+00]),\n", 200 | " 'c': array([[ 0.75000093, -0.24999997],\n", 201 | " [-0.25000073, 0.74999977]])}" 202 | ] 203 | }, 204 | "execution_count": 5, 205 | "metadata": {}, 206 | "output_type": "execute_result" 207 | } 208 | ], 209 | "source": [ 210 | "# start params have to fulfill the constraint\n", 211 | "start_params[\"c\"] = np.tile(0.25, (2, 2))\n", 212 | "\n", 213 | "res = em.minimize(\n", 214 | " criterion=criterion,\n", 215 | " params=start_params,\n", 216 | " algorithm=\"nlopt_bobyqa\",\n", 217 | " constraints=[\n", 218 | " {\"type\": \"linear\", \"selector\": lambda x: x[\"c\"], \"value\": 1, \"weights\": 1.0}\n", 219 | " ],\n", 220 | ")\n", 221 | "\n", 222 | "res.params" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 6, 228 | "id": "1517e64f", 229 | "metadata": {}, 230 | "outputs": [ 231 | { 232 | "data": { 233 | "text/plain": [ 234 | "1.0" 235 | ] 236 | }, 237 | "execution_count": 6, 238 | "metadata": {}, 239 | "output_type": "execute_result" 240 | } 241 | ], 242 | "source": [ 243 | "res.params[\"c\"].sum()" 244 | ] 245 | } 246 | ], 247 | "metadata": { 248 | "kernelspec": { 249 | "display_name": "Python 3 (ipykernel)", 250 | "language": "python", 251 | "name": "python3" 252 | }, 253 | "language_info": { 254 | "codemirror_mode": { 255 | "name": "ipython", 256 | "version": 3 257 | }, 258 | "file_extension": ".py", 259 | "mimetype": "text/x-python", 260 | "name": "python", 261 | "nbconvert_exporter": "python", 262 | "pygments_lexer": "ipython3", 263 | "version": "3.9.0" 264 | } 265 | }, 266 | "nbformat": 4, 267 | "nbformat_minor": 5 268 | } 269 | -------------------------------------------------------------------------------- /exercises/solutions/07_automatic_differentiation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "5b4c54e4", 6 | "metadata": {}, 7 | "source": [ 8 | "# Automatic differentiation\n", 9 | "\n", 10 | "In this exercise you will use automatic differentiation in JAX and estimagic to solve the previous problem.\n", 11 | "\n", 12 | "## Resources\n", 13 | "\n", 14 | "- https://jax.readthedocs.io/en/latest/jax.numpy.html\n", 15 | "- https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "id": "b819cbd7", 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import jax \n", 26 | "import jax.numpy as jnp\n", 27 | "import estimagic as em\n", 28 | "\n", 29 | "jax.config.update(\"jax_enable_x64\", True)" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "id": "84a4abd1", 35 | "metadata": {}, 36 | "source": [ 37 | "## Task 1: Switch to JAX\n", 38 | "\n", 39 | "- Use the code from exercise 2, task 2, and convert the criterion function and the parameters to JAX. Hint: look at the [`jax.numpy` documentation](https://jax.readthedocs.io/en/latest/jax.numpy.html) and slides if you have any questions." 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 2, 45 | "id": "7fe73bcd", 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "def criterion(x):\n", 50 | " first = (x[\"a\"] - jnp.pi) ** 2\n", 51 | " second = jnp.sum((x[\"b\"] - jnp.arange(3)) ** 2)\n", 52 | " third = jnp.sum((x[\"c\"] - jnp.eye(2)) ** 2)\n", 53 | " return first + second + third\n", 54 | " \n", 55 | " \n", 56 | "start_params = {\n", 57 | " \"a\": 1.,\n", 58 | " \"b\": jnp.ones(3).astype(float),\n", 59 | " \"c\": jnp.ones((2, 2)).astype(float)\n", 60 | "}" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 3, 66 | "id": "910b0b5e", 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "data": { 71 | "text/plain": [ 72 | "DeviceArray(8.58641909, dtype=float64)" 73 | ] 74 | }, 75 | "execution_count": 3, 76 | "metadata": {}, 77 | "output_type": "execute_result" 78 | } 79 | ], 80 | "source": [ 81 | "criterion(start_params)" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "id": "c690e3bf", 87 | "metadata": {}, 88 | "source": [ 89 | "## Solution, Task 1 (Windows):" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 4, 95 | "id": "22bfb278", 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "import numpy as np\n", 100 | "\n", 101 | "def criterion_windows(x):\n", 102 | " first = (x[\"a\"] - jnp.pi) ** 2\n", 103 | " second = np.sum((x[\"b\"] - np.arange(3)) ** 2)\n", 104 | " third = np.sum((x[\"c\"] - np.eye(2)) ** 2)\n", 105 | " return first + second + third\n", 106 | " \n", 107 | " \n", 108 | "start_params_windows = {\n", 109 | " \"a\": 1.,\n", 110 | " \"b\": np.ones(3).astype(float),\n", 111 | " \"c\": np.ones((2, 2)).astype(float)\n", 112 | "}" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "id": "9c2814c9", 118 | "metadata": {}, 119 | "source": [ 120 | "## Solution, Task 2: Gradient\n", 121 | "\n", 122 | "- Compute the gradient of the criterion (the whole function). Hint: look at the [`autodiff_cookbook` documentation](https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html) and slides if you have any questions." 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 5, 128 | "id": "122f2831", 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "data": { 133 | "text/plain": [ 134 | "{'a': DeviceArray(-4.28318531, dtype=float64, weak_type=True),\n", 135 | " 'b': DeviceArray([ 2., 0., -2.], dtype=float64),\n", 136 | " 'c': DeviceArray([[0., 2.],\n", 137 | " [2., 0.]], dtype=float64)}" 138 | ] 139 | }, 140 | "execution_count": 5, 141 | "metadata": {}, 142 | "output_type": "execute_result" 143 | } 144 | ], 145 | "source": [ 146 | "gradient = jax.grad(criterion)\n", 147 | "gradient(start_params)" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 6, 153 | "id": "7aefa2e9", 154 | "metadata": {}, 155 | "outputs": [ 156 | { 157 | "name": "stdout", 158 | "output_type": "stream", 159 | "text": [ 160 | "11.5 ms ± 2.05 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\n" 161 | ] 162 | } 163 | ], 164 | "source": [ 165 | "%timeit gradient(start_params)" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": 7, 171 | "id": "dd8ffcc6", 172 | "metadata": {}, 173 | "outputs": [ 174 | { 175 | "name": "stdout", 176 | "output_type": "stream", 177 | "text": [ 178 | "17.2 µs ± 7.57 µs per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n" 179 | ] 180 | } 181 | ], 182 | "source": [ 183 | "jitted_gradient = jax.jit(gradient)\n", 184 | "%timeit jitted_gradient(start_params)" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "id": "92f96dcc", 190 | "metadata": {}, 191 | "source": [ 192 | "## Solution, Task 2 (Windows):\n", 193 | "\n", 194 | "The analytical gradient of the function is given by:\n", 195 | "\n", 196 | "- $\\partial_a f(a, b, C) = 2 (a - \\pi)$\n", 197 | "- $\\partial_b f(a, b, C) = 2 (b - \\begin{pmatrix}0,1,2\\end{pmatrix}^\\top)$\n", 198 | "- $\\partial_C f(a, b, C) = 2 (C - I_2)$\n", 199 | "\n", 200 | "---\n", 201 | "\n", 202 | "- Implement the analytical gradient\n", 203 | " - return the gradient in the form of `{\"a\": ..., \"b\": ..., \"C\": ...}`" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 8, 209 | "id": "2201091d", 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "def gradient(params):\n", 214 | " return {\n", 215 | " \"a\": 2 * (params[\"a\"] - np.pi),\n", 216 | " \"b\": 2 * (params[\"b\"] - np.array([0, 1, 2])),\n", 217 | " \"c\": 2 * (params[\"c\"] - np.eye(2))\n", 218 | " }" 219 | ] 220 | }, 221 | { 222 | "cell_type": "markdown", 223 | "id": "e9e578b5", 224 | "metadata": {}, 225 | "source": [ 226 | "## Solution, Task 3: Minimize\n", 227 | "\n", 228 | "- Use estimagic to minimize the criterion\n", 229 | " - pass the gradient function you computed above to the minimize call.\n", 230 | " - use the `\"scipy_lbfgsb\"` algorithm." 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": 9, 236 | "id": "f23ead7a", 237 | "metadata": {}, 238 | "outputs": [ 239 | { 240 | "data": { 241 | "text/plain": [ 242 | "{'a': 3.141592653589793,\n", 243 | " 'b': DeviceArray([3.33066907e-16, 1.00000000e+00, 2.00000000e+00], dtype=float64),\n", 244 | " 'c': DeviceArray([[1.00000000e+00, 3.33066907e-16],\n", 245 | " [3.33066907e-16, 1.00000000e+00]], dtype=float64)}" 246 | ] 247 | }, 248 | "execution_count": 9, 249 | "metadata": {}, 250 | "output_type": "execute_result" 251 | } 252 | ], 253 | "source": [ 254 | "res = em.minimize(\n", 255 | " criterion=criterion,\n", 256 | " derivative=jitted_gradient,\n", 257 | " params=start_params,\n", 258 | " algorithm=\"scipy_lbfgsb\",\n", 259 | ")\n", 260 | "\n", 261 | "res.params" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 10, 267 | "id": "1ef9fc6e", 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "data": { 272 | "text/plain": [ 273 | "{'a': 3.141592653589793,\n", 274 | " 'b': array([3.33066907e-16, 1.00000000e+00, 2.00000000e+00]),\n", 275 | " 'c': array([[1.00000000e+00, 3.33066907e-16],\n", 276 | " [3.33066907e-16, 1.00000000e+00]])}" 277 | ] 278 | }, 279 | "execution_count": 10, 280 | "metadata": {}, 281 | "output_type": "execute_result" 282 | } 283 | ], 284 | "source": [ 285 | "res = em.minimize(\n", 286 | " criterion=criterion_windows,\n", 287 | " derivative=gradient,\n", 288 | " params=start_params_windows,\n", 289 | " algorithm=\"scipy_lbfgsb\",\n", 290 | ")\n", 291 | "\n", 292 | "res.params" 293 | ] 294 | } 295 | ], 296 | "metadata": { 297 | "kernelspec": { 298 | "display_name": "Python 3 (ipykernel)", 299 | "language": "python", 300 | "name": "python3" 301 | }, 302 | "language_info": { 303 | "codemirror_mode": { 304 | "name": "ipython", 305 | "version": 3 306 | }, 307 | "file_extension": ".py", 308 | "mimetype": "text/x-python", 309 | "name": "python", 310 | "nbconvert_exporter": "python", 311 | "pygments_lexer": "ipython3", 312 | "version": "3.9.0" 313 | } 314 | }, 315 | "nbformat": 4, 316 | "nbformat_minor": 5 317 | } 318 | -------------------------------------------------------------------------------- /exercises/solutions/08_jaxopt.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "749bb281", 6 | "metadata": {}, 7 | "source": [ 8 | "# Using JAXopt\n", 9 | "\n", 10 | "In this exercise you will use JAXopt to solve a *batched* version of the first and second exercise.\n", 11 | "\n", 12 | "> Note. You cannot tackle these exercises on Windows.\n", 13 | "\n", 14 | "## Resources\n", 15 | "\n", 16 | "- [JAX documentation](https://jax.readthedocs.io/en/latest/notebooks/quickstart.html)\n", 17 | "- [jax.numpy documentation](https://jax.readthedocs.io/en/latest/jax.numpy.html)\n", 18 | "- [JAXopt documentation](https://jaxopt.github.io/stable/unconstrained.html)\n", 19 | "\n", 20 | "---\n", 21 | "\n", 22 | "Lets start by defining the criterion function from the solutions of the second exercise, but this time we use jax.numpy instead of numpy, and we parametrize the function." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "id": "2a2c352d", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import jax\n", 33 | "import jax.numpy as jnp\n", 34 | "\n", 35 | "jax.config.update(\"jax_enable_x64\", True)" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 2, 41 | "id": "b887acb6", 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "x0 = {\"a\": jnp.array(0.0), \"b\": jnp.zeros(3), \"c\": jnp.zeros((2, 2))}\n", 46 | "b0 = jnp.arange(3, dtype=\"float64\")\n", 47 | "\n", 48 | "\n", 49 | "def f(x, b0):\n", 50 | " value = (\n", 51 | " (x[\"a\"] - jnp.pi) ** 2\n", 52 | " + jnp.sum((x[\"b\"] - b0) ** 2)\n", 53 | " + jnp.sum((x[\"c\"] - jnp.eye(2)) ** 2)\n", 54 | " )\n", 55 | " return value" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "id": "300e82c5", 61 | "metadata": {}, 62 | "source": [ 63 | "## Task 1: Optimize using `JAXopt`\n", 64 | "\n", 65 | "- Create a solver instance of the class `LBFGS` for your problem. You need to make sure to\n", 66 | " - pass the function `f`,\n", 67 | " - set `tol=1e-6` for increased accuracy.\n", 68 | "- Run the optimization using `solver.run`. You need to make sure to\n", 69 | " - pass the initial parameters,\n", 70 | " - pass additional arguments of `f`.\n", 71 | "- Look at the output of the results. How do you access the parameters?" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 3, 77 | "id": "36c70848", 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "from jaxopt import LBFGS\n", 82 | "\n", 83 | "solver = LBFGS(fun=f, tol=1e-6)\n", 84 | "\n", 85 | "res = solver.run(init_params=x0, b0=b0)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 4, 91 | "id": "1ef505af", 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "from numpy.testing import assert_array_almost_equal\n", 96 | "import numpy as np\n", 97 | "\n", 98 | "assert_array_almost_equal(res.params[\"a\"], np.pi)\n", 99 | "assert_array_almost_equal(res.params[\"b\"], np.arange(3))\n", 100 | "assert_array_almost_equal(res.params[\"c\"], np.eye(2))" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "id": "11a8775f", 106 | "metadata": {}, 107 | "source": [ 108 | "## Task 2: Batched optimization\n", 109 | "\n", 110 | "Now you will optimize `f` not only for a single value of `b`, but for many.\n", 111 | "\n", 112 | "- Write a wrapper for `solver.run` that takes starting values for `x` and a single vector-valued parameter `b0`.\n", 113 | "- Use `vmap` and `jit` to create a vectorized and jitted version of this wrapper that allows for array-valued `b0`.\n", 114 | "- Execute your vectorized function on `b_arr` (this should perform 500 optimizations.)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 5, 120 | "id": "b3180c08", 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "from jax import jit\n", 125 | "from jax import vmap\n", 126 | "\n", 127 | "\n", 128 | "b_arr = jnp.arange(1500, dtype=\"float64\").reshape(500, 3)\n", 129 | "\n", 130 | "\n", 131 | "def solve(x, b0):\n", 132 | " return solver.run(init_params=x, b0=b0)" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": 6, 138 | "id": "b3080dca", 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "batch_solve = jit(vmap(solve, in_axes=(None, 0)))" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 7, 148 | "id": "5f4f8710", 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "result = batch_solve(x0, b_arr)\n", 153 | "\n", 154 | "assert_array_almost_equal(result.params[\"b\"], b_arr)" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "id": "775a46e5", 160 | "metadata": {}, 161 | "source": [ 162 | "## Optional: Speed comparison\n", 163 | "\n", 164 | "Lets compare the speed of `jaxopt`'s batch optimization to a loop with scipy's `minimize`.\n", 165 | "\n", 166 | "- Finish the loop in `batch_solve_scipy` using `method=\"L-BFGS-B\"` in scipy's `minimize`.\n", 167 | "- Time the functions (Use `%timeit func()` in a notebook cell.)" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 8, 173 | "id": "114f8ad3", 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [ 177 | "import numpy as np\n", 178 | "from scipy.optimize import minimize\n", 179 | "\n", 180 | "\n", 181 | "x0_numpy = np.zeros(8)\n", 182 | "b_arr_numpy = np.array(b_arr)\n", 183 | "\n", 184 | "\n", 185 | "def f_numpy(x, b0):\n", 186 | " a = x[0]\n", 187 | " b = x[1:4]\n", 188 | " C = x[4:].reshape(2, 2)\n", 189 | "\n", 190 | " value = (a - np.pi) ** 2 + np.sum((b - b0) ** 2) + np.sum((C - np.eye(2)) ** 2)\n", 191 | " return value" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 9, 197 | "id": "618c9e1e", 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "def batch_solve_scipy(x0, b_arr):\n", 202 | "\n", 203 | " results = []\n", 204 | " for b0 in b_arr:\n", 205 | " res = minimize(f_numpy, x0, method=\"L-BFGS-B\", args=(b0,))\n", 206 | " results.append(res.x[1:4])\n", 207 | "\n", 208 | " return np.stack(results)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 10, 214 | "id": "6993143e", 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "results = batch_solve_scipy(x0_numpy, b_arr_numpy)\n", 219 | "\n", 220 | "assert_array_almost_equal(results, b_arr_numpy, decimal=5)" 221 | ] 222 | }, 223 | { 224 | "cell_type": "markdown", 225 | "id": "570eef91", 226 | "metadata": {}, 227 | "source": [ 228 | "### Timing" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": 11, 234 | "id": "1299716a", 235 | "metadata": {}, 236 | "outputs": [ 237 | { 238 | "name": "stdout", 239 | "output_type": "stream", 240 | "text": [ 241 | "6.4 ms ± 655 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n" 242 | ] 243 | } 244 | ], 245 | "source": [ 246 | "%timeit batch_solve(x0, b_arr)" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 12, 252 | "id": "2f9e4a7a", 253 | "metadata": {}, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "1.87 s ± 65 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n" 260 | ] 261 | } 262 | ], 263 | "source": [ 264 | "%timeit batch_solve_scipy(x0_numpy, b_arr_numpy)" 265 | ] 266 | } 267 | ], 268 | "metadata": { 269 | "kernelspec": { 270 | "display_name": "Python 3 (ipykernel)", 271 | "language": "python", 272 | "name": "python3" 273 | }, 274 | "language_info": { 275 | "codemirror_mode": { 276 | "name": "ipython", 277 | "version": 3 278 | }, 279 | "file_extension": ".py", 280 | "mimetype": "text/x-python", 281 | "name": "python", 282 | "nbconvert_exporter": "python", 283 | "pygments_lexer": "ipython3", 284 | "version": "3.9.0" 285 | } 286 | }, 287 | "nbformat": 4, 288 | "nbformat_minor": 5 289 | } 290 | -------------------------------------------------------------------------------- /slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/optimagic-dev/scipy-estimagic/694befecd441a390728c2613222e6539ad597f79/slides.pdf -------------------------------------------------------------------------------- /test_installation.py: -------------------------------------------------------------------------------- 1 | """Test that the environment is correctly installed.""" 2 | from distutils.spawn import find_executable 3 | 4 | 5 | def is_installed(executable): 6 | return bool(find_executable(executable)) 7 | 8 | 9 | # ====================================================================================== 10 | # Check Installations 11 | # ====================================================================================== 12 | 13 | required_estimagic_version = "0.4.0" 14 | 15 | try: 16 | import estimagic # noqa: F401 17 | except ModuleNotFoundError: 18 | msg = ( 19 | "\n\nestimagic is not installed. Please install it via " 20 | f"'conda install -c conda-forge estimagic=={required_estimagic_version}'" 21 | ) 22 | raise ModuleNotFoundError(msg) # noqa: TC200 23 | 24 | from estimagic import __version__ # noqa: E402 25 | 26 | if required_estimagic_version not in __version__: 27 | msg = ( 28 | f"You've installed estimagic version {__version__}, but we require version" 29 | f"{required_estimagic_version}" 30 | ) 31 | raise Exception(msg) # noqa: TC002 32 | 33 | from estimagic.config import IS_DFOLS_INSTALLED # noqa: E402 34 | from estimagic.config import IS_PYGMO_INSTALLED # noqa: E402 35 | from estimagic.config import IS_PYBOBYQA_INSTALLED # noqa: E402 36 | from estimagic.config import IS_NLOPT_INSTALLED # noqa: E402 37 | from estimagic.config import IS_FIDES_INSTALLED # noqa: E402 38 | from estimagic.config import IS_JAX_INSTALLED # noqa: E402 39 | 40 | not_installed = [] 41 | 42 | if not is_installed("jupyter"): 43 | not_installed.append(("jupyter", "conda")) 44 | 45 | if not IS_NLOPT_INSTALLED: 46 | not_installed.append(("nlopt", "conda")) 47 | 48 | if not IS_JAX_INSTALLED: 49 | not_installed.append(("jax", "conda")) 50 | 51 | if not IS_PYGMO_INSTALLED: 52 | not_installed.append(("pygmo", "conda")) 53 | 54 | if not IS_FIDES_INSTALLED: 55 | not_installed.append(("fides", "pip")) 56 | 57 | if not IS_DFOLS_INSTALLED: 58 | not_installed.append(("DFO-LS", "pip")) 59 | 60 | if not IS_PYBOBYQA_INSTALLED: 61 | not_installed.append(("Py-BOBYQA", "pip")) 62 | 63 | try: 64 | import jaxopt # noqa: E402, F401 65 | except ModuleNotFoundError: 66 | not_installed(("jaxopt", "pip")) 67 | 68 | 69 | # ====================================================================================== 70 | # Raise Error 71 | # ====================================================================================== 72 | 73 | not_installed_conda = [pkg for pkg, pkg_mng in not_installed if pkg_mng == "conda"] 74 | not_installed_pip = [pkg for pkg, pkg_mng in not_installed if pkg_mng == "pip"] 75 | 76 | 77 | msg = "The following packages are missing and need to be installed via:" 78 | 79 | if not_installed_conda: 80 | msg += f"\nconda: {not_installed_conda}." 81 | 82 | if not_installed_pip: 83 | msg += f"\npip: {not_installed_pip}." 84 | 85 | if not_installed_conda or not_installed_pip: 86 | raise ModuleNotFoundError(msg) 87 | else: 88 | print("\nYour installation is working!!!\n") # noqa: T201 89 | --------------------------------------------------------------------------------