├── CONTRIBUTORS.md ├── LICENSE.md ├── README.md ├── examples ├── Turbo1.ipynb └── TurboM.ipynb ├── requirements.txt ├── setup.py └── turbo ├── __init__.py ├── gp.py ├── turbo_1.py ├── turbo_m.py └── utils.py /CONTRIBUTORS.md: -------------------------------------------------------------------------------- 1 | Code written by: 2 | - David Eriksson 3 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by the text below. 2 | 3 | "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. 4 | 5 | "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. 6 | 7 | "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. 8 | 9 | "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. 10 | 11 | "Work" shall mean the work of authorship, whether in Source or Object form, made available under this License. 12 | 13 | This License governs use of the accompanying Work, and your use of the Work constitutes acceptance of this License. 14 | 15 | You may use this Work for any non-commercial purpose, subject to the restrictions in this License. Some purposes which can be non-commercial are teaching, academic research, and personal experimentation. You may also distribute this Work with books or other teaching materials, or publish the Work on websites, that are intended to teach the use of the Work. 16 | 17 | You may not use or distribute this Work, or any derivative works, outputs, or results from the Work, in any form for commercial purposes. Non-exhaustive examples of commercial purposes would be running business operations, licensing, leasing, or selling the Work, or distributing the Work for use with commercial products. 18 | 19 | You may modify this Work and distribute the modified Work for non-commercial purposes, however, you may not grant rights to the Work or derivative works that are broader than or in conflict with those provided by this License. For example, you may not distribute modifications of the Work under terms that would permit commercial use, or under terms that purport to require the Work or derivative works to be sublicensed to others. 20 | 21 | In return, we require that you agree: 22 | 23 | 1. Not to remove any copyright or other notices from the Work. 24 | 25 | 2. That if you distribute the Work in Source or Object form, you will include a verbatim copy of this License. 26 | 27 | 3. That if you distribute derivative works of the Work in Source form, you do so only under a license that includes all of the provisions of this License and is not in conflict with this License, and if you distribute derivative works of the Work solely in Object form you do so only under a license that complies with this License. 28 | 29 | 4. That if you have modified the Work or created derivative works from the Work, and distribute such modifications or derivative works, you will cause the modified files to carry prominent notices so that recipients know that they are not receiving the original Work. Such notices must state: (i) that you have changed the Work; and (ii) the date of any changes. 30 | 31 | 5. If you publicly use the Work or any output or result of the Work, you will provide a notice with such use that provides any person who uses, views, accesses, interacts with, or is otherwise exposed to the Work (i) with information of the nature of the Work, (ii) with a link to the Work, and (iii) a notice that the Work is available under this License. 32 | 33 | 6. THAT THE WORK COMES "AS IS", WITH NO WARRANTIES. THIS MEANS NO EXPRESS, IMPLIED OR STATUTORY WARRANTY, INCLUDING WITHOUT LIMITATION, WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE OR ANY WARRANTY OF TITLE OR NON-INFRINGEMENT. ALSO, YOU MUST PASS THIS DISCLAIMER ON WHENEVER YOU DISTRIBUTE THE WORK OR DERIVATIVE WORKS. 34 | 35 | 7. THAT NEITHER UBER TECHNOLOGIES, INC. NOR ANY OF ITS AFFILIATES, SUPPLIERS, SUCCESSORS, NOR ASSIGNS WILL BE LIABLE FOR ANY DAMAGES RELATED TO THE WORK OR THIS LICENSE, INCLUDING DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL OR INCIDENTAL DAMAGES, TO THE MAXIMUM EXTENT THE LAW PERMITS, NO MATTER WHAT LEGAL THEORY IT IS BASED ON. ALSO, YOU MUST PASS THIS LIMITATION OF LIABILITY ON WHENEVER YOU DISTRIBUTE THE WORK OR DERIVATIVE WORKS. 36 | 37 | 8. That if you sue anyone over patents that you think may apply to the Work or anyone's use of the Work, your license to the Work ends automatically. 38 | 39 | 9. That your rights under the License end automatically if you breach it in any way. 40 | 41 | 10. Uber Technologies, Inc. reserves all rights not expressly granted to you in this License. 42 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Overview 2 | 3 | This is the code-release for the TuRBO algorithm from ***Scalable Global Optimization via Local Bayesian Optimization*** appearing in NeurIPS 2019. This is an implementation for the noise-free case and may not work well if observations are noisy as the center of the trust region should be chosen based on the posterior mean in this case. 4 | 5 | Note that TuRBO is a **minimization** algorithm, so please make sure you reformulate potential maximization problems. 6 | 7 | ## Benchmark functions 8 | 9 | ### Robot pushing 10 | The original code for the robot pushing problem is available at https://github.com/zi-w/Ensemble-Bayesian-Optimization. We have made the following changes to the code when running our experiments: 11 | 12 | 1. We turned off the visualization, which speeds up the function evaluations. 13 | 2. We replaced all instances of ```np.random.normal(0, 0.01)``` by ```np.random.normal(0, 1e-6)``` in ```push_utils.py```. This makes the function close to noise-free. Another option is to average over several evaluations using the original code 14 | 3. We flipped the sign of the objective function to turn this into a minimization problem. 15 | 16 | Dependencies: ```numpy ```, ```pygame```, ```box2d-py``` 17 | 18 | ### Rover 19 | The original code for the robot pushing problem is available at https://github.com/zi-w/Ensemble-Bayesian-Optimization. We used the large version of the problem, which has 60 dimensions. We have flipped the sign of the objective function to turn this into a minimization problem. 20 | 21 | Dependencies: ```numpy```, ```scipy``` 22 | 23 | ### Lunar 24 | 25 | The lunar code is available in the OpenAI gym: https://github.com/openai/gym. The goal of the problem is to learn the parameter values of a controller for the lunar lander. The controller we learn is a modification of the original heuristic controller which takes the form: 26 | 27 | ``` 28 | def heuristic_Controller(s, w): 29 | angle_targ = s[0] * w[0] + s[2] * w[1] 30 | if angle_targ > w[2]: 31 | angle_targ = w[2] 32 | if angle_targ < -w[2]: 33 | angle_targ = -w[2] 34 | hover_targ = w[3] * np.abs(s[0]) 35 | 36 | angle_todo = (angle_targ - s[4]) * w[4] - (s[5]) * w[5] 37 | hover_todo = (hover_targ - s[1]) * w[6] - (s[3]) * w[7] 38 | 39 | if s[6] or s[7]: 40 | angle_todo = w[8] 41 | hover_todo = -(s[3]) * w[9] 42 | 43 | a = 0 44 | if hover_todo > np.abs(angle_todo) and hover_todo > w[10]: 45 | a = 2 46 | elif angle_todo < -w[11]: 47 | a = 3 48 | elif angle_todo > +w[11]: 49 | a = 1 50 | return a 51 | ``` 52 | 53 | We use the constraints 0 <= w_i <= 2 for all parameters. We use ```INITIAL_RANDOM = 1500.0``` to make the problem more challenging. 54 | 55 | For more information about the logic behind this controller and how to integrate it with ```gym```, take a look at the original heuristic controller source code: https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py#L364 56 | 57 | Dependencies: ```gym```, ```box2d-py``` 58 | 59 | ### Cosmological constant 60 | The code for the cosmological constant problem is available here: https://ascl.net/1306.012. You need to follow the instructions and compile the FORTRAN code. This gives you an executable ```CAMB``` that you can call to run the simulation. 61 | 62 | The parameter names and bounds that we tune are the following: 63 | 64 | ``` 65 | ombh2: [0.01, 0.25] 66 | omch2: [0.01, 0.25] 67 | omnuh2: [0.01, 0.25] 68 | omk: [0.01, 0.25] 69 | hubble: [52.5, 100] 70 | temp_cmb: [2.7, 2.8] 71 | hefrac: [0.2, 0.3] 72 | mneu: [2.9, 3.09] 73 | scalar_amp: [1.5e-9, 2.6e-8] 74 | scalar_spec_ind: [0.72, 5] 75 | rf_fudge: [0, 100] 76 | rf_fudge_he: [0, 100] 77 | ``` 78 | 79 | ## Examples 80 | Check the examples folder for two examples on how to use Turbo-1 and Turbo-n. 81 | 82 | ## Citing us 83 | 84 | The final version of the paper is available at: http://papers.nips.cc/paper/8788-scalable-global-optimization-via-local-bayesian-optimization. 85 | 86 | ``` 87 | @inproceedings{eriksson2019scalable, 88 | title = {Scalable Global Optimization via Local {Bayesian} Optimization}, 89 | author = {Eriksson, David and Pearce, Michael and Gardner, Jacob and Turner, Ryan D and Poloczek, Matthias}, 90 | booktitle = {Advances in Neural Information Processing Systems}, 91 | pages = {5496--5507}, 92 | year = {2019}, 93 | url = {http://papers.nips.cc/paper/8788-scalable-global-optimization-via-local-bayesian-optimization.pdf}, 94 | } 95 | ``` 96 | -------------------------------------------------------------------------------- /examples/Turbo1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Simple example of TuRBO-1" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "from turbo import Turbo1\n", 17 | "import numpy as np\n", 18 | "import torch\n", 19 | "import math\n", 20 | "import matplotlib\n", 21 | "import matplotlib.pyplot as plt" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "## Set up an optimization problem class" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 2, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "class Levy:\n", 38 | " def __init__(self, dim=10):\n", 39 | " self.dim = dim\n", 40 | " self.lb = -5 * np.ones(dim)\n", 41 | " self.ub = 10 * np.ones(dim)\n", 42 | " \n", 43 | " def __call__(self, x):\n", 44 | " assert len(x) == self.dim\n", 45 | " assert x.ndim == 1\n", 46 | " assert np.all(x <= self.ub) and np.all(x >= self.lb)\n", 47 | " w = 1 + (x - 1.0) / 4.0\n", 48 | " val = np.sin(np.pi * w[0]) ** 2 + \\\n", 49 | " np.sum((w[1:self.dim - 1] - 1) ** 2 * (1 + 10 * np.sin(np.pi * w[1:self.dim - 1] + 1) ** 2)) + \\\n", 50 | " (w[self.dim - 1] - 1) ** 2 * (1 + np.sin(2 * np.pi * w[self.dim - 1])**2)\n", 51 | " return val\n", 52 | "\n", 53 | "f = Levy(10)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "## Create a Turbo optimizer instance" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 3, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "Using dtype = torch.float64 \n", 73 | "Using device = cpu\n" 74 | ] 75 | } 76 | ], 77 | "source": [ 78 | "turbo1 = Turbo1(\n", 79 | " f=f, # Handle to objective function\n", 80 | " lb=f.lb, # Numpy array specifying lower bounds\n", 81 | " ub=f.ub, # Numpy array specifying upper bounds\n", 82 | " n_init=20, # Number of initial bounds from an Latin hypercube design\n", 83 | " max_evals = 1000, # Maximum number of evaluations\n", 84 | " batch_size=10, # How large batch size TuRBO uses\n", 85 | " verbose=True, # Print information from each batch\n", 86 | " use_ard=True, # Set to true if you want to use ARD for the GP kernel\n", 87 | " max_cholesky_size=2000, # When we switch from Cholesky to Lanczos\n", 88 | " n_training_steps=50, # Number of steps of ADAM to learn the hypers\n", 89 | " min_cuda=1024, # Run on the CPU for small datasets\n", 90 | " device=\"cpu\", # \"cpu\" or \"cuda\"\n", 91 | " dtype=\"float64\", # float64 or float32\n", 92 | ")" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "# Run the optimization process" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 4, 105 | "metadata": {}, 106 | "outputs": [ 107 | { 108 | "name": "stdout", 109 | "output_type": "stream", 110 | "text": [ 111 | "Starting from fbest = 20.98\n", 112 | "50) New best: 15.65\n", 113 | "80) New best: 11.27\n", 114 | "90) New best: 9.325\n", 115 | "100) New best: 8.288\n", 116 | "110) New best: 6.944\n", 117 | "120) New best: 5.974\n", 118 | "140) New best: 5.951\n", 119 | "160) New best: 5.905\n", 120 | "170) New best: 5.905\n", 121 | "180) New best: 5.822\n", 122 | "190) New best: 5.785\n", 123 | "200) New best: 5.759\n", 124 | "220) New best: 5.738\n", 125 | "230) New best: 5.683\n", 126 | "240) Restarting with fbest = 5.683\n", 127 | "Starting from fbest = 32.5\n", 128 | "320) New best: 5.526\n", 129 | "330) New best: 3.95\n", 130 | "350) New best: 1.736\n", 131 | "370) New best: 1.229\n", 132 | "410) New best: 1.206\n", 133 | "420) New best: 1.193\n", 134 | "430) New best: 1.191\n", 135 | "440) New best: 1.163\n", 136 | "450) New best: 1.145\n", 137 | "460) New best: 1.06\n", 138 | "480) New best: 1.024\n", 139 | "490) New best: 1.01\n", 140 | "500) New best: 1.001\n", 141 | "530) Restarting with fbest = 1.001\n", 142 | "Starting from fbest = 12.85\n", 143 | "730) Restarting with fbest = 8.634\n", 144 | "Starting from fbest = 9.62\n", 145 | "890) Restarting with fbest = 5.87\n", 146 | "Starting from fbest = 25.71\n" 147 | ] 148 | } 149 | ], 150 | "source": [ 151 | "turbo1.optimize()" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "## Extract all evaluations from Turbo and print the best" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": 5, 164 | "metadata": {}, 165 | "outputs": [ 166 | { 167 | "name": "stdout", 168 | "output_type": "stream", 169 | "text": [ 170 | "Best value found:\n", 171 | "\tf(x) = 1.001\n", 172 | "Observed at:\n", 173 | "\tx = [-3.006 0.914 3.659 0.853 0.033 -0.203 1.199 0.812 -0.301 2.42 ]\n" 174 | ] 175 | } 176 | ], 177 | "source": [ 178 | "X = turbo1.X # Evaluated points\n", 179 | "fX = turbo1.fX # Observed values\n", 180 | "ind_best = np.argmin(fX)\n", 181 | "f_best, x_best = fX[ind_best], X[ind_best, :]\n", 182 | "\n", 183 | "print(\"Best value found:\\n\\tf(x) = %.3f\\nObserved at:\\n\\tx = %s\" % (f_best, np.around(x_best, 3)))" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "## Plot the progress\n", 191 | "Each trust region is independent and finds different solutions" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": 6, 197 | "metadata": {}, 198 | "outputs": [ 199 | { 200 | "data": { 201 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAFTCAYAAAAKvWRNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9eZgc9X3n/6qqnhuEEGYM2Bg8sjEGCc1ASeaIibHHmQchjAABtgNrYsdyNo5J7F8ikInxxhxBApIY40OK411WJEFIgPhJDDtGXswlIanRDEjCwobhcsAM1yCYu6tq/6iu6eru6u7q6auq+vN6Hj3qo7q7po7v+/s5v4plWQiCIAiCUDvUWu+AIAiCINQ7IsaCIAiCUGNEjAVBEAShxogYC4IgCEKNETEWBEEQhBojYiwIgiAINSZW6x0QhEqhKMoyYKFlWVd6vLcCGATmAFiWtTb5egfwDWAFsBtYn/zIYcBsYINlWVsL/K77OwaBNcBGy7IGy/BnlYxzXIAO4B8ty9pdw31ZA2BZ1jdqtQ+CEAQUqTMWooaiKN3AycDngcHMgV5RlFXALsuyNno9T772PLDGsqzVGZ99EFuQ1/rYD8/vqCWKoswGnrQsa25SlHdXa5KgKMryzOOWPFdv13JCIAhBQCxjIXIkLdetiqI41mwmyzOs5QeBK4GNHttmchHwjqIoW4Ni6RaJjm2t4558VIlTMl8o5GUQhHpBYsZCXaEoyskeL78NdPv5vGVZw8BWYFU596vWJC3mnM9L/e6kO3qOx+vdOc6JINQVIsZCvTEHW3zdDENRAvQgthu8ZJKCtEpRlGXJ/7uTry9TFOX55L+TXa9ZiqKsyfF+d/J9z4lCcruLgA5FUVYkt+9WFOVJ4Feu/dmAbf3Pdn3vk4qibHB9ZpWiKMs9fsP5W7qTbnCwJzpzgJOTv+t8bg52bH2V6/Ozk9ssS/5b4XrP934IQuiwLEv+yb9I/sMe5NdkvLYMeCfjtdmABXS4XnseWJHje5fbt07B38/5HRnbZP7u7OTjbuD5zN92Pc77fo7f6wYe9DgmT2a8Zjn74dpmel+dY5bxmSeBk13P33GeJz+/odD+JL9jdsb7a4rZD/kn/8L4Tyxjod4Y9njNcZ9mWsy5mJ3je4rCsRyt9NjzbpIucysZT3VbvsBdzoaF3i8CP3/LsHtfLdtdj8t6PhlbIN2JWKdYRSRmOV4B57uTj7cCy11ei7z7IQhhRcRYqDfeJjupazaki0AB5gLxme6Ay33bAQy7XK7dwC7SxXENtisXbLHL3MdVwMrk49lF/A0zId93TyeGOVjFJ7idjPeEaDj5/X72QxBCiWRTC3WFZVm7FUXJHMznYCdl+eVi4HMl7EZH8n/HunP/duZ+rAVeSMaBvcTtLmBV0jKtZXb3IKm/Ky/JOuzZHlbzdN13BrOp7d8mCBVHLGOhHlnrsk7Brkde4+eDyeSmGTfKSArR5yFVWpR8zXl/tju72Eplb6+xPMqAku/fBaya6T5hC920t2Am2c3JfRt0XM3J7+lwfZdbrDPd2c53bARmZxyPZQSoYYogVAqxjIXIkRSAbuxknznJ5htbHQGwLOtKJ2MXWyCezxDGbyRfv0RRFOdrnZplT1HM+P1c37EwuV9uF/fngJWKouxyXrCy63//kXQ3bSZrCrzv7NfJ2PXUejJLea1lWcOWZQ0qirIxmZX8NrZwDmNb3Fcm/44rSWZDY1vrjmt8laIoq5Ji+bnk847k9ww7xyrpkYgnPz/ssT9OQ5BTXMdjDrYFfVHG9oX2QxBCh3TgEoSQoyjKMg8BFwQhRBS0jHVdn41dyjGMnbhCPB6/MmObtD6/8Xi8YKtAQRBmTtIy3F3IShcEIRwUtIx1XV/lFl9d158E1jiCq+v6KmBXPB7f6PVcEITyk4zNdkBqkQtBEMKLnwSuZbquuzvcDJJMQEmyPEN4HyRViiEIQgWwLGurZVlrRYgFIRr4SeD6fDwedydFdJBcVk7X9ZL6/AqCIAiC4EOM3ULsiG88HneWhMvZ51fX9dnxeNyzOP+qq66SrDFBEAQhlNx4441K4a2Kw1dpUzKJ62LsJvNfd701m+wifUec55CnU86Nq1Yx+etfY516qv+9ddHbq3LZZTFGRlLHpK3NYt26BIsXmzP6zrAzNDREe3t7rXej4hgG9PWpPPWUwoIFFj09JppW2d+sl2PrxjnOAwMKnZ2VOc71cFwNA5YsibFzp8roKLS2wqJFJlu2JKaPp9d41tpqMXeuxeCgwugotLTA3Lkm559v+Tof9XBs81GpceL73/9+6V/igS8xTlq4a4G1uq4/qeu6k8BVWp/fIsqqMgeG7m6TRYvMrAu8p6c+hbie0DRYvNhk8eJa70l08SMggj/6+lR27lSnhXZkBHbuVOnrU6cNh4EBW3DdjI7Cb36jkEgo088HB1U6O+vX4CiGsI0TvkqbMtzNa5L/1pKnz28uF3UaPsU418Bw330Jtm5NzXy6u82Kz+QFoR7wIyCCP3IJ7VNPKdNC0dlp0dpqH2cHRYFEIv/nhOiQN5ta1/Vu4J2kmzrzvdnxeHw32dax/z6/PsXYPTBYlsLIiMLOnSpbt9oDw8qVBj09JuedF+Oyy2Jce63GZZfFWLIkhmH4+glBEFzkExChOByhddPaCgsWpMa/nh7b09fWZqEoFk1Nznvpx7upKf1zQnQoVNoUB9ZmWLmfBza6Xlur6/qM+vz6FWM/A0Muwe7rk/bbglAsfgRE8Eem0La1WVkhNU2DLVsSrFuX4JprDM4/3/QYHi2OPNKSUFxEyeumjsfjw7qur0l22AK7P++guwlIPB6/Utf1FUlB7gCe993ww6cYe7lwMgcGL8EeGYFbb7XFWFzWguAfR0AkJ6N0HKEtlEzkjnH29qps3qymjXlNTbB6tRHaccxvQmA1EgeDiJ/Spt3YC57n22Z1vvdzYvq7sf0MDF6CDfDwwyq7dqmSfCIIReBXQAR/FJtMlGvMC2u83m9CYD0nDtZ21SaflrGfgcF98aYEWcGyJPlEEGZC2LJRo0TUJkN+EwLrOXGwtgHVIkqbnIFh5UqDxYu9XTxOzOUznzFRMvJMJPlEEIQwUWjMCxN+EwLrOXEwFJaxX5yLF2DXLjVvjFkQBEGoDn7yforZLoqExjIuBj/Zi0K4MAw7qeWGGzR6e1UpWROEEOF3TK7nsTtSlrFD1OIt9U49J3UIQhQoJqO8XsfuSIoxSPJJlKjnpA5BiAp+x+R6Hbsj6aYWokU9J3UIglAf1FSMFRFjwQfSDUoQhKgTOstYEnnqj3pO6hAEoT4IVcxYEnnqk3pO6ggL9drCUBDKRajEWBJ56pd6TeoIAzJJFoTSCZWbWhJ5BCF4yIppQrFIuDGbUFnG9dydRRCCSr5JsngyhEzEk+JNbaeuPldtcpBEHkEIHpLtLhSDeFK8CZVlLIk80UESfqKDrH0sFIN4UrwJlRiDJPJEAXFTRQuZJAvFIOFGb0InxvkQayscSFZ89JBJsuCXIHhSgqgVkRFjsbbCg7ipBKF+qaUnxcniXrFC47XXFCYmgqMVoSptyockBYQHr4SfxkZ49llFyhwEoQ5wPCkrVxosXlw9IV6yJMall8Z44QWF8fFgaUVkxFhqkIOBn/pBd1Y8WKiqxdQUrF+vctllMZYsiYkgC4JQVhyDbWJCAdJ1IQhaUVs3dZGlTfmQpIDa4zdU4HZTbdigcu+9KuPjEj8WBKFyeBlsDkHQishYxlKDXHuKCRU4bqqPf9xiYiL9vdFR6O9XpEOPEGqky1Sw8AqPgUVzczC0IjIJXFJeUXtmkpjl5dFoaYFNm1T+6Z8UScYTQokklFafQhnS7izukRFoaoKjjrK46SaDs8+uvVZERoxByitqzUxCBV5lDh0dJoODUvokhBcp36sufiY/QTfYIuGmFndQMJhJqMC5QdatS3DNNQbr1iVYutSSZDwh1EhCaXXxGyKrRRa3X0JvGTszoh077BlRQwMcf7zJ448naGwswz4KvpnpzDPbo6FKMp4QaiShtLpEoXdB6C3jvj41KcR2uvrUlMKePSpnnNEgFnINKMfMU5LxhLAj13B1icJiJaEvbfJOV1d49lkkPhMQ3IkV8+dbKIp93kzTFu+urnQLOuixHUEohFzD1SUILTZLJfRu6s5Oi4YGmJpKf31qKlwuirDht7erO7FiZARU1T7t7lPf1uadbCHJeEKYKeYaDmKv5CDjdbzCPvkJvRj39Jgcf7zJnj0q7q4q7vaKYTspQaeYso3MrFIvZ4hkmgr1jJRBFUe+4xXmCXzoY8aaBo8/nmD+fIvGRmmvWA2Kae6Rr+uNG8k0FeoV6avvTa4qmager8CKcTHlSo2N8MQTU9x5Z4IvfcmksRFMM1onKkgUU7bh3fUmm7AlWwhCuZAyqOzxfnLStn4vuyzGtddqaUaV1/EaGbG79oWZQLqpZ+K2ceIzAwOKZ3tFiR+Xj2LKNjK73uSLGYcp2UIQykW9l0F5jff5Gv90dlq0tJAlyJs2qVx1lRFa134gxbiU7jX1fmFXg2IyFzOzSufNs7Opn3pKwTAgFkMSVoS6JgqZwKXgNd7v36+SSKRv5xhVK1YYzJ2bmSekMDgY7gqaQIpxoQLufJmH9X5hV4NCZRte5yczsWLJktrsuyAEjXovg/Ia76em7PDj5GTqNbdRdcIJsGdP+mfC7gENpBjns24LubDr/cKuFrnKNiQzVBCKp55L+XKN93PnWgwOkjaOdHebLFkSY9u27BygsHtAfYmxrusrkg8XArvi8fhq13vLgA5gI/A2sBzYGI/HBwt+cQ4xzmfd+nFh1/OFXWukQb4gCMWQa7y/774EW7emG1XO+DIx4U7WsmhuDr8HtKAY67q+Jh6Pf8P1/Eld13EJ8hxgVfLfMPB1X0IMOcU4n3UbhR6kUUbOT/0ijSuEmZBvvM80qnKVSp5/vsnPf14e71utruO8Yqzr+mxsgXWzBlt4V7teOxSY41uEHZJinOuP97JuJUEr2Mj5qU8kPCGUgl9vptf40tYGF11UHsGs5XVcqPh2DrBC1/WOjNdnu5/E4/HhooUYwErFgL3qybyQBuzBRs5PfRLVRgxCsKj0+PLAAyrbt9fmOs5rGcfj8UFd10/JENrPA1vd2+m6vhw7XjwHmO2OKefj/QMH2LT+ADt2HJZcdcme8ezYobB+/QG6u8c9P/eLX8BDDzXzzDONnHDCJGedNc5bb/n5xWgzPJzpxKgNUTw/QTm2QeXxx2cxOjor7bXRUdi2bRRdP5Dzc3Jci8Mw7Htr375GTjzRvrdyWWxRPbaVGl8MA77znSMYz5AdP9dxOSgYM47H47udx0m3dTdwimuTrcDb8Xh8OLnNGl3Xl8fj8bWFvvvgtjZefPFQxsbSO6eMjSm89NKhtLfnbrv15S87j5qBWTm3qzfa29trvQtANM9PUI5tEDnjDJWf/Sw7PHH66a20tzfn/awc18I4HapWrNB47TW7sZEfF2pUj20lxpfeXpXXX4/hXuMAoKnJ33VcKsWWNm0APue2lD3c0w9ix5QLijGWJTFGoSCSGBR8pL6/cjihvG3b1GR3QalUqARe3RvB4sgjrapcx77FWNf1VcAqD0v5HeBQxzLGTvjKjDF7Y1lyEwt5kcSgcCD1/ZXDu5zHRioVykdnp0VbW7ph2NQEq1dXp8Wm3zrjZcCD8Xh8a/L5yS5RXu0SYrCF2Hdpk9zEQj6kbjk8SH1/Zci38pl4EctHLsOwWuOMnzrjbuzErK1JS3gOcAmwOx6PD+u6nhk6vwi40tevJ0ub5CYWciF1y0K94xXKi0qjiyBRa8PQT53xg8mna1xvbXQ9Xpvs0DUMzAXWxONx9/u5KcN6xl5IjDE6SE6BUO9krnzW1ARHHWVx000GZ58tY1s5qaVhWKi0aZjM1DLvbXyVMmVRATGWGGO0kJyCyiIT1+BTa4tNqA61XSjCLP+AKjHGaCEDUeWQiWt4kFCef8I6wQzkqk2lIDHG6CEDUWWQiasQNcI8waxtr7oSxNgpgr/hBo3eXnW6faYTY3QjMUZByCbfxFUQgkqusR/C3ZY1lJZxvtmPxBgFwR+SHCeEjUKWbzGe0aC5s0MpxoXcaxJjFITCyMRVCBuFxn6/E8wgurNDKcaFZj8SYxSEwkhynBA2Co39fieYQcyXCGU2tbjXokPQXEX1hkxchTBRaOz3O8EMYqJvKC1jca9FgyC6igRBCC5+xn4/E8wgGnShFGNxr0WDILqKBEEILuUa+zNFvaUFOjpM+vsVQK2JnoRSjEHca1EgiK4iQRCCTTnGfreo9/crbNqkMjioct11tfPQhbbOOBf5atCEYCE14YIg1ApH1Lu6LAYHlZrXJofWMvZCYpDhQmL/giDUmqB46CIlxhKDDBcS+xcEodYEJZkrlKVNuQjKDEfwj8T+o4eUq3kjxyWYBMVDV1sxLjNBmeEIQr0ioSJv5LhUH7+Tn6B46CLlpg7KDEeoLmJxBAcJFXkjx6W6FDv5CYKHLlJiHJQZjuCPcoioWBzBQkJF3shxqS5hnPxESowhGDMcoTDlEtEw3nRRRkJF3shxqS5hnPxErs5YCAflWndU1uQNFk6oqK3NQlEs2tosCRUhx6VUiu0fEcYeBpGzjIVwUK6Zq1gcwUJCRd7IcZk5M/GihTF/KFKlTUJ48BLRlhaYmoIbbtB8x5DDeNNFHQkVeSPHZWbMJBQVxsmPWMZC1XAnbM2fb7FwocmuXalG7U1N8MMfakXFkMN40wlCJchMiOzqqvUelYeZetHCNvkRMRaqgperaeFCk9tvT7Bnj8LUlC3EM0nECttNJwjlxuv+6uw8nF/+ktBPTOslFCUJXEJJ+E2s8ErY2rVLRVVh5UqDWAxJxAo4sghLcPG6v/r7G6u+2EElqJfkt5paxoqIcagpJrHCy9U0MgIbNthrh9bL7DesSD137fBTj+91f42NKYEu5fFLvYSixE0tzJhiEiu8xBbg3ntV/vCHGPfck6Cjw2L/fkgkJBEraEg9d23wOwnyToi0IjOZrYdQlLiphRlTTI2v42pqarIA57wrjI8r7NihcuaZDTz/vB07jsWgo8PkvvvE6goKUs9dG/zW43u5cru6JiMzmc0XIjEM2LxZ5fLLY1x+eYwtW8IZQgl9aZP0Ja4dxbiWHVfT174W48470weS0VF49lmFyUl7YJ+agsFBla1bxeoKChJGqA1+M4m9XLldXW+gae3V3eEKMDkJZ5zRwP79SprXbMuWBADnnBPjkUfUaTm56y6VM880uf/+cE3mayrGL78EHzRmnu0ncazaUmyNr6bBxRebbN6spg3qDQ22ALsJeuu6ekPquWtDsRNetyt3aKhKO1lBDAPOOCPGnj0KkB0iAdi+XcU0Ux4a04QnnghfCKWmYvz61r30LvgxX/+6iToDh/n2xKns3HE6I6MSx6oFM0ms8BrUOzpMBgdVsboCTL0k0QQNP5OgKHsH+/pU9u9XcYTYwZmsWxZMTGR/bnw8fJP5morxqeZ2Tn1uO1w5s89/BljAY2zjjOnXxKKqLsUmVngN6t3dJuedl+3hEKsrWNRDEk3QKDQJyucdjAIDA0qW1wxsb5ozWW9qyhbk5ubwTeZrGzMuA59teJRtUykxFosq+HgN6mJ1CYI3+SZB+bLcdb3KO1oBHDd9etzc4hOfsMcIw4CPf9xi375UPrCqwqmnhm8yXzMx/hf+GoCGmEVPj8WxH/UvoOr27ahPPgnAMR9O0DZkiUUVcsTqEoTiyZfgFQUx7ukx+dSnTHbssC3/hgY4/niTxx+3Lf/zzosxOGi7qzUNPvABi1tvNViyJHyT+ZqJ8XeUf54Wzz+7K4FRzIH7/venxfi/XZbgsM6EWFSCINQdUc9yz+em7+1Vk+552ytgGPD++7Zgh1EDaibG11xjzFw8XdleqmmIRSUIQl2SL8HrrbdqvXflIZfXrFzLsAaFmonxypUlVGW71dtV3R3lrELBRs5xMJHzUhvqOcs9al4BX2Ks6/qK5MOFwK54PL7a4/1BYA5APB5fW86dzMJ9pSUrvaXmOPrIOQ4mcl5qS73mW0St9r2gGOu6viYej3/D9fxJXddxBFnX9VXYAr3Rea7r+jLneUVwFyUnLWPpnRt95BwHEzkvM6cYj4J4H9LJVSYZ1mOUV4x1XZ8NDGe8vAZYBTjW8fJ4PO6uFH4Qu3K4cmLsYRlHLX4gZCPnOJjIeZkZxXgUxPvgjdsrEPZjVKjv1Rxgha7rHRmvzwbQdf1kj8+8DXSXYd9y4yHGTvzATWMjzJsXzvhBPVDs+rhe5zjMMaKoIOdlZvhdBKLYbeuVsB+jvJZxPB4f1HX9lHg8Puh6+fPA1uTjOdji62YYbKs6Ho9nWtXTDJXQOPWg0VEOTT4ee+89hoeG6OqCBQsO54knmqYbhk9NwT//s4GuvxGKmVGpDA/nPNyBwzDg0ksPp79fY2xMoaXFoqvL4I47cp+rri7o7Dyc/v7G6c90dk7S1fVGxfvwhunYVptSzks9H9fHH5/F6OistNdGR2HbtlF0/YCvbR97bJTh4Un27WvkxBMnOeus8en7p96Obb7j2dV1gIceavY8TkGhYMw4Ho/vdh4n3dbdwCnJl2aTTNpy4YjzHLJd3NO0t898NRHtkEOmH7c0NdGY/K7vfEfl0kthfNyOXZkmPPVUE/39R9RN7KqU41pNentVBgZi0zWCo6MKAwOFz9Uvfwl9fYYrc1Sp2so0YTm2taCU81Kvx/WMM1R+9rPsbODTT2+lvb254LYtLfDggwezZo2S0y1bT8c21/H81Kda+epXDwq8+7pY+30D8DmXpewlto44Z1rMZcNyHUHFtQzj008rWT1KZc3V2lDIBT3T9XGdGNHKlXZ9eZBupnpGzkvxeK1BnCsb2GvbuXNNBgeV0Lply4lh2P/a2y2am9OPp6IQCve17zrjZNb0KreljC24szM2nQ2Qz0VdMh7Z1BC9urOw4ieRotRz5Yj93Xfb18KyZSZnny0iIISHYmqEvbbt71e47rr07eoxcc493oyM2AtHHHusxU03GZx9tsmqVVooEgz91hkvAx6Mx+Nbk89Pjsfju+Px+G5d1zNFdw6pmHJlyCHGUas7Cyt+Sl1KOVeGEZ0FxYX6ppga4extVTE+sMcbu3e1Pd5MTMDrr9syoWnhMdIK2um6rndjC2xc1/XZyczqS1ybrE2KtcPnscufKodHNrXz8pYtCdatS3DNNQbr1iUCFxeoB/y4oEs5V319qmtBcfufaSrTC4oLQtTwCvt4ua4XLjQxTbjhBo2tW5sLVihEgd27vceb/n57vCkmHFBL/NQZP5h86hbY6RrieDx+pa7rK5KC3AE8X9GGH5DTMob67UYTJPzORGd6rgYGsnMDIJwLigtCIfKFfdyu63nzLH78Y5WvfCXG6Ci0tBzGunVW5A2SRI6lmx1pCEvL0EKlTcPYpkdeMttjVpwcvamFYFDpcEFnpxWZBcUFoRCFwj7OhLa3V2XXrtR2o6MKO3dake6EZhhw553e3rCYS93CYKSF06eXw00tBINKhwt6ekxOO81EVS3A/qeqVigXFBeEQvitPJhphUKY6etTee01J1yVoqnJnrSHiZqt2lQSedzUQjCo5ExU0+D++xP09qrcc499LVx4oWRTC9HEb9gnLIlK5cQ7ZGVx1FFW6Cbm4RRjsYzrHk2Dc881OfdcOf9C9HAvCjF/vp2YtWtX/rBPZniopcVi0aLwiVIxdHZatLWlT0CammD1aiN0E3MRY6EqyIozguAPr4SthQtNbr89wZ49uROQMhOVjjnmHS65ZFZk7zN3o4/XXrNzSJyJShhj5OEUY3FTh4pCTUBKEWoReSFqeCVs7dqloqp2h7N8uMNDQ0PjaNqsvNuHlUKNPsI4BoRTjH1mU8tAHQzyZYP29JgzXvYs7Eum1QtyHxaHLElZmMwxZWIChoZSjT7CSPjFOIebWgbq4JA/y3PmC9PLovbBR+7D4qnHRKxiqfSEpRYTyHCWNrnc1EoOy/iBB+wuTUFvDl4P5FvvtpRyjHos5QgbYV9jthaEpWNULankGtrOBPKyy2Jce63GZZfFWLIkVvGIaCjvCKuAm9ow4O/+TmN8PP11GahrQ77BpZSbSha1Dz4yYSoeaetbmJlOWAqtJge1m0CG003tTuDycFP39am8+qp3IbgM1NUnXzu6Qt268rmLZGGQ4DMTl6vEmMPRMaqWzKTFpd+QSa1i9uEXY4+pTa5C8COOsOjuNpML29fvjV4Lcg0u+W6qQjdPWHrO1jPFTpgkxiz4pdgJi98ck1rF7MMpxgXc1M7BzJzdtLXBF74Qyyqelxu9tuS6qfzcPGJBBJtiJ0ySlCdUCi+Ld2QENmxIN85q5XELvxh7uKl7ekzmzjXZs0cl5apWeO45eP55hfFxudHDgJR4hJtMd/OKFYW7Isk5FyqFl8WrqnDvvWpaw5DM1bCq5XELpxgXiBlrGixdarF3L1guz4LXsntyowcXKfEILzN1N8s5FypFpsXb2AhTU+Q0zqrtcQtlNrWfph9dXdmZtk1N9jJ7buRGDy5S4hFeZpqRKudcqBSZWernn2+mGWtQ20z/cFrGPpp+ZDdNh44Ok/ffV/jDH9L7mMqNHkwkQSu8zNTdLOdcqCTuHJPeXpXNm9XAeGHCKcY+elO7b+r+foVNm1QGB6PTxzSqeJW1SIJW+CjF3SxJeUI1CFppZDjF2GdvauemBpV/+iclUn1Mo4iUtUSHoA10gpBJ0Lww4RTjAglcmUiGZjiQspbw4/Zs/OVfmnzzm2beZf9yfVZ6AAjVIEhemHCKsU/L2EEyNMOBTJrCTSmeDfGKlIbXREYIF5HIpi7Ub1QyNMOB9JoON6X09PX67GOPqVx/vSZLlhegVgsbCOUlnJZxhpu60Iw6MzYwb56FosCqVZq4wwJErgz4/n57qUU5T8GmFM+G12enpuDmmzW2b1fEQs5DrvDOQw818+Uv13jnBN+EUowNl0E//JbJ9lfVgl21nNhAT4+4w4JKrgz4666T8xQGSgkHeX0WFCYnJW+gELkmQc8801ibHRJmROjc1IYBXwJi7/wAACAASURBVP160/TzOe++yPh4+s3uLtzOdGE/8ICsrxpknElTV5fF4KAi5ylElBIOcj7b0GAB6ffzyAhJ74jghVd4JxaDyUlfKTWRw88yiUEkdJZxX5/KroGmtNc+xU52cOr0c2epRK+kkPZ2S5KEQoAkcwWPQtnOjmejt1flnntULAuWLfNnzTqfvf56jZtu0piaSn9/0yaVq64q3Nu6HnEmMjt2qNP3zNQU/PznB7NnjxUpb1KhazDMiYChE+OBAYXBsSPTXvsYz7nE2OLII+2T5BVLeeUVW6zHx1OflySh4CEZ8MGimEHuJz9Rp7fbvFn1PRhqGlx9tcHmzUrWIi+Dg4irOgPHArz7bpXDD7c9SvfdpzI1ZR+30VGFnTutyBw3P9dgmMsjQ+fz6+y0aG1T+Dlfm37tIN6fftzUBKtX2zPogQElIwYFiYT9v2RWBxvJgA8WfjOlS8mohtQiL0qGV7qWPYODiGHAOefEuOSSGHfeqXLXXSobN6pZHoUoHTc/11Y+j1rQCZ1l7AzSI48cDMlYQBuO4locd5w1PQPq7LRoaspcrUnBsiz++q8NGhqoedcVwZugdcepd/yGDXJt52TEDwwozJ9vi+3TTysce2wzl1ySXq3oLPIiXpHc9PWpbN+uYpqZIpN+jKJ03Pxcg2H2qIVOjJ1B+oFFbbDPfs1tGS9dmhqwe3pMjjrK4oUXIOXyshMbGhpg5cqQRPYjTL4YUJC649Q7fgc5r+1aWuyYr92SNlWZaFnQ0nIY69alxzWllWZhBgYUzyVhU+OcRVOTxaJF0WkA4ucaDPO1EzoxBnuQ/kRX67QYO5ZxW5s9q3Zvd9NNBpdeGpMYcQAJc7JFveF3kPParqPDTC7SYguFu4OtV1xTvCKF8fb6pbN48Rh33BGLzHHzcw3O9NoJQivWUIoxwHEnt8Ed9uODeS9nTPHss01OOy2cM6WoE+Zki3rD7yDntV1/v8J11+X+bi93t3hF8tPTY49rjzyiuiY3Ke9fWxuce+4omjarJvtXCYq5Bou5doJiFIRWjC1XYd2Jx45w+80Jz+UQZZYdXKR8KVz4HeQytzNNNauCwY14qopH0+D+++0yso0bVX79a5V337XS1mk/66xxIDpiDJWZpAXFKAilGBsGrP7RIfwP5/nLr/LAqj0s/pCB4iGyMeCcjkYWn/0JDFOpuTtCsAlzsoXgD8OA225TmZwEd3KRqjox42jFNauJpsG555qce6457WZ1GxxvvVXrPSydariPg2IUhFKM+/pUnnruoOnnZ5m/4qxdp8Bp+T9nnPVZzrb62LlLkxhlAAhzsoXgj74+lV270rN+GxstvvMdg6YmOOaYd7jkklly/5VIFN36M3Efz0S8g2IUhFKMBwYUfjtxTNGf0x76v7za8l+MjNmflRhlbZEQQvTJtQBEUxOsWGGwfr0s2CJ4U6z7eKax36AYBb7EWNf1ZcDCeDx+pcfrHcBG4G1gObAxHo8PlntH3XR2WtzSdgIrR27gIjagYaCq8JGjLQ72CJEo+/ejJKvhzbH09EOJUdaWKM7ohRS5rI558yyWLImxY8dhjI0p4qUSsijWfTzT2G9QjIK8YqzrejdwMvB5wEtg5wCrkv+Gga9XWoghNZP50c6rWDV6VdqNPOVxABvmz0f53e8AOKjFgLHUexKjFITKkcvqUBSSr0km/UwJQjlOJSnWfVxK7FfTSFrC9vGsxZKtecU4Ho9vBbbqun4YMDvHZocCc6ohwg5Fz2RcbyyYn+C3+yyJUQpCFch1r65apeUdOKMoNOX8m4JSjlNJinUfFxLvfMc/CMez5JhxPB4fxraKq0pR7k011bv0Jz+a5Av/lZAYpSBUEGfg271bwTTt+7WrK3W/5Rs4gzAwlpty/01BKcepJMUaXfnEO/P4t7TA3LkmS5dadHVZmCY1P54li7Gu68ux48VzgNnxeHx1yXtVblxnT1NMiVEKQgVxBj73kn5gN6JwBCi17J+SFjPOtdpa2IWm3H+T1yI4IyPRy38pxujKJ969venHf3QU9uxR2bs3OEvrlirGW4G3k9Yxuq6v0XV9eTweX1v6rpURl2UcmpWmBSGkOMLjxIMdMgVoy5YE69cf4KWXDk0bOINS91lOyv03zZ9voarprUVV1U6Mq2dyibfX8bcXDbKvy1dfrf3SuiWJsUec+EHsZK6CYjw0NFTKTxfFB02TxuTjd958k8kq/nY1GR6uerSgbpBj65/HH5/F6Kh356fRUdi2bRRdPwCArg/T3W2PgE6TimOPbaal5bA0MW9psTjmmHcYGsrRxivglPtvevfdZuADaa9ZFjzyiH1sNU2uWTdex9/NxAR85CMJ3nxTY2xMoaXForNzkq6uN6iWXMxYjHVdnw28AxzqWMbYseMOP59vb2+f6U8XTay5efrxoYccglXF36421Tyu9YYcW3+ccYbKz35GlhsVbGvj9NNbaW9P3ZOZx/WSS2DdOoudO92JllayOUg42zuW+296+WUNK8NosyxYs2YWe/cexJYt9sLtcs3auI9/6rpM7+V9yy2gaYbLxa2gaVXUqRI/v9olxGALcdWyqn0jbmpBqBqpeLB3zLhQ9UJQ6j7LSbn/ppNOsjz6fStMTqZCAbpejj2PBu7j39+vsGmTyuAgaYleixebSTd3bfZxxmIcj8eHdV3P7H56EXCl1/Y1xS3GZjgTQAQhLGQOfIYBsRhFlfNEsRlMuf4m737fKSvPiUWLGKfjPv5XXWUEbrJXqOnHyUA3sAyYo+v688DWeDy+O7nJWl3XV2C7p+cCa+Lx+MZK7vCMEMtYEKpKFMU0KHj1+3YjjYwKE8Trs1DTj93AbsCzXCnpog5eKVMm7imPiLEgCCHGOzPYFl93KCAKqzbVE6FcKKJo3GIsbmpBEEKMV8OUpia44AKTiy4yA+FyFYqnPsRYYsaCEAqi2Aaz3OTqNPXzn4e3Q5lQL2LsukIV00SiKeFCBuho4HUeM9+PWhvMShDFbPNaEpTxpS7E2HId2bv+06LVqP6KHMLMkAE6GuQ6j7/4RWqbKLbBrBRBTEAKI0EaX9TCm4QfS0n9mRvutLjsshhLlsR85XIZht3X9IYbNHp7Vcn/qjLuAdqyFEZGlOkBWggPXufx0UdVrrhiDtdfb99bTz7p3W/ZXtJOEMpPkMaXurCMh97UOCr5WMFKO+D5ZtxBmjXVK1HsU1yPeJ3HRAI2b25l82ZoaIDZs73vxUSiCjso1CVBGl/qwrwYPuBatQnbtHUOeD6CNGuqV5zMUTdSRxk+OjstWloyX1Wm/01NKbzxhvd9Vc8TX/HMVZYgjS91YRnPmpO6yR0x9nPAgzRrqleKXWBcCBZOckw8rqAo4NUxKoXiet+mtdVeB7keEc9c5ck1vnR3m/T2Vjepqy7E+Iij0sW4rc3yNaDnWwBdyE+5MhQlczS8uMXEqzm/N+73LebOzc66rhckoa3yeI0v3d0m551X/UlQXYixoqXE+IxTE1jHmixbVvhiFqtsZpR7Ru/OHA1KGYJQmEwxmQlLl9bv+RXPXHXIzEzv7a3NJKguxNh9Nw/stli/Q2XzZrWgQIhVVhyOUN51l8r27Srj4+W9mMVtFy76+73aNuYi233d1la/LmoQz1w18Jrc12oSVB9i7OrAlZg0sVB8C4TU8/nD2yWZYqYXs/tmSSQQt10FqIS3wTBg0yYla83dFBaqajfEa2qCI4+0OPhgspa1q2cvlHjmKkuuyf03v2nWZBJUH2KsZWdTg7h8ykkhl6SfizlTFDJjN7EYTE2lf0bOYWlUytvQ16fy/PMq+WLEF11kcvTR73P66a3TAiNeqBTimaschgHXX6/x2GMqU1Ppk/u//MvaTILqQowtNXX1qqQOqDO4G0Z9l0+Ug3wrybS2QkeHSX+/Anh3P/MShY4Ok8HBlMDbQpydbStuu5lTqSShgQGFsbHc77e1wSWXmOj6Adrbm6dfFy9UOuKZKz/OWGMLcfp7o6Owd69Sk0lQ5MXYtrZifCH5XCOBM6BPTcEPf6ixfbsicccSybWSzNKlJs88ozA4qHLddbktLy9R2L9f9Wz40NhoMTUlbrtyUKn42EknWTQ1wfi4+1ULRUk/b7LMX/nI9Cx1ddV6j4KJM9Y4FrEbZ3Jfi0lQ5MW4r0/ljSG3ZZyeKCJxx/KQK7510UUmX/lKrKDl5SUKU1O298ItyK2t8Dd/Y9DQgLjtykAlkoQMA267TWVyEpyJr6LAiSeaXHCBJVnwFcDLs9TZeTi//KV4/TLJ5cVrbKzt5D7yYjwwoHB4IrvphxuJO5ZOrvjWqlWaL8vLSxTAHmRU1cKyUgJ/9dWGDDBlohJJQn19Krt2qZhmyvLQNIsTT4T58y1ME/7xHzVME8bGZvHpT5e+cEu9l7x5eZb6+xvp6zPEyMjAa6xpaIC//VujpmNL5MW4s9Pi1ZgGSevKS4wl7lgevFw7fi0vRxS2bVOZmACnTaJl2W5pWTi9MlQiSShXH+r16+2yt/QM61msWQNz55osXWrR1VX870vJm/cxHxtTxMjwINcEtNaT/MiLcU+PyZYjVXjFft6oGTQ3AFhMTEjcsdL4tbwcUfja12LceWd6j+LJSTjuOEtm+BWi3PExby+HbbF5lTqNjsKePSp7985MSKVTlfcxb2mxxMjwIKhZ6pFf8UDT4PwLUu6ymGIwPm4PCscea3H77Ym6mkFXG+fCX7cuwTXXGKxbl/t4axpcfLFJW1v66+K5CBfOBKytzcKOGfs5dzNfjCVfElrUyLVwhPuYK4pFW5tFV9ekGBk5cCagK1fabvwgjP+Rt4wBlIbUkTYSFqAwMQFDQ3Y/kCCciCiTaXk5A4pXfE8aHYQft+WxYYPKvfeqGVnV+RkZgQ0b/Md/66VTVSF3fKa119X1BprWXuvdFnxSF2Ls7sAlTT9qS6EBJaguJKE4nAlYT4/JH/6Qb7GI7DaYqgr33qumhZHyea/qZQJXyB2fOekdGqrhzlaJKCXu1bUYR3H2HHT8xPek0UF0cE+u+vsV7r1X5bnn7PrjWAwOOcRA0zQOHLBzOBob7ZK2Yvqa18sEzssdb2dN16dBEbXEvcjHjIE0P3RzgzEdU4ni7Dno1FN8T7BxJldXX22wY8cU//EfCb73PYNPftJibExlaCiVw7F0qZmV5OXn+ghiDLDcdHZatLRkv75pUyp2XE+4J/YzzTcIEuHc62JxWcYXnDeVlUiUKylCKD9OfM+NeCjqB0c0u7osBgcVRkft/tUTEwpDQwpz52ZfH+62tfVMT4/J3Lkm6QlxCoODSmgFqBSiNrGvjzPomiZ/7KOJtNmzYcA558T48pdj/OAHGl/+coxzzonV/Y1fKbyyPsVDUX/kGkg1zXY1tramMrGdtrVLltT3falpsHSp3VLUTZgFqBSiNrGvCzE2lZQYDz5npd3QDzyg8sgjztq7CuPjCo88ovLAA3VxaKpOMaVOQnTJNZB2dVls2ZLgb/7GoLERnOYvflyQ9eDh6uqKlgCVQtQm9pFP4DIMWHd7jG8knz+y+T36Pj/E//7fCdQjP8jGjTHMjHNnmnD33SpLloTzpAYdSdASnIF0xw6FsTElLQNa03Ivl9nfr2AYdsnUq6/aPa+POgouuMDkpz+123BGIZknF/WSOe6HqCXuRV6M+/pUXvx96s/888Qa/nzbGvgYWMccw6GdjwIfrt0OCkId4gyk69cf4KWXDmXePNv9umqVRmenxUkneXWUskuerrtOyUryWr9eRVHAsqLdhStqAlQqUZrYR16MBwYUXp083PM95aWXWH7WBtao306zjlXVnmkLglBeMutCzzprnPZ2g3POibF9u11b3NQEp55qsnBhygLUNGhosHj2WWVacDOxMhQ6qn0EoiRAQaRWtcuRF+POTouftl7IhtEtnMHjAMziAAdhT7lP+MgBzjzTTBsITjvNjNRsOixEqYBfyCbXMn9XXGHnbTirPE1MwKOPqtxxR4I331TYt0/BMODddwslKaW/H4vBs88q9PaWviqUUB/UsnY58mLc02My71Mt/NnOu6YP7q3tP+CrL/wPAJTxcb75TZMjjrDjTxdeaHL22XLjVpuoFfAL2eRa5u+22yzPvI2f/Uzld7/LbQnnx5peKWrzZlWuJcEXtVx0JPJi7BVjOWdfA3zPfv9/rknw17fGplvv/eEPcPbZYhVXG1l5J/p4lzMpvPyy9/bvvKMkl9PMJDNzOJfbWq4loTjy1S5XOixQF/U7md15aGqafm/83QnGx6PRwSXMRK2AX8jGq5xJUeCll7LPsarak+LMmlqw3c9/9EcmRx/tJax2mUsmUbmW6qF8q5bUsnY58paxF/ueb6Er+biZ9Km331mQxDfLS72svFPPZJblNDbaa1VnW7YWJ5xg0dVleax/bPGJT1jEYvDmm+r0a6nvUNA0+333SlFRuJbyhXIgezwSiqeWpWN1KcYvvtY0LcZNGWLs56aV+Gb5kfrJ6JMZMnr2WYU77/T2Qp13nsl3v+t9M82ZY7Fzp8rYmLelm0jA0UdbDA0RqWspVyint1flJz9Rs+6dX/yixjscQmpZOuZLjHVdXwYsjMfjV3q8twIYBOYAxOPxtWXdwwrw4bmN049TYmzR0AALFxa+aSW+WX6kfrI+cJfl9Paq3Huv3ZfaTXOzvXDEq6/a3bcyefxxNSvhy01bG6xebaBpROpayrVq049+pPLkk9nj0UMPNfPlL9dgR0NOrUrH8oqxruvdwMnA57EFN/P9VcCueDy+0Xmu6/oy53lQOWlRKmbczDhOQoiq2oPAAw+oPP10bvdzLYP8UUbqJ+uLnh6TU04xeOKJpmlxVVW7xljTyJG8pSS39fJe2RPqjg6Tp59W6OqyWLHCDqpGIaTkFcoB2LZNJZFIf210FJ55phEhPOQV43g8vhXYquv6YcBsj02WZ1jLDwJXAoEWY7UlJca2ZZxe37hjR/6FzSW+KQilo2nw7//+BvH4Edxzj+2udkoL+/pU2tqyhSeFd9JXWxv87ncq115r52keeaTFwQeTXCEq3CElJ5SzbZuanKjYngNbiNPHntZWOOGESaC56vsZJMKU2zPjmLGu6yd7vPw20D3z3akOVqOXm9rGNAsvbC7xTUEoD5oG555rcu656feO+x5LCXKubGgr2QoThodT201MwIsvpn92ZAQee0zl+us1rr7aCOzA7IUTyvna12KesfbGRnuFK2c8OuuscWBW9Xc0IIQtt6eUGp452OLrZhhA13UvKzo4NKdmi7abOjdeJRGy8pAgVBb3Pfa97xnMn2+hac6yit7b23XFmYKd/drUFNx0UziXZNQ0uPhik7a29NdbW+Fv/9aQ8ciFO7cnDKWrpWRTzyaZtOXCEec5JIU5kDRluqkdUjNsh1zuZ4lv5idM7iGh9uS6Xpx77O/+zuCrX42xcWP2QKooZMVM86MwNRXepMtcnrmwWfqVJmy5PaWIsZfYOuKcaTFnMTQ0VMJPl8jwCEcnH9pibIttS4tJZ+cUTz3VyNiYQkuLRWfnJF1db1DL3fXL8HAw5j+GAZdeejj9/dr0cezqMrjjjjdCO1gE5dhGjeHh4YLXy+QknHvuB/ntb911xSmya5H9MTICN99sMDz8HmedNV6xa9Mw4KGHmtm3r5ETT5wsy2/94hf2dz7zTCMnnGB/51tvpW9Tb9ds5nH+yEegpeUwRkdTnpGWFotjjnmHoaHxnJ/LPD+Z71eKUsT4bbKTumYDxOPxgldBe3t7CT89cwwDLv72CFuSz+exj1FaWa1eyfH/82rOPVehr89wlUQoaFpt9nUm1Oq4uuntVRkYiE3fBKOjCgMDTfT3HxE6K8RNEI5tFInHj8h5vfT0mHzqUzF+8xuVdHezo8ClddXatq2Jp55qqlgssZJxy1TZUjO5YsP1cs16HeeFC00WLbLYtctyHXuLSy6ZhabNyvk59/nxev+v/qoyf8OMxTgej+/WdT1TdOcAW0vbpcrS16eye3AOJgqqYxEzztXmddw0cAXa0jYWLzbp6bG3XbVKY/58232dr9xJSFFJ95C4v6NHvuvFNFX27csUYpLPC5nEXoKd3q0LbAv50UdVLr88xvHH252/ynVdVaIngdwD2Xgd5127VP7X/0qgaWbOevNC58fr/UpRageutRl1xZ8H1pT4nRVlYEDhtcTh3ML/xxXcShO22yGGQevQi2zZMp/duxX+/d9VXn3Vji2pSe+Yadq5X6eeanL//amZk9wY6VSq9Cts2ZGCP/JdL3fdpc7YDe1N7q5dGzbYN3pbW/muq3JPTOUe8CbXcd67V0muSVDc55zz4/V+pcibVqbr+snJDlvLgIt1XV/hLmlK1hh36Lq+LLnd80Fv+OHc+Cu4iWYm6ONPpt9r/8VNXLpsihtu0HjpJYWpKTsT0zSV5FqrCuPjCo88ovLAA+r0jXHZZTGuvVbjsstioczQLDdOgklbm920v63NKkvpV9iyIwV/lHa9ZCq1+7l3By9vlOl/IyMK27apfO1rsZIXYyj3wgNyD3gz0+Nc6HNe71eKQk0/dgO7gdV5tsn5XhCxY1AmO3bYM8uX+Mj0e1/iTv7AEXyHf877HaYJd9+toqpIW0wPKtXaMmzZkYI/8l0vF15octddudtfxmJ+MqmLjy9PTMCdd6ps3KjyoQ+ZXHqpxSmnFH8dl7snQTH3gLPCUz147WZynA3D/tfebvHaa6Q1enI+5/W9laLuFopw3/i33qry6K/PZDk/n37/j3nY1/eYpohDPipR+iWdz6JLrutl8WKTM880eewxp+VjSlBbW2HuXIvBQaZXgfJuoengjhcXwt4ukYCXXlK5/vqZua/LPTH1ew84Gep2Ylz03dnFHme3u39kxK52PfZYi5tuMjj77NTnvL53x47K/A116dtwbvwrrjC5r/VLXMvfT783iwO+vmPfPoWTTqrd2pf1SKXc30Jw0TS4//4E//mfCT76UYvm5tS5/9SnTB5/fGq6+c4ddyQ48USTbNe1QnEu60zS3deO69rvusKZ66mXIoZ+74G+PpX+/sa6cmcXc5zTE7PsxUqGhhRUlazPZX5vpag7y9hNT4+J/qkY/3PHN/ne6HUAHMK7ObZOz8J84QW7vlHaYlYPWdmpPnFaZjrZrZnn3m1R9/SYnH56A888Q9K17SXAmXFl/0xMwEUXaWhaKjckFoPjjzfZti1BY2Nlkzr93gMDA0rWEpPitUsRRK9mXYuxc2H/n01t8Kf2a7YYF3ZnOZl6Ig7VRTqf1S9+zn1jI+zYMcX112vcfLPGpEePhlgMWlstDhyYiaVsf8YW4pQre+9elUMOaWDOHBgbUxgbsyfrmUJdDvwch85Oi5YWK63hhXjtUgQx5BVdn4VPNA2U5iYmaQCgkSlXVy77XyxmTZc3OTgnzrkxnKXaVq0q7LYSBKFyaBpcfbXBGWeYtLam7mOw3bqf/rRZ4qDr3QPbshTeesu2uJw+2YmEwt69Kmec0VDVMaGnx6Sra1JCOjnw6+4vJhxRKnVtGTsMPKXyRxzC4bwJ2NbxEO2ceabJZz5jMX++xY9/rLJrl7c7Wmr/BCFYuN25/f0KhmFbqY7b+KtfrcTQl8vSVvjNb6hqlYWmwR13vEF//xHitfPAj7t/chLOOKOB/fvtZSqdcX3Rosrsk4gxcNJJFu+6xHgP85mgifGDz+cjK1eDouSMV0FluuwIglAa+dy5y5aZbNiQu2Sq3CQS1Y1HOv2UX3wx+mVNMyXf9WEYcMYZMfbsSXlBRkZg2zZVxLiSWBa8yQf4GM8D0M4b9hv3/4hf/8vFLLpiYdqJy0zQ6O8PXjKAIAi5cUqmtm1TXXHl0vpc5yMWq148MmXRfSDNotuyxS7Ilo6BhenrU9m/P7sNa/7SudIQMQb27FG4l7/iJJ6mlbG09z783T/jsi0Ps+6Xh2U1Dnfq02bPtmhqgnHX0si1TgYQBCE3TslUX5/Kk08q3HGHyu9/zwxign4E3OKTn7SqEq/NZdHt3GmXZP3kJ6qE03wwMGC3Qs6mghO2in1ziOjstLil7U/ZNHI+h/Auy1nL/+AfAPiY9Rzf2f5F+vp+5dk4fGICXn/dXlNVUWzxlRInQQg+bm/Xd79rpMWXVdUujXL/n0jAiy8qWBYcc4zdN/uVVwoL+Lx5Jo8/Xh3By2XRjY7CPfdIOM0vTrZ1usezmKYxxSNijLvlWSuvjbRyNxdMizHAQvMJbsjbOFxJNrO3iMWgo8PkvvtktikIYWEmJXPf/74xnUcyb56FadqC9+qr9vsf+pAdm3Z3dKo0uSw6RYFXXsledWhkxP6MhNPSyWybrGl2OLOSOQYixqRn1m3YoHLvvfP5zPhD/JqzALvcacG8BGB33cp0Saews+7271e58UaNq682RJAFIaJ4Cfh559XWwsxl0ZkmPPaYdyVr4d7e9UdmtvW8eamKmkpR93XGDs6N9fOfJzjtNJN42x8zQqrXZc+ZIxgG3Habk/Dh1C5mMzUFN9+syQpOgiBUFceia2pyj0/524GKweCNuw3muefay+auW1e5mYuIcQbOjGjdugRKS8v06z+6ye7qs2uXOr2cYmqB8+xeuJOT0e8FKwhCsHDGr/PP92eht7ZCV5ckmrrJ1ejDEedKIW5qD5yD3nBoM05y9Y9vnuQPDZpnPObQQ+G996ysVWWkvEkQhFrQ0WGhKCRzWdKJxeyxqqEB5s416e6W5C2HWjZwErMtD6NWyjJuYoKpqWw3T1MT/PSnCVasMLJ6z0p5kyAI1cQRk3/5F206qTSTtja7h3ciAYODKuedJ+E0B3e1TLVXuxIxzsOI0TT9uCWt/jjV63ZqCn72M5WrrrJ74Uov2JlRzR6wghBVHDGxF4jwjhO/+y5MTtbP0orFkG81p0ojbuo8NMxqgSH7cTNO+nT6STFN2LVLZetWVVZwmiHS21sQckeQSQAAGppJREFUysPAgJJVvpSOk+eSQsJpKWq5mpNMh3JgGPDKG83Tz1vIKi6eZmQErrlG5Wtfi2GasGJF6YuI1xOVcA2JpS3UIyedZMeKi0HCaSn8ruZUCcQyzkFfn8rB76dKm5rJ35R0716VvXth/XqVE08s7/qlUafcC32LpS3UK14JW5moKjQ2WkxMSLfATLzqixUFrr9e43e/UxgaglNPrcxvixjnYGBA4WTDbRlnxozdpKailsX0+qVPPDElg78Pyu0aklW0hHplzx4lhyCnXmxogCuuMGhutu+97m5TFo9w4VTT9PTYk/onnlAZcw3/IsZV5qSTLMZJZVP/d35KN1sxUXlAWUzbBZ/nV79SGB728gkpPPtsddcvDTOpdqTe60UXS7ktbUEIC356Kk9MWNx6q8app5qceKLFggUNvPaakmYp16MXKXM1PtO0J/FjY5VP3gIR45xYFoyTsox7+CU9/BKAb1o/5sHPPsOWLceRq6vN5CT88Icq8biCptmF9fU+48yFn4W+i6GWSRiCUEsyeyo7C12kozA+Dg8/rPLoo04+RcqLtGNH/XmRvEJb7e2WxzoElUPEOAd79ii8yGf4Crdnvadh8sCqfUxMHJf3Ox5+WOXhh+3Hra32AhInnmg3ba92A/mgM5NG/W7cs9r58y0WLjTZtas8lrYghAX3xHbbtlEaG9v44Q81zwxrywLDyDYmRkehv7++vEheoa1XXyXPOgTlR8Q4B52dFv/Uehm/Hz2aT/IbAP5MvZ1TzDgAb73sVT/gtryylzBzkrwA7rpL5cwz7X6nIsil4TWrXbjQ5PbbE+zZI2VmQn3hTGx1/QCHHdbM9u0K27apTEyA3yUA6636wCu0NTEBzc2Qaw2CciOlTTno6TFZ+CnY0fZZfqJ8k//V9k1eOKRz+v2DGCF1YVvEYk5JQe6G7O6G7aapsH27FNsXQ65yJa/SqF27VFQVVq6UMjOhftE0uO++BB/7mCMoFvkWuXEYHFTqqizQCW25aWpystNT47ZaweFaLOMceMUxP7iqDXbY7x/E+2nbz5ljMTRUXKB/YqL+3EEzJV+5kiRsCUJutm5Vee65fEZCOqoKmzapdZXQ5ZVE2t5u8eKL6cfMT+nYTBExzkNmHPOF/0yJcRvpbupihdjh4YcVurrU6fKCu++2p14SU04nX7mSJGwJQm4GBpSki9oPdhbx+HjqPnvsMZXrr09fnz0z8zjsYSAv48s04StfiWWNK5VCxLgIOk5qgw3244N4j/T1Qh0s/M5AAR55RGX7dpWGhvRyhLvuUvn0p016e6M9I/VLPut3xQqjrKVRghAlOjstmpooQpDTmZqCf/xHjZ//XOXMMy0sC371K5WREXuxiShYz16TC8OwV7/6zW+YXuWqo0OWUAwGBx00/fBg3sdbdIuxkO1tp6ZgaipdxE3Tzsa+/PIYX/qS6Wvm6cRUozJbdZPP+i13aZQgRImeHpNTTzV5+OHMgKffscpuJDI0BBs3Kmmvg31PPvqoyn33qTQ3Mz3+dHebbN0a/PEoVwIowHPPKcmlce1xenCwckFjEWOfGAb8cO0sViafdzDIWfzfnNubqDzLJ3iXQzzfm6CJ9JvB+8bYsEFlwwaVxkaLc86xhef11+Goo+CCC+yLe2DArhtcs+Yo3nvPvlgaG+G440zOP9+KRI1zocYgpZZGCUKUURR8Wsf5BDr3e4kEfPnLheTEYtYsmD0bvvhFk7//++xlZ2uBVwhs+3YVRUm56x3yL8JRGiLGPunrU9kzePD08z/hQf6EB2f8fS9zNMPMBuBNPsBzfAzTldz+DofyQ/6a1zkCsJuI3Htv+oWxfr3XLE2Z3t5dSqVpFi0ttqtlasq+KWMx+MAHQNfTRf6ii4IXr/Zr/UYtliUIpdLXp7Jrl8rERGY4rZw4q0HlD9kdOGD/W71aY+1ajZdfnqy5IOcqa6o2IsY+GRhQ2DfxsbJ930d4hY/wyvTzz/JQ1jYruZEnORkLhSHaGaCTA8zCQsFEnf5/nGb2c/y0uGdiofCscRzvv5+efTA1Ba+8Aq+8kn7DODXQQYtXF7J+ZYEIQcjGS2wqQ6blnP/58LDFjTdqXHNNbWunvEJgTU0kLePq7YeIsU86Oy1uaTuJK0Z+yBf4/1ExURU4/HCL9na7tAngxZcUXn5J4Qhe4xheyvoeBYsW/J/hU9g9/XgxD5T0N4zSwl7mcTcX5t/QAuURi+eXm3zyk9XPSLYOOQTzggvgsMOK+pwsECEI2XiJTTbFJZ6Wi+3bq/+bmXiFwJyYsfOaQ1tb5fZDxNgnzgn7xc5vcdvot7KsrmSMnyMN+PNzYjz8sJpWk9bebnHccRaPPaYyiwPTQn0Q73Mcv6XZJdDf5p/5BL8t+9/QyhiL2MUidhXe2AL+vey74Jvf/ctm9t60pSg3s9QbC0I2mWLT0uLEj608/asdvFzPbkoT09NOq335Ya4QGNgT/P5+BcOww3qdnRY7dlRmP0SMfeI3ZqlpcP/9CXp7Ve65x47pXnihHYMFOOecGI8/Pos9U/OnP7Od09K+Yw3L+SgvMoe3ATicIRbwNIfwLiomChYKFiomGgaf4Fk+yOs5972dIT7Mf5XjMFSNjz//S1ou7OC/L/w//PShj/kSZKk3FoRsvMYuJ9PZWbP3tttUtm1TmZpK/5xlgWmm7h9FgcMOs5gzx+Lll1XGx2d+b82eDVddFYz2XrlCYF6vVUqMFauSLUVycNVVV1n/8A//UPXfDQJOgpFzEyQS8JOfqLz0ksJBB1mccALs36/w7LNK2o1RGgrNjLGctRzFq8nXvGqk3Z+wOO88k7nlC5MXZPB5OG7TLWmv/ST2LY686xZfbmbDsCc7TzyhMj5u95U99dTy9P8eGhqivb29tC8RspDjWjmKObbOuJRpBZ51lslNN2ls365w2mkWV12VyoA2DNi8WeW221SeeSbllWppsfNRJifthNGmJovRUXucaWwMXjZ1sXz/+9/nxhtvLLt/vWTLWNf1ZUAHsBF4G1gObIzH44OlfncU8ZqBnX9+utC4BXvBAoszzzT5i7+I8cgjCo2N0NZm8eabyvTFrigWiYTCBz+Y4JxzVHbuVNi/3y53Gh9XMAyLcZq5lStcv5I7RqQocOaZJn/1HwmMKiY+/fsNGo9sOoeH+Oz0a4ck3srpZs7MnO7uto+jM7+swTxTEEJJvuTIXAlWmgZLl5osXSr5GOWgHG7qOcCq5L9h4OsixKXhdWPccUci9weSeM2Ec814nQ4z112nceedCqOjCu3tFvPmwcUX16a0yU6S+wyXjNzJer4IQJs24elm9sqc7ugwGRxMlXBMTMCuXZLAJQhC8ClXzPhQYI6IcPDIN+PVNPjBDwx+8IPq75cXTqKJua0FknV+H5w9xgKPtpZemdP796vT3XIcJIFLEIQwUBYxjsfjw9hWsSDMGCfRZPeqBkhOEBadNE7Cw0L3ypyemrJjUpOTqdckgUsQhDBQFjHWdX05drx4DjA7Ho+vLsf3CvWHpsGiTzdMP1cmvVvh5MqcnjvXYnAQWTBCEIRQUQ4x3gq8nbSO0XV9ja7ry+Px+Np8HxoaGirDTwtuhoej4ZxoHB3lg8nHU++/73mtdHVBZ+fh9Pc3Mjam0NJi0dk5yeWXv0dvbyuKAuecM8rnPjfOW2+Vvk9RObZBQ45r5ZBjGy5KFmOPOPGD2MlcecVYyhkqQxSOq3LEEdOPG0wz59/0y19CX58xXSb24x838O1vf2DaKh4ebuZLXypfK8woHNsgIse1csixDQ8lrQel6/psXdctXdfdTZGHsUudBGFmNDenHufp2O4kp61caaAo8MQTdkKXZSmMjCjTrTAFQRCCTjlGqtWOizpJByBZ1cKMsVydABQfndonJ+Ev/1LLauruZFILgiAEnZLEOCnCmRG5i4ArS/leoc5pako9dqdGuzAM6O1Vue46jfnzGxgaUshsYtLUJJnUgiCEg3IkcK3VdX0Ftnt6LrAmHo9vLMP3CvWKW4w93NROw48dO9wrqmRawBZHHmlJJrUgCKGgHAlcw4CUMgnlwx0z9nBT9/WpSSHO7YKOxWD1akPWMRYEIRRIdosQPFyWsTU+QW+viuFqj1t4sXSLD33I4umnlazPCoIgBBFZQlEIHIYSw0BFw0S1TD5+4Sm81qrw0Y9aoMB3DsCFKGkrqz7OGfwN/4yhNtLSAq+/rnDttVpZV24SBEGoFCLGQuDo61P5LAdxCAcAmGftgRFgr/3+IcBJGZ9ZwNM8pv0xU+cv4557VEzTdmGPj8PDD6tce63G974nbmtBEIKJuKmFwDEwoPBvfK3oz/3RR3+PpoGZkbNlWXDzzRpLlsTEZS0IQiARMRYCR2enxTVtt3AsL7CAARYwwKnN/fzqll1M7tw5/W9s+05eOuu/TX/uz//bOIpnTpdCIqHw2GMq11+viSALghA4xE0tBA5nKcWdO4/hZdeCD6f9RQLL5WZWgQ8tPAIeSj43prjwQpP161Usj/LiqSlYvVpj82aFL3zBwrLsLl5dXXYJlLiwBUGoFSLGQuBwllLs61N56imFBQvyiGVDaoUnpqZYvNjkmGMsXnwRsmuPFRIJ2LNHZc+e1KttbbbYb9kiSV6CINQGcVMLgcTdd3rx4jxWa4YYaxrcfLOR1jckGyXtn/SxFgSh1sjoI4QbVx9rEgkAzj7b5PTTTRoaLMBfO8yRETtxTBAEoRaIGAvhxm0ZJ/tYO27uv/s7g1gRgZiklguCIFQdEWMh1FguMVampqYfaxpcfbXB8ceb+LWOJV4sCEKtEDEWwo3b9HWJscP773u5nrPd162tdla1IAhCLZBsaiHcuCxja3KK3l6VgQGFzk4L04TXXsteWhHgkENgZMQikbC/Yu5ck+5uWeFJEITaIGIshBtXAtevtxpctinGaLI2ub3d8liB0bZ+Jyft9SgUxY4VDw6qnHdeTMqbBEGoCeKmFsKNyzJ+780pRkYULMsuV3r1VSVtNUYb21IeG1MYGYGpqdT2Ut4kCEKtkJFHCDfuBC4jPWY8MQFHHGGhKP5KnEZH4amnpLxJEITqI2IshBuXGDdr6WLc1gZf/KLTMKSwyLa2woIFksQlCEL1ETEWQo27tOkDsyZpa7Mt4bY2i4ULTV54QclTP6wAqe0XLTLp6ZEkLkEQqo8kcAnhxiXGnfMm+etPG2zbprBokUVvr8qjjxa2iP/4j02uuMKUxSIEQagZIsZCuHGJ8f6npvjhbo2REXj4YZIrN7nF2HFBpwv0W28pkRBiw4C+vlRpVxT+JkGoF0SMhXDjEuPx9xOMmLbQei2h6I3Cb39ri9jixeF1URsGLFkSY+dOlZERu2zrqKMsbrrJ4OyzRZQFIehIzFgINYaaEmPNzO7A5SaXIE1MwIYNKjfcoNHbq2IY5dzD6tDXpyaF2C7dmphQeOEFhUsvjbFkSSyUf5Mg1BNiGQuhxTDgW99u5t+Sz4/mFW7ib3NsbaEcdAg/G/sKz00ek/Xuxo0qiYSdUV2LtY1LdTEPDCiMjma+qjA+znT9dJgtf0GIOiLGQmjp61PZvTfV1eMDvMXfckvuD7wLfzKrj87EY5gZujQ1Zbu3R0bgkUdUrr1W43vfM6oiyIYB55wTY/t2lYkJu9328cebbNuWSFshMh+dnRYtLXgIsv039fcrLF5c3v0WBKF8iJtaCC0DAwrPjH2UVznS92fmjexg/X9O8ulPO2qc3bvaMODGGzU++tEGvvKVGFu2FO+6NgzYvFnl8stjXH556jsMA3p7013iDzyg8vDDKhMT9r4kEgp796rMn9/A9df7c5339JjMnZt7hapNm8LpfheEekEsYyG0dHZaNLQ1curIEyxlE41M0tho8aUvmnzyk+mipF13HcrICIphcO4Zb3L3vUfk+WZbnIeGYP16hfXrVQ4++CiWL7f4/e8VFAWWLTNzJkZNTsJppzWwb19K5O+8U+XDHzaZmFB56y0wTbut9sc/bvL++4pHwpnCSy/BtddqtLWlXOdgi/yPf6zyzjsKS5aYfPe7Bo2N8IUvWOzZ4/33/Pa39iTg3HPFVS0IQUTEWAgtPT0mixaZ7Nx5NLeNfms63vv3P01gZIik+otfoPzudwA8evebWFY+MXZIiel776nc4vKA33mnClg0NdnCqqq2e3lqCiYnlazPA/z+92ra65OTsG9fPudUynX+xBO26/w//kPl5ZdT37t3r8att2r8678m+Nd/zf1dExPw9a9rfPObCqecImVPghA0RIyF0KJpsGVLgr4+laeeUliwwFtkDAOeHWrnJGwxbvz23/DVg+awzEeLTIcpGniWTzDMbAw0DDQSaLw/cRBTNGChYE0o9v9J1/fv+TDPcELyeToJYkzRiJ82nQBjY7br3Cb9M6OjFn/6pzHP91IoDA/D9ddrKAocfbTJU08laGnx9fOCIFQYEWMh1GgaLF5s5k1O6utTUd4/gpOSz880fw0HqrF3uTFRuI/zWM5axsitiJM0JkUb8gmtP1I12C+/rHLooY20tFg0NsLnPmfxb/8m4iwItULEWIg8AwMKg8Z5LOXuWu/KNCoW57OJ89mUd7txmtjP8fwXHwLAQuFVjmKcrLUhPfGyyt3vMgaMgXWPwu33wl98w0TLNyooPoXf53azR0fR2trK+p2B364YSvjtQ0ZH0Vpby/Z9JW1Xy9+u5fkrAhFjIfJ0dlrc0nYpC0ZO4uNJV3Vzk8W3vmXQ2Wmxe7fCtm0qTzyhuHKRFdcji2N5kf/X3t28RnWFcRz/TibVYscypmApoovoKqCV8bgQcSGNC8GAYKwbd9JkIbhwEXVjXhCr8R9oLP0DrFFcqFmYrkU8CFLpznRRUAxWA5oW02RuF/fcyc04b7lnMhMzvw+EZOYOdyaHc85zz8t95hteugnq8KedeTbyjnbmWZygDs/wFX/TxR9s4ON7jTbwb82f/XM+sJun7OZp0n+/dgHw08q/TdzGxr5dS/my2R9grTp3bkVOq2Asa97iRq9d/P7PrsJGr53D85CGXC/kgL45GBkJN0TNza3c52ljgZ/p4xi3SFP+fqMMsyv3IURkVVEwljWv1o1e69bBpUsLDA8vcP9+G7dvt5HPw5YtAXfv5nn1qp1MBoIg4O3bVCFxSHw39fx8uCa7fj1s3RrQ1QXbtgVMTKR48SJFJgMvX6Y5lf+FU4XcYbB+fcCOHXD4cJ6JiTZ3W1TATp6xlb/cqwIyzPI1rwqPy68XB2z8IuD9LBVeE1qcDwjY/W3AyZNlbn+qNeF37YnBef/uHZlMpn7nXO2va+B7v5+dJRNfAmiB/3nVvC6BVLCCJy/n/PnzwfDwcMPfd62bnp5m8+bNzf4Ya1I9y3ZuDi5fTnPvXopNm+D06TxHjixeHEQJQ86caWd6evnnb2+H/fvz3Lkzz4EDn/HsWW1rYek0vH4919BNXKqzK0dluzIGBwe5cuVK3ReYNTIWabB162BoaIGhodLH02k4ejRPT8/cktH8wYN5rl1L8/Bhin37As6eXWB0NM2NGymCIIUxAV1dAbnc4sj/0aP/CqN8gCNH8ty61cbkZIoPH8IRvXZTizSfgrHIKlXqtq2LF5euMY+MLDAyUvkcPT35JZm3jh1TFi6R1aYuwdgYMwBMAR0A1trr9TiviIhIK/D+oghjzFVgylo77oLwdmNMr/9HExERaQ31+NamPmvteOzxA6C/DucVERFpCV7B2BiTK/H0G6Db57wiIiKtxHdk3EEYfONmAIwxWc9zi4iItATfDVxZ3KatmCg4d+ACcymDg4Oeby0iIrI2+AbjUsE2Cs7FI+aClbhhWkRE5FPlO039hnB0HJcFsNaWHRWLiIjIIq9gbK19wsej4w5g0ue8IiIiraQetzZdL7qv+BAwVofzioiItIS6fFFELANXJzCjDFwiIiK1a8q3NomsRsaYMWttf9FzFVO9KhWsyKfJzejutdaeK3HMq90n6RcaGozVcSXnyg5gL/DYWjta4riCRkIurWu3tXZP0XOPowxzy33cylyegQvAY8I6Z90ek+i46mtCrmyivTpZ9QXLY4zpBnKES6pTJS7Avdp90n6hHmvGNVEO6+TciG3U/RwHTsSCc9WyVdlXZozpLHOoWqpXpYItwQXi36y152LlcyF2XPU1IWPMgOsHrruymVRfsDzW2kl3AfOkzEt8232ifqFhwRh1XIm4jq14x/oYsc4NBQ1f3YRlUlAt1atSwVZ0ldgmTtfp/xA7rvqa3In4AzfbsDf2lMrWg2+79+kXGhKM1XF56QAGSozesqCg4ctNWf1a4lC1VK9KBVteH0W3N0Z5B1Rfvb0xxtyM6pgxpg+44f5W2frzbfeJ+4VGjYzVcSVkrZ0C9rjfkUMsdnYKGn6yZRLUVEv1Wu14S4pdNHYaY3qNMX3xaVRUX331E653/unK9U1spKuy9efb7hP3C40Kxuq4PBRtfMkSXslGU0sKGgkZY3orbKqoluo1USrYFlCYwYmtS0ZrlaD66sVdlI8RlslVlk5Rq2z9+bb7xP1Co4KxOq76uQl8FxspK2gk4EZwlVK2Vkv1qlSwpUV1ysaemwSi0bHqqwdjzBjwxFq7nfCCvM8Yc9MdVtn68233ifuFRgVjdVx14EYXV+MjZRQ0ksoBOWPMgJvu6wey7nFntVSvSgVb1gx8VLfiU6GqrwlFa77W2qgOXgf2ANFuaJWtJ99279MvNCQYq+Py524/eBA1xFjDVNBIwE2hRreLjRLuKp1xj6NZh2qpXpUKtogru5miDYeFDl/11UsH8Dz+hCvvcfe3yrY+fNt9on6hkbc2qeNKyO347QCsMSbrOrr4LQ4KGh7cjtTjhJuOBqLNLC4zT7QRaQB4Hl9jrna8hf3I0h26J4B4liPV1wTchXh8jTiabYhv7lTZVmGMybn22gt879p8Yae5b7tP2i80KwOXcljXyDW2tyUOjbsEINHrKpatyl4aqWgHNRWyRKm+LoO7EO8nNkJebtmpbFcn5aYWERFpskZOU4uIiEgJCsYiIiJNpmAsIiLSZArGIiIiTaZgLCIi0mQKxiIiIk2mYCwiItJkCsYiIiJN9j+BuXn6yi2bzwAAAABJRU5ErkJggg==\n", 202 | "text/plain": [ 203 | "
" 204 | ] 205 | }, 206 | "metadata": { 207 | "needs_background": "light" 208 | }, 209 | "output_type": "display_data" 210 | } 211 | ], 212 | "source": [ 213 | "fig = plt.figure(figsize=(7, 5))\n", 214 | "matplotlib.rcParams.update({'font.size': 16})\n", 215 | "plt.plot(fX, 'b.', ms=10) # Plot all evaluated points as blue dots\n", 216 | "plt.plot(np.minimum.accumulate(fX), 'r', lw=3) # Plot cumulative minimum as a red line\n", 217 | "plt.xlim([0, len(fX)])\n", 218 | "plt.ylim([0, 30])\n", 219 | "plt.title(\"10D Levy function\")\n", 220 | "\n", 221 | "plt.tight_layout()\n", 222 | "plt.show()" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [] 231 | } 232 | ], 233 | "metadata": { 234 | "kernelspec": { 235 | "display_name": "Python 3", 236 | "language": "python", 237 | "name": "python3" 238 | }, 239 | "language_info": { 240 | "codemirror_mode": { 241 | "name": "ipython", 242 | "version": 3 243 | }, 244 | "file_extension": ".py", 245 | "mimetype": "text/x-python", 246 | "name": "python", 247 | "nbconvert_exporter": "python", 248 | "pygments_lexer": "ipython3", 249 | "version": "3.6.8" 250 | } 251 | }, 252 | "nbformat": 4, 253 | "nbformat_minor": 2 254 | } 255 | -------------------------------------------------------------------------------- /examples/TurboM.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Simple example of TuRBO-m" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "from turbo import TurboM\n", 17 | "import numpy as np\n", 18 | "import torch\n", 19 | "import math\n", 20 | "import matplotlib\n", 21 | "import matplotlib.pyplot as plt" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "## Set up an optimization problem class" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 2, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "class Levy:\n", 38 | " def __init__(self, dim=10):\n", 39 | " self.dim = dim\n", 40 | " self.lb = -5 * np.ones(dim)\n", 41 | " self.ub = 10 * np.ones(dim)\n", 42 | " \n", 43 | " def __call__(self, x):\n", 44 | " assert len(x) == self.dim\n", 45 | " assert x.ndim == 1\n", 46 | " assert np.all(x <= self.ub) and np.all(x >= self.lb)\n", 47 | " w = 1 + (x - 1.0) / 4.0\n", 48 | " val = np.sin(np.pi * w[0]) ** 2 + \\\n", 49 | " np.sum((w[1:self.dim - 1] - 1) ** 2 * (1 + 10 * np.sin(np.pi * w[1:self.dim - 1] + 1) ** 2)) + \\\n", 50 | " (w[self.dim - 1] - 1) ** 2 * (1 + np.sin(2 * np.pi * w[self.dim - 1])**2)\n", 51 | " return val\n", 52 | "\n", 53 | "f = Levy(10)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "## Create a Turbo optimizer instance" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 3, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "Using dtype = torch.float64 \n", 73 | "Using device = cpu\n" 74 | ] 75 | } 76 | ], 77 | "source": [ 78 | "turbo_m = TurboM(\n", 79 | " f=f, # Handle to objective function\n", 80 | " lb=f.lb, # Numpy array specifying lower bounds\n", 81 | " ub=f.ub, # Numpy array specifying upper bounds\n", 82 | " n_init=10, # Number of initial bounds from an Symmetric Latin hypercube design\n", 83 | " max_evals=1000, # Maximum number of evaluations\n", 84 | " n_trust_regions=5, # Number of trust regions\n", 85 | " batch_size=10, # How large batch size TuRBO uses\n", 86 | " verbose=True, # Print information from each batch\n", 87 | " use_ard=True, # Set to true if you want to use ARD for the GP kernel\n", 88 | " max_cholesky_size=2000, # When we switch from Cholesky to Lanczos\n", 89 | " n_training_steps=50, # Number of steps of ADAM to learn the hypers\n", 90 | " min_cuda=1024, # Run on the CPU for small datasets\n", 91 | " device=\"cpu\", # \"cpu\" or \"cuda\"\n", 92 | " dtype=\"float64\", # float64 or float32\n", 93 | ")" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "# Run the optimization process" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 4, 106 | "metadata": {}, 107 | "outputs": [ 108 | { 109 | "name": "stdout", 110 | "output_type": "stream", 111 | "text": [ 112 | "TR-0 starting from: 24.79\n", 113 | "TR-1 starting from: 20.77\n", 114 | "TR-2 starting from: 14.87\n", 115 | "TR-3 starting from: 27.97\n", 116 | "TR-4 starting from: 23.89\n", 117 | "80) New best @ TR-2: 12.43\n", 118 | "90) New best @ TR-2: 6.42\n", 119 | "110) New best @ TR-2: 5.467\n", 120 | "180) New best @ TR-2: 2.888\n", 121 | "230) New best @ TR-1: 1.944\n", 122 | "280) New best @ TR-1: 1.54\n", 123 | "310) New best @ TR-1: 1.052\n", 124 | "340) New best @ TR-1: 1.038\n", 125 | "390) New best @ TR-1: 0.9689\n", 126 | "410) New best @ TR-1: 0.877\n", 127 | "420) New best @ TR-1: 0.7794\n", 128 | "460) New best @ TR-1: 0.7509\n", 129 | "470) New best @ TR-1: 0.7264\n", 130 | "480) New best @ TR-1: 0.7238\n", 131 | "530) New best @ TR-1: 0.7044\n", 132 | "540) New best @ TR-1: 0.695\n", 133 | "550) New best @ TR-1: 0.6823\n", 134 | "560) New best @ TR-1: 0.6656\n", 135 | "590) New best @ TR-1: 0.6614\n", 136 | "600) New best @ TR-1: 0.6604\n", 137 | "640) TR-1 converged to: : 0.6604\n", 138 | "640) TR-1 is restarting from: : 23.66\n" 139 | ] 140 | } 141 | ], 142 | "source": [ 143 | "turbo_m.optimize()" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "## Extract all evaluations from Turbo and print the best" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 5, 156 | "metadata": {}, 157 | "outputs": [ 158 | { 159 | "name": "stdout", 160 | "output_type": "stream", 161 | "text": [ 162 | "Best value found:\n", 163 | "\tf(x) = 0.660\n", 164 | "Observed at:\n", 165 | "\tx = [-2.968 1.072 0.173 0.973 3.698 0.883 0.946 0.872 0.006 0.927]\n" 166 | ] 167 | } 168 | ], 169 | "source": [ 170 | "X = turbo_m.X # Evaluated points\n", 171 | "fX = turbo_m.fX # Observed values\n", 172 | "ind_best = np.argmin(fX)\n", 173 | "f_best, x_best = fX[ind_best], X[ind_best, :]\n", 174 | "\n", 175 | "print(\"Best value found:\\n\\tf(x) = %.3f\\nObserved at:\\n\\tx = %s\" % (f_best, np.around(x_best, 3)))" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "## Plot the progress\n", 183 | "\n", 184 | "TuRBO-5 converges to a solution close to the global optimum" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 6, 190 | "metadata": {}, 191 | "outputs": [ 192 | { 193 | "data": { 194 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAFTCAYAAAAKvWRNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9e3Qc1ZXv/63qllqPxLFNEITJxCBnElawiWTKBltZTDwR0WDMmIeNJxl8uTOsIb+BZE3u3DvGXmScu0LwxeSxAoQEe3G5Q8z9BT/A9tgWUdD9EQyWX42lsUyCyVjYmZvYEQFrSNR6ddX5/XG61NXV9eyq6j7VvT9radlqdVfX45zz3XufffaRGGMgCIIgCKJyyJU+AYIgCIKodUiMCYIgCKLCkBgTBEEQRIUhMSYIgiCICkNiTBAEQRAVhsSYIAiCICpMstInQBBRIUnSSgALGWP3W/xtLYAhALMBgDG2Jfd6K4AvAVgL4DiAbbmPXARgJoAdjLFel+81HmMIwGYAOxljQyFcVmD0+wKgFcD/YIwdr+C5bAYAxtiXKnUOBCECEq0zJqoNSZI6ASwAcAOAIfNAL0nSJgDHGGM7rX7PvXYawGbG2COmz74ELshbPJyH5TEqiSRJMwG8zhibmxPl4+UyEiRJusd833LP6r1KGgQEIQLkGRNVR85z7ZUkSfdmzdxj8pZfAnA/gJ0W7zWzCsAFSZJ6RfF0faKAe+swGh9l4hrzC25RBoKoFWjOmKgpJElaYPHyewA6vXyeMTYCoBfApjDPq9LkPGbb34MeOxeOnm3xeqfNMyGImoLEmKg1ZoOLr5ERwJcAvQQeBg9MTpA2SZK0MvdvZ+71lZIknc79LDC8xiRJ2mzz987c3y0Nhdz7VgFolSRpbe79nZIkvQ7g/xjOZwe49z/TcNzXJUnaYfjMJkmS7rH4Dv1aOnNhcIAbOrMBLMh9r/652eBz65sMn5+Ze8/K3M9aw988nwdBxA7GGP3QT1X+gA/ym02vrQRwwfTaTAAMQKvhtdMA1toc9x7edVy/3/YYpveYv3dm7v+dAE6bv9vwf8e/23xfJ4CXLO7J66bXmH4ehvdMn6t+z0yfeR3AAsPvF/Tfc5/f4XY+uWPMNP19s5/zoB/6ieMPecZErTFi8ZoePjV7zHbMtDmOL3TPkRXOPR9HLmTOcvOpRs8XwHb9jW5/94GXaxkxnivj4XoYvOcF4AJpTMS6hvlIzNKjAvqxc//vBXCPIWrheB4EEVdIjIla4z0UJ3XNBApFwIW5ANKlnoAhfNsKYMQQcu0EcAyF4rgZPJQLcLEzn+MmAOtz/5/p4xpKwenY04lhOsx/gtsCWBtEI7njezkPgogllE1N1BSMseOSJJkH89ngSVleuQPA5wKcRmvuX927M363+Ty2AHg7Nw9sJW7bAWzKeaaVzO4eQv66HMmtw55p4TVPr/s2MROVvTaCiBzyjIlaZIvBOwX4euTNXj6YS24quVBGTohuAPJLi3Kv6X+facwuZvns7c3MYhlQ7u/bAWwq9ZzAhW46WlBKdnPu3Ib0UHPuOK2GYxnF2hzO1o+xE8BM0/1YCYEKphBEVJBnTFQdOQHoBE/2mZ0rvtGrCwBj7H49YxdcIE6bhPFLuddXS5KkH1Zfs2wpiqbvtzvGwtx5GUPcnwOwXpKkY/oLrHj97/9AYZjWzGaXv+vntQB8PbWSy1LewhgbYYwNSZK0M5eV/B64cI6Ae9z3567jfuSyocG9dT00vkmSpE05sfxc7vfW3HFG9HuVi0ikc58fsTgfvSDINYb7MRvcg15ler/beRBE7KAKXAQRcyRJWmkh4ARBxAhXz1hRlJngSzlGwBNXkE6n7ze9p6DObzqddi0VSBBE6eQ8w+NuXjpBEPHA1TNWFGWTUXwVRXkdwGZdcBVF2QTgWDqd3mn1O0EQ4ZObm20F8ptcEAQRX7wkcK1UFMVY4WYIuQSUHPeYhPcl5JdiEAQRAYyxXsbYFhJigqgOvCRw3ZBOp41JEa3IbSunKEqgOr8EQRAEQXgQY6MQ6+KbTqf1LeFs6/wqijIznU5bLs5ft24dZY0RBEEQseThhx+W3N/lD09Lm3JJXHeAF5n/W8OfZqJ4kb4uzrPhUCnn4U2b8I26b+DrU/80/VpzM8PWrVksW6Z5OS3fqCqwfHkSR4/KyGSApiZg0SIN+/ZlkUhE8pVlZXh4GC0tLY7v6e6WsWZNEqOj+bYU9X2vBrzcW8I/5vtq1UdbWzUMDcllbbN2/eSf/5mPFf/6rxI+/WmGri5N2LGD2mw0fP3rX4/kuJ7EOOfhbgGwRVGU1xVF0RO4AtX5/ehHGZqHWYEwdnZq6O6WMTAgoa0t3MaeSAD79mXR0yPHojNFwcCAhEym8LVMhg8uy5ZV5pwIQqenR8bRo3nhHR0F3nxTRjZb+L6o22xXl4ZFi7Qiw33ZMj5eUF8hwsbT0iZTuHlz7mcLHOr82oWojdx5p4pZ7dlpYezs1LBiRbSeK+9IWiw6k6rywSlMw6StjaGpiQ9yOk1NwKc/TTMHROWxMhanpoD6emByMv9aFG3W3N/27Mmit7d2DXeivDiKsaIonQBeUhRllllccyJ9XFGUkuv8ymAFwtjdXWwVHz0qo6dHrrkQalQhdTuLv6urtu4vISZ2xuLcuQxDQ4iszTr1tzgY7kT8cfOM0wC2mIT4BgA7Da9tURRlpWF5k+c6v2YohJrHKlwXhmFCoXpCZOyMxai9VLv+1t0tI5FAJNNmBGHEUYzT6fSIoiibcxW2AF6fd8hYBCSdTt+vKMpaRVGm6/x6LvhhKjhCIdQ8VobJ6CiwY0fwsHWcQvVEbeFkLEbZZu3629q1CQwPS1WZ8EmIhZelTcfBNzx3es8jTn+3xSTGFELNY2WYyDKwa5eMiQkaGIjqpRLGolV/S6WAc+ckjI/TtBkRPUJtoahbxVu3ZrFhg4qtW7M1Kza6YdLczCBJDKkUN1zGxyUwJmF0VJoeGAiCCIa5vzU3M1x2GcPEROH79Gkzggibym6haFEXm0KoHHO47tQpCdu2FQpvrc6nE0TYWIXHNQ24664kTZsRZYH2MxYYo2HS3S1j716ZBgaCiAizI6CqCGXaLIolikT1IZxnXAni0FloPl0s4tBmiGCEsfKg2qv+EeFR82Icl85CS5LEQW8zR47wNlNXB1x5pYaDB7Oor6/02RFhEnTaLKolikT1UXXZP6rKQ7obNybQ3S1DVZ3fb+wsoidG6QPD+vXqdFk+IH/Njz46w9M1E8Ho6ZFzQiwBkDA1JWFwUEZHRx3d+zLht59XCqfaCQRhpKo841K83LgXGlFV4Kabkjh8WMb4+Aw88QRw3XUa9u8Xy7OPK3o4+uDBGejokNHVpVm2GUDCqVMgj6cM+O3nlZxSoNoJhFeqSoxLCQnFvbO8+KKMAwdkaBq/5vFx4JVXZDz4YAL19aD5zAAUDvoz8OSTfNC/7z4NdXW8ZrKRqan4GHFxxk8/r/Q0FOV6EF4RLxYbgFJCQlbrC8PoLOUKo+3cKUMznSpjwLe/ncCDDyawZk0Sy5cnhQ3jiYzdFAZjfI4YKDTY4mTExRk//bzS01BUO4HwSmw8Yy+hplK8XC+JUX7DXJW2xgEgm6WEkaDYDfonT0o4eDCLjo46nDrFPWLyeMqHn34uwjQU1U4gvBALMfYqbqWGhJw6SynCWs4Myttv17B9e7F3bCROc+Ai4TTo19cDhw9PUXZ7xFgZwn76edynoYjaIRZFP7yKWxTLf0oR1nJa48uWabj+eg2HDvGa1ckkoGkoEGcafErDbdAnjydanAxhr/2c5myJuBALMfYjbmEPkKUIazmt8UQC2L+fD0x9fRlce20TnnhCnh586uqA1lYNnZ00+PjFaNz19WWwZEkTeb9lxM0Q9tLPaX0+ERdiIcaVDDWV8t3ltsZ1A0RR3sdFFzVAVYFf/ELG5CSQzQJDQzJWrEhS4kgJGO9tS0tDpU+npggrwuTXQKfqakQliMWccSVDTaV8d6WscT2s19cn53aboSQuIr5UwggXIfmSqE1iIcaVDDV5/W4ra7rc84kvv9yAo0dlTEwUL/GgJC4iblTCCKfylUSliEWYGqhssozbd4tiTb/xRr1FZSgOJXERcaMSRnhYoXEKdRN+iYVnLDqiWNNXXTVZFNYDGBoaKIOUiCflNsLDCI2LYpwT8aKyFbiqRIxFKQa/dOn4dDUxgCGVYrjiCoZnn6WqPwThhTAq8lW66hcRT2ITphYZUQoLlDusR6E4otoIow+JUPWLiB8Upg4BkQoLlCusR6E4oloJ2odEMc6NkOEsPuQZh0AtFhYQZZ6cIERDJOMcIMM5LpBnHBK1VhqRQnEEYY1oxjkZzvGAxJgoCRFDcQQhCmbjXN9StRJhYjKc4wGFqQVF9Dke0UJxYSD6PSfiiV2YeM+eLHp7o29vZDjHA/KMBSQOczyiheKCEod7TsQTqzDxkSMyOjrqMDQkRd7eqtFwrkZIjAUkLnM81TRPHpd7TsQPuzDxqVMSJiejb2/VZjhXK7QK3YA+r7NxYwLd3TJUtTLfIUoRkVqC7jkRFXqY2EhdHTA1VfhalO1NN5zXr1exbBkJsYiQZ5zDb5jSPL/Y2am5zv94/Q6a4yk/dM+JqLAKE7e2ahgakqm9EdNUVIwlgcTYT5jSLKqNjUAqBUxOwlFkvX4HzfGUH7rnRFRYhYk7OzWsWFFsmFN7q10omzqHn/R/s6hmMkAmw+C2f7DX76A5nvJD95yIEqv8Cr/trZzZ/rSyoPxQmDqHnzCllaiaMYqs3rDfektCKgWMj7t/RzUlR8UFL1tl0gBFhIWfPl7ObH9aWVAZyDPO4SdMaSXcZnSRNTbs0VFAlgFZZmCMQlNxggYocahFo6ic2f60sqAykGecw0+Y0izc+TljViTk5oataUAqxXDbbRpWrdJqYiCpBl5+uYEGKAGoVaOonFW0qGJXZSAxNuA1bGSXkNHbWyzkVg17chL4xCcYDeIx4o036mmAEoBa9drKme1PKwsqA4WpS8RKuK2EPKyGXYuhOZG46qpJGqAEIKjXFtd+VM5sf1pZUBk8ibGiKGtz/10I4Fg6nX7E8LeVAFoB7ATwHoB7AOxMp9NDrgcWzDOOgjAadq2G5kRi6dJxGqAEIIhxG0Y/qpSYlzPbn1YWVAZXMVYUZXM6nf6S4ffXFUWBQZBnA9iU+xkB8LeehBioCTEOo2HXamhOJGiAEoMgxm3QflRpo7icKyxoNUf5cRRjRVFmgguskc3gwvuI4bVZAGZ7FuEaw6lhe7G0KaFCDGiAqjxBjKKg/YiMYiJK3Dzj2QDW5rxjo9DONL4pnU6PoFi0XfnVWeASFTXrXVB5TILwT6lGUdB+REYxESWOYpxOp4cURbnGJMQ3AOg1vk9RlHvA54tnA5hpnFN24rX/bwo/+tzv8dRTv5sWH5ZIAA0NPi4hXFSVL2N54416XHXVJJYuHY/MWOjtbcCRIxchkzFurSZh27b30dmZrwzS3g60tV2M/v56jI1JaGhg+NjHsnj11TGMjOTPcWTEtz1EeITubTSU876a+1FjI0Nb2yTa29/B8LD75y+/vAGNjfn+CgCNjQxz5lzA8PC4wycrA7XZeOE6Z5xOp4/r/8+FrTsBXGN4Sy+A93LeMRRF2awoyj3pdHqL27HXqM9gzeFngHn511giAfXv/g7qt7/t/SpCotxzQmfOJDA2VrhLy9iYhLNnZ6GlpXA7p5/+FOjpUdHfL2H3bhlDQ3X43vfqCs4RAFpaWsI/UQIA3duoKOd91ftRPsQtAWjxlJS1ejWwdSvD0aPGegIMq1fPQCIxI/JzLyV5jNpsfPC7tGkHgM8ZPWWLeeKXwOeUXcXYCklVkXjiCahf+xowc6b7BzzipSGXe07IT9hMD80BMr77XcnyHBUl9FMkiKrCHOL2Y4BHkcTnVWArnTxGRI9nMVYUZROATRae8gUAs3TPGHzuuNXteH9A8/T/Gxp4Q5dyqiQxxidjAoixsZHPn8/wxBMyjh1zbsjlnhMqJTPU6RxJjAnCH34N8DCT+PwILCWPVT9e1xmvBPBSOp3uzf2+wCDKjxiEGOBC7JpVPUP6fVHjq587F9Kvf83foKrOB3DA3Mjr6/lG3prm3JDLnShViqXt9RzjWtyAIMpJJZOy/AgsJY9VP17WGXeCJ2b15jzh2QBWAzieTqdHFEV51/SRVQDudzvuhg1qsfhIhvnTAGJsbuQTEwBQKFZWDbkSlWf8WtpO5/hu7klQSIsgvFHJlQp+BJZWVFQ/XtYZv5T7dbPhTzsN/9+Sq9A1AmAugM3pdNr4d0vWr7cQW6NSBBBjL1scWjXkOBR28HKOFNIiCG9UsvSjH4GlEpXVj9vSphEAkof3eFrK5IpBUSTGUKrNZ9XIZRmor2eYmHBuyHEo7OB2jhTSIghveDFuo5ry8SOwcXAUiGAItVEESyTyyu/RM7bqKFaNfOFCDV/+sobBwfg1ZL+Dgd+QFs0vE7WMW4W8qKZ8/ApsHBwFonSEEmO/YWqnjmLXyJcvj/D8I6CUwcCPxU3zywRhT9RTPiSwhE6sxdito1RDIy9lMPBjcdP8MlEtRBHhoSkfolzEWoxroaOUeo1eLe5auIdE9RNVhIeymIlyIVf6BAqQDafjQYz1jmKk2jpK1Nd49dUMqVR0xyeIcmCM8DDGK9TpEZ4g6FM+zc0MksTQ3MyEzGJWVaC7W8bGjQl0d8tBFqMQFSLWnnEtpPtHeY2qCnz/+zImJwF9HbYs82S3arqHRPUTVYQnDlnMdlGBp5+u9JkRfhBXjDV3MYhDRwlKlNfY08NLhOqVyQCgro7hvvuq6x4S1U+U4eSwS2CGPa9tl/fx8ssN+OIXg58zUR7EEmOfYWqgNrIRo7pGK29ichI4eVLCzTeH+10EESXljpKVIqpRzWvbRQV+/vP60g9KlB2xxDikClxxpdzrfSk5hagWyhklK1VUo1q5YNePP/WpSQCV2xue8AeJsSBUYr1vLcy5E7WDWwTJz3aFTu8rVVSjmte268dLl44DiH6fZSIcSIwFoRLrfWthzp0gAO/Grpf3lSqqUUWi7Prxu+YtfAihEUuMjXPGHhK4ykU5wseVWu9r501QiUyimvBq7Hp5X6miGmUkqhZyZ6odscTYuFGEppW8UUSYlCt8LNL8LZXIJKoNr8aul/eVKqoUiSKcEEqMmYBh6nKEj1WV/7S0MJw7B9edpaKGSmQS1YZXY9fL+4KIKnmwhB1iVeASUIydLOUw0L3Q//yfk3j7bQmMAZdfzvDMM9mKeaJRX3NcMFY16u1tEKVJEiXgtZKW1/fporp+vYply8i7JYIjlGcsohhHHT42e6ETE8DwMJ8+L3cH1+eJ33pLQioFjI/n/1ZrS57MofrGxouwdSujUH1M8erNUiiZqBQkxi5EvfxHlI0ajOIzOsqNAVlmYKw2lzyZjaRMRsLRo4xC9THGa4iYQslEJSAxdsHKUu7s1ELLNBYlccssPpoGpFIMt92mYdUqrea8A1GMJIIgagOxxFgyzEkKIsZAoaUcdqaxKIU37EpjfuITrCY9QVGMJCI+eFkOSEsGCTvEEmOfG0VUgrAzjUWZo4qT+JRjQDMbSY2NDIsWsZoK1RPe8WKk05JBwglxxVggz9hIFOFLEeaoRPHQ3SjXgGY2kubMuYDVq2fQoElY4sVIpyWDhBO0tMknugdpRFQP0g+6+GzdmsWGDSq2bq3c0ionotpE3grdSFq7lrfFTZto43bCGi/LAWnJIOEEecY+iYsHWQoieOhulDuxSvfEjxy5CGNjEoUWawS/UyFepnniNBVElB9xxVjQOWNR5nhLJe4JJOUe0HRPPJOh0GKtUMpUiBcjvZoNeSI44oqxoJ4xEA8P0opqSCAp94BGS5xqj1Lmdo1Gen+/BFXlr/X0yNMGb9wNeSJahBJjZti1SRJYjONKNSSQlHtAo9Bi7VGqAZZIcGPx8cftDd64GvJE9IibwCVomDrOVEsCSTnrAuueeFOT5lirmKgegiRp+kkwNNY+p8RAQijPOC5h6rhCXp5/dE9827b3cfbsLAot1gBBpkK8etXVMGVEhAuJcYUpZ0IVJZCURiIBdHaOo6WlNtpkrRNkKsSrwVsNU0ZEuIglxoY541oQY1UFbropicOHZYyPAw0NwHXXadi/3791rKpAb28DzpxJ2Io6JZAQhDdKndv1avBSYiBhRigxZnJeFd56U8Plavm3ESwnL74o48ABGZrGrePxceDAARkvvihj+XLv1rGftbCUQEIQ0eHV4KUpI8KMMAlcqgr8v9vqpn//vzuO4sn2p6H++nwFzypadu6Ui/LUNA14/nl/jyW/Flb2XJWKkkeIuCNqG/aSYKh70M3NjBIDCQACecY9PTL+/df5Vvt59Sf4/Fs/wejiy4G334jURY57IQy/IS9KHiHiTtzbME0ZEWaEEeOBAQkDU21FrzcPn8HEr38NfOxjkXxvJTv17bdr2L690DuWZeC22/xZx35DXpQ8QsSdamjDNGVEGBEmTN3WxvBy0034KzyLx/AV/B4fyP+RRTePUs6NB8wsW6bh+us1pFIMAEMqxXD99ZrvwcTvWthqWW9M1C7UholqQxjPuKtLg3JtEnuOfhE/znwRf4G9+CD7A/9jhAVAKpnVmEgA+/cHD1X5XQtLySNE3KE2TFQbnsRYUZS1uf8uBHAsnU4/YvH3IQCzASCdTm/xeyLmOZQPPykDv839MUIxrnSnDitU5WctLK03JuIOtWGi2nAVY0VRNqfT6S8Zfn9dURTogqwoyiZwgd6p/64oykr9dz8YhanuWWlajCXGEJU0Gjv16CiQSgEtLQyahuli72FT6YQxSh4h4g61YaLacBRjRVFmAhgxvbwZwCYAund8Tzqdvt/w95cA3A/AtxgXYCwAEqFnrHfq7m4Za9cmcO6chDNnJNx1VzKSRC5RskApeYSIO6W04UobwgRhh5tnPBvA2px3PGR4fSYAKIqywOIz7wHoDHxmZRJjANO7qQwPSxgfjzY7sxqyQAmiUgQRU7Mh3NgIzJ2r4ZZbGNrbSZijhgwhZxzFOJ1ODymKco1JiG8A0Jv7/2xw8TUyAnCvOp1Om73qaYaHhx1P7FJNm071fvedd5CdPdvx/UE5eHAGMpkZBa9lMkBfXwaK8n4svmdkxPZ2EwGhexsNfu6rqgJ33nkx+vsTGBuT0NjI0N6u4tln3/E0qPf2NuDIkYuQyXBDOJMBBgdlDA4CqRTDJZcAGzaM4HOfG68KkRCpzQZ9drWA65xxOp0+rv8/F7buBHBN7qWZyCVtGdDFeTaKQ9zTtLS0OH5vor5++v8XzZoF5vL+oHR0yHjyyeJEriVLmtDS0hCb73G7r0Tp0L2NBq/3tbtbxsBA0iCmEgYGUujvv9RTVOnMGS4EhfDfJyYk/OpXEr785Q9j8eL4FA9xQ5Q2G/TZ1QJ+F9PuAPA5g6dsJba6OJs9Zn+UMUwNlK88HZXBI4jSCLq22Gqf4kL4NFW56gzUErQu3B3P64xzWdObjJ4yuODONL11JgA4hag9UWYxLld2JmWBhgfNQdUWQZchmldOcIrFIOo6A7XYbiu9hDQOeF1nvBLAS+l0ujf3+4J0On08nU4fVxTFLLqzkZ9TLp0yizFQvgxjymQOjihZ6UT50MX0yBH+zOvqgNZWDZ2d3sYHoyHc3y9h924Zb70FTEwARlGOUiTs2u2ePVn09lavQNO6cHe8rDPuRE5gc3PGswGsBqB7yFtM64pvAF/+FIwKiDERH6o1K70WvSavJBLAnj1ZdHTU4dQpCVNTwNCQjBUrkp6NMKMhvG6daljSyEW5FJHw88ys2u2RIzI6OuowNCRVrWFJEUF3vKwzfin3q1Fgp9cQp9Pp+xVFWZvznlsBnC6l4EcRJMY1j9MgV42bs5O3705vr4yhIQmTk8GNsEQCuPlmXgu+VJHw+8zs2u2pU+Fck8hQRNAZt6VNI7CaVCl+3yNu7/ENiXFN4zbIVeMcVLV6+2EShREWRCT8PjOrdltXB0xNFb6vlGsyG6/t7f6vh6gc4qYMSgYbgMS45nDbTasas9Ip49Qdq4zoShphfp+ZVbu98kot8DXpxuuaNUk8+GACa9YkceedF0N1L1VPCIIwuzYVEYJnTPNv8cXNA6rGOahq9PbDRrREIL/PzKrddnZqWLGiOArk55qsPPT+/nr09KgUVYkJVSvGNP8Wb7wMctU2ByWa0IiIaEZYKc/Mqt0GvSYr43VsTIp1DkWtUbVi7GUuJ6jnTJ53dNSiMIkmNKIikhEW1jMLek1WxmtjI6OoSoyoWjF2C3MG9ZzJ846WWhUmkYSG8IYIz8zKeG1rm0RXF+UbxAVxxdiYwMX8W3duYc6gmauU+Ro9IgxyBBEHrIzX9vZ3kEiIUZuacEfcbOqAnrFbtm3QzFXKfCUIwi+qyjdN2Lgxge5uOdRsZ914Xb+eJ21VexSp2hDXMzaKcQmesVuYM2jmKmW+EgThBy9TW5SHUrsIK8bMIMaSpqEUiXMKcwZNEKrFBKNyQoMSUW24TW1RHkptI6wYR12BK2iCUFwSjOIoajQoEdWIW1Ip5aHUNjUrxkDwBCHRE4ysRG3hQg1f/rKGEyfEFWcalIhqxG5qa948hu5uGY89JlddvXXCOzUtxtWOlagdOCDj8GG5YIca0TzOOG4CEccIBFFerKa2Fi7U8MQTMo4dkwtEWofyUGoHEuMqxkrUNA0YH8+Lc1+fjLvvTuKOOzRhBCRuyXEUVifsMBtp+r7F+tSWpgF33ZWcNpg5DJJEeSi1BolxFWMlamYmJoDnnpOxd68sjIDELTmOwuqEFU5Gmh7h2bgxUWQwA8BnP6vhK18Rx0Amoqdq1xkTxWutUylWcFs5EoDiXZHCoNQ1lXpy3NatWWzYoGLr1qwQRoIdtOacsMJt5zHAeheq5mbgK1/RYrdWOMo11LUAecZVjDnje948ZjE/lReMMOdlg4ZuRU+OMxK3sHo1EIc5+v5+qSgqNTrKjTe9XYsQBQrjXk5OAh0ddXjzTQnZrHt/14X7+ef5OL9ypYYbb4z+GYrcbsQVY9rPOL05ljYAACAASURBVBTMorZsmYaeHhk7dsjYtUvG+Hj+vaUIiF3jrqXQrQgDai0h4hy9VT+w8wyz2fz/K71EMox7qapAR0cSg4M8ygY493dVBW66KYkDB+TpoX37dhnXX69h//7onqGI7caIuGIsqGcssmXlBV2cu7o0nD8fbA9Vp8Ydx4zoUqn0gFpriGbo2fWDJUvs9zQ2/16pKFAY97KnR8abb8owRtkA+/7e0yPj0CEZmpZ/v6YBhw9H+wxFazdmSIx9ILpl5YcwBMSpcdda6DZOYfW4I5qhZ9cPFi9W0dSEgnNtagLa2637QCUM/TDu5cCAhKmp4tfr6qz7+8CAhImJ4vePj0f7DEVrN2ZIjH0gumXll6AC4tS4165VKXRLRIJohp5dP0gkgGuv9dYHKmXol3ovjYaDPkdceA8YPvlJZnmtbW0MqRSKBLmhIdpnKFq7MUNi7APRLaty49S4KXRLREXUc/R+PVS7ftDezrBuneqpD1TK0C/lXpoNh8ZGIJUCAIZMhnvEV16p4eBBa0Oiq0vD4sVawZyxLAPXXRetsS56bgeJsQ9Et6zKjVvjptAtEQVRGnqleKhO/cBrH6iUoV/KvTQbDvy8Gb76VXU6NK0fw86w2b8/i+5uGS+8wMf522+PPptadAeBxNgHoltW5Ub0xk1UL1EZeqV4qFZLCCUJ2LQp4Xnu18rQb2wEpqZ4YZAo55D93ksrw2FsjHvE69fnU8jdDJubb9Zw883lHTtFdhDiIcYl7GccBSKKj9HyvPzyBqxeXZytGSUiN26iuoki4alUDzW/SqG0uV+zoa+Hfh99NCFcsujVV/M5X7dlkdWWYxM18RBjQTxjQCzxKZ67uQhbtzIhOixBRImT1wWULtJBp6JKFSCzoT81xYVYNCFTVeD735cxOQkgt8u8LPMNL8wRQsqx8QeVw4wx5nJ7mYwcqKRltZSzq5brIOyxKzXZ3S1j+fIk1qxJ4sEHE1izJonly5Oe24C5hGxzM/M1FRWkNKpu6K9fryKZNGcni1FitaeHV/Dja4T5T10dcN99xQaPVanPUgsL1UJ/FtYzZgYxlkiMLQnT8qyWNdT6dRw5IhdldtbXV/rsiLCwa/svvBAsNBp0KiqsJE9Rk0Wt7vvkJHDypISbby58PYwcm2oZl7wgrmdsLIcpyJyxaIRleQLeitrHgZ4eOSfE3GqfmpIwOCijo6Ouai3qWsSu7TMW3KM0eqh+N2sI6lmHfZyw8TPmhLHhS7WMS14Q1jOmMLU7xUkfDIsWWS+0d0PU+R2/STpW1wFIOHUKFZ9vI8LDzutauVLD3r1yaB6l3/YXZpLnvfdquOQS7peUY+mPF4z3fXSUJ5m1tPB9mVU1/FKfoo5LUUBiHGPMHX/OnAtYvXpGSR1WxLBYKSGqtjaGujoUleebmqrODlyr2IkegNCWH5YaIg0qQFbfe/48cOONlR8H9fve3S1j7doEzp2TcOaMhLvuSkYSPhZxXIoKcX19EmNPGENqnZ3jJXcEc1isqYmhtVVDf79UsaSJUkJUXV0arrxSg57pqVOtHbiWsQonh7kXdqVCpKKHZvX7PDwsYXw82nMUNVwfBeQZEwAKPY3+fgm7d8sYGpLxzW9WLmmilBBVIgEcPJhFR0cdTp3iHnGtF2epNcJaflipEGkcQrNW5zg6CuzYEe66bxFrO0SFuGJM+xmXHX0QA2R897tSxdc4lhqiqq8HDh+eqokOTERHpUKkcQjNWp2jLAO7dsmYmAjXgBeptkOUiBH3sII844oRZK1kmAQJUQXJiCUIIPoQqd362TiEZs3nmEpxQyHqsHU1I65nTGJcMUSxzGspREWIRyU3pBC93ZvP8dQpCdu2FQqvaKF10fEkxoqirASwMJ1O32/xeiuAnQDeA3APgJ3pdHoo8JlVWIwrsdG3KIi0IUathKgIManUhhRxaPfGc+zulkNdUlaLOIqxoiidABYAuAGAlcDOBrAp9zMC4G9DEWKgomJcS1Vf7IwO0S1zgogzTglQcexrIhnwccVRjNPpdC+AXkVRLgIw0+ZtswDMDk2EdSooxn6LvcfVi3YzOkS3zAkirlhNBQE8Aer8+WTsDH8rA76zU4vluFgpAs8Zp9PpEXCvOFwqKMZ+lhbE0YvWjYft22UcOiRjfFysnWEIolqwM9R1T7Kvj2cf65sujI/Htw8aDfg4jouVJrAYK4pyD/h88WwAM9Pp9COBzwqAZkj0Pv1vDH9sUWotKvwkMMVtz05jJzFb5YC10VGq5x/XiIFXqv36CE6Q9u+WpHX33Uk891z1JT7FbVwUgaBi3AvgvZx3DEVRNiuKck86nd4S5KCqCjz7oyTuyf0u79qN1+f8EouXsILlx0FhixZB/fu/B5KFt8HP/EccFugbMXcSM2ajo1QLt9ot42q/PoIT5Dl7SdK64w7nWtpxNfjiNi6KQCAxtpgnfgk8mctVjIeHh23/1tvbgDP/nj+1P2G/xJ/87pfAv5R4onbs2YORD38YYzfeWPSnp58GXn65AT//eT0+9alJLF06jnffLT7E5Zc3oLHxotwuQZzGRoY5cy5geHg85BN2ZmTEfbbg4MEZyGRmmF7lHb+piaGtbRLt7e9Afzy9vQ04ciR/faOjwJEjErZtex+dnfbXV+rnRMV8b6vt+iqFlzZbSYI8Z6u+lskAfX0ZKMr7AID2dqCt7WL099djbExCY2O+D547B9x558Xo709M/629XcWzz77jSZCd7q2q8vHtjTfqcdVVfHwLU+RFGhfjQslirCjKTAAXAMzSPWPwueNWL59vaWmx/duZMwnsn/o8HsQ6JBBtSGPmO+/ggzbn8sUv6pZpE/7X/7K2TFevBrZuZTh6lBksZ5bbsMEsetHjdF8BoKNDxpNPFobgUyngtts0rFqloatLQiKRP8aZM3wgMDI2JuHs2VloabEvWF3q50TGeG+r8foqhVubrSRBnrNVX2tqApYsaUJLS8P0az/9KdDToxpWLvA+2N0tY2AgOS1omYyEgYEU+vsv9Rzqtbq35YjqiDYuxoGgYepHDEIMcCEOnFXd1sbwnearcfnoGSzEMQBAQ4rh7/9exYIFwdetydu2IbFrF//FvL2PAS+NNk7LgFSV/7S0MJw7h4KydU89Zd0RSy0AIkrhkKio9usjOEGes9fpLruVC1GFessxnxuncVEUShbjdDo9oiiKOXC7CsD9Vu/3Q74RfxS7Mx+dbsTzv56FFsLDlAYHAQ9i7LfRMoHHYXPiVioFXH45w7e+pTruk1rq+sFqX3dY7ddXbkSdGw3ynIMKkpUhUF8PzJsXbKAp13wuLY/0h1vRjwUAOgGsBDBbUZTTAHrT6fTx3Fu2KIqyFjw8PRfA5nQ6vTPoSUVuVRkTtrJZ27d5abRxSeQxGxYTE8DwMF9BFsVm6dVuGVf79ZUTkftQ0OccRJC6ujQsXKjhwAF5enXn1BTwxBP5BLBSoKiOmLgV/TgO4DgAy+VKuRB1KEuZzERqVdXV5f/vIMZeGq2IKfxWXkYQa7jUZ1HtlnG1X1+5ELEPGSn3czb23+uuYzh8GNO1ADQNOHYs2L3RRf7wYRnj40BDA7BwIUV1Ko24G0VEiVGMHcLUXkJUoqXw23kZ992nkTVMCIlofaiSmPtvMlk8RIV1b/RpNZGn12qJmtzfihnEWHIQYz1EtXVrFhs2qNi6NVsUOtO9ZyOVFDmjl2HcyowxCL8tG1GbiNaHKom5/05NFdcDCHpvenpkHDsmY2KCV/2amJCmvW2ictSmZ+xxzhiwDlEZw0jz5zMsXKjh2DExEnnsvIyTJyWa4ySEhJLh8lj1XwCor2eYmgrn3lAkQkxIjB08YyuswsALF2p45pksBgcrL3JO89w0x0mICCXD5bHrv1/9qoq6OoRybyiBS0xqU4w9zhlbYZVscuyYDFnWsH595Ys9kJdBxBEyFDl2/feBB9TQjBMaI8SExNinGIse4imnlyHq2lCCKDd++4Ld+8vRfykSISa1KcbGMLXqz5uNQ4inHF6GyGtDCaKc+O0Lbu8vR/+lSIR41Gb6XADPWA/x6FnJTU0Mra0a+vsldHfLfrU9tlhlbff1ybj77mRN3QeCsFvBYJed7Pf9RPioKtDdLWPjxoQw41VtesYBxNgY4unvl7B7t4yhIRnf/GZteYdW4fqJCeC552Ts3SvXzH0gCL9TV6JPdVU7okb1atIU87rO2A49xNPezjA0JMXCwg3bErRaGwrwdYsi3weCCBu/66S9vF9Ez61aEDUyUZuesdH8cVln7ERcLNwoLEFjRmZ+/jxfoEDE+0AQUeA3O9nt/eb+2tgIzJ2r4ZZbGNrb45FsJXJyp6jjdm2KcYAwtZE4JHMB0dT+NYbrd+yQsWsXr3OrI+J9IIgo8Jud7PZ+c3/NZIDBQRknT4oTUnVC1DCwbiC89ZaEVArCjVckxgE847is14vKEtTD9V1dGs6fL+58ot0HgogKv9nJTu+3rsIlgTHxNtGwQsSNP8xbyMoyIMsMjIkzXpEYB/CM47JeL2oPPi73gSDKRZAwrVV/NSJCSNUJEcPAZgNB04BUiuG22zSsWqUJMV7VphgHKIdpxs7CFWnOpBwevCjrFkW674TYRNVWgoZp3fIxRAipOiHi9F1/f7GBMDkJfOITTJgIQ02KsSrnPeM/jGSRVOHaSfx0XNHmTLx4rtUgYqLdd0JcomwrQcO0en/t7pbxj/+YwNmzElhun0NZFn/vYdGm71QV2L1bKtoqstIGgpmaE2NVBf6fLzfgR7nfM78ewQ+u3YcHHlAhGzLb2aWXgi1aBEiS744r4pyJk+daLSIm4n0nxCTKthJGmFavxPXOO3z5jU5dHcN994ltKIs2bdXTI+P0aRnG6ALA0NrKhDJqam4haE+PjNdPpKZ/v5SdxzdOrkTqC6tRtzr/U/+nf4rEN74x/Rk/69KcOqOIiLruzi9xu+9E5YiyrYS1P7PVOU5O8u1QRUc3/tevV7FsWWWNh4EBCWNjxa/fcotYRk28RtsQGBiQcHasBf+BGa7vlXt6pj/jp+PGbbN0q+sbHQV27IhXsYG43XeickTZVswlc5ubWUlhWmrP4WB1H5ubgfZ2se5jzYWp29oY0NyMVaM78Dd4GilMIJkArrlGwyWXAtLICOQDB/ibc0rkNyFBtDkTN+yyN3ftknH+fDI24eq43XeickTZVsIK00bdnq3yRIDwc0cqnY8Sl3Gh5sRYfzB9R29Ab+aGgvnRbAKQTpxA/aJF/M2aVvAZrw9TtDkTN6bvSZ+MiQlAL2s5Ps7n0bq7ZSQSED65K273nagcUbeVMFYXRHmOVnkiCxfy8ezYsfByR8qVj+Ik+HEZF2pOjF0fjDGLKyfGpTxMUZb6eEG/vrvvTuK55wpnLkZHgbVrExgelmKR3BWn+05Ulji0lajO0SqB7dAhGZIEjI+Hl9TmNVEuiPfsRfDj8KxrTowBlwdjIcaun6kCEgngjjs07N0rF4SrUyng3Dkp1A5KECJS6XBqObHbdc1M0GIdXjLL3cTU7blUyyqKmhRjR2zEuBawCse3tDCcOVOYqFbpajoEETbVsrzPK1Z5IqkUcp5x/rWgCWNe8m2cxLSrS3N9LiJW/CoFEmMzNSzGVuF4TQPuuivpKXmtljwLorqoFu/KK1aGt92ccZBEJy/5NnarOfr7JQDuz+XqqxkkCQVFPSQJmD9frGxpN0iMzdSwGAPF4XhVhafktVrzLIjqwqpcYhy9K6/Y5cEACDXRyUu+TVsbQ2Mjiu7/7t0yAM31uZgra+mvpdMSTpxIxMYxIDE2waR8SFaqQTE24zV5rVKeBXnjRKnobef4cQk/+pEsfLnEsLHLgwk7N8Yt36arS8PcuRoGB41VsiQMDfFn5BbmHhyUivwmxoDvfjeBqan4OAYkxmYMYmxpctUgXpLXBgakonXKo6PRehbkjROlYt5Sj1P+colkTPLx5ZZbGE6eLBxyMxn+N7fIXFsbQ3NzcZ2Eycl4TTmQGJsxhKkzowyPbIxPmKOSzJ/PIMuFkX1ZBubNi86gqbV5PiI8zG3Hik99ikXa58mYzNPebp3o1d7OsG6d6hiZM89LJ5PFm/HFYcqBxNiMQYzf+52GBx9M1FwnKcVal2zGNLvXw6BasiiJ8mPVdszIERcLJmMyj1Oil1tkzjyVNjUFPPpoQqgtHL1AYmzG0AMlpoFBKqmTlCP8ZP6O9vZwjmkM36VSwGWXMXzrWypuvNH+Gk6cKN6ijDE+n7N8efDzskLEfVOJeGBXAlZHloHbbos212H7drno+2vVmAxaJcso2KoKHDokCV/+0gyJsRmDGMvIPzw/naQc4Ser72hruxg//an73sxOmK31iQng7beBO+9MYvFi+2sISxj9GDFxqTlLiIex7YyO5iM4jHEDdPFiLTLvVO+7fX3FrrfIxmTUDkZYhZXiUv7SDImxGRsx9tNJyhF+svqO/v569PSogb7DOnyXr1Ntdw1hCKNfIyaunY6oPOa2M28eX6s6OBh9O9L77sREYcJYQ4N9n6l0olfc5rfjWDGRxNiMQYwT0CBJzLewlGMu0+o7xsakwN9ht+YPcL6GMISxFCMmjp2OEAOrthPmlIqdgNrNV996q4annioWNxGEMEoHo9KGhiiQGJsxiPGMD2rY8A+qb2Epx1ym1Xc0NrLA32G95o+TSjlfQ1BhpIQsolpwElCrvtvcDKxaZT3GhCGEdoLnVQjD6Jt2WzZW2tAQBRJjMwYxrk9qWL9e9X2IcsxlWn1HW9skurqCpS/ra/4GB81/YfjIR6Jdd0kJWUS14FZv2c/4EFQI7QyDPXuyWLHCmxAG7Zt253DffVrRferr49u23nxzbeV+eBJjRVFWAliYTqfvt/jbWgBDAGYDQDqd3hLqGZabEMphlmMu0+o72tvfQSLREvjY7e3Fi+hTKeCRR9RIrVVKyCKqBTcB9TM+BBVCK8Ogr0/GTTclceyYXLAjm50QBu2bdsbJJZcUZ7RPTPBtW5cts78n1RjadhRjRVE6ASwAcAO44Jr/vgnAsXQ6vVP/XVGUlfrvsSSk2tTlmMs0f8fwcDjHtet4Ua99pIQsolpwE1A/40NQIbTbLvHVV4uzue2EMGjftDNOJIkb+oXbN0o4dw62YXgR5tCjwFGM0+l0L4BeRVEuAjDT4i33mLzllwDcD6DmxTjOGDtef78EVeWvdXfzzcdPnIjOGqWELKIaCDPKE1QIrddU201n2QthkL5pZ5zcdpuGw4clvP124TlNTNiH4au1WErJc8aKoiywePk9AJ2ln44AUG1qALzjdXVpePzxfAEQ3U5hrHDeqbe3usJFBBEUvwLqFnYNIoTmNdWcwmVVXoXQzznbnYM52iZJvI6B132UqzXRM0gC12xw8TUyAgCKosxMp9MjAY5dOcgznubFF2UcOpSfUzLejtFR4MgRGR0ddRgakoQIF1XjPBIhHl7bmVcBjTrsajQMduyQsWuXXCB8ZrzMR4dZE+DGGzUsXuw9ilCtiZ5BxHgmcklbBnRxno2cMMcOEmMAvLP94z8mHDttJgOcOiUJsTtKtc4jEWIRRTsrR9hVNwy6ujScP58//8ZGPmc7Ocl8hdPDrAngN4pQrYmeQcTYSmx1cTZ7zEUMh5VtFDZTU/hj/f+aJu55WjAykn8kqgq8/HID3nijHlddNYmlS8d9DRa9vQ34zW8ugv3cElBXxzA1Vfj3TAbo68tAUd73e/qB6O1twJEjFyGTyQ8OR45I2LbtfXR2OlgUHjHeWyI84nZfo2hnBw/OQCYzo+C1MPqR3b19+mk+Nvz85/X41Kcmcf314zhwIP/70qXjePfd8p+zovAfAK7fb74GL+csOkHE+D0UJ3XNBAAvIeqWluBLcCJBNawr1jRxz9OGlpaWUKz3M2cSpnJ9AMCmp9SbmoDWVoahIakoXLRkSRNaWhpCuR6vnDmTwNhY4fmOjUk4e3YWWlr8rxW3Im5tIS7E6b5G0c46OmQ8+WRx2DWMfmR3b7/4Rf1/DQBmFP3uhtM5X3RRQ1mmi/yes+iULMbpdPq4oihm0Z0NoDfYKVUY465NMQ1Tew0hOc19WW3YXV8P/MM/qNOVuDo7NcuiAZUIF1XrPBIhFlG0s7DDrnq/PnhwBjo65EjE0O6cOzs1R0eA8jrsCVqBa4tpXfENADYHPGZlMW/Ay1i0m/JGgJdsQzfv2a6z/dM/FRb+EGVdcLXOIxFiEUU7C3N9fWG/noEnn4wmd8LunN0qj1Fehz1uRT8WgC9VWglgtqIopwH0ptPp4wCQTqfvVxRlba5CVyuA07Eu+JGDyXLeK9a0YHsSVgAv1rub9+x1gBBlXXA1FQwh70FcompnYfWjcq7BtTpnJ0cAqM71wWHhVvTjOIDjAB5xeI/t32KLLOczqWMoxl6sdy/es58BQgQBEcUwCAJlhYtPOduZ335V6TW4To6A13MTYSypBLRRhBUxX97kxXoPc+6LBCQ8qrW6EOGfUvqVU78uh8g5OwKy65hTy2MJibEVMRdjwN16D3PuSyQBibtVXWnPhhCHUvpVqYlVYeHkCHgZc3p6ZBw5IpuWjoU3log8PpAYW1EFYuyGudPMm8eXLW3alPDdSEURkGqwqikrnNAppV8Z+3VfXwZLljS5JlaVYy7ZfG52Ebvjx62vub+fX3MQMRV9fCAxtsIoxi71qUW2tNzIV+Vxb6Ruy6BEEBCRPPRSoaxwQqfUfqX3a0V5f3qdsigGs1vEzs73UdXgYir6+EBibIVxKZODZyy6peUVt0Za6jKocguIKANOEKopK5wIRpj9ShSD2Q2nVaQPPZTAa6/J01X//Iqp6OMDibEVHsPUoltaXnFrpHabk+ubkIsiIHEZcNyohqxwIjhh9itRDGY37AKR//t/y/jNbyRMTRW+7kdMRR8fSIyt8CjGoltaThjDztksihppMglMTfH3DQwUlrwEijchF0FA4jLgEIRXwupXohjMbhiHXiNciIvdZj9iKvr4QGJshUcxFt3SssMcdtZ3bmGMTRsXU1PAo48mcOiQhPvu05BKcQHOY78JebmwmseOw4BDEJVABIPZjQUL+JhqdHLq6oBs1vxOhvp6YOFCDZoGbNzonngqukFCYmyFRzEW3dLSMYuWpqEg7MwbPsNNN2nYs0cu2hLx3ns1XHYZw9tvA6VsQh4FTvPYog84BEHkMY5P8+czLFqk4dixfL9ubdUwNCQXOD11dbxO/uHDEu66K+k5Z0dkg4TE2AqPYiy6pQXwhn7TTUkcOiRjYoJ7wB/5CCsKr4+NAe+8A8s5mZMnJXzrWyruvDNZsL9xJaMA1TJfT9QOXlZeqCrQ3S3j+ef5GLRypYYbbxRrTAkTqyhda6uGZcs0/Pa3wEc+Atx2m4Yf/hAFAr1okQZFYXj88UTVjAEkxlb4WGcssqUF8I594IAMTeMNdmICOHuWi7JZWJcsYTh2zDrs3tWlYfFicaIAcZ6vJ6oHr0sbvay80A1n3l/5a9u3y7j+eg3794u9QqPUJZ5mo5ob/zJOnsy/Z98+GQsXanjmmSwGB/NOz6ZNiaoaA0iMraiioh/PPy8XXQJjwIc+BCQSrGBgWLdOxaFDkqXgihYFiOt8PVE9+Fna6CWS09Mj49ChvOEM8OHn8GGxvb0gSzytjGrjVBjA79WxYzJkWcP69fk9o6ttDLDJXatxXNYZ66GkjRsT6O6WoYazd31Z+exnNWzdmsWGDSq2bs1i374s6uu54Jpf1zuUHgVYv16dzqKuFPp8fXMzgyQxNDczIefrierFKLCMSRgdlaYF1ozzbkb59xQmSXLGxwvfJxpW96GvT8bddyddx0ddUN0w3yug+sYA8oytcPCM41boY+VKDdu3F3rHsgysWqXZlqwTOeyuI5qnTtQefqZKvHhxbW3MYtUC0NCQf5+IFf/6+4vvw8QE8NxzMvbulR3HR2MSbP7eeFvCZDUGdHZqwt0fr5AYW+EgxnFLHLrxRg3XX6/h8GEZ4+O8Y193HU8KiTtxMRyI6sRPmNTLygs9L8M4ZyzLvL92dblXwgMKxfryyxuwenW0O8CqKrB7t2RRrMPb+GgU1P5+Cbt3yzh9unBpU3NzfrOL7u5iodXHgLg5SmZIjK1wqE0dt8ShRALYv588SIIIGz9LG71EcvS+2t0t44UX+Bh0++35bOrubn9laxsbL8LWrSxSMerpkXH6tIxCb5YV/O5lcwtdUNetU6eFWVV58aG2Nu7xrljhLLRxc5TMkBhbwCRpuilJjMEox3FMGqhmD9JPNmtcw1eEmPidKvHSDxMJ4OabNdx8c7F4+C1bm8lIOHqURSpGAwMSxsac3+NnfLS7R26GiH4ucXKUzJAYW+EQpo5LoY9awGtYKu7hK0JcymnoujkClRAjq3OSZaC+nmFiIrzx0cu1xdFRMkJibIWDGFPikDh4DUvFLXxFXnx1EvS5ujkC5RQj/VqOH5fQ2sowNITpc1q4UMOXv6wVrAkOGq2yuja9fv7kJD/Ojh0yPvABQFXDNQTKBYmxFS7rjM1JA8ZSbpIEnDhBg2g5cLKWu7ryz+Wtt+ITviIvvjoJ47m6OQJmsW5sZFi0iHkSI6/VwXQB3rNHxtAQ30Cmvh6YNYvh5ptZQcWw5ctLvx/m8+ns5Nd25IhcUD//e99L4Ac/SOD99/NDtSQBc+YwfPvbqufqZSIYwCTGVngs+mFsUKOj+Y9pGq9wddllDN/6lvcG4YUgjUaEBhcmdp7AvHmsoKPX1/MOaszFEzV8FTcvnvBGWM/VKSxuFus5cy5g9eoZrn3ca5a2cazj8GuZnAR++1ueVf3b32J6pYbTeON0P7q6NMvz2bMni4cfTuDb305M18/PZIBMpjBhjDHg/Hk+HnsVYhEMYBJjK0rcz9j41okJ4O23gTvvTGLx4nAebJBGI0qDCxO7sJ0kFW6EMTEBKQ4UUQAAIABJREFUSBJDMsmgqmKHr+KehEJYU67nahTr4eFxJBIzXD/jtTqY8T3FSBgfdxdUfbxxLoJifT69vfJ0aNoNvVCKl3srigFMFbisCLCfcSESxsftq/L4xU/FnzA/Kyq6J2CuGHbiRPFzYYxvw5ZM8kL0e/aIaYRYVSQS1YsnvBPFcw2rEqCX6mBWhT2s0D/nNt443Q+r8xkdBXbskHH11d4qdhkLpbjh5frLQXxH4ijxuZ+xG2E92CCNJmiDE7UEqFWJTuvnIgHgG5QPDXErWxSM91ZVeQKM3xJ/oj4fghN26cbJSeC66+qwenUSDz6YwJo1SSxfnizpubsZCvaFPVjup/hzToKqqs73w25c3bVLxve/Lxf0j6YmhpkzAVnWz4VBlllBoRS3fiGKAUxhaitK2M9YnzNmTJ+bzItcWA82SLZkkM/GLcTtVmJPpLCv1b212qHG6T7H7fnUImGuwlBVoKMjicFBbmACxXOuPT0yDh6cgY4O2fV73LK07Qp7fOxjDIkEcO4cLLKX5aLxBuCCev58Evv2ZW1LWeoZ2m+9pZcG5dc5Ps43jPjnf84ikdCKPrdzp4xz5/i2i6tWcSF2KxTi5frLBYmxBUzKi3Hfa8CidutEAHMHmzePQdOAdesSNg00GEEaTZDPhjmnEmUSmfHY996r4b77NDz/vIxdu2Rh9mE2Y3VvrXao8XuMUp5PtSX4iUZYa5J7emS8+aZZHLmR2d8v4fHHdQGagSefzAuQ/tnjxyVoGj+f9nb+nJ0MBbvCHnfdpU1XzDJ/Th9v+vrkIkE1tk27UpaNjcCHPsQwPFx8jSdPSrlIWP71Zcs0/OAHMl5/nX9+3z4Zra0ahobc+4Uoy1VJjE2oKvBvb8m4Kvf78ANPYN8Tl+LWWxkkGYAsQ/vzPwe7/noA1h1s+XItkgcbpNEE+WxYySdRenB2x96zJ4vz54tfFyV5K4x7G8YxyLuODwMDkmUSU10df45Whll3t4wf/EAuWBoE5Os+79uXtTUUrKJqzc1cyO0MDH28ufvuJJ57rnBKyKptWu1rrKp87tdoSNfXA6dOSejultHZqaG3lxuP2Wzxdb/5poxstvC87PqFCFUKSYxN9PTIuPQP+dHnDvXHwL8DeCz/Hvboo5j8+c+BOXMsjxHlgw1y7FI/G1YxgSizFu2O3dsrC2H12hHGvQ3jGKJklBJ57CIV+vMuNMAYPvlJLo5WhtkLL8g5Q6t4r2C351xqVC2RAO64Q8PevbJr27QyKCcmgCuuYBgexvQ04NQUsG2bjH/5FxmpFJ87z2RgmWU9NcXFe3LS+btFQZwsFkEYGJDQp13n+B5JVfH8hpM1kygTVvJJlFmLTscWaR9mM2Hc2zCOIUpGKcHRIxVr1hQnaHV1abj2Wg1NTTxhqa6OYf58DQcPTqG9vTgZqb4eGBy0z4Z2e852qxa89COvbdMqiaq5GXjkEf59X/iChvp6QNN4dnYmI+HCBUxna09NFZ+/JAGf/CSLzX7H5BmbaGtjuLvpIRzLLMRH8X8B8DqrX/hLDVed2AZ5oB8A0L0tg937kjURygtrTiXKcn1uxxZ1PjSMexvGMeJe17facItU2D1vq6TSqSk+z2qHl+dcalTNa9u0875143lgQCra59kNSQIeeEBFfT2EjIqZITE20dWloe3aOuw7urqgUXzth1mcveXfcAW4GDdjtGDtXJxDeV6EKozQu97h9Hmrujq+5rezM9rkNtHnQ8O4t0GPIUpGKcFxywNwm6vt6ZGxdeskurubMDFh3t4wjz5nHOVz9rpblZNoWxmLbmga8ItfFCd7uVEpw53E2IRTozj3+w/gitz7PojfAxBrmUwplFOoEglgz54sOjrqcOoUT0IZGpKxYkUy8Pc5PTcv26/VOqJklNYCXgb7IJEKXfxefTVr6U3qpWF1Y7jUAjhhi5aTaBfX3UZuzphNl7ydmipcidrc7D+yo4+HRofhyis1HDyYRX196dfmBRJjC+waxayPNQOH+f8/gD8AiH8or9yJO729vMC8XlvW7/c5DQB2z41KTHpDhIzSaser8es1UuHUH+zyWRjjfU83hnt7S1uiGLURb762PXuy6O0tXJes/z5vHsMTT8g4dixYZKenR884z9+jwUEZHR11OHx4KlLjlMTYB3Ovbga28/9/EL8XMiHAr7UatlC5DQ7bt8tFoSav31fqAEDzoYQoeDV+vUQq3PqDF+EYHQU2bJCxfbtcsONSWNdRKk4eqnl9sf77smXFS0oBHhkLMh4CEk6dQuSRNBJjP3zgA9P//fTH/4BnHs6GuiOTH6xErxSxClOonL4f4H/r6ytO4Pf6faUOADQfSoiCH+PXLVLh1h/mz5+0WAJVzMmTMk6e5Iby9ddr2L/f3bsNy4ifnAQefjiBvj4JS5YwrFvHE67sPNQlS+rw3/+7arlNrfl+ed2NyjiOXn01Q12d9TKpqCNpJMYeUVXg0c0zsD73+weGBnH8a88gMawWVM+MlBkzoN14I9T6RstG9ld/1eC4LZmVxxqmUDkNDgD/vzmZpKHB+/eVOgDQfCghCl6MX6/RLbf+sHTpOObO1TA4aKzWJaEwiSvfHzUNOHzYm3cbhhE/OQnMmVOPCxf47z/7GfDDHyZw9uykrYf6xht8JzxjdUM7Z8PNWLErRfvJT2o4ebKwwlk5Imkkxh7p6ZFxYuiD079fpx3CdW8eAu4t73loN9yA7r/bb9nIPvShJsvOWVgir7gRhyVUToMDY9YW+q23anjqKW/zTH4GAKsBjeZDiUrjZvz6iW5Z9Yf6er6fN8CN0FtuYTh5EkWbPFxxBcPbbxcvd/K69WAYRvzDDydyQpw/jwsXGB5+OAFFYUgmUVRBizFgfLy4upi+/Env64D7lJhdKdr/+T+zeOghHpqemipfJC2wGCuKshJAK4CdAN4DcA+Anel0eijosUViYEBCemI+NEiQUbm5RunllzGw2Fr0JAmWYmVXIk+3EIMm7ujC99ZbElKpwvJ1emWcBQusS+qtWuVd+P0ktYi8lImoXdyMXz9TMV1dGhYu1HDggDydRTw1BTzxRP69ehEQc7/7whc0fOc7iaJsa69bD1rV5ZckYNOmhOfM6r4+67XPhw5JeOABFX/0RxrOni2uwW1kdBRYuzaB4WGpwLvlx3GeErNzHn7xCwmHD0+VPZIWhmc8G8Cm3M8IgL+tNiEGuBX6neaP49bRXbgZeyFDQzLJ8KfXM3z0o9GLc+JHPwIASNks2j6toakpUdDBkkngj/4oi4ULtaKMQrsSeWHMgRiFTy8yIEls2hKfmgIefTSBhQs1y3NzsjbN3m1np4Z779VwySXc8Lj9duuEk3JliItaSIQQGyfj1++c8pe/rOHwYXnaW9Q07t319MhQlLxg8/dwsV24kG/w0NcnFQi5LGN660E/19HVVZrxu2QJw89+Vvz64sW8rOeddzI89JD5r4XCnEoB585JBd7yoUMyJAmuU2JOkbZKrCwIK0w9C8DsahRhHd0r+z9H/wJ7M38x3eBW7s0iW4YBWH72WUi5XtN1QxaLFiUKir5PTQFPP/1BLFrEirbf6+kp3s4srDkQs/BpGpBM8uNms4XhH+PWZ26WtNUuLsZatE1NwPnzwI03Fg8cdnupfv3rCWgaQkm6C9P7FkHURTgHwv9c7IkTxZWp9KmpkZEGDA0l8Lvf5fci1v9NJID9+7Po7pbxwgvcg7Qzbt0o1fhdt07FD3+YwIUL+WubNYu/DgDXXMPQ3GxX6IMnWlnt7GRXqcs8JSZaYmcoYpxOp0fAveKqpeJJQHV1060soU1h374EHnoogW9/OzG9ZjeTkXDsGCvafi9Io3MbpK2EzzzPw88tv/WZF0vaaheXTIbBav9Wc4e3q9YzOCjhjjuSnjNGnQhz28JKh9RFOAeC47evWrX1hgbgRz+S8ZvffNiQFczb6cRE3nNetkzDzTfznyCUmlhZXw+cPTuJhx9O4NAhCYsX57OpAbttGDnJJHDZZQznz1vXpDbPkVtNiVV8TDcRihgrinIP+HzxbAAz0+n0I2EcVzQqWhQhmcybfNksEg3WO5WMjgKPPcYtXb1hldrovAzSVoNBKsU7hN0ewl6EzDqbshC7Dm+3lypQmDFql2HuhbCWdoiwW5II50Bw/PRVVeU/LS1sev/0xkbe986ezbd5M2EXvLFLJNO3OnTqV/X1wIYN9rvt3HuvhosvBl55RcZ//AebzqLW9yo2h6Lr6nRnoPD11tZ8Ypd1cmepVx8eYYhxL4D3ct4xFEXZrCjKPel0eovTh4aHh0P46trhjxKJ6eb1zm9+AzZzJi6/vAGNjRcVbYv2yisyjh6V0N6u4tln35nuCIrCfwDg3Xfdv7O3twFHjuSPPzoKHDkiYdu299HZyZW2vR1oa7sY/f31GBuT0NjI0NbG9ywbGCh8rb39HQwPA6++OgOZzIyC78pkgL6+DBTlfQCwvTYjjY0Mc+ZcwPDweNHfnn4a+C//ZTZ2724q+tv4OPDaaxl85zsp9Pcnps/RfL+MjIwUBn6szs/pfOw4eND9XkRNJc/BfF8JjltfVVXgzjsvRn9/ApmMhFSK4Y//WMWKFRk8+eQH4bQhXynt1AnjGJDJSKatDovHISdUFXj55QYMDtbjJz9pxNmzvH82NDBccUUWN944hnnzJjE4WI/vfW9G0ec//vEpvPlmXdHrN9zwe7z77vsF981Lvy8ngcXYYp74JfBkLkcxbmlpCfrVNYVkKIx68axZwMUXY/VqYOtWhqNHmcEq1ZcRSRgYSKG//9KSvZszZ3iDNTI2JuHs2Vloaclbsz/9KdDToxosef4Z82uJRAtUFejtTRaFkZqagCVLmtDS0gAABddmVYuWe+kMq1fPQCJR3CkB4D/9Jxkvvlg8h5RKAQ0NzRgYSEyLqZf71dLSMm1VDw1JmDsXGBryfj5WdHTIePLJ4jlC472ImkqfA40F/tm3T8brryenE5cmJiS8+66Ed975gOV2ggCbXm1h106D5A3oY8COHTJ27conlPkZh8zJoBx+nLExCb/6VR0+8xkJy5Y1oLtbxpYtxVnit94q49FHi1//zGd4W+bVuJK++n25CCTGiqLMBHABwCzdMwafO24NemKEiTqDtZeLTRtDWo89JuOVV+QCkQsajvKaTGIXvrd6radHxunT5uUKfK9RTeMd0i60bqxF6yXU3tWl4brrNLzySqGXoIfy/IaZzSX6eAa7hjVrGBYsKG2+SYQkEhHOgfDO5CRw772JgmkgQM8k5ssLCw1QhjlzGO66S8P8+daJk0HzBvQxwGqrQ6/jkHm6xIzxOHZtdt06FYcOSbZt2S658/XXJQCVTWAMI0z9iEGIAS7EVZtVXTGMLcOQIaV3AgA4elQqCJsGzZiOYpAeGJAwNlb8+vCwhLvuKtwf2krk/czZJxLAV77Cl3UY55YmJ/nSK78Z5uYSfdkscPasjD17GNavL62IvAhJJCKcA+GNyUlg/vy6XAZxsWjxuWJAr7JlrOmcSBSvUJg7V8MttzDXWgReCVKZyy1PxHgcpzbrthVjY2OxIf697/E3VDKBMZAYp9PpEUVRzDMaqwDcH+S4hAVJw6MyZ22BC2d7u4qBgVRg4TSGq+69V8N992kFS6X8NFBV5YXan3+ee6eXX26V6Rxd4tCJExImJwtfGxvjnXnRIi7UY2N8jeWHP6xh6VL7742qiLwIuyWJcA6EM6oKdHQkHZKzCqNNF1+s4oc/ZNPLlcxbiWYyvN7zyZPWyaClRNaCGPDWqyCM4fXC47i1WfNUmH5+ViVCR0e9rdSIkjA84y2KoqwFD0/PBbA5nU7vDOG4hAFWV5dvOqpaVAMskQCeffYd9Pdf6ujduM0Lhb1+9qabkgWFBSQJ+NCHgOZmVjQvBPCOsGOHPB2SDho2srPU29sZ/uEfVHz0o3wuXtO4l9vaWo+zZyct9y5ta6tcEXmC6OmR8eabzhWp8kj43e8SSKdVnDjBQ9L9/dbGJGN6my4cVUqJrAWJsljtWTx3roZbb2WexwC3zWp6emTMnu1+LpXYYjWMBK4RAFW5lEkoXDxjwN1S9CK0YS5z6emRceiQDE3LDx6MAWNjDP/1v6p4+20pl+xR+LkXXpDxk5/UFxT4KNUgsOrgra0a+vsl/PjHyZxIF9bGXb48ia9+VSvq/F1dGq680mxV03aMRHkYGJBsun7eqyt4lfG5YSC/HMgqRGukvp4FrsdsNQ55SQ4LY7rEbvzq7pbxgx/IpuQweyrRp2mjiLhgTOCyqqrhgVLX95ZqJVolcwA8uaSuDnjqqSzOn08WrQeemAAmJoKHjfQBoKODYfFivrvWnj0yhoZkfPOb3Eu34sABGa+/LhcZAIkEcPBgFh0ddWUvIk8QepSnsH/qGypYCXK+8tboKPDLX8q5tmy9a1NTE/DVr6qoq0OoeQN+om1Bp0vsErQef5w7BoWZ5jwErq/UmJhg03snt7Zq6Owsb58u1+Z/RFA8eMZuOAmtjt7hjZRqJba1MaRSxa/rxeh1S/jWW90bvfk83dAHgDVrknjooQQefTSBPXskDA1JGB2VwJhU4LHn4QbB6KhUsP2jTn09cPjwFJ57LosNG1Rs3ZqlSlXI5wZs3JhAd7cM1b6OA1EiXV0arr1WQ1MTAy9wwTB/vobz5ydx1VX8NXOo2cj4OHL5E5Lhh0GS+GqGa6/V8MADKtavV6c3kAkDoxPAGO9bfX0y7r47GXpbsRq/ZBno65Mth83PflbDs89mMTQ0iblzGerrua8zNCRjxYpkWdsxecZxIQTP2C3T0aqij9POSG5hp64uDYsXa47F6BMJ4I47NOzd6xw+8msQWEUB3nzTukPaeQqlbvpea+i5AcbNCK67LnjJUaIQpzDu0aNTePBBXh7XaXiwE6SvfKV4WiaseuVWTsDEBPDcczL27i2OQPnBajMZ47RUfT2/Zr1OvpHmZr7aYtkyDd3dvHaAXlp4dBR47TUZDz7It3M8caJwe8YoIDGOC8aWWqJn7JTpaF5wn0rxzOdvfUvF5z+vFTX4FSvcw05ei9F3dmpobWV4801uZ1gX+PAXCu7vl4rE3e62dXRoeP/9YrGu9blgr4Pxiy/KOYOLD2Tj4zzU/+KLMpYvj3/4XqRNNMyGoB6RGBiQcM01DJ/5DN8ZzSo5EkBRAqJRkHT0Y65dm8C5c1KBUV6KaFpnSQfPSbELf+/Zk52uR3DqlIRt28wBYO4Bm9cfW40XDz+cgCxjeinkokUaFi3yd/1eITGOCyF4xk6WtXnZw8QEMDzMG6FZePW6sFZzz1b1np2K0asqP/7p0zw5pa6OZ1AeOJDFyy+XnsjhJ7z0Z3/GC9RbdexanQv2M8+3c2c+8qGjacDzz8dfjEXeRMPq3BYu1PDMM1k8/7yMF16QCtbXNzUBc+cyDA3Bto3rxzRvzhBENI1OgJWRUGpOil0OTG+vPG2wdHfLRVG3ujrgv/03FQ88oE4/w/nzGWQZpnacr2dvPD6Jca1jnDMuUYwB+xCr3XzyCy9Yh3vNpzA6ymvRPvaYXLRnsW6pWnkWeofSC2lMTfH5mt5enmxitVbQC7LHbAh9mZPXTE7dSzp+XIKm8fvZ3l6atySKx2V1HrR5BEfk+2B1bseOyZBlvlXgr37FiuoOGL1GqzauH7NwAwZOqaJp7Fv5cpn5v5cagfKSbGoXDTQKMWCfzGnGbfOaIJAYxwRm8IylbNYhTaMYL4O+3Xwyr3Nd+N6pKeQyOAtff+EFnoyhhytHR4FXX5Uxf34dfvtbyXI+0S77ce3aBIaHpZK9kQULeKWd4mpfhdmUc+eygvlrL0vDjPtIAzzU5/f8wvC4whBzu/P4zGeY56z622/XsH17oXcsy8Btt8VftMNcXRA2VuuGjef27LPvIJ2+FC+8wMvkrlypubZxpypYVqLptQ0mEpieDjt8WHLNSfGCl2pfTka28dzfekvyZPibk8PChMQ4LpToGXsd9O0syJUrrZOr+CnorZdnZloVDuAlI/MVg8bH+a5Sf/M3SaxereGqq3jGtdFSTqWAc+ek6WLzpXgjXV0aPv7x4ko7Zm65xbuAmb14nVLOL6jHFVb41O48Fi9WPZc1XLZMw/XXazh0iIc2Uylg8WKt4p5jGAQp7xglqgrs3l0sIOZz09fWZjLwlCxlVwWrocE+pO2lDTrlpJhzSLzitdqX3bpn47nX11vvg6xfv7EKWFTQ0qa4YLFRhBeslhVYLdnRLcitWwuX7Nx4I2/wzc182YQkFQqwezWg4vcwxkPad96ZxJe+lMwtt+DHl2WGj3yE2Rab90oiAdxyC3MMPzU38xCzjtvyHCevwe/5eVlm5nROXp9rqeehlwttbs4vfbHzYPREvR//OIuvf13Fj3+crZpMan3A93IfyoWqAg89lMAvflG84Ypx396XX27w3UaM18sTnRguuYRhxQpeFlf//u5uvjTp0CFvxy80+vhc9vAw326x1HZiN2Z5OZ65/0xMSLnIDiv4SaUY5s3T8E//lD9+VJBnHBdK9Iz9hNnsQljG+Z7t22Woqr3oyDIgy8xig28j+dq4mUxhsYK6Ooa//EsNjz+eKLDOGxu5DbJxY8I1JKuHn375S6nI67ardevFyrf2Gjh+vSUvHpfTOYUVPnUqF7puneq5GlK1LvcSbRMNvU289lpx3gZQGOl54416320kkQD27Mni4YcTOHhQwpkzEn73Ownbt0vYt0+Gomh47z0Jb75pXQ3M7vh201GPPcaFu9R76mVqyRxGB4Dt262WUubHoWSSYeVKDXfcUbzkKypIjGMCk/Ot4V/TWXzqdm8WZRhhNr3BHz8u2WQp5wVu4UINCxcyfOc7iaIMWy9MTvIggLmEZSoFPPpownc4TDcOGHOudeslbKx7DXZzxn68JS8hNqdzCit86nQe1SqwfhHpPuhtwmrPYnOk56qrJj23kclJFAmwOfN5dBSG7UitDW39+OZEx9OnrQxj4Gc/k3HwoDy9s5RVTfhSscs2B4BDh5wjSKoKfPKTLHYbRRARo6rAy6+k8Oe53ye//zR+tuNVfP4GrSAMO2t8HMmGws3gb2bA9hky3pkA1CyQSAIXzwA+v1uDtCf/Pm3RImh//deOacjW4spw8cUMf/d3GtraGJYu1XDFFfWGkA9gDqU5hbatvLKJCb7FmZc5ZLOAaRqQSjHcdpuGVavsrVwvnqbRS+rv54ZJMomSkqe8eFxO57R2rRrK9paieX6EM9ZTJcXrZgFg6dJxT21kchKYM6ceFy4YX3WacjH/jfdz3Sjt7NQsEx11wzg/juRXUAwOyujoqMPhw6VtRWqFlTF76JAMSYIpW7x4nKLa1IQlPT0yRn6XnzNeor0GnHsN+FHh+z5g8/kCgz4L4ByKPpt45hlMfeQj0BzMf7v5189+luH++/lSgW98I4GREcDbzjKcVIphcrJwEweAr1nu6gLmzasrsqj9hMMmJ4FPfMLZyvXqaYbpJbkdy+mcwhRRkTw/whmrNmG1bhbwbmg9/HAiJ8TedoMyk0jwwjmXXQbcequGjRsTeO21Yu9dN4w/8QmGwcHiOtpBtyI1Y1f5yx7e12WZe9DlzgsgMY4BAwMS3sr+Gf4aT0X6PdLgIJxGZLvU/927ZSxfnsS+fVn09Vl16PyG4FZh7j/5Ew1XXQX8/OcSTp+W8eCD+U3Rv/Y1Db/5TXESWCplbbmWGr4Nsg9rVLidE4lo7eF13ayOlzZi3We9wjOtX39dxoEDfC6WMfuxYmICmDmTIZmUiua8Jyed55D9LuW7+mrrlRqSVBwuN+et3Hdf+aNDJMYxoK2N4TtNq3Fd5gpchTf+//bOPkaKMs/jn6qe7mEYkHGEQUBA1CWcwDliqYgr0V1WL64IBpWNyW4um1U8454XL0HEsEYT3+Pd6vpy4Jn9ZzeKuIgLepmTy/qG40uLEwUdZQGZUcHh3XEYZqa7nvvjqZqurq5+re6Zwfl9kg5TVd1PVz/U8/yel9/v+wOgOqb4zW9sZs9OGYvOzk5Gjx5dVNlmUxORDRv0wfff535v4Aq2duRw05SNGxf82bo6bSCDjPH27SY7dujGqFT60tVvfxuU+Ul7XAcZylKN6lBcrh2K9yQMLpV4JubNU7z+eimf1FtUx44ZadtC+XjrLbcjydyyeuMNLRrk9wkpNpQvmYQnnjA9kRraCJ91lk1Xl8G+fXpgUFWVGZzS2wvbthksXFhwRZQFMcYnAFdcYXPBhSbvv38h7x+7sP9BnPFIAtvzIHZ1dFDb0FBU2caxY+AYYyOPMZ4zJyiFm/PdjlDHvn1u40oNjWtr9QPvT1/m3AFKuctH/uG0waFDWijEP7p9+OHsM4FSO6uhONMcivckDC7lfiZWrEjy1FMRjhwJ8vHwkt4+TVO37QMHcpXuT8KSfeXM7QuCfEKC9n/fe8/kvvsigX4bTU3aqKdnZlPs3KnVv6qrYepUhWUpXn55aGjSizE+AajkDEnV1qYO8hhjd9b5xhumswyVetD9Qh2gnTWWLLGZPl1x//2l3WwiAZMnKzo60vV0c+0riQEThjvJJGzePIIvv8wfChiLwerVCW64oSpn2KLfkNo27N1LoJe0lylTFG1txS2F+31CsjkzPvhgJC2Jw8sv637ynnsiGaFLStHfP/X0QFub7rO8YkWlREaUCzHGJwgVMzCjPG5fuXIYOvdw662u0lL6iHPMGMX+/ZniHmefrTjnnMx93Ox7N+lLV7W1qVmwLNUKQn5Ssq2n0N1tFKTOtn17trDF3PT0wLRperCcLVPUvHmKgwczsyK5RKO6b8mlV50txt8rvfveeybz5kX59FOjoOVy2073qo5GFbfdlukIN1CIMR7ueIyx0dmZ9+0ff2w4+zDpXHaZYtMmI9Bxyh+fG43qvRvDMPrTJqZIN/Jjx6r+ROcy0xWE/PhlWwsaB+PQAAAQIElEQVSRWm1sVNTW5h2PZ+AdLGdLAnH66YqGBkV7OwFiQIpJkxSnn64yEsx4Z6f5Mj+Bnim3thq+pWn9HUHv95NI6EFBuRLBFIsY42FOcuQo3KCpw+1d1CRzi4kEjVBra7UIfUcHWcUj1q9P0NgYpb1dLwt99llm2r2gxjJtmm5Ibs7W2bN1nOJLL2knkGuvzcyPLAjDmVLU2YIFbXLvHbtLuu5g+YorbPbtq8oQ63HV9KqqMvWfq6vhkUeSXHmlnXMbzrtV99hjpkd8JP092cUJgwx0+rlYTEeGPPpoev1VVcHEiTZz5+p98qlTs31HOMQYD2OSSfi3lWNY4x63/p2/zriLJUsURhbtj6ts+O96g729Bok+qIrChHrFoi2KRefA7jGwv8NgXINi2unAun+kb8n1zJ8fS0sYUag619y5KlBAwOWFF0zmz7d/MFrIghCWUsL7vMbuww8N/vQnk2++8XsapxKuBC3p+n1b+vq0ap7rdJVIaD+S6mqVlrEptfKVexvONfh33x3UOSlOO02xb19Q9EUh+9WKqirFjh3pfi/ufbe1mbS16eM77iiguBIQYzyMaWoyeW/7Sf3HDXTwi/ZH4fe5P/cL70Ef0E7/Z6Y7Ly9bt4+gtfV6ihECAdUv7/f445GMTEkutg3vvjs08ssKwlAgNcs10vaMCwnvcw3iypUpBbzWVqM/ftglkUjt9WYr4/77IxkDaNvWwiDTp6uS/D+amkx27sxMkDF1qqKlpY9rrqnizTfNDJWvdBSmmTkhcAcNwYSJxS4Mydo0jGlpMfis+3TaOa2i32O/uaWYRFO48YtffdXLtm3ZMyW5HD9eXMakwSJfVihBKAfuDPWJJw4Wnc3IW8aVV9rceWeSpUvtjDy+hYT/uDN0L7W1cN11ulx3RlwMLS1GQI5y+NWvbGpqdPawtWsTzJ9vp+XW0ejZ76xZNpdckspMlVqyLiQLXeWQmfEwprFREauNcnHXFq7hJarpIeZkTZoxI2Sc3SfbqHr+OQBGH/k6ID45WLfaMGDmTJt33tGdx4YN+RtHLFZ4RqdsFKvuUyzlyj8sCIUQicCCBcdpaAg/4itVSKcSqnbZfFbcBBmRCCxcqJe+/fmTJ05M5U8G3d4ff9z0hGoOLmKMhzGpxjKZPxz71/7GsvK/EiRDGIhkEu78cTOPoY3x+M/f5M/RpSQdOUzTgFGjFQbQ2WlgK+3YUR1TnHuu4tQJsOeGSfyH+e98/vlkco9WFdFoYRmd3HvzG91kEi6+OEprq5boq4ShLCQrVDmo9KBCGH6UqnNQCX2EQg18Id995ZV2/zZX9jjp3IltyokY42FMpcREmppMXmtNuRyO4wCL+9al3qCA73wfUkAP8K4+PAu4iEM87c9oEcCxYykZza4ueOcdLc25cGF6A/WnV6yuhgkTtHe2FiWonKEsV/5hP27qu3feMZg7V9HcbBCPy+xbKC+l6hyUWx+hmD6rkFzHfslMIHA/eSAQYzzMqYSYSEuLwefdU9jKuczho5LLaaSFQkal/iWmnh649dYIV1xhp+VH9c9Oe3rgyy/dq+nf4098HpYg0fqwsnv+1HcpfeHUoOLtt7Vk4GAJGQhCuSlXnxUkmRmLKW6/PcmePQYvvhicN7pSiDEWyk5jo6KmNsKPu95mAZsZwXFGVCtuvTWZlvw8iP95vpOrN/0LAOP51jmba6ko6LzBt9+SkR+1pSVIBSh7Y3v9db2fdNJJioceGsEvf5npPepdFp49W+95f/xx+t8zZihWrIg4hrh8adqCU9+l129fHzzySITmZkNmyILgIWi1qq9Pr5j96EcqwOm0shvLYoyFspPa16lh07GF/culs+5NT2wRhB1LYm+6BRPFWA4QIYFtRDBNFeB9nDsUwZ8ftbFRz05z5zQFr/FXCo4ehZtvHstzz9ls3Jhg8+aU8X3ySbN//woyRQ2y3a9tKzo6DDZtMrnqqtK2BrZsKSz/rJtVS8K/BCFFvnhsvyJZLAa3356smLOXGGOh7ITZi/7pP0U4aIxlnNqPiaKbGpQyiADKCcTLli91LxPopiZ1ohfG3gjRU/SbFwHbbYNCo6yOMob7uIvdTNPHbyv++TxFe7vBweMj2WOeQdJOD4copqF++qnB0qVVjBunuPRSxbRpij17DAxDx2JCutLY5ZfbaQOB3buDSg020F1dsHWrAYhzlyBAfmewoGurViW5997K3I8YY6EilLqvs3mzyVRjCuPUfgCiOPp2BURoTKUt8+RB5+VwZnG3w19ZlDpIAjtSh3vsKdzCk4CBwmAXZ9BBA13U0kt1npJTRnP/fli3Lt2IPv+8mfM4qJx8PPBAxLe6oKip0eINM2cqLrtMe7PbNvzlLyZ798LEiTouNEhytLdXh5Nt3KjvYeZMuP56PWhoajJZu9bk00+1c51Siu5ugwsuUKxZk6CmBkEYVPJNGgY6l7gYY2FI0dJi8Gd7BatZxikcGuzbyclU2niFzAzkSUze4hKe4caiyzzAWNqYQjc1HGUMRzjZc9W/d17cepk2xOnGu7tbv5qbDZqbgz+3dq2JG0JmGPqVSJCRbm/7di1PGox+7549BuvWxaiu1gOB3t6JJBIGsRicdJKir89gxAiYNElx9KjO6mkYBlOnKm65xSYa1fvwMrMXykGuScNAp2IVYywMKRobFY/WLqGh6xoiznS4dqTij39M9AfrJ5OweHEVH3xg0uXs1Y6ghwnsxUARrVL8+tdJbr45uLNOJuHpp02eeSaCymLcrmMdV7OREaTcn6NRSPTBLLbl/A0RbC7lDS7ljdIqwcM+xnOUMWnnlM+gFnvsP3ecEfyds+imBhuTJJG0l43JQeo50leXUUY7U+iiNqP0fLHh9AA9nvvoBbzptNshbfLcBmve0jHqSkF8BLw9w+a++23MMDqCRkhv2TCfr/B3xw4fxjj55Oxv+AH/9op/vgKIMRaGFOn7OCYjR8K5F9hcfhXgCtIDG17VzlkffWSwYYPJrl0xdhwb3b+3s+w/E5iR4LmjCdz8e9i4o8rJzaydM6ZPt7n6asUXXxg8/r+ruK9zFUppz+fp0/toblbMn19F+ydHeYCVTKa9v8x6DjGN3Zza7wFeHk7l27KXGYTFhxX/jrLg/oceB1oASauZlfGDfQM/VCqUKUKMsTCkKNT5y7uEtGJFsiR1oFdeyf49bsiSe+3cc7+lpqaBLVsSXHxxHbd9/jS9vZne02PZz13czzj2F/3bY/Qyjd2MppORHGMyXxVdhiAIJyaGGgRRzhUrVqh77rlnwL/3h05HRwcNDQ2DfRs/SLx16zXUs2Zph6f1602++kpL63mdpEaOhGefTbBtm05L9/XXRo6cq+mMppMJ7HWOdDs1fJreqWPF6FHQ26Po9biLGwFLxgZ22vWptHEKB4mQxMT2LVInqSLBJL4m6vNDH8NRJvJNYT8mACNk3GaVqfiHsxWn1JdYTti+L8znB+C7+/r6iEajwRcH87eHZZD/31b+5Cc8+OCDZV/nLsvM2LKs5cAuoB4gHo+vyf0JQThxCXLsWLRIGzhXnrK52eCiixQrViSJxeCaa9LT0nmN+N690NCg+4jWVv3v/v0mhw+P5ovEaACiUUVVlV4y7+42MuT66uqgeXcvf/ubXrpPJrWk37PPRtjvm6SbpiIW04OKvj6DbcyuZHWVFdPU9eOV+uwTJ65AZHBeIe6+uyLFhjbGlmU9BHwQj8dfdI8ty7rWPRaE4UQsBr/7XXAcVi4j7se/TO5fQt+40eSpp0wOHzb4+c9tVq7URt9f/qpVSV591WT9eu3ltGRJepiSO3jYskWLgxw9qo1dXR0cPgzff2+QTCq++04nbXe9qW1b/55Ro2DcOEV9vS7v0CH9GaUUpmkwebJi/Hg4cECrG7W1wcGDelKhvam1B3UsBmPGKHp7U97U330HnZ2Z3tSffDIwoSaCMJCUY2Z8Uzwe9+5ovwbcAYgxFoQSyRdysXixzeLF+dW03JRy/qQZLrkGDwNBKbO3q66q0M0IwiASJigAy7LmBJw+BCwIU64gCIIgDCdCGWP0HrFfmeEIgGVZdZlvFwRBEATBT9hl6jocpy0PrnGuxzHMQdxdoU1wQRAEQTjRCGuMg4yta5yzahlWwi1cEARBEE5Uwi5TH0LPjr3UAcTj8ayzYkEQBEEQUoQyxvF4fCuZs+N6YHOYcgVBEARhOBF2ZgywxrKsaz3HPwNWl6FcQRAEQRgWlEUO06PAdQZwRBS4BEEQBKFwBkWbWhCGIpZlrY7H48t853JKvYoUrCCcmDgruuf7RKvca6HafSn9woAaY+m4SsepO4Dz0fKjDwdcF6NRIo6s64J4PH6e71ya1Gsxx8MZR2fgTuAD9DMXd3xM3OvyvJaIUzeur06d9AXFYVnWAmAOekt1V8AAPFS7L7VfKMeecUE4N7QrHo+/6Pznn+nbaxay4MzYHnZe1wFLPcY5b91K3efGsqwzsly6ydeAXgOWFXF9WOIY4v+Lx+N3eOrnTs91eV5LxLKs5U4/sMapm83SFxRHPB7f7AxgtmZ5S9h2X1K/MGDGGOm4SsLp2Pwe66vxdG6I0QjLAnSd9JNP6lWkYHPyEB4nTqfTv9FzXZ7X0lnqPXBWG873nJK6DUHYdh+mXxgQYywdVyjqgeUBs7c6EKMRFmfJ6oWAS/mkXkUKNjs34QtvdHUH5HkNzSHLsta5z5hlWTcBa52/pW7DE7bdl9wvDNTMWDquEonH47uA85x/XX5GqrMToxGOuiwCNfmkXvNdH5Z4Bo1nWJZ1rWVZN3mXUZHnNSzL0Pudu516PeSZ6Urdhidsuy+5XxgoYywdVwh8ji916JGsu7QkRqNE8uTdzif1WpIU7DCgfwXHsy/p7lWCPK+hcAblq9F18hDpS9RSt+EJ2+5L7hcGyhhLx1U+1gE/9cyUxWiUgDODyyXZmk/qVaRgg3Gfqbjn3GbAnR3L8xoCy7JWA1vj8fiZ6AH5TZZlrXMuS92GJ2y7L7lfGChjLB1XGXBmFw95Z8qI0SiVOcAcy7KWO8t9y4A65/iMfFKvIgWblSOQ8Wx5l0LleS0Rd883Ho+7z+Aa4DzA9YaWug1J2HYfpl8YEGMsHVd4nPCD19yG6GmYYjRKwFlCdcPFHkZ7lR5xjt1Vh3xSryIF68OpuyM+h8P+Dl+e11DUAzu9J5z6ftH5W+q2PIRt9yX1CwMZ2iQdV4k4Hr/1QNyyrDqno/OGOIjRCIHjkXod2ulouevM4ijzuI5Iy4Gd3j3mfNeHMQ+Q7qG7FPCqHMnzWgLOQNy7R+yuNnidO6Vu82BZ1hynvV4LXO+0+X5P87DtvtR+YbAUuETDukCcxnY44NKLjgCI+76cdSt1LwwkPg9qcqhEyfNaBM5AfBmeGXKxdSd1OzQRbWpBEARBGGQGcplaEARBEIQAxBgLgiAIwiAjxlgQBEEQBhkxxoIgCIIwyIgxFgRBEIRBRoyxIAiCIAwyYowFQRAEYZARYywIgiAIg8z/Az3PWSOXCaXsAAAAAElFTkSuQmCC\n", 195 | "text/plain": [ 196 | "
" 197 | ] 198 | }, 199 | "metadata": { 200 | "needs_background": "light" 201 | }, 202 | "output_type": "display_data" 203 | } 204 | ], 205 | "source": [ 206 | "fig = plt.figure(figsize=(7, 5))\n", 207 | "matplotlib.rcParams.update({'font.size': 16})\n", 208 | "plt.plot(fX, 'b.', ms=10) # Plot all evaluated points as blue dots\n", 209 | "plt.plot(np.minimum.accumulate(fX), 'r', lw=3) # Plot cumulative minimum as a red line\n", 210 | "plt.xlim([0, len(fX)])\n", 211 | "plt.ylim([0, 30])\n", 212 | "plt.title(\"10D Levy function\")\n", 213 | "\n", 214 | "plt.tight_layout()\n", 215 | "plt.show()" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": null, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [] 224 | } 225 | ], 226 | "metadata": { 227 | "kernelspec": { 228 | "display_name": "Python 3", 229 | "language": "python", 230 | "name": "python3" 231 | }, 232 | "language_info": { 233 | "codemirror_mode": { 234 | "name": "ipython", 235 | "version": 3 236 | }, 237 | "file_extension": ".py", 238 | "mimetype": "text/x-python", 239 | "name": "python", 240 | "nbconvert_exporter": "python", 241 | "pygments_lexer": "ipython3", 242 | "version": "3.6.8" 243 | } 244 | }, 245 | "nbformat": 4, 246 | "nbformat_minor": 2 247 | } 248 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.17.3 2 | torch==1.3.0 3 | gpytorch==0.3.6 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | setup( 4 | name="turbo", 5 | version="0.0.1", 6 | packages=find_packages(), 7 | install_requires=["numpy>=1.17.3", "torch>=1.3.0", "gpytorch>=0.3.6"], 8 | ) 9 | -------------------------------------------------------------------------------- /turbo/__init__.py: -------------------------------------------------------------------------------- 1 | from .turbo_1 import Turbo1 2 | from .turbo_m import TurboM 3 | -------------------------------------------------------------------------------- /turbo/gp.py: -------------------------------------------------------------------------------- 1 | ############################################################################### 2 | # Copyright (c) 2019 Uber Technologies, Inc. # 3 | # # 4 | # Licensed under the Uber Non-Commercial License (the "License"); # 5 | # you may not use this file except in compliance with the License. # 6 | # You may obtain a copy of the License at the root directory of this project. # 7 | # # 8 | # See the License for the specific language governing permissions and # 9 | # limitations under the License. # 10 | ############################################################################### 11 | 12 | import math 13 | 14 | import gpytorch 15 | import numpy as np 16 | import torch 17 | from gpytorch.constraints.constraints import Interval 18 | from gpytorch.distributions import MultivariateNormal 19 | from gpytorch.kernels import MaternKernel, ScaleKernel 20 | from gpytorch.likelihoods import GaussianLikelihood 21 | from gpytorch.means import ConstantMean 22 | from gpytorch.mlls import ExactMarginalLogLikelihood 23 | from gpytorch.models import ExactGP 24 | 25 | 26 | # GP Model 27 | class GP(ExactGP): 28 | def __init__(self, train_x, train_y, likelihood, lengthscale_constraint, outputscale_constraint, ard_dims): 29 | super(GP, self).__init__(train_x, train_y, likelihood) 30 | self.ard_dims = ard_dims 31 | self.mean_module = ConstantMean() 32 | base_kernel = MaternKernel(lengthscale_constraint=lengthscale_constraint, ard_num_dims=ard_dims, nu=2.5) 33 | self.covar_module = ScaleKernel(base_kernel, outputscale_constraint=outputscale_constraint) 34 | 35 | def forward(self, x): 36 | mean_x = self.mean_module(x) 37 | covar_x = self.covar_module(x) 38 | return MultivariateNormal(mean_x, covar_x) 39 | 40 | 41 | def train_gp(train_x, train_y, use_ard, num_steps, hypers={}): 42 | """Fit a GP model where train_x is in [0, 1]^d and train_y is standardized.""" 43 | assert train_x.ndim == 2 44 | assert train_y.ndim == 1 45 | assert train_x.shape[0] == train_y.shape[0] 46 | 47 | # Create hyper parameter bounds 48 | noise_constraint = Interval(5e-4, 0.2) 49 | if use_ard: 50 | lengthscale_constraint = Interval(0.005, 2.0) 51 | else: 52 | lengthscale_constraint = Interval(0.005, math.sqrt(train_x.shape[1])) # [0.005, sqrt(dim)] 53 | outputscale_constraint = Interval(0.05, 20.0) 54 | 55 | # Create models 56 | likelihood = GaussianLikelihood(noise_constraint=noise_constraint).to(device=train_x.device, dtype=train_y.dtype) 57 | ard_dims = train_x.shape[1] if use_ard else None 58 | model = GP( 59 | train_x=train_x, 60 | train_y=train_y, 61 | likelihood=likelihood, 62 | lengthscale_constraint=lengthscale_constraint, 63 | outputscale_constraint=outputscale_constraint, 64 | ard_dims=ard_dims, 65 | ).to(device=train_x.device, dtype=train_x.dtype) 66 | 67 | # Find optimal model hyperparameters 68 | model.train() 69 | likelihood.train() 70 | 71 | # "Loss" for GPs - the marginal log likelihood 72 | mll = ExactMarginalLogLikelihood(likelihood, model) 73 | 74 | # Initialize model hypers 75 | if hypers: 76 | model.load_state_dict(hypers) 77 | else: 78 | hypers = {} 79 | hypers["covar_module.outputscale"] = 1.0 80 | hypers["covar_module.base_kernel.lengthscale"] = 0.5 81 | hypers["likelihood.noise"] = 0.005 82 | model.initialize(**hypers) 83 | 84 | # Use the adam optimizer 85 | optimizer = torch.optim.Adam([{"params": model.parameters()}], lr=0.1) 86 | 87 | for _ in range(num_steps): 88 | optimizer.zero_grad() 89 | output = model(train_x) 90 | loss = -mll(output, train_y) 91 | loss.backward() 92 | optimizer.step() 93 | 94 | # Switch to eval mode 95 | model.eval() 96 | likelihood.eval() 97 | 98 | return model 99 | -------------------------------------------------------------------------------- /turbo/turbo_1.py: -------------------------------------------------------------------------------- 1 | ############################################################################### 2 | # Copyright (c) 2019 Uber Technologies, Inc. # 3 | # # 4 | # Licensed under the Uber Non-Commercial License (the "License"); # 5 | # you may not use this file except in compliance with the License. # 6 | # You may obtain a copy of the License at the root directory of this project. # 7 | # # 8 | # See the License for the specific language governing permissions and # 9 | # limitations under the License. # 10 | ############################################################################### 11 | 12 | import math 13 | import sys 14 | from copy import deepcopy 15 | 16 | import gpytorch 17 | import numpy as np 18 | import torch 19 | from torch.quasirandom import SobolEngine 20 | 21 | from .gp import train_gp 22 | from .utils import from_unit_cube, latin_hypercube, to_unit_cube 23 | 24 | 25 | class Turbo1: 26 | """The TuRBO-1 algorithm. 27 | 28 | Parameters 29 | ---------- 30 | f : function handle 31 | lb : Lower variable bounds, numpy.array, shape (d,). 32 | ub : Upper variable bounds, numpy.array, shape (d,). 33 | n_init : Number of initial points (2*dim is recommended), int. 34 | max_evals : Total evaluation budget, int. 35 | batch_size : Number of points in each batch, int. 36 | verbose : If you want to print information about the optimization progress, bool. 37 | use_ard : If you want to use ARD for the GP kernel. 38 | max_cholesky_size : Largest number of training points where we use Cholesky, int 39 | n_training_steps : Number of training steps for learning the GP hypers, int 40 | min_cuda : We use float64 on the CPU if we have this or fewer datapoints 41 | device : Device to use for GP fitting ("cpu" or "cuda") 42 | dtype : Dtype to use for GP fitting ("float32" or "float64") 43 | 44 | Example usage: 45 | turbo1 = Turbo1(f=f, lb=lb, ub=ub, n_init=n_init, max_evals=max_evals) 46 | turbo1.optimize() # Run optimization 47 | X, fX = turbo1.X, turbo1.fX # Evaluated points 48 | """ 49 | 50 | def __init__( 51 | self, 52 | f, 53 | lb, 54 | ub, 55 | n_init, 56 | max_evals, 57 | batch_size=1, 58 | verbose=True, 59 | use_ard=True, 60 | max_cholesky_size=2000, 61 | n_training_steps=50, 62 | min_cuda=1024, 63 | device="cpu", 64 | dtype="float64", 65 | ): 66 | 67 | # Very basic input checks 68 | assert lb.ndim == 1 and ub.ndim == 1 69 | assert len(lb) == len(ub) 70 | assert np.all(ub > lb) 71 | assert max_evals > 0 and isinstance(max_evals, int) 72 | assert n_init > 0 and isinstance(n_init, int) 73 | assert batch_size > 0 and isinstance(batch_size, int) 74 | assert isinstance(verbose, bool) and isinstance(use_ard, bool) 75 | assert max_cholesky_size >= 0 and isinstance(batch_size, int) 76 | assert n_training_steps >= 30 and isinstance(n_training_steps, int) 77 | assert max_evals > n_init and max_evals > batch_size 78 | assert device == "cpu" or device == "cuda" 79 | assert dtype == "float32" or dtype == "float64" 80 | if device == "cuda": 81 | assert torch.cuda.is_available(), "can't use cuda if it's not available" 82 | 83 | # Save function information 84 | self.f = f 85 | self.dim = len(lb) 86 | self.lb = lb 87 | self.ub = ub 88 | 89 | # Settings 90 | self.n_init = n_init 91 | self.max_evals = max_evals 92 | self.batch_size = batch_size 93 | self.verbose = verbose 94 | self.use_ard = use_ard 95 | self.max_cholesky_size = max_cholesky_size 96 | self.n_training_steps = n_training_steps 97 | 98 | # Hyperparameters 99 | self.mean = np.zeros((0, 1)) 100 | self.signal_var = np.zeros((0, 1)) 101 | self.noise_var = np.zeros((0, 1)) 102 | self.lengthscales = np.zeros((0, self.dim)) if self.use_ard else np.zeros((0, 1)) 103 | 104 | # Tolerances and counters 105 | self.n_cand = min(100 * self.dim, 5000) 106 | self.failtol = np.ceil(np.max([4.0 / batch_size, self.dim / batch_size])) 107 | self.succtol = 3 108 | self.n_evals = 0 109 | 110 | # Trust region sizes 111 | self.length_min = 0.5 ** 7 112 | self.length_max = 1.6 113 | self.length_init = 0.8 114 | 115 | # Save the full history 116 | self.X = np.zeros((0, self.dim)) 117 | self.fX = np.zeros((0, 1)) 118 | 119 | # Device and dtype for GPyTorch 120 | self.min_cuda = min_cuda 121 | self.dtype = torch.float32 if dtype == "float32" else torch.float64 122 | self.device = torch.device("cuda") if device == "cuda" else torch.device("cpu") 123 | if self.verbose: 124 | print("Using dtype = %s \nUsing device = %s" % (self.dtype, self.device)) 125 | sys.stdout.flush() 126 | 127 | # Initialize parameters 128 | self._restart() 129 | 130 | def _restart(self): 131 | self._X = [] 132 | self._fX = [] 133 | self.failcount = 0 134 | self.succcount = 0 135 | self.length = self.length_init 136 | 137 | def _adjust_length(self, fX_next): 138 | if np.min(fX_next) < np.min(self._fX) - 1e-3 * math.fabs(np.min(self._fX)): 139 | self.succcount += 1 140 | self.failcount = 0 141 | else: 142 | self.succcount = 0 143 | self.failcount += 1 144 | 145 | if self.succcount == self.succtol: # Expand trust region 146 | self.length = min([2.0 * self.length, self.length_max]) 147 | self.succcount = 0 148 | elif self.failcount == self.failtol: # Shrink trust region 149 | self.length /= 2.0 150 | self.failcount = 0 151 | 152 | def _create_candidates(self, X, fX, length, n_training_steps, hypers): 153 | """Generate candidates assuming X has been scaled to [0,1]^d.""" 154 | # Pick the center as the point with the smallest function values 155 | # NOTE: This may not be robust to noise, in which case the posterior mean of the GP can be used instead 156 | assert X.min() >= 0.0 and X.max() <= 1.0 157 | 158 | # Standardize function values. 159 | mu, sigma = np.median(fX), fX.std() 160 | sigma = 1.0 if sigma < 1e-6 else sigma 161 | fX = (deepcopy(fX) - mu) / sigma 162 | 163 | # Figure out what device we are running on 164 | if len(X) < self.min_cuda: 165 | device, dtype = torch.device("cpu"), torch.float64 166 | else: 167 | device, dtype = self.device, self.dtype 168 | 169 | # We use CG + Lanczos for training if we have enough data 170 | with gpytorch.settings.max_cholesky_size(self.max_cholesky_size): 171 | X_torch = torch.tensor(X).to(device=device, dtype=dtype) 172 | y_torch = torch.tensor(fX).to(device=device, dtype=dtype) 173 | gp = train_gp( 174 | train_x=X_torch, train_y=y_torch, use_ard=self.use_ard, num_steps=n_training_steps, hypers=hypers 175 | ) 176 | 177 | # Save state dict 178 | hypers = gp.state_dict() 179 | 180 | # Create the trust region boundaries 181 | x_center = X[fX.argmin().item(), :][None, :] 182 | weights = gp.covar_module.base_kernel.lengthscale.cpu().detach().numpy().ravel() 183 | weights = weights / weights.mean() # This will make the next line more stable 184 | weights = weights / np.prod(np.power(weights, 1.0 / len(weights))) # We now have weights.prod() = 1 185 | lb = np.clip(x_center - weights * length / 2.0, 0.0, 1.0) 186 | ub = np.clip(x_center + weights * length / 2.0, 0.0, 1.0) 187 | 188 | # Draw a Sobolev sequence in [lb, ub] 189 | seed = np.random.randint(int(1e6)) 190 | sobol = SobolEngine(self.dim, scramble=True, seed=seed) 191 | pert = sobol.draw(self.n_cand).to(dtype=dtype, device=device).cpu().detach().numpy() 192 | pert = lb + (ub - lb) * pert 193 | 194 | # Create a perturbation mask 195 | prob_perturb = min(20.0 / self.dim, 1.0) 196 | mask = np.random.rand(self.n_cand, self.dim) <= prob_perturb 197 | ind = np.where(np.sum(mask, axis=1) == 0)[0] 198 | mask[ind, np.random.randint(0, self.dim - 1, size=len(ind))] = 1 199 | 200 | # Create candidate points 201 | X_cand = x_center.copy() * np.ones((self.n_cand, self.dim)) 202 | X_cand[mask] = pert[mask] 203 | 204 | # Figure out what device we are running on 205 | if len(X_cand) < self.min_cuda: 206 | device, dtype = torch.device("cpu"), torch.float64 207 | else: 208 | device, dtype = self.device, self.dtype 209 | 210 | # We may have to move the GP to a new device 211 | gp = gp.to(dtype=dtype, device=device) 212 | 213 | # We use Lanczos for sampling if we have enough data 214 | with torch.no_grad(), gpytorch.settings.max_cholesky_size(self.max_cholesky_size): 215 | X_cand_torch = torch.tensor(X_cand).to(device=device, dtype=dtype) 216 | y_cand = gp.likelihood(gp(X_cand_torch)).sample(torch.Size([self.batch_size])).t().cpu().detach().numpy() 217 | 218 | # Remove the torch variables 219 | del X_torch, y_torch, X_cand_torch, gp 220 | 221 | # De-standardize the sampled values 222 | y_cand = mu + sigma * y_cand 223 | 224 | return X_cand, y_cand, hypers 225 | 226 | def _select_candidates(self, X_cand, y_cand): 227 | """Select candidates.""" 228 | X_next = np.ones((self.batch_size, self.dim)) 229 | for i in range(self.batch_size): 230 | # Pick the best point and make sure we never pick it again 231 | indbest = np.argmin(y_cand[:, i]) 232 | X_next[i, :] = deepcopy(X_cand[indbest, :]) 233 | y_cand[indbest, :] = np.inf 234 | return X_next 235 | 236 | def optimize(self): 237 | """Run the full optimization process.""" 238 | while self.n_evals < self.max_evals: 239 | if len(self._fX) > 0 and self.verbose: 240 | n_evals, fbest = self.n_evals, self._fX.min() 241 | print(f"{n_evals}) Restarting with fbest = {fbest:.4}") 242 | sys.stdout.flush() 243 | 244 | # Initialize parameters 245 | self._restart() 246 | 247 | # Generate and evalute initial design points 248 | X_init = latin_hypercube(self.n_init, self.dim) 249 | X_init = from_unit_cube(X_init, self.lb, self.ub) 250 | fX_init = np.array([[self.f(x)] for x in X_init]) 251 | 252 | # Update budget and set as initial data for this TR 253 | self.n_evals += self.n_init 254 | self._X = deepcopy(X_init) 255 | self._fX = deepcopy(fX_init) 256 | 257 | # Append data to the global history 258 | self.X = np.vstack((self.X, deepcopy(X_init))) 259 | self.fX = np.vstack((self.fX, deepcopy(fX_init))) 260 | 261 | if self.verbose: 262 | fbest = self._fX.min() 263 | print(f"Starting from fbest = {fbest:.4}") 264 | sys.stdout.flush() 265 | 266 | # Thompson sample to get next suggestions 267 | while self.n_evals < self.max_evals and self.length >= self.length_min: 268 | # Warp inputs 269 | X = to_unit_cube(deepcopy(self._X), self.lb, self.ub) 270 | 271 | # Standardize values 272 | fX = deepcopy(self._fX).ravel() 273 | 274 | # Create th next batch 275 | X_cand, y_cand, _ = self._create_candidates( 276 | X, fX, length=self.length, n_training_steps=self.n_training_steps, hypers={} 277 | ) 278 | X_next = self._select_candidates(X_cand, y_cand) 279 | 280 | # Undo the warping 281 | X_next = from_unit_cube(X_next, self.lb, self.ub) 282 | 283 | # Evaluate batch 284 | fX_next = np.array([[self.f(x)] for x in X_next]) 285 | 286 | # Update trust region 287 | self._adjust_length(fX_next) 288 | 289 | # Update budget and append data 290 | self.n_evals += self.batch_size 291 | self._X = np.vstack((self._X, X_next)) 292 | self._fX = np.vstack((self._fX, fX_next)) 293 | 294 | if self.verbose and fX_next.min() < self.fX.min(): 295 | n_evals, fbest = self.n_evals, fX_next.min() 296 | print(f"{n_evals}) New best: {fbest:.4}") 297 | sys.stdout.flush() 298 | 299 | # Append data to the global history 300 | self.X = np.vstack((self.X, deepcopy(X_next))) 301 | self.fX = np.vstack((self.fX, deepcopy(fX_next))) 302 | -------------------------------------------------------------------------------- /turbo/turbo_m.py: -------------------------------------------------------------------------------- 1 | ############################################################################### 2 | # Copyright (c) 2019 Uber Technologies, Inc. # 3 | # # 4 | # Licensed under the Uber Non-Commercial License (the "License"); # 5 | # you may not use this file except in compliance with the License. # 6 | # You may obtain a copy of the License at the root directory of this project. # 7 | # # 8 | # See the License for the specific language governing permissions and # 9 | # limitations under the License. # 10 | ############################################################################### 11 | 12 | import math 13 | import sys 14 | from copy import deepcopy 15 | 16 | import gpytorch 17 | import numpy as np 18 | import torch 19 | 20 | from .gp import train_gp 21 | from .turbo_1 import Turbo1 22 | from .utils import from_unit_cube, latin_hypercube, to_unit_cube 23 | 24 | 25 | class TurboM(Turbo1): 26 | """The TuRBO-m algorithm. 27 | 28 | Parameters 29 | ---------- 30 | f : function handle 31 | lb : Lower variable bounds, numpy.array, shape (d,). 32 | ub : Upper variable bounds, numpy.array, shape (d,). 33 | n_init : Number of initial points *FOR EACH TRUST REGION* (2*dim is recommended), int. 34 | max_evals : Total evaluation budget, int. 35 | n_trust_regions : Number of trust regions 36 | batch_size : Number of points in each batch, int. 37 | verbose : If you want to print information about the optimization progress, bool. 38 | use_ard : If you want to use ARD for the GP kernel. 39 | max_cholesky_size : Largest number of training points where we use Cholesky, int 40 | n_training_steps : Number of training steps for learning the GP hypers, int 41 | min_cuda : We use float64 on the CPU if we have this or fewer datapoints 42 | device : Device to use for GP fitting ("cpu" or "cuda") 43 | dtype : Dtype to use for GP fitting ("float32" or "float64") 44 | 45 | Example usage: 46 | turbo5 = TurboM(f=f, lb=lb, ub=ub, n_init=n_init, max_evals=max_evals, n_trust_regions=5) 47 | turbo5.optimize() # Run optimization 48 | X, fX = turbo5.X, turbo5.fX # Evaluated points 49 | """ 50 | 51 | def __init__( 52 | self, 53 | f, 54 | lb, 55 | ub, 56 | n_init, 57 | max_evals, 58 | n_trust_regions, 59 | batch_size=1, 60 | verbose=True, 61 | use_ard=True, 62 | max_cholesky_size=2000, 63 | n_training_steps=50, 64 | min_cuda=1024, 65 | device="cpu", 66 | dtype="float64", 67 | ): 68 | self.n_trust_regions = n_trust_regions 69 | super().__init__( 70 | f=f, 71 | lb=lb, 72 | ub=ub, 73 | n_init=n_init, 74 | max_evals=max_evals, 75 | batch_size=batch_size, 76 | verbose=verbose, 77 | use_ard=use_ard, 78 | max_cholesky_size=max_cholesky_size, 79 | n_training_steps=n_training_steps, 80 | min_cuda=min_cuda, 81 | device=device, 82 | dtype=dtype, 83 | ) 84 | 85 | self.succtol = 3 86 | self.failtol = max(5, self.dim) 87 | 88 | # Very basic input checks 89 | assert n_trust_regions > 1 and isinstance(max_evals, int) 90 | assert max_evals > n_trust_regions * n_init, "Not enough trust regions to do initial evaluations" 91 | assert max_evals > batch_size, "Not enough evaluations to do a single batch" 92 | 93 | # Remember the hypers for trust regions we don't sample from 94 | self.hypers = [{} for _ in range(self.n_trust_regions)] 95 | 96 | # Initialize parameters 97 | self._restart() 98 | 99 | def _restart(self): 100 | self._idx = np.zeros((0, 1), dtype=int) # Track what trust region proposed what using an index vector 101 | self.failcount = np.zeros(self.n_trust_regions, dtype=int) 102 | self.succcount = np.zeros(self.n_trust_regions, dtype=int) 103 | self.length = self.length_init * np.ones(self.n_trust_regions) 104 | 105 | def _adjust_length(self, fX_next, i): 106 | assert i >= 0 and i <= self.n_trust_regions - 1 107 | 108 | fX_min = self.fX[self._idx[:, 0] == i, 0].min() # Target value 109 | if fX_next.min() < fX_min - 1e-3 * math.fabs(fX_min): 110 | self.succcount[i] += 1 111 | self.failcount[i] = 0 112 | else: 113 | self.succcount[i] = 0 114 | self.failcount[i] += len(fX_next) # NOTE: Add size of the batch for this TR 115 | 116 | if self.succcount[i] == self.succtol: # Expand trust region 117 | self.length[i] = min([2.0 * self.length[i], self.length_max]) 118 | self.succcount[i] = 0 119 | elif self.failcount[i] >= self.failtol: # Shrink trust region (we may have exceeded the failtol) 120 | self.length[i] /= 2.0 121 | self.failcount[i] = 0 122 | 123 | def _select_candidates(self, X_cand, y_cand): 124 | """Select candidates from samples from all trust regions.""" 125 | assert X_cand.shape == (self.n_trust_regions, self.n_cand, self.dim) 126 | assert y_cand.shape == (self.n_trust_regions, self.n_cand, self.batch_size) 127 | assert X_cand.min() >= 0.0 and X_cand.max() <= 1.0 and np.all(np.isfinite(y_cand)) 128 | 129 | X_next = np.zeros((self.batch_size, self.dim)) 130 | idx_next = np.zeros((self.batch_size, 1), dtype=int) 131 | for k in range(self.batch_size): 132 | i, j = np.unravel_index(np.argmin(y_cand[:, :, k]), (self.n_trust_regions, self.n_cand)) 133 | assert y_cand[:, :, k].min() == y_cand[i, j, k] 134 | X_next[k, :] = deepcopy(X_cand[i, j, :]) 135 | idx_next[k, 0] = i 136 | assert np.isfinite(y_cand[i, j, k]) # Just to make sure we never select nan or inf 137 | 138 | # Make sure we never pick this point again 139 | y_cand[i, j, :] = np.inf 140 | 141 | return X_next, idx_next 142 | 143 | def optimize(self): 144 | """Run the full optimization process.""" 145 | # Create initial points for each TR 146 | for i in range(self.n_trust_regions): 147 | X_init = latin_hypercube(self.n_init, self.dim) 148 | X_init = from_unit_cube(X_init, self.lb, self.ub) 149 | fX_init = np.array([[self.f(x)] for x in X_init]) 150 | 151 | # Update budget and set as initial data for this TR 152 | self.X = np.vstack((self.X, X_init)) 153 | self.fX = np.vstack((self.fX, fX_init)) 154 | self._idx = np.vstack((self._idx, i * np.ones((self.n_init, 1), dtype=int))) 155 | self.n_evals += self.n_init 156 | 157 | if self.verbose: 158 | fbest = fX_init.min() 159 | print(f"TR-{i} starting from: {fbest:.4}") 160 | sys.stdout.flush() 161 | 162 | # Thompson sample to get next suggestions 163 | while self.n_evals < self.max_evals: 164 | 165 | # Generate candidates from each TR 166 | X_cand = np.zeros((self.n_trust_regions, self.n_cand, self.dim)) 167 | y_cand = np.inf * np.ones((self.n_trust_regions, self.n_cand, self.batch_size)) 168 | for i in range(self.n_trust_regions): 169 | idx = np.where(self._idx == i)[0] # Extract all "active" indices 170 | 171 | # Get the points, values the active values 172 | X = deepcopy(self.X[idx, :]) 173 | X = to_unit_cube(X, self.lb, self.ub) 174 | 175 | # Get the values from the standardized data 176 | fX = deepcopy(self.fX[idx, 0].ravel()) 177 | 178 | # Don't retrain the model if the training data hasn't changed 179 | n_training_steps = 0 if self.hypers[i] else self.n_training_steps 180 | 181 | # Create new candidates 182 | X_cand[i, :, :], y_cand[i, :, :], self.hypers[i] = self._create_candidates( 183 | X, fX, length=self.length[i], n_training_steps=n_training_steps, hypers=self.hypers[i] 184 | ) 185 | 186 | # Select the next candidates 187 | X_next, idx_next = self._select_candidates(X_cand, y_cand) 188 | assert X_next.min() >= 0.0 and X_next.max() <= 1.0 189 | 190 | # Undo the warping 191 | X_next = from_unit_cube(X_next, self.lb, self.ub) 192 | 193 | # Evaluate batch 194 | fX_next = np.array([[self.f(x)] for x in X_next]) 195 | 196 | # Update trust regions 197 | for i in range(self.n_trust_regions): 198 | idx_i = np.where(idx_next == i)[0] 199 | if len(idx_i) > 0: 200 | self.hypers[i] = {} # Remove model hypers 201 | fX_i = fX_next[idx_i] 202 | 203 | if self.verbose and fX_i.min() < self.fX.min() - 1e-3 * math.fabs(self.fX.min()): 204 | n_evals, fbest = self.n_evals, fX_i.min() 205 | print(f"{n_evals}) New best @ TR-{i}: {fbest:.4}") 206 | sys.stdout.flush() 207 | self._adjust_length(fX_i, i) 208 | 209 | # Update budget and append data 210 | self.n_evals += self.batch_size 211 | self.X = np.vstack((self.X, deepcopy(X_next))) 212 | self.fX = np.vstack((self.fX, deepcopy(fX_next))) 213 | self._idx = np.vstack((self._idx, deepcopy(idx_next))) 214 | 215 | # Check if any TR needs to be restarted 216 | for i in range(self.n_trust_regions): 217 | if self.length[i] < self.length_min: # Restart trust region if converged 218 | idx_i = self._idx[:, 0] == i 219 | 220 | if self.verbose: 221 | n_evals, fbest = self.n_evals, self.fX[idx_i, 0].min() 222 | print(f"{n_evals}) TR-{i} converged to: : {fbest:.4}") 223 | sys.stdout.flush() 224 | 225 | # Reset length and counters, remove old data from trust region 226 | self.length[i] = self.length_init 227 | self.succcount[i] = 0 228 | self.failcount[i] = 0 229 | self._idx[idx_i, 0] = -1 # Remove points from trust region 230 | self.hypers[i] = {} # Remove model hypers 231 | 232 | # Create a new initial design 233 | X_init = latin_hypercube(self.n_init, self.dim) 234 | X_init = from_unit_cube(X_init, self.lb, self.ub) 235 | fX_init = np.array([[self.f(x)] for x in X_init]) 236 | 237 | # Print progress 238 | if self.verbose: 239 | n_evals, fbest = self.n_evals, fX_init.min() 240 | print(f"{n_evals}) TR-{i} is restarting from: : {fbest:.4}") 241 | sys.stdout.flush() 242 | 243 | # Append data to local history 244 | self.X = np.vstack((self.X, X_init)) 245 | self.fX = np.vstack((self.fX, fX_init)) 246 | self._idx = np.vstack((self._idx, i * np.ones((self.n_init, 1), dtype=int))) 247 | self.n_evals += self.n_init 248 | -------------------------------------------------------------------------------- /turbo/utils.py: -------------------------------------------------------------------------------- 1 | ############################################################################### 2 | # Copyright (c) 2019 Uber Technologies, Inc. # 3 | # # 4 | # Licensed under the Uber Non-Commercial License (the "License"); # 5 | # you may not use this file except in compliance with the License. # 6 | # You may obtain a copy of the License at the root directory of this project. # 7 | # # 8 | # See the License for the specific language governing permissions and # 9 | # limitations under the License. # 10 | ############################################################################### 11 | 12 | import numpy as np 13 | 14 | 15 | def to_unit_cube(x, lb, ub): 16 | """Project to [0, 1]^d from hypercube with bounds lb and ub""" 17 | assert np.all(lb < ub) and lb.ndim == 1 and ub.ndim == 1 and x.ndim == 2 18 | xx = (x - lb) / (ub - lb) 19 | return xx 20 | 21 | 22 | def from_unit_cube(x, lb, ub): 23 | """Project from [0, 1]^d to hypercube with bounds lb and ub""" 24 | assert np.all(lb < ub) and lb.ndim == 1 and ub.ndim == 1 and x.ndim == 2 25 | xx = x * (ub - lb) + lb 26 | return xx 27 | 28 | 29 | def latin_hypercube(n_pts, dim): 30 | """Basic Latin hypercube implementation with center perturbation.""" 31 | X = np.zeros((n_pts, dim)) 32 | centers = (1.0 + 2.0 * np.arange(0.0, n_pts)) / float(2 * n_pts) 33 | for i in range(dim): # Shuffle the center locataions for each dimension. 34 | X[:, i] = centers[np.random.permutation(n_pts)] 35 | 36 | # Add some perturbations within each box 37 | pert = np.random.uniform(-1.0, 1.0, (n_pts, dim)) / float(2 * n_pts) 38 | X += pert 39 | return X 40 | --------------------------------------------------------------------------------