├── .dask └── config.yaml ├── .github └── workflows │ └── ci-binder.yml ├── .gitignore ├── LICENSE ├── README.md ├── binder ├── environment.yml ├── jupyterlab-workspace.json └── start ├── data └── .gitkeep ├── notebooks ├── 0_Dask_what_and_when.ipynb ├── 1_Delayed.ipynb ├── 2_Schedulers.ipynb ├── 3_DataFrames.ipynb └── 4_Machine_learning.ipynb └── prep_data.py /.dask/config.yaml: -------------------------------------------------------------------------------- 1 | distributed: 2 | dashboard: 3 | link: "{JUPYTERHUB_BASE_URL}user/{JUPYTERHUB_USER}/proxy/{port}/status" 4 | -------------------------------------------------------------------------------- /.github/workflows/ci-binder.yml: -------------------------------------------------------------------------------- 1 | name: Binder 2 | on: [push] 3 | 4 | jobs: 5 | build: 6 | runs-on: ubuntu-latest 7 | steps: 8 | 9 | - name: Build and cache on mybinder.org 10 | uses: jupyterhub/repo2docker-action@master 11 | with: 12 | NO_PUSH: true 13 | MYBINDERORG_TAG: ${{ github.event.ref }} -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | *.dot 3 | *.pdf 4 | *.png 5 | .ipynb_checkpoints 6 | *.gz 7 | data/accounts.*.csv 8 | data/accounts.h5 9 | data/random.hdf5 10 | data/weather-big 11 | data/myfile.hdf5 12 | data/flightjson 13 | data/holidays 14 | data/nycflights 15 | data/myfile.zarr 16 | data/accounts.parquet 17 | dask-worker-space/ 18 | profile.html 19 | log 20 | .idea/ 21 | _build/ 22 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Coiled 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Dask Live by Coiled Tutorial 2 | 3 | The purpose of this tutorial is to introduce folks to Dask and show them how to scale their python data-science and machine learning workflows. The materials covered are: 4 | 5 | 0. Overview of Dask - How it works and when to use it. 6 | 1. Dask Delayed: How to parallelize existing Python code and your custom algorithms. 7 | 2. Schedulers: Single Machine vs Distributed, and the Dashboard. 8 | 3. From pandas to Dask: How to manipulate bigger-than-memory DataFrames using Dask. 9 | 4. Dask-ML: Scalable machine learning using Dask. 10 | 11 | ## Prerequisites 12 | 13 | To follow along and get the most out of this tutorial it would help if you Know: 14 | 15 | - Programming fundamentals in Python (e.g variables, data structures, for loops, etc). 16 | - A bit of or are familiarized with `numpy`, `pandas` and `scikit-learn`. 17 | - Jupyter Lab/ Jupyter Notebooks 18 | - Your way around the shell/terminal 19 | 20 | However, the most important prerequisite is being willing to learn, and everyone is 21 | welcomed to tag along and enjoy the ride. If you would like to watch and not code along, 22 | not a problem. 23 | 24 | ## Get set up 25 | 26 | We have two options for you to follow this tutorial: 27 | 28 | 1. Click on the binder button right below, this will spin up the necessary computational environment for you so you can write and execute the notebooks directly on the browser. Binder is a free service so resources are not guaranteed, but they usually work. One thing 29 | to keep in mind is that the amount of resources are limited and sometimes you won't be able to see the benefits of parallelism due to this limitation. 30 | 31 | *IMPORTANT*: If you are joining the live session, make sure to click on the button few minutes before we start so we are ready to go. 32 | 33 | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/coiled/dask-mini-tutorial/HEAD) 34 | 35 | 36 | 2. You can create your own set-up locally. To do this you need to be comfortable with the git and github as well as installing packages and creating software environments. If so, follow the next steps: 37 | 38 | *IMPORTANT:* If you are joining for a live session please make sure you do the setup in advance, and be ready to go once the session starts. 39 | 40 | 1. **Clone this repository** 41 | In your terminal: 42 | 43 | ``` 44 | git clone https://github.com/coiled/dask-mini-tutorial.git 45 | ``` 46 | Alternatively, you can download the zip file of the repository at the top of the main page of the repository. This is a good option if you don't have experience with git. 47 | 48 | 2. Download Anaconda 49 | If you do not have anaconda already install, you will need the Python 3 [Anaconda Distribution](https://www.anaconda.com/products/individual). If you don't want to install anaconda you can install all the packages with `pip`, if you take this route you will need to install `graphviz` separately before installing `pygraphviz`. 50 | 51 | 3. Create a conda environment 52 | In your terminal navigate to the directory where you have cloned/downloaded th `dask-mini-tutorial` repo and install the required packages by doing: 53 | 54 | ``` 55 | conda env create -f binder/environment.yml 56 | ``` 57 | 58 | This will create a new environment called `dask-mini-tutorial`. To activate the environment do: 59 | 60 | ``` 61 | conda activate dask-mini-tutorial 62 | ``` 63 | 64 | 4. Open Jupyter Lab 65 | Once your environment has been activated and you are in the `dask-mini-tutorial` repository, in your terminal do: 66 | 67 | ``` 68 | jupyter lab 69 | ``` 70 | 71 | You will see a notebooks directory, click on there and you will be ready to go. 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | -------------------------------------------------------------------------------- /binder/environment.yml: -------------------------------------------------------------------------------- 1 | name: dask-mini-tutorial 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python=3.9.7 6 | - dask=2021.11.2 7 | # JupyterLab extensions 8 | - jupyterlab>=3 9 | - dask-labextension=5.1.0 10 | - ipywidgets=7.6.5 11 | - graphviz=2.49.0 12 | - python-graphviz=0.17 13 | - scikit-learn=1.0.1 14 | - dask-ml=2021.11.16 15 | - coiled=0.0.56 -------------------------------------------------------------------------------- /binder/jupyterlab-workspace.json: -------------------------------------------------------------------------------- 1 | { 2 | "data": { 3 | "file-browser-filebrowser:cwd": { 4 | "path": "" 5 | }, 6 | "dask-dashboard-launcher": { 7 | "url": "DASK_DASHBOARD_URL" 8 | } 9 | }, 10 | "metadata": { 11 | "id": "/lab" 12 | } 13 | } -------------------------------------------------------------------------------- /binder/start: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Replace DASK_DASHBOARD_URL with the proxy location 4 | sed -i -e "s|DASK_DASHBOARD_URL|${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}/proxy/8787|g" binder/jupyterlab-workspace.json 5 | 6 | # Import the workspace 7 | jupyter lab workspaces import binder/jupyterlab-workspace.json 8 | export DASK_TUTORIAL_SMALL=1 9 | 10 | exec "$@" -------------------------------------------------------------------------------- /data/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/coiled/dask-mini-tutorial/38ffa24ed47abe66c305345a3ef7f3b00ef73095/data/.gitkeep -------------------------------------------------------------------------------- /notebooks/0_Dask_what_and_when.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "fa397203", 6 | "metadata": {}, 7 | "source": [ 8 | "\"Dask\n", 11 | "\n", 12 | "\n", 13 | "# What is it and when to use it? \n", 14 | "\n", 15 | "\n", 16 | "If you ever heard of Dask you might have some form of these questions. If you have never heard of Dask but you want to know what it is and when/if you should use it, then you are in the right place. \n", 17 | "\n", 18 | "Before we give a short overview and attempt to answer these questions, we strongly recommend you to check the amazing documentation that the Dask community has in place. \n", 19 | "\n", 20 | "- Documentation: https://docs.dask.org\n", 21 | "\n", 22 | "Contribute to the project:\n", 23 | "\n", 24 | "- Github: https://github.com/dask/dask\n", 25 | "\n", 26 | "Engage with the community:\n", 27 | "\n", 28 | "- Slack: https://dask.slack.com/\n", 29 | "\n", 30 | "Looking for answers about how to use Dask:\n", 31 | "\n", 32 | "- Discourse: https://dask.discourse.group/" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "id": "5c317728", 38 | "metadata": {}, 39 | "source": [ 40 | "### What is Dask? \n", 41 | "\n", 42 | "Dask is a flexible library for parallel computing in Python, that follows the syntax of the PyData ecosystem. If you are familiar with NumPy, pandas and scikit-learn then think of Dask as their faster cousin. For example:\n", 43 | "\n", 44 | "```python\n", 45 | "import pandas as pd import dask.dataframe as dd\n", 46 | "df = pd.read_csv('2015-01-01.csv') df = dd.read_csv('2015-*-*.csv')\n", 47 | "df.groupby(df.user_id).value.mean() df.groupby(df.user_id).value.mean().compute()\n", 48 | "```\n", 49 | "\n", 50 | " Since they are all family, Dask allows you to scale your existing workflows with a small amount of changes. Dask enables you to accelerate computations and perform those that don't fit in memory. It works in your laptop but it also scales out to large clusters while providing a dashboard with great diagnostic tools. " 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "id": "c70d4db3", 56 | "metadata": {}, 57 | "source": [ 58 | "\"Dask" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "id": "3a8359e2", 66 | "metadata": {}, 67 | "source": [ 68 | "### Dask jargon: Client, Scheduler and Workers \n", 69 | "\n", 70 | "- Client: The user-facing entry point for cluster users. In other words, the client lives where your python code lives, and it communicates to the scheduler, passing along the tasks to be executed.\n", 71 | "- Scheduler: The task manager, it sends the tasks to the workers.\n", 72 | "- Workers: The ones that compute the tasks.\n", 73 | "\n", 74 | "Note: The Scheduler and the Workers are on the same network, they could live in your laptop or on a separate cluster\n", 75 | "\n", 76 | "\"Dask" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "id": "d78d7794", 84 | "metadata": { 85 | "tags": [] 86 | }, 87 | "source": [ 88 | "## When to use Dask?\n", 89 | "\n", 90 | "Before trying to use Dask, there are some questions to determine if Dask might be suitable for you. \n", 91 | "\n", 92 | "- Does your data fit in memory? \n", 93 | " - Yes: Use pandas or NumPy. \n", 94 | " - No : Dask might be able to help. \n", 95 | "- Do your computations take for ever?\n", 96 | " - Yes: Dask might be able to help. \n", 97 | " - No : Awesome.\n", 98 | "- Do you have embarrassingly parallelizable code?\n", 99 | " - Yes: Dask might be able to help.\n", 100 | " - No?: If you are not sure here are some [examples](https://examples.dask.org/applications/embarrassingly-parallel.html) \n", 101 | " - No: I'm sorry, although Dask might have some hope for you.\n", 102 | " \n", 103 | " \n", 104 | "**Bottom Left:** You don't need Dask. \n", 105 | "**Elsewhere:** Dask fair game.\n", 106 | "\n", 107 | "\n", 108 | "\"Dask\n", 111 | "\n", 112 | "\n", 113 | "**Disclaimers:**\n", 114 | "\n", 115 | "1. When we say \"Dask might be able to help\" it is because you should try first to accelerate your code with NumPy and or Numba, checking types used on your DataFrames, and then maybe consider Dask. Now even when using Dask, we can't guarantee that things will be faster, it depends on what is the code behind. \n", 116 | "\n", 117 | "2. Even when you have large datasets, at some point you want to double check if you have reduced things to a manageable level where going back to pandas or NumPy might be the best call.\n", 118 | "\n", 119 | "**Best practices:**\n", 120 | "\n", 121 | "The learning curve to use Dask can be a bit intimidating, that's why we want to point you out to some best practices links that will make the process smoother. We will go over some of these topics but we want to leave here these links for future reference\n", 122 | "\n", 123 | "- Are you working with arrays? Check this [array best practices](https://docs.dask.org/en/latest/array-best-practices.html)\n", 124 | "- Dealing with DataFrames? Check this [DataFrames best practices](https://docs.dask.org/en/latest/dataframe-best-practices.html)\n", 125 | "- Are you trying to accelerate your code using `delayed`? Check this [delayed best practices](https://docs.dask.org/en/latest/delayed-best-practices.html)\n", 126 | "- For overall good practices check [Dask good practices](https://docs.dask.org/en/latest/best-practices.html)" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "id": "ea50907f", 132 | "metadata": {}, 133 | "source": [ 134 | "## Why Dask? \n", 135 | "\n", 136 | "If you are interested in knowing why Dask might be a good option for you we recommend you to check the Dask documentation [Why Dask?](https://docs.dask.org/en/latest/why.html)\n", 137 | "\n", 138 | "But if you are already convinced that Dask is right for you and/or want to learn more about it. The topics that we will cover on this mini-tutorial are:\n", 139 | "\n", 140 | "1. Dask Delayed: How to parallelize existing Python code and your custom algorithms. \n", 141 | "2. Schedulers: Single Machine vs Distributed, and the Dashboard. \n", 142 | "3. From pandas to Dask: How to manipulate bigger-than-memory DataFrames using Dask. \n", 143 | "4. Dask-ML: Scalable machine learning using Dask.\n", 144 | "\n", 145 | "## Extra learning material:\n", 146 | "\n", 147 | "1. Self-paced Dask-Tutorial: https://tutorial.dask.org/\n", 148 | "2. Dask training by Coiled: [Scaling Python with Dask](https://coiled.io/course/scaling-python-with-dask/)" 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 3 (ipykernel)", 155 | "language": "python", 156 | "name": "python3" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 3 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython3", 168 | "version": "3.9.7" 169 | } 170 | }, 171 | "nbformat": 4, 172 | "nbformat_minor": 5 173 | } 174 | -------------------------------------------------------------------------------- /notebooks/1_Delayed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "be4dd2fc", 6 | "metadata": {}, 7 | "source": [ 8 | "\"Dask\n", 11 | "\n", 12 | "This notebook was inspired in the materials from: \n", 13 | "\n", 14 | "- https://github.com/coiled/pydata-global-dask/\n", 15 | "- https://github.com/dask/dask-tutorial/\n", 16 | "\n", 17 | "# Dask Delayed\n", 18 | "\n", 19 | "Sometimes we have problems that are parallelizable. Dask Delayed is an interface that can be use to parallelize existing Python code and custom algorithms. \n", 20 | "\n", 21 | "A first step to determine if we can use `dask.delayed` is to identify if there is some level of parallelism that we haven't exploit and hopefully `dask.delayed` will take care of it. We will start showing a simple example inspired on the main [Dask tutorial](https://tutorial.dask.org/), and we will it parallelize using `dask.delayed`.\n", 22 | "\n", 23 | "The following two functions will perform simple computations, where we use the `sleep` to simulate work. " 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "id": "c7cbb4e5", 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "from time import sleep\n", 34 | "\n", 35 | "def inc(x):\n", 36 | " \"\"\"Increments x by one\"\"\"\n", 37 | " sleep(1)\n", 38 | " return x + 1\n", 39 | "\n", 40 | "def add(x, y):\n", 41 | " \"\"\"Adds x and y\"\"\"\n", 42 | " sleep(1)\n", 43 | " return x + y" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "id": "700f0a97", 49 | "metadata": {}, 50 | "source": [ 51 | "Let's do some operations and time these functions using the `%%time` magic at the beginning of the cell. " 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 2, 57 | "id": "1c0053fb", 58 | "metadata": {}, 59 | "outputs": [ 60 | { 61 | "name": "stdout", 62 | "output_type": "stream", 63 | "text": [ 64 | "CPU times: user 933 µs, sys: 1.62 ms, total: 2.55 ms\n", 65 | "Wall time: 3.01 s\n" 66 | ] 67 | } 68 | ], 69 | "source": [ 70 | "%%time\n", 71 | "\n", 72 | "x = inc(1)\n", 73 | "y = inc(2)\n", 74 | "z = add(x, y)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "id": "a99dd0ec", 80 | "metadata": {}, 81 | "source": [ 82 | "The execution of the cell above took three seconds, this happens because we are calling each function sequentially. The computations above can be represented by the following graph:\n", 83 | "\n", 84 | "\"Dask\n", 87 | "\n", 88 | "\n", 89 | "Where the circles are function calls, squares represent objects that are created by one task as output and can be inputs into other tasks, and arrows represent the dependencies between the tasks. From looking at the task graph, the opportunity for parallelization is more evident since the the two calls to the `inc` function are completely independent of one-another. Let's explore how `dask.delayed` can help us with this.\n", 90 | "\n", 91 | "\n", 92 | "### `dask.delayed` \n", 93 | "\n", 94 | "Using the `dask.delayed` decorator we'll transform the `inc` and `add` functions. " 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 3, 100 | "id": "31abdfa5", 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "from dask import delayed" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 4, 110 | "id": "03aed361", 111 | "metadata": {}, 112 | "outputs": [ 113 | { 114 | "name": "stdout", 115 | "output_type": "stream", 116 | "text": [ 117 | "CPU times: user 119 µs, sys: 24 µs, total: 143 µs\n", 118 | "Wall time: 132 µs\n" 119 | ] 120 | } 121 | ], 122 | "source": [ 123 | "%%time\n", 124 | "\n", 125 | "a = delayed(inc)(1)\n", 126 | "b = delayed(inc)(2)\n", 127 | "c = delayed(add)(a, b)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "id": "f5546859", 133 | "metadata": {}, 134 | "source": [ 135 | "When we call the `delayed` version of the functions by passing the arguments, the original function is isn't actually called yet, that's why the execution finishes very quickly. When we called the `delayed` version of the functions, a `delayed` object is made, which keeps track of the functions to call and what arguments to pass to it. \n", 136 | "\n", 137 | "If we inspect `c`, we will notice that it instead of having the value five, we have what is called a `delayed` object." 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 5, 143 | "id": "33c59a9b", 144 | "metadata": {}, 145 | "outputs": [ 146 | { 147 | "name": "stdout", 148 | "output_type": "stream", 149 | "text": [ 150 | "Delayed('add-9b51a77b-5b91-4b92-88d8-48db94783550')\n" 151 | ] 152 | } 153 | ], 154 | "source": [ 155 | "print(c)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "id": "4ae1bfee", 161 | "metadata": {}, 162 | "source": [ 163 | "We can visualize this object by doing:" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": 6, 169 | "id": "ba3c6980", 170 | "metadata": {}, 171 | "outputs": [ 172 | { 173 | "data": { 174 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAALMAAAF7CAYAAACZ2LqeAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dd3BU5foH8O+mkbKQopCEwM9IQggQpYgQgaElqNSRJkUDCBeVJk3BuVev3LEil6bAFVGEhHJJMPROCiIEEYEAARIIEHpPW9K2PL8/vGSIKWSTs/vuvuf5zGQclpNzvnn8stk9++5ZDRERGLN/cQ6iEzCmFC4zkwaXmUnDSXQApV27dg2HDh0SHcPmvf7666IjKE4j2xPA2NhYDB06VHQMmyfZ/3ZA5ieARMRfFXytX79e9P8ai5G2zEx9uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwaXGYmDS4zkwaXmUmDy8ykwWVm0uAyM2lwmZk0uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwaXGYmDS4zkwaX2QpMJpPoCKrgJDqApcTGxoqOAAAgIuzatQu9evUSHQUAkJKSIjqCxUhb5qFDh4qOUMbKlStFR5CedA8zXn/9dRCRzXyNHz8eAJCYmCg8y+NfMpKuzLZEr9dj7dq1AFD6X2Y5XGYL2rNnD3JzcwEA69evR3FxseBEcuMyW9DatWvh7OwMANDpdNi9e7fgRHLjMltIQUEBNm7cCL1eDwBwdHTEmjVrBKeSG5fZQrZu3YqioqLSPxsMBmzevBk6nU5gKrlxmS1kzZo1cHR0LHObXq/Hli1bBCWSH5fZAnJycrBr1y4YDIYyt2s0GsTExAhKJT8uswVs2LChwpewjUYj9u7di3v37glIJT8uswU86d73559/tlISdeEyK+zmzZv49ddfYTQaK/x7IkJ0dLSVU6kDl1lh69evh4ND5WM1mUxISUlBVlaWFVOpA5dZYTExMeWe+P0VESEuLs5KidRD2lVzIty4cQPZ2dkICAgovc1gMKCoqAharbbMtjIvxRRFQ7IuobIRsbGxGDp0qLQr1WxIHD/MYNLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwaXGYmDS4zkwaXmUmDy8ykwWVm0uAyM2lwmZk0uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYMvNq4wIsL169dx+/ZtPHz4ECdOnAAA7Nu3D1qtFh4eHmjUqBG8vb0FJ5UPX2y8FogIaWlpSExMxKFDh5CRkYH09HQUFBQ88XsbNGiA0NBQtGzZEl26dEH37t3h6+trhdTSiuMym8lkMiEpKQkxMTHYuXMn7ty5A29vb3Tu3BktWrRASEgIQkJC4O/vDw8Pj9J745ycHOh0Ouh0OmRlZSE9PR3p6elITU3FkSNHYDQa0bJlSwwaNAhRUVEICgoS/aPamzgQq5Y7d+7Qxx9/TI0bNyYA1L59e5o7dy798ccfZDQaa7Xv/Px82rFjB02dOpX8/f1Jo9FQp06daPXq1aTX6xX6CaQXy2V+gmvXrtGUKVPI3d2dGjRoQP/4xz/o7NmzFjuewWCgnTt30rBhw8jJyYmCgoLo+++/p+LiYosdUxJc5sqUlJTQwoULSavVkq+vL3311Vf08OFDq2a4dOkSvffee+Tq6krBwcG0a9cuqx7fznCZK7J//34KDQ0ld3d3+uKLL4TfK16+fJlee+01AkDDhg2j27dvC81jo7jMjzMYDDR79mxydHSkfv360eXLl0VHKmPbtm0UGBhI/v7+lJiYKDqOreEyP3L79m3q3r07ubq60pIlS0THqVROTg4NGjSIHB0d6V//+heZTCbRkWwFl5mIKDMzk4KDgykoKIiOHTsmOk61fPvtt+Ts7EwjR46kkpIS0XFsQazqzzOnpaXhlVdegZ+fH3bs2IEGDRqIjlRtCQkJGDBgADp06ID4+HjUrVtXdCSR1P0JrWlpaejSpQtCQ0ORlJRkV0UGgIiICOzbtw8nTpzAoEGDUFJSIjqSUKq9Z7527Ro6deqEhg0bIiEhAe7u7qIj1dipU6fQpUsXvPrqq1izZg0cHFR5H6XOe+bc3Fz07NkTXl5e2Llzp10XGQCee+45/Pzzz9i4cSNmzpwpOo4wqizz3/72N+Tk5GDnzp3w8vISHUcRPXr0wIoVKzB//nzEx8eLjiOG2Ceg1rd48WJycHCgPXv2iI5iEW+//TZ5eXnRxYsXRUexNnWdzUhPT0fr1q0xa9YszJ49W3QciygqKkJ4eDi0Wi0OHDgAjUYjOpK1qGsJaGRkJO7cuYNjx47ByUne9yWcOnUKbdu2xbJlyzBmzBjRcaxFPU8A161bh6SkJCxbtkzqIgN/PiGcMGECZs6ciXv37omOYzWquGcuKSlBcHAwXnnlFSxfvlx0HKvIzc1FaGgoRowYgXnz5omOYw3quGeOjo7GrVu38NFHH4mOYjWenp6YOXMmvvvuO9y9e1d0HKuQvsxGoxFz587FqFGj8Mwzz4iOY1XvvPMOtFotFi1aJDqKVUhf5q1bt+LChQuYNWuW6ChW5+7ujqlTp2LJkiUoKioSHcfipC9zdHQ0IiIiEBwcLDqKEGPHjoVOp8OWLVtER7E4qcv84MED7NixA1FRUaKjCNOgQQNERkYiJiZGdBSLk7rMGzZsgJOTEwYMGCA6ilAjR47E7t278eDBA9FRLErqMu/duxfdu3eHVqsVHUWoXr16wWQyITk5WXQUi5K2zESE5ORkdO/eXXSUUidPnsSiRYuQnZ2tyHbV5eXlhTZt2iApKUmR/dkqact86tQp3Lt3z6bK/Ouvv2Lq1Km4deuWItuZo0ePHkhMTFRsf7ZI2jIfP34cbm5uaNWqlegoNiE8PBznzp1DYWGh6CgWI22Zz507h6ZNm6r1XRflNGvWDCaTCefPnxcdxWKk/T+dnp6OZs2aKbrP5ORkTJw4ESEhIWjcuDGGDx+O7777Dkajsdy2v//+O4YMGYImTZogMjISixcvRkXLYKq7XW0FBwfDyckJ586dU3zftkLa5WMXL15E7969FdtfUlISevbsCU9PT4wYMQJPP/009u7di/Hjx+PixYv4+uuvS7dNTk5Gnz594OrqioEDB8LBwQEff/xxuXe1VHc7Jbi4uKBx48a4ePGi4vu2GcLeF2BhgYGBNGfOHMX2N27cOKpTpw5lZ2eX3lZYWEj+/v4UGhpaZttWrVqRt7c3Xbp0qfS2jIwMcnd3JwB05swZs7ZTSps2bejDDz9UdJ82JFbahxn5+fmKXkdi+vTp+P3338vca5aUlMDLywt5eXmltx0+fBipqamYMGECAgMDS29v2rRpmVciq7udkurWrYv8/HyL7NsWSPswQ+kyh4aG4v79+5g3bx5SUlJw+fJlnD9/Hnl5eWjYsGHpdo8ek7Zu3brcPlq2bGn2dkqqV6+e1GWW9p5Zo9Eo+kRq7ty5aNSoET799FPo9XpERkZi5cqV6NSpU5ntHr1k7OjoWG4frq6uZm+nJCXnYYukLbNWq4VOp1NkX3fv3sWHH34IT09PXL16FZs3b8ZXX32FAQMGlFta+eyzzwIA9u/fX24/ly9fNns7JSn928rWSFtmJR8fZmVlwWQyYeDAgWXKcPXq1dJPk3qkXbt2cHZ2Lvdqm8FgwNq1a83eTklcZjvl5eWl2NqGZs2aQavVYv369di6dSvOnz+PlStXomPHjqhXrx50Oh3S09MBAI0bN8bEiRNx6tQpjB07FseOHcPx48cxePBg5Obmlu6zutspKScnB56enhbZt00QfT7FUgYOHEhDhgxRbH+xsbGk1WoJAAEgHx8fWrVqFW3YsIE8PDzIycmpdNuioiIaN25c6bYAKCIigmJiYsqccqvudkooLi4mJycnio2NVWyfNkbei8D8/e9/x/bt25GamqrYPu/fv4/jx4/D398fLVq0KL3Ayv3795GdnV3u3SxXr17FqVOn0Lx589LHyBWp7na1kZaWhrCwMKSmpuL555+3yDEEk/ciMKtWrcL48eORn59f4RkDtYmPj8eQIUOg0+ng5uYmOo4lyHupgbZt26KwsFDRe2Z7dvjwYTRv3lzWIgOQ+AlgWFgYfH19pV+QXl2JiYno0aOH6BgWJW2ZNRoNunTpwmXGn2cxTpw4YVNvVLAEacsMAD179kRSUpLUL+FWx/bt2+Hg4IBu3bqJjmJRUpd58ODBMBqN6r349v/ExMSgV69e8Pb2Fh3FoqQus7e3N/r06aOKa0ZU5vbt20hISFDFtUOkLjMAjBo1CklJSVK/Xagqy5cvR926ddG3b1/RUSxO+jL36dMHTZs2xVdffSU6itU9fPgQ33zzDSZNmmSxlXi2RPoyOzo6YtasWYiJiUFWVpboOFb1n//8BwUFBZg8ebLoKFYh7SuAj9Pr9WjatGnpJzKpQU5ODpo1a4aRI0di7ty5ouNYg7yvAD7O2dkZX3/9NVauXFnh+mEZffTRRzAajfjwww9FR7EaVdwzP9KnTx9cuXIFx44dg7Ozs+g4FnPs2DG0b98eP/74I0aNGiU6jrXIu9CoIufPn0erVq0wY8YMfPrpp6LjWERhYSE6dOgALy8v7N+/X1UfnaaKhxmPNG3aFAsWLMAXX3yBPXv2iI5jEZMnT8a1a9ewatUqNRX5T6JWUos0bNgw8vX1patXr4qOoqhVq1aRRqOhTZs2iY4igryL86uSl5eHl156CRqNBr/88gt8fHxER6q1ffv2oU+fPpg2bZoqz6lDbY+ZH3f9+nV06tQJfn5+SEhIgIeHh+hINXby5El06dKl9KV7lV4sUl2PmR8XEBCAvXv34tKlS+jTp4/F3kRqaUeOHEFkZCTatWuHn376Sa1FBqCCVwCr0rRpUyQmJiIzMxOdO3fG9evXRUcyy969exEZGYnw8HBs2bIFLi4uoiMJpeoyA39eCuvAgQPQ6/Xo3Lkzfv/9d9GRnoiIsGDBAvTu3RtDhgxBfHw83N3dRccST+jzTxty9+5devnll8nFxYUWLlxIJpNJdKQK3b9/n/r3709OTk705Zdf2mxOAWK5zI8xGo30+eefk5OTE7366qt04cIF0ZHKiI+Pp8aNG1OjRo3ol19+ER3H1nCZK3Lw4EEKCwsjV1dX+uSTT6igoEBonvPnz1Pv3r1Jo9FQVFQU3b17V2geG6XO88zVYTAYsGTJEvzzn/9EnTp1MGHCBPj5+WHkyJFWeXy6fft2uLi4YNu2bVi2bBmeeeYZLF68GD179rT4se1UHN8zP8GtW7fogw8+IFdXV9JoNDRz5kw6deqUxY5XUlJCW7dupc6dOxMAatKkCa1cuZL0er3FjikJea+crxRfX1+0b98eJSUl8PT0RGxsLJ577jm0bdsWX375JX777TcYDIZaHSM3NxdbtmzBpEmTEBAQgP79+6OgoAAAUFxcjIiICDg5SXtdeMXww4wn2LJlCwYOHAij0Yjw8HAcOnQIBw4cQExMDLZv346bN2+iXr166NSpE1q0aIGQkBA0a9YMvr6+0Gq1pV/5+fnIycmBTqdDVlYW0tPTkZ6ejhMnTuDYsWMgIjz//PMYNGgQoqKiUKdOHfj7+0Oj0SAwMBAHDx6Ev7+/6HHYMvW+nF0de/fuRZ8+fWA0GkFEGDJkCNavX19mm3PnziExMREpKSk4d+4c0tPTq3WdjoYNGyI0NBQtW7ZE165d0a1bNzz11FOlf28ymeDi4gKj0QhnZ2c0btwYBw8ehJ+fn+I/pyS4zJX59ddf0bNnT+j1ehiNRri4uGDChAlYsGDBE7/35s2buHv3LnQ6XelXvXr14OXlBa1Wi4CAgGpd9Pvpp5/G/fv3Afz5bpng4GAcOHCgTOlZqTh+IFaBlJQUvPzyy6VFBv683Fd17xX9/f0VeUjg6+tbWma9Xo8LFy6ga9euOHDggPQXdKkJfgL4F8ePH8crr7yCkpKSMp+8ajAYrP4rvnHjxmX+rNfrkZGRgZdffln1lxyrCJf5MampqejWrRsKCgrKfYSw0Wi0+hOwRo0alTuLodfrkZqaip49eyr2AUSy4DL/T3p6Onr06FFhkR+x9j2zn59fhRdK1+v1+OOPP9C/f/9yn3alZlxm/PlG186dOyMvL6/Kc8bWvmf29/eHyWSq8O8MBgMOHDiAfv36obi42Kq5bJXqy/xoLXN2dnaVRXZ0dLT6WQQ/Pz/o9fpK/95gMCAhIQFDhw6t9Qs3MlB9ma9fv44mTZqUns+tjI+Pj9XfxfGk3wTOzs5wdHSEj49P6VkPNVN9mbt06YKUlBQcPXoUQ4YMgYODQ4WlFvFiRWXHdHR0hJubG8aPH49Lly5hxYoV8PX1tXI628Pnmf/nhRdewJo1a9ChQwe8//77cHJygkajKf0136hRI6tnevyeWaPRQKPRwMPDAwaDAefPn0dAQIDVM9ky1d8z/9W6devQv39/ZGVlYdq0aaXv2hZRHDc3t9LjBwUFYfny5cjMzISLiwvWrVtn9Tw2T9yKPduTmJhIAOjQoUOlt+Xm5tKcOXNo8eLFQjKNHj2aNm/eTEajsfS26dOnU0BAABUXFwvJZKN4cf7jevfujYKCAiQnJ4uOUqVr164hKCgIy5Ytw+jRo0XHsRW80OiRU6dOoVWrVti2bRt69+4tOs4TjRo1CkeOHEFaWpqqr5XxGC7zI1FRUThx4gROnjxpFxccPHv2LMLCwrBp0yb069dPdBxbwGUG/vy13aRJE/zwww8YOXKk6DjV1rdvX+Tl5eGXX34RHcUWqPfyXI+bN28efH19MWzYMNFRzPLBBx/gwIEDOHTokOgoNkH1Zc7OzsYPP/yAadOm2d3lrbp27YqXXnoJ//73v0VHsQmqL/OSJUvg6OiIsWPHio5SIzNmzMCmTZtw5swZ0VGEU3WZi4uLsWTJEkycOBGenp6i49TIgAEDEBoaWq23c8lO1WX+6aefkJ2djUmTJomOUmMODg6YMmUKYmJicOPGDdFxhFJtmY1GI+bPn4/Ro0fb/Vv4R48eDR8fH3z77beiowil2jLHx8cjMzMTU6ZMER2l1urUqYNJkyZh6dKldnvRdCWotszz58/HgAED0Lx5c9FRFDFx4kRoNBosX75cdBRhVFnmpKQkHD58GDNmzBAdRTGenp4YO3YsFi5ciJKSEtFxhFDlK4C9evVCYWGhzS8oMtejBUjfffcd3nrrLdFxrE19L2fb24Iic6l4AZL6yhwVFYU//vgDaWlpdrGgyFwqXoCkrjLb64Iic6l0AZK6FhrZ64Iic6l1AZJqymzPC4rM9WgB0ty5c0VHsSrVlNneFxSZa8aMGdi8ebOqFiCposxFRUV2v6DIXGpcgKSKMsuwoMhcDg4OmDp1qqoWIElfZqPRiAULFkixoMhco0aNUtUCJOnLLNOCInOpbQGS9GWWbUGRudS0AEnqMsu4oMhcalqAJPUrgLIuKDKXShYgyftytuwLisylggVI8pZZ9gVF5lLBAiQ5y6yWBUXmknwBkpwLjdSyoMhcsi9Akq7MDx48UM2CInPJvgBJujKrbUGRuWRegCRVmYuKirB06VJVLSgyl8wLkCp8AljZBynaumXLlmH69OnIzMy0+KdDWfL0lqXnv3z5ckyZMgWZmZl2u16lgvnHlftMk/Xr1xMA/nrCl6Xw/Gs8/9hKPzotLi6usr9StZSUFMyfP9/ix+H5V6yq+Vda5sGDB1sskD2z1kMwnn/Fqpq/VE8AmbpxmZk0uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwaXGYmDS4zkwaXmUmDy8ykwWVm0uAyM2lwmZk0uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwadlFma31eNauYvcy/0g+Ct5UfwGQyYdGiRZg2bZroKAAAIrLKcXj+Fatq/pWW2dHR0SJhaur9998XHcGqeP7mK1fmjh07Ii4uTkSWCn3//ffYu3cvPvnkE4SFhYmOY3E8/1ogG1ZSUkKenp4EgMaOHSs6jurY2fxjbfoJ4O7du5GbmwsAiI2NRXFxseBE6mJv87fpMq9duxbOzs4AAJ1Oh127dglOpC72Nn+bLXNBQQE2bdoEvV4P4M8nRGvWrBGcSj3scf42W+YtW7agqKio9M8GgwFbtmyBTqcTmEo97HH+Nlvm1atXw8GhbDy9Xo/NmzcLSqQu9jh/myxzdnY29uzZA6PRWOZ2jUaDmJgYQanUw17nb5Nl3rBhQ7lBAoDRaMS+fftw584dAanUw17nb5Nlrupfv0ajQXx8vBXTqI+9zt/mynzz5k0cPHiw0rUJJpMJ0dHRVk6lHvY8f5sr83//+99yTzweZzKZcPjwYWRlZVkxlXrY8/xtrswxMTEwGAxVbkNEiI2NtVIidbHn+Ve6ak6EGzduIDs7GwEBAaW3lZSUQKfTwcfHp8y2KSkp1o4nPXufv4bISgt0ayg2NhZDhw612jpiVpYdzT/O5h5mMFZTXGYmDS4zkwaXmUmDy8ykwWVm0uAyM2lwmZk0uMxMGlxmJg0uM5MGl5lJg8vMpMFlZtLgMjNpcJmZNLjMTBpcZiYNLjOTBpeZSYPLzKTBZWbS4DIzaXCZmTS4zEwaXGYmDS4zkwaXmUmDy8ykwWVm0uAyM2nYdJnz8vKQnZ0NALh37x4KCwsFJ1IXe5u/TVxs/Pbt20hOTkZKSgrOnj2LjIwMXLlypcIPiXF1dUVISAhCQkIQFhaGrl27Ijw8HK6urgKSy0GS+ccJK/OVK1cQExOD9evX4/Tp03B0dESbNm3QsmVLNGvWDIGBgfD29oaHhwfc3NyQm5uLhw8f4v79+8jIyEBGRgaOHz+Oixcvws3NDd26dcObb76JAQMGwM3NTcSPZFcknH8cyMoSEhKoZ8+e5ODgQPXr16fJkyfT9u3bKT8/v0b7u3z5Mq1YsYL69etHzs7OVK9ePZo8eTJlZWUpnFwOEs8/1mplTkpKovDwcAJAERERtHnzZiopKVH0GHfu3KH58+fT//3f/5GLiwuNHTuWbty4oegx7JUK5m/5Mt+8eZPeeOMN0mg01Lt3bzp8+LClD0klJSX0448/UmBgIHl6etKiRYvIYDBY/Li2SEXzt2yZt2zZQk899RQFBATQqlWrLHmoChUUFNAnn3xCderUoRdffJEuXrxo9QwiqWz+limzXq+n6dOnk0ajoTFjxpBOp7PEYart9OnT1LJlS3rqqado+/btQrNYg0rnr3yZCwoKqH///uTh4UExMTFK777GHj58SKNGjSJHR0f6/vvvRcexGBXPX9ky5+bmUpcuXcjHx4dSUlKU3LVi/vWvf5FGo6HPP/9cdBTFqXz+ypW5sLCQunfvTv7+/nT69GmldmsRS5cuJY1GQwsXLhQdRTE8f4XKbDAYaODAgeTt7U0nT55UYpcWN3fuXHJwcKC1a9eKjlJrPH8iUqrMs2fPJldXVzpw4IASu7OaGTNmkKurK504cUJ0lFrh+ROREmVOTk4mR0dHWrp0qRKBrMpoNFJkZCQFBwdTbm6u6Dg1wvMvVbsy5+bmUkBAAA0ZMqS2QYS5fv06NWjQgMaNGyc6itl4/mXUrsxTp04lHx8funv3bm2DCLVu3TpycHCgQ4cOiY5iFp5/GTUvc2pqKjk5OdHy5ctrE8BmREREUJs2bchoNIqOUi08/3JqXubBgwdT27Zt7eZ//pOkpaWRg4MDxcXFiY5SLTz/cmpW5rNnz5KDgwNt2LChpge2SYMHD6bWrVuTyWQSHaVKPP8K1azM48aNo+bNm0tzr/DI8ePHCQDt3btXdJQq8fwrFGv2ewCLiooQFxeHt99+Gw4ONv0WQrO1bt0a4eHhiI6OFh2lUjz/ypk9jY0bN0Kn02HYsGE1OqCti4qKQnx8PPLz80VHqRDPv3Jmlzk+Ph6RkZHw8/Mz+2CP27x5M2JjY2u1D0sYOnQoioqKsHv3btFRKsTzr4I5D0pMJhM9/fTTNG/evJo8pimjXbt29Oyzz9Z6P5bQvn17mjBhgugY5fD8qxTrZE7xT548iXv37qFHjx7m/6v5i0mTJtnsdRh69OiBTZs2iY5RDs//Ccyp/g8//EBardamnkWbTCbFT6Vt3bqVANjceg2ef5XMO5tx7tw5hISEKPIs+r333sNbb71V+udx48Zh0qRJuHHjBkaMGIFnnnkGQUFBGDNmDB4+fFju+1NTUxEZGQkvLy+4u7ujQ4cO2LlzZ61zAUBoaCgAICMjQ5H9KYXn/wTmVL9fv340fPhw8/6ZVeKvj9natWtHgYGBFBAQQJ07d6aZM2dS165dCQANHDiwzPcmJSWRq6srBQQE0LRp02jMmDHk6elJTk5OdPDgwVpnMxgM5OLiQmvWrKn1vpTE86+SeS+avPDCCzRz5kzzklWiomECoFmzZpX+2jIajdS2bVvy9PQs3c5oNFKrVq3I09OTzp8/X3r72bNnSaPR0BtvvKFIvsDAQJozZ44i+1IKz79K5j3MyM/PR926dc276zeDm5sbZs+eDY1GAwBwcHBAp06dkJubi2vXrgEAjh8/jtTUVLz22msIDg4u/d7Q0FB88803aN++vSJZ6tatC51Op8i+lMLzr5pZZzMsPcwGDRqUuwCft7c3AJT+YBcuXAAAPPfcc+W+f9KkSYplqVu3LvLy8hTbnxJ4/lUz657ZZDJZ9CXUqi64R/+7vuPdu3cBAAEBARbLAQBOTk4wGo0WPYa5eP5VM2syWq22wme21hQYGAgA+O2338r9XXR0NFauXKnIcfLy8ix6L1gTPP+qmV1m0WsWXnzxRbi5uSExMbHM7WfOnMHo0aOxf/9+RY5j6V/pNcHzr5pZZW7QoAFu3bpl1vDQ6PEAAATnSURBVAGU5uvri6lTp+LkyZN49913cfToUURHR2P48OFwcnLCu+++W+tjEBFu376N+vXrK5BYOTz/qpn1BDAkJASpqalmHcASPv30UxAR5s6di2XLlgEA/P39sWbNGnTo0KHW+79x4wZ0Oh2aNWtW630pief/BOacyPvmm2+ofv365nyLRel0Ojp06BCdPn2aiouLFdtvYmIiAaBbt24ptk8l8PyrZN5Co1atWuHu3bu4dOkSnn32WfP+1ViAh4cHXnrpJcX3e+TIEfj5+cHX11fxfdcGz79qZj1mDg8Ph7u7O5KSksw6iL1JSEhQZGWa0nj+VTOrzC4uLujYsSMSEhLMPpC9KCoqwsGDB22yzDz/qpl9Br5fv37Ytm0bCgoKzD6YPdi+fTuKi4vRq1cv0VEqxPOvnNllHj58OAoLC21y8boSoqOjERERgYYNG4qOUiGef+XMLnP9+vXx6quv4scffzT7YLbuxo0b2LVrF6KiokRHqRTPvwo1OXWyb98+AmB312Z7kmnTppGfnx8VFhaKjlIlnn+Fan55rg4dOlD//v1r+u02586dO+Th4UHz588XHaVaeP7l1LzM27dvJ41GQ0lJSTXdhU155513yM/PT/gnM1UXz7+c2l3Stm/fvtSiRQvFP+nT2o4ePUqOjo60evVq0VHMwvMvo3ZlzszMJDc3N5o9e3ZtdiNUUVERtW7dmrp162bzF0z8K55/GbX/GIhvv/2WHB0d7fbX3YQJE8jT05MuXLggOkqN8PxLKfMBPYMGDaKGDRvStWvXlNid1axZs4Y0Go3dXJO5Mjx/IlKqzNnZ2dSiRQsKCwujBw8eKLFLi9u3bx/VqVOHZsyYITpKrfH8iUjJD7W8evUqNW7cmDp27Eh5eXlK7dYifvvtN6pXrx6NGDHCpq4OVBs8f4U/bvjMmTPk5+dHbdu2pdu3byu5a8Xs3r2btFot9e3bV9E1uLZA5fNX/oPgMzMzKTg4mIKDgyktLU3p3dfK8uXLycXFhUaOHGn3p7Mqo+L5K19mIqJbt25Rp06dyN3dnVasWGGJQ5glPz+f3nzzTdJoNPTRRx/Z3Sk4c6l0/pYpMxGRXq+nWbNmkUajoddee42ysrIsdagq7dixg4KCgqh+/fq0c+dOIRlEUOH8LVfmR/bt20chISHk4eFBn332mdWenJw9e5YGDBhAAGjIkCF0/fp1qxzX1qho/pYvM9Gfr/J89tlnVLduXfLx8aHZs2fTnTt3LHKso0eP0uuvv04ODg7UokUL2r17t0WOY09UMn/rlPmRBw8e0OzZs8nHx4ecnZ2pX79+tH79esrOzq7VfjMyMmjOnDkUFhZGAKh169YUFxcnzWk3pUg+/1gN0f8uImZFhYWF2LhxI1avXo09e/YAANq2bYsuXbqgRYsWaNasGYKCgqDVaqHVah+tu0ZOTg6ys7ORkZGB9PR0HD9+HElJSbhy5Qp8fHwwdOhQvPnmm+jYsaO1fyS7Iun844SU+XEPHjxAcnIyEhMTcejQIaSnp5d7f5uHh0e5a6w1aNAAYWFh6Nq1KyIiItC+fXs4OztbM7oUJJq/+DL/FRHh6tWruHz5MnQ6HXQ6HR4+fAitVgtvb294enqiadOm8PLyEh1VSnY8f9srM2M1FCfX59UyVeMyM2k4AcgWHYIxBTz8f0BtYAlbB9dFAAAAAElFTkSuQmCC", 175 | "text/plain": [ 176 | "" 177 | ] 178 | }, 179 | "execution_count": 6, 180 | "metadata": {}, 181 | "output_type": "execute_result" 182 | } 183 | ], 184 | "source": [ 185 | "c.visualize()" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "id": "2714579a", 191 | "metadata": {}, 192 | "source": [ 193 | "Up to this point the object `c` holds all the information we need to compute the result. We can evaluate the result with `.compute()`." 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": 7, 199 | "id": "3cfea4e1", 200 | "metadata": {}, 201 | "outputs": [ 202 | { 203 | "name": "stdout", 204 | "output_type": "stream", 205 | "text": [ 206 | "CPU times: user 1.41 ms, sys: 1.33 ms, total: 2.73 ms\n", 207 | "Wall time: 2.01 s\n" 208 | ] 209 | }, 210 | { 211 | "data": { 212 | "text/plain": [ 213 | "5" 214 | ] 215 | }, 216 | "execution_count": 7, 217 | "metadata": {}, 218 | "output_type": "execute_result" 219 | } 220 | ], 221 | "source": [ 222 | "%%time\n", 223 | "\n", 224 | "c.compute()" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "id": "b69fe21d", 230 | "metadata": {}, 231 | "source": [ 232 | "Notice that now the computation took 2s instead of 3s, this is because the two `inc` computations are run in parallel. \n", 233 | "\n", 234 | "**Note for Binder users**\n", 235 | "\n", 236 | "If you are running this notebook using binder, you will probably not see a speed-up. This happens because binder instances tend to have only one core with no threads so you can't see any parallelism. We can \"fix\" this by setting the number of workers to a higher number, but there is no guarantee that we will get these resources. \n", 237 | "\n", 238 | "For now, you can try copying the following lines in a cell and executing the same computation as before and see what happens. In one cell execute:\n", 239 | "\n", 240 | "\n", 241 | "```python\n", 242 | "import dask\n", 243 | "dask.config.set(scheduler='threads', num_workers=4) #setting num_workers\n", 244 | "```\n", 245 | "\n", 246 | "and in a separate cell try to run this again:\n", 247 | "\n", 248 | "```python\n", 249 | "%%time\n", 250 | "c.compute()\n", 251 | "```\n", 252 | "\n", 253 | "Don't worry about the syntax for now, we will explain this in the next lesson. " 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "id": "833397dd", 259 | "metadata": {}, 260 | "source": [ 261 | "## Parallelizing a `for`-loop\n", 262 | "\n", 263 | "When we perform the same group of operations multiple times in the form of a `for-loop`, there is a chance that we can perform these computations in parallel. For example, the following serial code can be parallelized using `delayed`: " 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 8, 269 | "id": "c79c75b0", 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "data = list(range(8))" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "id": "5926bbf6", 279 | "metadata": {}, 280 | "source": [ 281 | "#### Sequential code" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": 9, 287 | "id": "8153b1b6", 288 | "metadata": {}, 289 | "outputs": [ 290 | { 291 | "name": "stdout", 292 | "output_type": "stream", 293 | "text": [ 294 | "CPU times: user 1.31 ms, sys: 1.35 ms, total: 2.67 ms\n", 295 | "Wall time: 8.02 s\n" 296 | ] 297 | } 298 | ], 299 | "source": [ 300 | "%%time\n", 301 | "results = []\n", 302 | "for i in data:\n", 303 | " y = inc(i) # do somthing here\n", 304 | " results.append(y)\n", 305 | " \n", 306 | "total = sum(results) # do something here" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 10, 312 | "id": "a18b54f1", 313 | "metadata": {}, 314 | "outputs": [ 315 | { 316 | "name": "stdout", 317 | "output_type": "stream", 318 | "text": [ 319 | "total = 36\n" 320 | ] 321 | } 322 | ], 323 | "source": [ 324 | "print(f'{total = }')" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "id": "5d269dee", 330 | "metadata": {}, 331 | "source": [ 332 | "### Exercise: \n", 333 | "\n", 334 | "Notice that both the `inc` and `sum` operations can be done in parallel, use `delayed` to parallelize the sequential code above, compute the `total` and time it using `%%time` " 335 | ] 336 | }, 337 | { 338 | "cell_type": "code", 339 | "execution_count": 11, 340 | "id": "d6905529", 341 | "metadata": {}, 342 | "outputs": [], 343 | "source": [ 344 | "#solution\n", 345 | "results = []\n", 346 | "for i in data:\n", 347 | " y = delayed(inc)(i) \n", 348 | " results.append(y)\n", 349 | " \n", 350 | "total = delayed(sum)(results) " 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "id": "fd452def", 356 | "metadata": {}, 357 | "source": [ 358 | "In the code above, the `sum` step is not run in parallel, but it depends on each of the `inc` steps, that's why it needs the `delayed` decorator too. The `inc`steps will be parallelized, then aggregated with the `sum` step.\n", 359 | "\n", 360 | "Notice that we can apply delayed to built-in functions, as we did in the case of `sum` in the code above. " 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": 12, 366 | "id": "6f52a461", 367 | "metadata": {}, 368 | "outputs": [ 369 | { 370 | "data": { 371 | "text/plain": [ 372 | "Delayed('sum-c14bce50-fd0f-4eae-a781-b94226956e95')" 373 | ] 374 | }, 375 | "execution_count": 12, 376 | "metadata": {}, 377 | "output_type": "execute_result" 378 | } 379 | ], 380 | "source": [ 381 | "total" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": 13, 387 | "id": "bd0270ec", 388 | "metadata": {}, 389 | "outputs": [ 390 | { 391 | "data": { 392 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvMAAAGACAYAAAAgW+0lAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdeVjU5f4+8Hs2QDYBARH3FTTNhdx3xTTDytxKJW2jU3nwtIll3zMdOxX9rhYqW0hNMUPFUsPSEpVyKxXFXcF9RxNEZJPt/fujL/MFxX2YZz7D/bourithmLnnwfCezzyLTkQERERERESkNYv1qhMQEREREdGdYZknIiIiItIolnkiIiIiIo0yqg5AROQoTp06hU2bNqmOYfdGjx6tOgIRkcPQcQEsEZF1JCQkYMyYMapj2D3+s0NEZDVcAEtEZG0iwo8qPhYtWqT6R0NE5HBY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYiIiIg0imWeiIiIiEijWOaJiIiIiDSKZZ6IiIiISKNY5omIiIiINIplnoiIiIhIo1jmiYgcWH5+vuoIRERUjYyqAxAROZqEhATVEQAAJSUlWLBgAcLDw1VHAQD88ccfqiMQETkclnkiIisbM2aM6giV/PTTT6ojEBFRNeE0GyIiKxk9ejRExG4+Ro4cCQD4/vvvlWep+EFERNbDMk9E5IByc3OxfPlyAMB3332nOA0REVUXlnkiIge0bNkyFBcXA/h7ms2lS5cUJyIiourAMk9E5IDmz58PnU4HACgtLcWyZcsUJyIiourAMk9E5GAuXLiA1atXo7S0FACg0+nw7bffKk5FRETVgWWeiMjBLF68uNKfS0tLkZycjHPnzilKRERE1YVlnojIwXz77bfX7Bqj1+uvKflERKR9LPNERA7k5MmT+PPPP1FWVlbp86WlpZg3b56iVEREVF1Y5omIHMiCBQtgMBiu+byIICUlBUePHlWQioiIqgvLPBGRA5k3b55l4evVjEYjFi5caONERERUnXTC4/iIiBzCgQMH0Lp16xvepmXLlkhPT7dRIiIiqmaLeWWeiMhBxMfHw2Qy3fA2Bw8exJ49e2yUiIiIqhvLPBGRg5g3b57l1NcbWbBggQ3SEBGRLRhVByAiort39uxZdOnSBV26dLF8LjMzE+np6ejevXul2xYVFdk6HhERVRPOmSciclAJCQkYM2bMNXvOExGRw+CceSIiIiIirWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijTKqDkBERNaTk5ODCxcuIC8vDwcPHgQAbNu2DW5ubnBzc4Ofnx9cXFwUpyQiImvRiYioDkFERLenrKwMO3fuRHJyMvbs2YMDBw4gLS0NWVlZN/w+vV6PRo0aoVWrVggODkbnzp0xYMAABAYG2ig5ERFZ0WKWeSIijcjLy8PSpUuxZMkS/P7778jKyoKfnx86duyIoKAgBAcHo2nTpvD397dciXdzc0N2djZyc3ORl5eHs2fPIj09HWlpadi/fz9SU1NRVFSEoKAghIaG4rHHHkPPnj2h0+lUP10iIro5lnkiInu3fv16zJ49Gz/88AOKioowaNAgDBo0CP3790e7du3uqnjn5+dj48aNSE5Oxs8//4xdu3ahefPmCA8Px9NPP40GDRpY8ZkQEZGVscwTEdkjEcHKlSvx7rvvYuPGjejcuTOeeOIJPPbYY/D19a22x925cyfmzZuH+Ph4ZGVlYcKECYiKikLz5s2r7TGJiOiOLeZuNkREdmbDhg3o3LkzwsLC4OXlhY0bN2LLli2YNGlStRZ5AGjfvj0+/PBDHD9+HJ9//jnWrl2L4OBgPP300zh//ny1PjYREd0+lnkiIjvx119/4amnnkKfPn3g6+uL7du346effkKPHj1snsXJyQnPPPMMDhw4gDlz5mDVqlUIDg7Gl19+ibKyMpvnISKiqrHMExHZgaVLlyI4OBhJSUlISEjAL7/8gg4dOqiOBaPRiPHjx2P//v145plnMHnyZPTq1QvHjx9XHY2IiMAyT0SkVFFRESZPnowRI0Zg5MiR2L9/P0aOHKk61jXc3d3x//7f/8P27duRm5uLjh074scff1Qdi4ioxmOZJyJS5Ny5c+jVqxfmzJmD7777DrGxsXB3d1cd64batm2LzZs3Y8SIERg+fDhef/11cB8FIiJ1eAIsEZECR44cweDBg6HT6ZCSkoJWrVqpjnTLatWqhZkzZ6JXr16IiIjAmTNnMGvWLJhMJtXRiIhqHJZ5IiIb27VrF4YMGYLAwECsWLEC/v7+qiPdkQkTJiAgIAAjR47EhQsXsHjxYri6uqqORURUo3CaDRGRDR08eBCDBg1CcHAwkpOTNVvkyw0ePBhr1qzB5s2bMWbMGJSUlKiORERUo/DQKCIiGzlz5gx69eoFX19frF271u7nx9+OrVu3YsCAAXj44Yfx7bff3tWptEREdMt4aBQRkS3k5eXhgQcegIuLC1asWOFQRR4AOnfujISEBCQkJOB//ud/VMchIqoxWOaJiGzghRdewJkzZ/DLL79U+ymuqjzwwAP48ssv8e6772LFihWq4xAR1QhcAEtEVM3mzJmDb7/9FsuWLUOjRo1Ux6lWTz/9NNavX4/w8HBs374djRs3Vh2JiMihcc48EVE1OnjwIDp06IB//vOfiI6OVh3HJvLy8tC5c2f4+/sjOTmZ8+eJiKrPYpZ5IqJqNHjwYGRkZGDbtm0wGmvOm6Hbt29Hly5dMGvWLEycOFF1HCIiR8UFsERE1SUhIQFJSUn49NNPa1SRB4BOnTrh+eefx6uvvooLFy6ojkNE5LB4ZZ6IqBoUFBSgVatWGDRoEL755hvVcZS4dOkSgoODMWLECMyYMUN1HCIiR8Qr80RE1WHWrFnIzMzEe++9pzqKMrVr18Z//vMfzJo1C6dPn1Ydh4jIIbHMExFZWXFxMT788EM8++yzqFu3ruo4Sj355JOoW7cuPvjgA9VRiIgcEss8EZGVffvttzh79ixeffVV1VGUM5lMeOWVV/D111/jr7/+Uh2HiMjhsMwTEVnZV199hcceewwNGzZUHcUuPPPMM3BycsK8efNURyEicjgs80REVpSeno6tW7diwoQJqqPYDVdXV4waNQrz589XHYWIyOGwzBMRWdHcuXNRv3599O3bV3UUuxIeHo4dO3Zg165dqqMQETkUlnkiIitauHAhwsPDYTAYVEexK7169UKTJk2waNEi1VGIiBwKyzwRkZUcOXIER48exQMPPKA6it3R6XQYMmQI1qxZozoKEZFDYZknIrKS5ORkuLq6omvXrqqj2KUBAwYgJSUF2dnZqqMQETkMlnkiIitJTk5Gz5494ezsrDqKXerfvz9EBOvXr1cdhYjIYbDMExFZybZt29C9e3fVMeyWr68vWrVqhW3btqmOQkTkMFjmiYisoKSkBIcPH0ZwcLDqKHatVatWSEtLUx2DiMhhsMwTEVnBkSNHUFxcjKCgINVR7FpQUBDLPBGRFbHMExFZQXp6OgCgZcuWVrvPwsJCmM1mNG/eHM7OzmjZsiWee+45XL58udLtnnjiCYwfP/6a74+Ojkbv3r1RUlICAHj22WcxYcIEHDp0CM888wwaNmyIAQMGWA5z+uijjxASEgJ/f3888MADOHjwoNWeS7mgoCDLWBER0d0zqg5AROQILly4AFdXV3h4eFjtPl944QXMmzcP4eHh6NixIw4fPoyZM2di9+7d2LRpk+V227ZtQ1lZ2TXff/DgQWzYsMHytR07duDUqVNYvXo1vLy80L9/fyxatAi//fYb4uPjkZSUhKFDh6Jx48b4+eefERoaiqNHj0Kvt951H39/f+Tl5aGwsBAuLi5Wu18iopqKZZ6IyAouX75s1SJ/5coVzJ8/Hw8++CDmzJlj+Xzz5s0xefJkpKeno1WrVrd9vxkZGfjvf/+LadOmAQAef/xxDB06FL/99hv27t1ruc+JEyciLi4Ohw4duqPHuZ7yMbp8+TLLPBGRFXCaDRGRFeTm5sLd3d1q91daWgoA+O2335Cammr5/KRJk5Cbm4vmzZvf0f0aDAa89tprlj+3b98ewN97wFcs7f369QMA7Nu3744e53o8PT0BADk5OVa9XyKimoplnojICvLz8+Hm5ma1+3N1dYXZbEZOTg46deqENm3a4MUXX8TKlSvh7OwMg8FwR/cbGBgIJycny5/Lr44HBgZWul35/RcVFd3hM6ha+Rjl5eVZ9X6JiGoqlnkiIitwdnZGYWGhVe9z2rRpOHToEP7nf/4Hrq6u+OqrrxAWFoZ77rkHGRkZN/3+rKysaz53vRcc1pwXfyMFBQUAgFq1atnk8YiIHB3LPBGRFXh4eFyzy8zdKCoqQnZ2Npo0aYLp06cjJSUFp0+fxqRJk5Ceno7PPvvMcludTlflAlh73AKyfIysub6AiKgmY5knIrICDw8P5ObmWu3+1q5dC29vbyxYsMDyuYCAAMt894sXL1o+36RJExw7dgzFxcWWz+3duxeHDh2yWh5rYZknIrIulnkiIivw9vZGbm4urly5YpX769mzJ/z9/TF9+nT89ttvuHTpErZt24Z//etfAIAHH3zQctuuXbuiqKgIEydOxG+//YZZs2bhkUceQe3ata2SxZoyMzPh5OQEV1dX1VGIiBwCt6YkIrKCFi1aQERw6NAh3HPPPXd9fx4eHvjuu+8wYcIE9O/f3/J5FxcXvPPOO5XK/CuvvII//vgD8fHxiI+PR/369REeHg7g74Oj7El6ejpatGgBnU6nOgoRkUPQiYioDkFEpHWFhYVwd3dHQkICHn30Uavdb35+Pnbt2oUTJ07A19cXbdu2hb+/f5W3/euvv3D69Gm0b9/ebsvy6NGjUVJSgiVLlqiOQkTkCBbzyjwRkRW4uLigUaNGVl906urqim7duqFbt243va2fnx/8/Pys+vjWlp6ejiFDhqiOQUTkMFjmiYiu8v333yM9PR2BgYHw9/dHYGAg6tatC39//xvu796+fXts3brVhkm1JTc3F/v27UNUVNQNb1dQUICMjAycPXsW58+fx+nTp3H+/HmMGzfOqqfREhE5ApZ5IqKrZGZmYtq0adDr9ZW2fNTpdPD29oafnx8aNWqEwMBABAQEoF69evD390fDhg0xd+5cZGVlwcfHR+EzsD9lZWVITExESUkJRARxcXE4e/YsMjIycO7cORw/ftzy3/n5+ZW+V6/XQ0QQGRmpKD0Rkf3inHkioqukpaUhODj4prczGAwwGo0QERQXF6P81+nzzz+PTz75BCaTqbqjasbp06fRt29fHD58GEDlsSspKalyn/yKgoODsX//fltEJSLSksXcmpKI6CqtWrVCnTp1bnq70tJSXLlyBUVFRdDpdOjUqRN8fHzQqFEjFvmr1K9fH+7u7rj//vvh5eUFvV5vGbubFXknJycMGjTIRkmJiLSFZZ6I6Co6nQ4DBw6E0XjzmYgmkwlOTk549913sWXLFowePRrz58+3QUpt2bdvH3bu3Ilp06YhLS0No0ePBvD3FJqbKS4uRr9+/ao5IRGRNrHMExH9r7y8PKxatQqvv/46tm7dipvNQtTr9ejUqRN27dqFqKgoGAwGhIeHY+/evUhNTbVRam2Ii4tD48aN0atXL/j7+2P+/PlYvnw5fH19b/ouhoggNjYWH3/8MXbs2HHTK/lERDUJyzwR1VgFBQVYu3Yt/v3vf6N3797w9vbG4MGDsXTpUnTu3BmlpaVVfp/JZIKzszPeffddbNq0CUFBQZavde/eHS1atMC8efNs9TTsXmlpKeLj4zF+/PhKV+LDwsKQlpaGCRMmQKfTXfcqff369eHm5oZ33nkHHTt2hL+/Px599FF89tln2LNnz01fdBEROTIugCWiGqOkpAQ7d+7E6tWrsXr1amzYsAGFhYWoV68eevXqhdDQUAwePBiNGzeGiMDX1xdZWVmV7kOn02HAgAH45ptv0KhRI8vnc3NzcfHiRWRlZeHzzz/Hd999h+joaIwdO/aW5t87qpSUFHzzzTeIjY3FggUL0K5dO3h7e8PHxwdOTk6W2/3666946qmncP78eZSUlFg+bzKZEBERgRkzZgAAjhw5Yvn5rVmzBllZWfDz80PXrl0tP8NOnTrZ7aFZRERWtphlnogcVmlpKXbs2GEp7uvWrUNOTk6l8j5o0CA0bdq0yu8fMmQIkpKSUFZWBoPBAL1ej5CQEPj4+OCvv/5CZmYmsrOzkZOTU6mAlnv00Ufxww8/VPfTtGuFhYUICAjApUuXrvlarVq14OnpCW9vb/j6+sLHxwfHjh3D7t27odPpUFZWBp1Oh/nz52Ps2LHXfH9paSkOHDiAjRs3YvXq1UhKSkJ2djbq1q2LPn36IDQ0FD179sQ999xji6dKRKQCyzwROY7y8r5hwwZs3LjxrsvdG2+8gffeew/A3/PjDQYDysrKrjv9Bvj7yr3RaMTw4cOxZs0aHDt2DO7u7lZ5flq0dOlSjBgx4pZf2BiNRuh0Ost+9ABw9uxZBAQE3PR7r/fiLSAgAL17977pizciIg1imScibas47WL16tW4ePEi/P390bdvX/Ts2RO9evW642kXe/bsQadOnVBaWnpLiy4NBgOcnJyQmJiIjh07olmzZpg8eTKmT59+J09N84qKitChQwe0adMG33//PaKjo/H666/f8vfrdDoEBATgzJkzd/T4tzOtiohIo1jmiUhb0tPTsWbNGqxZswbJycnIysqCr68v+vbti379+qF///5Wm1ZRflLpk08+edPbGo1GeHp6YvXq1ejYsSMA4OOPP8bUqVOxa9euSotka4r3338f//nPf7Bnzx40a9YMADBjxgzLSa43++fH1dUVe/bssdqV9MLCQvz5559ITk7G2rVrsWXLFhQVFSEoKAgDBw7EwIED0a9fP57eS0RawjJPRPbt7NmzlvK+Zs0anDx5Eh4eHujbty8GDhyIAQMGoG3btre0X/mdKC0tRZMmTXDq1Knr3sZkMsHf3x/Jyclo2bJlpe+977774Ofnh1WrVlVLPnt18uRJtGnTBlOnTsW0adMqfS0+Ph5PPPEEROS673iYTCb885//xIcfflhtGfPz87Fx40asXbsWa9aswfbt2wEAHTt2tJT7Xr16oVatWtWWgYjoLrHME5F9ycvLwx9//GGZGrF9+3YYDAa0b98eoaGhCA0NRZ8+fSrthFJdiouLsWTJEvz73//G4cOHq5wrbzKZ0LRpU6xduxb169e/5usbN25Enz598OWXXyIiIqLaM9uDsrIy3H///Th16hR27twJZ2fna26zevVqDBs2DMXFxVWOq06nw/DhwzFt2jR06tTJFrGRm5uLP//80y7+7hER3SKWeSJS6+p5zevWrUNRURGaNWtmKVCDBw+Gp6enzTJduHABX3/9Nb744gtkZGRg6NChWLVqFa5cuVLpdkajEe3bt8evv/56w+0n33zzTXz44Yf45ptv0L59e7Rp06a6n4Iya9aswcqVKzFjxgxs2rTphkV8y5YtuP/++5Gfn4/i4mLL541GIzp27IgrV65g165d6NWrFyIjIzF8+PBbOpXXWs6dO4d169Zh9erV+OWXX3DixAm4ubmhe/fulr+b3AaTiBRjmSci2yorK0Nqaqplx5lff/0VOTk5lvLes2dPhIaGIjAw0ObZ0tLS8MUXX2DWrFkwGo2YOHEiXnrpJTRp0gTPPvss4uLiLKXTYDBg0KBB+OGHH+Dq6lrl/RUXF+P333/HsmXLMHPmTAB/Tz/x9/e32XOytU8//RSTJ09Gw4YN8cILL+Chhx664YuXffv2YcCAAcjKyqpU6JOSkhAaGooNGzbg008/xdKlS+Hn54eIiAhMmjQJvr6+tng6ldxosTUX0xKRIizzRFT9KpagtWvXIjMzE35+fujXr5/y7QLLysrw888/49NPP8WaNWvQokULvPjii3jmmWfg5uZmud3u3btx7733Avh7m8qxY8dizpw511wpzsnJwS+//IJly5Zh+fLlyM3NhU6ng06ng5eXF0JCQvDTTz855FSNvXv3ok+fPvD398eBAweg0+kgImjcuDFGjx6Nhx56CD169LhmfcPRo0cxYMAAnD59GiUlJWjRogXS0tIqXfE+evQoYmNjMXPmTOTl5WH06NF49dVXLT8TW6u4DWbFnXIqvqM0YMCAGn1gGBHZxGIIEZGVnTt3ThISEiQiIkKaNGkiAMTNzU1CQ0MlOjpaUlJSpKysTGnGS5cuSUxMjDRp0kT0er2EhoZKYmLiDXP17t1bAMhLL71U6Xbnz5+XuLg4eeCBB8RkMolOpxOTySQABIAYjUZ54403ZNeuXeLl5SWPPPKIlJSU2OJp2sypU6ekUaNG0q1bN8nKypLg4GAxGo2WMSgfD09PTxk/frwkJCTI5cuXLd+fkZEh7dq1EwDy5ZdfXvdxCgoKJC4uTtq2bSsApGfPnpKQkCDFxcW2eJrXlZ+fL+vXr5fo6GgJDQ0Vk8kker1eQkJCJDIyUhISEiQnJ0dpRiJySAm8Mk9Edy0/Px/r1q3DqlWrsGrVKuzduxdOTk7o1q2bZVeQrl272nS+8/UcPHgQM2bMwOzZs6HX6/H444/jX//6F1q3bn3T7122bBkOHDiAqVOn4siRI1i+fDni4+OxdetWy9XmqxdzGo1GNG3aFLt27YKLiwvWrl2LoUOHYsKECfjyyy+rbRceWzpz5gxCQ0NhNBqxbt06eHl5ITU1FZ07d77uouGSkhKYTCYMHDgQjzzyCB566CG4urpi4sSJ+Pbbbyu9K3I95VNwlixZgkaNGuG5557Ds88+axdbS168eBG//fYb1qxZg9WrVyMtLQ3Ozs7o1asXBg0ahPvvvx8dOnTgfHsiulu8Mk9Et6+srEy2b98u77//vgwcOFCcnZ0FgLRr105efvllWblypeTm5qqOaVFaWipJSUkSFhYmOp1OmjdvLtHR0ZKVlXXb97Vy5UrLuw3lV+Hxv1efq/owGAySmppa6T6WLVsmzs7OMmrUKCksLLTW01Ri//790rhxY2nTpo2cPn260tfMZrMYDIabjo9erxe9Xi99+/aVzMzM285w6NAhiYqKEi8vL3FxcZHw8HDZs2ePtZ6iVZw8eVLmzp0rY8eOFX9/fwEg/v7+Mm7cOJk7d66cOXNGdUQi0qYElnkiuiUVp87Ur19fAIivr6+MGjVKYmNj5cSJE6ojXiMnJ0diY2OldevWlaZk3M0Ul9zcXGnZsuVNS2p5UZ0+fXqV95OcnCy1a9eWAQMGSHZ29h3nUWnTpk3i6+sr3bt3r7KEFxUVyb333ltpytH1PnQ6nbz22mt3lac6ft7V5fDhwxITEyOhoaHi4uIiAKRZs2YSERHBKTlEdDs4zYaIqlZQUICNGzdWued2WFgYhg0bho4dO9rlNJHDhw9j5syZ+Prrr1FQUIBRo0ZhypQpaNu2rVXuf+/evQgJCblmq8qKjEYjWrdujW3btsFkMlV5mx07dmDo0KFwdXXFokWLEBISYpV81U1E8Omnn2LKlCm4//77sWjRouvu6LN//360b9++0k41VzOZTGjXrh3++OMPqywMLisrw9q1a/HJJ5/g559/RrNmzfDss8/iueeeg5eX113fv7VV9f9a+ZQcboFJRDfBaTZE9H8OHz4ssbGxMmrUKHF3d7/mauGlS5dUR7yh9evXy6hRo8RgMEhgYKCYzWa5cOFCtTxWbGzsDafYmEymW5rqkZGRIaGhoeLs7Cyffvqp8oXBN5OVlSXDhw8Xg8Egb7/9tpSWlt70e95//33R6/XXvSLv7u4uR44cqZa8aWlpEhkZKW5ubuLh4SERERGyb9++anksa8nIyLC8CxYYGCgAxM/Pz/Iu2MmTJ1VHJCL7wWk2RDXZ+fPnLaWhQYMG10ydOX78uOqIN6Vqd5OLFy9KixYtKu3YUv6h1+vlww8/vOX7Kisrk5iYGDGZTNK7d2/ZtWtXNSa/cwkJCVK3bl2pW7eurFq16pa/r7S0VHr06HHd6TbPPPNMtb+Iyc7Ovu3di+zFnj17LLvklK9PKX+RnZiYKAUFBaojEpE6LPNENUlxcbGsX79eoqKiJCQkRHQ6nRiNRgkJCZGoqChZv379LV1ptQenT58Ws9ksderUEWdnZwkPD5cdO3bY5LE3b94sTZs2lYCAAKlfv/41WzB27dr1juZpp6SkSJcuXcRkMsmrr756Rwt0q0Nqaqr06dNH9Hq9PP/883eU6/Dhw5a54eUfRqNRevToIQaDQYYPH26T51taWiqJiYkSGhoqAKRVq1YSExNjVwu2byQvL0+SkpIq/T9cq1Ytu9r2lYhsimWeyNFVnDrj4eFxzdQZrS2+LJ9KYzQaJSAgQMxms/z11182ecQbw1wAACAASURBVOzyK+hOTk4ycOBAOXv2rKSkpFS64uzi4iIHDx6848coLS2VL7/8Unx8fMTT01OmTp0q27dvl1OnTlnxmdzc5s2b5Y8//pBhw4aJTqeTLl26yNatW+/qPj/77DPLdBuTySTBwcGSn58vv//+u9SvX18aNmwoGzZssNIzuLnU1FSJiIgQV1dXqV27tkRGRsrRo0dt9vjWUHFKTr169Sy75JS/u2brvzdEZHMs80SOJjs7WxYvXixPP/20Zb5tnTp1ZMyYMTJr1iy73HXmZgoLCyUuLk7at28vACQkJETi4uKkqKjIZhn++usvefDBB8VoNIrZbK70DkZMTIxle8UvvvjCKo+Xk5Mj7733nvj6+oper5fHH39cfv/992q/6nr58mWJi4uTFi1aCADp1q2bLF++3CqPW1ZWJn369LFcTU5LS7N87UbjW93Onz8v0dHR0rBhQ9Hr9RIWFiZJSUk2e3xrKS0tlS1btsg777wjffv2rXRw1bRp02TDhg12ubMPEd0VlnkirSsrK5PU1FR57733pE+fPmI0GsVgMEiPHj3kv//9r2zZskUzU2eudvbsWTGbzeLr6ytOTk4yatQo2bhxo81z3OzKcVlZmYSFhUn//v2tWrYzMzMlODhYAEiHDh0EgDRp0kSmTp0qa9assdpc6bNnz0p8fLyMGzdO3NzcxMnJSe677z4BIP/617+s8hjljh07Jh4eHvLtt99e87Wq3vmwpaKiIklISLBMwenQoYPExsZKfn6+TXNYy+XLlyUxMVFeeOEFadasmQAQHx8fGTNmjMyZM0cyMjJURySiu8cyT6RFubm5kpiYKBEREdKwYcNKu13ExcXd0cE79mTz5s0yduxYMZlMUrduXTGbzTYvdiKVF6Y+/PDDNxzXzMxMq+4ykpOTI506dRIA4uHhISIiu3fvlilTpliumru4uMiAAQPktddek1mzZsm6devk7Nmz133HIjc3Vw4dOiQrVqyQjz76SJ577jm55557LNNe+vTpI1988YVkZmZKUlKSZerQv//9b6s9LxGRnTt33vDrW7dulebNm4u/v7/8+uuvVn3sW/Xnn3/K448/bvk7+NZbb2m+/JZPuQsLC7OsX2jTpo1ERUVJUlKSTd/pIiKrYZkn0oqKh8w4OTmJwWDQ5MLV67l6YaLqq6Lnzp2TwYMHi7Ozs8TExNh0UWF+fr706tXLsrC2adOm19zmxIkTMnfuXHniiSekU6dO4ubmVmlxqbOzs/j4+EijRo3E29v7mm00AwICpF+/fvLaa6/JihUr5PLly5Xuf+fOnZVuHx0dbaunLyIily5dkscee0x0Op1ERkYqK5pVvTv0xx9/KMliTfn5+ZaFtOWHbLm7u0tYWBi3vyTSFh4aRWSv8vLysHbtWvz0009YsWIFTp06BT8/P/Tr1w9hYWEICwuDj4+P6ph37dKlS5g7dy4+/vhjnDx5EkOHDsXkyZMxcOBAZYfkrFmzBuHh4XB2dsbChQvRtWtXmz12UVERHn74YaxevRolJSUAgB49emDjxo03/d6TJ0/iyJEjuHjxInJzc5Gbm4u8vDx4eHjAy8sL7u7u8PX1RVBQEGrXrn3D+8rIyEC9evUsf9bpdPj888/x/PPP390TvE3z5s3D888/j86dO+O7775D/fr1bfr45a5cuYJFixbhgw8+wO7duxESEoLIyEiMHTsWRqNRSSZrOnLkiOXQqpUrVyI3Nxdt2rTBsGHDEBoair59+1738DMiUoqHRhHZE0e/+l5Renr6NYf5HDhwQGmm4uJiMZvNotfrZcSIEXLx4kWbPn5JSYmMHDmy0laXOp1ORo4cadMc5Vmuvpqv0+lk1qxZNs+yd+9eadu2rfj6+spPP/1k88e/WsXDyZo2bSrR0dGan9pWEa/aE2kKp9kQqVRx7nv5oU2ONPf9amVlZZKUlCRhYWGi0+mkefPmEh0dbfPSXJUTJ05Iz549xcXFRWJiYmz++GVlZfLkk0+KwWC45iTZF1980eZ5RERq165d5YFYixYtsnmW/Px8efbZZy3Tbq5cuWLzDFc7dOiQREVFiZeXl7i7u0tERITs3btXdSyrq+pk6Ipz7e3hZ0FUg7HME9laxdMcHf3qe7nyU1rLF1uWn9JqL9vk/fjjj+Lj4yPBwcE3XZxZHcrKyuQf//iHZQ92XDX3/a233rJ5JhGR5s2bX5On/KAxVVfI4+LixN3dXe677z45dOiQkgxXy8nJkdjYWAkKCtLc6bK3q6qr9m5ubrxqT6QOyzxRdbt48aIkJCTIk08+aTnUJSAgQCZOnCiLFi2ym1M+q8OZM2euOaV1165dqmNZFBYWSmRkpOh0OgkPD1d2CmhUVNQ1U1oqXpm31t71t6tnz55VZtLr9eLk5CTJyclKch04cEDat28vnp6esnDhQiUZqlJxEbdOp7OcLpuXl6c6WrVJS0uTjz/+WAYPHiwuLi6i0+mkY8eO8vrrr8u6deukuLhYdUQiR8cyT1QdKm4BV1OuvleUkpIi4eHhSk5pvVVHjx6Vbt26iYeHh8yfP19ZjunTp1dZmCt+/PDDD0qyjRo16rovMvR6vbi6usqWLVuUZCsoKJDIyEgBIOHh4XZXmMtPl61Vq5bldFktHth2O6q6au/t7W2ZNujIFy6IFGKZJ7KG4uJiSU5OlldeeUVatWplOZxl7NixsmDBghrxj9iVK1ckISFBunfvLgCkU6dONj+l9VZ9//334uXlJZ06dZL09HRlOT7++OObFnkAsm7dOiX5Jk2aJE5OTtfNZTAYxNPTU+m7LUuWLBFvb2+55557ZM+ePcpyXM+5c+ckOjpaGjRoIAaDQcLCwqo8eMwRpaWlyYcffij9+/cXo9EoRqNR+vfvLx988EGl03+J6K6wzBPdqaysLElISJDw8HDx9vYWANKsWTOJjIysUQewnD9/XqKjo6Vhw4ai1+slLCxMkpKSVMeqUsWruREREVJYWKgsy549e8TV1bXKefJXf6ja5Wf69Oni7Ox83Vzl2bt166Z0OsWxY8eke/fu4u7uXuXJsvag/MVut27dBICEhITY7Yvd6lBxsX/dunWv+X3JRbREd4xlnuh2VNw60mQyicFgkJ49e0p0dLTs27dPdTybOnDggERGRoqrq6tlGsGxY8dUx7qu/fv3y7333iu1a9dWshtLVbKzsyUmJkb8/f1Fr9dft9ir2u3nq6++EpPJVOU8fgDSrl07iYuLs4t50RW3FVW5/uFWVJyGVq9ePTGbzXLhwgXVsWympKREUlJSxGw2S0hIyDWLaLV+0i6RjbHME91ISUmJrF+/XqKioqRNmzaW6TPlc0DtYUtFWyotLa20tWTLli0lJibGrouTyN87oLi5uUnnzp3l8OHDquNc48qVKzJ79mzx8PAQAJX2mTcajcp2RVmyZMk1Jb78526v774kJiZKnTp1JDg4WHbs2KE6zg2dPn1azGaz+Pj4WBaI7969W3Usmzty5EiVa4zMZrOkpKSojkdk71jmia6WmZlpmT7j5eVVY6fPVFS+9V7r1q1Fp9NpZuu9nJwcGTdunGVvcnv+2c2dO1dMJpPExcVJv379LNNY/Pz8lGXasGGDZW68s7Oz/OMf/5A333xTXF1d7W5Bc0UnT56U3r17Kzsz4HZdvnxZYmNjLRcM7G3rVluqOB0nICDA8vs3IiJCEhMTOR2H6Fos80QinD5zPYcPH5aoqCjx9vYWFxcXCQ8Pt8tFhlXZtm2btGzZUvz8/GTFihWq49xUhw4dZNy4cZY/p6amytixY6Vz587KMh08eFD8/f3l7bfftkwDycvLkzp16sjbb7+tLNetKCkpEbPZLAaDQR599FFNLEK/+p2vFi1aaOKdr+rC6ThEt4RlnmqmitNnyrdQq1OnTo2dPnO18uPqtTqnNy4uTmrVqiX9+vWT06dPq45zU6tWrRIAVW7zePnyZQWJ/lZUVFTlIuGpU6eKv7+/FBQUKEh1e9auXSv16tWTxo0by6ZNm1THuWVpaWmWNSmenp4SGRkpR48eVR1LKU7HIaoSyzzVHJw+c2Plu2107dq10m4b9rC48VZlZ2fL6NGjxWAwiNls1sw0hcGDB0v//v1Vx7hlp0+fFicnJ5k9e7bqKLfk/PnzMmTIEDEajWI2mzV1zkP5IulGjRrZ/W5RtlTVdJymTZtyOg7VRCzz5NjS09Pl/fffl969e4vBYBAnJycZNGiQfPLJJ3a5EFKF8+fPy/Tp0yUgIEBMJpOMGzdOtm7dqjrWbduyZYs0a9ZM6tatK6tWrVId55bt3r1bdDqd/PTTT6qj3JYJEyZIcHCwZopxWVmZxMTEiMlkkmHDhmnqnSaRv98liY+Pl86dOwsA6dq1qyxcuFBTL7arS1UbFXh5ecmYMWPku+++k+zsbNURiaoTyzw5nm3btsmbb74pbdu2FQDi6+srEyZMkMWLF8ulS5dUx7Mb6enp12wtqcUTKstLmpOTk4SGhsrZs2dVR7otEydOlKCgIM2U4nLlL0K0sB6hos2bN0vTpk2lQYMGyg7jultVbW2ZmZmpOpbduHoNlJOTkwwePFi++uorzf1+ILoFLPOkfaWlpZarMi1bthQA0rBhQ8vbrTV9+szV1q9f7zAL7M6fPy9Dhw7V5PQJEZGMjAxxcXGRmTNnqo5yRwYNGiQDBw5UHeO2ZWdny8iRIzX796bckSNHLAvU3d3dJSIiokYv2K/KxYsXLdMrPTw8RK/XW+bZqzqMjcjKWOZJmwoLCyUpKUkiIyOlXr16lea/r1+/3u63TLS1wsJCiYuLs7xb4Qhb3yUnJ0tgYKA0atRINm7cqDrOHXnjjTfE399f8vPzVUe5I7/88osAkO3bt6uOctsqvqPTv39/OXPmjOpId6x869igoCDR6/Wa2TrW1goKCq45hbZNmzYSFRXFfzdIy1jmSTvy8vIkMTFRwsPDxdPT0/KL2Gw2y969e1XHs0sZGRliNpvF19dXnJycJDw8XHbu3Kk61l2puOXgI488ooktB6tSvsXjf/7zH9VR7kr79u1l/PjxqmPcsZSUFGnRooX4+fnJypUrVce5K6WlpZKYmCihoaECQDp06CCxsbGa2HXI1srn2UdGRkqDBg0EgDRu3NiyIQLXIpCGsMyTfbtw4YLExcVJWFiYODs7W/Z/j4mJkZMnT6qOZ7dSU1MlIiJCXFxcxN/fX6KiouTUqVOqY921jIwMGTRokDg7O2viMKAb+eyzz6RWrVp2ffjSrZgzZ46YTCZNrrcol5OTI48//rgmDhe7Vdu2bZPw8HAxmUxSt25dMZvNmv+7Vp327NkjZrPZsoC2Tp06Eh4eLomJiVVuz0pkR1jmyf4cO3bMsnjJaDSKi4sLDwm5BVcfONOqVSuJiYnR7BSOqyUlJUlAQIC0atVKUlNTVce5K6WlpdKiRQv5xz/+oTrKXbty5YrUq1dPpkyZojrKXYuLixNXV1fp0qWLHDlyRHUcqzh79qyYzWapU6eOODs7S3h4uOzevVt1LLtWvoC2Z8+eotPpxNXVVcLCwiQuLo6bKJA9Ypkn+7Bnzx6Jjo62/PL08vKyHOCUk5OjOp5dKz8KvnXr1qLT6RxuvmxxcbGYzWbR6/USHh6u9BAla/n+++9Fp9M5zGLFd955Rzw9PR1iC8B9+/ZJu3btpHbt2rJ48WLVcaymoKBA4uLiLFeee/bs6VC/J6rLiRMnLAdVmUwmcXFxkdDQUImJieHOOGQvWOZJnfK3NYODgy1bSJa/rckDP27uzJkzYjabxcfHx3LFbc+ePapjWdXx48elR48eUqtWLc1Pq6moe/fu8vDDD6uOYTVZWVni7u4uH3/8seooVpGfny+RkZGWaTeO9Pvoeu/g5eXlqY5m9zIzMyUuLk5GjRol7u7uotfrpWfPnhIdHS3p6emq41HNxTJPtlNxwVH9+vUtJ/aV70Cj1e3hbK18j2mTySQBAQEOOxd26dKl4uPjI61bt5Zdu3apjmM1W7ZsEQCa3eP8eiZNmiSNGzd2qIWD8+bNE3d3dwkJCZGDBw+qjmN1O3bskIiICKlVq5b4+fk5zNoaW8jPz7dsyFB+onj5hgwpKSmq41HNwjJP1SsvL09++OEHGT9+vHh7ewsA6dixo0yfPp3zNm/D1btUdOzY0WF3qSgsLLRcFQ0PD3e4K4YjRoyQ++67T3UMqzty5IgYDAZZtGiR6ihWlZaWJh06dBBPT0+Jj49XHadanDt3TqKjoyUwMFCcnJxk1KhRsnnzZtWxNOPKlSuycuXKSlteBgUFydSpU2Xz5s2cykTVjWWerK/iFYvyQzrK34pMS0tTHU9TcnJyJCYmRpo0aSJ6vV7CwsIkKSlJdaxq4+jFqbzwLly4UHWUauGoL1TKX2ACcMgXmOXKz6No166dw5xHYWulpaWSkpIiZrNZgoKCLIcY8gwUqkYs82Qdly9floULF8qIESPE1dVVDAaDDBw4UL766is5d+6c6niaU36yo5eXl+Vkx/3796uOVa0WL14stWvXdtgpDSKOORWlIkedQlRu6dKl4u3tLW3atHH4dxYrnhTdvHlzTZ8UrVJqaqpMmzZNWrVqZSn2L730kmzcuJHFnqyFZZ7uXMUr8O7u7pX2gOcq/ztTPh/eaDRKvXr1xGw2S2ZmpupY1cqRFxtW5GiLRK/H0Rb3Xs1RF2VfT3p6ukRGRoqrq6t4enpKZGSkHD9+XHUsTSrf9KF169YCQBo0aGA5pIrvftBdYJmn21PxFFY3N7dKBZ57wN+ZoqIiSUhIkG7dugkACQkJkbi4OIe9eluRo24DWBVH2r7xRhxt282qOOJ2qTeTnZ0tMTEx0qBBAzEYDBIWFiYbN25UHUuzrj6kquJubjXhdz9ZFcs83RwLfPUo/8exYcOGlvnwGzZsUB3LZhzxgJ7rKSoqkgYNGjjEwUo340gHYt2MIx1kdquuXLlSYy8+VBcWe7pLLPNUtcuXL0t8fLw89NBD4uzsLCaTSYYMGSKzZ892+Gkf1S09PV2ef/55cXV1FS8vL5kyZYqcOHFCdSybycnJkccff9wyraaoqEh1pGo3Z84cMZlMNebn/Nlnn0mtWrUccsvUq2VkZMigQYPE2dm5Rky7qSg5OVkefvhh0ev10qxZM/nkk09qxLsU1enqYl+vXj355z//KRs2bOAce7oelnn6PwUFBbJkyRIZPXq0uLq6itFolCFDhsg333zDAm8F69evl0ceeUT0er00b95cPvvssxr3D19KSoq0aNFC/Pz8ZOXKlarj2Ez79u1l/PjxqmPYTF5entSpU0emT5+uOopNlJSUiNlsFoPBII888ohkZWWpjmRTBw8elBdffFHc3NzEy8uL+9Vbyd69e+Wtt96yzLFv1KiRvPbaa7Jt2zbV0ci+sMzXdOUHOUVEREjt2rUt20hyCo11lO8P36NHjxr9lnRZWZnExMSIk5OT9O/fX86cOaM6ks388ssvAkC2b9+uOopNvfHGG+Lv7y/5+fmqo9hMcnKyBAYGSqNGjWrkfPJLly5Z5tWbTCbuV29F5VfsW7ZsKQCkcePGEhkZyWJPIizzNVNpaanlJFZ/f3/LyXXR0dFy+vRp1fEcwuXLlyU2NlZatWpVI+fDV5SdnS0jR44Ug8EgZrO5xu3aMGjQIBk4cKDqGDaXkZEhLi4uMnPmTNVRbOr8+fMydOhQMRqNYjaba+TJ1leuXLlmv/rExEROE7GS8mLfrFmzSifP8hyXGotlvibZs2ePREVFSb169Sr9AnDUPb1VOHv2rJjNZvHx8bHsD3/gwAHVsZTZvHmzNG3aVBo0aOCwe4/fyO7du0Wn08mKFStUR1Fi4sSJEhQUVOMKbcV3okJDQ2v0Vr0V96tv2bKlxMTE1Kh3a6pTxQtzAQEBlf5dP3z4sOp4ZDss845u+/btMmXKFGncuLEAkNatW8tbb71VowtmddixY4dERESIi4uL1K1bV8xms1y4cEF1LGXKy4zJZJJhw4bV2LGYMGFCjSyz5cpfzPz000+qoyixZcsWadasmdStW1dWrVqlOo5SO3futPyO9Pf3F7PZXCMWSNtKcXGx/Prrr/LUU0+Jt7e36PV66d27t8yYMYMHNzo+lnlHdOLECYmJiZGOHTtaFs2UHyVN1lXxqtO9994rsbGxUlBQoDqWUufPn5chQ4bU6GkGIiKnT58WJycnmTVrluooSg0ePFj69++vOoYy2dnZMnr06Bo7zexqGRkZYjabpU6dOuLs7Czh4eEOfSaBCiUlJZKUlCTh4eHi4eFh2U46NjZWcnJyVMcj62OZdxQXLlyQL774Qnr27Ck6nU78/Pxk0qRJsmnTJtXRHE75fNB77rmH80GvsnbtWqlXr540bty4xv/dmzp1qvj7+9f4F3erVq0SADV+oV5cXJzUqlVL+vXrx7VJ8vfuaXFxcRIcHGxZV5SUlKQ6lsPJy8uTBQsWSFhYmJhMJnFzc5Nx48bJihUratxGDA6MZV7LCgoKJDExUUaNGiVOTk7i4uIiYWFhkpCQUCP27ra1v/76S6Kjo6V+/fri5OQk4eHhsmvXLtWx7ELFrfmGDx9e47bmu1r51oxvv/226ih2oUOHDjJu3DjVMZTbtm2btGzZUvz8/GrsOoqrle/4FRoaKgCkU6dONXLHL1vIysqSuLg4CQ0NFZ1OJz4+PhIRESHr16/nxShtY5nXmvIFLxEREeLp6cm3z2zg0KFDEhkZKa6urlK7dm2JjIyUkydPqo5lN06ePCm9e/cWFxeXGndozvV88skn4urqyjnB/2vu3LliMpnk+PHjqqMol5OTI+PGjatRh6bdqpSUFAkPDxej0ShNmjSR6OhouXjxoupYDql8Om6HDh0s03GjoqK4nk6bWOa1oqqdaKKjo2v0LgnVbf369TJq1CgxGAzSrFkziYmJkdzcXNWx7MqqVaukbt26EhQUJDt27FAdxy6UlJRI8+bN5YUXXlAdxW4UFRVJgwYN5NVXX1UdxW7ExcWJm5ubdO7cmTuPXOXIkSMSFRUltWvXFg8PD4mMjOQLwWpUvtVlkyZN2C+0iWXenh07dkzefvttCQ4OFgDSqlUreeutt7iVZDUqf8u3e/fulQ55qumL1q5WXFwsZrNZ9Hq9hIeH17iTbG9k8eLFotfrZf/+/aqj2JX33ntPPD09JTs7W3UUu7F//3659957pXbt2rJo0SLVcexO+SFUDRs2tBxC9eeff6qO5bDKF85OnDjR8s7/kCFDZP78+ZKXl6c6Hl0fy7y9uXz5ssydO1f69+8ver1e/P39JTIykqfoVbOcnByJiYmRxo0bWxZj1cQTHG/FsWPHpHv37lKrVi35+uuvVcexO927d5dHHnlEdQy7k5WVJe7u7vLRRx+pjmJXCgoKJDIyUgBIRESEFBYWqo5kd65cuSIJCQnSuXNny6YDCQkJvMhSjfLz8yUhIUEeeughMZlM4unpKU899ZT8/vvvnF9vf1jm7UHFefAeHh7i5OTEhaw2Un7Ik7e3Nw95ugVLliwRb29vadOmjezevVt1HLuzYcMGAcBtYK8jMjJSGjRowN9rVfj+++/Fy8tLOnXqJOnp6arj2K2K2wG3aNGCh1DZQFZWlsTGxkrPnj0FgDRs2FCioqI4S8B+sMyrdPz4cYmOjq50JHN0dLScP39edTSHl5qaKuHh4WIymSQgIEDMZrNkZmaqjmW3Kl49DA8P51uu1zF8+HDp3Lmz6hh26+jRo2I0GmXBggWqo9ilo0ePSrdu3cTDw0Pmz5+vOo5dS09Pl8jISKlVq5b4+flJVFSUnDlzRnUsh7dv3z4xm82WgyhDQkIkJiamxh4MaCdY5m0tOzu70tZQgYGBEhkZKampqaqjObyysjJJSkqyXNVp3749D3m6BQcOHJD27duLp6cnS9gNHDlyRAwGgyQkJKiOYtdGjhwpISEhqmPYrcLCQomMjBSdTifh4eFcdH8T586dE7PZLL6+vpZDqPbu3as6lsMrLS21HEzl6uoqLi4uMmrUKElMTOS2orbHMm8LFf/Su7m58S+9jRUWFvKQpzsUFxcn7u7uct9998mhQ4dUx7FrL774ojRp0oT/T9/Eli1bBID89ttvqqPYtR9//FF8fHwkODhYdu7cqTqO3Sv/Pd+6dWvR6/USGhoqiYmJqmPVCLxIqVyCTkQEFSQlJWHKlCmgG0tNTb3pbQ4dOoTZs2cjLi4OGRkZ6NmzJyZMmIBRo0ahdu3aVX4Px//W3Mr4Z2Zm4osvvsCMGTNw6dIljB8/Hi+99BLuueee634Px///lJWVIT09HZ6enqhXrx50Op3la7cy/ndCy+N/+fJllJWVXff/bWvS+vifP38eXl5ecHJyqvbHqg62Gv+ioiIcP34cbm5uCAwMrJbH1KIbjX9ZWRl+/PFHfPTRR9iwYQPuu+8+vPbaaxgxYgQMBsMN71fLv39s6Ubjn56ejnnz5mHevHk4efIkunTpgmeeeQaPPfYYPDw8bni/HP9bU8X4LzZe/ZmLFy9ix44dmDp1qm1SacyBAwewbNmy6379ypUrWLp0KWbOnInk5GQEBgbi6aefxoQJE9CiRYub3j/H/8ZuNv4AcOzYMXz88ceYPXs2nJ2d8fzzz2PSpEkICAi46f1z/CsbNGgQTCaT5c+3Mv53g+N/Yxx/tVSMf1lZGUTkpkW0JriV8dfr9Rg+fDiGDx+OzZs344MPPsDjjz+Opk2b4uWXX8aTTz6JWrVqVfm9/Pt/Y7cy/q1atcJ///tfTJ8+HcnJyZgzZw4iIyPx8ssv47HHHsOzzz6LLl26VPm9HP8bu+H4X32tftGiRVLFp+l/XW98Dh48KC+//LL4+vqKwWCQYcOGSWJi4m1vncXxv7Ebjc/OnTstpwfWq1dPzGbzbe9pzfG/seoeH47/jXH81eL4q3Wn43PkyBHLKd6+vr5iNpuroReRCQAAF0pJREFUXLDJ8b+xOx2fzMxMiYmJkbZt2woAad++vXz++efXnFrP8b+xG4xPgr5aX0Y4uLKyMqxcuRIPPvgggoKC8MMPP2Dy5Mk4fvw4EhMTMWzYMF5NsYENGzZg2LBh6NChA3bu3InZs2fj+PHjeOutt2wy5YGIiOxX06ZN8cknn+DYsWN48cUXMWPGDNSvXx9PPPEE0tPTVcdzeD4+Ppg8eTJ2796NTZs2oVOnTnj11VfRoEEDTJ48mT8DK2CZv0Nff/012rVrh6FDh+LSpUtYuHAhDh06hDfffBP169dXHc/hlZWVYfny5ejWrRt69+6Nixcv4scff8SOHTvwxBNPVJoaQkRE5Ofnh7feegvHjx/Hp59+ij///BOtW7fGsGHDsGXLFtXxaoTu3bvjm2++wdmzZzF9+nQsX74cwcHBGDRoEP7880/V8TSLZf4OvfLKK+jVqxd27dqFDRs2YNSoUTAar1mCQNWkRYsWGD58OBo3boytW7dars5XXKRJRER0NTc3N0RERGDv3r2WhZpdu3bFO++8ozpajVG7dm1MnjwZhw4dwqpVq+Di4oJvvvlGdSzNYpm/Q6dPn0ZsbCzatWunOkqNNGTIEKSlpWHRokW47777VMchIiKNMZlMGDduHFJTU/HLL7+grKxMdaQaR6/XIzQ0FMuXL0dMTIzqOJrFS8l3yNPTU3WEGu2LL75QHYGIiByATqfD4MGDcenSJYwZM0Z1nBrL1dVVdQTN4pV5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo1imSciIiIi0iiWeSIiIiIijWKZJyIiIiLSKJZ5IiIiIiKNYpknIiIiItIolnkiIiIiIo3SRJlftmyZ6gg1GsdfLY6/Whx/tTj+anH81eL4q6WV8Tde7wsdO3a0ZY7rKiwsxLFjxxAcHKw6CgAgOzvbJo/D8a8ax18tjr9aHH+1OP5qcfzV4virdaPxv6bMt27dGlOnTq3WQLfj999/x4EDB9CxY0fUrVtXdZxqx/FXi+OvFsdfLY6/Whx/tTj+anH874LYsbKyMmnQoIEAkClTpqiOU+Nw/NXi+KvF8VeL468Wx18tjr9aGhv/BJ2IiLqXEjf2xx9/oEePHgCAgIAAnDlzBjqdTnGqmoPjrxbHXy2Ov1ocf7U4/mpx/NXS2PgvtusFsPHx8XBycgIAZGRkYMOGDYoT1Swcf7U4/mpx/NXi+KvF8VeL46+W1sbfbst8aWkp4uPjUVRUBAAwmUxYsGCB4lQ1B8dfLY6/Whx/tTj+anH81eL4q6XF8bfbMr9mzRpkZWVZ/lxcXIz4+HgUFxcrTFVzcPzV4virxfFXi+OvFsdfLY6/Wlocf7st8/Hx8TCZTJU+d+nSJSQlJSlKVLNw/NXi+KvF8VeL468Wx18tjr9aWhx/uyzzhYWF+P777695FWQymfDdd98pSlVz/P/27jQ2yrKLw/jpYks7dNNYFVARSguuAbGAoigSKSCJmvhBqVExuEFcYuISxWBAJGiCMSZGE3eWgLjFIFGhWhHFSNyiAgUUiCiIINCWAp3peT9BXuwiT/tM//O01+8bhbnvw0U/nNLODP216K9Ffy36a9Ffi/5aUe2fksv8smXL7MCBA80+3tjYaO+8847V19cLpuo+6K9Ffy36a9Ffi/5a9NeKav+UXOYXLFhgGRkZLf7eoUOHbNmyZZ08UfdCfy36a9Ffi/5a9Neiv1ZU+6fcMr9//35btmyZxePxFn8/IyPD5s+f38lTdR/016K/Fv216K9Ffy36a0W5f8ot8++8806rIc3M4vG4LV++/JhnGiM89Neivxb9teivRX8t+mtFuX/KLfPH81VPPB63d999txOm6X7or0V/Lfpr0V+L/lr014py/5Ra5v/66y+rrq62pqam//yzqf4C/lFEfy36a9Ffi/5a9Neiv1bU+6e5u6uHOCIej1ttbe0xH3vvvfds8uTJzb6tkZaWZoWFhZ05XpdHfy36a9Ffi/5a9Neiv1bE+7+VqZ7g/2VmZlpRUdExH4vFYmZmzT6O8NFfi/5a9Neivxb9teivFfX+KfVjNgAAAACOH8s8AAAAEFEs8wAAAEBEscwDAAAAEcUyDwAAAEQUyzwAAAAQUSzzAAAAQESxzAMAAAARxTIPAAAARBTLPAAAABBRLPMAAABARLHMAwAAABHFMg8AAABEFMs8AAAAEFEs8wAAAEBEscwDAAAAEcUyDwAAAEQUyzwAAAAQUSzzAAAAQESxzAMAAAARxTIPAAAARBTLPAAAABBRLPMAAABARLHMAwAAABHFMg8AAABEFMs8AAAAEFEs8wAAAEBEscwDAAAAEcUyDwAAAEQUyzwAAAAQUSzzAAAAQESluburhzAz27x5s3311Ve2bt06q6mpsa1bt9q+ffts3759Vltba8XFxRaLxaywsNDKysqstLTUzjnnHLv00kutoKBAPX7k0V+L/lr016K/Fv216K/VBfq/JVvmE4mErVixwhYvXmwrV660bdu2WXZ2tg0YMMDKysqsb9++VlRUZLFYzHJzc23v3r1WX19vu3fvtpqaGqupqbFt27ZZenq6DRkyxCoqKmzSpElWVlam+OtEDv216K9Ffy36a9Ffi/5aXbD/W+adbMeOHf7II494r1693Mx8xIgR/sQTT/iqVav88OHDgc76+++/fenSpT516lTv3bu3m5kPHz7cX3/99cBndRf016K/Fv216K9Ffy36a3Xh/ks6bZnfvn27T5061XNycvzUU0/16dOne01NTWjnJxIJ//jjj33SpEl+wgkneN++ff2FF17wxsbG0O6IMvpr0V+L/lr016K/Fv21ukH/5C/zjY2NPm/ePM/Pz/czzzzTn3/+eW9oaEjqnb/99pvffffdnp2d7eeff76vWrUqqfelMvpr0V+L/lr016K/Fv21ulH/5C7zGzZs8CFDhnh2drZPnz7dDxw4kMzrmqmpqfGxY8d6WlqaT506Nen/iKmG/lr016K/Fv216K9Ff61u1j95y/zChQs9Ly/PL7roIt+wYUOyrjkuixYt8oKCAh88eHCo31pJZfTXor8W/bXor0V/LfprdcP+yVnm58yZ42bmt99+ux86dCgZVwS2ZcsWHz58uBcVFXX5bzvRX4v+WvTXor8W/bXor9VN+4e7zCcSCb/rrrs8IyPDX3755TCPDkVDQ4Nfc801npub6x988IF6nNDRX4v+WvTXor8W/bXor9XN+4e7zN91113eo0cPf/fdd8M8NlTxeNxvu+02z8rK8o8++kg9Tqjor0V/Lfpr0V+L/lr01+rm/cNb5h9//HHPyMjwpUuXhnVk0jQ1Nfmtt97qubm5vnr1avU4oaC/Fv216K9Ffy36a9Ffi/4hLfMLFy70tLS0lPzWRmsOHz7s48eP91NPPdX//PNP9TgdQn8t+mvRX4v+WvTXor8W/d09jGV+48aNnp+f7/fee28YA3Wq2tpaLysr88svv9zj8bh6nHahvxb9teivRX8t+mvRX4v+R3VsmW9sbPQhQ4b40KFDU+ZZw0F9++23np2d7U899ZR6lMDor0V/Lfpr0V+L/lr016L/MTq2zM+bN8+zs7N9/fr1HR1Eas6cOZ6Tk+O//vqrepRA6K9Ffy36a9Ffi/5a9Nei/zHav8xv377d8/Pz/bHHHuvIACnh0KFDPmjQIJ84caJ6lONGfy36a9Ffi/5a9Neivxb9m2n/Mj9t2jQ/44wzOv0tcpPlk08+cTPzL774Qj3KcaG/Fv216K9Ffy36a9Ffi/7NtG+Z37lzp+fk5Pjzzz/f3otT0siRI338+PHqMf4T/bXor0V/Lfpr0V+L/lr0b1H7lvlHH33UTznllC7zVdERy5cvdzPz77//Xj1Km+ivRX8t+mvRX4v+WvTXon+LlqRbQIlEwl577TWbMmWK5eTkBH14SquoqLDS0lJ79dVX1aO0iv5a9Neivxb9teivRX8t+rcu8DK/cuVK2759u1VWVrbrwlRXWVlpixYtssbGRvUoLaK/Fv216K9Ffy36a9Ffi/6tC7zML1682IYNG2ZlZWWBL/t/77//vi1ZsqRDZyRDZWWl/fXXX/bpp5+qR2kR/bXor0V/Lfpr0V+L/lr0b13gZb6qqsrGjRsX+KJ/mzVrlj388MMdPidsZ511lg0cONCqqqrUo7SI/lr016K/Fv216K9Ffy36ty4zyB/+9ddfbcuWLTZ69OjAF/3btGnTrKGhocPnJMOVV16Zkp/M9Neivxb9teivRX8t+mvR/z8Eebrs/PnzPTs7O6XeOrepqcmbmppCPXPx4sWekZHhDQ0NoZ7bUfTXor8W/bXor0V/Lfpr0b9NwV7NZv369da/f3/LysoK/lXDv9xzzz126623Hv31lClTbNq0afbHH3/YjTfeaGeeeab179/fJk+ebPX19c0e/8MPP9iYMWOssLDQcnNzbdiwYbZ8+fIOz2VmdvbZZ1sikbBNmzaFcl5Y6K9Ffy36a9Ffi/5a9Nei/38Isvpff/31fu211wb7MqMVQ4cO9bPOOuuYX/ft29d79+7tI0eO9AcffNBHjRrlZubXXXfdMY/99NNPvUePHt67d2+///77ffLkyV5QUOCZmZm+evXqDs928OBBz8jI8KVLl3b4rDDRX4v+WvTXor8W/bXor0X/NgV706jy8nJ/4IEHgk3WipZimpk/9NBDR79tkUgkfMiQIV5QUHD0zyUSCb/gggu8oKDAN27cePTj69at87S0NJ80aVIo8/Xp08efeeaZUM4KC/216K9Ffy36a9Ffi/5a9G9TsB+z2b9/v+Xn5wf7r/8AcnJybMaMGZaWlmZmZunp6XbJJZfYvn377Pfffzczs++++85++OEHu+aaa6ykpOToYwcOHGjPPfeclZeXhzJLfn6+1dbWhnJWWOivRX8t+mvRX4v+WvTXon/bAr2aTV1dnfXs2TPQBUEUFxdbjx49jvlYUVHR0bvN7OjPEZ133nnNHj9t2rTQZsnLy0u5T2b6a9Ffi/5a9Neivxb9tejftkD/M3/o0CHLzs4OdEEQbb09r7ubmdmuXbvMzKx3795Jm8PMrEePHnbw4MGk3hEU/bXor0V/Lfpr0V+L/lr0b1ugZT4Wi7X4zN7O1LdvXzMz+/rrr5v93htvvGGvvfZaKPfU1dVZXl5eKGeFhf5a9Neivxb9teivRX8t+rct0DKfl5d39NsNKhdddJHl5OQ0e1H9X375xW655Rarrq4O5Z79+/en3Ccz/bXor0V/Lfpr0V+L/lr0b1ugZf7EE0+0v//+O9AFYTvllFPsvvvusx9//NHuvPNOW7t2rb3xxht2ww03WGZmpt15552h3LN79+6jPy+VKuivRX8t+mvRX4v+WvTXon/bAj0BtqSkxGpqagJdkAwzZ840d7enn37aXnzxRTMzO+2002zBggU2bNiwDp+/Z88e27Nnjw0YMKDDZ4WJ/lr016K/Fv216K9Ffy36/4cgL2Q5d+5c79OnT5CHJFVdXZ1/+eWX/tNPP4X6Fr+rV692M/OtW7eGdmYY6K9Ffy36a9Ffi/5a9Neif5uWBPqf+XPPPde2b99uu3btspNPPjnYVw1JEIvFbMSIEaGf+/3331t+fr716dMn9LM7gv5a9Neivxb9teivRX8t+rct0M/Mjxw50jIyMuyzzz4LdEnUVFVV2ahRoyw9PVCepKO/Fv216K9Ffy36a9Ffi/5tC/xqNhdeeGGzZ/J2JU1NTfbZZ5/ZFVdcoR6lGfpr0V+L/lr016K/Fv216N+2wF96VVRU2Pvvv2+JRCLwZVFQXV1tu3fvtnHjxqlHaRH9teivRX8t+mvRX4v+WvRvXeBlvrKy0nbs2GErVqwIfFkUvPnmmzZ06FAbOHCgepQW0V+L/lr016K/Fv216K9F/9YFXuZLSkrs4osvDu2drlJJXV2dvf3221ZZWakepVX016K/Fv216K9Ffy36a9G/De156Zz58+d7Zmamb968uT0PT1lz5871vLw83717t3qUNtFfi/5a9Neivxb9teivRf8WLWnXMh+Px33AgAF+xx13tOfhKamhocFPO+00f+ihh9Sj/Cf6a9Ffi/5a9Neivxb9tejfovYt8+7uL730kmdlZfn69evbe0RKmT17tufm5vqOHTvUoxwX+mvRX4v+WvTXor8W/bXo30z7l/l4PO6DBw/2MWPGtPeIlLFt2zaPxWI+e/Zs9SjHjf5a9Neivxb9teivRX8t+jfT/mXe3f3LL7/09PR0f/PNNztyjFRTU5NfffXVXlZWFupb8nYG+mvRX4v+WvTXor8W/bXof4yOLfPu7vfcc4/n5+d7TU1NR4+SePbZZz0zM9M///xz9SjtQn8t+mvRX4v+WvTXor8W/Y/q+DJ/8OBBv/DCC33w4MF+4MCBjh7XqdasWeNZWVn+5JNPqkdpN/pr0V+L/lr016K/Fv216H9Ux5d5d/dNmzb5SSed5BMnTvTGxsYwjky6jRs3enFxsU+YMMETiYR6nA6hvxb9teivRX8t+mvRX4v+7h7WMu/u/tVXX3ksFvNbbrkl5T85/vjjD+/Xr5+Xl5d7bW2tepxQ0F+L/lr016K/Fv216K9F/xCXeXf3Dz/80LOzs/2GG25I2SdTbNy40fv16+eDBg3yXbt2qccJFf216K9Ffy36a9Ffi/5a3bx/uMu8u/uKFSs8Ly/Px44d63v37g37+A5Zs2aNFxcXe3l5eZf7RD6C/lr016K/Fv216K9Ff61u3D/8Zd7dfe3atd6rVy/v16+ff/PNN8m4IpCmpiafN2+eZ2Vl+YQJE7rMt5ZaQ38t+mvRX4v+WvTXor9WN+2fnGXe3X3nzp1+1VVXeVZWls+cOdMPHjyYrKva9Ntvv/mECRM8MzPTZ8+enfI/TxUW+mvRX4v+WvTXor8W/bW6Yf/kLfPu7olEwufOneuxWMxLS0t9+fLlybzuGPX19T5r1izPzc31gQMH+qpVqzrt7lRBfy36a9Ffi/5a9Neiv1Y365/cZf6IrVu3+rXXXutm5sOHD/cPPvjAm5qaknLX/v37fc6cOV5cXOyxWMyfeuqplH0yRGehvxb9teivRX8t+mvRX6ub9O+cZf6INWvW+MSJEz0tLc379+/vM2bM8E2bNnX43Hg87itWrPCbb77Z8/LyPD8/3x9++GHfuXNnCFN3HfTXor8W/bXor0V/LfprdfH+S9Lc3a2T/fzzz/bKK6/YokWL7M8//7TS0lIbPXq0XXbZZTZo0CArKyuznJycVh+/a9cu27Bhg3333XdWVVVl1dXV9s8//1h5ebnddNNNNmnSJCsqKurEv1G00F+L/lr016K/Fv216K/VRfu/JVnmj0gkElZdXW0rV660qqoqW7t2rcXjcUtLS7NevXpZz549rWfPnhaLxayurs727t1re/bssb1795qZWWFhoY0aNcpGjx5tFRUVVlpaqvqrRBL9teivRX8t+mvRX4v+Wl2sv3aZ/7fDhw/b5s2bbf369bZlyxarq6uzuro6q6+vt549e1pRUZEVFhZaSUmJlZaW2umnn64euUuhvxb9teivRX8t+mvRXyvi/VNrmQcAAABw3N5KV08AAAAAoH1Y5gEAAICIYpkHAAAAIup/6FbdD7WKy2IAAAAASUVORK5CYII=", 393 | "text/plain": [ 394 | "" 395 | ] 396 | }, 397 | "execution_count": 13, 398 | "metadata": {}, 399 | "output_type": "execute_result" 400 | } 401 | ], 402 | "source": [ 403 | "total.visualize()" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 14, 409 | "id": "ab0269d9", 410 | "metadata": {}, 411 | "outputs": [ 412 | { 413 | "name": "stdout", 414 | "output_type": "stream", 415 | "text": [ 416 | "CPU times: user 1.83 ms, sys: 1.35 ms, total: 3.18 ms\n", 417 | "Wall time: 1.01 s\n" 418 | ] 419 | }, 420 | { 421 | "data": { 422 | "text/plain": [ 423 | "36" 424 | ] 425 | }, 426 | "execution_count": 14, 427 | "metadata": {}, 428 | "output_type": "execute_result" 429 | } 430 | ], 431 | "source": [ 432 | "%%time\n", 433 | "total.compute()" 434 | ] 435 | }, 436 | { 437 | "cell_type": "markdown", 438 | "id": "ba56c847", 439 | "metadata": {}, 440 | "source": [ 441 | "**Note:**\n", 442 | "\n", 443 | "When we used `dask.delayed` without having a distributed scheduler (will see this later), we are relying on a single-machine scheduler and dask will use the threadpool executor, which by default will use the resources available on your machine. This can cause you to see different time values for the parallel version, since it'll depend on the resources you have available.\n", 444 | "\n", 445 | "You can check this by doing:" 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 15, 451 | "id": "fc6aab81", 452 | "metadata": {}, 453 | "outputs": [ 454 | { 455 | "data": { 456 | "text/plain": [ 457 | "8" 458 | ] 459 | }, 460 | "execution_count": 15, 461 | "metadata": {}, 462 | "output_type": "execute_result" 463 | } 464 | ], 465 | "source": [ 466 | "import os\n", 467 | "os.cpu_count()" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "id": "38659821", 473 | "metadata": {}, 474 | "source": [ 475 | "### The `@delayed` syntax \n", 476 | "\n", 477 | "The `delayed` decorator can be also used by \"decorating\" with `@delayed` the function you want to parallelize." 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": 16, 483 | "id": "294b1737", 484 | "metadata": {}, 485 | "outputs": [], 486 | "source": [ 487 | "@delayed \n", 488 | "def double(x):\n", 489 | " \"\"\"Decrease x by one\"\"\"\n", 490 | " sleep(1)\n", 491 | " return 2*x " 492 | ] 493 | }, 494 | { 495 | "cell_type": "markdown", 496 | "id": "acf79fa0", 497 | "metadata": {}, 498 | "source": [ 499 | "Then when we call this new `dec` function we obtain a delayed object:" 500 | ] 501 | }, 502 | { 503 | "cell_type": "code", 504 | "execution_count": 17, 505 | "id": "dd9c01f2", 506 | "metadata": {}, 507 | "outputs": [ 508 | { 509 | "name": "stdout", 510 | "output_type": "stream", 511 | "text": [ 512 | "Delayed('double-ea458133-3bf3-4e8d-8111-531e9158f458')\n" 513 | ] 514 | } 515 | ], 516 | "source": [ 517 | "d = double(4)\n", 518 | "print(d)" 519 | ] 520 | }, 521 | { 522 | "cell_type": "markdown", 523 | "id": "abdb64b5", 524 | "metadata": {}, 525 | "source": [ 526 | "### Exercise\n", 527 | "\n", 528 | "Using the `delayed` decorator create the parallel versions of `inc` and `add`" 529 | ] 530 | }, 531 | { 532 | "cell_type": "code", 533 | "execution_count": 18, 534 | "id": "57f47d98", 535 | "metadata": {}, 536 | "outputs": [], 537 | "source": [ 538 | "#solution\n", 539 | "\n", 540 | "@delayed\n", 541 | "def inc(x):\n", 542 | " \"\"\"Increments x by one\"\"\"\n", 543 | " sleep(1)\n", 544 | " return x + 1\n", 545 | "\n", 546 | "@delayed\n", 547 | "def add(x, y):\n", 548 | " \"\"\"Adds x and y\"\"\"\n", 549 | " sleep(1)\n", 550 | " return x + y" 551 | ] 552 | }, 553 | { 554 | "cell_type": "markdown", 555 | "id": "b3866d3d", 556 | "metadata": {}, 557 | "source": [ 558 | "``Delayed`` objects support several standard Python operations, each of which creates another ``Delayed`` object representing the result:\n", 559 | "\n", 560 | "- Arithmetic operators, e.g. `*`, `-`, `+`\n", 561 | "- Item access and slicing, e.g. `x[0]`, `x[1:3]`\n", 562 | "- Attribute access, e.g. `x.size`\n", 563 | "- Method calls, e.g. `x.index(0)`\n", 564 | "\n", 565 | "For example you can do:" 566 | ] 567 | }, 568 | { 569 | "cell_type": "code", 570 | "execution_count": 19, 571 | "id": "bfa8a187", 572 | "metadata": {}, 573 | "outputs": [ 574 | { 575 | "data": { 576 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAJPCAYAAACO6jTZAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3deVxU9eI+8GeAYZEdTVzTXIBCU8vrkisormnuS2VaVjfNSnMrb90W+5Xp1bLU8lqmoiBg7oaGggruiisqigYumRuy7zOf3x9+5UaiMTBnPmfOPO/Xy9d9XRznPD7R03Bm5oxOCCFARETWLMpOdgIiIqo6jjkRkQZwzImINMBBdgDSlitXrmDv3r2yY6jesGHDZEcgjdHxCVAyp8jISAwfPlx2DNXjv3ZkZnwClJQhhOCvcn5FRETI/kdDGsUxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxz0jyj0Sg7ApHiHGQHIG2KjIyUHQEAIITA1q1b0bt3b9lRAAD79u2THYE0imNOihg+fLjsCGUsW7ZMdgQiRfE0C5nVsGHDIIRQza9x48YBAGJjY6Vn+fMvInPjmJNmFRcXIywsDABK/5dIqzjmpFm//vorMjMzAQAREREoLCyUnIhIORxz0qywsDDo9XoAQE5ODrZt2yY5EZFyOOakSXl5eVi3bh2Ki4sBAPb29li1apXkVETK4ZiTJm3atAkFBQWl/7+kpAQbNmxATk6OxFREyuGYkyatWrUK9vb2Zb5WXFyMjRs3SkpEpCyOOWlORkYGtm7dipKSkjJf1+l0CA0NlZSKSFkcc9KcNWvWlPsWfoPBgJiYGNy6dUtCKiJlccxJc/7u0ffPP/9soSRElsMxJ025du0aEhISYDAYyv19IQRWrFhh4VREyuOYk6ZERETAzu7B39ZGoxH79u1DWlqaBVMRKY9jTpoSGhp63xOffyWEQFRUlIUSEVkGr5pImvH777/jzp07qFu3bunXSkpKUFBQADc3tzK35aVoSWt0gpdwIw2LjIzE8OHDeaVC0roonmYhItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAMcZAcgMichBK5evYrr168jNzcXx44dAwBs374dbm5ucHV1Rb169eDt7S05KZF56YQQQnYIosoQQiApKQmxsbHYu3cvzp07h+TkZOTl5f3tn61ZsyYCAgIQGBiIzp07IygoCL6+vhZITaSIKI45WRWj0Yi4uDiEhoYiOjoaN27cgLe3Nzp27IgnnngCfn5+8PPzQ+3ateHq6lr6aDwjIwM5OTnIyclBWloakpOTkZycjOPHj+PgwYMwGAwIDAzE4MGDMWrUKDRu3Fj2X5XIFBxzsg43b97Et99+i2XLluHy5cto06YNhg4diuDgYLRs2RJ2dpV/+icnJwfx8fH49ddfERERgT/++APPPPMMxo0bh+HDh8PBgWcjSfU45qRuV69exZw5c7BkyRK4ubnhtddew4svvoiAgABFjmcwGBATE4Ply5djzZo1aNCgAaZPn47Ro0fD0dFRkWMSmQHHnNSpuLgYixYtwgcffABXV1dMmjQJb731FqpVq2axDKmpqfjqq6/w3//+F/Xq1cOCBQvQs2dPix2fyAQcc1Kf3bt345///CcuXbqEDz74AJMnT5b6qDgtLQ0TJ07E+vXrMWLECMyfPx81a9aUloeoHFF8nTmphsFgwCeffILg4GA0bdoUp0+fxvvvvy/99EaDBg2wbt06bN68Gfv370fLli0RFxcnNRPRX3HMSRVu3LiBkJAQzJo1C9988w02btyIBg0ayI5VRt++fXHs2DE888wzCAkJwaeffgr+YEtqwafpSbqLFy+iZ8+eEEJg7969aNWqlexID+Tp6Yk1a9ZgwYIFePfdd3HhwgX88MMP0Ov1sqORjeM5c5IqKSkJPXv2RK1atfDLL79Y1bnoHTt2YODAgWjbti3Wrl0Ld3d32ZHIdvGcOcmTlJSEzp07IyAgAHFxcVY15ADQrVs3bN++HceOHcPgwYNRVFQkOxLZMD4yJymuXLmCDh06oE6dOtixY4dFX3JobidPnkTnzp3Rq1cvrFq1qkpvYCKqJD4yJ8vLzMxESEgIvLy8EB0dbdVDDgDNmzfHzz//jHXr1mHatGmy45CN4piTxb366qvIyMhAdHQ0vLy8ZMcxi+DgYCxduhTz5s3D2rVrZcchG8RXs5BFLVy4EGvXrsXWrVtRp04d2XHM6vnnn8euXbswduxYtGrVCo899pjsSGRDeM6cLCY5ORktW7bE9OnT8fHHH8uOo4iCggK0a9cObm5uiI+Ph06nkx2JbAPfzk+W0717d9y4cQOJiYmavhLhyZMn8dRTT2Hx4sV45ZVXZMch28AnQMkywsPDERcXh8WLF2t6yIG7T4iOHz8e06ZNw61bt2THIRvBR+akuKKiIjRp0gQ9e/bEkiVLZMexiMzMTAQEBOD555/H3LlzZcch7eMjc1LeihUr8Mcff+CDDz6QHcViPD09MW3aNHz//fe4efOm7DhkAzjmpCiDwYA5c+Zg9OjRqrtwltL++c9/ws3NDfPnz5cdhWwAx5wUtWnTJqSkpGD69Omyo1hctWrVMHHiRCxcuBAFBQWy45DGccxJUStWrEC3bt3QpEkT2VGkGDt2LHJycrBx40bZUUjjOOakmPT0dPzyyy8YNWqU7CjS1KxZE927d0doaKjsKKRxHHNSzJo1a+Dg4ICBAwfKjiLVSy+9hG3btiE9PV12FNIwjjkpJiYmBkFBQXBzc5MdRarevXvDaDRi586dsqOQhnHMSRFCCOzcuRNBQUGyo5Q6ceIE5s+fjzt37pjldhXl5eWFVq1a8XNDSVEcc1LEyZMncevWLVWNeUJCAiZOnIg//vjDLLczRXBwMGJjY812f0R/xTEnRRw9ehQuLi5o0aKF7Ciq0K5dO5w9exb5+fmyo5BGccxJEWfPnkXTpk35qTv/x9/fH0ajEefPn5cdhTSK/6aRIpKTk+Hv72/W+9y5cyfefPNN+Pn5oX79+hg5ciS+//57GAyG+2576NAhDB06FI0aNUL37t2xYMEClHcZoorerqqaNGkCBwcHnD171uz3TQTwwylIIRcvXkSfPn3Mdn9xcXEICQmBp6cnnn/+edSoUQMxMTEYN24cLl68iNmzZ5fedufOnejbty+cnZ0xaNAg2NnZ4cMPP7zvU40qejtzcHR0RP369XHx4kWz3zcRwDEnhWRmZpp1FMPDw+Hg4IALFy6U3u/06dPRqFEjbNq0qcyYT5w4EU5OTjhy5AgaNmwIAJgyZQpatmxZ5j4rejtz8fLyQmZmpiL3TcTTLKSI7OxsuLu7m+3+3n33XRw6dKjMfyCKiorg5eWFrKys0q/t378fx48fx/jx40sHGgCaNm1a5p2oFb2dObm7uyM7O1uR+ybiI3NShLnHPCAgALdv38bcuXOxb98+pKam4vz588jKyirzWaL3zkmX9+g6MDDQ5NuZk4eHB8ecFMNH5qQInU5n1icS58yZg3r16mHmzJkoLi5G9+7dsWzZMnTo0KHM7e69Zd7e3v6++3B2djb5dubEz4EhJXHMSRFubm7Iyckxy33dvHkT7733Hjw9PXH58mVs2LABs2bNwsCBA++7tOxjjz0GANi1a9d995Oammry7czJ3D+tEP0Zx5wUYc7zw2lpaTAajRg0aFCZMbx8+TKOHTtW5ratW7eGXq+/792WJSUlCAsLM/l25sQxJyVxzEkRXl5eZru2ib+/P9zc3BAREYFNmzbh/PnzWLZsGZ555hl4eHggJycHycnJAID69evjzTffxMmTJzF27FgkJibi6NGjGDJkSJlXklT0duaUkZEBT09PRe6bCIJIAYMGDRJDhw412/1FRkYKNzc3AUAAED4+PmL58uVizZo1wtXVVTg4OJTetqCgQLz22multwUgunXrJkJDQwUAcfr0aZNuZw6FhYXCwcFBREZGmu0+if4kUicEn5Uh85sxYwa2bNmC48ePm+0+b9++jaNHj6J27dp44oknoNPpSr9+586d+z7N6PLlyzh58iQef/zx0nPk5ano7aoiKSkJzZo1w/Hjx/Hkk08qcgyyaVEcc1LE8uXLMW7cOGRnZ5f7ihFbs3btWgwdOhQ5OTlwcXGRHYe0J4rnzEkRTz31FPLz8836yNya7d+/H48//jiHnBTDMSdFNGvWDL6+vvxAhv8TGxuL4OBg2TFIwzjmpAidTofOnTtzzHH3VSzHjh1T1Qd1kPZwzEkxISEhiIuLs/m3sG/ZsgV2dnbo2rWr7CikYRxzUsyQIUNgMBiwdu1a2VGkCg0NRe/eveHt7S07CmkYx5wU4+3tjb59+yI0NFR2FGmuX7+OHTt2KHYlRqJ7OOakqNGjRyMuLs5mPy5tyZIlcHd3x7PPPis7Cmkcx5wU1bdvXzRt2hSzZs2SHcXicnNz8c0332DChAmKXYmR6B6OOSnK3t4e06dPR2hoKNLS0mTHsajvvvsOeXl5eOutt2RHIRvAd4CS4oqLi9G0aVMEBwdj6dKlsuNYREZGBvz9/fHSSy9hzpw5suOQ9vEdoKQ8vV6P2bNnY9myZeVeP1yLPvjgAxgMBrz33nuyo5CN4CNzspi+ffvi0qVLSExMhF6vlx1HMYmJiWjTpg1+/PFHjB49WnYcsg280BZZzvnz59GiRQtMnjwZM2fOlB1HEfn5+Wjbti28vLywa9eu0is7EimMp1nIcpo2bYqvvvoKn3/+OX799VfZcRTx1ltv4cqVK1i+fDmHnCyKj8zJ4kaOHIm4uDgcPnwY9erVkx3HbFasWIExY8Zg3bp1eO6552THIdvC0yxkeVlZWWjfvj10Oh12794NHx8f2ZGqbPv27ejbty8mTZpkk6+pJ+k45iTH1atX0aFDB9SqVQs7duyAq6ur7EiVduLECXTu3Ln00gV2djx7SRbHc+YkR926dRETE4PffvsNffv2VexDlJV28OBBdO/eHa1bt8ZPP/3EISdp+J1H0jRt2hSxsbG4cOECOnbsiKtXr8qOZJKYmBh0794d7dq1w8aNG+Ho6Cg7EtkwjjlJFRgYiPj4eBQXF6Njx444dOiQ7Eh/SwiBr776Cn369MHQoUOxdu1aVKtWTXYssnEcc5KuYcOGSEhIgJ+fHzp27Ij58+dDrU/lpKenY8CAAZg2bRpmzpyJH374AQ4ODrJjEXHMSR1q1KiB6OhofPTRR5gyZQr69OmDCxcuyI5Vxrp169CyZUskJiYiNjYW7733Hl9LTqrBMSfVsLOzw4wZM7Br1y5cuXIFzZo1w8cff4z8/HypuVJSUtC3b18MHjwYXbt2xdGjR9GpUyepmYj+ii9NJFUqKSnBwoUL8e9//xtOTk4YP348atWqhZdeeski56e3bNkCR0dHbN68GYsXL0aDBg2wYMEChISEKH5sokrg68xJ3a5fv465c+fi22+/RWFhIaZOnYpRo0ahWbNmihyvuLgY27Ztw5dffomEhAQ0atQI//73v/HCCy/w3DipGV9nTurm6+uLNm3aoKioCJ6enoiMjETz5s3x1FNP4YsvvsCBAwdQUlJSpWNkZmZi48aNmDBhAurWrYv+/fsjLy8PAFBYWIhu3bpxyEn1+MicVG3jxo0YNGgQDAYD2rVrh7179yI+Ph6hoaHYsmULrl27Bg8PD3To0AFPPPEE/Pz84O/vD19fX7i5uZX+ys7ORkZGBnJycpCWlobk5GQkJyfj2LFjSExMhBACTz75JAYPHoxRo0bByckJtWvXhk6nQ8OGDbFnzx7Url1bdh1ED8LTLKReMTEx6Nu3LwwGA4QQGDp0KCIiIsrc5uzZs4iNjcW+fftw9uxZJCcnIzs7+2/vu06dOggICEBgYCC6dOmCrl27onr16qW/bzQa4ejoCIPBAL1ej/r162PPnj2oVauW2f+eRGbAMSd1SkhIQEhICIqLi2EwGODo6Ijx48fjq6+++ts/e+3aNdy8eRM5OTmlvzw8PODl5QU3NzfUrVsX7u7uf3s/NWrUwO3btwHc/bSkJk2aID4+vszoE6lEFE8Ekurs27cPPXr0KB1yANDpdBV+VFy7dm2znBLx9fUtHfPi4mKkpKSgS5cuiI+Ph7e3d5Xvn8ic+AQoqcrRo0fRs2dPFBUVlQ45cPelipY+xVG/fv0y/7+4uBjnzp1Djx49KnQqh8iSOOakGsePH0fXrl2Rl5dXZsgBwGAwWPwJyHr16t33Kpbi4mIcP34cISEhyMnJsWgeoofhmJMqJCcnIzg4uNwhv8fSj8xr1aoFe3v7+75eXFyMI0eOoH///igoKLBoJqIH4ZiTdOfPn0fHjh2RlZX10NeMW/qRee3atWE0Gsv9vZKSEsTHx6Nfv34oLCy0aC6i8nDMSap71zK/c+fOQ4fc3t7e4q8iqVWrFoqLix/4+yUlJdixYweGDx9e5TcuEVUVx5ykunr1Kho1alT6eu4H8fHxsfin+PzdTwJ6vR729vbw8fEpfdULkSwcc5Kqc+fO2LdvHw4fPoyhQ4fCzs6u3FGX8WadBx3T3t4eLi4uGDduHH777TcsXboUvr6+Fk5HVBZfZ06q8PTTT2PVqlVo27YtpkyZAgcHB+h0utLTHPXq1bN4pj8/MtfpdNDpdHB1dUVJSQnOnz+PunXrWjwT0YPwkTmpSnh4OPr374+0tDRMmjQJrq6uACBlOF1cXEqP37hxYyxZsgQXLlyAo6MjwsPDLZ6H6GH4dn5Sjbi4OAQHB2Pv3r1o3749ACArKwvff/89XF1d8eabb1o808svv4yBAwfi2WefLT1nP3nyZERERODixYv8EGdSC16bhdSjT58+yMvLw86dO2VHeagrV66gcePGWLx4McaMGSM7DhHAMSe1OHnyJFq0aIHNmzejT58+suP8rdGjR+PgwYNISkqy+KtsiMrBMSd1GDVqFI4dO4YTJ05YxYcknzlzBs2aNcP69evRr18/2XGIOOYk35UrV9CoUSP88MMPeOmll2THqbBnn30WWVlZ2L17t+woRPzYOJJv7ty58PX1xYgRI2RHMcnUqVMRHx+PvXv3yo5CxJcmklx37tzBDz/8gEmTJlndK0O6dOmC9u3b4z//+Y/sKEQcc5Jr4cKFsLe3x9ixY2VHqZTJkydj/fr1OH36tOwoZOM45iRNYWEhFi5ciDfffBOenp6y41TKwIEDERAQUKGPsyNSEsecpPnpp59w584dTJgwQXaUSrOzs8M777yD0NBQ/P7777LjkA3jmJMUBoMB8+bNw5gxYyx+nXJzGzNmDHx8fPDtt9/KjkI2jGNOUqxduxYXLlzAO++8IztKlTk5OWHChAlYtGgRMjMzZcchG8UxJynmzZuHgQMH4vHHH5cdxSzefPNN6HQ6LFmyRHYUslEcc7K4uLg47N+/H5MnT5YdxWw8PT0xduxYfP311ygqKpIdh2wQ3wFKFte7d2/k5+er/oJaprp3Aa7vv/8eL7/8suw4ZFv4dn6yLGu7oJapeAEukoRjTpY1atQoHDlyBElJSVZxQS1T8QJcJAnHnCzHWi+oZSpegIsk4IW2yHKs9YJapuIFuEgGjjlZhDVfUMtU9y7ANWfOHNlRyIZwzMkirP2CWqaaPHkyNmzYwAtwkcVwzElxBQUFVn9BLVPxAlxkaRxzUpwWLqhlKjs7O0ycOJEX4CKL4ZiTogwGA7766itNXFDLVKNHj+YFuMhiOOakKC1dUMtUvAAXWRLHnBSltQtqmYoX4CJL4ZiTYrR4QS1T8QJcZCl8BygpRqsX1DIVL8BFFsC385MytH5BLVPxAlykMI45KUPrF9QyFS/ARQrjmJP52coFtUzFC3CRgnihLTI/W7mglql4AS5SEseczCo9Pd1mLqhlKl6Ai5TEMSezsrULapmKF+AipXDMyWwKCgqwaNEim7qglql4AS5SCp8AVSGj0Sg7QqUsXrwY7777Li5cuIBatWopeiwlX96ndP9LlizBO++8gwsXLljt9Wr48krV4atZ1CYyMhLDhw+XHUP1lPq2Zf8Vw9lQnSgH2QmofFFRUbIjqNK+ffswb948xY/D/stnqf7JdBxzlRoyZIjsCKpkqVNQ7L981noK0BbwxBcRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg1wkB2Aymc0GmVHKJWYmIinnnpKdgwAgBDCIsdh/+WzVP9kOo65Stnb28uOYNPYP1kbjrnKPPPMM4iKipIdo9TGjRuxcuVKLFy4EI888ojsOIpj/2StdII/N9FDNG/eHKdOncIXX3yB9957T3Ycm8P+qYKiOOb0QMnJyQgICAAABAQE4MyZM5IT2Rb2TyaI4qtZ6IHCwsKg1+sBAGfPnsWpU6ckJ7It7J9MwTGnBwoNDUVxcTEAwNHREatXr5acyLawfzIFT7NQuQ4dOoQ2bdqU+Vq9evVw6dIl6HQ6SalsB/snE/E0C5UvPDy89Ef8e65cuYIDBw5ISmRb2D+ZimNO9zEajVi1alXpj/j3ODo6Ijw8XFIq28H+qTI45nSfXbt24caNG/d9vaioCKGhoSgpKZGQynawf6oMjjnd58+vovirO3fuIC4uzsKJbAv7p8rgmFMZxcXFiIyMvO9H/Hv0ej3CwsIsnMp2sH+qLI45lREdHY2srKwH/v69scnPz7dgKtvB/qmyOOZURlhYGBwcHn7Jnry8PGzdutVCiWwL+6fK4phTqdzcXGzcuLFCT7DxVRXmx/6pKnjVRCpVUlKCjRs3lvnarl278NlnnyEmJqbM152cnCwZzSawf6oKjjmV8vT0RPfu3ct8LT09HQDu+zqZH/unquBpFiIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0QCeEELJDkHyXL1/G2bNnkZycjOTkZFy/fh3Z2dm4efMm/vjjD/j7+8PNzQ2urq5o2LAh/Pz8EBAQgMDAQLi7u8uOb/XYP1VRFMfcRt25cwdbtmxBbGws4uLikJqaCgCoUaMG/P39UadOndLxcHV1RUZGBnJycpCbm4uLFy/i3LlzKCgogIODA55++mkEBQWhe/fu6Nq1K+zt7eX+5awA+ycz45jbkpKSEkRHRyM0NBSbNm2C0WhEu3btEBQUhKCgIDRv3hw+Pj4Vui+j0YhLly7h0KFDiIuLQ1xcHM6ePYs6derghRdewOjRoxEYGKjw38i6sH9SUBQEaV5hYaFYvny58PPzEwDE008/Lb7++mtx+/Ztsx4nLS1NzJo1SzRp0kQAEB06dBA7duww6zGsEfsnC4jkmGuY0WgUy5YtE/Xq1RNOTk7i9ddfFykpKRY5bnR0tOjYsaMAIIKDg8WJEycUP67asH+yII65Vp04cUJ06tRJ2Nvbi/Hjx4srV65IybFz507Rrl074eDgIN59912RlZUlJYelsX+yMI651hiNRjF79myh1+tF27ZtxZEjR2RHEgaDQfz3v/8V1atXFw0aNBD79u2THUkx7J8k4Zhryc2bN0Xfvn2FXq8Xs2fPFgaDQXakMm7cuCF69+4t9Hq9+M9//iOMRqPsSGbF/kkijrlWXLx4UTRt2lTUr19fJCQkyI7zQEajUXz99ddCr9eLF154QRQVFcmOZBbsnyTjmGtBYmKiqFWrlmjdurW4ceOG7DgVsm3bNuHm5iZ69eolcnJyZMepEvZPKhDJt/NbucTERAQFBaFZs2aIjY3FI488IjtShfTo0QOxsbE4cuQIevXqhfz8fNmRKoX9k1rwTUNWLCUlBR06dECrVq2wceNGODo6yo5kstOnT6Nz58545plnsHbtWjg4OMiOVGHsn1Qkio/MrdT169fRo0cPNGzYEGvWrLHKIQGAJ554Aps3b0ZsbCzeeOMN2XEqjP2T2nDMrZDRaMSLL74IOzs7bNmyBW5ubrIjVUm7du0QERGBn376CUuWLJEd52+xf1Il2WftyXSffvqpcHJyEocPH5YdxaxmzJghnJ2dRWJiouwoD8X+SYUiec7cyhw4cAAdOnTA119/jQkTJsiOY1YlJSXo1q0bbt26haNHj6ry1AX7J5XiVROticFgQJs2beDl5YUdO3bIjqOI1NRUBAYG4sMPP8R7770nO04Z7J9UjE+AWpNFixbh1KlTWLBggewoimnYsCFmzJiBmTNnll7jWy3YP6kZH5lbiezsbDRs2BCvv/46vvjiC9lxFI0fzLQAACAASURBVFVYWIgnn3wSbdu2xYoVK2THAcD+SfX4yNxafPfddygpKcG0adNkR1Gck5MTPvjgA4SFheH8+fOy4wBg/6R+HHMrUFBQgK+//hrjx4+Ht7e37DgW8fzzz6NRo0b4z3/+IzsK+yerwDG3AqtXr0Z6ejomTpwoO4rF2NvbY/LkyVi+fDnu3LkjNQv7l9s/VQzH3AosX74c/fv3h6+vr+woFjVy5EjY29sjMjJSag72L7d/qhiOucqlpaVh9+7dGDVqlOwoFufh4YEBAwYgNDRUWgb2L7d/qjiOucqtWbMG3t7e6NWrl+woUrz44ovYu3cvrly5IuX47F9u/1RxHHOV27FjB7p16wa9Xi87SoX98ssvWL16tVnuKygoCE5OToiNjTXL/ZmK/cvtnyqOY65iJSUl2LNnD4KCgmRHMcns2bMxdepUs9yXs7Mz2rVrh7i4OLPcnynYv9z+yTQccxVLTExEVlaW1Y2JuQUFBWHXrl0WPy77v0tW/2QajrmKJSUlwcXFBU2bNpUdRaoWLVogNTUVOTk5Fj0u+79LVv9kGo65iiUnJ8PPzw92dub5x/Taa69h9OjRSElJwauvvor69esjODgYK1euBADMmzcPTz/9NGrWrInevXvf9+6/l156CS+++OJ99ztr1ix06tQJJSUlZsn5V/7+/hBCICUlRZH7fxD2f5es/sk0/IwoFUtOToa/v7/Z7u/YsWO4cuUKtm/fDi8vLwQFBSEiIgI7d+5EWFgYYmJi0KdPHzRo0ABbtmxB9+7d8dtvv5WO2ZEjR2A0Gu+73/PnzyMhIaHc3zOHxo0bw8HBAWfPnkXLli0VOUZ52P9dsvon0/CRuYr98ccfqFevntnvc/z48UhKSsKKFSuwfv16CCGwc+dOJCUlYcOGDVi7di1GjhyJS5cuqeLRmF6vh6+vL65du2bR47L/u2T1T6bhmKtYVlYW3N3dzXqf9vb2ZV7p0KJFCwBAcHAw/Pz8Sr/etWtXAHc/8FcN3N3dkZ2dbdFjsv//kdE/mYZjrmLZ2dlmH5M6deqU+QQZZ2fn0q//mb29PQCgqKjIrMevLBljwv7/h2OufhxzFSsoKCj9l91cXF1dy/16VZ7kS09Pr/Sfrahq1aohLy9P8eP8Gfv/Hxn9k2k45irm4uKC/Px82TFK6XS6cp9kS05OVvzYubm5DxxCpbD//5HRP5mGY65iavvRtmHDhkhNTUVxcXHp15KSkizyJJ0Spzz+Dvv/Hxn9k2k45irm4eGBrKws2TFKtW3bFkVFRRgzZgx27tyJH374AQMGDICnp6fix5YxJuz/fzjm6sfXmatYnTp1cPnyZdkxSk2ePBn79u1DWFgYwsLCULdu3dJLw86aNUux4xYWFuL69etmf5ng32H/d8nqn0zDD3RWsQ8++AAbNmzAyZMnZUcp4+bNm7h69SpatGgBnU6n+PGSkpLQrFkznDhxAs2bN1f8ePew/7tk9U8mieIjcxXz9/fH+fPnYTAYSl+qpgaPPPIIHnnkEYsdLzk5GXZ2dha/Rgr7v0tW/2QanjNXsWbNmqGwsFA1bxyRJTExEU2aNDH7ywT/Dvu/S1b/ZBqOuYq1aNEC3t7eNn8t6djYWCmXoWX/d8nqn0zDMVcxOzs7dOnSxabHJDs7G4cPH5YyJuxfbv9kGo65ynXr1g1xcXGqevOKJcXExMBgMJReq8TS2L/c/qniOOYqN3ToUOTm5mLTpk2yo0ixcuVKBAcHw9fXV8rx2b/c/qniOOYq5+vri5CQEISGhsqOYnHp6en45ZdfSl9LLQP7l9s/VRzH3AqMHj0aW7duxaVLl2RHsajly5dDr9dj0KBBUnOwf7n9U8VwzK3AoEGDUK9ePcyZM0d2FIspLCzE3Llz8frrr8PNzU1qFvYvt3+qGI65FdDr9Zg6dSp++OEH/P7777LjWMTSpUtx69YtvPvuu7KjsH+yCnw7v5UoKChA48aN8dxzz2HRokWy4ygqJycHjz/+OPr374+FCxfKjgOA/ZPqRfGRuZVwdnbGrFmzsHjxYhw4cEB2HEV9+umnyM3NxUcffSQ7Sin2T2rHR+ZWRAiBoKAg5OXlYd++faq6Xoi5JCUloVWrVpg/fz7GjRsnO04Z7J9ULIpjbmWSkpLQunVrvPfee5p75FRQUID27dvDyckJe/furdJHqSmF/ZNKRUGQ1VmwYIGws7MTv/76q+woZvXGG28ILy8vceHCBdlRHor9kwpFcsyt1JAhQ0StWrVEamqq7ChmsXz5cqHT6cSaNWtkR6kQ9k8qE8nTLFYqMzMTnTt3RlFREeLj41GjRg3ZkSpt69at6N+/P6ZMmYLPP/9cdpwKYf+kMjxnbs1+//13dOjQATVr1sT27dut8jMa9+/fj+7du2Po0KFYunSpRT45x1zYP6kIX5pozerUqYNt27bh0qVLCAoKwtGjR2EwGGTHqrBff/0VISEh6NatG5YsWWJ1Q/LX/m/cuCE7UoVdu3YNmzdvtur+qSyOuZXz8/PD+vXrkZKSgk6dOiEtLU12pApZtWoV+vXrh4EDB2LNmjVwcLDOTzD08/NDQkICMjIy0LFjR1y8eFF2pApZv349+vfvj2bNmmH16tVW2z/9idxz9lQVd+7cEf/617+Es7OzACAaN24sfHx8xIYNG2RHe6DCwkLxzjvvCJ1OJ6ZMmSKMRqPsSGbxxx9/iFatWllV/82bNxcARMOGDUV4eLgwGAyy41Hl8dUs1ig3N1fMmjVLeHh4CL1eLwCIJk2aiJycHPHKK68InU4nJk2aJPLz82VHLeP8+fPiH//4h3B3dxerVq2SHcfscnNzrar/kydPCp1OV/orMDBQbNmyRXZMqhyOuTUpKioSixcvFjVr1hQODg4CgAAg7O3txeLFi0tvFxoaKtzd3UWTJk1EdHS0xMR35efni48//lg4OzuLli1birNnz8qOpChr6j8oKKj0e8ne3l4AEE8//bSIjY2VmJgqgWNuDQwGg4iMjBQNGjQQdnZ2QqfTlQ45AOHh4SFycnLK/JnLly+LoUOHCgBiwIAB4vjx4xbPXVJSIsLCwkSTJk2Em5ubmD17tigqKrJ4Dhmspf9NmzaV+V7686h37dpVHDlyxOK5qVI45moXExMjmjVrVvqj8F//xdPr9WLGjBkP/PPbtm0TLVq0EDqdTvTv31/s2bNH8cz5+fli6dKlws/PT9jb24sXXnhBXL58WfHjqpHa+zcajaJRo0blfm85ODgInU4nBg0aJJKTkxXPTVXCMVerPXv2iPbt2wsAws7O7r5/0f78KOrvhtJoNIpNmzaJdu3aCQAiICBAfPbZZ+K3334zW96SkhKxe/du8eqrrwovLy+h1+vFK6+8Is6dO2e2Y1grtfe/YMGC0kfj5f3S6/VCp9OJIUOGiIsXL5otM5kVx1yN3n///TI/7j7sX7Lhw4ebdN+HDh0Sb731lnjkkUcEAOHn5yf++c9/ivDwcJGUlCQKCwvv+zMpKSn3fS09PV3s379ffPPNN2LAgAHC29tbABAtW7YU8+bNE9euXav031/L1Nh/dna2cHNze+j32r0HFU5OThb56YJMxrfzq1F2djaeffZZ7N27FyUlJQ+97YEDB9CmTRuTj1FcXIxdu3YhLi4OcXFxOHToEEpKSmBvb4+GDRuidu3acHNzg6OjIxITE9GuXTtkZGQgMzMTqampuHnzJgDAy8sLXbp0QXBwMEJCQvD4449X6u9sa9TW/9SpUzF//nwUFxc/9HazZs3C9OnTK3UMUhTfzq9WhYWFGDlyJDZu3Fjuuzrt7OzQunVrs31QQkFBAc6dO4fk5GScO3cO169fR05ODo4fP46jR4+iX79+qFWrFtzd3fHoo4/C398ffn5+aNCgAS+Vagay+7906RIee+wxGI3Gcn/fzs4O3333HV5//fUqH4sUwUvgqllJSYkYMWJEuefMdTqdiIyMVDzDvfO83377reLHovtZsv/BgweXvm/hz99n9vb2Ijw8XPHjU5VE8iGVimVlZeH06dPw8/O777oZtWrVwsCBAxU9/uXLl0sf+a9YsULRY9H9LN3/pEmTypxmsbOzg16vh6urKw4dOqT48alqOOYqlZeXh/79+yM9PR1bt27FRx99VDroDg4OmDRpkuLX0wgPDy/9aLTDhw/jt99+U/R4VJal++/QoQNatmwJOzs72Nvbo1q1aoiNjcVPP/2E+fPn47PPPlP0+FRFsn82oPsVFhaK3r17ixo1aojTp0+Xfn3+/PlCp9MJZ2dnkZ6erniOwMDA0tcf6/V68fnnnyt+TPofGf2vWrVKABA+Pj7i2LFjpV//7rvvBADx1VdfKZ6BKoUvTVSbkpISMWzYMOHh4SEOHz583++vXLlSTJw4UfEcZ86cue88fdOmTRU/Lt0lq/+ioiLRoUOHct8k9PnnnwudTieWLl2qeA4yGcdcTYxGo3jttdeEi4uL2Llz5wNvV1JSoniWDz/88L4nwwCIkydPKn5sktv/w76/pk2bJvR6vdi0aZPiOcgkfAJUTaZPn45ly5YhKioKXbp0eeDt7p1HVdKKFSvue82xo6MjwsPDFT82ye3/Yd9fs2bNwpgxYzBs2DDs2rVL8SxkAtn/OaG7/t//+39Cp9OJn376SXYUceDAgQe+C7BOnTqauQa5Wqm9/787FUhS8JG5Gnz//ff417/+hXnz5mHMmDGy4yA8PBx6vb7c3/v999+xf/9+CyeyLWrv397eHqGhoejQoQN69eqFM2fOSM1Dd3HMJVu7di0mTJiAzz77DBMnTpQdB0ajEatWrXrg27p5qkVZ1tK/o6Mj1qxZg4CAAPTo0QOpqamyI9k8vp1fopiYGPTr1w+vv/46vvnmG9lxAACxsbHo1q3bQ2/j5eWFmzdv8nMjFWBt/WdmZiIoKAjZ2dlISEiAr6+v7Ei2KoqPzCXZv38/Bg4ciGHDhmH+/Pmy45QKCwt74I/492RkZCA2NtZCiWyLtfXv6emJrVu3wt7eHj169MCdO3dkR7JZHHMJTpw4gT59+qB79+5YunTpfW/Vl6WoqAhRUVEwGAzQ6/XQ6/VwcHCAg4ND6f+/NzRq+FFfa6y1/5o1ayImJgYZGRno27cvcnNzZUeySfJ/TrMxKSkp6NmzJ1q1aoXVq1er4kfle7KysjBnzpwyXzt8+DCWLFmCxYsXl/m6h4eHJaPZBGvuv379+vjll1/QpUsXDBw4EJs3b4ajo6PsWDaF58wt6OrVq+jYsSNq1qyJHTt2wM3NTXakvxUZGYnhw4eD3yZyWFv/Bw8eRPfu3dGvXz+Ehoby8siWw3PmlnLr1i306NEDrq6u+OWXX6xiyIlM1aZNG6xfv770VVpkOer5GV/DsrKy0Lt3bxQUFCA+Ph7Vq1eXHYlIMcHBwYiIiMDgwYNRvXp1zJw5U3Ykm8AxV1hRURGGDBmCS5cuIT4+HnXq1JEdiUhx/fv3x9KlSzFmzBh4enpiypQpsiNpHsdcQQaDAc8//zwOHjyInTt3ws/PT3YkIosZNWoUMjMz8fbbb8PLywuvvvqq7EiaxjFXiBACr7/+OqKjo7Ft2za0bNlSdiQii5swYQJu3LiBN954A56enhg6dKjsSJrFMVfIlClTsHLlSqxfvx4dO3aUHYdImk8//RQ5OTl48cUX4e7ujl69esmOpEkccwV89NFHmD9/PsLCwtC7d2/ZcYikmzt3LjIyMjB48GD8+uuv6NChg+xImsOXJprZwoULMXPmTHz33XcYNmyY7DhEqqDT6bBkyRL06tULzz77LI4dOyY7kuZwzM1o5cqVePvtt/Hll1/itddekx2HSFXs7e2xcuVKtGjRAj179sS5c+dkR9IUjrmZbNy4ES+//DLef/99TJ06VXYcIlVycXHBxo0bUb9+ffTu3RvXrl2THUkzOOZmEBcXh+HDh+PVV1/FZ599JjsOkap5eHhg69atcHJyQo8ePZCeni47kiZwzKvo0KFDeO6559CnTx8sWLBAdhwiq1CjRg3ExMQgJycHvXv3Rk5OjuxIVo9jXgXnzp3Ds88+i3bt2iEsLMwiH7RMpBV169ZFTEwMLl26hOeeew6FhYWyI1k1jnklXb58GSEhIWjcuDHWrVsHJycn2ZGIrE6TJk2wbds2HD16FCNGjEBJSYnsSFaLY14JN2/eREhICLy8vLBlyxa4urrKjkRktZ588kls2bIFMTExGDt2rNVc7ldtOOYmyszMRM+ePWEwGLBt2zZ4e3vLjkRk9dq3b49169YhIiIC77zzjuw4VonvADVBXl4e+vXrhxs3biAhIQG1atWSHYlIM0JCQhAWFoZhw4ahVq1amDFjhuxIVoVjXkHFxcUYMmQIzpw5g927d6Nhw4ayIxFpzqBBg/Dtt99i/PjxqFatGiZOnCg7ktXgmFeA0WjEqFGjsGfPHuzYsQOPP/647EhEmjVu3Dikp6dj8uTJ8PX1xciRI2VHsgoc878hhMC4ceOwYcMGREdHo3Xr1rIjEWnev/71L2RmZmL06NHw8PBA3759ZUdSPY7533j//ffx448/IiIiAl27dpUdh8hmfPnll7hz5w6GDh2KrVu3onPnzrIjqRpfzfIQs2bNwuzZs7FkyRIMHjxYdhwim6LT6fD999+jX79+6NevH44cOSI7kqpxzB9g+fLlmDFjBubOnYuXX35Zdhwim2Rvb4/Q0FA888wz6N27N86ePSs7kmpxzMuxbt06jB07Fp988gkmTZokOw6RTXN0dMTPP/8Mf39/hISEIC0tTXYkVeKY/8X27dsxcuRIjBs3Dh9++KHsOEQEoFq1ati8eTNq1KiBkJAQXL9+XXYk1eGY/8n+/fsxcOBADB06FPPnz5cdh4j+xNPTE9u2bYNOp0PPnj2RkZEhO5KqcMz/z8mTJ9G3b18EBwfjp59+gp0dqyFSm5o1ayI6Oho3b95E3759kZubKzuSanCxAFy4cAE9e/ZEixYtEBERAQcHvmKTSK0aNWqEbdu24ezZsxgxYgSKi4tlR1IFmx/zq1evIiQkBPXq1cOGDRvg7OwsOxIR/Y1mzZohOjoaO3fuxMsvvwyj0Sg7knQ2Pea3b99Gjx49oNfrsXnzZri7u8uOREQV1KZNG2zYsAE///wz3nrrLdlxpLPZ8wl5eXno378/srOzkZCQgJo1a8qOREQmCg4OxurVqzFkyBDUqFEDn3zyiexI0tw35hcuXMC2bdtkZLGo8PBwnDp1Cu+++y42b95s8p8fP368AqnU139iYiIAYNGiRZKTlMX+5VJb/yNGjMDMmTORkZEBf39/BZKpS3n968RfPtYjMjISw4cPR/Xq1S0WTAYhBIxGo8mf21lUVITs7GzFPg1Fjf0LIaDT6WTHAMD+ZVNz/yUlJZp/8cJD+o964N/81q1byqayUve+2ZTG/svH/uVi/3I9rH+bfgKUiEgrOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGWMWYX716VXYEm8b+5WL/cllL/w4P+o1FixZZMscD5efnY9WqVXj11VdlRwEAJCYmWuQ47L987F8u9i/Xw/rXCSHEn7+wYcMGjB07VvFQFVVYWIicnBz4+PhAp9PJjlPq1q1bitwv+68Y9i8X+5ernP6jIFSuS5cuAoBYvHix7Cg2if3Lxf7lsqL+I1V9zvzatWuIj48HAKxYsUJyGtvD/uVi/3JZW/+qHvOIiAjY2d2NuHfvXly5ckVyItvC/uVi/3JZW/+qHvMVK1bAaDQCABwcHBARESE5kW1h/3Kxf7msrf/7ngBViwsXLqBp06a4F0+n06F58+Y4fvy45GS2gf3Lxf7lssL+o1T7yDwsLAwODv975aQQAidOnMCZM2ckprId7F8u9i+XNfav2jEPDQ1FcXFxma85Ojqq/kcdrWD/crF/uayxf1WeZjl27BhatWpV7u81aNAAqamplg1kY9i/XOxfLivtX52nWcLDw6HX68v9vbS0NBw5csTCiWwL+5eL/ctlrf2rbsyFEFi5cuV9P+Lc4+joiPDwcAunsh3sXy72L5c196+60ywJCQno1KnTQ2/zyCOP4Nq1a7C3t7dQKtvB/uVi/3JZcf/qO83ysB9x7rl582bpO7PIvNi/XOxfLmvuX1VjXlJSgtWrV8NoNMLJyan0l6OjY5n/D0C1P+pYM/YvF/uXy9r7f+AlcGVIT0/HtGnTynztxIkTCAsLw6xZs8p83dvb25LRbAL7l4v9y2Xt/avunPlfRUZGYvjw4VB5TM1i/3Kxf7msqH/1nTMnIiLTccyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrAMSci0gCOORGRBnDMiYg0gGNORKQBHHMiIg3gmBMRaQDHnIhIAzjmREQawDEnItIAjjkRkQZwzImINIBjTkSkARxzIiIN4JgTEWkAx5yISAM45kREGsAxJyLSAI45EZEGcMyJiDSAY05EpAEccyIiDeCYExFpAMeciEgDOOZERBrgIDvAnxmNRly6dAnnzp1DWloaMjIycPLkSdSqVQuff/45XF1d4eXlBT8/P/j7+8PHx0d2ZE1h/3Kxf7msvX+dEELIOrjRaMSBAwewfft2xMXF4cCBA8jLywMAeHh4wMfHB9WqVYOLiwuysrKQm5uL9PR0FBQUAABq166NLl26ICgoCD179kSDBg1k/VWsEvuXi/3LpbH+o6SMeUpKCpYvX47Q0FCkpaWhfv36CAoKQufOnREYGAg/P78H/ldPCIG0tDScO3cOx44dQ1xcHOLj45GXl4dOnTph9OjRGDZsGNzc3Cz8t7Ie7F8u9i+XRvuPgrCgo0ePimHDhgk7OztRt25dMW3aNHHy5Mkq329RUZHYtGmTGDZsmHB2dhbVq1cXH3/8sbh9+7YZUmsH+5eL/cul8f4jLTLmV69eFSNGjBA6nU60bNlSREZGipKSEkWOdfv2bfHxxx8LHx8f4eHhIebNmyeKi4sVOZa1YP9ysX+5bKR/ZcfcaDSKb7/9Vnh4eIjHHntMrF+/XhiNRiUPWSorK0t8+OGHwsnJSTz55JPi4MGDFjmumrB/udi/XDbWv3Jjfvv2bdGvXz/h4OAgPvjgA5GXl6fUoR7q3Llzonv37sLR0VHMmzfPYv8wZWP/crF/uWywf2XG/MSJE+LRRx8V9evXFwkJCUocwiRGo1F88cUXwsHBQQwaNEjaP1hLYf9ysX+5bLR/8495QkKC8Pb2Fl27dhW3bt0y991Xye7du4WPj4/o2LGjSE9Plx1HEexfLvYvlw33b94xj42NFdWqVRMDBw4U+fn55rxrs0lKShL16tUTLVq00Nw3NPuXi/3LZeP9m2/Mjxw5Ijw8PMSIESMUe6bYXFJTU0X9+vVFx44dNfMjJ/uXi/3Lxf7NNOapqamiZs2aomfPnqKwsNAcd6m4pKQkUb16dTFgwACrf1KI/cvF/uVi/0IIc4x5UVGRaNeunWjevLnIyckxRyiLiY+PF3q9XsyZM0d2lEpj/3Kxf7nYf6mqj/mUKVOEq6urOH36tDkCWdzs2bOFXq8X+/btkx2lUti/XOxfLvZfqmpjfvDgQWFnZyd+/PHHqgaRxmg0ip49e4rAwEBRVFQkO45J2L9c7F8u9l9G5cfcYDCItm3bik6dOmninFu1atXE7NmzZUepMPYvF/uXi/3fp/Jj/uOPPwq9Xi9OnTpVlQCq8cknnwg3Nzdx48YN2VEqhP3Lxf7lYv/3qdyYl5SUCD8/P/Hqq69W9sCqk5ubK2rWrClmzJghO8rfYv9ysX+52H+5Kjfmq1evFvb29uLcuXOVPbAqff7558LT01PcuXNHdpSHYv9ysX+52H+5KjfmnTt3FkOGDKnMH1W1zMxM4ebmJr799lvZUR6K/cvF/uVi/+WKNPkDnVNTUxEfH4+XX35ZiU/LkMrDwwODBg3CypUrZUd5IPYvF/uXi/0/mMljHhoaCl9fX/To0aNSB1S7UaNG4cCBAzh37pzsKOVi/3Kxf7nY/4OZPObR0dHo378/HBwcTD7Yn23YsAGRkZFVug8lBAUFwdvbG9HR0bKjlIv9y8X+5WL/D2HKSZns7Gyh1+vF6tWrK3NOp4zWrVuLxx57rMr3o4SBAweK5557TnaM+7B/udi/XOz/oSJN+s/bnj17UFJSgq5du5r+X42/mDBhAvLz86t8P0oIDg7Ghx9+CCEEdDqd7Dil2L9c7F8u9v9wJo35yZMnUadOHfj6+poc8K9Gjx5d5fsAACEEAJj1m65Vq1bIyMjA1atXUa9ePbPdb1Wxf7nYv1zs/+FMOmeenJwMf39/k8OV5+233y7zjPRrr72GCRMm4Pfff8fzzz+PBg0aoHHjxnjllVeQm5t7358/fvw4unfvDi8vL1SrVg1t27Y123k+Pz8/AHf/vmrC/uVi/3Kx/79hykmZLl26iDfeeMPUcznl+us5q9atW4uGDRuKunXrio4dO4pp06aJLl26CABi0KBBZf5sXFyccHZ2FnXr1hWTJk0Sr7zyivD09BQODg5iz549Zsnn7e0tFi1aZJb7Mhf2Lxf7l4v9P5Rpbxpq3ry5+PDDD01L9QDllQlATJ8+vfTCOQaDQTz11FPC09Oz9HYGg0G0aNFCeHp6ivPnz5d+/cyZM0Kn04kXXnjBLPmaNm0qPvvsM7Pcl7mw7pLM4wAABuhJREFUf7nYv1zs/6FMe9NQTk4O3NzcTHvobwIXFxd8/PHHpeef7Ozs0KFDB2RmZuLKlSsAgKNHj+L48eMYMGAAmjRpUvpnAwIC8M0336BNmzZmyeLu7o7s7Gyz3Je5sH+52L9c7P/hTHoCNDc3F66uriYdwBQ1a9aEs7Nzma95e3sDuPsPEgBSUlIAAM2bN7/vz0+YMMFsWdzd3UuPqRbsXy72Lxf7fziTHpnr9XoUFRWZdABTuLi4PPD3xP89a3zz5k0AQN26dRXLAQCFhYVwcnJS9BimYv9ysX+52P/DmTTmavjRq2HDhgCAAwcO3Pd7K1aswLJly8xynOzsbLi7u5vlvsyF/cvF/uVi/w9n8phnZWWZdABz+8c//gEXFxfExsaW+frp06cxZswY7Nq1yyzHycrKUuU3M/uXh/3Lxf4fzqQxf/TRR5GammrSAczN19cXEydOxIkTJ/DGG2/g8OHDWLFiBUaOHAkHBwe88cYbVT5GYWEhfv/9dzRo0MAMic2H/cvF/uVi/w9n0hOgAQEB2LBhg0kHUMLMmTMhhMCcOXOwePFiAEDt2rWxatUqtG3btsr3n5KSAoPBYLY3KJgL+5eL/cvF/v+GKS9kDA0NFc7Ozqr5FO+cnByxd+9ecerUKVFYWGi2+42KihJ2dnYiLy/PbPdpDuxfLvYvF/t/KNNeZ96uXTsUFBTg4MGDpv0XQyGurq5o3749AgMD4ejoaLb73b17N1q1avXQZ7dlYP9ysX+52P/DmTTmTZo0QYMGDe47+a81O3bsQHBwsOwY92H/crF/udj/w5n84RRBQUHYunWryQeyFpcuXcKZM2fQrVs32VHKxf7lYv9ysf8HM3nMhw0bhr1796r2Y6WqauXKlfDx8UFQUJDsKOVi/3Kxf7nY/4OZPOY9evRA7dq1ERYWZvLBrEFYWBhGjhxp1nNg5sT+5WL/crH/h6jMs63Tp08XderUEQUFBZX546oVExMjAIjDhw/LjvJQ7F8u9i8X+y+XaZfAvefatWvCxcVFfP/995X546rVpUsX0aNHD9kx/hb7l4v9y8X+y1W5MRdCiHHjxomGDRuK/Pz8yt6FqsTFxQkAYufOnbKjVAj7l4v9y8X+71P5Mb98+bJwc3MTn3zySWXvQjWKi4tF8+bNRa9evWRHqTD2Lxf7l4v936fyYy6EEHPmzBHOzs4iJSWlKncj3dy5c4Wzs3OZTw6xBuxfLvYvF/svo2pjXlRUJJ588knRvn171bzF1lQnTpwQLi4uVvlfePYvF/uXi/2XUbUxF0KI5ORk4e7uLqZOnVrVu7K4nJwc8fjjj4tOnTqJ4uJi2XEqhf3Lxf7lYv+lqj7mQgixbNkyodPpRGhoqDnuziJKSkrEgAEDxCOPPCKuXLkiO06VsH+52L9c7F8IYa4xF0KIadOmCb1eL7Zs2WKuu1TUhAkThIuLi4iPj5cdxSzYv1zsXy72b8YxNxqNYsyYMaJatWoiOjraXHdrdgaDQbz99tvCwcFBbNy4UXYcs2H/crF/udi/GcdciLsvsRk9erTQ6/Vi5cqV5rxrsygsLBQjRowQTk5OIioqSnYcs2P/crF/uWy8f/OOuRB3/ws5depUodPpxPTp01XzxEpaWpp45plnhIeHh4iNjZUdRzHsXy72L5cN92/+Mb9n6dKlolq1aqJDhw7i4sWLSh2mQtauXSt8fHxEYGCgSEpKkprFUti/XOxfLhvsX7kxF0KIU6dOiWbNmolq1aqJzz77zOIXxvntt99E//79BQDxyiuviNzcXIseXzb2Lxf7l8vG+ld2zIW4e57oiy++EK6urqJx48bixx9/NOvn5ZXn6tWrYtKkScLFxUUEBASI7du3K3o8NWP/crF/uWyof+XH/J60tDQxduxY4ejoKB599FHxxRdfiEuXLpnt/o1Go0hISBCvvfaacHZ2FnXr1hVfffWV4v/grAX7l4v9y2UD/VtuzO+5dOmSmDhxoqhevbqws7MTQUFBYvbs2eLw4cPCYDCYdF/Z2dliy5YtYvLkyaJRo0YCgHjyySfFd999p7lrHZsL+5eL/cul4f4jdUIIUfXPxzBdUVERoqOjERERgR07duDGjRvw8PCAv78//P390bhxY7i5ucHDwwNOTk7Iy8tDVlYWbt26hfPnzyM5ORkpKSkwGAwIDAxEr1698OKLL6JFixYy/jpWh/3Lxf7l0mD/UdLG/M+EEDh16lTpZ/udPXsWqampyMnJQXZ2NvLz8+Hq6gpPT0/4+PigcePG8Pf3R7NmzdC5c2f4+vrK/itYNfYvF/uXSyP9q2PMiYioSqJM/kBnIiJSH445EZEGcMyJiDTAAcCXskMQEVGVnPz/KIQySWe+aSIAAAAASUVORK5CYII=", 577 | "text/plain": [ 578 | "" 579 | ] 580 | }, 581 | "execution_count": 19, 582 | "metadata": {}, 583 | "output_type": "execute_result" 584 | } 585 | ], 586 | "source": [ 587 | "result = (inc(5) * inc(7)) + (inc(3) * inc(2))\n", 588 | "result.visualize()" 589 | ] 590 | }, 591 | { 592 | "cell_type": "code", 593 | "execution_count": 20, 594 | "id": "93cee5c9", 595 | "metadata": {}, 596 | "outputs": [ 597 | { 598 | "name": "stdout", 599 | "output_type": "stream", 600 | "text": [ 601 | "CPU times: user 1.54 ms, sys: 1.39 ms, total: 2.93 ms\n", 602 | "Wall time: 1.01 s\n" 603 | ] 604 | }, 605 | { 606 | "data": { 607 | "text/plain": [ 608 | "60" 609 | ] 610 | }, 611 | "execution_count": 20, 612 | "metadata": {}, 613 | "output_type": "execute_result" 614 | } 615 | ], 616 | "source": [ 617 | "%%time\n", 618 | "result.compute()" 619 | ] 620 | }, 621 | { 622 | "cell_type": "markdown", 623 | "id": "c5c755fb", 624 | "metadata": {}, 625 | "source": [ 626 | "## Extra resources\n", 627 | "\n", 628 | "For more examples on `dask.delayed` check:\n", 629 | "- Main Dask tutorial: [Delayed lesson](https://github.com/dask/dask-tutorial/blob/main/01_dask.delayed.ipynb)\n", 630 | "- More examples on Delayed: [PyData global - Dask tutorial - Delayed](https://github.com/coiled/pydata-global-dask/blob/master/1-delayed.ipynb)\n", 631 | "- Short screencast on Dask delayed: [How to parallelize Python code with Dask Delayed (3min)](https://www.youtube.com/watch?v=-EUlNJI2QYs)\n", 632 | "- [Dask Delayed documentation](https://docs.dask.org/en/latest/delayed.html)\n", 633 | "- [Delayed Best Practices](https://docs.dask.org/en/latest/delayed-best-practices.html)\n" 634 | ] 635 | } 636 | ], 637 | "metadata": { 638 | "kernelspec": { 639 | "display_name": "Python 3 (ipykernel)", 640 | "language": "python", 641 | "name": "python3" 642 | }, 643 | "language_info": { 644 | "codemirror_mode": { 645 | "name": "ipython", 646 | "version": 3 647 | }, 648 | "file_extension": ".py", 649 | "mimetype": "text/x-python", 650 | "name": "python", 651 | "nbconvert_exporter": "python", 652 | "pygments_lexer": "ipython3", 653 | "version": "3.9.7" 654 | } 655 | }, 656 | "nbformat": 4, 657 | "nbformat_minor": 5 658 | } 659 | -------------------------------------------------------------------------------- /notebooks/2_Schedulers.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "ba6b2bc0", 6 | "metadata": {}, 7 | "source": [ 8 | "\"Dask\n", 11 | " \n", 12 | "This notebook was inspired in the materials from: \n", 13 | "\n", 14 | "- https://github.com/coiled/pydata-global-dask/\n", 15 | "\n", 16 | " \n", 17 | "# Schedulers\n", 18 | "\n", 19 | "So far we have only seen the power of `dask.delayed` and we got familiarized with the idea of task graphs and we learn that these task graphs need to be executed to get the results of our computation. But what does it mean \"to be executed\"? Who takes care of this? Well, as you might have guess from the title of this notebook, this is the job of the Dask task scheduler. \n", 20 | "\n", 21 | "\n", 22 | "\"Grid\n", 25 | "\n", 26 | "\n", 27 | "There are different task schedulers in Dask, and even though they will all compute the same result, but they might have different performances. There are two different classes of schedulers: single-machine and distributed schedulers.\n", 28 | "\n", 29 | "\n", 30 | "## Single Machine Schedulers\n", 31 | "\n", 32 | "Single machine schedulers require no setup, they only use the Python standard library, and they provide basic features on on a local process or threadpool. Dask provides different single machine schedulers:\n", 33 | "\n", 34 | "\n", 35 | "- \"threads\": The threaded scheduler executes computations with a local `concurrent.futures.ThreadPoolExecutor`. The threaded scheduler is the default choice for Dask arrays, Dask DataFrames, and Dask delayed.\n", 36 | "\n", 37 | "- \"processes\": The multiprocessing scheduler executes computations with a local `concurrent.futures.ProcessPoolExecutor`. The multiprocessing scheduler is the default choice for Dask Bag.\n", 38 | "\n", 39 | "- \"single-threaded\": The single-threaded synchronous scheduler executes all computations in the local thread, with no parallelism at all. This is particularly valuable for debugging and profiling, which are more difficult when using threads or processes.\n", 40 | "\n", 41 | "### Single machine schedulers in action\n", 42 | "\n", 43 | "Using the same examples we used in the Delayed lesson, let's see how we can modify the scheduler and how this affects the performance of our computations. " 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 1, 49 | "id": "e15fbe82", 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "import dask\n", 54 | "from dask import delayed\n", 55 | "from time import sleep" 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": 2, 61 | "id": "d0b88f61", 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "@delayed\n", 66 | "def inc(x):\n", 67 | " \"\"\"Increments x by one\"\"\"\n", 68 | " sleep(1)\n", 69 | " return x + 1" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 3, 75 | "id": "a73e948a", 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "data": { 80 | "text/plain": [ 81 | "Delayed('sum-de7db1d6-1e32-477b-aded-a34ba2c60cd9')" 82 | ] 83 | }, 84 | "execution_count": 3, 85 | "metadata": {}, 86 | "output_type": "execute_result" 87 | } 88 | ], 89 | "source": [ 90 | "data = list(range(8))\n", 91 | "\n", 92 | "results = []\n", 93 | "for i in data:\n", 94 | " y = inc(i) \n", 95 | " results.append(y)\n", 96 | " \n", 97 | "total = delayed(sum)(results)\n", 98 | "total" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "id": "4bf0cf9f", 104 | "metadata": {}, 105 | "source": [ 106 | "### The multi-threading scheduler (default)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 4, 112 | "id": "0d6e3709", 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "name": "stdout", 117 | "output_type": "stream", 118 | "text": [ 119 | "CPU times: user 2.09 ms, sys: 1.19 ms, total: 3.29 ms\n", 120 | "Wall time: 1.01 s\n" 121 | ] 122 | }, 123 | { 124 | "data": { 125 | "text/plain": [ 126 | "36" 127 | ] 128 | }, 129 | "execution_count": 4, 130 | "metadata": {}, 131 | "output_type": "execute_result" 132 | } 133 | ], 134 | "source": [ 135 | "%%time \n", 136 | "dask.config.set(scheduler='threads')\n", 137 | "total.compute()" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 5, 143 | "id": "1bc1197d", 144 | "metadata": {}, 145 | "outputs": [ 146 | { 147 | "name": "stdout", 148 | "output_type": "stream", 149 | "text": [ 150 | "CPU times: user 4.79 ms, sys: 2.42 ms, total: 7.21 ms\n", 151 | "Wall time: 2.01 s\n" 152 | ] 153 | }, 154 | { 155 | "data": { 156 | "text/plain": [ 157 | "36" 158 | ] 159 | }, 160 | "execution_count": 5, 161 | "metadata": {}, 162 | "output_type": "execute_result" 163 | } 164 | ], 165 | "source": [ 166 | "%%time \n", 167 | "dask.config.set(scheduler='threads', num_workers=4) #setting num_workers\n", 168 | "total.compute()" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "id": "6a294197", 174 | "metadata": {}, 175 | "source": [ 176 | "### The multi-process scheduler \n", 177 | "\n", 178 | "Notice that we can also set the scheduler as a context manager " 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 6, 184 | "id": "e158bf20", 185 | "metadata": {}, 186 | "outputs": [ 187 | { 188 | "name": "stdout", 189 | "output_type": "stream", 190 | "text": [ 191 | "CPU times: user 10.6 ms, sys: 19 ms, total: 29.7 ms\n", 192 | "Wall time: 6.19 s\n" 193 | ] 194 | } 195 | ], 196 | "source": [ 197 | "%%time\n", 198 | "with dask.config.set(scheduler='processes'): \n", 199 | " total.compute() " 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "id": "d0403b14", 205 | "metadata": {}, 206 | "source": [ 207 | "### The single-threaded scheduler \n", 208 | "\n", 209 | "Tools like `pdb` do not work well with multi threads or process, but you can work around this by using the single-threaded scheduler when debugging." 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 7, 215 | "id": "4ecb2d51", 216 | "metadata": {}, 217 | "outputs": [ 218 | { 219 | "name": "stdout", 220 | "output_type": "stream", 221 | "text": [ 222 | "CPU times: user 5.29 ms, sys: 1.45 ms, total: 6.74 ms\n", 223 | "Wall time: 8.04 s\n" 224 | ] 225 | }, 226 | { 227 | "data": { 228 | "text/plain": [ 229 | "36" 230 | ] 231 | }, 232 | "execution_count": 7, 233 | "metadata": {}, 234 | "output_type": "execute_result" 235 | } 236 | ], 237 | "source": [ 238 | "%%time\n", 239 | "total.compute(scheduler=\"single-threaded\") " 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "id": "b88c0692", 245 | "metadata": {}, 246 | "source": [ 247 | "For more information about single-machine schedulers, and which one to choose you can visit the detailed the Dask documentation on [single-machine schedulers](https://docs.dask.org/en/latest/setup/single-machine.html). " 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "id": "0ffcbd5f", 253 | "metadata": {}, 254 | "source": [ 255 | "## Distributed Scheduler\n", 256 | "\n", 257 | "The Dask distributed scheduler, despite having \"distributed\" in its name, also works well on a single machine. We recommend using the distributed scheduler as it offers more features and diagnostics. You can think of the distributed scheduler as an \"advanced scheduler\". \n", 258 | "\n", 259 | "The distributed scheduler can be used in a cluster as well as locally. Deploying a remote Dask cluster involves additional setup that you can read more about on the Dask [setup documentation](https://docs.dask.org/en/latest/setup.html). Alternatively, you can use [Coiled](https://docs.coiled.io/user_guide/index.html#what-is-coiled) which provides a cluster-as-a-service functionality to provision hosted Dask clusters on demand, and you can try it for free. \n", 260 | "\n", 261 | "For now, we will set up the scheduler locally. To set up the distributed scheduler locally we need to create a `Client` object, which will let you interact with the \"cluster\" (local threads or processes on your machine)" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 8, 267 | "id": "5cd38299", 268 | "metadata": {}, 269 | "outputs": [], 270 | "source": [ 271 | "from dask.distributed import Client" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": 9, 277 | "id": "0354e700", 278 | "metadata": {}, 279 | "outputs": [ 280 | { 281 | "data": { 282 | "text/html": [ 283 | "
\n", 284 | "
\n", 285 | "
\n", 286 | "

Client

\n", 287 | "

Client-3142c864-167e-11ec-9fb4-1e00ea0a0276

\n", 288 | " \n", 289 | "\n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | "\n", 297 | " \n", 298 | " \n", 299 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | "\n", 306 | "
Connection method: Cluster objectCluster type: distributed.LocalCluster
\n", 300 | " Dashboard: http://127.0.0.1:8787/status\n", 301 | "
\n", 307 | "\n", 308 | " \n", 309 | "
\n", 310 | "

Cluster Info

\n", 311 | "
\n", 312 | "
\n", 313 | "
\n", 314 | "
\n", 315 | "

LocalCluster

\n", 316 | "

a6338b33

\n", 317 | " \n", 318 | " \n", 319 | " \n", 322 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 330 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | "\n", 339 | "\n", 340 | " \n", 341 | "
\n", 320 | " Dashboard: http://127.0.0.1:8787/status\n", 321 | " \n", 323 | " Workers: 4\n", 324 | "
\n", 328 | " Total threads: 8\n", 329 | " \n", 331 | " Total memory: 16.00 GiB\n", 332 | "
Status: runningUsing processes: True
\n", 342 | "\n", 343 | "
\n", 344 | " \n", 345 | "

Scheduler Info

\n", 346 | " \n", 347 | "\n", 348 | "
\n", 349 | "
\n", 350 | "
\n", 351 | "
\n", 352 | "

Scheduler

\n", 353 | "

Scheduler-3d8c0db7-910f-4c72-8c3f-79d76d73f0b5

\n", 354 | " \n", 355 | " \n", 356 | " \n", 359 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 367 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 375 | " \n", 378 | " \n", 379 | "
\n", 357 | " Comm: tcp://127.0.0.1:57649\n", 358 | " \n", 360 | " Workers: 4\n", 361 | "
\n", 365 | " Dashboard: http://127.0.0.1:8787/status\n", 366 | " \n", 368 | " Total threads: 8\n", 369 | "
\n", 373 | " Started: Just now\n", 374 | " \n", 376 | " Total memory: 16.00 GiB\n", 377 | "
\n", 380 | "
\n", 381 | "
\n", 382 | "\n", 383 | "
\n", 384 | " \n", 385 | "

Workers

\n", 386 | " \n", 387 | "\n", 388 | " \n", 389 | "
\n", 390 | "
\n", 391 | "
\n", 392 | "
\n", 393 | " \n", 394 | "

Worker: 0

\n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 401 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 409 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 423 | " \n", 424 | "\n", 425 | " \n", 426 | "\n", 427 | " \n", 428 | "\n", 429 | "
\n", 399 | " Comm: tcp://127.0.0.1:57665\n", 400 | " \n", 402 | " Total threads: 2\n", 403 | "
\n", 407 | " Dashboard: http://127.0.0.1:57666/status\n", 408 | " \n", 410 | " Memory: 4.00 GiB\n", 411 | "
\n", 415 | " Nanny: tcp://127.0.0.1:57652\n", 416 | "
\n", 421 | " Local directory: /Users/ncclementi/Documents/git/dask-mini-tutorial/notebooks/dask-worker-space/worker-dop_ge5z\n", 422 | "
\n", 430 | "
\n", 431 | "
\n", 432 | "
\n", 433 | " \n", 434 | "
\n", 435 | "
\n", 436 | "
\n", 437 | "
\n", 438 | " \n", 439 | "

Worker: 1

\n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 446 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 454 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 468 | " \n", 469 | "\n", 470 | " \n", 471 | "\n", 472 | " \n", 473 | "\n", 474 | "
\n", 444 | " Comm: tcp://127.0.0.1:57668\n", 445 | " \n", 447 | " Total threads: 2\n", 448 | "
\n", 452 | " Dashboard: http://127.0.0.1:57669/status\n", 453 | " \n", 455 | " Memory: 4.00 GiB\n", 456 | "
\n", 460 | " Nanny: tcp://127.0.0.1:57654\n", 461 | "
\n", 466 | " Local directory: /Users/ncclementi/Documents/git/dask-mini-tutorial/notebooks/dask-worker-space/worker-1wivjaz7\n", 467 | "
\n", 475 | "
\n", 476 | "
\n", 477 | "
\n", 478 | " \n", 479 | "
\n", 480 | "
\n", 481 | "
\n", 482 | "
\n", 483 | " \n", 484 | "

Worker: 2

\n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 491 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 499 | " \n", 502 | " \n", 503 | " \n", 504 | " \n", 507 | " \n", 508 | " \n", 509 | " \n", 510 | " \n", 513 | " \n", 514 | "\n", 515 | " \n", 516 | "\n", 517 | " \n", 518 | "\n", 519 | "
\n", 489 | " Comm: tcp://127.0.0.1:57660\n", 490 | " \n", 492 | " Total threads: 2\n", 493 | "
\n", 497 | " Dashboard: http://127.0.0.1:57662/status\n", 498 | " \n", 500 | " Memory: 4.00 GiB\n", 501 | "
\n", 505 | " Nanny: tcp://127.0.0.1:57651\n", 506 | "
\n", 511 | " Local directory: /Users/ncclementi/Documents/git/dask-mini-tutorial/notebooks/dask-worker-space/worker-l9wsw5bz\n", 512 | "
\n", 520 | "
\n", 521 | "
\n", 522 | "
\n", 523 | " \n", 524 | "
\n", 525 | "
\n", 526 | "
\n", 527 | "
\n", 528 | " \n", 529 | "

Worker: 3

\n", 530 | " \n", 531 | " \n", 532 | " \n", 533 | " \n", 536 | " \n", 539 | " \n", 540 | " \n", 541 | " \n", 544 | " \n", 547 | " \n", 548 | " \n", 549 | " \n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 558 | " \n", 559 | "\n", 560 | " \n", 561 | "\n", 562 | " \n", 563 | "\n", 564 | "
\n", 534 | " Comm: tcp://127.0.0.1:57659\n", 535 | " \n", 537 | " Total threads: 2\n", 538 | "
\n", 542 | " Dashboard: http://127.0.0.1:57661/status\n", 543 | " \n", 545 | " Memory: 4.00 GiB\n", 546 | "
\n", 550 | " Nanny: tcp://127.0.0.1:57653\n", 551 | "
\n", 556 | " Local directory: /Users/ncclementi/Documents/git/dask-mini-tutorial/notebooks/dask-worker-space/worker-6c3pjpi7\n", 557 | "
\n", 565 | "
\n", 566 | "
\n", 567 | "
\n", 568 | " \n", 569 | "\n", 570 | "
\n", 571 | "
\n", 572 | "\n", 573 | "
\n", 574 | "
\n", 575 | "
\n", 576 | "
\n", 577 | " \n", 578 | "\n", 579 | "
\n", 580 | "
" 581 | ], 582 | "text/plain": [ 583 | "" 584 | ] 585 | }, 586 | "execution_count": 9, 587 | "metadata": {}, 588 | "output_type": "execute_result" 589 | } 590 | ], 591 | "source": [ 592 | "client = Client(n_workers=4)\n", 593 | "client" 594 | ] 595 | }, 596 | { 597 | "cell_type": "markdown", 598 | "id": "8dc55a56", 599 | "metadata": {}, 600 | "source": [ 601 | "When we create a distributed scheduler `Client`, by default it registers itself as the default Dask scheduler. From now on, all `.compute()` calls will start using the distributed scheduler unless otherwise is specified. \n", 602 | "\n", 603 | "The distributed scheduler has many features that you can learn more about in the [Dask distributed documentation](https://distributed.dask.org/en/latest/) but a nice feature to explore is diagnostic the Dashboard. We will be taking a look at the dashboard as we perform computations but for a brief overview of the main components of the dashboard you can check the Dask documentation on [diagnosing performance](https://distributed.dask.org/en/latest/diagnosing-performance.html).\n", 604 | "\n", 605 | "If you click on the link of the dashboard on the cell above and run the computation of `total` as we did before you will see now some action happening on the dashboard. " 606 | ] 607 | }, 608 | { 609 | "cell_type": "code", 610 | "execution_count": 10, 611 | "id": "3c9e1199", 612 | "metadata": {}, 613 | "outputs": [ 614 | { 615 | "data": { 616 | "text/plain": [ 617 | "36" 618 | ] 619 | }, 620 | "execution_count": 10, 621 | "metadata": {}, 622 | "output_type": "execute_result" 623 | } 624 | ], 625 | "source": [ 626 | "total.compute()" 627 | ] 628 | }, 629 | { 630 | "cell_type": "code", 631 | "execution_count": null, 632 | "id": "adfd314d", 633 | "metadata": {}, 634 | "outputs": [], 635 | "source": [ 636 | "client.close()" 637 | ] 638 | }, 639 | { 640 | "cell_type": "markdown", 641 | "id": "ffbd7734", 642 | "metadata": {}, 643 | "source": [ 644 | "## Extra resources\n", 645 | "\n", 646 | "- [Dask documentation on scheduling](https://docs.dask.org/en/latest/scheduling.html)\n", 647 | "- Example Dynamic computations using Futures: [PyData Global Dask tutorial - schedulers](https://github.com/coiled/pydata-global-dask/blob/master/3-schedulers.ipynb)\n", 648 | "- Advance Delayed with distributed scheduler: [Dask tutorial - Advanced delayed](https://github.com/dask/dask-tutorial/blob/main/06_distributed_advanced.ipynb)" 649 | ] 650 | } 651 | ], 652 | "metadata": { 653 | "kernelspec": { 654 | "display_name": "Python 3 (ipykernel)", 655 | "language": "python", 656 | "name": "python3" 657 | }, 658 | "language_info": { 659 | "codemirror_mode": { 660 | "name": "ipython", 661 | "version": 3 662 | }, 663 | "file_extension": ".py", 664 | "mimetype": "text/x-python", 665 | "name": "python", 666 | "nbconvert_exporter": "python", 667 | "pygments_lexer": "ipython3", 668 | "version": "3.9.7" 669 | } 670 | }, 671 | "nbformat": 4, 672 | "nbformat_minor": 5 673 | } 674 | -------------------------------------------------------------------------------- /notebooks/4_Machine_learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "33012a14", 6 | "metadata": {}, 7 | "source": [ 8 | "\"Dask\n", 11 | "\n", 12 | "# Parallel and Distributed Machine Learning\n", 13 | "\n", 14 | "The material in this notebook was based on the open-source content from [Dask's tutorial repository](https://github.com/dask/dask-tutorial) and the [Machine learning notebook](https://github.com/coiled/data-science-at-scale/blob/master/3-machine-learning.ipynb) from data science at scale from coiled\n", 15 | "\n", 16 | "So far we have seen how Dask makes data analysis scalable with parallelization via Dask DataFrames. Let's now see how [Dask-ML](https://ml.dask.org/) allows us to do machine learning in a parallel and distributed manner. Note, machine learning is really just a special case of data analysis (one that automates analytical model building), so the 💪 Dask gains 💪 we've seen will apply here as well!\n", 17 | "\n", 18 | "(If you'd like a refresher on the difference between parallel and distributed computing, [here's a good discussion on StackExchange](https://cs.stackexchange.com/questions/1580/distributed-vs-parallel-computing).)\n", 19 | "\n", 20 | "\n", 21 | "## Types of scaling problems in machine learning\n", 22 | "\n", 23 | "There are two main types of scaling challenges you can run into in your machine learning workflow: scaling the **size of your data** and scaling the **size of your model**. That is:\n", 24 | "\n", 25 | "1. **CPU-bound problems**: Data fits in RAM, but training takes too long. Many hyperparameter combinations, a large ensemble of many models, etc.\n", 26 | "2. **Memory-bound problems**: Data is larger than RAM, and sampling isn't an option.\n", 27 | "\n", 28 | "Here's a handy diagram for visualizing these problems:\n", 29 | "\n", 30 | "\"scaling\n", 33 | "\n", 34 | "\n", 35 | "In the bottom-left quadrant, your datasets are not too large (they fit comfortably in RAM) and your model is not too large either. When these conditions are met, you are much better off using something like scikit-learn, XGBoost, and similar libraries. You don't need to leverage multiple machines in a distributed manner with a library like Dask-ML. However, if you are in any of the other quadrants, distributed machine learning is the way to go.\n", 36 | "\n", 37 | "Summarizing: \n", 38 | "\n", 39 | "* For in-memory problems, just use scikit-learn (or your favorite ML library).\n", 40 | "* For large models, use `dask_ml.joblib` and your favorite scikit-learn estimator.\n", 41 | "* For large datasets, use `dask_ml` estimators.\n", 42 | "\n", 43 | "## Scikit-learn in five minutes\n", 44 | "\n", 45 | "\"sklearn\n", 48 | "\n", 49 | "\n", 50 | "In this section, we'll quickly run through a typical scikit-learn workflow:\n", 51 | "\n", 52 | "* Load some data (in this case, we'll generate it)\n", 53 | "* Import the scikit-learn module for our chosen ML algorithm\n", 54 | "* Create an estimator for that algorithm and fit it with our data\n", 55 | "* Inspect the learned attributes\n", 56 | "* Check the accuracy of our model\n", 57 | "\n", 58 | "Scikit-learn has a nice, consistent API:\n", 59 | "\n", 60 | "* You instantiate an `Estimator` (e.g. `LinearRegression`, `RandomForestClassifier`, etc.). All of the models *hyperparameters* (user-specified parameters, not the ones learned by the estimator) are passed to the estimator when it's created.\n", 61 | "* You call `estimator.fit(X, y)` to train the estimator.\n", 62 | "* Use `estimator` to inspect attributes, make predictions, etc. \n", 63 | "\n", 64 | "Here `X` is an array of *feature variables* (what you're using to predict) and `y` is an array of *target variables* (what we're trying to predict)." 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "id": "b6e7379d", 70 | "metadata": {}, 71 | "source": [ 72 | "### Generate some random data" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "id": "8613c202", 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "from sklearn.datasets import make_classification\n", 83 | "\n", 84 | "# Generate data\n", 85 | "X, y = make_classification(n_samples=10000, n_features=4, random_state=0)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "id": "1fbcf18d", 91 | "metadata": {}, 92 | "source": [ 93 | "**Refreshing some ML concepts**\n", 94 | "\n", 95 | "- `X` is the samples matrix (or design matrix). The size of `X` is typically (`n_samples`, `n_features`), which means that samples are represented as rows and features are represented as columns.\n", 96 | "- A \"feature\" (also called an \"attribute\") is a measurable property of the phenomenon we're trying to analyze. A feature for a dataset of employees might be their hire date, for example.\n", 97 | "- `y` are the target values, which are real numbers for regression tasks, or integers for classification (or any other discrete set of values). For unsupervized learning tasks, `y` does not need to be specified. `y` is usually 1d array where the `i`th entry corresponds to the target of the `i`th sample (row) of `X`." 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "id": "85824219", 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "# Let's take a look at X\n", 108 | "X[:8]" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "id": "7dd8ef9a", 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "# Let's take a look at y\n", 119 | "y[:8]" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "id": "80d61bf2", 125 | "metadata": {}, 126 | "source": [ 127 | "### Fitting and SVC\n", 128 | "\n", 129 | "For this example, we will fit a [Support Vector Classifier](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)." 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "id": "5d638ce9", 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [ 139 | "from sklearn.svm import SVC\n", 140 | "\n", 141 | "estimator = SVC(random_state=0)\n", 142 | "estimator.fit(X, y)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "id": "4068406a", 148 | "metadata": {}, 149 | "source": [ 150 | "We can inspect the learned features by taking a look a the `support_vectors_`:" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "id": "b891e47a", 157 | "metadata": {}, 158 | "outputs": [], 159 | "source": [ 160 | "estimator.support_vectors_[:4]" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "id": "a175852d", 166 | "metadata": {}, 167 | "source": [ 168 | "And we check the accuracy:" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "id": "8300ec8e", 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "estimator.score(X, y)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "id": "1efac1bd", 184 | "metadata": {}, 185 | "source": [ 186 | "There are [3 different approaches](https://scikit-learn.org/0.15/modules/model_evaluation.html) to evaluate the quality of predictions of a model. One of them is the **estimator score method**. Estimators have a score method providing a default evaluation criterion for the problem they are designed to solve, which is discussed in each estimator's documentation.\n", 187 | "\n", 188 | "### Hyperparameter Optimization\n", 189 | "\n", 190 | "There are a few ways to learn the best *hyper*parameters while training. One is `GridSearchCV`.\n", 191 | "As the name implies, this does a brute-force search over a grid of hyperparameter combinations. scikit-learn provides tools to automatically find the best parameter combinations via cross-validation (which is the \"CV\" in `GridSearchCV`)." 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "id": "b4659297", 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "from sklearn.model_selection import GridSearchCV" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "id": "f6c889f4", 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [ 211 | "%%time\n", 212 | "estimator = SVC(gamma='auto', random_state=0, probability=True)\n", 213 | "param_grid = {\n", 214 | " 'C': [0.001, 10.0],\n", 215 | " 'kernel': ['rbf', 'poly'],\n", 216 | "}\n", 217 | "\n", 218 | "# Brute-force search over a grid of hyperparameter combinations\n", 219 | "grid_search = GridSearchCV(estimator, param_grid, verbose=2, cv=2)\n", 220 | "grid_search.fit(X, y)" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": null, 226 | "id": "ad134157", 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "grid_search.best_params_, grid_search.best_score_" 231 | ] 232 | }, 233 | { 234 | "cell_type": "markdown", 235 | "id": "228deb1b", 236 | "metadata": {}, 237 | "source": [ 238 | "## Compute Bound: Single-machine parallelism with Joblib\n", 239 | "\n", 240 | "\"Joblib\n", 243 | "\n", 244 | "In this section we'll see how [Joblib](https://joblib.readthedocs.io/en/latest/) (\"*a set of tools to provide lightweight pipelining in Python*\") gives us parallelism on our laptop. Here's what our grid search graph would look like if we set up six training \"jobs\" in parallel:\n", 245 | "\n", 246 | "\"grid\n", 249 | "\n", 250 | "With Joblib, we can say that scikit-learn has *single-machine* parallelism.\n", 251 | "Any scikit-learn estimator that can operate in parallel exposes an `n_jobs` keyword, which tells you how many tasks to run in parallel. Specifying `n_jobs=-1` jobs means running the maximum possible number of tasks in parallel." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "id": "d9bead16", 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "%%time\n", 262 | "grid_search = GridSearchCV(estimator, param_grid, verbose=2, cv=2, n_jobs=-1)\n", 263 | "grid_search.fit(X, y)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "id": "e7c73f87", 269 | "metadata": {}, 270 | "source": [ 271 | "Notice that the computation above it is faster than before. If you are running this computation on binder, you might not see a speed-up and the reason for that is that binder instances tend to have only one core with no threads so you can't see any parallelism. " 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "id": "1fe6255f", 277 | "metadata": {}, 278 | "source": [ 279 | "## Compute Bound: Multi-machine parallelism with Dask\n", 280 | "\n", 281 | "\n", 282 | "In this section we'll see how Dask (plus Joblib and scikit-learn) gives us multi-machine parallelism. Here's what our grid search graph would look like if we allowed Dask to schedule our training \"jobs\" over multiple machines in our cluster:\n", 283 | "\n", 284 | "\"merged\n", 287 | " \n", 288 | "We can say that Dask can talk to scikit-learn (via Joblib) so that our *cluster* is used to train a model. \n", 289 | "\n", 290 | "If we run this on a laptop, it will take quite some time, but the CPU usage will be satisfyingly near 100% for the duration. To run faster, we would need a distributed cluster. For details on how to create a LocalCluster you can check the Dask documentation on [Single Machine: dask.distributed](https://docs.dask.org/en/latest/setup/single-distributed.html). \n", 291 | "\n", 292 | "Let's instantiate a Client with `n_workers=4`, which will give us a `LocalCluster`." 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "id": "cdf776a5", 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "import dask.distributed\n", 303 | "\n", 304 | "client = dask.distributed.Client(n_workers=4)\n", 305 | "client" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "id": "bb9c77aa", 311 | "metadata": {}, 312 | "source": [ 313 | "**Note:** Click on Cluster Info, to see more details about the cluster. You can see the configuration of the cluster and some other specs. \n", 314 | "\n", 315 | "We can expand our problem by specifying more hyperparameters before training, and see how using `dask` as backend can help us. " 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "id": "367f1c5a", 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "param_grid = {\n", 326 | " 'C': [0.001, 0.1, 1.0, 2.5, 5, 10.0],\n", 327 | " 'kernel': ['rbf', 'poly', 'linear'],\n", 328 | " 'shrinking': [True, False],\n", 329 | "}\n", 330 | "\n", 331 | "grid_search = GridSearchCV(estimator, param_grid, verbose=2, cv=2, n_jobs=-1)" 332 | ] 333 | }, 334 | { 335 | "cell_type": "markdown", 336 | "id": "2854c927", 337 | "metadata": {}, 338 | "source": [ 339 | "### Dask parallel backend\n", 340 | "\n", 341 | "We can fit our estimator with multi-machine parallelism by quickly *switching to a Dask parallel backend* when using joblib. " 342 | ] 343 | }, 344 | { 345 | "cell_type": "code", 346 | "execution_count": null, 347 | "id": "efb681a4", 348 | "metadata": {}, 349 | "outputs": [], 350 | "source": [ 351 | "import joblib" 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": null, 357 | "id": "4d8228e2", 358 | "metadata": {}, 359 | "outputs": [], 360 | "source": [ 361 | "%%time\n", 362 | "with joblib.parallel_backend(\"dask\", scatter=[X, y]):\n", 363 | " grid_search.fit(X, y)" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "id": "4b9194cf", 369 | "metadata": {}, 370 | "source": [ 371 | "**What did just happen?**\n", 372 | "\n", 373 | "Dask-ML developers worked with the scikit-learn and Joblib developers to implement a Dask parallel backend. So internally, scikit-learn now talks to Joblib, and Joblib talks to Dask, and Dask is what handles scheduling all of those tasks on multiple machines.\n", 374 | "\n", 375 | "The best parameters and best score:" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": null, 381 | "id": "626f0886", 382 | "metadata": {}, 383 | "outputs": [], 384 | "source": [ 385 | "grid_search.best_params_, grid_search.best_score_" 386 | ] 387 | }, 388 | { 389 | "cell_type": "markdown", 390 | "id": "e5ea35f8", 391 | "metadata": {}, 392 | "source": [ 393 | "## Memory Bound: Single/Multi machine parallelism with Dask-ML\n", 394 | "\n", 395 | "We have seen how to work with larger models, but sometimes you'll want to train on a larger than memory dataset. `dask-ml` has implemented estimators that work well on Dask `Arrays` and `DataFrames` that may be larger than your machine's RAM." 396 | ] 397 | }, 398 | { 399 | "cell_type": "code", 400 | "execution_count": null, 401 | "id": "01ed30f6", 402 | "metadata": {}, 403 | "outputs": [], 404 | "source": [ 405 | "import dask.array as da\n", 406 | "import dask.delayed\n", 407 | "from sklearn.datasets import make_blobs\n", 408 | "import numpy as np" 409 | ] 410 | }, 411 | { 412 | "cell_type": "markdown", 413 | "id": "3ebd444f", 414 | "metadata": {}, 415 | "source": [ 416 | "We'll make a small (random) dataset locally using scikit-learn." 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "id": "5f703b07", 423 | "metadata": {}, 424 | "outputs": [], 425 | "source": [ 426 | "n_centers = 12\n", 427 | "n_features = 20\n", 428 | "\n", 429 | "X_small, y_small = make_blobs(n_samples=1000, centers=n_centers, n_features=n_features, random_state=0)\n", 430 | "\n", 431 | "centers = np.zeros((n_centers, n_features))\n", 432 | "\n", 433 | "for i in range(n_centers):\n", 434 | " centers[i] = X_small[y_small == i].mean(0)\n", 435 | " \n", 436 | "centers[:4]" 437 | ] 438 | }, 439 | { 440 | "cell_type": "markdown", 441 | "id": "c28f3881", 442 | "metadata": {}, 443 | "source": [ 444 | "**Note**: The small dataset will be the template for our large random dataset.\n", 445 | "We'll use `dask.delayed` to adapt `sklearn.datasets.make_blobs`, so that the actual dataset is being generated on our workers. \n", 446 | "\n", 447 | "If you are not in binder and you machine has 16GB of RAM you can make `n_samples_per_block=200_000` and the computations takes around 10 min. If you are in binder the resources are limited and the problem below is big enough. " 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "id": "17d91e43", 454 | "metadata": {}, 455 | "outputs": [], 456 | "source": [ 457 | "n_samples_per_block = 60_000 #on binder replace this for 15_000\n", 458 | "n_blocks = 500\n", 459 | "\n", 460 | "delayeds = [dask.delayed(make_blobs)(n_samples=n_samples_per_block,\n", 461 | " centers=centers,\n", 462 | " n_features=n_features,\n", 463 | " random_state=i)[0]\n", 464 | " for i in range(n_blocks)]\n", 465 | "arrays = [da.from_delayed(obj, shape=(n_samples_per_block, n_features), dtype=X.dtype)\n", 466 | " for obj in delayeds]\n", 467 | "X = da.concatenate(arrays)\n", 468 | "X" 469 | ] 470 | }, 471 | { 472 | "cell_type": "markdown", 473 | "id": "dc609b5f", 474 | "metadata": {}, 475 | "source": [ 476 | "### KMeans from Dask-ml\n", 477 | "\n", 478 | "The algorithms implemented in Dask-ML are scalable. They handle larger-than-memory datasets just fine.\n", 479 | "\n", 480 | "They follow the scikit-learn API, so if you're familiar with scikit-learn, you'll feel at home with Dask-ML." 481 | ] 482 | }, 483 | { 484 | "cell_type": "code", 485 | "execution_count": null, 486 | "id": "095e644f", 487 | "metadata": {}, 488 | "outputs": [], 489 | "source": [ 490 | "from dask_ml.cluster import KMeans" 491 | ] 492 | }, 493 | { 494 | "cell_type": "code", 495 | "execution_count": null, 496 | "id": "51a4b716", 497 | "metadata": {}, 498 | "outputs": [], 499 | "source": [ 500 | "clf = KMeans(init_max_iter=3, oversampling_factor=10)" 501 | ] 502 | }, 503 | { 504 | "cell_type": "code", 505 | "execution_count": null, 506 | "id": "a0a1e790", 507 | "metadata": {}, 508 | "outputs": [], 509 | "source": [ 510 | "%time clf.fit(X)" 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": null, 516 | "id": "7cfc4c3d", 517 | "metadata": {}, 518 | "outputs": [], 519 | "source": [ 520 | "clf.labels_" 521 | ] 522 | }, 523 | { 524 | "cell_type": "code", 525 | "execution_count": null, 526 | "id": "30535766", 527 | "metadata": {}, 528 | "outputs": [], 529 | "source": [ 530 | "clf.labels_[:10].compute()" 531 | ] 532 | }, 533 | { 534 | "cell_type": "code", 535 | "execution_count": null, 536 | "id": "1d9a83d7", 537 | "metadata": {}, 538 | "outputs": [], 539 | "source": [ 540 | "client.close()" 541 | ] 542 | }, 543 | { 544 | "cell_type": "markdown", 545 | "id": "8103498a", 546 | "metadata": {}, 547 | "source": [ 548 | "## Multi-machine parallelism in the cloud with Coiled\n", 549 | "\n", 550 | "
\n", 551 | "\"Coiled\n", 554 | "
\n", 555 | "\n", 556 | "In this section we'll see how Coiled allows us to solve machine learning problems with multi-machine parallelism in the cloud.\n", 557 | "\n", 558 | "Coiled, [among other things](https://coiled.io/product/), provides hosted and scalable Dask clusters. The biggest barriers to entry for doing machine learning at scale are \"Do you have access to a cluster?\" and \"Do you know how to manage it?\" Coiled solves both of those problems. \n", 559 | "\n", 560 | "We'll spin up a Coiled cluster (with 10 workers in this case), then instantiate a Dask Client to use with that cluster.\n", 561 | "\n", 562 | "If you are running on your local machine and not in binder, and you want to give Coiled a try, you can signup [here](https://cloud.coiled.io/login?redirect_uri=/) and you will get some free credits. If you installed the environment by following the steps on the repository's [README](https://github.com/coiled/dask-mini-tutorial/blob/main/README.md) you will have `coiled` installed. You will just need to login, by following the steps on the [setup page](https://docs.coiled.io/user_guide/getting_started.html), and you will be ready to go. \n", 563 | "\n", 564 | "To learn more about how to set up an environment you can visit Coiled documentation on [Creating software environments](https://docs.coiled.io/user_guide/software_environment_creation.html). But for now you can use the envioronment we set up for this tutorial. " 565 | ] 566 | }, 567 | { 568 | "cell_type": "code", 569 | "execution_count": null, 570 | "id": "58d6d915", 571 | "metadata": {}, 572 | "outputs": [], 573 | "source": [ 574 | "import coiled\n", 575 | "from dask.distributed import Client" 576 | ] 577 | }, 578 | { 579 | "cell_type": "code", 580 | "execution_count": null, 581 | "id": "f4268dc5", 582 | "metadata": {}, 583 | "outputs": [], 584 | "source": [ 585 | "# Spin up a Coiled cluster, instantiate a Client\n", 586 | "cluster = coiled.Cluster(n_workers=10, software=\"ncclementi/dask-mini-tutorial\",)" 587 | ] 588 | }, 589 | { 590 | "cell_type": "code", 591 | "execution_count": null, 592 | "id": "e50ddd7c", 593 | "metadata": {}, 594 | "outputs": [], 595 | "source": [ 596 | "client = Client(cluster)\n", 597 | "client" 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "id": "ff3980aa", 603 | "metadata": {}, 604 | "source": [ 605 | "### Memory bound: Dask-ML" 606 | ] 607 | }, 608 | { 609 | "cell_type": "markdown", 610 | "id": "efccea68", 611 | "metadata": {}, 612 | "source": [ 613 | "We can use Dask-ML estimators on the cloud to work with larger datasets." 614 | ] 615 | }, 616 | { 617 | "cell_type": "code", 618 | "execution_count": null, 619 | "id": "dc52c251", 620 | "metadata": {}, 621 | "outputs": [], 622 | "source": [ 623 | "n_centers = 12\n", 624 | "n_features = 20\n", 625 | "\n", 626 | "X_small, y_small = make_blobs(n_samples=1000, centers=n_centers, n_features=n_features, random_state=0)\n", 627 | "\n", 628 | "centers = np.zeros((n_centers, n_features))\n", 629 | "\n", 630 | "for i in range(n_centers):\n", 631 | " centers[i] = X_small[y_small == i].mean(0)\n" 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "execution_count": null, 637 | "id": "ba1c77b3", 638 | "metadata": {}, 639 | "outputs": [], 640 | "source": [ 641 | "n_samples_per_block = 200_000\n", 642 | "n_blocks = 500\n", 643 | "\n", 644 | "delayeds = [dask.delayed(make_blobs)(n_samples=n_samples_per_block,\n", 645 | " centers=centers,\n", 646 | " n_features=n_features,\n", 647 | " random_state=i)[0]\n", 648 | " for i in range(n_blocks)]\n", 649 | "arrays = [da.from_delayed(obj, shape=(n_samples_per_block, n_features), dtype=X.dtype)\n", 650 | " for obj in delayeds]\n", 651 | "X = da.concatenate(arrays)\n" 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": null, 657 | "id": "d891a7b6", 658 | "metadata": {}, 659 | "outputs": [], 660 | "source": [ 661 | "X = X.persist()" 662 | ] 663 | }, 664 | { 665 | "cell_type": "code", 666 | "execution_count": null, 667 | "id": "9fef03bb", 668 | "metadata": {}, 669 | "outputs": [], 670 | "source": [ 671 | "from dask_ml.cluster import KMeans" 672 | ] 673 | }, 674 | { 675 | "cell_type": "code", 676 | "execution_count": null, 677 | "id": "8a186141", 678 | "metadata": {}, 679 | "outputs": [], 680 | "source": [ 681 | "clf = KMeans(init_max_iter=3, oversampling_factor=10)" 682 | ] 683 | }, 684 | { 685 | "cell_type": "code", 686 | "execution_count": null, 687 | "id": "56241fad", 688 | "metadata": {}, 689 | "outputs": [], 690 | "source": [ 691 | "%time clf.fit(X)" 692 | ] 693 | }, 694 | { 695 | "cell_type": "markdown", 696 | "id": "9a3682c1", 697 | "metadata": {}, 698 | "source": [ 699 | "Computing the labels:" 700 | ] 701 | }, 702 | { 703 | "cell_type": "code", 704 | "execution_count": null, 705 | "id": "f0b9980e", 706 | "metadata": {}, 707 | "outputs": [], 708 | "source": [ 709 | "clf.labels_[:10].compute()" 710 | ] 711 | }, 712 | { 713 | "cell_type": "code", 714 | "execution_count": null, 715 | "id": "9988ef14", 716 | "metadata": {}, 717 | "outputs": [], 718 | "source": [ 719 | "client.close()" 720 | ] 721 | }, 722 | { 723 | "cell_type": "markdown", 724 | "id": "6c5f839e", 725 | "metadata": {}, 726 | "source": [ 727 | "## Extra resources:\n", 728 | "\n", 729 | "- [Dask-ML documentation](https://ml.dask.org/)\n", 730 | "- [Getting started with Coiled](https://docs.coiled.io/user_guide/getting_started.html)" 731 | ] 732 | } 733 | ], 734 | "metadata": { 735 | "kernelspec": { 736 | "display_name": "Python 3 (ipykernel)", 737 | "language": "python", 738 | "name": "python3" 739 | }, 740 | "language_info": { 741 | "codemirror_mode": { 742 | "name": "ipython", 743 | "version": 3 744 | }, 745 | "file_extension": ".py", 746 | "mimetype": "text/x-python", 747 | "name": "python", 748 | "nbconvert_exporter": "python", 749 | "pygments_lexer": "ipython3", 750 | "version": "3.9.7" 751 | } 752 | }, 753 | "nbformat": 4, 754 | "nbformat_minor": 5 755 | } 756 | -------------------------------------------------------------------------------- /prep_data.py: -------------------------------------------------------------------------------- 1 | #This script was modify from original https://github.com/coiled/pydata-global-dask/blob/master/prep.py 2 | import time 3 | import sys 4 | import argparse 5 | import os 6 | from glob import glob 7 | import tarfile 8 | import urllib.request 9 | 10 | import pandas as pd 11 | 12 | 13 | DATASETS = ["flights", "all"] 14 | here = os.path.dirname(__file__) 15 | data_dir = os.path.abspath(os.path.join(here, "data")) 16 | 17 | print(f"{data_dir=}") 18 | 19 | def parse_args(args=None): 20 | parser = argparse.ArgumentParser( 21 | description="Downloads, generates and prepares data for the Dask tutorial." 22 | ) 23 | parser.add_argument( 24 | "--no-ssl-verify", 25 | dest="no_ssl_verify", 26 | action="store_true", 27 | default=False, 28 | help="Disables SSL verification.", 29 | ) 30 | parser.add_argument( 31 | "--small", 32 | action="store_true", 33 | default=None, 34 | help="Whether to use smaller example datasets. Checks DASK_TUTORIAL_SMALL environment variable if not specified.", 35 | ) 36 | parser.add_argument( 37 | "-d", "--dataset", choices=DATASETS, help="Datasets to generate.", default="all" 38 | ) 39 | 40 | return parser.parse_args(args) 41 | 42 | 43 | if not os.path.exists(data_dir): 44 | raise OSError( 45 | "data/ directory not found, aborting data preparation. " 46 | 'Restore it with "git checkout data" from the base ' 47 | "directory." 48 | ) 49 | 50 | 51 | def flights(small=None): 52 | start = time.time() 53 | flights_raw = os.path.join(data_dir, "nycflights.tar.gz") 54 | flightdir = os.path.join(data_dir, "nycflights") 55 | if small is None: 56 | small = bool(os.environ.get("DASK_TUTORIAL_SMALL", False)) 57 | 58 | if small: 59 | N = 500 60 | else: 61 | N = 10_000 62 | 63 | if not os.path.exists(flights_raw): 64 | print("- Downloading NYC Flights dataset... ", end="", flush=True) 65 | url = "https://storage.googleapis.com/dask-tutorial-data/nycflights.tar.gz" 66 | urllib.request.urlretrieve(url, flights_raw) 67 | print("done", flush=True) 68 | 69 | if not os.path.exists(flightdir): 70 | print("- Extracting flight data... ", end="", flush=True) 71 | tar_path = os.path.join(data_dir, "nycflights.tar.gz") 72 | with tarfile.open(tar_path, mode="r:gz") as flights: 73 | flights.extractall(data_dir) 74 | 75 | if small: 76 | for path in glob(os.path.join(data_dir, "nycflights", "*.csv")): 77 | with open(path, "r") as f: 78 | lines = f.readlines()[:1000] 79 | 80 | with open(path, "w") as f: 81 | f.writelines(lines) 82 | 83 | print("done", flush=True) 84 | 85 | else: 86 | return 87 | 88 | end = time.time() 89 | print("** Created flights dataset! in {:0.2f}s**".format(end - start)) 90 | 91 | 92 | def main(args=None): 93 | args = parse_args(args) 94 | 95 | if args.no_ssl_verify: 96 | print("- Disabling SSL Verification... ", end="", flush=True) 97 | import ssl 98 | 99 | ssl._create_default_https_context = ssl._create_unverified_context 100 | print("done", flush=True) 101 | 102 | if args.dataset == "flights" or args.dataset == "all": 103 | flights(args.small) 104 | 105 | 106 | if __name__ == "__main__": 107 | sys.exit(main()) 108 | --------------------------------------------------------------------------------