├── criugm.jpg ├── favicon.png ├── requirements.txt ├── MAIN_BASC064_subsamp_features.npz ├── _config.yml ├── data_fetch.sh ├── Dockerfile ├── sklearn └── requirements.txt ├── LICENSE ├── README.md ├── course-outline.md ├── dl-course-outline.md ├── MAIN_tutorial_intro_to_nilearn.ipynb ├── plot_variance_linear_regr.ipynb ├── MAIN_tutorial_machine_learning_with_nilearn.ipynb └── model_validation.ipynb /criugm.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brainhack101/introML/HEAD/criugm.jpg -------------------------------------------------------------------------------- /favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brainhack101/introML/HEAD/favicon.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | git+https://github.com/KamalakerDadi/nilearn.git@16f2df26401e5b6a16ff134f4da71ff920e3ac40 2 | -------------------------------------------------------------------------------- /MAIN_BASC064_subsamp_features.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brainhack101/introML/HEAD/MAIN_BASC064_subsamp_features.npz -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal 2 | title: IntroML 3 | logo: http://www.crm.umontreal.ca/2018/MAIN2018/img/MAIN2018_Poster_FR_web.jpg 4 | -------------------------------------------------------------------------------- /data_fetch.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | python -c "import nilearn; from nilearn import datasets; datasets.fetch_main(); datasets.fetch_atlas_basc_multiscale_2015()" 3 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM jupyter/datascience-notebook 2 | 3 | LABEL maintainer="Pierre Bellec " 4 | 5 | USER jovyan 6 | 7 | # Copying the repository inside the container 8 | COPY . /home/jovyan 9 | 10 | # Instaling the Kamalaker's main fecther 11 | RUN pip install -r requirements.txt 12 | 13 | # Downloading the data 14 | RUN ["/bin/bash", "/home/jovyan/data_fetch.sh"] 15 | -------------------------------------------------------------------------------- /sklearn/requirements.txt: -------------------------------------------------------------------------------- 1 | appnope==0.1.0 2 | backcall==0.1.0 3 | bleach==3.0.2 4 | cycler==0.10.0 5 | decorator==4.3.0 6 | defusedxml==0.5.0 7 | entrypoints==0.2.3 8 | ipykernel==5.1.0 9 | ipython==7.2.0 10 | ipython-genutils==0.2.0 11 | ipywidgets==7.4.2 12 | jedi==0.13.1 13 | Jinja2==2.10 14 | jsonschema==2.6.0 15 | jupyter==1.0.0 16 | jupyter-client==5.2.3 17 | jupyter-console==6.0.0 18 | jupyter-core==4.4.0 19 | kiwisolver==1.0.1 20 | MarkupSafe==1.1.0 21 | matplotlib==3.0.2 22 | mistune==0.8.4 23 | nbconvert==5.4.0 24 | nbformat==4.4.0 25 | notebook==5.7.2 26 | numpy==1.15.4 27 | pandocfilters==1.4.2 28 | parso==0.3.1 29 | pexpect==4.6.0 30 | pickleshare==0.7.5 31 | prometheus-client==0.4.2 32 | prompt-toolkit==2.0.7 33 | ptyprocess==0.6.0 34 | Pygments==2.3.0 35 | pyparsing==2.3.0 36 | python-dateutil==2.7.5 37 | pyzmq==17.1.2 38 | qtconsole==4.4.3 39 | scikit-learn==0.20.1 40 | scipy==1.1.0 41 | Send2Trash==1.5.0 42 | six==1.11.0 43 | terminado==0.8.1 44 | testpath==0.4.2 45 | tornado==5.1.1 46 | traitlets==4.3.2 47 | wcwidth==0.1.7 48 | webencodings==0.5.1 49 | widgetsnbextension==3.4.2 50 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 BrainHack 101 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## IntroML Resources @ MAIN 2018 2 | 3 | Welcome to the educational workshops @ MAIN 2018! On this page you'll find resources for the courses entitled, 4 | 5 | "[Machine learning for neuroimaging with Scikit-learn and nilearn](./course-outline.md)," 6 | 7 | and, 8 | 9 | "[Deep Learning for Neuroimaging](./dl-course-outline.md)" 10 | 11 | Register on [EventBrite](https://www.eventbrite.ca/e/deep-learning-in-neuroimaging-machine-learning-scikit-learn-nilearn-tickets-53388406160){: .btn} ! 12 | 13 | Join the [brainhack slack](https://brainhack-slack-invite.herokuapp.com/) and the #main-dl-2018 (Dec 11th) and/or #main-nilearn-2018 (Dec 12th) channel. 14 | 15 | Breakfast (8:30 am) and lunch are included. The training sessions will run from 9 am to 5 pm both days. All training sessions will be at the Groupe Maurice amphitheatre, at the [centre de recherche de l'institut de gériatrie de Montréal](https://goo.gl/maps/ouhdXKKWtko). 4545 Queen Mary Rd, Montreal, QC H3W 1W6, Canada. Metro station: snowdon (orange/blue lines), côte-des-neiges (blue line). 16 | ![CRIUGM](criugm.jpg) 17 | 18 | ### Usage 19 | 20 | To use the docker image, first after cloning the repository and cd to it, build it : 21 | ``` 22 | sudo docker build --tag=introml . 23 | ``` 24 | You can now run the container : 25 | ``` 26 | sudo docker run -p 8888:8888 -it introml jupyter notebook --no-browser --ip=0.0.0.0 27 | ``` 28 | -------------------------------------------------------------------------------- /course-outline.md: -------------------------------------------------------------------------------- 1 | ## Machine learning for neuroimaging ... 2 | ## ... with Scikit-learn and nilearn 3 | 4 | **Team:** Pierre Bellec, Elizabeth DuPre, Greg Kiar, Jacob Vogel 5 | 6 | **Date:** December 12th, 9h-17h. Breakfast/registration at 8h30. 7 | 8 | **Location:** Amphithéâtre “le groupe Maurice”, CRIUGM 9 | 10 | **Summary:** This course will be a hands-on/type-along introduction to machine learning for neuroimaging problems with scikit-learn and nilearn. 11 | 12 | ### Morning (9h-12h30): introduction to machine-learning with scikit-learn 13 | 14 | This part of the course will follow the scikit-learn chapter of the scipy-lectures, found [here](http://www.scipy-lectures.org/packages/scikit-learn/index.html). This includes: 15 | - Basic principles 16 | - Supervised learning: classification, the example of handwritten digits 17 | - Supervised learning: regression, the example of housing data 18 | - Measuring prediction performance 19 | - Unsupervised learning: dimension reduction and visualization 20 | - Chaining estimators: the example of eigenfaces 21 | - Parameter selection, validation, and testing 22 | 23 | ### Afternoon (13h30-17h): introduction to nilearn 24 | 25 | This part of the course will provide a general introduction to nilearn, building off of [several example analyses](http://nilearn.github.io/auto_examples/index.html#general-examples). 26 | 27 | ### Prerequisites 28 | 29 | - Basic familiarity with Python would be preferable 30 | - You will need enough space for Anaconda and all the course data (~4GB). 31 | 32 | If you are already savvy with Python and just want a tl;dr summary, here’s all you need to know: 33 | 34 | 1. Join the [Brainhack Slack](https://brainhack-slack-invite.herokuapp.com/) group and join the [main-nilearn-2018](https://brainhack.slack.com/messages/CEQB7U15M/) channel 35 | 2. Download and install python with the full-suite 64-bit [Anaconda](https://www.anaconda.com/download/) distribution 36 | 3. Download the [data](https://osf.io/5hju4/files/) and remember where you store it! 37 | 4. Download or clone the Intro to ML [repository](https://github.com/brainhack101/introML) 38 | 5. Install the necessary packages: `pip install -U nilearn scipy matplotlib scikit-learn jupyter pandas seaborn` 39 | 6. Test everything by opening one of the MAIN-tutorial .ipynb notebooks and running the first few cells 40 | 41 | For detailed instructions, view the full [installation instructions](https://docs.google.com/document/d/1G0QHtkZDklE5EEwbtTSruSijHhAoFIXoeDxk0AyVjM0/edit?usp=sharing). 42 | 43 | -------------------------------------------------------------------------------- /dl-course-outline.md: -------------------------------------------------------------------------------- 1 | ## Deep Learning for Neuroimaging 2 | 3 | **Team:** Andrew Doyle, Joseph Paul Cohen, Thomas Funck, Christopher Beckham 4 | 5 | **Date:** December 11th, 9h-17h. Breakfast/registration at 8h30. 6 | 7 | **Location:** Amphithéâtre “le groupe Maurice”, CRIUGM 8 | 9 | **Summary:** Deep learning is one of the most promising avenues towards achieving artificial general intelligence, and a strong new tool for the analysis of neuroimaging data. This course will offer an introduction into the theory behind how representations are automatically learned from data, and offer students an introduction into how to use the Keras library to formulate and solve a variety of deep learning problems using hands-on examples. 10 | 11 | **Learning Objectives:** 12 | * Understand how **representations are learned** in deep neural networks 13 | * Implement a **convolutional neural network** in Keras on neuroimaging data 14 | * Learn how **embeddings** can be learned 15 | 16 | 17 | ### Schedule: 18 | 19 | #### Morning (9h-12h30): Introduction & Segmentation with Deep Learning 20 | 21 | 9:00 am – 10:00 am: 22 | [Introduction to Deep Learning for Neuroimaging](https://www.dropbox.com/s/bju7auqwjslhhwy/IntroDL%20MAIN.pdf?dl=0) (Andrew Doyle) 23 | 24 | 10:00 am – 11:00 am: 25 | Deep Learning in Keras – [Hands-on Defacing Detector](https://colab.research.google.com/drive/1EgdnWZeNqmzqEmnSR9PUnYXlTjeu1wAU) (Andrew Doyle) 26 | 27 | 11:00 am – 11:15 am: 28 | Break 29 | 30 | 11:15 am – 12:30 am: 31 | Deep Learning for Segmentation - with [hands-on U-Net](https://colab.research.google.com/github/tfunck/minc_keras/blob/master/main2018.ipynb) (Thomas Funck) 32 | 33 | 12:30 pm - 1:30 pm: 34 | Lunch 35 | 36 | #### Afternoon (13h30-17h00): Getting Deeper 37 | 1:30 pm – 2:45 pm: 38 | [Looking Inside the Black Box](https://www.dropbox.com/s/dgmzittmthc41um/Interpretability.pdf?dl=0) - with Interpretability Hands-on (Andrew Doyle) 39 | 40 | 2:45 pm – 4:00 pm: 41 | [Clinical data successes using machine learning](https://docs.google.com/presentation/d/155oZORo29kpr1MNTwYbO2qEoYOIzHeBsDZCbI0NBmx8/edit) - with [Word2vec hands-on](https://colab.research.google.com/drive/1g4zvEg921sLQK-VsBk5mMb2-h4goCGyd) 42 | (Joseph Paul Cohen) 43 | 44 | 4:00 pm – 5:00 pm: 45 | [Generative Adversarial Networks](https://www.dropbox.com/s/vy5vdmubowv9g2h/gans.pdf?dl=0) - with [hands-on GAN](https://colab.research.google.com/drive/1KN0E_sORG-Bi7evOVtl6jONphI05ZiVL) (Christopher Beckham) 46 | 47 | ### Requirements 48 | * Basic familiarity with programming in Python is an asset, but not a requirement. 49 | * Examples will be presented in Google Collaboratory, and participants should create a Google account to run & write code along with the instructors: https://colab.research.google.com. 50 | * For students who wish to continue their analyses after the course, Python 3.x should be installed (ideally through [Anaconda](https://www.anaconda.com/)) with the [Keras](https://keras.io/) package. 51 | -------------------------------------------------------------------------------- /MAIN_tutorial_intro_to_nilearn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline\n", 10 | "import nilearn" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "# Let's keep our notebook clean, so it's a little more readable!\n", 20 | "import warnings\n", 21 | "warnings.filterwarnings('ignore')" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "## [Understanding neuroimaging data](http://nilearn.github.io/manipulating_images/input_output.html)\n", 29 | "\n", 30 | "### Text files: phenotype or behavior\n", 31 | "\n", 32 | "Phenotypic or behavioral data are often provided as a text or CSV (Comma Separated Values) file. They can be loaded with the [pandas package](https://pandas.pydata.org/) but you may have to specify some options. Here, we'll specify the `sep` field, since our data is tab-delimited rather than comma-delimited.\n", 33 | "\n", 34 | "For our dataset, let's load participant level information:" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "import os\n", 44 | "from nilearn import datasets\n", 45 | "import pandas as pd\n", 46 | "\n", 47 | "data_dir = '/home/jovyan/nilearn_data/main/main/'\n", 48 | "participants = 'participants.tsv'\n", 49 | "phenotypic_data = pd.read_csv(os.path.join(data_dir, participants), sep='\\t')\n", 50 | "phenotypic_data.head()" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "### Nifti data\n", 58 | "\n", 59 | "For volumetric data, nilearn works with data stored in the [Nifti structure](http://nipy.org/nibabel/nifti_images.html) (via the [nibabel package](http://nipy.org/nibabel/)).\n", 60 | "\n", 61 | "The NifTi data structure (also used in Analyze files) is the standard way of sharing data in neuroimaging research. Three main components are:\n", 62 | "\n", 63 | " * data:\traw scans in form of a numpy array: \n", 64 | " `data = img.get_data()`\n", 65 | " * affine:\treturns the transformation matrix that maps from voxel indices of the `numpy` array to actual real-world locations of the brain: \n", 66 | " `affine = img.affine`\n", 67 | " * header:\tlow-level informations about the data (slice duration, etc.): \n", 68 | " `header = img.header`\n", 69 | "\n", 70 | "It is important to appreciate that the representation of MRI data we'll be using is a big 4D matrix representing (3D MRI + 1D for time), stored in a single Nifti file.\n", 71 | "\n", 72 | "### Niimg-like objects\n", 73 | "\n", 74 | "Nilearn functions take as input argument what we call \"Niimg-like objects\":\n", 75 | "\n", 76 | "Niimg: A Niimg-like object can be one of the following:\n", 77 | "\n", 78 | " * A string with a file path to a Nifti image\n", 79 | " * A SpatialImage from `nibabel`, i.e., an object exposing the get_data() method and affine attribute, typically a Nifti1Image from `nibabel`.\n", 80 | "\n", 81 | "Niimg-4D: Similarly, some functions require 4D Nifti-like data, which we call Niimgs or Niimg-4D. Accepted input arguments are:\n", 82 | "\n", 83 | " * A path to a 4D Nifti image\n", 84 | " * List of paths to 3D Nifti images\n", 85 | " * 4D Nifti-like object\n", 86 | " * List of 3D Nifti-like objects\n", 87 | "\n", 88 | "**Note:** If you provide a sequence of Nifti images, all of them must have the same affine !" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": {}, 94 | "source": [ 95 | "## [Manipulating and looking at data](http://nilearn.github.io/auto_examples/plot_nilearn_101.html#sphx-glr-auto-examples-plot-nilearn-101-py)\n", 96 | "\n", 97 | "There is a whole section of the [Nilearn documentation](http://nilearn.github.io/plotting/index.html#plotting) on making pretty plots for neuroimaging data ! But let's start with a simple one." 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "# Let's use a Nifti file that is shipped with nilearn\n", 107 | "from nilearn import datasets\n", 108 | "\n", 109 | "# Note that the variable MNI152_FILE_PATH is just a path to a Nifti file\n", 110 | "print('Path to MNI152 template: {}'.format(datasets.MNI152_FILE_PATH))" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "In the above, MNI152_FILE_PATH is nothing more than a string with a path pointing to a nifti image. You can replace it with a string pointing to a file on your disk. Note that it should be a 3D volume, and not a 4D volume." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "from nilearn import plotting\n", 127 | "plotting.plot_img(datasets.MNI152_FILE_PATH)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "We can also directly manipulate these images using Nilearn ! As an example, let's try smoothing this image." 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "from nilearn import image\n", 144 | "smooth_anat_img = image.smooth_img(datasets.MNI152_FILE_PATH, fwhm=6)\n", 145 | "\n", 146 | "# While we are giving a file name as input, the function returns\n", 147 | "# an in-memory object:\n", 148 | "print(smooth_anat_img)" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "plotting.plot_img(smooth_anat_img)" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": {}, 163 | "source": [ 164 | "We can then save this manipulated image from in-memory to disk as follows:" 165 | ] 166 | }, 167 | { 168 | "cell_type": "code", 169 | "execution_count": null, 170 | "metadata": {}, 171 | "outputs": [], 172 | "source": [ 173 | "smooth_anat_img.to_filename('smooth_anat_img.nii.gz')\n", 174 | "os.getcwd() # We'll' check our \"current working directory\" (cwd) to see where the file was saved" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": {}, 180 | "source": [ 181 | "## [Visualizing neuroimaging volumes](https://nilearn.github.io/auto_examples/01_plotting/plot_visualization.html#visualization)\n", 182 | "\n", 183 | "What if we want to view not a structural MRI image, but a functional one ?\n", 184 | "No problem ! Let's try loading one:" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "fmri_filename = 'downsampled_derivatives:fmriprep:sub-pixar109:sub-pixar109_task-pixar_run-001_swrf_bold.nii.gz'\n", 194 | "plotting.plot_epi(os.path.join(data_dir, fmri_filename))" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": {}, 200 | "source": [ 201 | "Uh-oh, what happened ?! Let's look back at the error message:\n", 202 | "\n", 203 | "> DimensionError: Input data has incompatible dimensionality: Expected dimension is 3D and you provided a 4D image. See http://nilearn.github.io/manipulating_images/input_output.html.\n", 204 | "\n", 205 | "We can fix that ! Let's take an average of the EPI image and plot that instead:" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "from nilearn.image import mean_img\n", 215 | "\n", 216 | "plotting.view_img(mean_img(os.path.join(data_dir, fmri_filename)))" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "## [Convert the fMRI volumes to a data matrix](http://nilearn.github.io/auto_examples/plot_decoding_tutorial.html#convert-the-fmri-volume-s-to-a-data-matrix)\n", 224 | "\n", 225 | "These are some really lovely images, but for machine learning we want matrices so that we can use all of the techniques we learned this morning !\n", 226 | "\n", 227 | "To transform our Nifti images into matrices, we'll use the `nilearn.input_data.NiftiMasker` to extract the fMRI data from a mask and convert it to data series.\n", 228 | "\n", 229 | "First, let's do the simplest possible mask—a mask of the whole brain. We'll use a mask that ships with Nilearn and matches the MNI152 template we plotted earlier." 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "metadata": {}, 236 | "outputs": [], 237 | "source": [ 238 | "brain_mask = datasets.load_mni152_brain_mask()\n", 239 | "plotting.plot_roi(brain_mask, cmap='Paired')" 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": null, 245 | "metadata": {}, 246 | "outputs": [], 247 | "source": [ 248 | "from nilearn.input_data import NiftiMasker\n", 249 | "masker = NiftiMasker(mask_img=brain_mask, standardize=True)\n", 250 | "masker" 251 | ] 252 | }, 253 | { 254 | "cell_type": "code", 255 | "execution_count": null, 256 | "metadata": {}, 257 | "outputs": [], 258 | "source": [ 259 | "# We give the masker a filename and retrieve a 2D array ready\n", 260 | "# for machine learning with scikit-learn !\n", 261 | "fmri_masked = masker.fit_transform(os.path.join(data_dir, fmri_filename))\n", 262 | "print(fmri_masked)" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": null, 268 | "metadata": {}, 269 | "outputs": [], 270 | "source": [ 271 | "print(fmri_masked.shape)" 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": {}, 277 | "source": [ 278 | "One way to think about what just happened is to look at it visually:\n", 279 | "\n", 280 | "![](http://nilearn.github.io/_images/masking.jpg)\n", 281 | "\n", 282 | "Essentially, we can think about overlaying a 3D grid on an image. Then, our mask tells us which cubes or \"voxels\" (like 3D pixels) to sample from. Since our Nifti images are 4D files, we can't overlay a single grid -- instead, we use a series of 3D grids (one for each volume in the 4D file), so we can get a measurement for each voxel at each timepoint. These are reflected in the shape of the matrix ! You can check this by checking the number of positive voxels in our brain mask.\n", 283 | "\n", 284 | "There are many other strategies in Nilearn [for masking data and for generating masks](http://nilearn.github.io/manipulating_images/manipulating_images.html#computing-and-applying-spatial-masks). I'd encourage you to spend some time exploring the documentation for these !" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "We can also [display this time series](http://nilearn.github.io/auto_examples/03_connectivity/plot_adhd_spheres.html#display-time-series) to get an intuition of how the whole brain signal is changing over time.\n", 292 | "\n", 293 | "We'll display the first three voxels by sub-selecting values from the matrix. You can also find more information on [how to slice arrays here](https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.indexing.html#basic-slicing-and-indexing)." 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "import matplotlib.pyplot as plt\n", 303 | "plt.plot(fmri_masked[5:150, :3])\n", 304 | "\n", 305 | "plt.title('Voxel Time Series')\n", 306 | "plt.xlabel('Scan number')\n", 307 | "plt.ylabel('Normalized signal')\n", 308 | "plt.tight_layout()" 309 | ] 310 | }, 311 | { 312 | "cell_type": "markdown", 313 | "metadata": {}, 314 | "source": [ 315 | "## [Extracting signals from a brain parcellation](http://nilearn.github.io/auto_examples/03_connectivity/plot_signal_extraction.html#extracting-signals-from-a-brain-parcellation)\n", 316 | "\n", 317 | "Now that we've seen how to create a data series from a single region-of-interest (ROI), we can start to scale up ! What if, instead of wanting to extract signal from one ROI, we want to define several ROIs and extract signal from all of them ? Nilearn can help us with that, too ! 🎉\n", 318 | "\n", 319 | "For this, we'll use `nilearn.input_data.NiftiLabelsMasker`. `NiftiLabelsMasker` which works like `NiftiMasker` except that it's for labelled data rather than binary. That is, since we have more than one ROI, we need more than one value ! Now that each ROI gets its own value, these values are treated as labels." 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": null, 325 | "metadata": {}, 326 | "outputs": [], 327 | "source": [ 328 | "# First, let's load a parcellation that we'd like to use\n", 329 | "multiscale = datasets.fetch_atlas_basc_multiscale_2015(resume=True)\n", 330 | "print('Atlas ROIs are located at: %s' % multiscale.scale064)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": null, 336 | "metadata": {}, 337 | "outputs": [], 338 | "source": [ 339 | "plotting.plot_roi(multiscale.scale064)" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "metadata": {}, 346 | "outputs": [], 347 | "source": [ 348 | "from nilearn.input_data import NiftiLabelsMasker\n", 349 | "label_masker = NiftiLabelsMasker(labels_img=multiscale.scale064, standardize=True)\n", 350 | "label_masker" 351 | ] 352 | }, 353 | { 354 | "cell_type": "code", 355 | "execution_count": null, 356 | "metadata": {}, 357 | "outputs": [], 358 | "source": [ 359 | "fmri_matrix = label_masker.fit_transform(os.path.join(data_dir, fmri_filename))\n", 360 | "print(fmri_matrix)" 361 | ] 362 | }, 363 | { 364 | "cell_type": "code", 365 | "execution_count": null, 366 | "metadata": {}, 367 | "outputs": [], 368 | "source": [ 369 | "print(fmri_matrix.shape)" 370 | ] 371 | }, 372 | { 373 | "cell_type": "markdown", 374 | "metadata": {}, 375 | "source": [ 376 | "### [Compute and display a correlation matrix](http://nilearn.github.io/auto_examples/03_connectivity/plot_signal_extraction.html#compute-and-display-a-correlation-matrix)\n", 377 | "\n", 378 | "Now that we have a matrix, we'd like to create a _connectome_. A connectome is a map of the connections in the brain. Since we're working with functional data, however, we don't have access to actual connections. Instead, we'll use a measure of statistical dependency to infer the (possible) presence of a connection.\n", 379 | "\n", 380 | "Here, we'll use Pearson's correlation as our measure of statistical dependency and compare how all of our ROIs from our chosen parcellation relate to one another." 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": null, 386 | "metadata": {}, 387 | "outputs": [], 388 | "source": [ 389 | "from nilearn import connectome\n", 390 | "correlation_measure = connectome.ConnectivityMeasure(kind='correlation')\n", 391 | "correlation_measure" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "correlation_matrix = correlation_measure.fit_transform([fmri_matrix])\n", 401 | "correlation_matrix" 402 | ] 403 | }, 404 | { 405 | "cell_type": "code", 406 | "execution_count": null, 407 | "metadata": {}, 408 | "outputs": [], 409 | "source": [ 410 | "import numpy as np\n", 411 | "\n", 412 | "correlation_matrix = correlation_matrix[0]\n", 413 | "# Mask the main diagonal for visualization:\n", 414 | "# np.fill_diagonal(correlation_matrix, 0)\n", 415 | "plotting.plot_matrix(correlation_matrix)" 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "metadata": {}, 421 | "source": [ 422 | "### [The importance of specifying confounds](http://nilearn.github.io/auto_examples/03_connectivity/plot_signal_extraction.html#same-thing-without-confounds-to-stress-the-importance-of-confounds)\n", 423 | "\n", 424 | "In fMRI, we're collecting a noisy signal. We have artifacts like physiological noise (from heartbeats, respiration) and head motion which can impact our estimates. Therefore, it's strongly recommended that you control for these and related measures when deriving your connectome measures. Here, we'll repeat the correlation matrix example, but this time we'll control for confounds. " 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "conf_filename = 'sub-pixar109_task-pixar_run-001_ART_and_CompCor_nuisance_regressors.tsv'\n", 434 | "clean_fmri_matrix = label_masker.fit_transform(os.path.join(data_dir, fmri_filename),\n", 435 | " confounds=os.path.join(data_dir, conf_filename))\n", 436 | "clean_correlation_matrix = correlation_measure.fit_transform([clean_fmri_matrix])[0]\n", 437 | "np.fill_diagonal(clean_correlation_matrix, 0)\n", 438 | "plotting.plot_matrix(clean_correlation_matrix)" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "That looks a little different !\n", 446 | "\n", 447 | "Looking more closely, we can see that our correlation matrix is symmetrical; that is, that both sides of the diagonal contain the same information. We don't want to feed duplicate information into our machine learning classifier, and Nilearn has a really easy way to remove this redundancy ! " 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "metadata": {}, 454 | "outputs": [], 455 | "source": [ 456 | "vectorized_correlation = connectome.ConnectivityMeasure(kind='correlation',\n", 457 | " vectorize=True, discard_diagonal=True)\n", 458 | "clean_vectorized_correlation = vectorized_correlation.fit_transform([clean_fmri_matrix])[0]\n", 459 | "clean_vectorized_correlation.shape # Why is this value not 64 * 64 ?" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "## [Interactive connectome plotting](http://nilearn.github.io/plotting/index.html#d-plots-of-connectomes)\n", 467 | "\n", 468 | "It can also be helpful to project these connection weightings back on to the brain, to visualize these connectomes ! Here, we'll use the interactive connectome plotting in Nilearn." 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": null, 474 | "metadata": {}, 475 | "outputs": [], 476 | "source": [ 477 | "coords = plotting.find_parcellation_cut_coords(multiscale.scale064)\n", 478 | "plotting.view_connectome(clean_correlation_matrix, coords=coords)" 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": null, 484 | "metadata": {}, 485 | "outputs": [], 486 | "source": [ 487 | "?plotting.view_connectome" 488 | ] 489 | }, 490 | { 491 | "cell_type": "code", 492 | "execution_count": null, 493 | "metadata": {}, 494 | "outputs": [], 495 | "source": [] 496 | } 497 | ], 498 | "metadata": { 499 | "kernelspec": { 500 | "display_name": "Python 3", 501 | "language": "python", 502 | "name": "python3" 503 | }, 504 | "language_info": { 505 | "codemirror_mode": { 506 | "name": "ipython", 507 | "version": 3 508 | }, 509 | "file_extension": ".py", 510 | "mimetype": "text/x-python", 511 | "name": "python", 512 | "nbconvert_exporter": "python", 513 | "pygments_lexer": "ipython3", 514 | "version": "3.6.7" 515 | } 516 | }, 517 | "nbformat": 4, 518 | "nbformat_minor": 2 519 | } 520 | -------------------------------------------------------------------------------- /plot_variance_linear_regr.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "\n", 17 | "# Plot variance and regularization in linear models\n", 18 | "\n", 19 | "\n", 20 | "\n", 21 | "\n" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": 2, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "import numpy as np\n", 31 | "\n", 32 | "# Smaller figures\n", 33 | "from matplotlib import pyplot as plt\n", 34 | "plt.rcParams['figure.figsize'] = (3, 2)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "We consider the situation where we have only 2 data point\n", 42 | "\n" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "X = np.c_[ .5, 1].T\n", 52 | "y = [.5, 1]\n", 53 | "X_test = np.c_[ 0, 2].T" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "Without noise, as linear regression fits the data perfectly\n", 61 | "\n" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 4, 67 | "metadata": {}, 68 | "outputs": [ 69 | { 70 | "data": { 71 | "text/plain": [ 72 | "[]" 73 | ] 74 | }, 75 | "execution_count": 4, 76 | "metadata": {}, 77 | "output_type": "execute_result" 78 | }, 79 | { 80 | "data": { 81 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAANAAAACPCAYAAACLfFVWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAELlJREFUeJzt3Xl8VPW5x/HPAwREQVCD7BBR6oKg0IggFfeFiERRK6WiKK0vFxSs6EWtYrn2umCpIoJF4FbcW1AaMIqgXKkUKGGRHQwqsil72AJkee4fvxHSEMiEOTO/mczzfr3ycibnvOY8HvP1zPzmd56fqCrGmGNTxXcBxiQyC5AxEbAAGRMBC5AxEbAAGRMBC5AxEbAAGRMBC5AxEbAAGROBar4OnJqaqmlpab4Ob8xRzZs3b4uq1itvP28BSktLIycnx9fhjTkqEVkTzn7lvoUTkaYiMl1ElonIUhHpV8Y+IiLDRCRXRBaJSLtjKdqYRBPOFagQeFhV54tIbWCeiExV1WUl9ukCtAz9XAiMDP3TmPikCiIRv0y5VyBV3aiq80OPdwHLgcaldssExqkzG6grIg0jrs6YoB3YA588BtMGBfJyFRqFE5E0oC0wp9SmxsDaEs/XcXjIEJG7RSRHRHI2b95csUqNidQ3/wcjOsLsEVCwz12FIhR2gESkFjAB6K+qO4/lYKo6SlXTVTW9Xr1yBziMCUb+DvhHXxiXCVWqQe9syHghkLdwYY3CiUgKLjxvq+oHZeyyHmha4nmT0O+M8WvFRzD5d7BnM3TqD5cOhJSagb18uQESEQHGAMtVdegRdssC+orIe7jBgzxV3RhYlcZU1O5N8PGjsPRDqN8aer4HjdoGfphwrkCdgF7AYhFZGPrd40AzAFV9DcgGMoBcYC9wZ+CVGhMOVVj0Pnwy0A0YXP4kdOoHVVOicrhyA6SqXwJHfbOorrHC/UEVZcwx2bEWJj8EuVOhSXvIHA71zozqIb3NRDAmMMXFkDMGpj3trkBdXoALfgNVqkb90BYgk9i25ELWA/D9v6DFZXD9y3BS85gd3gJkElNRIcx6BaY/CynHQeYIOL9nIEPTFWEBMoln4yLI6gsbv4Kzr4eMP0Ht+l5KsQCZxFGwD2a8AF++BMefAr8cB+dkei3JAmQSw/dz3FVnyyo4rydc80c4/mTfVVmATJzbvxs+Gwz/HgV1msBtE+CMK31XdZAFyMSv3M9gUn/IWwvtfwtXPAU1avuu6j9YgEz8yd8OU56AhW/DKS3hrk+gWQffVZXJAmTiy7IsyB4Ae7bAxQ9D50fdMHWcsgCZ+LDrRxec5VnQoA38ejw0bOO7qnJZgIxfqrDwHZjyOBTkwxWD4KIHojb5M2gWIOPP9jUwuT+s/hyadYRur0BqS99VVYgFyMRecTHMfR2m/cFNvcl4EdL7QJXE6/NpATKxtXmVm/y5drb7Pqfrn6FuM99VHbNw+sKNFZFNIrLkCNsvFZE8EVkY+nkq+DJNwisqgBkvwmudYMtKuPEvbqAggcMD4V2B/goMB8YdZZ9/qmrXQCoylc+GhW4azg+L4ZwbIGMI1DrVd1WBCOeO1BmhdlbGVExBPnzxPMwcBiekwq1vudnTlUhQn4E6ishXwAZggKouDeh1TaJaM8tddbbmQtvb4OpnoOZJvqsKXBABmg80V9XdIpIBTMS1+D2MiNwN3A3QrFliv/c1R7B/lxtdm/u6+3zTayKcfpnvqqIm4nFDVd2pqrtDj7OBFBFJPcK+1lixMvt6muv8OXc0dLgP7p1VqcMDAVyBRKQB8KOqqoi0x4Vya8SVmcSxd5ubSfDVu5B6JvT5FJq2911VTITTWPFd4FIgVUTWAYOAFDjYE+5m4F4RKQTygR6hNlemslOFZRMh+xE3g7rzo9B5AFSr4buymAlnFO5X5WwfjhvmNslk1w/w0cOwYjI0PB96fQgNWvuuKuZsJoKpGFVY8Ja7X6doP1w1GDrcD1WT808pOf+tzbHZ/h1M6ueWCWneCa4fBqln+K7KKwuQKV9xketJ8NlgkKpw3VD4+Z0JOfkzaBYgc3SbVrgvRNfNhZZXu8mfdZr4ripuWIBM2QoPwMyXYMYQqF4Lur8OrW+JeefPeGcBModbP9/dcvDjEjj3Jrj2eahlX3yXxQJkDinIh+n/A7OGQ6360ONdOCvDd1VxzQJknO++dFedbd9Auzvc8HTNur6rinsWoGS3b6db8j1nLJyUBrdnQYtLfFeVMCxAyWzVFLei266N0LEvXPYEVD/ed1UJxQKUjPZsdWuILv4b1DvbrXLQJN13VQnJApRMVGHJBLd69b6dcMlA1/2zWnXflSUsC1Cy2LnBTf5cmQ2N2rkFeOu38l1VwrMAVXaqMP8N+PRJ1xnn6j9Ch3tjsgBvMrAAVWbbvoGsB+G7f0LaxdBtGJzcwndVlYoFqDIqLoLZI+HzZ1yP6etfdt/t2DScwIVzR+pYoCuwSVXPLWO7AC8DGcBeoLeqzg+6UFO2iQvWM2TKSjbsyKdR3ZoM7liFK1YNhvXz4GddoOtQOLGR7zIrrSAaK3bBdeFpCVwIjAz900TZxAXreeyDxeQXFJFCITfveouLP5/I/honUuOmMW4em111oqrcGzpUdQaw7Si7ZALj1JkN1BWRhkEVaI5syJSV5BcUcZ7kMqn6EzyUMoHs4gu5UV6C1jdbeGIgiM9AjYG1JZ6vC/1uY+kdrS9csLbt2MHj1cbTp2o2mziJuw4M4PPidkie78qSR0wHEVR1FDAKID093Tr3ROLbGUyrOZDG+iNvFV7Bc4W/YjduGk6jujU9F5c8ggjQeqBpiedNQr8z0bAvz32nM/8N6pzQjNt3DmJG4ZkHN9dMqcoj15x5lBcwQQripvYs4HZxOgB5qnrY2zcTgJUfw6sXwoI34aIHqdVvDt2730rjujURoHHdmjzbvTU3tG3su9KkEURjxWzcEHYubhj7zmgVm7T2bHHz15ZMgFNbQY93oHE7AG5oe7wFxqMgGisqcH9gFZlDVGHxeBee/bvc7Qad+tvkzzhiMxHiVd46mPw7+HoKNLnALcB76tm+qzKlWIDiTXExzPtfmDoItAiufQ7a322TP+OUBSiebF3tJn+u+RJOu8TNYTv5NN9VmaOwAMWDokKY/arriFO1BnQb7lZ1s5kEcc8C5NsPS1znzw0L4Mzr4Lo/wYk2EypRWIB8Kdzvln3/cqhbO/SWv7oVrO2qk1AsQD6sneuuOptXQJsecO2zcPzJvqsyx8ACFEsH9rib3GaPhBMbw6/HQ8urfFdlImABipXV02HSg7Dje7jgt3DlIKhR23dVJkIWoGjL3wGfPuFWdTv5dLjzY2h+ke+qTEAsQNG0fLJrJbVnM/ziIbjkvyDFbjWoTCxA0bB7k1u5etlEqN8aer4Hjdr6rspEgQUoSKqw6H3XNvfAHrj8SejUz3XGMZWSBSgoO9bC5P6QOw2aXuhmE9T7me+qTJRZgCJVXAw5Y2Da0+4K1OUFN8pmC/AmBQtQJLZ87Ral+n4WtLjMTf48qbnvqkwMhRUgEbkW1zyxKjBaVZ8rtb03MIRDvRCGq+roAOv0rmQDw6Z1qjOyxUxarRoBKcdB5gg4v6dNw0lC4dzSXRV4FbgK17JqrohkqeqyUru+r6p9o1CjdyUbGJ4j3/F8/ihaLf+ODQ2volHPV6F2fd8lGk/CuQK1B3JV9RsAEXkP10yxdIAqrSFTVlJckM+Aah9yT9VJbKc29xzoz+LtlzDTwpPUwglQWY0Ty2rde5OIdAZWAQ+p6trSOyRqY8WGeQt5o/rrnFFlA38v7MwzhbeRRy1kR77v0oxnQQ0VTQLSVLUNMBV4o6ydVHWUqqaranq9evUCOnQU7d8N2Y/ytxqDOU4O0OvAQB4pvIc8agHWwNCEdwUqt3Giqm4t8XQ08ELkpXmW+xlM6g95a/n2tJ78MvdqthYf+kLUGhgaCO8KNBdoKSKniUh1oAeumeJBpZrJdwOWB1dijO3dBhPvg7e6Q7UacNcnnH7HCJ7sfoE1MDSHCacvXKGI9AWm4Iaxx6rqUhEZDOSoahbwoIh0AwpxKzn0jmLN0bPsH/DRANi71S2+2/lRN0wN3NC2sQXGHEZcX8TYS09P15ycHC/HPsyuHyF7ACzPggZtIPNVaNjGd1XGIxGZp6rp5e2X3DMRVGHhOzDlcSjIhyufho4PQNXkPi0mfMn7l7J9DUzqB99Mh2YdXefP1Ja+qzIJJvkCVFwMc1+HaX9wU28yXoT0Pjb50xyT5ArQ5pVu8ufaOXDGldD1z1A3cb7QNfEnOQJUVAAzX4YvnofqJ8CNf4E2t9rkTxOxyh+gDQtdD7YfFkOrG939OrVO9V2VqSQqb4AK8t0VZ+YwOCEVbn0bzu7quypTyVTOAK35l/usszUX2vaCq//btc81JmCVK0D7d7lbq+eOdoMDvSbC6Zf5rspUYpUnQF9PdZM/d66HDvfB5b93AwbGRFHiB2jvNvjkMVj0HtQ7C/p8Ck3b+67KJInEDZCqa1yY/Qjkb3cTPzsPcDOojYmRxAzQzo1u8ueKydDwfPdZp8G5vqsySSixAqQKC96EKb+Hov1w1WDocL9N/jTeJM5f3rZv3eTPb7+A5p3c5M9TTvddlUlyYc2gFJFrRWSliOSKyMAyttcQkfdD2+eISFpgFRYXwawRMPIiWD8frhsKd0y28Ji4EFRfuD7AdlU9Q0R6AM8Dt0Zc3aYVbhrOurnQ8hroOhTqNIn4ZY0JSlB94TKBp0OPxwPDRUT0WG93LTwAM1+CGUOgei3oPhpa32yTP03cCaov3MF9Qj0U8oBTgC0VrmhfHoztApuWwrk3ucmfJ6RW+GWMiYWYDiKE1VjxuDqQ1snNJDgrI4bVGVNx4QwilNsXruQ+IlINqANsLbVP+I0VM4ZYeExCCKQvXOj5HaHHNwOfH/PnH2MSSFB94cYAb4pILq4vXI9oFm1MvAjrM5CqZgPZpX73VInH+4Bbgi3NmPjnrbGiiGwG1hxll1SOZRQv+qyuiknUupqrarkrIHgLUHlEJCeczpCxZnVVTGWvy5qhGRMBC5AxEYjnAI3yXcARWF0VU6nritvPQMYkgni+AhkT97wHyOu9RpHV1VtENovIwtDPb2JQ01gR2SQiS46wXURkWKjmRSLSLto1hVnXpSKSV+JcPVXWflGoq6mITBeRZSKyVET6lbFPZOdMVb394GY2rAZaANWBr4BzSu1zH/Ba6HEP4P04qas3MDzG56sz0A5YcoTtGcDHgAAdgDlxUtelwGQPf18NgXahx7VxK8iX/u8Y0TnzfQU6eK+Rqh4AfrrXqKRMDq36PR64QiTqNwaFU1fMqeoM3FSpI8kExqkzG6hbav1aX3V5oaobVXV+6PEu3Nq9pdfpjOic+Q5QWfcalf4X/I97jYCf7jXyXRfATaHL/ngRaVrG9lgLt24fOorIVyLysYi0ivXBQ2/92wJzSm2K6Jz5DlAimwSkqWobYCqHrpLmcPNxU2POA14BJsby4CJSC5gA9FfVnUG+tu8ABXavUazrUtWtqro/9HQ08PMo1xSOcM5nzKnqTlXdHXqcDaSISExuMxaRFFx43lbVD8rYJaJz5jtA8XqvUbl1lXqf3A33/tq3LOD20MhSByBPVTf6LkpEGvz0uVVE2uP+7qL9P0FCxxwDLFfVoUfYLbJzFuuRkTJGSjJwoyOrgSdCvxsMdAs9Pg74O5AL/BtoESd1PQssxY3QTQfOikFN7wIbgQLce/U+wD3APaHtguugtBpYDKTH6FyVV1ffEudqNnBRjOr6BaDAImBh6CcjyHNmMxGMiYDvt3DGJDQLkDERsAAZEwELkDERsAAZEwELkDERsAAZEwELkDER+H9TzlTozKDfbQAAAABJRU5ErkJggg==\n", 82 | "text/plain": [ 83 | "
" 84 | ] 85 | }, 86 | "metadata": {}, 87 | "output_type": "display_data" 88 | } 89 | ], 90 | "source": [ 91 | "from sklearn import linear_model\n", 92 | "regr = linear_model.LinearRegression()\n", 93 | "regr.fit(X, y)\n", 94 | "plt.plot(X, y, 'o')\n", 95 | "plt.plot(X_test, regr.predict(X_test))" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "In real life situation, we have noise (e.g. measurement noise) in our data:\n", 103 | "\n" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 5, 109 | "metadata": {}, 110 | "outputs": [ 111 | { 112 | "data": { 113 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMYAAACPCAYAAAChzKT5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VGWe7/HPU1uqsockZGELIQkhJJWEgIABBUFQtEVGW223UdvW7rZbe6bbuffa3XfueHumZ6Zvz2vE7mnBpUfalnZttV0aEFFZZA9UyEL2fd8qVamk1uf+USWiBglkqQSf9+uV16tInZznV4d885znnOecI6SUKIryeZpgF6Aok5EKhqIMQwVDUYahgqEow1DBUJRhqGAoyjBUMBRlGCoYijIMFQxFGYZuPFYaFxcnU1JSxmPVijIqx44d65JSxp9vuXEJRkpKCkePHh2PVSvKqAgh6keynNqVUpRhqGAol5SxmhSrgqFcMioPH2D7/34U19DgqNc1LmMMRZlIA329fPD7LVQc3Ed8SioOqxWD0TSqdapgKFOWlJLSjz/gw+efxu0cYsVtd7P4G3+DVjf6X2sVDGVK6u/qYNfTv6XuxDGSMxaw7rsPEztj1pitXwVDmVKkz8eJXe+y98XnQUquuvdB8tZdh9CM7XBZBUOZMnpamti5ZTPN5aXMMedz9Xd+QNT0hHFpSwVDmfS8Hg9H3/4zn7z6InpDCNd8/+/IuuIqhBDj1qYKhjKptddWs/OpzXTUVZO+9HLW3Pc9wqJjxr1dFQxlUvK4XBx8/U8cfvNVTBGRfOPv/xcZSwsnrH0VDGXSaS4vZceWzfS2NLFw1VquvOvbmMIjJrQGFQxl0nANDbJv+zaKdrxNZFw8Nz32OCm5i4JSiwqGMinUnTjGzqd/g627i/xrrmfFbXeP+uz1aKhgKEE1aLfx0bZnKPloN9OSZ3Lb//k3ZmRmBbssFQwleCoO7Wf3s79j0NbP0k23suxvbkVnMAS7LEAFQwkCe28PHzz3FJWHDzA9ZR43PfY401NSg13W56hgKBNGSknJR7v5cNvTeFwuVt5+D4uv34RGqw12aV+igqFMCGtHO7ue/g31liJmZC5k3YMPMy15RrDLOicVDGVc+XxeTux4h33bt4EQrLnve+Refe2YT/obayoYyrjpbmpk55bNtFSUkZJXwNXfeYjIuOnBLmtEVDCUMef1eDjy1mscfG07eqOJax/6exasXD2uk/7GmgqGMqbaa6rY8dQTdNbXkrF8JVfd88CETPobayoYyphwu5x88up2jv7ldUKjornhJz8lfcnyYJd10VQwlFFrKjvFzi1P0tvaTPbqdVx5130Yw8KDXdaoqGAoF83pcLB3+/Oc3PkOUdMTuPlnv2BOTl6wyxoTKhjKRaktOsqup3+LraeLRRs2suLWu9AbjcEua8yoYCgXZNDWz4fPP03p3j3EzpzNtx7/FckZmcEua8ypYCgjIqWk4uA+dj/3FM4BO8tuuo2lm25Fp9cHuzQ/jwtP2Ts0HNtF6l1PgHZ0dalgKOdl7+lm93O/o+rIQRJS01n/s18QP2dusMsCwNdWQuPHL2KpqKHEM4chpvODGgtx6QWjWq8KhnJOUkpO7dnFR394Fq/bzRV33kfBho3Bn/Q3ZKXz4EtYjh2k2BZFH1HoRTqZKdMxX341Malpo25CBUMZVl97G7u2bqbhlIWZWdmse/BhYhKTg1eQlNjKPuDUvnewtAzRynQEs0iNNbB66SoycxcTEhIyZs2pYCif4/N5KXrvbfa9tA2NRsPa+x/CvGZ90Cb9ObvqKf/gRSyVDdS445FEkhwaynrzQrILryEiYnxuknDeYAghZgHbgARAAlullE+MSzVKUHU11rPzqc20Vp0mddES1t7/EBGxcRNeh9c1RM3eV7CcOEa5LQw3eqK10azMiCdn1Y3EJ4/dPWrPZSQ9hgf4sZTyuBAiAjgmhNglpSwd59qUCeL1uDn85qscfO0lDKGhbPjhT8gsvHJCJ/1JKWk5dQDL/r9yqs3FACaMmMhN1GMuXM+s7GUTWs95gyGlbAVaA69tQogyYAaggnEJaKuqYMeWzXQ11JFZeCWr73mA0MioCWu/p62B4j2vY6luodsTihbJ/Agv5vxM0lZsQmcYu3HDhbigMYYQIgXIBw4N894DwAMAs2fPHoPSlPHkdg5x4JUXOfb2G4TFxHDjP/yceQVLJ6Rtx8AAJXv/gsViodFhACQp+kEKs6az4KrbMMXNnJA6vsqIgyGECAdeA34kpez/4vtSyq3AVoDFixePzYPQlHHRWGJh59Yn6WtrxbzmGq64815CQsPGtU23283pE4ewHPqQqi4XPjRMF3bWJhvJueIbRM1fAWOwq9Tv8RKpG/3h5BEFQwihxx+KP0opXx91q0pQOB0DfPzH32N5/69EJyTxzZ//C7OzzePWns/no66mGsv+v1Ja34HLpyUCO8sibZgLlpG4/FEwjC6Q/R4vB/vs7O+1s7/PTql9EEthNnGG0R1wHclRKQE8C5RJKf9jVK0pQVNz/Ai7nv4NA729FFy/icJb7kAfMj6T/tra2rAc+pDikjJsLoEBJ1m6ZsxZqaRc+SCa+PSLXrfd4+WgdYADvXb299kotg3iA4waweLIMH4yN3FMPsNIYlUI3AUUCyFOBL73mJTy3TGpQBlXjn4re/57K+X7PyJu1hxu+PFjJKXNH/N2rFYrxUVHsRw7SIfNjQYvadSzPtnE/MKN6DPXgfbC/4oPeL0csQ6c6RFO2hx4JRiEYFFkKH+XkkBhdASLIkMxasfuXMtIjkrtA6bOxboK4D/8efrAx3zw+y04HQ4u/+YdXHbjzWh1Yzfpb3BwkLLSUixH9lHX1gMIZtHCdWE9ZC1ZSdjiH0L4hd38YNDr41j/Z0Eo6nfglhKdgPyIMH44O4HC6HAKosIIHS4IPh+MwclIdeb7EmTr7uL9Z/+LmmOHSUzLYP2DDxM3O2VM1u3xeKiqqsJy/DCnq2rw+iCWHlZra8jJnMe0Zd+BmYtHPJB2+nwc73cEgmDjeL8Dp0+iAXIjQnlwVjyF0eFcFhVG2HCDakcPNB2FpsPQdASai+BHJ8E0uuvMVTAuIdLno/iDnXz0wnP4vF5W3X0/+dd+A41mdEdppJQ0NjZiOXmCkuKTDLq8hOJgMacxJ+hIXroJkf3LEQ2k3T7JCZuD/b029vfZOWodYNAnEUBOuIl7Z8RRGB3OsuhwIr4YBJ8XOsr8IWg84g9Cd6W/RqHBo0lgyBpHmMOGRgVDAehta2HXlidpLC1m1kIz6x74IdGJSaNaZ2dnJxaLheITx+mzDaDDwwIqMRvbSM1fhbbg1xD31TNZPT6Jxe7vEQ702TlkHcDh9QGQFWbkzuRYCqMjWBYdRrT+C7+Ojh7/L3/jp73BcXDZAPBpI3C643A0p2CvsDHYo0dKLSHzUwgZ1DDaW0OrYExxPp+X4++8yf6X/4hGq+XqB35IzlXrLnr6hM1m49SpU1hOnqC1rR2BJJV6VosKMtNSCVl8D6Rdfc6BtFdKSuyDZ8YIh/rs2AJByAg1cmviNAqjw1keHU7s2YdUfV5oKw6EILBr1F0FfNobJDJkTcBWGY6j2YvbrkUbZcSUl0fYHXnE5eVhzM5BGz4252NUMKawroY6djz1BG3VlcxbvJQ13/4eEdMufNKf0+mkvLwci8VCTU01UkKS6GQ9JWRP8xBRcAvk/vuwA2mflJQPDJ0ZI3zSN4DV4wVgnimETQkxXB4dTmFMOPGGswb+A91Qe+SsscFxcNn969RF4nLHMdA8F1uFjaFuHdInCEmfiSkvj7i/zcOUl4dhbsq4zZ9SwZiCvB43h/78Mof+/AohYWFc98g/MH/5ygv6JfF6vdTU1GCxWCgvK8Pt8RClcbBCFmPWNxCfvRry/w1mXfa5gbSUkgqH88wY4ZM+Oz1ufxDmGA1cFx9FYXQ4l8eEkxRi+LRg6Cj9/Nigp9q/PqHBo01iyJqEvXKAgUYP7gEtmkgDptxcwm7NJS4vD5PZjHacppgPRwVjimmtPM2Op56gu6mBBStXs+ru+0c86U9KSUtLCxaLhVOnTjEwMIBR48XsK8VMKbOSZ6MpuAuyboSQ8DM/U+MYOrNrdKDPTqfLA8CMED1rYyMpjI6gMCacmcZAEAa6oG7358cG7gEAfLooXJ44BlrmYSvvZ6hbi/RKDGmJ/t7gzkBvkJoa1Bs/q2BMEe6hIfa//AeOvfsW4dNi2fQ//pHURUtG9LM9PT0UFxdjsVjo7u5GKyTzNQ2YOUGa0YEu/1bI/1eIS0dKScOQi/0t3ewPTLVoc7kBSDTouSImgsLArtFsowHh80JHCVjOGhv01AAghRaPLokh60zsVXYG6t3+3iBM4+8Nbs4jLj/QG0RN3IzekVDBmAIaTp1k59Ynsba3kXv1Blbefg8hoaFf+TMOh4OSkhL/DNbGRgBSQqwUcogFsgZT2lWQ/8+QfjVNbunvEcrq2d9rp9npD0K8QecfHwSCkGoKQTi6ofEQlAaC8KXeYDqO1jT6y/sZ6tIgvT4Mc+Mw5a0l9lZ/bxCSNg8R7OvGz0MFYxIbGrDz8QvPUfzBTqITk7jlH3/JrKyccy7vdrupqKjAYrFQWVmJz+cj3uhhreYY2b5iosMTYeWdtC24hXfdRn+PcLiS+iEXANP0WpZHh/NQdDiFMRFkGLWI9hJo2gUHA7tFvbWAvzfw6pIZ7J+FvWqAgTon7gEtIlRiMpsJ3ZRLbF4eptxcdDHqps7KGKk6eojdz/yWgb4+ltxwE8u/eTv6YS7a8fl81NXVUVxcTGlpKU6nk4gQwbKQKsyDB0jwDNKVdzsfpv0TBzRJ7O+zU23pACBKp2V5dBj3z4ynMCacTGxomo5AWaA3aDkOboe/HX00Ls90BtoysJX1MdSlRXq96OdEE5q3itibA71BejpCN/V/rab+J7jEOKx9fPD7LZz+ZC/xs1PY+OjPSZz35dmobW1t/pNvxcXYbDYMOg1Zob2YnbuJ8Fo5NPcmnpi1lf3aRCoG3dAO4dpelkWH+0+qRYawcKAGbdP7UBE4bNpbB4AUOjz6ZIZsc7BXObDXDOJxaBEmL6bsbEJvyCM2P9AbxMZO8BaaGELKsb+maPHixfLo0aNjvt5LTcWhNj55sxp7j5OwGAOzMzop2/cyiZo5FCStR+fSoY0OoTz7GD3a54gy9GB3RtHRtoyWhuloNIK0KB/N/f28OLSUXm84OqMGR3oUvuQwQoQPg7MC38AJsryN/Dwuncs8vs/GBp5BAHz6GFze6ThaBbayXgY7NUivQD/Tf97g0y/j/AzEZLnz4EUSQhyTUi4+33KqxwiSikNt7PljOR6XD+mz0dvwPt01tSyYcSXm0OXg8v/B6jDuxhvyPNFa/4A4wmhFzNqP1biKPk8km915tLabED7/8p4hSWRZD3frT6DveIKsQRu5TiczAyfdfEKLzzCTwYFU7NUO7FUD/t7A4MSYk4Ppulxi8vIIzctDFx8fnI0zCahgBMknb1bjdnrxuix4HHsBHzrTKtINS/F6vDRpuqnSthGXsR20Wk6zkFKyKSWbWm0qMklLCF4MezuZ7utmkaaSfE0l+ZoqzKIGU6l/QN0jtXQ49BR3hGGsNeDt0iN9bnTJRkLzlhF7faA3yMxETJKHz08GKhhB0t/ZhtuxE+lpRqObjTZ0LT6j4JiunAp9Jw1RUbRHJ9BnfIwa0vAKHVrpZr6vnEcGfsdyaxGFplxaPR8zy9gJgFPqKJEpvOhdw3FfGj/668t4HFrcWqhJhIoUQWWhhv/60W70CQlB3gKTmwrGBPN5vRx9+8+4bC8AWkTkGhzRUTQkdNEUG0lLdCodkQV4NRq0PsmSoePcbv81S/pPkdrfRpTdhdYXWJehl9Mylf92r6PIl06JTMEZmFca7e4lYrmeimRBbSJ4dP5pHUlhSSoUI6CCMYE66mrYuWUzTY2N9CxezenoGTROi6I9chperRaD18W17UVsqK8mz15OsrsMvfAfWvUJ6I/Q0ZxkxBpqQO40oD9k4PTsJF7IW49T89l/pUmvYePqGN7tDGXIO3Tm+0atkUcWPTLhn3sqUsGYAEMuFy+89SbvNXXQsGg9rWvi8Wh1JAx1sqajiJXVpeQNlDBHVqER/u7AQxwu33zsnvV0uC20LWnCEycQvRoMlkwWXHEjpofz+N7Chcwo6+ZXO07T0jdIcrSJR9fP58b8GVxWA08cf4K2gTYSwxJ5ZNEjXJd6XZC3xtSgDteOA6+UFNsG2dfTz3u1DVg8AjSQba/kyu7jrBooJaf/FGFuKwBSglfEMuRKxF7nxlZmxTOoBa0WY2bmZ4dM8/PQz5gxpZ6XPdmow7UTyCclpfbBM5PuDvTaCB/sYHF/CTd1n+DfrMXMd9ahl/5ZqT4PuAYM9FoT6a/TMNgmkT6BdprAlHcZMd/Lw5SXiyk7G8155kQp40MF4yLITy/O6bNzoNfO0Z4eZvaVs7i/hG/1FvOr/lMkeHoA8KLF6wnF2amluzUUW4MJz5AWNBpC5s/HdGUu0YEeQT97tuoNJgkVjBGQUlLlcJ7pEarbapjXc5ICawkP20pZaDt9pjewEkHzYCSV/fOIrB5E3yTBJ9CGGjDl5hCzYYX/vMEYXoapjL2vdTBa296kpvr/sac2mTeqN9I9GElydCg/WZdBXmY8H1ha2FfXjEtTS9pQKQW2Uh7vLyHJ6T9v4EZHK9M54i2gvz+Mpi4H/W4dqR29LBzqJDx1OqaNazGtuwNDWrrqDaaQr20wWtvepLz8p+xvymJb6W24fP7j/76+Jt5/fQ8dcQ0UeCu4y16JQfqnY/QTQ72M54RYSLN3Fr1D2egHYvD1H0S6KwnHwKakFmZ/+2a0y++ByNHdpUMJnq9tMI5UPYPFU0BxVTZ3i13k6ytZpKkiSfjHBkO9egbEPGp9KykTUVSJOJwyihRXFN19kQjfTEJc5bgHXwbpQWdcwUBYDnP/c8OY3LVbCa6vTTDah1wUNVXQWXMAQ/MR0nq6uMZ+AANvgB4affEc9mVy3JfOcV86ZXIWdxpPIIRgpi+WJd5E5vji0KDhTU83Hsef8XnqEdpk9GHr0GinofVJFYpLxCUbjE6HjdOVB7HWfkJY63Hm9xZzjasLAKcmhObwaZTPiOW/m25g71A+nUR/7ucjcXO5O4O53gRMZ92+q0f04rQ9j5ACnWk12pC8M2MHh1aF4lJxaQRDSqzdddRU7MVRf4iYtiLS+k+zInCkqN2YRLcpg77BbIzHmhmqtWNfIum/w8c0YwPW0kLwfbY6o07w0zmCzJoEzr6nnVu6ON2xFxk7Ha3rWnTazy7gdyOZcaUaU1wqpmYw3IPYG4/RXLUfT+MREjqKiHN2kQ8MakKoCc/gaMQa9I1uwvdVgs2LoA7dnNn0Lrqa2nUJVDocRFSVsyj1JGRt589VG+kZiiI5ZIhHNS9yY/NuBiJvpn3om+idoTg8/RRZP8R3bRI/ufnfefGVMpo/aiPUK3FoBTOuTOKOW7KCvWWUMTL5p4RICX0NDNUfoqPmANrmoyT0lKKT/gtv6o3J1IZlMOCMxVjaQ/zhSvQeH8JkwpSdjSkvj8H5GVQKQUlVFb29veh0OjIzMzGbzcxLjEJb8ioUveC/KZjOBAtvhPw7afclsnPLk3TUVZOxtJCr7vsuYdFT78J+5TMjnRIy+YLhHoSWIlwNh7DWHcTUcpTwQf/YwKExcjJiPo3GeTj7Qgk/3khqWR0Gj/tLl2F6ZiRTcvo0FouFlpYWhBDMnTsXs9nMgvkZhDTth+Pb4PR74HPDjALIvxOyb8KjMfHJa9s58tZrmCIiWfvt75O+9PIx3EJKsEyNYEgJffXQeARP42EG6w8R2lmCNjA2qDXO4HhkFi26OXhbIf54LVk11ZgE/ssw83IxnXUZpsvlOnMP1urqaqSUJCUlYTabyc7OJsLdCUV/hBMvgq0FQmPBfJs/EAn+3aDm8lJ2bNlMb0sTC1etZdVd92MMDx/zbaQEx5hOIhRCXAM8AWiBZ6SU/3pRVbkc0FIETUfwNh7G23gYg8N/FtmlMWKJyOTYzFtpl0loah2knawhp+Y05tgeQvPyMN1045cuw/R6vdTW1mLZu5eysjLcbjdRUVGsWLGCnJwcpkeHQ9lf4LV/hrq9IDQwbw1c+6+QcS3o/OtxDTrYu30bJ3a+Q2RcPDc99jgpuYsu6mMqU99IHk6pBX4LXA00AUeEEG9JKUsvtLGet37MtFMvAtBgmsHRyHyOJi+k0xVHZFUPZks5hfVHiZ03z79L9OB9/qnWX7ji7Ox7sBYXF/vvwWo0YjabMZvNzJo1C01rERz+JRS/Cs5+iEmBq34GubdD1IzPra/uxDF2Pv0bbN1d5F9zPStuuxuD0XShH0+5hIykx7gMqJJS1gAIIf4EbAQuOBgvyAKOLMymxxFFSlUjecdKuLXvINMXBK45+OY3MC5ciOYcF+X39vaeuQdrV1cXWq2WjIwMzGYz6enp6JxWsLwE77zgv5+qzghZGyH/LphT+KVnsw3abXy07RlKPtrNtOSZ3PZP/86M+Qsu9GMpl6CRBGMG0HjWv5uApRfT2FqrlpUfHyEpM4PQ5XmYvve36JKSvnJy3XD3YJ0zZw7Lly8nKysLU4gBqvfA67+E8nf9A+nkRXDdf0DOzWAc/mbBFQf3sfu5pxiy21i66VaW/c2t6NRdMpSAMTuPIYR4AHgAYPbs2cMuk3X/vXD/vedd17D3YI2PZ82aNeTk5BAdHQ09tfDJr/0D6f5mME2Dy74TGEgvPOe67b09fPDcU1QePsD0ufO46bHHmZ6SenEfWrlkjSQYzcCss/49M/C9z5FSbgW2gv+o1IUW4vP5qK+vx2KxfHYP1ogIli5ditlsJjExEeEZ8g+k39jmH0gjIG0NrP8XmH8t6L58b9ez6qPkw/f58A/P4HG5WHn7PSy+fhOaSX7XbSU4RhKMI0C6EGIu/kDcBtw+VgW0t7efGUT39/djMBjIysrCbDaTkpKCRgj/kax3fhUYSFsheg6s/hnkfQuiZp63DWtHO7ue/g31liJmZC5k3YMPMy15xnl/Tvn6Om8wpJQeIcQPgB34D9c+J6UsGU2jVqvV/wBEi4X29nY0Gg1paWmsW7eOjIwMDAaD/4mdh7dC0R+g/ZR/IL3gBlh0F8xZMaKHnPt8Xk7seId927eBEKz59vfJXXtNUJ/Uo0wNIxpjSCnfBd4dbWNlZWUcPnyY2lr/MxZmzpzJhg0bWLhwIWFhYf4nd9bs8U/PKH8HvC5Izofrfg3ZN4Mp+jwtfKa7qZEdW56gtaKcuXkFrP3OQ0TGffnhiooynAmdRNjW1obVamXVqlXk5OQQ++kt5Hvr4NB/BgbSTWCKgcX3+QfSied+UMpwvB4PR956jYOvbUdvNHHtD37MghWr1GWlygWZ0GCsXLmSVasCv6TuQbC8AkXboPZjQMC8q2Dd/4XM675yIH0u7TVV7Pjdf9LZUEfG8pWsufdBQqNG3ssoyqcmNBg6rRZaT/rHDcWvwJAVomfD6p9C7rcgetb5VzIMt8vJJ69u5+hfXic0KpobfvJT0pcsH+Pqla+Tib0e491H4cjToA2BrBv8Z6RTVo5oIH0uTaWn2Ll1M72tLeRctY4r7rwPY5ia9KeMzsQGY8E3IH6+/4y0aXTXNTgdDvZuf56TO98hanoCN//sF8zJyRujQpWvu4kNRuqV/q9Rqik6wvtP/xe2ni4KrttI4S13oTcax6BARfGbUpe2OvqtfLjtGcr27iF25my+9fivSM7IDHZZyiVoSgRDSnlm0p9zwM6ym77F0k23oJviD0pUJq9JHwx7TzfvP/s7qo8eJCE1nfU/+wXxc+YGuyzlEjdpgyGl5NSeXXz0h2fxut1cced9FGzYqCb9KRNiUgajr72NXVs303DKwsysbNY9+DAxicnBLkv5GplUwfD5vBS99xf2/ekPaLQarv7OD8i5ap2a9KdMuEkTjK7GenY+tZnWqtOkLlrC2vsfIiI2LthlKV9TQQ+G1+Pm8BuvcvD1lwgJDWXDw4+SefkVatKfElRBDUZbVQU7tmymq6GOzMIrWX3PA4RGDn+NtqJMpKAEw+0c4sArL3Ls7TcIi4nhxn/4OfMKLur+CooyLiY8GI0lFnZueZK+9lbMa6/hijvuJSRUPYtOmVwmNBj7X36Bg6/9ieiEJL75839hdrZ5IptXlBGb0GAkpc2n4PpNFN5yB/oQNelPmbwmNBipi5aQumjJRDapKBdFnTlTlGGoYCjKMFQwFGUY4/LgGCFEJ1B/jrfjgK4xb3T0VF0XZqrWNUdKGX++lYxLML6yQSGOjuSJNhNN1XVhLvW61K6UogxDBUNRhhGMYGwNQpsjoeq6MJd0XRM+xlCUqUDtSinKMMYtGEKIa4QQp4UQVUKI/znM+yFCiJcC7x8SQqSMVy0XWNc9QohOIcSJwNf9E1DTc0KIDiHEqXO8L4QQmwM1W4QQE/Kc5RHUtUoIYT1rW/3vCaprlhBijxCiVAhRIoR4ZJhlRrfNpJRj/oX/ATPVQCpgAE4CWV9Y5vvAU4HXtwEvjUctF1HXPcBvxruWL7R5BbAIOHWO9zcA7wECWAYcmiR1rQLenshtFWg3CVgUeB0BVAzz/ziqbTZePcaZRyBLKV3Ap49APttG4PnA61eBNWL8r2cdSV0TTkr5MdDzFYtsBLZJv4NAtBAiaRLUFRRSylYp5fHAaxtQhv/pwmcb1TYbr2AM9wjkLxZ+ZhkppQewArHjVM+F1AVwU6D7fVUIcXHPJhhbI607GJYLIU4KId4TQpz7cbnjJLALng8c+sJbo9pmavD9ZX8BUqSUZmAXn/Vqypcdxz/FIhd4EnhjIhsXQoQDrwE/klL2j+W6xysYI3kE8pllhBA6IAroHqd6RlyXlLJbSukM/PMZoGCcaxqJET1SeqJJKfullPbA63cBvRBiQu55JITQ4w/FH6WUrw+zyKi22XgF48wjkIUQBvyD67e+sMxbwN8GXt8MfCADo6ZxdN66vrAfegP+/ddgewu4O3CkZRlglVK2BrsoIUTip+NCIcSjqiiSAAAAqklEQVRl+H+fxvuPG4E2nwXKpJT/cY7FRrfNxvHIwQb8RwuqgZ8Gvvc4cEPgtRF4BagCDgOpE3RE43x1/RIowX/Eag+QOQE1bQdaATf+feFvA98Fvht4XwC/DdRcDCyeoG11vrp+cNa2OghcPkF1rQAkYAFOBL42jOU2U2e+FWUYavCtKMNQwVCUYahgKMowVDAUZRgqGIoyDBUMRRmGCoaiDEMFQ1GG8f8B4nOonKPWFdsAAAAASUVORK5CYII=\n", 114 | "text/plain": [ 115 | "
" 116 | ] 117 | }, 118 | "metadata": {}, 119 | "output_type": "display_data" 120 | } 121 | ], 122 | "source": [ 123 | "np.random.seed(0)\n", 124 | "for _ in range(6):\n", 125 | " noisy_X = X + np.random.normal(loc=0, scale=.1, size=X.shape)\n", 126 | " plt.plot(noisy_X, y, 'o')\n", 127 | " regr.fit(noisy_X, y)\n", 128 | " plt.plot(X_test, regr.predict(X_test))" 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "metadata": {}, 134 | "source": [ 135 | "As we can see, our linear model captures and amplifies the noise in the\n", 136 | "data. It displays a lot of variance.\n", 137 | "\n", 138 | "We can use another linear estimator that uses regularization, the\n", 139 | ":class:`~sklearn.linear_model.Ridge` estimator. This estimator\n", 140 | "regularizes the coefficients by shrinking them to zero, under the\n", 141 | "assumption that very high correlations are often spurious. The alpha\n", 142 | "parameter controls the amount of shrinkage used.\n", 143 | "\n" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 6, 149 | "metadata": {}, 150 | "outputs": [ 151 | { 152 | "data": { 153 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAANAAAACPCAYAAACLfFVWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VFX+x/H3mUxmEtJ7Qui9l0gJiPTeQhEQUcSOirqoKGLbVVdUFPta1r76QxAQsCCsKyJFeiAEUkjvPZlkkulzfn8MIigKmpAJcF7Pk+eZmXuY+z2XfJg7h3vuEVJKFEX5azTuLkBRLmYqQIpSDypAilIPKkCKUg8qQIpSDypAilIPKkCKUg8qQIpSDypAilIPWnftODQ0VLZp08Zdu1eU33Xw4MEyKWXY+bR1W4DatGnDgQMH3LV7RfldQojs822rTuEUpR5UgJTLUkNdRK0CpFxWSq02bjqQxKspWQ3yfm77DqQojckhJe9nF7E8oxCzBE1hHnRpW+/3VQFSLnkJ1bXccySNZLukeVU5d3lL5o0Z2iDvrQKkXLJq7A6eTMrik9JqvGwWZpTlsmzYIFq0aNFg+1ABUi45Uko2FFewLCmbSinoWZTNQ20iGDF6Bh4aDRzbAL6R0Dq23vtSAVIuKZl1Fv6WcIK9JjuhNdXcZ63i9kkjCAgIgIoM2Hg3ZO+EyJ6wcGe996cCpFwSzA4nL2fk83puGRqHnVGFmSzr35PuXYeD3QLfL4edL4DTDhpP6Dm7QfarAqRc9H4or+b+xAzyndChNJ+7A/VMmxOHXq+HjO3wxe1QU+hq3HEsTHoRAls1yL5VgJSLVrHFxtJjmWw21OFfZ+SGynz+NmooUVFRYCyBdfdA6mZXY/8WEPc6tB+BJT0dYchG17p1vWtQAVIuOg4peS+nhOUZBVidTgbmZ7Ckc2sGT5iFBgk/vQHf/QMcFvDQwYhlMGgRDpOFsudXUPHxx/iNGEGL116tdy0qQMpFJb66jr8dTSPF6qRFRQk3YmL+9PH4+flBfjysXQCVWa7GnSfB5JVI3wiqv/qKkudXYC8rI/DqmYQtXtwg9agAKRcFg83OU6m5fFpchbfVTFxhBvdf2Z9OnTqB2QBrb4HEz12NA1rCjH9D60GYk5MpeuoBTAcP4tWzJy3eeB3vXr0arC4VIKVJk1KyrriSx5KzqXJCj4JM7o4MYPy8Weg8PSH+P/DNg2CrAw89jHocYu/AUWOk9KmnqVy1Co+AAKKefoqAGTMQGg1WUx12m41m/gH1rk8FSGmy0urM3JeYwb5aK+HVldxZU8ztY0cRHh4OZSdg9XVQmuxq3HUKTHkV6RVI1bp1lK58CYfBQNDcuYTdczceAQFY6mqJ3/wlB7/ZSIf+gxi38J5616gCpDQ5JoeTlzIL+FdOKR52GyPzTrC4d1eu6DscjcMKm+6FQx8BEgJbw6wPIToGU0ICRU8txHz0KN5XXEHkY4/i1aULJmMNh9Z8SvzmTVjqamkX059eo8c1SK0qQEqT8r/yapYcy6TAIelYksfNeidXz47D19fXdQnOpkVgqQGtHkY/CQNuw15VRcmjj2JYuw6PsFCar3ge/8mTMdVUs2PVRxze8hVWk4kO/QcRO2MOEe06NFi9KkBKk1BgtvJwcjZbKmsJrKthXnEmdw0fQrt27cCQB29PgsLDrsZdp0Lc60itD5WrVlH6yqs46+oIvukmQu+8A7Pdxo+ffsCRrd9gs1roFDuE2OmzCWtd/+kLv6YCpLiV3Sl5N6+E59JP/p9OzgkWtYlkxPxr0WoEbF4K+94G6YTANjD7Y2jem7oDByh66mksKSn4DB5ExCOPYAsK5Md1q0j47lscNhudB19F7Iw5hLRomKsOzkYFSHGbA4Za7juWSarFTqvyYq4xVzJ/yhhCQ0MhdQusvw3MVa7TtTFPwYDbsJWUUrLkQaq//BJtVBTRr7wCMX3Y+eU6jn6/FafDQberRjBg2myCm0df8D6oACmNrtJm58kTeawqrsLHYmJKbgp39e9D797jELWl8O4YyNvnatxlCkx/Cyl0VLz/AWVvvIG02Qi5YyGe06exd8uXJH70L0DSfdgoBkybTWBEZKP1RQVIaTRSStYUVfJ4ag7VDie98jK4JVDP5HmzaebtDd/9HXa/BtLhGl2b8wlE9aJ2926Knv4n1owMfIcPR3/rTcTv383xpfcghKDnyDEMiJuFf1h4o/dJBUhpFCm1Zh44nsV+o5kIQzlzynK4ecxIWrdu7bpieu2NUFfu+s/QMc9A7EJs+fkU33MvNVu34tmqFb7L/0lifiZJK55Co/Wg95iJ9J86E7+QULf1SwVIuaDqHE5WZhbyZm4JnjYbI7KTWNi5HVdOvh6ttRo+mADZu12NO0+Eme/hlB5UvPkmZW+/A4D21ptJ1thJWf0BWk8dMROn0m/KDHyDgt3YMxcVIOWC2Vpm4KGkbArtTjoX5jCXOmbPnEJwUBBsWw47X3RNcAtoCXM+hea9qdm2jeJnlmPLzcU+cjgZLcJI2/cDnnov+k+ZQb/J02kWEOjurp2iAqQ0uDyzlYeTc/hvpZGg2mquyUvlliGxdO/eHZG9G95bALUlrqkGY5+GwXdjzc6m+PaFGLdvp65je7KmTyArIxVdXRkDp88hZuLUBrl2raGpACkNxuaUvJ1bwgsZhTgcDgZmJ3FTZBDjFlyHl9MEH02FrB9djTuOhas/wOnQUPbyy1S89z5Vfj5kj76SvNIi9MX5DLr6WmImTMXL19e9HfsDKkBKg9hbZeSBpGxOmG20KStkmqGQ68aPpUV0NGx/Dn5c4Tpd82sOc/6DjL6Cmi1bKX7uOUqqK8mK6UqRuRYvUy1DrplPn3GT0DfzcXe3zkkFSKmXcqudJ9PyWV1ciZ+ljkkZx7i5T3cGzhiLR+4eeHEkGItdN/IY9TgMuQ9LejqFN95I3tEjpLdrQVl4M7x1Wq6asYA+Yyei827m7m6dNxUg5S9xSsmqwgqePJFHjcNBn9w0rtE7mXbtLAJ1Ej6ZAZk/uBq3HQ6zP8Lh0FL63POkfrGW9MggKjpE4xMYyPCpM+k1ajyeXl7u7NJfogKk/GnHjSaWJGVz0GgmqqqMmYXpzB85jC6dOsGOFbD9edfpmk84zPoQ2Xowhk2bOPray6R6eVDVJgLfwCBGTJtNz1Fj8dTp3d2lv0wFSDlvtXYHKzKLeCevFL3NyoiMRG5oG82ImxegLzwAK7u4TteEBwxdAsOXYUpOJv7aORw3VmAI8cHXP5DRs6+l+/AxaD093d2lelMBUs5JSsnmMgPLUnIpsjnoWpBFnKmCOXETiPLTwuo5kLHN1bjVIJj1MXablkMP3s/h1ERqvPX4hYUxZt4NdB82Cg/txR+cn6kAKX8o22RhWWoe/6uoIaTWwOzM41w38Ar6xUxEs3PlL6Nr3kEw4x0cbUdw+NWXOLjje2p0WvwCAhg7bwHdx0xA4+Hh7u40OBUg5aysTidv5pSyMqsQabczKDOJOYFeTLxhHv5lh+DlHidP1zQQexfOEY+RsHYNex+ZgVFI/HQ6xlw9lx4zZ6PRXHrB+dk5AySEeB+YDJRIKXucZbsAXgEmAnXAAinloYYu9HKVureInzamY6yw4BusZ1BcezoNdF2uXxtfQvWWLBxVFjwC9ST3OEiFx/sE6CowWIPRB93F5NgFp95rQ3w+K7akUFBlonmgN0vGdWZa32i+zviaVw69QlFtEZE+kYzrtoT1huakm220K81nfGkOc8eOpmN4M/ji+l9G16L64pjxHol7E9hz8zyMNit+dgcjR46n9x2LLslPnF8T51orUggxFDACH/9OgCYCd+MK0EDgFSnlwHPtuF+/flKt0v3HUvcWse3TZOxW56nXtDoNI+Z1IVqnoWr9CaTNtc0QuZvi7h8iPayn2locnjj9ljI5dgEb4vN5eP1RTDbHqe3enh7MHuLgm9K/Y3aYcWr8MQbOxeI7BH9zLVelHWVu1w4MHTwQ3Z5XYedK1+mazg/HxJdILGnGnv/7GKOpFn+ThT49Yui79FG0/n6Nd5AuACHEQSllv/Npe85PICnlj0KINn/QJA5XuCSwRwgRKISIklIWnle1yu/6aWP6GeEBsFud/LQxnbH+nqfCA1DWcd0Z4QHQe9ioqnwDWMCKLSlnhAfAZHPw6a5qvNpbMPuOpC5gDlLoiclOoUPpAR6Z/wARVYfhjRjXtWuAvff1JOpGsvf1tRhrqgmoM3NlUCR9nvkHXu3bX5gD0YQ1xHegaCD3tOd5J1/7TYCEELcBtwG0anXh5qlfKowVlt993eH8VbC8ys/aNkBXAUBBlems2+1WP6oinsCub0/zyhJi07eT672bE4FpRGxOgqwdrnbBXUiIuIl9m3dQa3ifoFoTsTZBj/sfxH/MGFxn8pefRh1EkFK+A7wDrlO4xtz3xcg3WH/WEPkG6/Hw98RR9cs2rTkEu/dvQ2SwuubMNA/0Jv8sIXJ6eaAVzRmadABd3S6OBB3hemMpN5XXgCzA4eHN4dAF7IvPo27XlwTXWehRaqDT3HmE3nYbGm/vBuzxxachlrnPB1qe9rzFydeUehoU1x6t7sy/Iq1Ow6C49viPa4Pw/GVb6ImZCIfujLYWhyf6oLsAWDKuM96eZ36plxpBRKiFKfFfUiQ+JajZj3xemMXCqmpKhAeHPEfzdtYIfvghEZ/ySgam5TO6RUcGrFlL+L33XvbhgYb5BNoELBJCfIZrEMGgvv80jJ9H235vFA44NQoXbB5BicWfyt8ZhZvWN5pSq40VW1Kw1NnR6AV99EXc270t+LXixm0fE1NbQ4bWk28tbUkvaoe5zkSkTkOfE3mEh0UQufJlfIcNc8ehaLLOZxRuFTAcCAWKgScATwAp5Vsnh7FfB8bjGsa+UUp5zuE1NQrXeCxOJ69nl/BKdhHC4aBf+jEm6ZzEjR1GSMI7sPctpHSSURvKt0XdMVudtAgJp/WhRAItdkIXLiT4xgVodLpz7+wS0NCjcHPPsV0Cd51nbUoj+7GihodScsk0W+lQksfIgnRmjhhGL5IQ/xmBNFVSZPLlq8LuVFt0tGnbntaHj+Nz5Cf8Jown4sEH8YyKcnc3mix1JcIlqthi44m0fDaUVBFkqWNSSjxxbVsyZlos3t/dB/kHqbbp+aagF/mmADr0jCE2Mw/thm/RdWhP5Icf4BNb/2XgL3UqQJcYh5R8mF/G8vQCzA4H/bJTGGOqZMaEq2iZ+gHyow+wOjVsL27PUUMUnfsPZqoFHKtWI3Q6Qh96iODr5iEugSulG4MK0CXkcHUdD6bkkmA00bKqlGHpicQN6MtgfS1i7SSclmqOVEaxu6w17fsNYlZ0F8xvvo29qIiAuDjC7r8Pz/DGvznhxUwF6BJgsNlZnlnER/ll+NqtjE49wrgAb+Im9MTn+6V4lCeRW+vPtuK+RHbpzax512B68x2M76xG37Ur0StfpFlMjLu7cVFSAbqISSlZX1zJ39PyKbPa6VGQwfCSHGYOjaFt6nvoP99IjU3HjpJO6CI7MGXZQ1g3bKH85tvQ+PgQ+cTjBM6ejbgMLvq8UFSALlJpdWaWpuSxs8pIZK2BGcmHmNS5HUMCq/H+6mo00sb+imjqPCMYcvcyZGYdxbfcjaO8nMBZswhb/De0QUHu7sZFTwXoImNyOHk1u5jXc4rxcDi4Ku0ow50mJvbwIXTfffg5y8gyBlJsDaJ73PVoI8dS9MyzmA4dwqt3L1q++SbePX9zUb3yF6kAXUS+L6/m4dQ8ss1WOpfkMSQ7iXEdwmh//AOii1MwWPUk1oXTrk9/Wo58jJIP1lC1+ho8AgOJ+uc/CZg+DaFpiKu3lJ+pAF0ECi1WHjuRz1elBkIsJqYkH2Sojwd9rD/Q5bDrxuzZdQGEhwfS/frnqUqoJm/2TTiqqwmaN4+wuxfh4e/v5l5cmlSAmjC7U/JefinPZxRidTjpn5XE4PI8+pkT6V+yhUCdmXK7Lz6eNlpPuBVTwFiyHnsBc2Iizfr1I+KxR/Hq3Nnd3bikqQA1UQcNtTyYksuxWjNtq0oZnBpPb3MeV9Z8SXvfCkyeXlgdGkI69cU++HEKPtyIYf0CtOHhNH/hBfwnTbxs5+g0JhWgJqbSZueZjEI+KSjH325lbEo8vcqyuLJyMwMCs8FXg11q8PbxRY55moqjNkrnLcJpNhNyy82ELLwDD9+mf0/pS4UKUBPx8/KHT6bnU2mz0ysvg/4Zx7iicgdj/BLwDTLj9PBG4zBB/5upDZhM8ROvYElNxWfwYCIefQR9u3bu7sZlRwWoCUipNfNQSi57DLU0N1YxI/kQPYrjmaz7iRaBpUidL1hBE9YRW+zjlPxnC9Vf34Vn8+ZEv/YqfqNHq9M1N1EBcqM6h5OXsopcyx/a7QxNO0rfnKOMlzvp43sCtDpwaBES5OhnqDjuSelNS8FuJ/TOOwm59RY1K9TNVIDcZGuZgWWpeeRZbHQpzCY2PYHhpr2M8klAa6tG6P3AUg3dp2MMmEHx029izcrCd+RIIh5eiq5ly3PvRLngVIAaWZ7ZyiOpeWwprybEaCAuLYGB5UeYFRCPnyMNtMFgk9AsBOvQFyn5bAc1/30Iz9ataPn2W2pKdROjAtRIbE7J2znFvJBZiMPuWv5wcN5Rrgk8TivtToTNF7ReYKnBOegBylP8Kb/zGdBoCFu8+LKaUn0xUQFqBD+VG1iccIIstLQpyWdI+lFmeiUT67UNYagBn1CoLUW2HYYxYA7FKz7ElpenplRfBFSALqBSk4kHdsezRdMMvzoz4zMSGV97jOk+e9FXprjWCzUbQGiwDn6eos8PUrvjaXQd2tPqww/xiT3nHZIVN1MBugCsFgsrf9zF2w5vLJ56+uakMjwnnhtCjhFp3g4iGLwCoKYAZ++bKDsRQcX9byD0eiIeXkrQtdeqKdUXCRWgBmSzmNn0/fc8a5TkhkYTbSxl6JF4rvVK5ArNd2jKrBDYCqpykJG9qfG7huJXPndNqZ42jfD770MbFububih/ggpQA7CaTez577e8lFvGns4xePvbGZFyiGmV+4jz3IVXZQ4Et4eqHDBVYem1lKKNydTtfQ19t65Er1xJs5i+7u6G8heoANWD1VTHoS1f83+HE9l8xUhquralZ2keo1N+5HrfAzS3HARdJPhGQEU6jo7TKUtvRcVjn6Lx9SXy708QOGuWmlJ9EVMB+gvMtUbiv/2S/23/gW9iRpAxbDpRljrGH/of1zl20Z+daGo1EN4NSo4jg9tjaH4jJW9+g6N8H4GzZxP2t3vVlOpLgArQn2Ay1nDom43s//YrdnW6gj1Tb0Wj0TAk8zhX521lmm4XXuYSiOwF5elQno65/UKKvs7GdPhj15Tqt97Cu0d3d3dFaSAqQOehrtrAwa++IH7L16QHR7J91iKKvHzpVlPBuKObuU67g2hnEni3Ae92UJSAPWoYpZkdqHrmSzyCg4l65hkCpsWpKdWXGBWgP1BbVcmBr77g8NavMWg8iZ+6gJ+Cowlz2pme8AMLjFvob9+HEN7QciDk7kX6RFEVcAel7/+Aw5hB0PXXEbZITam+VKkAnYWxopz9m9aR8N232O0Oiidfw4aW3ahzSmILMpibtpY47W68bJXQajCUpUDefuoi5lC8tQRz0kaa9e9PxKOP4tW5k7u7o1xAKkCnqS4rYd/GdSRu24rT4cBn9FTWdYslwWyng8nIlPgNXO/8L81lJgR2BY9WkLMbe2AfSgrGYli13TWl+sUX8J+oplRfDlSAAENJEfs2rCXxh+8AaDdyLLv6j+aTijr8zDamJe/mptK19HceBu8AaD0W0r9HevpQqZ1H6Sf7cVp2E3LrLYQuXIjGR02pvlxc1gGqLCpg7xdrOP7j92g0GnqMGkf5sIk8UWykuKKOfhUFLDj2IZPELvSyDtFxDBQfg7St1PqOp3hbDZb0bfhceSURjzyCvl1bd3dJaWSXZYDK83PZ+8Uakndux0Orpc+4SYSPi+OfJbVsy62klcPKvYfWcIP5a6KcBcjmMQi9L5zYgs2rAyWFcVRv349n8+a0eP01fEeNUqdrl6nLKkBludnsWb+alJ92oNXpiJkUR69J0/nAYOXVlGI8kEzL3M+tOR9yBYlInwhoNwuR/A3S7qDcOpWyjcfAcYTQu+5yTan28nJ3txQ3Oq8ACSHGA68AHsC7Uspnf7V9AbCCX1bnfl1K+W4D1lkvJVkZ7Fn/GSf27sbTy5v+U2fSb9I0Djg8mJyaR4bJQt/acm6P/xcTnDvRCTv0mI0oOgpHPycrvDtZUWXsrLXwhed9VFgCaV7bjCVJ5UzrG82uLVvw2WUl2BpAhc5A7ZU6rhw3jg3x+axZl0KPcif+UoPWV8vIWZ3OWCRYubidzyLDHkAqMAbIA/YDc6WUx09rswDoJ6VcdL47boxFhovST7Bn/WekH9iLzrsZMROmEDMxjmqd96nlDyNxcN3h1cyvXk+4LEW2HoLwCYXjG7DSnBMyiuLYbHaXX8HHx+didf4yK9Tb04MbOziZkazHS/7yullY+am7gc+P+DKyxgNPfjm9E1rB6Ou7qhA1YQ26yDAwAEiTUmacfPPPgDjg+B/+KTcqSE1mz/rPyIw/gN7Hh8Gz5tF3whQ8m/nwYX4Zz2ZkYXY4ics/zF1p/6IXKTj9W0Dn2xCJa3EaDZTXjKJ8Wwalj+Ug9fBF2pQzwgNgsjn4PElyLWe+7iV1dEvWEmvUnBEeAGmX/LQxXQXoEnE+AYoGck97ngecbarkTCHEUFyfVoullLm/biCEuA24DaBVq1Z/vtpzyEs+xp51n5GdEI+Xnz9DrplPn3GT0Tdr5lr+8GAqCTUmelkN3L3/JcbZd6HVADE3oylKRO57B6OlJ8V7WmIrSsJ/4gTygzcBUG4++4WfZZx98CDMHoi/tJ11m7HC0iD9VdyvoQYRvgRWSSktQojbgY+Akb9uJKV8B3gHXKdwDbFjKSW5x46yZ90qco8fpVlAIEPn3UjvsRPReXljsNn5e2oeH+aXESTgoWOrmF+2mhAqcXYYhyagORz6CIvJj+L0IdQmZKDv2IFWz67AJ3YgWbv2Y7YUEOJVSbk5+Df7D+Xs3SjVVlEtmhEgfxsw32B9Q3RdaQLOJ0D5wOk3IWvBL4MFAEgpy097+i7wfP1L+2NSSrIT4tmz/jPyk4/jExTM8Pm30mv0ODz1XqeWP3wiLZ9yq51ppYncffwlupGGPaAt9LkdzeH/w5m0lbLygZTvLEDjVULEsocJmjv31JTqdu0fIDn5EaZ3+PKs34FmdXBiTrb+5jvQ8S529hxxMrJG/OY70KC49hf68CiN5HwCtB/oKIRoiys41wDXnt5ACBElpSw8+XQqkNSgVZ5GSknm4QPsWfsZhWkp+IaEMvKmhfQcMRbtyds+pdWZeTg1jx2VRro5jKzYt4LR1h0ID0+csYvRliYjf3iWmqr2FB/shr0sh4Dp011TqkNDz9hfVGQcADrPF4DP2JAeR7nJn+aBzVgyrvNpo3B1Z4zCXT9uBn7d1Sjcpe6co3AAQoiJwMu4hrHfl1L+UwjxJHBASrlJCLEcV3DsQAVwh5Qy+Y/e88+OwkkpST+wlz3rP6M4Iw3/sHAGxM2i+/DRaE9+Wvy8/OEbOSXopJN7UlYzv+gTAqnB1iUOz9AOsPdNzJWC4hNdqEspxKtbNyIee5RmfdWUasXlz4zCnVeALoTzDZB0Ojmxbzd71q+mNDuTgIhIBk6fTberRuKh/eUD9PTlD6dXHWdxwnN0kllYgjqij70NDn6AIy+JsvxeVBwox8PXj7DFiwmcdbWaUq2coaGHsd3C6XSQ8tNO9q5fTXleDkFR0Uy46z66XDkMzWm/8Kcvf9jZaeSTA88xwrQTp9Ybx1WPoa9IR36zBENxNCXxHXFUlxE4ZzZh96op1Ur9NbkAOR0OkndtZ88Xa6gsyCOkRSsm3rOEzoOGoNH8EpxTyx9mFmF3OHgifQ3X5X2EL3WYu87Cu1Uf+HEFpkITxSk9MWWW4927PRGPP4Z3dzWlWmkYTS5Axspytrz1CiHRLZmyeCkdBwz+zTTog4ZaHkzN5ZjRzLTq4zx05BnaOnOpDeoKI+7He/+/sW/6nNLMjlQdqcUjGKKWLycgbqqaUq00qCYXIP/QcOY98xJhrdr85pf99OUP2zlqWH34Oa4y7sSm9cUycjk+tfnIdbdTmRNM6dF2OEwmgudfT+iiRXj4+bmpR8qlrMkFCCC8zZlLFUop+by4kn+kFVBttfBU9hrmZn+ENxaMXefi32UY/O8p6jKKKTreAUtBDc0G9CTi0Ufw6qSmVCsXTpMM0OlSas0sTc3lp6paZtQc5eEjz9DSUUBVYA/04x/D/+D72FfdQUlKKwxJYWgjmhG98u/4TZig5ugoF1yTDdDpyx+2slawPvEFBlfvos4jAOOYlQQ6KpFrFlCR6kXpsVY4bU5Cbr2V0IW3qynVSqNpkgHaWmZg2Yk8iupMPJ23mmsyP8ITGyWdryPsijjE1kepPZZJcWILLKVmfK4aSMSyh9G3VVOqlcbV5AKUZbKw4Ggm06oP8fjRZ4m0F1Pi3wv/KcsJT/wE27tzKDkWRXV6KJ7RIbR442F8R45Up2uKWzS5AIVVF/Lfw0vobthHtUcwpaNfJVxvx7lmHmUJTsqSokFoCV10KyG33KymVCtu1eQCpPX2pUXNCbLbX0eLq+bj/90jGPcnUJwQibXShu/o4UQsXYquRQt3l6ooTS9Aev9QtPcdIGDni1jfmETBkRCMOSHo2kTT8vlH8L1qiLtLVJRTmlyAqMhE/HsipftqKE+JAE894Q/cSfD8+Qi1SrXSxDS5AFmNHuRs8MJWCf6TJhD+4BI8IyLcXZainFWTC5BndDTeQ8YQdfXV+Awc4O5yFOUPNbkACQ8Poldc8BnhitIg1KXJilIPKkCKUg8qQIpSD267J4IQohTI/oMmoUBZI5XzZ6i6/pyLsa7WUsqw83kTtwXoXIQQB873xg6NSdX151zqdalTOEWpBxUgRamHphygd9xdwO9Qdf05l3RdTfY7kKJcDJryJ5CiNHluDZAQYrwQIkUIkSaEWHqW7XoYmz2iAAACfUlEQVQhxOqT2/cKIdo0kboWCCFKhRCHT/7c0kh1vS+EKBFCJP7OdiGEePVk3QlCiJgmUtdwIYThtOP1eCPV1VIIsU0IcVwIcUwIce9Z2tTvmEkp3fKD60b16UA7QAccAbr9qs2dwFsnH18DrG4idS3AtQ5sYx+zoUAMkPg72ycCmwEBxAJ7m0hdw4Gv3HC8ooCYk4/9cC3+9uu/y3odM3d+Ap1aOlJKaQV+XjrydHG4FusCWAuMEhf+5gfnU5dbSCl/xLX6xe+JAz6WLnuAQCFEVBOoyy2klIVSykMnH9fgWnYn+lfN6nXM3Bmgsy0d+evOnWojpbQDBiCkCdQFriUtE4QQa4UQLc+y3R3Ot3Z3GCSEOCKE2CyEaPSbk588/e8L7P3VpnodMzWI8Nd8CbSRUvYC/ssvn5LK2R3CdXlMb+A1YENj7lwI4QusA/4mpaxuyPd2Z4DOuXTk6W2EEFogACjnwjqvJS2llD+vFPwucMUFrul8nc8xbXRSymoppfHk428ATyFE6Dn+WIMQQnjiCs+nUsr1Z2lSr2PmzgCdWjpSCKHDNUiw6VdtNgE3nHx8NfC9PPnNz511/eoc+YIuafknbQLmnxxZigUM8pelN91GCBH583dXIcQAXL93F/ofQk7u8z0gSUq58nea1e+YNfbIyFlGQFJxjXo9cvK1J4GpJx97AZ8DacA+oF0TqWs5cAzXCN02oEsj1bUKKARsuM7VbwYWAgtPbhfAGyfrPgr0ayJ1LTrteO0BBjdSXUMACSQAh0/+TGzIY6auRFCUelCDCIpSDypAilIPKkCKUg8qQIpSDypAilIPKkCKUg8qQIpSDypAilIP/w8aMwSpNIeF0wAAAABJRU5ErkJggg==\n", 154 | "text/plain": [ 155 | "
" 156 | ] 157 | }, 158 | "metadata": {}, 159 | "output_type": "display_data" 160 | } 161 | ], 162 | "source": [ 163 | "regr = linear_model.Ridge(alpha=.1)\n", 164 | "np.random.seed(0)\n", 165 | "for _ in range(6):\n", 166 | " noisy_X = X + np.random.normal(loc=0, scale=.1, size=X.shape)\n", 167 | " plt.plot(noisy_X, y, 'o')\n", 168 | " regr.fit(noisy_X, y)\n", 169 | " plt.plot(X_test, regr.predict(X_test))\n", 170 | "\n", 171 | "plt.show()" 172 | ] 173 | } 174 | ], 175 | "metadata": { 176 | "kernelspec": { 177 | "display_name": "Python 3", 178 | "language": "python", 179 | "name": "python3" 180 | }, 181 | "language_info": { 182 | "codemirror_mode": { 183 | "name": "ipython", 184 | "version": 3 185 | }, 186 | "file_extension": ".py", 187 | "mimetype": "text/x-python", 188 | "name": "python", 189 | "nbconvert_exporter": "python", 190 | "pygments_lexer": "ipython3", 191 | "version": "3.5.2" 192 | } 193 | }, 194 | "nbformat": 4, 195 | "nbformat_minor": 1 196 | } 197 | -------------------------------------------------------------------------------- /MAIN_tutorial_machine_learning_with_nilearn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# Let's keep our notebook clean, so it's a little more readable!\n", 12 | "import warnings\n", 13 | "warnings.filterwarnings('ignore')" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": null, 19 | "metadata": { 20 | "collapsed": true 21 | }, 22 | "outputs": [], 23 | "source": [ 24 | "%matplotlib inline" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": { 30 | "collapsed": true 31 | }, 32 | "source": [ 33 | "# Section 2: Machine learning to predict age from rs-fmri\n", 34 | "\n", 35 | "We will integrate what we've learned in the previous sections to extract data from *several* rs-fmri images, and use that data as features in a machine learning model\n", 36 | "\n", 37 | "The dataset consists of 50 children (ages 3-13) and 33 young adults (ages 18-39). We will use rs-fmri data to try to predict who are adults and who are children." 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "### Load the data\n" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "# change this to the location where you downloaded the data\n", 56 | "wdir = '/Users/jakevogel/Science/Nilearn_tutorial/reduced/'\n" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": true 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "# Now fetch the data\n", 68 | "\n", 69 | "from glob import glob\n", 70 | "import os\n", 71 | "data = sorted(glob(os.path.join(wdir,'*.gz')))\n", 72 | "confounds = sorted(glob(os.path.join(wdir,'*regressors.tsv')))" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "How many individual subjects do we have?" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "#len(data.func)\n", 89 | "len(data)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": {}, 95 | "source": [ 96 | "### Extract features" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "Here, we are going to use the same techniques we learned in the previous tutorial to extract rs-fmri connectivity features from every subject.\n", 104 | "\n", 105 | "How are we going to do that? With a for loop.\n", 106 | "\n", 107 | "Don't worry, it's not as scary as it sounds" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "# Here is a really simple for loop\n", 117 | "\n", 118 | "for i in range(10):\n", 119 | " print('the number is', i)" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "container = []\n", 129 | "for i in range(10):\n", 130 | " container.append(i)\n", 131 | "\n", 132 | "container" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "Now lets construct a more complicated loop to do what we want" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "First we do some things we don't need to do in the loop. Let's reload our atlas, and re-iniate our masker and correlation_measure" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "from nilearn.input_data import NiftiLabelsMasker\n", 156 | "from nilearn.connectome import ConnectivityMeasure\n", 157 | "from nilearn import datasets\n", 158 | "\n", 159 | "# load atlas\n", 160 | "multiscale = datasets.fetch_atlas_basc_multiscale_2015()\n", 161 | "atlas_filename = multiscale.scale064\n", 162 | "\n", 163 | "# initialize masker (change verbosity)\n", 164 | "masker = NiftiLabelsMasker(labels_img=atlas_filename, standardize=True, \n", 165 | " memory='nilearn_cache', verbose=0)\n", 166 | "\n", 167 | "# initialize correlation measure, set to vectorize\n", 168 | "correlation_measure = ConnectivityMeasure(kind='correlation', vectorize=True,\n", 169 | " discard_diagonal=True)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "Okay -- now that we have that taken care of, let's run our big loop!" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "**NOTE**: On a laptop, this might a few minutes." 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "all_features = [] # here is where we will put the data (a container)\n", 193 | "\n", 194 | "for i,sub in enumerate(data):\n", 195 | " # extract the timeseries from the ROIs in the atlas\n", 196 | " time_series = masker.fit_transform(sub, confounds=confounds[i])\n", 197 | " # create a region x region correlation matrix\n", 198 | " correlation_matrix = correlation_measure.fit_transform([time_series])[0]\n", 199 | " # add to our container\n", 200 | " all_features.append(correlation_matrix)\n", 201 | " # keep track of status\n", 202 | " print('finished %s of %s'%(i+1,len(data)))" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": { 209 | "collapsed": true 210 | }, 211 | "outputs": [], 212 | "source": [ 213 | "# Let's save the data to disk\n", 214 | "import numpy as np\n", 215 | "\n", 216 | "np.savez_compressed('MAIN_BASC064_subsamp_features',a = all_features)" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "In case you do not want to run the full loop on your computer, you can load the output of the loop here!" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": { 230 | "collapsed": true 231 | }, 232 | "outputs": [], 233 | "source": [ 234 | "feat_file = 'MAIN_BASC064_subsamp_features.npz'\n", 235 | "X_features = np.load(feat_file)['a']" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "X_features.shape" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "Okay so we've got our features." 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "We can visualize our feature matrix" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "metadata": {}, 265 | "outputs": [], 266 | "source": [ 267 | "import matplotlib.pyplot as plt\n", 268 | "\n", 269 | "plt.imshow(X_features, aspect='auto')\n", 270 | "plt.colorbar()\n", 271 | "plt.title('feature matrix')\n", 272 | "plt.xlabel('features')\n", 273 | "plt.ylabel('subjects')" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "### Get Y (our target) and assess its distribution" 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": null, 286 | "metadata": { 287 | "collapsed": true 288 | }, 289 | "outputs": [], 290 | "source": [ 291 | "# Let's load the phenotype data\n", 292 | "\n", 293 | "pheno_path = os.path.join(wdir, 'participants.tsv')" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "from pandas import read_csv\n", 303 | "\n", 304 | "pheno = read_csv(pheno_path, sep='\\t').sort_values('participant_id')\n", 305 | "pheno.head()" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "Looks like there is a column labeling children and adults. Let's capture it in a variable" 313 | ] 314 | }, 315 | { 316 | "cell_type": "code", 317 | "execution_count": null, 318 | "metadata": {}, 319 | "outputs": [], 320 | "source": [ 321 | "y_ageclass = pheno['Child_Adult']\n", 322 | "y_ageclass.head()" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "Maybe we should have a look at the distribution of our target variable" 330 | ] 331 | }, 332 | { 333 | "cell_type": "code", 334 | "execution_count": null, 335 | "metadata": {}, 336 | "outputs": [], 337 | "source": [ 338 | "import matplotlib.pyplot as plt\n", 339 | "import seaborn as sns\n", 340 | "sns.countplot(y_ageclass)" 341 | ] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": {}, 346 | "source": [ 347 | "We are a bit unbalanced -- there seems to be more children than adults" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "pheno.Child_Adult.value_counts()" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "metadata": {}, 362 | "source": [ 363 | "### Prepare data for machine learning\n", 364 | "\n", 365 | "Here, we will define a \"training sample\" where we can play around with our models. We will also set aside a \"test\" sample that we will not touch until the end" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "We want to be sure that our training and test sample are matched! We can do that with a \"stratified split\". Specifically, we will stratify by age class." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": {}, 379 | "outputs": [], 380 | "source": [ 381 | "from sklearn.model_selection import train_test_split\n", 382 | "\n", 383 | "# Split the sample to training/test with a 60/40 ratio, and \n", 384 | "# stratify by age class, and also shuffle the data.\n", 385 | "\n", 386 | "X_train, X_test, y_train, y_test = train_test_split(\n", 387 | " X_features, # x\n", 388 | " y_ageclass, # y\n", 389 | " test_size = 0.4, # 60%/40% split \n", 390 | " shuffle = True, # shuffle dataset\n", 391 | " # before splitting\n", 392 | " stratify = y_ageclass, # keep\n", 393 | " # distribution\n", 394 | " # of ageclass\n", 395 | " # consistent\n", 396 | " # betw. train\n", 397 | " # & test sets.\n", 398 | " random_state = 123 # same shuffle each\n", 399 | " # time\n", 400 | " )\n", 401 | "\n", 402 | "# print the size of our training and test groups\n", 403 | "print('training:', len(X_train),\n", 404 | " 'testing:', len(X_test))" 405 | ] 406 | }, 407 | { 408 | "cell_type": "markdown", 409 | "metadata": {}, 410 | "source": [ 411 | "Let's visualize the distributions to be sure they are matched" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": null, 417 | "metadata": {}, 418 | "outputs": [], 419 | "source": [ 420 | "fig,(ax1,ax2) = plt.subplots(2)\n", 421 | "sns.countplot(y_train, ax=ax1, order=['child','adult'])\n", 422 | "ax1.set_title('Train')\n", 423 | "sns.countplot(y_test, ax=ax2, order=['child','adult'])\n", 424 | "ax2.set_title('Test')" 425 | ] 426 | }, 427 | { 428 | "cell_type": "markdown", 429 | "metadata": {}, 430 | "source": [ 431 | "### Run your first model!\n", 432 | "\n", 433 | "Machine learning can get pretty fancy pretty quickly. We'll start with a very standard classification model called a Support Vector Classifier (SVC). \n", 434 | "\n", 435 | "While this may seem unambitious, simple models can be very robust. And we don't have enough data to create more complex models.\n", 436 | "\n", 437 | "For more information, see this excellent resource:\n", 438 | "https://hal.inria.fr/hal-01824205" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "First, a quick review of SVM!\n", 446 | "![](https://docs.opencv.org/2.4/_images/optimal-hyperplane.png)" 447 | ] 448 | }, 449 | { 450 | "cell_type": "markdown", 451 | "metadata": {}, 452 | "source": [ 453 | "Let's fit our first model!" 454 | ] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "execution_count": null, 459 | "metadata": {}, 460 | "outputs": [], 461 | "source": [ 462 | "from sklearn.svm import SVC\n", 463 | "l_svc = SVC(kernel='linear') # define the model\n", 464 | "\n", 465 | "l_svc.fit(X_train, y_train) # fit the model" 466 | ] 467 | }, 468 | { 469 | "cell_type": "markdown", 470 | "metadata": {}, 471 | "source": [ 472 | "Well... that was easy. Let's see how well the model learned the data!\n", 473 | "\n", 474 | "We can judge our model on several criteria:\n", 475 | "* Accuracy: The proportion of predictions that were correct overall.\n", 476 | "* Precision: Accuracy of cases predicted as positive\n", 477 | "* Recall: Number of true positives correctly predicted to be positive\n", 478 | "* f1 score: A balance between precision and recall\n", 479 | "\n", 480 | "Or, for a more visual explanation...\n", 481 | "\n", 482 | "![](https://upload.wikimedia.org/wikipedia/commons/2/26/Precisionrecall.svg)" 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": null, 488 | "metadata": { 489 | "collapsed": true 490 | }, 491 | "outputs": [], 492 | "source": [ 493 | "from sklearn.metrics import classification_report, confusion_matrix, precision_score, f1_score\n", 494 | "\n", 495 | "# predict the training data based on the model\n", 496 | "y_pred = l_svc.predict(X_train) \n", 497 | "\n", 498 | "# caluclate the model accuracy\n", 499 | "acc = l_svc.score(X_train, y_train) \n", 500 | "\n", 501 | "# calculate the model precision, recall and f1, all in one convenient report!\n", 502 | "cr = classification_report(y_true=y_train,\n", 503 | " y_pred = y_pred)\n", 504 | "\n", 505 | "# get a table to help us break down these scores\n", 506 | "cm = confusion_matrix(y_true=y_train, y_pred = y_pred) \n" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "metadata": {}, 512 | "source": [ 513 | "Let's view our results and plot them all at once!" 514 | ] 515 | }, 516 | { 517 | "cell_type": "code", 518 | "execution_count": null, 519 | "metadata": {}, 520 | "outputs": [], 521 | "source": [ 522 | "import itertools\n", 523 | "from pandas import DataFrame\n", 524 | "\n", 525 | "# print results\n", 526 | "print('accuracy:', acc)\n", 527 | "print(cr)\n", 528 | "\n", 529 | "# plot confusion matrix\n", 530 | "cmdf = DataFrame(cm, index = ['Adult','Child'], columns = ['Adult','Child'])\n", 531 | "sns.heatmap(cmdf, cmap = 'RdBu_r')\n", 532 | "plt.xlabel('Predicted')\n", 533 | "plt.ylabel('Observed')\n", 534 | "# label cells in matrix\n", 535 | "for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", 536 | " plt.text(j+0.5, i+0.5, format(cm[i, j], 'd'),\n", 537 | " horizontalalignment=\"center\",\n", 538 | " color=\"white\")" 539 | ] 540 | }, 541 | { 542 | "cell_type": "markdown", 543 | "metadata": {}, 544 | "source": [ 545 | "![](https://sebastianraschka.com/images/faq/multiclass-metric/conf_mat.png)" 546 | ] 547 | }, 548 | { 549 | "cell_type": "markdown", 550 | "metadata": {}, 551 | "source": [ 552 | "HOLY COW! Machine learning is amazing!!! Almost a perfect fit!\n", 553 | "\n", 554 | "...which means there's something wrong. What's the problem here?" 555 | ] 556 | }, 557 | { 558 | "cell_type": "code", 559 | "execution_count": null, 560 | "metadata": { 561 | "collapsed": true 562 | }, 563 | "outputs": [], 564 | "source": [ 565 | "from sklearn.model_selection import cross_val_predict, cross_val_score\n", 566 | "\n", 567 | "# predict\n", 568 | "y_pred = cross_val_predict(l_svc, X_train, y_train, \n", 569 | " groups=y_train, cv=10)\n", 570 | "# scores\n", 571 | "acc = cross_val_score(l_svc, X_train, y_train, \n", 572 | " groups=y_train, cv=10)" 573 | ] 574 | }, 575 | { 576 | "cell_type": "markdown", 577 | "metadata": {}, 578 | "source": [ 579 | "We can look at the accuracy of the predictions for each fold of the cross-validation" 580 | ] 581 | }, 582 | { 583 | "cell_type": "code", 584 | "execution_count": null, 585 | "metadata": {}, 586 | "outputs": [], 587 | "source": [ 588 | "for i in range(10):\n", 589 | " print('Fold %s -- Acc = %s'%(i, acc[i]))" 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "We can also look at the overall accuracy of the model" 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "execution_count": null, 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "from sklearn.metrics import accuracy_score\n", 606 | "overall_acc = accuracy_score(y_pred = y_pred, y_true = y_train)\n", 607 | "overall_cr = classification_report(y_pred = y_pred, y_true = y_train)\n", 608 | "overall_cm = confusion_matrix(y_pred = y_pred, y_true = y_train)\n", 609 | "print('Accuracy:',overall_acc)\n", 610 | "print(overall_cr)\n" 611 | ] 612 | }, 613 | { 614 | "cell_type": "code", 615 | "execution_count": null, 616 | "metadata": {}, 617 | "outputs": [], 618 | "source": [ 619 | "thresh = overall_cm.max() / 2\n", 620 | "cmdf = DataFrame(overall_cm, index = ['Adult','Child'], columns = ['Adult','Child'])\n", 621 | "sns.heatmap(cmdf, cmap='copper')\n", 622 | "plt.xlabel('Predicted')\n", 623 | "plt.ylabel('Observed')\n", 624 | "for i, j in itertools.product(range(overall_cm.shape[0]), range(overall_cm.shape[1])):\n", 625 | " plt.text(j+0.5, i+0.5, format(overall_cm[i, j], 'd'),\n", 626 | " horizontalalignment=\"center\",\n", 627 | " color=\"white\")" 628 | ] 629 | }, 630 | { 631 | "cell_type": "markdown", 632 | "metadata": {}, 633 | "source": [ 634 | "Not too bad at all!" 635 | ] 636 | }, 637 | { 638 | "cell_type": "markdown", 639 | "metadata": {}, 640 | "source": [ 641 | "### Tweak your model\n", 642 | "\n", 643 | "It's very important to learn when and where its appropriate to \"tweak\" your model.\n", 644 | "\n", 645 | "Since we have done all of the previous analysis in our training data, it's find to try different models. But we **absolutely cannot** \"test\" it on our left out data. If we do, we are in great danger of overfitting.\n", 646 | "\n", 647 | "We could try other models, or tweak hyperparameters, but we are probably not powered sufficiently to do so, and would once again risk overfitting.\n" 648 | ] 649 | }, 650 | { 651 | "cell_type": "markdown", 652 | "metadata": {}, 653 | "source": [ 654 | "But as a demonstration, we could see the impact of \"scaling\" our data. Certain machine learning algorithms perform better when all the input data is transformed to a uniform range of values. This is often between 0 and 1, or mean centered around with unit variance. We can perhaps look at the performance of the model after scaling the data" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": null, 660 | "metadata": { 661 | "collapsed": true 662 | }, 663 | "outputs": [], 664 | "source": [ 665 | "# Scale the training data\n", 666 | "from sklearn.preprocessing import MinMaxScaler\n", 667 | "scaler = MinMaxScaler().fit(X_train)\n", 668 | "X_train_scl = scaler.transform(X_train)" 669 | ] 670 | }, 671 | { 672 | "cell_type": "code", 673 | "execution_count": null, 674 | "metadata": {}, 675 | "outputs": [], 676 | "source": [ 677 | "plt.imshow(X_train, aspect='auto')\n", 678 | "plt.colorbar()\n", 679 | "plt.title('Training Data')\n", 680 | "plt.xlabel('features')\n", 681 | "plt.ylabel('subjects')" 682 | ] 683 | }, 684 | { 685 | "cell_type": "code", 686 | "execution_count": null, 687 | "metadata": {}, 688 | "outputs": [], 689 | "source": [ 690 | "plt.imshow(X_train_scl, aspect='auto')\n", 691 | "plt.colorbar()\n", 692 | "plt.title('Scaled Training Data')\n", 693 | "plt.xlabel('features')\n", 694 | "plt.ylabel('subjects')" 695 | ] 696 | }, 697 | { 698 | "cell_type": "code", 699 | "execution_count": null, 700 | "metadata": {}, 701 | "outputs": [], 702 | "source": [ 703 | "# repeat the steps above to re-fit the model \n", 704 | "# and assess its performance\n", 705 | "\n", 706 | "# don't forget to switch X_train to X_train_scl\n", 707 | "\n", 708 | "# predict\n", 709 | "y_pred = cross_val_predict(l_svc, X_train_scl, y_train, \n", 710 | " groups=y_train, cv=10)\n", 711 | "\n", 712 | "# get scores\n", 713 | "overall_acc = accuracy_score(y_pred = y_pred, y_true = y_train)\n", 714 | "overall_cr = classification_report(y_pred = y_pred, y_true = y_train)\n", 715 | "overall_cm = confusion_matrix(y_pred = y_pred, y_true = y_train)\n", 716 | "print('Accuracy:',overall_acc)\n", 717 | "print(overall_cr)\n", 718 | "\n", 719 | "# plot\n", 720 | "thresh = overall_cm.max() / 2\n", 721 | "cmdf = DataFrame(overall_cm, index = ['Adult','Child'], columns = ['Adult','Child'])\n", 722 | "sns.heatmap(cmdf, cmap='copper')\n", 723 | "plt.xlabel('Predicted')\n", 724 | "plt.ylabel('Observed')\n", 725 | "for i, j in itertools.product(range(overall_cm.shape[0]), range(overall_cm.shape[1])):\n", 726 | " plt.text(j+0.5, i+0.5, format(overall_cm[i, j], 'd'),\n", 727 | " horizontalalignment=\"center\",\n", 728 | " color=\"white\")" 729 | ] 730 | }, 731 | { 732 | "cell_type": "markdown", 733 | "metadata": {}, 734 | "source": [ 735 | "What do you think about the results of this model compared to the non-transformed model?" 736 | ] 737 | }, 738 | { 739 | "cell_type": "markdown", 740 | "metadata": {}, 741 | "source": [ 742 | "**Exercise:** Try fitting a new SVC model and tweak one of the many parameters. Run cross-validation and see how well it goes. Make a new cell and type SVC? to see the possible hyperparameters" 743 | ] 744 | }, 745 | { 746 | "cell_type": "code", 747 | "execution_count": null, 748 | "metadata": { 749 | "collapsed": true 750 | }, 751 | "outputs": [], 752 | "source": [ 753 | "# new_model = SVC() " 754 | ] 755 | }, 756 | { 757 | "cell_type": "markdown", 758 | "metadata": {}, 759 | "source": [ 760 | "### Can our model classify childrens from adults in completely un-seen data?\n", 761 | "Now that we've fit a model we think has possibly learned how to decode childhood vs adulthood based on rs-fmri signal, let's put it to the test. We will train our model on all of the training data, and try to predict the age of the subjects we left out at the beginning of this section." 762 | ] 763 | }, 764 | { 765 | "cell_type": "markdown", 766 | "metadata": {}, 767 | "source": [ 768 | "Because we performed a transformation on our training data, we will need to transform our testing data using the *same information!* \n" 769 | ] 770 | }, 771 | { 772 | "cell_type": "code", 773 | "execution_count": null, 774 | "metadata": { 775 | "collapsed": true 776 | }, 777 | "outputs": [], 778 | "source": [ 779 | "# Notice how we use the Scaler that was fit to X_train and apply to X_test,\n", 780 | "# rather than creating a new Scaler for X_test\n", 781 | "X_test_scl = scaler.transform(X_test)" 782 | ] 783 | }, 784 | { 785 | "cell_type": "markdown", 786 | "metadata": {}, 787 | "source": [ 788 | "And now for the moment of truth! \n", 789 | "\n", 790 | "No cross-validation needed here. We simply fit the model with the training data and use it to predict the testing data\n", 791 | "\n", 792 | "I'm so nervous. Let's just do it all in one cell" 793 | ] 794 | }, 795 | { 796 | "cell_type": "code", 797 | "execution_count": null, 798 | "metadata": {}, 799 | "outputs": [], 800 | "source": [ 801 | "l_svc.fit(X_train_scl, y_train) # fit to training data\n", 802 | "y_pred = l_svc.predict(X_test_scl) # classify age class using testing data\n", 803 | "acc = l_svc.score(X_test_scl, y_test) # get accuracy\n", 804 | "cr = classification_report(y_pred=y_pred, y_true=y_test) # get prec., recall & f1\n", 805 | "cm = confusion_matrix(y_pred=y_pred, y_true=y_test) # get confusion matrix\n", 806 | "\n", 807 | "# print results\n", 808 | "print('accuracy =', acc)\n", 809 | "print(cr)\n", 810 | "\n", 811 | "# plot results\n", 812 | "thresh = cm.max() / 2\n", 813 | "cmdf = DataFrame(cm, index = ['Adult','Child'], columns = ['Adult','Child'])\n", 814 | "sns.heatmap(cmdf, cmap='RdBu_r')\n", 815 | "plt.xlabel('Predicted')\n", 816 | "plt.ylabel('Observed')\n", 817 | "for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n", 818 | " plt.text(j+0.5, i+0.5, format(cm[i, j], 'd'),\n", 819 | " horizontalalignment=\"center\",\n", 820 | " color=\"white\")" 821 | ] 822 | }, 823 | { 824 | "cell_type": "markdown", 825 | "metadata": {}, 826 | "source": [ 827 | "***Wow!!*** Congratulations. You just trained a machine learning model that used real rs-fmri data to predict the age of real humans.\n", 828 | "\n", 829 | "It seems like something in this data does seem to be systematically related to age ... but what? " 830 | ] 831 | }, 832 | { 833 | "cell_type": "markdown", 834 | "metadata": {}, 835 | "source": [ 836 | "### Interpreting model feature importances\n", 837 | "Interpreting the feature importances of a machine learning model is a real can of worms. This is an area of active research. Unfortunately, it's hard to trust the feature importance of some models. \n", 838 | "\n", 839 | "You can find a whole tutorial on this subject here:\n", 840 | "http://gael-varoquaux.info/interpreting_ml_tuto/index.html\n", 841 | "\n", 842 | "For now, we'll just eschew better judgement and take a look at our feature importances" 843 | ] 844 | }, 845 | { 846 | "cell_type": "markdown", 847 | "metadata": {}, 848 | "source": [ 849 | "We can access the feature importances (weights) used my the model" 850 | ] 851 | }, 852 | { 853 | "cell_type": "code", 854 | "execution_count": null, 855 | "metadata": {}, 856 | "outputs": [], 857 | "source": [ 858 | "l_svc.coef_" 859 | ] 860 | }, 861 | { 862 | "cell_type": "markdown", 863 | "metadata": {}, 864 | "source": [ 865 | "lets plot these weights to see their distribution better" 866 | ] 867 | }, 868 | { 869 | "cell_type": "code", 870 | "execution_count": null, 871 | "metadata": {}, 872 | "outputs": [], 873 | "source": [ 874 | "plt.bar(range(l_svc.coef_.shape[-1]),l_svc.coef_[0])\n", 875 | "plt.title('feature importances')\n", 876 | "plt.xlabel('feature')\n", 877 | "plt.ylabel('weight')" 878 | ] 879 | }, 880 | { 881 | "cell_type": "markdown", 882 | "metadata": {}, 883 | "source": [ 884 | "Or perhaps it will be easier to visualize this information as a matrix similar to the one we started with\n", 885 | "\n", 886 | "We can use the correlation measure from before to perform an inverse transform" 887 | ] 888 | }, 889 | { 890 | "cell_type": "code", 891 | "execution_count": null, 892 | "metadata": {}, 893 | "outputs": [], 894 | "source": [ 895 | "correlation_measure.inverse_transform(l_svc.coef_).shape" 896 | ] 897 | }, 898 | { 899 | "cell_type": "code", 900 | "execution_count": null, 901 | "metadata": {}, 902 | "outputs": [], 903 | "source": [ 904 | "from nilearn import plotting\n", 905 | "\n", 906 | "feat_exp_matrix = correlation_measure.inverse_transform(l_svc.coef_)[0]\n", 907 | "\n", 908 | "plotting.plot_matrix(feat_exp_matrix, figure=(10, 8), \n", 909 | " labels=range(feat_exp_matrix.shape[0]),\n", 910 | " reorder=False,\n", 911 | " tri='lower')" 912 | ] 913 | }, 914 | { 915 | "cell_type": "markdown", 916 | "metadata": {}, 917 | "source": [ 918 | "Let's see if we can throw those features onto an actual brain.\n", 919 | "\n", 920 | "First, we'll need to gather the coordinates of each ROI of our atlas" 921 | ] 922 | }, 923 | { 924 | "cell_type": "code", 925 | "execution_count": null, 926 | "metadata": { 927 | "collapsed": true 928 | }, 929 | "outputs": [], 930 | "source": [ 931 | "coords = plotting.find_parcellation_cut_coords(atlas_filename)" 932 | ] 933 | }, 934 | { 935 | "cell_type": "markdown", 936 | "metadata": {}, 937 | "source": [ 938 | "And now we can use our feature matrix and the wonders of nilearn to create a connectome map where each node is an ROI, and each connection is weighted by the importance of the feature to the model" 939 | ] 940 | }, 941 | { 942 | "cell_type": "code", 943 | "execution_count": null, 944 | "metadata": {}, 945 | "outputs": [], 946 | "source": [ 947 | "plotting.plot_connectome(feat_exp_matrix, coords, colorbar=True)" 948 | ] 949 | }, 950 | { 951 | "cell_type": "markdown", 952 | "metadata": {}, 953 | "source": [ 954 | "Whoa!! That's...a lot to process. Maybe let's threshold the edges so that only the most important connections are visualized" 955 | ] 956 | }, 957 | { 958 | "cell_type": "code", 959 | "execution_count": null, 960 | "metadata": {}, 961 | "outputs": [], 962 | "source": [ 963 | "plotting.plot_connectome(feat_exp_matrix, coords, colorbar=True, edge_threshold=0.04)" 964 | ] 965 | }, 966 | { 967 | "cell_type": "markdown", 968 | "metadata": {}, 969 | "source": [ 970 | "That's definitely an improvement, but it's still a bit hard to see what's going on.\n", 971 | "Nilearn has a new feature that let's use view this data interactively!" 972 | ] 973 | }, 974 | { 975 | "cell_type": "code", 976 | "execution_count": null, 977 | "metadata": {}, 978 | "outputs": [], 979 | "source": [ 980 | "plotting.view_connectome(feat_exp_matrix, coords, threshold='90%')" 981 | ] 982 | }, 983 | { 984 | "cell_type": "code", 985 | "execution_count": null, 986 | "metadata": { 987 | "collapsed": true 988 | }, 989 | "outputs": [], 990 | "source": [ 991 | "#view = plotting.view_connectome(feat_exp_matrix, coords, threshold='90%') \n", 992 | "#view.open_in_browser() " 993 | ] 994 | } 995 | ], 996 | "metadata": { 997 | "anaconda-cloud": {}, 998 | "kernelspec": { 999 | "display_name": "Python 3", 1000 | "language": "python", 1001 | "name": "python3" 1002 | }, 1003 | "language_info": { 1004 | "codemirror_mode": { 1005 | "name": "ipython", 1006 | "version": 3 1007 | }, 1008 | "file_extension": ".py", 1009 | "mimetype": "text/x-python", 1010 | "name": "python", 1011 | "nbconvert_exporter": "python", 1012 | "pygments_lexer": "ipython3", 1013 | "version": "3.6.2" 1014 | } 1015 | }, 1016 | "nbformat": 4, 1017 | "nbformat_minor": 1 1018 | } 1019 | -------------------------------------------------------------------------------- /model_validation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Validation Curves\n", 17 | "\n", 18 | "Let us create an example dataset:" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np \n", 28 | "def generating_func(x, err=0.5):\n", 29 | " return np.random.normal(10 - 1. / (x + 0.1), err)\n", 30 | "\n", 31 | "# randomly sample more data\n", 32 | "np.random.seed(1)\n", 33 | "x = np.random.random(size=200)\n", 34 | "y = generating_func(x, err=1.)" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "Central to quantify bias and variance of a model is to apply it on test data, sampled from the same distribution as the train, but that will capture independent noise:" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": 5, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "from sklearn.model_selection import train_test_split\n", 51 | "xtrain, xtest, ytrain, ytest = train_test_split(x, y, test_size=0.4)" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "Let's visualize the data:" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 14, 64 | "metadata": {}, 65 | "outputs": [ 66 | { 67 | "data": { 68 | "text/plain": [ 69 | "" 70 | ] 71 | }, 72 | "execution_count": 14, 73 | "metadata": {}, 74 | "output_type": "execute_result" 75 | }, 76 | { 77 | "data": { 78 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnX+QHOWZ37/PjiTwIALcoFQcyzujq3ApkMACFg6XzzEcKayAg1119pXF4DtiW2sviYr84iy8lImTWl/lqLsy+IydvZjCYeZkHBL7qAtXRvaBDVeG3AK2+WF8gLS7Wrgcq7VFJC2KpZ0nf3Tvan70j7e73+7pnvl+qqZ2p+ft7vftH99++nmf93lFVUEIIWRwGOl3BQghhNiFwk4IIQMGhZ0QQgYMCjshhAwYFHZCCBkwKOyEEDJgUNgJIWTAoLATQsiAQWEnhJABY10/dnruuedqrVbrx64JIaSwPP3004dUdVNYOWNhF5F7AXwAwBuqus1ddieAfw7glwBeBfAvVPVw2LZqtRpmZmZMd00IIQSAiMyZlIviirkPwI6uZfsAbFPViwD8DYDbImyPEEJIChgLu6r+AMDPu5Y9oqon3a9PAthssW6EEEJiYLPz9OMA/sLvRxEZF5EZEZlZXFy0uFtCCCHtWOk8FZFJACcBNP3KqOo0gGkAGBsb68kVfOLECSwsLOD48eM2qpRbTj/9dGzevBnr16/vd1UIIQNKYmEXkZvgdKperQmSuy8sLODMM89ErVaDiCStVi5RVSwtLWFhYQFbtmzpd3UIIQNKIleMiOwA8HsArlfV5STbOn78OCqVysCKOgCICCqVysC/lZCYNJtArQaMjDh/m74vwIQEEiXccS+AKwGcKyILAO6AEwVzGoB9riA/qaqfjluZQRb1VYahjSQGzSYwPg4su/bR3JzzHQDq9f7VixQSY2FX1Z0ei79msS6EDC+Tk6dEfZXlZWc5hZ1EhCkFXA4fPox77rkn8nrXXnstDh8OHZNFSDDz89GWExJAcYXdsj/ST9hPnjzpUfoUDz/8MM4+++xE+yYEo6PRlhMSQDGFfdUfOTcHqJ7yRyYQ9z179uDVV1/F9u3bcdlll+G9730vrr/+elxwwQUAgA996EO49NJLsXXrVkxPT6+tV6vVcOjQIczOzuL888/Hrl27sHXrVlxzzTV46623EjeVDAlTU0C53LmsXHaWExIVVc38c+mll2o3L774Ys8yX6pVVUfSOz/Vqvk2ujhw4IBu3bpVVVUfffRRLZfLun///rXfl5aWVFV1eXlZt27dqocOHXKrUtXFxUU9cOCAlkolffbZZ1VV9SMf+Yjef//9nvuK1FYyPDQazjUs4vxtNPpdozxWaagBMKMGGtuX7I6JycAfefnll3fEmt9999341re+BQA4ePAgXn75ZVQqlY51tmzZgu3btwMALr30UszOzlqrDxkC6vVcdZQyUKe4FNMVk4E/8owzzlj7/7HHHsN3v/td/PCHP8SPf/xjXHzxxZ6x6Keddtra/6VSKdQ/T0ieCQrUIfmmmMKegj/yzDPPxJEjRzx/e/PNN3HOOeegXC7jpZdewpNPPhl7P4QUBQbqFJdiCnu9DkxPA9UqIOL8nZ5O9H5YqVTwnve8B9u2bcOtt97a8duOHTtw8uRJnH/++dizZw+uuOKKpC0gJPcMc6BO4QcBmzjibX8Sd54WnGFqKykujYZqudwZn1AuD1AHqk/PcJ7bDcPO02Ja7ISQ1EnhxTg/BIRMR+5byKF5T2EnhPhSrwOzs0Cr5fwtpKh7CW+AekfqW0hhTI0NKOyEkMHFT3jnfKYOnZ+P1reQ09AhCjshZHDxE95Sybv86Gi0oLuchg5R2AkhxSKKT9tPYFdWAtX7bW87tbhSCehbyGnoEIWdEFIcovq0/QR2tSe4q2e4iTrGx4GlpVNFA1M+5TTHD4XdJW7aXgD44he/iOXu1z1CiH2i+rSDhNejZziyyzynoUOFFXbbEUYUdkIKQFSfdkThjeUyz2HoUCGFPY0Io/a0vbfeeivuvPNOXHbZZbjoootwxx13AACOHTuG6667Du9617uwbds2PPDAA7j77rvx+uuv46qrrsJVV11lqYWEEE/i+LQjCK/v5kcWggUmb7HsJqOYbH+SjjxNIWtvR9re73znO7pr1y5ttVq6srKi1113nX7/+9/XBx98UD/5yU+urXP48GG3Pk7qXlM48pSQCLSPEK1UVDdsSG1YqOeoUxzVBnb67yfDoaoY5JGnaUcYPfLII3jkkUdw8cUX45JLLsFLL72El19+GRdeeCH27duHz3zmM3j88cdx1lln2dkhIcSb7tfzpSXnb6WSik97zXNTWoCghSpmMY1dqGOvv7M9h7HshczHPjrqPb7AVoSRquK2227Dpz71qZ7fnnnmGTz88MO4/fbbcfXVV+Nzn/ucnZ0SQnrxEs0TJ4CNG4FDh1LZZb0O1D82CkB7f/SyHnMYy25ssYvIvSLyhog837bsV0Rkn4i87P49J51qdpJGhFF72t73v//9uPfee3H06FEAwGuvvYY33ngDr7/+OsrlMm688UbceuuteOaZZ3rWJYSYE+qa7pdoRvHl5zCWPYor5j4AO7qW7QHwPVU9D8D33O+pk0aEUXva3n379uGGG27Au9/9blx44YX48Ic/jCNHjuC5557D5Zdfju3bt+Pzn/88br/9dgDA+Pg4duzYwc7TlMhbv9Qaua1YMTAKguiXaEaxHvMYy27iiF/9AKgBeL7t+88AvN39/+0AfmayHabtHdy22p4jM7cpVHNbseJgFARh8ThHvjajrJDR5LAw7DxNKuyH2/6X9u9BHwr7YLY1Da1LIwIqFt03bqWSk4oVF0HL8xCKdBVcPfaAaql06jhHuLBSfQ5nOON35sLufv9FwLrjAGYAzIyOjvZUeFDFzosittXk2k1DhEW8t9lz86eJlyr4fTKtWEB9MxKa2DQaWpU58+slQJn7dW2G1SsNshJ2q66YVquVwqHIF61Wq3DCbnrtpiHCubDY/SrR94p5UBQXUbWqDezUMo52VlWOeVfV5xw0Krv7c222v0VkeB1kJex3Atjj/r8HwB+YbMdL2Pfv36+Li4sDLe6tVksXFxd1//79/a5KJEzFNQ0RzoVO+alC9ycPApr1kzDu24F7TBvYqVUcUMGKVnFAG7ghsHz3p4oDya5NHIj+VmPyBrf6xLD89mQq7OKUDUdE9gK4EsC5AP4OwB0Avg3gmwBGAcwB+G1V/XnYtsbGxnRmZqZj2YkTJ7CwsIDjx48b1aeonH766di8eTPWr1/f76oYMzLiXK3diDijtFdZjXJoDzsulyNGLK3ObjM/70Q+TE2hiXr3omzTcdRq3gMnKhUnnrpvFfPA9GTZIMkJ9zum1aoz7N+w/AhW4DXO0ujaxLFTg4+iXKh+de9ux9SUhRuiExF5WlXHQguaqL/tj5fFTvJLFCMwkYGSC/O8QPXyIkuLPcm+oh5Tn/LVypHo1+ba28HOeMco7A1utR0pnAuk4Yqx9aGwF4vMdC0XDnUfitAhqZrtQyip4zrqMfUoH6u5hvX2rV6Yb321YAqdThR2YpVMdC0XITAZkPbBtLH9voaa2K9qBwb1DnxgmD5NaLETorkRCpt0JyasbHyr1xWQN7eOqXAVyUXVjkG9Qy9Fk6dJCseHwk4K4z1Yo6hC4dJ9vCcmgoMn1tLBej28Go3OQVCVip3jYNsSL9xF5hJS78gvj37b61NUDIV9QCmsRhZUKLyOt0mUZBUHehWj0ejNOQ6orl+f7Hj0c0BCTjC9vCK9PGZ4s1HYh5wB9GrkmihjmDq0Eiu9J8b2wJewwTSVilljCn7xRNHfSFqd4fEyFfZCTrRBwslhiuiBJu5xHcV8bybAoI1F3VF7CkU/lpY6UyqaZCu8+WZg3TonYHzdOud7TLJKkhllPgy/DLKAR10Dbra+JQA1UX/bH1rs6TOgRldu8TveQe6YMo5qo7LbPJoiygkMs9LDthvks5iY8N7GxETk4zYx0XuMYnkxDHwsvh4mrBjt0NeKr+z23LBpuoMogK6Y4aawPvaC4ne8Jya6omIqBt0HSX3sJkPevZ5Apg7o1QyL3Z9SKfIx8xPbSAaI4cUemFbA4ObwXb9yJPHgKVMo7CR//ZAZRQ70C6vNSBIVE8fhX6mYWwJB24lAUDUj9dMavp566n9QZFIXgX3KHic/jT5oCjvJF0EmbdRXiwF5EKSGadKy9uMdJb98XIu967z55WOPbNVGUNBGQ7uSju0MLN9O4PPD45pMwx1KYSf5wu8q9xMJv6ufPqZwwnz07f6h1QdjFPMyjo/d47z55WNf9Qolbm+p5P3wj6m4vpfexOOePzQmHqePnQw4Ua1IP+uJvcLhxHn4RT2uExOnHsqlUnjHqcf2G9ipZTnWc9oj98Ga9Cm0t99v0MHEROjLoOfvAcfO9sslhZ3EIjUvhy2LfYAHz1jDxz8feG7jvgmZXjA+562BG+xcb+31MLmmJiacfbe5ZSZGvqrlDSeiW9gZXpMUdhKZVL0ctnzsQZblkPje4wi0kVsg6vGLcsFk+aZlILSNyu6e2ZsEK/GqmMMBShR2skbq16eNqBibnbBpYOvh4rOdUC31OYnV0kH75zbKBZNl30hbvTpmaCodXNud38xLsQxvphSgsHeQMwuz314O48PhVTAPvvcA323c7UzgS1rCCQVaWhpZ0Y0bQ5rpcxL9rNFE5zbqBZPV9e4eP885VV299TseYZeQbxMyahuFPe/0I7oj5OLrpzaGHo6wG6cfT6X2OlUqqiMj/nWIcl7dEzGBLykCQgI9m5lXiz0OScRyYkKrmPWtnt/goW7BL+OoE/Wi+QjIorDnnaxV1OCqjCKujcpurVaOWDNQQmOEw+6oPBxPU7MvDPch5VjqETefxMdu4xjYUrok23bXDXpLaTS0t6MUx3QCX+qNc3cPbh5eCinseSdrCzPC6LxqVVXQcnySuOFU7LN7owW94sYl8HCY1D1rcyogVrzDfYITOoEvRTuv7rZNrXXTTlDr3gKf6BsrJFFRd10/P/rqJnqOB24IvCf77apUpbDnn6wf/1GuSj9fsft/2A0Th8DD4ado3XXPss/C53h6u09aOnHGfebbdo+/n8W+2jyrIYJRN9RoaGP9TZ3W7fqbzLeRpmvNXTeyARJyTw6dxQ7g3wB4AcDzAPYCOD2oPIVd82Nhuldlx31WOtg7g3vbJ42OON/DMfG4pUxRFmk0fGOl/cS4NLISeR8TZ9zn8ZCIlUTRuw0+LhsTrfcKF1zLVhlz39ZcawZRMXHqlfiWtWB4ZCbsAN4B4ACAt7nfvwngpqB1KOwuWVqYAVel50/tyZG6PmlY7KtV7Dkcfjd45HHnlgjxrQe5T+IQdYCnMR7H1WskqJ9wVfCG9zWAA7H23XMBWfCxx1434J6MfctaMuSyFvaDAH4FwDoAfw7gmqB1KOwxsPEQ8NmG732GA54/pOFj9yUoFYEFoobQV0sH/RNIVSr+FjtOpHLuYuNxXE0f2I2G+j7ABAZvJqZuloSuojyFEtvy42TtirkFwFEAiwCaYeWHRthtXVwpu20CJyDwEdUGbtDqxkPpZ+BN0WKPclhD32pcUfLOj9VyOlDjnrs0zr/HcTV1sQX0Gzu5yWPsO47IBZI3YbfU85qlxX4OgL8EsAnAegDfBnCjR7lxADMAZkZHR+MfoKJg82ZM+Ubw3XzpoHnuDU3p+dNo+N8U3XN1RiT0sLaJg28c+OpbTdux6HCfrEbF9JjwJfMDk8b59zhZftkWu3fj/xLVMmtS2v1LaWw/6YOiaBY7gI8A+Frb998BcE/QOkNhsdu8GVOOswq9Dwz3n9rzx888BBLdrIHN6joovtYsVoJFI8iVZCo2aZ3/LrEyjXX3O8+RnrNpWtS2L0QbD4oC+th/3Y2IKQMQAF8HsDtonaEQdps3o40LNSS2edWIXN1sx/VmuP/Unj+B7/7VsLUjb7Za7f3R1/9cOhh8cwbV3VQNY57/ONppsk7WAV2RsX0h2npQFCkqxtkXPg/gJTfc8X4ApwWVHwpht2k1uHdSR+iWzK0NdTZdv/tONB6NaHgnp2axO7111p8anhMpyzHPEMvYHcYhUTQKhG8khpKGrpJQZBoNx5++1pHsNSl3v7B9IeZhZJILByj1m5g3o9+91ph43DgUrQefCz1S2KKBEHg2eVUoY9Cxy5F57/DLmDerV10FK44/vFz2nCqugZ1OVEwULWw0/Kedi9KGiEIcqG05ci2kgu265WFkkguFPQ9EuBnDrsVE11aGGf8aE49rVeY6QwJj3FShUSgJb9bQEM8okztHaURGll+Qkdmo7Pae9zOKUOVI7Dyx6cPP0UOMwl4wwu6ToD640GvXhsVuqyFJN7MaqZPwZg0N8VwNp0wjEiLCsYlbhaAOTs8Ro9gZ7QGTI/dEJuQkfJLCXjDC7hMTjSjLsVNJuzr8ON6WYyoDjSzd8GnrRqjFbsPyDHoaGxzwJIai37p+XqEqDpi3udGIPqUhsQKFvWD4Ck3liBOKhht6fOyBwtStAO3hL13ivvZaXk1xQJEtiz3aZnwJdPXYes32a0SpZGT5JT0GXkZm4JtK3CdG9ytBHvzsAwqFvZ943FFhb3KeQrPhhJMxr12EZU4lIB9Jx2hRLwUoyOCQLNyaa+ekO0VxFsfC4NU+jbeWQAMiyQbSPFFkDQp7TBK70jxu5sb6m4xmP+/Zd2W3z11YDXclBClA2v5CS9tPXXizoDsqplIxnp81jbeWxA9ME/eSzVcr0gGFPQZWrESPuzF2J2WAyWYUNZJwyH2qmIp/jiISuqtl9OzyetDjBqOolLSanui5a9ohPKidqH2Gwh4DKxaShxjHDisMqVCHRYvZ3jjvDRv6LoCeRFGsHIbVRRLcrvp7dlgHRKWkHowRdQemIZy02FOBwh4DKz5NmxZ7FAXxDXcI20kfiCLWOQyri/Ss6aq/77UQJSrFFnFfCdofBpWK6vr10bdBYkFhj4EV4zCBj913eyYWVQ4F0Jcodc2hxR7pUHfVPzCZWNZiaOvY5iTGexigsMfAmk8zRlRMYnIogL5EqWsOfeyRDnVX/X0tdtOoFBv4hL7m2hggqkphj00cl2NUwU5lH54ZrXL6ShxVrHNmEUZ+1rTVv1HZHf/tLa3Kp2AM5OyUDQwU9gyIY0zG0bTQ8l6FRCxOkJkCBb/zk1S/r00Pi2qx8JTJ4UvWwGAq7OKUzZaxsTGdmZnJfL+2qdWAubne5dUqMDtrZx2j8nEqQoaTkRFHa72oVoGpKaBeT7QLXo7pISJPq+pYaDkKe3z87hERoNWys45ReRHvnQVVhAwnGahunPuCmGEq7CNZVGZQGR2NtjzqOs2mc5MElm82/YU9qCLuqrWas49azfluldR30Of9FZGpKaBc7lxWLjvLLRHnviCWMfHX2P7Qxx6+TlAfV0d5P5/patrZ7o1m1YmXtaOVjl1zUnby81SkB9h5mg0998jE450TGVSOeOtryLRiQYkBO4qaJmo3DburWjowWYdfFinccwgoeN94bqGw2yLKFdpoOIORuoeMbzgRHsXSZdIYD4IxzbbXNTI1jdmTOsh6wFSRBmgVESp1LqCw26BNgDvylntY4arqZF00sYQNrEtjA9Q0d0fXhxY7MYa+ldxAYbeBKxbGMw2JmFnCBtZlpHup3ZoyFPZUZk/qrhN97IMBH5q5gcJuA1coja1bixa7asy3X79te0zO3Fh/k+Prz0vmwKLtb1igmys3ZCrsAM4G8CCAlwD8FMC7g8oXRthdkTT2R1v0sccmaNsUPhIHWuy5IWth/zqAT7r/bwBwdlD5wgi7K5JR/NGNice1IocUaCnQ0srGt8LdJ7ZFlgJObEI3V27ITNgBnAXgANxRrCafwgi7qmOFV3Yb+aN5/ZOBhcZCLjAV9sQpBURkO4BpAC8CeBeApwHcoqrHusqNAxgHgNHR0UvnvIY155hmE5icBObnnRF0Xik1mCODEJImWaYUWAfgEgBfUdWLARwDsKe7kKpOq+qYqo5t2rTJwm6zpV53xLnVcv565Uman/de1295EBwdTwiJiw1hXwCwoKpPud8fhCP0Q4etHBnNJjA+7lj/qs7f8XGKOyHEjMTCrqr/B8BBEfnH7qKr4bhlhg7P/EqyjKm5eiSze/KWo1he7ly2vOy4ggghJAxb2R13A2iKyE8AbAfwBUvbzTdd/pI6mpiednzqAkVV5jGtn0Qdf2pudjebmF8qe/4Ux6VDCBk+rAi7qv7I9Z9fpKofUtVf2NhuJsR1Zvv4S+poOr746hbMahV17D21jonZPTmJUXgrONOeEkJMGO587Emc2ZOTCPSXxO1JnZ/HFD6LMjqCilDGMZspswkhA8xwC3uYOAfhJ9Bzc47l7xdGGmZ2j46ijr2Yxi5UMQtBC1XMYrpyW9IZywghQ8JwC3uS+EQ/gRbxDmYHzGaqcXtg69iLWWxBCyXMlreiftevh9eJEEIw7MKeJD7RKwRGxN9Sr1aB6enwiYLrdZzqgRXz9cJgYDwhQ8PQCnuzCdSOPo8RrKCGA2hip/OD6fyPXgLsJ+oi/qOa/LYdNhoqCgyMJ2SoGEphX9O5pY1QjGAONYzjT9Cs7A60jnuMXnQJcLXqvcN+h7Mk6UsghBSOoRR2T53DGZjceHegqIcavRnMAB8Lm7kOCCG5ZyiFPY7OGRm9afnHk2Ir1wEhpBAMpbDH0Tnjh4Ft/7gN8vomQQhJhaEU9jg6V2ijN69vEoSQVBhKYY+jc4U3evP4JkEISYV1/a5Av6jXo2nbatmwyTYIIaTfDKXFDsQbr+Nl9HLcDyEkbwylsHuFLt54I3DuudGEmeN+CCF5JPGcp3EYGxvTmZmZzPe7it/cpIDjNzftV+Qcp4SQLMlyztPCERSvHmVAJsf9EELyyFAKe1iIoqkwFzoEkhAysAyNsLd3ch49CmzY4F/WVJgLHwJJCBlIhkLYuzs5l5acvxs39paNIswc90MIySNDIexeeV5OnAAqFaDRiCHMbeZ/fbKG2akmx/0QQnLDUAi7byfnXAv1j41gFjW07m+aCTNjHAkhOceasItISUSeFZE/t7VNW/h2cmLe2iTWzVue4kAlQkgusGmx3wLgpxa3Zw3PTk4cwxQ+e2pBgkmsm9iJ8aXfpxFPCMkFVoRdRDYDuA7Af7WxPdv0dHJiFtPYhTr2dhaMOYn1JL6AZZzRsYwTFBFC+oUti/2LAH4PQMvS9qzTkeelemWvqAOxJ7Geh/d6HKhECOkHiYVdRD4A4A1VfTqk3LiIzIjIzOLiYtLdJiNJALpHjONoZdmzKAcqEUL6gQ2L/T0ArheRWQDfAPCbItLoLqSq06o6pqpjmzZtsrDbBCQNQO9K8zh110YOVCKE5AarScBE5EoA/15VPxBUrt9JwNKg2WSudkJIupgmARvaiTZsE3XiDkIISQurwq6qjwF4zOY2CSGERGMoRp4SQsgwQWE3gNPfEUKKBH3sIaymhlnNIrA6qhSgT50Qkk9osYfgkxqGo0oJIbmFwh4Cp78jhBQNCnsInP6OEFI0KOwhcPo7QkjRKL6wG4SsJIlq4fR3hJCiYTWlgCnWUgp0h6wAjjndprzNm5/A+FcvwbKW/YoQQkghME0pUGxhr9Wc+MNuqlUnSVezidrH3os57XWIrxYhhJCiYCrsxXbFhIWsTE5iXjdHWpUQQopOsYU9LGRlft6Z1zTCqoQQUnSKLexhISujo5jCZ1HGsY4ighbm5pgegBAymBRb2ANCVppNoHb0eXwMDbwNy6hgEUALghbUbTYnnSaEDCLF7jz1wTNYBsfwNjmOJa30lGdHKiGkCAxH52k3bsD65I2zvfldcIanqAPsSCWEDBaDI+yrZvrcHOYRrWfUsyOVuXoJIQVlYIS9ectTqC2/gBGsYAQtzzKVimF6gLaHBFTpjCeEFIqBEPZmExhf+n3MoQbFCFawDkBn30G5DNx1l2F6AObqJYQUmOJPtNFsYvJ334dldA9EEpRwEi2MYLQ6gqmpUwIemkqAuXoJIQWm2MLuukzmV454/tzCCFqNvdGTwoyOeqcq4KgmQkgBSOyKEZF3isijIvKiiLwgIrfYqJgRrsvEd3RpZdlX1AP7RpmrlxBSYGz42E8C+HeqegGAKwD8SxG5wMJ2w3FdI16jS8tlYOqujZ6rhfaNMlcvIaTAWB+gJCJ/BuCPVXWfX5k0sjs2sROT+ALmMYrR0uuY+vpmXx0OSwpJCCF5pC8DlESkBuBiAE/Z3K4v11679m8dezGLLWihhNnxLwQa1+wbJYQMMtaEXUQ2AvgfAP61qv5fj9/HRWRGRGYWFxft7PSb34y23IXzmBJCBhkrwi4i6+GIelNV/6dXGVWdVtUxVR3btGmTjd0CS0vRlruwb5QQMsjYiIoRAF8D8FNV/aPkVUof9o0SQgYZGxb7ewB8DMBvisiP3M+1YStZoXIqqVcTO1HDAYxgBbWR+dDR//W601Haajl/KeqEkEEh8QAlVX0CgFioS3Tuugv4+MfR/OVvYRx/gmWcAQCYa70T4+NOEQo2IWTYKHaumHoduPdeTJb+YE3UV2FqF0LIsFJsYXeZX/mH3ssZvkgIGUKKLezuEFJOWE0IIacotrC7uWJ8UwowfJEQMoQUW9hdX0sdezGNXahiFoIWqphl+CIhZGgptrC3+Vo6UgpUr6SoE0KGlmILO4eQEkJID8UWdg4hJYSQHoo9gxLgiDiFnBBC1ii2xU4IIaSHwgp78+YnUFu3gBFpobZuAc2bn+h3lQghJBcU0hXTvPkJjH/l4lO5YVY2Y/wr5wB4AvV7fqO/lSOEkD5TSIt9crrWmxsGZ2ByutafChFCSI4opLD75obxWU4IIcNEIYV9tPR6pOWEEDJMFFLYp8Zne3PD4Bimxmf7UyFCCMkRhRT2+j2/gemJZ1EtLTi5YUoLmJ54lh2nhBACQFQ1852OjY3pzMxM5vslhJAiIyJPq+pYWLlCWuyEEEL8KYywN5tArQaMjDh/wyarJoSQYaUQA5TciZKwvOx8n5sDJ6smhBAfCmGxuxMldcDJqgkhxBsuK8KpAAAGKklEQVQrwi4iO0TkZyLyiojssbHNdvwmpeZk1YQQ0ktiYReREoAvA/hnAC4AsFNELki63Xb8JqXmZNWEENKLDYv9cgCvqOp+Vf0lgG8A+KCF7a7BiZIIIcQcG8L+DgAH274vuMs6EJFxEZkRkZnFxcVIO+BESYQQYk5mUTGqOg1gGnAGKEVdnxMlEUKIGTYs9tcAvLPt+2Z3GSGEkD5gQ9j/GsB5IrJFRDYA+CiAhyxslxBCSAwSu2JU9aSI/CsA3wFQAnCvqr6QuGaEEEJiYcXHrqoPA3jYxrYIIYQkoxAjT1dhvhhCCAmnELliAOaLIYQQUwpjsTNfDCGEmFEYYWe+GEIIMaMwws58MYQQYkZhhJ35YgghxIzCCDvzxRBCiBmFiYoBmC+GEEJMKIzFTgghxAwKOyGEDBgUdkIIGTAo7IQQMmBQ2AkhZMAotLAzKRghhPRSqHDHdpgUjBBCvCmsxc6kYIQQ4k1hhZ1JwQghxJvCCjuTghFCiDeFFXYmBSOEEG8KK+xMCkYIId4UNioGYFIwQgjxIpHFLiJ3ishLIvITEfmWiJxtq2KEEELikdQVsw/ANlW9CMDfALgteZUIIYQkIZGwq+ojqnrS/fokgM3Jq0QIISQJNjtPPw7gLyxujxBCSAxCO09F5LsA/oHHT5Oq+mdumUkAJwH4ZmsRkXEA4wAwymBzQghJDVHVZBsQuQnApwBcrarLIcVX11kEMBdzl+cCOBRz3aLCNg8HbPNwkKTNVVXdFFYokbCLyA4AfwTgfaq6GHtD0fY5o6pjWewrL7DNwwHbPBxk0eakPvY/BnAmgH0i8iMR+aqFOhFCCElAogFKqvqPbFWEEEKIHYqYUmC63xXoA2zzcMA2Dweptzlx5ykhhJB8UUSLnRBCSAC5FXYR2SEiPxORV0Rkj8fvp4nIA+7vT4lILfta2sWgzf9WRF50c/N8T0Sq/ainTcLa3Fbut0RERaTwERQmbRaR33bP9Qsi8qdZ19E2Btf2qIg8KiLPutf3tf2op01E5F4ReUNEnvf5XUTkbveY/ERELrG2c1XN3QdACcCrAH4VwAYAPwZwQVeZmwF81f3/owAe6He9M2jzVQDK7v8Tw9Bmt9yZAH4AJ23FWL/rncF5Pg/AswDOcb///X7XO4M2TwOYcP+/AMBsv+ttod3/BMAlAJ73+f1aOKP1BcAVAJ6yte+8WuyXA3hFVfer6i8BfAPAB7vKfBDA193/HwRwtYhIhnW0TWibVfVRPTUIbBBy85icZwD4TwD+M4DjWVYuJUzavAvAl1X1FwCgqm9kXEfbmLRZAfw99/+zALyeYf1SQVV/AODnAUU+COC/qcOTAM4Wkbfb2Hdehf0dAA62fV9wl3mWUScR2ZsAKpnULh1M2tzOJ1D83DyhbXZfT9+pqv8ry4qliMl5/jUAvyYifyUiT7oDAYuMSZv/A4AbRWQBwMMAdmdTtb4S9Z43ptATbQwrInIjgDEA7+t3XdJEREbgjGy+qc9VyZp1cNwxV8J5K/uBiFyoqof7Wqt02QngPlX9QxF5N4D7RWSbqrb6XbEikleL/TUA72z7vtld5llGRNbBeX1byqR26WDSZojIPwUwCeB6Vf1/GdUtLcLafCaAbQAeE5FZOH7IhwregWpynhcAPKSqJ1T1AJy5Ds7LqH5pYNLmTwD4JgCo6g8BnA4np8ogY3TPxyGvwv7XAM4TkS0isgFO5+hDXWUeAvC77v8fBvCX6vZIFJTQNovIxQD+CxxRL7rfFQhps6q+qarnqmpNVWtw+hWuV9WZ/lTXCibX9rfhWOsQkXPhuGb2Z1lJy5i0eR7A1QAgIufDEfZM8k/1kYcA/I4bHXMFgDdV9W+tbLnfPccBPcrXwrFUXoWTIhgA/iOcGxtwTvx/B/AKgP8N4Ff7XecM2vxdAH8H4Efu56F+1zntNneVfQwFj4oxPM8CxwX1IoDnAHy033XOoM0XAPgrOBEzPwJwTb/rbKHNewH8LYATcN7CPgHg0wA+3Xaev+wek+dsXtsceUoIIQNGXl0xhBBCYkJhJ4SQAYPCTgghAwaFnRBCBgwKOyGEDBgUdkIIGTAo7IQQMmBQ2AkhZMD4/+3WNxosfpWPAAAAAElFTkSuQmCC\n", 79 | "text/plain": [ 80 | "
" 81 | ] 82 | }, 83 | "metadata": {}, 84 | "output_type": "display_data" 85 | } 86 | ], 87 | "source": [ 88 | "import matplotlib.pyplot as plt\n", 89 | "plt.plot(xtrain,ytrain,'ro')\n", 90 | "plt.plot(xtest,ytest,'bo')\n", 91 | "plt.legend(['train','test'])" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "metadata": {}, 97 | "source": [ 98 | "**Validation curve** A validation curve consists in varying a model parameter that controls its complexity (here the degree of the polynomial) and measures both error of the model on training data, and on test data (eg with cross-validation). The model parameter is then adjusted so that the test error is minimized:\n", 99 | "\n", 100 | "We use `sklearn.model_selection.validation_curve()` to compute train and test error, and plot it:" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 16, 106 | "metadata": {}, 107 | "outputs": [ 108 | { 109 | "name": "stderr", 110 | "output_type": "stream", 111 | "text": [ 112 | "/home/pbellec/env/nilearn/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n", 113 | " warnings.warn(CV_WARNING, FutureWarning)\n" 114 | ] 115 | }, 116 | { 117 | "data": { 118 | "text/plain": [ 119 | "" 120 | ] 121 | }, 122 | "execution_count": 16, 123 | "metadata": {}, 124 | "output_type": "execute_result" 125 | }, 126 | { 127 | "data": { 128 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VfWZ+PHPc5ckJCQQsrAkQFgCsoUtKkp1rFtRW7XudLE6rf7sVMfWVgdrx236+r0cp7/ptDN0Ouowth0rWlstbWldqrgCEjbZEnZCACEJIQmEJHd5fn+ck3AJWS4k5Cb3Pu/X677O9j33Pvfm5jnf+z3f8z2iqhhjjEkMnlgHYIwxpvdY0jfGmARiSd8YYxKIJX1jjEkglvSNMSaBWNI3xpgEYknfGGMSiCV9Y4xJIJb0jTEmgfiiKSQi84CfAF7gOVV9qs32UcAvgMFumQWqulRECoAtQJlbdIWq3tPZa2VnZ2tBQcFpvAVjjDGrV6+uUtWcrsp1mfRFxAssBK4AKoBVIrJEVTdHFPsB8LKq/qeITAaWAgXuth2qOiPawAsKCigpKYm2uDHGGEBE9kRTLprmnfOA7aq6U1WbgcXAdW3KKJDhzg8C9kcbqDHGmN4TTdLPA/ZGLFe46yI9DnxFRCpwavn3RWwbIyJrReRdEbmoO8EaY4zpnp46kTsfeF5V84GrgV+JiAc4AIxS1ZnAA8CvRSSj7c4icreIlIhISWVlZQ+FZIwxpq1okv4+YGTEcr67LtLXgZcBVHU5kAJkq2qTqla761cDO4AJbV9AVZ9R1WJVLc7J6fI8hDHGmDMUTdJfBRSKyBgRSQJuA5a0KVMOXAYgIpNwkn6liOS4J4IRkbFAIbCzp4I3xhhzerrsvaOqQRG5F3gdpzvmIlXdJCJPAiWqugT4LvCsiHwH56TuHaqqInIx8KSIBIAwcI+qHj5r78YYY0ynpK/dOau4uFity6YxxpweEVmtqsVdlYvq4ixjjIlrgePQUA3NxyAUgHAAwiF3Pugsh4In5sNBd7ll3i2vIdAwqALayZQ2y+4+GSOg+M6z+lYt6Rtj4osqNNVDQxU0HIZjVc5867T61OXAsVhH7cg/15K+MaYPU4VAg5Nkm+qd+WAzhJog2ASh5jbTpojt7ZQLB50ac9itVWvo5OVwxLJGrg8706Z6p8Yeamo/Xl8KpGZD6hBIy4as8c40NcuZJg0Erx88fvD4wOtzph6/u94bMe9ui5wXj/sQQKKbtpbvHZb0jUlEwSZoOgrN9U6TRst809ETCbypHprqOpiPWNbwmcchXvAlgzfJmXpaEqvv5Kl4TyRWjxd8SW6SjSzrdZJ2SwJPzY6YZjnTpLReTbB9kSV9Y/qCwHE4tBmO17RTmw21UwMOt78u0OAmcPfR1GbaMh8ORBeXPw2S009+pOVAcsap65PTwZ96chJvnSY7ifqkabKTqE2vsqRvTG87XgOfboADn8CnnzjTqq1Ogu8ujx+SB0JSujt1H+nD2qxzk3nSQHddmrs94pE00GneMHHF/qLGnC2qUH8gIrmvd6ZHyk+USR8Ow4pg0ued6cChJzdXRDZptG3iaNnetonDmE5Y0jcmWuGwc8Ix1Ox203PnQ4ETJyIP7zxRez+w3ukd0mLIOMibDbPvhOFFbpLPjd37MQnJkr7p21Q76M3RQQ+PYCM0Nzhd8ALHO5k/5rR/t50PNrpJPHByQg81R9/84vFD7jkwYd6J5D5sqtNkYkyMWdI3jqajTjtzU93JJwjbnjBsm3Rbu82FT679dtQ9L9TcTpe9iK57bZN5d3qGtMfjd042JqU605b5pIGQlnvi5KPX7067mvefvH5QPuSc4zyPMX2QJf1EFA5BZSlUlMC+EqhYDZVbeibBenwd99Ro6cnhS4aUjPZ7dnj9Tr/lyC574uWUbnwdlfGluAk9DfwDnBOU/tQT815/99+jMf2YJf1EUHfATe4lsG817F/rdN0DSBnstDOfc40zHZjT5sRgJwm2vROJCd4H2pi+zpJ+vGk6CgfWnVyLr3fvXunxO23L0+dDfjHkFUPWOEvUxiQQS/p9TSjQ5qrHjq6GbGd94xGn90hLM01mAYy+8ESCHzYN/CkxfXvGmNiypN9XVKyGV+6EI9Hc0F5OvSIyZZBzEnHqjU6Cz5vtXHpujDERLOn3BRtegd9/y7kw55LvOyc5k9MjEnubBG/jhxhjzpAl/VgKh+Hdp+Ddf4ZRF8Kt/2u1c2PMWWVJP1aaG+D3fwebXoUZX4HP/6v17TbGnHWW9GOh7gAsng/718EV/wQX3mfNNcaYXmFJv7ftXwsvznd628x/ESZeFeuIjDEJxJJ+b9r0Grx6jzMe+dffgKFTTtq8o/IowZAybFAKGSk+pBdr/6pKfVMQn0cY4Pf26msbY3qPJf3eoArv/Qje+SGMPB9ufcG58tXV0BzkqT+X8svlJ7prDvB7GT4ohWEtj4wUd3lA6/ohqUl4PJ0n53BYqWlo5lB9E4fqmzhY10hlfROH6hpb1x2qb+RQXRNNQad/v88jpKf4yBjgJyPF78yn+MkY4CM9JWLdAD8ZKe66AU4Zj0cIh5VgWAmFlbA605b5YFgJu8shVcJhCIbDbjnwesDn8eDzCn6vB5/Hmfq97jp3m88rJHk9+CLKeLv4LIwxlvTPvsBx+P29sPEVKLoNvvCTky6QWr3nMN99eT27qxu4c24Bs0dn8mltI5/WNnKgzpmu3HmYg3WNBMN60lP7vcLQiIPB0PRkGgIhDtU1UVnvJPXK+qZT9gNIT/GRm55MbnoKs0ZlkpueTE56MqEw1DUGqG8MUHc86Ewbg+ysOkrd8SB1jQEamnvgZh9ngUegICuNWaMzmT06k+LRmYzLGdjlgdGYRGJJ/2yqPwiLv+QMh3DZY/CZ77SesG0Khvjxm9t45r0djBg8gBfvmsMF4zrurhkKK9VHm/i0rpEDLQeF2kY+rT3Op3WNbKg4wpt1jQzwe8lNTyE3I5nxuenkZiS3JvfI+QFJZ36zjWAoTH1jkPpG5yBQ5x4g6hoDqCoecWriHhG8HsErgsedej3OvM8Tsd1D63zI/ZUQCIUJhpRgOEwgpARDzrpAKEwwrARD7np3eyAUpjkYZuvBo7xdeohXVlcAkJHicw4Co5wDwfSRg0lLtq+9SVz27T9bDqx3Ttger3H630/6Quumjftq+e7L6yk7WM/880byyDWTGdhFIvJ6hNyMFHIzUijKP9vBd87n9ZCZlkRmWlJsA+mAqrKr6hir99SwpryG1XtqWFZWCTif46Th6cweldn6iyBv8ICzcg4jFFZqjweoaWjmSEMzh4+dmK9pCFBzrJkad75lnaqS5PWQ7Pe6Uw/JPg9JPg/JvhPrTky9J5XJHpjMmOw0CrLSyB6YZOdmzClE9dSf/qcUEpkH/ATwAs+p6lNtto8CfgEMdsssUNWl7raHga8DIeDvVfX1zl6ruLhYS0pKzuCt9CFb/gi/uwsGZDo9dIZPB5wa8n8u28FP/rqNIWlJ/PONRXz2HLtzUm+obQiwdq9zAFi9p4Z1e4+0NlMNzUhm9uhMZo3KJGtgEoGg0uz+qmgOutOWXxptl1vLOfvUNwY40uAk99rjATr69/J7hcGpSWSm+lunmalJeD1CU9B53aZgKGK+83WB0KkvNDDZR0F2KgVZaa0HgoLsNAqyUhmSZgeEeCMiq1W1uMtyXSV9EfECW4ErgApgFTBfVTdHlHkGWKuq/ykik4Glqlrgzr8InAeMAN4CJqh2fAuifp30VeGDH8Nfn3DGv7nt15A+FIDth47y3ZfXsb6ilmunj+DJ66YwOLVv1pQTQTAUpvTT+tZfAqv31FBRc7zTfZK8Hvxewe9zTiy3LCf5Wk40e8hI8Z2SzIekJZ2U2Aen+hmY3LO9s8JhpSkY5tO6RnZXH2N3lfPYVd3A7qpjVNQ0EHlqJz3Fd9KBYIx7cJg4LJ3UJGsA6I+iTfrR/HXPA7ar6k73iRcD1wGbI8ookOHODwLcsXy5Dlisqk3ALhHZ7j7f8qjeRX+iCkvuhbX/C1Nvguv+A/wDCIeVRR/u4l9eLyM1ycvCL83imqLhsY424fm8HqbmDWJq3iBuv6AAgEP1jRxrCrlJXNyk3vKQPl0z9niEAUlexmQ7tXomnry9ORhmb41zANhVdYw91Q3srj7GmvIa/vDJ/tZfJD6PMDVvEOePHcL5Y4ZQXDCEjBS78Uw8iSbp5wF7I5YrgPPblHkceENE7gPSgMsj9l3RZt+8M4q0rzu800n4598D854CEfYebuB7v1nPyl2HuXxSLv/3hmnkptvQxn1VbnoKxOltbJN8HsblDGRczsBTtjUFQ+w93MDOymOsrzjCyp2HWfTBLv7r3Z14BCaPyOC8gizOHzuE8wqG9NlzOSY6PfU7bj7wvKr+PxG5APiViEyNdmcRuRu4G2DUqFE9FFIvK3ePbbPvRIHFH5fzwz9uRkR4+qYibp6d36driiZxJfu8jM9NZ3xuOldOGQZAYyDEmvIaVu48zMe7DvPCyj0s+nAXABOHpjsHgDHOI14qMupeU+LzemIdylkVTdLfB4yMWM5310X6OjAPQFWXi0gKkB3lvqjqM8Az4LTpRxt8n1K+HFIGczB5FP/w/CqWlVVy4bgsnr6piPzM1FhHZ8xpSfF7uXBcNheOywacXwMbKmpZueswK3ZW88rqitaLCcdmp7UeBC4Ym82wQf3vIPBJxRG+95v1bDt0lMzUJHIGJpOdnkT2wOSIRxLZ6cnkDHSuaRmSloS/Hx4gojmR68M5kXsZTsJeBXxJVTdFlPkz8JKqPi8ik4C/4jTjTAZ+zYkTuX8FCuPyRO5/nMtB3wiuPPgtmoIhHr5qEl+dM9ouDDJxKRgKs3F/HR/vqnZ+Dew+TH1jEIDxuQP5zPhsLirM5vyxWV12R46lQCjMwne28+9vbydnYDI3zs6jpiFAVX0TVUebqDzaRFV9M8cD7aeszFT/iYNCejLDMpLJz0wlP3MA+Zmp5GUO6LX332MnclU1KCL3Aq/jdMdcpKqbRORJoERVlwDfBZ4Vke/gnNS9Q52jySYReRnnpG8Q+FZnCb/fOlYFVVt5PjibgmGp/PjWGYxtp+3UmHjh83qYMXIwM0YO5u6LxxEKK1sO1PHRjire31bFix+X8/xHu/F5hFmjMvlMYTafKcymKG9Qn2k+2X6ongdeXs8nFbV8cWYej39hCoNS2z9pfawpSNVR90BQ3xwx3+TON/NJxRHeqG1sHc6kxeBUv3MQGOwcDPLcA4JzYBhAei+fKI+qn35v6pc1/dI/weIvcWPTYyz4P3dwbsGQWEdkTEw1BkKs3lPD+9uq+GB7JZv216HqdBW9cFwWnynM4aLx2YzOSu31c13hsPI/H+3m6b+Ukprk5f9+cRpXTeuZHnWqStXRZipqGqioOe4+Gth35MR8Y+Dkg0JGiq/1IDB95GC+9dnxZ/TaPdll03SlfAVB8bNFxjItb1CsozEm5lL8XuaOz2bu+GzgHA4fa+bD7VV8sK2KD7ZX8fqmgwDkZw7gosJsPjM+h7njs876tSsVNU6PuhU7z06POhEhxx3HauaozFO2qyrVx5rZF3FAqKg5zr4jx9ldfYxQO+Nk9TRL+j2hfAXbfYUUZmWT4j/zMW2MiVdD0pL4wvQRfGH6iNZhMj7Y7jQF/XH9AV78eC8iUDw6kxtm5XP1tOEMGtBzzR6qym9WV/DkH5zLi56+sYibi3u/R52ItJ4DmD5ycK++dgtL+t0VOI7uX8t7wavaPbIbY04mIozNGcjYnIHcfkEBwVCY9RW1vL+tkj+s38/Dv9vAY0s2ccWkodwwK4+LJ+R0q5dMZX0TD/9uA29tOcj5Y4bwo5unM3JI4vaos6TfXfvWIOEAK4OFXDfakr4xp8vn9TDbHfzu/ssK2bCvlt+t2ceS9fv504YDZLm/Em6clc/UvIzTqp3/ZeMBvv/qRo42BfnBNZP427ljEr5HnSX97trrXJS1OjyBJ0bF5ueaMfFCRCjKH0xR/mAeuWYS75ZV8uraffx6pdMbaHzuQG6Ylcf1M/IYMXhAh89TezzAE0s28bu1+5iWN4h/vWU6hUPj9HLr02RJv7vKV3AgqYDkpGzyOvkSGmNOj9/r4fLJQ7l88lBqGwL8acMBXl1bwdN/KeNfXi/jgrFZ3DArn3lTh53UF/6DbVU8+Mp6DtU3cf9lhdx76fh+eRHV2WJJvzvCYdi7klWh85g1JtOGWTDmLBmU6udL54/iS+ePYk/1MV5du49X1+7je79Zzz++tpHPTRnKdTPzWFZ6iF8s38O4nDR+980LY3aytC+zpN8dlaXQWMuy5nHMspO4xvSK0VlpfPvyCdx/WSFrymv43Zp9/GH9fl5b5wzu+7dzx/DQvInWk64DlvS7o9wZIXqVTuTLo61GYUxvEhFmjx7C7NFDePQLk3lvaxU56cnMsNp9pyzpd0f5Cur92XwaGMqUEXZRljGxkuzzcsXkobEOo1+wsxvdsXcFGz3nMGXEYPspaYzpFyzpn6nafXCknLcbxlp7vjGm37Ckf6bc/vnLgxOYbRdlGWP6CUv6Z6p8JQHvALboaGbZSVxjTD9hJ3LPVPlydiRPIteXxvBBdlGWMaZ/sJr+mWisg4Mb+bCp0NrzjTH9iiX9M1GxCjTMO8fHMtPG2zHG9COW9M/E3pUoHtaGxzPLTuIaY/oRS/pnonw5n6YWEvCmMWVERqyjMcaYqFnSP12hAFSUsEbPYVr+IJJ9dlGWMab/sKR/uj7dAIEG3qgvYJa15xtj+hlL+qer3Lkoa0XQeu4YY/ofS/qnq3w5dSl5HGSIncQ1xvQ7lvRPhyrsXUmpfzJ5gwcwNCMl1hEZY8xpsStyT0fNLjh6kHd8Y5k51trzjTH9j9X0T4fbnv/WMbtTljGmf4oq6YvIPBEpE5HtIrKgne0/FpF17mOriByJ2BaK2LakJ4PvdeXLCfgz2K4jbGRNY0y/1GXzjoh4gYXAFUAFsEpElqjq5pYyqvqdiPL3ATMjnuK4qs7ouZBjqHwlu1KnknTcx6ThdlGWMab/iaamfx6wXVV3qmozsBi4rpPy84EXeyK4PuVYNVSVsSI4gaL8QST5rGXMGNP/RJO58oC9EcsV7rpTiMhoYAzwdsTqFBEpEZEVInL9GUcaa3tXArC0drS15xtj+q2e7r1zG/CKqoYi1o1W1X0iMhZ4W0Q2qOqOyJ1E5G7gboBRo0b1cEg9ZO8Kwp4k1gbHcIclfWNMPxVNTX8fMDJiOd9d157baNO0o6r73OlOYBknt/e3lHlGVYtVtTgnJyeKkGKgfAWH0ifRRJLdKcsY029Fk/RXAYUiMkZEknAS+ym9cETkHCATWB6xLlNEkt35bGAusLntvn1eoBH2r2W9TCI/cwC56XZRljGmf+qyeUdVgyJyL/A64AUWqeomEXkSKFHVlgPAbcBiVdWI3ScB/yUiYZwDzFORvX76jf1rIdTMm0cLmD3emnaMMf1XVG36qroUWNpm3aNtlh9vZ7+PgGndiK9vKHd+vPz1aAHftvZ8Y0w/Zv0Oo1G+gvr0cdSQYT13jDH9miX9roTDsHclW5OmkOL3cM7w9FhHZIwxZ8wGXOtKVRk0HuF93ziK8gfj99px0hjTf1kG64rbnr/ksF2UZYzp/yzpd6V8Bc0DctgZzrHbIxpj+j1L+l0pX0HFwCJA7E5Zxph+z5J+Z+r2w5E9lIQnMjorleyBybGOyBhjusWSfmfcm6b88Yi15xtj4oMl/c7sXUnYN4CPjg239nxjTFywpN+Z8uVUDy4iiI+ZVtM3xsQBS/odaaqHTzewyTeZ1CQv5wyzi7KMMf2fJf2OVJSAhvnrsXEU5Q/CZxdlGWPigGWyjpSvQMXDkmq7CboxJn5Y0u9I+XIaMidRG06xnjvGmLhhSb89oSBUlLBjgDMqtJ3ENcbEC0v67Tm4AQLHWB4Yz5jsNIakJcU6ImOM6RGW9NvjXpT1WtVIZlr/fGNMHLGk357y5QQzRrKlId3a840xccWSfluqUL6S/RnTASzpG2PiiiX9tmp2w9FPWcc5pCV5mWgXZRlj4ogl/bbc9vy/1BUwY9RgvB6JcUDGGNNzLOm3tXcFmjKIN6oyrWnHGBN3LOm3Vb6CI1kzCYbFkr4xJu5Y0o/UcBgqSylNmgJg3TWNMXHHkn6kvSsBePf4eMbmpDE41S7KMsbEF0v6kcpXoB4/vz+Ua007xpi4FFXSF5F5IlImIttFZEE7238sIuvcx1YRORKx7Wsiss19fK0ng+9x5Stoyi3iQIPYyJrGmLjk66qAiHiBhcAVQAWwSkSWqOrmljKq+p2I8vcBM935IcBjQDGgwGp335oefRc9IdgE+9ewp+DLgF2UZYyJT9HU9M8DtqvqTlVtBhYD13VSfj7wojv/OeBNVT3sJvo3gXndCfisqdoGoWbWBkaTnuyjMHdgrCMyxpgeF03SzwP2RixXuOtOISKjgTHA26ezr4jcLSIlIlJSWVkZTdw9r7IUgHePZDNj1GA8dlGWMSYO9fSJ3NuAV1Q1dDo7qeozqlqsqsU5OTk9HFKUKktR8fJOZbqNn2+MiVvRJP19wMiI5Xx3XXtu40TTzunuG1uVpRxPH02j+pll/fONMXEqmqS/CigUkTEikoST2Je0LSQi5wCZwPKI1a8DV4pIpohkAle66/qeyjIO+EcDMHOk1fSNMfGpy6SvqkHgXpxkvQV4WVU3iciTInJtRNHbgMWqqhH7Hgb+CefAsQp40l3XtwSboHoHm4IjGJ87kEGp/lhHZIwxZ0WXXTYBVHUpsLTNukfbLD/ewb6LgEVnGF/vqN4BGuKjumxmT7FavjEmftkVudDac2dd43Cmj7T2fGNM/LKkD27PHQ+7dBgTh1n/fGNM/LKkD1BZSm1KPk0kMT7X7pRljIlflvQBKsso945i+KAUBg2wk7jGmPhlST/YDNXb2RIcQeFQq+UbY+KbJf3DOyEcZNWxXCbYeDvGmDhnSb9yCwCbgyOYMMxq+saY+GZJv7IMRdihI5hgzTvGmDhnSb+ylNqUPJpIsuGUjTFxz5L+oVL2+kaTN3gAaclRXaBsjDH9VmIn/VAAqrezOTCcidaeb4xJAImd9A/vgnCAkmO5FA61ph1jTPxL7KTf0nMnlMcEuxLXGJMAEjzpn+i5Y807xphEkNhJ/9AWalNG0CTJjMux5h1jTPxL7KRfWcZe7yhGDUllQJI31tEYY8xZl7hJPxSE6m1sDgyn0NrzjTEJInE7ptfsglAzq5qG2hj6xpiEkbg1ffduWWWhPBt+wRiTMBI+6e/QEda8Y4xJGImb9A+VUps8nCbPAMbmpMU6GmOM6RWJm/Tdu2WNzkolxW89d4wxiSExk344BFVb2RwYblfiGmMSSmL23qnZDaEmShqH2o1TjDEJJTFr+u5J3K3hPCbYQGvGmAQSVdIXkXkiUiYi20VkQQdlbhGRzSKySUR+HbE+JCLr3MeSngq8Ww45A61tV+uuaYxJLF0274iIF1gIXAFUAKtEZImqbo4oUwg8DMxV1RoRyY14iuOqOqOH4+6eyjJqk4bR1JxKQZb13DHGJI5oavrnAdtVdaeqNgOLgevalLkLWKiqNQCqeqhnw+xhlaWUe0cyNieNJF9itnAZYxJTNBkvD9gbsVzhros0AZggIh+KyAoRmRexLUVEStz113cz3u5ze+5sCoyg0Jp2jDEJpqd67/iAQuASIB94T0SmqeoRYLSq7hORscDbIrJBVXdE7iwidwN3A4waNaqHQurAkT0QbGRNYKh11zTGJJxoavr7gJERy/nuukgVwBJVDajqLmArzkEAVd3nTncCy4CZbV9AVZ9R1WJVLc7JyTntN3FaKssA2BbOs4HWjDEJJ5qkvwooFJExIpIE3Aa07YXzGk4tHxHJxmnu2SkimSKSHLF+LrCZWHJ77mzTPGveMcYknC6bd1Q1KCL3Aq8DXmCRqm4SkSeBElVd4m67UkQ2AyHgQVWtFpELgf8SkTDOAeapyF4/MVFZRl1SLs2BgYwekhrTUIwxprdF1aavqkuBpW3WPRoxr8AD7iOyzEfAtO6H2YMqt1DuGcm43IH4vNZzxxiTWBIr64XDULmVjYERdiWuMSYhJdbYO7XlEDzO2sAwuxLXGJOQEqumf8gZc2dbOI/CXKvpG2MST2IlfXegte2ax0QbXdMYk4ASq3mnsow6fzbNoXRGZlrPHWNM4kmwmv4W9nhGMT53IB6PxDoaY4zpdYmT9N2eO5sCw+0krjEmYSVO807tXggcY7313DHGJLDEqem7Y+5sDedbH31jTMJKoKR/oueO1fSNMYkqoZJ+vS+LYNIg8gYPiHU0xhgTEwmV9Pd4RzJ+aDoi1nPHGJOYEiPpq0JlGRubRzDBrsQ1xiSwxOi9U1sBzUfZEBhmV+IaYxJaYtT0I3ru2I1TjDGJLEGSvjvQmuZZd01jTEJLkKS/haO+TEIpmQzLSIl1NMYYEzMJkvTL2O0ZxQTruWOMSXDxn/RV0coyNjYPs6YdY0zCi/+kX7cfaapjY2AEhbl2EtcYk9jiP+m3nMQN51t3TWNMwkucpK95FFrzjjEmwSVE0j/qHUw4NYucgcmxjsYYY2Iq/q/IrSxjj2ckE7Kt544xxsR3TV8VPbSFDYHh1nPHGGOI96Rf/ynSVMemwAgbQ98YY4gy6YvIPBEpE5HtIrKggzK3iMhmEdkkIr+OWP81EdnmPr7WU4FHpXILYDdOMcaYFl226YuIF1gIXAFUAKtEZImqbo4oUwg8DMxV1RoRyXXXDwEeA4oBBVa7+9b0/FtphzvQ2rZwviV9Y4whupr+ecB2Vd2pqs3AYuC6NmXuAha2JHNVPeSu/xzwpqoedre9CczrmdCjUFnKMe8gSMtmSFpSr72sMcb0VdEk/Txgb8Ryhbsu0gRggoh8KCIrRGTeaeyLiNwtIiUiUlJZWRl99F05VMpuTz6FQzN67jmNMaYf66kTuT6gELgEmA88KyKDo91ZVZ9R1WJVLc7JyemZiFTRylI2Ng+3K3GNMcYVTdLfB4yMWM5310WqAJaoakBVdwFbcQ4C0ex7dhxjyJDEAAAQhUlEQVQ9hDQeYXPQrsQ1xpgW0ST9VUChiIwRkSTgNmBJmzKv4dTyEZFsnOaencDrwJUikikimcCV7rqzz+25s8167hhjTKsue++oalBE7sVJ1l5gkapuEpEngRJVXcKJ5L4ZCAEPqmo1gIj8E86BA+BJVT18Nt7IKSJ77tjomsYQCASoqKigsbEx1qGYbkhJSSE/Px+/339G+0c1DIOqLgWWtln3aMS8Ag+4j7b7LgIWnVF03VFZSoMnHU96LoNSz+zDMSaeVFRUkJ6eTkFBgQ1J0k+pKtXV1VRUVDBmzJgzeo74vSL3UCm7PSOZMMx67hgD0NjYSFZWliX8fkxEyMrK6tavtfhM+qpo5RY2NA+39nxjIljC7/+6+zeMz6R/rBI5XkNpaIQNtGaMOS3Lli3j85//PABLlizhqaeearfcwIGd55YjR47ws5/9rHV5//793HTTTT0X6BmKz6TfeuOUfAqtpm9MvxEMBmMdwkmuvfZaFixod7ixLrVN+iNGjOCVV17pqdDOWJwmfafnztZwPoW5VtM3pi/55S9/SVFREdOnT+erX/0qd9xxB/fccw/nn38+Dz30EIcPH+b666+nqKiIOXPm8MknnwDw7rvvMmPGDGbMmMHMmTOpr6/nwIEDXHzxxcyYMYOpU6fy/vvvn/J6c+bMYdOmTa3Ll1xyCSUlJXz88cdccMEFzJw5kwsvvJCysrJT9n3++ee59957Adi1axcXXHAB06ZN4wc/+EFrmaNHj3LZZZcxa9Yspk2bxu9//3sAFixYwI4dO5gxYwYPPvggu3fvZurUqYBzfuXOO+9k2rRpzJw5k3feeaf19W644QbmzZtHYWEhDz30UA996ifE501UDm3huCcN/6DhpKdYzx1j2nriD5vYvL+uR59z8ogMHvvClE7LbNq0iR/+8Id89NFHZGdnc/jwYR544AEqKir46KOP8Hq93HfffcycOZPXXnuNt99+m9tvv51169bxox/9iIULFzJ37lyOHj1KSkoKzzzzDJ/73Od45JFHCIVCNDQ0nPKat956Ky+//DJPPPEEBw4c4MCBAxQXF1NXV8f777+Pz+fjrbfe4vvf/z6//e1vO4z9/vvv55vf/Ca33347CxcubF2fkpLCq6++SkZGBlVVVcyZM4drr72Wp556io0bN7Ju3ToAdu/e3brPwoULERE2bNhAaWkpV155JVu3bgVg3bp1rF27luTkZCZOnMh9993HyJEj6SlxW9PfJSMptOEXjOlT3n77bW6++Ways7MBGDJkCAA333wzXq8XgA8++ICvfvWrAFx66aVUV1dTV1fH3LlzeeCBB/jpT3/KkSNH8Pl8nHvuufzP//wPjz/+OBs2bCA9/dT/+VtuuaW1WeXll19ubVevra3l5ptvZurUqXznO9856ddAez788EPmz58P0BofON0ov//971NUVMTll1/Ovn37OHjwYKfP9cEHH/CVr3wFgHPOOYfRo0e3Jv3LLruMQYMGkZKSwuTJk9mzZ0+nz3W64rKmr5WlbAwUWc8dYzrQVY28t6WlpXVZZsGCBVxzzTUsXbqUuXPn8vrrr3PxxRfz3nvv8ac//Yk77riDBx54gPT0dJ544gkAnnvuOYqLi8nKyuKTTz7hpZde4uc//zkA//iP/8hnP/tZXn31VXbv3s0ll1zSZQzt9Zx54YUXqKysZPXq1fj9fgoKCrrVpTI5+cS9vL1eb4+f54i/mv6xKqShirKQ3S3LmL7m0ksv5Te/+Q3V1dUAHD586gX6F110ES+88ALg9KTJzs4mIyODHTt2MG3aNP7hH/6Bc889l9LSUvbs2cPQoUO56667+MY3vsGaNWv44he/yLp161i3bh3FxcWA08Tz9NNPU1tbS1FREeDU9PPynEF/n3/++S5jnzt3LosXLwZoja/leXJzc/H7/bzzzjutNfP09HTq6+vbfa7I97h161bKy8uZOHFilzH0hPhL+m7Pna2ab901jeljpkyZwiOPPMLf/M3fMH36dB544JSL+Hn88cdZvXo1RUVFLFiwgF/84hcA/Nu//RtTp06lqKgIv9/PVVddxbJly5g+fTozZ87kpZde4v7772/3dW+66SYWL17MLbfc0rruoYce4uGHH2bmzJlR1aZ/8pOfsHDhQqZNm8a+fSfGjfzyl79MSUkJ06ZN45e//CXnnHMOAFlZWcydO5epU6fy4IMPnvRcf/d3f0c4HGbatGnceuutPP/88yfV8M8mcUZQ6DuKi4u1pKTkzJ/g42dh6feY0/jvvP3kl0hNissWLGNO25YtW5g0aVKswzA9oL2/pYisVtXirvaNw5p+Gcc9aSRl5lvCN8aYNuIw6ZeyS/KZYD13jDHmFHGX9LWylE2B4XYlrjHGtCO+kv6xauRYJWWhPDuJa4wx7YivpF/l3jhF8627pjHGtCO+kv4h5xaJOzSPcTlW0zfGmLbiK+lXltEoA/APGUWK3xvraIwxEdqOOhmtq6++miNHjnRa5tFHH+Wtt94609ASSpwl/S3Wc8eYPqqjpN/VhVFLly5l8ODBnZZ58sknufzyy7sVX6KIq6SvlWVsCtjwC8b0RZFDDZ977rlcdNFFXHvttUyePBmA66+/ntmzZzNlyhSeeeaZ1v0KCgqoqqpi9+7dTJo0ibvuuospU6Zw5ZVXcvz4cQDuuOOO1kHVCgoKeOyxx1qHOi4tda7Sr6ys5IorrmDKlCl84xvfYPTo0VRVVfXypxB78XP1UsNh5OhBtoYvZaolfWM69+cF8OmGnn3OYdPgqvbvMgWcNNTwsmXLuOaaa9i4cWPrDb4XLVrEkCFDOH78OOeeey433ngjWVlZJz3Htm3bePHFF3n22We55ZZb+O1vf9s6WmWk7Oxs1qxZw89+9jN+9KMf8dxzz/HEE09w6aWX8vDDD/OXv/yF//7v/+7Z999PxE9N3+Nj3fTHWBaeYd01jekHzjvvvNaED/DTn/6U6dOnM2fOHPbu3cu2bdtO2WfMmDHMmDEDgNmzZ580Rn2kG2644ZQyH3zwAbfddhsA8+bNIzMzswffTf8RPzX9lAzeSr2GnbKDsdmW9I3pVCc18t4SOZzysmXLeOutt1i+fDmpqalccskl7Q5P3HbY4ZbmnY7KnY2hifu7+KnpA2UH6ynITiPJF1dvy5i40NlQw7W1tWRmZpKamkppaSkrVqzo8defO3cuL7/8MgBvvPEGNTU1Pf4a/UH81PSBbQfrmTwiI9ZhGGPaETnU8IABAxg6dGjrtnnz5vHzn/+cSZMmMXHiRObMmdPjr//YY48xf/58fvWrX3HBBRcwbNiwdu+0Fe+iGlpZROYBPwG8wHOq+lSb7XcA/wK0DDL9H6r6nLstBLScMSpX1Ws7e60zHVq5MRBi0qN/4f7LCvn25RNOe39j4l2iD63c1NSE1+vF5/OxfPlyvvnNb7bev7a/6c7Qyl3W9EXECywErgAqgFUiskRVN7cp+pKq3tvOUxxX1RldvU53HW0K8oWiERSPHnK2X8oY0w+Vl5dzyy23EA6HSUpK4tlnn411SDERTfPOecB2Vd0JICKLgeuAtkk/prIHJvPT+TNjHYYxpo8qLCxk7dq1sQ4j5qI545kH7I1YrnDXtXWjiHwiIq+IyMiI9SkiUiIiK0Tk+u4Ea4wxpnt6qpvLH4ACVS0C3gR+EbFttNvO9CXg30RkXNudReRu98BQUllZ2UMhGWPa6mu3RzWnr7t/w2iS/j4gsuaez4kTti1BVKtqk7v4HDA7Yts+d7oTWAac0gajqs+oarGqFufk5JzWGzDGRCclJYXq6mpL/P2YqlJdXU1KSsoZP0c0bfqrgEIRGYOT7G/DqbW3EpHhqnrAXbwW2OKuzwQaVLVJRLKBucDTZxytMeaM5efnU1FRgf2a7t9SUlLIz88/4/27TPqqGhSRe4HXcbpsLlLVTSLyJFCiqkuAvxeRa4EgcBi4w919EvBfIhLG+VXxVDu9fowxvcDv95807IFJTFH10+9NZ9pP3xhjElm0/fRtvAJjjEkglvSNMSaB9LnmHRGpBPbEOo5OZAN9+c4LFl/3WHzdY/F1T3fiG62qXXZ/7HNJv68TkZJo2s1ixeLrHouveyy+7umN+Kx5xxhjEoglfWOMSSCW9E/fM10XiSmLr3ssvu6x+LrnrMdnbfrGGJNArKZvjDEJxJJ+GyIyUkTeEZHNIrJJRO5vp8wlIlIrIuvcx6MxiHO3iGxwX/+US5jF8VMR2e4OeT2rF2ObGPHZrBOROhH5dpsyvfoZisgiETkkIhsj1g0RkTdFZJs7zexg36+5ZbaJyNd6Mb5/EZFS9+/3qogM7mDfTr8LZzG+x0VkX8Tf8OoO9p0nImXud3FBL8b3UkRsu0Wk3dtk9dLn125eicl3UFXtEfEAhgOz3Pl0YCswuU2ZS4A/xjjO3UB2J9uvBv4MCDAHWBmjOL3Apzh9iGP2GQIXA7OAjRHrngYWuPMLgH9uZ78hwE53munOZ/ZSfFcCPnf+n9uLL5rvwlmM73Hge1H8/XcAY4EkYH3b/6ezFV+b7f8PeDSGn1+7eSUW30Gr6behqgdUdY07X48zYmh7N43p664DfqmOFcBgERkegzguA3aoakwvuFPV93AGA4x0HSfu/fALoL2b/HwOeFNVD6tqDc79Iub1Rnyq+oaqBt3FFTjDmsdEB59fNFrvvKeqzUDLnfd6VGfxiYgAtwAv9vTrRquTvNLr30FL+p0QkQKc8f9XtrP5AhFZLyJ/FpEpvRqYQ4E3RGS1iNzdzvZo73h2tt1Gx/9ssf4Mh+qJIcE/BYa2U6avfI5/i/PLrT1dfRfOpnvd5qdFHTRN9IXP7yLgoKpu62B7r35+bfJKr38HLel3QEQGAr8Fvq2qdW02r8FprpgO/DvwWm/HB3xGVWcBVwHfEpGLYxBDp0QkCef+Cr9pZ3Nf+AxbqfM7uk92ZRORR3CGLX+hgyKx+i78JzAOmAEcwGlC6Yvm03ktv9c+v87ySm99By3pt0NE/Dh/mBdU9Xdtt6tqnaoedeeXAn5xbhLTa/TEHckOAa/i/IyO1OUdz3rBVcAaVT3YdkNf+AyBgy1NXu70UDtlYvo5isgdwOeBL7tJ4RRRfBfOClU9qKohVQ0Dz3bwurH+/HzADcBLHZXprc+vg7zS699BS/ptuO1//w1sUdV/7aDMMLccInIezudY3YsxpolIess8zgm/jW2KLQFud3vxzAFqI35G9pYOa1ix/gxdS4CWnhBfA37fTpnXgStFJNNtvrjSXXfWicg84CHgWlVt6KBMNN+FsxVf5DmiL3bwuq133nN/+d2G87n3lsuBUlWtaG9jb31+neSV3v8Ons0z1v3xAXwG5yfWJ8A693E1cA9wj1vmXmATTk+EFcCFvRzjWPe117txPOKuj4xRgIU4PSc2AMW9HGMaThIfFLEuZp8hzsHnABDAaRP9OpAF/BXYBrwFDHHLFgPPRez7t8B293FnL8a3Hactt+V7+HO37AhgaWffhV6K71fud+sTnOQ1vG187vLVOL1VdvRmfO7651u+cxFlY/H5dZRXev07aFfkGmNMArHmHWOMSSCW9I0xJoFY0jfGmARiSd8YYxKIJX1jjEkglvSNMSaBWNI3xpgEYknfGGMSyP8HbBB1FufBvPkAAAAASUVORK5CYII=\n", 129 | "text/plain": [ 130 | "
" 131 | ] 132 | }, 133 | "metadata": {}, 134 | "output_type": "display_data" 135 | } 136 | ], 137 | "source": [ 138 | "from sklearn.model_selection import validation_curve\n", 139 | "from sklearn.preprocessing import PolynomialFeatures\n", 140 | "from sklearn.linear_model import LinearRegression\n", 141 | "from sklearn.pipeline import make_pipeline\n", 142 | "degrees = np.arange(1, 21)\n", 143 | "\n", 144 | "model = make_pipeline(PolynomialFeatures(), LinearRegression())\n", 145 | "\n", 146 | "# Vary the \"degrees\" on the pipeline step \"polynomialfeatures\"\n", 147 | "train_scores, validation_scores = validation_curve(\n", 148 | " model, x[:, np.newaxis], y,\n", 149 | " param_name='polynomialfeatures__degree',\n", 150 | " param_range=degrees)\n", 151 | "\n", 152 | "# Plot the mean train score and validation score across folds\n", 153 | "plt.plot(degrees, validation_scores.mean(axis=1), label='cross-validation') \n", 154 | "\n", 155 | "plt.plot(degrees, train_scores.mean(axis=1), label='training') \n", 156 | "\n", 157 | "plt.legend(loc='best') " 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": {}, 163 | "source": [ 164 | "This figure shows why validation is important. On the left side of the plot, we have very low-degree polynomial, which under-fit the data. This leads to a low explained variance for both the training set and the validation set. On the far right side of the plot, we have a very high degree polynomial, which over-fits the data. This can be seen in the fact that the training explained variance is very high, while on the validation set, it is low. Choosing d around 4 or 5 gets us the best tradeoff. The astute reader will realize that something is amiss here: in the above plot, d = 4 gives the best results. But in the previous plot, we found that d = 6 vastly over-fits the data. What’s going on here? The difference is the number of training points used. In the previous example, there were only eight training points. In this example, we have 100. As a general rule of thumb, the more training points used, the more complicated model can be used. But how can you determine for a given model whether more training points will be helpful? A useful diagnostic for this are learning curves." 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": {}, 170 | "source": [ 171 | "## Learning Curves" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "A learning curve shows the training and validation score as a function of the number of training points. Note that when we train on a subset of the training data, the training score is computed using this subset, not the full training set. This curve gives a quantitative view into how beneficial it will be to add training samples. scikit-learn provides `sklearn.model_selection.learning_curve()`:" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 39, 184 | "metadata": {}, 185 | "outputs": [ 186 | { 187 | "name": "stderr", 188 | "output_type": "stream", 189 | "text": [ 190 | "/home/pbellec/env/nilearn/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n", 191 | " warnings.warn(CV_WARNING, FutureWarning)\n" 192 | ] 193 | }, 194 | { 195 | "data": { 196 | "text/plain": [ 197 | "(0, 0.9)" 198 | ] 199 | }, 200 | "execution_count": 39, 201 | "metadata": {}, 202 | "output_type": "execute_result" 203 | }, 204 | { 205 | "data": { 206 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8HPV9//HXZ1daXb6wLTD4FGAwhhAOYUhIgwvOL4bws5NAWruEAKFxczgkbZrGNClp6JWUtmnakBSHJqQ5MEdC4lBTQ8C0zQUWN7YRCBvbcjBIPiXL0mp3P/1jRtZqrWNlr44dv5+PxzxmZ+a7O9/xyO+d/c7Md8zdERGRaImNdAVERKTwFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncpOmZ2l5n99UjXQ2Q0U7iLDCMzO8vM1ppZs5npJhMZMgp3EcDMSoZpVZ3AvcCNw7Q+OUYp3GXUM7NzzexpM2sxs3uA8qxlV5rZs2a218x+ZWZnZy07z8yeCd93n5nd09WcY2bzzazRzD5nZjuB7+TxeSeZ2Y/MrMnMtpjZTYPdFnevd/d/BzYcxT+JyIAU7jKqmVkC+AnwPWAicB9wVbjsXODbwB8Bk4A7gNVmVha+7wHgrvB9dwPvy/n4KeGymcCyAT4vBvwMeA6YClwGfNrM3h3W5Q/CL4S+hhlD8e8j0hdT3zIympnZO4FVwFQP/1jN7FfAYwQB3Ozuf5FVvh5YBjhBoE/Let8vgMfd/QtmNh94GBjn7u3h8m/283ntwH3uPiNr2c3Aae5+wxFs16nAK+5ug32vSD6Gq51R5EidBOzwnkchW8PxTOA6M/tk1rJE+B7v5X3bcz67qSvY8/i8NHCSme3NWhYH/newGyQyHNQsI6Pd68BUM8s+wu06et4O/I27T8gaKt397j7eNz3ns3N/tvb3eduBLTnLxrr7FQBmdo2ZtfYzqFlGhpXCXUa7XwMp4CYzKzWz9wPzwmXfAj5qZhdaoMrM3mNmY8P3pYHlZlZiZouz3teX/j7vSaAlPAFbYWbx8LLGCwDc/QfuPqafYRtA+LnlBL8IMLNyMysr7D+ZiMJdRjl3TwLvB64HdgO/D/w4XFYHfAT4OrAHaAjLZb/vRmAv8EHgQaCjn3X193lp4ErgHGAL0AzcCYwf5CbNBA7SfbXMQaB+kJ8hMiCdUJVjhpk9Afybu39npOsiMtR05C6RZWaXmNmUsFnmOuBs4L9Gul4iwyGvcDezhWZWb2YNZrail+UzzexRM3vezB43s2mFr6rIoJ1OcF36XuAzwNXu/vrIVklkeAzYLGNmceBl4F1AI7AeWOruG7PK3Ac86O7fNbNLgRvc/dqhq7aIiPQnnyP3eUCDu28OT1KtAhbnlJlLcFMJwLpelouIyDDK5yamqfS8+aMRuDCnzHMEVyZ8jeAW77FmNsndd2UXMrNlBHf7UVVVdf6cOXOOtN4iIsekp556qtndqwcqV6g7VP8U+LqZXQ/8D7CD4BrjHtx9JbASoLa21uvq6gq0ehGRY4OZbR24VH7hvoOed/ZNC+cd4u6/JThyx8zGAFe5e/Zt2iIiMozyaXNfD8w2s5qwp70lwOrsAmY2Oew1D+Bmgp71RERkhAwY7u6eApYDa4FNwL3uvsHMbjWzRWGx+UC9mb0MnAD8zRDVV0RE8jBid6iqzV1EZPDM7Cl3rx2onO5QFRGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkgvIKdzNbaGb1ZtZgZit6WT7DzNaZ2TNm9ryZXVH4qoqISL4GDHcziwO3A5cDc4GlZjY3p9gXCJ7QdC7BY/i+UeiKiohI/vI5cp8HNLj7ZndPAquAxTllHBgXvh4P/LZwVRQRkcEqyaPMVGB71nQjcGFOmb8EHjazTwJVwIKC1E5ERI5IoU6oLgXucvdpwBXA98zssM82s2VmVmdmdU1NTQVatYiI5Mon3HcA07Omp4Xzst0I3Avg7r8GyoHJuR/k7ivdvdbda6urq4+sxiIiMqB8wn09MNvMaswsQXDCdHVOmW3AZQBmdgZBuOvQXERkhAwY7u6eApYDa4FNBFfFbDCzW81sUVjsM8BHzOw54G7genf3oaq0iIj0L58Tqrj7GmBNzrxbsl5vBC4ubNVERORI6Q5VEZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBeYW7mS00s3ozazCzFb0s/6qZPRsOL5vZ3sJXVURE8jXgk5jMLA7cDrwLaATWm9nq8OlLALj7H2eV/yRw7hDUVURE8pTPkfs8oMHdN7t7ElgFLO6n/FKC56iKiMgIySfcpwLbs6Ybw3mHMbOZQA3wWB/Ll5lZnZnVNTU1DbauIiKSp0KfUF0C3O/u6d4WuvtKd69199rq6uoCr1pERLrkE+47gOlZ09PCeb1ZgppkRERGXD7hvh6YbWY1ZpYgCPDVuYXMbA5wHPDrwlZRREQGa8Bwd/cUsBxYC2wC7nX3DWZ2q5ktyiq6BFjl7j40VRURkXwNeCkkgLuvAdbkzLslZ/ovC1ctERE5GrpDVUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCMor3M1soZnVm1mDma3oo8zvmdlGM9tgZj8sbDVFRGQwBnxYh5nFgduBdwGNwHozW+3uG7PKzAZuBi529z1mdvxQVVhERAaWz5H7PKDB3Te7exJYBSzOKfMR4HZ33wPg7m8WtpoiIjIY+YT7VGB71nRjOC/bacBpZvZLM/uNmS3s7YPMbJmZ1ZlZXVNT05HVWEREBlSoE6olwGxgPrAU+JaZTcgt5O4r3b3W3Wurq6sLtGoREcmVT7jvAKZnTU8L52VrBFa7e6e7bwFeJgh7EREZAfmE+3pgtpnVmFkCWAKszinzE4KjdsxsMkEzzeYC1lNERAZhwHB39xSwHFgLbALudfcNZnarmS0Ki60FdpnZRmAd8Fl33zVUlRYRkf6Zu4/Iimtra72urm5E1i0iUqzM7Cl3rx2onO5QFRGJIIW7iEgEDXiHqogcoVQSdm+G5pchk4LSSiit6B4nKnvOi5eOdI1lqGUykGwN9nVpxZCuSuFepNydjlSGZDpDZzhOpjJ0pjPB/FSGzrSTTGVIptMkU04ynaGiNE712DKqx5YxeUyCspL4SG9K8etsh10N0PQSNNV3j3e/GoR6vmKlWWEfBn4i5wuhtKrn8tIKSPQyL1YCFgsGrPu19fU61nM+/Szr7XVX+XhpuG4bon/sYeYOnW3Q0RqEckdLOO5juseyVki29JzuPBB87pX/DLU3DGnVFe6jzIGOFBtf38+u1iR72pLsPpBkV2uS3Qc62HUgmO4aOlKZo17f+IrSIOzHBIF/fBj8PYYxZRxXmSAWO8L/sDtfhFcfgxPPhmnzgsAqRsk22PVKzwB/cxPs2QIe7guLwcSToXoOnHFlMJ58GpSUBSHReTAc2oLPy53X2cu8ZBsc3Ht4uVT7yP579MEx0rEyUrEE6ViCVKyMtCXojCVIheNOgtdJK6XTEiQJXidJkKSUDkrpIBGMvZQOSmj30mCglPZMKQe9JBgypbRlSjmYKaEtU4p5mkraw+EgVeFQSXs4DoYqbw9fd8+v4iAV3vW+dipoJ05+/8/aSdBGBQeo4KAF4zYrp40pHKSCNqvgQLyCNsqZ2zqdy4Z4PyjcR4lMxvnxMzv48kObaG5N9lhWmYgzsSrBpKoEx48tY86UcUwak2B8RSmJeIxESTjEY5SG47JwXmm8e1mixGhLpmlq6egeWrtfP7t9L2+2tNPeefgfczxmVI8p44Tx5UwZV8aUceXh6/Ier6vKsv6kdr0Kj/8dvHA/EF6VFSuFqefBzLfDzHfAjAuhbOyA/z7JVIaDyTRtnSnakmkOJtN0pNJ0ZP9CCX+ldKacjvCXTNevmWTWr5tkH8vcIRYzqmhnamobUzu3MjW1jROTWzkxuZWJna8TC7cjTZxd5dNpLp9F85RL2F1Zw+6qk9lfOQOPlxOPWTDsMeJ7gy/FjCfI+AQcxz3Y5xmHjDvujgOZmJNJQKa0u4zTVSYYBwOQyRDPtFOSDofMQUrT7ZRk2smk06Qz6WAcvk6nvXt+Jk0mnSGTyZDKpPFMhkw4juHEcMwcC1/HCOYbvc+LkcGAUlIkrJMysgbrpIwkZXRSbp2UWyvldJKwFBMOlUsSRH4npQzi185R6LQEyVglHfEKOmKVdMQqScYm0RqvZFc43RHvmh+Ua7fueT2HCtyCX8FOzysQe7sg8fzpuT24FJ4uhRwFXtyxj1t++iJPb9vLuTMm8PH5p3Li+HKOCwO9vHT4mk7cnQO5XwAt7TS1dvDG/g7e2N/Ozn3t7NzfTkv74f8Jx5aVcObYVpb5/VxyYC3pWCn1M6/hjdP/gPLdrzCh6Qmqd9UxuWUjcU+TJs6O8tm8VHY2LybewnM2h+ZURRDkyTQHkikOJtOkMkf3dxozsr7kYiRixpT4Pk6Ovc5M/y0zfQdT041MS23jhEx3v3dJStgem8prNp0tNo3NTKOBaWzNnEAHJaQzfmjIuJPKeK//mfOtY8yMmBkWvs4dxwwsZ9z1HoBYLJgujYdf7PGs14e+7C1c1n0wUBq3Q8u7lnVPW/fBQ/ie0rhRllW+NDygiMeMkrhREgtel8aDL7mSWOxQnQeUSUOqI/hlkvc46zUEBwyJMVA2Jhz3Ml2k5zjyvRRS4T6C9hxI8g8P1/PDJ7cxqSrB5xbO4arzph1588cwa0umDgX9G/vb2du0k9MbvsUFTT8Cd34Sexf/0L6INzLjD3tvBe1cVPoqby95iQvYxFx/mQQpMhjbE6fwauU5bB93DjsnnA+VE6lKxKlIlFCZiFOZiFNRGqesNH4oZBLxeBhGduiXTFk8TiJzkNJ9WyjZ0wDNDUGzSvMrwa+KZEt3hUoqYNKpcPwcqD4dqs8ImlSOmwXxwf3AdQ8D37uD/1Ag0zOou4I7r9ATQeE+qqUzzqr127htbT0t7Sk+9LaZfHrBaYyvKM4jCdr3wa9vD4bONnjrUrjkc3DcTNIZZ1drB3vaOikvjVEZBnRFabznl1jnQWisg62/hNd+AY3ru4/Cjp8LMy+GWRcH4zE5jwvIpGHf9u7w3tUQBngD7M/uBslg/HSYfCpMmh2EedfrcVODw16RUU7hPko9tXUPX1z9Ii/u2M+FNRO5dfFZnD5l4DbnUSnZBuu/Bb/4KhzcA3MXw+9+PjjyPVqpDtjxNGz9BWz9FWx7ovtKg8mnwfR5wZfKrleDId3R/d6y8d2hnR3kk04Z8svPRIZavuFefCdUW3bC5sfhrUtGuiZ5y2Sc7Xva+JdHG/jR041MGVfOvy49lyvPPrE4f46nkvDMf8B/3watO+HUBXDpF+Ckcwu3jpIymPm2YABId8LrzwVH9Vt/CfUPQcVEmDwbTr0sDPLZwbhqcnQuxRM5QsUX7k9/D9b9NUy7IDgSG0X2tiXZ3HyALU0H2NJ8gM3NrWxuOsBruw7Q3pmhNG58bP4pLP/dU3teVVIsMml44T5Y97ewdytMvwiu/nbQXDLU4qUwrTYY3vHpoV+fSJErvoQ579rg8rqn7oL/91dDvrofPLGVf320IThBVxKjvDROWUmMstIY5SXBSbw3WzrY0nyA3Qe6L2GMx4wZEyupmVzFO06dTE11MJ45qWrI61wwnQeDOyx3vRq0Xz9/T3B995S3wDX3B0fsOkIWGZWKL9zHToE5V8Az3w+aAkrKhmxVB5Np/vHhl5lYleCsk8bRkcrQ3hlcW30wmWZvWycdqQyTqhK8+8wTOHnyGGomV1FTXcWMiZWUxovgBF26E/ZsDe6m3NXQHeS7NwcnKbNVz4GrvwNz36uTjyKjXPGFO0Dth2HTz2Djajj7A0O2mlXrt7H7QJI7rj2fC2ZNHLL1HLG23cGJx0xnENKZVDjuhHQqGB+aF45bXg+PxhuCYc9W8HT3Z5aPD04+zngbTLo2aPqadApMPAXKx43ctorIoOQV7uEDr78GxIE73f3LOcuvB26j+/F7X3f3OwtYz55q5sNxNfDUd4Ys3JOpDCv/ZzPzZk0cfcHevh9+8jF46cEje39pZRDWU86GM98XhPnEU4Jx5UQ1tYhEwIDhbmZx4HbgXQTPSl1vZqvdfWNO0XvcffkQ1PFwsRicfz38/Ivw5kvBjScF9sAzjby+r52/e/9bCv7ZR2X3Zrh7aXAd9zv+GCbMCG7p7+qwqWscKw1uvjm0LJyunAzjTlKAi0RcPkfu84AGd98MYGargMVAbrgPr3M/CI/9dXBi9fIvD1h8MNIZ55uPv8pZU8dxyWnVBf3so/LqY3DfDUEwX/tjOHn+SNdIREapfM6KTQWyz6w1hvNyXWVmz5vZ/WY2vbcPMrNlZlZnZnVNTU1HUN0sVZNh7iJ47ofBzTRHau92+P5VwU0yoTUvvM5ru9r4xPxTR8d16O7B3Z/fvyo46v7IOgW7iPSrUJc8/AyY5e5nA48A3+2tkLuvdPdad6+tri7AEXHth4O7FDc8cGTvT3fC/R+Ghp/DPR+E/b/F3bl9XQOnVFfx7jOnHH0dj1Zne9C+vvbP4fQr4MZHYGLNSNdKREa5fMJ9B5B9JD6N7hOnALj7Lnfvuv/7TuD8wlRvADMvDm5Ff+o7R/b+dX8DjU/C/D8P+kS551rWbWjkpZ0tfGz+qSPfgdf+1+GuK+C5u2H+zfB73wt6tRMRGUA+4b4emG1mNWaWAJYAq7MLmNmJWZOLgE2Fq2I/zOD8G4JOpna+MLj3Njwa9Ily3odg/ufgvd+AHXV0PvhZpk6oYPE5Jw1NnfO1fT2snB+cMP7978P8Fbq2XETyNmBauHsKWA6sJQjte919g5ndamaLwmI3mdkGM3sOuAm4fqgqfJi3LoGScqgbxNF7yxvwwB8F3bou/Eowb+5idpz5Ud7d/hD/ePLTI3sD0jM/CI7YS8rgDx+BM/7/yNVFRIpSXte5u/saYE3OvFuyXt8M3FzYquWpciKc+f7g1vhLvxBM9yeTgQeWBc8zvO5nPR75tmLvYj5uv+ail74M2y+B6RcMbd3dg5uKmuqDSxub6+GNjbDtV1DzTvjAdwfeHhGRXhTnHaq53r4cXrgX1vxp0JFVf37xT0Gvkov+FY4/49DsZ7fv5X9f3cOlC/6Zt714Pdx7LSz7bxh7QmHq2NoE258IArz5le5Az35gRNm44BzCJZ+Dd/7ZoB8SISLSJRrpccKZcMmKoLfIOe+Bs67qXtZ5MGiTf+2XQVexW38JZ10N517b4yO+sa6BceUlfOB3zoazfgB3vgvuuw4+tBpKEkdeN/fghOiaP+sO8rEnQfVpcM7SIMwnnxb0gT7mBN1cJCIFEY1wh+BuzZcfgv/8DMQT8NtngkDf8VTQx4rFgt4M3/5JeOdne4Ro/c4WHt74BjddNpsxZSVBucVfhx/dCA9/Hq647cjq1LYbHvw0bPxpcGXPgr8Mfi3k8UBoEZGjEZ1wj5fA++6Af/ud4Jp1i8NJ58BFH4NZ74AZFwWdYvXim483UJmIc8PbZ3XPfMvVwRfEr78OJ741uCN2MBoehZ98HNp2wYIvBV8qseF70LWIHNuiE+4Ak2eTvu5BUgd2k5h1EdZHL4YdqTTPN+7jN6/u4oktu/nVq818+OIajqvKaX5Z8KXg6T8//URwo9Nlt8DEk/uvQ+dBeOSL8OQdQRe519wbfDmIiAyjaIU7cPXPOnhmWydm/0tlaZzKsuCBzF0PZgZ4ccc+OlIZAOZMGcsNF9dw04LZh39YvASWroJffi04gt/0IFxwY9CsUzX58PK/fRZ+vCw4aXrhx2DBF/XMThEZEZEK9627DvDMtr1c8ZYpnFo9hgPJNG3JNG3JFAc60hzsTNGZdq65cCYXnjyRebMmHn60nqtsDFz6+aCrg8f/Dp5cCc/+MHjU24UfCy6lzKSDL4B1fxuE/rUPwCmXDs9Gi4j0IlLh/vNNbwKwYuEZzJhUOUDpQRp3Iiz6F7jo4/Dol+DRW+HJO4MTuRseCK5Nn/teuPKrujZdREZcpML9kY07Of2EsYUP9mzHz4GldwdX4jzyF/DQZ4Pr0993B5z9+7qUUURGhciE+962JOtf28NHLxnghGehzLoY/vDRoI/16tNh/LThWa+ISB4iE+6P1zeRzjgLzijQHaX5MINTLxu+9YmI5Cky3Qw+sukNJo8p463TJox0VURERlwkwj2ZyvDf9U0sOOP4ke+DXURkFIhEuD+5ZTetHanhbZIRERnFIhHu9W8EHXKdN/O4Ea6JiMjokFe4m9lCM6s3swYzW9FPuavMzM2stnBVHFhzawclMWNCRelwrlZEZNQaMNzNLA7cDlwOzAWWmtncXsqNBT4FPFHoSg6kuaWDSWMSam8XEQnlc+Q+D2hw983ungRWAYt7KfdXwFeA9gLWLy/NrR1MHlM23KsVERm18gn3qcD2rOnGcN4hZnYeMN3d/7O/DzKzZWZWZ2Z1TU1Ng65sX5pbkwp3EZEsR31C1cxiwD8BnxmorLuvdPdad6+trq4+2lUfoiN3EZGe8gn3HcD0rOlp4bwuY4GzgMfN7DXgImD1cJ1UdXd2tSaZPPYoHoUnIhIx+YT7emC2mdWYWQJYAqzuWuju+9x9srvPcvdZwG+ARe5eNyQ1zrH/YIpkOkO1jtxFRA4ZMNzdPQUsB9YCm4B73X2Dmd1qZouGuoIDaWrtAKB6rMJdRKRLXh2HufsaYE3OvFv6KDv/6KuVv+Yw3NXmLiLSrejvUFW4i4gcrvjDvaUr3HVCVUSkS/GHe2uSeMw4rlLhLiLSJQLh3sHEKnU9ICKSLRLhrvZ2EZGeij7cm1qTam8XEclR9OHe3NKhG5hERHIUdbi7e9AsoxuYRER6KOpwb+1I0ZHKqFlGRCRHUYd7c2sS0A1MIiK5ijzcdXeqiEhvijvcWxTuIiK9Kepw7+oRUn25i4j0VNTh3tzSgRlMVNcDIiI9FHW4N7UmmViZoCRe1JshIlJweaWimS00s3ozazCzFb0s/6iZvWBmz5rZL8xsbuGrejh1PSAi0rsBw93M4sDtwOXAXGBpL+H9Q3d/i7ufA/w9wQOzh1xwA5OaZEREcuVz5D4PaHD3ze6eBFYBi7MLuPv+rMkqwAtXxb7pyF1EpHf5PGZvKrA9a7oRuDC3kJl9AvgTIAFc2tsHmdkyYBnAjBkzBlvXwzS3JBXuIiK9KNiZSHe/3d1PAT4HfKGPMivdvdbda6urq49qfQc6UhzsTCvcRUR6kU+47wCmZ01PC+f1ZRXw3qOpVD66705Vm7uISK58wn09MNvMaswsASwBVmcXMLPZWZPvAV4pXBV7dyjc1SOkiMhhBmxzd/eUmS0H1gJx4NvuvsHMbgXq3H01sNzMFgCdwB7guqGsNEBTS9BpmPpyFxE5XD4nVHH3NcCanHm3ZL3+VIHrNSB1GiYi0reivbWzK9wnqc1dROQwRR3uEypLKVXXAyIihynaZNQ17iIifSvecG/t0GWQIiJ9KPJw15G7iEhvijjc1SwjItKXogz39s40rR0pqnUDk4hIr4oy3JvCZ6fqBiYRkd4VZbg369mpIiL9KtJwD7oeUJu7iEjvijTc1fWAiEh/ijPcW9T1gIhIf4oz3Fs7GFdeQllJfKSrIiIyKhVpuCfVj7uISD+KMtybdHeqiEi/8gp3M1toZvVm1mBmK3pZ/idmttHMnjezR81sZuGr2q25tUPXuIuI9GPAcDezOHA7cDkwF1hqZnNzij0D1Lr72cD9wN8XuqLZmlvUaZiISH/yOXKfBzS4+2Z3TxI8AHtxdgF3X+fubeHkbwgeoj0kOlJp9ren1CwjItKPfMJ9KrA9a7oxnNeXG4GHeltgZsvMrM7M6pqamvKvZZZdXTcw6YSqiEifCnpC1cw+CNQCt/W23N1Xunutu9dWV1cf0Tp0A5OIyMDyeUD2DmB61vS0cF4PZrYA+Dxwibt3FKZ6h+sOd7W5i4j0JZ8j9/XAbDOrMbMEsARYnV3AzM4F7gAWufubha9mt+YW9SsjIjKQAcPd3VPAcmAtsAm41903mNmtZrYoLHYbMAa4z8yeNbPVfXzcUWsKj9zVl7uISN/M3UdkxbW1tV5XVzfo97UlU+zc187J1WOGoFYiIqObmT3l7rUDlSu6O1QrEyUKdhGRARRduIuIyMAU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBeYW7mS00s3ozazCzFb0sf6eZPW1mKTO7uvDVFBGRwRgw3M0sDtwOXA7MBZaa2dycYtuA64EfFrqCIiIyePk8IHse0ODumwHMbBWwGNjYVcDdXwuXZYagjiIiMkj5NMtMBbZnTTeG80REZJQa1hOqZrbMzOrMrK6pqWk4Vy0ickzJJ9x3ANOzpqeF8wbN3Ve6e62711ZXVx/JR4iISB7yCff1wGwzqzGzBLAEWD201RIRkaMxYLi7ewpYDqwFNgH3uvsGM7vVzBYBmNkFZtYIfAC4w8w2DGWlRUSkf/lcLYO7rwHW5My7Jev1eoLmGhERGQV0h6qISAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiaC8wt3MFppZvZk1mNmKXpaXmdk94fInzGxWoSsqIiL5GzDczSwO3A5cDswFlprZ3JxiNwJ73P1U4KvAVwpdURERyV8+R+7zgAZ33+zuSWAVsDinzGLgu+Hr+4HLzMwKV00RERmMfJ6hOhXYnjXdCFzYVxl3T5nZPmAS0JxdyMyWAcvCyVYzqz+SShfQZHLqWMS0LaN4Vm6bAAAEgElEQVRPVLYDtC2jycx8CuX1gOxCcfeVwMrhXGd/zKzO3WtHuh6FoG0ZfaKyHaBtKUb5NMvsAKZnTU8L5/VaxsxKgPHArkJUUEREBi+fcF8PzDazGjNLAEuA1TllVgPXha+vBh5zdy9cNUVEZDAGbJYJ29CXA2uBOPBtd99gZrcCde6+Gvh34Htm1gDsJvgCKAajpomoALQto09UtgO0LUXHdIAtIhI9ukNVRCSCFO4iIhF0TIS7mU03s3VmttHMNpjZp8L5E83sETN7JRwfN9J1zZeZxc3sGTN7MJyuCbt+aAi7gkiMdB3zYWYTzOx+M3vJzDaZ2duKdb+Y2R+Hf18vmtndZlZeLPvFzL5tZm+a2YtZ83rdDxb4l3Cbnjez80au5j31sR23hX9fz5vZA2Y2IWvZzeF21JvZu0em1kPjmAh3IAV8xt3nAhcBnwi7UFgBPOrus4FHw+li8SlgU9b0V4Cvhl1A7CHoEqIYfA34L3efA7yVYJuKbr+Y2VTgJqDW3c8iuPhgCcWzX+4CFubM62s/XA7MDodlwDeHqY75uIvDt+MR4Cx3Pxt4GbgZIMyAJcCZ4Xu+EXa3EgnHRLi7++vu/nT4uoUgQKbSs9uE7wLvHZkaDo6ZTQPeA9wZThtwKUHXD1Ak22Jm44F3Elxthbsn3X0vRbpfCK4+qwjv9agEXqdI9ou7/w/BlW7Z+toPi4H/8MBvgAlmduLw1LR/vW2Huz/s7qlw8jcE9+pAsB2r3L3D3bcADQTdrUTCMRHu2cIeK88FngBOcPfXw0U7gRNGqFqD9c/AnwGZcHoSsDfrD7iR4MtrtKsBmoDvhE1Md5pZFUW4X9x9B/APwDaCUN8HPEVx7pcufe2H3rokKZbt+jDwUPi6mLdjQMdUuJvZGOBHwKfdfX/2svCmq1F/XaiZXQm86e5PjXRdCqAEOA/4prufCxwgpwmmiPbLcQRHgjXASUAVhzcPFK1i2Q/9MbPPEzTR/mCk6zIcjplwN7NSgmD/gbv/OJz9RtfPyXD85kjVbxAuBhaZ2WsEPXReStBuPSFsDoDeu4gYjRqBRnd/Ipy+nyDsi3G/LAC2uHuTu3cCPybYV8W4X7r0tR/y6ZJkVDGz64ErgWuy7p4vuu0YjGMi3MM26X8HNrn7P2Utyu424Trgp8Ndt8Fy95vdfZq7zyI4GfSYu18DrCPo+gGKZ1t2AtvN7PRw1mXARopwvxA0x1xkZpXh31vXthTdfsnS135YDXwovGrmImBfVvPNqGNmCwmaMRe5e1vWotXAEgseNlRDcIL4yZGo45Bw98gPwDsIflI+DzwbDlcQtFU/CrwC/ByYONJ1HeR2zQceDF+fTPCH2QDcB5SNdP3y3IZzgLpw3/wEOK5Y9wvwJeAl4EXge0BZsewX4G6CcwWdBL+obuxrPwBG8ACfV4EXCK4QGvFt6Gc7Ggja1rv+7/9bVvnPh9tRD1w+0vUv5KDuB0REIuiYaJYRETnWKNxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhH0fw7dxkIA+whxAAAAAElFTkSuQmCC\n", 207 | "text/plain": [ 208 | "
" 209 | ] 210 | }, 211 | "metadata": {}, 212 | "output_type": "display_data" 213 | } 214 | ], 215 | "source": [ 216 | "from sklearn.model_selection import learning_curve\n", 217 | "model.set_params(polynomialfeatures__degree=1)\n", 218 | "train_sizes, train_scores, validation_scores = learning_curve(\n", 219 | " model, x[:, np.newaxis], y, train_sizes=np.logspace(-1, 0, 20))\n", 220 | "\n", 221 | "# Plot the mean train score and validation score across folds\n", 222 | "plt.plot(train_sizes, validation_scores.mean(axis=1), label='cross-validation') \n", 223 | "plt.plot(train_sizes, train_scores.mean(axis=1), label='training') \n", 224 | "plt.title('degree=1')\n", 225 | "plt.ylim(ymin=0,ymax=0.9)" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "\n", 233 | "Note that the validation score generally increases with a growing training set, while the training score generally decreases with a growing training set. As the training size increases, they will converge to a single value.\n", 234 | "\n", 235 | "From the above discussion, we know that d = 1 is a high-bias estimator which under-fits the data. This is indicated by the fact that both the training and validation scores are low. When confronted with this type of learning curve, we can expect that adding more training data will not help: both lines converge to a relatively low score.\n", 236 | "\n", 237 | "**When the learning curves have converged to a low score, we have a high bias model.**\n", 238 | "\n", 239 | "A high-bias model can be improved by:\n", 240 | "\n", 241 | "Using a more sophisticated model (i.e. in this case, increase d)\n", 242 | "Gather more features for each sample.\n", 243 | "Decrease regularization in a regularized model.\n", 244 | "Increasing the number of samples, however, does not improve a high-bias model.\n", 245 | "\n", 246 | "Now let’s look at a high-variance (i.e. over-fit) model:" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 42, 252 | "metadata": {}, 253 | "outputs": [ 254 | { 255 | "name": "stderr", 256 | "output_type": "stream", 257 | "text": [ 258 | "/home/pbellec/env/nilearn/lib/python3.5/site-packages/sklearn/model_selection/_split.py:2053: FutureWarning: You should specify a value for 'cv' instead of relying on the default value. The default value will change from 3 to 5 in version 0.22.\n", 259 | " warnings.warn(CV_WARNING, FutureWarning)\n" 260 | ] 261 | }, 262 | { 263 | "data": { 264 | "text/plain": [ 265 | "(0, 0.9)" 266 | ] 267 | }, 268 | "execution_count": 42, 269 | "metadata": {}, 270 | "output_type": "execute_result" 271 | }, 272 | { 273 | "data": { 274 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucVWd97/HPd2YYhhkugWFCuEMMJhCMScQk1luaoCUxglZridqaNkp9vcRLa88xeenJ6Umv1tOLPU1tMaaxVsUkrYopNuZaWy8RUhUDSILkAoTAcIcZYC77d/5Ya5g9k4HZA3uuz/f9eu3X3mutZ+/9LNbwXWs/az3PUkRgZmYjS8VgV8DMzMrP4W5mNgI53M3MRiCHu5nZCORwNzMbgRzuZmYjkMPdhjxJd0v6o8Guh9lw4nA360eSFkp6QNJeSS/pVCLpMUnHJR3NH1sGo5428jjcLUmSqgboq1qBe4CbT1NmZUSMzR8XDlC9bIRzuNuQI+kySf8t6YikrwE1RctukPQTSQclfV/SJUXLLpf04/x990r6WkdzjqSrJe2Q9AlJLwL/WMLnTZP0L5IaJT0j6SN9XZeI2BIRXwA2nsU/iVmfOdxtSJFUDXwD+BIwCbgXeEe+7DLgLuB3gHrgH4A1kkbn7/s6cHf+vq8Cb+/28efly2YDK3r5vArgW8BPgenAtcDHJP1KXpd35zuEUz1m9WG1/zRvtvmepKv78D6zU5LHlrGhRNIbgNXA9Mj/OCV9H3iELID3RsT/Kiq/BVgBBFmgzyh6338Bj0XEp/LQ/A4wPiKO58s/d5rPOw7cGxGzipbdCrw8In7rDNbrAuDpiFC3+VcCm4AWYDnwt8ClEfGLvn6HWbGBanc0K9U0YGd0Pep4Ln+eDbxP0oeLllXn74ke3re922c3dgR7CZ/XDkyTdLBoWSXwn31dodOJiMeLJr8o6UbgeuD/lfN7LD1ulrGhZhcwXVLxEW7H0fN24I8j4pyiR21EfPUU75vZ7bO7/0w93edtB57ptmxcRFwPIOk9RVe49PToS7NM9zqq11JmvXC421DzA6AN+IikUZJ+FbgiX/Z54IOSrlSmTtJbJI3L39cOrJRUJWlZ0ftO5XSf9yPgSH4CdoykyvyyxlcDRMSXi65w6enxPED+uTVkvwiQVCNpdP76HEm/ks+rkvQe4A3Av5fvn9NS5XC3ISUiWoBfBW4C9gO/Dvxrvmw98AGydukDwNa8XPH7bgYOAu8F7gdOnOa7Tvd57cANwKXAM8Be4E5gQh9XaTZwjM6rZY4BHdeyjwL+CGjMP//DwNsi4qk+fofZS/iEqo1Ykh4H/j4i/nGw62I20HzkbiOGpDdKOi9v4ngfcAlu4rBElRTukpZI2iJpq6Rbelg+W9LDkjYo6049o/xVNevVhWTXpR8EPg68MyJ2DW6VzAZHr80ykiqBp4A3ATuAdcCNEbGpqMy9wP0R8UVJ1wC/FRG/0X/VNjOz0ynlyP0KYGtEbMtPWq0GlnUrs4CskwnAoz0sNzOzAVRKJ6bpdO0MsgO4sluZn5JdqfBZsi7f4yTVR8S+4kKSVpD1/qOuru5VF1100ZnWe+A174ODz0NdA0xwq5OZDY4nnnhib0Q09FauXD1Ufx/4W0k3Ad8FdpJdc9xFRKwCVgEsWrQo1q9fX6avHyAPfBJ+8Ldw/e/AFR8Y7NqYWYIkPdd7qdLCfSdde/rNyOedFBEvkB25I2ks8I6IKO62PTK86XbYtxW+/QmYdD5ccO1g18jMrEelhPs6YJ6kuWShvhx4d3EBSZOB/RFRAG4lG2lv5KmohHfcCXctgXtvgvc/BA0efrtHTfvgxQ3ZY9cGaNoDL7sG5i+F+pcNdu3MRrySOjFJuh74a7KBk+6KiD+WdDuwPiLWSHon8Kdk42J8F/hQRJyyZyAM02aZDge3w+evgepaeP8jUFdfns+NgK9/ELY9Che9BS5+O8x+bbZTORttJ2DXT2H7j6C9BcZPyx7jpsH4qVBdd3Z1Pvh8HuQ/y4L8xQ1wuOjH3fgZUDMB9uSdNKe8AhYshQXLvHM06yNJT0TEol7LDVYP1WEd7gDb18Hdb4Hpr4Lf/AZUjT77z3zii/Ctj8CMV8PujdDanJ3Anb8ULn5b6UHftBe2P549nn8cXvgxtJ9mX1szIQ/6POzHT4dxU7vuBGonQaEd9j5VFOQ/zZ6P5y1wqoD6eTD1EjjvEjjvFdlzx87vwHOw+VuweU1WN4DJF2Yhv2ApTFkI8phZZqfjcB8IP7sP/uVmuPQ9sOyOswumxi2w6uos2H/jG9B2DJ5+EDZ+HZ7+Th7058L8t3YN+kIB9m7pDPLtj8P+fCjwymqYeinMvAJmXQUzroDRY+HwLjjyAhzOH0d2dX19dDdEoWv9KvOdV8dOoqoGplzcGeJTXwnnLsh+zZTi8Auw+f4s6J/7XvZ9E+d2Bv20yx30Zj1wuA+UR/8U/uPPYPH/gdd97Mw+o/U43Lk4C9wPfi87ei7W0pQF/MZvdA36KQuyo/Ljh7JytZNh5pUw68rseeqlMKrmpd/Xm/a2LOCP7MqaVw7nz0RnmNfPg8oyXWx1tBF+ngf9M9+FQhtMmJXtyBYsy3Z4FR4pwwwc7gMnAu777ewI+9f/Gebf0PfP+PYt8Pjn4MavwYVLTl+2OOj3/yI7wp11VRbmk84f/ke7zfthy7ezoP/FI9k5gnFT4aIbsqCf/Utnfw7CbBhzuA+k1mNZ+/uezfDb/541UZTqqQfgK++CKz8I1326/+o4HB0/nP37bP4mPP1Q1lRVOzk72Tx/KUy+AMZMgtHjhv9OzaxEDveBdmR3dgVNFOADj7y0aaXH97wIn/ul7ITl+x86syaUVLQ0ZecgNn0z++XScrRzWUUVjJmYBX3tpOx5zESonVj0etJLX5fj37vQnv26aG/JmrNOvm6FQmvn64oqGFULo8Z0PqrGuLnJ+szhPhhefBK+8GZoeDnctPb0JxcLBfjnt2cnQX/nP3xJYF+0Hs9Owh7ZBccOZE05x/bnzweK5h3IjvZPZVRt0U5hIlSMysK40C2kTz56CO7uJ577qnJ0Hva12c6mYwdQVfPSncGo2nx+T+WLyowak/3tVY/NLnOtqvEvmxGk1HD3DbLL6byF8M4vwFdvhG98EN5596mPzL7/N7DtMXjrZx3sfTWqpvTewa3HikJ/fw+vD3a+LjRD5ajsUV2XXW1UUZU9V1bny6o7yxTPrxjVrUx1dsK5sjpbVmjN6nLy0Qxtx7PnLvOPZTuk1mNw9MVsR9ZRvmNZX3coqugM+o7HqKLX3Zd1me7pdW32/nKdULd+4a1TbhdeB2/+Q/jOp+CxP4FrPvXSMjufgEf+MGs3vvx9A1/HlIwaAxOmZ4+RICL7xdDanAd/R+gX7yiaoaU5a8pqOZo/N0FrU+frliZo3gsHn+ssd+IoxEuGhDq1qpqXBn/l6Jf+SugyrW7z1EO57mVKnZc/V1Se5pfNmFP/MnpJmeHdbOZw7w+vWZl19vnuZ7JLBl/5653LThyB+26GsefB0r/xz2XrGynrMFc1GsaU+bM7dhxddgrNXXcQvb1ub8k+52Rzb3T9/OJ5XZqEu8870/eRNZu1Hev81dN2LKvXmTjZbNZT01j3nULHDqWHHcXJ9+VlJszImgT7kcO9P0hw/V/A/mdgzUqYOCe79hzg334/O1q6aW2/b1yzPinecdROGuzalFeh/RRNYj39+jl+6maz4nlH93RtRms9nv06KqXZ7C1/Aa9+f7+ussO9v1RVw7v+KeuctPrd2RU0z/8QNqyGq2+F2a8Z7BqapaOiMuudPXps/35PRNEvh1OcS2k9lp2f62cO9/5UOwnefQ/ceQ18+Z1ZT89Zr4HX//5g18zM+oOUHdhVVWdjNg2i4Xu2YLiYfEF2BL9/W3Zy5lc/76sMzKzfOWUGwvlXZ4OBjR4H58zsrbSZ2VlzuA+Uua8f7BqYWUJKapaRtETSFklbJd3Sw/JZkh6V9GNJG/Kbe5iZ2SDpNdwlVQJ3ANcBC4AbJS3oVuxTwD0RcRnZbfj+rtwVNTOz0pVy5H4FsDUitkVEC7AaWNatTADj89cTgBfKV0UzM+urUsJ9OrC9aHpHPq/YHwDvlbQDWAt8uCy1MzMbQSKC5/c1c6DpDHvM9kG5TqjeCNwdEX8h6TXAlyQtjOjaVUvSCmAFwKxZs8r01WZmQ097IXhmbxMbXzjEkzsP8eTOwzz5wiGOHG/jT97+Ct59Zf9mYCnhvhMovn5vRj6v2M3AEoCI+IGkGmAysKe4UESsAlZBNuTvGdbZzGxIaWsv8PSeozy58xAbXzjMkzsPsWnXYZpbsoHYRldVMH/qeJZdOo2F0ybw2gsm93udSgn3dcA8SXPJQn058O5uZZ4HrgXuljQfqAEay1lRM7Oh4ERbO0+9eJQnO47IXzjMz3cd5kRb1lBRW13JxdPG865FM1k4fQILp4/ngoaxVFUObJ/RXsM9ItokrQQeACqBuyJio6TbgfURsQb4OPB5Sb9LdnL1phisu4CYmZVJc0sbm3cd6dK08tTuI7QVsngbX1PFwukTeN8vzeHiaeNZOH0Cc+rrqKwY/NFeS2pzj4i1ZCdKi+fdVvR6E/Da8lbNzGzgHD7eyqa8SaWjaeUXjUfJc5z6umounj6BX76ogYXTJrBw+gRmTByDhuiw3e6hambJOdDUkjerZCc5N+48xLP7mk8uP298DQunj+f6V0w92bRy3viaIRvkPXG4m9mItufIcTbuPJy3j2eBvvNg5711Z04aw8JpE/i1RTO5eNp4Lp42gYZxowexxuXhcDezYa+9EBxsbmFfU0t2+WF+ovPJnYfYc+TEyXLnT67j8tkT+c3XzGbh9AlcPG0859RWD2LN+4/D3cyGnIjg8PE29je1sO/oCfYebWFf0wn2Hc2m9zW1ZK/zefubW7rcaa9CMO/ccbxu3uST7ePzp45jXM2owVupAeZwN7MBcaylvTOgm/LAzsN6f1MLe/Mg71je2t7zBXfja6qoHzua+rpq5k6uY9GcSUyuq6Z+7Ggm1VUzfeIY5p83njHVlQO8hkOLw91smDrW0s5z+5t4dm8zz+1r4tl92fP2A820tweVlaKqooLKClFVIaoqRWVFBVUVOjmvc1n3+fl0pRjVMV3Z+/tOtBXY19TC/uIAzwO9o0NPdzWjKqivG83ksdVMGV/DgqnjqR+bTU/KQ7u+rprJY0czsW4Uo6vSDu1SOdzNhrCjJ9p4dm8Tz+1r5tl9TV1CfPfhE13K1tdVM7u+lstnTaS6soL2QtBWiPy5QFt71+nW9gLHWrPp1vagvVDoXN5e9L5C0N7tvYXT9GKpqtDJUJ48tpo59bVZQI+tpr6umvq67PXkfF5ttWOoP/hf1WyQHTrW2hnaezvD+9l9zew92jXAG8aNZk59La+f18DcyXXMrq9lTn0ds+prGT+A7cmFbjuOjh1JVYWYMGbUsLpkcKRyuFvSDh9v5cGNu1n7s13sOXKCMaMqGT2qgjGjKhlTXUlNVfbcMa9mVGX+XEFNl+mu88dUZ/NqqrJmkQPNrZ1H3t2aUQ40t3ap09QJNcyur2Xx/HOZXV/HnPpaZtdnQV43emj8l62oENUne2G6mWQoGhp/KWYDqLmljYc27+H+n77AY1saaWkvMP2cMcybMpbjre0cOd5G45ETHG9t51hrO8dbCxxrbaelrdD7h/egqkInu6sDSDBtwhjmTq7j+ldMZU4e3HMm1zFrUi01oxyWdvYc7paE463tPLalkW9teIFHNu/hWGs7544bzXuumsVbXzmNy2ae02tTQqEQHG/rDPvjre0ca2nnRFs7x1oKRTuDjkdnufqxo08egc+cNMYnBa3fOdxtxGppK/BfWxu5/6e7+M6m3Rw90cakumre8arp3HDJNF49Z1KfBniqqBC11VWM0D4vNsI43G1EaWsv8MNt+7l/wwt8+8kXOXSslfE1VVz/ivN46yun8Zrz6wd86FWzweBwt2GvUAjWP3eAb/30Bb795C72Hm2hrrqSN198HjdcMpXXz2ugusqBbmlxuNuwFBH8ZPtB7t+wi3/bsIsXDx+nZlQF1140hbe+cipXX3iuT0xa0hzuNmy0thf40TP7eWjzbh7ctJsdB45RXVnBGy9s4NZLLmLx/ClD5lJBs8FW0v8ESUuAz5Jd0HpnRPxZt+V/BfxyPlkLnBsR55SzopamQ82tPPbUHh7avIfHtuzhyPE2RldV8NoLJvOxxS/nTQumMGFMOoNBmZWq13CXVAncAbwJ2AGsk7Qmv/sSABHxu0XlPwxc1g91tUQ8t6+JBzft5qHNu1n37AHaC8HksdVcv3AqixdM4XUXTE5+UCiz3pRy5H4FsDUitgFIWg0sAzadovyNwP8uT/UsBe2F4CfbD/Dgpj08tHk3W/ccBeCi88bxwTeez+L5U3jljHOoGAL3pTQbLkoJ9+nA9qLpHcCVPRWUNBuYCzxyiuUrgBUAs2bN6lNFbWRpOtHGfz69l4c27+aRn+9hf1MLVRXiyvMn8Z4rZ7F4/hRmTqod7GqaDVvlPvu0HLgvInoc2zMiVgGrABYtWnSaceVsJNp16BgPb86Ozr//i320tBUYX1PFL190LovnT+GNFzYM6OBXZiNZKeG+E5hZND0jn9eT5cCHzrZSNjS1tRdoammnuaWNphPZ89ETbTSfaKeppY3mlnaaTnQua8rLNZ1oY8eBY2zadRiA2fW1/MZVs1k8fwqL5kxklDsVmZVdKeG+DpgnaS5ZqC8H3t29kKSLgInAD8paQxs0be0FnnjuAA//fA8Pb97NLxqbSn5vzagK6qqrqB1dSV11FZPqqvnEkot404JzeVnDWA8Ja9bPeg33iGiTtBJ4gOxSyLsiYqOk24H1EbEmL7ocWB0Rbm4Zxg4da+U/nmrkkc27eXRLI4eOtTKqUlw5t54bLpnGuJoqxo6uonZ0FXXVldRWd0xXdgnzvozZYmblV1Kbe0SsBdZ2m3dbt+k/KF+1bCA9u7eJhzbv5uHNe1j37H7aCsGkumqunZ+1hb9+3uSkbixsNhK4O1+CTtXc8vIpY/nAG85n8fxzuXTmRB99mw1jDveE/OiZ/Xzl8ed47KlGDjZ3Nre896rZXHvRFGbV+9JDs5HC4Z6Q2775JDsOHOPNF09xc4vZCOdwT0hre4E3vryBv3zXpYNdFTPrZ77AOCFBdv9OMxv5HO4JicDXl5slwuGekIjAF8CYpcHhnpBCgLPdLA0O94QEQYWbZcyS4HBPSKGAD93NEuFwT4yP3M3S4HBPSCHCB+5miXC4JyTCR+5mqXC4J6QQQYW3uFkS/F89IdlA+z5yN0tBSeEuaYmkLZK2SrrlFGXeJWmTpI2SvlLealo5uBOTWTp6HThMUiVwB/AmYAewTtKaiNhUVGYecCvw2og4IOnc/qqwnbls+IHBroWZDYRSjtyvALZGxLaIaAFWA8u6lfkAcEdEHACIiD3lraaVQyHcicksFaWE+3Rge9H0jnxesZcDL5f0PUk/lLSkpw+StELSeknrGxsbz6zGdsY8/IBZOsp1QrUKmAdcDdwIfF7SOd0LRcSqiFgUEYsaGhrK9NVWqojwqJBmiSgl3HcCM4umZ+Tziu0A1kREa0Q8AzxFFvY2hLjN3SwdpYT7OmCepLmSqoHlwJpuZb5BdtSOpMlkzTTbylhPK4PAnZjMUtFruEdEG7ASeADYDNwTERsl3S5paV7sAWCfpE3Ao8D/iIh9/VVpOzMefsAsHSXdQzUi1gJru827reh1AL+XP2yIioAKX+hulgT3UE2Ij9zN0uFwT0h2g2zHu1kKHO4JyS6FHOxamNlAcLgnJBvyd7BrYWYDweGekKzN3elulgKHe0Ky69wHuxZmNhAc7gnJeqg63c1S4HBPRNYVwcMPmKXC4Z6IQpbtHn7ALBEO90ScPHIf5HqY2cBwuCfi5JG7z6iaJcHhnohCfuRuZmlwuCfGbe5maXC4J6Lgq2XMkuJwT0ScvFpmcOthZgOjpHCXtETSFklbJd3Sw/KbJDVK+kn+eH/5q2pn4+SRu6+XMUtCrzfrkFQJ3AG8iexeqeskrYmITd2Kfi0iVvZDHa0MOk6nulnGLA2lHLlfAWyNiG0R0QKsBpb1b7Ws3KKQPXv4AbM0lBLu04HtRdM78nndvUPSBkn3SZrZ0wdJWiFpvaT1jY2NZ1BdO1ORH7u7zd0sDeU6ofotYE5EXAI8CHyxp0IRsSoiFkXEooaGhjJ9tZWioxOTs90sDaWE+06g+Eh8Rj7vpIjYFxEn8sk7gVeVp3pWLh3DD7iHqlkaSgn3dcA8SXMlVQPLgTXFBSRNLZpcCmwuXxWtHHzkbpaWXq+WiYg2SSuBB4BK4K6I2CjpdmB9RKwBPiJpKdAG7Adu6sc62xnoaHP3CVWzNPQa7gARsRZY223ebUWvbwVuLW/VrJzCQ/6aJcU9VBPREe7OdrM0ONwT0dFD1edTzdLgcE+Ehx8wS4vDPRFuljFLi8M9EZ3h7nQ3S4HDPREefsAsLQ73RBTcLGOWFId7Ik4OP+B0N0uCwz0RBd8f2ywpDvdk+MjdLCUO90S4zd0sLQ73RHhsGbO0ONwT0dlD1cxS4HBPhDsxmaXF4Z6Ik0fuznazJJQU7pKWSNoiaaukW05T7h2SQtKi8lXRyslt7mZp6DXcJVUCdwDXAQuAGyUt6KHcOOCjwOPlrqSdPQ/5a5aWUo7crwC2RsS2iGgBVgPLeij3h8CngeNlrJ+ViUeFNEtLKeE+HdheNL0jn3eSpMuBmRHxb6f7IEkrJK2XtL6xsbHPlbUz19nm7nQ3S8FZn1CVVAH8JfDx3spGxKqIWBQRixoaGs72q60PTnZiGtxqmNkAKSXcdwIzi6Zn5PM6jAMWAo9Jeha4Cljjk6pDjYcfMEtJKeG+Dpgnaa6kamA5sKZjYUQciojJETEnIuYAPwSWRsT6fqmxnREPP2CWll7DPSLagJXAA8Bm4J6I2CjpdklL+7uCVh4efsAsLVWlFIqItcDabvNuO0XZq8++WlZuHn7ALC3uoZoIDz9glhaHeyLCww+YJcXhnoiOGzG5zd0sDQ73RHjgMLO0ONwT0Xm1zODWw8wGhsM9ER1H7r5exiwNDvdEdLa5D2o1zGyAONwTEeHhB8xS4nBPhIf8NUuLwz0RBQ8/YJYUh3si4uQJVTNLgcM9ET5yN0uLwz0RHn7ALC0O90R4+AGztDjcE+HhB8zSUlK4S1oiaYukrZJu6WH5ByX9TNJPJP2XpAXlr6qdDQ8/YJaWXsNdUiVwB3AdsAC4sYfw/kpEvCIiLgX+nOyG2TaEePgBs7SUcuR+BbA1IrZFRAuwGlhWXCAiDhdN1tHZxGtDjI/czdJQym32pgPbi6Z3AFd2LyTpQ8DvAdXANT19kKQVwAqAWbNm9bWudhY629yd7mYpKNsJ1Yi4IyJeBnwC+NQpyqyKiEURsaihoaFcX20lcJu7WVpKCfedwMyi6Rn5vFNZDbztbCpl5dfRiUluczdLQinhvg6YJ2mupGpgObCmuICkeUWTbwGeLl8VrRzcicksLb22uUdEm6SVwANAJXBXRGyUdDuwPiLWACslLQZagQPA+/qz0tZ3HhXSLC2lnFAlItYCa7vNu63o9UfLXC8rs8DjuZulxD1UE+GBw8zS4nBPhJtlzNLicE+Ex5YxS4vDPREnr5bxpZBmSXC4J6JzyN9BrYaZDRCHeyIKBQ8/YJYSh3sifORulhaHeyI8/IBZWhzuiTh5QtVb3CwJ/q+eiJPXuQ9uNcxsgDjcE+HhB8zS4nBPRME9VM2S4nBPRHhsGbOkONwT0XmDbDNLgcM9MT5yN0tDSeEuaYmkLZK2Srqlh+W/J2mTpA2SHpY0u/xVtbPR2UN1kCtiZgOi13CXVAncAVwHLABulLSgW7EfA4si4hLgPuDPy11ROzudPVSd7mYpKOXI/Qpga0Rsi4gWshtgLysuEBGPRkRzPvlDspto2xDS0ebu4QfM0lBKuE8HthdN78jnncrNwLd7WiBphaT1ktY3NjaWXks7a50363C6m6WgrCdUJb0XWAR8pqflEbEqIhZFxKKGhoZyfrX1IiLc3m6WkFJukL0TmFk0PSOf14WkxcAngTdGxInyVM/KpRAeesAsJaUcua8D5kmaK6kaWA6sKS4g6TLgH4ClEbGn/NW0sxWET6aaJaTXcI+INmAl8ACwGbgnIjZKul3S0rzYZ4CxwL2SfiJpzSk+zgZJIXwZpFlKSmmWISLWAmu7zbut6PXiMtfLyizCJ1PNUuIeqomICLe5myXE4Z6IwB2YzFLicE9EoeBLIc1S4nBPhI/czdLicE9EwW3uZklxuCcifCmkWVIc7onIhh9wupulwuGeiKzNfbBrYWYDxeGeiIKP3M2S4nBPRISP3M1S4nBPRMHDD5glxeGeCA8/YJYWh3sismYZx7tZKhzuiSj4TkxmSXG4J8LDD5ilpaRwl7RE0hZJWyXd0sPyN0j6b0ltkt5Z/mra2Sp03CHbzJLQa7hLqgTuAK4DFgA3SlrQrdjzwE3AV8pdQSuTgAr/TjNLRil3YroC2BoR2wAkrQaWAZs6CkTEs/myQj/U0cogGzjMzTJmqSjlWG46sL1oekc+z4YRDz9glpYB/aEuaYWk9ZLWNzY2DuRXJ8+dmMzSUkq47wRmFk3PyOf1WUSsiohFEbGooaHhTD7CzlD4UkizpJQS7uuAeZLmSqoGlgNr+rdaVm4RuMXdLCG9hntEtAErgQeAzcA9EbFR0u2SlgJIerWkHcCvAf8gaWN/Vtr6Lghf526WkFKuliEi1gJru827rej1OrLmGhuiCgXficksJb7yORE+cjdLi8M9EQV3UDVLisM9ER4V0iwtDvdERISHHzBLiP+7J8LDD5ilxeGeCA8/YJYWh3siCoGvhTRLiMM9ERHhI3ezhDjcE+HhB8zS4nBPhDsxmaXF4Z4IDz9glhaHeyKC8HjuZglxuCei4DZ3s6Q43FPh4QfMkuJwT0TBd2IyS4rDPRFZD1Wnu1kqSgp3SUskbZG0VdItPSwfLelr+fLHJc0pd0Xt7PjI3SwtvYa7pErgDuCDXdz9AAAE9klEQVQ6YAFwo6QF3YrdDByIiAuAvwI+Xe6K2tmJwFfLmCWklCP3K4CtEbEtIlqA1cCybmWWAV/MX98HXCsnyZASEb5axiwhpdxDdTqwvWh6B3DlqcpERJukQ0A9sLe4kKQVwIp88qikLWdS6TKaTLc6DmMlrcs/3TwANTl7I2W7jJT1AK/LUDK7lEIl3SC7XCJiFbBqIL/zdCStj4hFg12PcvC6DD0jZT3A6zIcldIssxOYWTQ9I5/XYxlJVcAEYF85KmhmZn1XSrivA+ZJmiupGlgOrOlWZg3wvvz1O4FHIsK3ZDYzGyS9NsvkbegrgQeASuCuiNgo6XZgfUSsAb4AfEnSVmA/2Q5gOBgyTURl4HUZekbKeoDXZdiRD7DNzEYe91A1MxuBHO5mZiNQEuEuaaakRyVtkrRR0kfz+ZMkPSjp6fx54mDXtVSSKiX9WNL9+fTcfOiHrflQENWDXcdSSDpH0n2Sfi5ps6TXDNftIul387+vJyV9VVLNcNkuku6StEfSk0XzetwOyvxNvk4bJF0+eDXv6hTr8Zn872uDpK9LOqdo2a35emyR9CuDU+v+kUS4A23AxyNiAXAV8KF8CIVbgIcjYh7wcD49XHwU2Fw0/Wngr/IhIA6QDQkxHHwW+PeIuAh4Jdk6DbvtImk68BFgUUQsJLv4YDnDZ7vcDSzpNu9U2+E6YF7+WAF8boDqWIq7eel6PAgsjIhLgKeAWwHyDFgOXJy/5+/y4VZGhCTCPSJ2RcR/56+PkAXIdLoOm/BF4G2DU8O+kTQDeAtwZz4t4BqyoR9gmKyLpAnAG8iutiIiWiLiIMN0u5BdfTYm7+tRC+ximGyXiPgu2ZVuxU61HZYB/xSZHwLnSJo6MDU9vZ7WIyK+ExFt+eQPyfrqQLYeqyPiREQ8A2wlG25lREgi3IvlI1ZeBjwOTImIXfmiF4Epg1Stvvpr4H8ChXy6HjhY9Ae8g2znNdTNBRqBf8ybmO6UVMcw3C4RsRP4v8DzZKF+CHiC4bldOpxqO/Q0JMlwWa/fBr6dvx7O69GrpMJd0ljgX4CPRcTh4mV5p6shf12opBuAPRHxxGDXpQyqgMuBz0XEZUAT3ZpghtF2mUh2JDgXmAbU8dLmgWFruGyH05H0SbIm2i8Pdl0GQjLhLmkUWbB/OSL+NZ+9u+PnZP68Z7Dq1wevBZZKepZshM5ryNqtz8mbA6DnISKGoh3Ajoh4PJ++jyzsh+N2WQw8ExGNEdEK/CvZthqO26XDqbZDKUOSDCmSbgJuAN5T1Ht+2K1HXyQR7nmb9BeAzRHxl0WLiodNeB/wzYGuW19FxK0RMSMi5pCdDHokIt4DPEo29AMMn3V5Edgu6cJ81rXAJobhdiFrjrlKUm3+99axLsNuuxQ51XZYA/xmftXMVcChouabIUfSErJmzKUR0Vy0aA2wXNnNhuaSnSD+0WDUsV9ExIh/AK8j+0m5AfhJ/rierK36YeBp4CFg0mDXtY/rdTVwf/76fLI/zK3AvcDowa5fietwKbA+3zbfACYO1+0C/B/g58CTwJeA0cNluwBfJTtX0Er2i+rmU20HQGQ38PkF8DOyK4QGfR1Osx5bydrWO/7v/31R+U/m67EFuG6w61/Oh4cfMDMbgZJoljEzS43D3cxsBHK4m5mNQA53M7MRyOFuZjYCOdzNzEYgh7uZ2Qj0/wEroqBF3LyluAAAAABJRU5ErkJggg==\n", 275 | "text/plain": [ 276 | "
" 277 | ] 278 | }, 279 | "metadata": {}, 280 | "output_type": "display_data" 281 | } 282 | ], 283 | "source": [ 284 | "model.set_params(polynomialfeatures__degree=15)\n", 285 | "train_sizes, train_scores, validation_scores = learning_curve(\n", 286 | " model, x[:, np.newaxis], y, train_sizes=np.logspace(-1, 0, 20))\n", 287 | "\n", 288 | "# Plot the mean train score and validation score across folds\n", 289 | "plt.plot(train_sizes, validation_scores.mean(axis=1), label='cross-validation') \n", 290 | "\n", 291 | "plt.plot(train_sizes, train_scores.mean(axis=1), label='training') \n", 292 | "plt.title('degree=15')\n", 293 | "plt.ylim(ymin=0,ymax=0.9)" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "Here we show the learning curve for d = 15. From the above discussion, we know that d = 15 is a high-variance estimator which over-fits the data. This is indicated by the fact that the training score is much higher than the validation score. As we add more samples to this training set, the training score will continue to decrease, while the cross-validation error will continue to increase, until they meet in the middle.\n", 301 | "\n", 302 | "\n", 303 | "**Learning curves that have not yet converged with the full training set indicate a high-variance, over-fit model.**\n", 304 | "\n", 305 | "A high-variance model can be improved by:\n", 306 | "\n", 307 | "Gathering more training samples.\n", 308 | "Using a less-sophisticated model (i.e. in this case, make d smaller)\n", 309 | "Increasing regularization.\n", 310 | "In particular, gathering more features for each sample will not help the results." 311 | ] 312 | } 313 | ], 314 | "metadata": { 315 | "kernelspec": { 316 | "display_name": "Python 3", 317 | "language": "python", 318 | "name": "python3" 319 | }, 320 | "language_info": { 321 | "codemirror_mode": { 322 | "name": "ipython", 323 | "version": 3 324 | }, 325 | "file_extension": ".py", 326 | "mimetype": "text/x-python", 327 | "name": "python", 328 | "nbconvert_exporter": "python", 329 | "pygments_lexer": "ipython3", 330 | "version": "3.5.2" 331 | } 332 | }, 333 | "nbformat": 4, 334 | "nbformat_minor": 2 335 | } 336 | --------------------------------------------------------------------------------