├── .gitignore ├── LICENSE ├── README.md ├── data ├── auto_mpg.csv └── casting_images.npz ├── images ├── descent.png └── residuals.png ├── notebooks_en ├── 1_Linear_Regression.ipynb ├── 2_Logistic_Regression.ipynb ├── 3_Multiple_Linear_Regression.ipynb ├── 4_Polynomial_Regression.ipynb └── 5_Multiple_Logistic_Regression.ipynb ├── scripts └── plot_helpers.py └── style └── custom.css /.gitignore: -------------------------------------------------------------------------------- 1 | #Ignore notebook checkpoints 2 | .ipynb_checkpoints 3 | .DS_Store 4 | *.pyc 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2019, engineersCode 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Engineering Computations Module 6 2 | 3 | _Engineering Computations_ is a collection of stackable learning modules, flexible for adoption in different situations. 4 | It aims to develop computational skills for students in engineering, but it can also be used by students in other science majors. 5 | The modules use the Python programming language and the Jupyter open-source tools for interactive computing. 6 | 7 | > Rather than "learning to code," our vision is "coding to learn." 8 | 9 | ## Module 6: deep learning 10 | 11 | *A step-by-step introduction to deep learning (a.k.a. neural network) models, aimed at scientists and engineers having a background in calculus and linear algebra.* 12 | 13 | [![Tweet](https://img.shields.io/twitter/url/http/shields.io.svg?style=social)](https://twitter.com/intent/tweet?text=Step-by-step%20Deep%20Learning%20tutorial%20with%20Python%20for%20engineering%20students%20%23EngineersCode%20&url=https://github.com/engineersCode/EngComp6_deeplearning) 14 | 15 | **Pre-requisite: learning modules [*EngComp 1*](https://github.com/engineersCode/EngComp1_offtheground) and [*EngComp 4*](https://github.com/engineersCode/EngComp4_landlinear) of our collection.** Recommended: [*EngComp 2*](https://github.com/engineersCode/EngComp2_takeoff), or basic use of `pandas` for data manipulation. 16 | 17 | ### [Lesson 1](http://go.gwu.edu/engcomp6lesson1): Linear regression by gradient descent 18 | 19 | Find the minimum of a function by gradient descent. Play with SymPy. Key ingredients of building a linear model from data with a single independent variable. Optimize a loss function to find the model parameters. 20 | 21 | ### [Lesson 2](http://go.gwu.edu/engcomp6lesson2): Logistic regression 22 | 23 | Composition of a linear model with the logistic function. Construct the logistic loss function by integration. Find the model parameters with `autograd`. Combine with a decision boundary to do classification. 24 | 25 | ### [Lesson 3](http://go.gwu.edu/engcomp6lesson3): Multiple linear regression 26 | 27 | Use multiple independent variables to build a linear model. Express multiple linear regression in matrix form. Find the weights by gradient descent. Scale (normalize) the features to ensure convergence. Get acquainted with `scikit-learn`. Model accuracy. Linear regression with `scikit-learn` and with pseudo-inverse. 28 | 29 | ### [Lesson 4](http://go.gwu.edu/engcomp6lesson4): Polynomial regression 30 | 31 | Fitting a polynomial to data is a special case of multiple linear regression. Build polynomial features, scale the data, and train the model like in Lesson 3. For predictions with the model, use the scaling from the training data on the new data. 32 | Observe underfitting and overfitting. Use regularization to avoid overfitting. This is also called _ridge regression_. 33 | Do it with scikit-learn's `Ridge()`. 34 | 35 | ### [Lesson 5](http://go.gwu.edu/engcomp6lesson5): Multiple logistic regression 36 | 37 | A taste of more practical machine learning applications: _multiple logistic regression_ for the problem of identifying defective metal-casting parts. 38 | Turn an image into a vector of grayscale values to use it as input data, and set up a classification problem from multi-dimensional feature vectors. 39 | Split data into training, validation, and test datasets to assess model performance. 40 | Normalize the data using z-score. 41 | Evaluate the performance of a classification model using F-score. 42 | 43 | ### [Lesson 6]() Multivariate regression (coming soon) 44 | 45 | ### [Lesson 7]() Neural network model (coming soon) 46 | 47 | ## Copyright and License 48 | 49 | (c) 2021 Lorena A. Barba, Pi-Yueh Chuang, Tingyu Wang. 50 | All content is under Creative Commons Attribution [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/legalcode.txt), and all [code is under BSD-3 clause](https://github.com/engineersCode/EngComp/blob/master/LICENSE). We are happy if you re-use the content in any way! 51 | 52 | [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause) [![License: CC BY 4.0](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/) 53 | -------------------------------------------------------------------------------- /data/auto_mpg.csv: -------------------------------------------------------------------------------- 1 | mpg,cylinders,displacement,horsepower,weight,acceleration,model year,origin,car name 2 | 18.0,8,307.0,130.0,3504.0,12.0,70,1,chevrolet chevelle malibu 3 | 15.0,8,350.0,165.0,3693.0,11.5,70,1,buick skylark 320 4 | 18.0,8,318.0,150.0,3436.0,11.0,70,1,plymouth satellite 5 | 16.0,8,304.0,150.0,3433.0,12.0,70,1,amc rebel sst 6 | 17.0,8,302.0,140.0,3449.0,10.5,70,1,ford torino 7 | 15.0,8,429.0,198.0,4341.0,10.0,70,1,ford galaxie 500 8 | 14.0,8,454.0,220.0,4354.0,9.0,70,1,chevrolet impala 9 | 14.0,8,440.0,215.0,4312.0,8.5,70,1,plymouth fury iii 10 | 14.0,8,455.0,225.0,4425.0,10.0,70,1,pontiac catalina 11 | 15.0,8,390.0,190.0,3850.0,8.5,70,1,amc ambassador dpl 12 | 15.0,8,383.0,170.0,3563.0,10.0,70,1,dodge challenger se 13 | 14.0,8,340.0,160.0,3609.0,8.0,70,1,plymouth 'cuda 340 14 | 15.0,8,400.0,150.0,3761.0,9.5,70,1,chevrolet monte carlo 15 | 14.0,8,455.0,225.0,3086.0,10.0,70,1,buick estate wagon (sw) 16 | 24.0,4,113.0,95.0,2372.0,15.0,70,3,toyota corona mark ii 17 | 22.0,6,198.0,95.0,2833.0,15.5,70,1,plymouth duster 18 | 18.0,6,199.0,97.0,2774.0,15.5,70,1,amc hornet 19 | 21.0,6,200.0,85.0,2587.0,16.0,70,1,ford maverick 20 | 27.0,4,97.0,88.0,2130.0,14.5,70,3,datsun pl510 21 | 26.0,4,97.0,46.0,1835.0,20.5,70,2,volkswagen 1131 deluxe sedan 22 | 25.0,4,110.0,87.0,2672.0,17.5,70,2,peugeot 504 23 | 24.0,4,107.0,90.0,2430.0,14.5,70,2,audi 100 ls 24 | 25.0,4,104.0,95.0,2375.0,17.5,70,2,saab 99e 25 | 26.0,4,121.0,113.0,2234.0,12.5,70,2,bmw 2002 26 | 21.0,6,199.0,90.0,2648.0,15.0,70,1,amc gremlin 27 | 10.0,8,360.0,215.0,4615.0,14.0,70,1,ford f250 28 | 10.0,8,307.0,200.0,4376.0,15.0,70,1,chevy c20 29 | 11.0,8,318.0,210.0,4382.0,13.5,70,1,dodge d200 30 | 9.0,8,304.0,193.0,4732.0,18.5,70,1,hi 1200d 31 | 27.0,4,97.0,88.0,2130.0,14.5,71,3,datsun pl510 32 | 28.0,4,140.0,90.0,2264.0,15.5,71,1,chevrolet vega 2300 33 | 25.0,4,113.0,95.0,2228.0,14.0,71,3,toyota corona 34 | 19.0,6,232.0,100.0,2634.0,13.0,71,1,amc gremlin 35 | 16.0,6,225.0,105.0,3439.0,15.5,71,1,plymouth satellite custom 36 | 17.0,6,250.0,100.0,3329.0,15.5,71,1,chevrolet chevelle malibu 37 | 19.0,6,250.0,88.0,3302.0,15.5,71,1,ford torino 500 38 | 18.0,6,232.0,100.0,3288.0,15.5,71,1,amc matador 39 | 14.0,8,350.0,165.0,4209.0,12.0,71,1,chevrolet impala 40 | 14.0,8,400.0,175.0,4464.0,11.5,71,1,pontiac catalina brougham 41 | 14.0,8,351.0,153.0,4154.0,13.5,71,1,ford galaxie 500 42 | 14.0,8,318.0,150.0,4096.0,13.0,71,1,plymouth fury iii 43 | 12.0,8,383.0,180.0,4955.0,11.5,71,1,dodge monaco (sw) 44 | 13.0,8,400.0,170.0,4746.0,12.0,71,1,ford country squire (sw) 45 | 13.0,8,400.0,175.0,5140.0,12.0,71,1,pontiac safari (sw) 46 | 18.0,6,258.0,110.0,2962.0,13.5,71,1,amc hornet sportabout (sw) 47 | 22.0,4,140.0,72.0,2408.0,19.0,71,1,chevrolet vega (sw) 48 | 19.0,6,250.0,100.0,3282.0,15.0,71,1,pontiac firebird 49 | 18.0,6,250.0,88.0,3139.0,14.5,71,1,ford mustang 50 | 23.0,4,122.0,86.0,2220.0,14.0,71,1,mercury capri 2000 51 | 28.0,4,116.0,90.0,2123.0,14.0,71,2,opel 1900 52 | 30.0,4,79.0,70.0,2074.0,19.5,71,2,peugeot 304 53 | 30.0,4,88.0,76.0,2065.0,14.5,71,2,fiat 124b 54 | 31.0,4,71.0,65.0,1773.0,19.0,71,3,toyota corolla 1200 55 | 35.0,4,72.0,69.0,1613.0,18.0,71,3,datsun 1200 56 | 27.0,4,97.0,60.0,1834.0,19.0,71,2,volkswagen model 111 57 | 26.0,4,91.0,70.0,1955.0,20.5,71,1,plymouth cricket 58 | 24.0,4,113.0,95.0,2278.0,15.5,72,3,toyota corona hardtop 59 | 25.0,4,97.5,80.0,2126.0,17.0,72,1,dodge colt hardtop 60 | 23.0,4,97.0,54.0,2254.0,23.5,72,2,volkswagen type 3 61 | 20.0,4,140.0,90.0,2408.0,19.5,72,1,chevrolet vega 62 | 21.0,4,122.0,86.0,2226.0,16.5,72,1,ford pinto runabout 63 | 13.0,8,350.0,165.0,4274.0,12.0,72,1,chevrolet impala 64 | 14.0,8,400.0,175.0,4385.0,12.0,72,1,pontiac catalina 65 | 15.0,8,318.0,150.0,4135.0,13.5,72,1,plymouth fury iii 66 | 14.0,8,351.0,153.0,4129.0,13.0,72,1,ford galaxie 500 67 | 17.0,8,304.0,150.0,3672.0,11.5,72,1,amc ambassador sst 68 | 11.0,8,429.0,208.0,4633.0,11.0,72,1,mercury marquis 69 | 13.0,8,350.0,155.0,4502.0,13.5,72,1,buick lesabre custom 70 | 12.0,8,350.0,160.0,4456.0,13.5,72,1,oldsmobile delta 88 royale 71 | 13.0,8,400.0,190.0,4422.0,12.5,72,1,chrysler newport royal 72 | 19.0,3,70.0,97.0,2330.0,13.5,72,3,mazda rx2 coupe 73 | 15.0,8,304.0,150.0,3892.0,12.5,72,1,amc matador (sw) 74 | 13.0,8,307.0,130.0,4098.0,14.0,72,1,chevrolet chevelle concours (sw) 75 | 13.0,8,302.0,140.0,4294.0,16.0,72,1,ford gran torino (sw) 76 | 14.0,8,318.0,150.0,4077.0,14.0,72,1,plymouth satellite custom (sw) 77 | 18.0,4,121.0,112.0,2933.0,14.5,72,2,volvo 145e (sw) 78 | 22.0,4,121.0,76.0,2511.0,18.0,72,2,volkswagen 411 (sw) 79 | 21.0,4,120.0,87.0,2979.0,19.5,72,2,peugeot 504 (sw) 80 | 26.0,4,96.0,69.0,2189.0,18.0,72,2,renault 12 (sw) 81 | 22.0,4,122.0,86.0,2395.0,16.0,72,1,ford pinto (sw) 82 | 28.0,4,97.0,92.0,2288.0,17.0,72,3,datsun 510 (sw) 83 | 23.0,4,120.0,97.0,2506.0,14.5,72,3,toyouta corona mark ii (sw) 84 | 28.0,4,98.0,80.0,2164.0,15.0,72,1,dodge colt (sw) 85 | 27.0,4,97.0,88.0,2100.0,16.5,72,3,toyota corolla 1600 (sw) 86 | 13.0,8,350.0,175.0,4100.0,13.0,73,1,buick century 350 87 | 14.0,8,304.0,150.0,3672.0,11.5,73,1,amc matador 88 | 13.0,8,350.0,145.0,3988.0,13.0,73,1,chevrolet malibu 89 | 14.0,8,302.0,137.0,4042.0,14.5,73,1,ford gran torino 90 | 15.0,8,318.0,150.0,3777.0,12.5,73,1,dodge coronet custom 91 | 12.0,8,429.0,198.0,4952.0,11.5,73,1,mercury marquis brougham 92 | 13.0,8,400.0,150.0,4464.0,12.0,73,1,chevrolet caprice classic 93 | 13.0,8,351.0,158.0,4363.0,13.0,73,1,ford ltd 94 | 14.0,8,318.0,150.0,4237.0,14.5,73,1,plymouth fury gran sedan 95 | 13.0,8,440.0,215.0,4735.0,11.0,73,1,chrysler new yorker brougham 96 | 12.0,8,455.0,225.0,4951.0,11.0,73,1,buick electra 225 custom 97 | 13.0,8,360.0,175.0,3821.0,11.0,73,1,amc ambassador brougham 98 | 18.0,6,225.0,105.0,3121.0,16.5,73,1,plymouth valiant 99 | 16.0,6,250.0,100.0,3278.0,18.0,73,1,chevrolet nova custom 100 | 18.0,6,232.0,100.0,2945.0,16.0,73,1,amc hornet 101 | 18.0,6,250.0,88.0,3021.0,16.5,73,1,ford maverick 102 | 23.0,6,198.0,95.0,2904.0,16.0,73,1,plymouth duster 103 | 26.0,4,97.0,46.0,1950.0,21.0,73,2,volkswagen super beetle 104 | 11.0,8,400.0,150.0,4997.0,14.0,73,1,chevrolet impala 105 | 12.0,8,400.0,167.0,4906.0,12.5,73,1,ford country 106 | 13.0,8,360.0,170.0,4654.0,13.0,73,1,plymouth custom suburb 107 | 12.0,8,350.0,180.0,4499.0,12.5,73,1,oldsmobile vista cruiser 108 | 18.0,6,232.0,100.0,2789.0,15.0,73,1,amc gremlin 109 | 20.0,4,97.0,88.0,2279.0,19.0,73,3,toyota carina 110 | 21.0,4,140.0,72.0,2401.0,19.5,73,1,chevrolet vega 111 | 22.0,4,108.0,94.0,2379.0,16.5,73,3,datsun 610 112 | 18.0,3,70.0,90.0,2124.0,13.5,73,3,maxda rx3 113 | 19.0,4,122.0,85.0,2310.0,18.5,73,1,ford pinto 114 | 21.0,6,155.0,107.0,2472.0,14.0,73,1,mercury capri v6 115 | 26.0,4,98.0,90.0,2265.0,15.5,73,2,fiat 124 sport coupe 116 | 15.0,8,350.0,145.0,4082.0,13.0,73,1,chevrolet monte carlo s 117 | 16.0,8,400.0,230.0,4278.0,9.5,73,1,pontiac grand prix 118 | 29.0,4,68.0,49.0,1867.0,19.5,73,2,fiat 128 119 | 24.0,4,116.0,75.0,2158.0,15.5,73,2,opel manta 120 | 20.0,4,114.0,91.0,2582.0,14.0,73,2,audi 100ls 121 | 19.0,4,121.0,112.0,2868.0,15.5,73,2,volvo 144ea 122 | 15.0,8,318.0,150.0,3399.0,11.0,73,1,dodge dart custom 123 | 24.0,4,121.0,110.0,2660.0,14.0,73,2,saab 99le 124 | 20.0,6,156.0,122.0,2807.0,13.5,73,3,toyota mark ii 125 | 11.0,8,350.0,180.0,3664.0,11.0,73,1,oldsmobile omega 126 | 20.0,6,198.0,95.0,3102.0,16.5,74,1,plymouth duster 127 | 19.0,6,232.0,100.0,2901.0,16.0,74,1,amc hornet 128 | 15.0,6,250.0,100.0,3336.0,17.0,74,1,chevrolet nova 129 | 31.0,4,79.0,67.0,1950.0,19.0,74,3,datsun b210 130 | 26.0,4,122.0,80.0,2451.0,16.5,74,1,ford pinto 131 | 32.0,4,71.0,65.0,1836.0,21.0,74,3,toyota corolla 1200 132 | 25.0,4,140.0,75.0,2542.0,17.0,74,1,chevrolet vega 133 | 16.0,6,250.0,100.0,3781.0,17.0,74,1,chevrolet chevelle malibu classic 134 | 16.0,6,258.0,110.0,3632.0,18.0,74,1,amc matador 135 | 18.0,6,225.0,105.0,3613.0,16.5,74,1,plymouth satellite sebring 136 | 16.0,8,302.0,140.0,4141.0,14.0,74,1,ford gran torino 137 | 13.0,8,350.0,150.0,4699.0,14.5,74,1,buick century luxus (sw) 138 | 14.0,8,318.0,150.0,4457.0,13.5,74,1,dodge coronet custom (sw) 139 | 14.0,8,302.0,140.0,4638.0,16.0,74,1,ford gran torino (sw) 140 | 14.0,8,304.0,150.0,4257.0,15.5,74,1,amc matador (sw) 141 | 29.0,4,98.0,83.0,2219.0,16.5,74,2,audi fox 142 | 26.0,4,79.0,67.0,1963.0,15.5,74,2,volkswagen dasher 143 | 26.0,4,97.0,78.0,2300.0,14.5,74,2,opel manta 144 | 31.0,4,76.0,52.0,1649.0,16.5,74,3,toyota corona 145 | 32.0,4,83.0,61.0,2003.0,19.0,74,3,datsun 710 146 | 28.0,4,90.0,75.0,2125.0,14.5,74,1,dodge colt 147 | 24.0,4,90.0,75.0,2108.0,15.5,74,2,fiat 128 148 | 26.0,4,116.0,75.0,2246.0,14.0,74,2,fiat 124 tc 149 | 24.0,4,120.0,97.0,2489.0,15.0,74,3,honda civic 150 | 26.0,4,108.0,93.0,2391.0,15.5,74,3,subaru 151 | 31.0,4,79.0,67.0,2000.0,16.0,74,2,fiat x1.9 152 | 19.0,6,225.0,95.0,3264.0,16.0,75,1,plymouth valiant custom 153 | 18.0,6,250.0,105.0,3459.0,16.0,75,1,chevrolet nova 154 | 15.0,6,250.0,72.0,3432.0,21.0,75,1,mercury monarch 155 | 15.0,6,250.0,72.0,3158.0,19.5,75,1,ford maverick 156 | 16.0,8,400.0,170.0,4668.0,11.5,75,1,pontiac catalina 157 | 15.0,8,350.0,145.0,4440.0,14.0,75,1,chevrolet bel air 158 | 16.0,8,318.0,150.0,4498.0,14.5,75,1,plymouth grand fury 159 | 14.0,8,351.0,148.0,4657.0,13.5,75,1,ford ltd 160 | 17.0,6,231.0,110.0,3907.0,21.0,75,1,buick century 161 | 16.0,6,250.0,105.0,3897.0,18.5,75,1,chevroelt chevelle malibu 162 | 15.0,6,258.0,110.0,3730.0,19.0,75,1,amc matador 163 | 18.0,6,225.0,95.0,3785.0,19.0,75,1,plymouth fury 164 | 21.0,6,231.0,110.0,3039.0,15.0,75,1,buick skyhawk 165 | 20.0,8,262.0,110.0,3221.0,13.5,75,1,chevrolet monza 2+2 166 | 13.0,8,302.0,129.0,3169.0,12.0,75,1,ford mustang ii 167 | 29.0,4,97.0,75.0,2171.0,16.0,75,3,toyota corolla 168 | 23.0,4,140.0,83.0,2639.0,17.0,75,1,ford pinto 169 | 20.0,6,232.0,100.0,2914.0,16.0,75,1,amc gremlin 170 | 23.0,4,140.0,78.0,2592.0,18.5,75,1,pontiac astro 171 | 24.0,4,134.0,96.0,2702.0,13.5,75,3,toyota corona 172 | 25.0,4,90.0,71.0,2223.0,16.5,75,2,volkswagen dasher 173 | 24.0,4,119.0,97.0,2545.0,17.0,75,3,datsun 710 174 | 18.0,6,171.0,97.0,2984.0,14.5,75,1,ford pinto 175 | 29.0,4,90.0,70.0,1937.0,14.0,75,2,volkswagen rabbit 176 | 19.0,6,232.0,90.0,3211.0,17.0,75,1,amc pacer 177 | 23.0,4,115.0,95.0,2694.0,15.0,75,2,audi 100ls 178 | 23.0,4,120.0,88.0,2957.0,17.0,75,2,peugeot 504 179 | 22.0,4,121.0,98.0,2945.0,14.5,75,2,volvo 244dl 180 | 25.0,4,121.0,115.0,2671.0,13.5,75,2,saab 99le 181 | 33.0,4,91.0,53.0,1795.0,17.5,75,3,honda civic cvcc 182 | 28.0,4,107.0,86.0,2464.0,15.5,76,2,fiat 131 183 | 25.0,4,116.0,81.0,2220.0,16.9,76,2,opel 1900 184 | 25.0,4,140.0,92.0,2572.0,14.9,76,1,capri ii 185 | 26.0,4,98.0,79.0,2255.0,17.7,76,1,dodge colt 186 | 27.0,4,101.0,83.0,2202.0,15.3,76,2,renault 12tl 187 | 17.5,8,305.0,140.0,4215.0,13.0,76,1,chevrolet chevelle malibu classic 188 | 16.0,8,318.0,150.0,4190.0,13.0,76,1,dodge coronet brougham 189 | 15.5,8,304.0,120.0,3962.0,13.9,76,1,amc matador 190 | 14.5,8,351.0,152.0,4215.0,12.8,76,1,ford gran torino 191 | 22.0,6,225.0,100.0,3233.0,15.4,76,1,plymouth valiant 192 | 22.0,6,250.0,105.0,3353.0,14.5,76,1,chevrolet nova 193 | 24.0,6,200.0,81.0,3012.0,17.6,76,1,ford maverick 194 | 22.5,6,232.0,90.0,3085.0,17.6,76,1,amc hornet 195 | 29.0,4,85.0,52.0,2035.0,22.2,76,1,chevrolet chevette 196 | 24.5,4,98.0,60.0,2164.0,22.1,76,1,chevrolet woody 197 | 29.0,4,90.0,70.0,1937.0,14.2,76,2,vw rabbit 198 | 33.0,4,91.0,53.0,1795.0,17.4,76,3,honda civic 199 | 20.0,6,225.0,100.0,3651.0,17.7,76,1,dodge aspen se 200 | 18.0,6,250.0,78.0,3574.0,21.0,76,1,ford granada ghia 201 | 18.5,6,250.0,110.0,3645.0,16.2,76,1,pontiac ventura sj 202 | 17.5,6,258.0,95.0,3193.0,17.8,76,1,amc pacer d/l 203 | 29.5,4,97.0,71.0,1825.0,12.2,76,2,volkswagen rabbit 204 | 32.0,4,85.0,70.0,1990.0,17.0,76,3,datsun b-210 205 | 28.0,4,97.0,75.0,2155.0,16.4,76,3,toyota corolla 206 | 26.5,4,140.0,72.0,2565.0,13.6,76,1,ford pinto 207 | 20.0,4,130.0,102.0,3150.0,15.7,76,2,volvo 245 208 | 13.0,8,318.0,150.0,3940.0,13.2,76,1,plymouth volare premier v8 209 | 19.0,4,120.0,88.0,3270.0,21.9,76,2,peugeot 504 210 | 19.0,6,156.0,108.0,2930.0,15.5,76,3,toyota mark ii 211 | 16.5,6,168.0,120.0,3820.0,16.7,76,2,mercedes-benz 280s 212 | 16.5,8,350.0,180.0,4380.0,12.1,76,1,cadillac seville 213 | 13.0,8,350.0,145.0,4055.0,12.0,76,1,chevy c10 214 | 13.0,8,302.0,130.0,3870.0,15.0,76,1,ford f108 215 | 13.0,8,318.0,150.0,3755.0,14.0,76,1,dodge d100 216 | 31.5,4,98.0,68.0,2045.0,18.5,77,3,honda accord cvcc 217 | 30.0,4,111.0,80.0,2155.0,14.8,77,1,buick opel isuzu deluxe 218 | 36.0,4,79.0,58.0,1825.0,18.6,77,2,renault 5 gtl 219 | 25.5,4,122.0,96.0,2300.0,15.5,77,1,plymouth arrow gs 220 | 33.5,4,85.0,70.0,1945.0,16.8,77,3,datsun f-10 hatchback 221 | 17.5,8,305.0,145.0,3880.0,12.5,77,1,chevrolet caprice classic 222 | 17.0,8,260.0,110.0,4060.0,19.0,77,1,oldsmobile cutlass supreme 223 | 15.5,8,318.0,145.0,4140.0,13.7,77,1,dodge monaco brougham 224 | 15.0,8,302.0,130.0,4295.0,14.9,77,1,mercury cougar brougham 225 | 17.5,6,250.0,110.0,3520.0,16.4,77,1,chevrolet concours 226 | 20.5,6,231.0,105.0,3425.0,16.9,77,1,buick skylark 227 | 19.0,6,225.0,100.0,3630.0,17.7,77,1,plymouth volare custom 228 | 18.5,6,250.0,98.0,3525.0,19.0,77,1,ford granada 229 | 16.0,8,400.0,180.0,4220.0,11.1,77,1,pontiac grand prix lj 230 | 15.5,8,350.0,170.0,4165.0,11.4,77,1,chevrolet monte carlo landau 231 | 15.5,8,400.0,190.0,4325.0,12.2,77,1,chrysler cordoba 232 | 16.0,8,351.0,149.0,4335.0,14.5,77,1,ford thunderbird 233 | 29.0,4,97.0,78.0,1940.0,14.5,77,2,volkswagen rabbit custom 234 | 24.5,4,151.0,88.0,2740.0,16.0,77,1,pontiac sunbird coupe 235 | 26.0,4,97.0,75.0,2265.0,18.2,77,3,toyota corolla liftback 236 | 25.5,4,140.0,89.0,2755.0,15.8,77,1,ford mustang ii 2+2 237 | 30.5,4,98.0,63.0,2051.0,17.0,77,1,chevrolet chevette 238 | 33.5,4,98.0,83.0,2075.0,15.9,77,1,dodge colt m/m 239 | 30.0,4,97.0,67.0,1985.0,16.4,77,3,subaru dl 240 | 30.5,4,97.0,78.0,2190.0,14.1,77,2,volkswagen dasher 241 | 22.0,6,146.0,97.0,2815.0,14.5,77,3,datsun 810 242 | 21.5,4,121.0,110.0,2600.0,12.8,77,2,bmw 320i 243 | 21.5,3,80.0,110.0,2720.0,13.5,77,3,mazda rx-4 244 | 43.1,4,90.0,48.0,1985.0,21.5,78,2,volkswagen rabbit custom diesel 245 | 36.1,4,98.0,66.0,1800.0,14.4,78,1,ford fiesta 246 | 32.8,4,78.0,52.0,1985.0,19.4,78,3,mazda glc deluxe 247 | 39.4,4,85.0,70.0,2070.0,18.6,78,3,datsun b210 gx 248 | 36.1,4,91.0,60.0,1800.0,16.4,78,3,honda civic cvcc 249 | 19.9,8,260.0,110.0,3365.0,15.5,78,1,oldsmobile cutlass salon brougham 250 | 19.4,8,318.0,140.0,3735.0,13.2,78,1,dodge diplomat 251 | 20.2,8,302.0,139.0,3570.0,12.8,78,1,mercury monarch ghia 252 | 19.2,6,231.0,105.0,3535.0,19.2,78,1,pontiac phoenix lj 253 | 20.5,6,200.0,95.0,3155.0,18.2,78,1,chevrolet malibu 254 | 20.2,6,200.0,85.0,2965.0,15.8,78,1,ford fairmont (auto) 255 | 25.1,4,140.0,88.0,2720.0,15.4,78,1,ford fairmont (man) 256 | 20.5,6,225.0,100.0,3430.0,17.2,78,1,plymouth volare 257 | 19.4,6,232.0,90.0,3210.0,17.2,78,1,amc concord 258 | 20.6,6,231.0,105.0,3380.0,15.8,78,1,buick century special 259 | 20.8,6,200.0,85.0,3070.0,16.7,78,1,mercury zephyr 260 | 18.6,6,225.0,110.0,3620.0,18.7,78,1,dodge aspen 261 | 18.1,6,258.0,120.0,3410.0,15.1,78,1,amc concord d/l 262 | 19.2,8,305.0,145.0,3425.0,13.2,78,1,chevrolet monte carlo landau 263 | 17.7,6,231.0,165.0,3445.0,13.4,78,1,buick regal sport coupe (turbo) 264 | 18.1,8,302.0,139.0,3205.0,11.2,78,1,ford futura 265 | 17.5,8,318.0,140.0,4080.0,13.7,78,1,dodge magnum xe 266 | 30.0,4,98.0,68.0,2155.0,16.5,78,1,chevrolet chevette 267 | 27.5,4,134.0,95.0,2560.0,14.2,78,3,toyota corona 268 | 27.2,4,119.0,97.0,2300.0,14.7,78,3,datsun 510 269 | 30.9,4,105.0,75.0,2230.0,14.5,78,1,dodge omni 270 | 21.1,4,134.0,95.0,2515.0,14.8,78,3,toyota celica gt liftback 271 | 23.2,4,156.0,105.0,2745.0,16.7,78,1,plymouth sapporo 272 | 23.8,4,151.0,85.0,2855.0,17.6,78,1,oldsmobile starfire sx 273 | 23.9,4,119.0,97.0,2405.0,14.9,78,3,datsun 200-sx 274 | 20.3,5,131.0,103.0,2830.0,15.9,78,2,audi 5000 275 | 17.0,6,163.0,125.0,3140.0,13.6,78,2,volvo 264gl 276 | 21.6,4,121.0,115.0,2795.0,15.7,78,2,saab 99gle 277 | 16.2,6,163.0,133.0,3410.0,15.8,78,2,peugeot 604sl 278 | 31.5,4,89.0,71.0,1990.0,14.9,78,2,volkswagen scirocco 279 | 29.5,4,98.0,68.0,2135.0,16.6,78,3,honda accord lx 280 | 21.5,6,231.0,115.0,3245.0,15.4,79,1,pontiac lemans v6 281 | 19.8,6,200.0,85.0,2990.0,18.2,79,1,mercury zephyr 6 282 | 22.3,4,140.0,88.0,2890.0,17.3,79,1,ford fairmont 4 283 | 20.2,6,232.0,90.0,3265.0,18.2,79,1,amc concord dl 6 284 | 20.6,6,225.0,110.0,3360.0,16.6,79,1,dodge aspen 6 285 | 17.0,8,305.0,130.0,3840.0,15.4,79,1,chevrolet caprice classic 286 | 17.6,8,302.0,129.0,3725.0,13.4,79,1,ford ltd landau 287 | 16.5,8,351.0,138.0,3955.0,13.2,79,1,mercury grand marquis 288 | 18.2,8,318.0,135.0,3830.0,15.2,79,1,dodge st. regis 289 | 16.9,8,350.0,155.0,4360.0,14.9,79,1,buick estate wagon (sw) 290 | 15.5,8,351.0,142.0,4054.0,14.3,79,1,ford country squire (sw) 291 | 19.2,8,267.0,125.0,3605.0,15.0,79,1,chevrolet malibu classic (sw) 292 | 18.5,8,360.0,150.0,3940.0,13.0,79,1,chrysler lebaron town @ country (sw) 293 | 31.9,4,89.0,71.0,1925.0,14.0,79,2,vw rabbit custom 294 | 34.1,4,86.0,65.0,1975.0,15.2,79,3,maxda glc deluxe 295 | 35.7,4,98.0,80.0,1915.0,14.4,79,1,dodge colt hatchback custom 296 | 27.4,4,121.0,80.0,2670.0,15.0,79,1,amc spirit dl 297 | 25.4,5,183.0,77.0,3530.0,20.1,79,2,mercedes benz 300d 298 | 23.0,8,350.0,125.0,3900.0,17.4,79,1,cadillac eldorado 299 | 27.2,4,141.0,71.0,3190.0,24.8,79,2,peugeot 504 300 | 23.9,8,260.0,90.0,3420.0,22.2,79,1,oldsmobile cutlass salon brougham 301 | 34.2,4,105.0,70.0,2200.0,13.2,79,1,plymouth horizon 302 | 34.5,4,105.0,70.0,2150.0,14.9,79,1,plymouth horizon tc3 303 | 31.8,4,85.0,65.0,2020.0,19.2,79,3,datsun 210 304 | 37.3,4,91.0,69.0,2130.0,14.7,79,2,fiat strada custom 305 | 28.4,4,151.0,90.0,2670.0,16.0,79,1,buick skylark limited 306 | 28.8,6,173.0,115.0,2595.0,11.3,79,1,chevrolet citation 307 | 26.8,6,173.0,115.0,2700.0,12.9,79,1,oldsmobile omega brougham 308 | 33.5,4,151.0,90.0,2556.0,13.2,79,1,pontiac phoenix 309 | 41.5,4,98.0,76.0,2144.0,14.7,80,2,vw rabbit 310 | 38.1,4,89.0,60.0,1968.0,18.8,80,3,toyota corolla tercel 311 | 32.1,4,98.0,70.0,2120.0,15.5,80,1,chevrolet chevette 312 | 37.2,4,86.0,65.0,2019.0,16.4,80,3,datsun 310 313 | 28.0,4,151.0,90.0,2678.0,16.5,80,1,chevrolet citation 314 | 26.4,4,140.0,88.0,2870.0,18.1,80,1,ford fairmont 315 | 24.3,4,151.0,90.0,3003.0,20.1,80,1,amc concord 316 | 19.1,6,225.0,90.0,3381.0,18.7,80,1,dodge aspen 317 | 34.3,4,97.0,78.0,2188.0,15.8,80,2,audi 4000 318 | 29.8,4,134.0,90.0,2711.0,15.5,80,3,toyota corona liftback 319 | 31.3,4,120.0,75.0,2542.0,17.5,80,3,mazda 626 320 | 37.0,4,119.0,92.0,2434.0,15.0,80,3,datsun 510 hatchback 321 | 32.2,4,108.0,75.0,2265.0,15.2,80,3,toyota corolla 322 | 46.6,4,86.0,65.0,2110.0,17.9,80,3,mazda glc 323 | 27.9,4,156.0,105.0,2800.0,14.4,80,1,dodge colt 324 | 40.8,4,85.0,65.0,2110.0,19.2,80,3,datsun 210 325 | 44.3,4,90.0,48.0,2085.0,21.7,80,2,vw rabbit c (diesel) 326 | 43.4,4,90.0,48.0,2335.0,23.7,80,2,vw dasher (diesel) 327 | 36.4,5,121.0,67.0,2950.0,19.9,80,2,audi 5000s (diesel) 328 | 30.0,4,146.0,67.0,3250.0,21.8,80,2,mercedes-benz 240d 329 | 44.6,4,91.0,67.0,1850.0,13.8,80,3,honda civic 1500 gl 330 | 33.8,4,97.0,67.0,2145.0,18.0,80,3,subaru dl 331 | 29.8,4,89.0,62.0,1845.0,15.3,80,2,vokswagen rabbit 332 | 32.7,6,168.0,132.0,2910.0,11.4,80,3,datsun 280-zx 333 | 23.7,3,70.0,100.0,2420.0,12.5,80,3,mazda rx-7 gs 334 | 35.0,4,122.0,88.0,2500.0,15.1,80,2,triumph tr7 coupe 335 | 32.4,4,107.0,72.0,2290.0,17.0,80,3,honda accord 336 | 27.2,4,135.0,84.0,2490.0,15.7,81,1,plymouth reliant 337 | 26.6,4,151.0,84.0,2635.0,16.4,81,1,buick skylark 338 | 25.8,4,156.0,92.0,2620.0,14.4,81,1,dodge aries wagon (sw) 339 | 23.5,6,173.0,110.0,2725.0,12.6,81,1,chevrolet citation 340 | 30.0,4,135.0,84.0,2385.0,12.9,81,1,plymouth reliant 341 | 39.1,4,79.0,58.0,1755.0,16.9,81,3,toyota starlet 342 | 39.0,4,86.0,64.0,1875.0,16.4,81,1,plymouth champ 343 | 35.1,4,81.0,60.0,1760.0,16.1,81,3,honda civic 1300 344 | 32.3,4,97.0,67.0,2065.0,17.8,81,3,subaru 345 | 37.0,4,85.0,65.0,1975.0,19.4,81,3,datsun 210 mpg 346 | 37.7,4,89.0,62.0,2050.0,17.3,81,3,toyota tercel 347 | 34.1,4,91.0,68.0,1985.0,16.0,81,3,mazda glc 4 348 | 34.7,4,105.0,63.0,2215.0,14.9,81,1,plymouth horizon 4 349 | 34.4,4,98.0,65.0,2045.0,16.2,81,1,ford escort 4w 350 | 29.9,4,98.0,65.0,2380.0,20.7,81,1,ford escort 2h 351 | 33.0,4,105.0,74.0,2190.0,14.2,81,2,volkswagen jetta 352 | 33.7,4,107.0,75.0,2210.0,14.4,81,3,honda prelude 353 | 32.4,4,108.0,75.0,2350.0,16.8,81,3,toyota corolla 354 | 32.9,4,119.0,100.0,2615.0,14.8,81,3,datsun 200sx 355 | 31.6,4,120.0,74.0,2635.0,18.3,81,3,mazda 626 356 | 28.1,4,141.0,80.0,3230.0,20.4,81,2,peugeot 505s turbo diesel 357 | 30.7,6,145.0,76.0,3160.0,19.6,81,2,volvo diesel 358 | 25.4,6,168.0,116.0,2900.0,12.6,81,3,toyota cressida 359 | 24.2,6,146.0,120.0,2930.0,13.8,81,3,datsun 810 maxima 360 | 22.4,6,231.0,110.0,3415.0,15.8,81,1,buick century 361 | 26.6,8,350.0,105.0,3725.0,19.0,81,1,oldsmobile cutlass ls 362 | 20.2,6,200.0,88.0,3060.0,17.1,81,1,ford granada gl 363 | 17.6,6,225.0,85.0,3465.0,16.6,81,1,chrysler lebaron salon 364 | 28.0,4,112.0,88.0,2605.0,19.6,82,1,chevrolet cavalier 365 | 27.0,4,112.0,88.0,2640.0,18.6,82,1,chevrolet cavalier wagon 366 | 34.0,4,112.0,88.0,2395.0,18.0,82,1,chevrolet cavalier 2-door 367 | 31.0,4,112.0,85.0,2575.0,16.2,82,1,pontiac j2000 se hatchback 368 | 29.0,4,135.0,84.0,2525.0,16.0,82,1,dodge aries se 369 | 27.0,4,151.0,90.0,2735.0,18.0,82,1,pontiac phoenix 370 | 24.0,4,140.0,92.0,2865.0,16.4,82,1,ford fairmont futura 371 | 36.0,4,105.0,74.0,1980.0,15.3,82,2,volkswagen rabbit l 372 | 37.0,4,91.0,68.0,2025.0,18.2,82,3,mazda glc custom l 373 | 31.0,4,91.0,68.0,1970.0,17.6,82,3,mazda glc custom 374 | 38.0,4,105.0,63.0,2125.0,14.7,82,1,plymouth horizon miser 375 | 36.0,4,98.0,70.0,2125.0,17.3,82,1,mercury lynx l 376 | 36.0,4,120.0,88.0,2160.0,14.5,82,3,nissan stanza xe 377 | 36.0,4,107.0,75.0,2205.0,14.5,82,3,honda accord 378 | 34.0,4,108.0,70.0,2245.0,16.9,82,3,toyota corolla 379 | 38.0,4,91.0,67.0,1965.0,15.0,82,3,honda civic 380 | 32.0,4,91.0,67.0,1965.0,15.7,82,3,honda civic (auto) 381 | 38.0,4,91.0,67.0,1995.0,16.2,82,3,datsun 310 gx 382 | 25.0,6,181.0,110.0,2945.0,16.4,82,1,buick century limited 383 | 38.0,6,262.0,85.0,3015.0,17.0,82,1,oldsmobile cutlass ciera (diesel) 384 | 26.0,4,156.0,92.0,2585.0,14.5,82,1,chrysler lebaron medallion 385 | 22.0,6,232.0,112.0,2835.0,14.7,82,1,ford granada l 386 | 32.0,4,144.0,96.0,2665.0,13.9,82,3,toyota celica gt 387 | 36.0,4,135.0,84.0,2370.0,13.0,82,1,dodge charger 2.2 388 | 27.0,4,151.0,90.0,2950.0,17.3,82,1,chevrolet camaro 389 | 27.0,4,140.0,86.0,2790.0,15.6,82,1,ford mustang gl 390 | 44.0,4,97.0,52.0,2130.0,24.6,82,2,vw pickup 391 | 32.0,4,135.0,84.0,2295.0,11.6,82,1,dodge rampage 392 | 28.0,4,120.0,79.0,2625.0,18.6,82,1,ford ranger 393 | 31.0,4,119.0,82.0,2720.0,19.4,82,1,chevy s-10 394 | -------------------------------------------------------------------------------- /data/casting_images.npz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/engineersCode/EngComp6_deeplearning/8f76ef44050625abe041583ae1989b86ff5a717b/data/casting_images.npz -------------------------------------------------------------------------------- /images/descent.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/engineersCode/EngComp6_deeplearning/8f76ef44050625abe041583ae1989b86ff5a717b/images/descent.png -------------------------------------------------------------------------------- /images/residuals.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/engineersCode/EngComp6_deeplearning/8f76ef44050625abe041583ae1989b86ff5a717b/images/residuals.png -------------------------------------------------------------------------------- /notebooks_en/1_Linear_Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2021 Lorena A. Barba" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Linear regression by gradient descent\n", 15 | "\n", 16 | "This module of _Engineering Computations_ takes a step-by-step approach to introduce you to the essential ideas of deep learning, an algorithmic technology that is taking the world by storm. \n", 17 | "It is at the core of the artificial intelligence boom, and we think every scientist and engineer should understand the basics, at least. \n", 18 | "\n", 19 | "Another term for deep learning is deep neural networks. \n", 20 | "In this module, you will learn how neural-network models are built, computationally. \n", 21 | "The inspiration for deep learning may have been how the brain works, but in practice what we have is a method to build models, using mostly linear algebra and a little bit of calculus. \n", 22 | "These models are not magical, or even \"intelligent\"—they are just about _optimization_, which every engineer knows about!\n", 23 | "\n", 24 | "In this lesson, we take the first step of model-building: linear regression. \n", 25 | "The very first module of the _Engineering Computations_ series discusses [linear regression with real data](http://go.gwu.edu/engcomp1lesson5), and there we found the model parameters (slope and $y$-intercept) analytically. \n", 26 | "Let's forget about that for this lesson. \n", 27 | "The key concept we introduce here will be _gradient descent_. Start your ride here!" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "## Gradient descent\n", 35 | "\n", 36 | "This lesson is partly based on a tutorial at the 2019 SciPy Conference by Eric Ma [1]. He begins his tutorial by presenting the idea of _gradient descent_ with a simple quadratic function: the question is how do we find this function's minimum?\n", 37 | "\n", 38 | "$$f(w) = w^2 +3w -5$$\n", 39 | "\n", 40 | "We know from calculus that at the minimum, the derivative of the function is zero (the tangent to the function curve is horizontal), and the second derivative is positive (the curve slants _up_ on each side of the minimum). \n", 41 | "The analytical derivative of the function above is $f^\\prime(w) = 2w + 3$ and the second derivative is $f^{\\prime\\prime}(w)=2>0$. Thus, we make $2w+3=0$ to find the minimum.\n", 42 | "\n", 43 | "Let's play with this function using SymPy. We'll later use NumPy, and make plots with Matplotlib, so we load all the libraries in one place." 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 1, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "import sympy\n", 53 | "import numpy\n", 54 | "\n", 55 | "from matplotlib import pyplot\n", 56 | "%matplotlib inline" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "We run this SymPy method to get beautiful typeset symbols and equations (in the Jupyter notebook, it will use [MathJax](https://en.wikipedia.org/wiki/MathJax) by default): " 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 2, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "sympy.init_printing()" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "Now we'll define the Python variable `w` to be a SymPy symbol, and create the expression `f` to match the mathematical function above, and plot it." 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 3, 85 | "metadata": {}, 86 | "outputs": [ 87 | { 88 | "data": { 89 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAG8AAAAVCAYAAABIfLDHAAAACXBIWXMAAA7EAAAOxAGVKw4bAAADoUlEQVRoBe2Z61HbQBCAhYcCCOnA6QBSAnQQkgoCHYThF/7HOB0AFSShA0IFAXdgOgjjDsj3nXUaSSNsZMuyTbwz691b3WNft3eSt56fn5N5oNfr7TD+LJ2jm9KvyEfzzLsZO90D29O7TO3RJ1AnsRf8JfwD+CHKNnQxHug0MO0xATvIzdOH7yLby8k27AI80ETw3HX3C9BtM+U0D3jmNYnn5+d9cNjknP/rXPhxr2w7sh2wq7yJMy/Lj7RUfkKwnwlbZtDBS1M8g71M2fZc/t2yKk0sd4fe2jBIJ5MXgn8bC17qNM+7ffil3DRTQ0+hMXgJvMl0Cz0Cb7R8jeAp1dX7wyOo/hfYEfzbSPCYzOzWaYfQJG1LXbAWMMbLjxeeq1oDx52PIV6gDFYMVNxxvs5E2QxTL2XIADuOXlp57gsLkxs4Xw8u4fdE+FMwZg1sLbA0xPJQayCdLS9mZchMB6NPxtt+S9DEzvOdTmdLM8BpWenKhAtmWNNd9i6/DDLLpmCCvSkIwcNAnW/JeQ8OaWclC96ddAINwYDqjOxso11wFs9WBtDNEqyu6p+3aWZ72zYOvY2L+hobq5xnXrjAxLJ5huA7D/6AGpsHzwoHRRjB2P4YBatGscXy/Q29TDgNvS/puC72GrSfxgb0KBIf4E3KpAPjzjJogheO8lllx1sfCvS3NOmQ2pcRx7cB6OhBr8Ee9j9ADQ7lE7o29qLrIehmCQCvz/V/OAIsm48IwzaE/wxegAGQa6jRjze28QMyOZ0otmtTxqtAyKDS4F3bPK86MyfevkrzhCbz3IA64BfUEr8Qe5lbP92B0teCry/R968dYwAPGNfd5kfDEqiZ6cLZ2QCvc0c8Ky8wRD4XMGdVcKIevipYxmsBY0w25yjra9nUFo0OrwvQRu1lPv3Y2McJ5rPa7UJfmnOnQ4cIlkwzOwQzFSor7Dqe66Cyc9LuSyfeeC2RJuE0WHV7vVNU2REr0yAfvC6dy+dYlewLzikEdJqXWnxu4sUymV9WRwh5vatsq5Ity94r/Fz1t5oVJNjRCSaNf9xNWaQZaCejrEEBUll2eYnyFaLexgr6oXMsj74uGNwIq26vHz0K76a0vUEL4avLll+nI+Q6a+Rf0PPvGnwClRXeAWk3Duigs2c681SG8SZd/pOSyVf5YXoV7FXnlwD91N2EFNxIxsHPkCEJC8Gzx7IBxeYK3rL1b3P9fNlsc91Ja5lVIbMmddo8S5J/Uya9Zk7UnmQAAAAASUVORK5CYII=\n", 90 | "text/latex": [ 91 | "$\\displaystyle w^{2} + 3 w - 5$" 92 | ], 93 | "text/plain": [ 94 | " 2 \n", 95 | "w + 3⋅w - 5" 96 | ] 97 | }, 98 | "execution_count": 3, 99 | "metadata": {}, 100 | "output_type": "execute_result" 101 | } 102 | ], 103 | "source": [ 104 | "w = sympy.Symbol('w', real=True)\n", 105 | "\n", 106 | "f = w**2 + 3*w - 5\n", 107 | "f" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 4, 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "data": { 117 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAawAAAEeCAYAAADWyiHSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA1+ElEQVR4nO3deVxU9eL/8dewqqi4ASKLgiAiihuupaWImZVWltnVotWWe1tuZWX39stWzbK9a1natWtl2aJlLqVmqbnhrrigiAIiO6LsMOf3hze+19xQYc4M834+Hj3SMzOctweY95xzPudzLIZhICIiYu9czA4gIiJSEyosERFxCCosERFxCCosERFxCCosERFxCCosERFxCCosERFxCCosERFxCCosERFxCG5mBxC5UBaLpQFwLTAAaAOUADuBHw3D2GVmNhGpOxZNzSSOxGKxTAKuA1YCm4AsoAHQARj03z8/bhjGdpMiikgdUWGJQ7FYLNcYhvHjOR73BYINw0iwYSwRsQEVljgki8USahhGstk5RMR2VFjikCwWy29AALAR+A1YZRjGDnNTiUhdUmGJw7JYLB5AL+BK4D6gsWEYLUwNJSJ1RqMExSFZLJbLOTlKcADQDFgIrDIzk4jUrfPtYWn3S+ySq6srMTExTJw4keHDh+Ph4WF2JACGDRvGkiVLzI4h4kgsNX2i9rDEIeXm5rJmzRp+++033nnnHVxcXOjXrx8vvviiqblycnJMXb9IfabCEofUrFkzQkNDSU1NJS0tjd9//52KigqzY4lIHVJhiUNq3749ERERDBgwgPvvv59PPvnEbg4LikjdUGGJQ0pKSsLFRVNhijgT/caLQ3nppZfIy8s7a1mtWLGChQsX2jiViNiC9rDEoXTp0oXrrruOBg0a0KNHD3x8fCgtLSUpKYmtW7cyZMgQnnnmGbNjikgdUGGJQ/n6669Zs2YNU6dOxdfXl4yMDJo2bcq4ceOYMWMGDRs2NDuiiNQRFZY4lE2bNnHo0CE+++wzfvnll1MeKykpUWGJ1GMqLHEo999/P8OGDSM5OZmYmJjq5YZhYLFYSE7WfLgi9ZVmuhCH9MADDzB9+nSzY5wmJiaGhATd2UTqr9KKKhq4u9bml6zxTBcaJSgOyR7LSsQZ/P3Lrdz9742mrFuFJSIiNZKcfYIlu47S0b+JKetXYYmISI3M+C0Zd1cX7ugfYsr6VVgiInJemYWlfLs5ndExgfg08TQlgwpLRETOa9bqg1RarYwf0N60DCoscRp33XUXvr6+dO7cuXrZhAkT6NixI9HR0dxwww0UFBRUPzZ58mTCwsKIiIhg6dKlJiQWsQ/HSir4bP1hroluQ3DLRqblUGGJ07jjjjtOu7liXFwcO3fuZPv27XTo0IHJkycDkJiYyNy5c9m1axdLlizhwQcfpKqqyozYIqabs+4QJ8oquW9gqKk5VFjiNAYOHEiLFi1OWTZ06FDc3E5eP9+3b1/S0tIAWLBgAWPGjMHT05OQkBDCwsLYsGGDzTOLmK20oopP1qQwsIMPnQO8Tc2iwhL5r1mzZnH11VcDkJ6eTlBQUPVjgYGBpKenmxVNxDRfb0oj50QZ919h7t4VaGomEQBefvll3NzcGDt2LHByqqc/s1jOfEH+jBkzmDFjBgDZ2dl1F1LExqqsBh+tSqZrUDP6hbY0O472sERmz57NwoUL+eyzz6pLKTAwkNTU1OrnpKWl0aZNmzO+fvz48SQkJJCQkICPj49NMovYwuKdGRzKLeaBK0LP+oHNllRY4tSWLFnCq6++yvfff0+jRv83+mnEiBHMnTuXsrIyDh48SFJSEr179zYxqYhtGYbBD9uOEOrjxdBOrc2OA+iQoDiRW2+9lZUrV5KTk0NgYCDPP/88kydPpqysjLi4OODkwIsPPviAqKgoRo8eTadOnXBzc+P999/H1bVWJ/wUsWsr92azdFcmb4yOxsXF/L0r0GztIrVKs7VLfWAYBjd9sJaMghJWThiEh1udHozTbO0iInJx1h/MY9OhfO67on1dl9UFsZ8kIiJiF97/ZT+tGntyS6+g8z/ZhlRYIiJSbVtqAauScrhnQEht36jxkqmwRESk2nu/7KdpAzfG9gk2O8ppVFgiIgLA3qPH+TkxkzsuC6FJA3ez45xGhSUiIgD8a+V+Gnm4cmf/dmZHOSMVloiIkJJTxA/bjjCub1uae3mYHeeMVFgiIsIHvx7AzdWFey4PMTvKWamwREScXMaxEr7ZnMYtMUH4Nm1gdpyzUmGJiDi5Gb8lYxhwnx3cQuRcVFgiIk4s+3gZ65JzufvyEAKbNzr/C0ykwhIRcWIzfjvA3qPHGdPb/q67+jMVloiIk8o5UcacdYcZ2S2AkFZeZsc5LxWWiIiT+mhVMmWVVfxtcJjZUWpEhSUi4oTyisr5z9pDXNe1De19Gpsdp0ZUWCIiTuijVcmUVFTxkIPsXYEKS0TE6eQXlfPp7ylc08WfMN8mZsepMRWWiIiTmbn6IMUVVTwcG252lAuiwhIRcSIFxeX8+/cUhnf2p4Of4+xdgQpLRMSpzFp9kBNllTwU6zjnrv6gwhIRcRLHSir4ZE0Kw6Ja07F1U7PjXDAVloiIk/hkzUGOl1U63LmrP6iwREScQGFpBbNWH2RoJz86tXG8vStQYYmIOIV5G1Px8nRz2L0rUGGJiNR7x4oreGt5EtEB3nQO8DY7zkVTYYmI1HMfr07meGklj8Z1MDvKJVFhidO466678PX1pXPnztXL8vLyiIuLIzw8nLi4OPLz86sfmzx5MmFhYURERLB06VIzIotcsryicmatPsg1XfyJ9HfMc1d/UGGJ07jjjjtYsmTJKcumTJlCbGwsSUlJxMbGMmXKFAASExOZO3cuu3btYsmSJTz44INUVVWZEVvkknz46wGKK6p4dIjjnrv6gwpLnMbAgQNp0aLFKcsWLFhAfHw8APHx8cyfP796+ZgxY/D09CQkJISwsDA2bNhg68gilyTreCmz16Ywsmsbwh1sVoszUWGJU8vMzMTf3x8Af39/srKyAEhPTycoKKj6eYGBgaSnp5uSUeRiTV95gIoqg0eGOPa5qz+4mR1AxB4ZhnHaMovFcsbnzpgxgxkzZgCQnZ1dp7lEairjWAmfrT/MqB6OcTfhmtAeljg1Pz8/MjIyAMjIyMDX1xc4uUeVmppa/by0tDTatGlzxq8xfvx4EhISSEhIwMfHp+5Di9TA+7/sxzAMHhrs+Oeu/qDCEqc2YsQIZs+eDcDs2bMZOXJk9fK5c+dSVlbGwYMHSUpKonfv3mZGFamxtPxivtyYyuiYIIJaNDI7Tq3RIUFxGrfeeisrV64kJyeHwMBAnn/+eZ5++mlGjx7NzJkzCQ4OZt68eQBERUUxevRoOnXqhJubG++//z6urq4m/wtEaubd5fuxWCz8zYHuJlwTljMdq/8f53xQRE4VExNDQkKC2THEiaXkFBH7xq/c1rctk0ZEmR2nJs58cvgMdEhQRKQeeWd5Eu6uFh4c1N7sKLVOhSUiUk/szzzOwh0ZxPdrh2+TBmbHqXUqLBGRemLaz/sIbNaQ8QNDzY5SJ1RYIiL1wLbUAhbvPMqIbm1o2djT7Dh1QoUlIlIPTF26hxZeHtwzoH7uXYEKS0TE4a1OymHN/lz+NiiMxp7192olFZaIiAMzDIOpS/cQ0KwhY/sGmx2nTqmwREQc2JKdR9medoxHh4Tj6Va/L25XYYmIOKjKKiuv/bSXcN/G3Ngj0Ow4dU6FJSLioL7dnE5ydhFPXBWBq0uNJ4xwWCosEREHVFpRxZvL9tEtqBlDO/mZHccmVFgiIg5ozrpDZBwr5clhEWe9V1t9o8ISEXEwx0sreP+X/QwIb0X/9q3MjmMzKiwREQczZ91hWjX25MmrOpodxaZUWCIiDiSzsJR3lifRwa8xXQK9zY5jUyosEREH8tayfVRarTw1LNLsKDanwhIRcRD7Mo/z5cZUbuvbjuCWjcyOY3MqLBERB/Hq4j14ebrx0OAws6OYQoUlIuIA1h7IZfmeLB68MozmXh5mxzGFCktExM5ZrQaTF++mjXcD7rysndlxTKPCEhGxcz/uyGB72jEeHxpBA/f6PcHtuaiwRETsWFllFVOX7iHSvynXdw8wO46pVFgiInZszrrDpOaVMPHqjk4xwe25qLBEROzUsZIK3l2RxIDwVgzs4GN2HNOpsERE7NT0lfuJ8GvC08Ocawqms1FhiYjYoUO5RcxcfZDgFo2ICnCuKZjORoUlImKHJi/ag7urCxOuijA7it1QYYkAb775JlFRUXTu3Jlbb72V0tJS8vLyiIuLIzw8nLi4OPLz882OKU5iXXIuS3Yd5YEr2uPbtIHZceyGCkucXnp6Ou+88w4JCQns3LmTqqoq5s6dy5QpU4iNjSUpKYnY2FimTJlidlRxAlarwUs/JtLGuwH3Dgw1O45dUWGJAJWVlZSUlFBZWUlxcTFt2rRhwYIFxMfHAxAfH8/8+fPNDSlO4ZvNaexML+TJYR2d+iLhM1FhidMLCAjgiSeeIDg4GH9/f7y9vRk6dCiZmZn4+/sD4O/vT1ZW1hlfP2PGDGJiYoiJiSE7O9uW0aWeKSqr5LWle+ka1IwRXduYHcfuqLDE6eXn57NgwQIOHjzIkSNHKCoqYs6cOTV+/fjx40lISCAhIQEfH10rIxfvw18PkHW8jP93bSQuTn6R8JmosMTpLVu2jJCQEHx8fHB3d+fGG2/k999/x8/Pj4yMDAAyMjLw9fU1OanUZ0cKSpixKplro/3p2baF2XHskgpLnF5wcDDr1q2juLgYwzBYvnw5kZGRjBgxgtmzZwMwe/ZsRo4caXJSqc+mLtmD1YCnr9ZFwmfjZnYAEbP16dOHm266iR49euDm5kb37t0ZP348J06cYPTo0cycOZPg4GDmzZtndlSpp7YczmdH+jHuuTyEwObOdyfhmrIYhnGux8/5oIicKiYmhoSEBLNjiAOxWg1umP47RwtKWPbYFTRp6G52JFur8ck6HRIUETHRN5vT2JZawFNXd3TGsrogKiwREZMcL63g1SV76RHcjOu7Ofe9rmpC57BEREzy7or95BaVMeuOGA1jrwHtYYmImGB/1glmrT7ILTFBRAc2MzuOQ1BhiYjYmGEYvLgwkYYerjyh2dhrTIUlImJjK/Zk8eu+bB4d0oFWjT3NjuMwVFgiIjZUVlnFCwsTCfNtzO392podx6GosEREbOizdYexWg2eu64T7q56C74Q2loiIjaSll/M1KV76BzgzYBwTZR8oVRYIiI28sIPiViw8M9rO5kdxSGpsEREbOCXPVn8lJjJQ7FhBDRraHYch6TCEhGpY6UVVUz6YRehPl7cc7lue3+xNNOFiEgd++DXAxzKLWbO3X3wcNN+wsXSlhMRqUOHc4v518oDXBPtz+XhrcyO49BUWCIidcQwDCb9sAt3FwvPXqOBFpdKhSUiUkeW7c5ixZ4sHh3SgdbeDcyO4/BUWCIidaC4vJIvNhwmpm1z7risndlx6gUVlohIHXh7eRIr9mQx4aoIzWhRS7QVRURq2e6MQj5edZAxvYLoE9rS7Dj1hgpLRKQWWa0GE7/dQbOG7jx9dUez49QrNimszYfzeW3pHlusSkTEVJ9tOMzW1AKevbYTzRp5mB2nXrFJYa1OyuH9Xw6wYk+mLVYnImKKrMJSpi7ew+VhrRjZrY3ZceodmxTW/Ve0J9y3Mc/O30VRWaUtVikiYnPPL0ykrMrKi9d3xmKxmB2n3rFJYXm4uTBlVBeOHCth2k/7bLFKERGb+mVPFj9uz+ChQWGEtPIyO069ZLNBFz3btmBcn7b8+/eDbEstsNVqRUTqXHFZJc8u2EmYb2PGX6HJbeuKTUcJThgWgU8TT57+dgcVVVZbrlpEpM688fNe/Jp48vL1nfF0czU7Tr1l08Jq2sCd50d0ZndGITNXH7TlqkXOqaCggJtuuomOHTsSGRnJ2rVrycvLIy4ujvDwcOLi4sjPzzc7ptihrakFzFqTQkf/prrmqo7Z/DqsYZ1bc1WUH28t28eh3CJbr17kjB555BGGDRvGnj172LZtG5GRkUyZMoXY2FiSkpKIjY1lypQpZscUO1NeaeWpr7fj26SBrrmyAVMuHH5+RGfcXVx4Z3kShmGYEUGkWmFhIb/99ht33303AB4eHjRr1owFCxYQHx8PQHx8PPPnzzcxpdij6SsPsDfzOC9d35kmDdzNjlPvmVJYrb0bMGlkFN9sTuerhFQzIohUS05OxsfHhzvvvJPu3btzzz33UFRURGZmJv7+/gD4+/uTlZV1xtfPmDGDmJgYYmJiyM7OtmV0MdG+zOO890sSI7q2YUgnP7PjOAXTpma6oVsAfUNb8NLC3Rw9VmpWDBEqKyvZvHkzDzzwAFu2bMHLy+uCDv+NHz+ehIQEEhIS8PHxqcOkYi+qrAZPfr2dxp5uPHed7nNlK6YVlouLhVdHRVNhtfKP73bo0KCYJjAwkMDAQPr06QPATTfdxObNm/Hz8yMjIwOAjIwMfH19zYwpduTfv6ewNbWASSOiaNnY0+w4TsPUyW/btvRiwlUdWb4ni/lb082MIk6sdevWBAUFsXfvXgCWL19Op06dGDFiBLNnzwZg9uzZjBw50syYYicO5xbz+tK9DIrwYURXTb9kS25mB7ijfzsW7chg0veJXBbWCt8muiun2N67777L2LFjKS8vJzQ0lE8++QSr1cro0aOZOXMmwcHBzJs3z+yYYjLDMJj43XZcLPDyDV00/ZKNWc5zKM4mx+n2Z51g+DurGBzhy/RxPfRDIA4rJiaGhIQEs2NIHflq42HmrD/MzTFB3Na3rdlx6osav+Hbxf2wwnwb8/chHViy6yiLdhw1O46IyGnS8ot5YeFuGrq7MLZ3kNlxnJJdFBbAvQNCiA705v8t2EnuiTKz44iIVLP+d1SgYRi8fnM3XFzs5q3TqdjNVndzdWHqTdFUWa3865cDZscREak2Z/0hfj+Qyz+u6URQi0Zmx3FadlNYAB1bN+X+K9ozc81BFm4/YnYcERFScoqYvGgPAzv4cKsOBZrKrgoL4J4BoXQNasY/5+8kq1AXFIuIeaqsBhO+3oabq4VXR2lUoNnsrrDcXF2YdnNXSsqrePpbXVAsIub5ZM1BNqbk89x1Ufh7NzQ7jtOzu8KCk6MGnxrWkRV7sjTXoIiYYn/WCaYu3cuQSF9G9QgwO45gp4UFJy8o7hfakhd+SCQ1r9jsOCLiRCqrrDw+bxuNPFx55UYdCrQXdltYLi4WXrs5GovFwhPztmG16tCgiNjGp2sPsS21gBdGdtbsO3bEbgsLILB5I/7ftZ1YfzCPT35PMTuOiDiBnenHeGVRInddFsJ10f5mx5H/YdeFBXBzTCCxHX35cuNhko4eNzuOiNRjJeVVPDx3C60aN+Dh2DAdCrQzdl9YFouFKTdGc6ykgke+3EpZZZXZkUSknnp5USLJ2UVMG92VZo08zI4jf2L3hQXg09STl6/vQmJGIdN+2md2HBGph1bsyWTOusPcOyCEy8JamR1HzsAhCgtgSCc/xvYJZsZvyazZn2N2HBGpR3JOlPHk19vp2LoJT1wVYXYcOQuHKSyAf17TiVAfLx7/ahv5ReVmxxGResAwTk5sW1hayTu3dsfTzdXsSHIWDlVYDT1ceWdMd3KLynjmO82CISKX7rP1h1mxJ4uJV3ekg18Ts+PIOThUYQF0DvDmiaERLN55lHkJaWbHEREHtj/rBC/9mMiA8FbE92tndhw5D4crLIB7B4TSv31LJv2wi4M5RWbHEREHVFZZxTvLk2jk4ca0m7vi4qIh7PbOIQvLxcXCtNFd8XC18PayfZRXWs2OJCIOZsriPXy/7QjTbo7Gt6lms3AEDllYAP7eDXn95m7M33qEqUv2mB1HRBzIz4mZfLImhTv6t2NQRz+z40gNOWxhwcmh7rf3a8vHqw+yLDHT7Dgi4gDSC0p4Yt42oto0ZeLwjmbHkQvg0IUF8MzwSDr5N+WJr7dxpKDE7DgiYscqq6w88sUWKqusvPeXHhrC7mAcvrAauLvy3l+6U1Fp5ZG5J38QRUTO5M1l+0g4lM8rN3YhpJWX2XHkAjl8YQGE+jTm5Ru6sDEln7eWJZkdR0Ts0KqkbP618gCjYwIZ2U03ZHRE9aKwAK7vHsDomEDeX7mf1UmauklE/k/W8VL+/uVWwnwaM2lElNlx5CLVm8ICmDQiijCfxjz65VYyC3U+S0TAajV47MttHC+t5L2/9KCRh5vZkeQi1avCauThxvtjexDm68WTX+/Q+Sy5IFVVVXTv3p1rr70WgLy8POLi4ggPDycuLo78/HyTE8rFmL5yPwUl5UwaEUVEa0295MjqVWEBdPBrwk09g/h1XzbTftatSKTm3n77bSIjI6v/PmXKFGJjY0lKSiI2NpYpU6aYmE4uxi97s3j9532E+zZmTK8gs+PIJap3hQVwU89Abu0dzPSVB/hp11Gz44gDSEtL48cff+See+6pXrZgwQLi4+MBiI+PZ/78+Salk4uRmlfMo3O3EuHXhFduiNbdg+uBellYAM9d14kuAd48Pm8bh3I136Cc26OPPsrUqVNxcfm/X4nMzEz8/f0B8Pf3Jysry6x4coFKK6p44LNNWA2DD8b1pKGHrreqD+ptYTVwd+VfY3vgYrFw/5zNlFZUmR1J7NTChQvx9fWlZ8+eF/X6GTNmEBMTQ0xMDNnZ2bWcTi6UYRg8O38nO9MLeXN0N9rpeqt6o94WFkBQi0a8NaYbe44W8s/5O3X/LDmjNWvW8P3339OuXTvGjBnDihUrGDduHH5+fmRkZACQkZGBr6/vGV8/fvx4EhISSEhIwMfHx5bR5Qzmbkxl3qY0HhocxpBOmiewPqnXhQUwKMKXhwaH8/WmNL7cmGp2HLFDkydPJi0tjZSUFObOncvgwYOZM2cOI0aMYPbs2QDMnj2bkSNHmpxUzmdbagHPLdjFgPBWPDqkg9lxpJY5xQUJj8SGc/RYCdN+3keH1k3oEdzc7EjiAJ5++mlGjx7NzJkzCQ4OZt68eWZHknPIKyrngTmb8GniyTtjuuOq+1vVO5bzHCarN8fQCorLGfHeGkorqvjhocvx0/1vpA7ExMSQkJBgdgynU2U1iJ+1gQ0peXx9fz+iA5uZHUlqrsafLOr9IcE/NGvkwUe3x3CirJL7/rOJskoNwhCpL6b9tJfV+3N4cWSUyqoec5rCAoho3YQ3Rndla2oBz2oQhki9sGhHBr/vz+X2fm25pVew2XGkDjlVYQEM6+zPw4PD+CohjU/XHjI7johcgp3px3jsq624uMA/rok8/wvEoTldYQE8OqQDQyJ9eWFhImsP5JodR0QuQlZhKfd+mkCLRh58eFuMbsboBJyysFxcLLx5SzfatWzEXz/fTFp+sdmRROQClFZUMf4/mygoruCj+Bh8mniaHUlswCkLC6BJA3c+uj0Gv6aePPXNdk6UVZodSURqwDAMJn67g62pBbx5S1ei2nibHUlsxGkLC07eqfjpYR1Zl5zHw19socqqQRgi9u6DX5P5bks6TwztwLDO/mbHERty6sICuCLCl0kjolixJ4uXfkw0O46InMPPiZlMXbqH67q24a+DwsyOIzbmFDNdnM9tfduSklPEzNUHCW3lxW392pkdSUT+ZHdGIY/M3UKXAG9eu0m3C3FGKqz/emZ4JIdyi5j0QyJBLRpxZcSZJzoVEdvLOl7KtKV78W548txzA3eNCHRGTn9I8A+uLhbeHtOdCL8m/O3zLew9etzsSCICFJVVcte/N7LmQC4f3d5T06o5MRXW//DydGPmHTE08nDlrn9vJPt4mdmRRJxaZZWVv32+mcQjhbw/tjudA5qZHUlMpML6E3/vhsyM70XuiTJeWbSb4nINdxcxg2EYPLtgF7/szebF6zszuKPubeXsVFhn0CXQmw9u68mCrek89PkWKqusZkcScTrTfz3AFxsO88CV7Rnbp63ZccQOqLDO4soIX14Y2Znle7J0t2IRG1uwNZ2pS/YyomsbJgyNMDuO2AmNEjyHcX3bkllYyrsr9uPXtAF/j9MdTEXq2toDuTwxbxt9Qlrw2s3RuOhGjPJfKqzzeCyuA0ePlfL28iRaezfg1t66fYFIXUk8Usj0lftp19KLGZrQVv5EhXUeFouFV27sQvaJMv7x3Q5aNfYgrlNrs2OJ1DuHcou4fdYG3F0tfP1Af7wbuZsdSeyMzmHVgLurC+//pQeDInx5aeFu1ifrliQitSmrsJRxM9dTZbXyn7t7E9CsodmRxA6psGrIy9ON127uirubC3fPTmBH2jGzI4nUC8eKK7ht5gZyT5TzyZ29CfNtYnYksVMqrAvQwsuDOXf3oVkjd+I/2cD+LM2GIXIpissruWv2Rg7mFDHjthi6BTUzO5LYMRXWBWrt3YA5d/fBxWJh3McbSM3TzR9FLkZ5pZUH5mxmy+F83h7TjcvDW5kdSeycCusitGvlxZx7elNSUcW4mevJKiw1O5KIQ6mssvLiwkQ2HcrnlRu6cHUX3ddKzk+FdZE6tm7KJ3f2Ivt4GU99s538onKzI4k4hCqrwYSvt/OfdYd4clgHxuhSEakhFdYl6BHcnI/jY9iWdoyxH6+noFilJXIuVqvBxG+3892WdCZcFcHt/ULMjiQORIV1ifq3b8Ubo7uyP+sE42au51hxhdmRROzSyclsd/JVQhoPDw7THYPlgqmwasGVEb58eFtP9h09wW2z1nOsRKXlSFJTUxk0aBCRkZFERUXx9ttvA5CXl0dcXBzh4eHExcWRn59vclLHZRgGz/+QyGfrD3PfFaGa5kwuigqrlgzq6Mv0cT3YnVHI7bM2UFiq0nIUbm5uTJs2jd27d7Nu3Tref/99EhMTmTJlCrGxsSQlJREbG8uUKVPMjuqQDMPg/V/28+/fU7jzsnY8Payjbm8vF0WFVYtiI/14/y892JV+jPhZGziu0nII/v7+9OjRA4AmTZoQGRlJeno6CxYsID4+HoD4+Hjmz59vYkrHZBgGLyxM5PWf9vHokHD+37WdVFZy0VRYtWxoVGve+0sPdqQd47kFu7Sn5WBSUlLYsmULffr0ITMzE3//k8Ot/f39ycrKMjmdY7FaDf45fyefrEnh7stDeCQ2XGUll0SFVQeGdW7NjNt68v22I4z9aL2GvDuIEydOMGrUKN566y2aNm1a49fNmDGDmJgYYmJiyM7OrsOEjqPKavD0t9v5bP3JGzD+85pIlZVcMhVWHRkc6ceM23uyN/M4Y2asI+u4Li62ZxUVFYwaNYqxY8dy4403AuDn50dGRgYAGRkZ+Pr6nvG148ePJyEhgYSEBHx8fGyW2V5VVll5Yt42vkpI45HYcJ68KkJlJbVChVWHBnf045M7enE4r5hbPlzHkYISsyPJGRiGwd13301kZCSPPfZY9fIRI0Ywe/ZsAGbPns3IkSPNiugwyiuqeOTLrdXXWf09roPKSmqN5Ty3ftd94WtBQkoed36ykaYN3fn83j60belldiT5H6tXr2bAgAF06dIFF5eTn+FeeeUV+vTpw+jRozl8+DDBwcHMmzePFi1anPNrxcTEkJCQYIvYdqekvIoHP9tEUXkVQyJ9GT+wvdmRxDHU+BONCstGdqQd47aZ6+gW3Jynr+5Ix9Y1P0cijsNZC+tYSQX3zN5IwqF8XhgRxW392pkdSRxHjQtLhwRtpEugN/Pu78/ujEJGf7CWjSl5ZkcSqRVZx0sZM2MdW1MLePfW7iorqTMqLBsK92vCNw/0p1UTT8Z9vJ6fEzPNjiRySQ7nFnPzB2tJySliZnwvro1uY3YkqcdUWDYW2LwRX9/fn47+TbnvPwl8tTHV7EgiF2VH2jFun7Wepg3c+ezePgzsoBGSUrdUWCZo4eXB5/f04fJwH/61cj/vLE/iPOcSRezKij2ZjP5wLRVVBm+MjqZHcHOzI4kTUGGZxMvTjY9vj2FABx/e+HkfT8zbTnml1exYIuf1+frD3DM7gfa+Xnz31/6E+2kAkdiGm9kBnJmHmwsvjIiilZcnby7bx5GCEj4Y1xPvRu5mRxM5jdVq8PpPe/nXygMMivDhvb/0wMtTbyFiO9rDMpnFYuGRIeG8eUtXEg7lceP0NaTmFZsdS+QUJeVVPP3tdr7YcJhbewfz0e0xKiuxORWWnbiheyD/ubsPOSfK+cf8HRr2LnYj41gJoz9cy7xNaTw6JJxXbuiMm6veOsT29FNnR/qGtuS7B/tzpKCUv3y0js/XHzY7kji5zYfzGfHeGpKzT/Dx7THE9w/RVEtiGhWWnQn1acw3D/Snf/tWPPPdDv45f4cGY4gpvt2cxpgZ62jo7sp3f72M2Eg/syOJk1Nh2SHvhu7MuqMX910Rypx1hxk3cz05J8rMjiVOorLKyjvLk3jsq230CG7Ggr9eRge/JmbHElFh2StXFwsTr47k7THd2JZawA3vr2FbaoHZsaSeyz5exm0zN/DhrwcYPzCU/9zdh+ZeHmbHEgFUWHZvZLeA6pkxbv5gLf9Zd0gXGUudSEjJ49p3V7H5cD7Pj+zMM8MjcdfgCrEj+ml0AF0CvXl1VDT9w1ry7PydPDJ3KyfKKs2OJfWEYRh8vCq5+nzV/L9exk09A82OJXIaFZaDaOHlwaz4Xky4KoKF248w4r3V7D163OxY4uCOFVfw/A+JvPTjbgZ39OX7hy4n0l8zV4h90v2wHNDvB3J4+IutBDZvyM0xgfyld7CGGtsJR7of1vrkXP7+5Vbyisp5Zngkt/Vrq58jMYPuh1Wf9W/fikWPXI6/dwP+8d1O7v10E7kaRSg1VFFlZdpPe7n1o3V4uLnw5X39uL1/O5WV2D3tYTkwq9Vg1pqDTF2yl6YN3Xnt5mgGRfiaHcup2fse1qHcIh6Zu5WtqQXc3DOQSSOiNMWSmK3Gn5RUWPXA7oxCHp27lb2Zx4nv15aJwyNp4O5qdiynZK+FZRgG3287wjPf7sDFxcLkG7voZotiL2pcWPpoVQ9E+jdlwd8uY+qSvcxac5DU/BIeGhxGd92jSICjx0r5x3c72HI4nz6hLXnx+s4ENGtodiyRC6Y9rHrmt31ZTF26l8Qjhdx1WQiPD42goYf2tmzFnvawDMPgq4RUXlq4mwqrlQlXdeSO/u1wddG5KrEr2sNyVgM7+NI9uDlTFu/h49UH+SkxkymjutC/fSuzo4kNpeUXM/HbHaxKyqFPSAteHRVNu1ZeZscSuSTaw6rH1iXn8vQ320nJLeYvvYN5clgEzRppmp26ZPYeVmWVlU/XpvDp2kNkHS9j4tUdGdunLS7aqxL7pUEXclJJeRVvLdvH9rQC9maeYMJVEYyOCdJhoTpiZmFtOpTPP+fvZHdGIWN6BfHXQWEEtWhkShaRC6DCklMlHilk0ve72JCSR5cAbyaNiKJnWw3KqG1mFFbeiTKmLt3L3I2ptG7agOeu68Swzq11XZU4ChWWnO6Poc2TF+3haGEpN/YI4OlhHfFt2sDsaPWGLQurrLKK/6w9xJx1h8grLufWXsE8HBuu66rE0WimCzmdxWJhZLcAlj9+BQ9e2Z6F2zIYPO1XZq46SFllldnx7NKSJUuIiIggLCyMKVOmmB0HOPnBY9GODOLe+I2XftxN25ZefPtAfyYOj1RZSb2mPSwndjCniBcXJnK8tIL0/BIeGRLOqB6BuOmWEgBUVVXRoUMHfv75ZwIDA+nVqxdffPEFnTp1Outr6noPa/OhfF5ZtJuEQ/lE+DXhmWsiuaKDT52tT8QGNKxdzi+klRez7ujF6qRsXvtpH099s4MPf03m0bgOXNvF3+lHlm3YsIGwsDBCQ0MBGDNmDAsWLDhnYdWVnenHeGvZPjKOlZJZWMbkG7twc099uBDnosISLg/34bKwVizbncXrS/fy8Bdb+Ncv+3liaASxkb5Oe/I+PT2doKCg6r8HBgayfv16m2ZIPFLIW8v28VNiJt4N3bl3QAh39A+hcQP96orzOechwWHDhhk5OTm1sqLs7Gx8fBzv0IWj5oaLz15QXEFmYSlVVgMPNxdaNfbEu5F7zffba4E9bPf8/HwKCwtp27YtALm5uRQVFREcHHzK87Kzs/nj96SsrIxu3bpd8rpLyqvIKSqjoLgCF4sFn8aetGzigWsdfniwh21+MRw1Nzhu9trMvWnTpqWGYQyryXNtdg7L7AsqL5aj5oZLy15RZWXxzgzeWpZEcnYRAc0actflIdzSK4jGNjixbw/bfe3atUyaNImlS5cCMHnyZAAmTpx41td4eXlRVFR0UeuzWg1+2ZvFR6uSWZecxxUdWtE1qDl3Xx6Cd0P3i/qaF8IetvnFcNTc4LjZazm3zmHJpXF3dWFE1wCu7dKG5Xuy+Oi3ZF5cmMjby/Yxtm9b7uzfrt4Ph+/VqxdJSUkcPHiQgIAA5s6dy+eff17r6yktr2T+1iN8tCqZA9lFtPFuwD+GR3JL7yCaNqj7ohJxFCosOScXFwtxnfyI6+THlsP5fLQqmQ9/PcDPiZlEtG7CzT0DGRDuUy9nznBzc+O9997jqquuoqqqirvuuouoqKha+/r7s47zxYZUftuXzYHsE0T6N+XtMd0Y3sUfdw2mEDmNzQpr/PjxtlpVrXLU3FD72bsHN+dfY3tyKLeIH7YdYebqg/y4PYPWTRswqmcAN/UMIqSWJli1l+0+fPhwhg8fXuPnt2p17kmGS8qr+HFHBnM3HCbhUD7urhaGdmrNKzd0IaZdc1MHuNjLNr9QjpobHDe7Wbl1HZZctLLKKpbvzmJeQiq/7svGakCvds25uWcQw6P9bXKuy96c6dh+ZZWV3w/k8uP2DDam5JGcU0RoKy/G9A7ixh6BtGrsaVJaEbugqZnEtjILS/lmcxpfJ6SRnFNEB7/GtPdpTGykH4MifGjpJG/KfxRWZZWVDSl5LNyewZKdR8krKqexpxu39WvLFR186BPSwmkvFxD5E3OmZpo3bx5RUVG4uLic9ilz8uTJhIWFERERUT3q6s/y8vKIi4sjPDycuLg48vPzazNejdxyyy1069aNbt260a5du7MOUW7Xrh1dunShW7duxMTE2DbkWUyaNImAgIDq/IsWLTrj8+piuiG/pg148Mowlj9+Bd880I9ruviz6VA+T8zbRszLy7hp+u9MX3mApMzj/PlD0oQJE+jYsSPR0dHccMMNFBQUnHEd9rTNz7QNMwtLyS8u56EvtjDo9ZWMnbGWz9ckUZi0kSf7Nyfhn0N4alhH+oa2NKWsUlNTGTRoEJGRkURFRfH222+f9pyVK1fi7e1d/TP0wgsv2Dzn2Zzv+28YBg8//DBhYWFER0ezefNmE1Kebu/evdXbs1u3bjRt2pS33nrrlOfYy3a/66678PX1pXPnztXLavq+bJNpzAzDONd/FyQxMdHYs2ePccUVVxgbN26sXr5r1y4jOjraKC0tNZKTk43Q0FCjsrLytNdPmDDBmDx5smEYhjF58mTjySefvNAIteqxxx4znn/++TM+1rZtWyM7O9vGic7tueeeM1577bVzPqeystIIDQ01Dhw4YJSVlRnR0dHGrl276iRPVZXV2Jaab0z7aa8x/O3fjLZPLTTaPrXQGPDqCmPqkj3Gou1HjOzjpcbSpUuNiooKwzAM48knnzzr991etvkf23DX3n3Gil1HjPCbnzIGTl5qtH1qoeHROsyIeelnY/S0hUb/0Q8YJ0rLjbVr1xq9e/c2O7Zx5MgRY9OmTYZhGEZhYaERHh5+2vf+l19+Ma655hoz4p3X+b7/P/74ozFs2DDDarXazTb/s8rKSsPPz89ISUk5Zbm9bPdff/3V2LRpkxEVFVW9rCbvy5f4vnK+Hqr+r1ZPMkRGRp5x+YIFCxgzZgyenp6EhIQQFhbGhg0b6Nev32nPW7lyJQDx8fFceeWVvPrqq7UZscYMw+Crr75ixYoVpqy/rthyuiEXFwvRgc2IDmzGY3EdyDhWwvLdWazYncm8hFTeP14GQHsfLxIq99G2pRcBXfqxdvHXtZ7lUhmGweG8YnZnFPL92l24D5/IQz+kcyC7CJfQy6g8nsfEq3vx7uLGbHgmlvvvv5/rr78SL093+vbtS0FBARkZGfj7+5v2b/D3969ef5MmTYiMjCQ9Pd2UqabqwoIFC7j99tuxWCx2s83/bPny5bRv3776YnR7M3DgQFJSUk5ZVpP3ZVu9r9jkrHh6ejp9+/at/ntgYCDp6emnPS8zM7P6h8vf35+srCxbxDujVatW4efnR3h4+Bkft1gsDB06FIvFwn333Wc3o33ee+89Pv30U2JiYpg2bRrNm596zyszpxvy927IuL5tGde3LeWVVnYeOcaGg3msT87lp12ZpOQWA+6ERNzENe+s4rKwVjRv5EG4X2OCmjcEN4863+aVlVYyCks5lFtMUtZxkrJOcCi3iO1pxzheWklIKy9Sc0tp7OHOkEg/nr22JQfWLWXLxvXcd8UYPnJ3xWKxnHE7p6en282bZ0pKClu2bKFPnz6nPbZ27Vq6du1KmzZteP3112t1KP+lON/vnL1vc4C5c+dy6623nvExe93uNXlfttX7ygUX1pAhQzh69Ohpy19++WVGjhx5xtcYZxjYYeYJ55r8G7744ouz/mABrFmzhjZt2pCVlUVcXBwdO3Zk4MCBdZb5D+fK/sADD/Dss89isVh49tlnefzxx5k1a9YpzzPre3Gu3J/cORLDMHj6pddZn3SUUXfcz4aUfLalFrD+YB4tvTxwc7HATW8SHeFDcUkZU9euYxs/E9YuGJ8mnjRp6E4jD1c83Vxo4OaGq8vJi58NDKwGVFRaKa+yUlpppbS8iqLySgpLKkjLL8HVxcK2tGN4urmw52gheUUVtGvRiJS8YrwbutPBrzGjY4II921MpzZN2bXmZ5b/vIqJw+MBSN1w+ja0t5/5/3XixAlGjRrFW2+9RdOmTU95rEePHhw6dIjGjRuzaNEirr/+epKSkkxKeqrz/c7Z8zYHKC8v5/vvv6+eMeV/2fN2rwlbbfsLLqxly5Zd8EoCAwNJTU2t/ntaWhpt2rQ57Xl+fn7Vu/AZGRn4+vpe8Lpq4nz/hsrKSr799ls2bdp01uf8kd/X15cbbriBDRs22KSwarr97733Xq699trTltf0e1Hbzpf7008/5bdF37J8+XIaNfq/27oXllaQnHWCIwWlJOecoLTCSnpBCUmt27Eh5RirjqTQtmUjGnq4UVpRhburBTcXF8qrrLhaLLi6gIEFq2FgtRrVFzhbgEqrwdFjpUQHNcPd1ULnAG+6BXnTtqUXbVt6EerjhU9jz9N+8YqCz78NzdrO51NRUcGoUaMYO3YsN95442mP/2+BDR8+nAcffJCcnJzzXl9mC+f7nbPXbf6HxYsX06NHD/z8/E57zJ63e03el2227c9zkuui/HnQxc6dO08ZdBESEnLGQRdPPPHEKSf3JkyYcLERLsnixYuNgQMHnvXxEydOGIWFhdV/7tevn7F48WJbxTurI0eOVP/5jTfeMG655ZbTnlNRUWGEhIQYycnJ1SdHd+7cacuYp1m8eLERGRlpZGVlnfU5Z9vmlZVVRv6JMuNwbpGReKTA2JySZ6xLzjFWJWUZv+3LMn7be/L/v+/PNhJSco2daQXG/qxCI6uw1CivOP1nsCbOtQ179uxpGIZhLFy48JQBAL169bqoddUmq9Vq3HbbbcYjjzxy1udkZGQYVqvVMAzDWL9+vREUFFT9dzPV5HfOHrf5/7rllluMWbNmnfExe9ruBw8ePGXQRU3ely/xfaXGgy5qtbC+/fZbIyAgwPDw8DB8fX2NoUOHVj/20ksvGaGhoUaHDh2MRYsWVS+/++67q8stJyfHGDx4sBEWFmYMHjzYyM3NvdAItSI+Pt6YPn36KcvS09ONq6++2jAMwzhw4IARHR1tREdHG506dTJeeuklM2KeZty4cUbnzp2NLl26GNddd111gf1vdsM4OZoqPDzcCA0NtYvs7du3NwIDA42uXbsaXbt2Ne677z7DMOx7m59pG06fPt0IDg42DONkOTz44INGaGio0blz51M+wJll1apVBmB06dKlelv/+OOPxvTp06t/3t99912jU6dORnR0tNGnTx9jzZo1Jqc+6Wzf///Nbo/b/A9FRUVGixYtjIKCgupl9rjdx4wZY7Ru3dpwc3MzAgICjI8//vis78u1+L5S48LShcMitchRZ98WMZE5Fw6LiIjUFRWWiIg4BBWWiIg4BBWWiIg4BBWWiIg4BBWWiIg4BBWWiIg4BBWWiIjYzNSpU3nnnXcA+Pvf/47FYlkBYLFYYi0Wy5xzvfZ8Fw6LyAWwWCxLDMMYZnYOEXtlsVj6Ao8bhnGzxWJZBXgClwHPAEcNw/jwbK/VHpZILVJZiZzXJqCnxWJpApQBa4EYYACw6lwvtMn9sERERAAMw6iwWCwpwJ3A78B2YBDQHth9rtdqD0tERGztN+CJ//5/FXA/sNU4zzkqFZaIiNjaKsAfWGsYRiZQynkOB4IGXYiIiIPQHpaIiDgEFZaIiDgEFZaIiDgEFZaIiDgEFZaIiDgEFZaIiDgEFZaIiDgEFZaIiDiE/w+nNHQACEDmawAAAABJRU5ErkJggg==\n", 118 | "text/plain": [ 119 | "
" 120 | ] 121 | }, 122 | "metadata": { 123 | "needs_background": "light" 124 | }, 125 | "output_type": "display_data" 126 | } 127 | ], 128 | "source": [ 129 | "sympy.plotting.plot(f);" 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "metadata": {}, 135 | "source": [ 136 | "A neat parabola. We can see from the plot that the minimum of $f(w)$ is reached somewhere between $w=-2.5$ and $w=0$. SymPy can tell us the derivative, and the value where it is zero:" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 5, 142 | "metadata": {}, 143 | "outputs": [ 144 | { 145 | "data": { 146 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAADwAAAAQCAYAAABKkhw/AAAACXBIWXMAAA7EAAAOxAGVKw4bAAACn0lEQVRIDdWX61EbQQyAD8YF8OgAOgCnA9MBhA6ggzD8sv9loAOSChjoIO4gQAfQQTzuwPk+ze6xdzmG4IwPohkhraRdPVbaw2uLxaKaTCY7VVWdgcIQnIFnyB8U/E+QcjlNMW9Aze0C+VTZ2ng8VnCF4ECBAH8B+QIeZEPlHx2I1QRNLidcwR8iuwGP4G/XNQBrA3iNvO05qGGvgO8ReLKkU/edsN8kM8TNsjhXYMIj8BEjq1OChhvI7YA+wTjasfytf0fQixIDiL/mFQxAE9trK1QmWNZ53t8bJQdz2SwdIsu3faV8gOCoNCj4PXn08XBBTdyW2QbtiG/QAHhtT6ExGlCdOCr78I0Kx4ae/uDb7o2RhY94bek/AKUJ2Mr55dbmHPkl9CfoISU4H2Xrm6TrYWnUF2/8oI+uF+CF3WXfnQmj9LG6TQlWHsDaRAVf81lwz3+s5I+8xN7W0tFTlvVJ8f8AXoJ27zV4Dx+t7Qw3AIW9/pSMs851tDaCz+DXrEBuMWx3kyzhDt2LCaPTj4Vqw5YC9I0vRzIykZdGsH1OrLH34uy4G+hmI2EEzugWtP4mu4t1zCHUKplcPb/wBj1HlwvCMuAxM10U+66EquRjB+r4vAnYY/E9ox2LLW2co7qlMTKZXWhdQXgdl7NpIaxyFABeUNa4XfQ6bjvVdtVwjwPb10vphEg4BfgJWj5SbrAIs2KnybfbtEt2zFmNIhRnrJL1InILl36GaTEdEJgB+0hN4eNbVViOkJWt5a1F22iDzjZx5jwjIMnqByzLe6LtCzPGPIZ+Nuf+L+2s1QG3ArN990sZ61wUq/kLdJ6/g3aCssY3mvWbIAW41AzriP1eQj2W8Ob2/OPBX0sfCf414ddyWX/N4B30dom4EvgNCwYAZwDIeMAAAAAASUVORK5CYII=\n", 147 | "text/latex": [ 148 | "$\\displaystyle 2 w + 3$" 149 | ], 150 | "text/plain": [ 151 | "2⋅w + 3" 152 | ] 153 | }, 154 | "execution_count": 5, 155 | "metadata": {}, 156 | "output_type": "execute_result" 157 | } 158 | ], 159 | "source": [ 160 | "fprime = f.diff(w)\n", 161 | "fprime" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 6, 167 | "metadata": {}, 168 | "outputs": [ 169 | { 170 | "data": { 171 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAACsAAAAzCAYAAAAO2PE2AAAACXBIWXMAAA7EAAAOxAGVKw4bAAACw0lEQVRoBe2a31HjMBCHk5srIFwJoQP+dBA6IC1ACcw9Ja/QAbQQOgA6IHQAJcB1cPf9fBYjyZYszTBrHrwzGknrtfV5vVopmsw3m81i1iPb7fZPj9pExdi9TD8YfU/5iMq1CVV6kF3EI77dHM++0rjkbR7T9457BbYLCNY/LTEYdMl4l+2Y+tTqX5c6ygwWIMFdUTvYGe1zdA/Ua8o97awoZq1En/KiBXRjutD77RS52hL2BRBlGJVGAP9sO12utgwDefHAh/G8fOvrU21LzwYMgK5QKEUqE90FFxMdM8+68QE7oi3QU4pC45lSJGPAClBlBriywZ7622UD8QUCpNKVJtmO9iK42NMxi1lgjlR6GFwYKDSyYgYLhfYg+uSDHkwRW8asPvcjsKp9OWk7boHwrwVtS9irYGQ6gGuCydNKX/FLxOYzM1hg7igrir8ALCE6QzfoVZGbwWqwFqoITPaxWE6weOzq/gRb7bLCGybPFjqq2mzybLXLCm/I5lnyolaXJ4rqUtF2r9kCuhvo/3XtoRrbecpmCFZL4HHq5lJ9DqD0GbKbYrbGWzW2k2drvFVjm51gNQ8qsWWiLbFz+1ptut/VRx9kj9SzzGBb0FvqMwdDW+cG+qlTtKe1jNnmQMOBqgZSXlZ61HnsoFjCrqB5BTBeYLQZX6BXiGTFElZQb0DJk30Sv0THxjJm153R/yuaswReYnCSWXq2wwqgQPX5XYbo2PiKUWEB0cS6B/rGh0q1R4MFUD/JFcOp8OgwjwILoI7sf1F/5twOWY/CHBZAncIc+h6lvVTp4QtUprAAaUKdUscTSi+gpTcrlqlLntOE0uGcf4QkQB0rDU4yM1iAHigCVrzGMphjdYMZLJ47jAlr+6YxWwsX20+wsUe+qu9iVnku2PXQT+2Ovmrs5HNiFgwbNhcGSiX+HyS+258jGp5/R7zM+TmH0KYAAAAASUVORK5CYII=\n", 172 | "text/latex": [ 173 | "$\\displaystyle \\left[ - \\frac{3}{2}\\right]$" 174 | ], 175 | "text/plain": [ 176 | "[-3/2]" 177 | ] 178 | }, 179 | "execution_count": 6, 180 | "metadata": {}, 181 | "output_type": "execute_result" 182 | } 183 | ], 184 | "source": [ 185 | "sympy.solve(fprime, w)" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "That looks about right: $-3/2$ or $-1.5$. \n", 193 | "We could have also solved this by hand, because it's a simple function. \n", 194 | "But for more complicated functions, finding the minimum analytically could be more difficult. \n", 195 | "Instead, we can use the iterative method of gradient descent. \n", 196 | "\n", 197 | "The idea in gradient descent is to find the value of $w$ at the function minimum by starting with an initial guess, then iteratively taking small steps down the slope of the function, i.e., in the negative gradient direction. \n", 198 | "To illustrate the process, we turn the symbolic expression `fprime` into a Python function that we can call, and use it in a simple loop taking small steps:" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 7, 204 | "metadata": {}, 205 | "outputs": [ 206 | { 207 | "data": { 208 | "text/plain": [ 209 | "function" 210 | ] 211 | }, 212 | "execution_count": 7, 213 | "metadata": {}, 214 | "output_type": "execute_result" 215 | } 216 | ], 217 | "source": [ 218 | "fpnum = sympy.lambdify(w, fprime)\n", 219 | "type(fpnum)" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "Yep. We got a Python function with the [`sympy.lambdify()`](https://docs.sympy.org/latest/modules/utilities/lambdify.html) method, whose return value is of type `function`. Now, you can pick any starting guess, say $w=10$, and advance in a loop taking steps of size $0.01$ (a choice we make; more on this later):" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 8, 232 | "metadata": {}, 233 | "outputs": [ 234 | { 235 | "name": "stdout", 236 | "output_type": "stream", 237 | "text": [ 238 | "-1.4999999806458753\n" 239 | ] 240 | } 241 | ], 242 | "source": [ 243 | "w = 10.0 # starting guess for the min\n", 244 | "\n", 245 | "for i in range(1000):\n", 246 | " w = w - fpnum(w)*0.01 # with 0.01 the step size\n", 247 | "\n", 248 | "print(w)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "That gave a result very close to the true value $-1.5$, and all we needed was a function for the derivative of $f(w)$. This is how you find the argument of the minimum of a function iteratively. \n", 256 | "\n", 257 | "##### Note\n", 258 | "\n", 259 | "> Implied in this method is that the function is differentiable, and that we can step *down* the slope, meaning its second derivative is positive, or the function is _convex_.\n", 260 | "\n", 261 | " \n", 262 | "\n", 263 | "#### Gradient descent steps in the direction of the negative slope to approach the minimum." 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "## Linear regression\n", 271 | "\n", 272 | "Suppose you have data consisting of one independent variable and one dependent variable, and when you plot the data it seems to noisily follow a trend line. \n", 273 | "To build a model with this data, you assume the relationship is _linear_, and seek to find the line's slope and $y$-intercept (the model parameters) that best fit the data. \n", 274 | "\n", 275 | "Though this sounds straightforward, some key ideas of machine learning are contained:\n", 276 | "\n", 277 | "- we don't _know_ the true relationship between the variables, we _assume_ it is linear (and go for it!)\n", 278 | "- the model we chose (linear) has some parameters (slope, intercept) that are unknown\n", 279 | "- we will need some data (observational, experimental) of the dependent and independent variables\n", 280 | "- we find the model parameters by fitting the \"best\" line to the data\n", 281 | "- the model with its parameters can then be used to make _predictions_\n", 282 | "\n", 283 | "Let's make some synthetic data to play with, following the example in Eric Ma's tutorial [1]." 284 | ] 285 | }, 286 | { 287 | "cell_type": "code", 288 | "execution_count": 9, 289 | "metadata": {}, 290 | "outputs": [ 291 | { 292 | "data": { 293 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAX70lEQVR4nO3db4xcV3nH8d/jzYoMgmYdxaTOEuOoQg6QtLZYISRLLXFIk0JKDLywqEojUdV9AVIchVQLvCBVVWE1lPCiVSu3oKYqalMpIYkSqtSN3SKsNmWdmIDlpEEF2mzcxJSsAHmVru2nL2bGzM7eM3Pu37l37vcjRbs7O7tzVpF+c/yc55xj7i4AQPNsmvQAAADZEOAA0FAEOAA0FAEOAA1FgANAQ11S5YtdccUVvn379ipfEgAa7/jx4z909y3Dj1ca4Nu3b9fS0lKVLwkAjWdmP0h6nBIKADQUAQ4ADUWAA0BDEeAA0FAEOAA0VKVdKADQNg8/s6x7n3heL62s6qq5ju6+eYf27pov5HcT4ABQkoefWdanHvq2VtfOS5KWV1b1qYe+LUmFhDglFAAoyb1PPH8xvPtW187r3ieeL+T3E+AAUJKXVlZTPZ4WAQ4AJblqrpPq8bQIcAAoyd0371BndmbdY53ZGd19845Cfj+LmABQkv5CJV0oANBAe3fNFxbYwyihAEBDEeAA0FAEOAA0FAEOAA1FgANAQxHgANBQBDgANBR94ACgco99LQsBDqD1yj72tSyUUAC0XtnHvpaFAAfQemUf+1oWAhxA65V97GtZxga4mV1qZv9uZt8ys5Nm9vu9xy83s8Nm9kLv4+byhwsAxSv72NeyxMzAX5O0x91/SdJOSbeY2bslLUp60t3fKunJ3tcA0Dh7d83rcx+6XvNzHZmkuc6sLp3dpDsfOKHdB4/o4WeWJz3ERGMD3Lt+2vtytvefS7pN0v29x++XtLeMAQJAFfbumtexxT26b99OvXbugl49uybXzzpS6hjiUTVwM5sxsxOSXpF02N2fknSlu5+WpN7HNwV+dr+ZLZnZ0pkzZwoaNgCUo0kdKVF94O5+XtJOM5uT9FUzuy72Bdz9kKRDkrSwsOBZBgkAVYntSKnDxp9UG3ncfcXM/lnSLZJeNrOt7n7azLaqOzsHgFobDN7LOrMyk1bOrl0M4avmOlpOCPHBjpS6bPyJ6ULZ0pt5y8w6kt4r6TlJj0q6vfe02yU9UtIYAaAQ/eBdXlmVS1pZXdtQ677h2i1jO1JCZZYDFS96xtTAt0o6ambPSvqmujXwxyQdlHSTmb0g6abe1wBQW0nBO2h17byOPndmXUfK/FxHn/vQ9etm1qM2+FS56Dm2hOLuz0ralfD4/0q6sYxBAUAZYnZWvrSyGryIuF9+GbeY11/0LLucwk5MAK0Rs7My9JzB8kuMKrbhE+AAWiNpx+WgUbsvx5VfhlWxDZ/jZAG0Rr+kMaoLJVT2GDWj7szOrAv3qrbhE+AAWiVU3x4n1F443wv+SfSEE+AAEOHum3es6/2WfjbTzvqmkBcBDgARhssvdbh2jQAHgEgxM+0qt9gT4ACmXlWhWvUWewIcQKOkDeMqQ3XUSYYEOIBWyxLGeUM1zRtG1XdrspEHQGNkOas7T6gOH3417pyTqu/WJMABNEaWMM4TqmnfMKq+W5MAB9AYWcI4T6imfcMYvlsz6STDIlEDB9AYozbThOTp34653CHp9arqDSfAATRG1jDOGqpZ3jCqRIADKE1MB0fatsAqZ7h13H05iAAHUIqYlr+ie7TL2LAzqXNOYhDgAHJLCs6Y/usiN77U5aLhKtGFAiCXUK906OaawQ6OIje+ZOkRbzoCHEAuoeCcMUt8/mAHR5EbX6reBVkHBDiAXEIBed59bP91kRtfqt4FWQcEOIBcRgXk6y7ZpM2vnw1uaili48vDzyxr98EjWl5Z1fCc39Qt6ew+eCS4/b3JzN0re7GFhQVfWlqq7PUAlG948XBYZ3amtN2ISa9tknzgYxXjKJuZHXf3heHHmYEDyGVwFp2kzIXEpPq7S5ox0/DUdBoXNGkjBJBbv1f6msXHNwSntLFOXlS/9qj6e5rnNxUzcACFiVlITHtEa5bXi+mAmQbMwAFskHWGHHN2SOzmncExXNaZlZm0cnZt3XhCr/fhd87rwePLtT3DpCgEOIB18uxojDk7JKZfe3gMK6trF7+XNJ6k11t4y+W1PcOkKHShAFg3291kllhDnp/r6Njintyv1W/5G/X7Q88pYzxNQBcKgETDNemyFwBjNu/EvNa0LUhmQYADLZdUk05S1AJgzOadmNeatgXJLKiBAy0XM5MtegFw3BGtSYuTZY6nqQhwoOVC14bNmOmCe6ULgMOdJ5fObtLK2bVgF0rbEeBAy4Va8aredp7UedKZndF9+3YS1gFja+BmdrWZHTWzU2Z20szu6D1+j5ktm9mJ3n/vK3+4AIpW9U3qIW08zzuvmBn4OUl3ufvTZvZGScfN7HDve/e5++fLGx6AKtTh2rA2nued19gZuLufdvene5//RNIpSfx7BkCh2nied16p2gjNbLukXZKe6j30CTN71sy+bGabAz+z38yWzGzpzJkz+UYLoDb653Bfs/h4IedtF3m5Q1tE78Q0szdI+hdJf+juD5nZlZJ+qO7pjX8gaau7f2zU72AnJlC9Mm5qTzqHu4iFzzLGOg1COzGjAtzMZiU9JukJd/9Cwve3S3rM3a8b9XsIcKBaoy48mM8RkDHb4ZPGQjhnEwrwsYuYZmaSviTp1GB4m9lWdz/d+/KDkr5T1GABFCN04YGU7pCqYWkXHPMckIWwmBr4bkkflbRnqGXwj8zs22b2rKQbJN1Z5kABpDeug2N17bwOPHAidQ077YIjLYLlGDsDd/dvSBvuCpWkrxU/HABFCu2yHJZ2Rhxz7vcgWgTLwWFWQE0V0eWR1NkREjMj7o/pzgdOjL1xfhAtguVgKz1QQ0XVjAcvPFheWd1wU/uwUTPiPFvd087YEYcAB2poXM04TTfH4C7LfidIqKwyakYcexVaaAxpx43xCHCgQGXftt6fiWedmffDPNTHPWpGnLeOXYft+tOGGjhQkKpuWy+imyPLAVbUseuHGThQkDwlhmGhmnHogoMs3RxpZ8TUseuHAAcKUmSrXKhmHKpfXzXXKX2nI3Xs+iHAgYKEeq6zlhhCi4/DnSSd2RndcO2WSnY6UseuF2rgQEHSnKaXpsd7sLYudcO7v7OuX7s++twZdjq2EDNwIKfQPY6hEkPaHu/QeSbzY8oqEjsdpx0BDuSQZXNL2sXO2JbCJEV0iHCKYH1RQgFyyHJIU9rFzjQthYOK6BApsjUSxSPAgRyydJ6M6qdOqo2HauvnR5zlX9TFxJwiWG8EOJBDls0toUDud5IMz3YlJW66mQ+8xmBtPO91Z5wiWG/UwIEcsmxuGdXjHZrtHlvckzibTnrtIlsKi26NRLEIcCCHrJtbkvqp73zgROJzQ7PdLG8ERe0IZfdlPRDgQE5FbW7JMtst4o1gFHZf1hsBDtREUbPdMneEol5YxARqIssJgUnS7AhFszEDB2qkiNkuZY/2IMCBEZq6C5GyRzsQ4EBA3nspmxr+aA5q4EBAnl2IbEFHFQhwICDPLkS2oKMKBDgQkOcOSLagowoEOBCQpx2PC4BRBRYx0SixC4NFLCDmacdjCzqqQICjMWK7QvJ2jwzK2o5HLzaqQICjMWIPaSryMKc86MVG2QhwNEbswmCWBUR6ttFEBDgaI/aQprSHORVZcgnhDQJloAsFjRHbFZL0PFM3mJNup4nt2U667iwGm3pQFmbgaIzYhcHB5y2vrMok9W+PTJpdx5Rc8szSx9XkmZ0jKwIcjRK7MNh/3u6DRzaUUwZn1/c+8bxCVwMPllzyLIyOeoOoonyD6TW2hGJmV5vZUTM7ZWYnzeyO3uOXm9lhM3uh93Fz+cMF0gmFZz8ok2rl0sbSTJ6dlaM29bDlHnnE1MDPSbrL3d8m6d2SPm5mb5e0KOlJd3+rpCd7XwMTk1SjDoXnjNmG4OxLukghz87KUbV7ttwjj7EB7u6n3f3p3uc/kXRK0ryk2yTd33va/ZL2ljRGYKzQQuEN125JDM/znlw4MSnxBvi0C6ODRt20w5Z75JGqBm5m2yXtkvSUpCvd/bTUDXkze1PgZ/ZL2i9J27ZtyzVYICRUijj63Bl97kPXJ97cnqbVMO3CaNLPJ32PLffIIzrAzewNkh6UdMDdf2xmUT/n7ockHZKkhYWF0HoRkMuoUkQoPNMGZ8zCaJXnrQBRAW5ms+qG91fc/aHewy+b2dbe7HurpFfKGiQwTtrNO6OCc1xbX9F1a7bcI6uxAW7dqfaXJJ1y9y8MfOtRSbdLOtj7+EgpIwQiZClFJAVnTFtf2jcLoCwxXSi7JX1U0h4zO9H7733qBvdNZvaCpJt6XwOFi9kBOWqhMI2Ytr4854QDRRo7A3f3b6i74J7kxmKHA6yXZqNLEaWImPIIdWvUBTsxUWtVHw0bWx6hbo064DAr1FK/bBLaKZl1wXBcOYbyCJqEGThqZ7hskiTLgmFMOYbyCJqEAEftJJVNBmWdEceWYyiPoCkIcNTOqPLIfI4ZMeeOYNoQ4Kid0ELi/FxHxxb3XPw67Tna9G9j2rCIiYlKWlSMWUjMcssNC5SYNgQ4JiYUwpLGbsrJco52UZt9gLqghIKJGRXCSUe6Dspaz2aBEtOEGTgmpqxbboC2IMARlPUW9lhl3XIDtAUBjkRZFgnTyhPC1LMBauAIqOIMkry7Hqlno+0I8JbKemnB8sqqrll8PNcW87T92wCSEeAtlOfSAkkbWv4Gw3dcOKc5HhbAaNTAWyjrpQXDhn8mpm6epX8bQDICvIViLy0YXCSM+V0x4cx5JEBxCPAWim3f27trXscW9+h7B9+v+YifiQln+reB4hDgLZSlfS/mZ2LCmf5toDgEeAtl6aGO+ZmkcDZ1a+H9jUD0bwPFMXev7MUWFhZ8aWmpstdD9fpdKMsrqzJ1O1b6OrMzhDWQgZkdd/eF4ceZgaNQ/br5/FxHw1OD1bXzOvDAiVK25QNtRB84SjGqqyRrDzmA9Qhw5JYUvKM2Akkbt+WzwQdIjxIKcglt3rnh2i1jNwKl7SEHsB4BjlxCwXv0uTMXu01C0vaQA1iPAEcuo4K3v6D5xX07C+khB7AeAY5M+pc9hJpQB4M3aw85G3yA0VjExLpFyMs6szKTVs6uBTtBhhcchyUF77izu/OeDQ60EQHecsNhvLK6dvF7oU6QpLp333yO4OWCBiAdSigtNyqMpeROkFDd26Sxt8kDKA4B3nIxXR7Dz2HBEagHArzlYkJ3+DksOAL1QIA3WL8T5JrFxzOfLzLu5p3QgiQnCgKTN3YR08y+LOlWSa+4+3W9x+6R9DuSzvSe9ml3/1pZg8RGsVvPx50vMtz9EdOF0v85AhuYrLHHyZrZL0v6qaS/Hgrwn7r759O8GMfJbpT1AKfdB48knjUyP9fRscU9F3/3cLsfR7oCzZP5OFl3/7qkH5UyqpaLuQQ4JGbrOeeLANMtTx/4J8zstyQtSbrL3V9NepKZ7Ze0X5K2bduW4+Wmz7iAHTUzD532x/kiQHtkXcT8M0m/IGmnpNOS/jj0RHc/5O4L7r6wZcuWjC83nUJB2p+Jj5qZF3VHJYDmyhTg7v6yu5939wuS/kLSu4odVjuEgnTGbGzpg/NFAGQqoZjZVnc/3fvyg5K+U9yQ2uPum3ckLjKGdkYOz9g5XwRot5g2wr+V9B5JV5jZi5I+K+k9ZrZT3Ttrvy/pd8sb4vQKBWz/UuBhV811Unet0O4HTC9upa+hUPvfh985rwePLye2BUrjZ9rcOQk0U6iNkACvqaQjXl89u5b43LnOrF47d2FdsJu6/zzqnw4oiZ5woKEI8IYad/Z2jM7sjC6d3ZT4BjC48QdAPYUCnPPAa27cca8xVtfORy+MAmgODrOquXEB25md0ebXz2b+/fSEA81FgNfcqIDt935/9tffMfJEQalbJ6cnHJgulFBqLtQrnrT42G8/7C9gDj7/ng+84+Jz6EIBpgOLmA2Qtv2PdkFgutCFUiACEkCV6EIpSOxFCgBQNgI8pTxHwAJAkQjwlMYdAZtnZk5pBkAatBEOGXdRcJ4jYMe9btbbeQC0EwE+ICZEQ2dsnw8sBr+0shp1ezzXnwFIiwAfEBOioYsU5gMz88s6s1Eza64/A5AWNfABsSEaOmM7acONmYJvCmnvuASAQczAB2S5Q7JfHrnzgRN63SWbtPn1s+tm5iuBI2CH3xS4/gxAWszAB4S2rd99847EDhFp/ax7ZXVNndkZ3bdv57rbdmJm1lx/BiAtdmIOiQlqKf6M7dDtOlykACBWa7fSF9FbvfvgkcRZdIhJ+t7B9yeOoX+7zsrZNWbZAKKEAnyqa+BF9Van7QRJKo8cW9yj+/bt1GvnLujVs2v0egPIbapr4KG2wAMPnNA9j568OBMeNysOdYgk3UU5auFxVJsis3AAaU11gI+aOa+sriV+PrgFXir2jG16vQEUaeoCfLDevMksuENylNW187rn0ZPrZteujTe994M6dvZMrzeAIk1VgA93fGQJ777BWXlfP7xjbnFPWjwd1aYIAGlN1SJm6Ab3GbPCXiOm3BFaPJWUuA2f+jeALKZqBh4K1wvu+uK+nRtmvyGjerxjyh2jFiuPLe4hsAEUYioCvF+uCBVMrprrbNjpONh5ktSFIiVv3okpd7BYCaAKjQ/wpJ2OgwZDN3QI1ShZNgGxWAmgCo0P8FDdW9rYLZJWlsCXRp+pAgBFaXyAh8oSJkV1i5SBg6kAVKHxAV7XckXW2TsAxGp8GyHnaANoq8bPwClXAGirxge4RLkCQDuNLaGY2ZfN7BUz+87AY5eb2WEze6H3cXO5wwQADIupgf+VpFuGHluU9KS7v1XSk72vAQAVGhvg7v51ST8aevg2Sff3Pr9f0t5ihwUAGCdrDfxKdz8tSe5+2szeFHqime2XtF+Stm3blvHluoq4Hg0ApkXpi5jufkjSIal7J2ban++H9vClCoMn/BHiANooax/4y2a2VZJ6H18pbkg/M3gsq6QNh1X1T/gDgDbKGuCPSrq99/ntkh4pZjjrjTrnpG95ZVW7Dx7hYmAArRPTRvi3kv5V0g4ze9HMflvSQUk3mdkLkm7qfV242ONXud0dQBuNrYG7+0cC37qx4LFsEDrnJAm3uwNom1qfhZJ0zsmoy9G4MAFAm9Q6wPfumt9wh+R9+3ZqPnDS4KRPIASAKtX+LJTQOSdcmACg7Wof4Ek4gRAAGhrgEicQAkCta+AAgDACHAAaigAHgIYiwAGgoQhwAGgoc099wmv2FzM7I+kHlb1gca6Q9MNJD6JCbft7Jf7mtmjq3/wWd98y/GClAd5UZrbk7guTHkdV2vb3SvzNbTFtfzMlFABoKAIcABqKAI9zaNIDqFjb/l6Jv7ktpupvpgYOAA3FDBwAGooAB4CGIsBTMLNPmpmb2RWTHkvZzOxeM3vOzJ41s6+a2dykx1QWM7vFzJ43s++a2eKkx1M2M7vazI6a2SkzO2lmd0x6TFUwsxkze8bMHpv0WIpCgEcys6vVvcD5vyY9looclnSdu/+ipP+Q9KkJj6cUZjYj6U8l/Zqkt0v6iJm9fbKjKt05SXe5+9skvVvSx1vwN0vSHZJOTXoQRSLA490n6fcktWLV193/0d3P9b78N0lvnuR4SvQuSd919/909/+T9HeSbpvwmErl7qfd/ene5z9RN9Sm+nB9M3uzpPdL+stJj6VIBHgEM/uApGV3/9akxzIhH5P0D5MeREnmJf33wNcvasrDbJCZbZe0S9JTEx5K2b6o7gTswoTHUajG3shTNDP7J0k/n/Ctz0j6tKRfrXZE5Rv1N7v7I73nfEbdf3J/pcqxVcgSHmvFv7LM7A2SHpR0wN1/POnxlMXMbpX0irsfN7P3THg4hSLAe9z9vUmPm9n1kq6R9C0zk7qlhKfN7F3u/j8VDrFwob+5z8xul3SrpBt9ejcMvCjp6oGv3yzppQmNpTJmNqtueH/F3R+a9HhKtlvSB8zsfZIulfRzZvY37v6bEx5XbmzkScnMvi9pwd2beKJZNDO7RdIXJP2Ku5+Z9HjKYmaXqLtIe6OkZUnflPQb7n5yogMrkXVnIvdL+pG7H5jwcCrVm4F/0t1vnfBQCkENHCF/IumNkg6b2Qkz+/NJD6gMvYXaT0h6Qt3FvL+f5vDu2S3po5L29P7fnujNTtEwzMABoKGYgQNAQxHgANBQBDgANBQBDgANRYADQEMR4ADQUAQ4ADTU/wMeoX4vTceKnQAAAABJRU5ErkJggg==\n", 294 | "text/plain": [ 295 | "
" 296 | ] 297 | }, 298 | "metadata": { 299 | "needs_background": "light" 300 | }, 301 | "output_type": "display_data" 302 | } 303 | ], 304 | "source": [ 305 | "# make sythetic data (from Eric's example)\n", 306 | "x_data = numpy.linspace(-5, 5, 100)\n", 307 | "w_true = 2\n", 308 | "b_true = 20\n", 309 | "\n", 310 | "y_data = w_true*x_data + b_true + numpy.random.normal(size=len(x_data))\n", 311 | "\n", 312 | "pyplot.scatter(x_data,y_data);" 313 | ] 314 | }, 315 | { 316 | "cell_type": "markdown", 317 | "metadata": {}, 318 | "source": [ 319 | "This situation arises often. In **Module 1** of _Engineering Computations_, we used a real data set of Earth temperature over time and we fit an ominously sloped line. \n", 320 | "We derived analytical formulas for the model coefficients and wrote our own custom functions, and we also learned that NumPy has a built-in function that will do it for us: `numpy.polyfit(x, y, 1)` will return the two parameters $w, b$ for the line\n", 321 | "\n", 322 | "$$y = w x + b $$\n", 323 | "\n", 324 | "Here, we will instead use gradient descent to get the parameters of the linear model. \n", 325 | "The first step is to define a function that represents the _deviation_ of the data from the model. \n", 326 | "For linear regression, we use the sum (or the mean) of the square _errors_: the differences between each data point and the predicted value from the linear model (also called _residuals_).\n", 327 | "\n", 328 | " \n", 329 | "\n", 330 | "#### Each data point deviates from the linear regression: we aim to minimize the sum of squares of the residuals.\n", 331 | "\n", 332 | "\n", 333 | "Let's review our ingredients:\n", 334 | "\n", 335 | "1. observational data, in the form of two arrays: $x, y$\n", 336 | "2. our linear model: $y = wx + b$\n", 337 | "3. a function that measures the discrepancy between the data and the fitting line: $\\frac{1}{N}\\sum (y_i - f(x_i))^2$\n", 338 | "\n", 339 | "The last item is called a \"loss function\" (also sometimes \"cost function\"). Our method will be to step down the slope of the loss function, to find its minimum.\n", 340 | "\n", 341 | "As a first approach, let's again use SymPy, which can compute derivatives for us. Below, we define the loss function for a single data point, and make Python functions with its derivatives with respect to the model parameters. \n", 342 | "We will call these functions in a sequence of steps that start at an initial guess for the parameters (we choose zero), and step in the negative gradient multiplied by a step size (we choose $0.01$). \n", 343 | "After $1000$ steps, you see that the values of $w$ and $b$ are quite close to the true values from our synthetic data." 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": 10, 349 | "metadata": {}, 350 | "outputs": [ 351 | { 352 | "data": { 353 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHgAAAAaCAYAAAB8WJiDAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAE+UlEQVRoBe2a7VHcMBCGjxsKIKSCQAcQKgjpIEAFQAcw/OMfAx0AFRDoAKiAjw5IByHXAXkfI+1Isuz4jvMRJ94ZnaTVStoP7WplmHt5eRn00G0NHB4eLkiCAyfFkqu3hR/Nd1u0nnungWMZc9drQ+1TtR9Uloce2ded1sCOjLoeSHCs9pJwK72BA610uIn33uf4n+vv4Jxauo2T5+LB31SXQ7SQ/pLutpT/EPfj2ES0KxhXZRUVRCFag3vCQdDD36UB7lNsUwvuIOC9q2qPILYQLQRWX1O9zwCgNun3pcpnlWf1l1X38A4akO4x8Ej1WW574Ym8+6qLbNr1Xw2sDoa8VV24dbqA8KTc96otFU9p+n77GnB2+KK68E6/o/oYl6eROafa2Grfh2jcGoIqIGxfVw02xYuRdZWdpvQ9XUkD2AhbpYAD8kyi9oWnk33o2FQn650Yxa124+q3VEQKSg8TaEC2OFP5pUIoNi9W+0PVckMNcvf+qCIQ/ivj4YI1tP1Q+xrAVptNt+FTJQas8048+FEG9lncmvoX6l+pngloL7ye0P5R5Ul9SzTU5vrYVe2TCw5slEmqP9C4X4MuMmyrcHdtqQB3onkPmUhcH7R3KBO8ngu3AWMJYCtsZvTJeNTlDiZDfoqwruOUggJRBOHhRDWKuVQb3KzgwO19pw3TO+hAuJAXQhd95AqB77UnwTrnGiQnIDFBoem64dw22l4mcpt0bzyUg5oDbBXKm6Mx3FAthHs2TNzw969lblIICqRUMRCv8Mae9uOAYViAk5vyCo+WAIqeE/6oYteOcHh/qETPv/eCxWRc3faggUzIWRVVkb+xgQnRCIfAOWAjwnM6viA84TILoifb84cjpGGvgcZzCR375EIS9z8GAzjZR0VLP8JjfHhJlcGTzgys8bTPc9DkEm1uX5G8gsbZ41aFuilsaJ7nO51TKZMjRHcmZzIZuRrzgYHrgI2ie0lMe8N5ryrNF03OgAPh8Xq+yhDqG4FoRxC6uQjmvQ40vPD4TxUZXTmZ8eigsFAdaD48ZL8R1M2rGnPrDVTDfySTcFWH1i9X55CexmpCNC7PJjkgFKSG9Kc99Zrc/GnictGkFMqcglKDGx+BUu3gCrdAMaLZNdClRRK3bdWh9VzBJzZrBPOiwuXrYrqFOqcETj9Z66jRDtMjgkfjxS0LzgzlcFvizb7oOJ4vNUaSxaEkuuD14VokPDbHrTOLKidT6dAmjODBIe/JcNwdqstpX4vR1kN5YTaKor5LGWGYNOKWG/BpXiYeOOkIi5IKcDhLuBwaOsqzxkun381Jo5Sb2noVGcrxAq+pDCEjRf4QIura8xq8UMFwOdgWkvcYbzWUw/PoPYw70L58vTmliA+ix0+VTyrw53HRG1ljAF4LzyhuIFqiz7Wbw2c9DJ9GAUhnAUQN+CfDRx7/x5y66w85sjmO8CUo/pqkDUhK6rK+0sRJENpn7CRrkn26OscZel11NqETnmjF4fQH4Y+iEqIBTlDjU1HMmOwHz6P894AxVYggBahNhOS9fvSKyf7i8eF7PksUIgsDa3FCGM8Xu89Comm1tf6N22taS3Z5HaIZ16OHczX4Wpi9LpxtsNFYVyR3sAdSdu5isrge2tcA3rgog+2pJuSSX9TdveQZY0dZ+48OTR5oAzyYf9Zq/CGCeT20qwHZg0NwpTrKupvs+ht4KM5n5o25AwAAAABJRU5ErkJggg==\n", 354 | "text/latex": [ 355 | "$\\displaystyle \\left(b + w x - y\\right)^{2}$" 356 | ], 357 | "text/plain": [ 358 | " 2\n", 359 | "(b + w⋅x - y) " 360 | ] 361 | }, 362 | "execution_count": 10, 363 | "metadata": {}, 364 | "output_type": "execute_result" 365 | } 366 | ], 367 | "source": [ 368 | "w, b, x, y = sympy.symbols('w b x y')\n", 369 | "\n", 370 | "loss = (w*x + b - y)**2\n", 371 | "loss" 372 | ] 373 | }, 374 | { 375 | "cell_type": "code", 376 | "execution_count": 11, 377 | "metadata": {}, 378 | "outputs": [], 379 | "source": [ 380 | "grad_b = sympy.lambdify([w,b,x,y], loss.diff(b), 'numpy')\n", 381 | "grad_w = sympy.lambdify([w,b,x,y], loss.diff(w), 'numpy')" 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "Be sure to read the documentation for [`sympy.lambdify()`](https://docs.sympy.org/latest/modules/utilities/lambdify.html), which explains the argument list.\n", 389 | "Now, we step down the slope. \n", 390 | "Note that we first compute the derivatives with respect to both parameters _at all the data points_ (thanks to NumPy array operations), and we take the average. \n", 391 | "Then we step both parameters (starting from an initial guess of zero)." 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 12, 397 | "metadata": {}, 398 | "outputs": [ 399 | { 400 | "name": "stdout", 401 | "output_type": "stream", 402 | "text": [ 403 | "2.03136593978268\n", 404 | "20.030956250406764\n" 405 | ] 406 | } 407 | ], 408 | "source": [ 409 | "w = 0\n", 410 | "b = 0\n", 411 | "\n", 412 | "for i in range(1000):\n", 413 | " descent_b = numpy.sum(grad_b(w,b,x_data,y_data))/len(x_data)\n", 414 | " descent_w = numpy.sum(grad_w(w,b,x_data,y_data))/len(x_data)\n", 415 | " w = w - descent_w*0.01 # with 0.01 the step size\n", 416 | " b = b - descent_b*0.01 \n", 417 | "\n", 418 | "print(w)\n", 419 | "print(b)" 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": 13, 425 | "metadata": {}, 426 | "outputs": [ 427 | { 428 | "data": { 429 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAobUlEQVR4nO3deZzNZf/H8dfHGJpKlpKYiCQUoqR+TcrS5m4xuEu7FtTdRiRLi9J9Z0riV9x1U35p0V2JCVkSUrQSFdnSPoSWEZms1++PMzPNcr5nm3Nm5sx5Px+PHsyZc77fax49Hu+5fL6f67rMOYeIiMSfSmU9ABERiYwCXEQkTinARUTilAJcRCROKcBFROJU5dK82RFHHOEaNmxYmrcUEYl7y5cv/9k5V7vo66Ua4A0bNmTZsmWleUsRkbhnZt/5e10lFBGROKUAFxGJUwpwEZE4pQAXEYlTCnARkThVql0oIiKJJnNFFqPmrWNTdg71aqQw6PympLdJjcq1FeAiIjGSuSKLodO+IGfvfgCysnMYOu0LgKiEuEooIiIxMmreuvzwzpOzdz+j5q2LyvUV4CIiMbIpOyes18OlABcRiZF6NVLCej1cCnARkRgZdH5TUpKTCr2WkpzEoPObRuX6eogpIhIjeQ8q1YUiIhKH0tukRi2wi1IJRUQkTinARUTilAJcRCROKcBFROKUAlxEJE4pwEVE4pQCXEQkTqkPXESE2G77GisKcBFJeLHe9jVWVEIRkYQX621fY0UBLiIJL9bbvsaKAlxEEl6st32NlaABbmYHmdnHZvaZma02swdzX69lZvPNbEPunzVjP1wRkeiL6bavW7ZA//6wfXvJr1VEKDPw3UAn59xJQGvgAjM7HRgCLHDONQEW5H4tIhJ30tukMrJ7S1JrpGBAjZRkDkquxJ2vrCQtYyGZK7LCv+ju3fDoo9CkCYwfD+++G/VxBw1w57Mz98vk3P8c0BWYnPv6ZCA96qMTESkl6W1SWTqkE2N6tmb3vgP8tmsvjr86UkIOcecgMxNOPBEGD4azz4bVq+Hii6M+5pBq4GaWZGYrga3AfOfcR0Ad59xm33jdZuBIj8/2NbNlZrZs27ZtURq2iEhslKgj5fPP4ZxzoFs3qFoV5s2DmTPh+ONjMtaQ+sCdc/uB1mZWA5huZi1CvYFzbgIwAaBt27YukkGKiJSWUDtSCi78OaHybsatnU6j6VOgRg0YNw5uugkqx3apTVhXd85lm9k7wAXAFjOr65zbbGZ18c3ORUTKtYLBWz0lGTPI3rU3f/VlvRopZPkJ8YIdKXkLf/b9+Sc3LJ9Fv/f/y8F7ctjYsxeNxz8GtWqVys8SShdK7dyZN2aWApwDrAVmAL1y39YLeCNGYxQRiYq84M3KzsEB2Tl7i9W6OzarHbQjZdTctfzPmg+YO+k27lv0LJ/Wa8b5N4yn8zE9SJuwMrKHnhEIZQZeF5hsZkn4Av9V59wsM/sAeNXMbgS+By6N4ThFRErMX327oJy9+1m0dhsju7f03hdl9WoyJgyi/bcr+KrW0Vz39+G80/jU/GuU5jL8oAHunPscaOPn9V+AzrEYlIhILISysnJTdo7/g4h/+YWvbxlIg6kv0Co5hQc79+GFNheyL6l4jOY99CzzABcRqSi86ttF31PI3r3w73+z5/7hNNixg5fadGHMmVeRnXJYwOuUxjJ8LaUXkYThb8VlQcVWX86eDS1bQv/+rDzyOLpc/yTDz/1H0PCG0lmGrxm4iCSMvJJGoC6U9DapsGYNDBgAc+f6VlLOnEnP98CZ+b1uSnJSodp61JbhB6EAF5GE4re+nefXX6FfP9/S90MOgcceg9tvhypVqLdqod/yS2pu8JfFYRAKcBGRffvg6adh+HDIzoY+feChh6B27fy3DDq/aaFDH+CvmXbAXwoxpAAXkcQ2fz7ceadvv5KOHWHsWGjVqtjbipZfysOxawpwEUlM69fDwIEwaxY0bgzTp0PXruBR54Yg5ZdcpXm2pgJcRCq8gqF6fNX9jN84i+Ne+T9ISfFt+XrHHb7Np6Jwn9I8W1MBLiJxJdwZbl6o7tm9hys/m8eA916kZs4Ovu12OQ2fGgN16kRtbIF2MlSAi0hCi2SGO2reOtpsWM59C5+h+bZv+bB+C0Z07sv2pieyNITwDucXRmmfrakAF5G4EfYM96uvGD7pHs7b8CE/VK/DzelDmXv8GWCGhRCq4f7CCGUnw2jSSkwRiRshz3C3b4e774YTTuDM71by6FnXck7vp5jbNC3/IWUooRru4Q4xPVvTD83ARSRuBJ3h7t8PkybBvffC1q1w3XUsvuJ2/u+9reyOYKVkuCWR0m41VICLSNwItJiGxYt9p7+vXAlpab59TE45hS7A7tqRtfZFUhIpzUU9CnARiRv+ZrjDW6Rw3r/6weuvQ4MG8N//wmWXFernjjRUA/7CKAcU4CISM6F0cITbFpgfxjt2wMMPQ4/HfWdPPvSQb2FOSvQeGJbH1ZcFKcBFJCZC6eCIaOHLgQMweTIMGwY//QTXXusL8tTUmKyCLKt9TkKhABeREvMXnKG0/IXdFrhkiW+3wE8/hdNPhzfegHbt8sdQmqsgywO1EYpIiRQ9KDgvOL1OvinYwRFyl8d338Hll0P79r7ukpdegvffzw9vCL/lryJQgItIiXgFZ5LHplAFOzi8ujnyX//jD7jvPmjWDGbM8G33unYtXHllsU2nSnsVZHmgABeREvEKyP3OBV3U4rnw5dwm8PzzcPzx8M9/QvfusG4dPPCA76AFP4L+MqiAFOAiUiKBArJq5UrUPDgZw3dyzcjuLQvVo9PbpDKye0tSa6Tkv+fpxntIv+Xv0KsXpKbC0qW+kkn9+n7vkbkii7QM32k5Ref8hq+kk5axkMwVWSX+Wcsbc86V2s3atm3rli1bVmr3E5HYK/rwsKiU5KRiwe3XDz/AkCEwZQrUrQsZGXD11VDJe57p794GuAJ/hj2OcsjMljvn2hZ9XTNwESmRgrNof4I+SNy1y1caadoUpk3zLYNfv97XHhggvMF//d0BSWYUnZpWxAeaaiMUkRLL65VuNOTNYsEJxevkmSuyGDV3LW0/mMuwdydTZ/s23+rJRx6Bhg1Dvm+g+ns4749XCnARiZpQ9g7JXJHFlCen8uS8pzh50zq+qNOYgV3v5u/9ryC9YXjlDa/7JZn5DfGK9kBTAS4ixUS6ojHo3iFZWVS98XpeXTGfrYfUZFCXfkxt2RlnlfimyOKdgmOonpKMGWTv2ltoPF7363FKKq8vzyq3e5hEiwJcRAopyYpGz71DmtXytQOOHEmn3Xv49+l/Z/zpl/FH1YPzP1uwvFF0DNk5e/O/5288/n7ZtD2mVrndwyRa1IUiIoVmu5U8yg+pNVJYOqRTeBd2Dl591Xe4wvffQ48eXHpsOp9Uqhnw+nltgYFENJ44pS4UEfGr6FL4qD0AXL4czjrLtwS+Zk1YtAimTuWqKzoGXeATyr0q2gPJSCjARRKcv1Y8f0J+ALh5M1x/PZx6qq8dcOJEX5h36AD4X7xTtD87lHtVtAeSkVANXCTBhTKTDekB4J9/wpgxvq1dd++Gu+6Ce+6B6tWLvTXYFq3+Hk6GPZ4EoAAXSXCBWvEOOBf8AaBzvtNwBg2Cb7+Frl3hscfguOPCHkvRzpODkiuRvWuvZxdKolOAiyQ4r1a8kJadr1zpO4dy8WJo2RLefhs6d45oHP46T1KSkxjTs7XC2kPQGriZ1TezRWa2xsxWm1m/3NcfMLMsM1uZ+9/fYj9cEYm2UGrSxWzZAn36wMknw+rV8NRTvkMWIgxvSMz9vEsqlBn4PmCgc+5TM6sGLDez+bnfG+Oceyx2wxOR0hDysWG7d8MTT/jOn8zJ8c2+778fatQo8RgScT/vkgoa4M65zcDm3L/vMLM1gP49I5JInPMdX3bXXbBxI1x4IYwe7duAKkpCWYYvhYXVRmhmDYE2wEe5L91mZp+b2SQzK96Z7/tMXzNbZmbLtm3bVrLRikjp+/xzOOcc6NYNqlaFefNg1iwydx1KWsZCGg15Myr7bXse7qBuE08hr8Q0s0OBxcC/nHPTzKwO8DO+3RsfAuo6524IdA2txBQpfRGf1L5tm+84s4kTfSWSESPgppugcmW/+3BHY7/tWJwqXxF4rcQMKcDNLBmYBcxzzj3u5/sNgVnOuRaBrqMAFyldgQ48SPUKyD17YNw4X2Dv3Am33OLbr7tWrfy3eC11D7S8XeEcOa8AD1oDNzMDngXWFAxvM6ubWx8H6AasitZgRSQ6vA48AD+bQjkHs2bBwIGwYQNccAE8/jg0b17suuE+cCzJBlniLZQaeBpwDdCpSMvgo2b2hZl9DnQE7ozlQEUkfME6OHL27qf/Kyu5uv+zbD2jA1xyCSQlwezZMGeO3/CG8A8QVotgbITShbIEip0VCjA7+sMRkWjy6uzIU3PXdu5cMoWrVs5hZ9WD+fyuB2j18DBITg543aD7fhehFsHY0EpMkXIqGjVjrz1FKu/fxzUr3qT/kikcuieHl1p3YcyZV3Lw4UexNEB4ey11DzY+tQjGhgJcpByKVs244IEHWdk5GHD2xk+4b+GzNP71R947pjUPde7N+toNAcgOMCMuyVL3cGfsEhoFuEg5FKxmHM7MPH+V5Zo1/NTnVo5auoiva9bjxh73saBxO7C/KqSBZsSBxhTxST16gFkiCnCRKIpWq5xXbThvJh7WzPzXX+HBB2H8eI469FAYPZovzuzG+zPXQRgz4pLWsUNeri8h04EOIlFS9GSbvHCNZIWi10w4ySz0bo59+3z93E2a+P7s3dvXHjhgAF3bNQp7A6twO08k9jQDF4mSkpQYivKqGXsdcFBsFvzWW3DnnfDll9Cxo++ghZNOKvSWcGfEqmOXP5qBi0RJNFvlvLZ4TQ0wC85ckcXlAyez4Lh2cP757Pz9D5g+HRYsKBbekYho21mJKc3ARaIk2q1yBWfIebX1vE6SghtgpCQn0aV+VbL/cTsvfDKDPytXYWSH63j59G6MOOYU0s3fMo7IqI5dvijARaIknBJDOA87i7bvOf7az6RBtSo8sWs5x9yUQfU/fueVVucy+qxr+PmQmuCIqHwj8UMBLlJC4S5uCbfH22s/k4t/Wcs/X3+G6l+t5aP6LRhxaR9W12lc6H1a6VixKcBFSiCSxS3hPuwsGsLH/LaJYYsmcf6GD/mxeh2GdB3CnKZphfq580SjQ0S7CJZfCnCREoik8yTch515tfVqu//g1vdf4YZlM9ibVJlRZ/fimbZd2V25it/PRaNDRLsIlm8KcJESiKTzJNDDTn+z3UHnHMenw0dz+zvPU3tXNq+1OIcnO1/P9wdV97yH517fYYpma6REn9oIRUogksUtXkeHdWxWu9hCoGljXqLD1X9jxOwn2FT7aC65dgxjrxrKgF5ne7YU5oX3qHnrSnzcmXYRLN80AxcpgUgWt3jtC1Jwtnt09k8MWzSJv61/n59qHAn//S8nXXYZM4rUuf3dO+8XQTTKHtpFsHxTgIuUQKSbNPnrp77zlZUcsnsXt3z4Gr0/yWR/pUo81v5qnjm1G2t7dg/53qWxIlSrL8sHBbhICUVlccuBA/T+ajF95j7DkX/8xusnduTRs3uxpdoRnqUSr3vf+cpKv++NdEUoaBfB8koBLlLWliyBfv2459NPWZnajL7d72VlPd8MN5LZbixXhEr5ooeYImXlu++gZ09o3x62boWXXuLbGfPZdkLrEu014vWQVGWPikczcJHStnMnPPIIPPaYb/HN8OEwaBAccgjpQPrJR5fo8ip7JA4FuEgAUV2FeOAAvPACDB0KmzfDlVdCRgbUrx/dQaOyR6JQgIt4KOkqxILhf972r8l471lqrv4MTj0Vpk6FM86I6fil4lMNXMRDsHMpA8kL/wPff8+YmaP4z9N3sOf7H1k+Yix8+KHCW6JCM3ARDyVZhfjEzM+4adGL3PTRNAzHE//Tk6dP/zs1k2uxtJLmTRIdCnARDxG14zkHL7/Mi4/1o96On5nVrD0jO1xPVvUjAcjREnSJIk0FRDyE3Y738ce+0shVV/F7tZpcemUGt3UdnB/eoCXoEl2agUtcCbUrJBrdIyG342Vl+TpLXngBjjoKJk1ibatzWJW5GrQEXWJIAS5xI9SukGjuYR2wHS8nx9fLnZEB+/f7QnzoUKhWjXSASpXUiy0xpQCXuBHqJk0x38PaOXj1Vbj7bvj+e+jRAx59FI49ttDb1IstsaYAl7gRaldIJN0jIZdcli+H/v19+5ecdBJMngwdOoT6I4hElQJc4kaoXSHhdo+EVHLZvBnuuQeeew6OOAImTIAbboCkJL/X9HcPlVMk2tSFInEj1K4Qf+8zfMHs73SagAt2/vwTRo6E44/nwAsv8lL7S2l11TjSfmlM5uc/hTTuvF8QBU/aGTrti4hPyRHJoxm4xI1Qu0IKvi8rOwcDXO73/M2u/ZZWnKPVR2/DCb3hm2/Y1PF8rm9+Geuq1QHg9zAejAaryWt2LpFSgEtcCfXBYN770jIWFiunFFwOP2reuvxwz3Pilo3cv2Aip/2wClq2hAULuPRjPK8TbDyBavI69V1KImgJxczqm9kiM1tjZqvNrF/u67XMbL6Zbcj9s2bshysSHq/wzAvKgqF8xB+/MXLOE8x8rj9NfvmelcMyYMUK6NSpRMvqAx18XJL9VkRCqYHvAwY655oDpwO3mtkJwBBggXOuCbAg92uRMpO5Iou0jIWFTmL3Cs8ks/zgrLJvLzd9NJVFE/ry91ULeDWtOx/M+YDW/xqc/5AyktPn8wSq3evUdymJoAHunNvsnPs09+87gDVAKtAVmJz7tsngW7sgUha8HhR2bFbbb3judw6c47z1H/DWs7cw9J3n+Kh+Cy64YTyXL5nKhWedUOgz4T4YLSi9TSoju7cktUZKsZN2SvKLQSSsGriZNQTaAB8BdZxzm8EX8mZ2pMdn+gJ9ARo0aFCiwYp48SpFLFq7jZHdWxZ7SPj65DncnPkkad99zvrDG3DNZSN4r9HJngcIh/tg1N/n/X1Pp75LSZhzRR/heLzR7FBgMfAv59w0M8t2ztUo8P3fnHMB6+Bt27Z1y5YtK8l4RfxqNOTNYg8jwTdL/ibjwr9e2LYN7rsPN3Ei26sewuNpVzKldRf2JVUmJTkppDMo/T0YBd/MeumQTmGPXV0oEoyZLXfOtS36ekgzcDNLBl4HXnLOTct9eYuZ1c2dfdcFtkZvuCLhCbp4Z88eGDcORoyAnTux225jabc+LPhwK/uzc0gtEJzBAjXadWstuZdIBQ1wMzPgWWCNc+7xAt+aAfQCMnL/fCMmIxQJgWcp4rzjYeZMGDgQNmyALl1g9Gho3pwLgQs7FL5OKG19Ee0TLhIDoXShpAHXAJ3MbGXuf3/DF9znmtkG4Nzcr0Wizl93SVH+HhSOa5VM+pAb4JJLfN0ks2f7/mve3PNeobT1hb1PuEiMBJ2BO+eW4Csl+tM5usMRKSychS75pYiff4bhw+He/0C1ajB2LNxyCyQnB71fKOWRkPcJF4kxrcSUci2srWH37oV//xseeAB27ICbb4YHH4TDDw/5fqGWR1S3lvJAm1lJuZRXNvEXpuBnpjx7tm/Ze//+0K4dfPaZ76FlkfAOVo5ReUTiiWbgUu4ULZv4kz8jXrMGBgyAuXOhSROYMQMuugiseNUvlHKMyiMSTxTgUu74K5sUlJKcxLDTj4R+/WD8eDj0UF9nyW23QZUqYV3XXzlG5RGJFwpwKXcC9VM3qFaFJ3//iJN6XAXZ2dC3r6+3u3btiK+rfUckXqkGLuWOVz91t62reHdKf0569D5o3ZqFU+aSdsylNBr9cdD9SAJdV/3bEq80A5cy5W/VY9FFOcf+8iP3vzOJDl99DI0bQ2YmmfVPYej0VeTs3QOEth+J9h2RikYzcCkzXjsIAozs3pKmVfdz74KJvDXpVtI2f+k7+X31aujalVFvrQ97H+1AuwKKxCPNwKXMeD1UfHz2l7xb6yvSx98Hv/4KN94I//wn1KmT/75I69l6QCkViQJcyoy/sP2f7z5j+NsT4Ofv4KyzfKso27Qp9j7tRyKiAJcAYr3NacEQPua3TdyzaBLnbfiQTTWPgqlToXt3v/3coHq2CKgGLh686tPBOj3CMej8ptTe/ydDFk1i/jO3cMZ3n/F4x+tYNnsJ9OjhGd6gerYIaAYuHsLagyQS+/eTvmw2F0waRpXffmFqi868eFFfbrj0DC4J8fqqZ0uiU4AnqEgPLcjKzqHRkDdLVFJZMvE16gwfSpPNG1nXsAW/PvE8l13Vhcsi/mlEEpMCPAGV5NACoFjLX8EQD/iL4euv2dTnNs5cOIcfD6vNrZcM5s1mZ5Ky1hi5IkuzaZEwqQaegCI9tKCoop/xqpvPWrIOhg6F5s2p+d4iHmt/NZ17P82bzduDWdD+bRHxTzPwBBTJoQVeR18X/EzRXwzmDnDR8vmc/r8vwI5f4Zpr6FitMz9VOyLkMYmINwV4Aork0AKvvbkLfqZgCJ/6wyruXzCRlls28mm9phwxfzacdhpJGQtB/dsiUaESSgKK5NCCUD5Tr0YKqdu3Mu6NR3htyhAO37WdOy6+i9tvGw+nnRbxvUXEP83AE1AkhxYE/czOnUzaOINjnnsKhzE27Qr+064Hf1Y5CLf9T9IyFhZ6vw5MECk5c86ruhl9bdu2dcuWLSu1+0kpOHAAXnzR95By0yZ+6JLOHa16soJqGBSqnackJ2mxjUgEzGy5c65t0ddVQpHIvf8+nH469OoFRx8N779P/dnTmZ5xOak1Uoo9+MzZu5/+r6wMae9uEQlOJRQJ3w8/wODB8PLLUK8ePP88XHUVVPprPhCoqySiHnIRKUYzcAndrl3wwAPQtClMnw733gvr1pHZohNpj75T6KT3YF0lofaQa6Yu4k0BLsE5B1Om+IL7wQfh4oth7Vp46CEyN2z3G7wdm9UOuhAoUA85BD+gQSTRKcAlsI8/hrQ0X4mkTh147z145RU45hjAO3gXrd2Wv1ugF68e8oK0wEfEmwJc/MvKgmuv9fVvf/01TJrkC/Mzzyz0tkDBm94mlaVDOjG2Z+uQesj90QIfEW8KcCksJ8d3fNnxx/tm2kOGwIYNcP31hR5SZq7IIi1joecS+4LBG8re3VrgIxI+daGIr/tj7lrafPgWwxY/R73tW5lz/BlM6voPrrqsI+nVqhV7f9HTcAryF7zB9u7WAh+R8CnAE1zmiixeGD+NMfOept2PX/LlkY0YcMXDfNigFQCr/LT7+at750ktQfDqgAaR8CjAE9nmzST3uZHXlr/FLwdXZ/AFt/Nay3M4UOmvUoa/U3i86t4GLB3SKdajFpFcCvBE9OefMGYMPPww5+76kwntujH+jJ7sqHqI37cXDWydCC9SPijAE4lz8PrrMGgQfPstdO3K1cf14OPKtQJ+rGgw60R4kfJBAR7Hwlp6vmIF3HknLF4MLVrA229D585cuSKLLyJ4IAl64ChS1oIGuJlNAi4CtjrnWuS+9gDQB9iW+7ZhzrnZsRqkFBfKuZYAc95eyd5h93DRJ3PYfshhfDdsJK0fvAsqVy703rwwrp6SjBlk79obMJj1wFGk7IUyA38OGAc8X+T1Mc65x6I+ogQT6QZOgZaep7dJhd27WXX3CNo/PYaq+/Ywqe0lPJF2BXsrVWfkF1sK3UNhLBKfgga4c+5dM2tYCmNJOKHOov3xXAH52y7IzIS77qLFxo3MP64dD3e8kW9q5V7PT1eJiMSnktTAbzOza4FlwEDn3G/+3mRmfYG+AA0aNCjB7SqeYBs4BZqZ++sEabb1G/61+Fl4dCWccALXXjaCdxudXOy+2l9EpGKIdCn9U0BjoDWwGRjt9Ubn3ATnXFvnXNvatWtHeLuKyStI82bigbZWLbj0vNau7fxr3jjefK4fLX/5FsaNg88+Y2ObNL/XV7ufSMUQUYA757Y45/Y75w4AE4F20R1WYvAK0iSzoFurprdJJePipgz4YhbvTOhLz8/f4tvLr6PK1xvh1luhcmXtLyJSwUUU4GZWt8CX3YBV0RlOYvEK2P0e55Tmz9idg5kz6Xr1+dwx+2kO69ieyqtW0XjKs1Drr57uUDaREpH4FUob4ctAB+AIM/sRGA50MLPW+M6s/Ra4KXZDrLi8+qlHzVvnudJxwWsLqDZsMO2+Ws63tRuw6cnnOeO2awLeQ4EtUjHpVPpyyN9uf3X37mTMl9M5dd5r7KySwtgzr+SFNheSfFBVRnZvCQRfWKMzJ0Xik9ep9ArwciovbLf+soM+q+Zy8zsvcvDuXbzUpgtjzryK7JTD8t9bIyWZ3fsOFAp8w/fPo7zdAQG/y99VUhEp/xTg8WjOHHbccgfVvv2Kdxu24aFOvdlQ+5iwL5OSnMRByZX4bdfeYt9LrZGiHQRFyjmvANdeKOXR2rUwYADMmcNvh6fSr8f9LGx8KphFdLmcvfs99zpRT7hI/FKAlye//uo79X38eDj0UBg9mnM2H8uepGTPjwSaXYdCPeEi8UtnYpYH+/b5Ft80aeL7s3dv3zmUAwZQ+/DDPD+W1xY4/OITi7UjFlUjJVk94SIVjGbgZe2tt3zbvH75JXTq5DtooVWr/G977b3t7+FjXvth3gPMgu9/4JIT89+jLhSRikEBXlbWr4eBA2HWLGjcGKZPh65di9W5Q917u2C/d6B2QQW2SMWhLpQIlKifOjsbRoyAJ5+ElBS47z644w6oWjWmYxaR+KUulCiJeAvYffvgmWd8gf3LL74690MPQZ06pTFsEamAFOBhimgL2AULoH9/WLUKzj4bxo6F1q1LfewiUrEowMMUbAvYgjPzp56Zx2lfvkrdd+ZBw4YwdSp07+7Zz62l7iISDgV4EcFC1N9BClB4C9hqu//g1vdf4YZlM9hXuTI8/LCv0+SggwLeN9LTeUQkMakPvIC8EA31IIU8eVvAVjqwn8tXzmXhhJvo+/F0Mk/sQIc+E8i84FrSxr5PoyFvkpaxsND18gQrzYiIFKUZeAFBDwrGu63v7ade4ZY3xnHC1m/4JPUErr/0AVYddRw1UpJDmll7nnGppe4i4kEBXkCoIVpoj+2vv4ZBd5A+bRqbDjuSWy8ZzJvNzgQzUpKTMCPoLwXwLs1oqbuIeFEJpQCvsPT7+o4dMHQo+5s1J2fmbEa3v5putz/D+6d0wszyl7lne+xRUvSXgo4/E5FwaQZegNey9UHnN81/uLn5tz/ovfFdBrwzmYN+2cbMlp15uP21bK12OOyDFDvAmJ6tC5VaQplZh7riUkQkjwK8AK8QBd9hCC2+/oynF0yk5ZaNrDi6OaP73MuSWo0LXaNoeSTQLwV/91dgi0ioKvxS+mj0VvcYPIXr33iKi9YtYVO1I8jocB0zmp/t2c9twDcZF/odQ/WUZMwge9dezbJFJCQJuZS+xL3VO3fCI48wZfQjHLBKjE27gv+060FOFe9+bvBfHklvk6pebxGJqgod4F5tgf1fWckDM1bnz4SLzYrPbUL66kUwdChs2sTikzoz/Iyr2XxY7ULX8ncWZaAHj6G0KYqIhKpCB3igHursnL1+/15n9acc++Q/YNN6fjvxJAZfMoS3qh9L0WJJJHtsq9dbRKKpwgV4wXpzJTP2h1jjr/v7Noa88xxd1yzmp0NrcW/6XUw7sSO79vk+7yh+0nu4e2yr11tEoqlCBXjRGnMo4Z2y509u/uh1+n48DcPxv2dczn9O68GuKimwr/Dn88I7lFPc/T08DacjRUQkmAq1kMdfjRl8G00V4xxdVy9i4cSb6Pf+y7x9XDs6936aMe2v9oW3h1DKHV57qgCM7N6S1BopGH+daan6t4hEokLNwL3C9YBzjO3ZOn/2e9KmdQxfMIGTN63j86OO4/aud7Ps6BPz3x/opPdQyh2BHlYuHdJJgS0iUVEhAjyvXOFVMKlXI4X0NqkctHUzbugwuqyYz7ZDa3F/+kBePP5sDju4KjWL9GYDEZc79LBSREpD3Ad40bp3USnJSQw+uwE89BAXZGTA/v0wdCi1hw5lRLVqjAhy/UgWAelhpYiUhrgPcK+6N0Bq9YMYa+s59fKb4PvvoUcPGDUKGjUK6dqRLm3Xw0oRKQ1xH+BeZYlWmzcwY8lUWLoUTjoJnn/edx5lKdDGVCJSGuI+wIuWK2rv/JW7Fz/PpavehiOPhIkT4frrISkpwFWiTxtTiUisxX0bYd4+2lX37eGWD17lnQl96frlO2y49mZYvx569y718BYRKQ1xPwNPb12Pem/P4ujxD1Dvt59YfEIaex7O4NyuZ5b10EREYiq+A3zFCujfn3bvvgstW8JrL3J2585lPSoRkVIRtIRiZpPMbKuZrSrwWi0zm29mG3L/rBnbYRaxZYuvNHLKKfDll/DUU/Dpp6DwFpEEEkoN/DnggiKvDQEWOOeaAAtyv4693bvh0UehSROYPBnuvBM2bICbb4bK8f2PCRGRcAUNcOfcu8CvRV7uCkzO/ftkID26wyo2CMjMhBNPhMGDfe2Aq1bB6NFQo0ZMby0iUl5F2oVSxzm3GSD3zyO93mhmfc1smZkt27ZtW2R3u/VW6NaN310Sd17/CI1OvJm06VlkrsiK7HoiIhVAzOsOzrkJwATwnYkZ7uczV2SxYG9japx7My+37sK+Sr6WQB1HJiKJLtIA32JmdZ1zm82sLrA1moPKk7/PyeHN4PBmxb6v48hEJJFFWkKZAfTK/Xsv4I3oDKewQPuc5MnKziEtY6HKKSKScEJpI3wZ+ABoamY/mtmNQAZwrpltAM7N/TrqQt1+Na+cohAXkUQStITinLvC41sxb7r22pbVH5VTRCTRlOu9UPL2OSnIz+Fo+XRggogkknId4OltUoudITmmZ2tSPQ5G0IEJIpJIyv3yRa9tWXVggogkunIf4P7owAQRkTgNcNCBCSIi5boGLiIi3hTgIiJxSgEuIhKnFOAiInFKAS4iEqfMubB3eI38ZmbbgO9K7YbRcwTwc1kPohQl2s8L+pkTRbz+zMc452oXfbFUAzxemdky51zbsh5HaUm0nxf0MyeKivYzq4QiIhKnFOAiInFKAR6aCWU9gFKWaD8v6GdOFBXqZ1YNXEQkTmkGLiISpxTgIiJxSgEeBjO7y8ycmR1R1mOJNTMbZWZrzexzM5tuZjXKekyxYmYXmNk6M/vKzIaU9Xhizczqm9kiM1tjZqvNrF9Zj6k0mFmSma0ws1llPZZoUYCHyMzq4zvA+fuyHkspmQ+0cM61AtYDQ8t4PDFhZknAeKALcAJwhZmdULajirl9wEDnXHPgdODWBPiZAfoBa8p6ENGkAA/dGOBuICGe+jrn3nLO7cv98kPg6LIcTwy1A75yzn3tnNsD/BfoWsZjiinn3Gbn3Ke5f9+BL9Qq9Ob6ZnY0cCHwTFmPJZoU4CEws0uALOfcZ2U9ljJyAzCnrAcRI6nADwW+/pEKHmYFmVlDoA3wURkPJdbG4puAHSjjcURV3J7IE21m9jZwlJ9v3QMMA84r3RHFXqCf2Tn3Ru577sH3T+6XSnNspcj8vJYQ/8oys0OB14H+zrnfy3o8sWJmFwFbnXPLzaxDGQ8nqhTguZxz5/h73cxaAo2Az8wMfKWET82snXPup1IcYtR5/cx5zKwXcBHQ2VXcBQM/AvULfH00sKmMxlJqzCwZX3i/5JybVtbjibE04BIz+xtwEHCYmb3onLu6jMdVYlrIEyYz+xZo65yLxx3NQmZmFwCPA2c757aV9Xhixcwq43tI2xnIAj4BrnTOrS7TgcWQ+WYik4FfnXP9y3g4pSp3Bn6Xc+6iMh5KVKgGLl7GAdWA+Wa20syeLusBxULug9rbgHn4Hua9WpHDO1cacA3QKff/7crc2anEGc3ARUTilGbgIiJxSgEuIhKnFOAiInFKAS4iEqcU4CIicUoBLiISpxTgIiJx6v8BRNxcYGyxzAUAAAAASUVORK5CYII=\n", 430 | "text/plain": [ 431 | "
" 432 | ] 433 | }, 434 | "metadata": { 435 | "needs_background": "light" 436 | }, 437 | "output_type": "display_data" 438 | } 439 | ], 440 | "source": [ 441 | "pyplot.scatter(x_data,y_data)\n", 442 | "pyplot.plot(x_data, w*x_data + b, '-r');" 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": {}, 448 | "source": [ 449 | "It works! That line looks to be fitting the data pretty well. Now we have a \"best fit\" line that represents the data, and that we can use to estimate the value of the dependent variable for any value of the independent variable, even if not present in the data. That is, _to make predictions_.\n", 450 | "\n", 451 | "##### Key idea\n", 452 | "\n", 453 | "> \"Learning\" means building a model by finding the parameters that best fit the data. We do it by minimizing a loss function (a.k.a. cost function), which involves computing derivatives with respect to the parameters in the model. \n", 454 | "\n", 455 | "Here, we used SymPy to help us out with the derivatives, but for more complex models (which may have many parameters), this could be a cumbersome approach. \n", 456 | "Instead, we will make use of the technique of _automatic differentiation_, which evaluates the derivative of a function written in code. \n", 457 | "You'll learn more about it in the next lesson, on **logistic regression**." 458 | ] 459 | }, 460 | { 461 | "cell_type": "markdown", 462 | "metadata": {}, 463 | "source": [ 464 | "## What we've learned\n", 465 | "\n", 466 | "- Gradient descent can find a minimum of a function.\n", 467 | "- Linear regression starts by assuming a linear relationship between two variables.\n", 468 | "- A model includes the assumed relationship in the data and model parameters.\n", 469 | "- Observational data allows finding the parameters in the model (slope, intercept).\n", 470 | "- A loss function captures the deviation between the observed and the predicted values of the dependent variable.\n", 471 | "- We find the parameters by minimizing the loss function via gradient descent.\n", 472 | "- SymPy computes derivatives with `sympy.diff()` and returns numeric functions with `simpy.lambdify()`." 473 | ] 474 | }, 475 | { 476 | "cell_type": "markdown", 477 | "metadata": {}, 478 | "source": [ 479 | "## References\n", 480 | "\n", 481 | "1. Eric Ma, \"Deep Learning Fundamentals: Forward Model, Differentiable Loss Function & Optimization,\" SciPy 2019 tutorial. [video on YouTube](https://youtu.be/JPBz7-UCqRo) and [archive on GitHub](https://github.com/ericmjl/dl-workshop/releases/tag/scipy2019)." 482 | ] 483 | }, 484 | { 485 | "cell_type": "code", 486 | "execution_count": 14, 487 | "metadata": {}, 488 | "outputs": [ 489 | { 490 | "data": { 491 | "text/html": [ 492 | "\n", 493 | "\n", 494 | "\n", 495 | "\n", 625 | "\n" 641 | ], 642 | "text/plain": [ 643 | "" 644 | ] 645 | }, 646 | "execution_count": 14, 647 | "metadata": {}, 648 | "output_type": "execute_result" 649 | } 650 | ], 651 | "source": [ 652 | "# Execute this cell to load the notebook's style sheet, then ignore it\n", 653 | "from IPython.core.display import HTML\n", 654 | "css_file = '../style/custom.css'\n", 655 | "HTML(open(css_file, \"r\").read())" 656 | ] 657 | } 658 | ], 659 | "metadata": { 660 | "kernelspec": { 661 | "display_name": "Python 3", 662 | "language": "python", 663 | "name": "python3" 664 | }, 665 | "language_info": { 666 | "codemirror_mode": { 667 | "name": "ipython", 668 | "version": 3 669 | }, 670 | "file_extension": ".py", 671 | "mimetype": "text/x-python", 672 | "name": "python", 673 | "nbconvert_exporter": "python", 674 | "pygments_lexer": "ipython3", 675 | "version": "3.8.5" 676 | } 677 | }, 678 | "nbformat": 4, 679 | "nbformat_minor": 2 680 | } 681 | -------------------------------------------------------------------------------- /notebooks_en/4_Polynomial_Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "###### Content under Creative Commons Attribution license CC-BY 4.0, code under BSD 3-Clause License © 2021 Lorena A. Barba, Tingyu Wang" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Polynomial Regression\n", 15 | "\n", 16 | "In this fourth lesson of the _Engineering Computations_ module on deep learning, we play with a different model-building method: polynomial regression. \n", 17 | "We already saw one case in which the observational data could not be fit by a line, and that was the classification problem that we tackled with logistic regression (Lesson 3). \n", 18 | "Now imagine that your data wiggles about on the $x, y$ plane—where $x$ is the independent variable or feature, and $y$ is the dependent variable. \n", 19 | "You look at it and think: a curvilinear relationship might work. Good idea. \n", 20 | "\n", 21 | "It may surprise you to learn that fitting a polynomial to the data is a special case of multiple linear regression. Yes: _linear_. \n", 22 | "When we talk about linear models, we mean linear with respect to the parameters!" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "## A special case of multiple linear regression\n", 30 | "\n", 31 | "Let's generate some synthetic data using a polynomial function of fourth order, $y = x^4 + x^3 - 4x^2 $, with a bit of added noise.\n", 32 | "\n", 33 | "As usuall, we start by loading some needed Python libraries and functions, including the useful tools from `autograd` that you've learned about already." 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 1, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "from matplotlib import pyplot\n", 43 | "from autograd import grad\n", 44 | "from autograd import numpy" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": 2, 50 | "metadata": {}, 51 | "outputs": [ 52 | { 53 | "data": { 54 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAATNUlEQVR4nO3df2xd513H8fcXE+BqA7lV3ZK4LRkoGAZlC7KqoSI01g1XgBZTqWj8UgSVokn8GBKYJUxiAlS1yBICIYSItkEQ+0G0pWnEYF6XMvFDbKu7lKVdalqNrY0TGjOw2MDa0uzLHz4uTuo0uT7n5vg89/2SrHvPc38836Pkfu45z/PceyMzkSSV6evaLkCSNDiGvCQVzJCXpIIZ8pJUMENekgr29W0XsN4NN9yQO3fubLsMSeqUxx577D8yc2yj27ZUyO/cuZP5+fm2y5CkTomIL1zuNodrJKlghrwkFcyQl6SCGfKSVDBDXpIKtqVW10jSsDl6YpHZuQXOLK+wY7THzNQE07vHG3t+Q16SWnL0xCIHjpxk5fwFABaXVzhw5CRAY0HvcI0ktWR2buHFgF+zcv4Cs3MLjfVhyEtSS84sr/TVvhmGvCS1ZMdor6/2zTDkJaklM1MT9LaNXNTW2zbCzNREY3048SpJLVmbXHV1jSQVanr3eKOhfimHaySpYIa8JBXMkJekghnyklQwQ16SCtZIyEfEaER8MCKeiohTEfEDEXF9RDwcEU9Xl9c10Zck6eo1dST/h8BHMvO7gNcAp4D9wPHM3AUcr7YlSddQ7ZCPiG8Bfgh4N0BmfjUzl4E9wKHqboeA6bp9SZL608SR/LcDS8CfRcSJiHhXRLwCuCkzzwJUlzdu9OCI2BcR8xExv7S01EA5kqQ1TYT81wPfD/xJZu4G/oc+hmYy82BmTmbm5NjYWAPlSJLWNBHyp4HTmfnJavuDrIb+8xGxHaC6PNdAX5KkPtQO+cz8d+C5iFj72rQ7gc8Cx4C9Vdte4KG6fUmS+tPUF5T9MvDeiPgG4HPAz7P6BnI4Iu4FngXuaagvSdJVaiTkM/NxYHKDm+5s4vklSZvjJ14lqWCGvCQVzJCXpIIZ8pJUMENekgpmyEtSwQx5SSqYIS9JBTPkJalghrwkFcyQl6SCGfKSVDBDXpIKZshLUsEMeUkqmCEvSQVr5EdDIuLzwJeAC8ALmTkZEdcDfwXsBD4P/GRm/lcT/UmSrk6TR/I/nJmvzcy1X4jaDxzPzF3A8WpbknQNDXK4Zg9wqLp+CJgeYF+SpA00FfIJfDQiHouIfVXbTZl5FqC6vHGjB0bEvoiYj4j5paWlhsqRJEFDY/LAHZl5JiJuBB6OiKeu9oGZeRA4CDA5OZkN1SNJoqEj+cw8U12eAx4Ebgeej4jtANXluSb6kiRdvdohHxGviIhvXrsO/AjwBHAM2FvdbS/wUN2+JEn9aWK45ibgwYhYe773ZeZHIuJR4HBE3As8C9zTQF+SpD7UDvnM/Bzwmg3avwjcWff5JUmb5ydeJalghrwkFcyQl6SCNbVOXpKG0tETi8zOLXBmeYUdoz1mpiaY3j3edlkvMuQlaZOOnljkwJGTrJy/AMDi8goHjpwE2DJB73CNJG3S7NzCiwG/ZuX8BWbnFlqq6KUMeUnapDPLK321t8GQl6RN2jHa66u9DYa8JG3SzNQEvW0jF7X1to0wMzXRUkUv5cSrJG3S2uSqq2skqVDTu8e3VKhfyuEaSSqYIS9JBTPkJalghrwkFayxkI+IkYg4ERF/XW1fHxEPR8TT1eV1TfUlSbo6TR7Jvw04tW57P3A8M3cBx6ttSdI11EjIR8TNwI8B71rXvAc4VF0/BEw30Zck6eo1dST/B8BvAF9b13ZTZp4FqC5vbKgvSdJVqh3yEfHjwLnMfGyTj98XEfMRMb+0tFS3HEnSOk0cyd8BvDkiPg98AHhDRPwl8HxEbAeoLs9t9ODMPJiZk5k5OTY21kA5kqQ1tUM+Mw9k5s2ZuRN4C/BIZv4scAzYW91tL/BQ3b4kSf0Z5Dr5B4A3RcTTwJuqbUnSNdToF5Rl5seBj1fXvwjc2eTzS5L64ydeJalghrwkFcyQl6SCGfKSVDBDXpIKZshLUsEMeUkqmCEvSQUz5CWpYIa8JBXMkJekgjX63TVtOXpikdm5Bc4sr7BjtMfM1ATTu8fbLkuSWtf5kD96YpEDR06ycv4CAIvLKxw4chLAoJc09Do/XDM7t/BiwK9ZOX+B2bmFliqSpK2j8yF/Znmlr3ZJGiadD/kdo72+2iVpmDTxQ97fFBGfioh/iYgnI+K3q/brI+LhiHi6uryufrkvNTM1QW/byEVtvW0jzExNDKI7SeqUJo7kvwK8ITNfA7wWuCsiXgfsB45n5i7geLXduOnd49x/922Mj/YIYHy0x/133+akqyTRwOqazEzgy9XmtuovgT3A66v2Q6z+LODb6/a3kend44a6JG2gkTH5iBiJiMeBc8DDmflJ4KbMPAtQXd54mcfui4j5iJhfWlpqohxJUqWRkM/MC5n5WuBm4PaI+N4+HnswMyczc3JsbKyJciRJlUZX12TmMqvDMncBz0fEdoDq8lyTfUmSrqyJ1TVjETFaXe8BbwSeAo4Be6u77QUeqtuXJKk/TXytwXbgUESMsPqmcTgz/zoi/hk4HBH3As8C9zTQlySpD02srvkMsHuD9i8Cd9Z9fknS5nX+E6+SpMsz5CWpYIa8JBXMkJekghnyklQwQ16SCmbIS1LBDHlJKpghL0kFM+QlqWCGvCQVzJCXpII18S2UktSaoycWmZ1b4MzyCjtGe8xMTfhzoOsY8pI66+iJRQ4cOcnK+QsALC6vcODISQCDvmLIS+qs2bmFFwN+zcr5C8zOLVx1yJd+JmDIS+qsM8srfbVfahjOBJr4+b9bIuLvIuJURDwZEW+r2q+PiIcj4unq8rr65UrS/9sx2uur/VIvdyZQiiZW17wA/FpmfjfwOuAXI+LVwH7geGbuAo5X25LUmJmpCXrbRi5q620bYWZq4qoeX/dMoAtqh3xmns3MT1fXvwScAsaBPcCh6m6HgOm6fUnSetO7x7n/7tsYH+0RwPhoj/vvvu2qh1rqngl0QaNj8hGxk9Xfe/0kcFNmnoXVN4KIuPEyj9kH7AO49dZbmyxH0hCY3j2+6fHzmamJi8bkob8zgS5o7MNQEfFK4EPAr2bmf1/t4zLzYGZOZubk2NhYU+VI0hXVPRPogkaO5CNiG6sB/97MPFI1Px8R26uj+O3AuSb6kqQm1TkT6IImVtcE8G7gVGb+/rqbjgF7q+t7gYfq9iVJ6k8TR/J3AD8HnIyIx6u23wQeAA5HxL3As8A9DfQlSepD7ZDPzH8E4jI331n3+SVJm+e3UEpSwQx5SSqYIS9JBTPkJalghrwkFcyQl6SCGfKSVDBDXpIKZshLUsEMeUkqmCEvSQUz5CWpYIa8JBXMkJekgjX6G6+S+nf0xCKzcwucWV5hx2iPmamJon+pSNeWIb8F+CIfXkdPLF70Q9KLyyscOHISwP8DakQjwzUR8Z6IOBcRT6xruz4iHo6Ip6vL65roqzRrL/LF5RWS/3+RHz2x2HZpugZm5xZeDPg1K+cvMDu30FJFKk1TY/J/Dtx1Sdt+4Hhm7gKOV9u6hC/y4XZmeaWvdqlfjYR8Zv498J+XNO8BDlXXDwHTTfRVGl/kw23HaK+vdqlfg1xdc1NmngWoLm/c6E4RsS8i5iNifmlpaYDlbE2+yIfbzNQEvW0jF7X1to0wMzXRUkUqTetLKDPzYGZOZubk2NhY2+Vcc77Ih9v07nHuv/s2xkd7BDA+2uP+u29z0lWNGeTqmucjYntmno2I7cC5AfbVWWsvZlfXDK/p3eP+e2tgBhnyx4C9wAPV5UMD7KvTfJFLGpRGQj4i3g+8HrghIk4D72Q13A9HxL3As8A9TfQ1CK5Tl1SqRkI+M3/qMjfd2cTzD1IJH0bxTUrS5bQ+8dq2rq9T98NUkl7O0Id819epd/1NStJgDX3Id32detffpCQN1tCHfNfXqXf9TUo6emKROx54hFft/zB3PPCIQ40NG/qQ7/qHUbr+JqX6uhySzikNnl81TLfXqfthquHW9dVhLzen1IX6u8CQL0CX36RUT9dD0jmlwTPkG+A6dbWl6yG5Y7TH4ga1OqfUHEO+pq6fLjeh629yXa6/6yE5MzVx0esHnFNq2tBPvNY17OvUuz5x1vX6uz7x3vWFD13gkXxNXT9drqvrY8Jdr38rTLzXPRNyTmmwDPmaun66XFfX3+S6Xj+0G5IOV259DtfU1PXT5bq6/mGsrtfftmEfruwCQ76mYR9T7PqbXBP1d/nDSHWVcCZUOodrGjDMY4pbYUy4jrr1D/twxbAPV3ZBZOZgO4i4C/hDYAR4V2Y+cLn7Tk5O5vz8/EDrkZp0xwOPbBhy46M9/mn/G1qo6Nq69E0OVs+EhulsdiuIiMcyc3Kj2wZ6JB8RI8AfA28CTgOPRsSxzPzsIPuVrpVhH67o+pncMBj0cM3twDOZ+TmAiPgAsAcw5FUEhyuGe7iyCwY98ToOPLdu+3TVJhWh6xPPMNwTx8Ng0EfysUHbRZMAEbEP2Adw6623DrgcqVldH64Y9onjYTDokD8N3LJu+2bgzPo7ZOZB4CCsTrwOuB4VqO3vnunycEXXP/GrKxt0yD8K7IqIVwGLwFuAnx5wnxoiHonWM+wTx8NgoGPymfkC8EvAHHAKOJyZTw6yTw0XP3FZj5/4Ld/AP/GamX+Tmd+Zmd+RmfcNuj8NF49E6ylh4lgvz681UKd5JFrPsH8txzDwaw3UujoTp/7oRH1dnjjWlRnyalXdidOuL2GUBs2QV6uaWMLnkah0eY7Jq1VOnEqDZcirVU6cSoNlyKtVLuGTBssxebXKiVNpsAx5tc6JU2lwHK6RpIIZ8pJUMENekgpmyEtSwQx5SSqYIS9JBTPkJalgtUI+Iu6JiCcj4msRMXnJbQci4pmIWIiIqXplSpI2o+6HoZ4A7gb+dH1jRLya1d9z/R5gB/CxiPjOzLzw0qeQJA1KrSP5zDyVmRv9mOYe4AOZ+ZXM/DfgGeD2On1Jkvo3qDH5ceC5ddunq7aXiIh9ETEfEfNLS0sDKkeShtMVh2si4mPAt25w0zsy86HLPWyDttzojpl5EDgIMDk5ueF9JEmbc8WQz8w3buJ5TwO3rNu+GTizieeRJNUwqOGaY8BbIuIbI+JVwC7gUwPqS5J0GXWXUP5ERJwGfgD4cETMAWTmk8Bh4LPAR4BfdGWNJF17tZZQZuaDwIOXue0+4L46zy9JqsdPvEpSwQx5SSqYIS9JBfM3XsXRE4v+kLZUKEN+yB09sciBIydZOb+6+GlxeYUDR04CGPRSARyuGXKzcwsvBvyalfMXmJ3b6CuJJHWNIT/kziyv9NUuqVsM+SG3Y7TXV7ukbjHkh9zM1AS9bSMXtfW2jTAzNdFSRZKa5MTrkFubXHV1jVQmQ15M7x431KVCOVwjSQUz5CWpYIa8JBXMkJekghnyklSwyNw6v50dEUvAF2o8xQ3AfzRUTptK2Q9wX7aiUvYD3Jc135aZYxvdsKVCvq6ImM/MybbrqKuU/QD3ZSsqZT/AfbkaDtdIUsEMeUkqWGkhf7DtAhpSyn6A+7IVlbIf4L5cUVFj8pKki5V2JC9JWseQl6SCFRXyEfG7EfGZiHg8Ij4aETvarmmzImI2Ip6q9ufBiBhtu6bNioh7IuLJiPhaRHRuuVtE3BURCxHxTETsb7uezYqI90TEuYh4ou1a6oqIWyLi7yLiVPV/621t17QZEfFNEfGpiPiXaj9+u/E+ShqTj4hvycz/rq7/CvDqzHxry2VtSkT8CPBIZr4QEb8HkJlvb7msTYmI7wa+Bvwp8OuZOd9ySVctIkaAfwXeBJwGHgV+KjM/22phmxARPwR8GfiLzPzetuupIyK2A9sz89MR8c3AY8B01/5dIiKAV2TmlyNiG/CPwNsy8xNN9VHUkfxawFdeAXT2HSwzP5qZL1SbnwBubrOeOjLzVGZ29ZfBbweeyczPZeZXgQ8Ae1quaVMy8++B/2y7jiZk5tnM/HR1/UvAKaBzP4qQq75cbW6r/hrNraJCHiAi7ouI54CfAX6r7Xoa8gvA37ZdxJAaB55bt32aDoZJySJiJ7Ab+GTLpWxKRIxExOPAOeDhzGx0PzoX8hHxsYh4YoO/PQCZ+Y7MvAV4L/BL7Vb78q60L9V93gG8wOr+bFlXsy8dFRu0dfYMsTQR8UrgQ8CvXnIm3xmZeSEzX8vq2frtEdHoUFrnfv4vM994lXd9H/Bh4J0DLKeWK+1LROwFfhy4M7f45Ekf/y5dcxq4Zd32zcCZlmrROtUY9oeA92bmkbbrqSszlyPi48BdQGOT4507kn85EbFr3eabgafaqqWuiLgLeDvw5sz837brGWKPArsi4lUR8Q3AW4BjLdc09KoJy3cDpzLz99uuZ7MiYmxt5VxE9IA30nBulba65kPABKsrOb4AvDUzF9utanMi4hngG4EvVk2f6PBKoZ8A/ggYA5aBxzNzqtWi+hARPwr8ATACvCcz72u3os2JiPcDr2f1K22fB96Zme9utahNiogfBP4BOMnq6x3gNzPzb9qrqn8R8X3AIVb/b30dcDgzf6fRPkoKeUnSxYoarpEkXcyQl6SCGfKSVDBDXpIKZshLUsEMeUkqmCEvSQX7PxmPnhXp37F8AAAAAElFTkSuQmCC\n", 55 | "text/plain": [ 56 | "
" 57 | ] 58 | }, 59 | "metadata": { 60 | "needs_background": "light" 61 | }, 62 | "output_type": "display_data" 63 | } 64 | ], 65 | "source": [ 66 | "numpy.random.seed(0) # fix seed for reproducibility\n", 67 | "x = numpy.linspace(-3, 3, 20)\n", 68 | "y = x**4 + x**3 - 4*x**2 + 8*numpy.random.normal(size=len(x))\n", 69 | "pyplot.scatter(x, y);" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "Suppose we were only given the data (not the function that generated them), and our goal is to fit a curve to these data. The nonlinear relationship between $x$ and $y$ suggests that a linear regression will fail. Intuitively, using polynomial functions may first come to your mind.\n", 77 | "\n", 78 | "Let's write the model as a $d$th-order polynomial on $x$, the only feature:\n", 79 | "\n", 80 | "$$\n", 81 | "\\hat{y} = w_0 + w_1 x + w_2 x^2 + \\cdots + w_d x^d, \n", 82 | "$$\n", 83 | "\n", 84 | "where $w$ denotes the weights. Keep in mind that in the model-fitting context, the objective is always to find the optimal values of these weights given $x$ and $y$. When viewed from a different perspective, the model above is just a linear combination of the weights. In fact, by creating polynomial features of $x$, namely, letting $x_i = x^i$, the model becomes:\n", 85 | "\n", 86 | "$$\n", 87 | "\\hat{y} = w_0 + w_1 x_1 + w_2 x_2 + \\ldots + w_d x_d.\n", 88 | "$$\n", 89 | "\n", 90 | "As you can see, the polynomial regression model is identical to multiple linear regression, with the matrix form being also $\\hat{\\mathbf{y}} = X\\mathbf{w}$, and the only gap is forming the matrix $X$ using the powers of $x$.\n", 91 | "\n", 92 | "Suppose we want to fit our data with a 3rd degree polynomial function; let's write a function to create these polynomial features first." 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 3, 98 | "metadata": {}, 99 | "outputs": [ 100 | { 101 | "name": "stdout", 102 | "output_type": "stream", 103 | "text": [ 104 | "(20, 4)\n" 105 | ] 106 | } 107 | ], 108 | "source": [ 109 | "degree = 3\n", 110 | "\n", 111 | "def polynomial_features(x, degree):\n", 112 | " \"\"\" Generate polynomial features for x.\"\"\"\n", 113 | " \n", 114 | " X = numpy.empty((len(x), degree+1))\n", 115 | " for i in range(degree+1):\n", 116 | " X[:,i] = x**i\n", 117 | " return X\n", 118 | "\n", 119 | "X = polynomial_features(x, degree)\n", 120 | "print(X.shape)" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": {}, 126 | "source": [ 127 | "##### Note\n", 128 | "\n", 129 | "> Unsurprisingly, **scikit-learn** offers a counterpart: [`PolynomialFeatures()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html).\n", 130 | "When your original data come with multiple features (e.g., $x_1$ and $x_2$), the polynomial features for regression will involve interaction terms (e.g., $x_1 x_2$ for 2nd-order polynomials).\n", 131 | "In that case, it is handy to use this function to generate all terms." 132 | ] 133 | }, 134 | { 135 | "cell_type": "markdown", 136 | "metadata": {}, 137 | "source": [ 138 | "## Scale the data, train the model\n", 139 | "\n", 140 | "Recall that for multiple linear regression, we should normalize each feature to the same scale.\n", 141 | "As in the previous lesson, let's use `MinMaxScaler()` to scale all features to $[0,1]$, except for the first column:\n", 142 | "$x_0$ is set to 1 for all entries, since $w_0$ represents the intercept." 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 4, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "from sklearn.preprocessing import MinMaxScaler\n", 152 | "\n", 153 | "min_max_scaler = MinMaxScaler()\n", 154 | "X_scaled = min_max_scaler.fit_transform(X)\n", 155 | "X_scaled[:,0] = 1 # the column for intercept" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "When scaling the matrix of polynomial features $X$ above, we used:\n", 163 | "```python\n", 164 | "X_scaled = min_max_scaler.fit_transform(X)\n", 165 | "```\n", 166 | "The function `fit_transform()` is actually a combination of two steps:\n", 167 | "- [`fit()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler.fit): compute $x_{\\min}$ and $x_{\\max}$ values for each feature and save the information to the variable `min_max_scaler`. You can access them via `min_max_scaler.data_min_` and `min_max_scaler.data_max_`.\n", 168 | "- [`transform()`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler.transform) use `min_max_scaler` to scale `X`.\n", 169 | "\n", 170 | "This will be helpful to remember a bit later." 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "We can reuse the same model and loss function from Lesson 3:" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": 5, 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "def linear_regression(params, X):\n", 187 | " '''\n", 188 | " The linear regression model in matrix form.\n", 189 | " Arguments:\n", 190 | " params: 1D array of weights for the linear model\n", 191 | " X : 2D array of input values\n", 192 | " Returns:\n", 193 | " 1D array of predicted values\n", 194 | " '''\n", 195 | " return numpy.dot(X, params)\n", 196 | "\n", 197 | "def mse_loss(params, model, X, y):\n", 198 | " '''\n", 199 | " The mean squared error loss function.\n", 200 | " Arguments:\n", 201 | " params: 1D array of weights for the linear model\n", 202 | " model : function for the linear regression model\n", 203 | " X : 2D array of input values\n", 204 | " y : 1D array of predicted values\n", 205 | " Returns:\n", 206 | " float, mean squared error\n", 207 | " '''\n", 208 | " y_pred = model(params, X)\n", 209 | " return numpy.mean( numpy.sum((y-y_pred)**2) )\n", 210 | "\n", 211 | "gradient = grad(mse_loss)" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "Remember, \"training\" a model simply means finding the best parameters by minimizing a loss function. \n", 219 | "We'll choose both a maximum number of iterations in the optimization loop, and an exit criterion based on the norm of the gradient at the current iteration. \n", 220 | "(If this is quite small, when multiplied by the also-small learning rate, the parameters will change very little.)" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": 6, 226 | "metadata": {}, 227 | "outputs": [ 228 | { 229 | "name": "stdout", 230 | "output_type": "stream", 231 | "text": [ 232 | "iteration 0, loss = 5434.768, mae = 11.057\n", 233 | "iteration 100, loss = 1300.477, mae = 6.885\n", 234 | "iteration 200, loss = 1281.309, mae = 6.864\n", 235 | "iteration 300, loss = 1272.990, mae = 6.808\n", 236 | "iteration 400, loss = 1267.448, mae = 6.760\n", 237 | "iteration 500, loss = 1263.750, mae = 6.722\n", 238 | "iteration 600, loss = 1261.282, mae = 6.690\n", 239 | "iteration 700, loss = 1259.636, mae = 6.664\n", 240 | "iteration 800, loss = 1258.537, mae = 6.643\n", 241 | "iteration 900, loss = 1257.804, mae = 6.625\n", 242 | "iteration 1000, loss = 1257.314, mae = 6.611\n", 243 | "iteration 1100, loss = 1256.988, mae = 6.600\n", 244 | "iteration 1200, loss = 1256.770, mae = 6.590\n", 245 | "iteration 1300, loss = 1256.625, mae = 6.583\n", 246 | "iteration 1400, loss = 1256.528, mae = 6.576\n", 247 | "iteration 1500, loss = 1256.463, mae = 6.571\n", 248 | "iteration 1600, loss = 1256.420, mae = 6.567\n", 249 | "iteration 1700, loss = 1256.391, mae = 6.564\n", 250 | "iteration 1800, loss = 1256.372, mae = 6.561\n", 251 | "iteration 1900, loss = 1256.359, mae = 6.558\n", 252 | "iteration 2000, loss = 1256.350, mae = 6.557\n", 253 | "iteration 2100, loss = 1256.345, mae = 6.555\n", 254 | "iteration 2200, loss = 1256.341, mae = 6.554\n", 255 | "iteration 2300, loss = 1256.338, mae = 6.553\n", 256 | "iteration 2400, loss = 1256.337, mae = 6.552\n", 257 | "iteration 2500, loss = 1256.336, mae = 6.551\n", 258 | "iteration 2600, loss = 1256.335, mae = 6.551\n", 259 | "iteration 2700, loss = 1256.334, mae = 6.550\n", 260 | "iteration 2800, loss = 1256.334, mae = 6.550\n", 261 | "iteration 2900, loss = 1256.334, mae = 6.550\n" 262 | ] 263 | } 264 | ], 265 | "source": [ 266 | "max_iter = 3000\n", 267 | "alpha = 0.01\n", 268 | "params = numpy.zeros(X_scaled.shape[1])\n", 269 | "descent = numpy.ones(X_scaled.shape[1])\n", 270 | "i = 0\n", 271 | "\n", 272 | "from sklearn.metrics import mean_absolute_error\n", 273 | "\n", 274 | "while numpy.linalg.norm(descent) > 0.01 and i < max_iter:\n", 275 | " descent = gradient(params, linear_regression, X_scaled, y)\n", 276 | " params = params - descent * alpha\n", 277 | " loss = mse_loss(params, linear_regression, X_scaled, y)\n", 278 | " mae = mean_absolute_error(y, X_scaled@params)\n", 279 | " if i%100 == 0:\n", 280 | " print(f\"iteration {i:4}, {loss = :.3f}, {mae = :.3f}\")\n", 281 | " i += 1" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "metadata": {}, 287 | "source": [ 288 | "Let's print out the weights." 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 7, 294 | "metadata": {}, 295 | "outputs": [ 296 | { 297 | "data": { 298 | "text/plain": [ 299 | "array([-22.51572398, 6.75930601, 41.30788709, 30.0105898 ])" 300 | ] 301 | }, 302 | "execution_count": 7, 303 | "metadata": {}, 304 | "output_type": "execute_result" 305 | } 306 | ], 307 | "source": [ 308 | "params" 309 | ] 310 | }, 311 | { 312 | "cell_type": "markdown", 313 | "metadata": {}, 314 | "source": [ 315 | "The first term is the intercept, and the rest are the weights of each **scaled** polynomial feature.\n", 316 | "\n", 317 | "Although our model has multiple weights, there's only one feature in our original data. \n", 318 | "Therefore, if we are given new values of $x$ to predict $y$, we need to first create the polynomial features, scale them to $[0,1]$ using the **same scaling function**, and then multiply by the weights. Since we have used the min-max scaling, it is important to use the same $x_{\\min}$ and $x_{\\max}$ of the training data to scale new data.\n", 319 | "\n", 320 | "Recall the explanation above for the scaling step. We now only need to call `min_max_scaler.transform()` to scale new data. Such design occurs very often in scikit-learn.\n", 321 | "\n", 322 | "With that in mind, let's plot the fitted curve together with the data. We generate some new values of $x$, `xgrid`, and predict $y$ at these locations. Don't forget to repeat the procedure of creating polynomial features and scaling them. Ponder over this code for a moment to wrap your head around it." 323 | ] 324 | }, 325 | { 326 | "cell_type": "code", 327 | "execution_count": 8, 328 | "metadata": {}, 329 | "outputs": [ 330 | { 331 | "data": { 332 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqzElEQVR4nO3deXxU1f3/8dcnC4QEwhIChIQsLLKvCTsqCihWCm64hV+pG1qrdrEqFqu2lqqV+tXaKqaiYolYNyqiRZBFQNawyS5LFsKWEAgkhOzn98dMMGACSeZO7szk83w8eNzcOzP3fiaa95w599x7xBiDUkop3+RndwFKKaXcR0NeKaV8mIa8Ukr5MA15pZTyYRrySinlwwLsLqCy1q1bm9jYWLvLUEopr7Jx48bjxpjwqh7zqJCPjY0lJSXF7jKUUsqriEh6dY9pd41SSvkwDXmllPJhGvJKKeXDPKpPviolJSVkZmZSWFhodyleLygoiKioKAIDA+0uRSlVTzw+5DMzM2nWrBmxsbGIiN3leC1jDDk5OWRmZhIXF2d3OUqpeuLx3TWFhYWEhYVpwLtIRAgLC9NvREp5muRkiI0FPz/HMjnZ0t17fEse0IC3iP4elfIwyckwZQoUFDjW09Md6wCJiZYcwuNb8kop5bOmTfsh4CsUFDi2W0RDvp4tX76ccePGATB//nxeeOGFap+bm5vL66+/XutjPPvss8yYMaPONSql6klGRu2214GGvEXKyspq/Zrx48czderUah+va8grpbxEdHTttteBhnwNpKWl0a1bNyZPnkyfPn245ZZbKCgoIDY2lj/96U+MGDGCjz76iEWLFjF06FAGDBjAxIkTyc/PB2DhwoV069aNESNG8Omnn57b77vvvstDDz0EwLFjx7jxxhvp27cvffv2ZfXq1UydOpX9+/fTr18/HnvsMQBeeuklBg4cSJ8+fXjmmWfO7Wv69Ol07dqV0aNHs2fPnnr87Sil6mz6dAgOPn9bcLBju0W84sRrhT9+voOdh09bus8e7UN55qc9L/m8PXv2MGvWLIYPH87dd999roUdFBTEqlWrOH78ODfddBNff/01ISEhvPjii7z88ss8/vjj3HfffSxdupTOnTtz2223Vbn/Rx55hCuvvJJ58+ZRVlZGfn4+L7zwAtu3b2fLli0ALFq0iL1797J+/XqMMYwfP54VK1YQEhLCBx98wObNmyktLWXAgAHEx8db9jtSSrlJxcnVadMcXTTR0Y6At+ikK3hZyNupQ4cODB8+HIBJkybx97//HeBcaK9du5adO3eee05xcTFDhw5l9+7dxMXF0aVLl3OvTUpK+tH+ly5dynvvvQeAv78/zZs35+TJk+c9Z9GiRSxatIj+/fsDkJ+fz969e8nLy+PGG28k2NkiGD9+vNVvXynlLomJlob6hbwq5GvS4naXC4cfVqyHhIQAjouNxowZw9y5c8973pYtWywbumiM4cknn+T+++8/b/srr7yiwyOVUlXSPvkaysjIYM2aNQDMnTuXESNGnPf4kCFD+Pbbb9m3bx8ABQUFfP/993Tr1o3U1FT2799/7rVVGTVqFG+88QbgOIl7+vRpmjVrRl5e3rnnXHvttbz99tvn+voPHTpEVlYWV1xxBfPmzePs2bPk5eXx+eefW/vmlVJeS0O+hrp3787s2bPp06cPJ06c4Be/+MV5j4eHh/Puu+9yxx130KdPH4YMGcLu3bsJCgoiKSmJ66+/nhEjRhATE1Pl/l999VWWLVtG7969iY+PZ8eOHYSFhTF8+HB69erFY489xjXXXMOdd97J0KFD6d27N7fccgt5eXkMGDCA2267jX79+nHzzTdz+eWX18evRCnlBcQYY3cN5yQkJJgLJw3ZtWsX3bt3t6kih7S0NMaNG8f27dttrcMKnvD7VEpZS0Q2GmMSqnpMW/JKKeXDLAl5EWkhIh+LyG4R2SUiQ0WklYgsFpG9zmVLK45lh9jYWJ9oxSulGh6rWvKvAguNMd2AvsAuYCqwxBjTBVjiXFdKKVWPXA55EQkFrgBmARhjio0xucAEYLbzabOBG1w9llJKqdqxoiXfEcgG3hGRzSLyloiEAG2NMUcAnMs2Vb1YRKaISIqIpGRnZ1tQjlJKqQpWhHwAMAB4wxjTHzhDLbpmjDFJxpgEY0xCeHi4BeUopZSqYEXIZwKZxph1zvWPcYT+MRGJAHAusyw4li30bpBKKW/lcsgbY44CB0Wkq3PTKGAnMB+Y7Nw2GfjM1WPZpbqQr8vthZVSqj5ZNbrmYSBZRL4D+gF/AV4AxojIXmCMc9393DBfYuVb/g4cOJCrrrqKO++8k969e5OWlkavXr3OPXfGjBk8++yzAOzfv5+xY8cSHx/P5Zdfzu7du12uRSmlasOSG5QZY7YAVV1tNcqK/deYm+ZLrHzL3+XLl3P99dezfft24uLiSEtLq/Z1U6ZMYebMmXTp0oV169bx4IMPsnTp0jrXoZRSteVVd6G8pIvNl2jhrTwHDRpEXFzcRZ+Tn5/P6tWrmThx4rltRUVFltWglFI14VshXw/zJcIPtxcGCAgIoLy8/Nx6YWEhAOXl5bRo0eLchB9KKWUH37p3jZvmS7zwlr+VtW3blqysLHJycigqKmLBggUAhIaGEhcXx0cffQQ47gW/detWl+pQSqna8q2Qd9N8iRfe8reywMBAnn76aQYPHsy4cePo1q3buceSk5OZNWsWffv2pWfPnnz2mdcOMFJKeSnfu9VwcrJb50v0dnqrYaU8z6MfbqVjeAi/vKpznV7fsG41nJgIaWlQXu5YasArpTzY7qOn+WRTJkWl5Zd+ch34XsgrpZQXeW3pPkIa+XP38Fi37N8rQt6TupS8mf4elfIs+7Ly+HLbESYPi6VFcCO3HMPjQz4oKIicnBwNKBcZY8jJySEoKMjuUpRSTv9Yuo+gAH/uGXHx625c4fHj5KOiosjMzERvQ+y6oKAgoqKi7C5DKQWkHj/D/K2HuffyjoQ1bey243h8yAcGBl7y6lKllPI2/1y2j0B/P+67vKNbj+Px3TVKKeVrDp4oYN7mQ9w5OJrwZu5rxYOGvFJK1bvXl+/D30944MpObj+WhrxSStWjQ7ln+XhjJrcldKBtqPsHQmjIK6VUPZq5fD8AD4x0fyseNOSVUqreHD1VyH82HOSW+CgiWzSpl2NaMrpGRNKAPKAMKDXGJIhIK+A/QCyQBtxqjDlpxfGUUsobvbliP2XG8ODIut2jpi6sbMlfZYzpV+kmOVOBJcaYLsAS57pSSjVIWXmFvL8ugxv7R9KhVfClX2ARd3bXTABmO3+eDdzgxmMppZRHe2tlKiVl5XW+02RdWRXyBlgkIhtFxDmpKm2NMUcAnMs2Vb1QRKaISIqIpOhVrUopX5STX8S/16QzoV8kca1DLv0CC1l1xetwY8xhEWkDLBaR3TV9oTEmCUgCx/3kLapHKaU8xqxVqRSWltV7Kx4saskbYw47l1nAPGAQcExEIgCcyywrjqWUUt4kt6CY99akc33vCDq3aVrvx3c55EUkRESaVfwMXANsB+YDk51Pmwzo3HdKqQbn7W/TyC8q5aGr678VD9Z017QF5olIxf7eN8YsFJENwIcicg+QAUy04FhKKeU1TheW8M63qYzt2Y5u7UJtqcHlkDfGHAD6VrE9Bxjl6v6VUspbzf42jbxC+1rxoFe8KqWUW+QXlTLr21RGd29Dr8jmttWhIa+UUm7wzqpUcgtKePjqLrbWoSGvlFIWy8kv4s0VB7i2Z1v6dmhhay0a8kop5YrkZIiNBT8/xzI5mX8s28fZkjIeH9vN7uo05JVSqs6Sk2HKFEhPB2MgPZ2M3z3FnNWp3JrQgU7h9T8u/kIa8kopVVfTpkFBwXmb/pZwC/4lJfx6tL198RU05JVSqq4yMs5b3d6mI5/1HMk9G/5bL7M+1YSGvFJK1VV09HmrL478OS0LTnH/4XU2FfRjGvJKKVVX06dDsOPe8Kti+rIybgAPpcwj9Nk/2FzYD6y6C6VSSjU8iYkAlE97ihdG3kXkmRwm/eKGc9s9gYa8Ukq5IjGRBb2uYvvczfzfbX1p3D/K7orOo901SinlguLScmZ8tYfuEaFM6Btpdzk/oiGvlFIumLs+g4wTBUy9rht+fmJ3OT+iIa+UUnWUV1jC35fsZVinMK7o0trucqqkIa+UUnX0r5Wp5JwpZup13XDOqeFxLAt5EfEXkc0issC53kpEFovIXueypVXHUkopu2XlFfLWygOM6xNBn6gWdpdTLStb8r8CdlVanwosMcZ0AZY415VSyie8tmQfxaXl/O6arnaXclGWhLyIRAHXA29V2jwBmO38eTZwgxXHUkopu6UeP8Pc9RncOTia2NYhdpdzUVa15F8BHgfKK21ra4w5AuBctrHoWEopZasZX+2hcYCf7ROC1ITLIS8i44AsY8zGOr5+ioikiEhKdna2q+UopZRbbTmYyxfbjnDfFR0Jb9bY7nIuyYqW/HBgvIikAR8AV4vIHOCYiEQAOJdZVb3YGJNkjEkwxiSEh4dbUI5SSrmHMYbnv9xF66aNuPfyjnaXUyMuh7wx5kljTJQxJha4HVhqjJkEzAcmO582GfjM1WMppZSdPv/uCOtST/CbMZfRtLF33BXGnePkXwDGiMheYIxzXSmlvFJ+USnTv9hJ78jm3D4w+tIv8BCWfhQZY5YDy50/5wCjrNy/UkrZ5bUlezl2uoiZk+Lx98DbF1RHr3hVSqlL2JeVx6xVqdyW0IH+0d51XaeGvFJKXYQxhqc/20FwI38eH+vZFz5VRUNeKaUu4ottR1i9P4fHru1KWFPPHzJ5IQ15pZSqxpmiUv68YBc924dy5+AYu8upE+8YA6SUUjZ4bek+jp4u5J+JA7zqZGtl2pJXSqkq7M/OZ9aqA0yMjyI+xrtOtlamIa+UUhcwxvDs/B00CfTnieu62V2OSzTklVLqAv/bfpSVe4/z6DVdae2FJ1sr05BXSqlKCopL+fOCnXSPCCVxsPdc2VodDXmllKrkH0v3cfhUIc9N6EmAv/dHpPe/A6WUssiB7Hz+tfIANw2IJCG2ld3lWMI3Qj45GWJjwc/PsUxOtrsipZSXMcbwzPwdBAX48+R13e0uxzLeP04+ORmmTIGCAsd6erpjHSAx0b66lFJe5asdjpOtT4/r4RWTgdSU97fkp037IeArFBQ4tiulVA2cLS7juQW76NauGT8b6p1XtlbH+1vyGRm1266UUhf426I9HMo9y3+mDPGJk62Vef+7ia5miFN125VSqpKN6SeY9W0qiYOjGdwxzO5yLGfFRN5BIrJeRLaKyA4R+aNzeysRWSwie51L91wXPH06BAefvy042LFdKaUuorCkjMc++o72zZvw5E9852RrZVa05IuAq40xfYF+wFgRGQJMBZYYY7oAS5zr1ktMhKQkiIkBEccyKUlPuiqlLmnGV3s4cPwMf72lj9fM2VpbLr8rY4wB8p2rgc5/BpgAjHRun41jWsAnXD1elRITNdSVUrWSkvZDN83wzq3tLsdtLOmTFxF/EdkCZAGLjTHrgLbGmCMAzmWbal47RURSRCQlOzvbinKUUuqiCkvKeOxj3+6mqWBJyBtjyowx/YAoYJCI9KrFa5OMMQnGmITw8HArylFKqYua8dUeUo+f4SUf7qapYOnoGmNMLo5umbHAMRGJAHAus6w8llJK1UXlbpphPtxNU8GK0TXhItLC+XMTYDSwG5gPTHY+bTLwmavHUkopV5wtbjjdNBWs+J4SAcwWEX8cHxofGmMWiMga4EMRuQfIACZacCyllKqzGYsc3TTv3zvY57tpKlgxuuY7oH8V23OAUa7uXymlrJCSdoK3v01l0pCG0U1TwfuveFVKqUuo3E0z1YfuMFkTDeP7ilKqQWuI3TQVtCWvlPJpGxpoN00FDXmllM86W1zG4x9/R2SLJj41EUhtNKzvLUqpBuWlr37opglpYN00FbQlr5TySUt2HePtb1P52dCYBtlNU0FDXinlcw7lnuXRj7bSPSKU3zeQi56qoyGvlPIpJWXlPPz+JkpKy3k9cQBBgf52l2SrhtlJpZTyWTO+2sOmjFz+fkd/4lqH2F2O7bQlr5TyGUt2HePNFQdIHBzN+L7t7S7HI2jIK6V8QkU/fI+IUP4wrofd5XgMDXmllNer6IcvLTP8U/vhz6Mhr5TybsnJzLjpt2zKyOX5pW8S99V/7a7Io2jIK6W8V3IyS/4ykze7j2HSpi/46ap5MGUKJCfbXZnH0JBXSnmtQ9Nn8OjoB+lxbD9PLX3LsbGgAKZNq/lOkpMhNhb8/BxLH/uA0CGUSimvVFJWzsPxiZT6BfDPz14kqKzkhwczMmq2k+RkR8u/oMCxnp7uWAdITLS2YJtYMf1fBxFZJiK7RGSHiPzKub2ViCwWkb3OZUvXy1VKKYcZX+1hU2R3nl/4GnEnD5//YHR0zXYybdoPAV+htt8EPJwV3TWlwKPGmO7AEOCXItIDmAosMcZ0AZY415VSymUV4+EntSripxkbz38wOBimT6/Zjqpr8df0m4AXcDnkjTFHjDGbnD/nAbuASGACMNv5tNnADa4eSymlMk8WnBsP/9RvJkBSEsTEgIhjmZRU866W6lr8Nf0m4AUsPfEqIrE45ntdB7Q1xhwBxwcB0Kaa10wRkRQRScnOzrayHKWUj8krLOGed1MoK680Hj4xEdLSoLzcsaxNX/r06Y6Wf2W1+SbgBSwLeRFpCnwC/NoYc7qmrzPGJBljEowxCeHh4VaVo5TyMaVl5Tz0/mb2Z+czc1K8NfelSUx07ZuAF7Ak5EUkEEfAJxtjPnVuPiYiEc7HI4AsK45VlZz8Im5+YzXL97jtEEopGxlj+OPnO/nm+2yeu6EXw628P7wr3wS8gBWjawSYBewyxrxc6aH5wGTnz5OBz1w9VnWOnCokJ7+In7+zgZ+9vZ49R/PcdSillA3eXZ3Gv9emc/8VHbljkO/0l9cHMca4tgOREcBKYBtQ7tz8exz98h8C0UAGMNEYc+Ji+0pISDApKSl1qqO4tJx/r03n1a+/J7+olNsHRfOb0ZcR3qxxnfanlPIMS3Yd4773UhjdvS0zJ8Xj5yd2l+RxRGSjMSahysdcDXkruRLyFU6eKebVJXuZszadoEB/HryqE3cPj9MbFinlhXYePs0tM1fTKbwp/7l/CMGN9PrNqlws5H3utgYtQxrx7PiefPWbKxjSMYy/LtzDqL99w+dbD+NJH2hKqYs7drqQe2ZvoHmTQN6anKABX0c+F/IVOoU35a3JCbx/72BCmwTy8NzN3PTGajZlnLS7NKXUJRQUl3Lv7BROny1h1uSBtA0Nsrskr+WzIV9hWOfWLHh4BH+9uQ+ZJ89y0+ureXjuZjJPFlz6xUqpeldebvj1B1vYcfgUr93Znx7tQ+0uyav5fMgD+PsJtw7swPLfjeThqzuzeOdRrv7bN/x14W7yi0rtLk8pVcmLC3ezaOcxnh7Xg6u7tbW7HK/XIEK+QkjjAB69pitLHx3J9b0jeH35fka+tIy56zMoK9f+eqXsNnd9Bm+uOMDkoTH8fHic3eX4hAYV8hXat2jC/93Wj//+cjixYSE8+ek2rv/7SlbtPW53aUo1WKv2HucP/93OyK7hOkerhRpkyFfo16EFHz0wlNcTB3CmuJRJs9Zxz7sb2JeVb3dpSjUoG9NPcv+/U+jcpimv3dGfAP8GHU2WavC/SRHhJ70jWPybK3nyum6sTz3B2FdW8Oz8HZw8U2x3eUr5vC0Hc/n52+tpExrE7LsH0Swo0O6SfEqDD/kKQYH+3H9lJ5Y9NpLbB3XgvTVpXPnSMt5aeYCi0jK7y1PKJ20/dIqfzVpHy5BGvH/fYB0q6QYa8hdo3bQxf76hNwt/fQX9o1vy5y92MeblFXy57YheTKWUhXYePs2kWetoFhTI+/cNJqJ5E7tL8kka8tW4rG0zZt89iPfuHkRwI38eTN7ExJlr2KwXUynlsj1H85g0ax1NAv2Ze98QoloGX/pFqk405C/hisvC+eKRy3nx5t6knyjgxtdX89D7mzh4Qi+mUqou9mXlkfjWWgL8hPfvG0J0mAa8O2nI14C/n3DbwGiW/24kj4zqwte7jjHqb9/w/Je7OHW25NI7UEoBcCA7nzv+tQ5wBLwlE3+oi9KQr4WQxgH8dsxlLP/dVYzv156klQcY+dIyZq9Oo6Ss/NI7UKoBS885w53/Wkd5ueH9+wbTuU1Tu0tqEDTk66Bd8yBmTOzL5w+NoHtEKM/M38G1/7eChduP6slZVXvJyRAbC35+jmVyst0VWe7giQLu/Nc6CkvLmHPvYC5r28zukhoMDXkX9IpsTvK9g5k1OQE/P+GBORu5ZeYaUtIuOjfKjzWAP3JVjeRkmDIF0tPBGMdyyhSf+n/gUO5Z7vjXWvIKS5hzz2C6R+gNx+qTJZOGiMjbwDggyxjTy7mtFfAfIBZIA241xlx0aIoVk4bYpbSsnI83ZvLy4u/Jyivi2p5teXxsNzqFX+IracUfeUGlE7nBwT43mbCqRmysI9gvFBPjmG/Uyx3KPcud/1rLifxi5tw7mL4dWthdkk9y+8xQInIFkA+8Vynk/wqcMMa8ICJTgZbGmCcuth9vDvkKBcWlzFqZypsrDnC2pIzbB3bgV6O70KZZNRd5+PgfuboEPz9HC/5CIo6Jpb3YjsOnuOudDZwtLuPduwcRH9PS7pJ8Vr1M/yciscCCSiG/BxhpjDkiIhHAcmNM14vtwxdCvsLx/CJeW7KX5HUZNArw477LOzLlio6ENL5gdhsf/iNXNeCjH/LffJ/Ng3M2EtokkHfuGki3dtpF4052Tf/X1hhzBMC5bFNNcVNEJEVEUrKzs91YTv1q3bQxf5zQi8W/vZKrurbh1SV7ufKl5fx7bfr5I3Giq5l5vrrtyrdMn+7onqssONix3Uv9Z0MGd7+7gQ6tgpn34HANeJvZfuLVGJNkjEkwxiSEh4fbXY7l4lqH8M/EAXz64DA6tg7hD//dzpiXv+GzLYcoLzc++UeuaiEx0XH+JSbG8e0tJsZrz8cYY3h50R6e+GQbwzqF8dEDQ2nXXO9FYzftrqlHxhiW7s7ipa/2sPtoHt3aNeOxa7ty9cavkaemQUaGowU/fbpX/pGrhqu4tJypn37Hp5sOMTE+ir/c1JtAvV1wvblYd407pz+fD0wGXnAuP3PjsbyCiDCqe1uu6tqGz787zMuLv+ee2SnEx8Tx2NKNDOkYZneJStXa6cISHpyziVX7jvOb0ZfxyKjOiIjdZSknSz5qRWQusAboKiKZInIPjnAfIyJ7gTHOdc9Uz+PU/fyECf0i+fq3VzL9xl5knizg9qS1/Ozt9WzLPOXWYytlpSOnznLrzDWsPZDDjIl9+dXoLhrwHsay7hor2NJd4wHj1AtLynhvTRqvL99PbkEJP+ndjt+O6Vrzy76Tk2Gadveo+rXryGnuemcD+UWlvDFpAJd38b1zat6iXoZQWsGWkPegIWynC0t4a2Uqs1Y6xtjf2D+Kh6/uTOzFbuLkAR9SquFZ8X02DyZvomnjAN65a6BexWozDfmL8cBx6jn5Rby+fD9z1qZTWm64sX8kD11VTdh70IeU8n2lZeX8fek+Xlu6l65tm/HOXQN1sg8PoCF/MR4cklmnC5n5zQGS1znC/oZ+kT9u2Xvgh5TyTUdOneVXH2xhfeoJbh4QxZ8m9PzxxX3KFnZdDOUdPHicepvQIJ7+aQ9WPn4VPx8Wy4LvDjPq5W949MOtpB0/43iSXkyl6sHS3cf4yasr2X7oFC/f2pe/3drXuoDXG/S5lYa8F1yM0iY0iD+M68HKJ6oI+6ef99gPKVVP3BiSxaXl/HnBTu5+N4V2zZvw+cMjuGlAlGX7bwh34bSbdtd4oay8QpK+OcCcdekUl5ZzQ/NiHnz/BTpv36CjaxoaN554z8gp4OG5m9iaeYqfDY3h9z/pTlCgv4sFX8CDu0u9ifbJ+6jKYV9YUs6YHm154MpOere/hsRNIbngu8M8+ck2ROCvt/RhbK+IOu/rovSckiW0T97dbOpTbNMsiKfG9eDbJ67mkVFd2JB2gpvfWM2tM9ewbHeWzlLVEGRk1G77JRSWlPHkp9t46P3NdG7blC8eudx9AQ96TqkeaMi7ygP6FMOaNua3Yy7j2yeu5g/jepB5soC73t3Ada+uZN7mTPfPP+vtJ868uX4LQ3Jb5ikm/ONb5q7P4IErO/Hh/UPp0Cr40i90hQcPfPAZxhiP+RcfH2+8TkyMMY54P/9fTIxtJRWXlpmPUw6aMS8vNzFPLDDDnl9i3l51wJwpKrH+YHPmGBMcfP57Dw52bPcGWr/JPVNsnpq3zcROXWDin1tslu0+5saCqzBnjuPvRcSx9JbfvQcBUkw1uap98q7y4D7F8nLDsj1ZzPxmPxvSTtIyOJD/NySGxCExtA216Baw3n7izNvrhzrf1sIYwyebDvH8l7s4WVDMz4bG8ttrLiM0KLBejq+soyde3clLQiIl7QQzvznAkt3H8BdhbK92/HxYLPExLV27oZQHf8jViLfXX0e7j57mD//dzoa0k/SPbsFzE3rRK7J57Xekt9XwCBry7uRl/5On55zh32vS+TDlIKcLS+kREcrPh8Uyvl/7ug2P85IPuWp5e/21lF9UyiuLv+ed1WmEBgUw9bpuTIzvgJ9fHT/oG9jvz1Pp6Bp38oKLqSqLCQvhqXE9WPv7Ufzlxt6UlRse/+Q7hjy/hOf/t4vMkwWX3kll3n7izIr6veDErTHGcRHd35bz1qpUbk2IYumjI7ltYHTdAx4sH92jrKct+QbOGMPaAyd4b00ai3YewxjD6O5tmTwslqEdw2oWAN7eJ+tK/V7wTW5fVj7Pzt/Bqn3H6dk+lOdu6MWAaIuupdCWvEewtbtGRMYCrwL+wFvGmGonD9GQt9fh3LPMWZvOBxsOcuJMMdGtgrl5QBQ3x0cS1dLNQ+m8lQeH3O6jp/nnsv188d1hQhoH8LtrujJpSAz+rrTcL+QFH3INgW0hLyL+wPc4ZobKBDYAdxhjdlb1fA15z1BYUsbC7Uf5aONBVu/PwRgY1imMiQlRjO0ZQZNGFl/a7s088MTt1oO5/GPZPhbvPEZII38mDY3hvss70rppY/cc0Nu/yfkAO0N+KPCsMeZa5/qTAMaY56t6voa858k8WcCnmw7x8cZMMk4U0LRxAOP6RHBLfJTrI3N8gQe15NennuAfy/ax4vtsQoMCuGt4HHcNj6VFcKN6rUPVP7sm8gaIBA5WWs8EBrv5mMpCUS2DeWRUFx6+ujPrU0/w8cZM5m89zAcbDhLXOoRb4qO4oX8kkS0a6MQR06dX3V1RTyeejTGs2nec15buY33qCcJCGvHE2G5MGhJNs5qOd9eWuE9zd0t+InCtMeZe5/r/AwYZYx6u9JwpwBSA6Ojo+PSqWkXKo5wpKuXLbUf4eGMm61JPANA3qjnX9mrHdb0iiLvYdIW+yIaQLCs3LN2dxT+W7WPrwVzahQZx/5UduX1gdO2607RP3Sdod41ym4ycAr7YdoSFO46y9WAuAF3bNmNsr3aM7dWObu2aub9LpwG1RL8/lscnmzL5bPNhjp4upEOrJvziys7cHB9J44AGeJ2DAuwN+QAcJ15HAYdwnHi90xizo6rna8h7t8O5Z1m4/SgLdxxlQ9oJjIHYsOBzLfy+Uc2tD/wG0BLNziti/tbDzNucyfZDp/H3E0ZeFs5NA6K4tmdbAvxduNzFA08cq9qzewjlT4BXcAyhfNsYU21npYa878jOK2LxzmP8b/sR1uzPobTcENE8iBGdWzO8c2uGdQqjjRX3z/HRlmhhSRmLdx7j002ZrNh7nLJyQ+/I5tw0IJKf9m1v3UgZH/39NTR6WwNlq1MFJXy96xiLdx5jzYEcTp0tAaBzm6YM7xTGsM6tGRIXRvPgWt4YC3yqJXq2uIwNaSf44rsjfLntCHlFpUQ0D+KG/pHc1D+SLm2bWX/QBvBNqCHQkFceo6zcsPPwaVbvP863+3PYkHqCsyVl+An0imzOsE6OVn58TMuaTRTtxS3RwpIyNmfksuZADmv2H2fLwVxKygwhjfy5rncENw2IZEhcDa86dkUDOqfhqzTklccqLi1n89sfsnr+ClY3j2VzZDdK/fwRgbjWIfRq35ye7UPp6Vy2DLlgzLcXtUSLS8v5LjOXNftzWL0/h00ZJykqLcdPoHdkc4Z2as3QTmEMim2lF5ypWtGQV57rgpA+ExjE+s4D2Hrvb9jROpYdh05x+FThuae3bx5Ez8gfgr9H+1DaLfgU/6c8pyVqjCE7r4gDx89wIPsMB7Lz2XMsj5S0k5wtKQOgR0QoQzuFMaxTGAPjWtX+Hu5KVaIhrzxXDbpbTpwpZufh0+w4fIrtzmXq8TPnuuID/IR2zYNo36IJkS2a0L6F4+eK9YjmQTW/MKiGSsrKOVNUyuHcQg4czz8X5geOnyE1+wx5RaXnnts4wI+O4U0ZFNuSoZ3CGBwX9uNvJEq5QENeea46njg9U1TK7qOn2XUkj8O5Z53/CjmUe5ajpwspKz9/n82CAmgV0oigAH+CAv1oHOBP40A/ggL9CQr0p3GAH0GBfgQF+BPg78fZ4lLyikrJLywlv6iUvPOWJRSW/Li2yBZN6BgeQsfWIcS1DqFjeFM6hofQvnkT9/erqwbNztsaKHVx0dFVt+QvMRF1SOMA4mNaER/T6kePlZUbsvIKOZxbeO4D4FDuWU6dLaGwpIyi0nIKS8rIKywlO6+IYud6oXNZUlZOcKMAmjYOoFmQ41/rpo2IbR1yblvTxo5/bUIb07F1U+Jah2g/uvJIGvLKXm6494u/nxDRvAkRzZsQH2PRfdOV8lI6M5Syl5fNrKWUt9GWvLJfYqKGulJuoi15pZTyYRrySinlwzTklVLKh2nIK6WUD9OQV0opH6Yhr5RSPkxDXimlfJhLIS8iE0Vkh4iUi0jCBY89KSL7RGSPiFzrWplKKaXqwtWLobYDNwFvVt4oIj2A24GeQHvgaxG5zBhT5uLxlFJK1YJLLXljzC5jzJ4qHpoAfGCMKTLGpAL7gEGuHEsppVTtuatPPhI4WGk907ntR0RkioikiEhKdna2m8pRSqmG6ZLdNSLyNdCuioemGWM+q+5lVWyr8sb1xpgkIAkc95O/VD1KKaVq7pIhb4wZXYf9ZgIdKq1HAYfrsB+llFIucFd3zXzgdhFpLCJxQBdgvZuOpZRSqhquDqG8UUQygaHAFyLyFYAxZgfwIbATWAj8UkfWKKVU/XNpCKUxZh4wr5rHpgN1n95HKaWUy/SKV6WU8mEa8kop5cM05JVSyodpyCtITobYWPDzcyyTk+2uSCllEZ3Iu6FLToYpU6CgwLGenu5YB51cWykfoC35hm7atB8CvkJBgWO7Usrracg3dBkZtduulPIqGvINXXR07bYrpbyKhnxDN306BAefvy042LFdKeX1NOQbusRESEqCmBgQcSyTkvSkq1I+QkfXKEega6gr5ZO0Ja+UUj5MQ14ppXyYhrxSSvkwDXmllPJhGvJKKeXDxBjPmTtbRLKBdBd20Ro4blE5dvKV9wH6XjyRr7wP0PdSIcYYE17VAx4V8q4SkRRjTILddbjKV94H6HvxRL7yPkDfS01od41SSvkwDXmllPJhvhbySXYXYBFfeR+g78UT+cr7AH0vl+RTffJKKaXO52steaWUUpVoyCullA/zqZAXkedE5DsR2SIii0Skvd011ZWIvCQiu53vZ56ItLC7proSkYkiskNEykXE64a7ichYEdkjIvtEZKrd9dSViLwtIlkist3uWlwlIh1EZJmI7HL+v/Uru2uqCxEJEpH1IrLV+T7+aPkxfKlPXkRCjTGnnT8/AvQwxjxgc1l1IiLXAEuNMaUi8iKAMeYJm8uqExHpDpQDbwK/M8ak2FxSjYmIP/A9MAbIBDYAdxhjdtpaWB2IyBVAPvCeMaaX3fW4QkQigAhjzCYRaQZsBG7wtv8uIiJAiDEmX0QCgVXAr4wxa606hk+15CsC3ikE8NpPMGPMImNMqXN1LRBlZz2uMMbsMsbssbuOOhoE7DPGHDDGFAMfABNsrqlOjDErgBN212EFY8wRY8wm5895wC4g0t6qas845DtXA53/LM0tnwp5ABGZLiIHgUTgabvrscjdwP/sLqKBigQOVlrPxAvDxJeJSCzQH1hncyl1IiL+IrIFyAIWG2MsfR9eF/Ii8rWIbK/i3wQAY8w0Y0wHIBl4yN5qL+5S78X5nGlAKY7347Fq8l68lFSxzWu/IfoaEWkKfAL8+oJv8l7DGFNmjOmH49v6IBGxtCvN66b/M8aMruFT3we+AJ5xYzkuudR7EZHJwDhglPHwkye1+O/ibTKBDpXWo4DDNtWiKnH2YX8CJBtjPrW7HlcZY3JFZDkwFrDs5LjXteQvRkS6VFodD+y2qxZXichY4AlgvDGmwO56GrANQBcRiRORRsDtwHyba2rwnCcsZwG7jDEv211PXYlIeMXIORFpAozG4tzytdE1nwBdcYzkSAceMMYcsrequhGRfUBjIMe5aa0XjxS6EXgNCAdygS3GmGttLaoWROQnwCuAP/C2MWa6vRXVjYjMBUbiuKXtMeAZY8wsW4uqIxEZAawEtuH4ewf4vTHmS/uqqj0R6QPMxvH/lh/woTHmT5Yew5dCXiml1Pl8qrtGKaXU+TTklVLKh2nIK6WUD9OQV0opH6Yhr5RSPkxDXimlfJiGvFJK+bD/D3YEE9Xrncp1AAAAAElFTkSuQmCC\n", 333 | "text/plain": [ 334 | "
" 335 | ] 336 | }, 337 | "metadata": { 338 | "needs_background": "light" 339 | }, 340 | "output_type": "display_data" 341 | } 342 | ], 343 | "source": [ 344 | "xgrid = numpy.linspace(x.min(), x.max(), 30)\n", 345 | "Xgrid_poly_feat = polynomial_features(xgrid, degree)\n", 346 | "Xgrid_scaled = min_max_scaler.transform(Xgrid_poly_feat)\n", 347 | "Xgrid_scaled[:,0] = 1 \n", 348 | "pyplot.scatter(x, y, c='r', label='true')\n", 349 | "pyplot.plot(xgrid, Xgrid_scaled@params, label='predicted')\n", 350 | "pyplot.legend();" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "## Observe underfitting & overfitting\n", 358 | "\n", 359 | "In the model above, we just randomly picked a polynomial degree of $3$ for the fitted curve. Is it good enough to model our dataset? Should we try higher-order polynomials?\n", 360 | "\n", 361 | "We can repeat our study with different polynomial degrees varying from $1$ to $15$, and see what happens.\n", 362 | "To faciliate this task, we provide you with a script to train the model and plot these fitted curves interactively using `ipywidget`.\n", 363 | "\n", 364 | "Run the cell below and drag the slider to see how the curve changes." 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": 9, 370 | "metadata": {}, 371 | "outputs": [ 372 | { 373 | "data": { 374 | "application/vnd.jupyter.widget-view+json": { 375 | "model_id": "71f0d094cc19484483274e0b52147109", 376 | "version_major": 2, 377 | "version_minor": 0 378 | }, 379 | "text/plain": [ 380 | "interactive(children=(IntSlider(value=8, description='degree', max=15, min=1), Output()), _dom_classes=('widge…" 381 | ] 382 | }, 383 | "metadata": {}, 384 | "output_type": "display_data" 385 | } 386 | ], 387 | "source": [ 388 | "import sys\n", 389 | "sys.path.append('../scripts/')\n", 390 | "from plot_helpers import interact_polyreg\n", 391 | "\n", 392 | "max_degree = 15\n", 393 | "interact_polyreg(max_degree, x, y)" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "metadata": {}, 399 | "source": [ 400 | "### Underfitting\n", 401 | "\n", 402 | "When `degree` is $1$, the straight line clearly fails to capture the underlying relationship between $x$ and $y$. Specifically, a line is too simple to explain how far the data are spread out, namely, the **variance** in the data. We say that the linear model **underfits** the data in this case.\n", 403 | "\n", 404 | "Underfitting happens when the model is too naive or the weights need to be trained with more iterations. It is often easy to detect, since a large training error is a good indicator.\n", 405 | "\n", 406 | "##### Challenge question\n", 407 | "\n", 408 | "> Would having more training data help resolve underfitting?\n", 409 | "\n", 410 | "### Overfitting\n", 411 | "\n", 412 | "As we increase the polynomial degree, the training error (MAE from the figure title) keeps decreasing. \n", 413 | "But does it mean that the 15th-order polynomial gives the best fit?\n", 414 | "Probably not.\n", 415 | "\n", 416 | "Drag the slider to the right, and you will find that the curve passes exactly through many points and looks very odd.\n", 417 | "If we are given new data, this model may not predict well because it fits too closely to the old data.\n", 418 | "As real-world data tend to be noisy (due to missing and erroneous values), our synthetic data also have noise added.\n", 419 | "Models with high polynomial degrees are so flexible that they fit the noise rather than the true relationship! \n", 420 | "In this case, these models are **overfitting**, or have a **high variance**.\n", 421 | "Overfitting usually happens when the model is overly complicated.\n", 422 | "The high-order polynomials in our example have too many degrees of freedom (weights) for our data.\n", 423 | "\n", 424 | "##### Challenge question\n", 425 | "\n", 426 | "> Would having more training data help resolve overfitting?\n", 427 | "\n", 428 | "Compared to underfitting, overfitting is in general harder to detect because the training error tends to be small. We will discuss how to identify overfitting later in this module.\n", 429 | "Now let's focus on how to prevent overfitting using regularization." 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": {}, 435 | "source": [ 436 | "## Regularization\n", 437 | "\n", 438 | "Regularization is used to reduce or avoid overfitting.\n", 439 | "The idea is to introduce a term in the loss function that penalizes complicated models. In our polynomial regression model:\n", 440 | "\n", 441 | "$$\n", 442 | "\\hat{y} = w_0 + w_1 x + w_2 x^2 + \\cdots + w_d x^d,\n", 443 | "$$\n", 444 | "\n", 445 | "every polynomial term contributes to the overall complexity of the model.\n", 446 | "One idea is to add constraints to the magnitudes of these weights.\n", 447 | "\n", 448 | "A common approach is to add a regularization term $\\lambda\\sum_{j=1}^d w_j^2$ to the mean-squared error loss:\n", 449 | "\n", 450 | "$$\n", 451 | "L(\\mathbf{w})=\\frac{1}{N} {\\lVert \\mathbf{y} - X\\mathbf{w} \\rVert}^2 + \\lambda \\sum_{j=1}^d w_j^2\n", 452 | "$$\n", 453 | "\n", 454 | "This new loss function favors smaller weights, because larger weights will increase the term on the right.\n", 455 | "As weights are smaller, the model is less likely to overfit by large amplitude higher-order polynomial terms.\n", 456 | "\n", 457 | "Notice that we don't penalize the intercept $w_0$ in the regularization term. This can be justified mathematically, but one way to think about it is that a constant term won't affect the overall model complexity.\n", 458 | "\n", 459 | "Above, $\\lambda$ denotes the regularization parameter, or the strength of penalty. It controls the tradeoff between fitting the data well (minimizing the first term) and keeping the model simple to avoid overfitting (minimizing the second term). When $\\lambda\\rightarrow 0$, the loss function falls back to standard mean-square error loss. And when $\\lambda$ is large, all penalized weights will be close to 0 after training and our model would approach the constant value $w_0$.\n", 460 | "\n", 461 | "##### Note\n", 462 | "\n", 463 | "> This regularization is also called $L^2$-penalty, or Tikhonov regularization, while the name _ridge regression_ is also used for this model-building method." 464 | ] 465 | }, 466 | { 467 | "cell_type": "markdown", 468 | "metadata": {}, 469 | "source": [ 470 | "Let's code the regularized mean-squared error loss and set $\\lambda = 1$." 471 | ] 472 | }, 473 | { 474 | "cell_type": "code", 475 | "execution_count": 10, 476 | "metadata": {}, 477 | "outputs": [], 478 | "source": [ 479 | "def regularized_loss(params, model, X, y, _lambda=1.0):\n", 480 | " '''\n", 481 | " The mean squared error loss function with an L2 penalty.\n", 482 | " Arguments:\n", 483 | " params: 1D array of weights for the linear model\n", 484 | " model : function for the linear regression model\n", 485 | " X : 2D array of input values\n", 486 | " y : 1D array of predicted values\n", 487 | " _lambda: regularization parameter, default 1.0\n", 488 | " Returns:\n", 489 | " float, regularized mean squared error\n", 490 | " '''\n", 491 | " y_pred = model(params, X)\n", 492 | " return numpy.mean( numpy.sum((y-y_pred)**2) ) + _lambda * numpy.sum( params[1:]**2 )\n", 493 | "\n", 494 | "gradient = grad(regularized_loss) " 495 | ] 496 | }, 497 | { 498 | "cell_type": "code", 499 | "execution_count": 11, 500 | "metadata": {}, 501 | "outputs": [], 502 | "source": [ 503 | "no_regularization_params = params.copy()" 504 | ] 505 | }, 506 | { 507 | "cell_type": "markdown", 508 | "metadata": {}, 509 | "source": [ 510 | "And train the 3rd-degree polynomial model using gradient descent." 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": 12, 516 | "metadata": {}, 517 | "outputs": [ 518 | { 519 | "name": "stdout", 520 | "output_type": "stream", 521 | "text": [ 522 | "iteration 0, loss = 5434.768, mae = 11.05718775676392\n", 523 | "iteration 100, loss = 1785.985, mae = 6.983886996350374\n", 524 | "iteration 200, loss = 1764.260, mae = 6.966518785947395\n", 525 | "iteration 300, loss = 1763.570, mae = 6.965728405272657\n" 526 | ] 527 | } 528 | ], 529 | "source": [ 530 | "max_iter = 3000\n", 531 | "alpha = 0.01\n", 532 | "params = numpy.zeros(X_scaled.shape[1])\n", 533 | "descent = numpy.ones(X_scaled.shape[1])\n", 534 | "i = 0\n", 535 | "\n", 536 | "from sklearn.metrics import mean_absolute_error\n", 537 | "\n", 538 | "while numpy.linalg.norm(descent) > 0.01 and i < max_iter:\n", 539 | " descent = gradient(params, linear_regression, X_scaled, y)\n", 540 | " params = params - descent * alpha\n", 541 | " loss = mse_loss(params, linear_regression, X_scaled, y)\n", 542 | " mae = mean_absolute_error(y, X_scaled@params)\n", 543 | " if i%100 == 0:\n", 544 | " print(f\"iteration {i:4}, {loss = :.3f}, {mae = }\")\n", 545 | " i += 1" 546 | ] 547 | }, 548 | { 549 | "cell_type": "markdown", 550 | "metadata": {}, 551 | "source": [ 552 | "Let's compare the optimal weights before and after regularization. We can reuse the `xgrid` to plot both curves." 553 | ] 554 | }, 555 | { 556 | "cell_type": "code", 557 | "execution_count": 13, 558 | "metadata": {}, 559 | "outputs": [ 560 | { 561 | "name": "stdout", 562 | "output_type": "stream", 563 | "text": [ 564 | "weights without regularization\n", 565 | "[-22.51572398 6.75930601 41.30788709 30.0105898 ]\n", 566 | "weights with regularization\n", 567 | "[-11.13750882 12.48522096 28.26626633 11.09211867]\n" 568 | ] 569 | }, 570 | { 571 | "data": { 572 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAA6bUlEQVR4nO3deVhV1frA8e9iEAREFFBxAJxREec5p7S0MjNNLanMW5nZdKtbWnbLunnr+rPbZDczs6zIMss0K3PKecQpZ3MARBARRUGmczjr98cGRGXm4Bl8P89zns3e5+y93yP4nnXWXvtdSmuNEEII5+Ri6wCEEEJUHUnyQgjhxCTJCyGEE5MkL4QQTkySvBBCODE3WwdQWEBAgA4NDbV1GEII4VB27NhxVmsdWNRzdpXkQ0NDiY6OtnUYQgjhUJRSscU9J901QgjhxCTJCyGEE5MkL4QQTsyu+uSLYjKZiI+PJysry9ahCBvx9PSkYcOGuLu72zoUIRyO3Sf5+Ph4atSoQWhoKEopW4cjrjOtNSkpKcTHx9O4cWNbhyOEw7H77pqsrCz8/f0lwd+glFL4+/vLNznhvKKiIDQUXFyMZVSUVQ9v9y15QBL8DU5+/8JpRUXB+PGQkWGsx8Ya6wCRkVY5hd235IUQwmlNmXI5wefLyDC2W4kkeSt56623iLLy16yKeOihh1i4cGG59pk1axZffvlluc+1Zs0aNm3aVOnjCHHDiosr3/YKcIjuGkewfPlyFixYUO79zGYzbm62+zWYzWYmTJhQoX3XrFmDj48PPXv2BKjwcYS4YQUHG100RW23EmnJl2L69Ol88MEHADz77LPcfPPNAKxatYr7778fgIsXL5KTk0NgYCCxsbEMGDCAiIgIBgwYQFwRn8hTp05l/Pjx3HrrrTz44IMkJyczYsQIunTpQpcuXdi4cSMAycnJ3HLLLXTs2JHHHnuMkJAQzp49S0xMDOHh4QXHmzFjBlOnTr3mPG+88QZdunQhPDyc8ePHkz8LWL9+/Xj55Zfp27cv77//PlOnTmXGjBkkJCTQvn37goerqyuxsbH8/PPPdOvWjQ4dOjBw4ECSkpKIiYlh1qxZvPvuu7Rv357169cXHAdg9+7ddO/enYiICO6++27Onz9fcO5JkybRtWtXWrRowfr16630mxLCAU2bBl5eV27z8jK2W4lDteRf/3k/BxIuWvWYrev78tqdbYp9vk+fPrzzzjs8/fTTREdHk52djclkYsOGDfTu3RuAlStXMmDAAACefPJJHnzwQcaOHcvcuXN5+umn+emnn6457o4dO9iwYQPVq1dnzJgxPPvss9x0003ExcUxaNAgDh48yOuvv87NN9/MSy+9xLJly5g9e3a53tuTTz7Jq6++CsADDzzA0qVLufPOOwFITU1l7dq1AAUfEPXr12f37t0AfPTRR6xdu5aQkBB8fX3ZsmULSinmzJnD9OnTeeedd5gwYQI+Pj784x//AIwPvnwPPvggH374IX379uXVV1/l9ddf57333gOMbw/btm3j119/5fXXX2flypXlel9COI38i6tTphhdNMHBRoK30kVXcLAkbwudOnVix44dpKWl4eHhQceOHYmOjmb9+vUFLfxly5Yxbtw4ADZv3syPP/4IGIn1xRdfLPK4Q4cOpXr16oDxIXHgwIGC5y5evEhaWhobNmxg0aJFAAwePJhatWqVK/Y//viD6dOnk5GRwblz52jTpk1Bkh89enSx+23cuJE5c+YUtLLj4+MZPXo0iYmJ5OTklDpe/cKFC6SmptK3b18Axo4dy8iRIwueHz58OGD828bExJTrPQnhdCIjrZrUr+ZQSb6kFndVcXd3JzQ0lM8//5yePXsSERHBH3/8wbFjx2jVqhUA27Zt4+OPPy5y/+KG/3l7exf8bLFY2Lx5c0HSz1fcJOtubm5YLJaC9aLGkGdlZTFx4kSio6Np1KgRU6dOveJ1hc9fWGJiIg8//DBLlizBx8cHgKeeeornnnuOoUOHsmbNmiK7hsrDw8MDAFdXV8xmc6WOJYQomfTJl0GfPn2YMWMGffr0oXfv3syaNYv27dujlGL//v2EhYXh6uoKQM+ePfn2228BiIqK4qabbir1+LfeeiszZ84sWM/vMrnpppsKLuYuX768oF+7bt26nDlzhpSUFLKzs1m6dOk1x8xP6AEBAaSnp5dpxI3JZGLUqFH85z//oUWLFgXbL1y4QIMGDQCYN29ewfYaNWqQlpZ2zXFq1qxJrVq1Cr4JfPXVVwWteiHE9SVJvgx69+5NYmIiPXr0oG7dunh6ehb0x//2228MHjy44LUffPABn3/+OREREXz11Ve8//77pR7/gw8+IDo6moiICFq3bs2sWbMAeO2111i+fDkdO3bkt99+IygoiBo1auDu7s6rr75Kt27dGDJkCGFhYdcc08/Pj0cffZS2bdsybNgwunTpUmocmzZtYvv27bz22msFF18TEhKYOnUqI0eOpHfv3gQEBBS8/s4772TRokUFF14LmzdvHi+88AIRERHs3r274NqAEOL6UsV1CdhC586d9dWThhw8eLCgW8Qe3XLLLXz55ZcEBQVZ/djZ2dm4urri5ubG5s2befzxxwta+Tcae/87EMKWlFI7tNadi3rOofrk7dGKFSuq7NhxcXGMGjUKi8VCtWrV+PTTT6vsXEII52SVJK+U8gPmAOGABv4GHAa+A0KBGGCU1vq8Nc53o2jevDm7du2ydRhCCAdmrT7594FlWuswoB1wEJgMrNJaNwdW5a0LIYS4jiqd5JVSvkAf4DMArXWO1joVuAvIH4oxDxhW2XMJIYQoH2u05JsAycDnSqldSqk5SilvoK7WOhEgb1mnqJ2VUuOVUtFKqejk5GQrhCOEECKfNZK8G9AR+Fhr3QG4RDm6ZrTWs7XWnbXWnQMDA60QjhBCiHzWSPLxQLzWemve+kKMpJ+klAoCyFuescK57NLtt99Oamoqqamp/O9//yvYvmbNGoYMGWLDyCA0NJSzZ8+Wa59HHnnkijILZfXFF1+QkJBQ6eMIIayn0klea30aOKmUapm3aQBwAFgCjM3bNhZYXNlz2atff/0VPz+/a5J8Rdj6Nv/c3FzmzJlD69aty73v1Um+oscRQliPtUbXPAVEKaX+BNoD/wbeBm5RSv0F3JK37nDKUmo4v7U8efJkjh07Rvv27XnhhRcASE9P55577iEsLIzIyMgi69FcXfp3x44d9O3bl06dOjFo0CASExMB2L59OxEREfTo0YMXXnihoNzwF198wZNPPllwvCFDhrBmzZprzjNs2DA6depEmzZtrqho6ePjU3AH7ebNm+nXrx/R0dEsWbKk4M7Xli1bFhQmK6qE8cKFC4mOjiYyMpL27duTmZlZcByA+fPn07ZtW8LDw5k0adIV554yZQrt2rWje/fuJCUlVewXJYQoklXGyWutdwNF3W01wBrHL/DbZDi916qHpF5buK34z5+ylBrO9/bbb7Nv376Cu1LXrFnDrl272L9/P/Xr16dXr15s3LixyHo2+aV/TSYTffv2ZfHixQQGBvLdd98xZcoU5s6dy7hx45g9ezY9e/Zk8uTyj0idO3cutWvXJjMzky5dujBixAj8/f25dOkS4eHhvPHGG1e8fujQoQwdOhSAUaNGFdSfKaqE8T333MPMmTOZMWMGnTtf+aeQkJDApEmT2LFjB7Vq1eLWW2/lp59+YtiwYVy6dInu3bszbdo0XnzxRT799FNeeeWVcr83IUTRpHZNKa4uNdyjR4+CUsNXJ/midO3alYYNG+Li4kL79u2LLa2bX/r38OHD7Nu3j1tuuYX27dvz5ptvEh8fT2pqKmlpaQWzMI0ZM6bc7+WDDz4oaDGfPHmSv/76CzCqQY4YMaLY/aZPn0716tV54oknAKOEcbdu3Wjbti2rV69m//79JZ53+/bt9OvXj8DAQNzc3IiMjGTdunUAVKtWreC6hZQeFsL6HKusQQkt7qpSllLDJckvqwsll9bNL/2rtaZNmzZs3rz5iufzK1AWpSylh9esWcPKlSvZvHkzXl5e9OvXr+B1np6eBVU0r7Zq1Sq+//77gqRcWgnjopRUH8nd3b2gHLOUHhbC+qQlXwYllRourLjSu+XRsmVLkpOTC5K8yWRi//791KpVixo1arBlyxaAgnLGYFwT2L17NxaLhZMnT7Jt27ZrjnvhwgVq1aqFl5cXhw4dKjhOSWJjY5k4cSILFiwoqHVfUgnj4t5/t27dWLt2LWfPniU3N5f58+dL6WEhrhPHasnbSO/evZk2bRo9evTA29v7ilLDhfn7+9OrVy/Cw8O57bbbuOOOO8p9rmrVqrFw4UKefvppLly4gNls5u9//ztt2rThs88+49FHH8Xb25t+/fpRs2ZNAHr16kXjxo0LLmx27NjxmuMOHjyYWbNmERERQcuWLenevXupsXzxxRekpKRw9913A8b0gL/++mtBCePQ0NArShg/9NBDTJgwgerVq1/xTSQoKIi33nqL/v37o7Xm9ttv56677ir3v40Qovyk1LADSU9PL5it6e233yYxMbFM9eqdgfwdCGf2/II9NAn05on+zSq0f0mlhqW7xoH88ssvtG/fnvDwcNavXy+jUIRwAodOX+SHnfFkmy2lv7gCpLvGgYwePbrECbiFEI7nw9VH8a7myt96hVbJ8aUlL4QQNnL0TBq/7k1kbM9Q/LyqVck5JMkLIYSNzFx9FE83Vx6+qXGVnUOSvBBC2MCJs5dYsieBB3qE4O/jUfoOFSRJXgghbOCjP47i7urCo72bVOl5JMmXgTWqSwohRL6T5zJYtOsUY7oFE1ij6lrxIEm+TIpL8rm5uTaIRgjh6P635iiuLooJfZtW+bmcL8lHRUFoKLi4GMuoqEofsnAJ4S5dutC/f3/GjBlD27ZtiYmJKSj5CzBjxgymTp0KwLFjxxg8eDCdOnWid+/eHDp0qNKxCCEc26nUTBbuiGd050bU9fWs8vM51zj5qCgYPx4yMoz12FhjHSAyssKHLVxCeM2aNdxxxx3s27ePxo0bl1g1cfz48cyaNYvmzZuzdetWJk6cyOrVqyschxDC8c1acwyACf2qvhUPzpbkp0y5nODzZWQY2yuR5K/WtWvXggk0ipOens6mTZsYOXJkwbbs7GyrxSCEcDynL2Tx3faT3NOpIQ38ql+Xc1olySulYoA0IBcwa607K6VqA98BoUAMMEprXXy9XGuIiyvf9grKLwsMxZf5tVgs+Pn5FUwgIoQQn6w7Rq7WTOxXsRo1FWHNPvn+Wuv2hYrkTAZWaa2bA6vy1qtWcHD5tpdRSSWE69aty5kzZ0hJSSE7O5ulS5cC4OvrS+PGjfn+++8Bo6b6nj17KhWHEMJxnUnL4putcdzdoQGNantdt/NW5YXXu4B5eT/PA4ZV4bkM06aB11X/eF5exvZKKFxCOH/u1nzu7u4F86MOGTKEsLCwgueioqL47LPPaNeuHW3atGHxYqedy1wIUYo5609gyrVUuNJkRVml1LBS6gRwHtDAJ1rr2UqpVK21X6HXnNda1ypi3/HAeIDg4OBOsbGxVzxf7hKzUVFGH3xcnNGCnzbNqv3xwjak1LBwZCnp2dz0nz8YHF6Pd0e3t/rxSyo1bK0Lr7201glKqTrACqVUmccKaq1nA7PBqCdf6UgiIyWpCyHsymcbTpBlzr3urXiwUneN1johb3kGWAR0BZKUUkEAecsz1jiXEEI4ktSMHL7cHMsdbYNoVsfnup+/0kleKeWtlKqR/zNwK7APWAKMzXvZWKDCHdL2NHuVuP7k9y8c2dyNMaRnm3ny5uvfigfrdNfUBRblTWrtBnyjtV6mlNoOLFBKPQzEASNLOEaxPD09SUlJwd/f/5qJs4Xz01qTkpKCp2fV3xkohLVdzDLx+cYTDG5Tj7B6vjaJodJJXmt9HGhXxPYUYEBlj9+wYUPi4+NJTk6u7KGEg/L09KRhw4a2DkOIcpu3MYa0LNu14sEB7nh1d3cv9e5SIYSwN+nZZj7beIKBreoQ3qCmzeJwvgJlQghhBz7fcILUDBNP3dzcpnFIkhdCCCtLSc/mk3XHGdSmLu0a+dk0FknyQghRGUWUN5/5x1EyTbm8ODistL2rnCR5IYSoqPzy5rGxoDXExhL3j1f4etMJRnVuRNPA6z8u/mqS5IUQoqKKKG/+Tud7cDWZ+PtA2/bF55MkL4QQFXVVGfN9dZqwuE0/Ht7+03WZ9aksJMkLIURFXVXG/D/9HqJWxgUeS9hqo4CuJUleCCEqqlB58w0h7VjfuCNPRi/Cd+o/bRzYZXZ/M5QQQtitvIq3limv8Ha/cTS4lML9jw+zq0q4kuSFEKIyIiNZGt6fffN38e7odnh0sK8SHNJdI4QQlZBjtjDj98O0CvLlrnYNbB3ONSTJCyFEJczfFkfcuQwm3xaGi4v9VcqVJC+EEBWUlmXig1V/0bOpP32aB9g6nCJJkhdCiAr6dP0JUi7lMPm2MLud78JqSV4p5aqU2qWUWpq3XlsptUIp9Vfe8ppJvIUQwlGdSctizvrjDIkIIqKhX+UOlnkeLBarxHU1a7bknwEOFlqfDKzSWjcHVuWtCyGEU/hw1VFyzBb+cWvLyh3IlAVfDYefHrdOYFexSpJXSjUE7gDmFNp8FzAv7+d5wDBrnEsIIWztxNlLzN8Wx5huwYQGeFf8QFrD0mchYSe0Hmq9AAuxVkv+PeBFoPD3jbpa60SAvGUdK51LCCFsasbvh/Fwc6n8hCDbZsOeb6DfSxB2h3WCu0qlk7xSaghwRmu9o4L7j1dKRSulomUeVyGEvdt9MpVf9ibyaJ8mBNbwqPiBTqyHZS9Byzugz4vWC/Aq1mjJ9wKGKqVigG+Bm5VSXwNJSqkggLzlmaJ21lrP1lp31lp3DgwMtEI4QghRNbTWvPXrQQJ8qvFI7yYVP1BqHHw/Fvybwt2zjAlHqkilj6y1fklr3VBrHQrcC6zWWt8PLAHG5r1sLLC4sucSQghb+vnPRLaeOMezt7TAx6OCVWFyMuDbSMg1wb3zwdPXukFepSpr17wNLFBKPQzEASOr8FxCCFGl0rPNTPvlAG0b1OTeLsGl71AUreHnZ+D0XhjzHQQ0s26QRbBqktdarwHW5P2cAgyw5vGFEMJWPlz1F0kXs5l1fydcK1q+YPNHsHcB3PwKtBhk3QCLIXe8CiFEKY6eSeOzDScY3bkRHYIreF/nsT9gxT+h1VDo/Q/rBlgCSfJCCFECrTWvLt6PVzVXXhxcwRufzp2AheMgMAyGfQzXsQSCJHkhhCjBL3sT2XQshRcGtcTfpwJDJnMuGRdatQXujQIPH+sHWQKZNEQIIYpxKdvMm0sP0qa+L2O6hZT/AFrD4icg+SBEfg+1KzHssoKkJS+EEMX4cPVRTl/M4o27wit2sXXje7B/EQx4DZoNtHp8ZSFJXgghinAsOZ3PNhxnZKeGdAqpwMXWw8tg5evQZjj0esb6AZaRJHkhhLiK1pqpS/ZT3d2VSbeFlf8ACbuMC61BEXDXzOt6ofVqkuSFEOIqv+07zfq/zvL8rS0JKO/F1vOx8M1o8PKHMQugWiWqVFqBXHgVQohCMnLMvLn0AK2CfInsVs47WzPPQ9RIo0b8g0ugRr2qCbIcJMkLIUQhM1cfJeFCFh/c1wE313J0dpiz4bsH4NxxeOBHqFOBbp4qIEleCCHyHE9O59P1xxnesQGdQ2uXfUetYfGTELMehn8KjftUXZDl5Bx98lFREBpqlOsMDTXWhRCiHLTWvLZkP55urrx0W6vy7bz6zcs1aSJGVU2AFeT4LfmoKBg/HjIyjPXYWGMdIDLSdnEJIRzK7/uNi62vDmldvslAdsyD9TOg44PXtSZNWTl+S37KlMsJPl9GhrFdCCHKIDMnl38tPUhYvRo82KMcd7YeXWnM0dp0ANzxX5sOlSyO47fk4+LKt10IIa7yzvLDnErN5Lvx3ct+sTXxT1gwFuq2hlHzwNW9aoOsIMdvyQcXM8SpuO1CCFHIjthzfLbxBJHdgunWxL9sO12Ih29GgWdNGPM9eNSo2iArwRoTeXsqpbYppfYopfYrpV7P215bKbVCKfVX3rKCRZhLMW0aeHlduc3Ly9guhBAlyDLl8sL3f1K/ZnVeur2MF1uzLkDUKKO6ZOT34BtUtUFWkjVa8tnAzVrrdkB7YLBSqjswGViltW4OrMpbt77ISJg9G0JCjP6wkBBjXS66CiFKMeP3wxw/e4np90SUbc5WU5YxFv7sYRj9FdRtU/VBVlKl++S11hpIz1t1z3to4C6gX972eRjTAk6q7PmKFBkpSV0IUS7RMZe7aXo1Cyh9h1wTfP8QnFgLd38CTfpVdYhWYZU+eaWUq1JqN3AGWKG13grU1VonAuQt6xSz73ilVLRSKjo5Odka4QghRImyTLm8sLAc3TSWXPhxPBz5De54B9rdW/VBWolVkrzWOldr3R5oCHRVSoWXY9/ZWuvOWuvOgYGB1ghHCCFKNOP3w5w4e4n/K0s3jcUCPz8N+3+EW96ALo9cnyCtxKqja7TWqRjdMoOBJKVUEEDe8ow1zyWEEBVRuJumZ2ndNFrD7y/Brq+h7ySb1oWvKGuMrglUSvnl/VwdGAgcApYAY/NeNhZYXNlzCSFEZWTmlLObZvW/YOss6PEk9Hup6gOsAta4GSoImKeUcsX40FigtV6qlNoMLFBKPQzEASOtcC4hhKiwGcuNbppvHulWejfNuhmw/h3o9BDc+qZd3s1aFtYYXfMn0KGI7SnAgMoeXwghrCE65hxzN57g/u5l6KbZMstoxbcdZbflCsrK8e94FUKIUhTupplcWoXJnV/BskkQNgSGfQwurtcnyCri+LVrhBCiFGXuptm7EJY8ZRQcu2cuuDp+ipSWvBDCqW0vazfNoV9h0WMQ3ANGfw1u5Zzb1U5JkhdCOK3MnFxeXPgnDfyqlzwRyF8r4PuxUC8CxnwH1byKf62DkSQvhHBa/5d309P0ERF4F9dNc2AJzL8PAsPg/h/A0/f6BlnFJMkLIZzSqoNJzN14ggd7hBTfTfPnAqMeTf0OMPZn8CrHvK4OQpK8EMLpnErN5Pnv99AqyJeXi7vpaccXRj2akJ7wwCKo7nc9Q7xuJMkLIZyKKdfCU9/sxGS28L/Ijni6FzEEcsvH8PMz0GygURPew+f6B3qdOP74ICGEKGTG74fZGZfKB/d1oHGA97UvWDfDuNGp1Z0w4jOnGUVTHEnyQginsepgEp+sO05kt2CGtqt/5ZNaG8l9/TvGnazDPnaKcfClcf53KIS4IeT3w7cO8uWfQ1pf+aTWsOwl2PoxdBwLQ951+DtZy0qSvBDC4eX3w5tzNR9d3Q9vyYWlz8LOedDtcRj8lkPXoikvufAqhHBsUVHMGP4cO+NSeWv1JzT+/afLz+WaYdEEI8H3fv6GS/AgLXkhhCOLimLVv2fxyZ2TuX/nL9y5YRHs/N14btRw+OEROLQUbv4n9PmHbWO1EWnJCyEc1qlpM3h+4ERaJx3jldVzjI0ZGfCvl+CLIXDoF7hteskJPioKQkPBxcVYRkVdj9CvG2nJCyEckinXwlOdIjG7uPHR4v/gmWsynvB3gUHnIWk/jP7KGCpZnKgoGD/e+GAAiI011gEiI6v2DVwn1pj+r5FS6g+l1EGl1H6l1DN522srpVYopf7KW9aqfLhCCGGY8fthdjZoxVvLPqTx+QRjY7ArPOwNnq7w0C8lJ3iAKVMuJ/h8GRnGdidhje4aM/C81roV0B14QinVGpgMrNJaNwdW5a0LIUSl5Y+Hv792NnfG7TA2tnWDB7wgA2j1OjTsVPqB4uLKt90BVTrJa60TtdY7835OAw4CDYC7gHl5L5sHDKvsuYQQIv58RsF4+FeevQs++QTurAPDvSDZHbq8Aw89U7aDBQeXb7sDsuqFV6VUKMZ8r1uBulrrRDA+CIA6xewzXikVrZSKTk5OtmY4Qggnk5Zl4uEvosm15I2Hd7GAz2bomAUR98JHifDgo2U/4LRp4HVV7XgvL2O7k7BakldK+QA/AH/XWl8s635a69la685a686BgYHWCkcI4WTMuRae/GYXx5LTmXV/Jxp7m+DrEbD7a+g7Ge6eBW7VynfQyEiYPRtCQozx8yEhxrqTXHQFKyV5pZQ7RoKP0lr/mLc5SSkVlPd8EHDGGucqSkp6NiM+3sSaw1V2CiGEDWmtef3nA6w9ksy/hoXTKyAD5g6G2I1GDZr+L1X8JqfISIiJAYvFWDpRggfrjK5RwGfAQa31fws9tQQYm/fzWGBxZc9VnMQLWaSkZ/PQ59t5cO42Dp9Oq6pTCSFs4ItNMXy1JZbH+jThvqAkmDMQLibA/T9C+zG2Ds+uKa115Q6g1E3AemAvYMnb/DJGv/wCIBiIA0Zqrc+VdKzOnTvr6OjoCsWRY7bw1ZZY3l95hPRsM/d2DebZgS0IrOHcZUSFcHarDibx6JfRDAyrw6ywXbj8/jL41jfmYq1TwrytNxCl1A6tdecin6tskremyiT5fOcv5fD+qr/4ekssnu6uTOzflL/1alz0xAFCCLt2IOEi98zaROsAN76t/y1u+76H5rfC8NlQXW69yVdSkneOsgbZ6bDwYUj8k1re1Zg6tA2/P9uH7k38mb7sMAPeWcvPexKwpw80IUTJki5m8fC87bTxPMt8l1dw27cQ+k+B+76TBF8OzpHkzxyAoyvhk95GQaJzJ2ga6MOcsZ355pFu+FZ356n5uxj+8SZ2xp23dbRCiFJk5Jh5ZF40nTI38y0v4X7pNNy/EPq+aNSYEWXmPN01mamw8X1j7kaLGTqPgz4vgk8guRbNDzvi+b/lh0lOy+bOdvWZNLglDWt5lXpYIcT1ZbFoHv9qG+3+mslEtyUQ1B5GfQm1Qmwdmt26ofrkuZgIa9+GnV+Bmyf0fAp6PgkeNbiUbWbW2mN8uv44Fg2P3NSYif2b4eMhddqEsBfvLd5E5+h/cJPrfmMWp9umg7unrcOyazdWks939i9jPscDi8ErAPq8YLTu3TxISM3k/34/zKJdpwjwqcbzt7ZkVOdGuLrcWJMJCGFvli9fSvjGpwh0Scd96LvQ4X5bh+QQbswkny9+B6x8DWLWg1+wMXlA+D3g4sLuk6m8ufQA0bHnCatXg1fuaM1NzQOse34hROm05tgv79Fo+7+44BZIrXHf4tawg62jchg3dpIHYxLfY6tgxVRI2gt12hh3yIUNQQO/7TvNW78d5OS5TAaE1eGl21vRrI6P9eMQQlwr7TQXvn2MmqfWsM2tM62emE+NWkWWuhLFkCSfz2KBfT/Amrfg3DGoFwH9X4YWg8kyW5i3KYaZq4+Sacrl/u4hPDOgObW8y1kLQwhRdvsXYV7yd0xZGXziOY77Jr5O3ZrVbR2Vw5Ekf7VcM+xdAGv/A+djoH4HY/xts4GcvZTDeyuP8M3WOHw83Hh6QHMe6BGCh5vcTCWE1WSmwm8vwp/fsZemvO35HDMeH0GQJPgKkSRfnFwT7JkPa/8PLsRBwy5Gy75Jf46cSWfaLwdZeySZ4NpeTL4tjNvC66FusJnehbC642vhp4notERmMYL51UbyzYSbZEhzJUiSL405xyhXum4GXDwFwT2MZN+4D+uOJPPvXw9y6HQanUNqMeWOVnQIlrvthCg3UyasegO2/I9sv6Y8fPFRjrm34LvxPQj2lwRfGZLky8qcDTu/hPXvQFoihPaGPi+QG9KbhTvjmbH8CMlp2QyJCGLS4DAa1ZY/TCHKJGE3/Dgezh4mte1D3HHgFkwunnz3WA8aB3jbOjqHJ0m+vExZsOML2PBfSE+CBp2h9/NcCh3IJ+tjmL3uGBYLjOsVysT+zahZ3d3WEQthn3LNsPFdWPM2eAeS2P+/DP2tGlrDt+O7yyg2K5EkX1GmLNgdBRvfg9Q4qNMabnqW041uZ8bKY/ywMx6/6u78fWALxnQLxt1VamoIUSBuC/zyPCTtg/ARnOzxBiPnHcaUa2H++O60qFvD1hE6DeevQllV3D2hy8Pw1C64ezZoC/z4KPW+7MWMxjtYOqEzrYJ8eW3Jfga9u45l+05LpUtRflFREBpqFN4KDTXWHdmlFFj8BMwdZIyiGf01J2+eyb1fHSHLnMvXj3STBH8dSZIvC1c3aDcaHt8Mo6Ogem1Y+ixtvu9DVOttfDGmFS4uiglf7+CeWZuJjilxbpRrOdt/clF2UVEwfjzExho37cXGGuuO+DdgscCOeTCzE+z5Fno9A09s5VTQQO77dAtpWSa+frgbrYJ8bR3pDcUq3TVKqbnAEOCM1jo8b1tt4DsgFIgBRmmtS6zza3fdNcXRGk6sNS7QnlgH1Wth6fwoi6vdxlvrznEmLZtBbery4uAwmgaW0ueY/588I+PyNi8vp5tMWBQjNNRI7FcLCTHmG3UUiX/CL89B/HYI6QV3vAN1WnEqNZMxn27hXHoOXz/SjXaN/GwdqVOq8j55pVQfIB34slCSnw6c01q/rZSaDNTSWk8q6TgOk+QLO7nduEB7+FdwrYa5zT0scBvKv3e4kGnK5d4ujXhmYHPq1Cimip6z/CcXFePiYjQarqaU0TK2d1kX4Y9/w7ZPjG+4t74J7e4FpdifcIFxn28nMyeXL/7WlU4hMvS4qlyXC69KqVBgaaEkfxjop7VOVEoFAWu01i1LOoZDJvl8Z4/C1o9h9zdgyiAnuDcL3Iby+qH6uLu58WjvJozv0wTvq8saO/p/clE5jvohrzXs/xGWvWyMQOv8Nxjwz4IZm9YeSWbi1zvwre7O5+O6EFZPumiqkq2SfKrW2q/Q8+e11td8lCulxgPjAYKDgzvFFvUH70gyzsHOebB1NqQlkOPXlB/ch/L6yQh8fHx5ZmBz7u3S6PJIHEf9Ty6swxG76wpXdg1qD0P+Cw06FTz93fY4Xl60j+Z1fPhiXFfq1ZRa8FXNrpN8YQ7dkr9argn2/wRbPoKEXZg9/PjZbRBvpfTBy78Bz97Sgjsj6uMy/xvH+08urCsqCqZMgbg4CA6GadPs83d/5pAxR8OhpcYcDf0mGy14F6Ouk9aad1cc4YPVR+ndPID/RXakhqfcQ3I9SHeNLWltjBfe8hH60C9oXFjv1p1Zl/pyPrAbLwwO4+YdK1GvOMB/cnFjSo0zbmbaMx/cvY3Z1npMBI/LwyBzzBYm//gnP+48xchODfn38LZy38h1VFKSr8p575YAY4G385aLq/Bc9kspCOkBIT1Q506gts+hz+4o+lbbQNzF+sz7uj9f1h/K46t30L2Jv62jFeKy9GRjBFn0Z4CC7hPhpufA+8q/04tZJiZ+vZMNR8/y7MAWPD2gmRTysyPWGl0zH+gHBABJwGvAT8ACIBiIA0ZqrUscQG6zlvz1/rpsyoIDi7FEf4bLya1k487S3G4cqD+CYUOG01aGmQlbyroIm2fC5o/AlAHtI42umZoNr3lp4oVMxn2+naNn0nl7RAT3dLr2NaLqSVmDktj6wlfSfszb5mLZ/S3VctM5ZGnE7rp303Xo4zRpVL9sx3CUPl1h33IyIHqu0XrPPAet74L+r0BgiyJffjDxIuM+3056tpmP7+9I7+aB1zlgkU+SfEnsZXRLziUydy3gwvpPqJd+kAztwb5aAwju9zfqRQwwhloWxdYfUsLxpZ+BbbNh+2dGcm/SHwa8Cg06FrvLuiPJTIzaiY+HG5+P6yJ3sdqYJPmS2OE49dSjWzm+bCYtklfgozI5714X1e5e/Lo/AAHNr3yxvXxICceTfNjoltnzHeTmQMvbjYuqIT2K3cWca+GD1Uf5cPVftKxbg8/HdZHZnOyAJPmS2HGSPJNyjvVLvyTw2I/0Un/iqjTZdTvg0SkS2gw3LoDZ4YeUsGNaQ8wG2PQh/PU7uHlC+zHQ/QkIaFbirokXMnnm291sO3GOER0b8sZdba69uU/YhCT5kjhAd8eZi1lErdxG1q4F3KXW0dolFu3ijmoxCP73B2xKhNyrdrKDDylhR3JNcGAxbPoAEvcY49y7jjeqrHoHlLr76kNJPL9gD9lmC28OC2d4RyteYJVrSpUmSb40DvJHdiYti0/WHmf7lnUMVesY5bEZX/M5yNRwyAQHzXDcDB729SElqlhJf79pScb49u1z4MJJ8G8OPZ4w6su4l97NkmO2MH3ZIeZsOEGrIF9mjulQetG98sZu540sRyBJ3smcScti9trjzN96nM65f/KU5zo6ZG/F1d0COYB/F7jtaWg2EKrJFIVOragkWcMLpk+AGifh6ErQuUZlyJ5PQfNBxV/Ev0pcSgZPzd/JnvgLPNgjhJdvb4Wnu6t147fj7lJHIkneSeUn+6+3xmIxZfNE6Ckia+wmIH6lMUrC3Qua3wKthkKLQVfcoSicROEkWc8FOlSDcDfwcoEa9Y0We/sx116wL8XSPxN46Ye9KAXT74lgcHiQ9WMHuaZkJZLkq5qNu3tS0rOZtzmWLzfHkJphontITSa3Oku79HWoQ0uNKoGuHtD0Zgi7A5oNAN8yjsEX9s3HFdq6QXt3qOsKZg2HzLDHBIezCurKlFWWKZfXfz7A/G1xdAj244N7O1TthPXSkrcKmf6vKtnBzD7+Ph48d0sLNk66mX8OaU1cajbDlrlz29FhLLp5BeaxvxoX2E7vhSVPwn9bwf96wvJ/wvG1YM6uXACOPrOVo8V/KQV2z4f598FzPjDIE0zA0kx4Jw1+yARTg3In+L3xF7hr5kbmb4tjQt+mLHisR9UmeDAaRF5XncPLy9gurEJa8pVlhy0RU66FJbsT+GTdMY4kpdPArzqP9G7M6M4N8Tp/GI6tMvpqYzeDxWQUnWrc2+jDbzYAajcp+8kc/cKZo8R/9qgxMc3h3+DkFmO+4RpB4NYG3lkGJyse/4UMEzOWH+brrbH4e3swY2QE/VrWqaI3UgQHGfhgz6S7pirZcZ+ixaL54/AZZq09xvaY89TycueB7iFEdg+hrq8nZKcbNcGProKjK+B8jLFj7SbQdACE9oJG3cG3hP5YO/yQKxd7jd+SCye3XU7sKX8Z2+u2hZa3GY+g9sbfXwWTpNaaH3ae4q1fD3I+I4cHe4Ty3K0t8C1veWBJ0jYnSb4q2WuSuEp0zDlmrT3OqkNJuCrF4PB6PNQzlE4htS5XDEw5lpfwVxrJ35TXOvQLgeAeENzdWAa0uDxCw44/5MrEnuK/cAriNhv//kd+Ny6eu7gb37Ja3m5cPPcLtsqpDp2+yD9/2sf2mPN0CPbjX3eFE96gZvkP5CjfhJycJPmq5GB/5LEpl/hqcywLok9yMctM6yBfHuoZytD29a8cHpdrgtN/GrXw4zYby0vJxnOefnkJvzs89w7sOgXmq05kZx9yxbLVh7TFAskH8/598x4X4oznPP2MhN7yNuMblaf16sKkZ5t5b8URPt8Ug6+nG5NvC2Nkp0a4uFSwNLCDNHKcnST5quaAX1czcsz8tCuBeZtiOJyUhp+XO6O7NOKB7iE0rFXExTat4dzxy0n/5FY4e8R4zqLhrAWSLJCUC+fd4cXp8ODjRovYnlnjQ7osv39TFiTsvPyBeXIrZF0wnvOpe/lbUnB3o0vG1brlArTW/LI3kX8tPUDSxWzu69qIFweFUcu7WuUObE/fhG5gkuRFsbTWbDl+ji83x7D8QBJaawa2qsvYnqH0aOJfcgvv0lkjWS3/CnavBN9s8Cs0YMsrAOqFQ91wqBcBddsY/f32doNWZT6kr/6QcAHqe8GUCRBWx2itnzkEKUeNi9wAAS2vTOq1Qqv0w/DomXSmLtnPhqNnaVPfl38NC6djcIkzcZadtOTtgk2TvFJqMPA+4ArM0Vq/XdxrJcnbVkJqJl9vieXb7Sc5dymH4NpejOjYkBGdGhTdui9K5nk4vQ+S9uUt9xpJLrfQME3vOkZiK+pRI6jMd2TajMVi9JenJcLoQUAKBLpCHRfwdwHX/IStoFYIBLaCOmHQqJvx8Kp9XcI8dPoiH/1xjF/+TMDbw41/3NqS+7uH4FrRrpmiOFh3pbOyWZJXSrkCR4BbgHhgO3Cf1vpAUa+XJG8fsky5LNt3mu93nGTTsRS0hp5N/RnZuSGD2wRRvVo5b23PNcHZv+DMAWMET8EjFi7GG8MB87lWMy4u+tYHL3/jUb325Z+9al+5dPeqfCvYYgFzJpgyjYvN2emQfhrSThuJ/Ipl3iO/VQ5Gd8V5Dcm5kGy5/Ii5aJNvLXtOpjLzj6OsOJCEdzVX7u8RwqO9mxDg41E1J3TA7kpnY8sk3wOYqrUelLf+EoDW+q2iXi9J3v7En8/gx52nWLgjnrhzGfh4uDEkIoh7OjW8cmRORZlzjMJZqbFXfgCkJUFGivHIPA8U83fqWs24m9fV3Xi4uBv92a7VLv/s4m6s61wjiZsKJXRTJpizSo7Rs6bxDaNGvWuXDzwBf9rHhedtJ84x84+jrDuSjK+nG+N6NWZcr1D8vCrZ7y7snq0m8gZoAJwstB4PdKvicworaljLi6cHNOepm5ux7cQ5Fu6IZ8meBL7dfpLGAd7c06khwzo0oIFfBSeOcKsG/k2NR3EsuZCZanSR5Cf+wh8AuSZj0otcE1jMeUtT3vZCP7tUy2v9V897eBW/rFHPePjUK7k1/ly60V1hvqq74jrdsam1ZsPRs3y4+ijbTpzD37sakwaHcX/3YGqUdby7tMSdWlW35EcCg7TWj+StPwB01Vo/Veg144HxAMHBwZ1ii7qII+zKpWwzv+5NZOGOeLaeMOZmb9ewJoPC63FbeBCNA7xtHOF1ZoMkmWvRrD50hpl/HGXPyVTq+XryWN8m3NsluHzdadKn7hSku0ZUmbiUDH7Zm8iy/afZczIVgJZ1azA4vB6Dw+sRVq9G5bt0SnMDtUSPJKXxw854Fu9K4PTFLBrVrs7jfZsxolMDPNwqUAZYRsc4BVsmeTeMC68DgFMYF17HaK33F/V6SfKOLSE1k2X7TrNs/2m2x5xDawj19ypo4bdrWNP6Cf8GaIkmp2WzZE8Ci3bFs+/URVxdFP1aBDK8Y0MGtamLm2slRiPJOHenYOshlLcD72EMoZyrtS62s1KSvPNITstmxYEkftuXyOZjKZgtmqCantzULIBezQLo2dSfOr6elT+Rk7ZEs0y5rDiQxI8741n311lyLZq2DWoyvGMD7mxX33ojZZz03+9GIzdDCZu6kGFi5cEkVhxIYvPxFC5kGsMPm9XxoVdTf3o2C6B7Y39qepWzMBY4VUs0MyeX7THn+OXPRH7dm0hatpmgmp4M69CA4R0a0LxuFUz6cgN8E7oRSJIXdiPXojmQcJFNx86y8VgK20+cI9OUi4uC8AY16dnUaOV3CqmFt0cZBn85cEs0y5TLrrhUNh9PYfOxs+w+mYopV+NdzZXb2gYxvGMDujcu5a5ja7iBrmk4K0nywm7lmC3smruATUvWsalmKLsahGF2cUUpaBzgTXj9mrSp70ubvOU1tVYcqCWaY7bwZ3wqm4+lsOlYCjvjzpNttuCioG2DmvRoGkCPpv50Da1d/hvOxA1NkrywX1cl6Uvunmxr1pE9jzzL/oBQ9p+6QMKFyzcr1a/pSZsGlxN/6/q+1Fv6I66v2E9LVGtNclo2x89e4njyJY4np3M4KY3omPNkmnIBaB3kS4+m/vRs6k+XxrXLX8NdiEIkyQv7VYbulnOXcjiQcJH9CRfYl7c8cfZSQVe8m4uiXk1P6vtVp4Ffder7GT/nrwfV9Cz7jUFlZMq1cCnbTEJqFsfPphck8+NnL3Ei+RJp2ZdvgfVwc6FJoA9dQ2vRo6k/3Rr7V776oxCFSJIX9quCF04vZZs5dPoiBxPTSEjNzHtkcSo1k9MXs8i1XHnMGp5u1PauhqebK57uLni4ueLh7oKnuyue7q54uLng6e6Cp5srbq4uZOaYScs2k55lJj3bTNoVSxNZpmtja+BXnSaB3jQJ8KZxgDdNAn1oEuhN/ZrVq75fXdzQbFnWQIiSBQcX3ZIPLnkGJG8PNzqF1KZTyLUVHXMtmjNpWSSkZhV8AJxKzeRCpoksUy7ZZgtZplzSsswkp2WTk7eelbc05VrwquaGj4cbNTyNR4BPNUIDvAu2+XgYjzq+HjQJ8KFxgLf0owu7JEle2Na0aUVfOK1E7RdXF0VQzeoE1axOpxAr1U0XwkHZeeFu4fQiI42RMCEhRhdNSIhdjowRwlFJS17YXmSkJHUhqoi05IUQwolJkhdCCCcmSV4IIZyYJHkhhHBikuSFEMKJSZIXQggnJkleCCGcWKWSvFJqpFJqv1LKopTqfNVzLymljiqlDiulBlUuTCGEEBVR2Zuh9gHDgU8Kb1RKtQbuBdoA9YGVSqkWWuvcSp5PCCFEOVSqJa+1Pqi1PlzEU3cB32qts7XWJ4CjQNfKnEsIIUT5VVWffAPgZKH1+Lxt11BKjVdKRSulopOTk6soHCGEuDGV2l2jlFoJ1CviqSla68XF7VbEtiIL12utZwOzwagnX1o8Qgghyq7UJK+1HliB48YDjQqtNwQSKnAcIYQQlVBV3TVLgHuVUh5KqcZAc2BbFZ1LCCFEMSo7hPJupVQ80AP4RSn1O4DWej+wADgALAOekJE1Qghx/VVqCKXWehGwqJjnpgEVn95HCCFEpckdr0II4cQkyQshhBOTJC+EEE5MkryAqCgIDQUXF2MZFWXriIQQViITed/ooqJg/HjIyDDWY2ONdZDJtYVwAtKSv9FNmXI5wefLyDC2CyEcniT5G11cXPm2CyEciiT5G11wcPm2CyEciiT5G920aeDldeU2Ly9juxDC4UmSv9FFRsLs2RASAkoZy9mz5aKrEE5CRtcII6FLUhfCKUlLXgghnJgkeSGEcGKS5IUQwolJkhdCCCcmSV4IIZyY0tp+5s5WSiUDsZU4RABw1krh2JKzvA+Q92KPnOV9gLyXfCFa68CinrCrJF9ZSqlorXVnW8dRWc7yPkDeiz1ylvcB8l7KQrprhBDCiUmSF0IIJ+ZsSX62rQOwEmd5HyDvxR45y/sAeS+lcqo+eSGEEFdytpa8EEKIQiTJCyGEE3OqJK+U+pdS6k+l1G6l1HKlVH1bx1RRSqn/U0odyns/i5RSfraOqaKUUiOVUvuVUhallMMNd1NKDVZKHVZKHVVKTbZ1PBWllJqrlDqjlNpn61gqSynVSCn1h1LqYN7f1jO2jqkilFKeSqltSqk9ee/jdaufw5n65JVSvlrri3k/Pw201lpPsHFYFaKUuhVYrbU2K6X+A6C1nmTjsCpEKdUKsACfAP/QWkfbOKQyU0q5AkeAW4B4YDtwn9b6gE0DqwClVB8gHfhSax1u63gqQykVBARprXcqpWoAO4BhjvZ7UUopwFtrna6Ucgc2AM9orbdY6xxO1ZLPT/B5vAGH/QTTWi/XWpvzVrcADW0ZT2VorQ9qrQ/bOo4K6goc1Vof11rnAN8Cd9k4pgrRWq8Dztk6DmvQWidqrXfm/ZwGHAQa2Daq8tOG9LxV97yHVfOWUyV5AKXUNKXUSSASeNXW8VjJ34DfbB3EDaoBcLLQejwOmEycmVIqFOgAbLVxKBWilHJVSu0GzgArtNZWfR8Ol+SVUiuVUvuKeNwFoLWeorVuBEQBT9o22pKV9l7yXjMFMGO8H7tVlvfioFQR2xz2G6KzUUr5AD8Af7/qm7zD0Frnaq3bY3xb76qUsmpXmsNN/6e1HljGl34D/AK8VoXhVEpp70UpNRYYAgzQdn7xpBy/F0cTDzQqtN4QSLBRLKKQvD7sH4AorfWPto6nsrTWqUqpNcBgwGoXxx2uJV8SpVTzQqtDgUO2iqWylFKDgUnAUK11hq3juYFtB5orpRorpaoB9wJLbBzTDS/vguVnwEGt9X9tHU9FKaUC80fOKaWqAwOxct5yttE1PwAtMUZyxAITtNanbBtVxSiljgIeQErepi0OPFLobuBDIBBIBXZrrQfZNKhyUErdDrwHuAJztdbTbBtRxSil5gP9MEraJgGvaa0/s2lQFaSUuglYD+zF+P8O8LLW+lfbRVV+SqkIYB7G35YLsEBr/YZVz+FMSV4IIcSVnKq7RgghxJUkyQshhBOTJC+EEE5MkrwQQjgxSfJCCOHEJMkLIYQTkyQvhBBO7P8BYAELyEydIo4AAAAASUVORK5CYII=\n", 573 | "text/plain": [ 574 | "
" 575 | ] 576 | }, 577 | "metadata": { 578 | "needs_background": "light" 579 | }, 580 | "output_type": "display_data" 581 | } 582 | ], 583 | "source": [ 584 | "print(\"weights without regularization\")\n", 585 | "print(no_regularization_params)\n", 586 | "print(\"weights with regularization\")\n", 587 | "print(params)\n", 588 | "\n", 589 | "pyplot.scatter(x, y, c='r', label='true')\n", 590 | "pyplot.plot(xgrid, Xgrid_scaled@no_regularization_params, label='w/o regularization')\n", 591 | "pyplot.plot(xgrid, Xgrid_scaled@params, label='with regularization')\n", 592 | "pyplot.legend();" 593 | ] 594 | }, 595 | { 596 | "cell_type": "markdown", 597 | "metadata": {}, 598 | "source": [ 599 | "Using our helper script again to display both models with varying polynomial degree in an `ipywidget`, interact with the slider to explore the results. We set `regularized=True` this time." 600 | ] 601 | }, 602 | { 603 | "cell_type": "code", 604 | "execution_count": 14, 605 | "metadata": {}, 606 | "outputs": [ 607 | { 608 | "data": { 609 | "application/vnd.jupyter.widget-view+json": { 610 | "model_id": "e0c720ccb75d4994b1bb51d650dd8ab5", 611 | "version_major": 2, 612 | "version_minor": 0 613 | }, 614 | "text/plain": [ 615 | "interactive(children=(IntSlider(value=8, description='degree', max=15, min=1), Output()), _dom_classes=('widge…" 616 | ] 617 | }, 618 | "metadata": {}, 619 | "output_type": "display_data" 620 | } 621 | ], 622 | "source": [ 623 | "interact_polyreg(max_degree, x, y, regularized=True)" 624 | ] 625 | }, 626 | { 627 | "cell_type": "markdown", 628 | "metadata": {}, 629 | "source": [ 630 | "Thanks to the regularization term, you won't see the wiggling curves even with a high degree polynomial, since the magnitude of the weights (the coefficients before each polynomial term) is much smaller." 631 | ] 632 | }, 633 | { 634 | "cell_type": "markdown", 635 | "metadata": {}, 636 | "source": [ 637 | "## Ridge regression with scikit-learn\n", 638 | "\n", 639 | "You won't be surprised by now that **scikit-learn** offers a method to obtain a linear model with regularization: [`Ridge()`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge). \n", 640 | "Look and ponder at the code cell below, where we call `Ridge()` and its `.fit()` method to fit the ridge regression model with our scaled matrix of features (also called the _design matrix_) and the $y$ data.\n", 641 | "The default value of the penalization parameter of $1.0$, so we don't specify it, but if you wanted to try a different value, you specify it as `alpha=value` in the argument list. Do explore the documentation page." 642 | ] 643 | }, 644 | { 645 | "cell_type": "code", 646 | "execution_count": 15, 647 | "metadata": {}, 648 | "outputs": [ 649 | { 650 | "name": "stdout", 651 | "output_type": "stream", 652 | "text": [ 653 | "[12.48291164 28.26642615 11.09583654]\n", 654 | "-11.138315887456029\n" 655 | ] 656 | }, 657 | { 658 | "data": { 659 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAArgklEQVR4nO3deXxU1d348c83ISEEAiQhhDUJSMIWCUsC+LCjIK0oqMUKUXiqNm59bG1xK32pqPwen2pbatXSWLVYxwUX1LqyCCoKQlgEJEAAkxDAJATIQhaSmfP7YyYxSBJIZiaz5Pt+vfK6c++dued7JzPfe+bcc+8RYwxKKaX8U4CnA1BKKeU+muSVUsqPaZJXSik/pkleKaX8mCZ5pZTyY+08HUB93bp1M3FxcZ4OQymlfMrWrVuPG2OiGlrnVUk+Li6OjIwMT4ehlFI+RURyGlunzTVKKeXHNMkrpZQf0ySvlFJ+zKva5BtSXV1NXl4elZWVng5FtQEhISH06dOHoKAgT4eilEt4fZLPy8sjLCyMuLg4RMTT4Sg/ZoyhqKiIvLw8+vXr5+lwlHIJr2+uqaysJDIyUhO8cjsRITIyUn81qtZlsUBcHAQE2KcWi0s37/U1eUATvGo1+llTrcpigbQ0KC+3z+fk2OcBUlNdUoTX1+SVUspvLVr0Q4KvVV5uX+4imuRbKC4ujuPHj5+zvFOnTh6I5gcZGRncddddDa5rLGZ/sGzZMl566SVPh6FU8+TmNm95C/hEc01bZLVaCQwMbNZrampqSE5OJjk52U1RNVxmu3Yt/xgZYzDGEBDgXH3jtttuc+r1SnlETIy9iaah5S6iNfnzOH36NFdccQVJSUkkJiby+uuvn7W+oqKCGTNm8Nxzz53z2ieeeIKUlBSGDRvGQw89VLd89uzZjBo1iqFDh5Kenl63vFOnTjz44IOMGTOGjRs30qlTJxYtWkRSUhJjx44lPz//nDIefvhh0tLSmD59OvPnz2f9+vXMnDkTgKKiIqZPn86IESO49dZbqT8K2KOPPsqgQYOYNm0ac+fO5cknnwTg4MGDzJgxg1GjRjFhwgT27t173jILCwu59tprSUlJISUlhS+//BKAwsJCpk2bxsiRI7n11luJjY3l+PHjZGdnM3jwYO644w5GjhzJ4cOHG3yvGnvv77//foYMGcKwYcNYuHBhXUy1+7Bjxw7Gjh3LsGHDuPrqqzl58iQAkydP5r777mP06NEkJCTwxRdfNPm/V8rtliyB0NCzl4WG2pe7iE/V5Bf/51v2HC1x6TaH9OrMQ1cObXT9xx9/TK9evfjggw8AKC4urltXVlbG9ddfz/z585k/f/5Zr1u1ahVZWVls3rwZYwxXXXUVn3/+ORMnTuSFF14gIiKCiooKUlJSuPbaa4mMjOT06dMkJibyyCOPAPYkN3bsWJYsWcK9997Lc889xx/+8IdzYty6dSsbNmygQ4cOrF+/vm754sWLGT9+PA8++CAffPBB3QElIyODt956i+3bt1NTU8PIkSMZNWoUAGlpaSxbtoz4+Hi+/vpr7rjjDj799NMmy5w3bx53330348ePJzc3l8svv5zMzEwWL17M1KlTeeCBB/j444/POqDt27ePF198kWeffbbR96qwsPCc9/7EiROsXLmSvXv3IiKcOnXqnNjmz5/P3/72NyZNmsSDDz7I4sWLWbp0KWD/5bF582Y+/PBDFi9ezJo1axr93yvldrUnVxctsjfRxMTYE7yLTrqCjyV5T7j44otZuHAh9913HzNnzmTChAl162bNmsW9995LagP/kFWrVrFq1SpGjBgB2A8IWVlZTJw4kaeeeoqVK1cCcPjwYbKysoiMjCQwMJBrr722bhvBwcF1tfJRo0axevXqBmO86qqr6NChwznLP//8c95++20ArrjiCsLDwwHYsGEDs2bNqnvNlVdeWRfjV199xZw5c+q2UVVVdd4y16xZw549e+rWlZSUUFpayoYNG+r2c8aMGXXlA8TGxjJ27Ngm36sJEyac897X1NQQEhLCLbfcwhVXXFH3/tQqLi7m1KlTTJo0CYAFCxactT/XXHNN3fuZnZ3d4L4p1apSU12a1H/Mp5J8UzVud0lISGDr1q18+OGHPPDAA0yfPp0HH3wQgHHjxvHRRx8xb968c7reGWN44IEHuPXWW89avn79etasWcPGjRsJDQ1l8uTJdf2yQ0JCzmqHDwoKqttuYGAgNTU1DcbYsWPHRuNvqEtgY4O322w2unbtyo4dOxrdXkNl2mw2Nm7ceM6BpqlB4uu/vrH3Cmjwvd+8eTNr167ltdde4+mnn27wl0Zj2rdvDzT9firlT7RN/jyOHj1KaGgoN9xwAwsXLmTbtm116x555BEiIyO54447znnd5ZdfzgsvvEBZWRkAR44coaCggOLiYsLDwwkNDWXv3r1s2rTJbbFPnDgRi+PCio8++qiubXr8+PH85z//obKykrKysrrmkM6dO9OvXz/eeOMNwJ58v/nmm/OWM336dJ5++um6+dqDxPjx41mxYgVgr63Xlv9jjb1XDb33ZWVlFBcX89Of/pSlS5eec0Dq0qUL4eHhde3t//73v+tq9Uq1RT5Vk/eEXbt2cc899xAQEEBQUBB///vfz1q/dOlSbrrpJu69917++Mc/1i2fPn06mZmZXHLJJYD9pOrLL7/MjBkzWLZsGcOGDWPgwIF1TRbu8NBDDzF37lxGjhzJpEmTiHGcsU9JSeGqq64iKSmJ2NhYkpOT6dKlCwAWi4Xbb7+dxx57jOrqaq6//nqSkpKaLOepp57izjvvZNiwYdTU1DBx4kSWLVtWV/7rr7/OpEmT6NmzJ2FhYXXJvFZj79WBAwfOee9LS0uZNWsWlZWVGGP4y1/+ck48y5cv57bbbqO8vJz+/fvz4osvuuLtVMonSVM/qVtbcnKy+fGgIZmZmQwePNhDEfmvsrIyOnXqRHl5ORMnTiQ9PZ2RI0e6tIyqqioCAwNp164dGzdu5Pbbb7+gpiBP08+c8jUistUY02Dfaa3Jt1FpaWns2bOHyspKFixY4PIED5Cbm8t1112HzWYjODi4wW6mSin3ckmSF5GuwD+BRMAANwH7gNeBOCAbuM4Y03CjrGp1r7zyitvLiI+PZ/v27W4vRynVOFedeP0r8LExZhCQBGQC9wNrjTHxwFrHvFJKqVbkdJIXkc7AROB5AGPMGWPMKWAWsNzxtOXAbGfLUkop1TyuqMn3BwqBF0Vku4j8U0Q6AtHGmGMAjmn3hl4sImkikiEiGYWFhS4IRymlVC1XJPl2wEjg78aYEcBpmtE0Y4xJN8YkG2OSo6KiXBCOUkqpWq5I8nlAnjHma8f8m9iTfr6I9ARwTAtcUJZHnDp1imeffdbTYSilVLM5neSNMd8Dh0VkoGPRpcAe4D1ggWPZAuBdZ8vylMaSvNVq9UA0Sil14VzVu+Z/AIuI7ASGA/8PeByYJiJZwDTHvPu5YbzE+++/n4MHDzJ8+HBSUlKYMmUK8+bN4+KLLyY7O5vExMS65z755JM8/PDDwIXdtlcppdzJJf3kjTE7gIautrrUFdu/YG4aL/Hxxx9n9+7d7Nixg/Xr13PFFVewe/du+vXr1+SdDC/0tr1KKeUu/nXFa1PjJbrwVp6jR4+mX79+TT6nObftVUopd/GvJN8K4yXC2bfJbdeuHTabrW6+9rbBzbltr1JKuYt/3Wq4sXERnRwvMSwsjNLS0gbXRUdHU1BQQFFREVVVVbz//vtAy2/bq5RSruRfSd5N4yVGRkYybtw4EhMTueeee85aFxQUVDcu68yZMxk0aFDdOovFwvPPP09SUhJDhw7l3Xd9toORUspH+d+thi0Wt46XqPyf3mpY+Zq2dathN4+XqJRSvsS/mmuUUkqdxSeSvDc1KSn/pp815W+8PsmHhIRQVFSkXz7ldsYYioqKCAkJ8XQoSrmM17fJ9+nTh7y8PPQ2xKo1hISE0KdPH0+HoZTLeH2SDwoKOu/VpUoppRrm9c01SimlWk6TvFJK+TFN8kop5cc0ySullB/TJK+UUn7MJb1rRCQbKAWsQI0xJllEIoDXgTggG7jOGHPSFeUppZS6MK6syU8xxgyvd5Oc+4G1xph4YK1jXimlVCtyZ3PNLGC54/FyYLYby1JKKdUAVyV5A6wSka0i4hhUlWhjzDEAx7R7Qy8UkTQRyRCRDL2qVSmlXMtVV7yOM8YcFZHuwGoR2XuhLzTGpAPpYL+fvIviUUophYtq8saYo45pAbASGA3ki0hPAMe0wBVlKaWUunBOJ3kR6SgiYbWPgenAbuA9YIHjaQsAHftOKaVamSuaa6KBlSJSu71XjDEfi8gWYIWI3AzkAnNcUJZSSqlmcDrJG2MOAUkNLC8CLnV2+0oppVpOr3hVSik/pkleKaX8mCZ5pZTyY5rklVLKGRYLxMVBQIB9arF4OqKzeP3wf0op5bUsFkhLg/Jy+3xOjn0eIDXVc3HVozV5pZRqqUWLfkjwtcrL7cu9hCZ5pZRqqdzc5i33AE3ySinVUjExzVvuAZrklVKqpZYsgdDQs5eFhtqXewlN8kop1VKpqZCeDrGxIGKfpqd7zUlX0N41SinlnNRUr0rqP6Y1eaWU8mOa5JVSyo9pkldKKT+mSV4ppfyYy5K8iASKyHYRed8xHyEiq0UkyzENd1VZSinlT4rLq7HZ3DPEtStr8r8GMuvN3w+sNcbEA2sd80oppeqprLZy4wtf87s3vnHL9l2S5EWkD3AF8M96i2cByx2PlwOzXVGWUkr5C2MMi1buZmdeMT9J7OGWMlxVk18K3AvY6i2LNsYcA3BMu7uoLKWU8gv/+iqbt7bl8ZvL4pk+1EuTvIjMBAqMMVtb+Po0EckQkYzCwkJnw1FKKZ/w1cHjPPZBJtOGRHPX1Hi3leOKmvw44CoRyQZeA6aKyMtAvoj0BHBMCxp6sTEm3RiTbIxJjoqKckE4Sinl3fJOlvOrV7YTFxnKn69LIiBA3FaW00neGPOAMaaPMSYOuB741BhzA/AesMDxtAXAu86WpZRSvq7ijJVb/72V6hobz81PJiwkyK3lubOf/OPANBHJAqY55pVSqs0yxnD/2zvZc6yEv84dTv+oTm4v06U3KDPGrAfWOx4XAZe6cvtKKeXL/vnFd7y74ygLpycwdVB0q5SpV7wqpVQr2JB1nP/9KJOfJPbgzikDWq1cTfJKKeVmuUXl/OrVbcR3D+PJOUmIuO9E649pkldKKTcqP1ND2r8zsNkM6fNH0bF96w7joYOGKKWUmxhjuOfNnezPL+XFX4wmNrJjq8egNXmllHKTZZ8d4oOdx7h3xiAmJXjmOiBN8kop5QZrM/P54yd7mTmsJ7dO7O+xODTJK6WUi+3KK+ZXr2xnaK/O/PFnw1r1ROuPaZJXSikXOnyinJuWbyGiYzAvLEghNNizpz71xKtSSrlIcXk1v/jXFiqrrbxyyxi6dw7xdEhak1dKKVeoqrFy68sZ5BSd5h83jiI+OszTIQFak1dKKacZY7jvzZ1sOnSCpT8fzn9d1M3TIdXxj5q8xQJxcRAQYJ9aLJ6OSCnVhvxp1X7ecdyTZvaI3p4O5yy+X5O3WCAtDcrL7fM5OfZ5gNRUz8WllGoTXtucy9PrDnB9St9WvSfNhfL9mvyiRT8k+Frl5fblSinlRp/tL2TRO7uZmBDFo7MTPdpVsjG+n+Rzc5u3XCmlXODbo8Xc8fJWBkaH8WzqSIICvTOdemdUzRET07zlSinlpKOnKrjpX1vo3CGIF3+RQqdWvulYc7hiIO8QEdksIt+IyLcistixPEJEVotIlmMa7ny4DViyBEJDz14WGmpfrpRSLlZSWc1N/9pCeZWVF3+RQrQX9IVviitq8lXAVGNMEjAcmCEiY4H7gbXGmHhgrWPe9VJTIT0dYmNBxD5NT9eTrkopl6ustnL7y1s5UFDGshtHMahHZ0+HdF5O/8YwxhigzDEb5PgzwCxgsmP5cuzDAt7nbHkNSk3VpK6Ucqtqq41fvbKNLw8U8efrkhg3wHv6wjfFJW3yIhIoIjuAAmC1MeZrINoYcwzAMe3eyGvTRCRDRDIKCwtdEY5SSrmU1Wa4+/UdrMks4NHZiVwzso+nQ7pgLknyxhirMWY40AcYLSKJzXhtujEm2RiTHBXlmfstK6VUY2w2wwNv7+T9ncd44CeDuHFsrKdDahaX9q4xxpzC3iwzA8gXkZ4AjmmBK8tSSil3M8bwyPt7WJGRx12XxnPrpIs8HVKzuaJ3TZSIdHU87gBcBuwF3gMWOJ62AHjX2bKUUqo1PblqH//6Kptbxvfj7sviPR1Oi7iic2dPYLmIBGI/aKwwxrwvIhuBFSJyM5ALzHFBWUop1SqeWXeAZ9YdZO7oGBZdMdgrr2a9EK7oXbMTGNHA8iLgUme3r5RSre3FL7/jiU/2MXt4Lx7z0tsVXCjfv+JVKaVcaMWWwyz+zx4uHxrNk3OSCAzw3QQPmuSVUqrOe98c5b63dzIxIYqn5o6gnZfej6Y5fH8PlFLKBVbvyee3r+8gJTaCf9wwivbtAj0dkktokldKtXnr9hVwp2UbQ3t15vn/TqZDsH8keNAkr5Rq4z7efYy0lzKIj+7E8ptGExYS5OmQXEqTvFKqzXpn+xHufGU7F/fuwiu/HEvX0GBPh+Ry3nsTZKWUcqNXN+fy+5W7GNMvgucXpNDRi+8J7wz/3CullGrCCxu+45H39zB5YBTLbhhFSJD/tMH/mCZ5pVSb8sy6AzzxyT5mDO3BX+cO95teNI3RJK+UahOMMTy5ah/PrDvI7OG9eHJOkl/0gz8fTfJKKb9XezfJF7/MZu7ovjw2+2Kfv5L1QmmSV0r5NavN8Id3dvHq5sP8YlwcD84c4tP3omku//+topTybxYLxMVBQIB9arHUraqx2vjdih28uvkwd065qM0leNCavFLKl1kskJYG5eX2+Zwc+zxQed31/Pq17XzybT73XD6QO6cM8GCgnqM1eaWU71q06IcEX6u8nOOP/C/Xp29i1Z58Hr5ySNMJvolfAv5Aa/JKKd+Vm3vOogMRffjFlLsp/L6Ev6eOYkZij8Zf38QvAVJT3RBw63PF8H99RWSdiGSKyLci8mvH8ggRWS0iWY5puPPhKqVUPTExZ81+3Wco197wBBUhobyWdknTCR4a/SXAokUuDtRzXNFcUwP8zhgzGBgL3CkiQ4D7gbXGmHhgrWNeKaVcZ8kSCA0F4J0hk7nx54/RraKYlUkwvG/X87++gV8CTS73Qa4Y/u8YcMzxuFREMoHewCxgsuNpy4H1wH3OlqeUUnVSUzEGnn7lC/407ErG5u/nH1Oi6TJ/7oW9PibG3kTT0HI/4dI2eRGJwz7e69dAtOMAgDHmmIh0b+Q1aUAaQIwfvbFKKferttr4fftE3hjWlWtG9Obxa39CcLtmNFAsWXJ2mzzYfxksWeL6YD3EZb1rRKQT8BbwG2NMyYW+zhiTboxJNsYkR0VFuSocpZSfK66o5r9f3MwbW/P49aXx/Om6pOYleLCfXE1Ph9hYELFP09P95qQruKgmLyJB2BO8xRjztmNxvoj0dNTiewIFrihLKaXyTpZz07+2cKjwNE/OSeJno/q0fGOpqX6V1H/MFb1rBHgeyDTG/LneqveABY7HC4B3nS1LKaW25Z7k6me/4lhxJS/dNNq5BN8GuKImPw64EdglIjscy34PPA6sEJGbgVxgjgvKUkq1UcYY/r0ph0ff30OPLiFYbhlDQnSYp8Pyeq7oXbMBaOxmEJc6u32llCo/U8OilbtZuf0IUwZGsfTnI+gS6l9jsbqLX9zW4HRVDXe9up1vjxZ7OhSllItlHz/NNc9+xTs7jvDbaQk8vyBFE3wz+MVtDfZ+X8pn+wt575ujzBrei99NG0hMZKinw1JKOWn1nnx+u2IHgQHCv34xmkkJ2gOvucQY4+kY6iQnJ5uMjIwWvba4opp/fHaQF778DqvNkDomll9NHUC3Tu1dHKVSyt2sNsOfVu3j2fUHubh3F55NHUnfCK24NUZEthpjkhtc5y9JvlZ+SSVL12SxIuMwIe0C+OXE/twyoT+d/HQkdqX8TVFZFXe9tp0vDxQxd3RfHrpyqF8PtO0KbSrJ1zpYWMafVu3jw13fE9kxmP+ZOoB5Y2Kbf7GEUqrVbM89yR2WbRSdPsNjsxK5LqWvp0PyCW0yydfacfgU//fRXjYeKqJvRAcWTh/IlcN6EdBGxndUyhfU7x4Z3TmEZTeMIrF3F0+H5TPadJIH+wfo86zjPP7RXjKPlTCoRxh3T0tg+pDoNjcUmFLepqCkknvf2sn6fYVMHhjF0p8Pp2tosKfD8iltPsnXstkM/9l5lKVrsvju+GkSe3fm7ssSmDqouyZ7pTzgg53HWPTOLiqrrfz+p4O5cWysfhdbQJP8j9RYbbyz4yhPrc0i90Q5SX26cPe0BCYlROkHTKlWUFxRzcPvfcvK7UdI6tOFP/98OBdFdfJ0WD5Lk3wjqq023t6Wx1NrD3DkVAUjY7ry22kDGTcgUpO9Um7y1YHjLHzjG/JLq/ifqQO4c8oAggK1Q4QzNMmfx5kaG29sPczTnx7gWHElo+MiuHtaApdcFNnqsSjlryqrrfzx43288OV39O/Wkb/8fDhJFzJ6kzovTfIXqKrGyutbDvPMugPkl1Qxtn8Ed02N55KLtGavlDN2HynmN6/v4EBBGQsuieX+nwymQ7D2fXcVTfLNVFlt5dXNuTy7/iCFpVUM79uVO6cM4NJB3bXrpVLNUGO1seyzgyxdk0Vkp2Ce+FkSE/XWBC6nSb6FKqutvLk1j2WfHSTvZAUDo8O4ffJFzBzWk3bahqhUkzKyT/CHd3az9/tSrkzqxaOzhmrXSDdpKslrpmpCSFAgN4yNZf3Cyfzl50nYjOE3r+9g6p8+w/J1DpXVVk+HqPyBxQJxcRAQYJ9aLJ6OyCknTp/h3je/4WfLNlJSUc2yG0bxt7kjNMF7iNbkm8FmM6zOzOfZdQf4Jq+Y7mHt+eWE/swbE0NHZ+6NY7HAokWQm2sfJX7JEr8ejkzVY7E0PJC0D44zarMZVmQc5vGP91JWWcPNE/px19R4574b6oK4vblGRF4AZgIFxphEx7II4HUgDsgGrjPGnGxqO96e5GsZY/jqYBHPrDvAVweL6BoaxPxL4rhxbCxRYc2866UffclVC8TFQU7OuctjYyE7u7WjabFvjxbzh3d2sz33FKP7RfDY7EQdtakVtUaSnwiUAS/VS/J/BE4YYx4XkfuBcGPMfU1tx1eSfH3bck/y7LqDrMnMJzgwgFnDe3HzhH4M6tH5wjbgJ19y1UIBAdDQd1AEbLbWj6eZSiur+fPq/Sz/Kpvw0GB+/9PBXDOyt/ZGa2WtcuJVROKA9+sl+X3AZGPMMRHpCaw3xgxsahu+mORrHSos48Uvs3lzax4V1VbGDYjk5vH9mJxwnh45Pv4lV07y0YO8MYb3dx7j0ff3UFhWReqYGO6ZPkhHbPIQTyX5U8aYrvXWnzTGhDfwujQgDSAmJmZUTkMfeB9yqvwMr24+zPKvsvm+pJL+UR25aVw/rh3Zp+F+wT76JVcu4oPNdfXv7Hpx7y48NjtRL2ryMK9O8vX5ck3+x6qtNj7cdYznN3zHzrxiuoYGMW90DAv+K47oziE/PNEHv+TKxXzkxHtWfilPrtrHJ9/mE9kxmN9cFs+8MbEE6rUjHqfNNR5kjCEj5yTPf/Edq/Z8T4AIlyf2IHVMDJf0d1xJ6yNfctU25Z0sZ+maLN7elkdocDt+OaE/N0/op6OteZGmkrw7/0vvAQuAxx3Td91YltcSEVLiIkiJiyC3qJyXNmbz5rY8Pth5jP7dOjJvTAzXzp5DuCZ15WWOl1XxzLoDWDblgsBN4/pxx5QBRHTU/u6+xFW9a14FJgPdgHzgIeAdYAUQA+QCc4wxJ5rajsdq8q1ck66stvLR7mNYNuWSkXOS4HYBzLy4J/PGxDAqNlx7JiiPKq2s5rkvvuP5Lw5RUW1lzqi+/PqyeHp17eDp0FQj9LYGTfFwm/je70t45etcVm47QmlVDQOjw0gdG8PsEb3pHHKBPRW0uUe5QMUZK5avc3hm3QFOllfz04t78NtpAxnQXe/z7u00yTfFS3q3lJ+p4T/fHMXydS4784rpEBTIzGE9uWZkH8b0i2i8G6aeuFVOKiyt4qWN2by8KYeT5dVMiO/GPZcPZFifrp4OTV0gTfJN8cJ+6jvzTvHK17m8v/MYZVU19O7agatH9Obqkb3PHT3HSw5SyvccKCjln198x9vbj1BttXHZ4GjSJvYnJS7C06GpZtIk3xQvTpIVZ6ys2vM9b287whdZhdgMJPXtyrUjezNzWC/7CTAvPEgp72WMYdOhEzz3xSE+3VtA+3YB/GxUH24e34/+Ovyez9Ik3xQfae4oKKnkvW+O8ta2I2QeKyEoUJgysDvXpD/KlE0f0t5ac/YLvOAgpbxH7XUbz31xiN1HSojsGMz8S+K4YWwMkZ2aeb8lV9NzSk7TJH8+PvYh23O0hJXb83hnx1EKS6voUlnG9P0b+cm+LxmXs4P27YO97iCl3KiJz29BaSVvbzvCvzfmcORUBf2jOnLL+P5cM7I3IUFeMDKTj1SyvJ0meT9VY7Xx5cEi3nl7A2sKrJQGdyDsTAVTowL5yZVjmZTQXYdY83cNJMmqTp359P+e443Qfny2vxCrzTC6XwRpE/oz1dtGN/Pi5lJfokm+DaiqsfLVgSI+2n2M1XvyOVleTYegQKYMimJGYk+mDuquVyj6I0eSNMC30RfxZuKlvDN0Mqc6dKZH5xCuGdmba0f1OfeEvbfQc0ou4akrXtsOL2juad8ukCmDujNlUHdqrDa+/u4EH+0+xiff5vPhru8JbhfAxPhuTB/Sg4kJUfToEnL+jSqvd/x4Me8kz+LNiy9jb/d+BNecYXrWJubsWsP4Axnef1+ZmJiGa/IxMa0fi5/SmryzvLxN0WozbMs9yUe7vufj3cc4WlwJwKAeYUxMiGJSQhTJceG0b+dEs44XHOSc4mPxnzx9hnX7Cvho9/es232UmoBAko7uY86uNVyZ+Tldqk77TnOHl39/fIU217iTD7UpGmPY+30pn+8v5LP9hWzJPkG11dAhKJD/uiiSSQOjmBgfRVy3jhe+UV//kvpI/IcKy1ibWcDqzHwysk9gMxDduT2zg4v52dIHiD+S9cOTvTD+JvnYQdYbaZJ3Jx9uUzxdVcOmQ0V85kj6OUX2RBcbGcrE+CjG9I8gOTai6aYdHzrINchL46/9BbZmTz6rM/M5VHgagME9OzNtcHcuGxJNYq8u9pOonk6Sni5faZJ3Ky9NEi2Rffw0n2cV8tm+QjYeKqL8jBWAvhEdSI6NIDkunJS4CAZEdfqhh4YPH+QAr4r/WHEFW7JP8tm+Qj7daz95HhQojO0fybQh0Uwd1J0+4aGtGtN5+cgvIX+nSd6d/PRDXm21kXmshC3ZJ8nIPsGW7JMcL6sCoEuHIJJjw0mOiyB5YRoXf7OBkJozZ2/AVw5yHjpI22yGrIIytmSfqHt/j5yqAOzv79RB3blscDQTE7oRdqE3qvMEP6rk+DJN8u7WBn6uGmPIKSonI6c26Z/goKMJIdBmpf+JPAYXfGf/Kz7KkPt+RdSCud5/22RXHKQv4P9fWW1l15FiR1K3v4cllfarlKPC2pMSF05yrH3cgcE9w2gXGOCqPXQvL/ol1JZpklduUVRWxdack+z6+Esyd+wns1MPjnTpXrc+smMwg3t2ZnDPMIb06sygHp2Ji+zofRdoOXOQ/tFBojogkJye/cn6/WPsTxhBVkEpWfllHDpeRrXV/l0b0L3TWUm9b0QH7z8YNkZr8l7Bo0leRGYAfwUCgX8aYx5v7Lma5H1fcXk1e46VkFn7930J+/PLOFPzQ62uW6f2xER0ICYilJiIUPo6pjGRoUSHhXjXFZkNsNkMJ8vPkF9SRfZ1C8gyIezvFsOBbjEciuhNdaC9eUUE+oaHkhDdifjoMEbFhDMqNpxwfxpZyU+bK32Nx5K8iAQC+4FpQB6wBZhrjNnT0PM1yfunaquNQ4Wn2ZdfyuET5eQWlZN7wv53rLgCW72PYHBgAH3CO9CjSwjhHYOJCA0mPDTI/rhjMOGh9mnX0CAiOgbTISjQ6VqwzWaorLFSccZKRbWV01VWCkoryS+pIr+kkoISx+PSSgpKqigorayrlQOIsdH3VD4Jx3MZUJRLwvFcEooOc1HOXu/71eIObaC50tt58orX0cABY8whRyCvAbOABpO88k9BgQEM7BHGwB5h56w7U2Pj6KkKDp/8IfEfPlFOQUkVmcdKOHn6DKcqqhts9gX7QSG4XQBBgUK7wACCAhzTQCEoMIB2gUK7gACCAwOwGkPFGSuV1fZkXlFtT+xVNU23HXcOaUd05xCiO4cwpn9H++Ow9kR3DqHP/OsYsHsLHWqqzn5RbCy0hQQP9oSuSd1ruTvJ9wYO15vPA8a4uUzlQ4LbBRDXrWOTF2BZbYbiimpOlp/h5OkznDh9hpPlZzhxuppTFWeorjFUW23U2GxUWw01VhvVNkN1jY0am31dtdVGcEAg4aHBdAgOpENQAB2CAgkJDqRDkOMvOJCQoEBCgwPpHhZCdOf2dA8Labo2vvA2SNsG9e/0HBpqr836Cq2J+zV3J/mGfkefVScTkTQgDSBG71ehGhAYIEQ4mmuI8nQ0P1KbDH01Sf64TT0nxz4PvrMPqknu7qeVB/StN98HOFr/CcaYdGNMsjEmOSrK277ByidYLPZeHgEB9qnF0rrlp6bae5LYbPapLyXHRYvOPmkK9vlFizwTj3I5d9fktwDxItIPOAJcD8xzc5mqLdGaqHNyc5u3XPkct9bkjTE1wK+AT4BMYIUx5lt3lqnaGK2JOqexJlJtOvUbbr+szhjzoTEmwRhzkTHGh85GKZ+gNVHnLFliP1Fcn6+dOFZN8pFrp5VqhNZEnZOaar9wKTbWfvVWbKxeyORnNMkrz3PmxKnWRJ3nyyeO1XlpkleeVXviNCfHfqOr2hOnF5rotSaqVJP0BmXKs/QGV0o5ranbGmhNXnmWnjhVyq00ySvP0hOnSrmVJnnlWXriVCm30iSvPEtPnCrlVu6+rYFS56e3qlXKbbQmr5RSfkyTvFJK+TFN8kop5cc0ySullB/TJK+UUn5Mk7xSSvkxTfJKKeXHnEryIjJHRL4VEZuIJP9o3QMickBE9onI5c6FqZRSqiWcvRhqN3AN8I/6C0VkCPbxXIcCvYA1IpJgjLE6WZ5SSqlmcKomb4zJNMbsa2DVLOA1Y0yVMeY74AAw2pmylFJKNZ+72uR7A4frzec5lp1DRNJEJENEMgoLC90UjlJKtU3nba4RkTVAjwZWLTLGvNvYyxpY1uDoJMaYdCAd7IOGnC8epZRSF+68Sd4Yc1kLtpsH9K033wc42oLtKKWUcoK7mmveA64XkfYi0g+IBza7qSyllFKNcLYL5dUikgdcAnwgIp8AGGO+BVYAe4CPgTu1Z41SSrU+p7pQGmNWAisbWbcE0OF9lFLKg/SKV6WU8mOa5JVSyo9pkldKKT+mSV6BxQJxcRAQYJ9aLJ6OSCnlIjqQd1tnsUBaGpSX2+dzcuzzoINrK+UHtCbf1i1a9EOCr1Vebl+ulPJ5muTbutzc5i1XSvkUTfJtXUxM85YrpXyKJvm2bskSCA09e1loqH25UsrnaZJv61JTIT0dYmNBxD5NT9eTrkr5Ce1do+wJXZO6Un5Ja/JKKeXHNMkrpZQf0ySvlFJ+TJO8Ukr5MU3ySinlx8QY7xk7W0QKgRwnNtENOO6icDzJX/YDdF+8kb/sB+i+1Io1xkQ1tMKrkryzRCTDGJPs6Tic5S/7Abov3shf9gN0Xy6ENtcopZQf0ySvlFJ+zN+SfLqnA3ARf9kP0H3xRv6yH6D7cl5+1SavlFLqbP5Wk1dKKVWPJnmllPJjfpXkReRREdkpIjtEZJWI9PJ0TC0lIk+IyF7H/qwUka6ejqmlRGSOiHwrIjYR8bnubiIyQ0T2icgBEbnf0/G0lIi8ICIFIrLb07E4S0T6isg6Ecl0fLZ+7emYWkJEQkRks4h849iPxS4vw5/a5EWkszGmxPH4LmCIMeY2D4fVIiIyHfjUGFMjIv8HYIy5z8NhtYiIDAZswD+AhcaYDA+HdMFEJBDYD0wD8oAtwFxjzB6PBtYCIjIRKANeMsYkejoeZ4hIT6CnMWabiIQBW4HZvvZ/EREBOhpjykQkCNgA/NoYs8lVZfhVTb42wTt0BHz2CGaMWWWMqXHMbgL6eDIeZxhjMo0x+zwdRwuNBg4YYw4ZY84ArwGzPBxTixhjPgdOeDoOVzDGHDPGbHM8LgUygd6ejar5jF2ZYzbI8efSvOVXSR5ARJaIyGEgFXjQ0/G4yE3AR54Ooo3qDRyuN5+HDyYTfyYiccAI4GsPh9IiIhIoIjuAAmC1Mcal++FzSV5E1ojI7gb+ZgEYYxYZY/oCFuBXno22aefbF8dzFgE12PfHa13IvvgoaWCZz/5C9Dci0gl4C/jNj37J+wxjjNUYMxz7r/XRIuLSpjSfG/7PGHPZBT71FeAD4CE3huOU8+2LiCwAZgKXGi8/edKM/4uvyQP61pvvAxz1UCyqHkcb9luAxRjztqfjcZYx5pSIrAdmAC47Oe5zNfmmiEh8vdmrgL2eisVZIjIDuA+4yhhT7ul42rAtQLyI9BORYOB64D0Px9TmOU5YPg9kGmP+7Ol4WkpEomp7zolIB+AyXJy3/K13zVvAQOw9OXKA24wxRzwbVcuIyAGgPVDkWLTJh3sKXQ38DYgCTgE7jDGXezSoZhCRnwJLgUDgBWPMEs9G1DIi8iowGfstbfOBh4wxz3s0qBYSkfHAF8Au7N93gN8bYz70XFTNJyLDgOXYP1sBwApjzCMuLcOfkrxSSqmz+VVzjVJKqbNpkldKKT+mSV4ppfyYJnmllPJjmuSVUsqPaZJXSik/pkleKaX82P8Hv6QEveVo2RgAAAAASUVORK5CYII=\n", 660 | "text/plain": [ 661 | "
" 662 | ] 663 | }, 664 | "metadata": { 665 | "needs_background": "light" 666 | }, 667 | "output_type": "display_data" 668 | } 669 | ], 670 | "source": [ 671 | "from sklearn.linear_model import Ridge\n", 672 | "\n", 673 | "model = Ridge().fit(X_scaled[:, 1:], y) # Ridge() by default fits an intercept\n", 674 | "y_pred_sklearn = model.predict(Xgrid_scaled[:,1:])\n", 675 | "\n", 676 | "print(model.coef_)\n", 677 | "print(model.intercept_)\n", 678 | "\n", 679 | "pyplot.scatter(x, y, c='r', label='true')\n", 680 | "pyplot.plot(xgrid, y_pred_sklearn, label='sklearn ridge regression')\n", 681 | "pyplot.legend();\n" 682 | ] 683 | }, 684 | { 685 | "cell_type": "markdown", 686 | "metadata": {}, 687 | "source": [ 688 | "The `model` variable we created above using `Ridge()` stores the model weights and the intercept in the `.coef_` and `intercept_` attributes. \n", 689 | "Compare the values with those we obtained above using the regularization and 3rd-order polynomial. Pretty close!" 690 | ] 691 | }, 692 | { 693 | "cell_type": "markdown", 694 | "metadata": {}, 695 | "source": [ 696 | "## What we've learned\n", 697 | "\n", 698 | "- Polynomial regression is a special case of multiple linear regression.\n", 699 | "- Pick too low-order a polynomial (e.g., a line) and the model could _underfit_ the data, giving large residuals (training error).\n", 700 | "- Pick too high a polynomial order, and you will get _overfitting_, as the model tries to fit the noise in the data resulting in wiggles.\n", 701 | "- Regularization is used to reduce or avoid overfitting.\n", 702 | "- Adding the sum-of-squares of the weights to the loss function _penalizes_ large weights and controls overfitting. \n", 703 | "- Linear regression with this so-called Tikhonov regularization is called _ridge regression_.\n", 704 | "- With **scikit-learn**, you can use `Ridge()` to fit a linear model with regularization. " 705 | ] 706 | }, 707 | { 708 | "cell_type": "code", 709 | "execution_count": 16, 710 | "metadata": {}, 711 | "outputs": [ 712 | { 713 | "data": { 714 | "text/html": [ 715 | "\n", 716 | "\n", 717 | "\n", 718 | "\n", 848 | "\n" 864 | ], 865 | "text/plain": [ 866 | "" 867 | ] 868 | }, 869 | "execution_count": 16, 870 | "metadata": {}, 871 | "output_type": "execute_result" 872 | } 873 | ], 874 | "source": [ 875 | "# Execute this cell to load the notebook's style sheet, then ignore it\n", 876 | "from IPython.core.display import HTML\n", 877 | "css_file = '../style/custom.css'\n", 878 | "HTML(open(css_file, \"r\").read())" 879 | ] 880 | } 881 | ], 882 | "metadata": { 883 | "kernelspec": { 884 | "display_name": "Python 3", 885 | "language": "python", 886 | "name": "python3" 887 | }, 888 | "language_info": { 889 | "codemirror_mode": { 890 | "name": "ipython", 891 | "version": 3 892 | }, 893 | "file_extension": ".py", 894 | "mimetype": "text/x-python", 895 | "name": "python", 896 | "nbconvert_exporter": "python", 897 | "pygments_lexer": "ipython3", 898 | "version": "3.8.5" 899 | } 900 | }, 901 | "nbformat": 4, 902 | "nbformat_minor": 5 903 | } 904 | -------------------------------------------------------------------------------- /scripts/plot_helpers.py: -------------------------------------------------------------------------------- 1 | import numpy as _np 2 | from sklearn.pipeline import make_pipeline 3 | from sklearn.linear_model import LinearRegression, Ridge 4 | from sklearn.preprocessing import PolynomialFeatures, MinMaxScaler 5 | from sklearn.metrics import mean_absolute_error 6 | from matplotlib import pyplot as plt 7 | from ipywidgets import interact 8 | 9 | def interact_polyreg(max_degree, x, y, regularized=False, verbose=True): 10 | """ 11 | The function to plot polynomial linear regression. 12 | 13 | Args: 14 | max_degree: int 15 | Max polynomial degree. 16 | x,y: numpy.ndarray 17 | 1D Training data. 18 | regularized: bool 19 | Whether to add l2-norm regularization term in loss. Default to False. 20 | verbose: bool 21 | Whether to print trained weights. Default to True. 22 | """ 23 | x_plot = _np.linspace(x.min(), x.max(), 30).reshape(-1,1) 24 | 25 | def polyreg_helper(degree): 26 | plt.figure(figsize=(10,6)) 27 | plt.scatter(x, y, c='r', label='true') 28 | linear = make_pipeline(PolynomialFeatures(degree, include_bias=False), 29 | MinMaxScaler(), 30 | LinearRegression(fit_intercept=True)) 31 | linear.fit(x.reshape(-1,1), y) 32 | mae_linear = mean_absolute_error(y, linear.predict(x.reshape(-1,1))) 33 | 34 | if regularized: 35 | ridge = make_pipeline(PolynomialFeatures(degree, include_bias=False), 36 | MinMaxScaler(), 37 | Ridge(alpha=1.0)) 38 | ridge.fit(x.reshape(-1,1), y) 39 | mae_ridge = mean_absolute_error(y, ridge.predict(x.reshape(-1,1))) 40 | plt.plot(x_plot, linear.predict(x_plot), label='predicted, w/o regularization') 41 | plt.plot(x_plot, ridge.predict(x_plot), label='predicted, with regularization') 42 | plt.title(f"Poly degree = {degree:2}, MAE_no_reg = {mae_linear:.3f}, MAE_reg = {mae_ridge:.3f}", fontsize=16) 43 | if verbose: 44 | print('weights without regularization') 45 | print(linear.named_steps['linearregression'].coef_) 46 | print('weights with regularization') 47 | print(ridge.named_steps['ridge'].coef_) 48 | else: 49 | plt.plot(x_plot, linear.predict(x_plot), label='predicted') 50 | plt.title(f"Polynomial degree = {degree:2}, MAE = {mae_linear:.3f}", fontsize=16) 51 | plt.legend(fontsize=16) 52 | plt.show() 53 | 54 | interact(polyreg_helper, degree=(1, max_degree)) 55 | -------------------------------------------------------------------------------- /style/custom.css: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 134 | 150 | --------------------------------------------------------------------------------