├── .gitignore ├── README.md ├── excel └── britlit.xlsx └── nbs ├── 0. Course Logistics.ipynb ├── 1. Why are we here.ipynb ├── 2. Topic Modeling with NMF and SVD.ipynb ├── 3. Background Removal with Robust PCA.ipynb ├── 4. Compressed Sensing of CT Scans with Robust Regression.ipynb ├── 5. Health Outcomes with Linear Regression.ipynb ├── 6. How to Implement Linear Regression.ipynb ├── 7. PageRank with Eigen Decompositions.ipynb ├── 8. Implementing QR Factorization.ipynb ├── Homework 1.ipynb ├── Homework 2.ipynb ├── Homework 3.ipynb ├── Project_ideas.txt ├── convolution-intro.ipynb ├── graddesc.xlsm ├── gradient-descent-intro.ipynb └── images ├── AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9.url ├── Finite_element_sparse_matrix.png ├── Halide1.gif ├── Halide2.gif ├── Halide3.gif ├── InsideImagenet.png ├── L1vsL2.jpg ├── L1vsL2_2.png ├── adjaceny_matrix.jpg ├── adjaceny_matrix.png ├── arnoldi.jpg ├── background.jpg ├── block_matrix.png ├── bloom_filter.png ├── candes.png ├── cholesky_factorization_speed.png ├── cincinnati_reds.png ├── cincinnati_reds2.png ├── cnn1.png ├── cnn2.png ├── cnn3.png ├── ct_1.png ├── cython_vs_numba.png ├── data.png ├── data_xray.png ├── digit.gif ├── document_term.png ├── face_nmf.png ├── face_outputs.png ├── face_pca.png ├── faces_rpca.png ├── faces_rpca_gross.png ├── faces_rpca_gross2.png ├── figures.jpg ├── fltscale-wh.png ├── fltscale.svg ├── fpa.png ├── full_qr.JPG ├── full_svd.JPG ├── graph.png ├── grid.jpg ├── hermitian_eigen.JPG ├── image1.bmp ├── image1.jpg ├── lesson4.png ├── markov_health.jpg ├── matrix_operations.png ├── nimfa.png ├── nmf_doc.png ├── nonhermitian_eigen.JPG ├── page_rank_graph.png ├── pcp_algorithm.png ├── pentium_nytimes.png ├── prob1.png ├── prob2.png ├── proj.png ├── projection.png ├── projection_line.png ├── randomizedSVDbenchmarks.png ├── reduced_qr.JPG ├── reduced_svd.JPG ├── rpca_inexact.png ├── shop.png ├── skinny_image.jpg ├── sklearn_ct.png ├── sparse.png ├── sportspredict.jpeg ├── surveillance3.png ├── svd_fb.png ├── svddiagram.gif ├── svp_value.png ├── top10.png ├── what_is_pytorch.png ├── whole_image.jpg ├── why_python.png ├── word2vec_analogies.png ├── word2vec_sexist.png ├── xray.png └── zoom.bmp /.gitignore: -------------------------------------------------------------------------------- 1 | nbs/.ipynb_checkpoints 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Computational Linear Algebra for Coders 2 | 3 | This course is focused on the question: **How do we do matrix computations with acceptable speed and acceptable accuracy?** 4 | 5 | This course was taught in the [University of San Francisco's Masters of Science in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) program, summer 2017 (for graduate students studying to become data scientists). The course is taught in Python with Jupyter Notebooks, using libraries such as Scikit-Learn and Numpy for most lessons, as well as Numba (a library that compiles Python to C for faster performance) and PyTorch (an alternative to Numpy for the GPU) in a few lessons. 6 | 7 | Accompanying the notebooks is a [playlist of lecture videos, available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY). If you are ever confused by a lecture or it goes too quickly, check out the beginning of the next video, where I review concepts from the previous lecture, often explaining things from a new perspective or with different illustrations, and answer questions. 8 | 9 | ## Getting Help 10 | You can ask questions or share your thoughts and resources using the [**Computational Linear Algebra** category on our fast.ai discussion forums](http://forums.fast.ai/c/lin-alg). 11 | 12 | ## Table of Contents 13 | The following listing links to the notebooks in this repository, rendered through the [nbviewer](http://nbviewer.jupyter.org) service. Topics Covered: 14 | ### [0. Course Logistics](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb) ([Video 1](https://www.youtube.com/watch?v=8iGzBMboA0I&index=1&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 15 | - [My background](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Intro) 16 | - [Teaching Approach](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Teaching) 17 | - [Importance of Technical Writing](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Writing-Assignment) 18 | - [List of Excellent Technical Blogs](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Excellent-Technical-Blogs) 19 | - [Linear Algebra Review Resources](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Linear-Algebra) 20 | 21 | 22 | ### [1. Why are we here?](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb) ([Video 1](https://www.youtube.com/watch?v=8iGzBMboA0I&index=1&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 23 | We start with a high level overview of some foundational concepts in numerical linear algebra. 24 | - [Matrix and Tensor Products](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Matrix-and-Tensor-Products) 25 | - [Matrix Decompositions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Matrix-Decompositions) 26 | - [Accuracy](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Accuracy) 27 | - [Memory use](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Memory-Use) 28 | - [Speed](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Speed) 29 | - [Parallelization & Vectorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Scalability-/-parallelization) 30 | 31 | ### [2. Topic Modeling with NMF and SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb) ([Video 2](https://www.youtube.com/watch?v=kgd40iDT8yY&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=2) and [Video 3](https://www.youtube.com/watch?v=C8KEtrWjjyo&index=3&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 32 | We will use the newsgroups dataset to try to identify the topics of different posts. We use a term-document matrix that represents the frequency of the vocabulary in the documents. We factor it using NMF, and then with SVD. 33 | - [Topic Frequency-Inverse Document Frequency (TF-IDF)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#TF-IDF) 34 | - [Singular Value Decomposition (SVD)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Singular-Value-Decomposition-(SVD)) 35 | - [Non-negative Matrix Factorization (NMF)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Non-negative-Matrix-Factorization-(NMF)) 36 | - [Stochastic Gradient Descent (SGD)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Gradient-Descent) 37 | - [Intro to PyTorch](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#PyTorch) 38 | - [Truncated SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Truncated-SVD) 39 | 40 | ### [3. Background Removal with Robust PCA](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb) ([Video 3](https://www.youtube.com/watch?v=C8KEtrWjjyo&index=3&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY), [Video 4](https://www.youtube.com/watch?v=Ys8R2nUTOAk&index=4&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY), and [Video 5](https://www.youtube.com/watch?v=O2x5KPJr5ag&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=5)) 41 | Another application of SVD is to identify the people and remove the background of a surveillance video. We will cover robust PCA, which uses randomized SVD. And Randomized SVD uses the LU factorization. 42 | - [Load and View Video Data](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Load-and-view-the-data) 43 | - [SVD](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#SVD) 44 | - [Principal Component Analysis (PCA)](https://github.com/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb) 45 | - [L1 Norm Induces Sparsity](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#L1-norm-induces-sparsity) 46 | - [Robust PCA](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Robust-PCA-(via-Primary-Component-Pursuit)) 47 | - [LU factorization](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#LU-Factorization) 48 | - [Stability of LU](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Stability) 49 | - [LU factorization with Pivoting](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#LU-factorization-with-Partial-Pivoting) 50 | - [History of Gaussian Elimination](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#History-of-Gaussian-Elimination) 51 | - [Block Matrix Multiplication](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Block-Matrices) 52 | 53 | ### [4. Compressed Sensing with Robust Regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#4.-Compressed-Sensing-of-CT-Scans-with-Robust-Regression) ([Video 6](https://www.youtube.com/watch?v=YY9_EYNj5TY&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=6) and [Video 7](https://www.youtube.com/watch?v=ZUGkvIM6ehM&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=7)) 54 | Compressed sensing is critical to allowing CT scans with lower radiation-- the image can be reconstructed with less data. Here we will learn the technique and apply it to CT images. 55 | - [Broadcasting](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Broadcasting) 56 | - [Sparse matrices](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Sparse-Matrices-(in-Scipy)) 57 | - [CT Scans and Compressed Sensing](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Sparse-Matrices-(in-Scipy)) 58 | - [L1 and L2 regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Regresssion) 59 | 60 | ### [5. Predicting Health Outcomes with Linear Regressions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb) ([Video 8](https://www.youtube.com/watch?v=SjX55V8zDXI&index=8&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 61 | - [Linear regression in sklearn](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Linear-regression-in-Scikit-Learn) 62 | - [Polynomial Features](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Polynomial-Features) 63 | - [Speeding up with Numba](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Speeding-up-feature-generation) 64 | - [Regularization and Noise](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Regularization-and-noise) 65 | 66 | ### [6. How to Implement Linear Regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb)([Video 8](https://www.youtube.com/watch?v=SjX55V8zDXI&index=8&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 67 | - [How did Scikit Learn do it?](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#How-did-sklearn-do-it?) 68 | - [Naive solution](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Naive-Solution) 69 | - [Normal equations and Cholesky factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Normal-Equations-(Cholesky)) 70 | - [QR factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#QR-Factorization) 71 | - [SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#SVD) 72 | - [Timing Comparison](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Timing-Comparison) 73 | - [Conditioning & Stability](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Conditioning-&-stability) 74 | - [Full vs Reduced Factorizations](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Full-vs-Reduced-Factorizations) 75 | - [Matrix Inversion is Unstable](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Matrix-Inversion-is-Unstable) 76 | 77 | ### [7. PageRank with Eigen Decompositions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb) ([Video 9](https://www.youtube.com/watch?v=AbB-w77yxD0&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=9) and [Video 10](https://www.youtube.com/watch?v=1kw8bpA9QmQ&index=10&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 78 | We have applied SVD to topic modeling, background removal, and linear regression. SVD is intimately connected to the eigen decomposition, so we will now learn how to calculate eigenvalues for a large matrix. We will use DBpedia data, a large dataset of Wikipedia links, because here the principal eigenvector gives the relative importance of different Wikipedia pages (this is the basic idea of Google's PageRank algorithm). We will look at 3 different methods for calculating eigenvectors, of increasing complexity (and increasing usefulness!). 79 | - [SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Motivation) 80 | - [DBpedia Dataset](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#DBpedia) 81 | - [Power Method](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Power-method) 82 | - [QR Algorithm](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#QR-Algorithm) 83 | - [Two-phase approach to finding eigenvalues](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#A-Two-Phase-Approach) 84 | - [Arnoldi Iteration](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Arnoldi-Iteration) 85 | 86 | ### [8. Implementing QR Factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb) ([Video 10](https://www.youtube.com/watch?v=1kw8bpA9QmQ&index=10&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY)) 87 | - [Gram-Schmidt](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Gram-Schmidt) 88 | - [Householder](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Householder) 89 | - [Stability Examples](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Ex-9.2:-Classical-vs-Modified-Gram-Schmidt) 90 | 91 |
92 | 93 | ## Why is this course taught in such a weird order? 94 | 95 | This course is structured with a *top-down* teaching method, which is different from how most math courses operate. Typically, in a *bottom-up* approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures. The problems with this are that students often lose motivation, don't have a sense of the "big picture", and don't know what they'll need. 96 | 97 | Harvard Professor David Perkins has a book, [Making Learning Whole](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719) in which he uses baseball as an analogy. We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game. Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on. 98 | 99 | If you took the fast.ai deep learning course, that is what we used. You can hear more about my teaching philosophy [in this blog post](http://www.fast.ai/2016/10/08/teaching-philosophy/) or [this talk I gave at the San Francisco Machine Learning meetup](https://vimeo.com/214233053). 100 | 101 | All that to say, don't worry if you don't understand everything at first! You're not supposed to. We will start using some "black boxes" or matrix decompositions that haven't yet been explained, and then we'll dig into the lower level details later. 102 | 103 | To start, focus on what things DO, not what they ARE. 104 | -------------------------------------------------------------------------------- /excel/britlit.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/excel/britlit.xlsx -------------------------------------------------------------------------------- /nbs/0. Course Logistics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n", 8 | "\n", 9 | "You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# 0. Course Logistics" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Ask Questions" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "Let me know how things are going. This is particularly important since I'm new to MSAN, I don't know everything you've seen/haven't seen." 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "## Intro" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "**My background and linear algebra love**:\n", 45 | "\n", 46 | "- **Swarthmore College**: linear algebra convinced me to be a math major! (minors in CS & linguistics) I thought linear algebra was beautiful, but theoretical\n", 47 | "- **Duke University**: Math PhD. Took numerical linear algebra. Enjoyed the course, but not my focus\n", 48 | "- **Research Triangle Institute**: first time using linear algebra in practice (healthcare economics, markov chains)\n", 49 | "- **Quant**: first time working with lots of data, decided to become a data scientist\n", 50 | "- **Uber**: data scientist\n", 51 | "- **Hackbright**: taught software engineering. Overhauled ML and collaborative filtering lectures\n", 52 | "- **fast.ai**: co-founded to make deep learning more accessible. Deep Learning involves a TON of linear algebra" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "## Teaching" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "**Teaching Approach**\n", 67 | "\n", 68 | "I'll be using a *top-down* teaching method, which is different from how most math courses operate. Typically, in a *bottom-up* approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures. The problems with this are that students often lose motivation, don't have a sense of the \"big picture\", and don't know what they'll need.\n", 69 | "\n", 70 | "If you took the fast.ai deep learning course, that is what we used. You can hear more about my teaching philosophy [in this blog post](http://www.fast.ai/2016/10/08/teaching-philosophy/) or [in this talk](https://vimeo.com/214233053).\n", 71 | "\n", 72 | "Harvard Professor David Perkins has a book, [Making Learning Whole](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719) in which he uses baseball as an analogy. We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game. Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on.\n", 73 | "\n", 74 | "All that to say, don't worry if you don't understand everything at first! You're not supposed to. We will start using some \"black boxes\" or matrix decompositions that haven't yet been explained, and then we'll dig into the lower level details later.\n", 75 | "\n", 76 | "To start, focus on what things DO, not what they ARE." 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "People learn by:\n", 84 | "1. **doing** (coding and building)\n", 85 | "2. **explaining** what they've learned (by writing or helping others)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "**Text Book**\n", 93 | "\n", 94 | "The book [**Numerical Linear Algebra**](https://www.amazon.com/Numerical-Linear-Algebra-Lloyd-Trefethen/dp/0898713617) by Trefethen and Bau is recommended. The MSAN program has a few copies on hand.\n", 95 | "\n", 96 | "A secondary book is [**Numerical Methods**](https://www.amazon.com/Numerical-Methods-Analysis-Implementation-Algorithms/dp/0691151229) by Greenbaum and Chartier." 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "## Basics" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "**Office hours**: 2:00-4:00 on Friday afternoons. Email me if you need to meet at other times.\n", 111 | "\n", 112 | "My contact info: **rachel@fast.ai**\n", 113 | "\n", 114 | "Class Slack: #numerical_lin_alg\n", 115 | "\n", 116 | "Email me if you will need to miss class.\n", 117 | "\n", 118 | "Jupyter Notebooks will be available on Github at: https://github.com/fastai/numerical-linear-algebra Please pull/download before class. **Some parts are removed for you to fill in as you follow along in class**. Be sure to let me know **THIS WEEK** if you are having any problems running the notebooks from your own computer. You may want to make a separate copy, because running Jupyter notebooks causes them to change, which can create github conflicts the next time you pull." 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "Check that you have MathJax running (which renders LaTeX, used for math equations) by running the following cell:" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "$$ e^{\\theta i} = \\cos(\\theta) + i \\sin(\\theta)$$ " 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "check that you can import:" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 2, 145 | "metadata": { 146 | "collapsed": true 147 | }, 148 | "outputs": [], 149 | "source": [ 150 | "import numpy as np\n", 151 | "import sklearn" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "**Grading Rubric**:\n", 159 | "\n", 160 | "| Assignment | Percent |\n", 161 | "|-------------------|:-------:|\n", 162 | "| Attendance | 10% |\n", 163 | "| Homework | 20% |\n", 164 | "| Writing: proposal | 10% |\n", 165 | "| Writing: draft | 15% |\n", 166 | "| Writing: final | 15% |\n", 167 | "| Final Exam | 30% |" 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "**Honor Code** \n", 175 | "\n", 176 | "No cheating nor plagiarism is allowed, please see below for more details." 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "**On Laptops**\n", 184 | "\n", 185 | "I ask you to be respectful of me and your classmates and to refrain from surfing the web or using social media (facebook, twitter, etc) or messaging programs during class. It is absolutely forbidden to use instant messaging programs, email, etc. during class lectures or quizzes." 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "## Syllabus" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "Topics Covered:\n", 200 | "\n", 201 | "1\\. Why are we here?\n", 202 | " - Matrix and Tensor Products\n", 203 | " - Matrix Decompositions\n", 204 | " - Accuracy\n", 205 | " - Memory use\n", 206 | " - Speed\n", 207 | " - Parallelization & Vectorization\n", 208 | "\n", 209 | "\n", 210 | "2\\. Topic Modeling with NMF and SVD\n", 211 | " - Topic Frequency-Inverse Document Frequency (TF-IDF)\n", 212 | " - Singular Value Decomposition (SVD)\n", 213 | " - Non-negative Matrix Factorization (NMF)\n", 214 | " - Stochastic Gradient Descent (SGD)\n", 215 | " - Intro to PyTorch\n", 216 | " - Truncated SVD, Randomized SVD\n", 217 | "\n", 218 | "\n", 219 | "3\\. Background Removal with Robust PCA\n", 220 | " - Robust PCA\n", 221 | " - Randomized SVD\n", 222 | " - LU factorization\n", 223 | "\n", 224 | "\n", 225 | "4\\. Compressed Sensing for CT scans with Robust Regression\n", 226 | " - L1 regularization\n", 227 | "\n", 228 | "\n", 229 | "5\\. Predicting Health Outcomes with Linear Regression\n", 230 | " - Linear regression\n", 231 | " - Polynomial Features\n", 232 | " - Speeding up with Numba\n", 233 | " - Regularization and Noise\n", 234 | " - Implementing linear regression 4 ways\n", 235 | "\n", 236 | "\n", 237 | "6\\. PageRank with Eigen Decompositions\n", 238 | " - Power Method\n", 239 | " - QR Algorithm\n", 240 | " - Arnoldi Iteration\n", 241 | "\n", 242 | "\n", 243 | "7\\. QR Factorization\n", 244 | " - Gram-Schmidt\n", 245 | " - Householder\n", 246 | " - Stability" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "metadata": {}, 252 | "source": [ 253 | "## Writing Assignment" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": {}, 259 | "source": [ 260 | "**Writing Assignment:** Writing about technical concepts is a hugely valuable skill. I want you to write a technical blog post related to numerical linear algebra. [A blog is like a resume, only better](http://www.fast.ai/2017/04/06/alternatives/). Technical writing is also important in creating documentation, sharing your work with co-workers, applying to speak at conferences, and practicing for interviews. (You don't actually have to publish it, although I hope you do, and please send me the link if you do.)\n", 261 | "- [List of ideas here](Project_ideas.txt)\n", 262 | "- Always cite sources, use quote marks around quotes. Do this even as you are first gathering sources and taking notes. If you plagiarize parts of someone else's work, you will fail.\n", 263 | "- Can be done in a Jupyter Notebook (Jupyter Notebooks can be turned into blog posts) or a [Kaggle Kernel](https://www.kaggle.com/xenocide/content-based-anime-recommender)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "For the proposal, write a brief paragraph about the problem/topic/experiment you plan to research/test and write about. You need to include **4 sources** that you plan to use: these can include Trefethen, other blog posts, papers, or books. Include a sentence about each source, stating what it's in it.\n", 271 | "\n", 272 | "Feel free to ask me if you are wondering if your topic idea is suitable!" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "### Excellent Technical Blogs\n", 280 | "\n", 281 | "Examples of great technical blog posts:\n", 282 | "- [Peter Norvig](http://nbviewer.jupyter.org/url/norvig.com/ipython/ProbabilityParadox.ipynb) (more [here](http://norvig.com/ipython/))\n", 283 | "- [Stephen Merity](https://smerity.com/articles/2017/deepcoder_and_ai_hype.html)\n", 284 | "- [Julia Evans](https://codewords.recurse.com/issues/five/why-do-neural-networks-think-a-panda-is-a-vulture) (more [here](https://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-tcp-stack-in-python/))\n", 285 | "- [Julia Ferraioli](http://blog.juliaferraioli.com/2016/02/exploring-world-using-vision-twilio.html)\n", 286 | "- [Edwin Chen](http://blog.echen.me/2014/10/07/moving-beyond-ctr-better-recommendations-through-human-evaluation/)\n", 287 | "- [Slav Ivanov](https://blog.slavv.com/picking-an-optimizer-for-style-transfer-86e7b8cba84b)\n", 288 | "- [Brad Kenstler](https://hackernoon.com/non-artistic-style-transfer-or-how-to-draw-kanye-using-captain-picards-face-c4a50256b814)\n", 289 | "- find [more on twitter](https://twitter.com/math_rachel)" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "## Deadlines" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "| Assignment | Dates |\n", 304 | "|-------------------|:--------:|\n", 305 | "| Homeworks | TBA |\n", 306 | "| Writing: proposal | 5/30 |\n", 307 | "| Writing: draft | 6/15 |\n", 308 | "| Writing: final | 6/27 |\n", 309 | "| Final Exam | 6/29 |" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "## Linear Algebra" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": {}, 322 | "source": [ 323 | "We will review some linear algebra in class. However, if you find there are concepts you feel rusty on, you may want to review on your own. Here are some resources:\n", 324 | "\n", 325 | "- [3Blue1Brown Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab) videos about *geometric intuition* (fantastic! gorgeous!)\n", 326 | "- Lectures 1-6 of Trefethen\n", 327 | "- [Immersive linear algebra](http://immersivemath.com/ila/) free online textbook with interactive graphics\n", 328 | "- [Chapter 2](http://www.deeplearningbook.org/contents/linear_algebra.html) of Ian Goodfellow's Deep Learning Book\n" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "## USF Policies" 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "**Academic Integrity** \n", 343 | "\n", 344 | "USF upholds the standards of honesty and integrity from all members of the academic community. All students are expected to know and adhere to the University’s Honor Code. You can find the full text of the [code online](www.usfca.edu/academic_integrity). The policy covers:\n", 345 | "- Plagiarism: intentionally or unintentionally representing the words or ideas of another person as your own; failure to properly cite references; manufacturing references.\n", 346 | "- Working with another person when independent work is required.\n", 347 | "- Submission of the same paper in more than one course without the specific permission of each instructor.\n", 348 | "- Submitting a paper written (entirely or even a small part) by another person or obtained from the internet.\n", 349 | "- Plagiarism is plagiarism: it does not matter if the source being copied is on the Internet, from a book or textbook, or from quizzes or problem sets written up by other students.\n", 350 | "- The penalties for violation of the policy may include a failing grade on the assignment, a failing grade in the course, and/or a referral to the Academic Integrity Committee." 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "**Students with Disabilities**\n", 358 | "\n", 359 | "If you are a student with a disability or disabling condition, or if you think you may have a disability, please contact USF Student Disability Services (SDS) at 415 422-2613 within the first week of class, or immediately upon onset of disability, to speak with a disability specialist. If you are determined eligible for reasonable accommodations, please meet with your disability specialist so they can arrange to have your accommodation letter sent to me, and we will discuss your needs for this course. For more information, please visit [this website]( http://www.usfca.edu/sds) or call (415) 422-2613." 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "**Behavioral Expectations**\n", 367 | "\n", 368 | "All students are expected to behave in accordance with the [Student Conduct Code and other University policies](https://myusf.usfca.edu/fogcutter). Open discussion and disagreement is encouraged when done respectfully and in the spirit of academic discourse. There are also a variety of behaviors that, while not against a specific University policy, may create disruption in this course. Students whose behavior is disruptive or who fail to comply with the instructor may be dismissed from the class for the remainder of the class period and may need to meet with the instructor or Dean prior to returning to the next class period. If necessary, referrals may also be made to the Student Conduct process for violations of the Student Conduct Code." 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": {}, 374 | "source": [ 375 | "**Counseling and Psychological Services**\n", 376 | "\n", 377 | "Our diverse staff offers brief individual, couple, and group counseling to student members of our community. CAPS services are confidential and free of charge. Call 415-422-6352 for an initial consultation appointment. Having a crisis at 3 AM? We are still here for you. Telephone consultation through CAPS After Hours is available between the hours of 5:00 PM to 8:30 AM; call the above number and press 2." 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "**Confidentiality, Mandatory Reporting, and Sexual Assault**\n", 385 | "\n", 386 | "As an instructor, one of my responsibilities is to help create a safe learning environment on our campus. I also have a mandatory reporting responsibility related to my role as a faculty member. I am required to share information regarding sexual misconduct or information about a crime that may have occurred on USFs campus with the University. Here are other resources:\n", 387 | "\n", 388 | "- To report any sexual misconduct, students may visit Anna Bartkowski (UC 5th floor) or see many other options by visiting [this website](https://myusf.usfca.edu/title-IX)\n", 389 | "- Students may speak to someone confidentially, or report a sexual assault confidentially by contacting Counseling and Psychological Services at 415-422-6352\n", 390 | "- To find out more about reporting a sexual assault at USF, visit [USF’s Callisto website](https://usfca.callistocampus.org/)\n", 391 | "- For an off-campus resource, contact [San Francisco Women Against Rape](http://www.sfwar.org/about.html) 415-647-7273" 392 | ] 393 | } 394 | ], 395 | "metadata": { 396 | "kernelspec": { 397 | "display_name": "Python 3", 398 | "language": "python", 399 | "name": "python3" 400 | }, 401 | "language_info": { 402 | "codemirror_mode": { 403 | "name": "ipython", 404 | "version": 3 405 | }, 406 | "file_extension": ".py", 407 | "mimetype": "text/x-python", 408 | "name": "python", 409 | "nbconvert_exporter": "python", 410 | "pygments_lexer": "ipython3", 411 | "version": "3.6.1" 412 | } 413 | }, 414 | "nbformat": 4, 415 | "nbformat_minor": 2 416 | } 417 | -------------------------------------------------------------------------------- /nbs/5. Health Outcomes with Linear Regression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n", 8 | "\n", 9 | "You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# 5. Health Outcomes with Linear Regression" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 26, 22 | "metadata": { 23 | "collapsed": true 24 | }, 25 | "outputs": [], 26 | "source": [ 27 | "from sklearn import datasets, linear_model, metrics\n", 28 | "from sklearn.model_selection import train_test_split\n", 29 | "from sklearn.preprocessing import PolynomialFeatures\n", 30 | "import math, scipy, numpy as np\n", 31 | "from scipy import linalg" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "## Diabetes Dataset" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "We will use a dataset from patients with diabates. The data consists of 442 samples and 10 variables (all are physiological characteristics), so it is tall and skinny. The dependent variable is a quantitative measure of disease progression one year after baseline." 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "This is a classic dataset, famously used by Efron, Hastie, Johnstone, and Tibshirani in their [Least Angle Regression](https://arxiv.org/pdf/math/0406456.pdf) paper, and one of the [many datasets included with scikit-learn](http://scikit-learn.org/stable/datasets/)." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 27, 58 | "metadata": { 59 | "collapsed": true 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "data = datasets.load_diabetes()" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": 28, 69 | "metadata": { 70 | "collapsed": true 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "feature_names=['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 29, 80 | "metadata": { 81 | "collapsed": true 82 | }, 83 | "outputs": [], 84 | "source": [ 85 | "trn,test,y_trn,y_test = train_test_split(data.data, data.target, test_size=0.2)" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 30, 91 | "metadata": {}, 92 | "outputs": [ 93 | { 94 | "data": { 95 | "text/plain": [ 96 | "((353, 10), (89, 10))" 97 | ] 98 | }, 99 | "execution_count": 30, 100 | "metadata": {}, 101 | "output_type": "execute_result" 102 | } 103 | ], 104 | "source": [ 105 | "trn.shape, test.shape" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "metadata": {}, 111 | "source": [ 112 | "## Linear regression in Scikit Learn" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "Consider a system $X\\beta = y$, where $X$ has more rows than columns. This occurs when you have more data samples than variables. We want to find $\\hat{\\beta}$ that minimizes: \n", 120 | "$$ \\big\\vert\\big\\vert X\\beta - y \\big\\vert\\big\\vert_2$$" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": {}, 126 | "source": [ 127 | "Let's start by using the sklearn implementation:" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": 173, 133 | "metadata": {}, 134 | "outputs": [ 135 | { 136 | "name": "stdout", 137 | "output_type": "stream", 138 | "text": [ 139 | "458 µs ± 62.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n" 140 | ] 141 | } 142 | ], 143 | "source": [ 144 | "regr = linear_model.LinearRegression()\n", 145 | "%timeit regr.fit(trn, y_trn)" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 7, 151 | "metadata": { 152 | "collapsed": true 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "pred = regr.predict(test)" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "It will be helpful to have some metrics on how good our prediciton is. We will look at the mean squared norm (L2) and mean absolute error (L1)." 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": 128, 169 | "metadata": { 170 | "collapsed": true, 171 | "scrolled": true 172 | }, 173 | "outputs": [], 174 | "source": [ 175 | "def regr_metrics(act, pred):\n", 176 | " return (math.sqrt(metrics.mean_squared_error(act, pred)), \n", 177 | " metrics.mean_absolute_error(act, pred))" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": 129, 183 | "metadata": {}, 184 | "outputs": [ 185 | { 186 | "data": { 187 | "text/plain": [ 188 | "(75.36166834955054, 60.629082113104403)" 189 | ] 190 | }, 191 | "execution_count": 129, 192 | "metadata": {}, 193 | "output_type": "execute_result" 194 | } 195 | ], 196 | "source": [ 197 | "regr_metrics(y_test, regr.predict(test))" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "## Polynomial Features" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "Linear regression finds the best coefficients $\\beta_i$ for:\n", 212 | "\n", 213 | "$$ x_0\\beta_0 + x_1\\beta_1 + x_2\\beta_2 = y $$\n", 214 | "\n", 215 | "Adding polynomial features is still a linear regression problem, just with more terms:\n", 216 | "\n", 217 | "$$ x_0\\beta_0 + x_1\\beta_1 + x_2\\beta_2 + x_0^2\\beta_3 + x_0 x_1\\beta_4 + x_0 x_2\\beta_5 + x_1^2\\beta_6 + x_1 x_2\\beta_7 + x_2^2\\beta_8 = y $$\n", 218 | "\n", 219 | "We need to use our original data $X$ to calculate the additional polynomial features." 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 172, 225 | "metadata": {}, 226 | "outputs": [ 227 | { 228 | "data": { 229 | "text/plain": [ 230 | "(353, 10)" 231 | ] 232 | }, 233 | "execution_count": 172, 234 | "metadata": {}, 235 | "output_type": "execute_result" 236 | } 237 | ], 238 | "source": [ 239 | "trn.shape" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": {}, 245 | "source": [ 246 | "Now, we want to try improving our model's performance by adding some more features. Currently, our model is linear in each variable, but we can add polynomial features to change this." 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": 130, 252 | "metadata": { 253 | "collapsed": true 254 | }, 255 | "outputs": [], 256 | "source": [ 257 | "poly = PolynomialFeatures(include_bias=False)" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": 131, 263 | "metadata": { 264 | "collapsed": true 265 | }, 266 | "outputs": [], 267 | "source": [ 268 | "trn_feat = poly.fit_transform(trn)" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": 132, 274 | "metadata": {}, 275 | "outputs": [ 276 | { 277 | "data": { 278 | "text/plain": [ 279 | "'age, sex, bmi, bp, s1, s2, s3, s4, s5, s6, age^2, age sex, age bmi, age bp, age s1, age s2, age s3, age s4, age s5, age s6, sex^2, sex bmi, sex bp, sex s1, sex s2, sex s3, sex s4, sex s5, sex s6, bmi^2, bmi bp, bmi s1, bmi s2, bmi s3, bmi s4, bmi s5, bmi s6, bp^2, bp s1, bp s2, bp s3, bp s4, bp s5, bp s6, s1^2, s1 s2, s1 s3, s1 s4, s1 s5, s1 s6, s2^2, s2 s3, s2 s4, s2 s5, s2 s6, s3^2, s3 s4, s3 s5, s3 s6, s4^2, s4 s5, s4 s6, s5^2, s5 s6, s6^2'" 280 | ] 281 | }, 282 | "execution_count": 132, 283 | "metadata": {}, 284 | "output_type": "execute_result" 285 | } 286 | ], 287 | "source": [ 288 | "', '.join(poly.get_feature_names(feature_names))" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": 133, 294 | "metadata": { 295 | "scrolled": true 296 | }, 297 | "outputs": [ 298 | { 299 | "data": { 300 | "text/plain": [ 301 | "(353, 65)" 302 | ] 303 | }, 304 | "execution_count": 133, 305 | "metadata": {}, 306 | "output_type": "execute_result" 307 | } 308 | ], 309 | "source": [ 310 | "trn_feat.shape" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 134, 316 | "metadata": {}, 317 | "outputs": [ 318 | { 319 | "data": { 320 | "text/plain": [ 321 | "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)" 322 | ] 323 | }, 324 | "execution_count": 134, 325 | "metadata": {}, 326 | "output_type": "execute_result" 327 | } 328 | ], 329 | "source": [ 330 | "regr.fit(trn_feat, y_trn)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": 135, 336 | "metadata": {}, 337 | "outputs": [ 338 | { 339 | "data": { 340 | "text/plain": [ 341 | "(55.747345922929185, 42.836164292252235)" 342 | ] 343 | }, 344 | "execution_count": 135, 345 | "metadata": {}, 346 | "output_type": "execute_result" 347 | } 348 | ], 349 | "source": [ 350 | "regr_metrics(y_test, regr.predict(poly.fit_transform(test)))" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "Time is squared in #features and linear in #points, so this will get very slow!" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": 136, 363 | "metadata": {}, 364 | "outputs": [ 365 | { 366 | "name": "stdout", 367 | "output_type": "stream", 368 | "text": [ 369 | "635 µs ± 9.25 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n" 370 | ] 371 | } 372 | ], 373 | "source": [ 374 | "%timeit poly.fit_transform(trn)" 375 | ] 376 | }, 377 | { 378 | "cell_type": "markdown", 379 | "metadata": {}, 380 | "source": [ 381 | "## Speeding up feature generation" 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "We would like to speed this up. We will use [Numba](http://numba.pydata.org/numba-doc/0.12.2/tutorial_firststeps.html), a Python library that compiles code directly to C." 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": {}, 394 | "source": [ 395 | "**Numba is a compiler.**" 396 | ] 397 | }, 398 | { 399 | "cell_type": "markdown", 400 | "metadata": {}, 401 | "source": [ 402 | "#### Resources" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "metadata": {}, 408 | "source": [ 409 | "[This tutorial](https://jakevdp.github.io/blog/2012/08/24/numba-vs-cython/) from Jake VanderPlas is a nice introduction. Here Jake [implements a non-trivial algorithm](https://jakevdp.github.io/blog/2015/02/24/optimizing-python-with-numpy-and-numba/) (non-uniform fast Fourier transform) with Numba.\n", 410 | "\n", 411 | "Cython is another alternative. I've found Cython to require more knowledge to use than Numba (it's closer to C), but to provide similar speed-ups to Numba.\n", 412 | "\n", 413 | "\"Cython\n", 414 | "\n", 415 | "Here is a [thorough answer](https://softwareengineering.stackexchange.com/questions/246094/understanding-the-differences-traditional-interpreter-jit-compiler-jit-interp) on the differences between an Ahead Of Time (AOT) compiler, a Just In Time (JIT) compiler, and an interpreter." 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "metadata": {}, 421 | "source": [ 422 | "### Experiments with vectorization and native code" 423 | ] 424 | }, 425 | { 426 | "cell_type": "markdown", 427 | "metadata": {}, 428 | "source": [ 429 | "Let's first get aquainted with Numba, and then we will return to our problem of polynomial features for regression on the diabates data set." 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 140, 435 | "metadata": { 436 | "collapsed": true 437 | }, 438 | "outputs": [], 439 | "source": [ 440 | "%matplotlib inline" 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "execution_count": 141, 446 | "metadata": { 447 | "collapsed": true 448 | }, 449 | "outputs": [], 450 | "source": [ 451 | "import math, numpy as np, matplotlib.pyplot as plt\n", 452 | "from pandas_summary import DataFrameSummary\n", 453 | "from scipy import ndimage" 454 | ] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "execution_count": 142, 459 | "metadata": { 460 | "collapsed": true 461 | }, 462 | "outputs": [], 463 | "source": [ 464 | "from numba import jit, vectorize, guvectorize, cuda, float32, void, float64" 465 | ] 466 | }, 467 | { 468 | "cell_type": "markdown", 469 | "metadata": {}, 470 | "source": [ 471 | "We will show the impact of:\n", 472 | "- Avoiding memory allocations and copies (slower than CPU calculations)\n", 473 | "- Better locality\n", 474 | "- Vectorization\n", 475 | "\n", 476 | "If we use numpy on whole arrays at a time, it creates lots of temporaries, and can't use cache. If we use numba looping through an array item at a time, then we don't have to allocate large temporary arrays, and can reuse cached data since we're doing multiple calculations on each array item." 477 | ] 478 | }, 479 | { 480 | "cell_type": "code", 481 | "execution_count": 175, 482 | "metadata": { 483 | "collapsed": true 484 | }, 485 | "outputs": [], 486 | "source": [ 487 | "# Untype and Unvectorized\n", 488 | "def proc_python(xx,yy):\n", 489 | " zz = np.zeros(nobs, dtype='float32')\n", 490 | " for j in range(nobs): \n", 491 | " x, y = xx[j], yy[j] \n", 492 | " x = x*2 - ( y * 55 )\n", 493 | " y = x + y*2 \n", 494 | " z = x + y + 99 \n", 495 | " z = z * ( z - .88 ) \n", 496 | " zz[j] = z \n", 497 | " return zz" 498 | ] 499 | }, 500 | { 501 | "cell_type": "code", 502 | "execution_count": 176, 503 | "metadata": { 504 | "collapsed": true 505 | }, 506 | "outputs": [], 507 | "source": [ 508 | "nobs = 10000\n", 509 | "x = np.random.randn(nobs).astype('float32')\n", 510 | "y = np.random.randn(nobs).astype('float32')" 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": 177, 516 | "metadata": {}, 517 | "outputs": [ 518 | { 519 | "name": "stdout", 520 | "output_type": "stream", 521 | "text": [ 522 | "49.8 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n" 523 | ] 524 | } 525 | ], 526 | "source": [ 527 | "%timeit proc_python(x,y) # Untyped and unvectorized" 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "#### Numpy" 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": {}, 540 | "source": [ 541 | "Numpy lets us vectorize this:" 542 | ] 543 | }, 544 | { 545 | "cell_type": "code", 546 | "execution_count": 146, 547 | "metadata": { 548 | "collapsed": true 549 | }, 550 | "outputs": [], 551 | "source": [ 552 | "# Typed and Vectorized\n", 553 | "def proc_numpy(x,y):\n", 554 | " z = np.zeros(nobs, dtype='float32')\n", 555 | " x = x*2 - ( y * 55 )\n", 556 | " y = x + y*2 \n", 557 | " z = x + y + 99 \n", 558 | " z = z * ( z - .88 ) \n", 559 | " return z" 560 | ] 561 | }, 562 | { 563 | "cell_type": "code", 564 | "execution_count": 147, 565 | "metadata": {}, 566 | "outputs": [ 567 | { 568 | "data": { 569 | "text/plain": [ 570 | "True" 571 | ] 572 | }, 573 | "execution_count": 147, 574 | "metadata": {}, 575 | "output_type": "execute_result" 576 | } 577 | ], 578 | "source": [ 579 | "np.allclose( proc_numpy(x,y), proc_python(x,y), atol=1e-4 )" 580 | ] 581 | }, 582 | { 583 | "cell_type": "code", 584 | "execution_count": 148, 585 | "metadata": {}, 586 | "outputs": [ 587 | { 588 | "name": "stdout", 589 | "output_type": "stream", 590 | "text": [ 591 | "35.9 µs ± 166 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n" 592 | ] 593 | } 594 | ], 595 | "source": [ 596 | "%timeit proc_numpy(x,y) # Typed and vectorized" 597 | ] 598 | }, 599 | { 600 | "cell_type": "markdown", 601 | "metadata": {}, 602 | "source": [ 603 | "#### Numba" 604 | ] 605 | }, 606 | { 607 | "cell_type": "markdown", 608 | "metadata": {}, 609 | "source": [ 610 | "Numba offers several different decorators. We will try two different ones:\n", 611 | "\n", 612 | "- `@jit`: very general\n", 613 | "- `@vectorize`: don't need to write a for loop. useful when operating on vectors of the same size" 614 | ] 615 | }, 616 | { 617 | "cell_type": "markdown", 618 | "metadata": {}, 619 | "source": [ 620 | "First, we will use Numba's jit (just-in-time) compiler decorator, without explicitly vectorizing. This avoids large memory allocations, so we have better locality:" 621 | ] 622 | }, 623 | { 624 | "cell_type": "code", 625 | "execution_count": 149, 626 | "metadata": { 627 | "collapsed": true, 628 | "scrolled": false 629 | }, 630 | "outputs": [], 631 | "source": [ 632 | "@jit()\n", 633 | "def proc_numba(xx,yy,zz):\n", 634 | " for j in range(nobs): \n", 635 | " x, y = xx[j], yy[j] \n", 636 | " x = x*2 - ( y * 55 )\n", 637 | " y = x + y*2 \n", 638 | " z = x + y + 99 \n", 639 | " z = z * ( z - .88 ) \n", 640 | " zz[j] = z \n", 641 | " return zz" 642 | ] 643 | }, 644 | { 645 | "cell_type": "code", 646 | "execution_count": 150, 647 | "metadata": {}, 648 | "outputs": [ 649 | { 650 | "data": { 651 | "text/plain": [ 652 | "True" 653 | ] 654 | }, 655 | "execution_count": 150, 656 | "metadata": {}, 657 | "output_type": "execute_result" 658 | } 659 | ], 660 | "source": [ 661 | "z = np.zeros(nobs).astype('float32')\n", 662 | "np.allclose( proc_numpy(x,y), proc_numba(x,y,z), atol=1e-4 )" 663 | ] 664 | }, 665 | { 666 | "cell_type": "code", 667 | "execution_count": 151, 668 | "metadata": {}, 669 | "outputs": [ 670 | { 671 | "name": "stdout", 672 | "output_type": "stream", 673 | "text": [ 674 | "6.4 µs ± 17.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" 675 | ] 676 | } 677 | ], 678 | "source": [ 679 | "%timeit proc_numba(x,y,z)" 680 | ] 681 | }, 682 | { 683 | "cell_type": "markdown", 684 | "metadata": {}, 685 | "source": [ 686 | "Now we will use Numba's `vectorize` decorator. Numba's compiler optimizes this in a smarter way than what is possible with plain Python and Numpy." 687 | ] 688 | }, 689 | { 690 | "cell_type": "code", 691 | "execution_count": 152, 692 | "metadata": { 693 | "collapsed": true 694 | }, 695 | "outputs": [], 696 | "source": [ 697 | "@vectorize\n", 698 | "def vec_numba(x,y):\n", 699 | " x = x*2 - ( y * 55 )\n", 700 | " y = x + y*2 \n", 701 | " z = x + y + 99 \n", 702 | " return z * ( z - .88 ) " 703 | ] 704 | }, 705 | { 706 | "cell_type": "code", 707 | "execution_count": 153, 708 | "metadata": {}, 709 | "outputs": [ 710 | { 711 | "data": { 712 | "text/plain": [ 713 | "True" 714 | ] 715 | }, 716 | "execution_count": 153, 717 | "metadata": {}, 718 | "output_type": "execute_result" 719 | } 720 | ], 721 | "source": [ 722 | "np.allclose(vec_numba(x,y), proc_numba(x,y,z), atol=1e-4 )" 723 | ] 724 | }, 725 | { 726 | "cell_type": "code", 727 | "execution_count": 154, 728 | "metadata": { 729 | "scrolled": true 730 | }, 731 | "outputs": [ 732 | { 733 | "name": "stdout", 734 | "output_type": "stream", 735 | "text": [ 736 | "5.82 µs ± 14.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" 737 | ] 738 | } 739 | ], 740 | "source": [ 741 | "%timeit vec_numba(x,y)" 742 | ] 743 | }, 744 | { 745 | "cell_type": "markdown", 746 | "metadata": {}, 747 | "source": [ 748 | "Numba is **amazing**. Look how fast this is!" 749 | ] 750 | }, 751 | { 752 | "cell_type": "markdown", 753 | "metadata": { 754 | "heading_collapsed": true 755 | }, 756 | "source": [ 757 | "### Numba polynomial features" 758 | ] 759 | }, 760 | { 761 | "cell_type": "code", 762 | "execution_count": 155, 763 | "metadata": { 764 | "collapsed": true, 765 | "hidden": true 766 | }, 767 | "outputs": [], 768 | "source": [ 769 | "@jit(nopython=True)\n", 770 | "def vec_poly(x, res):\n", 771 | " m,n=x.shape\n", 772 | " feat_idx=0\n", 773 | " for i in range(n):\n", 774 | " v1=x[:,i]\n", 775 | " for k in range(m): res[k,feat_idx] = v1[k]\n", 776 | " feat_idx+=1\n", 777 | " for j in range(i,n):\n", 778 | " for k in range(m): res[k,feat_idx] = v1[k]*x[k,j]\n", 779 | " feat_idx+=1" 780 | ] 781 | }, 782 | { 783 | "cell_type": "markdown", 784 | "metadata": { 785 | "heading_collapsed": true, 786 | "hidden": true 787 | }, 788 | "source": [ 789 | "#### Row-Major vs Column-Major Storage" 790 | ] 791 | }, 792 | { 793 | "cell_type": "markdown", 794 | "metadata": { 795 | "hidden": true 796 | }, 797 | "source": [ 798 | "From this [blog post by Eli Bendersky](http://eli.thegreenplace.net/2015/memory-layout-of-multi-dimensional-arrays/):\n", 799 | "\n", 800 | "\"The row-major layout of a matrix puts the first row in contiguous memory, then the second row right after it, then the third, and so on. Column-major layout puts the first column in contiguous memory, then the second, etc.... While knowing which layout a particular data set is using is critical for good performance, there's no single answer to the question which layout 'is better' in general.\n", 801 | "\n", 802 | "\"It turns out that matching the way your algorithm works with the data layout can make or break the performance of an application.\n", 803 | "\n", 804 | "\"The short takeaway is: **always traverse the data in the order it was laid out**.\"\n", 805 | "\n", 806 | "**Column-major layout**: Fortran, Matlab, R, and Julia\n", 807 | "\n", 808 | "**Row-major layout**: C, C++, Python, Pascal, Mathematica" 809 | ] 810 | }, 811 | { 812 | "cell_type": "code", 813 | "execution_count": 156, 814 | "metadata": { 815 | "collapsed": true, 816 | "hidden": true 817 | }, 818 | "outputs": [], 819 | "source": [ 820 | "trn = np.asfortranarray(trn)\n", 821 | "test = np.asfortranarray(test)" 822 | ] 823 | }, 824 | { 825 | "cell_type": "code", 826 | "execution_count": 157, 827 | "metadata": { 828 | "collapsed": true, 829 | "hidden": true 830 | }, 831 | "outputs": [], 832 | "source": [ 833 | "m,n=trn.shape\n", 834 | "n_feat = n*(n+1)//2 + n\n", 835 | "trn_feat = np.zeros((m,n_feat), order='F')\n", 836 | "test_feat = np.zeros((len(y_test), n_feat), order='F')" 837 | ] 838 | }, 839 | { 840 | "cell_type": "code", 841 | "execution_count": 158, 842 | "metadata": { 843 | "collapsed": true, 844 | "hidden": true 845 | }, 846 | "outputs": [], 847 | "source": [ 848 | "vec_poly(trn, trn_feat)\n", 849 | "vec_poly(test, test_feat)" 850 | ] 851 | }, 852 | { 853 | "cell_type": "code", 854 | "execution_count": 159, 855 | "metadata": { 856 | "hidden": true 857 | }, 858 | "outputs": [ 859 | { 860 | "data": { 861 | "text/plain": [ 862 | "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)" 863 | ] 864 | }, 865 | "execution_count": 159, 866 | "metadata": {}, 867 | "output_type": "execute_result" 868 | } 869 | ], 870 | "source": [ 871 | "regr.fit(trn_feat, y_trn)" 872 | ] 873 | }, 874 | { 875 | "cell_type": "code", 876 | "execution_count": 160, 877 | "metadata": { 878 | "hidden": true 879 | }, 880 | "outputs": [ 881 | { 882 | "data": { 883 | "text/plain": [ 884 | "(55.74734592292935, 42.836164292252306)" 885 | ] 886 | }, 887 | "execution_count": 160, 888 | "metadata": {}, 889 | "output_type": "execute_result" 890 | } 891 | ], 892 | "source": [ 893 | "regr_metrics(y_test, regr.predict(test_feat))" 894 | ] 895 | }, 896 | { 897 | "cell_type": "code", 898 | "execution_count": 161, 899 | "metadata": { 900 | "hidden": true 901 | }, 902 | "outputs": [ 903 | { 904 | "name": "stdout", 905 | "output_type": "stream", 906 | "text": [ 907 | "7.33 µs ± 19.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n" 908 | ] 909 | } 910 | ], 911 | "source": [ 912 | "%timeit vec_poly(trn, trn_feat)" 913 | ] 914 | }, 915 | { 916 | "cell_type": "markdown", 917 | "metadata": { 918 | "hidden": true 919 | }, 920 | "source": [ 921 | "Recall, this was the time from the scikit learn implementation PolynomialFeatures, which was created by experts: " 922 | ] 923 | }, 924 | { 925 | "cell_type": "code", 926 | "execution_count": 136, 927 | "metadata": { 928 | "hidden": true 929 | }, 930 | "outputs": [ 931 | { 932 | "name": "stdout", 933 | "output_type": "stream", 934 | "text": [ 935 | "635 µs ± 9.25 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n" 936 | ] 937 | } 938 | ], 939 | "source": [ 940 | "%timeit poly.fit_transform(trn)" 941 | ] 942 | }, 943 | { 944 | "cell_type": "code", 945 | "execution_count": 162, 946 | "metadata": { 947 | "hidden": true 948 | }, 949 | "outputs": [ 950 | { 951 | "data": { 952 | "text/plain": [ 953 | "78.57142857142857" 954 | ] 955 | }, 956 | "execution_count": 162, 957 | "metadata": {}, 958 | "output_type": "execute_result" 959 | } 960 | ], 961 | "source": [ 962 | "605/7.7" 963 | ] 964 | }, 965 | { 966 | "cell_type": "markdown", 967 | "metadata": { 968 | "hidden": true 969 | }, 970 | "source": [ 971 | "This is a big deal! Numba is **amazing**! With a single line of code, we are getting a 78x speed-up over scikit learn (which was optimized by experts)." 972 | ] 973 | }, 974 | { 975 | "cell_type": "markdown", 976 | "metadata": { 977 | "heading_collapsed": true 978 | }, 979 | "source": [ 980 | "## Regularization and noise" 981 | ] 982 | }, 983 | { 984 | "cell_type": "markdown", 985 | "metadata": { 986 | "hidden": true 987 | }, 988 | "source": [ 989 | "Regularization is a way to reduce over-fitting and create models that better generalize to new data." 990 | ] 991 | }, 992 | { 993 | "cell_type": "markdown", 994 | "metadata": { 995 | "heading_collapsed": true, 996 | "hidden": true 997 | }, 998 | "source": [ 999 | "### Regularization" 1000 | ] 1001 | }, 1002 | { 1003 | "cell_type": "markdown", 1004 | "metadata": { 1005 | "hidden": true 1006 | }, 1007 | "source": [ 1008 | "Lasso regression uses an L1 penalty, which pushes towards sparse coefficients. The parameter $\\alpha$ is used to weight the penalty term. Scikit Learn's LassoCV performs cross validation with a number of different values for $\\alpha$.\n", 1009 | "\n", 1010 | "Watch this [Coursera video on Lasso regression](https://www.coursera.org/learn/machine-learning-data-analysis/lecture/0KIy7/what-is-lasso-regression) for more info." 1011 | ] 1012 | }, 1013 | { 1014 | "cell_type": "code", 1015 | "execution_count": 163, 1016 | "metadata": { 1017 | "collapsed": true, 1018 | "hidden": true 1019 | }, 1020 | "outputs": [], 1021 | "source": [ 1022 | "reg_regr = linear_model.LassoCV(n_alphas=10)" 1023 | ] 1024 | }, 1025 | { 1026 | "cell_type": "code", 1027 | "execution_count": 164, 1028 | "metadata": { 1029 | "hidden": true 1030 | }, 1031 | "outputs": [ 1032 | { 1033 | "name": "stderr", 1034 | "output_type": "stream", 1035 | "text": [ 1036 | "/home/jhoward/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:484: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n", 1037 | " ConvergenceWarning)\n" 1038 | ] 1039 | }, 1040 | { 1041 | "data": { 1042 | "text/plain": [ 1043 | "LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,\n", 1044 | " max_iter=1000, n_alphas=10, n_jobs=1, normalize=False, positive=False,\n", 1045 | " precompute='auto', random_state=None, selection='cyclic', tol=0.0001,\n", 1046 | " verbose=False)" 1047 | ] 1048 | }, 1049 | "execution_count": 164, 1050 | "metadata": {}, 1051 | "output_type": "execute_result" 1052 | } 1053 | ], 1054 | "source": [ 1055 | "reg_regr.fit(trn_feat, y_trn)" 1056 | ] 1057 | }, 1058 | { 1059 | "cell_type": "code", 1060 | "execution_count": 165, 1061 | "metadata": { 1062 | "hidden": true 1063 | }, 1064 | "outputs": [ 1065 | { 1066 | "data": { 1067 | "text/plain": [ 1068 | "0.0098199431661591518" 1069 | ] 1070 | }, 1071 | "execution_count": 165, 1072 | "metadata": {}, 1073 | "output_type": "execute_result" 1074 | } 1075 | ], 1076 | "source": [ 1077 | "reg_regr.alpha_" 1078 | ] 1079 | }, 1080 | { 1081 | "cell_type": "code", 1082 | "execution_count": 166, 1083 | "metadata": { 1084 | "hidden": true 1085 | }, 1086 | "outputs": [ 1087 | { 1088 | "data": { 1089 | "text/plain": [ 1090 | "(50.0982471642817, 40.065199085003101)" 1091 | ] 1092 | }, 1093 | "execution_count": 166, 1094 | "metadata": {}, 1095 | "output_type": "execute_result" 1096 | } 1097 | ], 1098 | "source": [ 1099 | "regr_metrics(y_test, reg_regr.predict(test_feat))" 1100 | ] 1101 | }, 1102 | { 1103 | "cell_type": "markdown", 1104 | "metadata": { 1105 | "heading_collapsed": true, 1106 | "hidden": true 1107 | }, 1108 | "source": [ 1109 | "### Noise" 1110 | ] 1111 | }, 1112 | { 1113 | "cell_type": "markdown", 1114 | "metadata": { 1115 | "hidden": true 1116 | }, 1117 | "source": [ 1118 | "Now we will add some noise to the data" 1119 | ] 1120 | }, 1121 | { 1122 | "cell_type": "code", 1123 | "execution_count": 167, 1124 | "metadata": { 1125 | "collapsed": true, 1126 | "hidden": true 1127 | }, 1128 | "outputs": [], 1129 | "source": [ 1130 | "idxs = np.random.randint(0, len(trn), 10)" 1131 | ] 1132 | }, 1133 | { 1134 | "cell_type": "code", 1135 | "execution_count": 168, 1136 | "metadata": { 1137 | "collapsed": true, 1138 | "hidden": true 1139 | }, 1140 | "outputs": [], 1141 | "source": [ 1142 | "y_trn2 = np.copy(y_trn)\n", 1143 | "y_trn2[idxs] *= 10 # label noise" 1144 | ] 1145 | }, 1146 | { 1147 | "cell_type": "code", 1148 | "execution_count": 169, 1149 | "metadata": { 1150 | "hidden": true 1151 | }, 1152 | "outputs": [ 1153 | { 1154 | "data": { 1155 | "text/plain": [ 1156 | "(51.1766253181518, 41.415992803872754)" 1157 | ] 1158 | }, 1159 | "execution_count": 169, 1160 | "metadata": {}, 1161 | "output_type": "execute_result" 1162 | } 1163 | ], 1164 | "source": [ 1165 | "regr = linear_model.LinearRegression()\n", 1166 | "regr.fit(trn, y_trn)\n", 1167 | "regr_metrics(y_test, regr.predict(test))" 1168 | ] 1169 | }, 1170 | { 1171 | "cell_type": "code", 1172 | "execution_count": 170, 1173 | "metadata": { 1174 | "hidden": true 1175 | }, 1176 | "outputs": [ 1177 | { 1178 | "data": { 1179 | "text/plain": [ 1180 | "(62.66110319520415, 53.21914420254862)" 1181 | ] 1182 | }, 1183 | "execution_count": 170, 1184 | "metadata": {}, 1185 | "output_type": "execute_result" 1186 | } 1187 | ], 1188 | "source": [ 1189 | "regr.fit(trn, y_trn2)\n", 1190 | "regr_metrics(y_test, regr.predict(test))" 1191 | ] 1192 | }, 1193 | { 1194 | "cell_type": "markdown", 1195 | "metadata": { 1196 | "hidden": true 1197 | }, 1198 | "source": [ 1199 | "Huber loss is a loss function that is less sensitive to outliers than squared error loss. It is quadratic for small error values, and linear for large values.\n", 1200 | "\n", 1201 | " $$L(x)= \n", 1202 | "\\begin{cases}\n", 1203 | " \\frac{1}{2}x^2, & \\text{for } \\lvert x\\rvert\\leq \\delta \\\\\n", 1204 | " \\delta(\\lvert x \\rvert - \\frac{1}{2}\\delta), & \\text{otherwise}\n", 1205 | "\\end{cases}$$" 1206 | ] 1207 | }, 1208 | { 1209 | "cell_type": "code", 1210 | "execution_count": 171, 1211 | "metadata": { 1212 | "hidden": true 1213 | }, 1214 | "outputs": [ 1215 | { 1216 | "data": { 1217 | "text/plain": [ 1218 | "(51.24055602541746, 41.670840571376822)" 1219 | ] 1220 | }, 1221 | "execution_count": 171, 1222 | "metadata": {}, 1223 | "output_type": "execute_result" 1224 | } 1225 | ], 1226 | "source": [ 1227 | "hregr = linear_model.HuberRegressor()\n", 1228 | "hregr.fit(trn, y_trn2)\n", 1229 | "regr_metrics(y_test, hregr.predict(test))" 1230 | ] 1231 | }, 1232 | { 1233 | "cell_type": "markdown", 1234 | "metadata": { 1235 | "collapsed": true 1236 | }, 1237 | "source": [ 1238 | "# End" 1239 | ] 1240 | } 1241 | ], 1242 | "metadata": { 1243 | "anaconda-cloud": {}, 1244 | "kernelspec": { 1245 | "display_name": "Python 3", 1246 | "language": "python", 1247 | "name": "python3" 1248 | }, 1249 | "language_info": { 1250 | "codemirror_mode": { 1251 | "name": "ipython", 1252 | "version": 3 1253 | }, 1254 | "file_extension": ".py", 1255 | "mimetype": "text/x-python", 1256 | "name": "python", 1257 | "nbconvert_exporter": "python", 1258 | "pygments_lexer": "ipython3", 1259 | "version": "3.6.1" 1260 | } 1261 | }, 1262 | "nbformat": 4, 1263 | "nbformat_minor": 1 1264 | } 1265 | -------------------------------------------------------------------------------- /nbs/Homework 1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Homework 1" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This is due on Thurs, 6/1" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "1. Consider the polynomial $p(x) = (x-2)^9 = x^9 - 18x^8 + 144x^7 - 672x^6 + 2016x^5 - 4032x^4 + 5376x^3 - 4608x^2 + 2304x - 512$\n", 22 | "\n", 23 | " a. Plot $p(x)$ for $x=1.920,\\,1.921,\\,1.922,\\ldots,2.080$ evaluating $p$ via its coefficients $1,\\,,-18,\\,144,\\ldots$\n", 24 | "\n", 25 | " b. Plot the same plot again, now evaluating $p$ via the expression $(x-2)^9$.\n", 26 | "\n", 27 | " c. Explain the difference.\n", 28 | " \n", 29 | " *(The numpy method linspace will be useful for this)*" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "2\\. How many different double-precision numbers are there? Express your answer using powers of 2" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "3\\. Using the updated [Numbers Every Programmer Should Know](https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html), how much longer does a main memory reference take than an L1 cache look-up? How much longer does a disk seek take than a main memory reference?" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "4\\. From the Halide Video, what are 4 ways to traverse a 2d array?" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "5\\. Using the animations below ([source](https://www.youtube.com/watch?v=3uiEyEKji0M)), explain what the benefits and pitfalls of each approach. Green squares indicate that a value is being read; red indicates a value is being written. Your answers should be longer in length (give more detail) than just two words.\n", 58 | "\n", 59 | " a. \"Halide\"\n", 60 | " \n", 61 | " b. \"Halide\"\n", 62 | " \n", 63 | " c. \"Halide\"" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "6\\. Prove that if $A = Q B Q^T$ for some orthnogonal matrix $Q$, the $A$ and $B$ have the same singular values." 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "7\\. What is the *stochastic* part of *stochastic gradient descent*?" 78 | ] 79 | } 80 | ], 81 | "metadata": { 82 | "kernelspec": { 83 | "display_name": "Python [conda root]", 84 | "language": "python", 85 | "name": "conda-root-py" 86 | }, 87 | "language_info": { 88 | "codemirror_mode": { 89 | "name": "ipython", 90 | "version": 3 91 | }, 92 | "file_extension": ".py", 93 | "mimetype": "text/x-python", 94 | "name": "python", 95 | "nbconvert_exporter": "python", 96 | "pygments_lexer": "ipython3", 97 | "version": "3.6.1" 98 | } 99 | }, 100 | "nbformat": 4, 101 | "nbformat_minor": 2 102 | } 103 | -------------------------------------------------------------------------------- /nbs/Homework 2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Homework 2" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This homework covers the material in Lessons 3 & 4. It is due **Thurday, June 15**. Please submit your **answers as a PDF**." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "1\\. Modify the LU method (without pivoting) to work **in-place**. That is, it should not allocate any new memory for L or U, but instead overwrite A." 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": { 28 | "collapsed": true 29 | }, 30 | "outputs": [], 31 | "source": [ 32 | "def LU(A):\n", 33 | " U = np.copy(A)\n", 34 | " m, n = A.shape\n", 35 | " L = np.eye(n)\n", 36 | " for k in range(n-1):\n", 37 | " for j in range(k+1,n):\n", 38 | " L[j,k] = U[j,k]/U[k,k]\n", 39 | " U[j,k:n] -= L[j,k] * U[k,k:n]\n", 40 | " return L, U" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "2\\. Modify our LU method from class to add pivoting, as described in the lesson. *Hint: the swap method below will be useful* " 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": { 54 | "collapsed": true 55 | }, 56 | "outputs": [], 57 | "source": [ 58 | "def swap(a,b):\n", 59 | " temp = np.copy(a)\n", 60 | " a[:] = b\n", 61 | " b[:] = temp" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "3\\. For each of the following sets of dimensions, either\n", 69 | " - give the dimensions of the output of an operation on A and B, **or** \n", 70 | " - answer *incompatible* if the dimensions are incompatible according to the [numpy rules of broadcasting](https://docs.scipy.org/doc/numpy-1.10.0/user/basics.broadcasting.html)." 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": { 76 | "collapsed": true 77 | }, 78 | "source": [ 79 | " a. A (2d array): 3 x 3\n", 80 | " B (1d array): 1\n", 81 | "\n", 82 | " b. A (2d array): 2 x 1\n", 83 | " B (3d array): 6 x 4 x 2\n", 84 | "\n", 85 | " c. A (2d array): 5 x 4\n", 86 | " B (1d array): 4\n", 87 | "\n", 88 | " d. A (3d array): 32 x 64 x 8\n", 89 | " B (3d array): 32 x 1 x 8\n", 90 | "\n", 91 | " e. A (3d array): 64 x 1\n", 92 | " B (3d array): 32 x 1 x 16\n", 93 | "\n", 94 | " f. A (3d array): 32 x 64 x 2\n", 95 | " B (3d array): 32 x 1 x 8" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "source": [ 104 | "4\\. Write how this matrix would be stored in compressed row format:\n", 105 | "\n", 106 | "\\begin{pmatrix}\n", 107 | " 1 & & & & -2 & -3 \\\\\n", 108 | " & 3 & & & & -9 \\\\\n", 109 | " & & & -7 & 4 & \\\\ \n", 110 | " -1 & 2 & & 7 & & \\\\\n", 111 | " -3 & & & 26 & &\n", 112 | " \\end{pmatrix}" 113 | ] 114 | } 115 | ], 116 | "metadata": { 117 | "kernelspec": { 118 | "display_name": "Python [conda root]", 119 | "language": "python", 120 | "name": "conda-root-py" 121 | }, 122 | "language_info": { 123 | "codemirror_mode": { 124 | "name": "ipython", 125 | "version": 3 126 | }, 127 | "file_extension": ".py", 128 | "mimetype": "text/x-python", 129 | "name": "python", 130 | "nbconvert_exporter": "python", 131 | "pygments_lexer": "ipython3", 132 | "version": "3.6.1" 133 | } 134 | }, 135 | "nbformat": 4, 136 | "nbformat_minor": 2 137 | } 138 | -------------------------------------------------------------------------------- /nbs/Homework 3.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Homework 3" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "1\\. Add shifts to the QR algorithm\n", 15 | "\n", 16 | "Instead of factoring $A_k$ as $Q_k R_k$ (the way the pure QR algorithm without shifts does), the shifted QR algorithms:\n", 17 | "\n", 18 | "i. Get the QR factorization $$A_k - s_k I = Q_k R_k$$\n", 19 | "ii. Set $$A_{k+1} = R_k Q_k + s_k I$$\n", 20 | "\n", 21 | "Choose $s_k = A_k(m,m)$, an approximation of an eigenvalue of A. \n", 22 | "\n", 23 | "The idea of adding shifts to speed up convergence shows up in many algorithms in numerical linear algebra (including the power method, inverse iteration, and Rayleigh quotient iteration). " 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": { 30 | "collapsed": true 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "def practical_qr(A, iters=10):\n", 35 | " # Fill method in\n", 36 | " return Ak, Q" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": { 43 | "collapsed": true 44 | }, 45 | "outputs": [], 46 | "source": [] 47 | } 48 | ], 49 | "metadata": { 50 | "kernelspec": { 51 | "display_name": "Python [conda root]", 52 | "language": "python", 53 | "name": "conda-root-py" 54 | }, 55 | "language_info": { 56 | "codemirror_mode": { 57 | "name": "ipython", 58 | "version": 3 59 | }, 60 | "file_extension": ".py", 61 | "mimetype": "text/x-python", 62 | "name": "python", 63 | "nbconvert_exporter": "python", 64 | "pygments_lexer": "ipython3", 65 | "version": "3.6.1" 66 | } 67 | }, 68 | "nbformat": 4, 69 | "nbformat_minor": 2 70 | } 71 | -------------------------------------------------------------------------------- /nbs/Project_ideas.txt: -------------------------------------------------------------------------------- 1 | Suggestions for Writing Topics: 2 | 3 | - Algorithms to research and implement: 4 | -- Cholesky Factorization 5 | -- Gauss-Legendre Quadrature 6 | -- Conjugate Gradient 7 | -- Rayleigh Quotient Iteration 8 | -- Power Method with shifts and deflation 9 | -- QR algorithm with Wilkinson shifts 10 | -- Golub-Kahan bidiagonalization with Householder reflectors 11 | -- Divide-and-conquer eigenvalue algorithm (Cuppen) 12 | -- GMRES (generalized minimal residuals) 13 | -- Sparse NMF 14 | -- Monte Carlo Methods 15 | -- Feel free to ask me about additional ideas 16 | 17 | - Speed up an algorithm of your choice using one or more of...: 18 | -- The GPU with PyTorch 19 | -- Numba or Cython 20 | -- Parallelization 21 | -- Randomized projections 22 | 23 | - Find an interesting paper (related to linear algebra). Summarize it and implement it. Some interesting ideas in: 24 | -- https://arxiv.org/abs/1608.04481 25 | 26 | - Look at the Matrix Factorization Jungle for ideas: https://sites.google.com/site/igorcarron2/matrixfactorizations 27 | -------------------------------------------------------------------------------- /nbs/graddesc.xlsm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/graddesc.xlsm -------------------------------------------------------------------------------- /nbs/gradient-descent-intro.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "toc": "true" 7 | }, 8 | "source": [ 9 | "# Gradient Descent Intro\n", 10 | "

" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": { 17 | "collapsed": true 18 | }, 19 | "outputs": [], 20 | "source": [ 21 | "%matplotlib inline\n", 22 | "import math,sys,os,numpy as np\n", 23 | "from numpy.random import random\n", 24 | "from matplotlib import pyplot as plt, rcParams, animation, rc\n", 25 | "from __future__ import print_function, division\n", 26 | "from ipywidgets import interact, interactive, fixed\n", 27 | "from ipywidgets.widgets import *\n", 28 | "rc('animation', html='html5')\n", 29 | "rcParams['figure.figsize'] = 3, 3\n", 30 | "%precision 4\n", 31 | "np.set_printoptions(precision=4, linewidth=100)" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 1, 37 | "metadata": { 38 | "collapsed": true 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "def lin(a,b,x): return a*x+b" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 3, 48 | "metadata": { 49 | "collapsed": true 50 | }, 51 | "outputs": [], 52 | "source": [ 53 | "a=3.\n", 54 | "b=8." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 7, 60 | "metadata": { 61 | "collapsed": true 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "n=30\n", 66 | "x = random(n)\n", 67 | "y = lin(a,b,x)" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 8, 73 | "metadata": {}, 74 | "outputs": [ 75 | { 76 | "data": { 77 | "text/plain": [ 78 | "array([ 0.3046, 0.918 , 0.7925, 0.8476, 0.2508, 0.3504, 0.8326, 0.6875, 0.4449, 0.4687,\n", 79 | " 0.5901, 0.2757, 0.6629, 0.169 , 0.8677, 0.6612, 0.112 , 0.1669, 0.6226, 0.6174,\n", 80 | " 0.3871, 0.4724, 0.3242, 0.7871, 0.0157, 0.8589, 0.7008, 0.2942, 0.3166, 0.5847])" 81 | ] 82 | }, 83 | "execution_count": 8, 84 | "metadata": {}, 85 | "output_type": "execute_result" 86 | } 87 | ], 88 | "source": [ 89 | "x" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 9, 95 | "metadata": {}, 96 | "outputs": [ 97 | { 98 | "data": { 99 | "text/plain": [ 100 | "array([ 8.9138, 10.7541, 10.3775, 10.5428, 8.7525, 9.0511, 10.4977, 10.0626, 9.3347,\n", 101 | " 9.4062, 9.7704, 8.827 , 9.9888, 8.507 , 10.603 , 9.9836, 8.336 , 8.5006,\n", 102 | " 9.8678, 9.8523, 9.1614, 9.4172, 8.9725, 10.3614, 8.0471, 10.5766, 10.1025,\n", 103 | " 8.8827, 8.9497, 9.7542])" 104 | ] 105 | }, 106 | "execution_count": 9, 107 | "metadata": {}, 108 | "output_type": "execute_result" 109 | } 110 | ], 111 | "source": [ 112 | "y" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 10, 118 | "metadata": {}, 119 | "outputs": [ 120 | { 121 | "data": { 122 | "text/plain": [ 123 | "" 124 | ] 125 | }, 126 | "execution_count": 10, 127 | "metadata": {}, 128 | "output_type": "execute_result" 129 | }, 130 | { 131 | "data": { 132 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAANYAAADFCAYAAAAooQwbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADyRJREFUeJzt3X+MHOV9x/H3x+dLe0ZVz8Hm15GriYRQIA4hPRnLJRFp\nqgBOKC5CkVFQK5TWKgK1jSJLICF+REhQuf0nTZvISlESpYAIMReTAiY/pCQKgvbMmR5W6tbl90KL\nAxyUcBVn8+0fO2vW69sft7vP7Ozu5yWtbnd2dufxig8z88zzfEcRgZl114peN8BsEDlYZgk4WGYJ\nOFhmCThYZgk4WGYJOFhmCThYZgk4WGYJrOx1A5ayZs2aWLduXa+bYXacvXv3/ioi1jZbr5DBWrdu\nHTMzM71uhtlxJD3Xyno+FDRLwMEyS8DBMkugkOdYZnmYni2xY88BXppf4LTxMbZfdBZbzpvoync7\nWDaUpmdL3LBrjoXFIwCU5he4YdccQFfC5UNBG0o79hw4GqqKhcUj7NhzoCvf72DZUHppfmFZy5fL\nwbKhdNr42LKWL5eDZUNp+0VnMTY6csyysdERtl90Vle+350XNpQqHRTuFTTrsi3nTXQtSLUcLBtY\nKa9TNeNg2UBKfZ2qGXde2EC6Zff+pNepmnGwbOBMz5aYX1hc8r1uXadqxsGygTI9W+JL9z5Z9/1u\nXadqxsGygVE5rzrS4H4E3bpO1UzTYEm6U9Irkp6qWvZ+ST+U9J/Z39V1PvuspDlJ+yR5SrAlU9lT\n1Z5XVVu9ajS3XsFW9ljfBC6uWXY98OOIOBP4cfa6nk9GxEcjYqq9Jpo11sqeamx0hJsvPSe3NjUN\nVkT8DHitZvFlwLey598CtnS5XWYtW2qkerURidsvX5/b3graP8c6OSJezp7/N3BynfUC+JGkvZK2\nNfpCSdskzUiaOXToUJvNsmHUqKdvbHSEv/3cubmGCrrQeRHlO9fV2wdfEBEfBS4BrpX0iQbfszMi\npiJiau3aptWlzI6q19PXiz1VRbvB+h9JpwJkf19ZaqWIKGV/XwHuBza0uT2zuuqNVO/Fnqqi3SFN\nu4E/Ae7I/n6/dgVJJwArIuJ/s+efBr7cbkPN6o39Sz1SvR1NgyXpbuBCYI2kF4GbKQfqXklfAJ4D\nPpetexrwjYjYTPm8635Jle3cFREPp/hH2OC7cXqOf3rs+aPnHLVj/1KOVG9H02BFxJV13vrUEuu+\nBGzOnj8NnNtR68wo76mqQ1VRGftXpEBVeOSFFd6OPQfq9o7lNfZvuRwsK7xG4clr7N9yOVhWePXC\nI/Ib+7dcDpYV3lLd6QI+v3GykOdX4BnE1geK2J3ejINlfaFo3enN+FDQLAEHyywBB8ssAQfLLAEH\nyywBB8ssAXe3W656WfY5Tw6W5abXZZ/z5ENBy0W98mR5ln3Ok4NlyTUrT1bUqR+dcLAsuWblyYo6\n9aMTDpYl16w8WVGnfnQidYnpiyUdkHRQUqNquTZgpmdL/N4dP+GM6/+ZFeW6J8fpZXmy1JKVmJY0\nAvw95ZqCZwNXSjq7o9ZaX5ieLbH9u09Sml8gYMlzq16XJ0stZYnpDcDBiHg6It4B7sk+ZwPult37\nWXz3+DApe0yMjw3snqqi3etYrZSYngBeqHr9InB+vS/MSlBvA5icnGyzWVYE9W76FsCzd3wm38b0\nSOoS08v5HpeYtoGRssR0CfhA1evTs2U24FavGl3W8kHUbrAqJaahTolp4F+BMyWdIel9wNbsczbg\nbr70HEZHju0JHB1Rrven6rVkJaYj4rCk64A9wAhwZ0TsT/PPsF5pNKh2GAbb1qNocBe8XpmamoqZ\nGd9Ztehq66lDuRt9kHv8JO1t5e6kHnlhbWlWT33YOVjWln6sp54nB8va0o/11PPkiY7WktpOivFV\no7z+9vEXgotcTz1PDpY1tdRN30ZXiNERsXjkvQPCotdTz5ODZQ3V66RYfDcYHxvlhN9YObRd6o04\nWNZQo06KNxYW2Xfzp3NtT79w54U15E6K9jhY1lA/3vStCBwsa6gfb/pWBD7HsoY87q89DpY11W83\nfSsCHwqaJeBgmSXgYJkl4GCZJeBgmSXgYJkl0FGwJP2lpKck7Zf0V0u8f6GkNyTtyx43dbI9s37R\n9nUsSR8G/oxyxdt3gIcl/SAiDtas+vOI+GwHbTTrO53ssT4EPB4Rb0fEYeCnwOXdaZZZf+skWE8B\nH5d0oqRVwGaOLdBZsUnSv0l6SFLdwnKStkmakTRz6NChDppl1nttHwpGxC8l/TXwCPBrYB9Qe3ex\nJ4DJiHhL0mZgGjizzvftBHZCufxZu+2y9wzLjbSLqKPOi4j4x4j43Yj4BPA68B81778ZEW9lzx8E\nRiWt6WSb1prp2RLb73vvVjql+QW23/ck07Ou8p2HTnsFT8r+TlI+v7qr5v1TpPJdxyRtyLb3aifb\ntNbc+sD+Y+pRACweCW59wMWI89Dp6PbvSToRWASujYh5SX8OEBFfB64ArpF0GFgAtkYRS+8OoKUq\nKDVabt3VUbAi4uNLLPt61fOvAl/tZBtm/cjzsQZIdWeFWPqmZeNjw3MrnV5ysAbE9GyJG3bNsbBY\n2zH7ntEV4pY/HJ5b6fSSgzUgduw5sGSoRiTejXB3e84crAFRr0zZuxE8MyT3/S0Sj24fEPXKlLn2\nX284WANiqTJlY6Mjrv3XIz4UHBAuU1YsDtYAcZmy4vChoFkC3mP1AY9S7z8OVsHVXvgtzS9ww645\nAIerwHwoWHBLXfj1nemLz8EquHoXfn1n+mJzsArOF377k4NVcL7w25/ceVEgjXr/3CvYXxysgmjW\n++cg9RcfChaEe/8GS+oS05L0FUkHs9qCH+tke4PMvX+DJXWJ6Uso1xE8Ezgf+Fr21zj2nGqFxJEl\n6uy4968/pS4xfRnw7Sh7DBiXdGoH2xwYlXOqSt2/pULl3r/+lbrE9ATwQtXrF7Nlxxm2EtO37N5f\ndyq9gInxMW6/fL07LfpU6hLTy/m+oSkxPT1bYn5h6fp+nko/GJKWmAZKHLsXOz1bNrSmZ0t86d4n\n677vc6rB0NF1LEknRcQrVSWmN9asshu4TtI9lDst3oiIlzvZZr+ani1x6wP7m1ai9TnVYEhdYvpB\nyudeB4G3gas73F5faqXmH8DqVaM+pxoQqUtMB3BtJ9sYBPVq/lUbGx3h5ktdTHNQeEhTQpXrVKUm\nF3lHJPcADhgHK5FWD//GRkccqgHksYKJtHL4Nz426lANKO+xEmk0xm/CUz8GnoPVRa2M/ZsYH+MX\n1/9+D1pneXKwuqT2nMpj/4abg9Ulvo2OVXOwusS30bFq7hXsEldTsmoOVpe4mpJV86Fgl7iaklVz\nsLrI1ZSswoeCZgk4WGYJOFhmCThYZgk4WGYJOFhmCXRaTOaLwJ8CAcwBV0fE/1W9fyHwfeCZbNGu\niPhyJ9vMy43Tc9z9+AsciWBE4srzP8BtW9b3ulnWJzopMT0B/AVwdkQsSLoX2Ap8s2bVn0fEZ9tv\nYv5unJ7jO489f/T1kYijrx0ua0Wnh4IrgTFJK4FVwEudN6n37n78hWUtN6vVdrAiogT8DfA88DLl\nmoGPLLHqpuxOIw9JqluGqEglppeaS9VouVmttoMlaTXlmx6cAZwGnCDpqprVngAmI+IjwN8B0/W+\nLyJ2RsRUREytXbu23WZ1xYi0rOVmtTo5FPwD4JmIOBQRi8AuYFP1ChHxZkS8lT1/EBiVtKaDbebi\nyvNr7+3QeLlZrU6C9TywUdIqSQI+BfyyegVJp2TvIWlDtr1XO9hmLm7bsp6rNk4e3UONSFy1cdId\nF9ayTu428rik+ygf7h0GZoGdNSWmrwCukXQYWAC2ZtVxC++2LesdJGubivjf+dTUVMzMzPS6GWbH\nkbQ3IqaarTeU87Gqy5R5QqKlMHTBqi1TVppf4IZdcwAOl3XN0I0VXKpM2cLiEXbsOdCjFtkgGrpg\n1StT1qgktNlyDV2wXKbM8jB0wXKZMsvD0HVeuEyZ5WHoggUuU2bpDd2hoFkeHCyzBBwsswQcLLME\nHCyzBBwsswQcLLME+vI6lqd9WNH1XbA87cP6Qd8dCnrah/WDjoIl6YuS9kt6StLdkn6z5n1J+oqk\ng1ltwY911lxP+7D+0EldwUqJ6amI+DAwQrnEdLVLgDOzxzbga+1ur8LTPqwfpC4xfRnw7Sh7DBiX\ndGonG/S0D+sHqUtMTwDVBc9fzJYdp9US01vOm+D2y9czMT6GgInxMW6/fL07LqxQOrnbSHWJ6Xng\nu5KuiojvtPN9EbET2Anl8meN1vW0Dyu6pCWmgRJQXZf59GyZ2UBLWmIa2A38cdY7uJHy4eLLHWzT\nrC+kLjH9ILAZOAi8DVzdcYvN+oBLTJstQ6slpgsZLEmHgOcarLIG+FVOzWlH0dsHxW9jUdv3OxHR\n9AZuhQxWM5JmWvm/Rq8UvX1Q/DYWvX3N9N1YQbN+4GCZJdCvwdrZ6wY0UfT2QfHbWPT2NdSX51hm\nRdeveyyzQnOwzBIoVLAkXSzpQDYx8vol3q87cbLZZ3Ns4+ezts1JelTSuVXvPZst3ycpyRXwFtp3\noaQ3sjbsk3RTq5/NsY3bq9r3lKQjkt6fvZf8N+yKiCjEg/JEyf8CPgi8D3gSOLtmnc3AQ4CAjcDj\nrX42xzZuAlZnzy+ptDF7/Sywpse/4YXAD9r5bF5trFn/UuAnef2G3XoUaY+1ATgYEU9HxDvAPZSn\npVSrN3Gylc/m0saIeDQiXs9ePkZ5RH9eOvkdCvMb1rgSuDtBO5IqUrBamRRZb52WJ1Tm0MZqX6C8\nh60I4EeS9kra1sP2bcoOVx+SdM4yP5tXG5G0CrgY+F7V4tS/YVf0XfmzfiHpk5SDdUHV4gsioiTp\nJOCHkv49In6Wc9OeACYj4i1Jm4FpyjVJiuhS4BcR8VrVsiL8hk0VaY/VyqTIeuvkNaGype1I+gjw\nDeCyiHi1sjzK5QyIiFeA+ykfFuXavoh4MyLeyp4/CIxKWtPKZ/NqY5Wt1BwG5vAbdkevT/KqTkpX\nAk9TnupfOak9p2adz3Bs58W/tPrZHNs4SXn+2aaa5ScAv1X1/FHg4h607xTeGxiwgfKEVRXpN8zW\n+23gNeCEPH/Dbj0KcygYEYclXQfsodxzdGdE7G9l4mS9z/aojTcBJwL/UJ5YzeEoj9I+Gbg/W7YS\nuCsiHu5B+64ArpF0GFgAtkb5v9Qi/YYAfwQ8EhG/rvp48t+wWzykySyBIp1jmQ0MB8ssAQfLLAEH\nyywBB8ssAQfLLAEHyyyB/we26e4GZgQSIAAAAABJRU5ErkJggg==\n", 133 | "text/plain": [ 134 | "" 135 | ] 136 | }, 137 | "metadata": {}, 138 | "output_type": "display_data" 139 | } 140 | ], 141 | "source": [ 142 | "plt.scatter(x,y)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": 11, 148 | "metadata": { 149 | "collapsed": true 150 | }, 151 | "outputs": [], 152 | "source": [ 153 | "def sse(y,y_pred): return ((y-y_pred)**2).sum()\n", 154 | "def loss(y,a,b,x): return sse(y, lin(a,b,x))\n", 155 | "def avg_loss(y,a,b,x): return np.sqrt(loss(y,a,b,x)/n)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": 12, 161 | "metadata": {}, 162 | "outputs": [ 163 | { 164 | "data": { 165 | "text/plain": [ 166 | "9.1074" 167 | ] 168 | }, 169 | "execution_count": 12, 170 | "metadata": {}, 171 | "output_type": "execute_result" 172 | } 173 | ], 174 | "source": [ 175 | "a_guess=-1.\n", 176 | "b_guess=1.\n", 177 | "avg_loss(y, a_guess, b_guess, x)" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": 13, 183 | "metadata": { 184 | "collapsed": true 185 | }, 186 | "outputs": [], 187 | "source": [ 188 | "lr=0.01\n", 189 | "# d[(y-(a*x+b))**2,b] = 2 (b + a x - y) = 2 (y_pred - y)\n", 190 | "# d[(y-(a*x+b))**2,a] = 2 x (b + a x - y) = x * dy/db" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 14, 196 | "metadata": { 197 | "collapsed": true 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "def upd():\n", 202 | " global a_guess, b_guess\n", 203 | " \n", 204 | " # make a prediction using the current weights\n", 205 | " y_pred = lin(a_guess, b_guess, x)\n", 206 | " \n", 207 | " # calculate the derivate of the loss\n", 208 | " dydb = 2 * (y_pred - y)\n", 209 | " dyda = x*dydb\n", 210 | " \n", 211 | " # update our weights by moving in direction of steepest descent\n", 212 | " a_guess -= lr*dyda.mean()\n", 213 | " b_guess -= lr*dydb.mean()" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": 15, 219 | "metadata": {}, 220 | "outputs": [ 221 | { 222 | "data": { 223 | "text/html": [ 224 | "" 591 | ], 592 | "text/plain": [ 593 | "" 594 | ] 595 | }, 596 | "execution_count": 15, 597 | "metadata": {}, 598 | "output_type": "execute_result" 599 | } 600 | ], 601 | "source": [ 602 | "fig = plt.figure(dpi=100, figsize=(5, 4))\n", 603 | "plt.scatter(x,y)\n", 604 | "line, = plt.plot(x,lin(a_guess,b_guess,x))\n", 605 | "plt.close()\n", 606 | "\n", 607 | "def animate(i):\n", 608 | " line.set_ydata(lin(a_guess,b_guess,x))\n", 609 | " for i in range(10): upd()\n", 610 | " return line,\n", 611 | "\n", 612 | "ani = animation.FuncAnimation(fig, animate, np.arange(0, 40), interval=100)\n", 613 | "ani" 614 | ] 615 | }, 616 | { 617 | "cell_type": "code", 618 | "execution_count": null, 619 | "metadata": { 620 | "collapsed": true 621 | }, 622 | "outputs": [], 623 | "source": [] 624 | } 625 | ], 626 | "metadata": { 627 | "kernelspec": { 628 | "display_name": "Python [conda root]", 629 | "language": "python", 630 | "name": "conda-root-py" 631 | }, 632 | "language_info": { 633 | "codemirror_mode": { 634 | "name": "ipython", 635 | "version": 3 636 | }, 637 | "file_extension": ".py", 638 | "mimetype": "text/x-python", 639 | "name": "python", 640 | "nbconvert_exporter": "python", 641 | "pygments_lexer": "ipython3", 642 | "version": "3.6.1" 643 | }, 644 | "nav_menu": {}, 645 | "toc": { 646 | "navigate_menu": true, 647 | "number_sections": true, 648 | "sideBar": true, 649 | "threshold": 6, 650 | "toc_cell": true, 651 | "toc_section_display": "block", 652 | "toc_window_display": false 653 | }, 654 | "widgets": { 655 | "state": {}, 656 | "version": "1.1.2" 657 | } 658 | }, 659 | "nbformat": 4, 660 | "nbformat_minor": 1 661 | } 662 | -------------------------------------------------------------------------------- /nbs/images/AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9.url: -------------------------------------------------------------------------------- 1 | [InternetShortcut] 2 | URL=https://photos.google.com/photo/AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9 3 | -------------------------------------------------------------------------------- /nbs/images/Finite_element_sparse_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/Finite_element_sparse_matrix.png -------------------------------------------------------------------------------- /nbs/images/Halide1.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/Halide1.gif -------------------------------------------------------------------------------- /nbs/images/Halide2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/Halide2.gif -------------------------------------------------------------------------------- /nbs/images/Halide3.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/Halide3.gif -------------------------------------------------------------------------------- /nbs/images/InsideImagenet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/InsideImagenet.png -------------------------------------------------------------------------------- /nbs/images/L1vsL2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/L1vsL2.jpg -------------------------------------------------------------------------------- /nbs/images/L1vsL2_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/L1vsL2_2.png -------------------------------------------------------------------------------- /nbs/images/adjaceny_matrix.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/adjaceny_matrix.jpg -------------------------------------------------------------------------------- /nbs/images/adjaceny_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/adjaceny_matrix.png -------------------------------------------------------------------------------- /nbs/images/arnoldi.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/arnoldi.jpg -------------------------------------------------------------------------------- /nbs/images/background.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/background.jpg -------------------------------------------------------------------------------- /nbs/images/block_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/block_matrix.png -------------------------------------------------------------------------------- /nbs/images/bloom_filter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/bloom_filter.png -------------------------------------------------------------------------------- /nbs/images/candes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/candes.png -------------------------------------------------------------------------------- /nbs/images/cholesky_factorization_speed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cholesky_factorization_speed.png -------------------------------------------------------------------------------- /nbs/images/cincinnati_reds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cincinnati_reds.png -------------------------------------------------------------------------------- /nbs/images/cincinnati_reds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cincinnati_reds2.png -------------------------------------------------------------------------------- /nbs/images/cnn1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cnn1.png -------------------------------------------------------------------------------- /nbs/images/cnn2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cnn2.png -------------------------------------------------------------------------------- /nbs/images/cnn3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cnn3.png -------------------------------------------------------------------------------- /nbs/images/ct_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/ct_1.png -------------------------------------------------------------------------------- /nbs/images/cython_vs_numba.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/cython_vs_numba.png -------------------------------------------------------------------------------- /nbs/images/data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/data.png -------------------------------------------------------------------------------- /nbs/images/data_xray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/data_xray.png -------------------------------------------------------------------------------- /nbs/images/digit.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/digit.gif -------------------------------------------------------------------------------- /nbs/images/document_term.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/document_term.png -------------------------------------------------------------------------------- /nbs/images/face_nmf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/face_nmf.png -------------------------------------------------------------------------------- /nbs/images/face_outputs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/face_outputs.png -------------------------------------------------------------------------------- /nbs/images/face_pca.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/face_pca.png -------------------------------------------------------------------------------- /nbs/images/faces_rpca.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/faces_rpca.png -------------------------------------------------------------------------------- /nbs/images/faces_rpca_gross.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/faces_rpca_gross.png -------------------------------------------------------------------------------- /nbs/images/faces_rpca_gross2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/faces_rpca_gross2.png -------------------------------------------------------------------------------- /nbs/images/figures.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/figures.jpg -------------------------------------------------------------------------------- /nbs/images/fltscale-wh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/fltscale-wh.png -------------------------------------------------------------------------------- /nbs/images/fltscale.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 18 | 20 | 38 | 40 | 41 | 43 | image/svg+xml 44 | 46 | 47 | 48 | 49 | 50 | 54 | 59 | 64 | 69 | 74 | 79 | 84 | 89 | 94 | 99 | 104 | 109 | 114 | 119 | 124 | 129 | 134 | 139 | 144 | 149 | 154 | 159 | 164 | 169 | 174 | 179 | 184 | 189 | 194 | 199 | 204 | 209 | 214 | 219 | 224 | 229 | 234 | 239 | 244 | 249 | 254 | 259 | 264 | 269 | 274 | 279 | 284 | 289 | 294 | 299 | 304 | 309 | 314 | 319 | 324 | 329 | 334 | 339 | 344 | 349 | 354 | 359 | 364 | 369 | 374 | 379 | 384 | 389 | 394 | 399 | 404 | 409 | 414 | 419 | 424 | 429 | 434 | 439 | 444 | 449 | 454 | 459 | 464 | 469 | 474 | 479 | 484 | 489 | 494 | 499 | 504 | 509 | 514 | 519 | 524 | 529 | 534 | 539 | 544 | 549 | 554 | 559 | 564 | 569 | 574 | 579 | 584 | 589 | 594 | 599 | 604 | 609 | 614 | 619 | 624 | 629 | 634 | 639 | 644 | 649 | 654 | 659 | 664 | 669 | 674 | 679 | 684 | 689 | 694 | 699 | 704 | 709 | 714 | 719 | 724 | 729 | 734 | 739 | 744 | 749 | 754 | 759 | 764 | 769 | 774 | 779 | 784 | 789 | 794 | 799 | 804 | 809 | 814 | 819 | 824 | 829 | 834 | 839 | 844 | 849 | 854 | 859 | 864 | 869 | 874 | 879 | 884 | 889 | 894 | 899 | 904 | 909 | 914 | 919 | 924 | 929 | 934 | 939 | 944 | 949 | 954 | 959 | 964 | 969 | 974 | 979 | 984 | 989 | 994 | 999 | 1004 | 1009 | 1014 | 1019 | 1024 | 1029 | 1034 | 1039 | 1044 | 1049 | 1054 | 1059 | 1064 | 1069 | 1074 | 1079 | 1084 | 1089 | 1094 | 1099 | 1104 | 1109 | 1114 | 1119 | 1124 | 1129 | 1134 | 1139 | 1144 | 1149 | 1154 | 1159 | 1164 | 1169 | 1174 | 1179 | -8 1190 | 8 1201 | -15.5 1216 | -4 1227 | 4 1238 | 15.5 1249 | -2 1260 | 2 1271 | -1 1282 | 1 1293 | -0.5 1304 | 0.5 1315 | 1326 | 0 1337 | 1343 | denormalised 1354 | -0.015625 0.015625 1365 | 1366 | 1367 | -------------------------------------------------------------------------------- /nbs/images/fpa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/fpa.png -------------------------------------------------------------------------------- /nbs/images/full_qr.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/full_qr.JPG -------------------------------------------------------------------------------- /nbs/images/full_svd.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/full_svd.JPG -------------------------------------------------------------------------------- /nbs/images/graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/graph.png -------------------------------------------------------------------------------- /nbs/images/grid.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/grid.jpg -------------------------------------------------------------------------------- /nbs/images/hermitian_eigen.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/hermitian_eigen.JPG -------------------------------------------------------------------------------- /nbs/images/image1.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/image1.bmp -------------------------------------------------------------------------------- /nbs/images/image1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/image1.jpg -------------------------------------------------------------------------------- /nbs/images/lesson4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/lesson4.png -------------------------------------------------------------------------------- /nbs/images/markov_health.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/markov_health.jpg -------------------------------------------------------------------------------- /nbs/images/matrix_operations.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/matrix_operations.png -------------------------------------------------------------------------------- /nbs/images/nimfa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/nimfa.png -------------------------------------------------------------------------------- /nbs/images/nmf_doc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/nmf_doc.png -------------------------------------------------------------------------------- /nbs/images/nonhermitian_eigen.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/nonhermitian_eigen.JPG -------------------------------------------------------------------------------- /nbs/images/page_rank_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/page_rank_graph.png -------------------------------------------------------------------------------- /nbs/images/pcp_algorithm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/pcp_algorithm.png -------------------------------------------------------------------------------- /nbs/images/pentium_nytimes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/pentium_nytimes.png -------------------------------------------------------------------------------- /nbs/images/prob1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/prob1.png -------------------------------------------------------------------------------- /nbs/images/prob2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/prob2.png -------------------------------------------------------------------------------- /nbs/images/proj.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/proj.png -------------------------------------------------------------------------------- /nbs/images/projection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/projection.png -------------------------------------------------------------------------------- /nbs/images/projection_line.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/projection_line.png -------------------------------------------------------------------------------- /nbs/images/randomizedSVDbenchmarks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/randomizedSVDbenchmarks.png -------------------------------------------------------------------------------- /nbs/images/reduced_qr.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/reduced_qr.JPG -------------------------------------------------------------------------------- /nbs/images/reduced_svd.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/reduced_svd.JPG -------------------------------------------------------------------------------- /nbs/images/rpca_inexact.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/rpca_inexact.png -------------------------------------------------------------------------------- /nbs/images/shop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/shop.png -------------------------------------------------------------------------------- /nbs/images/skinny_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/skinny_image.jpg -------------------------------------------------------------------------------- /nbs/images/sklearn_ct.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/sklearn_ct.png -------------------------------------------------------------------------------- /nbs/images/sparse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/sparse.png -------------------------------------------------------------------------------- /nbs/images/sportspredict.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/sportspredict.jpeg -------------------------------------------------------------------------------- /nbs/images/surveillance3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/surveillance3.png -------------------------------------------------------------------------------- /nbs/images/svd_fb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/svd_fb.png -------------------------------------------------------------------------------- /nbs/images/svddiagram.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/svddiagram.gif -------------------------------------------------------------------------------- /nbs/images/svp_value.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/svp_value.png -------------------------------------------------------------------------------- /nbs/images/top10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/top10.png -------------------------------------------------------------------------------- /nbs/images/what_is_pytorch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/what_is_pytorch.png -------------------------------------------------------------------------------- /nbs/images/whole_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/whole_image.jpg -------------------------------------------------------------------------------- /nbs/images/why_python.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/why_python.png -------------------------------------------------------------------------------- /nbs/images/word2vec_analogies.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/word2vec_analogies.png -------------------------------------------------------------------------------- /nbs/images/word2vec_sexist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/word2vec_sexist.png -------------------------------------------------------------------------------- /nbs/images/xray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/xray.png -------------------------------------------------------------------------------- /nbs/images/zoom.bmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fastai/numerical-linear-algebra/694453e105b2a8f96b62b62d44d234c561eba268/nbs/images/zoom.bmp --------------------------------------------------------------------------------