├── Eigenvalue_conditioning.ipynb ├── Five_point_Laplacian.ipynb ├── Gaussian Elimination.ipynb ├── Gram-Schmidt.ipynb ├── Householder_stability.ipynb ├── Least_squares_algorithms.ipynb ├── Linear_regression_with_auto_data.ipynb ├── Performance modeling.ipynb ├── Power iteration illustrated.ipynb ├── README.md ├── Rayleigh_quotient_iteration.ipynb ├── SVD_image_compression.ipynb └── linear_solve_stability.ipynb /Eigenvalue_conditioning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "metadata": { 3 | "name": "", 4 | "signature": "sha256:63c7eab0c6b8aee3e180ef11dddca021c6ce9555323352c904039978fe74aed7" 5 | }, 6 | "nbformat": 3, 7 | "nbformat_minor": 0, 8 | "worksheets": [ 9 | { 10 | "cells": [ 11 | { 12 | "cell_type": "code", 13 | "collapsed": false, 14 | "input": [ 15 | "%matplotlib inline\n", 16 | "import matplotlib.pyplot as plt\n", 17 | "import numpy as np" 18 | ], 19 | "language": "python", 20 | "metadata": {}, 21 | "outputs": [] 22 | }, 23 | { 24 | "cell_type": "heading", 25 | "level": 1, 26 | "metadata": {}, 27 | "source": [ 28 | "Conditioning of eigenvalue problems" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "The condition of an eigenvalue problem depends strongly on whether the matrix in question is normal. Let's consider a simple non-normal matrix given by \n", 36 | "\n", 37 | "$$tridiag(1/4,0,1).$$\n", 38 | "\n", 39 | "This example is discussed in section 3 of [Spectra and Pseudospectra](http://pup.princeton.edu/titles/8113.html) by Trefethen & Embree." 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "collapsed": false, 45 | "input": [ 46 | "m=32\n", 47 | "v1 = np.ones(m-1)\n", 48 | "A = np.diag(v1,1)+np.diag(0.25*v1,-1)\n", 49 | "v2 = 2**np.arange(1,m+1)\n", 50 | "D = np.diag(v2)\n", 51 | "Dinv = np.diag(1./v2)" 52 | ], 53 | "language": "python", 54 | "metadata": {}, 55 | "outputs": [] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "The eigenvalues of this matrix are simply\n", 62 | "\n", 63 | "$$\\lambda_k = \\cos\\left(\\frac{k\\pi}{m+1}\\right).$$" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "collapsed": false, 69 | "input": [ 70 | "lamda = np.linalg.eigvals(A)\n", 71 | "plt.plot(np.real(lamda),np.imag(lamda),'o');" 72 | ], 73 | "language": "python", 74 | "metadata": {}, 75 | "outputs": [] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "To investigate conditioning, we perturb $A$ by random matrices with norm $10^{-5}$ and compute the eigenvalues of the resulting matrices." 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "collapsed": false, 87 | "input": [ 88 | "lamda = np.linalg.eigvals(A)\n", 89 | "plt.plot(np.real(lamda),np.imag(lamda),'ko')\n", 90 | "\n", 91 | "for i in range(10):\n", 92 | " E = np.random.randn(m,m)\n", 93 | " E = E/np.linalg.norm(E) * 1.e-5\n", 94 | " lamda = np.linalg.eigvals(A+E)\n", 95 | " plt.plot(np.real(lamda),np.imag(lamda),'rs')" 96 | ], 97 | "language": "python", 98 | "metadata": {}, 99 | "outputs": [] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "Clearly, the eigenvalues are perturbed by a much larger amount! In fact, you might be rightly suspicious of whether we computed the eigenvalues of $A$ itself correctly in the first place (the answer is that we did, but if you increase $m$ above to 256 you'll see that roundoff errors lead to large errors in the eigenvalues)." 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "collapsed": false, 111 | "input": [ 112 | "W,V = np.linalg.eig(A)" 113 | ], 114 | "language": "python", 115 | "metadata": {}, 116 | "outputs": [] 117 | }, 118 | { 119 | "cell_type": "code", 120 | "collapsed": false, 121 | "input": [ 122 | "np.log10(np.linalg.cond(V))" 123 | ], 124 | "language": "python", 125 | "metadata": {}, 126 | "outputs": [] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "collapsed": false, 131 | "input": [ 132 | "B=np.dot(D,np.dot(A,Dinv))\n", 133 | "lamda = np.linalg.eigvals(A)\n", 134 | "plt.plot(np.real(lamda),np.imag(lamda),'o');" 135 | ], 136 | "language": "python", 137 | "metadata": {}, 138 | "outputs": [] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "collapsed": false, 143 | "input": [ 144 | "lamda = np.linalg.eigvals(B)\n", 145 | "plt.plot(np.real(lamda),np.imag(lamda),'ko')\n", 146 | "\n", 147 | "for i in range(10):\n", 148 | " E = np.random.randn(m,m)\n", 149 | " E = E/np.linalg.norm(E) * 1.e-5\n", 150 | " lamda = np.linalg.eigvals(B+E)\n", 151 | " plt.plot(np.real(lamda),np.imag(lamda),'rs')" 152 | ], 153 | "language": "python", 154 | "metadata": {}, 155 | "outputs": [] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "collapsed": false, 160 | "input": [], 161 | "language": "python", 162 | "metadata": {}, 163 | "outputs": [] 164 | } 165 | ], 166 | "metadata": {} 167 | } 168 | ] 169 | } -------------------------------------------------------------------------------- /Five_point_Laplacian.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline\n", 12 | "import numpy as np\n", 13 | "import scipy.sparse\n", 14 | "import matplotlib.pyplot as plt" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "# The 5-point Laplacian\n", 22 | "The function below sets up a matrix that approximates the Laplacian operator in two dimensions:\n", 23 | "\n", 24 | "\\begin{align}\n", 25 | "\\nabla^2 u = u_{xx} + u_{yy}.\n", 26 | "\\end{align}\n", 27 | "\n", 28 | "The formula for the finite difference approximation is\n", 29 | "\n", 30 | "\\begin{align}\n", 31 | "\\nabla^2 u(x_i,y_j) \\approx \\frac{U_{i+1,j} + U_{i-1,j} + U_{i,j+1} + U_{i,j-1} - 4 U_{ij}}{h^2} = AU.\n", 32 | "\\end{align}\n", 33 | "\n", 34 | "Here we construct the matrix $A$, assuming that $U$ is order row-wise. The constant $h$ is the grid spacing (which is assumed to be the same in the $x$ and $y$ directions). The domain is assumed to be the square $[a,b]^2$ and it is discretized using $m$ points (not counting boundary values) in each direction." 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": 2, 40 | "metadata": { 41 | "collapsed": false 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "def five_pt_laplacian_sparse(m,a,b):\n", 46 | " \"\"\"Construct a sparse finite difference matrix that approximates the Laplacian.\"\"\"\n", 47 | " e=np.ones(m**2)\n", 48 | " e2=([1]*(m-1)+[0])*m\n", 49 | " e3=([0]+[1]*(m-1))*m\n", 50 | " h=(b-a)/(m+1)\n", 51 | " A=scipy.sparse.spdiags([-4*e,e2,e3,e,e],[0,-1,1,-m,m],m**2,m**2)\n", 52 | " A/=h**2\n", 53 | " A = A.todia()\n", 54 | " return A" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "In general, this matrix has the following form:\n", 62 | "\n", 63 | "\\begin{align}\n", 64 | " A = \\frac{1}{h^2}\\begin{bmatrix} T & I & & & \\\\ I & T & I & & \\\\ & I & T & I & \\\\ & & \\ddots & \\ddots & \\ddots \\\\ & & & I & T\n", 65 | " \\end{bmatrix}\n", 66 | "\\end{align}\n", 67 | "\n", 68 | "where $I$ is the $m \\times m$ identity matrix and $T$ is a tridiagonal $m\\times m$ matrix:\n", 69 | "\n", 70 | "\\begin{align}\n", 71 | " T = \\begin{bmatrix} -4 & 1 & & & \\\\ 1 & -4 & 1 & & \\\\ & 1 & -4 & 1 & \\\\ & & \\ddots & \\ddots & \\ddots \\\\ & & & 1 & -4\n", 72 | " \\end{bmatrix}\n", 73 | "\\end{align}" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "This structure can be seen using the `spy` function, which shows where the nonzero entries are:" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 4, 86 | "metadata": { 87 | "collapsed": false 88 | }, 89 | "outputs": [ 90 | { 91 | "data": { 92 | "text/plain": [ 93 | "" 94 | ] 95 | }, 96 | "execution_count": 4, 97 | "metadata": {}, 98 | "output_type": "execute_result" 99 | }, 100 | { 101 | "data": { 102 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPwAAAD7CAYAAABOrvnfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADrtJREFUeJzt3WusZXV5x/HvU8DWKSmENI5aJhwuxVhiiZQaQmuYWkim\nlIIvDEI0wpD4qrZoDBQwqedlK1GxbXihXApR6AUJoQmmDNoxNqQEcBimDBQRRxkMM0YEYxMFOk9f\n7DUnZ47nus76r33W+n8/yUn2Ze39/Neeec66nPXb/8hMJNXhV6Y9AEn9seGlitjwUkVseKkiNrxU\nERteqkjvDR8R2yLimYj4TkT8VeFaWyLiPyLiqYj474j4y5L1mppHRcSuiPi3HmodHxH3RMTTEbE3\nIs4pXO/65rPcExF3RcSvdvz+t0XEgYjYM++xEyJiR0Q8GxEPRsTxhevd2HyeuyPi3og4rmS9ec99\nMiIORcQJXdVbTK8NHxFHAf8AbAN+B7g8It5ZsOTrwCcy8wzgHODPC9cDuBrYC/RxgcMXgAcy853A\n7wJPlyoUETPAR4GzMvNdwFHAZR2XuZ3J/435rgN2ZObpwNeb+yXrPQickZlnAs8C1xeuR0RsAS4A\nvt9hrUX1vYV/D/BcZu7LzNeBfwIuKVUsM1/KzCea2z9j0hBvL1UvIk4ELgRuAaJUnabWccB7M/M2\ngMx8IzNfLVjyp0x+gW6KiKOBTcCLXRbIzG8BP1nw8MXAHc3tO4D3l6yXmTsy81Bz9xHgxJL1Gp8D\nru2qznL6bvjfAl6Yd39/81hxzRbq3Uz+EUv5PHANcGilBTtwMvCjiLg9Ir4dEV+KiE2limXmy8Bn\ngR8APwReycyHStWbZ3NmHmhuHwA291DzsKuAB0oWiIhLgP2Z+WTJOof13fBTuY43Io4F7gGubrb0\nJWpcBBzMzF0U3ro3jgbOAm7OzLOA/6Xb3d0jRMSpwMeBGSZ7ScdGxIdK1VtMTq4D7+X/UER8Cngt\nM+8qWGMTcAPw6fkPl6oH/Tf8i8CWefe3MNnKFxMRxwBfBb6cmfcVLHUucHFEfA+4G3hfRNxZsN5+\nJluGR5v79zD5BVDK2cDDmfnjzHwDuJfJOpd2ICLeChARbwMOli4YEVcyOTQr/QvtVCa/QHc3/29O\nBB6PiLeUKth3wz8G/HZEzETEm4APAveXKhYRAdwK7M3Mm0rVAcjMGzJzS2aezORk1jcy8yMF670E\nvBARpzcPnQ88Vaoe8AxwTkS8uflcz2dycrK0+4ErmttXACV/aRMR25gcll2SmT8vWSsz92Tm5sw8\nufl/s5/JSdFyv9Qys9cf4E+A/wGeA64vXOsPmRxPPwHsan629bCO5wH391DnTOBRYDeTLe5xhetd\ny+SXyh4mJ9CO6fj972ZyfuA1Jud6tgMnAA8xOWP+IHB8wXpXAd9hcrb88P+XmwvU+8Xh9Vvw/PPA\nCSX/DaMpJKkCXmknVcSGlypiw0sVseGlihxd4k0jwjOB0hRl5qIX8LiFl2pS6O+1SXMl5BLPz5b8\nW6P1rDeUel3XWqn3Wu/SN1ck3cQkJnlLZv7t4sud/Y9wysyRj542E3Hp1iMfe35f5mNXLv+6xRz5\nur61XT9pGlo1/Lxc+/lMro9/NCLuz8xF8tinzMC/nHfkY7PA7ElHPnbpKl63mIWv61vb9ZP61/YY\nfp259q0ty7a2s99yW/st1/v6WW+gtVo3/Dpz7Vtblm0nM3f2WnDk62e9YdaC9g3vn92kAWp70m5V\nufaImIXTZibHtFuZwq6uNHoRsZVVNlfbhp/LtTOJ+30QuHzhQpk5OzlbvfAElqSuNIcFOwEi4tPL\nLduq4TPzjYj4GPDvTP4sd+viZ+glbSSt/w6fmV8DvtbhWCQV5qW1UkWKhGeO9Py+1V108vy+bl7X\nt6GMU6LMV1wdTsvlEokdSWWs1Hvu0ksVseGlihQ/hm+beuv7dX0byjg1Lj2ctGubeuv7dX0byjg1\nJu7SSxWx4aWK2PBSRWx4qSI2vFQRG16qiA0vVcSGlyoywrTcUNJrQxmnxsS0nDQipuUkzelhl74d\nQzeLG8o4tTGtZ265LcCdwFuYfE/9FzPz77oamKGbpQxlnNqI1rOFfx34RGY+ERHHAo9HxA6/vVba\nuFofw2fmS5n5RHP7Z8DTwNu7Gpik7nVy0q6ZkOLdwCNdvJ+kMtZ90q7Znb8HuLrZ0s9/bnbe3Z39\nT+oojV8fU00dLnQM8FXgy5l538LnM3N2Pe8vaWVrmWqq9S59RARwK7A3M29q+z6S+rOeY/g/AD4M\n/FFE7Gp+tnU0LkkFrGduuf/EK/WkQdmwV9oZulmu/hDGqY3I8Iw0IoZnJM2x4aWKbOBj+HbGnpZr\na+zrp9UZXcOPPy3X1tjXT6vhLr1UERteqogNL1XEhpcqYsNLFbHhpYrY8FJFbHipIiO88Gbsabm2\nxr5+Wg3TctKImJaTNMeGlypS/Bh+7CmtsafzhjJOrc56v6b6KOAxYH9m/tniS409pTX2dN5QxqnV\nWO8u/dXAXiaTSUra4NbzvfQnAhcCtwCejZcGYD1b+M8D1wCHOhqLpMJaHcNHxEXAwczc1cxrtdRy\ns3DaDMwymfpqyUUltdTH3HLnAhdHxIXArwG/ERF3ZuZH5i+UmbMRl26F2ZNa1pG0guJzy2XmDZm5\nJTNPBi4DvrGw2SVtPF1deONZemkA1n3hTWZ+E/hmB2ORVFgPabmxp7TGns4byji1GqblpBExLSdp\njg0vVWTDfuPNUFJafaflxv65qKwN2/DDSWn1nZYb++eiktyllypiw0sVseGlitjwUkVseKkiNrxU\nERteqogNL1VkA194M5SUVt9pubF/LirJtJw0IqblJM2x4aWKbOBj+HaGktIyZbe4oYxzqFo3fEQc\nz2TWmTOYfInlVZn5X10NrL2hpLRM2S1uKOMcpvVs4b8APJCZH4iIo4Ff72hMkgppO/PMccB7M/MK\ngMx8A3i1y4FJ6l7bk3YnAz+KiNsj4tsR8aWI2NTlwCR1r+0u/dHAWcDHMvPRiLgJuA746/kLTeaW\nm7OzmRJHUof6mFtuP7A/Mx9t7t/DpOGPkJmzLd9f0ir1MbfcS8ALEXF689D5wFNt3ktSf9Zzlv4v\ngK9ExJuA7wLbuxmSpFJaN3xm7gZ+v8OxSCpsdFfaDSelZcpu6fpDGOcwmZaTRsS0nKQ5NrxUkREe\nw7czlJTW2NNybY19/bpiw88ZSkpr7Gm5tsa+ft1wl16qiA0vVcSGlypiw0sVseGlitjwUkVseKki\nNrxUES+8mTOUlNbY03JtjX39umFaThoR03KS5tjwUkU8hl+nsae0xp7OG8o4u7KeueWuBz4MHAL2\nANsz8xddDWw4xp7SGns6byjj7EarXfqImAE+CpyVme8CjgIu625Ykkpou4X/KfA6sCki/g/YBLzY\n2agkFdF2IoqXgc8CPwB+CLySmQ91OTBJ3Ws7e+ypwMeBGSazxv5rRHwoM7+yYLnZeXedW04qoI+5\n5c4GHs7MHzcF7wXOBY5oeOeWk8orPrcc8AxwTkS8OSKCydxye1u+l6SetD2G3w3cCTwGPNk8/MWu\nBiWpjPXMLfcZ4DMdjkVSYV5pt25jT2mNPZ03lHF2w7ScNCKm5STNseGlingMPyVDSWn1nZYb++cy\nbTb81AwlpdV3Wm7sn8t0uUsvVcSGlypiw0sVseGlitjwUkVseKkiNrxUERteqogX3kzNUFJafafl\nxv65TJdpOWlETMtJmmPDSxXxGH5ghpLSMmX3yzbCGJdt+Ii4DfhT4GAzpRQRcQLwz8BJwD7g0sx8\npcTgtJihpLRM2f2y6Y9xpV3624FtCx67DtiRmacDX2/uSxqAZRs+M78F/GTBwxcDdzS37wDeX2Bc\nkgpoc9Juc2YeaG4fADZ3OB5JBa3rpF1m5uG/+y3GueWk8krPLXcgIt6amS9FxNuAg0st6NxyUnml\n55a7H7iiuX0FcF+L95A0Bcs2fETcDTwMvCMiXoiI7cDfABdExLPA+5r7kgZg2V36zLx8iafOLzAW\nSYV5pd3gDCWlZcpu8drTHaNpOWlETMtJmmPDSxXxGL4SGyGptRpjTsu1tZZ1W2kJG74a009qrc6Y\n03JtrWXdHl92CXfppYrY8FJFbHipIja8VBEbXqqIDS9VxIaXKmLDSxXxwptqTD+ptTpjTsu1taZ1\nW/YCHdNy0oiYlpM0x4aXKuIxvJY15hQa1JfOW7Hhl5hf7kbgIuA14LvA9sx8teRANS1jTqFBbem8\n1ezSLza/3IPAGZl5JvAscH3XA5PUvRUbfrH55TJzR2Yeau4+ApxYYGySOtbFSburgAc6eB9Jha3r\npF1EfAp4LTPvWuL52Xl3nVtOKqD03HKHi1wJXAj88VLLOLecVN5a5pZr1fARsQ24BjgvM3/e5j0k\n9W/FY/hF5pe7Cvh74FhgR0TsioibC49TUgdW3MIvMb/cbQXGIqkwr7TTCsacQoPa0nmm5aQRMS0n\naY4NL1XEY3gVMZQ0WZ9puY3wmdjwKmQoabI+03LT/0zcpZcqYsNLFbHhpYrY8FJFbHipIja8VBEb\nXqqIDS9VxAtvVMhQ0mR9puWm/5mYlpNGxLScpDk2vFSRZY/hF5tmat5znwRuBH4zM18uN0TVZCMk\nylajz7TcWl630hIrnbS7nckXVt45/8GI2AJcAHx/5UFIazH9RNnq9JmWW8vrHl92iWV36RebZqrx\nOeDalQcgaSNZ8zF8RFwC7M/MJwuMR1JBa/o7fERsAm5gsjs/93CnI5JUzFovvDkVmAF2RwRMZo19\nPCLek5kHFy7s3HJSH3Y2P7tmVlpyTQ2fmXuAzYfvR8T3gN9b6iy9c8tJfdja/OzdB8+dtNySyx7D\nz5tm6vRmmqntCxbp/jI9ScUsu4VfYpqp+c+f0u1wJJXklXZSRUzLaYOZfqJsdfpMy63pdcteoGNa\nThoR03KS5tjwUkU8htcoDCFlN4S0nDQQQ0jZbfC0nKRxseGlitjwUkVseKkiNrxUERteqshUGj4i\ntlrPetaDyRdX9GdaW/it1rOe9aDvhvfCG43EEFJ2i41x18zkm2oWLrfS65Z6/ymm5SRNh2k5SWW2\n8JI2JrfwUkVseKkiNrxUERteqogNL1Xk/wETjwUC0E67IwAAAABJRU5ErkJggg==\n", 103 | "text/plain": [ 104 | "" 105 | ] 106 | }, 107 | "metadata": {}, 108 | "output_type": "display_data" 109 | } 110 | ], 111 | "source": [ 112 | "A = five_pt_laplacian_sparse(4,-1.,1.)\n", 113 | "plt.spy(A)" 114 | ] 115 | } 116 | ], 117 | "metadata": { 118 | "kernelspec": { 119 | "display_name": "Python 2", 120 | "language": "python", 121 | "name": "python2" 122 | }, 123 | "language_info": { 124 | "codemirror_mode": { 125 | "name": "ipython", 126 | "version": 2 127 | }, 128 | "file_extension": ".py", 129 | "mimetype": "text/x-python", 130 | "name": "python", 131 | "nbconvert_exporter": "python", 132 | "pygments_lexer": "ipython2", 133 | "version": "2.7.9" 134 | } 135 | }, 136 | "nbformat": 4, 137 | "nbformat_minor": 0 138 | } 139 | -------------------------------------------------------------------------------- /Gaussian Elimination.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "%matplotlib inline\n", 12 | "import numpy as np" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# Gaussian elimination" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "Here is an implementation of Gaussian elimination without pivoting." 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": 2, 32 | "metadata": { 33 | "collapsed": false 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "def LU(A):\n", 38 | " \"\"\"Factor a square matrix A into triangular matrices L, U via Gaussian elimination.\"\"\"\n", 39 | " m = A.shape[0]\n", 40 | " U = A.copy()\n", 41 | " L = np.eye(m)\n", 42 | " for j in range(m):\n", 43 | " for i in range(j+1,m):\n", 44 | " L[i,j] = U[i,j]/U[j,j]\n", 45 | " U[i,:] = U[i,:] - L[i,j]*U[j,:]\n", 46 | " return L, U" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "Let's test it on a $3 \\times 3$ matrix:" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 3, 59 | "metadata": { 60 | "collapsed": false 61 | }, 62 | "outputs": [ 63 | { 64 | "name": "stdout", 65 | "output_type": "stream", 66 | "text": [ 67 | "[[ 1. 0. 0.]\n", 68 | " [ 2. 1. 0.]\n", 69 | " [ 1. 2. 1.]]\n", 70 | "[[ 1. 2. 1.]\n", 71 | " [ 0. 1. 1.]\n", 72 | " [ 0. 0. 6.]]\n" 73 | ] 74 | } 75 | ], 76 | "source": [ 77 | "A = np.array([[1.,2,1],[2,5,3],[1,4,9]])\n", 78 | "L, U = LU(A)\n", 79 | "print(L)\n", 80 | "print(U)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "The factors are triangular, as desired. Does their product give $A$?" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 4, 93 | "metadata": { 94 | "collapsed": false 95 | }, 96 | "outputs": [ 97 | { 98 | "data": { 99 | "text/plain": [ 100 | "array([[ 0., 0., 0.],\n", 101 | " [ 0., 0., 0.],\n", 102 | " [ 0., 0., 0.]])" 103 | ] 104 | }, 105 | "execution_count": 4, 106 | "metadata": {}, 107 | "output_type": "execute_result" 108 | } 109 | ], 110 | "source": [ 111 | "A-np.dot(L,U)" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "Yes, it does.\n", 119 | "\n", 120 | "Next, let us investigate backward stability of this algorithm. If Gaussian elimination is backward stable, then our computed factors $L, U$ should satisfy\n", 121 | "\n", 122 | "$$LU = A + \\delta A$$\n", 123 | "\n", 124 | "for some matrix $\\delta A$ such that $\\|\\delta A\\|/\\|A\\| \\approx \\epsilon_{machine} \\approx 10^{-16}$. In other words, we should have\n", 125 | "\n", 126 | "$$\\frac{\\|LU - A \\|}{\\|A\\|} \\approx 10^{-16}.$$\n", 127 | "\n", 128 | "The following code computes this quantity for 100 random matrices of size $50 \\times 50$, and stops if it finds one that substantially violates the condition above." 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 6, 134 | "metadata": { 135 | "collapsed": false 136 | }, 137 | "outputs": [ 138 | { 139 | "name": "stdout", 140 | "output_type": "stream", 141 | "text": [ 142 | "1.41094330347e-11\n", 143 | "Random matrix 8 produced a backward error of 1.41094330347e-11\n", 144 | "1.41094330347e-11\n" 145 | ] 146 | } 147 | ], 148 | "source": [ 149 | "m = 500\n", 150 | "l = []\n", 151 | "for i in range(100):\n", 152 | " A = np.random.randn(m,m)\n", 153 | " L, U = LU(A)\n", 154 | " rel_err = np.linalg.norm(A-np.dot(L,U))/np.linalg.norm(A)\n", 155 | " l.append(rel_err)\n", 156 | " if rel_err > 1e-11:\n", 157 | " print(rel_err)\n", 158 | " print(\"Random matrix \"+str(i+1)+\" produced a backward error of \" + str(rel_err))\n", 159 | " break\n", 160 | "print(np.max(np.abs(l)))" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "Play around with the tolerance and the matrix size above.\n", 168 | "\n", 169 | "Clearly, Gaussian elimination (without pivoting) is not backward stable." 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "# Partial pivoting" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "What goes wrong in Gaussian elimination? Sometimes, the diagonal entry in the next column is very small. That means that the multipliers used to cancel the remaining entries in that column will be very large, leading to an amplification of roundoff errors. This can be alleviated via *pivoting*, which means choosing the largest entry in the next column and moving it to the diagonal.\n", 184 | "\n", 185 | "Here is an implementation of Gaussian Elimination with partial pivoting." 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": 7, 191 | "metadata": { 192 | "collapsed": false 193 | }, 194 | "outputs": [], 195 | "source": [ 196 | "def PLU(A):\n", 197 | " \"\"\"Factor a square matrix via Gaussian elimination with partial pivoting.\"\"\"\n", 198 | " m = A.shape[0]\n", 199 | " U = A.copy()\n", 200 | " L = np.eye(m)\n", 201 | " P = np.eye(m)\n", 202 | " for j in range(m):\n", 203 | " ii = np.argmax(np.abs(U[j:,j]))+j\n", 204 | " #temp = U[j,:].copy(); U[j,:] = U[ii,:]; U[ii,:] = temp # This would also work.\n", 205 | " U[[j,ii],:] = U[[ii,j],:] # Swap two rows via advanced slicing!\n", 206 | " P[[j,ii],:] = P[[ii,j],:]\n", 207 | " L[[j,ii],:j] = L[[ii,j],:j]\n", 208 | " \n", 209 | " for i in range(j+1,m):\n", 210 | " L[i,j] = U[i,j]/U[j,j]\n", 211 | " U[i,:] = U[i,:] - L[i,j]*U[j,:]\n", 212 | " return P, L, U" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "Let's check that it works:" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 8, 225 | "metadata": { 226 | "collapsed": false 227 | }, 228 | "outputs": [ 229 | { 230 | "name": "stdout", 231 | "output_type": "stream", 232 | "text": [ 233 | "[[ 1. 0. 0. ]\n", 234 | " [ 0.5 1. 0. ]\n", 235 | " [ 0.5 -0.33333333 1. ]]\n", 236 | "[[ 2. 5. 3. ]\n", 237 | " [ 0. 1.5 7.5]\n", 238 | " [ 0. 0. 2. ]]\n", 239 | "[[ 0. 1. 0.]\n", 240 | " [ 0. 0. 1.]\n", 241 | " [ 1. 0. 0.]]\n", 242 | "[[ 1. 4. 9.]\n", 243 | " [ 1. 2. 1.]\n", 244 | " [ 2. 5. 3.]]\n" 245 | ] 246 | } 247 | ], 248 | "source": [ 249 | "A = np.array([[1.,2,1],[2,5,3],[1,4,9]])\n", 250 | "P, L, U = PLU(A)\n", 251 | "print(L)\n", 252 | "print(U)\n", 253 | "print(P)\n", 254 | "print(np.dot(P,np.dot(L,U)))" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "The outputs should satisfy $PA = LU$:" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 9, 267 | "metadata": { 268 | "collapsed": false 269 | }, 270 | "outputs": [ 271 | { 272 | "name": "stdout", 273 | "output_type": "stream", 274 | "text": [ 275 | "[[ 2. 5. 3.]\n", 276 | " [ 1. 4. 9.]\n", 277 | " [ 1. 2. 1.]]\n", 278 | "[[ 2. 5. 3.]\n", 279 | " [ 1. 4. 9.]\n", 280 | " [ 1. 2. 1.]]\n" 281 | ] 282 | } 283 | ], 284 | "source": [ 285 | "print np.dot(P,A)\n", 286 | "print np.dot(L,U)" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": {}, 292 | "source": [ 293 | "Now, lets see how large the backward error is with pivoting:" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": 10, 299 | "metadata": { 300 | "collapsed": false 301 | }, 302 | "outputs": [], 303 | "source": [ 304 | "m = 100\n", 305 | "all_rel_errs = []\n", 306 | "for i in range(100):\n", 307 | " A = np.random.randn(m,m)\n", 308 | " P, L, U = PLU(A)\n", 309 | " rel_err = np.linalg.norm(np.dot(P,A)-np.dot(L,U))/np.linalg.norm(A)\n", 310 | " all_rel_errs.append(rel_err)\n", 311 | " if rel_err > 1e-15:\n", 312 | " print(rel_err)\n", 313 | " print(i)\n", 314 | " break" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 11, 320 | "metadata": { 321 | "collapsed": false 322 | }, 323 | "outputs": [ 324 | { 325 | "name": "stdout", 326 | "output_type": "stream", 327 | "text": [ 328 | "8.08395876658e-16\n" 329 | ] 330 | } 331 | ], 332 | "source": [ 333 | "print(np.max(np.array(rel_err)))" 334 | ] 335 | }, 336 | { 337 | "cell_type": "markdown", 338 | "metadata": {}, 339 | "source": [ 340 | "# A final example" 341 | ] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": {}, 346 | "source": [ 347 | "What's going on with this special matrix?" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": 12, 353 | "metadata": { 354 | "collapsed": false 355 | }, 356 | "outputs": [], 357 | "source": [ 358 | "m = 70\n", 359 | "A = np.zeros((m,m))\n", 360 | "for i in range(m):\n", 361 | " for j in range(m):\n", 362 | " if i==j:\n", 363 | " A[i,j] = 1.\n", 364 | " elif i>j:\n", 365 | " A[i,j] = -1.\n", 366 | " elif j==m-1:\n", 367 | " A[i,j] = 1." 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": 13, 373 | "metadata": { 374 | "collapsed": false 375 | }, 376 | "outputs": [ 377 | { 378 | "data": { 379 | "text/plain": [ 380 | "1296.7103487907095" 381 | ] 382 | }, 383 | "execution_count": 13, 384 | "metadata": {}, 385 | "output_type": "execute_result" 386 | } 387 | ], 388 | "source": [ 389 | "P, L, U = PLU(A)\n", 390 | "np.linalg.norm(np.dot(P,A)-np.dot(L,U))/np.linalg.norm(A)" 391 | ] 392 | } 393 | ], 394 | "metadata": { 395 | "kernelspec": { 396 | "display_name": "Python 2", 397 | "language": "python", 398 | "name": "python2" 399 | }, 400 | "language_info": { 401 | "codemirror_mode": { 402 | "name": "ipython", 403 | "version": 2 404 | }, 405 | "file_extension": ".py", 406 | "mimetype": "text/x-python", 407 | "name": "python", 408 | "nbconvert_exporter": "python", 409 | "pygments_lexer": "ipython2", 410 | "version": "2.7.11" 411 | } 412 | }, 413 | "nbformat": 4, 414 | "nbformat_minor": 0 415 | } 416 | -------------------------------------------------------------------------------- /Gram-Schmidt.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import numpy as np\n", 12 | "%matplotlib inline\n", 13 | "np.set_printoptions(precision=3) # So printed matrices aren't huge\n", 14 | "np.set_printoptions(suppress=True) # Don't use scientific notation" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "# QR Factorization by the Gram-Schmidt algorithm" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "Let's look at the accuracy of the classical and modified Gram-Schmidt algorithms (see, e.g. Trefethen & Bau lecture 7). Here is an implementation of the classical Gram-Schmidt algorithm; notice that it uses more storage than necessary." 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": { 35 | "collapsed": false 36 | }, 37 | "outputs": [], 38 | "source": [ 39 | "def clgs(A):\n", 40 | " \"QR factorization by the classical Gram-Schmidt algorithm.\"\n", 41 | " n = A.shape[1] # Number of columns of A\n", 42 | " R = np.zeros([n,n])\n", 43 | " V = np.zeros(A.shape)\n", 44 | " Q = np.zeros(A.shape)\n", 45 | " \n", 46 | " for j in range(n): # loop over columns of R\n", 47 | " V[:,j] = A[:,j]\n", 48 | " for i in range(j): # Make jth column of V orthogonal to ith column of Q\n", 49 | " R[i,j] = np.dot(Q[:,i].T,A[:,j])\n", 50 | " V[:,j] = V[:,j] - R[i,j]*Q[:,i]\n", 51 | " \n", 52 | " R[j,j] = np.linalg.norm(V[:,j],2)\n", 53 | " Q[:,j] = V[:,j]/R[j,j] # Normalize jth column of V to get jth column of Q\n", 54 | " \n", 55 | " return Q, R" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "Now, we'll generate a random $4 \\times 4$ matrix:" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": { 69 | "collapsed": false 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "n = 4\n", 74 | "A = np.random.rand(n,n)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "Let's factor it:" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": false 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "Q, R = clgs(A)\n", 93 | "print('Q =')\n", 94 | "print(Q)\n", 95 | "print('R =')\n", 96 | "print(R)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "Clearly $R$ is upper triangular. Is $Q$ unitary? We can check by computing $I-Q^T Q$, which should be zero:" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": { 110 | "collapsed": false 111 | }, 112 | "outputs": [], 113 | "source": [ 114 | "print(np.max(np.abs(np.dot(Q.T,Q)-np.eye(n))))" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "Not bad. We can also check our factorization by computing $QR-A$:" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": { 128 | "collapsed": false 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "print(np.max(np.abs(np.dot(Q,R)-A)))" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "That looks pretty good. " 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "# Instability of classical Gram-Schmidt" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "Now let's try it on a different matrix. The function below sets up a Hilbert matrix; these matrices are notoriously ill-conditioned (i.e., close to singular)." 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": null, 159 | "metadata": { 160 | "collapsed": false 161 | }, 162 | "outputs": [], 163 | "source": [ 164 | "def Hilbert(n):\n", 165 | " \"\"\"Return the n x n Hilbert matrix.\"\"\"\n", 166 | " A = np.zeros([n,n])\n", 167 | " for i in range(n):\n", 168 | " for j in range(n):\n", 169 | " A[i,j] = 1./(i+j+1)\n", 170 | " return A" 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "Now let's set up and factor a modest-sized Hilbert matrix:" 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": null, 183 | "metadata": { 184 | "collapsed": false 185 | }, 186 | "outputs": [], 187 | "source": [ 188 | "n = 10\n", 189 | "H = Hilbert(n)\n", 190 | "print(H)" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": { 197 | "collapsed": false 198 | }, 199 | "outputs": [], 200 | "source": [ 201 | "Q, R = clgs(H)" 202 | ] 203 | }, 204 | { 205 | "cell_type": "markdown", 206 | "metadata": {}, 207 | "source": [ 208 | "How good is our factorization? Let's check." 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "collapsed": false 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "print('QR-H')\n", 220 | "print(np.max(np.abs(np.dot(Q,R)-H)))\n", 221 | "print(\"Q*Q-I\")\n", 222 | "print(np.max(np.abs(np.dot(Q.T,Q)-np.eye(n))))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "Whoa! We still have an accurate factorization of $A$, but $Q$ isn't even close to orthogonal:" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "metadata": { 236 | "collapsed": false 237 | }, 238 | "outputs": [], 239 | "source": [ 240 | "print(np.dot(Q.T,Q)-np.eye(n))" 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "metadata": {}, 246 | "source": [ 247 | "# Modified Gram-Schmidt (to the rescue!)" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": {}, 253 | "source": [ 254 | "Let's try the modified algorithm on the same Hilbert matrix. Here is the modified algorithm (again, using more storage than necessary):" 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "metadata": { 261 | "collapsed": false 262 | }, 263 | "outputs": [], 264 | "source": [ 265 | "def mgs(A):\n", 266 | " \"QR factorization by the modified Gram-Schmidt algorithm.\"\n", 267 | " n = A.shape[1]\n", 268 | " R = np.zeros([n,n])\n", 269 | " V = np.zeros(A.shape)\n", 270 | " Q = np.zeros(A.shape)\n", 271 | " for i in range(n):\n", 272 | " V[:,i] = A[:,i]\n", 273 | " for i in range(n):\n", 274 | " R[i,i] = np.linalg.norm(V[:,i],2)\n", 275 | " Q[:,i] = V[:,i]/R[i,i]\n", 276 | " for j in range(i,n):\n", 277 | " R[i,j] = np.dot(Q[:,i].T,V[:,j])\n", 278 | " V[:,j] = V[:,j] - R[i,j]*Q[:,i]\n", 279 | " return Q, R" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": { 286 | "collapsed": false 287 | }, 288 | "outputs": [], 289 | "source": [ 290 | "Q2, R2 = mgs(H)" 291 | ] 292 | }, 293 | { 294 | "cell_type": "markdown", 295 | "metadata": {}, 296 | "source": [ 297 | "How good is our factorization?" 298 | ] 299 | }, 300 | { 301 | "cell_type": "code", 302 | "execution_count": null, 303 | "metadata": { 304 | "collapsed": false 305 | }, 306 | "outputs": [], 307 | "source": [ 308 | "print('QR-H')\n", 309 | "print(np.max(abs(np.dot(Q2,R2)-H)))\n", 310 | "print(\"Q*Q-I\")\n", 311 | "print(np.max(abs(np.dot(Q2.T,Q2)-np.eye(n))))" 312 | ] 313 | }, 314 | { 315 | "cell_type": "markdown", 316 | "metadata": {}, 317 | "source": [ 318 | "The orthogonality of $Q$ is far from perfect, but much better than what we got from the classical algorithm." 319 | ] 320 | }, 321 | { 322 | "cell_type": "markdown", 323 | "metadata": {}, 324 | "source": [ 325 | "We can also see the inaccuracy of cGS if we look at the diagonal entries of $R$, which approximate the singular values of $A$ (see Trefethen & Bau lecture 9):" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": null, 331 | "metadata": { 332 | "collapsed": false 333 | }, 334 | "outputs": [], 335 | "source": [ 336 | "import matplotlib.pyplot as plt\n", 337 | "plt.figure(figsize=(8,4))\n", 338 | "plt.semilogy(np.diag(R),'r',linewidth=3)\n", 339 | "plt.hold(True)\n", 340 | "plt.semilogy(np.diag(R2),'ok');\n", 341 | "plt.legend(('Classical','Modified'));\n", 342 | "plt.title('Diagonal entries of R');" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "# Householder transformations are even better" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "Numpy's built-in QR function uses Householder triangularization:" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": { 363 | "collapsed": false 364 | }, 365 | "outputs": [], 366 | "source": [ 367 | "Q3, R3 = np.linalg.qr(H)" 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": null, 373 | "metadata": { 374 | "collapsed": false 375 | }, 376 | "outputs": [], 377 | "source": [ 378 | "print(np.max(np.abs(np.dot(Q3,R3)-H)))\n", 379 | "print(np.max(np.abs(np.dot(Q3.T,Q3)-np.eye(n))))" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "metadata": { 385 | "collapsed": false 386 | }, 387 | "source": [ 388 | "Impressive!" 389 | ] 390 | } 391 | ], 392 | "metadata": { 393 | "kernelspec": { 394 | "display_name": "Python 2", 395 | "language": "python", 396 | "name": "python2" 397 | }, 398 | "language_info": { 399 | "codemirror_mode": { 400 | "name": "ipython", 401 | "version": 2 402 | }, 403 | "file_extension": ".py", 404 | "mimetype": "text/x-python", 405 | "name": "python", 406 | "nbconvert_exporter": "python", 407 | "pygments_lexer": "ipython2", 408 | "version": "2.7.11" 409 | } 410 | }, 411 | "nbformat": 4, 412 | "nbformat_minor": 0 413 | } 414 | -------------------------------------------------------------------------------- /Householder_stability.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "#Stability of Householder Triangularization\n", 8 | "This is a reproduction of the example from the beginning of Lecture 16 of *Numerical Linear Algebra* by Trefethen & Bau." 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": null, 14 | "metadata": { 15 | "collapsed": false 16 | }, 17 | "outputs": [], 18 | "source": [ 19 | "import numpy as np\n", 20 | "\n", 21 | "def householder(A):\n", 22 | " \"\"\"QR factorization via Householder triangularization.\"\"\"\n", 23 | " m, n = A.shape\n", 24 | " V = np.zeros(A.shape)\n", 25 | " R = A.copy()\n", 26 | " for k in range(n-1):\n", 27 | " x = R[k:,k].copy()\n", 28 | " x[0] = x[0] + np.sign(x[0])*np.linalg.norm(x,2)\n", 29 | " x = x/np.linalg.norm(x,2)\n", 30 | " V[k:,k] = x.copy()\n", 31 | " for j in range(k,n):\n", 32 | " R[k:,j] = R[k:,j] - 2*V[k:,k]*np.dot(V[k:,k].T,R[k:,j])\n", 33 | " return V,R[:n,:]\n", 34 | "\n", 35 | "def apply_Q(V,x):\n", 36 | " \"\"\"Algorithm 10.3 of Trefethen & Bau.\"\"\"\n", 37 | " m, n = V.shape\n", 38 | " for k in range(n-1,-1,-1):\n", 39 | " x[k:] = x[k:] - 2*np.dot(V[k:,k],x[k:])*V[k:,k]\n", 40 | " return x\n", 41 | "\n", 42 | "def compute_Q(V):\n", 43 | " m, n = V.shape\n", 44 | " Q = np.zeros((m,n))\n", 45 | " for k in range(n):\n", 46 | " x = np.zeros(m)\n", 47 | " x[k] = 1.\n", 48 | " Q[:,k] = apply_Q(V,x)\n", 49 | " return Q" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "We're going to investigate the accuracy of the QR factorization generated by Householder triangularization. In order to do so, we construct a matrix $A$ with a known QR factorization by creating a random upper-triangular matrix and a random unitary matrix:" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": false 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "N = 40\n", 68 | "R = np.triu(np.random.randn(N,N))\n", 69 | "Q, X = np.linalg.qr(np.random.randn(N,N))\n", 70 | "A = np.dot(Q,R)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "Now we compute the QR factorization of $A$:" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": { 84 | "collapsed": true 85 | }, 86 | "outputs": [], 87 | "source": [ 88 | "#Q2, R2 = np.linalg.qr(A)\n", 89 | "V, R2 = householder(A)\n", 90 | "Q2 = compute_Q(V)" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "...and check how close the factors are to the exact ones:" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": { 104 | "collapsed": false 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "print np.linalg.norm(Q-Q2)\n", 109 | "print np.linalg.norm(R-R2)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": {}, 115 | "source": [ 116 | "The accuracy is extremely poor! But it's all we can expect based on the condition number of $A$:" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": { 123 | "collapsed": false 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "print np.linalg.cond(A)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "The errors above are *forward errors*. But what if we multiply our factors back together and compare them to $A$? That difference is the *residual*, or *backward error*:" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": { 141 | "collapsed": false 142 | }, 143 | "outputs": [], 144 | "source": [ 145 | "A2 = np.dot(Q2,R2)\n", 146 | "np.linalg.norm(A-A2)" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "It's extremely accurate! Somehow all the errors cancel out, to very high precision.\n", 154 | "\n", 155 | "This cancellation is extremely lucky, and would never happen if the errors were random:" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": null, 161 | "metadata": { 162 | "collapsed": false 163 | }, 164 | "outputs": [], 165 | "source": [ 166 | "Q3 = Q + 1e-4*np.random.randn(N,N)\n", 167 | "R3 = R + 1e-4*np.random.randn(N,N)\n", 168 | "A3 = np.dot(Q3,R3)\n", 169 | "print np.linalg.norm(A-A3)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": { 175 | "collapsed": false 176 | }, 177 | "source": [] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": { 183 | "collapsed": false 184 | }, 185 | "outputs": [], 186 | "source": [ 187 | "for i in range(Q2.shape[1]):\n", 188 | " if Q2[0,i]*Q[0,i] < 0:\n", 189 | " Q2[:,i] = -Q2[:,i]\n", 190 | " R2[i,:] = -R2[i,:]\n", 191 | "print np.linalg.norm(Q-Q2)\n", 192 | "print np.linalg.norm(R-R2)" 193 | ] 194 | } 195 | ], 196 | "metadata": { 197 | "kernelspec": { 198 | "display_name": "Python 2", 199 | "language": "python", 200 | "name": "python2" 201 | }, 202 | "language_info": { 203 | "codemirror_mode": { 204 | "name": "ipython", 205 | "version": 2 206 | }, 207 | "file_extension": ".py", 208 | "mimetype": "text/x-python", 209 | "name": "python", 210 | "nbconvert_exporter": "python", 211 | "pygments_lexer": "ipython2", 212 | "version": "2.7.9" 213 | } 214 | }, 215 | "nbformat": 4, 216 | "nbformat_minor": 0 217 | } 218 | -------------------------------------------------------------------------------- /Least_squares_algorithms.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline\n", 10 | "import matplotlib.pyplot as plt\n", 11 | "import numpy as np\n", 12 | "from numpy.linalg import norm" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "# Stability of least squares algorithms" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "Let's set up a least squares problem. We'll fit a polynomial of degree 14 to the function \n", 27 | "\n", 28 | "$$e^{\\sin(4t)}.$$\n", 29 | "\n", 30 | "We'll normalize the function (based on knowledge of the exact answer) so that the correct answer for the leading coefficient of the interpolating polynomial is 1." 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "m = 100 # Sample at this many points\n", 40 | "n = 15 # Fit a polynomial of degree n-1\n", 41 | "\n", 42 | "t = np.linspace(0,1,m)\n", 43 | "A = np.zeros((m,n))\n", 44 | "for i in range(n):\n", 45 | " A[:,i] = t**i # Vandermonde matrix\n", 46 | " \n", 47 | "b = np.exp(np.sin(4*t)) # Sampled function\n", 48 | "b = b/2006.787453080206 # Normalization\n", 49 | "plt.plot(b);" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "Now let's determine the least squares condition number for this problem. Note that in order to do so, we have to actually solve the problem. We also plot the fit, which is very close." 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "x, res, rank, s = np.linalg.lstsq(A,b,rcond=-1)\n", 66 | "y = np.dot(A,x)\n", 67 | "kappa = np.linalg.cond(A)\n", 68 | "tantheta = norm(b-y)/norm(y)\n", 69 | "eta = norm(A)*norm(x)/norm(y)\n", 70 | "term2 = kappa**2 * tantheta/eta\n", 71 | "print('kappa: %0.2e' % kappa)\n", 72 | "print('theta: %0.2e' % np.arctan(tantheta))\n", 73 | "print('eta: %10.2e' % eta)\n", 74 | "print('kappa^2 * tan(theta)/eta: %10.2e' % term2)\n", 75 | "\n", 76 | "plt.plot(t,b,'-',t,y,'o'); plt.xlabel('x'); plt.ylabel('$f(x)$');" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "We see that the matrix is very ill-conditioned (because of the high polynomial degree), but $\\theta$ is very small which means we can fit the data very well. Recall our formula for the condition number:\n", 84 | "\n", 85 | "$$ \\kappa(A) + \\frac{\\kappa(A)^2 \\tan(\\theta)}{\\eta}. $$\n", 86 | "\n", 87 | "In this case, because $\\theta$ is small and $\\eta$ is fairly large, both terms are comparable. Thus we get a condition number of about $\\kappa$, rather than $\\kappa^2$. This is often the case when the problem is ill-conditioned and the fit is close." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "print('least squares condition number: %0.2e' % (kappa + kappa**2*tantheta/eta))\n", 97 | "print('kappa^2: %0.2e' % kappa**2)" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "Based on this, we should hope to get perhaps 6 digits of accuracy from a backward-stable algorithm (in double precision). Meanwhile, the error from using the normal equations will be governed by $\\kappa^2$, which is much larger." 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "# Solution methods and accuracy" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "## Numpy QR" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "First, let's try numpy's built-in QR factorization." 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "Q, R = np.linalg.qr(A)\n", 135 | "x = np.linalg.solve(R,np.dot(Q.T,b))\n", 136 | "print(x[14]-1)" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "It does a bit better than what the analysis guarantees. It seems this uses a backward-stable algorithm." 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "## Householder QR" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "Next, let's try using our own implementation of Householder factorization." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "def householder(A):\n", 167 | " \"\"\"QR factorization via Householder triangularization.\"\"\"\n", 168 | " m, n = A.shape\n", 169 | " V = np.zeros(A.shape)\n", 170 | " R = A.copy()\n", 171 | " for k in range(n-1):\n", 172 | " x = R[k:,k].copy()\n", 173 | " x[0] = x[0] + np.sign(x[0])*np.linalg.norm(x,2)\n", 174 | " x = x/np.linalg.norm(x,2)\n", 175 | " V[k:,k] = x.copy()\n", 176 | " for j in range(k,n):\n", 177 | " R[k:,j] = R[k:,j] - 2*V[k:,k]*np.dot(V[k:,k].T,R[k:,j])\n", 178 | " return V,R[:n,:]\n", 179 | "\n", 180 | "def apply_Q(V,x):\n", 181 | " \"\"\"Algorithm 10.3 of Trefethen & Bau.\"\"\"\n", 182 | " m, n = V.shape\n", 183 | " for k in range(n-1,-1,-1):\n", 184 | " x[k:] = x[k:] - 2*np.dot(V[k:,k],x[k:])*V[k:,k]\n", 185 | " return x\n", 186 | "\n", 187 | "def compute_Q(V):\n", 188 | " m, n = V.shape\n", 189 | " Q = np.zeros((m,n))\n", 190 | " for k in range(n):\n", 191 | " x = np.zeros(m)\n", 192 | " x[k] = 1.\n", 193 | " Q[:,k] = apply_Q(V,x)\n", 194 | " return Q" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": null, 200 | "metadata": {}, 201 | "outputs": [], 202 | "source": [ 203 | "V, R = householder(A)\n", 204 | "Q = compute_Q(V)\n", 205 | "x = np.linalg.solve(R,np.dot(Q.T,b))\n", 206 | "print(x[14]-1)" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": {}, 212 | "source": [ 213 | "This answer is totally wrong! What happened? Let's check that $Q$ is unitary:" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "print(np.linalg.norm(np.dot(Q.T,Q)-np.eye(15)))" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "It is, so what's wrong? Even though Householder is backward stable, my method for applying $Q^T$ is not. However, there is a sneaky way to avoid this problem. We just append $b$ to $A$ as a final column:" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "metadata": {}, 236 | "outputs": [], 237 | "source": [ 238 | "M = np.append(A,b.reshape((m,1)),axis=1)\n", 239 | "V, R = householder(M)\n", 240 | "R2 = R[:n,:n]\n", 241 | "Qb = R[:n,n]\n", 242 | "x = np.linalg.solve(R2,Qb)\n", 243 | "print(x[14]-1)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "markdown", 248 | "metadata": {}, 249 | "source": [ 250 | "Now we get the expected accuracy -- roughly the same as numpy's QR." 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "### Modified Gram-Schmidt" 258 | ] 259 | }, 260 | { 261 | "cell_type": "markdown", 262 | "metadata": {}, 263 | "source": [ 264 | "Next, let's try Modified Gram-Schmidt. We already know this is an unstable algorithm for computing $Q$, but here goes..." 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "def mgs(A):\n", 274 | " \"QR factorization by the modified Gram-Schmidt algorithm.\"\n", 275 | " n = A.shape[1]\n", 276 | " R = np.zeros([n,n])\n", 277 | " V = np.zeros(A.shape)\n", 278 | " Q = np.zeros(A.shape)\n", 279 | " for i in range(n):\n", 280 | " V[:,i] = A[:,i]\n", 281 | " for i in range(n):\n", 282 | " R[i,i] = np.linalg.norm(V[:,i],2)\n", 283 | " Q[:,i] = V[:,i]/R[i,i]\n", 284 | " for j in range(i,n):\n", 285 | " R[i,j] = np.dot(Q[:,i].T,V[:,j])\n", 286 | " V[:,j] = V[:,j] - R[i,j]*Q[:,i]\n", 287 | " return Q, R" 288 | ] 289 | }, 290 | { 291 | "cell_type": "code", 292 | "execution_count": null, 293 | "metadata": {}, 294 | "outputs": [], 295 | "source": [ 296 | "Q, R = mgs(A)\n", 297 | "x = np.linalg.solve(R,np.dot(Q.T,b))\n", 298 | "print(x[14]-1)" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": null, 304 | "metadata": {}, 305 | "outputs": [], 306 | "source": [ 307 | "print(np.linalg.norm(np.dot(Q.T,Q)-np.eye(15)))" 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": {}, 313 | "source": [ 314 | "Since the $Q$ matrix we get from MGS is not orthogonal, then $Q^T$ is not its inverse, and the algorithm fails. There is again a sneaky way to get around this and avoid the need to use $Q^T$ explicitly:" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": null, 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "M = np.append(A,b.reshape((m,1)),axis=1)\n", 324 | "Q2, R = mgs(M)\n", 325 | "R2 = R[:n,:n]\n", 326 | "Qb = R[:n,n]\n", 327 | "x = np.linalg.solve(R2,Qb)\n", 328 | "print(x[14]-1)" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "Implemented this way, MGS is just as good as the other algorithms we've tried." 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "### Normal equations" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "What about the normal equations?" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": {}, 356 | "outputs": [], 357 | "source": [ 358 | "x = np.linalg.solve(np.dot(A.T,A),np.dot(A.T,b))\n", 359 | "print(x[14]-1)" 360 | ] 361 | }, 362 | { 363 | "cell_type": "markdown", 364 | "metadata": {}, 365 | "source": [ 366 | "Since $\\kappa^2 \\approx 10^{20}$, we don't get a single accurate digit from this algorithm.\n", 367 | "However, the fact that $x$ is totally wrong doesn't necessarily mean $y$ is wrong. Observe:" 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": null, 373 | "metadata": {}, 374 | "outputs": [], 375 | "source": [ 376 | "y = np.dot(A,x)\n", 377 | "plt.plot(t,b,'-',t,y,'o',lw=10);" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "Remember: **The fact that your model fits the data very well says absolutely nothing about whether the model is correct!** \n", 385 | "\n", 386 | "Also, if your least squares problem is ill-conditioned, then there can be vastly different models that fit the data almost equally well. In this case it is a mistake to attach any meaning to the values of the model parameters." 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "### SVD" 394 | ] 395 | }, 396 | { 397 | "cell_type": "markdown", 398 | "metadata": {}, 399 | "source": [ 400 | "Finally, let's try the SVD:" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": null, 406 | "metadata": {}, 407 | "outputs": [], 408 | "source": [ 409 | "U, S, Vstar = np.linalg.svd(A)\n", 410 | "x = np.dot(Vstar.T,np.dot(U[:,:n].T,b)/S)\n", 411 | "print(x[14]-1)" 412 | ] 413 | }, 414 | { 415 | "cell_type": "markdown", 416 | "metadata": {}, 417 | "source": [ 418 | "This is almost exactly the same accuracy as we got from numpy's built-in QR function." 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": {}, 424 | "source": [ 425 | "# Lower-degree fitting" 426 | ] 427 | }, 428 | { 429 | "cell_type": "markdown", 430 | "metadata": {}, 431 | "source": [ 432 | "Let's try fitting a more-oscillatory function with just a quadratic." 433 | ] 434 | }, 435 | { 436 | "cell_type": "code", 437 | "execution_count": null, 438 | "metadata": {}, 439 | "outputs": [], 440 | "source": [ 441 | "m = 100 # Sample at this many points\n", 442 | "n = 3 # Fit a polynomial of degree n-1\n", 443 | "\n", 444 | "t = np.linspace(0,1,m)\n", 445 | "A = np.zeros((m,n))\n", 446 | "for i in range(n):\n", 447 | " A[:,i] = t**i # Vandermonde matrix\n", 448 | " \n", 449 | "b = np.exp(np.sin(40*t)) # Sampled function\n", 450 | "\n", 451 | "x, res, rank, s = np.linalg.lstsq(A,b,rcond=-1)\n", 452 | "y = np.dot(A,x)\n", 453 | "kappa = np.linalg.cond(A)\n", 454 | "eta = norm(A)*norm(x)/norm(y)\n", 455 | "tantheta = norm(b-y)/norm(y)\n", 456 | "term2 = kappa**2*tantheta/eta\n", 457 | "\n", 458 | "print('kappa: %0.2e' % kappa)\n", 459 | "print('tan(theta): %0.2e' % tantheta)\n", 460 | "print('eta: %10.2e' % eta)\n", 461 | "print('kappa^2 * tan(theta)/eta: %10.2e' % term2)\n", 462 | "print('kappa^2: %0.2e' % kappa**2)\n", 463 | "plt.plot(t,b,'-',t,y,'o');\n", 464 | "\n", 465 | "# Use sympy to get \"exact\" solution\n", 466 | "import sympy\n", 467 | "def f(i,j):\n", 468 | " return sympy.Rational(i,m-1)**j\n", 469 | "AA=sympy.Matrix(m,n,f)\n", 470 | "bb = sympy.Matrix(b)\n", 471 | "x_ex=AA.LDLsolve(bb)\n", 472 | "\n", 473 | "# Note that we don't bother to normalize this time because x_ex[-1] is approximately 1 anyway." 474 | ] 475 | }, 476 | { 477 | "cell_type": "markdown", 478 | "metadata": {}, 479 | "source": [ 480 | "As expected, the fit is not as good. The value of $\\theta$ is correspondingly much larger. Of course, the smaller vandermonde matrix $A$ is much better conditioned than before.\n", 481 | "\n", 482 | "Notice that now we expect the normal equations to be worse than a backward-stable algorithm by at most about a factor of 3. Let's check:" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "metadata": {}, 488 | "source": [ 489 | "## SVD" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": null, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "U, S, Vstar = np.linalg.svd(A)\n", 499 | "xx = np.dot(Vstar.T,np.dot(U[:,:n].T,b)/S)\n", 500 | "print(xx[-1]-x_ex[-1])" 501 | ] 502 | }, 503 | { 504 | "cell_type": "markdown", 505 | "metadata": {}, 506 | "source": [ 507 | "## Normal equations" 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": null, 513 | "metadata": {}, 514 | "outputs": [], 515 | "source": [ 516 | "xx = np.linalg.solve(np.dot(A.T,A),np.dot(A.T,b))\n", 517 | "print(xx[-1]-x_ex[-1])" 518 | ] 519 | }, 520 | { 521 | "cell_type": "markdown", 522 | "metadata": {}, 523 | "source": [ 524 | "Indeed, for this case the normal equations give a result just as good as the SVD." 525 | ] 526 | } 527 | ], 528 | "metadata": { 529 | "kernelspec": { 530 | "display_name": "Python 3", 531 | "language": "python", 532 | "name": "python3" 533 | }, 534 | "language_info": { 535 | "codemirror_mode": { 536 | "name": "ipython", 537 | "version": 3 538 | }, 539 | "file_extension": ".py", 540 | "mimetype": "text/x-python", 541 | "name": "python", 542 | "nbconvert_exporter": "python", 543 | "pygments_lexer": "ipython3", 544 | "version": "3.6.5" 545 | }, 546 | "toc": { 547 | "base_numbering": 1, 548 | "nav_menu": {}, 549 | "number_sections": true, 550 | "sideBar": true, 551 | "skip_h1_title": false, 552 | "title_cell": "Table of Contents", 553 | "title_sidebar": "Contents", 554 | "toc_cell": false, 555 | "toc_position": {}, 556 | "toc_section_display": true, 557 | "toc_window_display": false 558 | } 559 | }, 560 | "nbformat": 4, 561 | "nbformat_minor": 1 562 | } 563 | -------------------------------------------------------------------------------- /Linear_regression_with_auto_data.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Linear regression\n", 8 | "In this notebook we'll perform a multiple linear regression to look at correlations in a data regarding properties of automobiles. The data is obtained from the [UC Irvine machine learning repository](http://archive.ics.uci.edu/ml/datasets.html)." 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "First let's download the data. Numpy's `genfromtxt` function makes this easy." 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 139, 21 | "metadata": { 22 | "collapsed": false 23 | }, 24 | "outputs": [ 25 | { 26 | "data": { 27 | "text/plain": [ 28 | "(398, 8)" 29 | ] 30 | }, 31 | "execution_count": 139, 32 | "metadata": {}, 33 | "output_type": "execute_result" 34 | } 35 | ], 36 | "source": [ 37 | "import numpy as np\n", 38 | "data = np.genfromtxt(\"http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data\", \n", 39 | " usecols=range(8))\n", 40 | "data.shape" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "So we have 398 samples, and 8 characteristics for each. What are these properties?\n", 48 | "\n", 49 | "At http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.names we learn that they are\n", 50 | "\n", 51 | "1. mpg: continuous\n", 52 | "2. cylinders: multi-valued discrete\n", 53 | "3. displacement: continuous\n", 54 | "4. horsepower: continuous\n", 55 | "5. weight: continuous\n", 56 | "6. acceleration: continuous\n", 57 | "7. model year: multi-valued discrete\n", 58 | "8. origin: multi-valued discrete\n", 59 | "\n", 60 | "In fact the original dataset has a 9th column with the make and model, but I've omitted that since genfromtxt doesn't know how to handle it easily." 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "Unfortunately, some values are missing:" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": 140, 73 | "metadata": { 74 | "collapsed": false 75 | }, 76 | "outputs": [ 77 | { 78 | "name": "stdout", 79 | "output_type": "stream", 80 | "text": [ 81 | "[ 2.50000000e+01 4.00000000e+00 9.80000000e+01 nan\n", 82 | " 2.04600000e+03 1.90000000e+01 7.10000000e+01 1.00000000e+00]\n" 83 | ] 84 | } 85 | ], 86 | "source": [ 87 | "print data[32,:]" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": {}, 93 | "source": [ 94 | "Here `nan` means *not a number* -- the value in the data file is actually a question mark. We'd like to automatically find and eliminate the rows with missing data. We can detect them like this:" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": 141, 100 | "metadata": { 101 | "collapsed": false 102 | }, 103 | "outputs": [ 104 | { 105 | "data": { 106 | "text/plain": [ 107 | "True" 108 | ] 109 | }, 110 | "execution_count": 141, 111 | "metadata": {}, 112 | "output_type": "execute_result" 113 | } 114 | ], 115 | "source": [ 116 | "any(np.isnan(data[32,:]))" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": {}, 122 | "source": [ 123 | "Now let's delete the rows with missing data:" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 142, 129 | "metadata": { 130 | "collapsed": false 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "bad_rows = []\n", 135 | "\n", 136 | "for i, line in enumerate(data):\n", 137 | " if any(np.isnan(line)):\n", 138 | " bad_rows.append(i)\n", 139 | " \n", 140 | "count = 0\n", 141 | "for i in bad_rows:\n", 142 | " data = np.delete(data,i-count,0)\n", 143 | " count = count + 1" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "Check that it's clean:" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 143, 156 | "metadata": { 157 | "collapsed": false 158 | }, 159 | "outputs": [ 160 | { 161 | "data": { 162 | "text/plain": [ 163 | "False" 164 | ] 165 | }, 166 | "execution_count": 143, 167 | "metadata": {}, 168 | "output_type": "execute_result" 169 | } 170 | ], 171 | "source": [ 172 | "np.isnan(data).any()" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 145, 178 | "metadata": { 179 | "collapsed": false 180 | }, 181 | "outputs": [ 182 | { 183 | "data": { 184 | "text/plain": [ 185 | "(392, 8)" 186 | ] 187 | }, 188 | "execution_count": 145, 189 | "metadata": {}, 190 | "output_type": "execute_result" 191 | } 192 | ], 193 | "source": [ 194 | "data.shape" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": {}, 200 | "source": [ 201 | "So we removed six rows, which still leaves us plenty of data to work with.\n", 202 | "\n", 203 | "Let's create a model that predicts fuel economy (miles per gallon) based on the remaining characteristics. Since there isn't a numerical meaning to the values for *origin*, we'll omit the last column. Remember that Python indexes from zero!" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": 150, 209 | "metadata": { 210 | "collapsed": false 211 | }, 212 | "outputs": [ 213 | { 214 | "data": { 215 | "text/plain": [ 216 | "(392, 6)" 217 | ] 218 | }, 219 | "execution_count": 150, 220 | "metadata": {}, 221 | "output_type": "execute_result" 222 | } 223 | ], 224 | "source": [ 225 | "A = data[:,1:7]\n", 226 | "A.shape" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 151, 232 | "metadata": { 233 | "collapsed": true 234 | }, 235 | "outputs": [], 236 | "source": [ 237 | "y = data[:,0] # miles per gallon" 238 | ] 239 | }, 240 | { 241 | "cell_type": "markdown", 242 | "metadata": {}, 243 | "source": [ 244 | "Now we have the *inputs* for our model in the matrix $A$ and the *outputs* in the vector $y$. We'll solve\n", 245 | "\n", 246 | "$$Ax = y$$\n", 247 | "\n", 248 | "in the least squares sense, which will give us an idea of how fuel economy typically varies with each of the other factors." 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "metadata": {}, 254 | "source": [ 255 | "We can solve the system using this function from `numpy`:" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 152, 261 | "metadata": { 262 | "collapsed": true 263 | }, 264 | "outputs": [], 265 | "source": [ 266 | "np.linalg.lstsq?" 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": 153, 272 | "metadata": { 273 | "collapsed": false 274 | }, 275 | "outputs": [], 276 | "source": [ 277 | "x, resid, rank, s = np.linalg.lstsq(A,y)" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 154, 283 | "metadata": { 284 | "collapsed": false 285 | }, 286 | "outputs": [ 287 | { 288 | "name": "stdout", 289 | "output_type": "stream", 290 | "text": [ 291 | "[-0.5226089 0.01022108 -0.020873 -0.00639456 -0.05202195 0.61025869]\n" 292 | ] 293 | } 294 | ], 295 | "source": [ 296 | "print x" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "What do these results mean? As one might expect, fuel economy tends to be worse for cars with more cylinders, horsepower, or weight. Newer cars tend to be more fuel efficient, as do cars with greater displacement (i.e., greater engine volume)." 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "metadata": {}, 309 | "source": [ 310 | "Furthermore, we can make quantitative statements. Cars seem to be getting more fuel efficient by a little more than 1/2 mpg per year. Each thousand pounds of weight added to a car reduces its fuel efficiency by roughly 6 mpg." 311 | ] 312 | } 313 | ], 314 | "metadata": { 315 | "kernelspec": { 316 | "display_name": "Python 2", 317 | "language": "python", 318 | "name": "python2" 319 | }, 320 | "language_info": { 321 | "codemirror_mode": { 322 | "name": "ipython", 323 | "version": 2 324 | }, 325 | "file_extension": ".py", 326 | "mimetype": "text/x-python", 327 | "name": "python", 328 | "nbconvert_exporter": "python", 329 | "pygments_lexer": "ipython2", 330 | "version": "2.7.9" 331 | } 332 | }, 333 | "nbformat": 4, 334 | "nbformat_minor": 0 335 | } 336 | -------------------------------------------------------------------------------- /Performance modeling.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%matplotlib inline\n", 10 | "import matplotlib.pyplot as plt\n", 11 | "import numpy as np\n", 12 | "from time import time" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "## inner product" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "m = 1000000\n", 29 | "x = np.random.rand(m)\n", 30 | "y = np.random.rand(m)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "r_peak = 2.9e9 * 32 # 32 flops/cycle\n", 40 | "B_peak = 2.133e9 * 2 # 2 words/cycle" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "F = 2*m\n", 50 | "M = 2*m\n", 51 | "print(F/r_peak)\n", 52 | "print(M/B_peak)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "%%timeit\n", 62 | "np.dot(x,y)" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "x.dtype" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "## GEMV" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "m = 10000\n", 88 | "A = np.random.rand(m,m)\n", 89 | "x = np.random.rand(m)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "F = 2*m**2\n", 99 | "M = m**2\n", 100 | "print(F/r_peak)\n", 101 | "print(M/B_peak)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": null, 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "%%timeit\n", 111 | "np.dot(A,x)" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "## GEMM" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "m = 1000\n", 128 | "A = np.random.rand(m,m)\n", 129 | "B = np.random.rand(m,m)" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "F = 2*m**3\n", 139 | "M = 3*m**2\n", 140 | "print(F/r_peak)\n", 141 | "print(M/B_peak)" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "%%timeit\n", 151 | "C = np.dot(A,B)" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": {}, 158 | "outputs": [], 159 | "source": [] 160 | } 161 | ], 162 | "metadata": { 163 | "kernelspec": { 164 | "display_name": "Python 3", 165 | "language": "python", 166 | "name": "python3" 167 | }, 168 | "language_info": { 169 | "codemirror_mode": { 170 | "name": "ipython", 171 | "version": 3 172 | }, 173 | "file_extension": ".py", 174 | "mimetype": "text/x-python", 175 | "name": "python", 176 | "nbconvert_exporter": "python", 177 | "pygments_lexer": "ipython3", 178 | "version": "3.6.5" 179 | }, 180 | "toc": { 181 | "base_numbering": 1, 182 | "nav_menu": {}, 183 | "number_sections": true, 184 | "sideBar": true, 185 | "skip_h1_title": false, 186 | "title_cell": "Table of Contents", 187 | "title_sidebar": "Contents", 188 | "toc_cell": false, 189 | "toc_position": {}, 190 | "toc_section_display": true, 191 | "toc_window_display": false 192 | } 193 | }, 194 | "nbformat": 4, 195 | "nbformat_minor": 2 196 | } 197 | -------------------------------------------------------------------------------- /Power iteration illustrated.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "%matplotlib inline\n", 11 | "import matplotlib.pyplot as plt\n", 12 | "import matplotlib as mpl\n", 13 | "mpl.rcParams['axes.autolimit_mode'] = 'round_numbers'\n", 14 | "mpl.rcParams['axes.xmargin'] = 0\n", 15 | "mpl.rcParams['axes.ymargin'] = 0" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "A = np.random.rand(2,2)\n", 25 | "A = A+A.T\n", 26 | "Lambda, X = np.linalg.eig(A)\n", 27 | "Lambda = np.array([1., 0.99])\n", 28 | "A = np.dot(X,np.dot(np.diag(Lambda),np.linalg.inv(X)))\n", 29 | "Lambda, X = np.linalg.eig(A)\n", 30 | "print(Lambda)\n", 31 | "print(X)\n", 32 | "j = np.argmax(np.abs(Lambda))" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "theta = np.linspace(0,2*np.pi,200)\n", 42 | "unit_circle = np.zeros((2, len(theta)))\n", 43 | "for i, thet in enumerate(theta):\n", 44 | " unit_circle[0,i] = np.cos(thet)\n", 45 | " unit_circle[1,i] = np.sin(thet)\n", 46 | "plt.plot(unit_circle[0,:],unit_circle[1,:],'.')\n", 47 | "plt.plot([0,Lambda[j]*X[0,j]],[0,Lambda[j]*X[1,j]],'-k',lw=3)\n", 48 | "plt.plot([0,Lambda[1-j]*X[0,1-j]],[0,Lambda[1-j]*X[1,1-j]],'-k',lw=3)\n", 49 | "plt.axis('equal');" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [ 58 | "it = np.dot(A,unit_circle)\n", 59 | "plt.plot(it[0,:],it[1,:],'.')\n", 60 | "plt.plot([-2,2],[0,0],'--k')\n", 61 | "plt.plot([0,0],[-2,2],'--k')\n", 62 | "plt.plot([0,Lambda[j]*X[0,j]],[0,Lambda[j]*X[1,j]],'-k',lw=3)\n", 63 | "plt.plot([0,Lambda[1-j]*X[0,1-j]],[0,Lambda[1-j]*X[1,1-j]],'-k',lw=3)\n", 64 | "plt.axis('image');" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [ 73 | "it = np.dot(A,it)\n", 74 | "plt.plot(it[0,:],it[1,:],'.')\n", 75 | "plt.plot([-2,2],[0,0],'--k')\n", 76 | "plt.plot([0,0],[-2,2],'--k')\n", 77 | "plt.plot([0,Lambda[j]*X[0,j]],[0,Lambda[j]*X[1,j]],'-k',lw=3)\n", 78 | "plt.plot([0,Lambda[1-j]*X[0,1-j]],[0,Lambda[1-j]*X[1,1-j]],'-k',lw=3)\n", 79 | "plt.axis('image');" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "it = np.dot(A,it)\n", 89 | "plt.plot(it[0,:],it[1,:],'.')\n", 90 | "plt.plot([-2,2],[0,0],'--k')\n", 91 | "plt.plot([0,0],[-2,2],'--k')\n", 92 | "plt.plot([0,Lambda[j]*X[0,j]],[0,Lambda[j]*X[1,j]],'-k',lw=3)\n", 93 | "plt.plot([0,Lambda[1-j]*X[0,1-j]],[0,Lambda[1-j]*X[1,1-j]],'-k',lw=3)\n", 94 | "plt.axis('image');" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "def map_circle(A,k,zoom=False):\n", 104 | " Lambda, X = np.linalg.eig(A)\n", 105 | " j = np.argmax(np.abs(Lambda))\n", 106 | " theta = np.linspace(0,2*np.pi,200)\n", 107 | " unit_circle = np.zeros((2, len(theta)))\n", 108 | " for i, thet in enumerate(theta):\n", 109 | " unit_circle[0,i] = np.cos(thet)\n", 110 | " unit_circle[1,i] = np.sin(thet)\n", 111 | " it = np.dot(np.linalg.matrix_power(A,k),unit_circle)\n", 112 | " plt.plot(it[0,:],it[1,:],'.')\n", 113 | " plt.plot([-2,2],[0,0],'--k')\n", 114 | " plt.plot([0,0],[-2,2],'--k')\n", 115 | " plt.plot([0,Lambda[j]*X[0,j]],[0,Lambda[j]*X[1,j]],'-k',lw=3,alpha=0.5)\n", 116 | " plt.plot([0,Lambda[1-j]*X[0,1-j]],[0,Lambda[1-j]*X[1,1-j]],'-k',lw=3,alpha=0.5)\n", 117 | " plt.axis('image');\n", 118 | " if zoom:\n", 119 | " plt.axis([-2,2,-2,2])\n", 120 | " plt.show()" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "map_circle(A,0)" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": null, 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "from ipywidgets import interact, fixed, IntSlider\n", 139 | "\n", 140 | "interact(map_circle,A=fixed(A),k=IntSlider(min=0,max=20,value=0),zoom=fixed(False));" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "interact(map_circle,A=fixed(A),k=IntSlider(min=0,max=20,value=0),zoom=fixed(True));" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [] 158 | } 159 | ], 160 | "metadata": { 161 | "kernelspec": { 162 | "display_name": "Python 3", 163 | "language": "python", 164 | "name": "python3" 165 | }, 166 | "language_info": { 167 | "codemirror_mode": { 168 | "name": "ipython", 169 | "version": 3 170 | }, 171 | "file_extension": ".py", 172 | "mimetype": "text/x-python", 173 | "name": "python", 174 | "nbconvert_exporter": "python", 175 | "pygments_lexer": "ipython3", 176 | "version": "3.6.5" 177 | }, 178 | "toc": { 179 | "base_numbering": 1, 180 | "nav_menu": {}, 181 | "number_sections": true, 182 | "sideBar": true, 183 | "skip_h1_title": false, 184 | "title_cell": "Table of Contents", 185 | "title_sidebar": "Contents", 186 | "toc_cell": false, 187 | "toc_position": {}, 188 | "toc_section_display": true, 189 | "toc_window_display": false 190 | } 191 | }, 192 | "nbformat": 4, 193 | "nbformat_minor": 2 194 | } 195 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # IPython notebooks on numerical linear algebra 2 | 3 | This is a small collection of notebooks illustrating concepts in numerical linear algebra. 4 | They were created as part of the Fall 2014 course AMCS 251 at KAUST. They are meant to 5 | accompany a reading of [Numerical Linear Algebra](https://people.maths.ox.ac.uk/trefethen/text.html) 6 | by Trefethen & Bau. 7 | 8 | Links to notebooks on nbviewer: 9 | 10 | - [Stability of Householder Triangularization](http://nbviewer.ipython.org/github/ketch/numerical_linear_algebra_notebooks/blob/master/Householder_stability.ipynb) 11 | - [Backward stability of solving Ax=b by Householder QR](http://nbviewer.ipython.org/github/ketch/numerical_linear_algebra_notebooks/blob/master/linear_solve_stability.ipynb) 12 | - [Stability of Least Squares Algorithms](http://nbviewer.ipython.org/urls/raw.github.com/ketch/numerical_linear_algebra_notebooks/master/Least_squares_algorithms.ipynb) 13 | - [Gaussian Elimination](http://nbviewer.ipython.org/github/ketch/numerical_linear_algebra_notebooks/blob/master/Gaussian%20Elimination.ipynb) 14 | - [Methods for finding a single eigenvalue](http://nbviewer.ipython.org/github/ketch/numerical_linear_algebra_notebooks/blob/master/Rayleigh_quotient_iteration.ipynb) 15 | 16 | 17 | The material in this repository is released under a Creative Commons 3.0 CC-BY license. 18 | You may reuse it freely as long as you credit [David I. Ketcheson](http://davidketcheson.info). 19 | -------------------------------------------------------------------------------- /Rayleigh_quotient_iteration.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "%matplotlib inline\n", 11 | "import matplotlib.pyplot as plt" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "# Power iteration" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "def power_iteration(A,v0,niter=5):\n", 28 | " v = v0\n", 29 | " vv = [v0]\n", 30 | " ll = [np.dot(v0,np.dot(A,v0))]\n", 31 | " for k in range(niter):\n", 32 | " w = np.dot(A,v)\n", 33 | " v = w/np.linalg.norm(w)\n", 34 | " lamda = np.dot(v,np.dot(A,v))\n", 35 | " vv.append(v)\n", 36 | " ll.append(lamda)\n", 37 | " return ll, vv" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "# Inverse iteration" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "def inverse_iteration(A,v0,mu,niter=5):\n", 54 | " v = v0\n", 55 | " I = np.eye(len(v0))\n", 56 | " vv = [v0]\n", 57 | " ll = [np.dot(v0,np.dot(A,v0))]\n", 58 | " for k in range(niter):\n", 59 | " w = np.linalg.solve(A-mu*I,v)\n", 60 | " v = w/np.linalg.norm(w)\n", 61 | " lamda = np.dot(v,np.dot(A,v))\n", 62 | " vv.append(v)\n", 63 | " ll.append(lamda)\n", 64 | " return ll, vv" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "# Rayleigh Quotient Iteration" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "def rayleigh_quotient_iteration(A,v0,niter=5):\n", 81 | " v = v0\n", 82 | " I = np.eye(len(v0))\n", 83 | " lamda = np.dot(v0,np.dot(A,v0))/np.linalg.norm(v0)\n", 84 | " vv = [v0]\n", 85 | " ll = [np.dot(v0,np.dot(A,v0))/np.linalg.norm(v0)]\n", 86 | " for k in range(niter):\n", 87 | " w = np.linalg.solve(A-lamda*I,v)\n", 88 | " v = w/np.linalg.norm(w)\n", 89 | " lamda = np.dot(v,np.dot(A,v))\n", 90 | " vv.append(v)\n", 91 | " ll.append(lamda)\n", 92 | " return ll, vv" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "# Examples" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "First, let's try a symmetric matrix." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "A = np.array([[2.,1,1],[1,3,1],[1,1,4]])\n", 116 | "print(A)\n", 117 | "print(np.linalg.eigvals(A))\n", 118 | "lam =(np.linalg.eigvals(A)[0])" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "ll, vv = power_iteration(A,np.ones(3))\n", 128 | "plt.plot(range(len(ll)),ll,'-o')\n", 129 | "plt.ylabel('Eigenvalue approximation')\n", 130 | "plt.xlabel('Iteration');" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [ 139 | "plt.semilogy(np.abs(ll-lam))\n", 140 | "plt.ylabel('Error'); plt.title('Power iteration')\n", 141 | "plt.xlabel('Iteration');" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "mu=5\n", 151 | "ll, vv = inverse_iteration(A,np.ones(3),mu)\n", 152 | "plt.semilogy(np.abs(ll-lam))\n", 153 | "plt.ylabel('Error'); plt.title('Inverse iteration')\n", 154 | "plt.xlabel('Iteration');" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "ll, vv = rayleigh_quotient_iteration(A,np.ones(3))\n", 164 | "plt.semilogy(np.abs(ll-lam))\n", 165 | "plt.ylabel('Error'); plt.title('Rayleigh quotient iteration')\n", 166 | "plt.xlabel('Iteration');" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "Here's a comparison of all three methods." 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "ll1, vv = power_iteration(A,np.ones(3))\n", 183 | "ll2, vv = inverse_iteration(A,np.ones(3),mu)\n", 184 | "ll3, vv = rayleigh_quotient_iteration(A,np.ones(3))\n", 185 | "plt.semilogy(np.abs(ll1-lam))\n", 186 | "plt.semilogy(np.abs(ll2-lam))\n", 187 | "plt.semilogy(np.abs(ll3-lam))\n", 188 | "plt.ylabel('Error'); plt.xlabel('Iteration')\n", 189 | "plt.legend(['Power','Inverse','Rayleigh'],loc='best');" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "Now let's try a non-symmetric matrix (but with distinct real eigenvalues)." 197 | ] 198 | }, 199 | { 200 | "cell_type": "code", 201 | "execution_count": null, 202 | "metadata": {}, 203 | "outputs": [], 204 | "source": [ 205 | "A = np.array([[2.,5.,1],[1,3,1],[1,1,4]])\n", 206 | "print(A)\n", 207 | "print(np.linalg.eigvals(A))\n", 208 | "lam =(np.linalg.eigvals(A)[0])" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "ll1, vv = power_iteration(A,np.ones(3))\n", 218 | "ll2, vv = inverse_iteration(A,np.ones(3),5.)\n", 219 | "ll3, vv = rayleigh_quotient_iteration(A,np.ones(3))\n", 220 | "plt.semilogy(np.abs(ll1-lam))\n", 221 | "plt.semilogy(np.abs(ll2-lam))\n", 222 | "plt.semilogy(np.abs(ll3-lam))\n", 223 | "plt.ylabel('Error'); plt.xlabel('Iteration')\n", 224 | "plt.legend(['Power','Inverse','Rayleigh'],loc='best');" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "# QR iteration" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "A = np.array([[2.,1,1],[1,3,1],[1,1,4]])\n", 241 | "#A = np.array([[2.,5.,1],[1,3,1],[1,1,4]])\n", 242 | "print(A)\n", 243 | "print(np.linalg.eigvals(A))" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "niter = 20\n", 253 | "for i in range(niter):\n", 254 | " Q, R = np.linalg.qr(A)\n", 255 | " A = np.dot(R,Q)\n", 256 | "print(A)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "print(np.diag(A))" 266 | ] 267 | } 268 | ], 269 | "metadata": { 270 | "kernelspec": { 271 | "display_name": "Python 3", 272 | "language": "python", 273 | "name": "python3" 274 | }, 275 | "language_info": { 276 | "codemirror_mode": { 277 | "name": "ipython", 278 | "version": 3 279 | }, 280 | "file_extension": ".py", 281 | "mimetype": "text/x-python", 282 | "name": "python", 283 | "nbconvert_exporter": "python", 284 | "pygments_lexer": "ipython3", 285 | "version": "3.6.5" 286 | }, 287 | "toc": { 288 | "base_numbering": 1, 289 | "nav_menu": {}, 290 | "number_sections": true, 291 | "sideBar": true, 292 | "skip_h1_title": false, 293 | "title_cell": "Table of Contents", 294 | "title_sidebar": "Contents", 295 | "toc_cell": false, 296 | "toc_position": {}, 297 | "toc_section_display": true, 298 | "toc_window_display": false 299 | } 300 | }, 301 | "nbformat": 4, 302 | "nbformat_minor": 1 303 | } 304 | -------------------------------------------------------------------------------- /SVD_image_compression.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# Import key packages and set up plotting\n", 10 | "# You may need to:\n", 11 | "# pip install pillow\n", 12 | "# pip install imageio\n", 13 | "import numpy as np\n", 14 | "import imageio\n", 15 | "%matplotlib inline\n", 16 | "import matplotlib\n", 17 | "import matplotlib.pyplot as plt\n", 18 | "matplotlib.rcParams.update({'font.size': 15})\n", 19 | "plt.gray()" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "img = imageio.imread('kaust_logo.png')\n", 29 | "img = img.sum(2)/3. # Average RGB channels to get grayscale\n", 30 | "print(img.shape)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "plt.imshow(img)" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "U,S,V = np.linalg.svd(img,full_matrices=False)" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "new_img = np.dot(U,np.dot(np.diag(S),V))\n", 58 | "plt.imshow(new_img)" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "plt.spy(np.diag(S),markersize=5)" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "plt.semilogy(S,'o')" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "S1 = np.diag(S)\n", 86 | "n = 10\n", 87 | "S1[n:,n:] = 0\n", 88 | "plt.spy(S1,markersize=5)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "img_c = np.dot(U,np.dot(S1,V))" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "plt.imshow(img_c)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "from matplotlib import animation\n", 116 | "from IPython.display import HTML\n", 117 | "\n", 118 | "#from JSAnimation import IPython_display\n", 119 | "\n", 120 | "fig = plt.figure(figsize=[8,4])\n", 121 | "\n", 122 | "plot_handle = plt.imshow(img)\n", 123 | "\n", 124 | "def fplot(n):\n", 125 | " S1 = np.diag(S)\n", 126 | " S1[n:,n:] = 0\n", 127 | " img_c = np.dot(U,np.dot(S1,V))\n", 128 | " plot_handle.set_data(img_c)\n", 129 | " fig.axes[0].set_title('Number of singular values: '+str(n))\n", 130 | " return plot_handle,\n", 131 | "\n", 132 | "anim = animation.FuncAnimation(fig, fplot, frames=100, interval=20);" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": {}, 139 | "outputs": [], 140 | "source": [ 141 | "HTML(anim.to_jshtml())" 142 | ] 143 | } 144 | ], 145 | "metadata": { 146 | "kernelspec": { 147 | "display_name": "Python 3", 148 | "language": "python", 149 | "name": "python3" 150 | }, 151 | "language_info": { 152 | "codemirror_mode": { 153 | "name": "ipython", 154 | "version": 3 155 | }, 156 | "file_extension": ".py", 157 | "mimetype": "text/x-python", 158 | "name": "python", 159 | "nbconvert_exporter": "python", 160 | "pygments_lexer": "ipython3", 161 | "version": "3.6.5" 162 | }, 163 | "toc": { 164 | "base_numbering": 1, 165 | "nav_menu": {}, 166 | "number_sections": true, 167 | "sideBar": true, 168 | "skip_h1_title": false, 169 | "title_cell": "Table of Contents", 170 | "title_sidebar": "Contents", 171 | "toc_cell": false, 172 | "toc_position": {}, 173 | "toc_section_display": true, 174 | "toc_window_display": false 175 | } 176 | }, 177 | "nbformat": 4, 178 | "nbformat_minor": 1 179 | } 180 | -------------------------------------------------------------------------------- /linear_solve_stability.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Backward stability of solving $Ax=b$ by Householder QR factorization" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In this notebook we demonstrate the backward stability of solving $Ax=b$ by the following algorithm:\n", 15 | "\n", 16 | "1. Factor $A=QR$ using Householder triangularization.\n", 17 | "2. Compute $y=Q^* b$.\n", 18 | "3. Solve $Ry = x$ by backward substitution." 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "First, let's implement Householder (*note: we don't need to actually compute $Q$ in the algorithm above, so we could just apply $Q^*$ directly using algorithm 10.2 from Trefethen & Bau*)." 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "def householder(A):\n", 44 | " \"\"\"QR factorization via Householder triangularization.\"\"\"\n", 45 | " m, n = A.shape\n", 46 | " V = np.zeros(A.shape)\n", 47 | " R = A.copy()\n", 48 | " for k in range(n-1):\n", 49 | " x = R[k:,k].copy()\n", 50 | " x[0] = x[0] + np.sign(x[0])*np.linalg.norm(x,2)\n", 51 | " x = x/np.linalg.norm(x,2)\n", 52 | " V[k:,k] = x.copy()\n", 53 | " for j in range(k,n):\n", 54 | " R[k:,j] = R[k:,j] - 2*V[k:,k]*np.dot(V[k:,k].T,R[k:,j])\n", 55 | " return V,R[:n,:]\n", 56 | "\n", 57 | "def apply_Q(V,x):\n", 58 | " \"\"\"Algorithm 10.3 of Trefethen & Bau.\"\"\"\n", 59 | " m, n = V.shape\n", 60 | " for k in range(n-1,-1,-1):\n", 61 | " x[k:] = x[k:] - 2*np.dot(V[k:,k],x[k:])*V[k:,k]\n", 62 | " return x\n", 63 | "\n", 64 | "def compute_Q(V):\n", 65 | " \"\"\"Find Q given the Householder reflector vectors.\"\"\"\n", 66 | " m, n = V.shape\n", 67 | " Q = np.zeros((m,n))\n", 68 | " for k in range(n):\n", 69 | " x = np.zeros(m)\n", 70 | " x[k] = 1.\n", 71 | " Q[:,k] = apply_Q(V,x)\n", 72 | " return Q" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "In order to show stability, let's pick a really nasty matrix:" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": null, 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "def Hilbert(n):\n", 89 | " \"\"\"Return the n x n Hilbert matrix.\"\"\"\n", 90 | " A = np.zeros([n,n])\n", 91 | " for i in range(n):\n", 92 | " for j in range(n):\n", 93 | " A[i,j] = 1./(i+j+1)\n", 94 | " return A" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "N=10\n", 104 | "x = np.ones(N)\n", 105 | "A = Hilbert(N)\n", 106 | "b = np.dot(A,x)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "kappa = np.linalg.cond(A)\n", 116 | "np.log10(kappa)" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": {}, 122 | "source": [ 123 | "The condition number is about $10^{13}$, which means we could lose 13 digits of accuracy just due to rounding of the inputs. Hence a backward stable algorithm can be expected to give us $16-13 = 3$ accurate digits in double precision." 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "V, R = householder(A)\n", 133 | "Q = compute_Q(V)\n", 134 | "y = np.dot(Q.T,b)\n", 135 | "x_tilde = np.linalg.solve(R,y)" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "print(np.linalg.norm(x_tilde-x))" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "Indeed, we are left with about 3 significant accurate digits; in other words, the forward error is about $10^{-3}$." 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": {}, 158 | "outputs": [], 159 | "source": [ 160 | "print(x_tilde)" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "How large is the backward error?" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "norm = np.linalg.norm\n", 177 | "r = np.dot(A,x_tilde)-b\n", 178 | "print(norm(r)/norm(A)*norm(x_tilde))" 179 | ] 180 | }, 181 | { 182 | "cell_type": "markdown", 183 | "metadata": {}, 184 | "source": [ 185 | "Still on the order of unit roundoff, as we expect." 186 | ] 187 | } 188 | ], 189 | "metadata": { 190 | "kernelspec": { 191 | "display_name": "Python 3", 192 | "language": "python", 193 | "name": "python3" 194 | }, 195 | "language_info": { 196 | "codemirror_mode": { 197 | "name": "ipython", 198 | "version": 3 199 | }, 200 | "file_extension": ".py", 201 | "mimetype": "text/x-python", 202 | "name": "python", 203 | "nbconvert_exporter": "python", 204 | "pygments_lexer": "ipython3", 205 | "version": "3.6.5" 206 | }, 207 | "toc": { 208 | "base_numbering": 1, 209 | "nav_menu": {}, 210 | "number_sections": true, 211 | "sideBar": true, 212 | "skip_h1_title": false, 213 | "title_cell": "Table of Contents", 214 | "title_sidebar": "Contents", 215 | "toc_cell": false, 216 | "toc_position": {}, 217 | "toc_section_display": true, 218 | "toc_window_display": false 219 | } 220 | }, 221 | "nbformat": 4, 222 | "nbformat_minor": 1 223 | } 224 | --------------------------------------------------------------------------------