├── .gitignore ├── Ch 1 - Introduction.ipynb ├── Ch 2 - Rank Nullspace and Determinants.ipynb ├── Ch 3 - Eigenvalues and Eigenvectors.ipynb ├── Ch 4 - Least Squares and Least Norm.ipynb ├── Ch 5 - Singular Value Decomposition.ipynb ├── README.md ├── helpers.py ├── img ├── Loop-left-shad.png ├── determinant_1.png ├── determinant_2.png ├── disclaimer.png ├── dyn_eig.png ├── eig.png ├── excel.jpg ├── gps.png ├── hamster.png ├── hard_disk.jpg ├── independence.png ├── intro_least_norm_control.png ├── intro_naive_control.png ├── intro_system_modes.png ├── intro_system_ringing.png ├── markov_1.png ├── markov_matrix.png ├── nullspace.png ├── qr_factorization.png ├── radio_tower.png ├── receiver.png ├── receiver2.png ├── rigid_body.png ├── rocket.png ├── satellite-clipart-satellite-signal-21.zip ├── sensor_interp.png ├── svd.png ├── svd_matrices.png ├── taylor_series.png ├── vector.png └── vector_addition.png ├── requirements.txt └── runtime.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pptx 2 | -------------------------------------------------------------------------------- /Ch 1 - Introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# A Crash Course in Applied Linear Algebra\n", 8 | "Patrick Landreman | Spry Health" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "### Some Motivation: Predicting the Future\n", 16 | "\n", 17 | "Since we are all busy people with too much to do, why should you spend your precious time reading this notebook? I'd like to give you a sample of the kind of power you'll have by the end.\n", 18 | "\n", 19 | "Consider the following time series plot:\n", 20 | "\n", 21 | "\n", 22 | "\n", 23 | "This could be the value of a stock price, or the position of an aircraft, or some biological signal, or any number of things. You are tasked with predicting the value at some future time. Looks unpleasant, no? Maybe there is some periodicity emerging by the end, but surely this is not a trivial task." 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "Now suppose you know that the series above was a mixture of a number of much simpler functions, and you could tell the amount of each function:\n", 31 | "\n", 32 | "\n", 33 | "\n", 34 | "Predicting how each of these functions will evolve should be a much simpler task. The question is - how do you find these simpler functions, and how do you find the weight of each function?" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "Perhaps more interesting, suppose the system above comes from a robotic arm which can move in a 2D plane. The time series corresponds to the position of the end of the arm in one direction - we kicked the system, which caused some shaking and oscillating that subsides over time. \n", 42 | "\n", 43 | "We'll describe the complete position with a pair of numbers, $(y_1, y_2)$. We also can control our robot with some inputs, $(u_1, u_2)$. If we wanted to drive the arm to a specific location, say $(y_1, y_2) = (0.2, -0.2)$, we can expect that the arm will move, and then because of momentum or other issues the system might overshoot a bit and take some time to stabilize (this is exactly the kind of thing that happens with the read head in a spinning-disk hard drive, for instance). The next image shows what happens if at 50 seconds we apply a specific set of inputs to reach the target:" 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "\n", 51 | "\n", 52 | "Ok, we get there after ~3000 seconds, but it takes a while for the ringing to settle down." 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "Instead, however, it is possible to choose a more complicated sequence of inputs that brings the arm to the target in **exactly** a specified amount of time:\n", 60 | "\n", 61 | "\n", 62 | "\n", 63 | "Here, the arm reaches the specified coordinates at 800 seconds and remains there, perfectly stable. From looking at the sequence of inputs, there is **no way** a human could intuit the sequence.\n", 64 | "\n", 65 | "This is just one of the kinds of problems I hope to expose you to today." 66 | ] 67 | }, 68 | { 69 | "attachments": {}, 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "### Acknowledgements\n", 74 | "\n", 75 | "*This presentation is dedicated to Prof. Stephen Boyd and Prof. Sanjay Lall, who first exposed me to this way of seeing the world. Many of the ideas I present here were inspired largely by them and their teachers, and they deserve credit for the enormous amount of work they have done.*\n", 76 | "\n", 77 | "*If this subject interests you, I* strongly *suggest exploring their [books](http://vmls-book.stanford.edu/), [course notes](http://ee263.stanford.edu/), and lecture series on [YouTube](https://www.youtube.com/playlist?list=PL06960BA52D0DB32B).*" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "### Python Setup\n", 85 | "\n", 86 | "Everything in this talk can be done with a basic installation of Numpy and Scipy. The version should not be important. Scipy is used exclusively for some convenience functions, and Matplotlib is included only for visualization purposes. Neither are necessary for linear algebra. This notebook was written using Python 3.6, Numpy 1.16.4, Scipy 1.2.1 and Matplotlib 3.1.1" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 9, 92 | "metadata": { 93 | "ExecuteTime": { 94 | "end_time": "2019-11-04T04:34:25.444430Z", 95 | "start_time": "2019-11-04T04:34:25.431666Z" 96 | } 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "# Render MPL figures within notebook cells\n", 101 | "%matplotlib inline\n", 102 | "\n", 103 | "# Import python libraries\n", 104 | "import numpy as np\n", 105 | "import matplotlib.pyplot as plt\n", 106 | "from matplotlib import rcParams" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 10, 112 | "metadata": { 113 | "ExecuteTime": { 114 | "end_time": "2019-11-04T04:34:25.501434Z", 115 | "start_time": "2019-11-04T04:34:25.448882Z" 116 | } 117 | }, 118 | "outputs": [], 119 | "source": [ 120 | "# Configure some defaults for plots\n", 121 | "rcParams['font.size'] = 16\n", 122 | "rcParams['figure.figsize'] = (10, 3)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": 11, 128 | "metadata": { 129 | "ExecuteTime": { 130 | "end_time": "2019-11-04T04:34:25.513684Z", 131 | "start_time": "2019-11-04T04:34:25.506798Z" 132 | } 133 | }, 134 | "outputs": [], 135 | "source": [ 136 | "# Set Numpy's random number generator so the same results are produced each time the notebook is run\n", 137 | "np.random.seed(0)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "### Why did I make this notebook?\n", 145 | "\n", 146 | "\n", 147 | "If your exposure to the linear algebra was anything like my first course, you probably don't think back on the experience longingly, wishing you could relive those days. My professor literally opened the class by saying he didn't understand why linear algebra was interesting or useful. It was a pretty painful semester, full of abstract concepts like vector subspaces, and I just couldn't connect with the material. It wasn't until the end of my graduate school career when I took a class that beat me over the head with examples of how powerful and magical this subject can be. I want to share that experience with more people.\n", 148 | "\n", 149 | "Associating linear algebra with a kind of fascination or awe is especially helpful, because things like matrices and vectors have a way of showing up in just about every quantitative situation. In my world of signal processing and statistics, it's virtually impossible to get through a research paper without being comfortable manipulating vectors and matrices. \n", 150 | "\n", 151 | "Where the magic really becomes clear is when - for whatever discipline you're working in - you manage to cram your problem into something that looks like \n", 152 | "\n", 153 | "
\n", 154 | "\n", 155 | "
\n", 156 | "$\n", 157 | "\\begin{equation*}\n", 158 | "y = Ax\n", 159 | "\\end{equation*}\n", 160 | "$\n", 161 | "
\n", 162 | "
\n", 163 | "\n", 164 | "suddenly you can apply ideas from finance, quantum mechanics, operations research, RADAR, medical imaging, or any number of other fields to solve your problem. The only thing you need to learn to be dangerous in any of these fields is an intution for the properties and structure of the matrix $A$.\n", 165 | "\n", 166 | "So how can four ASCII characters (not including the spaces) apply to so many situations in the real world almost stupidly well? In part because each of these symbols encompass a great deal of complexity. Let's start by unpacking them one by one:" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "### Vectors and Matrices\n", 174 | "\n", 175 | "All of this should be material you've seen in courses before, so I'll go through it very quickly just to put us on the same footing as far as notation.\n", 176 | "\n", 177 | "Beginning with $y$:\n", 178 | "\n", 179 | "$y$ is a **vector**, which is essentially a list of numbers. It's often useful to think of vectors as arrows that point from the origin to some point in space, where each entry of $y$ corresponds to a dimension in that space. For a 2-dimensional example, we could have a vector with entries $[2, 3]$ that one could visualize as \n", 180 | "\n", 181 | "\n", 182 | "\n", 183 | "As convention, lowercase variables without subscripts will be vectors. If $y$ is a vector, then $y_1$, $y_2 \\ldots$ are the entries of $y$." 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "Vectors add elementwise - that is, you add the elements in the corresponding positions. Graphically, this has the effect of starting one vector from the point of the other vector. The vector sum is the final point that is reached:\n", 191 | "\n", 192 | "\n" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "Vectors can be written in row or column form. For this presentation, all vectors are column vectors, and the corresponding row vectors are notated as e.g. $y^\\mathsf{T}$. The inner product (or dot product) of two vectors is \n", 200 | "\n", 201 | "
\n", 202 | "$y^\\mathsf{T} x=\n", 203 | "\\begin{bmatrix}\n", 204 | "y_1 & y_2 & \\ldots & y_n\n", 205 | "\\end{bmatrix}\n", 206 | "\\begin{bmatrix}\n", 207 | "x_1 \\\\\n", 208 | "x_2 \\\\\n", 209 | "\\vdots \\\\\n", 210 | "x_n\n", 211 | "\\end{bmatrix}\n", 212 | "= x_1y_1 + x_2y_2 + \\ldots + x_ny_n$\n", 213 | "
\n", 214 | "\n", 215 | "(This is a scalar)" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "$A$ is a **matrix**. Matrices are a grid of numbers, and can be thought of as a horizontal stack of column vectors (or a vertical stack of row vectors). Matrices add elementwise. Multiplying a matrix by a vector is equivalent to a batch inner product of the vector with the rows of $A$.\n", 223 | "\n", 224 | "I'll use uppercase variables to indicate matrices, and lowercase letters with two subscripts, e.g. $a_{i,j}$, indicate the element in the $i$th row and $j$th column of $A$. If you encounter a matrix element with only one subscript, such as $a_i$, this refers to the vector containing the $i$th column of $A$. A lowercase character with a tilde, e.g. $\\tilde{a}_j$ means the $j$th row of $A$ *as a column vector*." 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "In Python, we'll use Numpy arrays for all of our matrices and vectors:" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": 12, 237 | "metadata": { 238 | "ExecuteTime": { 239 | "end_time": "2019-11-04T04:34:26.087811Z", 240 | "start_time": "2019-11-04T04:34:26.075573Z" 241 | } 242 | }, 243 | "outputs": [ 244 | { 245 | "data": { 246 | "text/plain": [ 247 | "array([0, 1, 2])" 248 | ] 249 | }, 250 | "execution_count": 12, 251 | "metadata": {}, 252 | "output_type": "execute_result" 253 | } 254 | ], 255 | "source": [ 256 | "m, n = 3, 2\n", 257 | "\n", 258 | "# vectors are row vectors by default in Numpy\n", 259 | "y = np.arange(m)\n", 260 | "y" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": 13, 266 | "metadata": { 267 | "ExecuteTime": { 268 | "end_time": "2019-11-04T04:34:26.148504Z", 269 | "start_time": "2019-11-04T04:34:26.091874Z" 270 | } 271 | }, 272 | "outputs": [ 273 | { 274 | "data": { 275 | "text/plain": [ 276 | "array([[0],\n", 277 | " [1],\n", 278 | " [2]])" 279 | ] 280 | }, 281 | "execution_count": 13, 282 | "metadata": {}, 283 | "output_type": "execute_result" 284 | } 285 | ], 286 | "source": [ 287 | "# To get column vectors we need to do a reshape operation\n", 288 | "y = y.reshape(-1, 1)\n", 289 | "y" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": 14, 295 | "metadata": { 296 | "ExecuteTime": { 297 | "end_time": "2019-11-04T04:34:26.166216Z", 298 | "start_time": "2019-11-04T04:34:26.154737Z" 299 | } 300 | }, 301 | "outputs": [ 302 | { 303 | "data": { 304 | "text/plain": [ 305 | "array([[1, 2, 3],\n", 306 | " [4, 5, 6]])" 307 | ] 308 | }, 309 | "execution_count": 14, 310 | "metadata": {}, 311 | "output_type": "execute_result" 312 | } 313 | ], 314 | "source": [ 315 | "# Matrices are defined from lists or Numpy's construction tools\n", 316 | "A = np.array([[1, 2, 3],\n", 317 | " [4, 5, 6]])\n", 318 | "A" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": 15, 324 | "metadata": { 325 | "ExecuteTime": { 326 | "end_time": "2019-11-04T04:34:26.190495Z", 327 | "start_time": "2019-11-04T04:34:26.172962Z" 328 | } 329 | }, 330 | "outputs": [ 331 | { 332 | "data": { 333 | "text/plain": [ 334 | "array([[1., 1.],\n", 335 | " [1., 1.],\n", 336 | " [1., 1.]])" 337 | ] 338 | }, 339 | "execution_count": 15, 340 | "metadata": {}, 341 | "output_type": "execute_result" 342 | } 343 | ], 344 | "source": [ 345 | "B = np.ones((m, n))\n", 346 | "B" 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "metadata": {}, 352 | "source": [ 353 | "### Where do matrices come from?\n", 354 | "\n", 355 | "A system of linear equations\n", 356 | "\n", 357 | "
\n", 358 | "$\n", 359 | "\\begin{align*}\n", 360 | " y_1 &= a_{1,1}x_1 + a_{1,2}x_2 + a_{1,3}x_3 + \\ldots + a_{1,n}x_{n} \\\\\n", 361 | " y_2 &= a_{2,1}x_1 + a_{2,2}x_2 + a_{2,3}x_3 + \\ldots + a_{2,n}x_{n} \\\\\n", 362 | " \\vdots \\\\\n", 363 | " y_m &= a_{m,1}x_1 + a_{m,2}x_2 + a_{m,3}x_3 + \\ldots + a_{m,n}x_{n} \\\\\n", 364 | "\\end{align*}\n", 365 | "$\n", 366 | "
\n", 367 | " \n", 368 | "can be re-written as \n", 369 | " \n", 370 | "
\n", 371 | "$\n", 372 | "\\begin{bmatrix}\n", 373 | "y_1 \\\\\n", 374 | "y_2 \\\\\n", 375 | "\\vdots \\\\\n", 376 | "y_m\n", 377 | "\\end{bmatrix}\n", 378 | "=\n", 379 | "\\begin{bmatrix}\n", 380 | "a_{1,1} & a_{1,2} & a_{1,3} & \\ldots & a_{1,m} \\\\\n", 381 | "a_{2,1} & a_{2,2} & a_{2,3} & \\ldots & a_{2,m} \\\\\n", 382 | "\\vdots \\\\\n", 383 | "a_{n,1} & a_{n,2} & a_{n,3} & \\ldots & a_{n,m} \\\\\n", 384 | "\\end{bmatrix}\n", 385 | "\\begin{bmatrix}\n", 386 | "x_1 \\\\\n", 387 | "x_2 \\\\\n", 388 | "\\vdots \\\\\n", 389 | "x_m\n", 390 | "\\end{bmatrix}\n", 391 | "$\n", 392 | "
\n", 393 | "\n", 394 | "which we simplify as\n", 395 | "\n", 396 | "
\n", 397 | "$y=Ax$\n", 398 | "
\n" 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "metadata": {}, 404 | "source": [ 405 | "We say that \n", 406 | "\n", 407 | "$y \\in \\mathbb{R}^m$ \n", 408 | "$x \\in \\mathbb{R}^n$ \n", 409 | "$A \\in \\mathbb{R}^{m \\times n}$\n", 410 | "\n", 411 | "...but of course these could also be complex.\n", 412 | "\n", 413 | "I'll stick to this convention of using $m$ as the number of rows and $n$ as the number of columns everywhere." 414 | ] 415 | }, 416 | { 417 | "cell_type": "markdown", 418 | "metadata": {}, 419 | "source": [ 420 | "### Examples\n", 421 | "\n", 422 | "#### Classical Mechanics\n", 423 | "\n", 424 | "\n", 425 | "\n", 426 | "* $x$ represents a series of forces being applied at different points\n", 427 | "* $y$ represents the net force and torque on the total body\n", 428 | "* $\\mathbf{A}$ is determined by the geometry of the system" 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": {}, 434 | "source": [ 435 | "#### Buisness Operations\n", 436 | "\n", 437 | "\n", 438 | "\n", 439 | "e.g. Cost of production\n", 440 | "\n", 441 | "* $x$ represents the cost per component\n", 442 | "* $y$ represents the cost per finished product\n", 443 | "* $\\mathbf{A}$ is the Bill of Materials for each product" 444 | ] 445 | }, 446 | { 447 | "cell_type": "markdown", 448 | "metadata": {}, 449 | "source": [ 450 | "#### Statistics\n", 451 | "\n", 452 | "e.g. Autoregressive Models\n", 453 | "\n", 454 | "
\n", 455 | "
\n", 456 | "$\n", 457 | "\\begin{align*}\n", 458 | " x(T) &= a_1x(T-1) + a_2x(T-2) + a_3x(T-3) + \\ldots + a_nx(T-n) \\\\\n", 459 | "\\end{align*}\n", 460 | "$\n", 461 | "
\n", 462 | "\n", 463 | "
\n", 464 | "
\n", 465 | "\n", 466 | "
\n", 467 | "$\n", 468 | "\\begin{bmatrix}\n", 469 | "x(T+1) \\\\\n", 470 | "\\vdots \\\\\n", 471 | "x(n+2) \\\\\n", 472 | "x(n+1) \\\\\n", 473 | "\\end{bmatrix}\n", 474 | "=\n", 475 | "\\begin{bmatrix}\n", 476 | "x(T-n+1) & x(T-n+2) & x(T-n+3) & \\ldots & x(T) \\\\\n", 477 | "\\vdots \\\\\n", 478 | "x(2) & x(3) & x(4) & \\ldots & x(n+1) \\\\\n", 479 | "x(1) & x(2) & x(3) & \\ldots & x(n) \\\\\n", 480 | "\\end{bmatrix}\n", 481 | "\\begin{bmatrix}\n", 482 | "a_1 \\\\\n", 483 | "a_2 \\\\\n", 484 | "\\vdots \\\\\n", 485 | "a_n\n", 486 | "\\end{bmatrix}\n", 487 | "$\n", 488 | "
\n", 489 | "\n", 490 | "\n", 491 | "* $x$ is the coefficients of the model for each lag\n", 492 | "* $y$ is the time series output of the model\n", 493 | "* $\\mathbf{A}$ is a stack of lagged windows of the time series" 494 | ] 495 | }, 496 | { 497 | "cell_type": "markdown", 498 | "metadata": {}, 499 | "source": [ 500 | "### More than just Lines\n", 501 | "\n", 502 | "This last example of an Autoregressive Model, as well as the time series forecasting example at the beginning, are great for highlighting that linear algebra applies to situations that look anything but linear! Let's explore this idea more closely - consider the case of a polynomial evaluated at $m$ different points:\n", 503 | "\n", 504 | "
\n", 505 | "\n", 506 | "
\n", 507 | "$\n", 508 | "\\begin{align*}\n", 509 | " y_1 &= a_1 + a_{2}x_1 + a_{3}x_1^2 + \\ldots + a_{n}x_1^{n-1} \\\\\n", 510 | " y_2 &= a_1 + a_{2}x_2 + a_{3}x_2^2 + \\ldots + a_{n}x_2^{n-1} \\\\\n", 511 | " \\vdots \\\\\n", 512 | " y_m &= a_1 + a_{2}x_m + a_{3}x_m^2 + \\ldots + a_{n}x_m^{n-1} \\\\\n", 513 | "\\end{align*}\n", 514 | "$\n", 515 | "
\n", 516 | "\n", 517 | "
\n", 518 | "\n", 519 | "It's certainly true that these equations are not linear as far as $x$ is concerned. BUT! The coefficients of the polynomial are all first order, which means we can write the system of equations this way:\n", 520 | "
\n", 521 | "\n", 522 | "
\n", 523 | "$\n", 524 | "\\begin{bmatrix}\n", 525 | "y_1 \\\\\n", 526 | "y_2 \\\\\n", 527 | "\\vdots \\\\\n", 528 | "y_m \\\\\n", 529 | "\\end{bmatrix}\n", 530 | "=\n", 531 | "\\begin{bmatrix}\n", 532 | "1 & x_1 & x_1^2 & \\ldots & x_1^{n-1} \\\\\n", 533 | "1 & x_2 & x_2^2 & \\ldots & x_2^{n-1} \\\\\n", 534 | "\\vdots \\\\\n", 535 | "1 & x_m & x_m^2 & \\ldots & x_m^{n-1} \\\\\n", 536 | "\\end{bmatrix}\n", 537 | "\\begin{bmatrix}\n", 538 | "a_1 \\\\\n", 539 | "a_2 \\\\\n", 540 | "\\vdots \\\\\n", 541 | "a_n\n", 542 | "\\end{bmatrix}\n", 543 | "$\n", 544 | "
\n" 545 | ] 546 | }, 547 | { 548 | "cell_type": "markdown", 549 | "metadata": {}, 550 | "source": [ 551 | "In fact, there is absolutely no requirement on where the entries of $A$ come from! They often will arise from some horrendous expressions involving sines, logarithms, or enhanced spherical Riemann functions of the 60th kind (I made that up). As long as you can remain calm, slowly step back, and recognize that the **mixture** of these functions is still linear, then you still have\n", 552 | "\n", 553 | "
\n", 554 | "\n", 555 | "
\n", 556 | "$\n", 557 | "\\begin{align*}\n", 558 | " y_1 &= a_{1}f_1(x_1) + a_{2}f_2(x_1) + a_{3}f_3(x_1) + \\ldots + a_{n}f_n(x_1) \\\\\n", 559 | " y_2 &= a_{1}f_1(x_2) + a_{2}f_2(x_2) + a_{3}f_3(x_2) + \\ldots + a_{n}f_n(x_2) \\\\\n", 560 | " \\vdots \\\\\n", 561 | " y_m &= a_{1}f_1(x_m) + a_{2}f_2(x_m) + a_{3}f_3(x_m) + \\ldots + a_{n}f_n(x_m) \\\\\n", 562 | "\\end{align*}\n", 563 | "$\n", 564 | "
\n", 565 | "\n", 566 | "
\n", 567 | "
\n", 568 | "\n", 569 | "
\n", 570 | "$\n", 571 | "\\begin{bmatrix}\n", 572 | "y_1 \\\\\n", 573 | "y_2 \\\\\n", 574 | "\\vdots \\\\\n", 575 | "y_m \\\\\n", 576 | "\\end{bmatrix}\n", 577 | "=\n", 578 | "\\begin{bmatrix}\n", 579 | "f_1(x_1) & f_2(x_1) & f_3(x_1) & \\ldots & f_n(x_1) \\\\\n", 580 | "f_1(x_2) & f_2(x_2) & f_3(x_2) & \\ldots & f_n(x_2) \\\\\n", 581 | "\\vdots \\\\\n", 582 | "f_m(x_m) & f_2(x_m) & f_3(x_m) & \\ldots & f_n(x_m) \\\\\n", 583 | "\\end{bmatrix}\n", 584 | "\\begin{bmatrix}\n", 585 | "a_1 \\\\\n", 586 | "a_2 \\\\\n", 587 | "\\vdots \\\\\n", 588 | "a_n\n", 589 | "\\end{bmatrix}\n", 590 | "$\n", 591 | "
" 592 | ] 593 | }, 594 | { 595 | "cell_type": "markdown", 596 | "metadata": {}, 597 | "source": [ 598 | "#### Linearization\n", 599 | "\n", 600 | "Another reason why linear algebra applies to so many situations is that many problems which are *technically* nonlinear can be approximated as linear with a negligible amount of error. A common way this comes up (and a favorite approach of physicists everywhere) is by invoking a \"Taylor series approximation\". Imagine you are standing at a point $a$ on a number line, and you know the value of some function $f$ at your current location. If you were to move from $a$ to a new point $x$, the change you would see in the value of the function is a weighted sum of different derivatives of $f$:\n", 601 | "\n", 602 | "$f(x) = f(a)+\\frac {f'(a)}{1!} (x-a)+ \\frac{f''(a)}{2!} (x-a)^2+\\frac{f'''(a)}{3!}(x-a)^3+ \\cdots$ \n", 603 | "\n", 604 | "The weighting (read: \"practical importance\") of the terms depends on how far you have moved. For any function, in any practical situation, there will be some point close to $a$ where the error from throwing away all the terms which are nonlinear in $x$ is irrelevant to what you are trying to accomplish.\n", 605 | "\n", 606 | "That is, for $x$ \"very\" close to $a$: \n", 607 | " \n", 608 | "$f(x) \\sim f(a)+\\frac {f'(a)}{1!} (x-a)$\n", 609 | "\n", 610 | "It can be helpful to see this visually. The figure below shows Taylor approximations to $f(x)=\\sin(x)$ for $a=0$. The order of the approximations are 1, 3, 5, 7, 9, 11 and 13. Notice how in the range (-1, 1), a straight line looks reasonably equivalent to $\\sin(x)$! Whether or not it is \"reasonable\" depends on your application.\n", 611 | "\n", 612 | "\n", 613 | "\n" 614 | ] 615 | }, 616 | { 617 | "cell_type": "markdown", 618 | "metadata": {}, 619 | "source": [ 620 | "#### Example - GPS\n", 621 | "\n", 622 | "\n", 623 | "\n", 624 | "GPS sattelites orbit at an altitude of ~20 km1. For a 1-meter deviation off-axis, the resulting fractional error from linearizing would be " 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": 16, 630 | "metadata": { 631 | "ExecuteTime": { 632 | "end_time": "2019-11-04T04:34:26.885021Z", 633 | "start_time": "2019-11-04T04:34:26.870692Z" 634 | } 635 | }, 636 | "outputs": [ 637 | { 638 | "data": { 639 | "text/plain": [ 640 | "1.2500000593718141e-09" 641 | ] 642 | }, 643 | "execution_count": 16, 644 | "metadata": {}, 645 | "output_type": "execute_result" 646 | } 647 | ], 648 | "source": [ 649 | "x = 20e3\n", 650 | "y = 1\n", 651 | "\n", 652 | "(np.sqrt(x**2 + y**2) - x) / x" 653 | ] 654 | }, 655 | { 656 | "cell_type": "markdown", 657 | "metadata": {}, 658 | "source": [ 659 | "Assuming that you can live with a few nano-percent of error in locating your position, then GPS may as well be a linear problem.\n", 660 | "\n", 661 | "\n", 662 | "1https://www.gps.gov/systems/gps/space/" 663 | ] 664 | } 665 | ], 666 | "metadata": { 667 | "author": "l", 668 | "kernelspec": { 669 | "display_name": "Python 3", 670 | "language": "python", 671 | "name": "python3" 672 | }, 673 | "language_info": { 674 | "codemirror_mode": { 675 | "name": "ipython", 676 | "version": 3 677 | }, 678 | "file_extension": ".py", 679 | "mimetype": "text/x-python", 680 | "name": "python", 681 | "nbconvert_exporter": "python", 682 | "pygments_lexer": "ipython3", 683 | "version": "3.6.7" 684 | }, 685 | "latex_envs": { 686 | "LaTeX_envs_menu_present": true, 687 | "autoclose": true, 688 | "autocomplete": true, 689 | "bibliofile": "biblio.bib", 690 | "cite_by": "apalike", 691 | "current_citInitial": 1, 692 | "eqLabelWithNumbers": true, 693 | "eqNumInitial": 1, 694 | "hotkeys": { 695 | "equation": "Ctrl-E", 696 | "itemize": "Ctrl-I" 697 | }, 698 | "labels_anchors": false, 699 | "latex_user_defs": false, 700 | "report_style_numbering": false, 701 | "user_envs_cfg": false 702 | }, 703 | "toc": { 704 | "base_numbering": 1, 705 | "nav_menu": {}, 706 | "number_sections": false, 707 | "sideBar": true, 708 | "skip_h1_title": false, 709 | "title_cell": "Table of Contents", 710 | "title_sidebar": "Contents", 711 | "toc_cell": false, 712 | "toc_position": {}, 713 | "toc_section_display": true, 714 | "toc_window_display": false 715 | } 716 | }, 717 | "nbformat": 4, 718 | "nbformat_minor": 2 719 | } 720 | -------------------------------------------------------------------------------- /Ch 2 - Rank Nullspace and Determinants.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Linear Algebra Basics\n", 8 | "\n", 9 | "### a.k.a. \"Stuff You've Seen Before\"\n", 10 | "\n", 11 | "Assuming you've drunk the Kool-Aid that linear algebra is useful, let's start dusting off some of those neurons from your introductory classes. In this notebook, I'll focus on visualizations and applications for these topics:\n", 12 | "\n", 13 | "* Rank\n", 14 | "* Nullspace\n", 15 | "* Determinant & Singularity\n" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "### Python Setup" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "metadata": { 29 | "ExecuteTime": { 30 | "end_time": "2019-11-04T04:37:09.194122Z", 31 | "start_time": "2019-11-04T04:37:08.385935Z" 32 | } 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "# Render MPL figures within notebook cells\n", 37 | "%matplotlib inline\n", 38 | "\n", 39 | "# Import python libraries\n", 40 | "import numpy as np\n", 41 | "import matplotlib.pyplot as plt\n", 42 | "from matplotlib import rcParams" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 2, 48 | "metadata": { 49 | "ExecuteTime": { 50 | "end_time": "2019-11-04T04:37:09.202091Z", 51 | "start_time": "2019-11-04T04:37:09.197194Z" 52 | } 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "# Configure some defaults for plots\n", 57 | "rcParams['font.size'] = 16\n", 58 | "rcParams['figure.figsize'] = (10, 3)" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 3, 64 | "metadata": { 65 | "ExecuteTime": { 66 | "end_time": "2019-11-04T04:37:09.260035Z", 67 | "start_time": "2019-11-04T04:37:09.205186Z" 68 | } 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "# Set Numpy's random number generator so the same results are produced each time the notebook is run\n", 73 | "np.random.seed(0)" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### The Sensor Interpretation of a Linear System\n", 81 | "\n", 82 | "It can be helpful to imagine $y=Ax$ as the following:\n", 83 | "\n", 84 | "* $x$ describes the outputs of $n$ transmitters\n", 85 | "* $y$ contains $m$ sensor measurements\n", 86 | "* $a_{i,j}$ is the impact of transmitter $j$ on sensor $i$\n", 87 | "\n", 88 | "\n", 89 | "\n", 90 | "With this picture, you can start to think of columns as the \"fingerprints\" of the inputs - if you crank up the transmission power from $x_3$, then $y$ starts to look more like $a_3$. Another perspective is that the vector $x$ is a recipe for mixture of columns of $A$." 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "### Rank\n", 98 | "\n", 99 | "The **rank** of a matrix is a number that tells you the size of the largest group of rows or columns of $A$ that form a [linearly independent](https://en.wikipedia.org/wiki/Linear_independence) set.\n", 100 | "\n", 101 | "Consider the following matrix:" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 4, 107 | "metadata": { 108 | "ExecuteTime": { 109 | "end_time": "2019-11-04T04:37:09.279271Z", 110 | "start_time": "2019-11-04T04:37:09.265044Z" 111 | } 112 | }, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "array([[1., 0., 0.],\n", 118 | " [0., 1., 0.],\n", 119 | " [0., 0., 1.]])" 120 | ] 121 | }, 122 | "execution_count": 4, 123 | "metadata": {}, 124 | "output_type": "execute_result" 125 | } 126 | ], 127 | "source": [ 128 | "# Create a matrix whose rows form a linearly-independent set\n", 129 | "A = np.eye(3)\n", 130 | "A" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "No matter how hard I want to, there is no way that I can mix the first two rows of $A$ by scaling and adding them that will result in the third row. Thus, the rank of $A$ is 3, which Numpy cheerfully tells us:" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": 5, 143 | "metadata": { 144 | "ExecuteTime": { 145 | "end_time": "2019-11-04T04:37:09.295786Z", 146 | "start_time": "2019-11-04T04:37:09.279271Z" 147 | } 148 | }, 149 | "outputs": [ 150 | { 151 | "data": { 152 | "text/plain": [ 153 | "3" 154 | ] 155 | }, 156 | "execution_count": 5, 157 | "metadata": {}, 158 | "output_type": "execute_result" 159 | } 160 | ], 161 | "source": [ 162 | "np.linalg.matrix_rank(A)" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "metadata": {}, 168 | "source": [ 169 | "A good property to remember is that $\\text{rank}(A) \\le \\min\\{m, n\\}$. As soon as we add a new row, we are guaranteed that it will be a linear combination of the previous rows:" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": 6, 175 | "metadata": { 176 | "ExecuteTime": { 177 | "end_time": "2019-11-04T04:37:09.312709Z", 178 | "start_time": "2019-11-04T04:37:09.296783Z" 179 | } 180 | }, 181 | "outputs": [ 182 | { 183 | "data": { 184 | "text/plain": [ 185 | "3" 186 | ] 187 | }, 188 | "execution_count": 6, 189 | "metadata": {}, 190 | "output_type": "execute_result" 191 | } 192 | ], 193 | "source": [ 194 | "# Row 3 = (Row 1) * 2 + (Row 2) + (Row 3)\n", 195 | "\n", 196 | "B = np.array([[1, 0, 0],\n", 197 | " [0, 1, 0],\n", 198 | " [0, 0, 1],\n", 199 | " [2, 1, 1]])\n", 200 | "\n", 201 | "np.linalg.matrix_rank(B)" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": 7, 207 | "metadata": { 208 | "ExecuteTime": { 209 | "end_time": "2019-11-04T04:37:09.325497Z", 210 | "start_time": "2019-11-04T04:37:09.316715Z" 211 | } 212 | }, 213 | "outputs": [ 214 | { 215 | "data": { 216 | "text/plain": [ 217 | "3" 218 | ] 219 | }, 220 | "execution_count": 7, 221 | "metadata": {}, 222 | "output_type": "execute_result" 223 | } 224 | ], 225 | "source": [ 226 | "# Transposing a matrix (flipping rows and columns) preserves rank\n", 227 | "np.linalg.matrix_rank(B.T)" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "When $\\text{rank}(A) = \\min\\{m, n\\}$, we say that $A$ is **full rank**. \n", 235 | "\n", 236 | "**Question:** what is the rank of the matrix $A$ with columns $[a_1 \\; a_2 \\; a_3]$, shown below?\n", 237 | "\n", 238 | "\n", 239 | "\n", 240 | "First, notice that $A \\in \\mathbb{R}^{2 \\times 3}$. Even though the set of vectors is not independent, the rank is still full (2) because at least 2 of the vectors are not colinear." 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "metadata": {}, 246 | "source": [ 247 | "#### Application: Compression + Efficiency\n", 248 | "\n", 249 | "As soon as there is some notion of redundant information, one can start thinking about applications in compression. Take the highly contrived but illustrative example of storing a matrix with 1,000,000 rows and 2 columns, where the 2nd column is exactly double the first column. That is,\n", 250 | "\n", 251 | "
\n", 252 | "$\n", 253 | "A =\n", 254 | " \\begin{bmatrix}\n", 255 | " a_{1,1} & 2a_{1,1} \\\\\n", 256 | " a_{2,1} & 2a_{2,1} \\\\\n", 257 | " \\vdots \\\\\n", 258 | " a_{n,1} & 2a_{n,1} \\\\\n", 259 | " \\end{bmatrix}\n", 260 | "$\n", 261 | "
\n", 262 | "\n", 263 | "This would be a silly thing to do, when one could immediately save half of the memory by storing only the first column, and remembering to double the value whenever we need to index into the 2nd column. We can achieve this by representing $A$ as the following:\n", 264 | "\n", 265 | "
\n", 266 | "$\n", 267 | "A =\n", 268 | " \\begin{bmatrix}\n", 269 | " a_{1,1} & 2a_{1,1} \\\\\n", 270 | " a_{2,1} & 2a_{2,1} \\\\\n", 271 | " \\vdots \\\\\n", 272 | " a_{n,1} & 2a_{n,1} \\\\\n", 273 | " \\end{bmatrix}\n", 274 | "=\n", 275 | " \\begin{bmatrix}\n", 276 | " a_{1,1} \\\\\n", 277 | " a_{2,1} \\\\\n", 278 | " \\vdots \\\\\n", 279 | " a_{n,1} \\\\\n", 280 | " \\end{bmatrix}\n", 281 | " \\begin{bmatrix}\n", 282 | " 1 & 2\n", 283 | " \\end{bmatrix}\n", 284 | "$\n", 285 | "
" 286 | ] 287 | }, 288 | { 289 | "cell_type": "markdown", 290 | "metadata": {}, 291 | "source": [ 292 | "This idea generalizes to matrices of any shape:\n", 293 | "\n", 294 | "> A matrix $A$ with rank $r$ can be factored into matrices $Q \\in \\mathbb{R}^{m \\times r}$ and $R \\in \\mathbb{R}^{r \\times n}$\n", 295 | "\n", 296 | "\n", 297 | "\n", 298 | "Our lovely Mathematician friends have given us multiple algorithms that will produce these smaller matrices for us (see [QR decomposition](https://en.wikipedia.org/wiki/QR_decomposition)). We'll see one such algorithm in a later chapter.\n", 299 | "\n", 300 | "Armed with this fact, we can quickly calculate how many numbers we have to store for the original matrix versus the factorized matrices:\n", 301 | "\n", 302 | "$\\;\\;$ Size of $A = mn$\n", 303 | "\n", 304 | "$\\;\\;$ Size of $Q$ + Size of $R = mr + rn = r(m+n)$\n", 305 | "\n", 306 | "As soon as $r$ becomes appreciably smaller than $m$ and $n$, this trick can save a lot of space:" 307 | ] 308 | }, 309 | { 310 | "cell_type": "code", 311 | "execution_count": 8, 312 | "metadata": { 313 | "ExecuteTime": { 314 | "end_time": "2019-11-04T04:37:09.341092Z", 315 | "start_time": "2019-11-04T04:37:09.330100Z" 316 | } 317 | }, 318 | "outputs": [ 319 | { 320 | "data": { 321 | "text/plain": [ 322 | "1000000" 323 | ] 324 | }, 325 | "execution_count": 8, 326 | "metadata": {}, 327 | "output_type": "execute_result" 328 | } 329 | ], 330 | "source": [ 331 | "m = 1000\n", 332 | "n = 1000\n", 333 | "r = 10\n", 334 | "\n", 335 | "# Storage cost of A\n", 336 | "m*n" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": 9, 342 | "metadata": { 343 | "ExecuteTime": { 344 | "end_time": "2019-11-04T04:37:09.354373Z", 345 | "start_time": "2019-11-04T04:37:09.345070Z" 346 | } 347 | }, 348 | "outputs": [ 349 | { 350 | "data": { 351 | "text/plain": [ 352 | "20000" 353 | ] 354 | }, 355 | "execution_count": 9, 356 | "metadata": {}, 357 | "output_type": "execute_result" 358 | } 359 | ], 360 | "source": [ 361 | "# Storage cost of QR\n", 362 | "m*r + r*n" 363 | ] 364 | }, 365 | { 366 | "cell_type": "markdown", 367 | "metadata": {}, 368 | "source": [ 369 | "**Multiplication Efficiency**\n", 370 | "\n", 371 | "Amusingly, the math is identical when you estimate how many floating-point operations (FLOPs) it takes to compute $Ax = Q(Rx)$\n", 372 | "\n", 373 | "$\\;\\;$ FLOPs to compute $Ax$ = (# rows) x FLOPs to compute ($\\tilde{a}^\\mathsf{T}x)$ $\\sim O(mn)$\n", 374 | "\n", 375 | "$\\;\\;$ FLOPs to compute $QRx\\sim O(mr + rn) = O(r(m+n))$" 376 | ] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "metadata": {}, 381 | "source": [ 382 | "### Nullspace\n", 383 | "\n", 384 | "\"The Nullspace\" is a term that haunted me from my first linear algebra class. I remember thinking it sounded cool, but I had no grasp of what it really meant. I think the simplest way of understanding the idea is this: every matrix has a nullspace, which is a list of vectors. If you multiply your matrix $A$ by a vector $z$ and the result is the zero vector, then you put $z$ in the list.\n", 385 | "\n", 386 | "> The nullspace of $A$ is the set of vectors, $z$, such that $Az = 0$\n", 387 | "\n", 388 | "If we were to express this in code, `A.nullspace()` would return a list of vectors. This list would be infinitely long, but one can return a list of vectors that can be used to reconstruct any vector in the nullspace via linear combinations (we would call this finite set a **basis** for the nullspace). There isn't a direct way to return the nullspace of $A$ with Numpy, but we will cover how to do this in a later chapter (see Ch 4 - Singular Value Decomposition).\n", 389 | "\n", 390 | "The **key idea** about vectors in the nullspace of $A$ is that they represent redundancy or flexibility in the system:\n", 391 | "\n", 392 | "$A(x + z) = Ax + Az = Ax + 0 = Ax$\n", 393 | "\n", 394 | "We can add any vector from the nullspace \"for free.\" Visually, if we plot the points $x$ where $Ax$ equals some constant $b$, then any vector that is parallel to the line is in the nullspace of $A$:\n", 395 | "\n", 396 | "\n", 397 | "\n", 398 | "The points making up the green line represent all the vectors in the nullspace of A, for this example.\n", 399 | "\n", 400 | "This is great if, for instance, we have $y=Ax$ where $y$ is a target we are trying to achieve. Lots of vectors in the nullspace means we have options to choose from for our input, and we can optimize on some other criteria.\n", 401 | "\n", 402 | "However, if you recall our transmitter/receiver analogy from above, having vectors in the nullspace of our matrix can be a very bad thing! Imagine you are trying to reconstruct what your transmitters were sending from the sensor measurements you detect. If there are multiple ways to transmit signals that produce the same received measurements, **you can never know which signals were sent.**" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "metadata": {}, 408 | "source": [ 409 | "### Singularity\n", 410 | "\n", 411 | "This last idea, that you can't \"undo\" the transformation from multiplying $x$ by $A$, is mathematically expressed by saying that $A$ is not invertible. Precisely, there is no matrix $B$ such that \n", 412 | "\n", 413 | "
\n", 414 | " $x = B(Ax)$\n", 415 | "
" 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "metadata": {}, 421 | "source": [ 422 | "This idea is also expressed by saying that \"$A$ has no left inverse\". In the special case where $A$ is square, you will encounter people using the term **singular** to mean a matrix which has no inverse. As best I can find, the term singular refers to an (incorrect) understanding that most square matrices are invertible, and so non-invertible matrices are special. " 423 | ] 424 | }, 425 | { 426 | "cell_type": "markdown", 427 | "metadata": {}, 428 | "source": [ 429 | "### The Determinant\n", 430 | "\n", 431 | "Another abstract concept you would undoubtedly find in a beginning linear algebra class is **the determinant** of a matrix. You probably were drilled at length on using [Cramer's Rule](https://en.wikipedia.org/wiki/Cramer%27s_rule) to calculate the determinant, and that was probably all you would have done with it.\n", 432 | "\n", 433 | "The determinant is connected to all of the previous topics in this section, which I have grouped together because they all indicate whether or not a matrix operation can be undone. In applied terms, whether or not an estimation problem has a unique solution, or if a design problem has multiple choices.\n", 434 | "\n", 435 | "The determinant is a number that **determines** if a square matrix is singular. That's where the name comes from. If $\\det(A)=0$, then $A$ has no inverse.\n", 436 | "\n", 437 | "Ok great. Why?\n", 438 | "\n", 439 | "There is a great visual interpretation - we'll work in a 3-dimensional space so that we can draw it, but this extends to any number of dimensions. Suppose we have a cube with volume 1. Applying the matrix $A$ to the vectors which define the points on the boundary of the cube results in a parallelopiped with volume equal to the determinant of $A$:\n", 440 | "\n", 441 | "\n", 442 | "\n", 443 | "If the determinant of $A$ is 0, the resulting parallelopiped has volume 0, which implies at least one dimension of the cube is flattened:\n", 444 | "\n", 445 | "\n", 446 | "\n", 447 | "In the picture above, any information in the vertical direction is lost - you can't unflatten the shape because you don't know how high to stretch!" 448 | ] 449 | }, 450 | { 451 | "cell_type": "markdown", 452 | "metadata": {}, 453 | "source": [ 454 | "We can demonstrate this volume concept in code:" 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": 10, 460 | "metadata": { 461 | "ExecuteTime": { 462 | "end_time": "2019-11-04T04:37:09.369929Z", 463 | "start_time": "2019-11-04T04:37:09.359000Z" 464 | } 465 | }, 466 | "outputs": [ 467 | { 468 | "data": { 469 | "text/plain": [ 470 | "array([[1., 0., 0.],\n", 471 | " [0., 1., 0.],\n", 472 | " [0., 0., 1.]])" 473 | ] 474 | }, 475 | "execution_count": 10, 476 | "metadata": {}, 477 | "output_type": "execute_result" 478 | } 479 | ], 480 | "source": [ 481 | "# Define a 3d unit cube\n", 482 | "C = np.eye(3)\n", 483 | "C" 484 | ] 485 | }, 486 | { 487 | "cell_type": "code", 488 | "execution_count": 11, 489 | "metadata": { 490 | "ExecuteTime": { 491 | "end_time": "2019-11-04T04:37:09.385231Z", 492 | "start_time": "2019-11-04T04:37:09.373653Z" 493 | } 494 | }, 495 | "outputs": [ 496 | { 497 | "data": { 498 | "text/plain": [ 499 | "array([[1, 4, 2],\n", 500 | " [1, 4, 4],\n", 501 | " [4, 4, 2]])" 502 | ] 503 | }, 504 | "execution_count": 11, 505 | "metadata": {}, 506 | "output_type": "execute_result" 507 | } 508 | ], 509 | "source": [ 510 | "# Define a transformation matrix\n", 511 | "A = np.random.randint(low=1, high=5, size=(3, 3))\n", 512 | "A" 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": 12, 518 | "metadata": { 519 | "ExecuteTime": { 520 | "end_time": "2019-11-04T04:37:09.400901Z", 521 | "start_time": "2019-11-04T04:37:09.389461Z" 522 | } 523 | }, 524 | "outputs": [ 525 | { 526 | "data": { 527 | "text/plain": [ 528 | "array([[1., 4., 2.],\n", 529 | " [1., 4., 4.],\n", 530 | " [4., 4., 2.]])" 531 | ] 532 | }, 533 | "execution_count": 12, 534 | "metadata": {}, 535 | "output_type": "execute_result" 536 | } 537 | ], 538 | "source": [ 539 | "# Since the unit cube vectors correspond to an Identity matrix, the output of the transform\n", 540 | "# is the same as the transformation matrix\n", 541 | "A @ C" 542 | ] 543 | }, 544 | { 545 | "cell_type": "code", 546 | "execution_count": 13, 547 | "metadata": { 548 | "ExecuteTime": { 549 | "end_time": "2019-11-04T04:37:09.417521Z", 550 | "start_time": "2019-11-04T04:37:09.404196Z" 551 | } 552 | }, 553 | "outputs": [ 554 | { 555 | "data": { 556 | "text/plain": [ 557 | "24.000000000000004" 558 | ] 559 | }, 560 | "execution_count": 13, 561 | "metadata": {}, 562 | "output_type": "execute_result" 563 | } 564 | ], 565 | "source": [ 566 | "# The volume of the transformed cube is given by the determinant\n", 567 | "np.linalg.det(A)" 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": 14, 573 | "metadata": { 574 | "ExecuteTime": { 575 | "end_time": "2019-11-04T04:37:09.434914Z", 576 | "start_time": "2019-11-04T04:37:09.420512Z" 577 | } 578 | }, 579 | "outputs": [ 580 | { 581 | "data": { 582 | "text/plain": [ 583 | "0.0" 584 | ] 585 | }, 586 | "execution_count": 14, 587 | "metadata": {}, 588 | "output_type": "execute_result" 589 | } 590 | ], 591 | "source": [ 592 | "# If we make two of the columns of A identical, this will have the effect of \n", 593 | "# Making the parallelopiped effectively 2-dimensional. The determinant will be\n", 594 | "# zero to reflect this.\n", 595 | "A_flat = A\n", 596 | "A_flat[:, 2] = A[:, 1]\n", 597 | "np.linalg.det(A_flat)" 598 | ] 599 | }, 600 | { 601 | "cell_type": "markdown", 602 | "metadata": {}, 603 | "source": [ 604 | "### Unification\n", 605 | "\n", 606 | "You may have noticed just now that, to create a matrix with a zero determinant, I made one column of A equal to another column. This dropped the rank of $A$. This was not a coincidence. In fact, the rank of $A$ can also be interpreted as the number of dimensions in the accessible output space (the **range**) of $A$ - if this number is less than the number of dimensions of $x$, then some kind of geometric flattening is going to happen and the determinant will be zero. \n", 607 | "\n", 608 | "At the same time, removing a dimension from the range of $A$ **adds** a dimension to the nullspace of $A$! There is a handy property:\n", 609 | "\n", 610 | "> $\\text{Rank}(A) + \\text{dim Null}(A) = n$\n", 611 | "\n", 612 | "All of the topics in this section are intimately connected, each communicating whether or not a matrix is invertible. To summarize, the following statements are equivalent:\n", 613 | "\n", 614 | "- $A$ is full rank\n", 615 | "- $A$ has only the zero vector in its nullspace\n", 616 | "- the determinant of $A$ is non-zero\n", 617 | "- $A$ has a left inverse" 618 | ] 619 | }, 620 | { 621 | "cell_type": "code", 622 | "execution_count": null, 623 | "metadata": {}, 624 | "outputs": [], 625 | "source": [] 626 | } 627 | ], 628 | "metadata": { 629 | "author": "", 630 | "kernelspec": { 631 | "display_name": "Python 3", 632 | "language": "python", 633 | "name": "python3" 634 | }, 635 | "language_info": { 636 | "codemirror_mode": { 637 | "name": "ipython", 638 | "version": 3 639 | }, 640 | "file_extension": ".py", 641 | "mimetype": "text/x-python", 642 | "name": "python", 643 | "nbconvert_exporter": "python", 644 | "pygments_lexer": "ipython3", 645 | "version": "3.6.7" 646 | }, 647 | "latex_envs": { 648 | "LaTeX_envs_menu_present": true, 649 | "autoclose": true, 650 | "autocomplete": false, 651 | "bibliofile": "biblio.bib", 652 | "cite_by": "apalike", 653 | "current_citInitial": 1, 654 | "eqLabelWithNumbers": false, 655 | "eqNumInitial": 1, 656 | "hotkeys": { 657 | "equation": "Ctrl-E", 658 | "itemize": "Ctrl-I" 659 | }, 660 | "labels_anchors": false, 661 | "latex_user_defs": false, 662 | "report_style_numbering": false, 663 | "user_envs_cfg": false 664 | }, 665 | "toc": { 666 | "base_numbering": 1, 667 | "nav_menu": {}, 668 | "number_sections": false, 669 | "sideBar": true, 670 | "skip_h1_title": false, 671 | "title_cell": "Table of Contents", 672 | "title_sidebar": "Contents", 673 | "toc_cell": false, 674 | "toc_position": {}, 675 | "toc_section_display": true, 676 | "toc_window_display": true 677 | } 678 | }, 679 | "nbformat": 4, 680 | "nbformat_minor": 2 681 | } 682 | -------------------------------------------------------------------------------- /Ch 3 - Eigenvalues and Eigenvectors.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Eigenvalues and Eigenvectors\n", 8 | "\n", 9 | "This chapter will revisit another significant topic that you would have seen in a typical linear algebra course. During my class experience, I recall going through mountainous piles of loose-leaf paper, solving characteristic polynomials to find eigenvectors of 2x2 matrices, without any motivation for the practice. \n", 10 | "\n", 11 | "Eigenvalues and vectors turn out to have myriad applications, and hopefully the content here will help remind you how these things work and provide a context to understand them when they show up elsewhere." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "### Python Setup" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 2, 24 | "metadata": { 25 | "ExecuteTime": { 26 | "end_time": "2019-11-04T15:28:26.182425Z", 27 | "start_time": "2019-11-04T15:28:24.643827Z" 28 | } 29 | }, 30 | "outputs": [], 31 | "source": [ 32 | "# Render MPL figures within notebook cells\n", 33 | "%matplotlib inline\n", 34 | "\n", 35 | "# Import python libraries\n", 36 | "import numpy as np\n", 37 | "import matplotlib.pyplot as plt\n", 38 | "from matplotlib import rcParams" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": 3, 44 | "metadata": { 45 | "ExecuteTime": { 46 | "end_time": "2019-11-04T15:28:26.193748Z", 47 | "start_time": "2019-11-04T15:28:26.187041Z" 48 | } 49 | }, 50 | "outputs": [], 51 | "source": [ 52 | "# Configure some defaults for plots\n", 53 | "rcParams['font.size'] = 16\n", 54 | "# rcParams['figure.figsize'] = (10, 3)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 4, 60 | "metadata": { 61 | "ExecuteTime": { 62 | "end_time": "2019-11-04T15:28:26.205917Z", 63 | "start_time": "2019-11-04T15:28:26.198928Z" 64 | } 65 | }, 66 | "outputs": [], 67 | "source": [ 68 | "# Set Numpy's random number generator so the same results are produced each time the notebook is run\n", 69 | "np.random.seed(0)" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "### A Quick Review\n", 77 | "\n", 78 | "It's helpful to start with a geometric picture. If you multiply a vector with a square matrix, the output will be another vector in the same space, but in general will \"point\" in another direction. For any matrix $A$, there are a set of special vectors $v$ whose output is a scaled version of the input:\n", 79 | "\n", 80 | "\n", 81 | "\n", 82 | "This relationship is expressed mathematically as \n", 83 | "\n", 84 | "
\n", 85 | " $Av = \\lambda v$\n", 86 | "
\n", 87 | "\n", 88 | "where $v$ is called an **eigenvector** and the scaling factor $\\lambda$ is the associated **eigenvalue**. To emphasize, since the output has the same dimensions as the input, eigenvectors only make sense in the context of square matrices. A square matrix of dimension $n$ may have up to $n$ eigenvectors." 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": {}, 94 | "source": [ 95 | "#### Finding Eigenvectors and Eigenvalues\n", 96 | "\n", 97 | "Given the definition of an eigenpair is $Av=\\lambda v$, it has to be true that $(A-\\lambda I)v=0$. Referring back to the previous chapter, this means that $v$ is in the nullspace of the matrix $M = (A - \\lambda I)$. This, in turn, means that $M$ **has** non-zero vectors in its nullspace, which means that the determinant must be zero.\n", 98 | "\n", 99 | "This is where in a linear algebra class, one would start writing out the determinants of matrices and solving for values of $\\lambda$ which produce a zero determinant. In practice, this is not an efficient way to find eigenpairs. Numpy uses the LAPACK routine `geev`:" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": 5, 105 | "metadata": { 106 | "ExecuteTime": { 107 | "end_time": "2019-11-04T15:28:26.225379Z", 108 | "start_time": "2019-11-04T15:28:26.211441Z" 109 | } 110 | }, 111 | "outputs": [ 112 | { 113 | "data": { 114 | "text/plain": [ 115 | "array([[1, 0, 0, 0, 0],\n", 116 | " [0, 2, 0, 0, 0],\n", 117 | " [0, 0, 3, 0, 0],\n", 118 | " [0, 0, 0, 4, 0],\n", 119 | " [0, 0, 0, 0, 5]])" 120 | ] 121 | }, 122 | "execution_count": 5, 123 | "metadata": {}, 124 | "output_type": "execute_result" 125 | } 126 | ], 127 | "source": [ 128 | "A = np.diag([1, 2, 3, 4, 5])\n", 129 | "A" 130 | ] 131 | }, 132 | { 133 | "cell_type": "code", 134 | "execution_count": 6, 135 | "metadata": { 136 | "ExecuteTime": { 137 | "end_time": "2019-11-04T15:28:26.251245Z", 138 | "start_time": "2019-11-04T15:28:26.230343Z" 139 | } 140 | }, 141 | "outputs": [ 142 | { 143 | "data": { 144 | "text/plain": [ 145 | "array([1., 2., 3., 4., 5.])" 146 | ] 147 | }, 148 | "execution_count": 6, 149 | "metadata": {}, 150 | "output_type": "execute_result" 151 | } 152 | ], 153 | "source": [ 154 | "λ, v = np.linalg.eig(A)\n", 155 | "\n", 156 | "# λ contains the eigenvalues\n", 157 | "λ" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 7, 163 | "metadata": { 164 | "ExecuteTime": { 165 | "end_time": "2019-11-04T15:28:26.267187Z", 166 | "start_time": "2019-11-04T15:28:26.256265Z" 167 | } 168 | }, 169 | "outputs": [ 170 | { 171 | "data": { 172 | "text/plain": [ 173 | "array([[1., 0., 0., 0., 0.],\n", 174 | " [0., 1., 0., 0., 0.],\n", 175 | " [0., 0., 1., 0., 0.],\n", 176 | " [0., 0., 0., 1., 0.],\n", 177 | " [0., 0., 0., 0., 1.]])" 178 | ] 179 | }, 180 | "execution_count": 7, 181 | "metadata": {}, 182 | "output_type": "execute_result" 183 | } 184 | ], 185 | "source": [ 186 | "# v contains the eigenvectors as columns\n", 187 | "v" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": {}, 193 | "source": [ 194 | "### Time Dynamics\n", 195 | "\n", 196 | "One of my favorite applications of eigenvectors is with systems that evolve over time - think object tracking or evolution of financial portfolios, etc. Any kind of time series data.\n", 197 | "\n", 198 | "In this context, a vector is a list of the parameters that describe the state of your system at a particular time. A nice, concrete example of \"the state\" from classical mechanics would be the position, velocity, and acceleration of an object. In finance it could be the dollar amounts in a list of assets.\n", 199 | "\n", 200 | "The matrix $A$ describes the process that updates the state from time $t$ to $t+1$: \n", 201 | "
\n", 202 | "
\n", 203 | " $x(t+1) = Ax(t)$\n", 204 | "
\n", 205 | "
\n", 206 | "It's possible to describe the state at an arbitrary time as a function of the input state using entirely terms that depend on the eigenvectors and eigenvalues:\n", 207 | "
\n", 208 | "
\n", 209 | " $x(t) = (\\lambda_1^t v_1 + \\ldots)x(0)$\n", 210 | "
\n", 211 | "
\n", 212 | "\n", 213 | "Now, the $\\ldots$ above are hiding an ugly mess involving binomial coefficients and such - my aim is to simply highlight that every term contains one eigenvector scaled by something that looks like the matching eigenvalue raised to a power which grows with time. In special cases, if $A$ is diagonalizable, then the terms in the equation above becomes extremely simple: \n", 214 | "
\n", 215 | "
\n", 216 | " $x(t) = (\\lambda_1^t v_1 + \\lambda_2^t v_2 + \\ldots)x(0) = \\sum_{i=1}^n \\lambda_i^t v_i x(0)$\n", 217 | "
\n", 218 | "
\n", 219 | "\n", 220 | "First of all, how amazing is it that eigenvectors appear in this setting at all! These weird geometric oddities can describe how systems evolve in time from any starting state. Second, and just as amazing, this structure means that linear systems can only evolve in time in a few specific ways:\n", 221 | "\n", 222 | "\n", 223 | "\n", 224 | "1. Terms associated with eigenvalues of magnitude = 1 stay the same over time\n", 225 | "2. Terms associated with eigenvalues of magnitude < 1 decay over time\n", 226 | "3. Terms associated with eigenvalues of magnitude > 1 stay grow exponentially over time\n", 227 | "\n", 228 | "For a matrix with all real entries, it's possible to have complex eigenvalues and eigenvectors. Complex values will be associated with oscillating behavior, shown by dashed lines in the image above. The asymptotic behavior is still determined by the magnitude of the eigenvalue. " 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "### Markov Processes\n", 236 | "\n", 237 | "A fun example to show this idea is that of **Markov Processes**. In a Markov process, at any point in time we can be in one of several states. Each time step, we can transition to one of the other states with some probability based on where we are currently. An example system could be an employee - at any point the employee could be thinking, working, or idle:\n", 238 | "\n", 239 | "\n", 240 | "\n", 241 | "If we ran a simulation using coin flips to decide if we change states, we would end up with a sequence like \n", 242 | "\n", 243 | "*idle - working - idle - thinking - thinking - working - thinking...* \n", 244 | "\n", 245 | "This sequence is called a **Markov Chain**. A classic question one might ask with a model like this is \"what is the probability at any time that the employee is producing results?\"" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "Markov processes are linear. The vector $x$ represents the probability at time $t$ that we are in state 1 through state $n$. The square matrix $P$ is called the state transition matrix, and the entry $p_{i,j}$ is the probability of moving from state $j$ to state $i$:\n", 253 | "\n", 254 | "" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "Ok, let's do an example. When I was in high school, I had a particularly boring hamster that had a set routine: *sleep - run on wheel - eat - sleep...*\n", 262 | "\n", 263 | "Here's the Markov process for my hamster:\n", 264 | "\n", 265 | "" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 8, 271 | "metadata": { 272 | "ExecuteTime": { 273 | "end_time": "2019-11-04T15:28:26.283172Z", 274 | "start_time": "2019-11-04T15:28:26.272017Z" 275 | } 276 | }, 277 | "outputs": [], 278 | "source": [ 279 | "# The associated transition matrix is P\n", 280 | "P = np.array([[0, 0, 1],\n", 281 | " [1, 0, 0],\n", 282 | " [0, 1, 0]])\n", 283 | "\n", 284 | "λ, v = np.linalg.eig(P)" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": 9, 290 | "metadata": { 291 | "ExecuteTime": { 292 | "end_time": "2019-11-04T15:28:26.303171Z", 293 | "start_time": "2019-11-04T15:28:26.289061Z" 294 | } 295 | }, 296 | "outputs": [ 297 | { 298 | "data": { 299 | "text/plain": [ 300 | "array([-0.5+0.8660254j, -0.5-0.8660254j, 1. +0.j ])" 301 | ] 302 | }, 303 | "execution_count": 9, 304 | "metadata": {}, 305 | "output_type": "execute_result" 306 | } 307 | ], 308 | "source": [ 309 | "λ" 310 | ] 311 | }, 312 | { 313 | "cell_type": "code", 314 | "execution_count": 10, 315 | "metadata": { 316 | "ExecuteTime": { 317 | "end_time": "2019-11-04T15:28:26.320703Z", 318 | "start_time": "2019-11-04T15:28:26.307590Z" 319 | } 320 | }, 321 | "outputs": [ 322 | { 323 | "data": { 324 | "text/plain": [ 325 | "array([[ 0.57735027+0.j , 0.57735027-0.j , -0.57735027+0.j ],\n", 326 | " [-0.28867513-0.5j, -0.28867513+0.5j, -0.57735027+0.j ],\n", 327 | " [-0.28867513+0.5j, -0.28867513-0.5j, -0.57735027+0.j ]])" 328 | ] 329 | }, 330 | "execution_count": 10, 331 | "metadata": {}, 332 | "output_type": "execute_result" 333 | } 334 | ], 335 | "source": [ 336 | "v" 337 | ] 338 | }, 339 | { 340 | "cell_type": "markdown", 341 | "metadata": {}, 342 | "source": [ 343 | "We have one eigenvalue of 1, corresponding to an eigenvector with equal weighting on all three states. The remaining eigenvectors form a complex conjugate pair - we would expect a complex value to appear given the clear, periodic nature of the example.\n", 344 | "\n", 345 | "We can look at another transition matrix, this time with all real eigenvectors. For sake of concreteness, this matrix could correspond to a chemical reaction, where the probabilities are now concentrations of mass in different reactants:" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": 11, 351 | "metadata": { 352 | "ExecuteTime": { 353 | "end_time": "2019-11-04T15:28:26.332138Z", 354 | "start_time": "2019-11-04T15:28:26.323908Z" 355 | } 356 | }, 357 | "outputs": [], 358 | "source": [ 359 | "# Construct an example where all the non-unity eigenvalues are associated with decaying exponentials\n", 360 | "\n", 361 | "P = np.array([[1, 0.05, 0.0],\n", 362 | " [0, 0.95, 0.2],\n", 363 | " [0, 0.00, 0.8]])\n", 364 | "\n", 365 | "λ, v = np.linalg.eig(P)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 12, 371 | "metadata": { 372 | "ExecuteTime": { 373 | "end_time": "2019-11-04T15:28:26.347257Z", 374 | "start_time": "2019-11-04T15:28:26.337148Z" 375 | } 376 | }, 377 | "outputs": [ 378 | { 379 | "data": { 380 | "text/plain": [ 381 | "array([1. , 0.95, 0.8 ])" 382 | ] 383 | }, 384 | "execution_count": 12, 385 | "metadata": {}, 386 | "output_type": "execute_result" 387 | } 388 | ], 389 | "source": [ 390 | "λ" 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": 13, 396 | "metadata": { 397 | "ExecuteTime": { 398 | "end_time": "2019-11-04T15:28:26.362670Z", 399 | "start_time": "2019-11-04T15:28:26.351972Z" 400 | } 401 | }, 402 | "outputs": [ 403 | { 404 | "data": { 405 | "text/plain": [ 406 | "array([[ 1. , -0.70710678, 0.19611614],\n", 407 | " [ 0. , 0.70710678, -0.78446454],\n", 408 | " [ 0. , 0. , 0.58834841]])" 409 | ] 410 | }, 411 | "execution_count": 13, 412 | "metadata": {}, 413 | "output_type": "execute_result" 414 | } 415 | ], 416 | "source": [ 417 | "v" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "Before reading further, try to inspect the eigenvalues and eigenvectors and predict what this system will do over time. Does your prediction make sense if you try to imagine the process by looking at the transition matrix?\n", 425 | "\n", 426 | "Ok, let's run a simulation to see what this does over time:" 427 | ] 428 | }, 429 | { 430 | "cell_type": "code", 431 | "execution_count": 17, 432 | "metadata": { 433 | "ExecuteTime": { 434 | "end_time": "2019-11-04T15:29:21.234730Z", 435 | "start_time": "2019-11-04T15:29:20.863133Z" 436 | } 437 | }, 438 | "outputs": [ 439 | { 440 | "data": { 441 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAF+CAYAAACvRkIWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3xV9f3H8dc3e5OdAGGFsJEZGYobQbQqLsS9qBRHtcOq1Eql1lbraPVXB3ULYquiqDgRFEVRAdkz7BWyyd7f3x8nICPCBZKcm9z38/E4j3PvGfd+gpg333O+5/s11lpERETE+/m5XYCIiIh4RqEtIiLSTCi0RUREmgmFtoiISDOh0BYREWkmFNoiIiLNRIDbBRxJfHy87dixo9tliIiINIlFixblWGsT6tvn9aHdsWNHFi5c6HYZIiIiTcIYs+Xn9unyuIiISDOh0BYREWkmFNoiIiLNhEJbRESkmVBoi4iINBMKbRERkWbCo0e+jDEpwN1AOtAXCAU6WWs3e3BuCPAX4GogGlgC3G2tnXeMNR+isLCQrKwsqqqqGuojW5TAwEASExOJiopyuxQRETkOnj6nnQaMARYBXwEjjuI7XgDOA+4CNgK3Ap8YY4Zaa5ccxefUq7CwkN27d9O2bVtCQ0MxxhzvR7Yo1lrKysrYsWMHgIJbRKQZ8/Ty+DxrbZK19lzgTU8/3BjTF7gS+I219j/W2s9xwn8rMPmoq61HVlYWbdu2JSwsTIFdD2MMYWFhtG3blqysLLfLERGR4+BRaFtra4/x8y8AqoD/7vdZ1cAbwEhjTPAxfu4+VVVVhIaGHu/HtHihoaG6fSAi0sw1dke0XsAma23pQdtXAkE4l92Pm1rYR6Y/IxGR5q+xxx6PBfLr2Z633/5DGGNuBm4GaN++feNUJiIiUqe21lJVW0tVjaW6xllX1dRSXeNsr977vtZSs+84S3VtLZ0TImgXG9YkdTZ2aBvA/sz2n2WtnQJMAUhPT6/vfBERaaastVTVWMqra6ioqqWiuobyunVFde2+bRXVtXXva6iscbZX1tRSWV231L2u2O991d51jbO9qu51VbUTunv3VdXYfcc6QXzsUfOnX/TkpmGdGvBP6Oc1dmjnAfU1lWP22y8iIl7CWktFdS2llTWUVFRTVlVDaWUNpRXVlFbWUFZVQ1llDaWV1ZRV1da9r65b11JeVUN5Vd1xVU4Y791WXvVTOB9HRgLgZyAowI8gfz+CAvwJDvAjKMCPQH+zb3ugvx8RwQH7XgcG+BHoZ+qO27sYAv39CNi79jMHbXdeB/g5xwT4GWdb3TrA35AS03T9qho7tFcCFxljwg66r90TqAQyGvn7RURaPGst5VW1FFVUUVReTVF5NcXl1RTv/76impIKZ733dUlFDSWVzvvSutellTVH1eo0BkID/QkN9Cck0J/QoL2vncCMj3C2hwT4OetAP4IDnHVIoD/BgU7ghtStncWf4EAnePcevzeI924P8PfNscEaO7TfAx4ALgNeATDGBACXA59aaysa+fubpYyMDHr06MHEiRN54IEH9m2fMGECU6dOZe7cuaSnp7tYoYg0tNpaS1F5NQVlleSXVlFQWsmesioKy6qcdXk1e0qrKCyvW8qqKSx3QrmwrIpqD4I2KMCPyOAAwuuWiGB/YsODaBcTRliQf912f8KCApz3QQGEBfsTFuRPaKCzLSzICeawoIB94ayOrk3H49A2xlxa93Jg3XqUMSYbyLbWfmmM6QBsACZbaycDWGuXGGP+C/zTGBMIbAImAJ2Aqxrqh2hp0tLSGDduHE888QS333478fHxTJ48mRdffJFZs2YpsEW8XG2tpbC8itySSvJKKsktriS/1HmdX1JJXmklBaVV5O+33lNWhT1M7oYE+tEqNJCokECiQgOJjwgiNSGcyJAAokICiQwJJDIkYN8SERxYt3behwcHEOijrdOW5Gha2gcPqvJ03fpL4HSczmX+HPoY2Q3AX4EHcYYxXQqcY61dfLTFeuqB91eyamdhY328R3q2iWLS+b2O+fxJkybx6quv8vDDD9O9e3ceeOABpk+fzvDhwxuwShHxVG2tJa+0kqzCCrKLK8guqiCnuIKcIud9bnEluSWV5BZXkFdS+bMt37Agf2LCgogJDyQmLIiUmDBiwgKJDg0kOiyI6LBAosMCaRXqLFF16+AA/yb+icUbeRza1toj9fjeTD29wq21ZcBv6xbxUHJyMnfeeSePPfYY1dXVPPnkk4wZM2bf/oceeohXXnmF9evXM2PGDEaPHu1itSLNl7WWPWVVZBaWk7nHWXYXVrC7qJyswrrXheXkllTWe683NNCf+Mgg4iOCaRsdSt+UVsSGBxEXEUxceBCxdUtcRBAxYUGEBCp85dg19j1tVxxPC9ebdOnShYqKCoYNG8att956wL6zzjqLyy+/nJtuusml6kSah9LKanYWlLGjoJydBWV1Szm79pSRuaecXXvKKauqOeS8uPAgEiKDSYoKoXtyJIlRwSRGhpAYGUxCZDDxEc46PLhF/hoVL6W/bV5qzpw5jB8/nqFDhzJ//nyWLl1K37599+0fPHiwi9WJeI+K6hq255exLa/UWfLL2J5fyvb8Mrbnl5FXUnnA8X4GkqJCaN0qhB5tojirR2Ld+1CSWzkhnRgZQlCA7v+K91Foe6HFixczevTofZ3RunbtysSJE5k1a5bbpYm4orSyms05pWzOLWFzbglbckrZklfC1txSdhWWH9CBKyjAj5ToUNrGhNKrTStSYkJpW/e+TXQoSZHBPvu4kDR/Cm0vk5GRwahRoxgxYgRPPfUUfn5+TJo0iRtvvJF58+Zx6qmnul2iSKOorbXsKCgjI7uYjdklbKxbb8opIbOw/IBj4yOCaB8bxuDUONrHhtEhLox2sWG0jw0jISIYPz89giQtk0Lbi2RmZjJixAh69OjBtGnT8PNzWgPXXnstjzzyCPfccw/ffPONy1WKHJ+aWsvWvFLWZhaxfncR67OKycgqZmNOMeVVP00oGBUSQGpCBCelxZEaH06HuHA6xYfTIS6MyJBAF38CEfcotL1IcnIyGzduPGS7v78/q1evdqEikeOTU1zB6l2FrM0sYvWuItbuLmT97mIqqn8K57bRoaQlRjC0cxxpiRF0Toigc0I4seFBGrRD5CAK7WbqwQcf5NlnnyU7O5sVK1Zw2223sXDhQpKTk90uTXyQtU7recWOQlbu3MOqXYWs2llIVtFPgx4mRAbTPTmSa4Z0oGtyJF2TIklLjCBCva9FPKb/W5qp++67j/vuu8/tMsQH7Q3oZdv3sGx7Act37GHlzkKKyqsBCPAzdEmK5JQuCfRoHUnP1lF0bx1FbHiQy5WLNH8KbRE5rILSSpZsK+DHrQX8uK2AZdsLKCitApye2j1aR3Fhvzb0btOK3m1b0SUpQqN3iTQShbaI7FNba9mYU8zCzfks2pLPoq35bMwuAZznm7smRTKyZzJ92rWib0o03ZIjNZ61SBNSaIv4sOqaWlbuLOT7TXl8vzmPHzbn7WtFx4QFMqB9DJcMSKF/+2j6pETr/rOIy/R/oIgPqam1rNixh2835vLthlwWbs6jpNIZwrNjXBhn90jixI6xDOwYQ2p8uHpvi3gZhbZIC2atZWNOCfMzcvh6fQ7fbszd12EsLTGCiwa0ZXCnOAZ1iiUpKsTlakXkSBTaIi3MnrIq5mfk8OXabOatz2bXHmc0sXaxofyiT2uGdo5nSGosiZEKaZHmRqEt0sxZa1mTWcScNVnMXZPFj9sKqKm1RIYEMCwtntvOjOeUtATax4W5XaqIHCeFtkgzVFFdwzcbcpm9ajdz12Sxs641fULbVtxyemdO65pAv3bRmhhDpIVRaIs0E4XlVcxdk8Wnq3bzxZosSiprCAvyZ1haPHcM78IZ3RJJ1H1pkRZNoS3ixQpKK/l01W4+XpHJV+uzqaqxxEcEc0G/NozomczQznGEBGogExFfodAW8TKF5VV8unI37y3dyTcZOVTXWlJiQrn+pI6c0zuZ/u1iNPWkiI9SaIt4gfKqGj5fncXMJTv4Ym02lTW1tIsNZdwpqZx3Qmt6t43SM9MiotAWcUttreX7zXm8s3gHHy7fRVFFNYmRwVw9pAPn921Nv3bRCmoROYBC2wtlZGTQo0cPJk6cyAMPPLBv+4QJE5g6dSpz584lPT3dxQrleGzLK+WtRdt5a9F2dhSUER7kzzm9W3PxgLYMSY3DX5e+ReRnKLS9UFpaGuPGjeOJJ57g9ttvJz4+nsmTJ/Piiy8ya9YsBXYzVFFdw8crMvnfwm3Mz8jFGBiWFs8fzunG2T2TCAvS/4oicmQt8zfFR/dA5nJ3a0g+AUb9/ZhPnzRpEq+++ioPP/ww3bt354EHHmD69OkMHz68AYuUxrYhu5g3vt/KW4u2k19aRbvYUH57dlcuGZhC2+hQt8sTkWamZYZ2C5CcnMydd97JY489RnV1NU8++SRjxowBoLy8nLFjx7J27VqCg4NJSkrimWeeITU11eWqBZyZsz5btZtXv93CtxtzCfAzjOiVxJWDOnBS5zj1/BaRY9YyQ/s4WrjepEuXLlRUVDBs2DBuvfXWA/ZNmDCBkSNHAvB///d/jBs3jjlz5rhRptTJKa7gje+3Mu27rezaU07b6FDuGtmNy9JTNM63iDSIlhnaLcCcOXMYP348Q4cOZf78+SxdupS+ffsCEBISsi+wAYYMGcKjjz7qVqk+b01mIS98tYmZS3ZSWVPLsLR4HrigF2f1SFKnMhFpUAptL7R48WJGjx69rzNa165dmThxIrNmzar3+KeeeooLL7ywiav0bdZa5q3P4fmvNvLV+hxCAv24LD2FG07uSFpipNvliUgLpdD2MhkZGYwaNYoRI0bw1FNP4efnx6RJk7jxxhuZN28ep5566gHH/+1vf2PdunV8/vnnLlXsW6prapm1fBfPfLGBNZlFJEYGc9fIblw5qD0x4UFulyciLZyx1rpdw2Glp6fbhQsX/uz+1atX06NHjyasqPFkZmZy0kkn0b59ez755BOCg4MBqKmpoXfv3sTExPDNN9/sO/7RRx/ljTfeYPbs2URHRx/x81vSn1VTK6+q4c1F25kybwPb8spIS4xg/KmpXNivLUEBmklLRBqOMWaRtbbeZ3vV0vYiycnJbNy48ZDt/v7+rF69+oBtjz/+ONOnT/c4sOXYlFXW8Pr3W3nuyw1kFVXQr100fzqvJ8N7JKkXuIg0OYV2M7R9+3Z+97vfkZqayhlnnAFAQEAAh7siIUentLKaqQu2MGXeRnKKKxmSGss/x/ZjaGqchhYVEdcotJuhlJQUvP22RnNVUV3D699t5d9zN5BTXMGwtHhuPzONwalxbpcmIqLQFgGoqqnl7UXbefLz9ezcU86Q1FievXoA6R1j3S5NRGQfhbb4NGstn6zczSMfr2FjTgl920XzyKV9OTlNl8FFxPsotMVnLdqSx0MfrmHRlnzSEiOYcs1Azu6ZpLAWEa+l0BafszW3lL99tJqPVmSSGBnM3y4+gcsGphDgr0e3RMS7KbTFZxRXVPP03Aye/2oT/n6G3wzvyi9P7aRpMUWk2dBvK2nxrLXMWLyDv3+8huyiCi7u35Y/nNOd5FaaxENEmheFtrRoq3cVcv/MFfywOZ9+7aKZcs1A+rePcbssEZFjotCWFqmovIonPlvPK99uplVoII9c0odLB6ZoFDMRadYU2tLifLxiF/fPXEl2cQVXDGrPH0Z2IzpMk3mISPOn0JYWI3NPOffPXMGnq3bTo3UUU65Np187jcsuIi2HQluavdpay7Tvt/LIR2uorKnl7nO6M+6UTgTqES4RaWEU2tKsbcsr5a63lrJgYx4np8Xx0EUn0CEu3O2yREQahUJbmqW9reu/fbgaP2P4+8UncPmJ7TSamYi0aB5dPzTGtDPGvGWM2WOMKTTGzDDGtPfw3PbGmFeMMVuNMaXGmHXGmAeNMWoO/YyMjAwCAwOZNGnSAdsnTJhAZGSkz0/BubOgjGte/I4/vbuCAe1j+OQ3pzJ2UHsFtoi0eEcMbWNMGDAH6A5cB1wDdAHmHil46/bPBk4F/gScBzwP/A548bgqb8HS0tIYN24cTzzxBDk5OQBMnjyZF198kXfeeYf09HSXK3TP+0t3cs4/5/Hj1gL+elFvXrtpEG2jQ90uS0SkSXhyefyXQCrQzVqbAWCMWQasB8YDjx/m3JNxAn6ktfbTum1zjTGxwO+NMWHW2tJjrv5nPPz9w6zJW9PQH3tUusd25+5Bdx/z+ZMmTeLVV1/l4Ycfpnv37jzwwANMnz6d4cOHN2CVzUdheRWTZq7knR930L99NP+8vJ/uXYuIz/EktC8AFuwNbABr7SZjzHzgQg4f2nsfji08aHsBTitf1zN/RnJyMnfeeSePPfYY1dXVPPnkk4wZM2bf/rPOOoucnByMMURGRvLUU0/Rr18/FytuPIu25PHr6UvILCznzuFduO2MNE3uISI+yZPQ7gXMrGf7SuCyI5w7G6dF/rAxZgKwFRgE3AE8a60tOYpaPXY8LVxv0qVLFyoqKhg2bBi33nrrAftmzJhBq1atAHjnnXe4/vrrWbJkiRtlNpraWsuz8zbw2KfraBMdwpu/GsoADUEqIj7Mk9COBfLr2Z4HHPY3qLW23BgzDHgbJ+T3eh64zdMifdGcOXMYP348Q4cOZf78+SxdupS+ffvu2783sAEKCw++kNH8ZRdV8Nv/LeGr9Tmc16c1f7v4BKJCAt0uS0TEVZ4+8mXr2XbES9vGmBDgv0AiTge2vS3t+4FqYMLPnHczcDNA+/YedVJvURYvXszo0aP3dUbr2rUrEydOZNasWQccd9VVV/Hll1/i5+fHhx9+6FK1DW/Bxlxun/4jhWVVPHTRCVwxSI9yiYiAZ4985eO0tg8WQ/0t8P3dBJwOnGutnWqtnWetfRSn9/ivjDF96zvJWjvFWpturU1PSEjwoMSWIyMjg1GjRjFixAieeuopgoKCmDRpEh9++CHz5s074Nhp06axfft27r//fu6+u/nfErDWMmXeBq56/jsiQwKYedvJXDlYj3KJiOzlSWivxLmvfbCewKojnHsCkG+t3XDQ9u/r1j08+H6fkZmZyYgRI+jRowfTpk3Dz8/5z3PttdfSvXt37rnnnnrPu+mmm/jss8/Izc1tynIbVHFFNbdMW8xDH65hRM8kZt56Mt2To9wuS0TEq3hyefw94FFjTKq1diOAMaYjzuNc9afITzKBGGNM2v69z4HBdesdR1duy5acnMzGjRsP2e7v78/q1av3vc/Pz6e8vJzWrVsD8Pbbb5OYmEhsbH0XRLxfRlYRN7+2iC25pfzx3B6MO6WTWtciIvXwJLT/g9NpbKYx5j6c+9t/AbYBz+09yBjTAdgATLbWTq7b/DLwW+BDY8xfce5pp+MMtLIImN8wP4Zvyc/P5/LLL6e8vBw/Pz8SExP54IMPmmXQzVmzm19PX0JIoB/Txg1mSGqc2yWJiHitI4a2tbbEGHMm8ATwGk4HtM+BO621xfsdagB/9rvkbq3dbIwZAvwZeBCIxwn7KcBfrbW1DfRz+JTU1FR++OEHt8s4LtZanv1yI498soZebaKYck06bTSymYjIYXnUe9xauxW45AjHbKaeHuXW2lXAmENOEJ9VXlXD3W8vY+aSnZzftw2PXNKH0CB/t8sSEfF6muVLmlR2UQW/fHUhS7cXcNfIbtxyeudmeVlfRMQNCm1pMut3F3HDyz+QW1zJs1cPZGSvZLdLEhFpVhTa0iTmZ+Twq6mLCAn057/jh9AnJdrtkkREmp0WEdrWWl1iPQJr6xvUrmm8uXAb985YTueECF64Pp2UmDDXahERac6afWgHBARQXV1NYKDGpT6c6upqAgKa9j+3tZanv9jAPz5Zy7C0eJ6+eoDGDxcROQ7NPrRDQkIoLi4mJkazPx1OUVERISEhTfZ9tbWWyR+s4uVvNnNhvzb849K+BAVoOk0RkePR7H+LJiQkkJ2dTWlpqauXgL2VtZbS0lJycnJoqnHcK6pruP2NH3n5m82MG9aJJ8b0U2CLiDSAFtHSTkpKIjMzk4qKCrfL8UrBwcEkJSU1SUu7pKKam19byPyMXCae252bT+3c6N8pIuIrmn1ogzO39P7zS4s79pRVccNL37NkWwGPXtaXSwemuF2SiEiL0iJCW9yXW1zBNS98z/qsIv595QBGndDa7ZJERFochbYct8w95Vz1/AJ2FJTx/HUnclpX35oDXUSkqSi05bjsKCjjiikLyCup5JUbBjFYs3SJiDQahbYcs+35pVzxnwUUlFYxddxg+rXTKGciIo1JoS3HZFueE9h7yqqYetNg+iqwRUQanUJbjtq2vFLGTllAUXkV08YN1jjiIiJNRKEtR2VnQRljpyyguKKaaeOGcEKKHrUTEWkqGqZKPJZVVM5Vz39HYd0lcQW2iEjTUktbPJJXUsnVz3/H7sJyXrtpkAJbRMQFamnLEe0pq+LaF79jS24pz1+XzsAOsW6XJCLikxTaclilldXc8NL3rM0s4tlrBnJS53i3SxIR8VkKbflZVTW1TJi6mCXbCnjqiv6c0S3R7ZJERHya7mlLvWprLb9/cylfrsvm7xefwDm9NZa4iIjb1NKWQ1hrmfzBKmYu2cldI7sxdlB7t0sSEREU2lKPp7/YwMvfbObGkztxy+maD1tExFsotOUAby7cxj8+Wcvofm2477weGGPcLklEROootGWfr9fncO+M5ZycFscjl/bFz0+BLSLiTRTaAsCazEImTF1E54QInrl6IEEB+qshIuJt9JtZyNxTzg0v/UBYsD8v3XAiUSGBbpckIiL1UGj7uOKKam54+QcKy6p48foTaRMd6nZJIiLyM/Sctg+rqbXcMf1H1u0u4oXr0unVRuOJi4h4M7W0fdgjH6/h8zVZTDq/J6drtDMREa+n0PZRby7cxnPzNnL1kPZcO7Sj2+WIiIgHFNo+6IfNeUx8x3m0a9L5vdwuR0REPKTQ9jHb8koZ/9oiUmLCePrKgQT666+AiEhzod/YPqSssobxry2iqqaW569Lp1WYHu0SEWlO1HvcR1hruXfGMlZnFvLCdel0TohwuyQRETlKamn7iBfnb+bdJTv57fCunNk9ye1yRETkGCi0fcA3G3J46MPVjOiZxK1npLldjoiIHCOFdgu3o6CM217/kU7x4Tx+eT9NAiIi0owptFuwiuoabpm2mMrqWp67ZiARwerCICLSnOm3eAv20KzVLN1WwDNXDVDHMxGRFkAt7RbqvaU7eeXbLdw0rBOjTmjtdjkiItIAFNotUEZWMfe8vYyBHWK4Z1R3t8sREZEGotBuYUorq7ll2iJCAv35vyv7a8QzEZEWRPe0W5j7Z65kfVYxr944iNatNDe2iEhLomZYC/LOj9t5a9F2bjsjjVO6JLhdjoiINDCFdguxKaeE+95ZwYkdY7jjrC5ulyMiIo3Ao9A2xrQzxrxljNljjCk0xswwxrT39EuMMT2MMW8aY3KMMWXGmLXGmDuOvWzZX0V1DbdPX0yAvx//GtufAN3HFhFpkY54T9sYEwbMASqA6wALPAjMNcb0sdaWHOH89LrzvwDGAXuALoAeHG4gj3y8lhU7CplyzUDaROs+tohIS+VJR7RfAqlAN2ttBoAxZhmwHhgPPP5zJxpj/IBXgM+ttRftt2vuMVcsB5izZjcvfL2J64Z2YESvZLfLERGRRuTJddQLgAV7AxvAWrsJmA9ceIRzTwd6cphgl2OXXVTBXW8uo0frKO49t4fb5YiISCPzJLR7ASvq2b4SJ5APZ1jdOsQYs8AYU2WMyTLGPGmM0XXc42Ct5Q9vLaW4oponx/YjJNDf7ZJERKSReRLasUB+PdvzgJgjnNumbv1f4FPgbOARnHvbr//cScaYm40xC40xC7Ozsz0o0fdMXbCFuWuzuXdUd7okRbpdjoiINAFPB1ex9WzzZI7Hvf8omGqtvb/u9RfGGH/g78aYntbaVYd8mbVTgCkA6enp9X23T8vIKuLBWas5rWsC153U0e1yRESkiXjS0s7HaW0fLIb6W+D7y61bf3bQ9k/r1v08+H7ZT2V1LXe8sYTw4AD+cVkfjNH82CIivsKTlvZKnPvaB+sJHNJKrudcOLSlvjdpaj34ftnPE7PXsXKn83hXYmSI2+WIiEgT8qSl/R4wxBiTuneDMaYjcHLdvsP5COf57nMO2j6ybr3QoyoFgEVb8njuyw2MPbGdHu8SEfFBnoT2f4DNwExjzIXGmAuAmcA24Lm9BxljOhhjqo0xe+9dY63NBf4G/MoY85AxZrgx5h7gfuCV/R8jk8Mrrazmd/9bSpvoUO77xZE67YuISEt0xMvj1toSY8yZwBPAaziXtj8H7rTWFu93qAH8OfQfApOBIuAW4PfALuAfwF+Ou3of8vBHa9icW8obNw8hIliTs4mI+CKPfvtba7cClxzhmM3U06PcWmtxBlfRACvHaH5GDq98u4UbT+7EkNQ4t8sRERGXaGYJL1dYXsVdby4lNSGcP5zTze1yRETERbrO6uX+8v4qMgvLmXHLyRr1TETEx6ml7cXmrs3izUXb+dVpnenXLtrtckRExGUKbS9VVF7FxBnLSUuM4I7hXdwuR0REvIAuj3uphz5cw+7Cct6ecBLBAbosLiIiaml7pfkZOUz/fivjTkmlf/sjzckiIiK+QqHtZUoqqrn77WV0ig/nt2d3dbscERHxIro87mUe+XgNOwrK+N/4oeotLiIiB1BL24ss2pLHqwu2cO2QDpzYsb6J1URExJcptL1ERXUNd7+9nNZRIdx1Tne3yxERES+ky+Ne4pkvNpCRVcyL16drbHEREamXWtpeYP3uIv49N4ML+rbhzO5JbpcjIiJeSqHtstpayz0zlhMeHMD952vKTRER+XkKbZdN+34ri7bkc995PYmPCHa7HBER8WIKbRftLizn4Y/WMCwtnksGtHW7HBER8XIKbRc98P5Kqmpq+etFvTHmkKnIRUREDqDQdsmcNbv5cHkmvz6rCx3iwt0uR0REmgGFtgtKK6v507sr6ZIYwS9PSXW7HBERaSb0QBOKqfoAACAASURBVLAL/jV7/b6hSoMC9O8mERHxjBKjia3aWcjzX29i7IntGNRJQ5WKiIjnFNpNqLbW8sd3lxMdGsg9ozRUqYiIHB2FdhOa/sNWftxawB/P60F0WJDb5YiISDOj0G4iOcUVPPLxWoakxnJRfz2TLSIiR0+h3UT+9uEaSiureXC0nskWEZFjo9BuAt9tzOXtxdv55SmppCVGul2OiIg0UwrtRlZVU8ufZq6gbXQot5/Zxe1yRESkGdNz2o3sha83sW53Mc9fm05okL/b5YiISDOmlnYj2lFQxr9mr+fsnkkM76l5skVE5PgotBvRX95fhcUySfNki4hIA1BoN5Iv1mbx8cpMbj+zCykxYW6XIyIiLYBCuxGUV9Uw6b2VpMaHM+6UTm6XIyIiLYQ6ojWCKfM2siW3lKk3DSY4QJ3PRESkYail3cC25ZXy77kZnNenNcO6xLtdjoiItCAK7Qb2wPsr8fcz3HdeD7dLERGRFkaXx4/Vnu2w8h3Y9h0ERUJoDBuKA4lbV8K9Z4yhdatQtysUEZEWRqF9NCqKYel0WPE2bP3W2RabCjVV2LJ8OlcW83Ag2G9eguyRMOAaSDsb/PXHLCIix09p4qnCnTDtMti9AhJ6wJn3Qa+LIa4zAE98upZn5qzhzUsT6Jf3ESx9A9bOgohkOGMi9L8G/HQ3QkREjp1C2xNZq2HqpVBeAFe9BV3OPmD35pwSnv1yI+f1a0+/9P7ASXDW/bD+M/jmSXj/17DkdfjFE5CkgVZEROTYqOl3JJu/hhdGQm0V3PDRIYFtrWXSeysJCvBj4rn7dT7zD4Tu5zrnXPhvyFkHz50Cn90PVWVN/EOIiEhLoNA+nLUfw2sXQWQSjJsNrfsccsgnK3fz5bpsfnN2VxKjQg79DGOg/9Vw20LoMxbm/wteOBvyNzd+/SIi0qIotH9OaR7MvAUSusONn0B0+0MPqaxm8vsr6Z4cyXVDOxz+88LjYPS/4cr/Qf5WmHI6ZHzeOLWLiEiLpND+OZ/eB+V7YPQzEBZb7yFPzclg555yJl/YmwB/D/8ou46Em+dCZGuYdil89ThY24CFi4hIS6XQrs/GL2HJNDjpdkjuXe8h63cX8Z95G7lkQAqDOtUf6j8rrrNzub3naPj8AZhxM9RUNUDhIiLSkqn3+MGqyuGD30BMJzjt7noPsdbyp5krCAvy595zux/b9wSFw6UvOr3J5zwIZfkw5hVnu4iISD3U0j7YV49C3gbn8azA+kc1m7lkJws25vGHc7oTHxF87N9lDJx6F5z/JGz4HF690LmXLiIiUg+F9v6yVsPXTzi9vDufUe8he8qqeHDWavqmtOKKQYd2TjsmA6+DMa/CrmXw0ijYs6NhPldERFoUj0LbGNPOGPOWMWaPMabQGDPDGHPUiWWMudcYY40xXx99qU3g0z9BcBSM/OvPHvL4p2vJLangwdEn4O9nGu67e5wPV7/tBPbL50LBtob7bBERaRGOGNrGmDBgDtAduA64BugCzDXGeHwD1hiTCvwRyDq2UhtZ/mbImA2Dbobw+qfUXLFjD68t2MI1QzpwQkqrhq+h0ylw7UwozYdXfqHgFhGRA3jS0v4lkAqMtta+a62dCVwAdADGH8V3PQNMA1YfdZVNYdErzj3mAdfWu7um1jLxneXEhgfzuxHdGq+OlIFwzTsKbhEROYQnoX0BsMBam7F3g7V2EzAfuNCTLzHGXAkMAO49liIbXU0V/DgVuoyEVm3rPeTVbzezbPse7j+/J61CAxu3HgW3iIjUw5PQ7gWsqGf7SuCIs18YY2KAJ4A/WGu9s2v0mllQkgXpN9S7e9eeMh79ZC2ndU3g/D6tm6amA4L7fCjKbJrvFRERr+VJaMcC+fVszwNiPDj/H8A64GXPy2pii16CVu0gbXi9uyfNXEmNtTw4ujfGNGDnsyNJGQjXzIDiLOdxsJLcpvtuERHxOp4+8lXfOJtHTC9jzCnAtcAEaz0fq9MYc7MxZqExZmF2dranpx2b3A2w8QvnXraf/yG7P1mZyaerdnPn8K60iw1r3Frqk5IOV/7X6Sg39SJnaFUREfFJnoR2Pk5r+2Ax1N8C399zwAvAdmNMtDEmGmcUNv+69/WOTGKtnWKtTbfWpickJHhQ4nFY/AoYf+h/zSG7iiuq+fN7zoQgNw3r1Lh1HE6nU2DMa7B7FUwbA5Ul7tUiIiKu8SS0V+Lc1z5YT2DVEc7tAfwKJ9z3LicDQ+peT/C40sZQXQk/ToNuoyDq0HvVj36ylszCch66+AQCPZ0QpLF0HQGXPA/bv4c3roLqCnfrERGRJufJ2OPvAY8aY1KttRsBjDEdccL3niOcW9+wYv8E/IHbgYx69jedNe9DaQ4MPLQD2g+b83jl281cO6QDA9p7cuu+CfQa7bSyZ97iTDJy6Yv1XtIXEZGWyZPQ/g9wGzDTGHMfzv3tvwDbcC5/A2CM6QBsACZbaycDWGu/OPjDjDEFQEB9+5rc4tecebI7n3nA5rLKGv7w1jJSYkL5wznHOCFIY+l/lTO5yKd/hFkxzhjpTdk5TkREXHPEa77W2hLgTJwe4K/hDJCyCTjTWlu836EGpwXdPMYzryiGzV8702P6HVjyE7PXsSmnhL9f3IfwYC+cCO2k22DYb51e73P+4nY1IiLSRDxKJGvtVuCSIxyzGQ96lFtrT/fkOxvdlvlQWwVpZx2w+cet+Tz/1UauGNSek9PqH87UK5x1P5TmwlePQWisE+QiItKieWEzsolkfA4BodBuyL5N5VU13PXWMpKjQph4rPNkNxVjnEvj5QXOpfKwOOh3hdtViYhII/Ld0N4wBzoOg8CQfZv+9fl6MrKKefmGE4kMaeShShuCnz9c/B/nHvfMWyEsFrqOdLsqERFpJM3j/nNDK9gKuesP6IC2YGMuz365gTHpKZzeLdHF4o5SQDCMfR2ST4D/XQdbv3O7IhERaSS+Gdob5jjruvvZBaWV/Oa/S+gYF86k8+t7JN3LBUfCVW9BVBt4/TJnEBYREWlxfDO0Mz6HqLYQ3xVrLffOWE52UQX/GtvPO3uLeyIiwRmnPCAUpl7sXE0QEZEWxfdCu6YaNn0Jnc8AY/jfwm18tCKT34/sRp+UaLerOz4xHZ3griqF1y6Ckhy3KxIRkQbke6G980dn0o3OZ7Ehu5g/v7eKkzrHcfMpqW5X1jCSesEV/4U922HapVBR5HZFIiLSQHwvtDd8DhhKU4Zx2+s/EhLox+Nj+uHn14JGFeswFC57GXYtg/9erXHKRURaCB8M7TnYNgP43QfbWJtZyBOX9yO5VciRz2tuuo2CC//PmXb0nfFQW+N2RSIicpx8K7TLCmD7Qr7z68tHKzKZeG6P5vV419HqdyWc/RdY+Q58+HvwfEpzERHxQs20q/Qx2jQPbA2PbkjhsoEp7s6R3VRO/rUzk9n8fzmjpp15n9sViYjIMfKp0N6+dBa1fmH4tTuRBy/qjfGV2bGGPwCleTDvH8445UNvcbsiERE5Bj4T2qXlVVxeNp8TYtvx72sGExzgQ/NQGwO/+KczTvkn9zrDnfYd63ZVIiJylHzmnnZYkD/tQjqwKyGWhMhgt8tpev4BcPHz0OlUePcWWPOh2xWJiMhR8pnQxs+Pod1HsLViNxU1PvoIVGCIM055m37w5vVOz3IREWk2fCe0gd5xvam21azJW+N2Ke7ZO055XGeYfiVs+97tikRExEM+Fdq94p3JQFbkrHC5EpeFxcI170BEojNqWuZytysSEREP+FRoJ4UlER8az8qclW6X4r7IZLh2JgRFOOOU52S4XZGIiByBT4W2MYZecb1YmavQBiCmA1zzrjPoyqsXQP5mtysSEZHD8KnQBucS+aY9myipKnG7FO+Q0NVpcVeVwivnOxONiIiIV/K50O4d1xuLZVXuKrdL8R7JvZ173GUF8MoFUJTpdkUiIlIP3wvt+N6AOqMdok1/p1d5USa8eqHm4hYR8UI+F9oxITG0jWir0K5P+8Fw5X8hf4vT4i7JdbsiERHZj8+FNqDOaIfT6RS48g3I2+B0TlNwi4h4Dd8M7fhe7CjeQX55vtuleKfU0+GKNyA3w7lUXprndkUiIoKPhnbvOOe+tlrbh9H5DGfI05x1TotbwS0i4jqfDO2ecT0xGN3XPpK0s+CK1yF7Xd09bnVOExFxk0+GdkRQBB1bddTIaJ5IGw5XTIfc9fDyL6Bot9sViYj4LJ8MbXAukevyuIfSzoKr3oSCLfDyeVC40+2KRER8ks+Gdq/4XmSXZbO7RC1Hj3Q6Fa6e4TzH/dK5ULDN7YpERHyO74Z2XN2MX7m6r+2xDkPh2nedTmkvjdIkIyIiTcxnQ7t7bHcCTIDuax+tlHS4/n2oKoOXztG0niIiTchnQzskIIQuMV1Ylr3M7VKan9Z94caPwT/Iuce97Xu3KxIR8Qk+G9oAA5MGsiR7CZU1lW6X0vzEd3GCOyzeGYBlwxy3KxIRafF8OrQHtx5MRU0FS7OXul1K8xTd3gnu2FSYNgaWv+V2RSIiLZpPh/bApIH4GT++z9Tl3WMWkQjXz4J2g+Dtm2DBM25XJCLSYvl0aEcGRdIrrhff7frO7VKat9Bo53GwHufDx/fA7D+DtW5XJSLS4vh0aAMMSh7E8uzllFaVul1K8xYYApe9Auk3wtdPwLsToFp9BUREGpJCu/Ugqm01i7MWu11K8+fnD+c9Dmf8EZZOh2mXQFmB21WJiLQYPh/a/RP7E+AXwPe7dF+7QRgDp/0BRj8LW76FF0dCwVa3qxIRaRF8PrRDA0Lpm9CX7zJ1X7tB9bsCrn4bCnfBf86CHbqSISJyvHw+tAEGJw9mde5q9lTscbuUliX1NLjpUwgIccYrX/mu2xWJiDRrCm2c+9oWy8LdC90upeVJ7A6//ByST4A3r4MvHlbPchGRY6TQBvrE9yHEP0T3tRtLRCJc/wH0vQK+eAjeugEq1VtfRORoBbhdgDcI9A9kQNIADbLSmAKCYfQzkNDdeY47bxOMnQatUtyuTESk2VBLu86g5EFkFGSQU5bjdiktlzEw7E64YjrkboDnToNNX7ldlYhIs6HQrjO49WAAfsj8weVKfEC3UXDzXAiLdSYb+fZp3ecWEfGAR6FtjGlnjHnLGLPHGFNojJlhjGnvwXnpxpgpxpg1xphSY8xWY8w0Y0yn4y+9YXWP7U5kYKSGNG0q8V1g3OdOgH9yL8y4GSpL3K5KRMSrHTG0jTFhwBygO3AdcA3QBZhrjAk/wuljgV7Ak8Ao4B5gALDQGNPuOOpucAF+AaQnp/PNzm+wavU1jZAoGPManPknWP4m/OdMyF7rdlUiIl7Lk5b2L4FUYLS19l1r7UzgAqADMP4I5z5srT3ZWvu0tfZLa+3rwDlATN3nepUz25/JrpJdrMpd5XYpvsPPD079PVzzDpTkwJQzYNmbblclIuKVPAntC4AF1tqMvRustZuA+cCFhzvRWptdz7YtQDbQ9uhKbXynp5yOv/Fn9tbZbpfiezqfAb/6Clr3gRnj4P07oarM7apERLyKJ6HdC1hRz/aVQM+j/UJjTA8gEVh9tOc2tuiQaE5MPpHZW2brErkbotrAdR/AyXfAopecy+VZXvfXRETENZ6EdiyQX8/2PJzL3B4zxgQAz+K0tF84zHE3G2MWGmMWZmcf0lhvVMPbD2dz4WY2FGxo0u+VOv4BcPZkuOptKMmGKafDDy+od7mICJ4/8lXfb0xzDN/3f8BJwNXW2vr+IeB8mbVTrLXp1tr0hISEY/iaY3dm+zMxGD7b+lmTfq8cpMtwmPANdDgZZv0W/ns1lOa5XZWIiKs8Ce18nNb2wWKovwVeL2PM34CbgRuttZ96el5TSwhLoF9iPz7f8rnbpUhEIlz1Fox4ENZ9Ak8PhfXqbyAivsuT0F6Jc1/7YD0Bj7pZG2P+iPO41x3W2tc8L88dw9sPZ23+WrYVbnO7FPHzg5Nuh1/OgdAYmHYJfPBbPdMtIj7Jk9B+DxhijEndu8EY0xE4uW7fYRljfg08CPzRWvvUsZXZtIZ3GA6gXuTepHUfuPkLGHobLHwRnh0GWzUQjoj4Fk9C+z/AZmCmMeZCY8wFwExgG/Dc3oOMMR2MMdXGmPv32zYW+CfwMTDHGDNkv+Woe543lTYRbegZ15PZWxTaXiUwBEb+1ZkxrKYaXhwJH9+rGcNExGccMbSttSXAmcA64DVgGrAJONNaW7zfoQbwP+gzz6nbfg7w7UHL0w1Qf6M5u8PZLMtZRmZJptulyME6DoNbvoETb4IFT8MzJ8Hmr92uSkSk0XnUe9xau9Vae4m1NspaG2mtHW2t3XzQMZuttcZa++f9tl1ft62+5fQG/Uka2FntzwLg863qkOaVgiPhvMec57qx8PJ58MFvoKzA7cpERBqNZvn6GZ1adSItOk2XyL1dp1OcR8OG3AqLXoZ/D4IVM/Rct4i0SArtwxjRcQSLdi9ie9F2t0uRwwkKh3MecnqYRybDWzfA62Mgf4vblYmINCiF9mFclHYRxhjeXv+226WIJ9r0h3FzYORDsHm+0+r+8hGoKne7MhGRBqHQPozk8GROTTmVGetnUFVT5XY54gn/ABh6K9z2PXQ9B+b+FZ4e4gzOIiLSzCm0j+CyrpeRV57HnG1z3C5FjkarFBjzClzzLvgFOJfLX78ccjKOfK6IiJdSaB/ByW1OpnV4a95cpzmem6XOZzgd1YY/4Fwyf3owfDwRyjwegVdExGsotI/A38+fS7teyne7vmNr4Va3y5FjERAEw+6E2xdBvyudZ7ufHADfTQHd9hCRZkSh7YGL0i4iwATw1rq33C5FjkdkElzwFIyfB0m94KO7nM5qK9/RI2Ii0iwotD2QEJbA6e1O592Md6msqXS7HDlerfvAde/Dlf+DgBB483p4/izY9JXblYmIHJZC20OXdb2M/Ip8DbbSUhgDXUfCr76GC5+Gokx45Rfw6mjYvtDt6kRE6qXQ9tCQNkNIiUhRh7SWxs8f+l/l3O8e+RBkLnda3a+PhV3L3K5OROQACm0P+Rk/xnQbw8LdC1mevdztcqShBYY6z3ffsRTOuh+2fgPPnQLTr4SdP7pdnYgIoNA+KmO6jSE6OJpnlj7jdinSWIIj4JTfwR3L4PSJsOVrmHI6TLsMtv3gdnUi4uMU2kchPDCc63pdx1c7vmJZti6dtmih0XD63XDnCqflvX0hvDAcXv4FrJ+t3uYi4gqF9lG6svuVxATH8PRSr54OXBpKSJTT8r5zOYz4K+RugGmXwLOnwPK3oKba7QpFxIcotI9SWGAY1/e+nvk75rMka4nb5UhTCY6Ak25z7nlf+DTUVMDbN8GT/WD+k5rHW0SahEL7GIztNpbYkFjd2/ZFAUFOb/NbvoOx0yGmI3z2J3iiF3x0t9MSFxFpJArtYxAWGMYNvW7gm53fqLXtq/z8oPu5cP0Hzghr3X8BP7wATw2AqZc6s4rV1rpdpYi0MArtYzSm2xhiQ2L595J/u12KuK11X7j4OfjNCqfHeeZyZ1axp/rD1/+E4my3KxSRFkKhfYzCAsO4qfdNLNi1gHnb57ldjniDyGSnx/lvVsClL0FkG5g9CR7v4QyVuvFL9ToXkeNirJf/EklPT7cLF3rnsJJVNVVc9v5llFaX8u6F7xIWGOZ2SeJtstbA4ldgyetQXuDcA+93NfS7wpnzW0TkIMaYRdba9Pr2qaV9HAL9A5l00iR2lezSZXKpX2J3OOdv8Ls1cNEUaNUO5j4IT/SG1y5yHhurLHW7ShFpJhTax6l/Yn8u63oZU1dPZVXuKrfLEW8VGAp9L3c6rv16CZz2B8hZ7zw29mgXeGcCbPwCamvcrlREvJgujzeAwspCLnz3QhLDEpl27jQC/ALcLkmag9pa2DIflr0Bq96DikKIbA29LoLel0Dbgc5sZCLiU3R5vJFFBUVx96C7WZW7iulrprtdjjQXfn7Q6RS48N/w+3VO57W2A+GH552Zxv7VF2b/GXYsVgc2EQHU0m4w1lpum3MbP2T+wNvnv027qHZulyTNVfkeWP0BrHirrsd5DbRqDz0vgB4XQMqJTuCLSIt0uJa2QrsB7SrexaXvX0pSeBJTR01Vb3I5fqV5sPZD5/L5hjlQWwXhidBtlDOgS6dTITDE7SpFpAEptJvQ/B3zmTB7AqM6jeLvp/wdo3uS0lDKCiBjNqz5ANZ/BpXFEBgOnc+AriOhywjnWXERadYOF9rqMdXATm57Mrf1v42nfnyKPgl9uKrHVW6XJC1FaDSccKmzVFfApq+cVvi6T5wgB2d0trSzIW24cxndX/+Li7Qkamk3glpby51z7+Sr7V/xnxH/IT253n8wiTQMayFrFaz7GNZ9Ctt/cO6DB7eC1FMh9QxIPR1iU9UbXaQZ0OVxFxRVFnHlrCspqixi+nnTaR3R2u2SxFeUFcCmLyHjc2cp3O5sj27vhHfHU51e67qULuKVFNou2VCwgas/vJqYkBheGvkSSeFJbpckvsZaZ7rQjXOdwVs2fQUVe5x9cV2c8O5wMnQ4CaLauFqqiDgU2i5amr2U8Z+NJz40npdGvkRCWILbJYkvq62BXUth81dOgG/91unQBs646B1OhnaDof0QiO+qy+kiLlBou2xJ1hLGfzaepPAkXhz5IvGh8W6XJOKoqYbMZU54b/nGWcrynH2hMU6Ap5wIKenQZgCERLlbr4gPUGh7gUW7FzFh9gTahLfh+ZHPK7jFO1kLuRmwdQFsWwBbv4Pc9XU7DSR0d0Zta9vfCfGk3hAQ5GrJIi2NQttL/JD5A7d+fiutglvxrzP+Rc+4nm6XJHJkZfmwYxFsX+T0TN+x6KfWuH+QE9yt+/60JPWCgGB3axZpxhTaXmR17mrumHsHeeV5TD5pMuemnut2SSJHx1oo2Ao7Fzvjou9a4twnL6/r4OYXAPHdIPkESO7trBN7QYT6c4h4QqHtZXLLcvntF79lcdZibup9E7f3vx1/P3+3yxI5dtZC/mYnwDOX/7QU7frpmPAESOzptMQTe0BCD0jopvvkIgdRaHuhqpoq/v793/nfuv8xIHEAfz7pz3Rq1cntskQaVkmOE95Zq2D3Kti9ArLXQHX5T8dEpUBCV6d1vncd38UJefVeFx+k0PZi7214j4e/f5jy6nIm9JvA9b2u13zc0rLV1kDBFsha7SzZayFnLeSsh6rSn44LbgVxnZ0Aj0tzRnSLTXW2hbRyr36RRqbQ9nI5ZTk89N1DfLblM3rE9uBPQ/7ECQknuF2WSNOqrXVGb8te5/Rgz13vBHluBhTuOPDYsDiI6QSxnZwgj+kI0R2cdWRrTV0qzZpCu5n4bMtn/HXBX8ktz+XMdmdyW//b6BLTxe2yRNxXWQr5m5zR3fI2QN4m533eZifobe1Px/oHQat2EN3OGbq1VXvndasUZ4lso8fUxKsptJuR4spiXlv9Gq+ufJWSqhJGdRrF+L7jSW2V6nZpIt6pugL2bHc6wuVvdi69F2ytW7ZBSdZBJxhn3PWoNhDV1gnyqDZOCz2qzU+v9diauESh3QwVlBfw0sqXeH3165TXlDOk9RDGdh/LaSmn6Z63yNGoKoM9O2DPNifc964Ld0DhTmdfVcmh54XGOuEemQwRyRCZtN86CcITISIRgiPVYU4alEK7Gcspy2HG+hn8b+3/2F26m+TwZC5Ku4iRHUfSObqz2+WJNH/WOs+YF+1yQrxwp/O6KNNZiveud0Nt9aHnB4TUBXiC0+M9PN5Zh8U7r8PiITzOWYfFQVBY0/+M0qwotFuA6tpqvtz+JW+seYPvdn2HxZIWncaIDiM4q8NZdInugtG/9kUaT22tMzrc3hAvyYbiLOfye3GW874kp27Jhtqq+j8nIMQJ79BYCIupW8fut45xlpBoCI3+aR0Y2rQ/r7hGod3CZJdm89mWz/h0y6cs3r0YiyU+NJ7BrQczpPUQBicPJjk8WSEu4pa9rffSXCfES3OgNM95v2/Jc4aD3bsuyz+wQ93B/IPrQrzVgUtwVN3rKOd1cNR+ryPrligIjtB9+mbiuEPbGNMOeAI4GzDAbOBOa+1WD84NAf4CXA1EA0uAu6218zwpXqF9eNml2Xy942u+3fUt3+36jrxyZ0zohNAEesf3pk9CH3rF9aJLTBfiQuIU5CLeqrYWKouc8C6tC/HyAigr2G+9Z7+lAMoLoaLQeV9TeeTv8AusC/EICNq7joCgcGd7UPh+SwQEhv30fu/rwDDnEn9guLMOCNUjdg3suELbGBMGLAUqgPsACzwIhAF9rLX19OA44PxpwHnAXcBG4FZgFDDUWrvkSMUrtD1Xa2tZn7+ehbsXsjxnOStyVrClcMu+/THBMaTFpNG5VWfaR7UnJSKFdpHtSIlMISQgxMXKReS4VZU7AV5R5IR4RSFUFDvvKwp/el+5d1vd68piqCxx3leVOK89+QfA/gJCncv3gWEQGOK83rct1LklsHdfQKjT4t+3ve793u0BIQet6xb/vduCnNf+gS22A+DxhvYdwONAN2ttRt22TsB64A/W2scPc25fnJb1jdbal+q2BQArgbXW2guOVLxC+/jsqdjDytyVbCjYwIaCDawvWM/Ggo0UVxUfcFxMcAxJ4UkkhTlLfGg8sSGxxIXGERsSS3RINK2CWhEVHEWgX6BLP42INInqSifMq0qdEN+7VJXWbSv9aV91ed2+/2/v3mPkOuszjn+fmdmLN9kQm7pSc3EuChIkTYCItkGhNUmkOEXgCJFwqUTbIKCEqqJClXoTV9FQlYq2gQpIm4IEEUEQIEZIkAtQaJu0pFFSMDRtaJI6reMEX2PvbS4//njPrM8ez+6ctSc7c7zPxz4+M+95z867jzzzmzmXObPZ8tnU1r3dnVpzS+fRPvFx1rsFffzovD6eK+zd291prDAfTxe46bbVxrJlvW43jrbVGtl62bKNRtIU0QAACplJREFU56SzDAbkRIv2vcBkRFxeaP9HgIjYusK67wHeA5weETO59g8AfwScFhHzKz2+i/bgRQT75/fz5LNPsuvZXex6dhd7Zvbw9MzT7Dmyhz0zezgwf2DZ9acaU0yPT3Pq2KmcMn4K02PTTI1NsaGxgQ2NDel2fQMTjQkm6hNM1ieZaEwwXhtnvJ5NtXHG6mOM1cZo1BpL5nXVadQa1Gt1GmpQU416rU5ddWryZjizk0K7Ba3ZdJ59ay5tKWjNpjcMrbncNJ+m9nzh9kLveXsB2s2s30JuaqZ5az6dBbCkvbn8gYNlbPswvPydA4tmpaJd5oTfi4A7e7TvBK4vse5j+YKdW3ccuCC7bWtIEpsmN7FpchOXbL6kZ59mp8mBuQPsm9vH3tm9HFw4yMH5NB2YP8Dh5mGONI9weOEwhxYOsfvIbmZbs8y2ZplpzrDQWeXmtVXoFu+eEzUkUVMNISQhlt4HlizrzrvZLP7JbXrLr7dkXmgvKh5DUOx3zP0Sm/uWe6wVVjA7udVIFWVJw4ZsKimyfyKOziOATrask1u29P6bp6e5ZgC/RhllivYmYH+P9n3AxhNYt7v8GJLeDrwdYMuWLSWGaIM2Vhtj89RmNk8d3zWQ25028+35xWmuNcdCZ4Fmu8l8e56FzgKtTotmu0mzk6ZWp0Wr06IdbZqdJu1Om050aEVr8XY7js4jYklbkO5HBO1s01u3rZMdlRsRLP7Jbqe/hbasb/dn9LyfzY/Olm61OuZ+9FlO/4NCy3RZ9c80sxMyNn3Gmj1W2a/W6vXML/P+XcezbkTcAtwCafN4icexEVOv1ZmqTTE15i+SMDMblDI7CPfT+xPxRnp/is7bt8K63eVmZmZWQpmivZO0b7roQuBHJdY9LzttrLjuAvBoicc3MzMzyhXtHcBlkhYvMyXpXODybFm/dcfIHbCWnfL1BuCufkeOm5mZ2VFlivbfAY8Dd0q6VtJ20tHku4BPdTtJOkdSS9J7u23Zl6d8AfhrSW+VdBVwO3Ae8L7B/RpmZmYnv75FO/vGsyuB/wI+C9wGPAZcGRH5b+gQUO/xM28APk36FrWvA2cD10TEgyc8ejMzs3Wk1NHj2XeMv65Pn8fpcVR4RMwC784mMzMzO07+eikzM7OKcNE2MzOrCBdtMzOzinDRNjMzqwgXbTMzs4pw0TYzM6sIF20zM7OKUPFygaNG0jPAEwP8kT8H/HSAP2+9co6D4RwHwzkOhnMcjBPN8ZyI6Hld5JEv2oMm6YGIeNmwx1F1znEwnONgOMfBcI6D8Vzm6M3jZmZmFeGibWZmVhHrsWjfMuwBnCSc42A4x8FwjoPhHAfjOctx3e3TNjMzq6r1+EnbzMysktZF0ZZ0tqQvSToo6ZCkL0vaMuxxjSpJ10m6Q9ITkmYlPSLpw5KmC/02Svp7ST+VdETSPZIuHta4R52kb0gKSR8qtDvHEiS9StJ3JR3OnscPSLoyt9w59iHpckl3SXo6y/BBSW8p9HGOGUlnSfqYpPskzWTP33N79CuVmaRJSR+RtDt7bb1P0q+tZkwnfdGWNAV8C3gh8FvAm4EXAN+WdMowxzbC/gBoA38CXAN8ArgRuFtSDUCSgB3Z8t8jXW99jJTrWcMY9CiT9CbgxT3anWMJkn4HuBP4d+C1wPXAF4GpbLlz7EPSJcA9pFzeRsro+8Ctkm7M+jjHpS4AXg/sB77Xq8MqM7uVlP17gVcDu4FvSnpJ6RFFxEk9Ae8iFaALcm3nAS3g3cMe3yhOwOYebb8JBHBldv/a7P4VuT7PA/YBNw/7dxilCTgdeAp4U5bZh3LLnGP//M4FZoHfX6GPc+yf403AAnBqof1+4D7n2DOzWu72W7Nszi30KZUZ6U17ADfk2hrAI8COsmM66T9pA9uB+yPi0W5DRDwG/DMpbCuIiGd6NH8/m5+ZzbcD/x8R386tdxD4Gs616C+AnRHx+R7LnGN/bwE6wCdX6OMc+xsHmqQ3QHkHOLrV1TnmRESnRLeymW0n5f+FXL8WcDuwTdJEmTGth6J9EfDDHu07gQvXeCxVtjWb/zibr5TrFkmnrsmoRpykV5C2UrxzmS7Osb9XAP8JvFHSTyS1JD0q6XdzfZxjf5/J5jdLOkPS6ZLeBlwF/FW2zDmuXtnMLgIei4iZHv3GSZvi+1oPRXsTaX9E0T5g4xqPpZIknQl8ELgnIh7ImlfKFZwtksaATwF/GRGPLNPNOfZ3Buk4lI8Afw5cDdwNfFzSu7I+zrGPiPgh8ErSp7//I+X1t8A7IuL2rJtzXL2ymfXrt6nMgzVWNbTq6nUyutZ8FBWUvUu8k3QMwA35RTjXfv4Q2AD82Qp9nGN/NWAa+O2I+HLW9q3sKN4/lnQzzrEvSS8A7iB9snsHaTP5tcAnJc1FxG04x+NRNrOBZLseivZ+er+D2Ujvdz2WkTRJOiryfGBrRDyZW7yP5XOFdZ5tdkrhn5IOXpko7K+akHQ68CzOsYy9pE/adxfa7yIdsfsLOMcybiLtU311RDSztnslPR/4G0mfxzkej7KZ7QN6nWq8Mbe8r/WweXwnaV9C0YXAj9Z4LJWRbdq9A/hl4FUR8YNCl5Vy/d+IOPwcD3HUnQ9MAp8jPWm7E6RT6vYDF+Mcy9i5THv3E0oH51jGxcDDuYLd9W/A84Gfxzkej7KZ7QTOy05DLvZbAB6lhPVQtHcAl0k6v9uQbVa7PFtmBdm52LeRDlC5NiLu79FtB3CmpK259U4DXoNzBXgIuKLHBKmQX0F6kjrH/r6SzbcV2rcBT0bEUzjHMp4CXiJpvND+K8Ac6ZOec1y9spntIJ2/fX2uXwN4A3BXRMyXerRhnwe3BufZnUJ6cfwBaf/NduBh4H8onK/oaTGzT5CdTwxcVpjOyvrUgH8BdgFvJL2Afof0xD972L/DqE4ce562c+yfmUhfkLSXtC/2atIFGYK0n9s5lsvxuiyzb2avhVcDH8/aPuocV8ztutzr4o3Z/a2rzYx0etd+0m6zq4Avkd4wXVp6PMMOZI1C30La1HuItB/xqxROkPe0JK/Hs/+cvab35/ptAv4h+885A9wLvHjY4x/lqVi0nWPp3E4jHem8h7Qp8T+A33COq87x17OC8kz2WvgQ6XTEunNcNrPlXgu/s9rMSAemfpS01WMO+FfglasZj6/yZWZmVhHrYZ+2mZnZScFF28zMrCJctM3MzCrCRdvMzKwiXLTNzMwqwkXbzMysIly0zczMKsJF28yWkHSapPdLetGwx2JmS7lom1nRy4D3kb4n2cxGiIu2mRW9FJjHV8EzGzn+GlMzWyTpx8ALC813RMR1wxiPmS3lom1miyT9EulKRDuBm7Lm3RHxxPBGZWZdjWEPwMxGysPAWcDHovd11M1siLxP28zyLgLGgQeHPRAzO5aLtpnlXUq6VvBDwx6ImR3LRdvM8l4K/CQiDg17IGZ2LBdtM8u7EJ/qZTayfCCameUdAC6VtA04CPx3ROwd8pjMLONTvsxskaRfBG4FLgEmgV+NiH8a7qjMrMtF28zMrCK8T9vMzKwiXLTNzMwqwkXbzMysIly0zczMKsJF28zMrCJctM3MzCrCRdvMzKwiXLTNzMwqwkXbzMysIn4Gy+mjIj8xopoAAAAASUVORK5CYII=\n", 442 | "text/plain": [ 443 | "
" 444 | ] 445 | }, 446 | "metadata": { 447 | "needs_background": "light" 448 | }, 449 | "output_type": "display_data" 450 | } 451 | ], 452 | "source": [ 453 | "# Simulate the evolution of the system, starting from a uniform state\n", 454 | "T = 100\n", 455 | "x = np.ones((3, T)) / 3.\n", 456 | "\n", 457 | "for t in np.arange(1, T):\n", 458 | " x[:, t] = P @ x[:, t-1]\n", 459 | "\n", 460 | "plt.figure(figsize=(8, 6))\n", 461 | "plt.plot(x.T)\n", 462 | "plt.legend(labels=['$x_1$', '$x_2$', '$x_3$'], loc='upper left')\n", 463 | "plt.xlabel('$t$')\n", 464 | "plt.show()" 465 | ] 466 | }, 467 | { 468 | "cell_type": "markdown", 469 | "metadata": {}, 470 | "source": [ 471 | "We have two eigenvalues with magnitude less than 1, so any contribution to the initial state from those eigenvectors will dampen out over time. The one eigenvalue with magnitude 1 will dominate over time - this eigenvalue corresponds to the eigenvector having 100% occupation of state 1, so eventually the probability of being in state 1 approaches 100%.\n", 472 | "\n", 473 | "Markov transition matrices, because of the structure imposed to contain probabilities summing to 1, will always have an eigenvalue with magnitude 1. This idea is one of the foundational principles of Google's PageRank algorithm. Thus, whether you like it or not, this theory impacts your life every single day!" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": null, 479 | "metadata": {}, 480 | "outputs": [], 481 | "source": [] 482 | } 483 | ], 484 | "metadata": { 485 | "kernelspec": { 486 | "display_name": "Python 3", 487 | "language": "python", 488 | "name": "python3" 489 | }, 490 | "language_info": { 491 | "codemirror_mode": { 492 | "name": "ipython", 493 | "version": 3 494 | }, 495 | "file_extension": ".py", 496 | "mimetype": "text/x-python", 497 | "name": "python", 498 | "nbconvert_exporter": "python", 499 | "pygments_lexer": "ipython3", 500 | "version": "3.6.7" 501 | }, 502 | "latex_envs": { 503 | "LaTeX_envs_menu_present": true, 504 | "autoclose": true, 505 | "autocomplete": false, 506 | "bibliofile": "biblio.bib", 507 | "cite_by": "apalike", 508 | "current_citInitial": 1, 509 | "eqLabelWithNumbers": false, 510 | "eqNumInitial": 1, 511 | "hotkeys": { 512 | "equation": "Ctrl-E", 513 | "itemize": "Ctrl-I" 514 | }, 515 | "labels_anchors": false, 516 | "latex_user_defs": false, 517 | "report_style_numbering": false, 518 | "user_envs_cfg": false 519 | }, 520 | "toc": { 521 | "base_numbering": 1, 522 | "nav_menu": {}, 523 | "number_sections": false, 524 | "sideBar": true, 525 | "skip_h1_title": false, 526 | "title_cell": "Table of Contents", 527 | "title_sidebar": "Contents", 528 | "toc_cell": false, 529 | "toc_position": {}, 530 | "toc_section_display": true, 531 | "toc_window_display": true 532 | } 533 | }, 534 | "nbformat": 4, 535 | "nbformat_minor": 2 536 | } 537 | -------------------------------------------------------------------------------- /Ch 5 - Singular Value Decomposition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Singular Value Decomposition\n", 8 | "\n", 9 | "The Singular Value Decomposition (SVD) is a topic that is not part of the standard linear algebra introduction, but is possibly the most incredibly powerful tool in the field. The SVD makes many mathematical proofs very straightforward, and has immediate applications in\n", 10 | "\n", 11 | "* compression,\n", 12 | "* error analysis,\n", 13 | "* convex **and** non-convex optimization,\n", 14 | "\n", 15 | "...just to name a few." 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "### Python Setup" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 1, 28 | "metadata": { 29 | "ExecuteTime": { 30 | "end_time": "2019-11-03T21:54:57.746169Z", 31 | "start_time": "2019-11-03T21:54:56.391964Z" 32 | } 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "# Render MPL figures within notebook cells\n", 37 | "%matplotlib inline\n", 38 | "\n", 39 | "# Import python libraries\n", 40 | "import numpy as np\n", 41 | "import matplotlib.pyplot as plt\n", 42 | "from matplotlib import rcParams" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 2, 48 | "metadata": { 49 | "ExecuteTime": { 50 | "end_time": "2019-11-03T21:54:57.758468Z", 51 | "start_time": "2019-11-03T21:54:57.750295Z" 52 | } 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "# Configure some defaults for plots\n", 57 | "rcParams['font.size'] = 16\n", 58 | "rcParams['figure.figsize'] = (10, 3)" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 3, 64 | "metadata": { 65 | "ExecuteTime": { 66 | "end_time": "2019-11-03T21:54:57.767433Z", 67 | "start_time": "2019-11-03T21:54:57.762978Z" 68 | } 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "# Set Numpy's random number generator so the same results are produced each time the notebook is run\n", 73 | "np.random.seed(0)" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### Derivation\n", 81 | "\n", 82 | "The SVD is a way of decomposing a matrix into smaller matrices with very well-defined structure. We can do a nice derivation of the derivation, again, geometrically:\n", 83 | "\n", 84 | "We start with the fun fact that if you transform a sphere of unit radius by **any** matrix $A$, the resulting points make an ellipse. Notice that we are **not restricted in any way** on the matrix $A$ here.\n", 85 | "\n", 86 | "\n", 87 | "\n", 88 | "The output ellipse is defined by a set of mutually orthogonal principal axis vectors, $u$. Orthogonality, as a reminder, means that the inner product between two different vectors in the set is zero: \n", 89 | "
\n", 90 | "
\n", 91 | " $u_1^{\\mathsf{T}}u_2 = 0$\n", 92 | "
\n", 93 | "
\n", 94 | "\n", 95 | "An absolutely remarkable result is that - working backwards - the input vectors $v$ that map onto the principle axes **also** form a mutually orthogonal set. For simplicity, we'll scale the $u$ and $v$ vectors to have unit length, and use coefficients $\\sigma$ to make the equality work out.\n", 96 | "\n", 97 | "
\n", 98 | "
\n", 99 | " $Av=\\sigma u$\n", 100 | "
\n", 101 | "
\n", 102 | "\n", 103 | "Now, we can horizontally stack all the vectors into matrices: \n", 104 | "\n", 105 | "
\n", 106 | "
\n", 107 | " $AV=U\\Sigma$\n", 108 | "
\n", 109 | "
\n", 110 | "\n", 111 | "The matrix $\\Sigma$ contains the scaling coefficients $\\sigma$ along its main diagonal:\n", 112 | "\n", 113 | "\n", 114 | "\n", 115 | "We need one last piece, which comes from exploiting the fact that these new matrices $U$ and $V$ are built from mutually orthogonal vectors with length 1. These are called **orthogonal** matrices, and have the property that \n", 116 | "\n", 117 | "
\n", 118 | "
\n", 119 | " $V^\\mathsf{T} V = VV^\\mathsf{T} = I \\\\\n", 120 | " U^\\mathsf{T} U = UU^\\mathsf{T} = I$\n", 121 | "
\n", 122 | "
\n", 123 | "\n", 124 | "Therefore, we can conclude that **any** matrix $A$ can be written in the form \n", 125 | "\n", 126 | "
\n", 127 | "
\n", 128 | " $A=U\\Sigma V^\\mathsf{T}$\n", 129 | "
\n", 130 | "
\n", 131 | "\n", 132 | "This is the singular value decomposition, where \"singular values\" refers to the scaling coefficients $\\sigma$. Let's dive into how we can use this thing for fun and profit.\n" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "### Pseudoinverse\n", 140 | "\n", 141 | "We saw in the previous section the least squares and least norm solutions when solving problems of the form $y = Ax$. The SVD provides an inconceivably simple way of arriving at these solutions *simultaneously*. \n", 142 | "\n", 143 | "1. Drop any rows or columns in $\\Sigma$ that have only zeros such that it becomes a square, diagonal matrix. If A has rank $r$, this will give $U \\in \\mathbb{R}^{m, r}$, $V \\in \\mathbb{R}^{n, r}$, and $\\Sigma \\in \\mathbb{R}^{r, r}$.\n", 144 | "2. Now, by construction the matrix $V\\Sigma^{-1}U^{\\mathsf{T}}$ is an inverse to $U\\Sigma V^{\\mathsf{T}}$ (this is super easy to test)\n", 145 | "3. Therefore, $x = V\\Sigma^{-1}U^{\\mathsf{T}}y$\n", 146 | "\n", 147 | "It's easy to show just by substituting the SVD into the least squares and least norm solutions that one arrives at the same result. Try this at home - cancelling everything out is immensely satisfying.\n", 148 | "\n", 149 | "The matrix $V\\Sigma^{-1}U^{\\mathsf{T}}$ is so significant that it has a fancy name, the **Moore-Penrose Pseudoinverse**. It also has a fancy notation, $A^{\\dagger}$." 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": {}, 155 | "source": [ 156 | "### Basis for the Nullspace\n", 157 | "\n", 158 | "I mentioned in the dicussion on the nullspace back in chapter 1 that there would be a way to get the basis vectors for the nullspace. If $A$ has rank $r$, then the last $n-r$ columns of $V$ are those basis vectors! These are the directions in the input space that get multiplied by singular values of zero, and are, thus, flattened." 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "### Error Analysis\n", 166 | "\n", 167 | "Suppose you have a linear system with some small error, $\\delta$ in the observed measurement such that $y = Ax + \\delta$. If you solve using the pseudoinverse, the resulting error in the estimate of $x$ will be $V\\Sigma^{-1} U^{\\mathsf{T}}\\delta$. A property of $U$ and $V$ as orthogonal matrices is that, when viewed as a transform, they only rotate or reflect - lengths of vectors are preserved. This means the only thing that affects the magnitude of the error is $\\Sigma^{-1}$. This is a diagonal matrix whose entries are $\\sigma_i^{-1}$ - if $\\sigma_i$ is very, very small, then this means you have very, very big numbers multiplying your errors!\n", 168 | "\n", 169 | "You will see people refer to matrices as **well-conditioned** or **ill-conditioned**. These terms indicate whether the matrix has these very small singular values and is at risk of producing large errors. How small is small, though? Since size is relative, the standard is to normalize to the largest singular value. This gives us the concept of the **condition number**: \n", 170 | "\n", 171 | "
\n", 172 | "
\n", 173 | " $k = \\frac{\\sigma_{max}}{\\sigma_{min}}$\n", 174 | "
\n", 175 | "
\n", 176 | "\n", 177 | "When $k$ is large, there is a wide spread in the singular values and our estimates may be very sensitive to small changes or noise." 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "### Low-Rank Approximation\n", 185 | "\n", 186 | "For any matrix that you would obtain from *measurements* of something, as opposed to analytically, it's basically guaranteed that the matrix will be full rank. Suppose you build a measurement system whose corresponding matrix is **not** full rank - if you so much as speak loudly nearby, the infinitesimal noise you introduce will add some floating point-level error that a computer will correctly identify as \"full rank\". Practically, though, your matrix is still singular.\n", 187 | "\n", 188 | "The SVD provides a handy solution - simply round the smallest singular values to zero. One can show this is actually an optimal approximation to the original matrix $A$ under a particular cost function (the details are beyond the scope of what I'm covering here, but the Boyd lectures explain it well!). How small should you set the threshold? That depends on your application. We'll see an example below." 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "metadata": {}, 194 | "source": [ 195 | "#### Compression\n", 196 | "\n", 197 | "I mentioned in the discussion on rank that any matrix $A$ can be factored into component matrices such that $A = QR$, which require storing fewer elements than the original matrix depending on the rank of $A$. The SVD provides us with an immediate way of getting $Q$ and $R$. If $A$ is low rank - implying that transforming by $A$ flattens some dimensions - then some of the singular values will be zero. We can drop the rows and columns of $U$ and $V$ corresponding to those singular values and merge $\\Sigma$ into either of them:\n", 198 | "\n", 199 | "
\n", 200 | "
\n", 201 | " $A=U\\Sigma V^\\mathsf{T}=U(\\Sigma V^\\mathsf{T})=QR$\n", 202 | "
\n", 203 | "
\n", 204 | "\n", 205 | "More interestingly, the SVD provides a mechanism for *lossy* compression as well, based on the low-rank approximation idea presented above. We round the singular values below some threshold to zero:" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": 4, 211 | "metadata": { 212 | "ExecuteTime": { 213 | "end_time": "2019-11-03T21:55:01.391815Z", 214 | "start_time": "2019-11-03T21:55:01.304112Z" 215 | } 216 | }, 217 | "outputs": [], 218 | "source": [ 219 | "# Construct a matrix with a relatively low rank\n", 220 | "\n", 221 | "m = 1000\n", 222 | "n = 500\n", 223 | "r = 10\n", 224 | "\n", 225 | "Q = np.random.rand(m, r)\n", 226 | "R = np.random.rand(r, n)\n", 227 | "A = Q @ R" 228 | ] 229 | }, 230 | { 231 | "cell_type": "code", 232 | "execution_count": 5, 233 | "metadata": { 234 | "ExecuteTime": { 235 | "end_time": "2019-11-03T21:55:01.965330Z", 236 | "start_time": "2019-11-03T21:55:01.586665Z" 237 | } 238 | }, 239 | "outputs": [ 240 | { 241 | "name": "stdout", 242 | "output_type": "stream", 243 | "text": [ 244 | "U: (1000, 1000)\n", 245 | "S: (500,)\n", 246 | "Vt: (500, 500)\n" 247 | ] 248 | } 249 | ], 250 | "source": [ 251 | "# Perform the Singular Value Decomposition\n", 252 | "U, Σ, Vt = np.linalg.svd(A)\n", 253 | "\n", 254 | "print('U:', U.shape)\n", 255 | "print('S:', Σ.shape)\n", 256 | "print('Vt:', Vt.shape)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "In the figure below, notice how the singular values beyond $\\text{rank(A)}$ (sorted from largest to smallest) are not precisely zero - this is due to floating point rounding error." 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 6, 269 | "metadata": { 270 | "ExecuteTime": { 271 | "end_time": "2019-11-03T21:55:06.888571Z", 272 | "start_time": "2019-11-03T21:55:04.051679Z" 273 | } 274 | }, 275 | "outputs": [ 276 | { 277 | "data": { 278 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAArQAAADbCAYAAACcAwJ8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAUqklEQVR4nO3df5Cd1X3f8ffHlRMHBcciqMQxEFmFQkTrICQ7EJqa4JqEUAsaU9pM4mGS8KPu2CltmsSm2PmBM3Y6DvyBSweM28bUaTImTJDjxlMbGEEoOBEYy4iEgkFGavB4qWRhwOMU69s/nmfTy2WXvXv3x92zer9mdu7e85x77nf1zJE+e3Se56aqkCRJklr1ikkXIEmSJC2EgVaSJElNM9BKkiSpaQZaSZIkNc1AK0mSpKYZaCVJktS0NZMuQIvv6KOPrg0bNky6DEmSpDndf//9T1fV+oWMYaBdhTZs2MDOnTsnXYYkSdKcknxloWO45UCSJElNM9BKkiSpaQZaSZIkNc1AK0mSpKYZaCVJktQ073Kgv7HhPZ9e8Bh7PnTeIlQiSZI0OldoJUmS1DQDrSRJkppmoJUkSVLTDLSSJElqmoFWkiRJTTPQSpIkqWkGWkmSJDXNQCtJkqSmGWglSZLUNAOtJEmSmmaglSRJUtPWTLoArX4b3vPpRRlnz4fOW5RxJEnS6mKgVbOWMigvxthLNe5MY/tn0e64kqSFc8uBJEmSmmaglSRJUtMMtJIkSWqagVaSJElNM9BKkiSpad7lYAVLcjtwNFDAN4B3V9WDk61KkiRpZTHQrmw/VVUHAZL8E+C/AKdOtCJJkqQVxi0HI0pybJLrktyb5PkklWTDLH2PS3JLkoNJnklya5Lj5/ue02G29+oxS5ckSVrVDLSjOwG4CDgA3D1bpyRHAHcAJwMXA+8ATgTuTLJ2vm+a5BNJ9gFXAz87Rt2SJEmrmlsORndXVR0DkOQS4JxZ+l0KbAROqqrH+v67gEeBy4Fr+rYHgNlWbTdX1V6AqvqZgff8bcCPGZIkSRrgCu2IqurQiF23AfdNh9n+tU8A9wDnD7SdVlVHz/K1d4ZxPwa8Ncn3LuTnkCRJWm0MtIvvFOChGdp3A5tGHSTJuiSvHWh6O/A1YP/CypMkSVpd3HKw+I6i22c7bD+wbh7jrAP+IMmrgEN0YfYfV1XN1DnJZcBlAMcfP+/rzyRJkpploF0aM4XOzGuAqseBN86j/43AjQBbt26dMfRKkiStRm45WHwH6FZph61j5pVbSZIkLYCBdvHtpttHO2wT8PAy1yJJkrTqGWgX33bg9CQbpxv6D2A4sz8mSZKkReQe2nlIcmH/7Zb+8dwkU8BUVe3o2z4KvAu4LclVdPtprwb2AjcsZ72SJEmHAwPt/Hxy6Pn1/eMO4CyAqnouydnAtcDNdBeD3Q5cUVXPLlOdkiRJhw0D7TxU1Uh3KqiqJ+nuGytJkqQl5h5aSZIkNc1AK0mSpKYZaCVJktQ0A60kSZKaZqCVJElS0wy0kiRJapqBVpIkSU0z0EqSJKlpBlpJkiQ1zUArSZKkphloJUmS1DQDrSRJkppmoJUkSVLTDLSSJElqmoFWkiRJTTPQSpIkqWkGWkmSJDXNQCtJkqSmGWglSZLUNAOtJEmSmrZm0gVoZkm+H/jvA01rgdcDf7uq9k+mKkmSpJXHQLtCVdVfAadOP0/yHuBHDLOSJEkv5paDESU5Nsl1Se5N8nySSrJhlr7HJbklycEkzyS5NcnxCyzh54GPLXAMSZKkVcdAO7oTgIuAA8Dds3VKcgRwB3AycDHwDuBE4M4ka8d54yT/EDgS+PQ4r5ckSVrNlnzLQZLXAv8b+NGqumep328J3VVVxwAkuQQ4Z5Z+lwIbgZOq6rG+/y7gUeBy4Jq+7QFgtlXbzVW1d+D5LwC/W1UvLPinkCRJWmWWYw/tBcAUcO8yvNeSqapDI3bdBtw3HWb71z6R5B7gfPpAW1WnjTJYklcDbwc2z69iSZKkw8NybDm4ANg+j0DYulOAh2Zo3w1sGmO8nwbur6pHX65TksuS7Eyyc2pqaoy3kSRJatPIgTbJ2iS/neSxJH/dXxQ1+PVLM7zm1cBZwB8tdKyGHEW3z3bYfmDdGOP9AnDTXJ2q6saq2lpVW9evXz/G20iSJLVppC0HSQLcCpwJ/BawEzgD+DVgD/DfePE9U6edB/w18LlFGKslNUNbxhqo6k0LrEWSJGlVG3UP7TuBtwI/XlWf7ds+m+RU4EeB91XVTCHuAuAzVfWtRRirFQfoVmmHrWPmlVtJkiQtwKhbDn4O+OxAAJ32l8C66QCaZF2SO/vvvwP4CYa2G4w6VsN20+2jHbYJeHiZa5EkSVr15gy0SY4BtgJ/MsPh1wJfnX5SVQeq6sf6p28BvouBe6fOZ6yGbQdOT7JxuqH/AIYz+2OSJElaRKOs0P5A//jUYGOSvwWcC9w20PabSd7fP70A2FFVX1/AWL+f5FNJHk/yR0lOTfLH/fObB/puSXJXkvv7C83eN3DshiQf7r///iQPJ/lHI/zcL5HkwiQXAlv6pnP7tjcPdPso3V7g25Kcn2Rb/3PtBW4Y530lSZI0u1H20E4H0pOH2n+Vbl/oYEjbAlzfX/j1NrqLvhYy1rfpPp3r28A+4F/R3ZM1wFeTvLaqngIeB86uqheSfBfwZJKPVNUB4H3AQ0l+D/hPwK9W1ecYzyeHnl/fP+6gu5sDVfVckrOBa4Gb+1pvB66oqmfHfF9JkiTNYpRA+yjwBeCXk0wBX6b78IB3Au+uql0DfbcADwCn020huG2BY51VVd8ESLIGeH9Vfatf0V0DPNP3/Qng8iTr6ALka+jurkBVfS3JR4B7gIur6lMj/MwzqqqR7lRQVU/SBW9JkiQtsTkDbVVVkguAjwD/nm6bwv3A+YPhMMmxwKGqeirJFcDOqtq3gLFeUVV/2T9/PfD1gY+D/UHgyX419DzgX/djPNWvjv6Hqnquf+33Aj8FHKT7b39JkiStIiPdtqtfcdw2R7ctdOEUuo94vXmmTvMYa+fA8zcOPR88vgV4sA+z6+mC8p8DJPke4DPAh4EX6D529ow53luSJEkNGfU+tKOY3m5AVQ3vkR1nrMEAu5XZA+3H6S7A2gXsorsga2eStXQf0HBTVf3Xfl/vLyX551X1+wusT5IkSSvEogXaqnr/3L3GG6uqfmXo+S8OfL8H+KFZhjpzoF/RrfRKkiRpFRn1gxUkSZKkFclAK0mSpKYZaCVJktQ0A60kSZKaZqCVJElS0wy0kiRJapqBVpIkSU0z0EqSJKlpBlpJkiQ1zUArSZKkphloJUmS1DQDrSRJkppmoJUkSVLTDLSSJElqmoFWkiRJTTPQTliSK5M8kuRQkgvme1ySJOlwZ6CdvNuBnwTuGvO4JEnSYc1AOyDJsUmuS3JvkueTVJINs/Q9LsktSQ4meSbJrUmOn+97VtXnq+rL4x6XJEk63BloX+wE4CLgAHD3bJ2SHAHcAZwMXAy8AzgRuDPJ2mWoU5IkSb01ky5ghbmrqo4BSHIJcM4s/S4FNgInVdVjff9dwKPA5cA1fdsDwGyrtpurau8i1i5JknRYMtAOqKpDI3bdBtw3HWb71z6R5B7gfPpAW1WnLX6VkiRJGuSWg/GcAjw0Q/tuYNMy1yJJknRYM9CO5yi6fbbD9gPr5jNQkquS7APOAG5Ksi/J9416fKDfZUl2Jtk5NTU1rx9GkiSpZQba8dUMbZn3IFUfqKpjq+o7q+ro/vuvjnp8oN+NVbW1qrauX79+vmVIkiQ1y0A7ngN0q7TD1jHzyq0kSZKWiIF2PLvp9tEO2wQ8vMy1SJIkHdYMtOPZDpyeZON0Q/8BDGf2xyRJkrRMvG3XkCQX9t9u6R/PTTIFTFXVjr7to8C7gNuSXEW3n/ZqYC9ww3LWK0mSdLgz0L7UJ4eeX98/7gDOAqiq55KcDVwL3Ex3MdjtwBVV9ewy1SlJkiQMtC9RVSPdqaCqngTevsTlSJIkaQ7uoZUkSVLTDLSSJElqmoFWkiRJTTPQSpIkqWkGWkmSJDXNQCtJkqSmGWglSZLUNAOtJEmSmmaglSRJUtMMtJIkSWqagVaSJElNM9BKkiSpaQZaSZIkNc1AK0mSpKYZaCVJktQ0A60kSZKaZqCVJElS0wy0kiRJapqBVpIkSU0z0E5YkiuTPJLkUJILZji+pz/+YP91ySTqlCRJWqnWTLoAcTvwB8DHXqbPP6uqB5epHkmSpKa4QjsgybFJrktyb5Lnk1SSDbP0PS7JLUkOJnkmya1Jjp/ve1bV56vqywutXZIk6XBloH2xE4CLgAPA3bN1SnIEcAdwMnAx8A7gRODOJGuXoK6PJ/lSko8ned0SjC9JktQstxy82F1VdQxAv1f1nFn6XQpsBE6qqsf6/ruAR4HLgWv6tgeA2VZtN1fV3hFqenNVfSXJGuBK4BbgjBF/HkmSpFXPFdoBVXVoxK7bgPumw2z/2ieAe4DzB9pOq6qjZ/kaJcxSVV/pH18ArgV+OMkrR/2ZJEmSVjsD7XhOAR6aoX03sGmx3iTJ2iSvGWj6GeChqvq/i/UekiRJrTPQjucoun22w/YD6+YzUJKrkuyj20ZwU5J9Sb6vP3wM3b7cXUm+RLcy/E9nGeeyJDuT7JyamppPCZIkSU1zD+34aoa2zHuQqg8AH5jl2OPA5hHHuRG4EWDr1q0z1SZJkrQquUI7ngN0q7TD1jHzyq0kSZKWiIF2PLvp9tEO2wQ8vMy1SJIkHdYMtOPZDpyeZON0Q/8BDGf2xyRJkrRM3EM7JMmF/bdb+sdzk0wBU1W1o2/7KPAu4LYkV9Htp70a2AvcsJz1SpIkHe4MtC/1yaHn1/ePO4CzAKrquSRn090X9ma6i8FuB66oqmeXqU5JkiRhoH2JqhrpTgVV9STw9iUuR9JhYMN7Pr0o4+z50HmLMo4ktcY9tJIkSWqaK7SStEot1crvUq4ot1azfxZLP+5Sjt3auIs1dot/xnNxhVaSJElNM9BKkiSpaQZaSZIkNc1AK0mSpKYZaCVJktQ0A60kSZKaZqCVJElS0wy0kiRJapqBVpIkSU0z0EqSJKlpBlpJkiQ1zUArSZKkphloJUmS1DQDrSRJkppmoJUkSVLTDLSSJElqmoFWkiRJTUtVTboGLbIk3wAemXQdGtvRwNOTLkJj8dy1zfPXNs9fu06qqiMXMsCaxapEK8ojVbV10kVoPEl2ev7a5Llrm+evbZ6/diXZudAx3HIgSZKkphloJUmS1DQD7ep046QL0IJ4/trluWub569tnr92LfjceVGYJEmSmuYKrSRJkppmoF0lkhyX5JYkB5M8k+TWJMdPui7NLclZSWqGr69Puja9WJJjk1yX5N4kz/fnacMM/dYluSnJ00meS/K5JH9/+SvWoFHOX5INs8zHSvKayVSuJBcm+cMkX0nyzSSPJPlgkiOH+jn3VphRzt1izDtv27UKJDkCuAP4FnAxUMAHgDuTvKGqnptkfRrZLwJ/PvD8hUkVolmdAFwE3A/cDZwz3CFJgO3A64F3AweA99LNx1Orat/ylashc56/AR+kO4+DvrFEdWlu/xZ4ErgS2AdsBn4d+LEkP1JVh5x7K9ac526g79jzzkC7OlwKbKS7MfFjAEl2AY8ClwPXTLA2je4vquq+SRehl3VXVR0DkOQSZg5E24B/AJxdVXf2fe8FngB+he4XF03GKOdv2uPOxxXlbVU1NfB8R5L9wO8CZ9Et6jj3VqZRzt20seedWw5Wh23AfdNhFqCqngDuAc6fWFXSKjO0kjCbbcBfTf+D2r/uIPApnI8TNeL50wo0FIimTf+P1uv6R+feCjTiuVswA+3qcArw0Aztu4FNy1yLxveJJN9O8n+S/J57oJv1cvPx+CTfvcz1aDwfTPJCf13Cdvdhrkhv7h//on907rVj+NxNG3veueVgdTiKbq/QsP3AumWuRfN3EPgdYAfwDN3+oiuBe5NsrqqvTbI4zdtRwJ4Z2vf3j+uAZ5etGs3Xt4AbgP8BTAEn083H/5nkTVU1/A+wJiDJ64DfBD5XVdMfm+rca8As527B885Au3rMdEPhLHsVmreq+gLwhYGmHUnuAv6Mbs/XVRMpTOMKzsdmVdVTwL8YaLo7yWfoVvn+HfCzEylMf6Nfab2N7sLZnxs8hHNvRZvt3C3GvDPQrg4H6H4zHbaOmVdutcJV1QNJ/hfwxknXonnbz+zzEZyTzamqvUn+FOfjxCV5Fd1V8BuBNw/ducC5t4LNce5eYr7zzj20q8Nuur1DwzYBDy9zLVo8s602aGV7ufn4ZFX5X55tcj5OWJJXAn8IvAn4yar60lAX594KNcK5m/WljDjvDLSrw3bg9CQbpxv6m4WfyUvv56YGJNkK/F3g85OuRfO2HXhdkumLHkjyauBtOB+b1F+geSbOx4lJ8grgE8BbgPNnubWTc28FGvHczfS6ec27VPkLZ+uSrAW+CHyTbr9lAVcDRwJv8LfSlS3JJ+juk/gA8HW6i8LeCzwPnFZVT0+wPA1JcmH/7Vvo9nz9S7qLGKaqakf/l/efAscBv8z/v7n7G4Afqqq9y1+1po1w/n6HbrHn3r79JLrz9z3AD1fVI8tftZL8R7rz9VvAHw8d3ldV+5x7K9OI527B885Au0r0v8lcC7yVbon+duCKqtozybo0tyTvBX4a+AHgCOCrwJ8Av9ZvlNcKkmS2vzR3VNVZfZ+jgA8DFwCvovtL+t9U1ReXpUjNaq7zl+TngXfSfarYkcDTdDd+/w3D7OQk2UP3d+RMfqOqfr3v59xbYUY5d4sx7wy0kiRJapp7aCVJktQ0A60kSZKaZqCVJElS0wy0kiRJapqBVpIkSU0z0EqSJKlpBlpJkiQ1zUArSRpJkjclqSRnTLoWSRrkBytIkkaSZD3wd4A/q6pDk65HkqYZaCVJktQ0txxIkkaS5ItJ/vOk65CkYQZaSdKcknwH8IPAFyZdiyQNM9BKkkbx94BXYqCVtAIZaCVJo9gMFPDFSRciScMMtJKkUWwGvlxVz0y6EEkaZqCVJI1iM243kLRCGWglSS8rySuAN2CglbRCGWglSXM5EfhuDLSSVigDrSRpLpv7xwcnWoUkzcJPCpMkSVLTXKGVJElS0wy0kiRJapqBVpIkSU0z0EqSJKlpBlpJkiQ1zUArSZKkphloJUmS1DQDrSRJkppmoJUkSVLT/h91GlkH01rn0gAAAABJRU5ErkJggg==\n", 279 | "text/plain": [ 280 | "
" 281 | ] 282 | }, 283 | "metadata": { 284 | "needs_background": "light" 285 | }, 286 | "output_type": "display_data" 287 | } 288 | ], 289 | "source": [ 290 | "plt.bar(np.arange(Σ.size) + 1, Σ/Σ.max())\n", 291 | "plt.xlim(0, 25)\n", 292 | "plt.yscale('log')\n", 293 | "plt.ylabel('$\\sigma_i / \\sigma_{max}$', rotation=0, labelpad=30)\n", 294 | "plt.xlabel('$i$')\n", 295 | "plt.show()" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": 8, 301 | "metadata": { 302 | "ExecuteTime": { 303 | "end_time": "2019-11-03T21:55:46.999891Z", 304 | "start_time": "2019-11-03T21:55:46.919192Z" 305 | } 306 | }, 307 | "outputs": [ 308 | { 309 | "data": { 310 | "text/plain": [ 311 | "500" 312 | ] 313 | }, 314 | "execution_count": 8, 315 | "metadata": {}, 316 | "output_type": "execute_result" 317 | } 318 | ], 319 | "source": [ 320 | "# Numpy will still identify A as full rank, depending on a tolerance argument\n", 321 | "np.linalg.matrix_rank(A, tol=1e-33)" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": 9, 327 | "metadata": { 328 | "ExecuteTime": { 329 | "end_time": "2019-11-03T21:55:57.336870Z", 330 | "start_time": "2019-11-03T21:55:57.219690Z" 331 | } 332 | }, 333 | "outputs": [ 334 | { 335 | "data": { 336 | "text/plain": [ 337 | "10" 338 | ] 339 | }, 340 | "execution_count": 9, 341 | "metadata": {}, 342 | "output_type": "execute_result" 343 | } 344 | ], 345 | "source": [ 346 | "np.linalg.matrix_rank(A, tol=1e-5)" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": 10, 352 | "metadata": { 353 | "ExecuteTime": { 354 | "end_time": "2019-11-03T21:56:18.184105Z", 355 | "start_time": "2019-11-03T21:56:18.123459Z" 356 | } 357 | }, 358 | "outputs": [], 359 | "source": [ 360 | "# Demonstrate compression savings\n", 361 | "# We want A = QR\n", 362 | "\n", 363 | "r = 10\n", 364 | "Q = U[:, :r]\n", 365 | "R = np.diag(Σ[:r]) @ Vt[:r, :]" 366 | ] 367 | }, 368 | { 369 | "cell_type": "code", 370 | "execution_count": 13, 371 | "metadata": { 372 | "ExecuteTime": { 373 | "end_time": "2019-11-03T21:56:35.556578Z", 374 | "start_time": "2019-11-03T21:56:35.546113Z" 375 | } 376 | }, 377 | "outputs": [ 378 | { 379 | "data": { 380 | "text/plain": [ 381 | "500000" 382 | ] 383 | }, 384 | "execution_count": 13, 385 | "metadata": {}, 386 | "output_type": "execute_result" 387 | } 388 | ], 389 | "source": [ 390 | "# Number of elements to store A\n", 391 | "A.size" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": 14, 397 | "metadata": { 398 | "ExecuteTime": { 399 | "end_time": "2019-11-03T21:56:44.934422Z", 400 | "start_time": "2019-11-03T21:56:44.926146Z" 401 | } 402 | }, 403 | "outputs": [ 404 | { 405 | "data": { 406 | "text/plain": [ 407 | "15000" 408 | ] 409 | }, 410 | "execution_count": 14, 411 | "metadata": {}, 412 | "output_type": "execute_result" 413 | } 414 | ], 415 | "source": [ 416 | "# Number of elements to store factorized matrices\n", 417 | "Q.size + R.size" 418 | ] 419 | }, 420 | { 421 | "cell_type": "markdown", 422 | "metadata": {}, 423 | "source": [ 424 | "#### Document Search\n", 425 | "\n", 426 | "Below is another fun example of how low-rank approximation can be applied. We have a set of $n$ documents, and within each document we count the number of instances of $m$ specific words. Ultimately, our goal is to be able to sort the documents by relevance given a target word. The problem is set up so that there are patterns in word usage. In fact, there are 3 general categories of document - music, math, and cats. The word usage definitely clusters by genre, but there are some spurious cases that break the pattern. SVD will help us identify the patterns despite the noise by finding the best approximation.\n", 427 | "\n", 428 | "We start by setting up the problem data:" 429 | ] 430 | }, 431 | { 432 | "cell_type": "code", 433 | "execution_count": 16, 434 | "metadata": { 435 | "ExecuteTime": { 436 | "end_time": "2019-11-04T04:11:00.537621Z", 437 | "start_time": "2019-11-04T04:11:00.515852Z" 438 | } 439 | }, 440 | "outputs": [ 441 | { 442 | "data": { 443 | "text/plain": [ 444 | "array([[ 5, 10, 0, 0, 0, 0, 0, 0, 0],\n", 445 | " [ 5, 3, 8, 0, 0, 0, 0, 0, 2],\n", 446 | " [ 2, 6, 12, 0, 0, 0, 0, 0, 0],\n", 447 | " [ 0, 0, 0, 5, 9, 2, 0, 0, 0],\n", 448 | " [ 0, 0, 0, 6, 4, 5, 0, 0, 0],\n", 449 | " [ 0, 0, 0, 4, 4, 6, 0, 0, 0],\n", 450 | " [ 0, 0, 0, 0, 0, 0, 10, 4, 5],\n", 451 | " [ 0, 0, 0, 0, 0, 0, 10, 5, 11]])" 452 | ] 453 | }, 454 | "execution_count": 16, 455 | "metadata": {}, 456 | "output_type": "execute_result" 457 | } 458 | ], 459 | "source": [ 460 | "words = np.array([\n", 461 | " 'drums',\n", 462 | " 'piano',\n", 463 | " 'chords',\n", 464 | " 'bayesian',\n", 465 | " 'matrix',\n", 466 | " 'coefficient',\n", 467 | " 'fluffy',\n", 468 | " 'meow'\n", 469 | "])\n", 470 | "\n", 471 | "titles = np.array([\n", 472 | " 'Music Theory for Beginners',\n", 473 | " 'A History of Jazz',\n", 474 | " 'Beethoven vs. Mechagodzilla',\n", 475 | " 'Elements of Statistical Learning',\n", 476 | " 'Finite Element Methods: the Hottest Trends',\n", 477 | " 'Introduction to Linear Algebra',\n", 478 | " '10 Fluffy Cats you need to see RIGHT NOW!!!',\n", 479 | " 'The Lost Art of Grooming',\n", 480 | " 'Piano Cat Strikes Again'\n", 481 | "])\n", 482 | "\n", 483 | "word_counts = [\n", 484 | " [ 5, 5, 2, 0, 0, 0, 0, 0],\n", 485 | " [10, 3, 6, 0, 0, 0, 0, 0],\n", 486 | " [ 0, 8, 12, 0, 0, 0, 0, 0],\n", 487 | " [ 0, 0, 0, 5, 6, 4, 0, 0],\n", 488 | " [ 0, 0, 0, 9, 4, 4, 0, 0],\n", 489 | " [ 0, 0, 0, 2, 5, 6, 0, 0],\n", 490 | " [ 0, 0, 0, 0, 0, 0, 10, 10],\n", 491 | " [ 0, 0, 0, 0, 0, 0, 4, 5],\n", 492 | " [ 0, 2, 0, 0, 0, 0, 5, 11]\n", 493 | "]\n", 494 | "\n", 495 | "A = np.array(word_counts).T\n", 496 | "m, n = A.shape\n", 497 | "\n", 498 | "A" 499 | ] 500 | }, 501 | { 502 | "cell_type": "markdown", 503 | "metadata": {}, 504 | "source": [ 505 | "Next, we'll look at our singular values:" 506 | ] 507 | }, 508 | { 509 | "cell_type": "code", 510 | "execution_count": 19, 511 | "metadata": { 512 | "ExecuteTime": { 513 | "end_time": "2019-11-04T04:11:16.516425Z", 514 | "start_time": "2019-11-04T04:11:16.506065Z" 515 | } 516 | }, 517 | "outputs": [ 518 | { 519 | "name": "stdout", 520 | "output_type": "stream", 521 | "text": [ 522 | "U: (8, 8)\n", 523 | "S: (8,)\n", 524 | "Vt: (9, 9)\n" 525 | ] 526 | } 527 | ], 528 | "source": [ 529 | "# Normalize the columns to remove the effect of word counts from each document\n", 530 | "A_norm = A / A.sum(axis=0)\n", 531 | "\n", 532 | "# Perform the Singular Value Decomposition\n", 533 | "U, Σ, Vt = np.linalg.svd(A_norm)\n", 534 | "\n", 535 | "print('U:', U.shape)\n", 536 | "print('S:', Σ.shape)\n", 537 | "print('Vt:', Vt.shape)" 538 | ] 539 | }, 540 | { 541 | "cell_type": "code", 542 | "execution_count": 20, 543 | "metadata": { 544 | "ExecuteTime": { 545 | "end_time": "2019-11-04T04:11:19.234855Z", 546 | "start_time": "2019-11-04T04:11:18.911093Z" 547 | } 548 | }, 549 | "outputs": [ 550 | { 551 | "data": { 552 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAADgCAYAAACpWqpmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZLklEQVR4nO3de5CddZ3n8ffHgBoY3YQdQBPkJjthoHBI0TPljM4usLMGL0DW28CqoI5geXdrTWkGdRBw0YkrjriW4lq7luBlccJtF42OYLQso3bIAAbNCEMEgqztQESgRYjf/eM5rZ3DSffpkO7zdPf7VXXqpH/nuXzPYwc/+f1+z+9JVSFJkqTBe8KgC5AkSVLDYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJQxmkiRJLdFXMEtyUJKLk3wnyUNJKsmhfew3lOSSJD/q7HdHksuSHNZj262d43a/Vk79a0mSJM0+e/W53RHAy4GNwLeA5/W532nA0cBHgc3AUuA9wHCSY6vqzq7t1wHndrVt6fNckiRJs1q/weybVXUgQJLX0X8w+2BVjYxvSPJt4HbgLOC9Xdv/vKo29HlsSZKkOaWvocyq+s3uHLw7lHXafgKM0PSeSZIkqWPGJ/8n+UPgAOCHPT4+uTMX7eEkG5xfJkmS5pN+hzL3iCR7AZ+g6TH7dNfH1wDfpxnmPBB4M3BFkldV1aW7ON7ZwNkA++6773FHHnnkdJUuSZK0x2zcuPHnVbV/d3um+hDzzhyzTwGHVdXWKe77CeCvgBdW1Vcn2XYBsAF4WlU9Y7JjDw0N1fDw8FTKkSRJGogkG6tqqLt9xoYyk1xI07v12slCGUBV7QAuBw5K8vTprk+SJGnQZmQoM8k5wLuAt1bVZ6eya+d9at16kiRJs9C0B7MkbwUuAM6pqounsN9ewMuAO6rqnumqrx9XbtrGmnVbuHv7KEsWLWTVimWsXO5NpZIkac/qO5gleWnnj8d13p+fZAQYqar1SQ4BbgPOq6rzOvucBnwE+ApwXZJnjzvk/VV1S2e704FTgWuBO2km/7+pc67Td/fL7QlXbtrG6rU3M/rIDgC2bR9l9dqbAQxnkiRpj5pKj9nlXT9/vPO+HjieZthxATvPWzup035S5zXe2H7Q3Il5ALAG2A94iOYOzZOqat0Uatzj1qzb8ttQNmb0kR2sWbfFYCZJkvaovoNZVWWSz7fyuzlhY22vBl7dx7E3ACf2W8tMunv76JTaJUmSdteMLzA72yxZtHBK7ZIkSbvLYDaJVSuWsXDvBTu1Ldx7AatWLBtQRZIkaa6a0ZX/Z6OxeWTelSlJkqabwawPK5cvNYhJkqRp51CmJElSSxjMJEmSWsJgJkmS1BIGM0mSpJYwmEmSJLWEwUySJKklDGaSJEktYTCTJElqCYOZJElSSxjMJEmSWsJgJkmS1BIGM0mSpJYwmEmSJLWEwUySJKklDGaSJEktYTCTJElqCYOZJElSS/QVzJIclOTiJN9J8lCSSnJon/s+OcmaJD9NMto5xr/tsd0TkqxOsjXJr5LcmOQlU/s6kiRJs1e/PWZHAC8H7gO+NcVzfBo4C3gv8CLgp8C6JMd2bXc+cC7wMeD5wAbg8iQvmOL5JEmSZqW9+tzum1V1IECS1wHP62enJH8E/CfgtVX1Pztt64HNwHnAKZ22A4B3AB+oqg91dr8+yRHAB4Br+6xTA3Llpm2sWbeFu7ePsmTRQlatWMbK5UsHXVZreH0kSf3oq8esqn6zm8c/BXgE+OK4Yz0KfAFYkeRJneYVwBOBS7v2vxQ4Jslhu3l+zYArN21j9dqb2bZ9lAK2bR9l9dqbuXLTtkGX1gpeH0lSv6Z78v/RwO1V9VBX+2aaIHbEuO0eBm7tsR3AUdNWoR63Neu2MPrIjp3aRh/ZwZp1WwZUUbt4fSRJ/ZruYLYfzby0bveO+3zsfXtV1STb7STJ2UmGkwyPjIw87mK1e+7ePjql9vnG6yNJ6td0B7MA3WFrrH13tttJVV1SVUNVNbT//vvvZol6vJYsWjil9vnG6yNJ6td0B7N76d3btXjc52Pvi5N0B7Hu7dRCq1YsY+HeC3ZqW7j3AlatWDagitrF6yNJ6td0B7PNwGFJ9ulqPwr4Nb+bU7YZeBLwzB7bAdwybRXqcVu5fCkXvvgYli5aSIClixZy4YuP8a7DDq+PJKlfeey0rkl2aJbL+BRwWFVtnWTbY4FNwKur6jOdtr2Am4Fbq+rkTtsBwJ3Af62q943b/x+AA6vqmMnqGhoaquHh4Sl9F0mSpEFIsrGqhrrb+13HjCQv7fzxuM7785OMACNVtT7JIcBtwHlVdR5AVf1jki8CH0myN3A78AbgMOAVY8euqp8luQhYneSXwA3AXwInAqdO8btKkiTNSn0HM+Dyrp8/3nlfDxxPM1F/AY8dHn0N8H7gAmARcCNwUlXd0LXdOcADwNuApwFbgJdX1TVTqFGSJGnWmvJQZls5lClJkmaLXQ1lTvfkf0mSJPXJYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJQxmkiRJLWEwkyRJagmDmSRJUksYzCRJklrCYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJQxmkiRJLWEwkyRJagmDmSRJUksYzCRJklrCYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJfoKZkmekeRLSX6R5P4ka5Mc3Md+5yapXbx+1bXt1l1st3J3v5wkSdJsstdkGyTZB7gOeBg4EyjgAuD6JM+qqgcn2P1/AF/patu303Z1j+3XAed2tW2ZrEZJkqS5YNJgBpwFHA4sq6pbAZLcBPwYeD3w4V3tWFV3AXeNb0vyqs55P9Njl59X1Yb+SpckSZpb+hnKPAXYMBbKAKrqduDbwKm7cc4zgf9H0zsmSZKkjn6C2dHAD3q0bwaOmsrJkhwEnABcVlWP9tjk5CQPJXk4yQbnl0mSpPmkn2C2H3Bfj/Z7gcVTPN+rOufsNYx5DfAWYAXwCuBXwBVJXrmrgyU5O8lwkuGRkZEpliJJktQu/cwxg2bCf7fsxvnOADZV1U2POUHVW3Y6eHIFsAG4ELi0Z1FVlwCXAAwNDfWqUZIkadbop8fsPppes26L6d2T1lOSPwGOpHdv2WNU1Q7gcuCgJE/v9zySJEmzVT/BbDPNPLNuRwG3TOFcZwKPAp+bwj5jvXL2hkmSpDmvn2B2NfDsJIePNSQ5FHgOvdcie4wkTwROA66tqr4mgyXZC3gZcEdV3dPPPpIkSbNZP8HsU8BW4KokpyY5BbgKuBP45NhGSQ5J8miS9/Y4xotohkN7DmMmOT3JF5KckeSEJKcB1wPHAe+c0jeSJEmapSad/F9VDyY5EbgI+CzN8OLXgbdX1QPjNg2wgN5h70yauzj/zy5OcztwALCGJsA9BHwfOKmqXO9MkiTNC33dlVlVdwAvmWSbreziTs2qmnAh2s5q/yf2U4skSdJc1ddDzCVJkjT9DGaSJEktYTCTJElqCYOZJElSSxjMJEmSWsJgJkmS1BIGM0mSpJYwmEmSJLWEwUySJKklDGaSJEktYTCTJElqCYOZJElSSxjMJEmSWsJgJkmS1BIGM0mSpJYwmEmSJLWEwUySJKklDGaSJEktYTCTJElqCYOZJElSSxjMJEmSWqKvYJbkGUm+lOQXSe5PsjbJwX3uW7t4Hdu13ROSrE6yNcmvktyY5CW786UkSZJmo70m2yDJPsB1wMPAmUABFwDXJ3lWVT3Yx3n+F/DJrrZ/6vr5fOAdwDnARuA04PIkL6qqa/s4hyRJ0qw2aTADzgIOB5ZV1a0ASW4Cfgy8HvhwH8fYVlUbdvVhkgNoQtkHqupDnebrkxwBfAAwmElz2JWbtrFm3Rbu3j7KkkULWbViGSuXLx10WZI04/oZyjwF2DAWygCq6nbg28Cpe6iOFcATgUu72i8Fjkly2B46j6SWuXLTNlavvZlt20cpYNv2UVavvZkrN20bdGmSNOP6CWZHAz/o0b4ZOKrP87whycNJHkpyXZI/73GOh4Fbu9o3d977PY+kWWbNui2MPrJjp7bRR3awZt2WAVUkSYPTTzDbD7ivR/u9wOI+9r8UeCPwF8DZwL8GrktyfNc5tldV9TjH2OePkeTsJMNJhkdGRvooRVLb3L19dErtkjSX9btcRndgAkhfO1a9qqq+WFXfqqpLgecCd9PcQDD+WFM+R1VdUlVDVTW0//7791OOpJZZsmjhlNolaS7rJ5jdR+8eq8X07kmbUFX9Evi/wB+Pa74XWJykO4gtHve5pDlo1YplLNx7wU5tC/dewKoVywZUkSQNTj93ZW6mmQPW7Sjglt08b3cP2WbgScAz2Xme2djcst09j6SWG7v70rsyJam/YHY18KEkh1fVPwMkORR4DvCuqZ4wyVOBFwLfHdf8FeDXwCuA941rfyXwg85doJLmqJXLlxrEJIn+gtmngDcDVyV5N01P1/nAnYxbNDbJIcBtwHlVdV6n7R3AMuB6mnllh9CsV/Y0mhAGQFX9LMlFwOokvwRuAP4SOJE9tySHJElSq00azKrqwSQnAhcBn6UZhvw68PaqemDcpgEWsPO8tS3Af+y8/hVwP836Z39VVd/rOtU5wAPA22iC2xbg5VV1zW58L0mSpFknj12hYnYaGhqq4eHhQZchSZI0qSQbq2qou73f5TIkSZI0zQxmkiRJLWEwkyRJagmDmSRJUksYzCRJklrCYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJQxmkiRJLWEwkyRJagmDmSRJUkvsNegCJEmTu3LTNtas28Ld20dZsmghq1YsY+XypYMuS9IeZjCTpJa7ctM2Vq+9mdFHdgCwbfsoq9feDGA4k+YYhzIlqeXWrNvy21A2ZvSRHaxZt2VAFUmaLgYzSWq5u7ePTqld0uxlMJOklluyaOGU2iXNXgYzSWq5VSuWsXDvBTu1Ldx7AatWLBtQRZKmi5P/Janlxib4e1emNPcZzCRpFli5fKlBTJoH+hrKTPKMJF9K8osk9ydZm+TgPvYbSnJJkh8leSjJHUkuS3JYj223Jqker5W788UkSZJmm0l7zJLsA1wHPAycCRRwAXB9kmdV1YMT7H4acDTwUWAzsBR4DzCc5NiqurNr+3XAuV1t3g8uSZLmhX6GMs8CDgeWVdWtAEluAn4MvB748AT7frCqRsY3JPk2cHvnuO/t2v7nVbWhz9olSZLmlH6C2SnAhrFQBlBVt3cC1qlMEMy6Q1mn7SdJRmh6zyRJetx8ZJXmin7mmB0N/KBH+2bgqKmeMMkfAgcAP+zx8cmduWgPJ9ng/DJJ0mTGHlm1bfsoxe8eWXXlpm2DLk2asn6C2X7AfT3a7wUWT+VkSfYCPgGMAJ/u+vga4C3ACuAVwK+AK5K8coLjnZ1kOMnwyMhjOuckSfOAj6zSXNLvchnVoy27cb6PAX8GvLCqdgp7VfWWnQ6eXAFsAC4ELu1ZVNUlwCUAQ0NDvWqUJM1xPrJKc0k/PWb30fSadVtM7560npJcCJwNvLaqvjrZ9lW1A7gcOCjJ0/s9jyRpfvGRVZpL+glmm2nmmXU7Criln5MkOQd4F/C2qvps/+X9tlfO3jBJUk8+skpzST/B7Grg2UkOH2tIcijwnM5nE0ryVpp1z86pqov7LawzH+1lwB1VdU+/+0mS5peVy5dy4YuPYemihQRYumghF774GO/K1KyUqok7o5LsC9wIjALvpum9Oh94CvCsqnqgs90hwG3AeVV1XqftNOBzNAvHvq/r0PdX1S2d7U6nWXrjWuBO4EDgTcBzgdOr6guTfZGhoaEaHh7u4ytLkiQNVpKNVTXU3T7p5P+qejDJicBFwGdphhe/Drx9LJSNnQNYwM69cCd12k/qvMZbDxzf+fPtNEtorKGZz/YQ8H3gpKpaN1mNkiRJc8GkPWazhT1mkiRptthVj1lfDzGXJEnS9Ot3HTNJkjSL+diq2cFgJknSHDf22KqxJySMPbYKMJy1jEOZkiTNcT62avYwmEmSNMf52KrZw2AmSdIc52OrZg+DmSRJc5yPrZo9nPwvSdIcNzbB37sy289gJknSPLBy+VKD2CzgUKYkSVJL2GMmSZLmvbYswGswkyRJ81qbFuB1KFOSJM1rbVqA12AmSZLmtTYtwGswkyRJ81qbFuA1mEmSpHmtTQvwOvlfkiTNa21agNdgJkmS5r22LMDrUKYkSVJLGMwkSZJaIlU16Br2iCQjwE+m+TS/D/x8ms8x23mNJub1mZjXZ3Jeo4l5fSbnNZrYTF2fQ6pq/+7GORPMZkKS4aoaGnQdbeY1mpjXZ2Jen8l5jSbm9Zmc12hig74+DmVKkiS1hMFMkiSpJQxmU3PJoAuYBbxGE/P6TMzrMzmv0cS8PpPzGk1soNfHOWaSJEktYY+ZJElSSxjMJEmSWsJgNokkz0jypSS/SHJ/krVJDh50XW2R5KAkFyf5TpKHklSSQwddV1skeWmSv0/ykySjSbYkuTDJUwZdW1skWZHkuiT3JHk4yV1J/neSowZdW1sl+Urn79oFg66lDZIc37ke3a/tg66tTZK8IMk3kzzQ+f+z4SQnDrquNkjyjV38DlWSr8xkLT4rcwJJ9gGuAx4GzgQKuAC4PsmzqurBQdbXEkcALwc2At8CnjfYclrnHcAdwF8DdwHLgXOBE5L8WVX9ZoC1tcV+NL8/HwdGgIOBdwEbkhxTVdO9cPSskuR04I8GXUdLvRX4/rifHx1UIW2T5PXAxzqv82k6Zo4F9hlkXS3yRuCpXW1/CnwYuHomCzGYTews4HBgWVXdCpDkJuDHwOtp/geb775ZVQcCJHkdBrNuJ1fVyLif1ye5F/gMcDxN8J/XqurzwOfHtyX5HvAj4KXAfxtEXW2UZBFwEfCfgc8NuJw2+mFVbRh0EW3TGcX4CLCqqj4y7qN1Aymoharqlu62JGcBvwa+MJO1OJQ5sVOADWOhDKCqbge+DZw6sKpaxB6fiXWFsjFj/6JfOpO1zDL/0nl/ZKBVtM/fAps7YVbq12uB3wCfGHQhs0WShcDLgGuq6t6ZPLfBbGJHAz/o0b4ZcP6Ldte/67z/cKBVtEySBUmemOTfAJ8E7mGG/6XaZkmeC5xBM+Si3i5LsiPJvyT5nPOBf+u5ND3QpyW5LcmjSW5N8qZBF9ZiLwaeQjO6MaMcypzYfsB9PdrvBRbPcC2aA5IsBc4D/qGqhgddT8t8Fziu8+dbgROr6mcDrKc1kuxNE1Y/VFVbBl1PC/2CZsh7PXA/zVzOvwa+k2S5v0cs6bzW0FyX22h6gz6WZK+q+rtBFtdSZwA/A7480yc2mE2u1wq8mfEqNOsl+T3gKpoJya8ZcDlt9CqaybeH09w08bUkz62qrQOtqh3eCSwE3j/oQtqoqjYBm8Y1rU/yTeB7NDcEvHsghbXHE2h6f15dVWs7bdd15p6tTvLRcrX530qyBPgL4O+qasZvIHEoc2L30fSadVtM7540qackT6a5s+dwYEVV3TXgklqnqn5YVd/tzJ/698Dv0dydOa91huPOAd4DPCnJos5NAIz7ecHgKmynqroB+CfgjwddSwuMzdn8Wlf7V4EDgafPbDmt90qafDTjw5hgMJvMZpp5Zt2OAh5zB4fUS2cY6u+BPwFeUFU3D7ik1quq7TTDmUcMupYWOBx4MnApzT8Ix17Q9CzeBxwzmNJaL/Qe9ZhvNu+ifWz0x5u4dnYGcGNV3TiIkxvMJnY18Owkh481dLp+n8MMr2ui2SnJE4DLaHqATvVW/v4kORA4kmYuzHz3j8AJPV7QhLUTaEKsxkkyBPwBzdzF+e6KzvuKrvYVwF1Vdc8M19Nand+boxlQbxk4x2wynwLeDFyV5N00//I6H7iTZiKuaFa37/xxbOL285OMACNVtX5AZbXFf6eZZPt+4MEkzx732V0OaUKSK4AbgJtoJm7/Ac06XY/iGmZjvYff6G5PAvCTqnrMZ/NNksuA22l+j7bTTP5fDWwDLh5gaW1xLXA98Mkkvw/8M80agc/D+a7dzqD5b8/A1gmM8/0m1pnfcRHwH2i6fb8OvN0Jyb+TZFe/ROur6viZrKVtkmwFDtnFx++rqnNnrpp2SvJOmqdHPBN4Is0/fL4BXOjfs13r/L17f1XN94ntJFkNnE7zd20fmqVWvgz8TVX9dJC1tUWSpwIX0gSyxTTLZ3ygqlyouKMz7eRumvVLTx5YHQYzSZKkdnCOmSRJUksYzCRJklrCYCZJktQSBjNJkqSWMJhJkiS1hMFMkiSpJQxmkiRJLWEwkyRJagmDmSR1JNk3yQeT3Jrk10mq6/VfBl2jpLnNZ2VKEpDm4ZNrgefQPNt0GPhT4G+ArcDnaZ45KEnTxkcySRKQ5I3Ax4AVVfW1ce1rgT8HDij/gylpmjmUKUmN1wBfGx/KOn4ELB4LZUkWJ7l+xquTNC8YzCTNe0kOBIaAL/f4+OnAPWM/VNV9VXXCTNUmaX4xmEkSHNJ5/+n4xiQLgOcDV41rOy/Je2ewNknziMFMkmB75/3IrvZ3AouBT45rOw7YOBNFSZp/vCtTkuDHwCZgVZIR4DbgFOANwFuq6qZx2x4H3DDzJUqaD7wrU5KAJAfT3JV5Is1owkbgb6vqmnHbHAR8r6qWDKZKSXOdPWaSBFTVHTS9ZBNxGFPStHKOmST1z2FMSdPKoUxJkqSWsMdMkiSpJQxmkiRJLWEwkyRJagmDmSRJUksYzCRJklrCYCZJktQSBjNJkqSWMJhJkiS1xP8H6YZLp06mBXkAAAAASUVORK5CYII=\n", 553 | "text/plain": [ 554 | "
" 555 | ] 556 | }, 557 | "metadata": { 558 | "needs_background": "light" 559 | }, 560 | "output_type": "display_data" 561 | } 562 | ], 563 | "source": [ 564 | "plt.plot(Σ, marker='o', lw=0)\n", 565 | "plt.xlabel('$\\sigma_i$')\n", 566 | "plt.show()" 567 | ] 568 | }, 569 | { 570 | "cell_type": "markdown", 571 | "metadata": {}, 572 | "source": [ 573 | "Notice how the first 3 singular values are more significant than the later ones. Using the language of Principal Component Analysis, we would say the first three vectors in $U$ and $V$ \"explain most of the variance\" of the word count data. In more pure linear algebra speak, we would say $A$ is approximately rank 3.\n", 574 | "\n", 575 | "It's enlightening to look at the first 3 vectors in $U$ and $V$:" 576 | ] 577 | }, 578 | { 579 | "cell_type": "code", 580 | "execution_count": 21, 581 | "metadata": { 582 | "ExecuteTime": { 583 | "end_time": "2019-11-04T04:11:23.368230Z", 584 | "start_time": "2019-11-04T04:11:22.893747Z" 585 | } 586 | }, 587 | "outputs": [ 588 | { 589 | "data": { 590 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFrCAYAAACnj/O7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd7gkZZmw8fuBYcgZFRRhPiUYQEVBMIAiQVF3EZEV17AjoCtiwAyYENe4GFHXBMyKAXUVcwIWMyIgugRRR0RFSYJkHcI83x/P205NcwbmDGemTve5f9dV10xXVfd56+3qqqfeGJmJJEmSZraV+k6AJEmS+mdQKEmSJINCSZIkGRRKkiQJg0JJkiRhUChJkiRgVt8JGHUbbbRRzpkzp+9kSJIk3amzzz77L5l5t4m2GRTeRXPmzOGss87qOxmSJEl3KiJ+v6RtVh9LkiTJoFCSJElWH0uSpBF14Lwz+07ClDp27g69/n1LCiVJkmRQKEmSJINCSZIkYVAoSZIkDAolSZKEQaEkSZIwKJQkSRIGhZIkScKgUJIkSRgUSpIkCYNCSZIkYVAoSZIkDAolSZKEQaEkSZIwKJQkSRIGhZIkScKgUJIkSRgUSpIkCYNCSZIkYVAoSZIkDAolSZKEQaEkSZKAWX0nQJIkTd6B887sOwlT6ti5O/SdhBnPkkJJkiQZFEqSJMmgUJIkSSxFUBgRcyMil7DsPtUJiogDImLuVH/unfzNx0TE6RHxt4i4NCKOjojVVmQaJEmS+jSZjib7AZcMrbtgCtMycABwKzBvOXz27UTEdsC3ga8DrwPuC7wT2AR45opIgyRJUt8mExT+PDPnL7eULEcRsWpmLljC5qOAi4GnZ+atwKkRcStwbES8IzP/b0WlU5IkqS9T0qYwIlaPiPdFxPkRcWOrgv1KRGw9wb73jYhPRcTlEbEgIi6KiHe3bT8EHgU8plNFfUrnvTtFxKkRcUNbTo6I7Yc+/5MRcXFEPGpQJQy8dQnpXg3YE/hsCwgHTgRuAfa+q3kjSZI0CiZTUrhyRHT3z8y8rf1/9bYcBVwGbAgcApweEffLzCugAkLgp8B1VFXtb4HNgEHbxOcDnwFuA17Y1l3b3rsd8F3gXGBu23Y48P2IeHhmntdJ2wbAp6lq4MOBm5ZwTFsAs4Hue8nMmyLiYuABd5QhkiRJ42IyQeGFQ69/BDwaIDOvpgI6ACJiZaqd3pXA04Fj2qY3A6sAj8jMyzqfNa99zgURcT1wa2b+ZOjvvZEK7nbLzOva3zmFqvp9A/AvnX3XBp6RmV+/k2PaoP371wm2Xd3ZLkmSNNYmExTuw+IdTa7vboyI/YGXA1sD63Q2dauQ9wS+MhQQLq1d2nuvG6zIzGsi4mvAHkP7LgC+sRSfGYOPuoNtkiRJY28yQeF5S+poEhH7UNW+xwNHAn8BFlKlhd2hXTbg9j2Yl9Z6wKUTrL+M25foXZ6ZEwV6w67upGvY+sBFS588SZKk0TVVcx/vD1yYmQcMVrROHOsN7XcVcK9l/BvXABtPsH7j9rldSxMQAvwGuBl4IPD5wcqIWAOYA5ww6VRKkiSNoKma0WQNamzBrudM8PnfAfaOiLvfwWctoDqtDPse8OSIWHOwIiLWBZ7Utk1aZv4dOBl4+lAnmn+h2j5+dVk+V5IkadRMVVD4LWCbNhPIbhFxGPB6qpdx1+upoV5Oj4iDIuKxEfHsiPhEZ58LgAdFxH4RsX1EbNXWH0V1IDklIp4aEfsCpwCrUh1YltUbgfsAn4mIx0XEQcB7gBMz8+d34XMlSZJGxlRVH3+YqhaeSw0lcwbwZGqWkH/IzIsiYkfgLcA7gDWBPwEndXZ7G7AlcBywFnAqsHtmnhMRu7b3nkBVEf8E2GVoOJpJycyzI+IJ7e9+g6qmPh547bJ+piRJ0qi506AwM+dxJ1POtfEKj2hL16YT7DufGqZmSZ/1Z+AJS9h2OvC4O0nLs+5o+xLecxqw02TfJ0mSNC6mqvpYkiRJI8ygUJIkSQaFkiRJMiiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEkCZvWdAGlpHTjvzL6TMKWOnbtD30mQJOkfLCmUJEmSQaEkSZIMCiVJkoRBoSRJkrCjiTRS7GwjSVpeDApHwLgFAmAwIEnSdGP1sSRJkgwKJUmSZFAoSZIkIDKz7zSMtI022ijnzJnTdzIkSZLu1Nlnn52ZOWGhoB1N7qI5c+Zw1lln9Z0MSZKkOxURP1vSNquPJUmSZFAoSZIkq4+lkTJuY1Y6XqUkTR+WFEqSJMmgUJIkSQaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJknBGE40QZ/OQJGn5saRQkiRJBoWSJEkyKJQkSRK2KRwJ49aWDmxPJ0nSdGNJoSRJkgwKJUmSZFAoSZIkDAolSZKEQaEkSZIwKJQkSRIGhZIkScKgUJIkSRgUSpIkCYNCSZIkYVAoSZIkliIojIi5EZFLWHaf6gRFxAERMXeqP/cO/t4uETEvIs6LiNsiYv6K+tuSJEnTxaxJ7LsfcMnQugumMC0DBwC3AvOWw2dPZHfg0cBZQACrrqC/K0mSNG1MJij8eWaOZClaRKyamQuWsPnIzHxD2+9EYPsVlzJJkqTpYUraFEbE6hHxvog4PyJujIhLI+IrEbH1BPveNyI+FRGXR8SCiLgoIt7dtv0QeBTwmE4V9Smd9+4UEadGxA1tOTkith/6/E9GxMUR8aiIOD0i/ga8dUlpz8yFU5EHkiRJo2wyJYUrR0R3/8zM29r/V2/LUcBlwIbAIcDpEXG/zLwCKiAEfgpcB7wO+C2wGVWFC/B84DPAbcAL27pr23u3A74LnAvMbdsOB74fEQ/PzPM6adsA+DTwzrbPTZM4TkmSpBlnMkHhhUOvf0S1xSMzr6YCOgAiYmXg28CVwNOBY9qmNwOrAI/IzMs6nzWvfc4FEXE9cGtm/mTo772RCu52y8zr2t85BbgYeAPwL5191waekZlfn8TxSZIkzViTCQr3YfGOJtd3N0bE/sDLga2BdTqbulXIewJfGQoIl9Yu7b3XDVZk5jUR8TVgj6F9FwDfWIa/IUmSNCNNJig8b0kdTSJiH6ra93jgSOAvwEKqtHC1zq4bcPsezEtrPeDSCdZf1j636/LMzGX8O5IkSTPOZILCO7I/cGFmHjBYERGrUYFc11XAvZbxb1wDbDzB+o3b53YZEEqSJE3CVM1osgY1tmDXcyb4/O8Ae0fE3e/gsxZQnVaGfQ94ckSsOVgREesCT2rbJEmStIymqqTwW8AHIuJo4JvADlTv4+uG9ns98ASqV/LbgPnAvYE9MvM5bZ8LgIMiYj/gd8B1mflrqmfz6cApEfGf1EDTh1GDTb95WRPeAtRd2stNgTUj4mnt9XmZOdzBRpIkaexMVVD4YapaeC41lMwZwJOBxXr/ZuZFEbEj8BbgHcCawJ+Akzq7vQ3YEjgOWAs4Fdg9M8+JiF3be0+gqoh/AuwyNBzNZD0I+PzQusHr1wP/cRc+W5IkaSTcaVCYmfO4kynn2niFR7Sla9MJ9p1PDVOzpM/6M1WaONG204HH3UlannVH2yfY/xSq1FGSJGnGmqo2hZIkSRphBoWSJEkyKJQkSZJBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEjCr7wRIkiQtiwPnndl3EqbUsXN36PXvGxRKkjSCDIg01aw+liRJkkGhJEmSDAolSZKEQaEkSZIwKJQkSRIGhZIkScKgUJIkSRgUSpIkCYjM7DsNI22jjTbKOXPm9J0MSZKkO3X22WdnZk5YKOiMJnfRnDlzOOuss/pOhiRJ0p2KiJ8taZvVx5IkSTIolCRJkkGhJEmSsE2hpBFz4Lwz+07ClDp27g6Tfo95IGl5sKRQkiRJBoWSJEkyKJQkSRIGhZIkScKOJpKkEWRnG2nqWVIoSZIkg0JJkiQZFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJGFQKEmSJAwKJUmShEGhJEmSMCiUJEkSBoWSJEnCoFCSJEkYFEqSJAmDQkmSJLEUQWFEzI2IXMKy+1QnKCIOiIi5U/25S/hbsyLi1RFxWkRcHhHXR8TZEfHciIgVkQZJkqTpYNYk9t0PuGRo3QVTmJaBA4BbgXnL4bOHrQUcDnwCeA9wE/Ak4Dhgq7ZNkiRp7E0mKPx5Zs5fbilZjiJi1cxcMMGmG4D7ZOZfO+tOiYgNgUMj4sglvE+SJGmsTEmbwohYPSLeFxHnR8SNEXFpRHwlIraeYN/7RsSnWnXtgoi4KCLe3bb9EHgU8JhOFfUpnffuFBGnRsQNbTk5IrYf+vxPRsTFEfGoiDg9Iv4GvHWidGfmrUMB4cCZwGrABsueK5IkSaNjMiWFK0dEd//MzNva/1dvy1HAZcCGwCHA6RFxv8y8AiogBH4KXAe8DvgtsBkwaJv4fOAzwG3AC9u6a9t7twO+C5wLzG3bDge+HxEPz8zzOmnbAPg08M62z02TOE6AxwBXA1dM8n2SJEkjaTJB4YVDr38EPBogM6+mAjoAImJl4NvAlcDTgWPapjcDqwCPyMzLOp81r33OBRFxPXBrZv5k6O+9kQrudsvM69rfOQW4GHgD8C+dfdcGnpGZX5/E8Q3S/kRgX+CwTtArSZI01iYTFO7D4h1Nru9ujIj9gZcDWwPrdDZ1q5D3BL4yFBAurV3ae68brMjMayLia8AeQ/suAL4x2T8QEdsCnwJOBo5ehjRKkiSNpMkEhectqaNJROxDVfseDxwJ/AVYSJUWrtbZdQNu34N5aa0HXDrB+su4fdu/yzMzJ/PhEbEF8B3gN8BTLSWUJEkzyWSCwjuyP3BhZh4wWBERq1GBXNdVwL2W8W9cA2w8wfqN2+d2TTYg3Aw4lWpHuFdm3rBMKZQkSRpRUzWjyRrU2IJdz5ng878D7B0Rd7+Dz1pAdVoZ9j3gyRGx5mBFRKxLjSv4vUmneNFn3AM4BbgF2CMzhwNMSZKksTdVQeG3gG0i4uiI2C0iDgNeT/Uy7no9FXydHhEHRcRjI+LZEfGJzj4XAA+KiP0iYvuI2KqtP4rqQHJKRDw1IvalgrlVqQ4skxYRa1BV3PemekNv1oa9GSxrL8vnSpIkjZqpqj7+MFUtPJcaSuYM4MnAYr1/M/OiiNgReAvwDmBN4E/ASZ3d3gZsSc0qshZVrbt7Zp4TEbu2955AVRH/BNhlaDiaybgn8OD2/89MsH1n4IfL+NmSJEkj406Dwsycx51MOdc6ZRzRlq5NJ9h3PjVMzZI+68/AE5aw7XTgcXeSlmfd0fYJ0uIcx5IkacabqupjSZIkjTCDQkmSJBkUSpIkyaBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJGBRKkiQJg0JJkiRhUChJkiQgMrPvNIy0iLgS+H3f6ZhCGwF/6TsRPTMPzAMwD8A8APMAzAMYrzzYPDPvNtEGg0ItJiLOyszt+05Hn8wD8wDMAzAPwDwA8wBmTh5YfSxJkiSDQkmSJBkU6vY+2ncCpgHzwDwA8wDMAzAPwDyAGZIHtimUJEmSJYWSJEkyKJQkSRIGhZIkScKgUJKkXkTEyn2nQeoyKNQyiYiVhl5HX2lR/zwfBH7vSyMiVoqItQEy87a27gGjmneDwDYiZkXEqn2nR3eNQaEmLSJWzsyF7f+PAsgZ1I29GwB1L+SjelG/q4bOh10iYv2ZdD5MxjifIxGx0uB7j4i1+k7PNPYM4DURsRlARHwbeDWwRq+pWkaZeVtErAZ8C9gGbv+QOBNFxP36TsOymPFfnCanBQCDp9uPAB+KiFf2nKwVJiJmZebC9lS8AXC/tj4yM8f5pj+RofPhY8A7gBdaLVY6pSiD8yLa67G79nYeDI4Dntj+P6N+D0vpJuAI4M0RcTKwLXBMZt7Yb7LuklWBRwHPgUXnwkwVEe8CjomIdfpOy2SN3YVJy1cnAPgssAdwFPCFXhO1grQA6NZW9fNF4Gzg/Ij4CXBoRKw60wLDofNhN+Bo4L8H69u2GZMfXYOAOSLWBI6OiBOoQOD/tQeLsbv+tt/GPwObD1b1mJxpKTNPAv61LTsCL8rMs2F6/1YGaVtCGm8E/hvYdVACOsP9groe7grT+3sdNnYXJS1/EfEq6mK2P/DlzPxdRKwZEXNa6dlY6tzgzwDWBT4CHADMpp783z0IDHtM5goXES8FHknd5L6cmZdExFoRsXlE3HOm5cdA53w5C3g88DDgIOD7EbHNOASGQ80nZgELgL8A9wZLjO7AJsDKwJrAUyNiS6hmONM4gBhc22/XfCYzbwXmUdXHu63wlE0/JwCfBt4fEVuM0jVwpC9I6s0WwNmZ+dNWcrY9cDJwGvCLiHgajNbT0SQcCqwCvDAz356Z84BdgO8A+wHPH9PjpgX+h0XERp11KwFbAT/PzJ8ACyNiB+pc+B7wm4h4Rtt3LPNlWAuOBg4A/gA8hbphvpAKmn4UEduOamAYZeXuzS4zb83Mm4FzgS3bfrMG+/eT0ulhgu/4o8Ac4HlUG8MjI2JrWNQ+ezrlWUTsAlwSEQ8btCFsJd7dYOcsqgblBRFxj35S2p+IeHtEPA/+8R0eC/wdeG5ErN5r4iZh5C5GmhbWAR4cEftFxLuBHwBXA+8BfgO8LSJWH6Wno0m4P3B9Zp4PEBGzM/MG4AXAn4D9x/S4oQKbtwKvH5QIt5KgW4BdIuJfgfcBP6Ty4l3AN6m2NRuMcb4spj0orRkRbwE2Bf43M+dn5sLM/AJwGPA74IejVmIYEatExBpZBk0HPhwRX42IN0XEnlTJ+RYRsXYrQZpRHdGGxeIdsdZq+XJjZv4hM48Dnk/VurwhIrZo+80GDo7Wka9PLVj9H+DjmXl2C3A+AVzQvvPd4R+lhV8DHkQFvGPZdnYiEfFkqrPQRyJiXkTskpmnAV+ivt/N237Tvq21cx9riaLTiWBo/RzgJOC+VKnAiZl5TNt2KPX0u3NmXr3iUrt8taf2AD4GPA7YJTP/2LbNzsybI+IgqgRga2D+uN0IW1Xo84D/pPLhjZl5ZURsBRxDVSGfBfxPZn6wvedQ4N+BR2bmX/tJ+YoXEU+hSk0AXp2ZR8finXIeD7yNunnulpnn9JPSpRcRa1Df7+sy84tt3RbAe6mes5vQqo3b63OB+cCFwEXUg8IFmfmHFZz03kT1yB4EhEcDjwUS+GZmvqGz34FUc5STgK8Aj6Z+a1tk5kUrOt1d7bp2BPAI4HrgLcDlVI3JwcAs4Ezgw8B3qV7IV2XmP/eR3j60ktH3Uc2KNgauAc4D/gP4OnB5Zj6p7RvT+d4w68530Uw0dAPbH7g7dUM4NzMvBrZrwcB1mXlZ22996sJxEVVsPrK6F3P4R0lHRsQPgecC+0bEx9oT/81tt02oktIrpvOPflm08+HGiPgQdWP4KHBdRLwzM38NPD4i7gPckJlXtPesD2wP/J4qTZxJTgMOpG4K+0bEvMz8y+C8ysxvR0QCx1M9tvfsM7FLIzNviohTgFMAImK1zJwPPLm9Xh3YjGpv/HYqcJhN9Uhdj7omPKyHpPei3fwHAeHxwO5UM5NNgde1doRzM3NBZh4bETdTnTUeS/VQflifAWFrJvJXqhZoDvU9vowKAN+UmddExInAdlQp2YeA26i0bxARD8zM86d7EHRXRMSGmXlVZl4eET+lOl++nPqODwb+DzgH2DkiXpCZH572eZGZLi5LXIATgRvaspB6mt12gv22o9pQXAU8oO9038VjntX+nU0NOfOgoe3zgL9RF8h7t3UPojqgfI5WAj8uC7By5/8vo6qQb2znw7uBDSd4z47Ax6kbygP7PoblnD8rLWH9elRgeC1V+rP28P7Aw7v5O12X4TRSpUKHAWt11g1qnh7Yzo0d2+tVgLWAe/R9HH2cE1T7ym8Du7fX6wIvar+hzwOrdva9P9Vj9Z49p/+xwK+owBTgOOrB7kLgvhMdL9WZ6j3tN78QOLLv72E559HH2+/6KZ1136dKgaFqlt5ONRVZ2PZdve903+lx9Z0Al+m7UENLnN0uUltST0ALW+CzbWe/Q4GfAxcMB1CjtnRubGsDpwNXtGM+EXh027Z+CwwXth/8z4HfAj8DVmn7TBgojPJCDT306xYUHka1F7yNqjresLPfIcBPqeqTkT4fliJPug8QW7Tfylqd9etRnU2uA74KrDPR+TEcdE23pXM8g9/HOdSD4gtpgWELDFZqx38pcPAoHNtyzrePUqXB3wLW7axfh2pWMQgMV+s7rZ203Q/4M/Dh9no1qt34VS0wfDGLB7Kzht7/YKp5yS+Bzfs+nuWYT69s+XItFQxv2O6TvwBe29lvN6p99f36TvNSHVffCXCZPsvwxRv4N6r0r/vU++8tGPr8IDCket8eCszp+xju4vGv1P6dRbWNOZlq1/OidlM/C9hjKH/+q+XRKzs3zlkrOu0rIG+eQZUA7D44T6i2M68Bbm0Xxbu19bu382TzvtO9nPNkkA9rt3Nlfvtt/B/wElqpAIsCw2uohufr9pXmZTzO6Pz/zcBD2/9PpZoSvIhOiWHbdhZwfN9p7znf1gI+086J3wxfHzuB4TVUSeKqfaRzgnQ/uwWFG1NB/ulU9fADqCD3lnZ+rzb0vu558s/turBz38eznPNqG+BVnfvDG6m2wp8AHtzZb2TuCb0nwGV6LCxeRfi8FuT9F3BUW7daZ/sgMDwR2KatG+mSMRaVgMymRub/7NCPerv2w/858PjO+uGn5LEoFRn+PttN4Fpg66H1G1BVSwup9nPrjcP5MInzZU2qQ8UpVAekjdpNfj7w+k5guC7VFnUh8M6+078s5wHVdORS4FGddd/tBIZrdtZ/iypJXqXvY+g5/zahBnRfCBzezaO2fe12rf0TcK++09vS9Mx2rXsXVQPyQ2Djtu1uncDwxcOBYecztgT+CDyn7+OZ4rx5NLA3sO/QPXNL4BvUA+FNVKnqy/tO7zIdY98JcJleCzX0wDUtAFhINRYfXBBmd/Y7qG2f110/ygvV9um09sM+p7N+dvv3Qe1i+bNuYDhuy9DFbpf27/OpdpRbDfKqs8/eVDXywnYjmSkB4SyqmuybwN3buk+1wOkMqnr1iE5guD7VKWMkSg1YvORnS+ohcZ8JtnUDw0G7yWfTHhhnwsLta1m6wXQ3kHoJsMbQvmvTHqamy0L1gr4JuJihak+q0+ESA0NqUO5jgZsZo9oCakDq31EloLcA57ff8+CcXx94OlUbsBC4kuqFP1JtzHtPgEvPJ8DiAcBTqSLwxwH3BF5Htak7E9ik7dMNDOcyZp0IqOFWbqDa+mzX1gWLqoa3papRLwEe3nd6l8Pxd2/2X6V6S27ezof5wI9YVM0+CI72pYZfORC4f9/HsJzyZUfgrUPr1qYamx/YXv831dN6DjUX7HzgMqrEcLiEaCQCw5bWo6kSo7/QSs+7v4n2+rvtd/Gq4aBn3Jeha+ihVBvCE1sAuGpbvx71ADFhYDhdFhY1ifgdVTBwNVUdus7QfnenOhv9jXrwmT20fS6tmcE4LO13/ocW9D2cGlfyjHZ/HC41DKo50Ug+FPWeAJfpsVAlQYdRnQYGF4aVgZe2G91PWVRiOC3avkzx8XdvcK+knga/NAhyhgLDh7VgaSyqijvH3b2wPZjqZLRzOw+CGmLhL1RV6fot8Nmk3QQ/xzRqLD+FeRJUCfKJwH9MsH0TqiH+Xu2m8fhBPrCol/b1wHP7PpbJHPPQ6zewaJipndu6f7S/7ex3DvWwtH7fx9BHXlG1LH9k0exOC6i2go9s2zeiSthupNriTpueqMPXMqoK+Z7tmK6ietGuN7TP3almNt8bPmfGaaHaUv6Wqh3rft8bUx1N5tPaCY/DPaH3BLj0v1Dt5Ra25SND26IFhn8AfkzPQyVM4TEPXwRXGXr9Oqoa/XO0IXYGAcIdfc44LFTJ1vFUtWh3yJG1qYD5z9QT8jlUe7rrGNGn4knkyaCKaHXgJZ31g+DotdTD0+qdbe+lBrR953Q/T6jAf2dgy866/2BRaflL2vd8bvf30P7tBoab930sPeXfES1w3oVFpYMvaNfU53XOk7tRc+JewTQLnqmqzo8xNMQUNerAkgLD9RmqORi3BdipfY9Pa6+7D8+PoYL/V/WdzqlaZsQUNFrcBFPtXEi1C/s98NiIeNBgQ9aZ/36qWvVBwCciYuXpNC/nZEXErKz5O9eMiNdGxKeAj0XEYwb7ZOZ/UO3j9gTeGBH3z7LYIMw5wYwvoywidqICgCcCf83MG9o8t7My83rqXNiL6l33a6racIfMPK+vNC9vbfDd69vLNwHvioi3wj+m+YNqe7shNSQNEfEAYN42GMUAABnsSURBVAfg9Mx8dTvfpvMUV/emBtF+R0RsFhFfoYabWQCQme8HjqQ6zBwz+D20vLl1cGyZ+ft+kr9i3MF3+Aiq1OiszFzQBqY+iiphPmFwnmTmldTv68E5/Wb4eRo1osLnBtNYAmTmvlTJ5/OA10TEup1tf802TWO7V4yja6lS8i3b64Wd8+AX1MPSBhO9cST1HZW69LcA/8qiKtHVqWEErqKqPLYY2jeoJ98tV3Q6p/iYB0+1a1Pj6P2EalT9daqzxP5D+7+eRVWmc/pO/wo8L35LPR0/qbN+Wpd2Lc/zpfN6U+pG/wfg7Z3121FDd1zZ/v0tVf0+Sm0Hn9Z+A7+nesMOSgm7JSOvbNv/l9YBgTEtIZogf9Zo14sdO+tmUUPP/G5wPlDVjVe382TNtu4tTPOeuO1YXtG+3+8CGwxt/zz18PMhhoYgGuel5csJVJOiJw5t25Ian/dF7fXI/xYsKZyhIuJ5VGnPu1op0N+oYPAAqoTjg4PJ2aFKDLOm6PlNPymeGllPtatRnSgup3pT7tM2B/CpiDigs/+bqSFXbqMCgbGxpFKPzPw0i6rD3hYRu7X1t7VSw+h8xsiWGN+Z9rtYGBGrRsROEbFRZl5CdSY4A3hmRLwDIGvu4tdQnU2upM6vHbNTijbdZeb/UN/5vamBh//e1t8WEbPa/4+m2h1vTv1Wts52N5wBdqSaTFwwWJGZt2bmDdSIBE+IiIdQbexOBQ7KmhryPlQHtS0iYpUe0n07g++z83p2Zt5KNXf4AHAf4ItDJYb7UT1u51DtIsdSRDwtIl4YEa9o391t1Dis5wCfj4gDImJORDyMGmZofeBr8I+atdHWd1Tq0s9CVXWdQD0Vvo9FJYarUlXJV1OlZ1v3mc4pOM6NGeoYA+xHtZcbjLE4aCC+N9WL9jbgGUPvGbSfGovhVli89Gcvqmro2VRV8GD9s6mmBecAuw3nxTgvnd/DWlTV2Rfb+TFYvzFVcvJH4D877xtuczpKJYWrUj2Nj6R6yX6ezugCLN528LVUSfvmfad7BefRoBPeu2jTm1EPk8+ihiK6BfhCZ/+7UQ+VFwL/r+/0Dx3DWsALOusHQ2/NonqRX0rVkAy3IxzbNoRUG/I/tuU6qiDgCGr82p2pqepuo4LiP1IlxA/pO91Tmgd9J8BlBXzJt+9UMfjxr9cCw0uotmLdwPCfqOrDLw7f6EZlAbaixtp69ND6nVk0jMhb2w97MMfn3u24FwCHDL1vLC6CLD6G2onUsCmDua2Hq0UHgeFPgSf0nfYVlD+DB4C1qBKzU6jxyIbHYxv0zlwsz0blfOEOHnCoGWxuacfXDQzXZNF839Oqo8SKyiuqZ/78dt3cq61brV1DL6XaFj6c6q1/IvBXep7ukZq67sWd17OoUq6FwBs76wf3htlUT+mF1EgLGy3tuTOqC1U48geq88jmbd2P2z3kn9rrjagh215JDUWzad/pnvJ86DsBLivwy168Lcxgjt71qWrky6gi8kFgOBhmY2RLCqmn9Ge1/6/M4mMszqbaCJ1NtRvsznn8i7b+B9P9xn4X8+eD1NPu46m2cg8Bvkz1un5PZ79nUFXt32MEB2NdxrxZmXpg+hFVVTooHblXy6vNOq8/2wKog/tO92SOr/P/ramq0W2G9nkGNQDx59q5sRbVO/UbDI27OM4LE7SlpR6af9gCwye3dau1YOEMahiiX1PtlXsdy5XqPfvrFvAc3Fl/P6opwELgTZ31g97TD6YemG8GPtT397Cc8+geVFvgV7AoML4b1cb+OBa1DR2Zkv9lzou+E+CyHL/cxZ9uD2s//v0767qB4WksmpFiJEsG7yAfZrcL9ctZfLq+Lakqgtd01j2Bag/0kE6gOHZBEFX9+QuqqrBbLbgZi2bleEpn/b7AfftO9wrMn3WoQdtf217PosZuO78FApcCO7Vt96LmBB6JjjhDAeHHqOYBC6jhZo4e2ncQGP6+5ccNwPZ9H0NPeXUIi1e3PpEqSeoGhiu3c+UBVBDda/AMPLJd4/6rpfW3wKGd7VtQD4eLBYZt20HAJ6mOZyNxbt+FfLovFcgf1F5vSZXwfo420Dg1TWWvJb4rJC/6ToDLcvpiF7+Y3YN66vvfdjN7Rmfb4KlwB2rsrGsZoblZ7yQPBkHdhtRA1DdS8zZ3SwxPpZ6in0m1NfwBNS7XWLUhnCBvtmo3gucNjpNFpWFbtwvk7QZrnikLVU36baoX5gupQHkh8G6qQ8kv2va1ht43MjfPdsP/PdXreJt2rAuBY4f225Uq8TqONkbhTFhY/KH6s1QHk/+mM0fxUGD4xM763s8DanDpC6ghcQYB4AVUW9DuWJvdwPAdVCeTHdv58ObpdEzLMa+2okoFX0Q1C7m6feeD8Um3a/eGp/Wd1uWeF30nwGU5f8HVoeJcqjH0zi0Iupzbd6SYS7Wd+gijP+zMRNU9a7cL+s0tMBxUB2xBtZdbSD0Z/ohFJahjV0LYyY+NqXZRJwF3a+u6N8FzgM/2nc6+zpe2/plUldKfqCrTbmebk4DP9532u3DMh1BzfD+6vX4J1dv4C+238NGh/VdlzGoQJpFXH2rB886d60b3oXsQGP4O+Oe+09tJ1zZUs6BnsejB+EiqxPe3LN7G8D5U++q/UU0hrmr7je133gK9h7FoBqJjqBLzv1OD9w/Wb0TN5XwWnQeCcV16T4DLFH+hnR8xVfJ1NjUA86Ct4GNaYHgFi9rbbdKCwdcv6QY5KkvnONdoN77DqSn81qaqyT/aAsMXsKjtSFDt6h7Jop55Y9F25I6+T6p0YJAXG3Ty4t5UacIb+k7/CjxfVqcejI6gHhru0davRw3BsVF7vQr1IPET4G19p38Zj3n19rt4eXv9onYj3IeqCv8GFRi+v++09pQ/3anM7gP8hprVabjDXjcwfBJVCnc+Vcrc+wMlFcj/uKX/j9RA6rTf90+5fWC4DhVIHkqnynhcroVDeXMCVQjw9/a97USVjp5EjUf4XKrEcFdqBpqrgW37TveKWAZVZBphbdy9zTPzV511B1DB3n2ptiGZ7cuOiJ2pKrAnUhexpG4Gj83Mc1dw8qdcRKxNtSFchwoGZ1GlPYdSF4AjqADgxVTVyk1D7185x2CmkohYNTMXtP//G9Vh4pfAzzLzt239N6kHhY8D86iR+Z9F3eQekZnze0j6CtFmYVjYzpcfU8HSbKra7VfUlI8f6uy/FjVzxZFt34dnjUMYOY0vpBOdzxHxQKqt2WyqNuEDwH9l5i0RsQfVa3Z9qsTwBSs6zSta+26fBXw8a7y+wfqdqQ5WD8/MswbnzBI+4wnAhZl58YpI8x3pnNs7U+3Fb6JKvr/ZztnNqJ7lGwLvzcxjlvA5Y3Et7IqIV1NB31uph75nU53HnkeVFM6lguIrqU5311DNbP6vj/SucH1HpS53baECnq9SJTsPaet2ZNFcxp/o7Dvc4/B5VJXC+2mzE4zqwuKdJU6gLoQPo6pJd6Mu7FdSHUnWowKgm6jOJ2NTRUKVkL6H1ju2rfsc1Vb0ChYNMfH4ofy6tm37MxUQPbjvY1lB+TUb+D7VfmpbKhBamypFuGCQD1Tpz4eoKrXvsKiJwbQuWR/6zW8HbM/ina0eR5WC7NpZ95KWJ68Y9evCJPLpeKrn/eyh9fenxqU7pLOum6cvBl7Rd/rv4Lg+Q5VqX0i1nd6RRTUkgxLD+QwNvzXOC9WZ8pWd11tTtWeXUb3KV2r59JR2D9mwr7T2kj99J8BlCr7EKvX7dQuEBoHh06gqg4uAh3b2HZ62a+XpfmObRD6s1X7U7+T2bSbv3gLDX7X91qaqCr7LNKjqmcI8eBFV/fFFqqR4F6pN6WOoYPgpVJvSH7H4FHbbU2PxPZJWdToTlnbx/yVVaj7oaLMnFSAf2l4P1j8HOJARrFZj8eqy86lSYNp5sZDqZRxUydHHqQ4Hq/WZ5hWUL+tSVcMrsaiX6V6dwGkzqlPRGcAjh967EdUB6ROD9063haoC3aj9vs9t94mdWPRQc2+q3eyNwL59p3c558VOwKOB99IegljUofC+LTC8Cti777T2mk99J8DlLnx5i7d9OaT94L9L6zYP/AtVAvQ1OvP2jlMQ1D0m4D/bDe4WFs020H2q35MqHZzbXm/AGI7OTw2PcgnVaeDNVNVgt63pE6nq9B/RBmWdKQu3fyh6ClVlNCgR3L+dQ4e31+tSgfbwPLAj8yAFvJoKfJ9NlWydQZWaP4UaW+/j7TdzOjVd21/peWy9FZQva1IPiWd2AsKXte//NZ3AcJ+27jTakF4tuDi+5eP9+z6WpTzeHagapeHAcE47B0bmnF6GY/9MC/gGNWgfG77mt8Dwm237E/tI53RYek+Ay1348hYPeDajGsReSpWIPaCtfzo1vMhXmSbTLC3H/LgXNWzGLcC7OusHgd+m1Dhrrxh631gMO8PiQ+28mRqs9lLgsMF2Fj0ZDwLD7zFDnow758GqnXV7USWr27UgaeEgv9r2p1FVqSMzNt/wzZ2Jq8tObjfJf2o3w1dTJSWfYIYMO0O1KbuQRZ2stmuB4inAxVRnnMGQXftRPZBvpXroXkp14BiZZhbUg/MgMPwVVUo+PC3j2AWGwOvad3Vg+42fS9WWvHSCfbekapC26jvdveVX3wlwWcYvbvFSwi9QYyidS3WbX9hu9tu27YPA8CTGfABiqg3hiS0PDh7atlO7mA/G5hun0sHuA8Kc9u9r2w3s1yyammxWJzDci6pO/BYzZIYKqvfwj+mMwdh+O4NShJd21m/dAsLPMCIPDkPnwdJUl11BG0aFNvBy38ewAvPquBYcrUe1rT65rV+PGtP1j0OB4bZUEH0E1dRi5IYnaYHh9lSV+LWMeYkw1SbwecDzO+sGbSkvXkJgODZtzJcpz/pOgMtd/AKr7c+VVFuwwbAZr6NKgb47+NFTVckL2w1urC/81GDdn2/H+9527M9hUfXYWD0NDwUCH2pBzvPb6ze18+PLg5sYi3fK2YMRH5dyknm1ETUG4/+010HNd3061cvw0dRAtnu3G8fPWDRszUgEhi2tS1td9i1mUHUZ1Zb4UKpE8JHUQ9Nv2m/kYZ39hgPD2X2leTnkQbRjP2HcroVDx3lIO7evZ1G1/6BJwKZUM4qL6QzL42JQONJLu7CdTI28HixeevgqFpUYbtPWPZWZ05vwHlSJ4a1U78G3Uu1mBk/9Y3cxpHoZ/47qNHDfzvq3UG0Mvwzcs62bcU/DLColGzwg7d5ez2pB4A+oBvfXtcDxq52AcFqfL0OB/rJUl43sHOeTyKM1qSYVp3WCgwvaufBNYOO2btDMoBsYvmqcAsOhfJnW5/ZdOK57UhMWLKAzpiiL2lJuSrWrvo7O9IUzfXGcwhEWEbOoSdmvzczHt3WrZOYt7f9fAh5LPQ09KzPP6ympvYiIjYG3U6WEczPzE2397My8udfETbGI+Hdq8PFnAj/IGqOsey68Ffg3aniKQzPzj/2ldsUYHmMtIqL9dw4V8H0feFm2sRzbPrtRVcyXABe0fJyVnbHrprOIeBjwUCoA/mhbd2+qicndgfdk5vuG3vOP82ScRcQ7qWrf7TJzQRuvbgeqCvkVVI/9V2bmpYOxJyNiPeph61HAEcN5p+mt3QPeR7UJfX5mfrytXyVrTM7NqGYEL8gxHpN1Mmb1nQDdJQupNoT7RsSumXlaO9EHN8PbqDZ0N1NF6DNKZl4WEYdRQ9DMaxeCY8ctIGweRDUZODPb4LrtXJidmTdn5hERcSM18PLfI+I5OWaD0g7LzNsiYg3ggcBFmXlV2/S7iDiZGqz4DcCCwWC/mXlq9zPa+lEJCA+hpuq6kWpHNXgA+mNEPJUKDF8WEQuzM1jxTAgIm6AC/jkR8Q4qeH4MNTbnRdQMP0RENzC8JiKeTpU4faOndGsZtXvAS6ghhz4aEWTmxzvXxj9ExOPH/Vo4GSv1nQAtu3bzP4YasPiIiHhEW39bRGxI9cI9FNgzM3/fX0r7k5mXUW1LPgt8LCKe3XOSplwrAdsCINvsLINSsUEAHBF7ZuZbqKrFN82Ei2DLg49SbYe+GhH/GhFz2uZ3UcMTvabd/CecpWJJ66epk6h2YrOBB0N9/+1h6BJgX+rB4S0RMfazlEzgp9QwVN+hOuE8PjN/10qKP00NPfRU4OiI2KSVFK6UmX+leuj/preUa5ll5uXUd/sFKjA8oK0fFA6M0m98ubOkcMRl5q8i4mnUDeH4iPgGVV28O9Vo/vDMvKbHJPYuMy+PiJdTbUvO7js9U63dvM4DDoyIPTLz5Oy0C4mIbYBXRsRNmfmO/lK6YrV8OZjqcLUH8Eng/yLiNCpYvIgqYZ0F3DLdp6u7M5n554h4DTX24Gsi4redUpFVMvOSiHgGVV12Sr+pXfEy8/MRcTQ1QsGPqeB5sO3vEfGp9vIDwG0RcXhm/qltH9nzQv+4BwwG9v94RNySmSe0bX63HbYpHBPtxv8uakLzlakSgefmTJmvcSmM4zyeAxHxAKok5CfAGzLzx239PaiOJo+gSoz/1F8ql687+34jYieqJOjZVMeLtYD7AM/MzM+smFQuf+07/wBVMthtRzW7lRyO7e9gSSJiJaq6+P3Ug+FzqCF5jsrMn3f2W43qqHUs1THt4JmWV+MsIjYB3ga8MzMv6Ds905FB4Rhp7afWoIZduDozr+05SVqBImIvqorkcqok6HrgIW3ZNTN/0WPylqtBZ5CIWJ1qVL4F1bP07OFqv4hYk5ra7P5Ux5wvtX8XjFh18RINBYYHZeZxnW0jXSK6tAbtRLuvqfE4r4+IV1DtSf8XOLL722iB4X5U+9wLV3S6tXzNxIeiyTAolMZIRDwUOAp4ADUw9S+o0pBf9pqw5WhwkY+ItalgeH2qs8X9qJv+MZn5rbbvYj1tI+LF1PSI249b7/wWGL6XGrz+3wbVZTNB98YfEQ8GVgX+3q05ab2PX0uVGL5pnB+apKVlm0JpjGTmz1pP01WoBtS3jnvv0k4v41Op0tFnZub8iPgxsCuwXutx+51B73xgYSst+zw1Q8U/UdN/jY1xb0u7JK2EcBAQzqPaVm9SL+ME4BOZ+aPMfGdE3EaVGN4WEW/PzBmTT9JEDAqlMdN61Y3jsDsTar2MD6UCwme3DhcnUXNhHw4cTfW4jcz8dgsiB2MWJjV001hWmbShVQ6cSdVlgyrjFhA+jhp4+nqqac3ngK0i4kWZeX5mvisiFlLtsf8eEQdlZ9xKaaYxKJQ0kgZt41ov4/OpmQr+HBEfpIZk2aeVnELd9F8REWtl5hfae1ahOhysQ7UrHEszKSAciIhHAjtSDwtfbSXEG1C9T38JzO+MTfmeiLgZOMWAUDOd4xRKGimt+nexoSQy88vAO6JmKHgc1eN60Lvwl1TJ6W7U0DSD99xCDUuzgx0Kxs7G1DRmv28B4ZbAfCr4f2UL/nZpnUrIzA9m5q/6S640PVhSKGlkdIZVWQN4LtXT/hrgU61X6b2ArYE/trHngup48hFqLuyz2ucMSom+0M+RaDlbkyr0+FXUVGdnUINWH5SZN0XEk4FXAwcD5/eXTGl6MSiUNK1FxA7AAzNzXgsI1wVOp6p916bG5Xxlm87qEuD/qOncLm3bXwLMz8wz2uc5JMWYGB52puPbwNXAycBWwNeAQzLzhoi4O7AP8Ddq+CZJjdXHkqatiJhFtQs7LiIObKs/QN3M96ZKBfemgsF5VC/TT1ODuJ8DfJGauWLw3hnZxm4cteB+0KnkfhHx8IjYsg07dAU1HM8cqunAS1tJ8v2oIYj+CXhZZv6lr/RL05HjFEqa1tosBO+mxts7ANgWOKs7C0kr/fkCVTL4KGrA7m2onsXHtx7HszLz1hWdfk29bglhRHycGnro/1Glf/8DvI8aYuhwat7bG6kHiQA2ojohOS6hNMSgUNK01wZi/iBV7Qewb2Z+aWiQ4j2Br1Dtxj459H6rjMdQG3fwsdSA7X+lqoqPosZlfDFwJvVw8HRgdappwWmZ+Yc+0itNd7YplDTtdSa0vwF4FrAz8KVWAjgoNbqAGnLk7hO834BwzETELtR58Crgs4Pe6BFxLvAZaraSuZl5LnBubwmVRohtCiWNhMy8DDiMqh58WUQc3NYPOhpsSg1SfH0/KdTy1OYu7roH9Z3Pb+NOrtzGrvwq8HLgycCDhj4jkLRElhRKGhmZeVlEvJTqcfzBiNga+DGwGjW8yKXAcT0mUcvBUDOBbVvp3w1t8xYRcXYrNV6Zakd6CjW930OB7w0+J20vJd0hSwoljZTMvJzqPPA54IVUb+MtqfHmHt4JDjQGhgLCDwHvj4gXUnNd/wZ4KTVYdbeZwAbUkDQXr/AESyPMoFDSyGmB4aHAJ6nr2A8z86DMvLX1MrYN4ZjoBISfA/YCPgp8u83x/SpgC+DEiNglItZus5ccQg1Fc2ZPyZZGkr2PJY2sNoPJwcCRDjczviLi34HXA88EftAZjmYN4InU2IMbA9cCV1ElhXtl5s/7SbE0mgwKJY0FxyEcXxHxQWB7YNfMvGmC7etR0x5uCvwe+Gpm/m7FplIafXY0kTQWDAjHU+sxvAXAICBsvYy7JRo7ZuZ7+kifNE5sUyhJmrZa8HcesHVE7NFZB0BEbAMcGhFPaK8ddkZaRgaFkqTp7liqZus1EfHIwco2082hwGbUbCUOOyPdBbYplCRNexGxFzW/9eXUOITXU3NcP4Rqa+hcxtJdZFAoSRoJEfFQam7jBwB/B34BHJWZv+w1YdKYMCiUJI2MiJgNrAIsBG7NzFt6TpI0NgwKJUmSZEcTSZIkGRRKkiQJg0JJkiRhUChJkiQMCiVJkoRBoSRJkjAolCRJEgaFkiRJwqBQkiRJwP8HN6YjrLXLD04AAAAASUVORK5CYII=\n", 591 | "text/plain": [ 592 | "
" 593 | ] 594 | }, 595 | "metadata": { 596 | "needs_background": "light" 597 | }, 598 | "output_type": "display_data" 599 | } 600 | ], 601 | "source": [ 602 | "# Plot the principal basis vectors for the search terms\n", 603 | "\n", 604 | "fig, axs = plt.subplots(3, sharex=True, figsize=(10, 5))\n", 605 | "plt.subplots_adjust(hspace=0)\n", 606 | "\n", 607 | "for i in range(3):\n", 608 | " plt.sca(axs[i])\n", 609 | " plt.bar(range(m), abs(U[:, i]), alpha=0.7)\n", 610 | " plt.yticks([])\n", 611 | " plt.ylabel('Factor {}'.format(i), rotation='horizontal', labelpad=40)\n", 612 | " \n", 613 | "plt.xticks(range(m), words, rotation=45)\n", 614 | "plt.show()" 615 | ] 616 | }, 617 | { 618 | "cell_type": "markdown", 619 | "metadata": {}, 620 | "source": [ 621 | "The vectors in $U$ contain information about the clustering of the keywords." 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": 22, 627 | "metadata": { 628 | "ExecuteTime": { 629 | "end_time": "2019-11-04T04:11:30.983510Z", 630 | "start_time": "2019-11-04T04:11:30.141717Z" 631 | } 632 | }, 633 | "outputs": [ 634 | { 635 | "data": { 636 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFHCAYAAADX4Tb2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAa3klEQVR4nO3de7RcZZ3m8e/ThDsohmijKMYL4vLuklEYMMi1UUEGkNu0MjGDl8ZZq7udbkdUEEGgUZc0rT2tzqhRLrHHVlpAVAgKLjVREqUFIipg7EYIiFwSbtHAb/7Yu7QozklSlcqpnHO+n7VqVdX7vnvvX51aZD3s9927UlVIkiRpevuTURcgSZKk0TMUSpIkyVAoSZIkQ6EkSZIwFEqSJAlDoSRJkoAZoy5gsps1a1bNnj171GVIkiSt09KlS++qqieP1Wco3ECzZ89myZIloy5DkiRpnZL8arw+p48lSZJkKJQkSZLTx5IkTRuHLjh01CUM5JLjLhl1CdOCZwolSZLkmUJJkjyDJnmmUJIkSRgKJUmShKFQkiRJuKZQkiRNMa4RHYxnCiVJkmQolCRJkqFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmSxHqEwiRzk9Q4jwOGXVCSeUnmDnu/6zjmPkkWJXkoye1JPppkq4msQZIkaZRm9DH2KODWnrZlQ6ylYx6wBpi/Efb9OEleDnwT+BrwfuA5wIeBpwJ/PhE1SJIkjVo/ofDaqrppo1WyESXZsqpWj9N9GrAcOKaq1gBXJlkDfCbJ2VX1k4mqU5IkaVSGsqYwydZJzk1yQ5IH2inYi5PsNsbY5yS5IMkdSVYnuSXJx9q+7wJ7Aft0TVEv7Np2jyRXJrm/fVyRZPee/Z+fZHmSvTpTwsCZ49S9FXAQ8M9tIOz4IvB74LAN/dtIkiRNBv2cKdwsSff4qqpH2tdbt4/TgBXAjsA7gUVJnl9Vd0ITCIEfAitppmpvBnYBOmsT3wYsAB4BTmzb7mu3fTlwFXAdMLftOwn4TpJXVtX1XbXNBC6kmQY+CXhwnM/0XGALoHtbqurBJMuBF6ztDyJJkjRV9BMKb+x5/z1gb4Cqupsm0AGQZDOadXq/AY4BPt52nQ5sDuxZVSu69jW/3c+yJKuANVW1uOd4H6AJd/tX1cr2OAtppn5PAY7uGrs9cFxVfW0dn2lm+3zPGH13d/VLkiRNaf2EwsN57IUmq7o7kxwLvAvYDXhCV1f3FPJBwMU9gXB9zWm3XdlpqKp7k1wKHNgzdjVw2XrsM51draVPkiRpyusnFF4/3oUmSQ6nmfb9HHAqcBfwKM3Zwu5bu8zk8Vcwr68dgNvHaF/B48/o3VFVYwW9Xnd31dXrScAt61+eJEnS5NVPKFybY4Ebq2pep6G9iGOHnnG/BXYe8Bj3AjuN0b5Tu99u6xMIAX4B/A54IfClTmOSbYDZwHl9VylJkjQJDesXTbahubdgt+PH2P/lwGFJnrKWfa2muWil19XAIUm27TQkeSLw+ravb1X1MHAFcEzPRTRH06x9vGSQ/UqSJE02wwqF3wBe1P4SyP5J3gOcTHOVcbeTaW71sijJCUlek+TNSb7QNWYZ8JIkRyXZPcnz2vbTaC4gWZjkiCRHAguBLWkuYBnUB4BnAwuS7JfkBOAc4ItVde0G7FeSJGnSGNb08SdppoXn0txK5gfAITS/EvIHVXVLklcBZwBnA9sCvwYu6hp2FrAr8FlgO+BK4ICq+nGSfdttz6OZIl4MzOm5HU1fqmppkoPb415GM039OeB9g+5TkiRpsllnKKyq+azjJ+fa+xW+t310e/oYY2+iuU3NePu6DTh4nL5FwH7rqOVNa+sfZ5tvA3v0u50kSdJUMazpY0mSJE1ihkJJkiQZCiVJkmQolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJAEzRl2A1u3QBYeOuoSBXXLcJaMuYZMzWb9Pv0tJmto8UyhJkiTPFGrT4Rk0SZJGxzOFkiRJMhRKkiTJUChJkiQMhZIkScJQKEmSJCBVNeoaJrVZs2bV7NmzR12GJEnSOi1durSqasyTgt6SZgPNnj2bJUuWjLoMSZKkdUryo/H6nD6WJEmSoVCSJElOH0sTzl9ukSRtijxTKEmSJEOhJEmSDIWSJEnCUChJkiS80ESbEC/AkCRpdAyFk8BkDUtgYJIkabJw+liSJEmGQkmSJBkKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJksR6hMIkc5PUOI8Dhl1QknlJ5g57v2s53pwk85Ncn+SRJDdN1LElSZI2FTP6GHsUcGtP27Ih1tIxD1gDzN8I+x7LAcDewBIgwJYTdFxJkqRNRj+h8NqqmpRn0ZJsWVWrx+k+tapOacd9Edh94iqTJEnaNAxlTWGSrZOcm+SGJA8kuT3JxUl2G2Psc5JckOSOJKuT3JLkY23fd4G9gH26pqgXdm27R5Irk9zfPq5IsnvP/s9PsjzJXkkWJXkIOHO82qvq0WH8DSRJkiazfs4Ubpake3xV1SPt663bx2nACmBH4J3AoiTPr6o7oQmEwA+BlcD7gZuBXWimcAHeBiwAHgFObNvua7d9OXAVcB0wt+07CfhOkldW1fVdtc0ELgQ+3I55sI/PKUmSNO30Ewpv7Hn/PZq1eFTV3TSBDoAkmwHfBH4DHAN8vO06Hdgc2LOqVnTta367n2VJVgFrqmpxz/E+QBPu9q+qle1xFgLLgVOAo7vGbg8cV1Vf6+PzSZIkTVv9hMLDeeyFJqu6O5McC7wL2A14QldX9xTyQcDFPYFwfc1pt13Zaaiqe5NcChzYM3Y1cNkAx5AkSZqW+gmF1493oUmSw2mmfT8HnArcBTxKc7Zwq66hM3n8Fczrawfg9jHaV7T77XZHVdWAx5EkSZp2+gmFa3MscGNVzes0JNmKJsh1+y2w84DHuBfYaYz2ndr9djMQSpIk9WFYv2iyDc29BbsdP8b+LwcOS/KUtexrNc1FK72uBg5Jsm2nIckTgde3fZIkSRrQsM4UfgP4RJKPAl8H/hPN1ccre8adDBxMc1XyWcBNwDOAA6vq+HbMMuCEJEcBvwRWVtXPaa5sXgQsTPIRmhtNv4fmZtOnD1p4G1DntG+fDmyb5I3t++urqvcCG0mSpClnWKHwkzTTwnNpbiXzA+AQ4DFX/1bVLUleBZwBnA1sC/wauKhr2FnArsBnge2AK4EDqurHSfZttz2PZop4MTCn53Y0/XoJ8KWets77k4EPbcC+JUmSJoV1hsKqms86fnKuvV/he9tHt6ePMfYmmtvUjLev22jOJo7VtwjYbx21vGlt/WOMX0hz1lGSJGnaGtaaQkmSJE1ihkJJkiQZCiVJkmQolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiQBM0ZdgCRJo3bogkNHXcJALjnukr7GT5fPqcF4plCSJEmGQkmSJBkKJUmShKFQkiRJGAolSZKEoVCSJEkYCiVJkoShUJIkSRgKJUmSBKSqRl3DpDZr1qyaPXv2qMuQJElap6VLl1ZVjXlS0J+520CzZ89myZIloy5DkiRpnZL8aLw+p48lSZJkKJQkSZKhUJIkSbimUNJGcOiCQ0ddwsAuOe6S9R47WT9nP59R0vRhKJQkrZXhV5oenD6WJEmSoVCSJEmGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkYSiUJEkS6xEKk8xNUuM8Dhh2QUnmJZk77P2Oc6wZSd6d5NtJ7kiyKsnSJG9JkomoQZIkaVMwo4+xRwG39rQtG2ItHfOANcD8jbDvXtsBJwFfAM4BHgReD3wWeF7bJ0mSNOX1EwqvraqbNlolG1GSLatq9Rhd9wPPrqp7utoWJtkR+Kskp46znSRJ0pQylDWFSbZOcm6SG5I8kOT2JBcn2W2Msc9JckE7Xbs6yS1JPtb2fRfYC9ina4p6Yde2eyS5Msn97eOKJLv37P/8JMuT7JVkUZKHgDPHqruq1vQEwo5rgK2AmYP/VSRJkiaPfs4Ubpake3xV1SPt663bx2nACmBH4J3AoiTPr6o7oQmEwA+BlcD7gZuBXYDO2sS3AQuAR4AT27b72m1fDlwFXAfMbftOAr6T5JVVdX1XbTOBC4EPt2Me7ONzAuwD3A3c2ed2kiRJk1I/ofDGnvffA/YGqKq7aQIdAEk2A74J/AY4Bvh423U6sDmwZ1Wt6NrX/HY/y5KsAtZU1eKe432AJtztX1Ur2+MsBJYDpwBHd43dHjiuqr7Wx+fr1P464EjgPV2hV5IkaUrrJxQezmMvNFnV3ZnkWOBdwG7AE7q6uqeQDwIu7gmE62tOu+3KTkNV3ZvkUuDAnrGrgcv6PUCSFwMXAFcAHx2gRkmSpEmpn1B4/XgXmiQ5nGba93PAqcBdwKM0Zwu36ho6k8dfwby+dgBuH6N9BY9f+3dHVVU/O0/yXOBy4BfAEZ4llCRJ00k/oXBtjgVurKp5nYYkW9EEuW6/BXYe8Bj3AjuN0b5Tu99u/QbCXYAradYRvraq7h+oQkmSpElqWL9osg3NvQW7HT/G/i8HDkvylLXsazXNRSu9rgYOSbJtpyHJE2nuK3h13xX/cR9/CiwEfg8cWFW9AVOSJGnKG1Yo/AbwoiQfTbJ/kvcAJ9NcZdztZJrwtSjJCUlek+TNSb7QNWYZ8JIkRyXZPcnz2vbTaC4gWZjkiCRH0oS5LWkuYOlbkm1oprifQXM19C7tbW86j+0H2a8kSdJkM6zp40/STAvPpbmVzA+AQ4DHXP1bVbckeRVwBnA2sC3wa+CirmFnAbvS/KrIdjTTugdU1Y+T7Ntuex7NFPFiYE7P7Wj68TTgpe3rBWP0vxr47oD7liRJmjTWGQqraj7r+Mm59qKM97aPbk8fY+xNNLepGW9ftwEHj9O3CNhvHbW8aW39Y9TibxxLkqRpb1jTx5IkSZrEDIWSJEkyFEqSJMlQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJwlAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkiRJAlJVo65hUkvyG+BXo65jA80C7hp1ERoKv8upw+9y6vC7nFom+/f5zKp68lgdhkKRZElV7T7qOrTh/C6nDr/LqcPvcmqZyt+n08eSJEkyFEqSJMlQqManR12Ahsbvcurwu5w6/C6nlin7fbqmUJIkSZ4plCRJkqFwWkryjCT/kuS+JCuTfCXJLqOuS/1L8sYkX07yqyQPJflZkrOSbD/q2rThknwjSSX50KhrUf+SvC7Jd5Lc3/5buyTJfqOuS/1LsleSy5Pc2X6XP0oyb9R1DZuhcJpJsg3wLeD5wH8D3gzsCnw7ybajrE0D+RvgEeC9wMHAPwF/AVyRxP++J7EkxwEvHXUdGkyStwNfBZYChwNHAV8CthllXepfkpcAC4HNgbcCRwLXAJ9J8hejrG3YXFM4zST5S+BjwG5VdVPb9izgF8C7q+pjo6xP/Uny5Kr6TU/b8cDngf2r6lujqUwbIskOwI3AXwMXAmdU1ftHW5XWV5LZwE+Bk6rq70dbjTZUkjNp/gd8ZlXd39W+GKiq2nNkxQ2ZZxKmnzcAizuBEKCqfgl8DzhsZFVpIL2BsHVN+7zzRNaiofowcENVLRh1IRrIPOBR4JOjLkRDsQXwe+ChnvZ7mWI5akp9GK2XFwLXj9F+A/CCCa5FG8c+7fNPR1qFBpJkb+B44MRR16KB7U1zpvfYJDcnWZPkpiTvHHVhGsj89vkfkjwtyQ5J3grsD5wzurKGb8aoC9CEmwncM0b73cCTJrgWDVmSnYHTgIVVtWTU9ag/STYHPgV8tKp+Nup6NLCntY+P0Kz3vZlmTeEnksyoqnNHWZz6U1XXJ3kNcBF//J+13wPvqKovjqywjcBQOD2NtZA0E16FhirJdjQL29cAbxlxORrM/wK2Bs4YdSHaIH8CbA/MraqvtG3fatcanpTkH8oF/ZNGkl2BL9PMqL2DZhr5MOCTSR6uqgtGWd8wGQqnn3tozhb2ehJjn0HUJJBkK+Bi4NnAPlV164hLUp/a20K9DzgB2DLJll3dW7YXn6yqqkdGUqD68Vuauzpc0dN+Oc1dAp4K3DbRRWlgZ9KcGTykqn7ftl2ZZEfg3CQLqurR0ZU3PK4pnH5uoFlX2OsFwLIJrkVD0E45fhl4JfC6qrpuxCVpMM8GtgLOp/kftM4Dmisf7wFePJrS1KcbxmnvzMhMiQAxjbwY+LeuQNjxQ2BH4CkTX9LGYSicfi4G9kjy7E5DO6WxV9unSaS9F+EFNAueD6uqxSMuSYO7Fth3jAc0QXFf4KaxN9Um5qL2+c962v8MuLWqVkxwPdowK4CXJdmip/1VwMM0a/KnBKePp5//A/wP4KtJ3k+zvvB04D9oFrhrcvlHmgXsZwAPJNmjq+9Wp5Enj6q6F7iqtz0JwK+q6nF92mRdBnwb+FSSWcAtwBuBg3C972T0CZobj1+S5H/TrCl8A3AccE5V/W6UxQ2TN6+ehtq1S+cAB9JMZ1wJ/FVVLR9lXepfkuXAM8fp/mBVnTpx1WhjSFJ48+pJJ8kTgLNowuCTaG5R83dVdeFIC9NAkryW5kKwF9Is87gZ+DTwqam0ztdQKEmSJNcUSpIkyVAoSZIkDIWSJEnCUChJkiQMhZIkScJQKEmSJAyFkqaYJHOTVNfjgSTLk1yU5Oj2V2DUpf2bzRtw213av/Oz2vf/lOSfh1uhpIngP46SpqqjgD2B1wEnA6uBBcDlSbYeZWGboLnAQKEQeAVwT1X9sn2/O7B0GEVJmliGQklT1bVVtbiqrq6q86rqWOBoYD/gwyOubSp5BW0ITLI58GIMhdKkZCiUNG1U1ZeBrwJvTbJNpz3JU5N8IcldSVYn+UmSN/Vun+RZSc5LsqIdd0uSc7v6r0py1RjbLU8yv+t9Z4r7Pyf5f0lWJbkjyUlt/8FJftxOfV+T5BVj7POIJIuTPJjk3iRfan/Csve45yc5NslP2/0tSbJ3d83APsBeXVPuj/sMa/EK4Eft6xcDW3a9lzSJGAolTTeX0QSX3QGSbAtcDbwWeC/wX4DrgPOSvK2zUbtm7ofAHOAD7fgPArM2oJbPt8c6HPhX4MwkZwMfAc4GjgG2Bf41yRZdtbwD+DKwjOa3dd8OvAi4Osn2Pcd4NfA/aabQjwE2Ay5NskPbfyLwY+AnNNPte7Zt40oyvxMggYOBd7evO2cI72775/b355A0SjNGXYAkTbB/b5+f2j6/BdgV2Leqrmrbvp7kT4EPJflM+4P3HwS2Bl5aVbd17e/zG1DLeVV1OvzhjN3hwLuA53XW6LUXxnyVJqxdnWQ7msD4uar6wzrAJD8Afg78d+Dvu47xBOBlVXVPO24FcA3NWssLq2pZkpXAjKpavJ51n9Ie4wXA+cDewIPAOcBtNKEW/vi3ljQJeKZQ0nST9rna5znAr7sCYcf5wJNpgg/AQcClPYFwQ32986Kq1gA3AT/vumgD4Mb2+Rnt8540Qe+CJDM6D+DWduycnmMs6gTC1nXt8y4MqKr+vaquBZ4E/LSqvt++3xX4elVd2z7uHvQYkiaeZwolTTedcHV7+zyz63W3FV39ADvSBK9huqfn/e/GaQPYqn1+Svu8cD33+ZhgVlWrk3Tvry/tmcvOCYW9ge+3ofSZwM7ANe37R6vq0UGOIWk0DIWSppvXAw/Ttf4N2G2McTu1z79tn++iCT1r8zDNWbxeM8doG1SnnrnADWP0rxriscZyCs2aym4ndL3unNn8IHDqRq5F0hAZCiVNG0mOAN4AnFtVD7bNVwNHJdmrqr7XNfy/AncCP23fXw4ckeSpVTXWmUWAXwFHJtmiqn7XHnMO0Hvxx4b4Pk3we25Vbch6xm6rWf8aPw1cCjyX5r6PBwD3AacDDwFntuOGOc0uaQIYCiVNVS9LMgvYgmb93CE0N7S+Ajipa9x84C+BryR5H80U8Z8DBwJvby8ygebs2OtppkvPpFn/tzNwcFV1bl/zReBtwGfbW9A8i+bCkfuG9aGqamWSvwX+McmTadYl3tfWsg9wVVVd2OdulwEnJjkGuBlYVVU/G+f4twG3JXk18G9VdSVAkpcCf11VSwb6YJJGzlAoaar6Uvv8MM0Zvx8BxwL/UlWdi0yoqgeS7ENzQ+u/ozlj9jPgzVV1fte45UleBXwIOKsd92uaK4M7Y77d3i7mb4AjaW718iaa28cMTVV9Ksl/AH9Lc0Zz87aW7wDXDrDLs2mm0P8vsB3N2dPXrGOb1wHfAEjyMpq1jt8c4NiSNhHp+rdRkiRJ05S3pJEkSZKhUJIkSYZCSZIkYSiUJEkShkJJkiRhKJQkSRKGQkmSJGEolCRJEoZCSZIkAf8fFEtpIZUPWgQAAAAASUVORK5CYII=\n", 637 | "text/plain": [ 638 | "
" 639 | ] 640 | }, 641 | "metadata": { 642 | "needs_background": "light" 643 | }, 644 | "output_type": "display_data" 645 | } 646 | ], 647 | "source": [ 648 | "# Plot the principal basis vectors for the documents\n", 649 | "\n", 650 | "fig, axs = plt.subplots(3, sharex=True, figsize=(10, 5))\n", 651 | "plt.subplots_adjust(hspace=0)\n", 652 | "\n", 653 | "for i in range(3):\n", 654 | " plt.sca(axs[i])\n", 655 | " plt.bar(range(n), abs(Vt.T[:, i]), color='g', alpha=0.7)\n", 656 | " plt.yticks([])\n", 657 | " plt.ylabel('Factor {}'.format(i), rotation='horizontal', labelpad=40)\n", 658 | " \n", 659 | "plt.xlabel('Document #')\n", 660 | "plt.show()" 661 | ] 662 | }, 663 | { 664 | "cell_type": "markdown", 665 | "metadata": {}, 666 | "source": [ 667 | "The vectors in $V$ contain information about the clustering of the documents.\n", 668 | "\n", 669 | "If we were to reconstruct $A$ using only these first 3 vectors from $U$ and $V$ (equivalent to setting the remining singular values to zero), this is similar to saying the information in the remaining vectors is just noise. In our example, that translates roughly as \"while we have 8 distinct words, there are really only 3 *things* that matter.\" Similarly, \"while there are 9 distinct documents, there are only 3 *things* that matter.\"" 670 | ] 671 | }, 672 | { 673 | "cell_type": "code", 674 | "execution_count": 23, 675 | "metadata": { 676 | "ExecuteTime": { 677 | "end_time": "2019-11-04T04:11:32.699370Z", 678 | "start_time": "2019-11-04T04:11:32.687087Z" 679 | } 680 | }, 681 | "outputs": [ 682 | { 683 | "data": { 684 | "text/plain": [ 685 | "array([[ 0.53, 0.53, 0.53, 0. , 0. , 0. , -0.02, -0.02, 0.03],\n", 686 | " [ 0.57, 0.57, 0.57, 0. , 0. , 0. , 0.04, 0.04, 0.09],\n", 687 | " [ 0.63, 0.63, 0.63, -0. , -0. , -0. , -0.02, -0.02, 0.04],\n", 688 | " [ 0. , -0. , -0. , 0.59, 0.59, 0.59, 0. , 0. , 0. ],\n", 689 | " [ 0. , -0. , -0. , 0.59, 0.59, 0.59, -0. , -0. , 0. ],\n", 690 | " [ 0. , -0. , -0. , 0.56, 0.56, 0.56, 0. , -0. , 0. ],\n", 691 | " [-0. , -0.02, -0. , -0. , -0. , -0. , 0.59, 0.59, 0.59],\n", 692 | " [ 0.01, -0.01, 0.01, 0. , 0. , 0. , 0.8 , 0.8 , 0.8 ]])" 693 | ] 694 | }, 695 | "execution_count": 23, 696 | "metadata": {}, 697 | "output_type": "execute_result" 698 | } 699 | ], 700 | "source": [ 701 | "# Construct a rank 3 approximation of A\n", 702 | "\n", 703 | "A_hat = U[:, :3] @ np.diag(Σ[:3]) @ Vt[:3, :]\n", 704 | "A_hat /= np.linalg.norm(A_hat, axis=0)\n", 705 | "np.round(A_hat, 2)" 706 | ] 707 | }, 708 | { 709 | "cell_type": "markdown", 710 | "metadata": {}, 711 | "source": [ 712 | "If we want to order the documents by relevance, we can now probe the \"overlap\" of a query vector with each document. If we did this with the original matrix $A$, then we would just retrieve the documents sorted by the prevalence of the target word. With our approximation, however, we have re-scored each document in terms of the 3 **latent factors** rather than words, and our search will now rank by the prevalence of those factors. \n", 713 | "\n", 714 | "The mathematical way of describing the similarity between two vectors is an inner product, so to create the ranking we take the inner product of our search vector with each column of $\\hat{A}$:" 715 | ] 716 | }, 717 | { 718 | "cell_type": "code", 719 | "execution_count": 426, 720 | "metadata": { 721 | "ExecuteTime": { 722 | "end_time": "2019-10-18T04:08:22.451965Z", 723 | "start_time": "2019-10-18T04:08:22.427733Z" 724 | } 725 | }, 726 | "outputs": [ 727 | { 728 | "name": "stdout", 729 | "output_type": "stream", 730 | "text": [ 731 | "Searching for piano...\n", 732 | "\n", 733 | "Music Theory for Beginners\n", 734 | "Beethoven vs. Mechagodzilla\n", 735 | "A History of Jazz\n", 736 | "Piano Cat Strikes Again\n", 737 | "The Lost Art of Grooming\n", 738 | "...\n" 739 | ] 740 | } 741 | ], 742 | "source": [ 743 | "# Perform a search\n", 744 | "search_term = 1\n", 745 | "print('Searching for {}...\\n'.format(words[search_term]))\n", 746 | "\n", 747 | "# The search vector, x, is a vector of zeros with a '1' for the desired term. If we were to search for\n", 748 | "# multiple words, we would normalize x to have unit norm\n", 749 | "x = np.zeros(m)\n", 750 | "x[search_term] = 1\n", 751 | "\n", 752 | "# Compute the scores by projecting x onto each \"document\" (the columns of A_hat)\n", 753 | "scores = abs(A_hat.T @ x)\n", 754 | "\n", 755 | "# Sort the titles based on their scores, in descending order\n", 756 | "rank = np.argsort(scores)[::-1]\n", 757 | "sorted_titles = titles[rank]\n", 758 | "\n", 759 | "# Print the results\n", 760 | "for i in range(5):\n", 761 | " print(sorted_titles[i])\n", 762 | "print('...')" 763 | ] 764 | } 765 | ], 766 | "metadata": { 767 | "kernelspec": { 768 | "display_name": "Python 3", 769 | "language": "python", 770 | "name": "python3" 771 | }, 772 | "language_info": { 773 | "codemirror_mode": { 774 | "name": "ipython", 775 | "version": 3 776 | }, 777 | "file_extension": ".py", 778 | "mimetype": "text/x-python", 779 | "name": "python", 780 | "nbconvert_exporter": "python", 781 | "pygments_lexer": "ipython3", 782 | "version": "3.6.7" 783 | }, 784 | "latex_envs": { 785 | "LaTeX_envs_menu_present": true, 786 | "autoclose": true, 787 | "autocomplete": true, 788 | "bibliofile": "biblio.bib", 789 | "cite_by": "apalike", 790 | "current_citInitial": 1, 791 | "eqLabelWithNumbers": false, 792 | "eqNumInitial": 1, 793 | "hotkeys": { 794 | "equation": "Ctrl-E", 795 | "itemize": "Ctrl-I" 796 | }, 797 | "labels_anchors": false, 798 | "latex_user_defs": false, 799 | "report_style_numbering": false, 800 | "user_envs_cfg": false 801 | }, 802 | "toc": { 803 | "base_numbering": 1, 804 | "nav_menu": {}, 805 | "number_sections": false, 806 | "sideBar": true, 807 | "skip_h1_title": false, 808 | "title_cell": "Table of Contents", 809 | "title_sidebar": "Contents", 810 | "toc_cell": false, 811 | "toc_position": {}, 812 | "toc_section_display": true, 813 | "toc_window_display": true 814 | } 815 | }, 816 | "nbformat": 4, 817 | "nbformat_minor": 2 818 | } 819 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # A Crash Course in Applied Linear Algebra 2 | 3 | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/plandrem/PyData-2019/master) 4 | 5 | --- 6 | 7 | These notebooks were written as companions to my talk at the PyData 2019 conference in NYC. They contain examples of topics from basic linear algebra courses like rank and eigenvectors, as well as some more advanced topics that are less familiar like the singular value decomposition. As much as possible, I've tried to include geometric 2D or 3D visualizations and coded python examples for all of the topics to make them more accessible. 8 | 9 | These notebooks are *faaaaaaaar* from comprehensive. I've skipped a lot of formal math in favor of brevity and readability. The goal is to pique your interest so that you can go on and indulge in a more rigorous course. I highly recommend the materials from Stanford's ["Introduction to Linear Dynamical Systems"](http://ee263.stanford.edu/ 10 | ) course, which is the material that first excited me about the subject. 11 | 12 | ### Python Setup 13 | 14 | > The easiest way to view the notebooks is with the Binder link above (no setup required) 15 | 16 | Everything in this talk can be done with a basic installation of Numpy and Scipy. The version should not be important. Scipy is used exclusively for some convenience functions, and Matplotlib is included only for visualization purposes. Neither are necessary for linear algebra. This notebook was written using Python 3.6, Numpy 1.16.4, Scipy 1.2.1, Matplotlib 3.1.1, notebook 5.7.8, and jupyter 1.0.0. 17 | 18 | There is a requirements.txt file for convenience if you want to setup an environment locally - this doesn't include Jupyter though. 19 | 20 | ### Viewing the Notebooks 21 | 22 | I've noticed that there are some rendering problems when viewing the notebooks on Github. I'm working to correct this, but they look fine on Binder and hopefully the materials should work if you download them and run Jupyter locally... 23 | 24 | Please feel free to contact me with feedback and corrections! 25 | -------------------------------------------------------------------------------- /helpers.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import scipy as sp 3 | import matplotlib.pyplot as plt 4 | from matplotlib import rcParams 5 | from scipy.stats import ortho_group 6 | 7 | 8 | def get_least_norm_matrices(show=False): 9 | np.random.seed(1) 10 | 11 | N = 16 # Number of Modes 12 | 13 | # Generate some random eigenvalue magnitudes 14 | a = 1 - 0.2 ** np.linspace(3, 5, N // 2) 15 | 16 | # Generate some random frequencies 17 | ϕ = np.logspace(-2, 0, N // 2) 18 | np.random.shuffle(ϕ) 19 | 20 | # Generate some random phase shifts 21 | δ = np.random.rand(N // 2) * 2 22 | 23 | # Create eigenvalues 24 | λ = a * np.exp(1j*ϕ) 25 | 26 | T = 1000 27 | ts = np.arange(T) 28 | 29 | if show: 30 | # Plot the oscillatory modes 31 | fig, axs = plt.subplots(N // 2, sharex=True, figsize=(10, 10)) 32 | plt.subplots_adjust(hspace=0) 33 | 34 | for λi, δi, ax in zip(λ, δ, axs): 35 | 36 | xs = λi**ts * np.exp(1j*δi) 37 | 38 | ax.plot(ts, xs) 39 | ax.set_yticks([]) 40 | 41 | plt.xlim(0, T) 42 | plt.xlabel('Time (s)') 43 | plt.show() 44 | 45 | # Construct A matrix 46 | 47 | # Build blocks for real modal form 48 | blocks = [] 49 | for λi in λ: 50 | block = np.array([[ λi.real, λi.imag], 51 | [-λi.imag, λi.real]]) 52 | 53 | blocks.append(block) 54 | 55 | Λ = sp.linalg.block_diag(*blocks) 56 | 57 | # - randomly rotate to get a non-modal matrix 58 | Q = ortho_group.rvs(dim=N) 59 | 60 | A = Q.T @ Λ @ Q 61 | 62 | # Build Linear Dynamic System matrices 63 | C = np.random.rand(2, A.shape[0]) 64 | B = np.random.randn(N, 2) 65 | 66 | # Give the system a kick and see what happens - the initial state will be an arbitrary displacement from x = 0 67 | if show: 68 | x = np.zeros(N) 69 | x[4] = 1 70 | 71 | ys = np.empty((len(ts), 2)) 72 | for t in ts: 73 | x = A @ x 74 | ys[t, :] = C @ x 75 | 76 | 77 | plt.figure(figsize=(10, 4)) 78 | plt.plot(ts, ys[:, 0]) 79 | plt.xlim(0, 350) 80 | plt.ylim(-1, 1) 81 | plt.xlabel('Time (s)') 82 | # plt.legend(['$y_1$', '$y_2$']) 83 | 84 | plt.show() 85 | 86 | return A, B, C, N -------------------------------------------------------------------------------- /img/Loop-left-shad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/Loop-left-shad.png -------------------------------------------------------------------------------- /img/determinant_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/determinant_1.png -------------------------------------------------------------------------------- /img/determinant_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/determinant_2.png -------------------------------------------------------------------------------- /img/disclaimer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/disclaimer.png -------------------------------------------------------------------------------- /img/dyn_eig.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/dyn_eig.png -------------------------------------------------------------------------------- /img/eig.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/eig.png -------------------------------------------------------------------------------- /img/excel.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/excel.jpg -------------------------------------------------------------------------------- /img/gps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/gps.png -------------------------------------------------------------------------------- /img/hamster.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/hamster.png -------------------------------------------------------------------------------- /img/hard_disk.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/hard_disk.jpg -------------------------------------------------------------------------------- /img/independence.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/independence.png -------------------------------------------------------------------------------- /img/intro_least_norm_control.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/intro_least_norm_control.png -------------------------------------------------------------------------------- /img/intro_naive_control.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/intro_naive_control.png -------------------------------------------------------------------------------- /img/intro_system_modes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/intro_system_modes.png -------------------------------------------------------------------------------- /img/intro_system_ringing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/intro_system_ringing.png -------------------------------------------------------------------------------- /img/markov_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/markov_1.png -------------------------------------------------------------------------------- /img/markov_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/markov_matrix.png -------------------------------------------------------------------------------- /img/nullspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/nullspace.png -------------------------------------------------------------------------------- /img/qr_factorization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/qr_factorization.png -------------------------------------------------------------------------------- /img/radio_tower.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/radio_tower.png -------------------------------------------------------------------------------- /img/receiver.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/receiver.png -------------------------------------------------------------------------------- /img/receiver2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/receiver2.png -------------------------------------------------------------------------------- /img/rigid_body.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/rigid_body.png -------------------------------------------------------------------------------- /img/rocket.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/rocket.png -------------------------------------------------------------------------------- /img/satellite-clipart-satellite-signal-21.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/satellite-clipart-satellite-signal-21.zip -------------------------------------------------------------------------------- /img/sensor_interp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/sensor_interp.png -------------------------------------------------------------------------------- /img/svd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/svd.png -------------------------------------------------------------------------------- /img/svd_matrices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/svd_matrices.png -------------------------------------------------------------------------------- /img/taylor_series.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/taylor_series.png -------------------------------------------------------------------------------- /img/vector.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/vector.png -------------------------------------------------------------------------------- /img/vector_addition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/plandrem/PyData-2019/7bb4aece20cfd7d25af725dca8ddde82b57e5fd8/img/vector_addition.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.16.* 2 | scipy==1.2.1 3 | matplotlib==3.1.1 -------------------------------------------------------------------------------- /runtime.txt: -------------------------------------------------------------------------------- 1 | python-3.6 --------------------------------------------------------------------------------