├── .gitignore
├── .travis.yml
├── 00_intro_numerical_methods.ipynb
├── 01_intro_to_python.ipynb
├── 02_NumPy.ipynb
├── 03_matplotlib.ipynb
├── 04_error.ipynb
├── 05_root_finding_optimization.ipynb
├── 06_interpolation.ipynb
├── 07_differentiation.ipynb
├── 08_quadrature.ipynb
├── 09_ODE_ivp_part1.ipynb
├── 09_ODE_ivp_part2.ipynb
├── 10.1_SVD.ipynb
├── 10_LA_intro.ipynb
├── 11_LA_QR.ipynb
├── 12_LA_conditioning_stability.ipynb
├── 13_LA_eigen.ipynb
├── 14_LA_iterative.ipynb
├── 15_LA_gaussian.ipynb
├── 16_ODE_BVP.ipynb
├── LICENSE
├── README.md
├── data
    └── sunspot.dat
├── extras
    └── 00_intro_numerical_methods-extra-examples.ipynb
├── images
    ├── CC-BY.png
    ├── Fenics_tc_vm.png
    ├── MakeGoodChoices.gif
    ├── RuiningMyLife.gif
    ├── bound_matrix_multiply.png
    ├── ellipses.png
    ├── ellipses_CG.png
    ├── golden_section.png
    ├── horners_method_big_count.png
    ├── householder.png
    ├── linear_regression.png
    ├── lsq_projection.png
    ├── mathModeling.png
    ├── mathModeling.svg
    └── siemens-volrender0000_kirby_modeling.png
├── peer_review.ipynb
├── requirements.txt
├── style.css
└── test.py


/.gitignore:
--------------------------------------------------------------------------------
1 | .ipynb_checkpoints/*
2 | *.slides.html
3 | peer_review.html
4 | *template.ipynb


--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
 1 | language: python
 2 | sudo: true
 3 | 
 4 | python:
 5 |   - "2.7"
 6 |   - "3.6"
 7 | 
 8 | before_install:
 9 |   - sudo apt-get update -qq
10 |   # - git clone --branch=master --depth=100 --quiet git://github.com/mandli/intro-numerical-methods
11 |   # - git checkout master
12 | 
13 | install: 
14 |   - pip install -U pip setuptools # Travis breaks without this
15 |   - pip install -r requirements.txt
16 | 
17 | script: 
18 |   - python test.py


--------------------------------------------------------------------------------
/02_NumPy.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 14 |     "</table>"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": null,
 20 |    "metadata": {
 21 |     "slideshow": {
 22 |      "slide_type": "skip"
 23 |     }
 24 |    },
 25 |    "outputs": [],
 26 |    "source": [
 27 |     "from __future__ import print_function"
 28 |    ]
 29 |   },
 30 |   {
 31 |    "cell_type": "markdown",
 32 |    "metadata": {
 33 |     "slideshow": {
 34 |      "slide_type": "slide"
 35 |     }
 36 |    },
 37 |    "source": [
 38 |     "# Introduction to NumPy\n",
 39 |     "\n",
 40 |     "NumPy is the basic library in Python that defines a number of essential data structures and routines for doing numerical computing (among other things).  Many of the semantics for manipulating the most basic data structure, the `ndarray`, are identical to manipulating `list`s with a few key exceptions. Other commands are similar to matlab commands and work in a similar manner.  We will cover those and some of the other important points when working with NumPy.\n",
 41 |     "\n",
 42 |     "Topics:\n",
 43 |     " - The `ndarray`\n",
 44 |     " - Mathematical functions\n",
 45 |     " - Array manipulations\n",
 46 |     " - Common array functions"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "markdown",
 51 |    "metadata": {
 52 |     "slideshow": {
 53 |      "slide_type": "slide"
 54 |     }
 55 |    },
 56 |    "source": [
 57 |     "## `ndarray`\n",
 58 |     "\n",
 59 |     "The `ndarray` forms the most basic type of data-structure for NumPy.  As the name suggests the `ndarray` is an array that can have as many dimensions as you specify.  For matlab users this should be familiar although note that the `ndarray` does not exactly behave as you might expect the same object to in matlab.  Here are some examples usages:"
 60 |    ]
 61 |   },
 62 |   {
 63 |    "cell_type": "code",
 64 |    "execution_count": null,
 65 |    "metadata": {
 66 |     "slideshow": {
 67 |      "slide_type": "fragment"
 68 |     }
 69 |    },
 70 |    "outputs": [],
 71 |    "source": [
 72 |     "import numpy"
 73 |    ]
 74 |   },
 75 |   {
 76 |    "cell_type": "markdown",
 77 |    "metadata": {
 78 |     "slideshow": {
 79 |      "slide_type": "fragment"
 80 |     }
 81 |    },
 82 |    "source": [
 83 |     "Define a 2x2 array, note that unlike MATLAB we need commas everywhere:"
 84 |    ]
 85 |   },
 86 |   {
 87 |    "cell_type": "code",
 88 |    "execution_count": null,
 89 |    "metadata": {
 90 |     "slideshow": {
 91 |      "slide_type": "-"
 92 |     }
 93 |    },
 94 |    "outputs": [],
 95 |    "source": [
 96 |     "my_array = numpy.array([[1, 2], [3, 4]])\n",
 97 |     "print(my_array)"
 98 |    ]
 99 |   },
100 |   {
101 |    "cell_type": "markdown",
102 |    "metadata": {
103 |     "slideshow": {
104 |      "slide_type": "subslide"
105 |     }
106 |    },
107 |    "source": [
108 |     "Get the `(0, 1)` component of the array:"
109 |    ]
110 |   },
111 |   {
112 |    "cell_type": "code",
113 |    "execution_count": null,
114 |    "metadata": {
115 |     "slideshow": {
116 |      "slide_type": "-"
117 |     }
118 |    },
119 |    "outputs": [],
120 |    "source": [
121 |     "print(my_array)"
122 |    ]
123 |   },
124 |   {
125 |    "cell_type": "code",
126 |    "execution_count": null,
127 |    "metadata": {
128 |     "slideshow": {
129 |      "slide_type": "-"
130 |     }
131 |    },
132 |    "outputs": [],
133 |    "source": [
134 |     "my_array[0, 1]"
135 |    ]
136 |   },
137 |   {
138 |    "cell_type": "markdown",
139 |    "metadata": {
140 |     "slideshow": {
141 |      "slide_type": "fragment"
142 |     }
143 |    },
144 |    "source": [
145 |     "Fetch the second row of the matrix:"
146 |    ]
147 |   },
148 |   {
149 |    "cell_type": "code",
150 |    "execution_count": null,
151 |    "metadata": {
152 |     "slideshow": {
153 |      "slide_type": "-"
154 |     }
155 |    },
156 |    "outputs": [],
157 |    "source": [
158 |     "my_array[1,:]"
159 |    ]
160 |   },
161 |   {
162 |    "cell_type": "markdown",
163 |    "metadata": {
164 |     "slideshow": {
165 |      "slide_type": "subslide"
166 |     }
167 |    },
168 |    "source": [
169 |     "Fetch the first column of the matrix:"
170 |    ]
171 |   },
172 |   {
173 |    "cell_type": "code",
174 |    "execution_count": null,
175 |    "metadata": {
176 |     "slideshow": {
177 |      "slide_type": "-"
178 |     }
179 |    },
180 |    "outputs": [],
181 |    "source": [
182 |     "print(my_array)"
183 |    ]
184 |   },
185 |   {
186 |    "cell_type": "code",
187 |    "execution_count": null,
188 |    "metadata": {
189 |     "slideshow": {
190 |      "slide_type": "-"
191 |     }
192 |    },
193 |    "outputs": [],
194 |    "source": [
195 |     "#my_array[?,?]"
196 |    ]
197 |   },
198 |   {
199 |    "cell_type": "markdown",
200 |    "metadata": {
201 |     "slideshow": {
202 |      "slide_type": "subslide"
203 |     }
204 |    },
205 |    "source": [
206 |     "Define a column vector:"
207 |    ]
208 |   },
209 |   {
210 |    "cell_type": "code",
211 |    "execution_count": null,
212 |    "metadata": {
213 |     "slideshow": {
214 |      "slide_type": "-"
215 |     }
216 |    },
217 |    "outputs": [],
218 |    "source": [
219 |     "my_vec = numpy.array([[1], [2]])\n",
220 |     "print(my_vec)"
221 |    ]
222 |   },
223 |   {
224 |    "cell_type": "markdown",
225 |    "metadata": {
226 |     "slideshow": {
227 |      "slide_type": "fragment"
228 |     }
229 |    },
230 |    "source": [
231 |     "Multiply `my_array` by the vector `my_vec` in the usual linear algebra sense (equivalent to MATLAB's `*`)"
232 |    ]
233 |   },
234 |   {
235 |    "cell_type": "code",
236 |    "execution_count": null,
237 |    "metadata": {
238 |     "slideshow": {
239 |      "slide_type": "-"
240 |     }
241 |    },
242 |    "outputs": [],
243 |    "source": [
244 |     "print(my_array)"
245 |    ]
246 |   },
247 |   {
248 |    "cell_type": "code",
249 |    "execution_count": null,
250 |    "metadata": {
251 |     "slideshow": {
252 |      "slide_type": "-"
253 |     }
254 |    },
255 |    "outputs": [],
256 |    "source": [
257 |     "print(numpy.dot(my_array, my_vec))"
258 |    ]
259 |   },
260 |   {
261 |    "cell_type": "markdown",
262 |    "metadata": {
263 |     "slideshow": {
264 |      "slide_type": "subslide"
265 |     }
266 |    },
267 |    "source": [
268 |     "Multiply `my_array` and `my_vec` by \"broadcasting\" the matching dimensions, equivalent to MATLAB's `.*` form:"
269 |    ]
270 |   },
271 |   {
272 |    "cell_type": "code",
273 |    "execution_count": null,
274 |    "metadata": {
275 |     "slideshow": {
276 |      "slide_type": "-"
277 |     }
278 |    },
279 |    "outputs": [],
280 |    "source": [
281 |     "print(my_array)\n",
282 |     "print()\n",
283 |     "print(my_vec)"
284 |    ]
285 |   },
286 |   {
287 |    "cell_type": "code",
288 |    "execution_count": null,
289 |    "metadata": {
290 |     "slideshow": {
291 |      "slide_type": "-"
292 |     }
293 |    },
294 |    "outputs": [],
295 |    "source": [
296 |     "my_array * my_vec"
297 |    ]
298 |   },
299 |   {
300 |    "cell_type": "markdown",
301 |    "metadata": {
302 |     "slideshow": {
303 |      "slide_type": "slide"
304 |     }
305 |    },
306 |    "source": [
307 |     "## Common Array Constructors\n",
308 |     "Along with the most common constructor for `ndarray`s above (`array`) there are number of other ways to create arrays with particular values inserted in them.  Here are a few that can be useful."
309 |    ]
310 |   },
311 |   {
312 |    "cell_type": "markdown",
313 |    "metadata": {
314 |     "slideshow": {
315 |      "slide_type": "subslide"
316 |     }
317 |    },
318 |    "source": [
319 |     "The `linspace` command (similar to MATLAB's `linspace` command) take three arguments, the first define a range of values and the third how many points to put in between them.  This is great if you want to evaluate a function at evently space points between two numbers."
320 |    ]
321 |   },
322 |   {
323 |    "cell_type": "code",
324 |    "execution_count": null,
325 |    "metadata": {
326 |     "slideshow": {
327 |      "slide_type": "-"
328 |     }
329 |    },
330 |    "outputs": [],
331 |    "source": [
332 |     "print(numpy.linspace(-1, 1, 10))\n",
333 |     "#numpy.linspace?"
334 |    ]
335 |   },
336 |   {
337 |    "cell_type": "markdown",
338 |    "metadata": {
339 |     "slideshow": {
340 |      "slide_type": "subslide"
341 |     }
342 |    },
343 |    "source": [
344 |     "Another useful set of functions are `zeros` and `ones` which create an array of zeros and ones respectively (again equivalent to the functions in MATLAB). Note that you can explicitly define the data type."
345 |    ]
346 |   },
347 |   {
348 |    "cell_type": "code",
349 |    "execution_count": null,
350 |    "metadata": {
351 |     "slideshow": {
352 |      "slide_type": "-"
353 |     }
354 |    },
355 |    "outputs": [],
356 |    "source": [
357 |     "numpy.zeros([3, 3])"
358 |    ]
359 |   },
360 |   {
361 |    "cell_type": "code",
362 |    "execution_count": null,
363 |    "metadata": {
364 |     "slideshow": {
365 |      "slide_type": "-"
366 |     }
367 |    },
368 |    "outputs": [],
369 |    "source": [
370 |     "numpy.ones([2, 3, 2], dtype=int)"
371 |    ]
372 |   },
373 |   {
374 |    "cell_type": "markdown",
375 |    "metadata": {
376 |     "slideshow": {
377 |      "slide_type": "subslide"
378 |     }
379 |    },
380 |    "source": [
381 |     "Another common array is the identity matrix. The `identity` command can be used to define an identity matrix of a given dimension."
382 |    ]
383 |   },
384 |   {
385 |    "cell_type": "code",
386 |    "execution_count": null,
387 |    "metadata": {
388 |     "slideshow": {
389 |      "slide_type": "-"
390 |     }
391 |    },
392 |    "outputs": [],
393 |    "source": [
394 |     "I = numpy.identity(3)\n",
395 |     "print(I)"
396 |    ]
397 |   },
398 |   {
399 |    "cell_type": "markdown",
400 |    "metadata": {
401 |     "slideshow": {
402 |      "slide_type": "subslide"
403 |     }
404 |    },
405 |    "source": [
406 |     "Note that NumPy arrays can be reshaped and expanded after they are created but this can be computational expense and may be difficult to fully understand the consequences of (`reshape` in particular can be difficult).  One way to avoid these issues is to create an empty array of the right size and storing the calculated values as you find them.  The array constructor to do this is called `empty`:"
407 |    ]
408 |   },
409 |   {
410 |    "cell_type": "code",
411 |    "execution_count": null,
412 |    "metadata": {
413 |     "slideshow": {
414 |      "slide_type": "-"
415 |     }
416 |    },
417 |    "outputs": [],
418 |    "source": [
419 |     "numpy.empty([2,3])"
420 |    ]
421 |   },
422 |   {
423 |    "cell_type": "markdown",
424 |    "metadata": {
425 |     "slideshow": {
426 |      "slide_type": "skip"
427 |     }
428 |    },
429 |    "source": [
430 |     "Note that here the IPython notebook is displaying zeros (or something close to this).  The values are almost always not zero but the display of values is truncated to help with displaying long numbers.  This can be controlled using `%precision 3` where 3 is upto the number of decimal points to display"
431 |    ]
432 |   },
433 |   {
434 |    "cell_type": "code",
435 |    "execution_count": null,
436 |    "metadata": {
437 |     "slideshow": {
438 |      "slide_type": "skip"
439 |     }
440 |    },
441 |    "outputs": [],
442 |    "source": [
443 |     "%precision 3\n",
444 |     "numpy.empty([2,3]) + 2"
445 |    ]
446 |   },
447 |   {
448 |    "cell_type": "markdown",
449 |    "metadata": {
450 |     "slideshow": {
451 |      "slide_type": "slide"
452 |     }
453 |    },
454 |    "source": [
455 |     "## Array Manipulations\n",
456 |     "Sometimes, despite our best efforts, we will need to manipulate the size or shape of our already created arrays.\n",
457 |     " - Note that these functions can be complex to use and can be computationally expensive so use sparingly!\n",
458 |     " - That being said, often these can still be a great way to avoid using too much memory and still may be faster than creating multiple arrays.\n",
459 |     " - Check out the [NumPy Docs](http://docs.scipy.org/doc/numpy/reference/routines.array-manipulation.html) for more functions beyond these basic ones"
460 |    ]
461 |   },
462 |   {
463 |    "cell_type": "markdown",
464 |    "metadata": {
465 |     "slideshow": {
466 |      "slide_type": "subslide"
467 |     }
468 |    },
469 |    "source": [
470 |     "One of the important aspects of an array is its `shape`."
471 |    ]
472 |   },
473 |   {
474 |    "cell_type": "code",
475 |    "execution_count": null,
476 |    "metadata": {
477 |     "slideshow": {
478 |      "slide_type": "-"
479 |     }
480 |    },
481 |    "outputs": [],
482 |    "source": [
483 |     "A = numpy.array([[1, 2, 3], [4, 5, 6]])\n",
484 |     "print(A)\n",
485 |     "print(\"A Shape = \", A.shape)\n"
486 |    ]
487 |   },
488 |   {
489 |    "cell_type": "markdown",
490 |    "metadata": {
491 |     "slideshow": {
492 |      "slide_type": "fragment"
493 |     }
494 |    },
495 |    "source": [
496 |     "We can reshape an array."
497 |    ]
498 |   },
499 |   {
500 |    "cell_type": "code",
501 |    "execution_count": null,
502 |    "metadata": {
503 |     "slideshow": {
504 |      "slide_type": "-"
505 |     }
506 |    },
507 |    "outputs": [],
508 |    "source": [
509 |     "B = A.reshape((6,1))\n",
510 |     "print(\"A Shape = \", A.shape)\n",
511 |     "print(\"B Shape = \", B.shape)\n",
512 |     "print(B)\n",
513 |     "#numpy.reshape?  "
514 |    ]
515 |   },
516 |   {
517 |    "cell_type": "markdown",
518 |    "metadata": {
519 |     "slideshow": {
520 |      "slide_type": "subslide"
521 |     }
522 |    },
523 |    "source": [
524 |     "Take the matrix `A` and make a larger matrix by tiling the old one the number of times specified."
525 |    ]
526 |   },
527 |   {
528 |    "cell_type": "code",
529 |    "execution_count": null,
530 |    "metadata": {
531 |     "slideshow": {
532 |      "slide_type": "-"
533 |     }
534 |    },
535 |    "outputs": [],
536 |    "source": [
537 |     "A"
538 |    ]
539 |   },
540 |   {
541 |    "cell_type": "code",
542 |    "execution_count": null,
543 |    "metadata": {
544 |     "slideshow": {
545 |      "slide_type": "-"
546 |     }
547 |    },
548 |    "outputs": [],
549 |    "source": [
550 |     "numpy.tile(A, (2,3))"
551 |    ]
552 |   },
553 |   {
554 |    "cell_type": "code",
555 |    "execution_count": null,
556 |    "metadata": {
557 |     "slideshow": {
558 |      "slide_type": "-"
559 |     }
560 |    },
561 |    "outputs": [],
562 |    "source": [
563 |     "A.flatten()"
564 |    ]
565 |   },
566 |   {
567 |    "cell_type": "markdown",
568 |    "metadata": {
569 |     "slideshow": {
570 |      "slide_type": "slide"
571 |     }
572 |    },
573 |    "source": [
574 |     "## Array Operations\n",
575 |     "\n",
576 |     "The numpy library also includes a number of basic operations on arrays. For example, a common operation is to determine the transpose of an array."
577 |    ]
578 |   },
579 |   {
580 |    "cell_type": "code",
581 |    "execution_count": null,
582 |    "metadata": {
583 |     "slideshow": {
584 |      "slide_type": "-"
585 |     }
586 |    },
587 |    "outputs": [],
588 |    "source": [
589 |     "B = numpy.array([[1,2,3],[1,4,9],[1,8,27]])\n",
590 |     "print(B)\n",
591 |     "print(B.transpose())"
592 |    ]
593 |   },
594 |   {
595 |    "cell_type": "markdown",
596 |    "metadata": {
597 |     "slideshow": {
598 |      "slide_type": "subslide"
599 |     }
600 |    },
601 |    "source": [
602 |     "One nice aspect of the numpy libary is that scalar multiplication is defined in the usual way."
603 |    ]
604 |   },
605 |   {
606 |    "cell_type": "code",
607 |    "execution_count": null,
608 |    "metadata": {
609 |     "slideshow": {
610 |      "slide_type": "-"
611 |     }
612 |    },
613 |    "outputs": [],
614 |    "source": [
615 |     "v = numpy.array([[1],[2],[3]])\n",
616 |     "print(v)\n",
617 |     "print(2*v)"
618 |    ]
619 |   },
620 |   {
621 |    "cell_type": "markdown",
622 |    "metadata": {
623 |     "slideshow": {
624 |      "slide_type": "subslide"
625 |     }
626 |    },
627 |    "source": [
628 |     "Another common operation is to multiply two arrays. Be careful to make sure that an operation is defined. In the example below the operation that is commented out is not defined. You should uncomment the line and execute the commands. It is important to learn how to read and interpret error messages."
629 |    ]
630 |   },
631 |   {
632 |    "cell_type": "code",
633 |    "execution_count": null,
634 |    "metadata": {
635 |     "slideshow": {
636 |      "slide_type": "-"
637 |     }
638 |    },
639 |    "outputs": [],
640 |    "source": [
641 |     "A = numpy.array([[1],[-1],[1]])\n",
642 |     "B = numpy.array([[1,2,3],[1,4,9],[1,8,27]])\n",
643 |     "print(numpy.matmul(B,A))\n",
644 |     "print(numpy.matmul(A.transpose(),B))\n",
645 |     "#print(numpy.matmul(A,B))"
646 |    ]
647 |   },
648 |   {
649 |    "cell_type": "markdown",
650 |    "metadata": {
651 |     "slideshow": {
652 |      "slide_type": "subslide"
653 |     }
654 |    },
655 |    "source": [
656 |     "An element within an array can be changed using the same notation above that is used to get the value of an entry within an array."
657 |    ]
658 |   },
659 |   {
660 |    "cell_type": "code",
661 |    "execution_count": null,
662 |    "metadata": {
663 |     "slideshow": {
664 |      "slide_type": "-"
665 |     }
666 |    },
667 |    "outputs": [],
668 |    "source": [
669 |     "B = numpy.array([[1,2,3],[1,4,9],[1,8,27]])\n",
670 |     "print(B)\n",
671 |     "B[0,0] = -5\n",
672 |     "print(B)"
673 |    ]
674 |   },
675 |   {
676 |    "cell_type": "markdown",
677 |    "metadata": {
678 |     "slideshow": {
679 |      "slide_type": "slide"
680 |     }
681 |    },
682 |    "source": [
683 |     "## Mathematical Functions\n",
684 |     "Similar to the built-in Python module `math`, NumPy also provides a number of common math functions such as `sqrt`, `sin`, `cos`, and `tan` along with a number of useful constants, the most important of which is $pi$.  The benefit of using NumPy's versions is that they can be used on entire arrays."
685 |    ]
686 |   },
687 |   {
688 |    "cell_type": "code",
689 |    "execution_count": null,
690 |    "metadata": {
691 |     "slideshow": {
692 |      "slide_type": "subslide"
693 |     }
694 |    },
695 |    "outputs": [],
696 |    "source": [
697 |     "x = numpy.linspace(-2.0 * numpy.pi, 2.0 * numpy.pi, 62)\n",
698 |     "y = numpy.sin(x)\n",
699 |     "print(y)"
700 |    ]
701 |   },
702 |   {
703 |    "cell_type": "markdown",
704 |    "metadata": {
705 |     "slideshow": {
706 |      "slide_type": "subslide"
707 |     }
708 |    },
709 |    "source": [
710 |     "This is often useful for plotting functions easily or setting up a problem (we will cover plotting later)."
711 |    ]
712 |   },
713 |   {
714 |    "cell_type": "code",
715 |    "execution_count": null,
716 |    "metadata": {
717 |     "hide_input": true
718 |    },
719 |    "outputs": [],
720 |    "source": [
721 |     "import matplotlib.pyplot as plt\n",
722 |     "%matplotlib inline\n",
723 |     "\n",
724 |     "fig = plt.figure(figsize=(8,6))\n",
725 |     "plt.plot(x,y,linewidth=2)\n",
726 |     "plt.grid()\n",
727 |     "plt.xlabel('$x$',fontsize=16)\n",
728 |     "plt.ylabel('$y$',fontsize=16)\n",
729 |     "plt.title('$\\sin{x}$',fontsize=18)\n",
730 |     "plt.show()"
731 |    ]
732 |   },
733 |   {
734 |    "cell_type": "markdown",
735 |    "metadata": {
736 |     "slideshow": {
737 |      "slide_type": "subslide"
738 |     }
739 |    },
740 |    "source": [
741 |     "One thing to watch out for (and this is true of the `math` module) is that contrary to what you might expect:"
742 |    ]
743 |   },
744 |   {
745 |    "cell_type": "code",
746 |    "execution_count": null,
747 |    "metadata": {
748 |     "slideshow": {
749 |      "slide_type": "-"
750 |     }
751 |    },
752 |    "outputs": [],
753 |    "source": [
754 |     "x = numpy.linspace(-1, 1, 20)\n",
755 |     "print(x)\n",
756 |     "numpy.sqrt(x)"
757 |    ]
758 |   },
759 |   {
760 |    "cell_type": "markdown",
761 |    "metadata": {
762 |     "slideshow": {
763 |      "slide_type": "subslide"
764 |     }
765 |    },
766 |    "source": [
767 |     "The problem is that if you take the `sqrt` of a negative number NumPy does not automatically use the `Complex` variable type to represent the output.  Unlike lists, NumPy requires the data stored within to be uniform (of the same type or record structure).  By default NumPy assumes we want `float`s which obey the IEEE compliant floating point rules for arithmetic (more on this later) and generates `nan`s instead (`nan` stands for \"not-a-number\", see more about this special value [here]()).\n",
768 |     "\n",
769 |     "If we want to deal with complex numbers there is still a way to tell NumPy that we want the `Complex` data type instead by doing the following:"
770 |    ]
771 |   },
772 |   {
773 |    "cell_type": "code",
774 |    "execution_count": null,
775 |    "metadata": {
776 |     "slideshow": {
777 |      "slide_type": "-"
778 |     }
779 |    },
780 |    "outputs": [],
781 |    "source": [
782 |     "x = numpy.linspace(-1, 1, 20, dtype=complex)\n",
783 |     "numpy.sqrt(x)"
784 |    ]
785 |   },
786 |   {
787 |    "cell_type": "markdown",
788 |    "metadata": {
789 |     "slideshow": {
790 |      "slide_type": "fragment"
791 |     }
792 |    },
793 |    "source": [
794 |     "There are number of other data types that NumPy understands, the most important one being `int` for integers."
795 |    ]
796 |   }
797 |  ],
798 |  "metadata": {
799 |   "celltoolbar": "Slideshow",
800 |   "kernelspec": {
801 |    "display_name": "Python 3",
802 |    "language": "python",
803 |    "name": "python3"
804 |   },
805 |   "language_info": {
806 |    "codemirror_mode": {
807 |     "name": "ipython",
808 |     "version": 3
809 |    },
810 |    "file_extension": ".py",
811 |    "mimetype": "text/x-python",
812 |    "name": "python",
813 |    "nbconvert_exporter": "python",
814 |    "pygments_lexer": "ipython3",
815 |    "version": "3.7.5"
816 |   },
817 |   "latex_envs": {
818 |    "bibliofile": "biblio.bib",
819 |    "cite_by": "apalike",
820 |    "current_citInitial": 1,
821 |    "eqLabelWithNumbers": true,
822 |    "eqNumInitial": 0
823 |   }
824 |  },
825 |  "nbformat": 4,
826 |  "nbformat_minor": 4
827 | }
828 | 


--------------------------------------------------------------------------------
/03_matplotlib.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 14 |     "</table>"
 15 |    ]
 16 |   },
 17 |   {
 18 |    "cell_type": "code",
 19 |    "execution_count": null,
 20 |    "metadata": {
 21 |     "slideshow": {
 22 |      "slide_type": "skip"
 23 |     }
 24 |    },
 25 |    "outputs": [],
 26 |    "source": [
 27 |     "from __future__ import print_function"
 28 |    ]
 29 |   },
 30 |   {
 31 |    "cell_type": "markdown",
 32 |    "metadata": {
 33 |     "slideshow": {
 34 |      "slide_type": "slide"
 35 |     }
 36 |    },
 37 |    "source": [
 38 |     "# Plotting with `matplotlib`\n",
 39 |     "\n",
 40 |     "The most common facility for plotting with the Python numerical suite is to use the `matplotlib` package.  We will cover a few of the basic approaches to plotting figures.  If you are interested in learning more about `matplotlib` or are looking to see how you might create a particular plot check out the matplotlib [gallery](http://matplotlib.org/gallery) for inspiration.\n",
 41 |     "\n",
 42 |     "Please see the [matplotlib](https://matplotlib.org) site for more details and examples."
 43 |    ]
 44 |   },
 45 |   {
 46 |    "cell_type": "markdown",
 47 |    "metadata": {
 48 |     "slideshow": {
 49 |      "slide_type": "slide"
 50 |     }
 51 |    },
 52 |    "source": [
 53 |     "## Interfaces\n",
 54 |     "\n",
 55 |     "There are two primary ways to interface with the plotting facilities in `matplotlib`\n",
 56 |     " - Influenced by `MATLAB`, a procedural interface and\n",
 57 |     " - An object oriented interface"
 58 |    ]
 59 |   },
 60 |   {
 61 |    "cell_type": "code",
 62 |    "execution_count": null,
 63 |    "metadata": {
 64 |     "slideshow": {
 65 |      "slide_type": "subslide"
 66 |     }
 67 |    },
 68 |    "outputs": [],
 69 |    "source": [
 70 |     "%matplotlib inline\n",
 71 |     "#%matplotlib notebook\n",
 72 |     "import numpy\n",
 73 |     "import matplotlib.pyplot as plt"
 74 |    ]
 75 |   },
 76 |   {
 77 |    "cell_type": "markdown",
 78 |    "metadata": {
 79 |     "slideshow": {
 80 |      "slide_type": "fragment"
 81 |     }
 82 |    },
 83 |    "source": [
 84 |     "These commands import the necessary functions for NumPy and `matplotlib` for our demos.  The first command is so that the IPython notebook will make our plots and display them in the notebook."
 85 |    ]
 86 |   },
 87 |   {
 88 |    "cell_type": "markdown",
 89 |    "metadata": {
 90 |     "slideshow": {
 91 |      "slide_type": "slide"
 92 |     }
 93 |    },
 94 |    "source": [
 95 |     "### Matlab Inspired Interface\n",
 96 |     "First off, lets plot a simple quadratic function $f(x) = x^2 + 2x + 3$:"
 97 |    ]
 98 |   },
 99 |   {
100 |    "cell_type": "code",
101 |    "execution_count": null,
102 |    "metadata": {
103 |     "slideshow": {
104 |      "slide_type": "fragment"
105 |     }
106 |    },
107 |    "outputs": [],
108 |    "source": [
109 |     "x = numpy.linspace(-5, 5, 100)\n",
110 |     "#print(x)\n",
111 |     "y = x**2 + 2 * x + 3\n",
112 |     "#print(y)\n",
113 |     "plt.plot(x, y, 'ro')\n",
114 |     "# plt.plot?\n",
115 |     "plt.show()"
116 |    ]
117 |   },
118 |   {
119 |    "cell_type": "markdown",
120 |    "metadata": {
121 |     "slideshow": {
122 |      "slide_type": "subslide"
123 |     }
124 |    },
125 |    "source": [
126 |     "We can also label our plot (please do this), set the bounds and change the style of the plot:"
127 |    ]
128 |   },
129 |   {
130 |    "cell_type": "code",
131 |    "execution_count": null,
132 |    "metadata": {
133 |     "slideshow": {
134 |      "slide_type": "fragment"
135 |     }
136 |    },
137 |    "outputs": [],
138 |    "source": [
139 |     "plt.plot(x, y, 'r--')\n",
140 |     "plt.xlabel(\"time (s)\")\n",
141 |     "plt.ylabel(\"Tribble Population\")\n",
142 |     "plt.title(\"Tribble Growth\")\n",
143 |     "plt.xlim([0, 4])\n",
144 |     "plt.ylim([0, 25])\n",
145 |     "plt.show()"
146 |    ]
147 |   },
148 |   {
149 |    "cell_type": "markdown",
150 |    "metadata": {
151 |     "slideshow": {
152 |      "slide_type": "slide"
153 |     }
154 |    },
155 |    "source": [
156 |     "It is possible to include the graph of more than one relationship. Simply use additional plot commands for each relationship you wish to graph. When doing so it is polite to include a legend. \n",
157 |     "\n",
158 |     "More information about using the [legend command](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.legend.html) is available at the matplotlib website."
159 |    ]
160 |   },
161 |   {
162 |    "cell_type": "code",
163 |    "execution_count": null,
164 |    "metadata": {
165 |     "slideshow": {
166 |      "slide_type": "fragment"
167 |     }
168 |    },
169 |    "outputs": [],
170 |    "source": [
171 |     "y1 = x\n",
172 |     "plt.plot(x,x,'g',label='Linear')\n",
173 |     "plt.plot(x,x*x,'r',label='Quadratic')\n",
174 |     "plt.legend()\n",
175 |     "plt.show()"
176 |    ]
177 |   },
178 |   {
179 |    "cell_type": "markdown",
180 |    "metadata": {
181 |     "slideshow": {
182 |      "slide_type": "subslide"
183 |     }
184 |    },
185 |    "source": [
186 |     "The other major way to plot data is via a pseudo-color plot.  A pseudo-color plot takes 3-dimensional data (or 2-dimensional depending on how you look at it) and plots it using color to represent the values of the function.  Say we have a function $f(x, y)$, we can plot a pseudo-color by using the following commands."
187 |    ]
188 |   },
189 |   {
190 |    "cell_type": "code",
191 |    "execution_count": null,
192 |    "metadata": {
193 |     "slideshow": {
194 |      "slide_type": "fragment"
195 |     }
196 |    },
197 |    "outputs": [],
198 |    "source": [
199 |     "x = numpy.linspace(-1, 1, 100)\n",
200 |     "y = numpy.linspace(-1, 1, 100)\n",
201 |     "X, Y = numpy.meshgrid(x, y)\n",
202 |     "F = numpy.sin(X**2) + numpy.cos(Y**2)\n",
203 |     "plt.pcolor(X, Y, F)\n",
204 |     "plt.colorbar(label=\"stuff\")\n",
205 |     "plt.title('Pseudo-Color plot')\n",
206 |     "plt.show()"
207 |    ]
208 |   },
209 |   {
210 |    "cell_type": "markdown",
211 |    "metadata": {
212 |     "slideshow": {
213 |      "slide_type": "subslide"
214 |     }
215 |    },
216 |    "source": [
217 |     "instead of pcolor,  you could also make a filled contour plot using contourf"
218 |    ]
219 |   },
220 |   {
221 |    "cell_type": "code",
222 |    "execution_count": null,
223 |    "metadata": {
224 |     "slideshow": {
225 |      "slide_type": "fragment"
226 |     }
227 |    },
228 |    "outputs": [],
229 |    "source": [
230 |     "plt.contourf(X, Y, F)\n",
231 |     "plt.colorbar(label=\"stuff\")\n",
232 |     "plt.title('Contour Plot')\n",
233 |     "plt.show()"
234 |    ]
235 |   },
236 |   {
237 |    "cell_type": "markdown",
238 |    "metadata": {
239 |     "slideshow": {
240 |      "slide_type": "subslide"
241 |     }
242 |    },
243 |    "source": [
244 |     "Here we have used a new way to create NumPy `ndarray`s so that we can easily evaluate 2-dimensional functions.  The function `meshgrid` takes two 1-dimensional arrays and turns them into 2, 2-dimensional arrays whose matching indices will provide an easy to use set of arrays that can be evaluate 2-dimensional functions.\n",
245 |     "\n",
246 |     "In the current version of Matplotlib (v3.x), the default colormap is 'viridis' while earlier versions used a rainbow colormap known as 'jet'.  I highly suggest looking at [this explanation of how to choose a colormap](http://matplotlib.org/users/colormaps.html) if you are interested.  \n",
247 |     "\n",
248 |     "Anyway, let us make this plot also more useful."
249 |    ]
250 |   },
251 |   {
252 |    "cell_type": "code",
253 |    "execution_count": null,
254 |    "metadata": {
255 |     "slideshow": {
256 |      "slide_type": "subslide"
257 |     }
258 |    },
259 |    "outputs": [],
260 |    "source": [
261 |     "color_map = plt.get_cmap(\"Oranges\")\n",
262 |     "plt.gcf().set_figwidth(plt.gcf().get_figwidth() * 2) \n",
263 |     "plt.subplot(1, 2, 1, aspect=\"equal\")\n",
264 |     "plt.pcolor(X, Y, F, cmap=color_map)\n",
265 |     "plt.colorbar(label=\"Density\")\n",
266 |     "\n",
267 |     "plt.xlabel(\"x (m)\")\n",
268 |     "plt.ylabel(\"y (m)\")\n",
269 |     "plt.title(\"Tribble Density ($Number/m^2$)\")\n",
270 |     "\n",
271 |     "plt.subplot(1, 2, 2, aspect=\"equal\")\n",
272 |     "plt.pcolor(X, Y, X*Y, cmap=plt.get_cmap(\"RdBu\"))\n",
273 |     "plt.xlabel(\"x (m)\")\n",
274 |     "plt.ylabel(\"y (m)\")\n",
275 |     "plt.title(\"Tribble Birth Rate Anomaly\")\n",
276 |     "plt.colorbar(label=\"Birth Rate Deviation\")\n",
277 |     "\n",
278 |     "plt.autoscale(enable=True, tight=False)\n",
279 |     "plt.show()"
280 |    ]
281 |   },
282 |   {
283 |    "cell_type": "markdown",
284 |    "metadata": {
285 |     "slideshow": {
286 |      "slide_type": "slide"
287 |     }
288 |    },
289 |    "source": [
290 |     "## Object-Oriented Interface\n",
291 |     "\n",
292 |     "A much better way to \"build\" plots is to use a more object-oriented approach.  In this case, we create the objects and manipulate them which allows us to have more control over how we create plots.  Here we will create the annotated plots so that we have examples on how to do the same things (notice that they are very similar).  The basic premise of this approach is that you generate objects that can be manipulated and remain persistent."
293 |    ]
294 |   },
295 |   {
296 |    "cell_type": "code",
297 |    "execution_count": null,
298 |    "metadata": {
299 |     "slideshow": {
300 |      "slide_type": "subslide"
301 |     }
302 |    },
303 |    "outputs": [],
304 |    "source": [
305 |     "x = numpy.linspace(-5, 5, 100)\n",
306 |     "\n",
307 |     "fig = plt.figure()\n",
308 |     "axes = fig.add_subplot(1, 1, 1)\n",
309 |     "growth_curve = axes.plot(x, x**2 + 2 * x + 3, 'r--')\n",
310 |     "axes.set_xlabel(\"time (s)\")\n",
311 |     "axes.set_ylabel(\"Tribble Population\")\n",
312 |     "axes.set_title(\"Tribble Growth\")\n",
313 |     "axes.set_xlim([0, 4])\n",
314 |     "axes.set_ylim([0, 25])\n",
315 |     "#growth_curve[0].set_color('b')\n",
316 |     "plt.show()"
317 |    ]
318 |   },
319 |   {
320 |    "cell_type": "markdown",
321 |    "metadata": {
322 |     "slideshow": {
323 |      "slide_type": "skip"
324 |     }
325 |    },
326 |    "source": [
327 |     "Here is the other example with a few extra tricks added in."
328 |    ]
329 |   },
330 |   {
331 |    "cell_type": "code",
332 |    "execution_count": null,
333 |    "metadata": {
334 |     "slideshow": {
335 |      "slide_type": "subslide"
336 |     }
337 |    },
338 |    "outputs": [],
339 |    "source": [
340 |     "x = numpy.linspace(-1, 1, 100)\n",
341 |     "y = numpy.linspace(-1, 1, 100)\n",
342 |     "X, Y = numpy.meshgrid(x, y)\n",
343 |     "\n",
344 |     "fig = plt.figure()\n",
345 |     "fig.set_figwidth(fig.get_figwidth() * 2)\n",
346 |     "\n",
347 |     "axes = fig.add_subplot(1, 2, 1, aspect='equal')\n",
348 |     "tribble_density = axes.pcolor(X, Y, numpy.sin(X**2) + numpy.cos(Y**2), cmap=plt.get_cmap(\"Oranges\"))\n",
349 |     "axes.set_xlabel(\"x (km)\")\n",
350 |     "axes.set_ylabel(\"y (km)\")\n",
351 |     "axes.set_title(\"Tribble Density ($N/km^2$)\")\n",
352 |     "cbar = fig.colorbar(tribble_density, ax=axes)\n",
353 |     "cbar.set_label(\"$1/km^2$\")\n",
354 |     "\n",
355 |     "axes = fig.add_subplot(1, 2, 2, aspect='equal')\n",
356 |     "klingon_population_density = axes.pcolor(X, Y, X * Y + 1, cmap=plt.get_cmap(\"RdBu\"))\n",
357 |     "axes.set_xlabel(\"x (km)\")\n",
358 |     "axes.set_ylabel(\"y (km)\")\n",
359 |     "axes.set_title(\"Tribble Birth Rate Anomaly\")\n",
360 |     "cbar = fig.colorbar(klingon_population_density, ax=axes)\n",
361 |     "cbar.set_label(\"Birth Rate Deviation\")\n",
362 |     "\n",
363 |     "plt.show()"
364 |    ]
365 |   },
366 |   {
367 |    "cell_type": "markdown",
368 |    "metadata": {
369 |     "slideshow": {
370 |      "slide_type": "slide"
371 |     }
372 |    },
373 |    "source": [
374 |     "Make sure to visit ([matplotlib.org](https://matplotlib.org/)) for all sorts of other hints, tricks, and documentation."
375 |    ]
376 |   }
377 |  ],
378 |  "metadata": {
379 |   "celltoolbar": "Slideshow",
380 |   "kernelspec": {
381 |    "display_name": "Python 3",
382 |    "language": "python",
383 |    "name": "python3"
384 |   },
385 |   "language_info": {
386 |    "codemirror_mode": {
387 |     "name": "ipython",
388 |     "version": 3
389 |    },
390 |    "file_extension": ".py",
391 |    "mimetype": "text/x-python",
392 |    "name": "python",
393 |    "nbconvert_exporter": "python",
394 |    "pygments_lexer": "ipython3",
395 |    "version": "3.7.5"
396 |   },
397 |   "latex_envs": {
398 |    "bibliofile": "biblio.bib",
399 |    "cite_by": "apalike",
400 |    "current_citInitial": 1,
401 |    "eqLabelWithNumbers": true,
402 |    "eqNumInitial": 0
403 |   }
404 |  },
405 |  "nbformat": 4,
406 |  "nbformat_minor": 1
407 | }
408 | 


--------------------------------------------------------------------------------
/10.1_SVD.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli and Rajath Kumar Mysore Pradeep Kumar</td>\n",
 14 |     "</table>\n",
 15 |     "\n",
 16 |     "Note:  This material largely follows the text \"Numerical Linear Algebra\" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there."
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": null,
 22 |    "metadata": {
 23 |     "slideshow": {
 24 |      "slide_type": "skip"
 25 |     }
 26 |    },
 27 |    "outputs": [],
 28 |    "source": [
 29 |     "from __future__ import print_function\n",
 30 |     "\n",
 31 |     "%matplotlib inline\n",
 32 |     "import numpy\n",
 33 |     "import matplotlib.pyplot as plt"
 34 |    ]
 35 |   },
 36 |   {
 37 |    "cell_type": "markdown",
 38 |    "metadata": {
 39 |     "slideshow": {
 40 |      "slide_type": "slide"
 41 |     }
 42 |    },
 43 |    "source": [
 44 |     "## Singular Value Decomposition\n",
 45 |     "\n",
 46 |     "Definition: Let $A \\in \\mathbb R^{m \\times n}$, then $A$ can be factored as\n",
 47 |     "$$\n",
 48 |     "    A = U\\Sigma V^{T}\n",
 49 |     "$$\n",
 50 |     "\n",
 51 |     "where,\n",
 52 |     "* $U \\in \\mathbb R^{m \\times m}$ and is the orthogonal matrix whose columns are the eigenvectors of $AA^{T}$\n",
 53 |     "* $V \\in \\mathbb R^{n \\times n}$ and is the orthogonal matrix whose columns are the eigenvectors of $A^{T}A$\n",
 54 |     "* $\\Sigma \\in \\mathbb R^{m \\times n}$ and is a diagonal matrix with elements $\\sigma_{1}, \\sigma_{2}, \\sigma_{3}, ... \\sigma_{r}$ where $r = rank(A)$ corresponding to the square roots of the eigenvalues of $A^{T}A$. They are called the singular values of $A$ and are non negative arranged in descending order. ($\\sigma_{1} \\geq \\sigma_{2} \\geq \\sigma_{3} \\geq ... \\sigma_{r} \\geq 0$).\n",
 55 |     "\n",
 56 |     "The SVD has a number of applications mostly related to reducing the dimensionality of a matrix."
 57 |    ]
 58 |   },
 59 |   {
 60 |    "cell_type": "markdown",
 61 |    "metadata": {
 62 |     "slideshow": {
 63 |      "slide_type": "subslide"
 64 |     }
 65 |    },
 66 |    "source": [
 67 |     "### Full SVD example\n",
 68 |     "\n",
 69 |     "Consider the matrix\n",
 70 |     "$$\n",
 71 |     "    A = \\begin{bmatrix} \n",
 72 |     "        2 & 0 & 3 \\\\\n",
 73 |     "        5 & 7 & 1 \\\\\n",
 74 |     "        0 & 6 & 2 \n",
 75 |     "    \\end{bmatrix}.\n",
 76 |     "$$\n",
 77 |     "Confirm the SVD representation using `numpy` functions as appropriate."
 78 |    ]
 79 |   },
 80 |   {
 81 |    "cell_type": "code",
 82 |    "execution_count": null,
 83 |    "metadata": {
 84 |     "slideshow": {
 85 |      "slide_type": "skip"
 86 |     }
 87 |    },
 88 |    "outputs": [],
 89 |    "source": [
 90 |     "A = numpy.array([\n",
 91 |     "    [2.0, 0.0, 3.0],\n",
 92 |     "    [5.0, 7.0, 1.0],\n",
 93 |     "    [0.0, 6.0, 2.0]\n",
 94 |     "])\n",
 95 |     "\n",
 96 |     "U, sigma, V_T = numpy.linalg.svd(A, full_matrices=True)\n",
 97 |     "print(numpy.dot(U, numpy.dot(numpy.diag(sigma), V_T)))"
 98 |    ]
 99 |   },
100 |   {
101 |    "cell_type": "markdown",
102 |    "metadata": {
103 |     "slideshow": {
104 |      "slide_type": "subslide"
105 |     }
106 |    },
107 |    "source": [
108 |     "### Eigenvalue Decomposition vs. SVD Decomposition\n",
109 |     "\n",
110 |     "Let the matrix $X$ contain the eigenvectors of $A$ which are linearly independent, then we can write a decomposition of the matrix $A$ as\n",
111 |     "$$\n",
112 |     "    A = X \\Lambda X^{-1}.\n",
113 |     "$$\n",
114 |     "\n",
115 |     "How does this differ from the SVD?\n",
116 |     " - The basis of the SVD representation differs from the eigenvalue decomposition\n",
117 |     "   - The basis vectors are not in general orthogonal for the eigenvalue decomposition where it is for the SVD\n",
118 |     "   - The SVD effectively contains two basis sets.\n",
119 |     " - All matrices have an SVD decomposition whereas not all have eigenvalue decompositions."
120 |    ]
121 |   },
122 |   {
123 |    "cell_type": "markdown",
124 |    "metadata": {
125 |     "slideshow": {
126 |      "slide_type": "subslide"
127 |     }
128 |    },
129 |    "source": [
130 |     "### Existence and Uniqueness\n",
131 |     "\n",
132 |     "Every matrix $A \\in \\mathbb{C}^{m \\times n}$ has a singular value decomposition. Furthermore, the singular values $\\{\\sigma_{j}\\}$ are uniquely determined, and if $A$ is square and the $\\sigma_{j}$ are distinct, the left and right singular vectors $\\{u_{j}\\}$ and $\\{v_{j}\\}$ are uniquely determined up to complex signs (i.e., complex scalar factors of absolute value 1)."
133 |    ]
134 |   },
135 |   {
136 |    "cell_type": "markdown",
137 |    "metadata": {
138 |     "slideshow": {
139 |      "slide_type": "subslide"
140 |     }
141 |    },
142 |    "source": [
143 |     "### Matrix Properties via the SVD\n",
144 |     "\n",
145 |     " - The $\\text{rank}(A) = r$ where $r$ is the number of non-zero singular values.\n",
146 |     " - The $\\text{range}(A) = [u_1, ... , u_r]$ and $\\text{null}(a) = [v_{r+1}, ... , v_n]$.\n",
147 |     " - The $|| A ||_2 = \\sigma_1$ and $||A||_F = \\sqrt{\\sigma_{1}^{2}+\\sigma_{2}^{2}+...+\\sigma_{r}^{2}}$.\n",
148 |     " - The nonzero singular values of A are the square roots of the nonzero eigenvalues of $A^{T}A$ or $AA^{T}$.\n",
149 |     " - If $A = A^{T}$, then the singular values of $A$ are the absolute values of the eigenvalues of $A$.\n",
150 |     " - For $A \\in \\mathbb{C}^{m \\times n}$ then $|det(A)| = \\Pi_{i=1}^{m} \\sigma_{i}$"
151 |    ]
152 |   },
153 |   {
154 |    "cell_type": "markdown",
155 |    "metadata": {
156 |     "slideshow": {
157 |      "slide_type": "subslide"
158 |     }
159 |    },
160 |    "source": [
161 |     "### Low-Rank Approximations\n",
162 |     "\n",
163 |     " - $A$ is the sum of the $r$ rank-one matrices:\n",
164 |     "$$\n",
165 |     "    A = U \\Sigma V^T = \\sum_{j=1}^{r} \\sigma_{j}u_{j}v_{j}^{T}\n",
166 |     "$$\n",
167 |     " - For any $k$ with $0 \\leq k \\leq r$, define\n",
168 |     "$$\n",
169 |     "    A = \\sum_{j=1}^{k} \\sigma_{j}u_{j}v_{j}^{T}\n",
170 |     "$$\n",
171 |     "Let $k = min(m,n)$, then\n",
172 |     "\n",
173 |     "$$\n",
174 |     "    ||A - A_{v}||_{2} = \\text{inf}_{B \\in \\mathbb{C}^{m \\times n}} \\text{rank}(B)\\leq k|| A-B||_{2} = \\sigma_{k+1}\n",
175 |     "$$\n",
176 |     "\n",
177 |     "- For any $k$ with $0 \\leq k \\leq r$, the matrix $A_{k}$ also satisfies\n",
178 |     "$$\n",
179 |     "    ||A - A_{v}||_{F} = \\text{inf}_{B \\in \\mathbb{C}^{m \\times n}} \\text{rank}(B)\\leq v ||A-B||_{F} = \\sqrt{\\sigma_{v+1}^{2} + ... + \\sigma_{r}^{2}}\n",
180 |     "$$"
181 |    ]
182 |   },
183 |   {
184 |    "cell_type": "markdown",
185 |    "metadata": {
186 |     "slideshow": {
187 |      "slide_type": "subslide"
188 |     }
189 |    },
190 |    "source": [
191 |     "#### Example:   \"Hello World\"\n",
192 |     "\n",
193 |     "How does this work in practice?"
194 |    ]
195 |   },
196 |   {
197 |    "cell_type": "code",
198 |    "execution_count": null,
199 |    "metadata": {
200 |     "slideshow": {
201 |      "slide_type": "skip"
202 |     }
203 |    },
204 |    "outputs": [],
205 |    "source": [
206 |     "data = numpy.zeros((15,40))\n",
207 |     "\n",
208 |     "#H\n",
209 |     "data[2:10,2:4] = 1\n",
210 |     "data[5:7,4:6] = 1\n",
211 |     "data[2:10,6:8] = 1\n",
212 |     "\n",
213 |     "#E\n",
214 |     "data[3:11,10:12] = 1\n",
215 |     "data[3:5,12:16] = 1\n",
216 |     "data[6:8, 12:16] = 1\n",
217 |     "data[9:11, 12:16] = 1\n",
218 |     "\n",
219 |     "#L\n",
220 |     "data[4:12,18:20] = 1\n",
221 |     "data[10:12,20:24] = 1\n",
222 |     "\n",
223 |     "#L\n",
224 |     "data[5:13,26:28] = 1\n",
225 |     "data[11:13,28:32] = 1\n",
226 |     "\n",
227 |     "#0\n",
228 |     "data[6:14,34:36] = 1\n",
229 |     "data[6:8, 36:38] = 1\n",
230 |     "data[12:14, 36:38] = 1\n",
231 |     "data[6:14,38:40] = 1\n",
232 |     "\n",
233 |     "plt.imshow(data)\n",
234 |     "plt.show()"
235 |    ]
236 |   },
237 |   {
238 |    "cell_type": "code",
239 |    "execution_count": null,
240 |    "metadata": {
241 |     "scrolled": false,
242 |     "slideshow": {
243 |      "slide_type": "skip"
244 |     }
245 |    },
246 |    "outputs": [],
247 |    "source": [
248 |     "u, diag, vt = numpy.linalg.svd(data, full_matrices=True)\n",
249 |     "fig = plt.figure()\n",
250 |     "fig.set_figwidth(fig.get_figwidth() * 3)\n",
251 |     "fig.set_figheight(fig.get_figheight() * 4)\n",
252 |     "for i in range(1, 16):\n",
253 |     "    diag_matrix = numpy.concatenate((numpy.zeros((len(diag[:i]) -1),), diag[i-1: i], numpy.zeros((40-i),)))\n",
254 |     "    reconstruct = numpy.dot(numpy.dot(u, numpy.diag(diag_matrix)[:15,]), vt)\n",
255 |     "    \n",
256 |     "    axes = fig.add_subplot(5, 3, i)\n",
257 |     "    mappable = axes.imshow(reconstruct, vmin=0.0, vmax=1.0)\n",
258 |     "    axes.set_title('Component = %s' % i)\n",
259 |     "    \n",
260 |     "plt.show()"
261 |    ]
262 |   },
263 |   {
264 |    "cell_type": "code",
265 |    "execution_count": null,
266 |    "metadata": {
267 |     "scrolled": true,
268 |     "slideshow": {
269 |      "slide_type": "skip"
270 |     }
271 |    },
272 |    "outputs": [],
273 |    "source": [
274 |     "u, diag, vt = numpy.linalg.svd(data, full_matrices=True)\n",
275 |     "fig = plt.figure()\n",
276 |     "fig.set_figwidth(fig.get_figwidth() * 3)\n",
277 |     "fig.set_figheight(fig.get_figheight() * 4)\n",
278 |     "for i in range(1, 16):\n",
279 |     "    diag_matrix = numpy.concatenate((diag[:i], numpy.zeros((40-i),)))\n",
280 |     "    reconstruct = numpy.dot(numpy.dot(u, numpy.diag(diag_matrix)[:15,]), vt)\n",
281 |     "\n",
282 |     "    axes = fig.add_subplot(5, 3, i)\n",
283 |     "    mappable = axes.imshow(reconstruct, vmin=0.0, vmax=1.0)\n",
284 |     "    axes.set_title('Component = %s' % i)\n",
285 |     "    \n",
286 |     "plt.show()"
287 |    ]
288 |   }
289 |  ],
290 |  "metadata": {
291 |   "celltoolbar": "Slideshow",
292 |   "kernelspec": {
293 |    "display_name": "Python 3",
294 |    "language": "python",
295 |    "name": "python3"
296 |   },
297 |   "language_info": {
298 |    "codemirror_mode": {
299 |     "name": "ipython",
300 |     "version": 3
301 |    },
302 |    "file_extension": ".py",
303 |    "mimetype": "text/x-python",
304 |    "name": "python",
305 |    "nbconvert_exporter": "python",
306 |    "pygments_lexer": "ipython3",
307 |    "version": "3.7.1"
308 |   }
309 |  },
310 |  "nbformat": 4,
311 |  "nbformat_minor": 2
312 | }
313 | 


--------------------------------------------------------------------------------
/12_LA_conditioning_stability.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 14 |     "</table>\n",
 15 |     "\n",
 16 |     "Note:  This material largely follows the text \"Numerical Linear Algebra\" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there."
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": null,
 22 |    "metadata": {
 23 |     "init_cell": true,
 24 |     "slideshow": {
 25 |      "slide_type": "skip"
 26 |     }
 27 |    },
 28 |    "outputs": [],
 29 |    "source": [
 30 |     "from __future__ import print_function\n",
 31 |     "\n",
 32 |     "%matplotlib inline\n",
 33 |     "import numpy\n",
 34 |     "import matplotlib.pyplot as plt"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {
 40 |     "slideshow": {
 41 |      "slide_type": "slide"
 42 |     }
 43 |    },
 44 |    "source": [
 45 |     "# Conditioning and Stability\n",
 46 |     "\n",
 47 |     "Once an approximation to a linear system is constructed the next question is how much trust can we put in the approximation? Since the true solution is not known, one of the few tools we have is to ask how well the approximation matches the original equation. In other words, we seek a solution to a system,\n",
 48 |     "$$\n",
 49 |     "    \\vec{f}(\\vec{x}) = \\vec{b}. \n",
 50 |     "$$\n",
 51 |     "\n",
 52 |     "We do not have $\\vec{x}$ but instead have an approximation, $\\hat{x}$, and we hope that \n",
 53 |     "$$\n",
 54 |     "    \\vec{f}(\\hat{x}) \\approx \\vec{b}.\n",
 55 |     "$$\n",
 56 |     "In this section the question we explore is to try to determine a bound on the relative error, $\\frac{||\\vec{x}-\\hat{x}||}{||\\vec{x}||}$ given the matrix, $A$. \n",
 57 |     "\n",
 58 |     "This leads to the notion of conditioning. Conditioning is the behavior of a problem when the solution is a changed a small bit (perturbed), and it is a  mathematical (analytic) property of the original system of equations.  Stability, on the other hand, is concerned with how the algorithm used to obtain an approximation behaves when the approximation is perturbed."
 59 |    ]
 60 |   },
 61 |   {
 62 |    "cell_type": "markdown",
 63 |    "metadata": {
 64 |     "slideshow": {
 65 |      "slide_type": "slide"
 66 |     }
 67 |    },
 68 |    "source": [
 69 |     "## Conditioning and Condition Numbers\n",
 70 |     "\n",
 71 |     "A **well-conditioned** problem is one where a small perturbation to the original problem leads to only small changes in the solution."
 72 |    ]
 73 |   },
 74 |   {
 75 |    "cell_type": "markdown",
 76 |    "metadata": {
 77 |     "slideshow": {
 78 |      "slide_type": "subslide"
 79 |     }
 80 |    },
 81 |    "source": [
 82 |     "Formally we can think of a function $f$ which maps $x$ to $y$\n",
 83 |     "\n",
 84 |     "$$\n",
 85 |     "    f(x) = y \\quad \\text{or} \\quad f: X \\rightarrow Y.\n",
 86 |     "$$\n",
 87 |     "\n",
 88 |     "Let $x \\in X$ where we perturb $x$ with $\\delta x$ and we ask how the result $y$ changes:\n",
 89 |     "\n",
 90 |     "$$\n",
 91 |     "    ||f(x) - f(x + \\delta x)|| \\leq C ||x - (x+\\delta x)||\n",
 92 |     "$$\n",
 93 |     "\n",
 94 |     "for some constant $C$ possible dependent on $\\delta x$ depending on the type of conditioning we are considering."
 95 |    ]
 96 |   },
 97 |   {
 98 |    "cell_type": "markdown",
 99 |    "metadata": {
100 |     "slideshow": {
101 |      "slide_type": "subslide"
102 |     }
103 |    },
104 |    "source": [
105 |     "### Absolute Condition Number\n",
106 |     "\n",
107 |     "If we let $\\delta x$ be the small perturbation to the input and $\\delta f = f(x + \\delta x) - f(x)$ be the result the **absolute condition number** $\\hat{~\\kappa}$ can be defined as\n",
108 |     "\n",
109 |     "$$\n",
110 |     "    \\hat{\\!\\kappa} = \\sup_{\\delta x} \\frac{||\\delta f||}{||\\delta x||}\n",
111 |     "$$\n",
112 |     "\n",
113 |     "for most problems (assuming $\\delta f$ and $\\delta x$ are both infinitesimal).  "
114 |    ]
115 |   },
116 |   {
117 |    "cell_type": "markdown",
118 |    "metadata": {
119 |     "slideshow": {
120 |      "slide_type": "subslide"
121 |     }
122 |    },
123 |    "source": [
124 |     "When $f$ is differentiable we can evaluate the condition number via the Jacobian.  Recall that the derivative of a multi-valued function can be termed in the form of a Jacobian $J(x)$ where\n",
125 |     "$$\n",
126 |     "    [J(x)]_{ij} = \\frac{\\partial f_i}{\\partial x_j}(x).\n",
127 |     "$$"
128 |    ]
129 |   },
130 |   {
131 |    "cell_type": "markdown",
132 |    "metadata": {
133 |     "slideshow": {
134 |      "slide_type": "subslide"
135 |     }
136 |    },
137 |    "source": [
138 |     "This allows us to write the infinitesimal $\\delta f$ as\n",
139 |     "$$\n",
140 |     "    \\delta f \\approx J(x) \\delta x\n",
141 |     "$$\n",
142 |     "with equality when $||\\delta x|| \\rightarrow 0$.  Then we can write the condition number as\n",
143 |     "$$\n",
144 |     "    \\hat{\\!\\kappa} = ||J(x)||\n",
145 |     "$$\n",
146 |     "where the norm is the one induced by the spaces $X$ and $Y$."
147 |    ]
148 |   },
149 |   {
150 |    "cell_type": "markdown",
151 |    "metadata": {
152 |     "slideshow": {
153 |      "slide_type": "subslide"
154 |     }
155 |    },
156 |    "source": [
157 |     "### Relative Condition Number\n",
158 |     "\n",
159 |     "The **relative condition number** is defined similarly and is related to the difference before between the absolute error and relative error as defined previously.  With the same caveats as before it can be defined as\n",
160 |     "$$\n",
161 |     "    \\kappa = \\sup_{\\delta x} \\left( \\frac{\\frac{||\\delta f||}{||f(x)||}}{\\frac{||\\delta x||}{||x||}} \\right).\n",
162 |     "$$"
163 |    ]
164 |   },
165 |   {
166 |    "cell_type": "markdown",
167 |    "metadata": {
168 |     "slideshow": {
169 |      "slide_type": "subslide"
170 |     }
171 |    },
172 |    "source": [
173 |     "Again if $f$ is differentiable we can use the Jacobian $J(x)$ to evaluate the relative condition number as\n",
174 |     "$$\n",
175 |     "    \\kappa = \\frac{||J(x)||}{||f(x)|| ~/ ~||x||}.\n",
176 |     "$$"
177 |    ]
178 |   },
179 |   {
180 |    "cell_type": "markdown",
181 |    "metadata": {
182 |     "slideshow": {
183 |      "slide_type": "subslide"
184 |     }
185 |    },
186 |    "source": [
187 |     "#### Examples\n",
188 |     "Calculate the following relative condition numbers of the following problems.\n",
189 |     "\n",
190 |     "$\\sqrt{x}$ for $x > 0$. "
191 |    ]
192 |   },
193 |   {
194 |    "cell_type": "markdown",
195 |    "metadata": {
196 |     "slideshow": {
197 |      "slide_type": "subslide"
198 |     }
199 |    },
200 |    "source": [
201 |     "$$\n",
202 |     "    f(x) = \\sqrt{x}, \\quad J(x) = f'(x) = \\frac{1}{2 \\sqrt{x}} \\\\\n",
203 |     "    \\kappa = \\frac{||J(x)||}{||f(x)|| / ||x||} = \\frac{1}{2 \\sqrt{x}} \\frac{x}{\\sqrt{x}} = \\frac{1}{2}\n",
204 |     "$$"
205 |    ]
206 |   },
207 |   {
208 |    "cell_type": "markdown",
209 |    "metadata": {
210 |     "slideshow": {
211 |      "slide_type": "subslide"
212 |     }
213 |    },
214 |    "source": [
215 |     "Calculate the relative condition number for the scalar function $f(x) = x_1 - x_2$ using the vector $\\vec{x} = (x_1, x_2)^T \\in \\mathbb R^2$ using an $\\ell_\\infty$ norm."
216 |    ]
217 |   },
218 |   {
219 |    "cell_type": "markdown",
220 |    "metadata": {
221 |     "slideshow": {
222 |      "slide_type": "subslide"
223 |     }
224 |    },
225 |    "source": [
226 |     "$$\n",
227 |     "    f(x) = x_1 - x_2, \\quad J(x) = \\left [ \\frac{\\partial f}{\\partial x_1}, \\frac{\\partial f}{\\partial x_2}\\right ] = [1, -1] \\\\\n",
228 |     "    \\kappa = \\frac{||J(x)||_\\infty}{||f(x)||_\\infty / ||x||_\\infty} = \\frac{2 \\max_{i=1,2} |x_i|}{|x_1 - x_2|}\n",
229 |     "$$\n",
230 |     "where\n",
231 |     "$$\n",
232 |     "    ||J||_\\infty = 2.\n",
233 |     "$$"
234 |    ]
235 |   },
236 |   {
237 |    "cell_type": "markdown",
238 |    "metadata": {
239 |     "slideshow": {
240 |      "slide_type": "subslide"
241 |     }
242 |    },
243 |    "source": [
244 |     "The condition number of a function was discussed in general terms above. Now, the more specific case of a linear function, a matrix-vector multiplication, is examined. Here we let $\\vec{f}(\\vec{x})=Ax$ and determine the condition number by perturbing $x$."
245 |    ]
246 |   },
247 |   {
248 |    "cell_type": "markdown",
249 |    "metadata": {
250 |     "slideshow": {
251 |      "slide_type": "subslide"
252 |     }
253 |    },
254 |    "source": [
255 |     "We begin with the definition above,\n",
256 |     "$$\\begin{aligned}\n",
257 |     "    \\kappa &= \\sup_{\\delta x} \\left ( \\frac{||A (\\vec{x}+\\delta x) - A \\vec{x}||}{||A\\vec{x}||} \\frac{||\\vec{x}||}{||\\delta x||}\\right ), \\\\\n",
258 |     "    &= \\sup_{\\delta x} \\frac{ ||A \\delta x||}{||\\delta x||} \\frac{||\\vec{x}||}{||A\\vec{x}||}, \\\\\n",
259 |     "    &= ||A|| \\frac{||\\vec{x}||}{||A \\vec{x}||},\n",
260 |     "\\end{aligned}$$\n",
261 |     "where $\\delta x$ is a vector.\n",
262 |     "\n",
263 |     "If $A$ has an inverse, then we note that\n",
264 |     "$$\n",
265 |     "\\begin{align}\n",
266 |     " \\vec{x} &= A^{-1}A \\vec{x}, \\\\\n",
267 |     " \\Rightarrow ||\\vec{x}|| &= || A^{-1}A \\vec{x} ||, \\\\\n",
268 |     "                         &\\leq ||A^{-1}|| || A \\vec{x} ||,\n",
269 |     "\\end{align}\n",
270 |     "$$\n",
271 |     "which implies that\n",
272 |     "$$\n",
273 |     "    \\frac{||x||}{||A x||} \\leq ||A^{-1}||.\n",
274 |     "$$\n",
275 |     "We can now bound the condition number for a matrix by\n",
276 |     "$$\n",
277 |     "    \\kappa \\leq ||A|| ||A^{-1}||.\n",
278 |     "$$"
279 |    ]
280 |   },
281 |   {
282 |    "cell_type": "markdown",
283 |    "metadata": {
284 |     "slideshow": {
285 |      "slide_type": "subslide"
286 |     }
287 |    },
288 |    "source": [
289 |     "### Condition Number of a Matrix\n",
290 |     "\n",
291 |     "The condition number of a matrix is defined by the product\n",
292 |     "$$\n",
293 |     "    \\kappa(A) = ||A||~||A^{-1}||.\n",
294 |     "$$\n",
295 |     "where here we are thinking about the matrix rather than a problem.  If $\\kappa$ is small than $A$ is said to be **well-conditioned**.  If $A$ is singular we assign $\\kappa(A) = \\infty$ as the matrix's condition number."
296 |    ]
297 |   },
298 |   {
299 |    "cell_type": "markdown",
300 |    "metadata": {
301 |     "slideshow": {
302 |      "slide_type": "subslide"
303 |     }
304 |    },
305 |    "source": [
306 |     "When we are considering the $\\ell_2$ norm then we can write the condition number as\n",
307 |     "\n",
308 |     "$$\n",
309 |     "    \\kappa(A) = \\frac{\\sqrt{\\rho(A^\\ast A)}}{\\sqrt{\\rho((A^\\ast A)^{-1})}} = \\frac{\\sqrt{\\max |\\lambda|}}{\\sqrt{\\min |\\lambda|}}.\n",
310 |     "$$"
311 |    ]
312 |   },
313 |   {
314 |    "cell_type": "markdown",
315 |    "metadata": {
316 |     "slideshow": {
317 |      "slide_type": "subslide"
318 |     }
319 |    },
320 |    "source": [
321 |     "### Condition Number of a System of Equations\n",
322 |     "\n",
323 |     "Another way to think about the conditioning of a problem we have looked at before is that the matrix $A$ itself is an input to the problem.  Consider than the system of equations $A\\vec{x} = \\vec{b}$ where we will perturb both $A$ and $\\vec{x}$ resulting in\n",
324 |     "$$\n",
325 |     "    (A + \\delta A)(\\vec{x} + \\delta x) = \\vec{b}.\n",
326 |     "$$"
327 |    ]
328 |   },
329 |   {
330 |    "cell_type": "markdown",
331 |    "metadata": {
332 |     "slideshow": {
333 |      "slide_type": "subslide"
334 |     }
335 |    },
336 |    "source": [
337 |     "Assuming we solve the problem exactly we know that $A\\vec{x} = \\vec{b}$ and that the infinitesimals multiplied $\\delta A \\delta x$ are smaller than the other term, and the above expression can be approximation by\n",
338 |     "$$\n",
339 |     "\\begin{aligned}\n",
340 |     "    (A + \\delta A)(\\vec{x} + \\delta x) &= \\vec{b}, \\\\\n",
341 |     "    A\\vec{x} + \\delta Ax + A \\delta x + \\delta A \\delta \\vec{x} &= \\vec{b} \\\\\n",
342 |     "    \\delta A\\vec{x} + A \\delta x & = 0\n",
343 |     "\\end{aligned}\n",
344 |     "$$"
345 |    ]
346 |   },
347 |   {
348 |    "cell_type": "markdown",
349 |    "metadata": {
350 |     "slideshow": {
351 |      "slide_type": "subslide"
352 |     }
353 |    },
354 |    "source": [
355 |     "Solving for $\\delta x$ leads to \n",
356 |     "$$\n",
357 |     "    \\delta x = -A^{-1} \\delta A \\vec{x}\n",
358 |     "$$\n",
359 |     "implying\n",
360 |     "$$\n",
361 |     "    ||\\delta x|| \\leq ||A^{-1}|| ~ ||\\delta A|| ~ ||\\vec{x}||\n",
362 |     "$$\n",
363 |     "and therefore\n",
364 |     "$$\n",
365 |     "    \\frac{\\frac{||\\delta x||}{||\\vec{x}||}}{\\frac{||\\delta A||}{||A||}} \\leq ||A^{-1}||~||A|| = \\kappa(A).\n",
366 |     "$$"
367 |    ]
368 |   },
369 |   {
370 |    "cell_type": "markdown",
371 |    "metadata": {
372 |     "slideshow": {
373 |      "slide_type": "subslide"
374 |     }
375 |    },
376 |    "source": [
377 |     "We can also say the following regarding the condition number of a system of equations then\n",
378 |     "\n",
379 |     "**Theorem:**  Let $\\vec{b}$ be fixed and consider the problem of computing $\\vec{x}$ in $A\\vec{x} = \\vec{b}$ where $A$ is square and non-singular.  The condition number of this problem with respect to perturbations in $A$ is the condition number of the matrix $\\kappa(A)$."
380 |    ]
381 |   },
382 |   {
383 |    "cell_type": "markdown",
384 |    "metadata": {
385 |     "slideshow": {
386 |      "slide_type": "slide"
387 |     }
388 |    },
389 |    "source": [
390 |     "## Stability\n",
391 |     "\n",
392 |     "We now return to the consideration of the fact that we are interested not only in the well-conditioning of a mathematical problem but in how we might solve it on a finite precision machine.  In some sense conditioning describes how well we can solve a problem in exact arithmetic and stability how well we can solve the problem in finite arithmetic.  "
393 |    ]
394 |   },
395 |   {
396 |    "cell_type": "markdown",
397 |    "metadata": {
398 |     "slideshow": {
399 |      "slide_type": "subslide"
400 |     }
401 |    },
402 |    "source": [
403 |     "### Accuracy and Stability\n",
404 |     "\n",
405 |     "As we have defined before we will consider **absolute error** as\n",
406 |     "$$\n",
407 |     "    ||F(x) - f(x)||\n",
408 |     "$$\n",
409 |     "where $F(x)$ is the approximation to the true solution $f(x)$.  Similarly we can define **relative error** as\n",
410 |     "$$\n",
411 |     "    \\frac{||F(x) - f(x)||}{||f(x)||}.\n",
412 |     "$$\n",
413 |     "In the ideal case we would like the relative error to be $\\mathcal{O}(\\epsilon_{\\text{machine}})$."
414 |    ]
415 |   },
416 |   {
417 |    "cell_type": "markdown",
418 |    "metadata": {
419 |     "slideshow": {
420 |      "slide_type": "subslide"
421 |     }
422 |    },
423 |    "source": [
424 |     "#### Forwards Stability\n",
425 |     "\n",
426 |     "A **forward stable** algorithm for $x \\in X$ has\n",
427 |     "\n",
428 |     "$$\n",
429 |     "    \\frac{||F(x) - f(x)||}{||f(x)||} = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
430 |     "$$\n",
431 |     "\n",
432 |     "In other words\n",
433 |     "> A forward stable algorithm gives almost the right answer to exactly the right question."
434 |    ]
435 |   },
436 |   {
437 |    "cell_type": "markdown",
438 |    "metadata": {
439 |     "slideshow": {
440 |      "slide_type": "subslide"
441 |     }
442 |    },
443 |    "source": [
444 |     "#### Backwards Stability\n",
445 |     "\n",
446 |     "A stronger notion of stability can also be defined which is satisfied by many approaches in numerical linear algebra.  We say that an algorithm $F$ is **backward stable** if for $x \\in X$ we have\n",
447 |     "\n",
448 |     "$$\n",
449 |     "    F(x) = f(\\hat{\\!x})\n",
450 |     "$$\n",
451 |     "\n",
452 |     "for some $\\hat{\\!x}$ with\n",
453 |     "\n",
454 |     "$$\n",
455 |     "    \\frac{||\\hat{\\!x} - x||}{||x||} = \\mathcal{O}(\\epsilon_{\\text{machine}}).\n",
456 |     "$$\n",
457 |     "\n",
458 |     "In other words\n",
459 |     "> A backward stable algorithm gives exactly the right answer to nearly the right question."
460 |    ]
461 |   },
462 |   {
463 |    "cell_type": "markdown",
464 |    "metadata": {
465 |     "slideshow": {
466 |      "slide_type": "subslide"
467 |     }
468 |    },
469 |    "source": [
470 |     "Combining these ideas along with the idea that we should not expect to be able to accurately compute the solution to a poorly conditioned problem we can form the mixed forward-backward sense of stability as for $x \\in X$ if\n",
471 |     "\n",
472 |     "$$\n",
473 |     "    \\frac{||F(x) - f(\\hat{\\!x})||}{||f(\\hat{\\!x})||} = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
474 |     "$$\n",
475 |     "\n",
476 |     "for some $\\hat{\\!x}$ with \n",
477 |     "\n",
478 |     "$$\n",
479 |     "    \\frac{||\\hat{\\!x} - x||}{||x||} = \\mathcal{O}(\\epsilon_{\\text{machine}}).\n",
480 |     "$$\n",
481 |     "\n",
482 |     "In other words\n",
483 |     "> A stable algorithm gives nearly the right answer to nearly the right question."
484 |    ]
485 |   },
486 |   {
487 |    "cell_type": "markdown",
488 |    "metadata": {
489 |     "slideshow": {
490 |      "slide_type": "subslide"
491 |     }
492 |    },
493 |    "source": [
494 |     "An important aspect of the above statement is that we can not necessarily guarantee an accurate result.  If the condition number $\\kappa(x)$ is small we would expect that a stable algorithm would give us an accurate result (by definition).  This is reflected in the following theorem."
495 |    ]
496 |   },
497 |   {
498 |    "cell_type": "markdown",
499 |    "metadata": {
500 |     "slideshow": {
501 |      "slide_type": "subslide"
502 |     }
503 |    },
504 |    "source": [
505 |     "**Theorem:**  Suppose a backward stable algorithm is applied to solve a problem $f: X \\rightarrow Y$ with condition number $\\kappa$ on a finite precision machine, then the relative errors satisfy\n",
506 |     "$$\n",
507 |     "    \\frac{||F(x) - f(\\hat{\\!x})||}{||f(\\hat{\\!x})||} = \\mathcal{O}(\\kappa(x) ~ \\epsilon_{\\text{machine}}).\n",
508 |     "$$"
509 |    ]
510 |   },
511 |   {
512 |    "cell_type": "markdown",
513 |    "metadata": {
514 |     "slideshow": {
515 |      "slide_type": "subslide"
516 |     }
517 |    },
518 |    "source": [
519 |     "**Proof:**  By the definition of the condition number of a problem we can write\n",
520 |     "$$\n",
521 |     "    \\frac{||F(x) - f(\\hat{\\!x})||}{||f(\\hat{\\!x})||} \\leq (\\kappa(x) + \\mathcal{O}(\\epsilon_{\\text{machine}}))\\frac{||\\hat{\\!x} - x||}{||x||}.\n",
522 |     "$$\n",
523 |     "Combining this with the definition of backwards stability we can arrive at the statement of the theorem."
524 |    ]
525 |   },
526 |   {
527 |    "cell_type": "markdown",
528 |    "metadata": {
529 |     "slideshow": {
530 |      "slide_type": "subslide"
531 |     }
532 |    },
533 |    "source": [
534 |     "**Backward Error Analysis** - Process of using the condition number of the problem and stability of the algorithm to determine the error.\n",
535 |     "\n",
536 |     "**Forward Error Analysis** - Considers the accrual of error at each step of an algorithm given slightly perturbed input."
537 |    ]
538 |   },
539 |   {
540 |    "cell_type": "markdown",
541 |    "metadata": {
542 |     "slideshow": {
543 |      "slide_type": "subslide"
544 |     }
545 |    },
546 |    "source": [
547 |     "### Stability of $A\\vec{x} = \\vec{b}$ using Householder Triangularization\n",
548 |     "\n",
549 |     "As an example lets consider the conditioning and algorithm for solving $A\\vec{x} = \\vec{b}$.  Here we will use a $QR$ factorization approach to solve $A\\vec{x} = \\vec{b}$ given by a Householder triangularization.  First off lets discuss the $QR$ factorization itself."
550 |    ]
551 |   },
552 |   {
553 |    "cell_type": "markdown",
554 |    "metadata": {
555 |     "slideshow": {
556 |      "slide_type": "subslide"
557 |     }
558 |    },
559 |    "source": [
560 |     "**Theorem:** Let the $QR$ factorization $A = QR$ of a matrix $A \\in \\mathbb C^{m \\times n}$ be computed using a Householder triangularization approach on a finite precision machine, then\n",
561 |     "\n",
562 |     "$$\n",
563 |     "    \\hat{\\!Q} \\cdot \\hat{\\!R} = A + \\delta A \\quad \\frac{||\\delta A||}{||A||} = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
564 |     "$$\n",
565 |     "\n",
566 |     "for some $\\delta A \\in \\mathbb C^{m \\times n}$ where $\\hat{\\!Q}$ and $\\hat{\\!R}$ are the finite arithmetic versions of $Q$ and $R$.  Householder triangularization is therefore backward stable."
567 |    ]
568 |   },
569 |   {
570 |    "cell_type": "markdown",
571 |    "metadata": {
572 |     "slideshow": {
573 |      "slide_type": "subslide"
574 |     }
575 |    },
576 |    "source": [
577 |     "#### Solving $A\\vec{x} = \\vec{b}$ with $QR$ Factorization\n",
578 |     "\n",
579 |     "So Householder triangularization is backwards stable but we also know that this does not guarantee accuracy if the problem itself is ill-conditioned.  Is backward stability enough to guarantee accurate results if we use it for $A\\vec{x} = \\vec{b}$ for instance?  It turns out that the accuracy of the product of $QR$ is enough to guarantee accuracy of a larger algorithm."
580 |    ]
581 |   },
582 |   {
583 |    "cell_type": "markdown",
584 |    "metadata": {
585 |     "slideshow": {
586 |      "slide_type": "subslide"
587 |     }
588 |    },
589 |    "source": [
590 |     "Consider the steps to solving $A \\vec{x} = \\vec{b}$ using $QR$ factorization:\n",
591 |     "1. Compute the $QR$ factorization of $A$\n",
592 |     "1. Multiply the vector $\\vec{b}$ by $Q^\\ast$ so that $\\vec{y} = Q^\\ast \\vec{b}$.\n",
593 |     "1. Solve using backward-substitution the triangular system $R \\vec{x} = \\vec{y}$ or $\\vec{x} = R^{-1} \\vec{y}$."
594 |    ]
595 |   },
596 |   {
597 |    "cell_type": "markdown",
598 |    "metadata": {
599 |     "slideshow": {
600 |      "slide_type": "subslide"
601 |     }
602 |    },
603 |    "source": [
604 |     "We know that step (1) is backward stable, what about step (2), the matrix-vector multiplication?  We can write the estimate of the backwards stability as\n",
605 |     "\n",
606 |     "$$\n",
607 |     "    (\\hat{\\!Q} + \\delta Q) \\cdot \\hat{\\!y} = b \\quad \\text{with} \\quad ||\\delta Q|| = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
608 |     "$$\n",
609 |     "\n",
610 |     "where we have inverted the matrix $ \\hat{\\!Q}$ since it is unitary.  Since this is exact we know also that the matrix-vector multiplication is also backwards stable since this is an equivalent statement to multiplying $b$ by a slightly perturbed matrix."
611 |    ]
612 |   },
613 |   {
614 |    "cell_type": "markdown",
615 |    "metadata": {
616 |     "slideshow": {
617 |      "slide_type": "subslide"
618 |     }
619 |    },
620 |    "source": [
621 |     "Step (3) is backward substitution (or the computation of $R^{-1}$).  Writing the backwards stability estimate we have\n",
622 |     "\n",
623 |     "$$\n",
624 |     "    (\\hat{\\!R} + \\delta R) \\cdot \\hat{\\!x} = \\hat{\\!y} \\quad \\text{with} \\quad \\frac{||\\delta R||}{||\\hat{\\!R}||} = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
625 |     "$$\n",
626 |     "\n",
627 |     "demonstrating that the results $\\hat{\\!x}$ is the exact solution to a slight perturbation of the original problem."
628 |    ]
629 |   },
630 |   {
631 |    "cell_type": "markdown",
632 |    "metadata": {
633 |     "slideshow": {
634 |      "slide_type": "subslide"
635 |     }
636 |    },
637 |    "source": [
638 |     "These results lead to the following two theorems:\n",
639 |     "\n",
640 |     "**Theorem:**  Using $QR$ factorization to solve $A\\vec{x}=\\vec{b}$ as described above is backward stable, satisfying\n",
641 |     "$$\n",
642 |     "    (A + \\Delta A) ~ \\hat{\\!x} = \\vec{b}, \\quad \\frac{||\\Delta A||}{||A||} = \\mathcal{O}(\\epsilon_{\\text{machine}})\n",
643 |     "$$\n",
644 |     "for some $\\Delta A \\in \\mathbb C^{m \\times n}$.\n",
645 |     "\n",
646 |     "**Theorem:** The solution $\\hat{x}$ computed by the above algorithm satisfies\n",
647 |     "$$\n",
648 |     "    \\frac{||\\hat{\\!x} - \\vec{x}||}{||\\vec{x}||} = \\mathcal{O}(\\kappa(x) ~ \\epsilon_{\\text{machine}}).\n",
649 |     "$$"
650 |    ]
651 |   }
652 |  ],
653 |  "metadata": {
654 |   "celltoolbar": "Slideshow",
655 |   "kernelspec": {
656 |    "display_name": "Python 3",
657 |    "language": "python",
658 |    "name": "python3"
659 |   },
660 |   "language_info": {
661 |    "codemirror_mode": {
662 |     "name": "ipython",
663 |     "version": 3
664 |    },
665 |    "file_extension": ".py",
666 |    "mimetype": "text/x-python",
667 |    "name": "python",
668 |    "nbconvert_exporter": "python",
669 |    "pygments_lexer": "ipython3",
670 |    "version": "3.6.9"
671 |   },
672 |   "latex_envs": {
673 |    "bibliofile": "biblio.bib",
674 |    "cite_by": "apalike",
675 |    "current_citInitial": 1,
676 |    "eqLabelWithNumbers": true,
677 |    "eqNumInitial": 0
678 |   }
679 |  },
680 |  "nbformat": 4,
681 |  "nbformat_minor": 1
682 | }
683 | 


--------------------------------------------------------------------------------
/15_LA_gaussian.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 14 |     "</table>\n",
 15 |     "\n",
 16 |     "Note:  This material largely follows the text \"Numerical Linear Algebra\" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there."
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": null,
 22 |    "metadata": {
 23 |     "collapsed": true,
 24 |     "slideshow": {
 25 |      "slide_type": "skip"
 26 |     }
 27 |    },
 28 |    "outputs": [],
 29 |    "source": [
 30 |     "from __future__ import print_function\n",
 31 |     "\n",
 32 |     "%matplotlib inline\n",
 33 |     "import numpy\n",
 34 |     "import matplotlib.pyplot as plt"
 35 |    ]
 36 |   },
 37 |   {
 38 |    "cell_type": "markdown",
 39 |    "metadata": {
 40 |     "slideshow": {
 41 |      "slide_type": "slide"
 42 |     }
 43 |    },
 44 |    "source": [
 45 |     "# Gaussian Elimination\n",
 46 |     "\n",
 47 |     "Gaussian elimination is the process where one transforms a matrix or linear system through a series of operations into one that is at the very least upper triangular (it is often also presented as including the operation that transforms $A$ completely to diagonal).  These series of operations can be written as a sequence of successive matrix-matrix multiplications by lower triangular matrices.  Letting these matrices be $L_j$ and the resulting upper triangular matrix be $U$ we can write this as\n",
 48 |     "$$\n",
 49 |     "    \\overbrace{L_{m-1} \\cdots L_2 L_1}^{L^{-1}} A = U.\n",
 50 |     "$$\n",
 51 |     "Labeling the successive operations as $L^{-1}$ allows us to move $L$ to the other side of the equation and we see that in fact what we have done is computed another factorization of the matrix $A$ called the $LU$ factorization."
 52 |    ]
 53 |   },
 54 |   {
 55 |    "cell_type": "markdown",
 56 |    "metadata": {
 57 |     "slideshow": {
 58 |      "slide_type": "subslide"
 59 |     }
 60 |    },
 61 |    "source": [
 62 |     "### Example\n",
 63 |     "\n",
 64 |     "As an example of this process lets consider the matrix\n",
 65 |     "$$\n",
 66 |     "    A = \\begin{bmatrix}\n",
 67 |     "        2 & 1 & 1 & 0 \\\\\n",
 68 |     "        4 & 3 & 3 & 1 \\\\\n",
 69 |     "        8 & 7 & 9 & 5 \\\\\n",
 70 |     "        6 & 7 & 9 & 8\n",
 71 |     "    \\end{bmatrix}\n",
 72 |     "$$"
 73 |    ]
 74 |   },
 75 |   {
 76 |    "cell_type": "markdown",
 77 |    "metadata": {
 78 |     "slideshow": {
 79 |      "slide_type": "subslide"
 80 |     }
 81 |    },
 82 |    "source": [
 83 |     "The first step is to remove the values in the first column below the diagonal, to do this we can multiply by the matrix\n",
 84 |     "$$\n",
 85 |     "    L_1 = \\begin{bmatrix}\n",
 86 |     "         1 &   &   &  \\\\\n",
 87 |     "        -2 & 1 &   &  \\\\\n",
 88 |     "        -4 &   & 1 &  \\\\\n",
 89 |     "        -3 &   &   & 1\n",
 90 |     "    \\end{bmatrix} \\text{ so that } L_1 A = \\begin{bmatrix}\n",
 91 |     "        2 & 1 & 1 & 0 \\\\\n",
 92 |     "          & 1 & 1 & 1 \\\\\n",
 93 |     "          & 3 & 5 & 5 \\\\\n",
 94 |     "          & 4 & 6 & 8\n",
 95 |     "    \\end{bmatrix}.\n",
 96 |     "$$"
 97 |    ]
 98 |   },
 99 |   {
100 |    "cell_type": "markdown",
101 |    "metadata": {
102 |     "slideshow": {
103 |      "slide_type": "subslide"
104 |     }
105 |    },
106 |    "source": [
107 |     "The next step is to remove the values below the diagonal of the second column.  This can be done with\n",
108 |     "$$\n",
109 |     "    L_2 = \\begin{bmatrix}\n",
110 |     "         1 &    &   &   \\\\\n",
111 |     "           &  1 &   &   \\\\\n",
112 |     "           & -3 & 1 &   \\\\\n",
113 |     "           & -4 &   & 1\n",
114 |     "    \\end{bmatrix} \\text{ so that } L_2 L_1 A = \\begin{bmatrix}\n",
115 |     "        2 & 1 & 1 & 0 \\\\\n",
116 |     "          & 1 & 1 & 1 \\\\\n",
117 |     "          &   & 2 & 2 \\\\\n",
118 |     "          &   & 2 & 4\n",
119 |     "    \\end{bmatrix}.\n",
120 |     "$$"
121 |    ]
122 |   },
123 |   {
124 |    "cell_type": "markdown",
125 |    "metadata": {
126 |     "slideshow": {
127 |      "slide_type": "subslide"
128 |     }
129 |    },
130 |    "source": [
131 |     "Finally we multiply $A$ by $L_3$ defined as\n",
132 |     "$$\n",
133 |     "    L_3 = \\begin{bmatrix}\n",
134 |     "         1 &   &    &   \\\\\n",
135 |     "           & 1 &    &   \\\\\n",
136 |     "           &   &  1 &   \\\\\n",
137 |     "           &   & -1 & 1\n",
138 |     "    \\end{bmatrix} \\text{ so that } L_3 L_2 L_1 A = \\begin{bmatrix}\n",
139 |     "        2 & 1 & 1 & 0 \\\\\n",
140 |     "          & 1 & 1 & 1 \\\\\n",
141 |     "          &   & 2 & 2 \\\\\n",
142 |     "          &   &   & 2\n",
143 |     "    \\end{bmatrix}\n",
144 |     "$$\n",
145 |     "completing the factorization with\n",
146 |     "$$\n",
147 |     "    L^{-1} = L_3 L_2 L_1 = \\begin{bmatrix}\n",
148 |     "         1 &   &    &   \\\\\n",
149 |     "           & 1 &    &   \\\\\n",
150 |     "           &   &  1 &   \\\\\n",
151 |     "           &   & -1 & 1\n",
152 |     "    \\end{bmatrix}\n",
153 |     "    \\begin{bmatrix}\n",
154 |     "         1 &    &   &   \\\\\n",
155 |     "           &  1 &   &   \\\\\n",
156 |     "           & -3 & 1 &   \\\\\n",
157 |     "           & -4 &   & 1\n",
158 |     "    \\end{bmatrix} \n",
159 |     "    \\begin{bmatrix}\n",
160 |     "         1 &   &   &   \\\\\n",
161 |     "        -2 & 1 &   &   \\\\\n",
162 |     "        -4 &   & 1 &   \\\\\n",
163 |     "        -3 &   &   & 1\n",
164 |     "    \\end{bmatrix}\n",
165 |     "$$ \n",
166 |     "and\n",
167 |     "$$\n",
168 |     "    U = \\begin{bmatrix}\n",
169 |     "        2 & 1 & 1 & 0 \\\\\n",
170 |     "          & 1 & 1 & 1 \\\\\n",
171 |     "          &   & 2 & 2 \\\\\n",
172 |     "          &   &   & 2\n",
173 |     "    \\end{bmatrix}\n",
174 |     "$$"
175 |    ]
176 |   },
177 |   {
178 |    "cell_type": "markdown",
179 |    "metadata": {
180 |     "slideshow": {
181 |      "slide_type": "subslide"
182 |     }
183 |    },
184 |    "source": [
185 |     "We can actually easily invert $L$ and finally can write $A$ as\n",
186 |     "$$\n",
187 |     "    A =\n",
188 |     "    \\begin{bmatrix}\n",
189 |     "         1 &   &   &   \\\\\n",
190 |     "         2 & 1 &   &   \\\\\n",
191 |     "         4 & 3 & 1 &   \\\\\n",
192 |     "         3 & 4 & 1 & 1\n",
193 |     "    \\end{bmatrix}\\begin{bmatrix}\n",
194 |     "        2 & 1 & 1 & 0 \\\\\n",
195 |     "          & 1 & 1 & 1 \\\\\n",
196 |     "          &   & 2 & 2 \\\\\n",
197 |     "          &   &   & 2\n",
198 |     "    \\end{bmatrix}\n",
199 |     "$$"
200 |    ]
201 |   },
202 |   {
203 |    "cell_type": "markdown",
204 |    "metadata": {
205 |     "slideshow": {
206 |      "slide_type": "slide"
207 |     }
208 |    },
209 |    "source": [
210 |     "## Pivoting\n",
211 |     "\n",
212 |     "As you may recall from your linear algebra course pivoting of the rows and columns of $A$ is often an important addition to Gaussian elimination to ensure that we can in fact factorize the matrix.  As a simple example take the matrix\n",
213 |     "$$\n",
214 |     "    A = \\begin{bmatrix} 0 & 1 \\\\ 1 & 1 \\end{bmatrix}.\n",
215 |     "$$\n",
216 |     "Without switch the rows Gaussian elimination would fail at the first step!  This is made more hazardous if we consider the matrix\n",
217 |     "$$\n",
218 |     "    A = \\begin{bmatrix} 10^{-17} & 1 \\\\ 1 & 1 \\end{bmatrix}.\n",
219 |     "$$\n",
220 |     "on a finite precision machine.  In principle any row and column can be **pivoted** so that at each step we have the maximum value being used (on the diagonal) to perform the operations that compose the matrices $L_j$.  In practice however we restrict ourselves to only pivoting rows of the matrix (called partial pivoting)."
221 |    ]
222 |   },
223 |   {
224 |    "cell_type": "markdown",
225 |    "metadata": {
226 |     "slideshow": {
227 |      "slide_type": "subslide"
228 |     }
229 |    },
230 |    "source": [
231 |     "Consider again the example from above and switch the 1st and 3rd rows using the criteria that we always want to use the largest value to do perform the reduction.  Defining the first pivot matrix as\n",
232 |     "$$\n",
233 |     "    P_1 = \\begin{bmatrix}\n",
234 |     "          &   & 1 &   \\\\\n",
235 |     "          & 1 &   &   \\\\\n",
236 |     "        1 &   &   &   \\\\\n",
237 |     "          &   &   & 1\n",
238 |     "    \\end{bmatrix} \\quad \\text{so that} \\quad\n",
239 |     "    P_1 A = \\begin{bmatrix}\n",
240 |     "        8 & 7 & 9 & 5 \\\\\n",
241 |     "        4 & 3 & 3 & 1 \\\\\n",
242 |     "        2 & 1 & 1 & 0 \\\\\n",
243 |     "        6 & 7 & 9 & 8\n",
244 |     "    \\end{bmatrix}\n",
245 |     "$$"
246 |    ]
247 |   },
248 |   {
249 |    "cell_type": "markdown",
250 |    "metadata": {
251 |     "slideshow": {
252 |      "slide_type": "subslide"
253 |     }
254 |    },
255 |    "source": [
256 |     "Now defining the first $L_1$ as\n",
257 |     "$$\n",
258 |     "    L_1 = \\begin{bmatrix}\n",
259 |     "        1 &   &   &   \\\\\n",
260 |     "        -\\frac{1}{2} & 1 &   &   \\\\\n",
261 |     "        -\\frac{1}{4} &   & 1 &   \\\\\n",
262 |     "        -\\frac{3}{4} &   &   & 1\n",
263 |     "    \\end{bmatrix} \\quad \\text{so that} \\quad\n",
264 |     "    L_1 P_1 A = \\begin{bmatrix}\n",
265 |     "        8 & 7 & 9 & 5 \\\\\n",
266 |     "          & -\\frac{1}{2} & -\\frac{3}{2} & -\\frac{3}{2} \\\\\n",
267 |     "          & -\\frac{3}{4} & -\\frac{5}{4} & -\\frac{5}{4} \\\\\n",
268 |     "          & \\frac{7}{4} & \\frac{9}{4} & \\frac{17}{4}\n",
269 |     "    \\end{bmatrix}.\n",
270 |     "$$"
271 |    ]
272 |   },
273 |   {
274 |    "cell_type": "markdown",
275 |    "metadata": {
276 |     "slideshow": {
277 |      "slide_type": "subslide"
278 |     }
279 |    },
280 |    "source": [
281 |     "Again examining the remaining values in column 2 the maximum value lies in row 4 so we want to interchange this with the second row (note that we do not want to move the first row as that will bring non-zero values into the first column below the diagonal).\n",
282 |     "$$\n",
283 |     "    P_2 = \\begin{bmatrix}\n",
284 |     "        1 &   &   &   \\\\\n",
285 |     "          &   &   & 1 \\\\\n",
286 |     "          &   & 1 &   \\\\\n",
287 |     "          & 1 &   &\n",
288 |     "    \\end{bmatrix}  \\quad \\text{and} \\quad \n",
289 |     "    L_2 = \\begin{bmatrix}\n",
290 |     "        1 &   &   &   \\\\\n",
291 |     "          & 1 &   &   \\\\\n",
292 |     "          & \\frac{3}{7} & 1 &   \\\\\n",
293 |     "          & \\frac{2}{7}  &   & 1\n",
294 |     "    \\end{bmatrix}  \\quad  \\text{so that}  \\quad \n",
295 |     "    L_2 P_2 L_1 P_1 A = \\begin{bmatrix}\n",
296 |     "        8 &            7 &            9 &             5 \\\\\n",
297 |     "          &  \\frac{7}{4} &  \\frac{9}{4} &  \\frac{17}{4} \\\\\n",
298 |     "          &              & -\\frac{2}{7} &   \\frac{4}{7} \\\\\n",
299 |     "          &              & -\\frac{6}{7} &  -\\frac{2}{7}\n",
300 |     "    \\end{bmatrix}.\n",
301 |     "$$"
302 |    ]
303 |   },
304 |   {
305 |    "cell_type": "markdown",
306 |    "metadata": {
307 |     "slideshow": {
308 |      "slide_type": "subslide"
309 |     }
310 |    },
311 |    "source": [
312 |     "Finally\n",
313 |     "$$\n",
314 |     "    P_3 = \\begin{bmatrix}\n",
315 |     "        1 &   &   &   \\\\\n",
316 |     "          & 1 &   &   \\\\\n",
317 |     "          &   &   & 1 \\\\\n",
318 |     "          &   & 1 &\n",
319 |     "    \\end{bmatrix}  \\quad \\text{and} \\quad \n",
320 |     "    L_3 = \\begin{bmatrix}\n",
321 |     "        1 &   &   &   \\\\\n",
322 |     "          & 1 &   &   \\\\\n",
323 |     "          &   & 1 &   \\\\\n",
324 |     "          &   & -\\frac{1}{3} & 1\n",
325 |     "    \\end{bmatrix} \\quad \\text{so that} \\quad\n",
326 |     "    L_3 P_3 L_2 P_2 L_1 P_1 A = \\begin{bmatrix}\n",
327 |     "        8 &            7 &            9 &             5 \\\\\n",
328 |     "          &  \\frac{7}{4} &  \\frac{9}{4} &  \\frac{17}{4} \\\\\n",
329 |     "          &              & -\\frac{6}{7} &  -\\frac{2}{7} \\\\\n",
330 |     "          &              &              &   \\frac{2}{3} \\\\\n",
331 |     "    \\end{bmatrix}.\n",
332 |     "$$"
333 |    ]
334 |   },
335 |   {
336 |    "cell_type": "markdown",
337 |    "metadata": {
338 |     "slideshow": {
339 |      "slide_type": "subslide"
340 |     }
341 |    },
342 |    "source": [
343 |     "### LU Factorization with Partial Pivoting\n",
344 |     "\n",
345 |     "Due to the nature of the pivot matrices we can disentangle them from the matrices $L_j$.  Right now we have\n",
346 |     "$$\n",
347 |     "    L_3 P_3 L_2 P_2 L_1 P_1 A = U\n",
348 |     "$$\n",
349 |     "where what we want is\n",
350 |     "$$\n",
351 |     "    (L'_3 L'_2 L'_1) P_3 P_2 P_1 A = U.\n",
352 |     "$$"
353 |    ]
354 |   },
355 |   {
356 |    "cell_type": "markdown",
357 |    "metadata": {
358 |     "slideshow": {
359 |      "slide_type": "subslide"
360 |     }
361 |    },
362 |    "source": [
363 |     "It turns out we can easily do this by\n",
364 |     "$$\n",
365 |     "    L'_3 = L_3, \\quad L'_2 = P_3 L_2 P_3^{-1}, \\quad \\text{and} \\quad L'_1 = P_3 P_2 L_1 P_2^{-1} P_3^{-1}.\n",
366 |     "$$\n",
367 |     "These new matrices $L'_j$ can easily be computed and it turns out that the $L'_j$ matrices have the same structure with the rows permuted accordingly."
368 |    ]
369 |   },
370 |   {
371 |    "cell_type": "markdown",
372 |    "metadata": {
373 |     "slideshow": {
374 |      "slide_type": "subslide"
375 |     }
376 |    },
377 |    "source": [
378 |     "In general then the $LU$ factorization with partial pivoting of the above example can be written as\n",
379 |     "$$\n",
380 |     "    \\underbrace{\\begin{bmatrix}\n",
381 |     "          &   & 1 &   \\\\\n",
382 |     "          &   &   & 1 \\\\\n",
383 |     "          & 1 &   &   \\\\\n",
384 |     "        1 &   &   & \n",
385 |     "    \\end{bmatrix}}_{P = P_3 P_2 P_1} \\underbrace{\\begin{bmatrix}\n",
386 |     "        2 & 1 & 1 & 0 \\\\\n",
387 |     "        4 & 3 & 3 & 1 \\\\\n",
388 |     "        8 & 7 & 9 & 5 \\\\\n",
389 |     "        6 & 7 & 9 & 8\n",
390 |     "    \\end{bmatrix}}_{A} = \n",
391 |     "    \\underbrace{\\begin{bmatrix}\n",
392 |     "        1           &              &              &   \\\\\n",
393 |     "        \\frac{3}{4} &            1 &              &   \\\\\n",
394 |     "        \\frac{1}{2} & -\\frac{2}{7} &            1 &   \\\\\n",
395 |     "        \\frac{1}{4} & -\\frac{3}{7} & \\frac{1}{3}  & 1 \\\\\n",
396 |     "    \\end{bmatrix}}_{L}\n",
397 |     "    \\underbrace{\\begin{bmatrix}\n",
398 |     "        8 &            7 &            9 &             5 \\\\\n",
399 |     "          &  \\frac{7}{4} &  \\frac{9}{4} &  \\frac{17}{4} \\\\\n",
400 |     "          &              & -\\frac{6}{7} &  -\\frac{2}{7} \\\\\n",
401 |     "          &              &              &   \\frac{2}{3} \\\\\n",
402 |     "    \\end{bmatrix}}_{U}\n",
403 |     "$$"
404 |    ]
405 |   },
406 |   {
407 |    "cell_type": "markdown",
408 |    "metadata": {
409 |     "slideshow": {
410 |      "slide_type": "subslide"
411 |     }
412 |    },
413 |    "source": [
414 |     "The algorithm for the general factorization with partial pivoting then can be written as\n",
415 |     "```\n",
416 |     "U = A\n",
417 |     "L = I\n",
418 |     "P = I\n",
419 |     "for k = 1 to m - 1\n",
420 |     "    Select i >= k to maximize |u_{i,k}|\n",
421 |     "    U[k, k:m] <==> U[i, k:m]\n",
422 |     "    L[k, 1:k-1] <==> L[i, 1:k-1]\n",
423 |     "    P[k, :] <==> P[i, :]\n",
424 |     "    for j = k + 1 to m\n",
425 |     "        L[j, k] = U[j, k] / U[k, k]\n",
426 |     "        U[j, k:m] = U[j, k:m] - L[j, k] * U[k, k:m]\n",
427 |     "```\n",
428 |     "where `<==>` represents swapping of the two rows indicated."
429 |    ]
430 |   },
431 |   {
432 |    "cell_type": "markdown",
433 |    "metadata": {
434 |     "slideshow": {
435 |      "slide_type": "slide"
436 |     }
437 |    },
438 |    "source": [
439 |     "## Solving $Ax = b$\n",
440 |     "\n",
441 |     "To complete our discussion lets consider the solution of the linear system $Ax = b$.  We now have a factorization of the matrix $PA = LU$ so that the new system is $LU x = b$ (note that pivoting is allowed as long as we also interchange the elements of $b$ via $P \\cdot b$).  We can do this in two steps:\n",
442 |     "\n",
443 |     "1. Compute the pivoted $LU$ factorization $PA = LU$.\n",
444 |     "1. Compute the solution to $L y = P b$ using forward substitution.\n",
445 |     "1. Compute the solution to $U x = y$ using backward substitution."
446 |    ]
447 |   },
448 |   {
449 |    "cell_type": "markdown",
450 |    "metadata": {
451 |     "slideshow": {
452 |      "slide_type": "subslide"
453 |     }
454 |    },
455 |    "source": [
456 |     "### Forward Substitution\n",
457 |     "\n",
458 |     "For forward substitution we proceed from the first row and progress downwards through the matrix.  We can then consider the general $i$th row with\n",
459 |     "$$\n",
460 |     "    L_{i,1} y_1 + L_{i,2} y_2 + \\cdots + L_{i,i-1} y_{i-1} + y_i = b_i\n",
461 |     "$$\n",
462 |     "noting that we are using the fact that the matrix $L$ has 1 on its diagonal.  We can now solve for $y_i$ as\n",
463 |     "$$\n",
464 |     "    y_i = b_i - \\left( L_{i,1} y_1 + L_{i,2} y_2 + \\cdots + L_{i,i-1} y_{i-1} \\right ).\n",
465 |     "$$"
466 |    ]
467 |   },
468 |   {
469 |    "cell_type": "markdown",
470 |    "metadata": {
471 |     "slideshow": {
472 |      "slide_type": "subslide"
473 |     }
474 |    },
475 |    "source": [
476 |     "### Backward Substitution\n",
477 |     "\n",
478 |     "Backwards substitution requires us to move from the last row of $U$ and move upwards.  We can consider again the general $i$th row with\n",
479 |     "$$\n",
480 |     "    U_{i,i} x_i + U_{i,i+1} x_{i+1} + \\ldots + U_{i,m-1} x_{m-1} + U_{i,m} x_m = y_i.\n",
481 |     "$$\n",
482 |     "We can now solve for $x_i$ as\n",
483 |     "$$\n",
484 |     "    x_i = \\frac{1}{U_{i,i}} \\left( y_i - ( U_{i,i+1} x_{i+1} + \\ldots + U_{i,m-1} x_{m-1} + U_{i,m} x_m) \\right )\n",
485 |     "$$"
486 |    ]
487 |   }
488 |  ],
489 |  "metadata": {
490 |   "anaconda-cloud": {},
491 |   "celltoolbar": "Slideshow",
492 |   "kernelspec": {
493 |    "display_name": "Python 3",
494 |    "language": "python",
495 |    "name": "python3"
496 |   },
497 |   "language_info": {
498 |    "codemirror_mode": {
499 |     "name": "ipython",
500 |     "version": 3
501 |    },
502 |    "file_extension": ".py",
503 |    "mimetype": "text/x-python",
504 |    "name": "python",
505 |    "nbconvert_exporter": "python",
506 |    "pygments_lexer": "ipython3",
507 |    "version": "3.7.0"
508 |   },
509 |   "latex_envs": {
510 |    "bibliofile": "biblio.bib",
511 |    "cite_by": "apalike",
512 |    "current_citInitial": 1,
513 |    "eqLabelWithNumbers": true,
514 |    "eqNumInitial": 0
515 |   }
516 |  },
517 |  "nbformat": 4,
518 |  "nbformat_minor": 1
519 | }
520 | 


--------------------------------------------------------------------------------
/16_ODE_BVP.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "slideshow": {
  7 |      "slide_type": "skip"
  8 |     }
  9 |    },
 10 |    "source": [
 11 |     "<table>\n",
 12 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 13 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 14 |     "</table>\n",
 15 |     "\n",
 16 |     "Note:  The presentation below largely follows part I in \"Finite Difference Methods for Ordinary and Partial Differential Equations\" by LeVeque (SIAM, 2007)."
 17 |    ]
 18 |   },
 19 |   {
 20 |    "cell_type": "code",
 21 |    "execution_count": null,
 22 |    "metadata": {
 23 |     "collapsed": true,
 24 |     "init_cell": true,
 25 |     "slideshow": {
 26 |      "slide_type": "skip"
 27 |     }
 28 |    },
 29 |    "outputs": [],
 30 |    "source": [
 31 |     "from __future__ import print_function\n",
 32 |     "\n",
 33 |     "%matplotlib inline\n",
 34 |     "import numpy\n",
 35 |     "import matplotlib.pyplot as plt"
 36 |    ]
 37 |   },
 38 |   {
 39 |    "cell_type": "markdown",
 40 |    "metadata": {
 41 |     "slideshow": {
 42 |      "slide_type": "slide"
 43 |     }
 44 |    },
 45 |    "source": [
 46 |     "# Solving Boundary Value Problems"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "markdown",
 51 |    "metadata": {
 52 |     "slideshow": {
 53 |      "slide_type": "slide"
 54 |     }
 55 |    },
 56 |    "source": [
 57 |     "## The Problem\n",
 58 |     "\n",
 59 |     "We want to solve an ODE (PDE) that instead of having initial conditions is contained to an interval and has values at the edges of the interval.  This naturally comes about when we consider spatial problems.  One of the simplest cases for this is the Poisson problem in one-dimension\n",
 60 |     "$$\n",
 61 |     "    u_{xx} = f(x)\n",
 62 |     "$$\n",
 63 |     "where we will use the short-hand\n",
 64 |     "$$\n",
 65 |     "    u_{xx} = \\frac{\\text{d}^2 u}{\\text{d} x^2} \\quad \\text{or} \\quad \\frac{\\partial^2 u}{\\partial x^2}.\n",
 66 |     "$$"
 67 |    ]
 68 |   },
 69 |   {
 70 |    "cell_type": "markdown",
 71 |    "metadata": {
 72 |     "slideshow": {
 73 |      "slide_type": "subslide"
 74 |     }
 75 |    },
 76 |    "source": [
 77 |     "Note that due to the order of the derivative we require two conditions to solve this.  The simplest case where we are on the domain $x \\in [a,b]$ is to have conditions such that we require $u(a) = u_a$ and $u(b) = u_b$ and are commonly termed boundary value problems (BVP).  If these conditions are both at one end of the domain then we can actually phrase the ODE (PDE) again as an initial value problem (IVP).  So what do we need to do to solve these types of problems?  We will consider two approaches to this problem:\n",
 78 |     "\n",
 79 |     "1. Rephrase the BVP to an IVP and use our standard methods for ODEs.\n",
 80 |     "1. Use finite differences to represent the unknowns as a linear system and solve the resulting system."
 81 |    ]
 82 |   },
 83 |   {
 84 |    "cell_type": "markdown",
 85 |    "metadata": {
 86 |     "slideshow": {
 87 |      "slide_type": "slide"
 88 |     }
 89 |    },
 90 |    "source": [
 91 |     "## The Shooting Method\n",
 92 |     "\n",
 93 |     "The shooting method takes the approach that we want to use our ability to solve IVP problems and so tries to term the problem as a root finding problem for the higher order initial condition that we are not given.  This is best illustrated in an example."
 94 |    ]
 95 |   },
 96 |   {
 97 |    "cell_type": "markdown",
 98 |    "metadata": {
 99 |     "slideshow": {
100 |      "slide_type": "subslide"
101 |     }
102 |    },
103 |    "source": [
104 |     "Consider the problem\n",
105 |     "$$\n",
106 |     "    u_{xx} = -\\sin u\n",
107 |     "$$\n",
108 |     "with\n",
109 |     "$$\n",
110 |     "    x \\in [0, 2] \\quad \\text{and} \\quad u(0) = 0.0, \\quad u(2.0) = \\frac{\\pi}{2}.\n",
111 |     "$$"
112 |    ]
113 |   },
114 |   {
115 |    "cell_type": "markdown",
116 |    "metadata": {
117 |     "slideshow": {
118 |      "slide_type": "subslide"
119 |     }
120 |    },
121 |    "source": [
122 |     "We can rewrite this problem as a system of two ODEs as\n",
123 |     "$$\n",
124 |     "    v = \\begin{bmatrix} u \\\\ u_x \\end{bmatrix} \\quad \\text{and} \\quad v_x = \\begin{bmatrix} u_x \\\\ u_{xx} \\end{bmatrix} = \\begin{bmatrix} v_2 \\\\ -\\sin v_1 \\end{bmatrix}.\n",
125 |     "$$"
126 |    ]
127 |   },
128 |   {
129 |    "cell_type": "markdown",
130 |    "metadata": {
131 |     "slideshow": {
132 |      "slide_type": "subslide"
133 |     }
134 |    },
135 |    "source": [
136 |     "We know that we want $v_1(0) = 0$ but what do we use for $v_2(0)$?  Making an initial guess at $v_2(0)$ and solving the associated IVP ODE we can then find out what these initial conditions produces on the right boundary of the problem.  Using a root-finding approach (or minimization routine) we can write this procedure as\n",
137 |     "$$\n",
138 |     "    \\min_{v_2(0)} \\left | \\pi / 2 - v_1(2) \\right |\n",
139 |     "$$\n",
140 |     "where the parameter we vary is $v_2(0)$."
141 |    ]
142 |   },
143 |   {
144 |    "cell_type": "code",
145 |    "execution_count": null,
146 |    "metadata": {
147 |     "slideshow": {
148 |      "slide_type": "skip"
149 |     }
150 |    },
151 |    "outputs": [],
152 |    "source": [
153 |     "# Basic Shooting Method solving u_xx = -sin(u)\n",
154 |     "import scipy.integrate as integrate\n",
155 |     "\n",
156 |     "# Algorithm parameters\n",
157 |     "TOLERANCE = 1e-8\n",
158 |     "MAX_ITERATIONS = 100\n",
159 |     "\n",
160 |     "# Problem Statement\n",
161 |     "a = 0.0\n",
162 |     "b = 2.0\n",
163 |     "N = 100\n",
164 |     "x = numpy.linspace(a, b, N)\n",
165 |     "u_a = 0.0\n",
166 |     "u_b = numpy.pi / 2.0\n",
167 |     "# RHS function\n",
168 |     "def f(x, u):\n",
169 |     "    return numpy.array([u[1], -numpy.sin(u[0])])\n",
170 |     "\n",
171 |     "# Initial guess\n",
172 |     "# Slope at RHS\n",
173 |     "u_prime_rhs = 1.0\n",
174 |     "# Initial step size\n",
175 |     "du_prime = 0.5\n",
176 |     "\n",
177 |     "# Plotting\n",
178 |     "fig = plt.figure()\n",
179 |     "fig.set_figwidth(fig.get_figwidth() * 2)\n",
180 |     "axes = fig.add_subplot(1, 2, 1)\n",
181 |     "\n",
182 |     "# Main loop\n",
183 |     "success = False\n",
184 |     "u = numpy.empty((2, N))\n",
185 |     "convergence = numpy.zeros(MAX_ITERATIONS)\n",
186 |     "for n in range(MAX_ITERATIONS):\n",
187 |     "    \n",
188 |     "    # Initial condition\n",
189 |     "    u[0, 0] = u_a\n",
190 |     "    u[1, 0] = u_prime_rhs\n",
191 |     "\n",
192 |     "    # Construct integrator\n",
193 |     "    integrator = integrate.ode(f)\n",
194 |     "    integrator.set_integrator(\"dopri5\")\n",
195 |     "    integrator.set_initial_value(u[:, 0])\n",
196 |     "\n",
197 |     "    # Compute solution - note that we are only producing the intermediate values\n",
198 |     "    # for demonstration purposes\n",
199 |     "    for (i, x_output) in enumerate(x[1:]):\n",
200 |     "        integrator.integrate(x_output)\n",
201 |     "        if not integrator.successful():\n",
202 |     "            raise Exception(\"Integration Failed!\")\n",
203 |     "        u[:, i + 1] = integrator.y\n",
204 |     "\n",
205 |     "    # Stopping Criteria\n",
206 |     "    convergence[n] = numpy.abs(u[0, -1] - u_b)\n",
207 |     "    if numpy.abs(u[0, -1] - u_b) < TOLERANCE:\n",
208 |     "        success = True\n",
209 |     "        break\n",
210 |     "    else:\n",
211 |     "        if u[0, -1] < u_b:\n",
212 |     "            u_prime_rhs += du_prime\n",
213 |     "        else:\n",
214 |     "            u_prime_rhs -= du_prime\n",
215 |     "            du_prime *= 0.5\n",
216 |     "\n",
217 |     "    axes.plot(x, u[0, :], 'b')\n",
218 |     "    axes.plot(b, u_b, 'ro')\n",
219 |     "\n",
220 |     "axes.set_title(\"Shooting Method Iterations\")\n",
221 |     "axes.set_xlabel(\"$x$\")\n",
222 |     "axes.set_ylabel(\"$u(x)$\")\n",
223 |     "\n",
224 |     "axes = fig.add_subplot(1, 2, 2)\n",
225 |     "n_range = numpy.arange(n)\n",
226 |     "axes.semilogy(n_range, convergence[:n])\n",
227 |     "axes.set_title(\"Convergence of Shooting Method\")\n",
228 |     "axes.set_xlabel(\"step\")\n",
229 |     "axes.set_ylabel(\"$|u(b) - U(b)|$\")\n",
230 |     "\n",
231 |     "plt.show()"
232 |    ]
233 |   },
234 |   {
235 |    "cell_type": "markdown",
236 |    "metadata": {
237 |     "slideshow": {
238 |      "slide_type": "subslide"
239 |     }
240 |    },
241 |    "source": [
242 |     "The tricky part of this procedure is coming up with the search criteria, i.e. coming up with the decision of how to change $v_2(0)$ with respect to the position of $v_2(2)$ compared to what we want $u(2)$."
243 |    ]
244 |   },
245 |   {
246 |    "cell_type": "markdown",
247 |    "metadata": {
248 |     "slideshow": {
249 |      "slide_type": "subslide"
250 |     }
251 |    },
252 |    "source": [
253 |     "In general any minimization routine can be used in a shooting method.  These approaches are generally very effective at approaching non-linear BVPs where the next method we will discuss is too expensive to perform."
254 |    ]
255 |   },
256 |   {
257 |    "cell_type": "markdown",
258 |    "metadata": {
259 |     "slideshow": {
260 |      "slide_type": "slide"
261 |     }
262 |    },
263 |    "source": [
264 |     "## Linear System Approach"
265 |    ]
266 |   },
267 |   {
268 |    "cell_type": "markdown",
269 |    "metadata": {
270 |     "slideshow": {
271 |      "slide_type": "subslide"
272 |     }
273 |    },
274 |    "source": [
275 |     "### Formulation\n",
276 |     "\n",
277 |     "The second approach we will consider involves the formation of a system of equations to solve based on finite difference approximations.  Again let's consider an example problem where\n",
278 |     "$$\n",
279 |     "    u_{xx} = f(x)\n",
280 |     "$$\n",
281 |     "with the initial conditions $u(a) = u_a$ and $u(b) = u_b$.  "
282 |    ]
283 |   },
284 |   {
285 |    "cell_type": "markdown",
286 |    "metadata": {
287 |     "slideshow": {
288 |      "slide_type": "subslide"
289 |     }
290 |    },
291 |    "source": [
292 |     "We know from our finite difference discussion that the second order centered difference approximation for the second derivative for a function $u(x)$ is\n",
293 |     "$$\n",
294 |     "    u_{xx} \\approx \\frac{u(x_{i-1}) - 2 u(x_i) + u(x_{i+1})}{\\Delta x^2}.\n",
295 |     "$$"
296 |    ]
297 |   },
298 |   {
299 |    "cell_type": "markdown",
300 |    "metadata": {
301 |     "slideshow": {
302 |      "slide_type": "subslide"
303 |     }
304 |    },
305 |    "source": [
306 |     "If we descretize the domain of the original BVP into $N$ points (not including the boundaries) such that\n",
307 |     "$$\n",
308 |     "    x_i = a + \\frac{b - a}{N+1} \\cdot i ~~~ \\text{where} ~~~ i = 1, \\ldots, N\n",
309 |     "$$\n",
310 |     "we can then write the finite difference approximation as a system of linear equations!  "
311 |    ]
312 |   },
313 |   {
314 |    "cell_type": "markdown",
315 |    "metadata": {
316 |     "slideshow": {
317 |      "slide_type": "subslide"
318 |     }
319 |    },
320 |    "source": [
321 |     "If for instance we take $N = 5$ then\n",
322 |     "$$\\begin{aligned}\n",
323 |     "    (U_{xx})_1 &\\approx \\frac{U_a - 2 U_1 + U_2}{\\Delta x^2} \\\\\n",
324 |     "    (U_{xx})_2 &\\approx \\frac{U_1 - 2 U_2 + U_3}{\\Delta x^2} \\\\\n",
325 |     "    (U_{xx})_3 &\\approx \\frac{U_2 - 2 U_3 + U_4}{\\Delta x^2} \\\\\n",
326 |     "    (U_{xx})_4 &\\approx \\frac{U_3 - 2 U_4 + U_5}{\\Delta x^2} \\\\\n",
327 |     "    (U_{xx})_5 &\\approx \\frac{U_4 - 2 U_5 + U_b}{\\Delta x^2} \\\\\n",
328 |     "\\end{aligned}$$\n",
329 |     "where we have used $U_a = u(a)$ and $U_b = u(b)$ as the boundary conditions.  "
330 |    ]
331 |   },
332 |   {
333 |    "cell_type": "markdown",
334 |    "metadata": {
335 |     "slideshow": {
336 |      "slide_type": "subslide"
337 |     }
338 |    },
339 |    "source": [
340 |     "Using these approximations to the derivatives we can then write the ODE as\n",
341 |     "$$\n",
342 |     "    \\frac{1}{\\Delta x^2}\\begin{bmatrix}\n",
343 |     "    -2 &  1 &    &    &    \\\\\n",
344 |     "     1 & -2 &  1 &    &    \\\\\n",
345 |     "       &  1 & -2 &  1 &    \\\\\n",
346 |     "       &    &  1 & -2 &  1 \\\\\n",
347 |     "       &    &    &  1 & -2 \\\\\n",
348 |     "    \\end{bmatrix} \\begin{bmatrix}\n",
349 |     "        U_1 \\\\ U_2 \\\\ U_3 \\\\ U_4 \\\\ U_5\n",
350 |     "    \\end{bmatrix} = \n",
351 |     "    \\begin{bmatrix}\n",
352 |     "        f(x_1) \\\\ f(x_2) \\\\ f(x_3) \\\\ f(x_4) \\\\ f(x_5) \\\\\n",
353 |     "    \\end{bmatrix}.\n",
354 |     "$$"
355 |    ]
356 |   },
357 |   {
358 |    "cell_type": "markdown",
359 |    "metadata": {
360 |     "slideshow": {
361 |      "slide_type": "notes"
362 |     }
363 |    },
364 |    "source": [
365 |     "Note that our previous example used for the shooting method is difficult in the current context as the unknown function is in the function $f$ so that we would need to actual solve a non-linear system of equations.  This is still possible in this context using an approach such as a Newton solver and has similar properties as the shooting method (although not as simple to implement)."
366 |    ]
367 |   },
368 |   {
369 |    "cell_type": "markdown",
370 |    "metadata": {
371 |     "slideshow": {
372 |      "slide_type": "subslide"
373 |     }
374 |    },
375 |    "source": [
376 |     "### Boundary Conditions\n",
377 |     "\n",
378 |     "This does not include the boundary conditions though.  We can add these values easily for Dirichlet boundary conditions by sending the values we know to the $b$ vector:\n",
379 |     "$$\\begin{aligned}\n",
380 |     "    \\frac{U_a - 2 U_1 + U_2}{\\Delta x^2} = f(x_1) &\\Rightarrow& \\frac{- 2 U_1 + U_2}{\\Delta x^2} = f(x_1) - \\frac{U_a}{\\Delta x^2} \\\\\n",
381 |     "    \\frac{U_4 - 2 U_5 + U_b}{\\Delta x^2} = f(x_1) &\\Rightarrow& \\frac{U_4 - 2 U_5}{\\Delta x^2} = f(x_5) - \\frac{U_b}{\\Delta x^2}\n",
382 |     "\\end{aligned}$$\n",
383 |     "so that final system looks like\n",
384 |     "$$\n",
385 |     "    \\frac{1}{\\Delta x^2} \\begin{bmatrix}\n",
386 |     "    -2 &  1 &    &    &    \\\\\n",
387 |     "     1 & -2 &  1 &    &    \\\\\n",
388 |     "       &  1 & -2 &  1 &    \\\\\n",
389 |     "       &    &  1 & -2 &  1 \\\\\n",
390 |     "       &    &    &  1 & -2 \\\\\n",
391 |     "    \\end{bmatrix} \\begin{bmatrix}\n",
392 |     "        U_1 \\\\ U_2 \\\\ U_3 \\\\ U_4 \\\\ U_5\n",
393 |     "    \\end{bmatrix} = \n",
394 |     "    \\begin{bmatrix}\n",
395 |     "        f(x_1) - \\frac{U_a}{\\Delta x^2} \\\\ f(x_2) \\\\ f(x_3) \\\\ f(x_4) \\\\ f(x_5) - \\frac{U_b}{\\Delta x^2} \\\\\n",
396 |     "    \\end{bmatrix}.\n",
397 |     "$$"
398 |    ]
399 |   },
400 |   {
401 |    "cell_type": "markdown",
402 |    "metadata": {
403 |     "slideshow": {
404 |      "slide_type": "subslide"
405 |     }
406 |    },
407 |    "source": [
408 |     "### Example\n",
409 |     "\n",
410 |     "Want to solve the BVP\n",
411 |     "$$\n",
412 |     "    u_{xx} = e^x, \\quad x \\in [0, 1] \\quad \\text{with} \\quad u(0) = 0.0, \\text{ and } u(1) = 3\n",
413 |     "$$\n",
414 |     "via the construction of a linear system of equations."
415 |    ]
416 |   },
417 |   {
418 |    "cell_type": "markdown",
419 |    "metadata": {
420 |     "slideshow": {
421 |      "slide_type": "subslide"
422 |     }
423 |    },
424 |    "source": [
425 |     "\\begin{align*}\n",
426 |     "    u_{xx} &= e^x \\\\\n",
427 |     "    u_x &= A + e^x \\\\\n",
428 |     "    u &= Ax + B + e^x\\\\\n",
429 |     "    u(0) &= B + 1 = 0 \\Rightarrow B = -1 \\\\\n",
430 |     "    u(1) &= A - 1 + e^{1} = 3 \\Rightarrow A = 4 - e\\\\ \n",
431 |     "    ~\\\\\n",
432 |     "    u(x) &= (4 - e) x - 1 + e^x\n",
433 |     "\\end{align*}"
434 |    ]
435 |   },
436 |   {
437 |    "cell_type": "code",
438 |    "execution_count": null,
439 |    "metadata": {
440 |     "slideshow": {
441 |      "slide_type": "skip"
442 |     }
443 |    },
444 |    "outputs": [],
445 |    "source": [
446 |     "# Problem setup\n",
447 |     "a = 0.0\n",
448 |     "b = 1.0\n",
449 |     "u_a = 0.0\n",
450 |     "u_b = 3.0\n",
451 |     "f = lambda x: numpy.exp(x)\n",
452 |     "u_true = lambda x: (4.0 - numpy.exp(1.0)) * x - 1.0 + numpy.exp(x)\n",
453 |     "\n",
454 |     "# Descretization\n",
455 |     "N = 10\n",
456 |     "x_bc = numpy.linspace(a, b, N + 2)\n",
457 |     "x = x_bc[1:-1]\n",
458 |     "delta_x = (b - a) / (N + 1)\n",
459 |     "\n",
460 |     "# Construct matrix A\n",
461 |     "A = numpy.zeros((N, N))\n",
462 |     "diagonal = numpy.ones(N) / delta_x**2\n",
463 |     "A += numpy.diag(diagonal * -2.0, 0)\n",
464 |     "A += numpy.diag(diagonal[:-1], 1)\n",
465 |     "A += numpy.diag(diagonal[:-1], -1)\n",
466 |     "\n",
467 |     "# Construct RHS\n",
468 |     "b = f(x)\n",
469 |     "b[0] -= u_a / delta_x**2\n",
470 |     "b[-1] -= u_b / delta_x**2\n",
471 |     "\n",
472 |     "# Solve system\n",
473 |     "U = numpy.empty(N + 2)\n",
474 |     "U[0] = u_a\n",
475 |     "U[-1] = u_b\n",
476 |     "U[1:-1] = numpy.linalg.solve(A, b)\n",
477 |     "\n",
478 |     "# Plot result\n",
479 |     "fig = plt.figure()\n",
480 |     "axes = fig.add_subplot(1, 1, 1)\n",
481 |     "axes.plot(x_bc, U, 'o', label=\"Computed\")\n",
482 |     "axes.plot(x_bc, u_true(x_bc), 'k', label=\"True\")\n",
483 |     "axes.set_title(\"Solution to $u_{xx} = e^x$\")\n",
484 |     "axes.set_xlabel(\"x\")\n",
485 |     "axes.set_ylabel(\"u(x)\")\n",
486 |     "plt.show()"
487 |    ]
488 |   },
489 |   {
490 |    "cell_type": "markdown",
491 |    "metadata": {
492 |     "slideshow": {
493 |      "slide_type": "subslide"
494 |     }
495 |    },
496 |    "source": [
497 |     "If we instead have Neumann boundary conditions it is no longer clear how to handle the boundary conditions using the above approach.  Instead a **ghost cell** approach is often used.  These **ghost cells** are added unknowns that represent the boundary values that we actually know.  "
498 |    ]
499 |   },
500 |   {
501 |    "cell_type": "markdown",
502 |    "metadata": {
503 |     "slideshow": {
504 |      "slide_type": "subslide"
505 |     }
506 |    },
507 |    "source": [
508 |     "For instance, if we had the BVP\n",
509 |     "$$\n",
510 |     "    u_{xx} = e^x, \\quad x \\in [-1, 1] \\quad \\text{with} \\quad u(-1) = 3, \\text{ and } u_x(1) = -5\n",
511 |     "$$\n",
512 |     "then we could keep the boundary values in the vector of unknowns so that now\n",
513 |     "$$\n",
514 |     "    U = \\begin{bmatrix} U_0 \\\\ U_1 \\\\ \\vdots \\\\ U_N \\\\ U_{N+1} \\end{bmatrix}\n",
515 |     "$$\n",
516 |     "where here $U_0$ and $U_{N+1}$ are actually the boundary points.  "
517 |    ]
518 |   },
519 |   {
520 |    "cell_type": "markdown",
521 |    "metadata": {
522 |     "slideshow": {
523 |      "slide_type": "subslide"
524 |     }
525 |    },
526 |    "source": [
527 |     "The matrix $A$ is then modified to have the appropriate relationships.  In this case the left boundary condition leads to\n",
528 |     "$$\n",
529 |     "    A = \\begin{bmatrix}\n",
530 |     "  1 &    &    &    &    &    \\\\\n",
531 |     "  \\frac{1}{\\Delta x^2} & \\frac{-2}{\\Delta x^2} &  \\frac{1}{\\Delta x^2} &    &    &    \\\\\n",
532 |     "    &  \\frac{1}{\\Delta x^2} & \\frac{-2}{\\Delta x^2} &  \\frac{1}{\\Delta x^2} &    &    \\\\\n",
533 |     "    & & \\ddots & \\ddots & \\ddots\n",
534 |     "    \\end{bmatrix} \\quad \\text{and} \\quad b = \\begin{bmatrix}\n",
535 |     "        u(a) \\\\ f(x_1) \\\\ f(x_2) \\\\ \\vdots\n",
536 |     "    \\end{bmatrix}\n",
537 |     "$$\n",
538 |     "which multiplied out simply gives\n",
539 |     "$$\n",
540 |     "    U_0 = u(-1) = 3.\n",
541 |     "$$"
542 |    ]
543 |   },
544 |   {
545 |    "cell_type": "markdown",
546 |    "metadata": {
547 |     "slideshow": {
548 |      "slide_type": "subslide"
549 |     }
550 |    },
551 |    "source": [
552 |     "For the right boundary condition we can use the second order backward finite difference approximation for the first derivative\n",
553 |     "$$\n",
554 |     "    u_x(b) \\approx \\frac{3 U_{N+1} - 4 U_{N} + U_{N - 1}}{2.0 \\Delta x} = -5\n",
555 |     "$$\n",
556 |     "which can be incorporated into the matrix $A$ and vector $b$ as\n",
557 |     "$$\n",
558 |     "    A =  \\begin{bmatrix}\n",
559 |     "     \\ddots & \\ddots & \\ddots &    &    \\\\\n",
560 |     "            & \\frac{1}{\\Delta x^2} &     \\frac{-2}{\\Delta x^2}&  \\frac{1}{\\Delta x^2} &    \\\\\n",
561 |     "            &        &      \\frac{1}{\\Delta x^2} & \\frac{-2}{\\Delta x^2} &  \\frac{1}{\\Delta x^2} \\\\\n",
562 |     "            &        &      \\frac{1}{2 \\Delta x} &  \\frac{-4}{2 \\Delta x} &  \\frac{3}{2 \\Delta x} \\\\\n",
563 |     "    \\end{bmatrix} ~~~~ \\text{and} ~~~~ b = \\begin{bmatrix}\n",
564 |     "        \\vdots \\\\ f(x_N) \\\\ u_x(b)\n",
565 |     "    \\end{bmatrix}.\n",
566 |     "$$"
567 |    ]
568 |   },
569 |   {
570 |    "cell_type": "markdown",
571 |    "metadata": {
572 |     "slideshow": {
573 |      "slide_type": "subslide"
574 |     }
575 |    },
576 |    "source": [
577 |     "All together the new system looks like\n",
578 |     "$$\n",
579 |     "    \\begin{bmatrix}\n",
580 |     "     1 &    &    &    &    &    \\\\\n",
581 |     "     \\frac{1}{\\Delta x^2} & \\frac{-2}{\\Delta x^2} &  \\frac{1}{\\Delta x^2} &    &    &    \\\\\n",
582 |     "       &  \\ddots & \\ddots &  \\ddots &    \\\\\n",
583 |     "       &    & \\frac{1}{\\Delta x^2} & \\frac{-2}{\\Delta x^2} &  \\frac{1}{\\Delta x^2} \\\\\n",
584 |     "            &        &      \\frac{1}{2 \\Delta x} &  \\frac{-4}{2 \\Delta x} &  \\frac{3}{2 \\Delta x} \\\\\n",
585 |     "    \\end{bmatrix} \\begin{bmatrix}\n",
586 |     "        U_0 \\\\ U_1 \\\\ \\vdots \\\\ U_N \\\\ U_{N+1}\n",
587 |     "    \\end{bmatrix} = \n",
588 |     "    \\begin{bmatrix}\n",
589 |     "        u(a) \\\\ f(x_1) \\\\ \\vdots \\\\ f(x_N) \\\\ u_x(b)\n",
590 |     "    \\end{bmatrix}.\n",
591 |     "$$"
592 |    ]
593 |   },
594 |   {
595 |    "cell_type": "markdown",
596 |    "metadata": {
597 |     "slideshow": {
598 |      "slide_type": "subslide"
599 |     }
600 |    },
601 |    "source": [
602 |     "### Example\n",
603 |     "\n",
604 |     "Want to solve the BVP\n",
605 |     "$$\n",
606 |     "    u_{xx} = e^x, \\quad x \\in [-1, 1] \\quad \\text{with} \\quad u(-1) = 3.0, \\text{ and } u_x(1) = -5.0\n",
607 |     "$$\n",
608 |     "via the construction of a linear system of equations.\n",
609 |     "\n",
610 |     "First find the true solution and then compute the solution."
611 |    ]
612 |   },
613 |   {
614 |    "cell_type": "markdown",
615 |    "metadata": {
616 |     "slideshow": {
617 |      "slide_type": "subslide"
618 |     }
619 |    },
620 |    "source": [
621 |     "\\begin{align*}\n",
622 |     "    u(x) &= A x + B + e^x \\\\\n",
623 |     "    u_x(1) &= A + e^1 = -5 \\Rightarrow A = -5 - e \\\\\n",
624 |     "    u(-1) &= (5 + e) + B + e^{-1} = 3 \\Rightarrow B = 3 - 5 - e - e^{-1} = -(2 + e + e^{-1}) \\\\\n",
625 |     "    ~\\\\\n",
626 |     "    u(x) &= -(5 + e) x -(2 + e + e^{-1}) + e^{x}\n",
627 |     "\\end{align*}"
628 |    ]
629 |   },
630 |   {
631 |    "cell_type": "code",
632 |    "execution_count": null,
633 |    "metadata": {
634 |     "slideshow": {
635 |      "slide_type": "skip"
636 |     }
637 |    },
638 |    "outputs": [],
639 |    "source": [
640 |     "# Problem setup\n",
641 |     "a = -1.0\n",
642 |     "b = 1.0\n",
643 |     "u_a = 3.0\n",
644 |     "u_x_b = -5.0\n",
645 |     "f = lambda x: numpy.exp(x)\n",
646 |     "u_true = lambda x: -(5.0 + numpy.exp(1.0)) * x - (2.0 + numpy.exp(1.0) + numpy.exp(-1.0)) + numpy.exp(x)\n",
647 |     "\n",
648 |     "# Descretization\n",
649 |     "N = 10\n",
650 |     "x_bc = numpy.linspace(a, b, N + 2)\n",
651 |     "x = x_bc[1:-1]\n",
652 |     "delta_x = (b - a) / (N + 1)\n",
653 |     "\n",
654 |     "# Construct matrix A\n",
655 |     "A = numpy.zeros((N + 2, N + 2))\n",
656 |     "diagonal = numpy.ones(N + 2) / delta_x**2\n",
657 |     "A += numpy.diag(diagonal * -2.0, 0)\n",
658 |     "A += numpy.diag(diagonal[:-1], 1)\n",
659 |     "A += numpy.diag(diagonal[:-1], -1)\n",
660 |     "\n",
661 |     "# Construct RHS\n",
662 |     "b = f(x_bc)\n",
663 |     "\n",
664 |     "# Boundary conditions\n",
665 |     "A[0, 0] = 1.0\n",
666 |     "A[0, 1] = 0.0\n",
667 |     "A[-1, -1] = 3.0 / (2.0 * delta_x)\n",
668 |     "A[-1, -2] = -4.0 / (2.0 * delta_x)\n",
669 |     "A[-1, -3] = 1.0 / (2.0 * delta_x)\n",
670 |     "\n",
671 |     "b[0] = u_a\n",
672 |     "b[-1] = u_x_b\n",
673 |     "\n",
674 |     "# Solve system\n",
675 |     "U = numpy.empty(N + 2)\n",
676 |     "U = numpy.linalg.solve(A, b)\n",
677 |     "\n",
678 |     "# Plot result\n",
679 |     "fig = plt.figure()\n",
680 |     "axes = fig.add_subplot(1, 1, 1)\n",
681 |     "axes.plot(x_bc, U, 'o', label=\"Computed\")\n",
682 |     "axes.plot(x_bc, u_true(x_bc), 'k', label=\"True\")\n",
683 |     "axes.set_title(\"Solution to $u_{xx} = e^x$\")\n",
684 |     "axes.set_xlabel(\"x\")\n",
685 |     "axes.set_ylabel(\"u(x)\")\n",
686 |     "plt.show()"
687 |    ]
688 |   },
689 |   {
690 |    "cell_type": "markdown",
691 |    "metadata": {
692 |     "slideshow": {
693 |      "slide_type": "subslide"
694 |     }
695 |    },
696 |    "source": [
697 |     "## Ways to Solve $A u = f$\n",
698 |     "\n",
699 |     "We have proposed solving the linear system $A u = f$ which we have implemented naively above with the `numpy.linalg.solve` command but perhaps given the special structure of $A$ here that we can do better.   "
700 |    ]
701 |   },
702 |   {
703 |    "cell_type": "markdown",
704 |    "metadata": {
705 |     "slideshow": {
706 |      "slide_type": "subslide"
707 |     }
708 |    },
709 |    "source": [
710 |     "### Direct Methods (Gaussian Elimination)\n",
711 |     "\n",
712 |     "We could use Gaussian elimination to solve the system (or some factorization) which leads to a solution in a finite number of steps.  For large, sparse methods however these direct solvers are much more expensive in general over iterative solvers.  As was discussed for eigenproblems, iterative solvers start with an initial guess and try to improve on that guess. "
713 |    ]
714 |   },
715 |   {
716 |    "cell_type": "markdown",
717 |    "metadata": {
718 |     "slideshow": {
719 |      "slide_type": "subslide"
720 |     }
721 |    },
722 |    "source": [
723 |     "Consider a 3D Poisson Problem:\n",
724 |     " - Discretize using $100 \\times 100 \\times 100 = 10^6$ unknowns\n",
725 |     " - Gaussian elimination requires $\\mathcal{O}(N^3)$ operations\n",
726 |     " - Solving this system would take $10^{18}$ floating point operations to complete\n",
727 |     " \n",
728 |     "**How long?**\n",
729 |     "\n",
730 |     "Today's computer is $\\approx 100$ gigaflops (floating point operations per second) - $10^{11}$ flops / second.  We would be waiting 115 days for the solve to finish!\n",
731 |     "\n",
732 |     "**Memory?**\n",
733 |     "\n",
734 |     "Matrix requires $N^2$ memory locations ($N = 10^6$ here).  Single precision floating point storage (4-bytes per number) would require $4 \\times 10^{12}$ bytes of 4 terabytes of memory.\n",
735 |     "\n",
736 |     "The situation really is not as bad as we are making it out to be as long as we take advantage of the sparse nature of the matrices.  In fact for 1 dimensional problems direct methods can be reduced to $\\mathcal{O}(N)$ in the case for a tridiagonal system.  The situation is not so great for higher-dimensional problems however unless most structure can be leveraged.  Examples of these types of solvers include fast Fourier methods such as fast Poisson solvers."
737 |    ]
738 |   },
739 |   {
740 |    "cell_type": "markdown",
741 |    "metadata": {
742 |     "slideshow": {
743 |      "slide_type": "subslide"
744 |     }
745 |    },
746 |    "source": [
747 |     "### Iterative Methods\n",
748 |     "\n",
749 |     "Iterative methods take another tact that direct methods.  If we have the system $A x = b$ we form an iterative procedure that applies a function, say $L$, such that\n",
750 |     "$$\n",
751 |     "    \\hat{x}^{(k)} = L(\\hat{x}^{(k-1)})\n",
752 |     "$$\n",
753 |     "where we want errot between the real solution $x$ and $\\hat{x}^{(k)}$ goes to zero as $k \\rightarrow \\infty$.  We will explore these methods in the next lecture."
754 |    ]
755 |   }
756 |  ],
757 |  "metadata": {
758 |   "celltoolbar": "Slideshow",
759 |   "kernelspec": {
760 |    "display_name": "Python 3",
761 |    "language": "python",
762 |    "name": "python3"
763 |   },
764 |   "language_info": {
765 |    "codemirror_mode": {
766 |     "name": "ipython",
767 |     "version": 3
768 |    },
769 |    "file_extension": ".py",
770 |    "mimetype": "text/x-python",
771 |    "name": "python",
772 |    "nbconvert_exporter": "python",
773 |    "pygments_lexer": "ipython3",
774 |    "version": "3.7.0"
775 |   },
776 |   "latex_envs": {
777 |    "bibliofile": "biblio.bib",
778 |    "cite_by": "apalike",
779 |    "current_citInitial": 1,
780 |    "eqLabelWithNumbers": true,
781 |    "eqNumInitial": 0
782 |   }
783 |  },
784 |  "nbformat": 4,
785 |  "nbformat_minor": 1
786 | }
787 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
  1 | Attribution 4.0 International
  2 | 
  3 | =======================================================================
  4 | 
  5 | Creative Commons Corporation ("Creative Commons") is not a law firm and
  6 | does not provide legal services or legal advice. Distribution of
  7 | Creative Commons public licenses does not create a lawyer-client or
  8 | other relationship. Creative Commons makes its licenses and related
  9 | information available on an "as-is" basis. Creative Commons gives no
 10 | warranties regarding its licenses, any material licensed under their
 11 | terms and conditions, or any related information. Creative Commons
 12 | disclaims all liability for damages resulting from their use to the
 13 | fullest extent possible.
 14 | 
 15 | Using Creative Commons Public Licenses
 16 | 
 17 | Creative Commons public licenses provide a standard set of terms and
 18 | conditions that creators and other rights holders may use to share
 19 | original works of authorship and other material subject to copyright
 20 | and certain other rights specified in the public license below. The
 21 | following considerations are for informational purposes only, are not
 22 | exhaustive, and do not form part of our licenses.
 23 | 
 24 |      Considerations for licensors: Our public licenses are
 25 |      intended for use by those authorized to give the public
 26 |      permission to use material in ways otherwise restricted by
 27 |      copyright and certain other rights. Our licenses are
 28 |      irrevocable. Licensors should read and understand the terms
 29 |      and conditions of the license they choose before applying it.
 30 |      Licensors should also secure all rights necessary before
 31 |      applying our licenses so that the public can reuse the
 32 |      material as expected. Licensors should clearly mark any
 33 |      material not subject to the license. This includes other CC-
 34 |      licensed material, or material used under an exception or
 35 |      limitation to copyright. More considerations for licensors:
 36 | 	wiki.creativecommons.org/Considerations_for_licensors
 37 | 
 38 |      Considerations for the public: By using one of our public
 39 |      licenses, a licensor grants the public permission to use the
 40 |      licensed material under specified terms and conditions. If
 41 |      the licensor's permission is not necessary for any reason--for
 42 |      example, because of any applicable exception or limitation to
 43 |      copyright--then that use is not regulated by the license. Our
 44 |      licenses grant only permissions under copyright and certain
 45 |      other rights that a licensor has authority to grant. Use of
 46 |      the licensed material may still be restricted for other
 47 |      reasons, including because others have copyright or other
 48 |      rights in the material. A licensor may make special requests,
 49 |      such as asking that all changes be marked or described.
 50 |      Although not required by our licenses, you are encouraged to
 51 |      respect those requests where reasonable. More_considerations
 52 |      for the public:
 53 | 	wiki.creativecommons.org/Considerations_for_licensees
 54 | 
 55 | =======================================================================
 56 | 
 57 | Creative Commons Attribution 4.0 International Public License
 58 | 
 59 | By exercising the Licensed Rights (defined below), You accept and agree
 60 | to be bound by the terms and conditions of this Creative Commons
 61 | Attribution 4.0 International Public License ("Public License"). To the
 62 | extent this Public License may be interpreted as a contract, You are
 63 | granted the Licensed Rights in consideration of Your acceptance of
 64 | these terms and conditions, and the Licensor grants You such rights in
 65 | consideration of benefits the Licensor receives from making the
 66 | Licensed Material available under these terms and conditions.
 67 | 
 68 | 
 69 | Section 1 -- Definitions.
 70 | 
 71 |   a. Adapted Material means material subject to Copyright and Similar
 72 |      Rights that is derived from or based upon the Licensed Material
 73 |      and in which the Licensed Material is translated, altered,
 74 |      arranged, transformed, or otherwise modified in a manner requiring
 75 |      permission under the Copyright and Similar Rights held by the
 76 |      Licensor. For purposes of this Public License, where the Licensed
 77 |      Material is a musical work, performance, or sound recording,
 78 |      Adapted Material is always produced where the Licensed Material is
 79 |      synched in timed relation with a moving image.
 80 | 
 81 |   b. Adapter's License means the license You apply to Your Copyright
 82 |      and Similar Rights in Your contributions to Adapted Material in
 83 |      accordance with the terms and conditions of this Public License.
 84 | 
 85 |   c. Copyright and Similar Rights means copyright and/or similar rights
 86 |      closely related to copyright including, without limitation,
 87 |      performance, broadcast, sound recording, and Sui Generis Database
 88 |      Rights, without regard to how the rights are labeled or
 89 |      categorized. For purposes of this Public License, the rights
 90 |      specified in Section 2(b)(1)-(2) are not Copyright and Similar
 91 |      Rights.
 92 | 
 93 |   d. Effective Technological Measures means those measures that, in the
 94 |      absence of proper authority, may not be circumvented under laws
 95 |      fulfilling obligations under Article 11 of the WIPO Copyright
 96 |      Treaty adopted on December 20, 1996, and/or similar international
 97 |      agreements.
 98 | 
 99 |   e. Exceptions and Limitations means fair use, fair dealing, and/or
100 |      any other exception or limitation to Copyright and Similar Rights
101 |      that applies to Your use of the Licensed Material.
102 | 
103 |   f. Licensed Material means the artistic or literary work, database,
104 |      or other material to which the Licensor applied this Public
105 |      License.
106 | 
107 |   g. Licensed Rights means the rights granted to You subject to the
108 |      terms and conditions of this Public License, which are limited to
109 |      all Copyright and Similar Rights that apply to Your use of the
110 |      Licensed Material and that the Licensor has authority to license.
111 | 
112 |   h. Licensor means the individual(s) or entity(ies) granting rights
113 |      under this Public License.
114 | 
115 |   i. Share means to provide material to the public by any means or
116 |      process that requires permission under the Licensed Rights, such
117 |      as reproduction, public display, public performance, distribution,
118 |      dissemination, communication, or importation, and to make material
119 |      available to the public including in ways that members of the
120 |      public may access the material from a place and at a time
121 |      individually chosen by them.
122 | 
123 |   j. Sui Generis Database Rights means rights other than copyright
124 |      resulting from Directive 96/9/EC of the European Parliament and of
125 |      the Council of 11 March 1996 on the legal protection of databases,
126 |      as amended and/or succeeded, as well as other essentially
127 |      equivalent rights anywhere in the world.
128 | 
129 |   k. You means the individual or entity exercising the Licensed Rights
130 |      under this Public License. Your has a corresponding meaning.
131 | 
132 | 
133 | Section 2 -- Scope.
134 | 
135 |   a. License grant.
136 | 
137 |        1. Subject to the terms and conditions of this Public License,
138 |           the Licensor hereby grants You a worldwide, royalty-free,
139 |           non-sublicensable, non-exclusive, irrevocable license to
140 |           exercise the Licensed Rights in the Licensed Material to:
141 | 
142 |             a. reproduce and Share the Licensed Material, in whole or
143 |                in part; and
144 | 
145 |             b. produce, reproduce, and Share Adapted Material.
146 | 
147 |        2. Exceptions and Limitations. For the avoidance of doubt, where
148 |           Exceptions and Limitations apply to Your use, this Public
149 |           License does not apply, and You do not need to comply with
150 |           its terms and conditions.
151 | 
152 |        3. Term. The term of this Public License is specified in Section
153 |           6(a).
154 | 
155 |        4. Media and formats; technical modifications allowed. The
156 |           Licensor authorizes You to exercise the Licensed Rights in
157 |           all media and formats whether now known or hereafter created,
158 |           and to make technical modifications necessary to do so. The
159 |           Licensor waives and/or agrees not to assert any right or
160 |           authority to forbid You from making technical modifications
161 |           necessary to exercise the Licensed Rights, including
162 |           technical modifications necessary to circumvent Effective
163 |           Technological Measures. For purposes of this Public License,
164 |           simply making modifications authorized by this Section 2(a)
165 |           (4) never produces Adapted Material.
166 | 
167 |        5. Downstream recipients.
168 | 
169 |             a. Offer from the Licensor -- Licensed Material. Every
170 |                recipient of the Licensed Material automatically
171 |                receives an offer from the Licensor to exercise the
172 |                Licensed Rights under the terms and conditions of this
173 |                Public License.
174 | 
175 |             b. No downstream restrictions. You may not offer or impose
176 |                any additional or different terms or conditions on, or
177 |                apply any Effective Technological Measures to, the
178 |                Licensed Material if doing so restricts exercise of the
179 |                Licensed Rights by any recipient of the Licensed
180 |                Material.
181 | 
182 |        6. No endorsement. Nothing in this Public License constitutes or
183 |           may be construed as permission to assert or imply that You
184 |           are, or that Your use of the Licensed Material is, connected
185 |           with, or sponsored, endorsed, or granted official status by,
186 |           the Licensor or others designated to receive attribution as
187 |           provided in Section 3(a)(1)(A)(i).
188 | 
189 |   b. Other rights.
190 | 
191 |        1. Moral rights, such as the right of integrity, are not
192 |           licensed under this Public License, nor are publicity,
193 |           privacy, and/or other similar personality rights; however, to
194 |           the extent possible, the Licensor waives and/or agrees not to
195 |           assert any such rights held by the Licensor to the limited
196 |           extent necessary to allow You to exercise the Licensed
197 |           Rights, but not otherwise.
198 | 
199 |        2. Patent and trademark rights are not licensed under this
200 |           Public License.
201 | 
202 |        3. To the extent possible, the Licensor waives any right to
203 |           collect royalties from You for the exercise of the Licensed
204 |           Rights, whether directly or through a collecting society
205 |           under any voluntary or waivable statutory or compulsory
206 |           licensing scheme. In all other cases the Licensor expressly
207 |           reserves any right to collect such royalties.
208 | 
209 | 
210 | Section 3 -- License Conditions.
211 | 
212 | Your exercise of the Licensed Rights is expressly made subject to the
213 | following conditions.
214 | 
215 |   a. Attribution.
216 | 
217 |        1. If You Share the Licensed Material (including in modified
218 |           form), You must:
219 | 
220 |             a. retain the following if it is supplied by the Licensor
221 |                with the Licensed Material:
222 | 
223 |                  i. identification of the creator(s) of the Licensed
224 |                     Material and any others designated to receive
225 |                     attribution, in any reasonable manner requested by
226 |                     the Licensor (including by pseudonym if
227 |                     designated);
228 | 
229 |                 ii. a copyright notice;
230 | 
231 |                iii. a notice that refers to this Public License;
232 | 
233 |                 iv. a notice that refers to the disclaimer of
234 |                     warranties;
235 | 
236 |                  v. a URI or hyperlink to the Licensed Material to the
237 |                     extent reasonably practicable;
238 | 
239 |             b. indicate if You modified the Licensed Material and
240 |                retain an indication of any previous modifications; and
241 | 
242 |             c. indicate the Licensed Material is licensed under this
243 |                Public License, and include the text of, or the URI or
244 |                hyperlink to, this Public License.
245 | 
246 |        2. You may satisfy the conditions in Section 3(a)(1) in any
247 |           reasonable manner based on the medium, means, and context in
248 |           which You Share the Licensed Material. For example, it may be
249 |           reasonable to satisfy the conditions by providing a URI or
250 |           hyperlink to a resource that includes the required
251 |           information.
252 | 
253 |        3. If requested by the Licensor, You must remove any of the
254 |           information required by Section 3(a)(1)(A) to the extent
255 |           reasonably practicable.
256 | 
257 |        4. If You Share Adapted Material You produce, the Adapter's
258 |           License You apply must not prevent recipients of the Adapted
259 |           Material from complying with this Public License.
260 | 
261 | 
262 | Section 4 -- Sui Generis Database Rights.
263 | 
264 | Where the Licensed Rights include Sui Generis Database Rights that
265 | apply to Your use of the Licensed Material:
266 | 
267 |   a. for the avoidance of doubt, Section 2(a)(1) grants You the right
268 |      to extract, reuse, reproduce, and Share all or a substantial
269 |      portion of the contents of the database;
270 | 
271 |   b. if You include all or a substantial portion of the database
272 |      contents in a database in which You have Sui Generis Database
273 |      Rights, then the database in which You have Sui Generis Database
274 |      Rights (but not its individual contents) is Adapted Material; and
275 | 
276 |   c. You must comply with the conditions in Section 3(a) if You Share
277 |      all or a substantial portion of the contents of the database.
278 | 
279 | For the avoidance of doubt, this Section 4 supplements and does not
280 | replace Your obligations under this Public License where the Licensed
281 | Rights include other Copyright and Similar Rights.
282 | 
283 | 
284 | Section 5 -- Disclaimer of Warranties and Limitation of Liability.
285 | 
286 |   a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE
287 |      EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS
288 |      AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF
289 |      ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS,
290 |      IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION,
291 |      WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR
292 |      PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS,
293 |      ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT
294 |      KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT
295 |      ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU.
296 | 
297 |   b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE
298 |      TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION,
299 |      NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT,
300 |      INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES,
301 |      COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR
302 |      USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN
303 |      ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR
304 |      DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR
305 |      IN PART, THIS LIMITATION MAY NOT APPLY TO YOU.
306 | 
307 |   c. The disclaimer of warranties and limitation of liability provided
308 |      above shall be interpreted in a manner that, to the extent
309 |      possible, most closely approximates an absolute disclaimer and
310 |      waiver of all liability.
311 | 
312 | 
313 | Section 6 -- Term and Termination.
314 | 
315 |   a. This Public License applies for the term of the Copyright and
316 |      Similar Rights licensed here. However, if You fail to comply with
317 |      this Public License, then Your rights under this Public License
318 |      terminate automatically.
319 | 
320 |   b. Where Your right to use the Licensed Material has terminated under
321 |      Section 6(a), it reinstates:
322 | 
323 |        1. automatically as of the date the violation is cured, provided
324 |           it is cured within 30 days of Your discovery of the
325 |           violation; or
326 | 
327 |        2. upon express reinstatement by the Licensor.
328 | 
329 |      For the avoidance of doubt, this Section 6(b) does not affect any
330 |      right the Licensor may have to seek remedies for Your violations
331 |      of this Public License.
332 | 
333 |   c. For the avoidance of doubt, the Licensor may also offer the
334 |      Licensed Material under separate terms or conditions or stop
335 |      distributing the Licensed Material at any time; however, doing so
336 |      will not terminate this Public License.
337 | 
338 |   d. Sections 1, 5, 6, 7, and 8 survive termination of this Public
339 |      License.
340 | 
341 | 
342 | Section 7 -- Other Terms and Conditions.
343 | 
344 |   a. The Licensor shall not be bound by any additional or different
345 |      terms or conditions communicated by You unless expressly agreed.
346 | 
347 |   b. Any arrangements, understandings, or agreements regarding the
348 |      Licensed Material not stated herein are separate from and
349 |      independent of the terms and conditions of this Public License.
350 | 
351 | 
352 | Section 8 -- Interpretation.
353 | 
354 |   a. For the avoidance of doubt, this Public License does not, and
355 |      shall not be interpreted to, reduce, limit, restrict, or impose
356 |      conditions on any use of the Licensed Material that could lawfully
357 |      be made without permission under this Public License.
358 | 
359 |   b. To the extent possible, if any provision of this Public License is
360 |      deemed unenforceable, it shall be automatically reformed to the
361 |      minimum extent necessary to make it enforceable. If the provision
362 |      cannot be reformed, it shall be severed from this Public License
363 |      without affecting the enforceability of the remaining terms and
364 |      conditions.
365 | 
366 |   c. No term or condition of this Public License will be waived and no
367 |      failure to comply consented to unless expressly agreed to by the
368 |      Licensor.
369 | 
370 |   d. Nothing in this Public License constitutes or may be interpreted
371 |      as a limitation upon, or waiver of, any privileges and immunities
372 |      that apply to the Licensor or You, including from the legal
373 |      processes of any jurisdiction or authority.
374 | 
375 | 
376 | =======================================================================
377 | 
378 | Creative Commons is not a party to its public
379 | licenses. Notwithstanding, Creative Commons may elect to apply one of
380 | its public licenses to material it publishes and in those instances
381 | will be considered the “Licensor.” The text of the Creative Commons
382 | public licenses is dedicated to the public domain under the CC0 Public
383 | Domain Dedication. Except for the limited purpose of indicating that
384 | material is shared under a Creative Commons public license or as
385 | otherwise permitted by the Creative Commons policies published at
386 | creativecommons.org/policies, Creative Commons does not authorize the
387 | use of the trademark "Creative Commons" or any other trademark or logo
388 | of Creative Commons without its prior written consent including,
389 | without limitation, in connection with any unauthorized modifications
390 | to any of its public licenses or any other arrangements,
391 | understandings, or agreements concerning use of licensed material. For
392 | the avoidance of doubt, this paragraph does not form part of the
393 | public licenses.
394 | 
395 | Creative Commons may be contacted at creativecommons.org.
396 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # intro-numerical-methods
2 | Jupyter notebooks and other materials developed for the Columbia course APMA 4300
3 | 
4 | [![Build Status](https://travis-ci.org/mandli/intro-numerical-methods.svg)](https://travis-ci.org/mandli/intro-numerical-methods)
5 | 
6 | <a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
7 | 
8 | [![Binder](http://mybinder.org/badge.svg)](http://mybinder.org:/repo/mandli/intro-numerical-methods)
9 | 


--------------------------------------------------------------------------------
/data/sunspot.dat:
--------------------------------------------------------------------------------
  1 | 1700    5.0
  2 | 1701   11.0
  3 | 1702   16.0
  4 | 1703   23.0
  5 | 1704   36.0
  6 | 1705   58.0
  7 | 1706   29.0
  8 | 1707   20.0
  9 | 1708   10.0
 10 | 1709    8.0
 11 | 1710    3.0
 12 | 1711    0.0
 13 | 1712    0.0
 14 | 1713    2.0
 15 | 1714   11.0
 16 | 1715   27.0
 17 | 1716   47.0
 18 | 1717   63.0
 19 | 1718   60.0
 20 | 1719   39.0
 21 | 1720   28.0
 22 | 1721   26.0
 23 | 1722   22.0
 24 | 1723   11.0
 25 | 1724   21.0
 26 | 1725   40.0
 27 | 1726   78.0
 28 | 1727  122.0
 29 | 1728  103.0
 30 | 1729   73.0
 31 | 1730   47.0
 32 | 1731   35.0
 33 | 1732   11.0
 34 | 1733    5.0
 35 | 1734   16.0
 36 | 1735   34.0
 37 | 1736   70.0
 38 | 1737   81.0
 39 | 1738  111.0
 40 | 1739  101.0
 41 | 1740   73.0
 42 | 1741   40.0
 43 | 1742   20.0
 44 | 1743   16.0
 45 | 1744    5.0
 46 | 1745   11.0
 47 | 1746   22.0
 48 | 1747   40.0
 49 | 1748   60.0
 50 | 1749   80.9
 51 | 1750   83.4
 52 | 1751   47.7
 53 | 1752   47.8
 54 | 1753   30.7
 55 | 1754   12.2
 56 | 1755    9.6
 57 | 1756   10.2
 58 | 1757   32.4
 59 | 1758   47.6
 60 | 1759   54.0
 61 | 1760   62.9
 62 | 1761   85.9
 63 | 1762   61.2
 64 | 1763   45.1
 65 | 1764   36.4
 66 | 1765   20.9
 67 | 1766   11.4
 68 | 1767   37.8
 69 | 1768   69.8
 70 | 1769  106.1
 71 | 1770  100.8
 72 | 1771   81.6
 73 | 1772   66.5
 74 | 1773   34.8
 75 | 1774   30.6
 76 | 1775    7.0
 77 | 1776   19.8
 78 | 1777   92.5
 79 | 1778  154.4
 80 | 1779  125.9
 81 | 1780   84.8
 82 | 1781   68.1
 83 | 1782   38.5
 84 | 1783   22.8
 85 | 1784   10.2
 86 | 1785   24.1
 87 | 1786   82.9
 88 | 1787  132.0
 89 | 1788  130.9
 90 | 1789  118.1
 91 | 1790   89.9
 92 | 1791   66.6
 93 | 1792   60.0
 94 | 1793   46.9
 95 | 1794   41.0
 96 | 1795   21.3
 97 | 1796   16.0
 98 | 1797    6.4
 99 | 1798    4.1
100 | 1799    6.8
101 | 1800   14.5
102 | 1801   34.0
103 | 1802   45.0
104 | 1803   43.1
105 | 1804   47.5
106 | 1805   42.2
107 | 1806   28.1
108 | 1807   10.1
109 | 1808    8.1
110 | 1809    2.5
111 | 1810    0.0
112 | 1811    1.4
113 | 1812    5.0
114 | 1813   12.2
115 | 1814   13.9
116 | 1815   35.4
117 | 1816   45.8
118 | 1817   41.1
119 | 1818   30.1
120 | 1819   23.9
121 | 1820   15.6
122 | 1821    6.6
123 | 1822    4.0
124 | 1823    1.8
125 | 1824    8.5
126 | 1825   16.6
127 | 1826   36.3
128 | 1827   49.6
129 | 1828   64.2
130 | 1829   67.0
131 | 1830   70.9
132 | 1831   47.8
133 | 1832   27.5
134 | 1833    8.5
135 | 1834   13.2
136 | 1835   56.9
137 | 1836  121.5
138 | 1837  138.3
139 | 1838  103.2
140 | 1839   85.7
141 | 1840   64.6
142 | 1841   36.7
143 | 1842   24.2
144 | 1843   10.7
145 | 1844   15.0
146 | 1845   40.1
147 | 1846   61.5
148 | 1847   98.5
149 | 1848  124.7
150 | 1849   96.3
151 | 1850   66.6
152 | 1851   64.5
153 | 1852   54.1
154 | 1853   39.0
155 | 1854   20.6
156 | 1855    6.7
157 | 1856    4.3
158 | 1857   22.7
159 | 1858   54.8
160 | 1859   93.8
161 | 1860   95.8
162 | 1861   77.2
163 | 1862   59.1
164 | 1863   44.0
165 | 1864   47.0
166 | 1865   30.5
167 | 1866   16.3
168 | 1867    7.3
169 | 1868   37.6
170 | 1869   74.0
171 | 1870  139.0
172 | 1871  111.2
173 | 1872  101.6
174 | 1873   66.2
175 | 1874   44.7
176 | 1875   17.0
177 | 1876   11.3
178 | 1877   12.4
179 | 1878    3.4
180 | 1879    6.0
181 | 1880   32.3
182 | 1881   54.3
183 | 1882   59.7
184 | 1883   63.7
185 | 1884   63.5
186 | 1885   52.2
187 | 1886   25.4
188 | 1887   13.1
189 | 1888    6.8
190 | 1889    6.3
191 | 1890    7.1
192 | 1891   35.6
193 | 1892   73.0
194 | 1893   85.1
195 | 1894   78.0
196 | 1895   64.0
197 | 1896   41.8
198 | 1897   26.2
199 | 1898   26.7
200 | 1899   12.1
201 | 1900    9.5
202 | 1901    2.7
203 | 1902    5.0
204 | 1903   24.4
205 | 1904   42.0
206 | 1905   63.5
207 | 1906   53.8
208 | 1907   62.0
209 | 1908   48.5
210 | 1909   43.9
211 | 1910   18.6
212 | 1911    5.7
213 | 1912    3.6
214 | 1913    1.4
215 | 1914    9.6
216 | 1915   47.4
217 | 1916   57.1
218 | 1917  103.9
219 | 1918   80.6
220 | 1919   63.6
221 | 1920   37.6
222 | 1921   26.1
223 | 1922   14.2
224 | 1923    5.8
225 | 1924   16.7
226 | 1925   44.3
227 | 1926   63.9
228 | 1927   69.0
229 | 1928   77.8
230 | 1929   64.9
231 | 1930   35.7
232 | 1931   21.2
233 | 1932   11.1
234 | 1933    5.7
235 | 1934    8.7
236 | 1935   36.1
237 | 1936   79.7
238 | 1937  114.4
239 | 1938  109.6
240 | 1939   88.8
241 | 1940   67.8
242 | 1941   47.5
243 | 1942   30.6
244 | 1943   16.3
245 | 1944    9.6
246 | 1945   33.2
247 | 1946   92.6
248 | 1947  151.6
249 | 1948  136.3
250 | 1949  134.7
251 | 1950   83.9
252 | 1951   69.4
253 | 1952   31.5
254 | 1953   13.9
255 | 1954    4.4
256 | 1955   38.0
257 | 1956  141.7
258 | 1957  190.2
259 | 1958  184.8
260 | 1959  159.0
261 | 1960  112.3
262 | 1961   53.9
263 | 1962   37.5
264 | 1963   27.9
265 | 1964   10.2
266 | 1965   15.1
267 | 1966   47.0
268 | 1967   93.8
269 | 1968  105.9
270 | 1969  105.5
271 | 1970  104.5
272 | 1971   66.6
273 | 1972   68.9
274 | 1973   38.0
275 | 1974   34.5
276 | 1975   15.5
277 | 1976   12.6
278 | 1977   27.5
279 | 1978   92.5
280 | 1979  155.4
281 | 1980  154.6
282 | 1981  140.4
283 | 1982  115.9
284 | 1983   66.6
285 | 1984   45.9
286 | 1985   17.9
287 | 1986   13.4
288 | 1987   29.3
289 | 


--------------------------------------------------------------------------------
/extras/00_intro_numerical_methods-extra-examples.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {
  6 |     "hide_input": false,
  7 |     "slideshow": {
  8 |      "slide_type": "skip"
  9 |     }
 10 |    },
 11 |    "source": [
 12 |     "<table>\n",
 13 |     " <tr align=left><td><img align=left src=\"./images/CC-BY.png\">\n",
 14 |     " <td>Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli</td>\n",
 15 |     "</table>"
 16 |    ]
 17 |   },
 18 |   {
 19 |    "cell_type": "code",
 20 |    "execution_count": null,
 21 |    "metadata": {
 22 |     "hide_input": false,
 23 |     "slideshow": {
 24 |      "slide_type": "skip"
 25 |     }
 26 |    },
 27 |    "outputs": [],
 28 |    "source": [
 29 |     "from __future__ import print_function\n",
 30 |     "\n",
 31 |     "%matplotlib inline\n",
 32 |     "import numpy\n",
 33 |     "import matplotlib.pyplot as plt"
 34 |    ]
 35 |   },
 36 |   {
 37 |    "cell_type": "markdown",
 38 |    "metadata": {
 39 |     "slideshow": {
 40 |      "slide_type": "skip"
 41 |     }
 42 |    },
 43 |    "source": [
 44 |     "Note to lecturers:  This notebook is designed to work best as a classic Jupyter Notebook with nbextensions \n",
 45 |     "* hide_input: to hide selected python cells particularly for just plotting\n",
 46 |     "* RISE:  Interactive js slide presentations"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "markdown",
 51 |    "metadata": {
 52 |     "slideshow": {
 53 |      "slide_type": "skip"
 54 |     }
 55 |    },
 56 |    "source": [
 57 |     "## Why should I care?"
 58 |    ]
 59 |   },
 60 |   {
 61 |    "cell_type": "markdown",
 62 |    "metadata": {
 63 |     "slideshow": {
 64 |      "slide_type": "skip"
 65 |     }
 66 |    },
 67 |    "source": [
 68 |     "*Because I have to be here for a requirement...*"
 69 |    ]
 70 |   },
 71 |   {
 72 |    "cell_type": "markdown",
 73 |    "metadata": {
 74 |     "slideshow": {
 75 |      "slide_type": "skip"
 76 |     }
 77 |    },
 78 |    "source": [
 79 |     "*or I might actually need to solve something important...like can I ever retire?*"
 80 |    ]
 81 |   },
 82 |   {
 83 |    "cell_type": "markdown",
 84 |    "metadata": {
 85 |     "slideshow": {
 86 |      "slide_type": "skip"
 87 |     }
 88 |    },
 89 |    "source": [
 90 |     "### An example: The Retirement Problem\n",
 91 |     "\n",
 92 |     "$$A = \\frac{P}{r} \\left((1+r)^n - 1 \\right)$$\n",
 93 |     "\n",
 94 |     "$A$ is the total amount after n payments\n",
 95 |     "\n",
 96 |     "$P$ is the incremental payment\n",
 97 |     "\n",
 98 |     "$r$ is the interest rate per payment period\n",
 99 |     "\n",
100 |     "$n$ is the number of payments\n",
101 |     "\n",
102 |     "\n",
103 |     "\n",
104 |     "Note that these can all be functions of $r$, $n$, and time"
105 |    ]
106 |   },
107 |   {
108 |    "cell_type": "markdown",
109 |    "metadata": {
110 |     "hide_input": true,
111 |     "slideshow": {
112 |      "slide_type": "subslide"
113 |     }
114 |    },
115 |    "source": [
116 |     "What is the minimum interest rate $r$ required to accrue \\$1M over 20 years saving \\\\$1000 per month?\n",
117 |     "\n",
118 |     "$$\n",
119 |     "A = \\frac{P}{r} \\left((1+r)^n - 1 \\right)\n",
120 |     "$$"
121 |    ]
122 |   },
123 |   {
124 |    "cell_type": "code",
125 |    "execution_count": null,
126 |    "metadata": {
127 |     "hide_input": false,
128 |     "jupyter": {
129 |      "source_hidden": true
130 |     },
131 |     "slideshow": {
132 |      "slide_type": "skip"
133 |     }
134 |    },
135 |    "outputs": [],
136 |    "source": [
137 |     "def A(P, r, n):\n",
138 |     "    return P / r * ((1 + r)**n - 1)\n",
139 |     "\n",
140 |     "P = 1000.\n",
141 |     "years = numpy.linspace(0,20,241)\n",
142 |     "n = 12*years\n",
143 |     "target = 1.e6"
144 |    ]
145 |   },
146 |   {
147 |    "cell_type": "code",
148 |    "execution_count": null,
149 |    "metadata": {
150 |     "hide_input": false,
151 |     "jupyter": {
152 |      "source_hidden": true
153 |     },
154 |     "slideshow": {
155 |      "slide_type": "skip"
156 |     }
157 |    },
158 |    "outputs": [],
159 |    "source": [
160 |     "# find the root using scipy's brentq method\n",
161 |     "from scipy.optimize import brentq\n",
162 |     "n_max = n[-1]\n",
163 |     "f = lambda r : target - A(P,r/12., n_max)\n",
164 |     "r_target = brentq(f,0.1,0.15)"
165 |    ]
166 |   },
167 |   {
168 |    "cell_type": "code",
169 |    "execution_count": null,
170 |    "metadata": {
171 |     "hide_input": true,
172 |     "jupyter": {
173 |      "source_hidden": true
174 |     },
175 |     "slideshow": {
176 |      "slide_type": "fragment"
177 |     }
178 |    },
179 |    "outputs": [],
180 |    "source": [
181 |     "fig = plt.figure(figsize=(8,6))\n",
182 |     "fig.set_figwidth(fig.get_figwidth() * 2)\n",
183 |     "\n",
184 |     "axes = fig.add_subplot(1, 2, 1)\n",
185 |     "for r in [0.02, 0.05, 0.08, 0.1, 0.12, 0.15]:\n",
186 |     "    axes.plot(years, A(P, r/12., n),label='r = {}'.format(r))\n",
187 |     "axes.plot(years, numpy.ones(years.shape) * target, 'k--',linewidth=2,label=\"Target\")\n",
188 |     "axes.legend(loc='best')\n",
189 |     "axes.set_xlabel(\"Years\",fontsize=18)\n",
190 |     "axes.set_ylabel(\"Annuity Value (Dollars)\",fontsize=18)\n",
191 |     "axes.set_title(\"The Forward Problem $A(P,r,n)$\",fontsize=20)\n",
192 |     "axes.grid()\n",
193 |     "\n",
194 |     "axes = fig.add_subplot(1, 2, 2)\n",
195 |     "r = numpy.linspace(1.e-6,0.15,100)\n",
196 |     "target = 1.e6\n",
197 |     "axes.plot(A(P,r/12.,n_max),r,linewidth=2)\n",
198 |     "axes.plot(numpy.ones(r.shape) * target, r,'k--',linewidth=2)\n",
199 |     "axes.scatter(A(P,r_target/12,n_max),r_target,s=100,c='r')\n",
200 |     "axes.set_xlabel('Annuity Value (Dollars)',fontsize=18)\n",
201 |     "axes.set_title('The Inverse Problem $r(A,P,n)$, $r\\geq$ {:.3f}'.format(r_target),fontsize=20)\n",
202 |     "axes.grid()\n",
203 |     "plt.show()"
204 |    ]
205 |   },
206 |   {
207 |    "cell_type": "markdown",
208 |    "metadata": {
209 |     "slideshow": {
210 |      "slide_type": "fragment"
211 |     }
212 |    },
213 |    "source": [
214 |     "This is a rootfinding problem for a function of a single variable"
215 |    ]
216 |   },
217 |   {
218 |    "cell_type": "markdown",
219 |    "metadata": {
220 |     "slideshow": {
221 |      "slide_type": "skip"
222 |     }
223 |    },
224 |    "source": [
225 |     "### Another Example:  Boat race (numerical quadrature of arc length)\n",
226 |     "Given a river with coordinates $(x,f(x))$ find the total length actually rowed over a given interval $x\\in [0,X]$\n",
227 |     "\n",
228 |     "Example:  a sinusoidal river $$f(x) = A \\sin x$$"
229 |    ]
230 |   },
231 |   {
232 |    "cell_type": "code",
233 |    "execution_count": null,
234 |    "metadata": {
235 |     "hide_input": true,
236 |     "slideshow": {
237 |      "slide_type": "skip"
238 |     }
239 |    },
240 |    "outputs": [],
241 |    "source": [
242 |     "A=.5\n",
243 |     "fig = plt.figure(figsize=(8,6))\n",
244 |     "x = numpy.linspace(0,10,100)\n",
245 |     "plt.plot(x,A*numpy.sin(x),'r',linewidth=2,)\n",
246 |     "plt.xlabel('distance (km)')\n",
247 |     "plt.grid()\n",
248 |     "plt.title('The Boat Race: $f(x)=A\\sin(x)')\n",
249 |     "plt.show()"
250 |    ]
251 |   },
252 |   {
253 |    "cell_type": "markdown",
254 |    "metadata": {
255 |     "slideshow": {
256 |      "slide_type": "skip"
257 |     }
258 |    },
259 |    "source": [
260 |     "We need to calculate the function $f(x)$'s arc-length from $[0, X]$ e.g.\n",
261 |     "\n",
262 |     "\\begin{align}\n",
263 |     "    L(X) &= \\int_0^{X} \\sqrt{1 + |f'(x)|^2} dx\\\\\n",
264 |     "         &= \\int_0^{X} \\sqrt{1 + A^2\\cos^2(x)} dx\\\\\n",
265 |     "\\end{align}\n",
266 |     "\n",
267 |     "In general the value of this integral cannot be solved analytically (the answer is actually an elliptic integral). We usually need to approximate the integral using a numerical quadrature."
268 |    ]
269 |   },
270 |   {
271 |    "cell_type": "markdown",
272 |    "metadata": {
273 |     "hide_input": true,
274 |     "slideshow": {
275 |      "slide_type": "skip"
276 |     }
277 |    },
278 |    "source": [
279 |     "## Why is this exciting?\n",
280 |     "\n",
281 |     "Computers have revolutionized our ability to explore...\n",
282 |     "\n",
283 |     "[Top 10 Algorithms of the 20th Century](http://www.siam.org/pdf/news/637.pdf?t=1&cn=ZmxleGlibGVfcmVjcw%3D%3D&refsrc=email&iid=658bdab6af614c83a8df1f8e02035eae&uid=755271476&nid=244+285282312)\n",
284 |     "\n",
285 |     "...and there is more to come!"
286 |    ]
287 |   }
288 |  ],
289 |  "metadata": {
290 |   "celltoolbar": "Slideshow",
291 |   "kernelspec": {
292 |    "display_name": "Python 3",
293 |    "language": "python",
294 |    "name": "python3"
295 |   },
296 |   "language_info": {
297 |    "codemirror_mode": {
298 |     "name": "ipython",
299 |     "version": 3
300 |    },
301 |    "file_extension": ".py",
302 |    "mimetype": "text/x-python",
303 |    "name": "python",
304 |    "nbconvert_exporter": "python",
305 |    "pygments_lexer": "ipython3",
306 |    "version": "3.7.5"
307 |   },
308 |   "latex_envs": {
309 |    "bibliofile": "biblio.bib",
310 |    "cite_by": "apalike",
311 |    "current_citInitial": 1,
312 |    "eqLabelWithNumbers": true,
313 |    "eqNumInitial": 0
314 |   }
315 |  },
316 |  "nbformat": 4,
317 |  "nbformat_minor": 4
318 | }
319 | 


--------------------------------------------------------------------------------
/images/CC-BY.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/CC-BY.png


--------------------------------------------------------------------------------
/images/Fenics_tc_vm.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/Fenics_tc_vm.png


--------------------------------------------------------------------------------
/images/MakeGoodChoices.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/MakeGoodChoices.gif


--------------------------------------------------------------------------------
/images/RuiningMyLife.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/RuiningMyLife.gif


--------------------------------------------------------------------------------
/images/bound_matrix_multiply.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/bound_matrix_multiply.png


--------------------------------------------------------------------------------
/images/ellipses.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/ellipses.png


--------------------------------------------------------------------------------
/images/ellipses_CG.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/ellipses_CG.png


--------------------------------------------------------------------------------
/images/golden_section.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/golden_section.png


--------------------------------------------------------------------------------
/images/horners_method_big_count.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/horners_method_big_count.png


--------------------------------------------------------------------------------
/images/householder.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/householder.png


--------------------------------------------------------------------------------
/images/linear_regression.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/linear_regression.png


--------------------------------------------------------------------------------
/images/lsq_projection.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/lsq_projection.png


--------------------------------------------------------------------------------
/images/mathModeling.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/mathModeling.png


--------------------------------------------------------------------------------
/images/mathModeling.svg:
--------------------------------------------------------------------------------
  1 | <?xml version="1.0" encoding="UTF-8" standalone="no"?>
  2 | <!-- Created with Inkscape (http://www.inkscape.org/) -->
  3 | 
  4 | <svg
  5 |    xmlns:dc="http://purl.org/dc/elements/1.1/"
  6 |    xmlns:cc="http://creativecommons.org/ns#"
  7 |    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  8 |    xmlns:svg="http://www.w3.org/2000/svg"
  9 |    xmlns="http://www.w3.org/2000/svg"
 10 |    xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
 11 |    xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
 12 |    width="210mm"
 13 |    height="297mm"
 14 |    viewBox="0 0 210 297"
 15 |    version="1.1"
 16 |    id="svg8"
 17 |    inkscape:version="0.92.4 5da689c313, 2019-01-14"
 18 |    sodipodi:docname="mathModeling.svg"
 19 |    inkscape:export-filename="/tmp/mathModeling.png"
 20 |    inkscape:export-xdpi="164.571"
 21 |    inkscape:export-ydpi="164.571">
 22 |   <defs
 23 |      id="defs2">
 24 |     <marker
 25 |        inkscape:isstock="true"
 26 |        style="overflow:visible;"
 27 |        id="marker16568"
 28 |        refX="0.0"
 29 |        refY="0.0"
 30 |        orient="auto"
 31 |        inkscape:stockid="Arrow1Lend">
 32 |       <path
 33 |          transform="scale(0.8) rotate(180) translate(12.5,0)"
 34 |          style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1;fill:#000000;fill-opacity:1"
 35 |          d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z "
 36 |          id="path16566" />
 37 |     </marker>
 38 |     <marker
 39 |        inkscape:isstock="true"
 40 |        style="overflow:visible"
 41 |        id="marker16492"
 42 |        refX="0.0"
 43 |        refY="0.0"
 44 |        orient="auto"
 45 |        inkscape:stockid="Arrow1Lstart">
 46 |       <path
 47 |          transform="scale(0.8) translate(12.5,0)"
 48 |          style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt;stroke-opacity:1;fill:#000000;fill-opacity:1"
 49 |          d="M 0.0,0.0 L 5.0,-5.0 L -12.5,0.0 L 5.0,5.0 L 0.0,0.0 z "
 50 |          id="path16490" />
 51 |     </marker>
 52 |     <inkscape:path-effect
 53 |        effect="powerstroke"
 54 |        id="path-effect4573"
 55 |        is_visible="true"
 56 |        offset_points="0.32,5.0799999"
 57 |        sort_points="true"
 58 |        interpolator_type="CubicBezierJohan"
 59 |        interpolator_beta="0.2"
 60 |        start_linecap_type="zerowidth"
 61 |        linejoin_type="extrp_arc"
 62 |        miter_limit="4"
 63 |        end_linecap_type="zerowidth" />
 64 |     <inkscape:path-effect
 65 |        effect="bspline"
 66 |        id="path-effect4571"
 67 |        is_visible="true"
 68 |        weight="33.333333"
 69 |        steps="2"
 70 |        helper_size="0"
 71 |        apply_no_weight="true"
 72 |        apply_with_weight="true"
 73 |        only_selected="false" />
 74 |     <marker
 75 |        inkscape:stockid="Arrow1Lstart"
 76 |        orient="auto"
 77 |        refY="0"
 78 |        refX="0"
 79 |        id="Arrow1Lstart-8"
 80 |        style="overflow:visible"
 81 |        inkscape:isstock="true">
 82 |       <path
 83 |          inkscape:connector-curvature="0"
 84 |          id="path4585-7"
 85 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
 86 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
 87 |          transform="matrix(0.8,0,0,0.8,10,0)" />
 88 |     </marker>
 89 |     <marker
 90 |        inkscape:stockid="Arrow1Lend"
 91 |        orient="auto"
 92 |        refY="0"
 93 |        refX="0"
 94 |        id="Arrow1Lend-4"
 95 |        style="overflow:visible"
 96 |        inkscape:isstock="true">
 97 |       <path
 98 |          inkscape:connector-curvature="0"
 99 |          id="path4588-1"
100 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
101 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
102 |          transform="matrix(-0.8,0,0,-0.8,-10,0)" />
103 |     </marker>
104 |     <marker
105 |        inkscape:stockid="Arrow1Lstart"
106 |        orient="auto"
107 |        refY="0"
108 |        refX="0"
109 |        id="Arrow1Lstart-3"
110 |        style="overflow:visible"
111 |        inkscape:isstock="true">
112 |       <path
113 |          inkscape:connector-curvature="0"
114 |          id="path4585-8"
115 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
116 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
117 |          transform="matrix(0.8,0,0,0.8,10,0)" />
118 |     </marker>
119 |     <marker
120 |        inkscape:stockid="Arrow1Lend"
121 |        orient="auto"
122 |        refY="0"
123 |        refX="0"
124 |        id="Arrow1Lend-5"
125 |        style="overflow:visible"
126 |        inkscape:isstock="true">
127 |       <path
128 |          inkscape:connector-curvature="0"
129 |          id="path4588-5"
130 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
131 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
132 |          transform="matrix(-0.8,0,0,-0.8,-10,0)" />
133 |     </marker>
134 |     <marker
135 |        inkscape:stockid="Arrow1Lstart"
136 |        orient="auto"
137 |        refY="0"
138 |        refX="0"
139 |        id="Arrow1Lstart-1"
140 |        style="overflow:visible"
141 |        inkscape:isstock="true">
142 |       <path
143 |          inkscape:connector-curvature="0"
144 |          id="path4585-2"
145 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
146 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
147 |          transform="matrix(0.8,0,0,0.8,10,0)" />
148 |     </marker>
149 |     <marker
150 |        inkscape:stockid="Arrow1Lend"
151 |        orient="auto"
152 |        refY="0"
153 |        refX="0"
154 |        id="Arrow1Lend-6"
155 |        style="overflow:visible"
156 |        inkscape:isstock="true">
157 |       <path
158 |          inkscape:connector-curvature="0"
159 |          id="path4588-4"
160 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
161 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
162 |          transform="matrix(-0.8,0,0,-0.8,-10,0)" />
163 |     </marker>
164 |     <marker
165 |        inkscape:isstock="true"
166 |        style="overflow:visible"
167 |        id="marker16568-5"
168 |        refX="0"
169 |        refY="0"
170 |        orient="auto"
171 |        inkscape:stockid="Arrow1Lend">
172 |       <path
173 |          inkscape:connector-curvature="0"
174 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
175 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
176 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
177 |          id="path16566-6" />
178 |     </marker>
179 |     <marker
180 |        inkscape:isstock="true"
181 |        style="overflow:visible"
182 |        id="marker16492-5"
183 |        refX="0"
184 |        refY="0"
185 |        orient="auto"
186 |        inkscape:stockid="Arrow1Lstart">
187 |       <path
188 |          inkscape:connector-curvature="0"
189 |          transform="matrix(0.8,0,0,0.8,10,0)"
190 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
191 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
192 |          id="path16490-2" />
193 |     </marker>
194 |     <marker
195 |        inkscape:isstock="true"
196 |        style="overflow:visible"
197 |        id="marker16568-6"
198 |        refX="0"
199 |        refY="0"
200 |        orient="auto"
201 |        inkscape:stockid="Arrow1Lend">
202 |       <path
203 |          inkscape:connector-curvature="0"
204 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
205 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
206 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
207 |          id="path16566-9" />
208 |     </marker>
209 |     <marker
210 |        inkscape:isstock="true"
211 |        style="overflow:visible"
212 |        id="marker16492-3"
213 |        refX="0"
214 |        refY="0"
215 |        orient="auto"
216 |        inkscape:stockid="Arrow1Lstart">
217 |       <path
218 |          inkscape:connector-curvature="0"
219 |          transform="matrix(0.8,0,0,0.8,10,0)"
220 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
221 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
222 |          id="path16490-1" />
223 |     </marker>
224 |     <marker
225 |        inkscape:isstock="true"
226 |        style="overflow:visible"
227 |        id="marker16568-52"
228 |        refX="0"
229 |        refY="0"
230 |        orient="auto"
231 |        inkscape:stockid="Arrow1Lend">
232 |       <path
233 |          inkscape:connector-curvature="0"
234 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
235 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
236 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
237 |          id="path16566-69" />
238 |     </marker>
239 |     <marker
240 |        inkscape:isstock="true"
241 |        style="overflow:visible"
242 |        id="marker16568-52-6"
243 |        refX="0"
244 |        refY="0"
245 |        orient="auto"
246 |        inkscape:stockid="Arrow1Lend">
247 |       <path
248 |          inkscape:connector-curvature="0"
249 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
250 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
251 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
252 |          id="path16566-69-0" />
253 |     </marker>
254 |     <marker
255 |        inkscape:isstock="true"
256 |        style="overflow:visible"
257 |        id="marker16492-3-7"
258 |        refX="0"
259 |        refY="0"
260 |        orient="auto"
261 |        inkscape:stockid="Arrow1Lstart">
262 |       <path
263 |          inkscape:connector-curvature="0"
264 |          transform="matrix(0.8,0,0,0.8,10,0)"
265 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
266 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
267 |          id="path16490-1-4" />
268 |     </marker>
269 |     <marker
270 |        inkscape:isstock="true"
271 |        style="overflow:visible"
272 |        id="marker16568-52-6-9"
273 |        refX="0"
274 |        refY="0"
275 |        orient="auto"
276 |        inkscape:stockid="Arrow1Lend">
277 |       <path
278 |          inkscape:connector-curvature="0"
279 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
280 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
281 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
282 |          id="path16566-69-0-0" />
283 |     </marker>
284 |     <marker
285 |        inkscape:isstock="true"
286 |        style="overflow:visible"
287 |        id="marker16492-3-7-0"
288 |        refX="0"
289 |        refY="0"
290 |        orient="auto"
291 |        inkscape:stockid="Arrow1Lstart">
292 |       <path
293 |          inkscape:connector-curvature="0"
294 |          transform="matrix(0.8,0,0,0.8,10,0)"
295 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
296 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
297 |          id="path16490-1-4-2" />
298 |     </marker>
299 |     <marker
300 |        inkscape:isstock="true"
301 |        style="overflow:visible"
302 |        id="marker16492-3-7-0-9"
303 |        refX="0"
304 |        refY="0"
305 |        orient="auto"
306 |        inkscape:stockid="Arrow1Lstart">
307 |       <path
308 |          inkscape:connector-curvature="0"
309 |          transform="matrix(0.8,0,0,0.8,10,0)"
310 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
311 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
312 |          id="path16490-1-4-2-0" />
313 |     </marker>
314 |     <marker
315 |        inkscape:isstock="true"
316 |        style="overflow:visible"
317 |        id="marker16568-52-6-9-3"
318 |        refX="0"
319 |        refY="0"
320 |        orient="auto"
321 |        inkscape:stockid="Arrow1Lend">
322 |       <path
323 |          inkscape:connector-curvature="0"
324 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
325 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
326 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
327 |          id="path16566-69-0-0-5" />
328 |     </marker>
329 |     <marker
330 |        inkscape:isstock="true"
331 |        style="overflow:visible"
332 |        id="marker16568-52-6-9-3-7"
333 |        refX="0"
334 |        refY="0"
335 |        orient="auto"
336 |        inkscape:stockid="Arrow1Lend">
337 |       <path
338 |          inkscape:connector-curvature="0"
339 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
340 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
341 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
342 |          id="path16566-69-0-0-5-1" />
343 |     </marker>
344 |     <marker
345 |        inkscape:isstock="true"
346 |        style="overflow:visible"
347 |        id="marker16492-3-7-0-9-7"
348 |        refX="0"
349 |        refY="0"
350 |        orient="auto"
351 |        inkscape:stockid="Arrow1Lstart">
352 |       <path
353 |          inkscape:connector-curvature="0"
354 |          transform="matrix(0.8,0,0,0.8,10,0)"
355 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
356 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
357 |          id="path16490-1-4-2-0-0" />
358 |     </marker>
359 |     <marker
360 |        inkscape:isstock="true"
361 |        style="overflow:visible"
362 |        id="marker16568-52-6-9-3-4"
363 |        refX="0"
364 |        refY="0"
365 |        orient="auto"
366 |        inkscape:stockid="Arrow1Lend">
367 |       <path
368 |          inkscape:connector-curvature="0"
369 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
370 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
371 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
372 |          id="path16566-69-0-0-5-13" />
373 |     </marker>
374 |     <marker
375 |        inkscape:isstock="true"
376 |        style="overflow:visible"
377 |        id="marker16492-3-7-0-9-0"
378 |        refX="0"
379 |        refY="0"
380 |        orient="auto"
381 |        inkscape:stockid="Arrow1Lstart">
382 |       <path
383 |          inkscape:connector-curvature="0"
384 |          transform="matrix(0.8,0,0,0.8,10,0)"
385 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
386 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
387 |          id="path16490-1-4-2-0-2" />
388 |     </marker>
389 |     <marker
390 |        inkscape:isstock="true"
391 |        style="overflow:visible"
392 |        id="marker16568-52-6-9-3-0"
393 |        refX="0"
394 |        refY="0"
395 |        orient="auto"
396 |        inkscape:stockid="Arrow1Lend">
397 |       <path
398 |          inkscape:connector-curvature="0"
399 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
400 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
401 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
402 |          id="path16566-69-0-0-5-9" />
403 |     </marker>
404 |     <marker
405 |        inkscape:isstock="true"
406 |        style="overflow:visible"
407 |        id="marker16492-3-7-0-9-3"
408 |        refX="0"
409 |        refY="0"
410 |        orient="auto"
411 |        inkscape:stockid="Arrow1Lstart">
412 |       <path
413 |          inkscape:connector-curvature="0"
414 |          transform="matrix(0.8,0,0,0.8,10,0)"
415 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
416 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
417 |          id="path16490-1-4-2-0-5" />
418 |     </marker>
419 |     <marker
420 |        inkscape:isstock="true"
421 |        style="overflow:visible"
422 |        id="marker16568-64"
423 |        refX="0"
424 |        refY="0"
425 |        orient="auto"
426 |        inkscape:stockid="Arrow1Lend">
427 |       <path
428 |          inkscape:connector-curvature="0"
429 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
430 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
431 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
432 |          id="path16566-3" />
433 |     </marker>
434 |     <marker
435 |        inkscape:isstock="true"
436 |        style="overflow:visible"
437 |        id="marker16492-9"
438 |        refX="0"
439 |        refY="0"
440 |        orient="auto"
441 |        inkscape:stockid="Arrow1Lstart">
442 |       <path
443 |          inkscape:connector-curvature="0"
444 |          transform="matrix(0.8,0,0,0.8,10,0)"
445 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
446 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
447 |          id="path16490-10" />
448 |     </marker>
449 |     <marker
450 |        inkscape:isstock="true"
451 |        style="overflow:visible"
452 |        id="marker16568-9"
453 |        refX="0"
454 |        refY="0"
455 |        orient="auto"
456 |        inkscape:stockid="Arrow1Lend">
457 |       <path
458 |          inkscape:connector-curvature="0"
459 |          transform="matrix(-0.8,0,0,-0.8,-10,0)"
460 |          style="fill:#000000;fill-opacity:1;fill-rule:evenodd;stroke:#000000;stroke-width:1.00000003pt;stroke-opacity:1"
461 |          d="M 0,0 5,-5 -12.5,0 5,5 Z"
462 |          id="path16566-4" />
463 |     </marker>
464 |   </defs>
465 |   <sodipodi:namedview
466 |      id="base"
467 |      pagecolor="#ffffff"
468 |      bordercolor="#666666"
469 |      borderopacity="1.0"
470 |      inkscape:pageopacity="0.0"
471 |      inkscape:pageshadow="2"
472 |      inkscape:zoom="1.5219183"
473 |      inkscape:cx="289.66944"
474 |      inkscape:cy="619.3352"
475 |      inkscape:document-units="mm"
476 |      inkscape:current-layer="layer1"
477 |      showgrid="true"
478 |      inkscape:snap-nodes="false"
479 |      inkscape:window-width="1295"
480 |      inkscape:window-height="948"
481 |      inkscape:window-x="92"
482 |      inkscape:window-y="38"
483 |      inkscape:window-maximized="0">
484 |     <inkscape:grid
485 |        type="xygrid"
486 |        id="grid4518" />
487 |   </sodipodi:namedview>
488 |   <metadata
489 |      id="metadata5">
490 |     <rdf:RDF>
491 |       <cc:Work
492 |          rdf:about="">
493 |         <dc:format>image/svg+xml</dc:format>
494 |         <dc:type
495 |            rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
496 |         <dc:title></dc:title>
497 |       </cc:Work>
498 |     </rdf:RDF>
499 |   </metadata>
500 |   <g
501 |      inkscape:label="Layer 1"
502 |      inkscape:groupmode="layer"
503 |      id="layer1">
504 |     <ellipse
505 |        style="color:#000000;clip-rule:nonzero;display:inline;overflow:visible;visibility:visible;opacity:1;isolation:auto;mix-blend-mode:normal;color-interpolation:sRGB;color-interpolation-filters:linearRGB;solid-color:#000000;solid-opacity:1;fill:#f2f2f2;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.21609715;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;color-rendering:auto;image-rendering:auto;shape-rendering:auto;text-rendering:auto;enable-background:accumulate"
506 |        id="path4520"
507 |        cx="61.764481"
508 |        cy="96.028145"
509 |        rx="19.455597"
510 |        ry="10.719258" />
511 |     <text
512 |        xml:space="preserve"
513 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
514 |        x="62.970844"
515 |        y="93.799995"
516 |        id="text4524"><tspan
517 |          sodipodi:role="line"
518 |          id="tspan4522"
519 |          x="62.970844"
520 |          y="93.799995"
521 |          style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;writing-mode:lr-tb;text-anchor:middle;stroke-width:0.26458332px">Physical</tspan><tspan
522 |          sodipodi:role="line"
523 |          x="62.970844"
524 |          y="100.85555"
525 |          style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;writing-mode:lr-tb;text-anchor:middle;stroke-width:0.26458332px"
526 |          id="tspan4526">Situation</tspan></text>
527 |     <ellipse
528 |        style="color:#000000;clip-rule:nonzero;display:inline;overflow:visible;visibility:visible;opacity:1;isolation:auto;mix-blend-mode:normal;color-interpolation:sRGB;color-interpolation-filters:linearRGB;solid-color:#000000;solid-opacity:1;fill:#f2f2f2;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.21609715;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;color-rendering:auto;image-rendering:auto;shape-rendering:auto;text-rendering:auto;enable-background:accumulate"
529 |        id="path4520-3"
530 |        cx="18.058458"
531 |        cy="121.02831"
532 |        rx="19.455597"
533 |        ry="10.719258" />
534 |     <ellipse
535 |        style="color:#000000;clip-rule:nonzero;display:inline;overflow:visible;visibility:visible;opacity:1;isolation:auto;mix-blend-mode:normal;color-interpolation:sRGB;color-interpolation-filters:linearRGB;solid-color:#000000;solid-opacity:1;fill:#f2f2f2;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.18788715;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;color-rendering:auto;image-rendering:auto;shape-rendering:auto;text-rendering:auto;enable-background:accumulate"
536 |        id="path4520-6"
537 |        cx="63.498337"
538 |        cy="142.89879"
539 |        rx="20.575281"
540 |        ry="7.6623073" />
541 |     <ellipse
542 |        style="color:#000000;clip-rule:nonzero;display:inline;overflow:visible;visibility:visible;opacity:1;isolation:auto;mix-blend-mode:normal;color-interpolation:sRGB;color-interpolation-filters:linearRGB;solid-color:#000000;solid-opacity:1;fill:#f2f2f2;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.18452899;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;marker:none;color-rendering:auto;image-rendering:auto;shape-rendering:auto;text-rendering:auto;enable-background:accumulate"
543 |        id="path4520-7"
544 |        cx="103.61116"
545 |        cy="117.10221"
546 |        rx="19.225697"
547 |        ry="7.9096704" />
548 |     <text
549 |        xml:space="preserve"
550 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:middle;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
551 |        x="18.585663"
552 |        y="120.53786"
553 |        id="text4557"><tspan
554 |          sodipodi:role="line"
555 |          id="tspan4555"
556 |          x="18.585663"
557 |          y="120.53786"
558 |          style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;writing-mode:lr-tb;text-anchor:middle;stroke-width:0.26458332px">Mathematical</tspan><tspan
559 |          sodipodi:role="line"
560 |          x="18.585663"
561 |          y="127.59341"
562 |          style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-feature-settings:normal;text-align:center;writing-mode:lr-tb;text-anchor:middle;stroke-width:0.26458332px"
563 |          id="tspan4559">Model</tspan></text>
564 |     <text
565 |        xml:space="preserve"
566 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
567 |        x="45.848404"
568 |        y="143.94269"
569 |        id="text4563"><tspan
570 |          sodipodi:role="line"
571 |          id="tspan4561"
572 |          x="45.848404"
573 |          y="143.94269"
574 |          style="stroke-width:0.26458332px">Approximation</tspan></text>
575 |     <text
576 |        xml:space="preserve"
577 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna';letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
578 |        x="93.133331"
579 |        y="119.2"
580 |        id="text4567"><tspan
581 |          sodipodi:role="line"
582 |          id="tspan4565"
583 |          x="93.133331"
584 |          y="119.2"
585 |          style="stroke-width:0.26458332px">Analysis</tspan></text>
586 |     <path
587 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.17828794px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker16568-52-6-9)"
588 |        d="m 80.017564,100.90044 10.607733,9.633"
589 |        id="path16482-4-3-2"
590 |        inkscape:connector-curvature="0" />
591 |     <path
592 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.19461012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker16492-3-7-0)"
593 |        d="M 81.887018,99.110056 93.281086,109.7955"
594 |        id="path16482-2-4-7-6"
595 |        inkscape:connector-curvature="0" />
596 |     <path
597 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.17828794px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker16568-52-6-9-3)"
598 |        d="m 38.929265,98.153039 -9.633,10.607741"
599 |        id="path16482-4-3-2-5"
600 |        inkscape:connector-curvature="0" />
601 |     <path
602 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.19461012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker16492-3-7-0-9)"
603 |        d="M 41.022713,100.49423 30.33727,111.8883"
604 |        id="path16482-2-4-7-6-8"
605 |        inkscape:connector-curvature="0" />
606 |     <path
607 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.17828794px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker16568-52-6-9-3-7)"
608 |        d="m 42.945521,139.76716 -10.60774,-9.633"
609 |        id="path16482-4-3-2-5-6"
610 |        inkscape:connector-curvature="0" />
611 |     <path
612 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.19461012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker16492-3-7-0-9-7)"
613 |        d="M 38.821212,142.1585 27.427142,131.47305"
614 |        id="path16482-2-4-7-6-8-0"
615 |        inkscape:connector-curvature="0" />
616 |     <path
617 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.17828794px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker16568-52-6-9-3-4)"
618 |        d="M 90.955264,124.48325 81.32226,135.09099"
619 |        id="path16482-4-3-2-5-9"
620 |        inkscape:connector-curvature="0" />
621 |     <path
622 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.19461012px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker16492-3-7-0-9-0)"
623 |        d="M 95.134887,126.30289 84.449448,137.69696"
624 |        id="path16482-2-4-7-6-8-6"
625 |        inkscape:connector-curvature="0" />
626 |     <path
627 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.165;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-end:url(#marker16568);stroke-miterlimit:4;stroke-dasharray:none"
628 |        d="m 59.630061,108.54814 -0.173848,24.51265"
629 |        id="path22947"
630 |        inkscape:connector-curvature="0" />
631 |     <path
632 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.165;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1;marker-start:url(#marker16492);stroke-miterlimit:4;stroke-dasharray:none"
633 |        d="m 64.09821,108.48718 -0.173848,24.51265"
634 |        id="path22947-1"
635 |        inkscape:connector-curvature="0" />
636 |     <path
637 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.20591417;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-end:url(#marker16568-9)"
638 |        d="M 81.536479,120.45375 41.078642,120.2897"
639 |        id="path22947-5"
640 |        inkscape:connector-curvature="0" />
641 |     <path
642 |        style="fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:0.20591417;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1;marker-start:url(#marker16492-9)"
643 |        d="M 81.450795,117.60746 40.992958,117.44341"
644 |        id="path22947-1-6"
645 |        inkscape:connector-curvature="0" />
646 |     <text
647 |        xml:space="preserve"
648 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna';letter-spacing:0px;word-spacing:0px;fill:#ff0000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
649 |        x="9.9093704"
650 |        y="108.54814"
651 |        id="text24515"><tspan
652 |          sodipodi:role="line"
653 |          x="9.9093704"
654 |          y="108.54814"
655 |          style="fill:#ff0000;stroke-width:0.26458332px"
656 |          id="tspan24535">Error</tspan></text>
657 |     <text
658 |        xml:space="preserve"
659 |        style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:5.64444447px;line-height:125%;font-family:'MgOpen Moderna';-inkscape-font-specification:'MgOpen Moderna';letter-spacing:0px;word-spacing:0px;fill:#ff0000;fill-opacity:1;stroke:none;stroke-width:0.26458332px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1"
660 |        x="56.918507"
661 |        y="155.39249"
662 |        id="text24515-2"><tspan
663 |          sodipodi:role="line"
664 |          id="tspan24513-5"
665 |          x="56.918507"
666 |          y="155.39249"
667 |          style="fill:#ff0000;stroke-width:0.26458332px">Error</tspan></text>
668 |   </g>
669 | </svg>
670 | 


--------------------------------------------------------------------------------
/images/siemens-volrender0000_kirby_modeling.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/mandli/intro-numerical-methods/e66f052dd406a1845754dc4be4ee6243406dd6b6/images/siemens-volrender0000_kirby_modeling.png


--------------------------------------------------------------------------------
/peer_review.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Peer Review Guidelines\n",
  8 |     "\n",
  9 |     "When writing a peer review give the code a look over and think about the questions below.  Giving a couple of sentences for each of the questions is sufficient.  Code examples are even better!\n",
 10 |     "\n",
 11 |     "1. Easy to read and understand?\n",
 12 |     "   - Are the `doc-strings` useful?\n",
 13 |     "   - Are variables and functions named descriptively when useful?\n",
 14 |     "   - Are the comments helpful?\n",
 15 |     "1. Does the code follow a consistent style?\n",
 16 |     "   - Is there a consistent style? [PEP8](https://www.python.org/dev/peps/pep-0008/)\n",
 17 |     "1. Is the code written succinctly and efficiently?\n",
 18 |     "   - Was there superfluous code sections?\n",
 19 |     "   - Was the code written clearly?\n",
 20 |     "   - Was the code written elegantly without decreasing readability?\n",
 21 |     "   \n",
 22 |     "Make sure to suggest **constructive** ways to improve all of the above along with:\n",
 23 |     " - Clever ways to use built-in functionality (when appropriate)\n",
 24 |     " - Simpler ways to implement the same functionality\n",
 25 |     " - General improvements to structure, style, and naming\n",
 26 |     " - Be courteous to the person you are reviewing!\n",
 27 |     "\n",
 28 |     "\n",
 29 |     "## Rubric\n",
 30 |     "<table>\n",
 31 |     "  <tr>\n",
 32 |     "    <th>Criteria</th>\n",
 33 |     "    <th>3 points</th>\n",
 34 |     "    <th>2 points</th>\n",
 35 |     "    <th>1 point</th>\n",
 36 |     "  </tr>\n",
 37 |     "  <tr>\n",
 38 |     "    <td>Readability</td>\n",
 39 |     "    <td><ul>\n",
 40 |     "        <li>Doc-strings are clear and accurate</li>\n",
 41 |     "        <li>Variables and functions are named descriptively when useful</li>\n",
 42 |     "        <li>Comments are placed in appropriate places and are clear and accurate</li>\n",
 43 |     "        </ul>\n",
 44 |     "    </td>\n",
 45 |     "    <td><ul>\n",
 46 |     "        <li>Doc-strings are present but not completely descriptive</li>\n",
 47 |     "        <li>Variables and functions are sometimes named helpfully</li>\n",
 48 |     "        <li>Comments are present but not always accurate or in the most helpful of places</li>\n",
 49 |     "        </ul>\n",
 50 |     "    </td>\n",
 51 |     "\n",
 52 |     "    <td><ul>\n",
 53 |     "        <li>No doc-strings</li>\n",
 54 |     "        <li>Variables and functions are named indecipherably</li>\n",
 55 |     "        <li>No or inaccurate comments</li>\n",
 56 |     "        </ul>\n",
 57 |     "    </td>\n",
 58 |     "  </tr>\n",
 59 |     "  <tr>\n",
 60 |     "    <td>Style</td>\n",
 61 |     "    <td><ul>\n",
 62 |     "        <li><a href=\"https://www.python.org/dev/peps/pep-0008/\">PEP 8</a> or other style is consistent</li>\n",
 63 |     "        <li>Indentation is clean and not mixed</li>\n",
 64 |     "       </ul>\n",
 65 |     "    <td><ul>\n",
 66 |     "        <li>Style is mostly consistent with something</li>\n",
 67 |     "       </ul>\n",
 68 |     "    </td>\n",
 69 |     "    <td><ul>\n",
 70 |     "        <li>Style is of a by-gone era but may come back someday (not today)</li>\n",
 71 |     "       </ul>\n",
 72 |     "    </td>\n",
 73 |     "  </tr>\n",
 74 |     "  <tr>\n",
 75 |     "    <td>Code Awesome</td>\n",
 76 |     "    <td><ul>\n",
 77 |     "        <li>Code was succinct and clean</li>\n",
 78 |     "        <li>Upon a glance you completely understood the code</li>\n",
 79 |     "        <li>This code defines elegance</li>\n",
 80 |     "    </td>\n",
 81 |     "    <td><ul>\n",
 82 |     "        <li>Code was overall clean but there were a couple spots</li>\n",
 83 |     "        <li>Code was mostly clear except for a couple of spots</li>\n",
 84 |     "        <li>Code worked but may have been a bit hard to follow</li>\n",
 85 |     "    </td>\n",
 86 |     "    <td><ul>\n",
 87 |     "        <li>Code had large sections that did nothing</li>\n",
 88 |     "        <li>Code was very hard to follow</li>\n",
 89 |     "        <li>Code got the job done but no one would ever understand why</li>\n",
 90 |     "    </ul></td>\n",
 91 |     "  </tr>\n",
 92 |     "</table>\n"
 93 |    ]
 94 |   }
 95 |  ],
 96 |  "metadata": {
 97 |   "kernelspec": {
 98 |    "display_name": "Python 2",
 99 |    "language": "python",
100 |    "name": "python2"
101 |   },
102 |   "language_info": {
103 |    "codemirror_mode": {
104 |     "name": "ipython",
105 |     "version": 3
106 |    },
107 |    "file_extension": ".py",
108 |    "mimetype": "text/x-python",
109 |    "name": "python",
110 |    "nbconvert_exporter": "python",
111 |    "pygments_lexer": "ipython3",
112 |    "version": "3.6.2"
113 |   },
114 |   "latex_envs": {
115 |    "bibliofile": "biblio.bib",
116 |    "cite_by": "apalike",
117 |    "current_citInitial": 1,
118 |    "eqLabelWithNumbers": true,
119 |    "eqNumInitial": 0
120 |   }
121 |  },
122 |  "nbformat": 4,
123 |  "nbformat_minor": 1
124 | }
125 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | six
2 | numpy >= 1.12
3 | matplotlib >= 2.0.0
4 | sympy
5 | scipy
6 | jupyter


--------------------------------------------------------------------------------
/style.css:
--------------------------------------------------------------------------------
  1 | <link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
  2 | <link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
  3 | <link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
  4 | <link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
  5 | 
  6 | <style>
  7 | 
  8 | #notebook_panel { /* main background */
  9 |     background: #888;
 10 |     color: #f6f6f6;
 11 | }
 12 | 
 13 | div.cell { /* set cell width to about 80 chars */
 14 |     width: 800px;
 15 | }
 16 | 
 17 | div #notebook { /* centre the content */
 18 |     background: #fff; /* white background for content */
 19 |     width: 1000px;
 20 |     margin: auto;
 21 |     padding-left: 1em;
 22 | }
 23 | 
 24 | #notebook li { /* More space between bullet points */
 25 | margin-top:0.8em;
 26 | }
 27 | 
 28 | /* draw border around running cells */
 29 | div.cell.border-box-sizing.code_cell.running { 
 30 |     border: 3px solid #111;
 31 | }
 32 | 
 33 | /* Put a solid color box around each cell and its output, visually linking them together */
 34 | div.cell.code_cell {
 35 |     background-color: #f4f3e0; 
 36 |     border-radius: 10px; /* rounded borders */
 37 |     padding: 1em;
 38 |     margin-top: 1em;
 39 | }
 40 | 
 41 | div.text_cell_render{
 42 |     font-family: 'Arvo' sans-serif;
 43 |     line-height: 130%;
 44 |     font-size: 115%;
 45 |     width:700px;
 46 |     margin-left:auto;
 47 |     margin-right:auto;
 48 | }
 49 | 
 50 | 
 51 | /* Formatting for header cells */
 52 | .text_cell_render h1 {
 53 |     font-family: 'Philosopher', sans-serif;
 54 |     font-weight: 400;
 55 |     font-size: 40pt;
 56 |     line-height: 100%;
 57 |     color: rgb(12,85,97);
 58 |     margin-bottom: 0.1em;
 59 |     margin-top: 0.1em;
 60 |     display: block;
 61 | }   
 62 | .text_cell_render h2 {
 63 |     font-family: 'Philosopher', serif;
 64 |     font-weight: 700;
 65 |     font-size: 24pt;
 66 |     line-height: 100%;
 67 |     color: rgb(171,165,131);
 68 |     margin-bottom: 0.1em;
 69 |     margin-top: 0.1em;
 70 |     display: block;
 71 | }   
 72 | 
 73 | .text_cell_render h3 {
 74 |     font-family: 'Philosopher', serif;
 75 |     margin-top:12px;
 76 |     margin-bottom: 3px;
 77 |     font-style: italic;
 78 |     color: rgb(95,92,72);
 79 | }
 80 | 
 81 | .text_cell_render h4 {
 82 |     font-family: 'Philosopher', serif;
 83 | }
 84 | 
 85 | .text_cell_render h5 {
 86 |     font-family: 'Alegreya Sans', sans-serif;
 87 |     font-weight: 300;
 88 |     font-size: 16pt;
 89 |     color: grey;
 90 |     font-style: italic;
 91 |     margin-bottom: .1em;
 92 |     margin-top: 0.1em;
 93 |     display: block;
 94 | }
 95 | 
 96 | .text_cell_render h6 {
 97 |     font-family: 'PT Mono', sans-serif;
 98 |     font-weight: 300;
 99 |     font-size: 10pt;
100 |     color: grey;
101 |     margin-bottom: 1px;
102 |     margin-top: 1px;
103 | }
104 | 
105 | .CodeMirror{
106 |     font-family: "PT Mono";
107 |     font-size: 100%;
108 | }
109 | 
110 | </style>


--------------------------------------------------------------------------------
/test.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env
 2 | """Basic test script that runs all notebooks."""
 3 | 
 4 | import os
 5 | import subprocess
 6 | import tempfile
 7 | import sys
 8 | import nbformat
 9 | import doctest
10 | 
11 | if sys.version_info >= (3,0):
12 |     kernel = 'python3'
13 | else:
14 |     kernel = 'python2'
15 | 
16 | # Note we leave out the python intro as there are purposeful exceptions
17 | notebooks = ["00_intro_numerical_methods.ipynb",
18 |              # "01_intro_to_python.ipynb",
19 |              "02_NumPy.ipynb",
20 |              "03_matplotlib.ipynb",
21 |              "04_error.ipynb",
22 |              "05_root_finding_optimization.ipynb",
23 |              "06_interpolation.ipynb",
24 |              "07_differentiation.ipynb",
25 |              "08_quadrature.ipynb",
26 |              "09_ODE_ivp_part2.ipynb",
27 |              "09_ODE_ivp_part1.ipynb",
28 |              "10_LA_intro.ipynb",
29 |              "11_LA_QR.ipynb",
30 |              "12_LA_conditioning_stability.ipynb",
31 |              "13_LA_eigen.ipynb",
32 |              "14_LA_iterative.ipynb",
33 |              "15_LA_gaussian.ipynb",
34 |              "16_ODE_BVP.ipynb",]
35 | 
36 | def _notebook_run(path):
37 |     """Execute a notebook via nbconvert and collect output.
38 |        :returns (parsed nb object, execution errors)
39 |     """
40 |     with tempfile.NamedTemporaryFile(suffix=".ipynb") as fout:
41 |         args = ["jupyter", "nbconvert", "--to", "notebook", "--execute",
42 |                 "--ExecutePreprocessor.timeout=60",
43 |                 "--ExecutePreprocessor.kernel_name="+kernel,
44 |                 "--output", fout.name, path]
45 |         subprocess.check_call(args)
46 | 
47 |         fout.seek(0)
48 |         nb = nbformat.reads(fout.read().decode('utf-8'), nbformat.current_nbformat)
49 | 
50 |     errors = [output for cell in nb.cells if "outputs" in cell
51 |               for output in cell["outputs"]
52 |               if output.output_type == "error"]
53 | 
54 |     return nb, errors
55 | 
56 | def run_tests():
57 | 
58 |     for filename in notebooks:
59 |         if (filename.split('.')[-1] == 'ipynb'):
60 |             nb, errors = _notebook_run(filename)
61 |             if errors != []:
62 |                 raise(Exception)
63 | 
64 | 
65 | if __name__ == '__main__':
66 |     run_tests()


--------------------------------------------------------------------------------