├── .gitignore ├── LICENSE ├── README.md ├── annual_reports └── 2022 │ ├── Ogranskad sammanstallning - kurt80463_ogranskad.pdf │ └── report.md ├── assets ├── filtering-smoothing.gif └── fpk.gif ├── exercises ├── README.md ├── exercise1.pdf ├── exercise2.pdf ├── exercise3.pdf ├── exercise_bosch_probnum_ode.pdf ├── exercise_emzir_continuous_filtering.md ├── exercise_hosttler_gp_sde.md ├── exercise_yildiz_sde_sgd_mcmc.md ├── exercise_zhao_brownian_motion.md └── exercise_zhao_jax_workshop.ipynb ├── history.md ├── lectures ├── README.md ├── lecture_notes │ ├── bibliography.pdf │ ├── lecture1.pdf │ ├── lecture2.pdf │ ├── lecture3.pdf │ ├── lecture4.pdf │ ├── lecture5.pdf │ ├── lecture8.pdf │ └── lectures_6_and_7.pdf ├── scripts │ ├── lec2_brownian_motion.py │ ├── lec2_ode_noisy.ipynb │ ├── lec3-em-vs-ito15-vs-tme.ipynb │ ├── lec3_em_vs_milstein.py │ ├── lec3_van_der_pol.py │ ├── lec4_kolmogorov_forward_equation_jax.ipynb │ ├── lec4_kolmogorov_forward_equation_np.ipynb │ ├── lec4_moments_estimation_compare.ipynb │ ├── lec4_moments_estimation_compare2.ipynb │ ├── lec4_parameter_estimation_mle_jax.ipynb │ ├── lec4_parameter_estimation_mle_np.ipynb │ ├── lec5_linear_sde_mean_cov.ipynb │ ├── lec5_ornstein_uhlenbeck.ipynb │ ├── lec6_kalman_filter_rts_smoother_np.ipynb │ └── lec7_extended_kf.ipynb └── seminar_lectures │ ├── README.md │ ├── bosch_probnumode.pdf │ ├── bosch_probnumode_demo.jl │ ├── emzir_continuous_filtering.pdf │ ├── hostettler_sysid_gp_sdes.pdf │ ├── yildiz_sde_sgd_mcmc.md │ ├── zhao_brownian_motion.pdf │ └── zhao_jax_workshop │ ├── 1-jax_clinic_intro.ipynb │ ├── 2-jax_clinic_jit.ipynb │ ├── 3-jax_clinic_randomness.ipynb │ ├── 4-jax_clinic_vectorisation.ipynb │ ├── 5-jax_clinic_autodiff.ipynb │ ├── 6-jax_clinic_control_flow.ipynb │ ├── README.md │ └── rise.css └── projects └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | out/ 2 | *.out 3 | *.bbl 4 | *.blg 5 | *.log 6 | *.aux 7 | *.nav 8 | *.snm 9 | *.toc 10 | *.synctex.gz 11 | 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Zheng Zhao and all the lecturers (add their names here) - All rights reserved 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # A computational introduction to stochastic differential equations 2 | 3 | SDE density evolution in time 4 | Stochasti filtering and smoothing 5 | 6 | This course aims to develop a computational view of stochastic differential equations (SDEs) for students that have an applied or engineering background, e.g., machine learning, signal processing, electrical engineering, control, and statistics. 7 | 8 | **The course for the year 2022 is now archived. Please feel free to find the annual report in the folder `./annual_reports/2022`.** 9 | 10 | # Prerequisites 11 | 12 | 1. Linear algebra 13 | 2. Real analysis (not essential) 14 | 3. Probability theory 15 | 4. Ordinary differential equations 16 | 17 | # How to register 18 | 19 | The registration is now closed. If you still would like to enroll the course, please send an email to the course responsible. 20 | 21 | # Lecture notes 22 | 23 | The lectures are given on chalkboard (numerical experiments are given in Jupyter Notebook). Hence, it is recommended that students take notes during the class, although the course responsible will upload the lecture notes (possibly hand-written) to this repository. 24 | 25 | # Essential lectures (6 credits) 26 | 27 | 1. [**Introduction to the course**](./lectures/lecture_notes/lecture1.pdf).
28 | 17 Oct, 2022. Room 4005 Ångström. 13:15 - 15:00. 29 | 30 | 2. [**Stochastic differential/integral equations**](./lectures/lecture_notes/lecture2.pdf).
31 | 21 Oct, 2022. Room 101132 Ångström. 13:15 - 15:00. 32 | 33 | 3. [**Numerical solution to stochastic differential equation**](./lectures/lecture_notes/lecture3.pdf).
34 | 24 Oct, 2022. Room 101127 Ångström. 13:15 - 15:00. 35 | 36 | 4. [**Statistical properties of SDE solutions**](./lectures/lecture_notes/lecture4.pdf).
37 | 28 Oct, 2022. Room 101142 Ångström. 13:15 - 15:00. 38 | 39 | 5. [**Linear SDEs and Gaussian processes**](./lectures/lecture_notes/lecture5.pdf).
40 | 31 Oct, 2022. Room 101146 Ångström. 13:15 - 15:00. 41 | 42 | - [**Exercise 1**](./exercises/exercise1.pdf).
43 | 2 Nov, 2022. Room 101127 Ångström. 13:15 - 15:00.
44 | In this exercise session, the students should deliver answers to the assignments that are associated with Lectures 2 - 3. 45 | 46 | 6. [**Filtering and smoothing problems I (i.e., regression with SDE prior)**](./lectures/lecture_notes/lectures_6_and_7.pdf).
47 | 4 Nov, 2022. Room 101127 Ångström. 13:15 - 15:00. 48 | 49 | 7. [**Filtering and smoothing problems II (i.e., regression with SDE prior)**](./lectures/lecture_notes/lectures_6_and_7.pdf).
50 | 7 Nov, 2022. Room 101150 Ångström. 13:15 - 15:00. 51 | 52 | - [**Exercise 2**](./exercises/exercise2.pdf).
53 | 9 Nov, 2022. Room 101150 Ångström. 13:15 - 15:00.
54 | In this exercise session, the students should deliver answers to the assignments that are associated with Lectures 4 - 5. 55 | 56 | 8. [**Parameter estimation for stochastic continuous-time dynamical models**](./lectures/lecture_notes/lecture8.pdf).
57 | 11 Nov, 2022. Room 101142 Ångström. 13:15 - 15:00.
58 | This lecture presents estimation methods for SDEs. These are typically based on discrete-time measurements, and therefore the transformation of the original model into a discrete-time model is important. The focus is on the linear time-invariant case. We will briefly discuss sampling of continuous-time models, the main challenges of estimating continuous-time models, and describe some of the available basic estimation approaches. The approaches can be divided into direct approaches and indirect approaches. Among the direct approaches is the maximum likelihood method, which is based on exact discretisation of the model and a discrete-time Kalman filter.
59 | Lecturer: [Mohamed Abdalmoaty](https://user.it.uu.se/~mohab408/) 60 | 61 | - [**Exercise 3**](./exercises/exercise3.pdf).
62 | 18 Nov, 2022. Room 101127 Ångström. 13:15 - 15:00.
63 | In this exercise session, the students should deliver answers to the assignments that are associated with Lectures 6 - 8. 64 | 65 | - **Student project work presentation**.
66 | 16 Dec, 2022. Room 101142 Ångström. 67 | 68 | The time is 13:15 - 15:00 for all the lectures (45 mins + 10 mins break + 45 mins). 69 | 70 | # Seminar lectures (9 credits) 71 | 72 | By attending (not necessarily all) the seminar courses and completing their writing assigments/exericses, you get upgrade to 9 credits. 73 | 74 | 1. **Differential geometry for continuous-time stochastic filtering: the projection filter**.
75 | The continuous-time stochastic filtering problem consists in solving the Kushner--Stratonovich equation which is an infinite-dimensional stochastic partial differential equation difficult to solve. In this lecture, we introduce a differential geometry approach for solving the equation: the projection filtering. The projection filters approximate the filtering solution by projecting the solution onto a manifold of parametric familiy, resulting in a finite-dimensional stochastic differential equation that is easy to solve. Then, we discuss the recent advances and challenges of the projection filtering.
76 | Date: 13:15 - 14:45, 17 Nov, 2022. Zoom https://uu-se.zoom.us/j/66304380076
77 | Lecturer: [Muhammad Fuady Emzir](https://scholar.google.com/citations?user=nfBRAHAAAAAJ&hl=en) (KFUPM)
78 | [Lecture note](lectures/seminar_lectures/emzir_continuous_filtering.pdf) 79 | 80 | 2. **SDEs and Markov chain Monte Carlo**.
81 | In this lecture, we present a general recipe for constructing Markov chain Monte Carlo (MCMC) samplers, including stochastic gradient (SG) versions, from stochastic continuous dynamics (SDEs). We also explore the connections between SG-MCMC and stochastic optimization methods via simple annealing techniques. Recommended readings: 1) Bayesian Learning via Stochastic Gradient Langevin Dynamics 2) A Complete Recipe for Stochastic Gradient MCMC.
82 | Date: 09:15 - 11:00, 22 Nov, 2022. Room 4101 Ångström
83 | Lecturer: [Cagatay Yildiz](https://cagatayyildiz.github.io/) (University of Tübingen)
84 | [Lecture note](lectures/seminar_lectures/yildiz_sde_sgd_mcmc.md) 85 | 86 | 3. **Probabilistic numerics for ordinary differential equations**.
87 | Probabilistic numerical methods aim to explicitly represent the numerical uncertainty that results from limited computational resources. In this lecture, we present a class of probabilistic numerical solvers for ODEs which pose the numerical solution of an ODE as a Gauss--Markov regression problem. The resulting methods, called "ODE filters", efficiently compute posterior distributions over ODE solutions with methods from Bayesian filtering and smoothing.
88 | Date: 13:15 - 15:00, 22 Nov, 2022. Room 2001 Ångström
89 | Lecturer: [Nathanael Bosch](https://nathanaelbosch.github.io/) (University of Tübingen)
90 | [Lecture note](lectures/seminar_lectures/bosch_probnumode.pdf) 91 | 92 | 4. **Compiled and differentiable scientific computing with JAX**.
93 | [JAX](https://github.com/google/jax) is a Python library that supports autodifferentiation and just-in-time (JIT) compilation. You can use JAX to differentiate Python and Numpy functions and make them faster with the JIT compilation. This helps you to achieve reliable and high-performance computations in your research. In this lecture, we go walk through the basics of JAX then see how we could apply it to the applications of SDEs.
94 | Date: 13:15 - 15:00, 28 Nov, 2022. Room 101127 Ångström
95 | [Lecture note](lectures/seminar_lectures/zhao_jax_workshop) 96 | 97 | 5. **Constructions of Wiener processes and stochastic integrals**.
98 | This lecture explains the constructions of Brownian motion and Ito integrals.
99 | Date: 13:15 - 15:00, 5 Dec, 2022. Room 101127 Ångström
100 | [Lecture note](lectures/seminar_lectures/zhao_brownian_motion.pdf) 101 | 102 | 6. **Gaussian process SDE models**.
103 | Stochastic differential equations (SDEs) are a versatile tool for modeling uncertain and stochastic systems. Typically, SDEs are derived from first principles. However, this is not always possible and it might be more appropriate to learn the SDE governing a system from data instead. In this talk, we look at data-driven SDE modeling using Gaussian processes. In particular, we model the drift (and possibly diffusion) function(s) using a Gaussian process. We discuss the general framework of this approach, some of the challenges that arise in such models, and look at some practical solutions.
104 | Date: 13:15 - 15:00, 7 Dec, 2022. Room 101127 Ångström.
105 | Lecturer: [Roland Hostettler](http://hostettler.co/) (Uppsala University)
106 | [Lecture note](lectures/seminar_lectures/hostettler_gp_sdes.pdf) 107 | 108 | # Reserved timeslots 109 | 110 | We have also reserved 09.12.2022 and 12.12.2022 at Room 101127 Ångström, 13:15 - 15:00 as backup. 111 | 112 | # Course arrangement 113 | 114 | The course consists of lectures, exercises, and project work. Specifically, in each week, there would be one/two lectures (45 + 45 mins) and an exercise session (60 mins). The students shall present and discuss their exercise solutions during the exercise session. 115 | 116 | Total credit is 6 or 9. 117 | 118 | In order to get 6 credits, you need to 119 | 120 | - actively participate all the essential lectures, 121 | - pass the three exercise assignments, 122 | - present the project work. Depending on the number of students, you may do the project work in group. 123 | 124 | If you would like to get 9 credits, you need to fulfill the requirements for the 6 credits as stated above, and in addition, 125 | 126 | - actively participate all the seminar lectures, 127 | - Select five from all the seminar lectures, then pass the exercises of the selected, or do a writing assignment if the lecture has no exercise. (We will define the writing assignment later). 128 | 129 | The course grade is based on pass/fail. 130 | 131 | # Project work 132 | 133 | See the folder `./projects`. 134 | 135 | # Reading materials 136 | 137 | This course is mainly based on the following textbooks. 138 | 139 | - Hui-Hsiung Kuo. Introduction to stochastic integration. Universitext. Springer, 2006. 140 | - Simo Särkkä and Arno Solin. Applied stochastic differential equations. Cambridge University Press, 2019. 141 | - Ioannis Karatzas and Steve E. Shreve. Brownian motion and stochastic calculus. Springer, 2nd edition, 1991. 142 | 143 | # Course history 144 | 145 | - Oct - Dec, 2022, Uppsala University, FTN0332 TN22H006. 146 | 147 | # Contact 148 | 149 | Zheng Zhao, Uppsala University. 150 | 151 | firstname.lastname@it.uu.se 152 | 153 | https://zz.zabemon.com 154 | -------------------------------------------------------------------------------- /annual_reports/2022/Ogranskad sammanstallning - kurt80463_ogranskad.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/annual_reports/2022/Ogranskad sammanstallning - kurt80463_ogranskad.pdf -------------------------------------------------------------------------------- /annual_reports/2022/report.md: -------------------------------------------------------------------------------- 1 | # Annual report 2022, winter 2 | 3 | In this report, we review the feedbacks from the students to evaluate the performance of the course, and consider how to improve the course thereupon. 4 | 5 | Please find the student feedbacks provided at `./Ogranskad sammanstallning - kurt80463_ogranskad.pdf`. 6 | 7 | We begin with a basic statistics of the course. 8 | 9 | # Basic statistics 10 | 11 | - **Course name**: A computational introduction to stochastic differential equations 12 | - **Date**: 17 October 2022 to 16 December 2022. 13 | - **Level**: Doctoral level (for IT department, Uppsala University). 14 | - **Credits**: 6 credits. Students can upgrade to 9 credits by doing the exercises of the seminar courses. 15 | - **Number of students**: After of the announcement of the course, around 20 students registered the course. However, there were actually only around 10 active students enrolled in the course. The number of attendees of each lecture is given as follows. 16 | 17 | - Lecture 1: Around 8 18 | - Lecture 2: Around 10 19 | - Lecture 3: 12 20 | - Lecture 4: 12 21 | - Lecture 5: 8 22 | - Lecture 6: 7 23 | - Lecture 7: 7 24 | - Lecture 8: 8 25 | - Seminar lectures: forgot to track 26 | 27 | There are **10** students passed the course and got **6** credits. Among these 10 students, there are a plenty of them were interested in getting the 9 credits. However, at the end of the course, only **2** students eventually did the seminar exercises and got 9 credits. This is not surprised due to the following reasons: 1) the topics of the seminar courses are very versatile, hence, it can be hard to find a student who is interested in all of them, not to mention to do all their exercises. 2) the assignments for the seminar lectures were not well defined and prepared. 3) one lecture was too technical. 28 | 29 | # Analysis of the course feedbacks 30 | 31 | The feedbacks are available at `./Ogranskad sammanstallning - kurt80463_ogranskad.pdf`. There are in total 7 (out of 10) students answered the poll. Compared to other courses from the IT department, for example, Advanced Probabilistic Machine Learning which has only 20% students answered, our answer rate is significantly much higher. We had decent students! 32 | 33 | 1. **How would you rate the course’s degree of difficulty?** Most students agreed that the course is not too difficult and not too hard either. 34 | 2. **How did you perceive the course’s workload in relation to its size (number of credits)?** Six students agreed that the amount of credits they got and workload are balanced. One student argues that the workload is slightly demanding. 35 | 3. **I feel that the treatment of students in the course has been good and that no one has been disadvantaged by the organization, content or execution of the teaching.** All completely agree. Of course ;) 36 | 4. **How valuable were the different activities in the course for your learning?** 37 | - **Lectures**. "somewhat valuable" and "very valuable" fifty fifty. 38 | - **Exercises**. Two students voted "invaluable" and other four voted "very valuable". The comments from the students also said that the exercises are super useful. 39 | - **Project works**. Most students are very positive of the project works, while only one found it "somewhat valuable". 40 | - **Seminars**. Most students voted "somewhat useful". This means that we should definitely improve the seminar course next year. 41 | - **Lecture notes**. Two voted "invaluable" while five voted "very valuable". Very useful it is then. 42 | 5. **Did you attend the seminar courses? How useful were they to you?** 43 | - **Differential geometry for continuous-time stochastic filtering: the projection filter, Muhammad Emzir**. A lot of students do not find it useful. 44 | - **SDEs and Markov chain Monte Carlo, Cagatay Yildiz**. Three voted useful. 45 | - **Probabilistic numerics for ordinary differential equations, Nathanael Bosch**. Two voted "valuable" and "somewhat valuable". Most students did not attend. 46 | - **Compiled and differentiable scientific computing with JAX, Zheng Zhao**. Three voted "valuable" and one voted "somewhat valuable". 47 | - **Constructions of Wiener processes, Zheng Zhao**. I originally though this may have the lowest usefulness, while all three voted "valuable". 48 | - **Gaussian process SDE models, Roland Hostettler**. Five "valuable" and one "somewhat valuable". 49 | 6. **Did you experience a mismatch between the prerequisites of this course and what you have learned from previous courses?** Six out of seven found no mismatch, while one found slightly mismatch, meaning that the course's prerequisites are in a good level. 50 | 7. **How well does this course fit in your degree program – did it help you obtain knowledge you expect from your degree program?** No students found it a bad fit. 51 | 8. **What do you think were the best thing(s) about this course? Description: Here, you can highlight efforts, characteristics or parts of the course you thought were good.** No statistics provided. Please see the PDF. 52 | 9. **Please provide constructive suggestions for course development. Description: With your help, the course can be made better, and something that is already good can be made even more prominent/effective.** No statistics provided. Please see the PDF. 53 | 10. **Overall, I am satisfied with this course. Description: Here you are asked how well you think the course worked in relation to everything from teacher, content, forms of instruction, and examination to scheduling.** All students agree to a very high extent. 54 | 55 | Text comments from the students: 56 | 57 | - The good thing was that it was possible to put in more effort to get more out of it. Generally, I wanted to solve (and solved) more of the assignments than was required. A possibility for the next occasion is to keep the ’give it an honest try and get full points’ for each assignment, but to require a few more points. (On the other hand, I personally enjoyed the greater freedom to explore the material where I wanted, so not a trivial solution here) 58 | - But then, as stated above, I put some extra effort into solving (but not necessarily presenting) the assignments, pretty much based on the expected credit workload. 59 | - As is a typical drawback of the lecture format, it could at times be a bit passive (although the python examples in class were very useful). Perhaps a flipped classroom style would do well here: present the lecture material ’off-site’ and work hands-on on the python code examples in class (or something like this) 60 | - Assignments were very nice to work with and to learn from. The freedom of the final project made it possible to try my wings and learn a lot as well. 61 | - It would be nice if the lecture notes were typed in LaTeX. 62 | - Feels like I learnt a lot, and that I now a good foundation to stand on! Well done, I certainly believe this to course to be useful to many students in the future. 63 | 64 | # Conclusions and action points for the next year 65 | 66 | Based on the feedbacks from the students, we conclude that our course organised for 2022 winter in Uppsala University is successful. To improve the course, we summarise the following action points. 67 | 68 | - A lot of students did not actively attend the lectures. I am not sure how to solve this, since PhD students are busy. 69 | - Improve the lecture presentation. We also need to find a better lecture room, so that we can use blackboard and screening at the same time. 70 | - Some students complained about the hand-written lecture notes. We should next try to typeset the lecture note in LaTeX. 71 | - Change the way of getting the credits for the seminar courses. In this year, students can get the credits if they can select 5 out of the 6 lectures and then solve their assignments. However, we found that the topics of these lectures are too broad, while the students are specialised, hence, the students felt reluctant to do the seminar assignments. We can fix this problem by changing the passing rule. For instance, students can get 9 credit, if they can select **one** of seminar lectures, and then do a **project work** of the selected seminar lecture. 72 | - Make the exercises a pinch harder. Basically, the students can already find the answers in the companion codes of the lectures, maybe we should not let this be so obvious. 73 | 74 | Reported by: Zheng Zhao, 25 December 2022. 75 | -------------------------------------------------------------------------------- /assets/filtering-smoothing.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/assets/filtering-smoothing.gif -------------------------------------------------------------------------------- /assets/fpk.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/assets/fpk.gif -------------------------------------------------------------------------------- /exercises/README.md: -------------------------------------------------------------------------------- 1 | # Exercises 2 | 3 | Please find the exercises in this folder. 4 | 5 | # Solutions 6 | 7 | Solutions are shared via a link to students. 8 | 9 | -------------------------------------------------------------------------------- /exercises/exercise1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/exercises/exercise1.pdf -------------------------------------------------------------------------------- /exercises/exercise2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/exercises/exercise2.pdf -------------------------------------------------------------------------------- /exercises/exercise3.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/exercises/exercise3.pdf -------------------------------------------------------------------------------- /exercises/exercise_bosch_probnum_ode.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/exercises/exercise_bosch_probnum_ode.pdf -------------------------------------------------------------------------------- /exercises/exercise_emzir_continuous_filtering.md: -------------------------------------------------------------------------------- 1 | # Exercise on "Differential geometry for continuous-time stochastic filtering: the projection filter" 2 | 3 | Write a one-page essay that reflects what you have learnt from the lecture. You are free to define the "reflection" by yourself. Examples of such are, for instance, 1) a summary of how to apply the differential geometry approach for solving the continuous-time filtering problem. 2) how is the projection filtering compared to the conventional Gaussian approximate filtering (e.g., extenden Kalman filter). 3) a feedback to the lecture. 4 | 5 | **Please submit your answer as an email to muhammad.emzir@kfupm.edu.sa and zheng.zhao@it.uu.se**. 6 | 7 | -------------------------------------------------------------------------------- /exercises/exercise_hosttler_gp_sde.md: -------------------------------------------------------------------------------- 1 | # Exercise on "Gaussian process SDE models" 2 | 3 | Write a reflection/summary of the lecture. 4 | 5 | -------------------------------------------------------------------------------- /exercises/exercise_yildiz_sde_sgd_mcmc.md: -------------------------------------------------------------------------------- 1 | # Exercise on "SDEs and Markov chain Monte Carlo" 2 | 3 | As your course assignment, we ask you to write a one-page “reflections” essay. In a nutshell, you are expected to comment on the presented material. In general, what did you (not) like and why? How do you think optimisation and MCMC are related? How can we bring ideas from one field to the other? In what settings one MCMC method would be more advantageous over the others? What could be the practical benefits of the temperature trick? What metrics other than Fisher information matrix can we consider? Why would one prefer HMC inference over Gaussian processes or variational inference? What potential drawbacks does the presented “complete recipe” have? Feel free to answer some of these or come up with your own questions! 4 | 5 | **Please submit your answer as an email to cagatay.yildiz@uni-tuebingen.de and zheng.zhao@it.uu.se**. 6 | 7 | -------------------------------------------------------------------------------- /exercises/exercise_zhao_brownian_motion.md: -------------------------------------------------------------------------------- 1 | # Exercise on "Constructions of Wiener processes and stochastic integrals" 2 | 3 | Write a reflection/summary of the lecture. 4 | 5 | -------------------------------------------------------------------------------- /exercises/exercise_zhao_jax_workshop.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "7905ec1e", 6 | "metadata": {}, 7 | "source": [ 8 | "# Homework\n", 9 | "\n", 10 | "Please select **one** of the two tasks in what follows. Then write you answer in this notebook.\n", 11 | "\n", 12 | "## Task 1\n", 13 | "\n", 14 | "Recall the infinitesimal generator\n", 15 | "\n", 16 | "$$\n", 17 | "(A \\phi)(x) = \\nabla_x \\phi \\cdot a(x) + \\frac{1}{2} \\mathrm{tr}\\big(\\Gamma(x) \\, \\mathrm{H}_x \\phi\\big), \\quad \\phi\\colon \\mathbb{R}^d\\to\\mathbb{R},\n", 18 | "$$\n", 19 | "\n", 20 | "associated with the SDE\n", 21 | "\n", 22 | "$$\n", 23 | "d X(t) = a(X(t)) dt + b(X(t)) dW(t), \\quad X(0) = X_0,\\\\\n", 24 | "\\Gamma(x) := b(x) \\, b(x)^\\top.\n", 25 | "$$\n", 26 | "\n", 27 | "Now define the SDE coefficients \n", 28 | "\n", 29 | "$$\n", 30 | "a(X(t)) := \n", 31 | "\\begin{bmatrix}\n", 32 | "X_2(t) \\\\\n", 33 | "X_1(t) \\, \\bigl(2 - X_1^2(t)\\bigr) - X_2(t)\n", 34 | "\\end{bmatrix}, \\\\\n", 35 | "b(X(t)) := \n", 36 | "\\begin{bmatrix}\n", 37 | "0 \\\\\n", 38 | "X_1(t).\n", 39 | "\\end{bmatrix}\n", 40 | "$$\n", 41 | "\n", 42 | "Approximate the mean $\\mathbb{E}[X(t) \\mid X(0)]$ and the second moment $\\mathbb{E}[X(t) \\, X(t)^\\top \\mid X(0)]$ by using the generator. That is, using the approximation $\\mathbb{E}[\\phi(X(t)) \\mid X(0)] \\approx X(0) + (A\\phi)(X(0))$. Think what the $\\phi$ are to give the mean and second moments?\n", 43 | "\n", 44 | "Implement the approximation as functions of $t$ and $X(0)$, and vectorise the functions over $t$.\n", 45 | "\n", 46 | "```python\n", 47 | "import jax\n", 48 | "import jax.numpy as jnp\n", 49 | "\n", 50 | "def cond_mean(t, x0):\n", 51 | " # This function takes (float, vector) as input and returns a vector\n", 52 | " return ...\n", 53 | "\n", 54 | "def cond_second_moment(t, x0):\n", 55 | " # This function takes (float, vector) as input and returns a matrix\n", 56 | " return ...\n", 57 | "\n", 58 | "x0 = jnp.array([-3., 0.])\n", 59 | "ts = jnp.linspace(0.01, 0.2, 100)\n", 60 | "\n", 61 | "means = ? # Vectorise the function cond_mean so that we can evalute cond_mean on ts. Expect means is a (100, 2)-shaped array\n", 62 | "second_moments = ? # Expect second_moments is a (100, 2, 2)-shaped array\n", 63 | "```\n", 64 | "\n", 65 | "## Task 2\n", 66 | "\n", 67 | "Implement the Kalman filtering and RTS smoothering in Assignment 1, Exercise 3, in JAX." 68 | ] 69 | } 70 | ], 71 | "metadata": { 72 | "kernelspec": { 73 | "display_name": "Python 3 (ipykernel)", 74 | "language": "python", 75 | "name": "python3" 76 | }, 77 | "language_info": { 78 | "codemirror_mode": { 79 | "name": "ipython", 80 | "version": 3 81 | }, 82 | "file_extension": ".py", 83 | "mimetype": "text/x-python", 84 | "name": "python", 85 | "nbconvert_exporter": "python", 86 | "pygments_lexer": "ipython3", 87 | "version": "3.10.4" 88 | } 89 | }, 90 | "nbformat": 4, 91 | "nbformat_minor": 5 92 | } 93 | -------------------------------------------------------------------------------- /history.md: -------------------------------------------------------------------------------- 1 | # Course history 2 | 3 | - Oct - Dec, 2022, Uppsala University, FTN0332 TN22H006. Lecturers are, Zheng Zhao (Uppsala University), Mohamed Abdalmoaty (Uppsala University), Muhammad Emzir (KFUPM), Cagatay Yildiz (University of Tübingen), and Nathanael Bosch (University of Tübingen). 4 | -------------------------------------------------------------------------------- /lectures/README.md: -------------------------------------------------------------------------------- 1 | # Lectures 2 | 3 | Please find the lecture notes in this folder. 4 | 5 | # Author(s) 6 | 7 | Zheng Zhao. 8 | 9 | # Note 10 | 11 | The lecture note was sketched very quickly, hence, it very possibly have errors. Please let me know if you find any. 12 | 13 | -------------------------------------------------------------------------------- /lectures/lecture_notes/bibliography.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/bibliography.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture1.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture2.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture3.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture3.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture4.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture5.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture5.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lecture8.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lecture8.pdf -------------------------------------------------------------------------------- /lectures/lecture_notes/lectures_6_and_7.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/lecture_notes/lectures_6_and_7.pdf -------------------------------------------------------------------------------- /lectures/scripts/lec2_brownian_motion.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generate realisations of Brownian motion. 3 | """ 4 | import numpy as np 5 | import matplotlib.pyplot as plt 6 | 7 | dt = 0.001 8 | T = 10000 9 | ts = np.linspace(dt, dt * T, T) 10 | 11 | np.random.seed(666) 12 | 13 | num_samples = 10 14 | 15 | # Set plotting parameters 16 | plt.rcParams.update({ 17 | 'text.usetex': True, 18 | 'font.family': "serif", 19 | 'text.latex.preamble': r'\usepackage{amsmath,amsfonts}', 20 | 'font.serif': ["Computer Modern Roman"], 21 | 'font.size': 20}) 22 | 23 | for i in range(num_samples): 24 | dws = np.sqrt(dt) * np.random.randn(T, ) 25 | ws = np.cumsum(dws) 26 | 27 | plt.plot(ts, ws, linewidth=0.5) 28 | 29 | # Plot 0.95 interval 30 | plt.fill_between(ts, 31 | -1.96 * np.sqrt(ts), 32 | 1.96 * np.sqrt(ts), 33 | color='black', 34 | edgecolor='none', 35 | alpha=0.15, 36 | label='0.95 quantile') 37 | 38 | plt.xlabel('$t$') 39 | plt.ylabel('$W(t)$') 40 | plt.grid(linestyle='--', alpha=0.3, which='both') 41 | 42 | plt.legend() 43 | 44 | plt.tight_layout(pad=0.1) 45 | plt.show() 46 | -------------------------------------------------------------------------------- /lectures/scripts/lec3_em_vs_milstein.py: -------------------------------------------------------------------------------- 1 | """ 2 | A computational introduction to stochastic differential equations. 3 | 4 | Lecture 3. 5 | 6 | https://github.com/spdes/computational-sde-intro-lecture. 7 | 8 | Euler--Maruyama vs Milstein's method. 9 | """ 10 | import math 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | 14 | np.random.seed(666) 15 | 16 | alp = 1. 17 | 18 | 19 | def drift(x): 20 | return - alp ** 2 / 2 * x 21 | 22 | 23 | def dispersion(x): 24 | return alp * np.sqrt(1 - x ** 2) 25 | 26 | 27 | def derivative_of_dispersion(x): 28 | return -alp * x / np.sqrt(1 - x ** 2) 29 | 30 | 31 | # Times 32 | T = 500 33 | dt = 0.01 34 | ts = np.linspace(dt, dt * T, T) 35 | 36 | # Brownian motion increments and the Brownian motion 37 | dws = math.sqrt(dt) * np.random.randn(T) 38 | ws = np.cumsum(dws) 39 | 40 | # Initial value 41 | x0 = 0. 42 | 43 | # True solution. See Exercise 1, Assignment 2. 44 | xs_true = np.sin(alp * ws) 45 | 46 | # Simulate using Euler--Maruyama 47 | xs_euler_maruyama = np.zeros((T,)) 48 | x = x0 49 | for i in range(T): 50 | x += drift(x) * dt + dispersion(x) * dws[i] 51 | xs_euler_maruyama[i] = x 52 | 53 | # Simulate using Milstein's method 54 | xs_milstein = np.zeros((T,)) 55 | x = x0 56 | for i in range(T): 57 | x += drift(x) * dt + dispersion(x) * dws[i] + 0.5 * derivative_of_dispersion(x) * dispersion(x) * (dws[i] ** 2 - dt) 58 | xs_milstein[i] = x 59 | 60 | # Plot 61 | plt.rcParams.update({ 62 | 'text.usetex': True, 63 | 'font.family': "serif", 64 | 'text.latex.preamble': r'\usepackage{amsmath,amsfonts}', 65 | 'font.serif': ["Computer Modern Roman"], 66 | 'font.size': 16}) 67 | 68 | # Plot trajectories 69 | plt.figure() 70 | plt.plot(ts, xs_true, label='Truth') 71 | plt.plot(ts, xs_euler_maruyama, label='Euler--Maruyama') 72 | plt.plot(ts, xs_milstein, label='Milstein') 73 | plt.grid(linestyle='--', alpha=0.3, which='both') 74 | plt.xlabel('$t$') 75 | plt.ylabel('$X(t)$') 76 | plt.legend() 77 | 78 | plt.tight_layout(pad=0.1) 79 | plt.show() 80 | 81 | # Plot error 82 | plt.figure() 83 | plt.plot(ts, np.abs(xs_euler_maruyama - xs_true), label='Euler--Maruyama error') 84 | plt.plot(ts, np.abs(xs_milstein - xs_true), label='Milstein error') 85 | plt.grid(linestyle='--', alpha=0.3, which='both') 86 | plt.xlabel('$t$') 87 | plt.ylabel('Absolute error') 88 | plt.legend() 89 | plt.yscale('log') 90 | 91 | plt.tight_layout(pad=0.1) 92 | plt.show() 93 | -------------------------------------------------------------------------------- /lectures/scripts/lec3_van_der_pol.py: -------------------------------------------------------------------------------- 1 | """ 2 | A computational introduction to stochastic differential equations. 3 | 4 | Lecture 3. 5 | 6 | https://github.com/spdes/computational-sde-intro-lecture. 7 | 8 | Simulating a modified Duffing--van der Pol SDE using Euler--Maruyama. 9 | """ 10 | import math 11 | import numpy as np 12 | import matplotlib.pyplot as plt 13 | 14 | np.random.seed(666) 15 | 16 | alpha = 2 17 | 18 | 19 | def drift(x): 20 | return np.array([x[1], 21 | x[0] * (alpha - x[0] ** 2) - x[1]]) 22 | 23 | 24 | def dispersion(x): 25 | return np.array([0., 26 | x[0]]) 27 | 28 | 29 | # Times 30 | T = 10000 31 | dt = 0.001 32 | ts = np.linspace(dt, dt * T, T) 33 | 34 | # Brownian motion increments 35 | dws = math.sqrt(dt) * np.random.randn(T) 36 | 37 | # Initial value 38 | x0 = np.array([-3, 0.]) 39 | 40 | # Simulate using Euler--Maruyama 41 | xs_euler_maruyama = np.zeros((T, 2)) 42 | x = x0.copy() 43 | for i in range(T): 44 | x += drift(x) * dt + dispersion(x) * dws[i] 45 | xs_euler_maruyama[i] = x 46 | 47 | # Plot 48 | plt.rcParams.update({ 49 | 'text.usetex': True, 50 | 'font.family': "serif", 51 | 'text.latex.preamble': r'\usepackage{amsmath,amsfonts}', 52 | 'font.serif': ["Computer Modern Roman"], 53 | 'font.size': 16}) 54 | 55 | plt.plot(xs_euler_maruyama[:, 0], xs_euler_maruyama[:, 1], c='black') 56 | plt.scatter(*x0, s=20, c='red', edgecolors=None) 57 | plt.grid(linestyle='--', alpha=0.3, which='both') 58 | plt.xlabel('$X_1(t)$') 59 | plt.ylabel('$X_2(t)$') 60 | 61 | plt.tight_layout(pad=0.1) 62 | plt.show() 63 | -------------------------------------------------------------------------------- /lectures/scripts/lec4_kolmogorov_forward_equation_jax.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "# A computational introduction to stochastic differential equations.\n", 7 | "\n", 8 | "Lecture 4.\n", 9 | "\n", 10 | "https://github.com/spdes/computational-sde-intro-lecture.\n", 11 | "\n", 12 | "Solve a Kolmogorov forward equation (a.k.a. Fokker--Planck) using finite difference." 13 | ], 14 | "metadata": { 15 | "collapsed": false 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "outputs": [], 22 | "source": [ 23 | "import math\n", 24 | "import jax\n", 25 | "import jax.numpy as jnp\n", 26 | "import jax.scipy.stats\n", 27 | "import matplotlib.pyplot as plt\n", 28 | "from jax.config import config\n", 29 | "\n", 30 | "config.update(\"jax_enable_x64\", True)" 31 | ], 32 | "metadata": { 33 | "collapsed": false 34 | } 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 2, 39 | "id": "d31926f9", 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "a, b = -2., 1.\n", 44 | "\n", 45 | "def drift(x):\n", 46 | " return a * x\n", 47 | "\n", 48 | "\n", 49 | "def dispersion(x):\n", 50 | " return b * x" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 3, 56 | "id": "ba3bf30c", 57 | "metadata": {}, 58 | "outputs": [ 59 | { 60 | "name": "stderr", 61 | "output_type": "stream", 62 | "text": [ 63 | "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n" 64 | ] 65 | } 66 | ], 67 | "source": [ 68 | "dt = 2e-5\n", 69 | "T = 10000\n", 70 | "ts = jnp.linspace(dt, dt * T, T)\n", 71 | "\n", 72 | "m0 = 2.\n", 73 | "var0 = 1." 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 4, 79 | "outputs": [], 80 | "source": [ 81 | "def simulate_true_trajectory(key):\n", 82 | " x0 = m0 + math.sqrt(var0) * jax.random.normal(key)\n", 83 | "\n", 84 | " key, _ = jax.random.split(key)\n", 85 | " ws = jnp.cumsum(math.sqrt(dt) * jax.random.normal(key, (T, )))\n", 86 | "\n", 87 | " return x0 * jnp.exp((a - b ** 2 / 2) * ts + b * ws)" 88 | ], 89 | "metadata": { 90 | "collapsed": false 91 | } 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": 5, 96 | "outputs": [], 97 | "source": [ 98 | "key = jax.random.PRNGKey(999)\n", 99 | "\n", 100 | "num_trajs = 2000\n", 101 | "keys = jax.random.split(key, num=num_trajs)\n", 102 | "true_trajectories = jax.vmap(simulate_true_trajectory, in_axes=[0])(keys)" 103 | ], 104 | "metadata": { 105 | "collapsed": false 106 | } 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 7, 111 | "outputs": [ 112 | { 113 | "data": { 114 | "text/plain": "
", 115 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAa2ElEQVR4nO3de2yV9f3A8U9bbCsCFW1oFRvrLUOiUqDQFeNl+XWyhF1MdmHGja7Z+GOC053MDLaFbrJZNpV1DibKxrboCGTJ1MU5nOnEzNmlCiHzsrFbEMS1QOZarUkxbX9/LFaLLfSUy7eX1yt5EvvwPOd8T4qHd77neb4np7e3tzcAABLJTT0AAGB8EyMAQFJiBABISowAAEmJEQAgKTECACQlRgCApMQIAJDUhNQDGIqenp549dVXY/LkyZGTk5N6OADAEPT29sbrr78e5557buTmDj7/MSpi5NVXX42ysrLUwwAAhmHfvn1x3nnnDfrnoyJGJk+eHBH/ezFTpkxJPBoAYCg6OjqirKys79/xwYyKGHn7o5kpU6aIEQAYZY51iYULWAGApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASU1IPQB4W/mK37xn3541ixKMBIBTycwIAJCUGAEAkhIjAEBSrhkhmYGuEQFg/DEzAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABIakLqAcDRlK/4Tb+f96xZlGgkAJwsZkYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABIyrf2Mib5tl+A0cPMCACQlBgBAJISIwBAUmIEAEhKjAAASQ0rRtavXx/l5eVRWFgYVVVV0dLSMuixP/vZzyInJ6ffVlhYOOwBAwBjS9YxsnXr1shkMlFfXx87d+6MWbNmxcKFC+PAgQODnjNlypT497//3be9/PLLxzVoAGDsyDpG1q5dG0uXLo26urqYOXNmbNiwISZOnBibNm0a9JycnJwoLS3t20pKSo5r0ADA2JFVjBw+fDh27NgRNTU17zxAbm7U1NREc3PzoOe98cYbcf7550dZWVl87GMfixdffPGoz9PV1RUdHR39NgBgbMoqRg4dOhTd3d3vmdkoKSmJ1tbWAc953/veF5s2bYpHHnkkHnzwwejp6YkFCxbEK6+8MujzNDQ0RFFRUd9WVlaWzTABgFHkpC8HX11dHdXV1X0/L1iwIC699NK47777YvXq1QOes3LlyshkMn0/d3R0CJIR7Mil1yMsvw7A0GUVI8XFxZGXlxdtbW399re1tUVpaemQHuO0006L2bNnxz/+8Y9BjykoKIiCgoJshgYAjFJZfUyTn58fc+fOjaampr59PT090dTU1G/242i6u7vj+eefj3POOSe7kQIAY1LWH9NkMpmora2NysrKmD9/fjQ2NkZnZ2fU1dVFRMSSJUti+vTp0dDQEBERt99+e7z//e+Piy++OP773//GnXfeGS+//HJ84QtfOLGvBAAYlbKOkcWLF8fBgwdj1apV0draGhUVFbFt27a+i1r37t0bubnvTLi89tprsXTp0mhtbY2pU6fG3Llz45lnnomZM2eeuFcBAIxaw7qAdfny5bF8+fIB/2z79u39fv7+978f3//+94fzNADAOOC7aQCApMQIAJDUSV9nBCIGXosEACLMjAAAiYkRACApMQIAJCVGAICkxAgAkJQYAQCScmsvo57bhgFGNzMjAEBSYgQASEqMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEjKcvCMCwMtGb9nzaIEIwHgSGZGAICkxAgAkJQYAQCSEiMAQFJiBABISowAAEm5tZdRZaBbdAEY3cyMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApy8FzUozGZdsHGvOeNYsSjARgfDEzAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASVlnhKyNxjVEABi5zIwAAEmJEQAgKR/TcFQ+kgHgZDMzAgAkNawYWb9+fZSXl0dhYWFUVVVFS0vLkM7bsmVL5OTkxPXXXz+cpwUAxqCsY2Tr1q2RyWSivr4+du7cGbNmzYqFCxfGgQMHjnrenj174itf+UpcddVVwx4sADD2ZB0ja9eujaVLl0ZdXV3MnDkzNmzYEBMnToxNmzYNek53d3fceOON8a1vfSsuvPDC4xowADC2ZBUjhw8fjh07dkRNTc07D5CbGzU1NdHc3DzoebfffntMmzYtPv/5zw/pebq6uqKjo6PfBgCMTVnFyKFDh6K7uztKSkr67S8pKYnW1tYBz3n66afjJz/5SWzcuHHIz9PQ0BBFRUV9W1lZWTbDBABGkZN6N83rr78en/3sZ2Pjxo1RXFw85PNWrlwZ7e3tfdu+fftO4igBgJSyWmekuLg48vLyoq2trd/+tra2KC0tfc/x//znP2PPnj3xkY98pG9fT0/P/554woTYvXt3XHTRRe85r6CgIAoKCrIZGmRtKGuoHHnMnjWLTtZwAMatrGZG8vPzY+7cudHU1NS3r6enJ5qamqK6uvo9x8+YMSOef/752LVrV9/20Y9+ND7wgQ/Erl27fPwCAGS/Amsmk4na2tqorKyM+fPnR2NjY3R2dkZdXV1ERCxZsiSmT58eDQ0NUVhYGJdddlm/888888yIiPfsBwDGp6xjZPHixXHw4MFYtWpVtLa2RkVFRWzbtq3vota9e/dGbq6FXQGAocnp7e3tTT2IY+no6IiioqJob2+PKVOmpB7OuOK7afpzzQjA0A31329TGABAUmIEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABISowAAEmJEQAgKTECACQlRgCApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABISowAAEmJEQAgKTECACQlRgCApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICkxAgAkJUYAgKQmpB4AI0v5it+kHgIA44yZEQAgKTECACQlRgCApFwzAlkY6JqaPWsWJRgJwNgxrJmR9evXR3l5eRQWFkZVVVW0tLQMeuyvfvWrqKysjDPPPDPOOOOMqKioiAceeGDYAwYAxpasY2Tr1q2RyWSivr4+du7cGbNmzYqFCxfGgQMHBjz+rLPOiq9//evR3Nwcf/7zn6Ouri7q6uri8ccfP+7BAwCjX05vb29vNidUVVXFvHnzYt26dRER0dPTE2VlZXHzzTfHihUrhvQYc+bMiUWLFsXq1auHdHxHR0cUFRVFe3t7TJkyJZvhkiW39mbPxzQAAxvqv99ZzYwcPnw4duzYETU1Ne88QG5u1NTURHNz8zHP7+3tjaampti9e3dcffXVgx7X1dUVHR0d/TYAYGzKKkYOHToU3d3dUVJS0m9/SUlJtLa2Dnpee3t7TJo0KfLz82PRokXxwx/+MD74wQ8OenxDQ0MUFRX1bWVlZdkMEwAYRU7Jrb2TJ0+OXbt2xbPPPhvf+c53IpPJxPbt2wc9fuXKldHe3t637du371QMEwBIIKtbe4uLiyMvLy/a2tr67W9ra4vS0tJBz8vNzY2LL744IiIqKiriL3/5SzQ0NMS111474PEFBQVRUFCQzdAAgFEqq5mR/Pz8mDt3bjQ1NfXt6+npiaampqiurh7y4/T09ERXV1c2Tw0AjFFZL3qWyWSitrY2KisrY/78+dHY2BidnZ1RV1cXERFLliyJ6dOnR0NDQ0T87/qPysrKuOiii6Krqysee+yxeOCBB+Lee+89sa8EABiVso6RxYsXx8GDB2PVqlXR2toaFRUVsW3btr6LWvfu3Ru5ue9MuHR2dsZNN90Ur7zySpx++ukxY8aMePDBB2Px4sUn7lUAAKNW1uuMpGCdkVPHOiPZs84IwMBOyjojAAAnmhgBAJLyrb1wgg3lm319+y/AO8yMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUtYZgeNkCX2A42NmBABISowAAEmJEQAgKTECACQlRgCApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICkxAgAkJUYAgKR8ay+cAr7ZF2BwZkYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJTl4GEUOXJZ+T1rFiUaCcCJY2YEAEhKjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABISowAAElZDn4cO3JpcdKy1DswXpkZAQCSEiMAQFJiBABIalgxsn79+igvL4/CwsKoqqqKlpaWQY/duHFjXHXVVTF16tSYOnVq1NTUHPV4AGB8yTpGtm7dGplMJurr62Pnzp0xa9asWLhwYRw4cGDA47dv3x433HBDPPnkk9Hc3BxlZWVx3XXXxf79+4978ADA6Jd1jKxduzaWLl0adXV1MXPmzNiwYUNMnDgxNm3aNODxv/jFL+Kmm26KioqKmDFjRvz4xz+Onp6eaGpqOu7BAwCjX1Yxcvjw4dixY0fU1NS88wC5uVFTUxPNzc1Deow333wz3nrrrTjrrLOyGykAMCZltc7IoUOHoru7O0pKSvrtLykpib/+9a9DeoyvfvWrce655/YLmiN1dXVFV1dX388dHR3ZDBMAGEVO6d00a9asiS1btsRDDz0UhYWFgx7X0NAQRUVFfVtZWdkpHCUAcCplFSPFxcWRl5cXbW1t/fa3tbVFaWnpUc+96667Ys2aNfG73/0urrjiiqMeu3Llymhvb+/b9u3bl80wAYBRJKsYyc/Pj7lz5/a7+PTti1Grq6sHPe973/terF69OrZt2xaVlZXHfJ6CgoKYMmVKvw0AGJuy/m6aTCYTtbW1UVlZGfPnz4/Gxsbo7OyMurq6iIhYsmRJTJ8+PRoaGiIi4rvf/W6sWrUqNm/eHOXl5dHa2hoREZMmTYpJkyadwJcCAIxGWcfI4sWL4+DBg7Fq1apobW2NioqK2LZtW99FrXv37o3c3HcmXO699944fPhwfOITn+j3OPX19fHNb37z+EYPAIx6w/rW3uXLl8fy5csH/LPt27f3+3nPnj3DeQoAYJwYVowAJ1/5it+kHgLAKeGL8gCApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJubV3HHGrKAAjkZkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABIyjojMIoNtHbMnjWLEowEYPjMjAAASYkRACApMQIAJCVGAICkxAgAkJQYAQCSEiMAQFJiBABISowAAEmJEQAgKTECACQlRgCApMQIAJCUb+2FMebIb/I9md/ieyqfCxi7zIwAAEmJEQAgKTECACQlRgCApMQIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICnLwY8RluVmMEf+3Yjw9wMYWcyMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUtYZgXHIujTASGJmBABISowAAEkNK0bWr18f5eXlUVhYGFVVVdHS0jLosS+++GJ8/OMfj/Ly8sjJyYnGxsbhjhU4hcpX/KbfBnCyZB0jW7dujUwmE/X19bFz586YNWtWLFy4MA4cODDg8W+++WZceOGFsWbNmigtLT3uAQMAY0vWMbJ27dpYunRp1NXVxcyZM2PDhg0xceLE2LRp04DHz5s3L+6888749Kc/HQUFBcc9YABgbMkqRg4fPhw7duyImpqadx4gNzdqamqiubn5hA2qq6srOjo6+m0AwNiUVYwcOnQouru7o6SkpN/+kpKSaG1tPWGDamhoiKKior6trKzshD02ADCyjMi7aVauXBnt7e192759+1IPCQA4SbJa9Ky4uDjy8vKira2t3/62trYTenFqQUGB60sAYJzIamYkPz8/5s6dG01NTX37enp6oqmpKaqrq0/44ACAsS/r5eAzmUzU1tZGZWVlzJ8/PxobG6OzszPq6uoiImLJkiUxffr0aGhoiIj/XfT60ksv9f33/v37Y9euXTFp0qS4+OKLT+BL4d2sC0EKA/29s9Q8cCxZx8jixYvj4MGDsWrVqmhtbY2KiorYtm1b30Wte/fujdzcdyZcXn311Zg9e3bfz3fddVfcddddcc0118T27duP/xUAAKPasL4ob/ny5bF8+fIB/+zIwCgvL4/e3t7hPA0AMA741l7Ax3pAUiPy1l4AYPwQIwBAUmIEAEhKjAAASYkRACApMQIAJCVGAICkrDMCDIm1SICTxcwIAJCUGAEAkhIjAEBSYgQASEqMAABJiREAICm39gKn1EC3CO9ZsyjBSICRwswIAJCUGAEAkhIjAEBSrhkBkjvyOhLXkMD4YmYEAEhKjAAASfmYZhTwbamMZv7+AsdiZgQASEqMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJKyzggwZpyoZeUtTw+nlpkRACApMQIAJOVjGmDEGWgJ+ZQflYy08cBYY2YEAEhKjAAASYkRACAp14wAY9ZA13oAI4+ZEQAgKTECACQlRgCApFwzMsL4jBsGNl7+37CmCeORmREAICkxAgAk5WOaxMbL1DOMNb7ZF04cMyMAQFJiBABISowAAEm5ZuQEcTsejG9Duf5roPeEE3Xd2HCef7jvW66X4UQzMwIAJCVGAICkhhUj69evj/Ly8igsLIyqqqpoaWk56vG//OUvY8aMGVFYWBiXX355PPbYY8MaLAAw9mR9zcjWrVsjk8nEhg0boqqqKhobG2PhwoWxe/fumDZt2nuOf+aZZ+KGG26IhoaG+PCHPxybN2+O66+/Pnbu3BmXXXbZCXkRx+NEfWY63GOA8WO47wkpr9E4le9jJ/P9eLiPM1avhxlp1/1kPTOydu3aWLp0adTV1cXMmTNjw4YNMXHixNi0adOAx//gBz+ID33oQ3HbbbfFpZdeGqtXr445c+bEunXrjnvwAMDol9XMyOHDh2PHjh2xcuXKvn25ublRU1MTzc3NA57T3NwcmUym376FCxfGww8/POjzdHV1RVdXV9/P7e3tERHR0dGRzXCHpKfrzffsG8rzDHQewKkw0HvUUN6TjjxvKO9/w3nc4TqZ78fDfZyT8e/OSHDkaz1Zr/Ptx+3t7T36gb1Z2L9/f29E9D7zzDP99t9222298+fPH/Cc0047rXfz5s399q1fv7532rRpgz5PfX19b0TYbDabzWYbA9u+ffuO2hcjcp2RlStX9ptN6enpif/85z9x9tlnR05OTsKRjQ4dHR1RVlYW+/btiylTpqQeDkfhdzV6+F2NHn5XI0dvb2+8/vrrce655x71uKxipLi4OPLy8qKtra3f/ra2tigtLR3wnNLS0qyOj4goKCiIgoKCfvvOPPPMbIZKREyZMsX/iKOE39Xo4Xc1evhdjQxFRUXHPCarC1jz8/Nj7ty50dTU1Levp6cnmpqaorq6esBzqqur+x0fEfHEE08MejwAML5k/TFNJpOJ2traqKysjPnz50djY2N0dnZGXV1dREQsWbIkpk+fHg0NDRERccstt8Q111wTd999dyxatCi2bNkSzz33XNx///0n9pUAAKNS1jGyePHiOHjwYKxatSpaW1ujoqIitm3bFiUlJRERsXfv3sjNfWfCZcGCBbF58+b4xje+EV/72tfikksuiYcffnhErDEyVhUUFER9ff17Pupi5PG7Gj38rkYPv6vRJ6e391j32wAAnDy+mwYASEqMAABJiREAICkxAgAkJUbGoPXr10d5eXkUFhZGVVVVtLS0pB4SR2hoaIh58+bF5MmTY9q0aXH99dfH7t27Uw+LY1izZk3k5OTErbfemnooDGL//v3xmc98Js4+++w4/fTT4/LLL4/nnnsu9bA4BjEyxmzdujUymUzU19fHzp07Y9asWbFw4cI4cOBA6qHxLk899VQsW7Ys/vSnP8UTTzwRb731Vlx33XXR2dmZemgM4tlnn4377rsvrrjiitRDYRCvvfZaXHnllXHaaafFb3/723jppZfi7rvvjqlTp6YeGsfg1t4xpqqqKubNmxfr1q2LiP+tkFtWVhY333xzrFixIvHoGMzBgwdj2rRp8dRTT8XVV1+dejgc4Y033og5c+bEj370o/j2t78dFRUV0djYmHpYHGHFihXxxz/+Mf7whz+kHgpZMjMyhhw+fDh27NgRNTU1fftyc3OjpqYmmpubE46MY2lvb4+IiLPOOivxSBjIsmXLYtGiRf3+32Lk+fWvfx2VlZXxyU9+MqZNmxazZ8+OjRs3ph4WQyBGxpBDhw5Fd3d332q4byspKYnW1tZEo+JYenp64tZbb40rr7zSysQj0JYtW2Lnzp19X3HByPWvf/0r7r333rjkkkvi8ccfjy9+8YvxpS99KX7+85+nHhrHkPVy8MCJtWzZsnjhhRfi6aefTj0UjrBv37645ZZb4oknnojCwsLUw+EYenp6orKyMu64446IiJg9e3a88MILsWHDhqitrU08Oo7GzMgYUlxcHHl5edHW1tZvf1tbW5SWliYaFUezfPnyePTRR+PJJ5+M8847L/VwOMKOHTviwIEDMWfOnJgwYUJMmDAhnnrqqbjnnntiwoQJ0d3dnXqIvMs555wTM2fO7Lfv0ksvjb179yYaEUMlRsaQ/Pz8mDt3bjQ1NfXt6+npiaampqiurk44Mo7U29sby5cvj4ceeih+//vfxwUXXJB6SAzg//7v/+L555+PXbt29W2VlZVx4403xq5duyIvLy/1EHmXK6+88j23yP/tb3+L888/P9GIGCof04wxmUwmamtro7KyMubPnx+NjY3R2dkZdXV1qYfGuyxbtiw2b94cjzzySEyePLnvmp6ioqI4/fTTE4+Ot02ePPk91/GcccYZcfbZZ7u+ZwT68pe/HAsWLIg77rgjPvWpT0VLS0vcf//9cf/996ceGsfg1t4xaN26dXHnnXdGa2trVFRUxD333BNVVVWph8W75OTkDLj/pz/9aXzuc587tYMhK9dee61be0ewRx99NFauXBl///vf44ILLohMJhNLly5NPSyOQYwAAEm5ZgQASEqMAABJiREAICkxAgAkJUYAgKTECACQlBgBAJISIwBAUmIEAEhKjAAASYkRACApMQIAJPX/x4k7HdpnTm8AAAAASUVORK5CYII=\n" 116 | }, 117 | "metadata": {}, 118 | "output_type": "display_data" 119 | } 120 | ], 121 | "source": [ 122 | "_ = plt.hist(true_trajectories[:, -1], density=True, bins=100)" 123 | ], 124 | "metadata": { 125 | "collapsed": false 126 | } 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 8, 131 | "id": "b14484d9", 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "N = 500\n", 136 | "z = 3\n", 137 | "xs = jnp.linspace(-2, 6, N)\n", 138 | "dx = xs[1] - xs[0]" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 9, 144 | "id": "8d6c7f83", 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "def fpk_operator(ps):\n", 149 | " \"\"\"Kolmogorov forward equation operator (for unidimensional process)\n", 150 | " \"\"\"\n", 151 | " derivative_drift = a\n", 152 | " gamma = dispersion(xs) ** 2\n", 153 | " derivative_gamma = 2 * b ** 2 * xs\n", 154 | " second_derivative_gamma = 2 * b ** 2\n", 155 | " derivative_p = jnp.gradient(ps, dx)\n", 156 | " second_derivative_p = jnp.gradient(jnp.gradient(ps, dx), dx)\n", 157 | "\n", 158 | " part1 = -(derivative_drift * ps + drift(xs) * derivative_p)\n", 159 | " return part1 + 0.5 * (second_derivative_gamma * ps + 2 * derivative_gamma * derivative_p + gamma * second_derivative_p)\n", 160 | "\n", 161 | "def euler(ps):\n", 162 | " r\"\"\"p_k \\approx p_{k-1} + fpk_operator(p_{k-1}) * dt\n", 163 | " \"\"\"\n", 164 | " return ps + fpk_operator(ps) * dt\n", 165 | "\n", 166 | "def solve_fpk(p0):\n", 167 | " def scan_body(carry, _):\n", 168 | " ps = euler(carry)\n", 169 | " return ps, ps\n", 170 | " _, densities = jax.lax.scan(scan_body, p0, ts)\n", 171 | " return densities" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 10, 177 | "id": "a0ffa0b9", 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "p0 = jax.scipy.stats.norm.pdf(xs, m0, jnp.sqrt(var0))\n", 182 | "densities = solve_fpk(p0)" 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": 11, 188 | "id": "beabceda", 189 | "metadata": {}, 190 | "outputs": [ 191 | { 192 | "data": { 193 | "text/plain": "
", 194 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/ZElEQVR4nO3deXxV5YH/8c85NzskYQkkLJGwCeJCZBVcwDZKK3a0m9Rpi82vQ6cqVpt2ptBpodpWsLUMM0pF6aDYaqXWtVqxmorWGgUSUURAEcJqNpaskOWe8/vj5CZECeSS5bnL9/163dddcpcvEck3z/Oc51iu67qIiIiIGGKbDiAiIiLRTWVEREREjFIZEREREaNURkRERMQolRERERExSmVEREREjFIZEREREaNURkRERMSoGNMBOsJxHA4ePEhycjKWZZmOIyIiIh3gui7V1dUMHjwY225//CMsysjBgwfJzMw0HUNERETOwL59+xg6dGi7Xw+LMpKcnAx4f5iUlBTDaURERKQjqqqqyMzMbPk53p6wKCOBqZmUlBSVERERkTBzuiUWWsAqIiIiRqmMiIiIiFEqIyIiImKUyoiIiIgYpTIiIiIiRqmMiIiIiFEqIyIiImKUyoiIiIgYpTIiIiIiRqmMiIiIiFEqIyIiImKUyoiIiIgYFRYnyhPpcfsL4cO/AS6M/AxkToXTnOhJRETOjMqIyImaGuD578Pbf2h97NW74JwvwDW/hQSdNVpEpKtpmkYkwHXhmZu8ImLZcO4X4fzrwI6FbX+BP3wJGupMpxQRiTgaGREJKFoDWx4HOwa+9iicPQuAazadx0Nxv6Lv/o089fOv8P3Gm9u8rHjpbBNpRUQihkZGRABqD8HfFnm3c37WUkQA3nFH8Z2GPJpcmy/6/sln7CIzGUVEIpTKiAjA68ugvhLSz4eLbvrUlze6Y/md3xsBuSP2IZI43tMJRUQilsqISN1h2PSgdztnMdi+kz7tf5q+yD5nAEOtCv495rkeDCgiEtlURkQ2PwKNtZBxPozKafdpx0jgzqZ/BSDXt44UansqoYhIRFMZkejmurD5j97tSf/vtHuJrHMms8MZSopVxw2+F3sgoIhI5FMZkehW8i6UbQVfvHco72m42NzT5D3v2zEvkEB9dycUEYl4KiMS3QKjImM+D4l9O/SSvzpT2esMoI9Vyxd8Bd0YTkQkOqiMSPRyHHjvCe929r92/GXYPOL31pZ83fdydyQTEYkqKiMSvT7eDLVlEJcMIy4P6qWP+2dQ78aQbe+CA9p3RESkM1RGJHp9+JJ3PXImxMQF9dLDpPBXZ6p3p+jhrs0lIhJlVEYken34N+969JVn9PI/+Wd6N7Y+BU1ayCoicqZURiQ61VbAgULv9qgrzugt3nLO4WO3Hxw/2jrKIiIiQVMZkei0Mx9wvY3OUgad0Vs42Dzjn+7deXdt12UTEYkyKiMSnfa87l0HuXD1k572X+Ld+GAdHK/sZCgRkeikMiLRae+b3vWw6Z16m+3uWZB2NvgbNFUjInKGVEYk+tRWQMUH3u3MqZ1/v7FXe9fb/tL59xIRiUIxpgOI9LjAqMiAc8i6owt2UD3nanh9Gex8GRqPQ2xC599TRCSKaGREos/e5gJy1kVd836DJ0DKEGiogV3ru+Y9RUSiiMqIRJ/AyMhZ07rm/SwLxs72bm/XVI2ISLBURiS6NB73toEHOKsL1osEBNaN7HgB/E1d974iIlFAZUSiS9lWcJogqT/0GdZ17zvsYu+sv3WHYN+bXfe+IiJRQGVEosvH73jXg8Z70ytdxRcDo2d5twPbzIuISIeojEh0ObjZux6U3fXvPSrHu96Z3/XvLSISwVRGJLoE1osMzu769x75GcCC0veg6uOuf38RkQilMiLRo6kBSt/3bg8a3/Xv36s/DJng3f5IoyMiIh2lMiLRo+x9cBohoU/XLl49UctUzcvd8/4iIhFIZUSix4lTNF25ePVEgTLy0d91iK+ISAedURlZsWIFWVlZJCQkMHXqVDZs2NDucx966CEsy2pzSUjQdtliQMl73nXGBd33GUMmeiMvxyvhQGH3fY6ISAQJuoysXbuWvLw8Fi9eTFFREePHj2fWrFmUlZW1+5qUlBQ+/vjjlsuePXs6FVrkjJRv967Tz+2+z7B9zQtZgZ06i6+ISEcEXUaWLVvGvHnzyM3NZdy4caxcuZKkpCRWr17d7mssyyIjI6Plkp6e3qnQIkFzXSjd6t0eeE73ftaoz3rXu17t3s8REYkQQZWRhoYGCgsLycnJaX0D2yYnJ4eCgvbPflpTU8OwYcPIzMzkmmuuYevWraf8nPr6eqqqqtpcRDqlthyOHQbLhrSzu/ezhl/mXR8ohPrq7v0sEZEIEFQZqaiowO/3f2pkIz09nZKSkpO+ZsyYMaxevZpnnnmGP/zhDziOw/Tp09m/f3+7n7NkyRJSU1NbLpmZmcHEFPm0suZDevsOh9jE7v2sPmdB3yxw/bCn/ZIuIiKebj+aZtq0acydO5fs7GxmzJjBk08+yYABA7j//vvbfc3ChQuprKxsuezbt6+7Y0qkK9vmXXf3FE1AYHRkt6ZqREROJyaYJ6elpeHz+SgtLW3zeGlpKRkZGR16j9jYWC688EJ27tzZ7nPi4+OJj48PJprIqQVGRgaO65nPGz4Dih6G3a/1zOeJiISxoEZG4uLimDhxIvn5rbtLOo5Dfn4+06ZN69B7+P1+tmzZwqBBg4JLKtIZZc1H0vTUyEjWJd51yRaoO9wznykiEqaCnqbJy8tj1apVrFmzhm3btnHjjTdSW1tLbm4uAHPnzmXhwoUtz7/jjjv429/+xq5duygqKuIb3/gGe/bs4d/+7d+67k8hcique8I0TQ+NjCRnQNoYwIU9/+yZzxQRCVNBTdMAzJkzh/LychYtWkRJSQnZ2dmsW7euZVHr3r17se3WjnPkyBHmzZtHSUkJffv2ZeLEibzxxhuMG9dDPxREqg5CQzXYMdB/ZM997vDLoGKHN1Vzzhd67nNFRMKM5bquazrE6VRVVZGamkplZSUpKSmm40i42fUqPPwv0H8U3NJ2V9SsBc93+u2Ll84++Rfefxb+9E0YMBZufqvTnyMiEm46+vNb56aRyHfoQ++6/6ie/dysSwDL2/m1pv0dikVEop3KiES+iuYjt3q6jCT1g4zzvNs6qkZEpF1BrxkRCTuHmstI2uie/+zhM6BkC39c+wcWPnLyzdbaneYREYkSGhmRyGdqmgYg61IAptjbe/6zRUTChMqIRLameji617vd38DIyFlTARhpf0walT3/+SIiYUBlRCLb4d3gOhCXDL0H9vznJ/Zlu+OdW2mSvaPnP19EJAyojEhkC0zRpI0CyzISYaMzBtBUjYhIe1RGJLIFFq+amKJpttEZC8BklRERkZNSGZHIdugj77ond179hA3NIyPjrD304pixHCIioUplRCLbkWLvuu9wYxFK6M8+ZwA+y2WC/aGxHCIioUplRCLbkT3edd8sozE2uN7oiKZqREQ+TWVEIpe/Ear2e7f7DjMaJbBuZIqOqBER+RSVEYlclfu8w3pjEqB3utEogSNqsq2dxNFoNIuISKhRGZHI1bJeJMvYYb0BH7mDOeQmk2A1cp6122gWEZFQozIikevEMmKcxSbtNyIiclIqIxK5AotX+5hdLxIQOMR3staNiIi0obP2SuRqHhm54591rH7tebNZaF3EOsnegYWDq98FREQAjYxIJGsuI3tdA+ekOYmtbha1bjypVh1jrP2m44iIhAyNjEjk6qEykrWgY6MufnwUOaO51Pcek+3tbPef1a25RETChUZGJDIdOwrHjwKw3x1gNMqJWqdqPjCcREQkdKiMSGQ66i1eLXdTqCPBcJhWha53wr4JlraFFxEJUBmRyNQ8RbMvRNaLBLzjjMTvWmTa5QzkiOk4IiIhQWVEIlPzYb2hVkZqSGKH660V0UnzREQ8KiMSmULsSJoTFTreVM1ErRsREQFURiRShXQZORtQGRERCVAZkcjUXEZC6UiagMAi1vOs3cTTYDiNiIh5KiMSeVwXKr1NxfaFYBnZ5w6k3E0hzvLrpHkiIqiMSCSqrQB/PWBR6vYzneYkLIo0VSMi0kJlRCJPVfNW673TaQzRTYZbF7HqiBoREZURiTzNUzSkDjWb4xQCi1gn2B9400oiIlFMZUQiTxiUkffc4TS4PgZYVS2LbUVEopXKiESeMCgj9cTxnjvcu7Nvg9kwIiKGqYxI5AmDMgKtUzXse8tsEBERw1RGJPKEXRnRyIiIRDeVEYk8YVJGipqPqKFsKxyvMhtGRMQglRGJLE0NUFPq3U7NNJvlNMroy343DVwHDhSajiMiYozKiESW6oOACzEJkNTfdJrT0lSNiIjKiESawBRNyhCwLLNZOiCw+ZkWsYpINFMZkcgSJutFAlpGRvZvBMcxG0ZExBCVEYksYVZGtrtnQWwS1FdB+XbTcUREjFAZkcgSZmXEjw+GTPTu7Ne6ERGJTiojElnCrIwAkDnVu9YiVhGJUiojElnCuYzsfdNsDhERQ1RGJLJUHfCuU8KojAyd5F0f/ghqD5nNIiJigMqIRI7jld5CUIDUIWazBCOpH6QFjqrRVI2IRB+VEYkcgSmaxL4Q18tslmBlTvGutd+IiEShGNMBRDoja8HzLbdn2O+wJg621Sbz+RMeD3VZC57nOl8iv4qFt157gTkvT27z9eKlsw0lExHpGRoZkYiRbh0GoMTtazhJ8AKbn11g7SKGJsNpRER6lsqIRIwMjgBQ4vYznCR4u9xBHHV7kWg1cI6113QcEZEedUZlZMWKFWRlZZGQkMDUqVPZsKFji+4ee+wxLMvi2muvPZOPFTmljOaRkVLCb2TExaao+Tw1E+0PDKcREelZQZeRtWvXkpeXx+LFiykqKmL8+PHMmjWLsrKyU76uuLiYH/7wh1x66aVnHFbkVNKt8B0ZgdapGpUREYk2QZeRZcuWMW/ePHJzcxk3bhwrV64kKSmJ1atXt/sav9/P17/+dW6//XZGjBjRqcAi7ckI4zUjAEWuNzIywf7QcBIRkZ4VVBlpaGigsLCQnJyc1jewbXJycigoKGj3dXfccQcDBw7k29/+9pknFTmNwMhIaZiOjLzjjMTvWgyxDjEIbX4mItEjqEN7Kyoq8Pv9pKent3k8PT2d7dtPfsbR119/nf/7v/9j8+bNHf6c+vp66uvrW+5XVVUFE1OiUByNpFne35NwHRmpI4Ft7jDOs4qZYH/I805/05FERHpEtx5NU11dzTe/+U1WrVpFWlpah1+3ZMkSUlNTWy6ZmZndmFIiwUDrKAD1bixHSDYbphMKtYhVRKJQUCMjaWlp+Hw+SktL2zxeWlpKRkbGp57/0UcfUVxczBe+8IWWxxzH8T44JoYdO3YwcuTIT71u4cKF5OXltdyvqqpSIZFTSqf5SBq3D2AZzdIZhc7Z3MBLTFAZEZEoElQZiYuLY+LEieTn57ccnus4Dvn5+cyfP/9Tzx87dixbtmxp89hPfvITqqur+Z//+Z92C0Z8fDzx8fHBRJMolxE4kobwXC8SUOR6R9Sca+0hngbqiTOcSESk+wW9HXxeXh433HADkyZNYsqUKSxfvpza2lpyc3MBmDt3LkOGDGHJkiUkJCRw3nnntXl9nz59AD71uEhntOwxEqbrRQL2u2mUun1It45ygbWLje5Y05FERLpd0GVkzpw5lJeXs2jRIkpKSsjOzmbdunUti1r37t2LbWtjV+lZ4b7HSCuLImc0n/dtZKL9ARv9KiMiEvnO6ER58+fPP+m0DMD69etP+dqHHnroTD5S5JTCfY+RExU6ZzeXkQ/BbzqNiEj30xCGRIRw32PkRIFt4b1FrK7ZMCIiPUBlRCJCBpEzMvKeO5x6N4b+VjVZVonpOCIi3U5lRCKAGzFH0wA0EMsW1zttwkRLW8OLSORTGZGw14ca4q1GAMoiYGQEWjc/03lqRCQaqIxI2AuMihxyk2kg1nCarlHUfAZfbX4mItFAZUTCXuseI+E/RRMQWMQ6xtoPxysNpxER6V4qIxL2WvcYiYwpGoBy+rDXGYBtubB/k+k4IiLdSmVEwl46gcN6I6eMABQ2bw3Pvg1mg4iIdDOVEQl7A5tHRsqIsDLiBMrIW2aDiIh0M5URCXsDraMAlLl9jOboaoF1I+zfBI62YhWRyKUyImFvgOUt8Cx3Uw0n6Vo73Exq3ARoqIby7abjiIh0G5URCXsDWkZGImuaxo+Pzc5I746makQkgqmMSJhzGcBRIPKmaUCLWEUkOqiMSFjzdl9tArzDYSNNkRaxikgUUBmRsBZYvHrY7U0jMWbDdIO3A9M0h3dBTbnZMCIi3URlRMJaYL1IeQRO0QBU0RsGjPXu7NdUjYhEJpURCWsDI3i9SIvMKd61pmpEJEKpjEhYa9ljJALXi7TInOpd79toNoeISDdRGZGwNrBlmiayDuttI1BGDhZBU4PZLCIi3UBlRMJa65qRyNrwrI3+oyCxLzQdh5ItptOIiHQ5lREJa5G6FXwblnXCVI3WjYhI5FEZkbDWuuFZBE/TgBaxikhEi7yNGSSqRMMC1qwFz3OR7fBYHJRsfZWLFjwHWC1fL14621w4EZEuoJERCV8NtSRbx4AIn6YB3nFG0OTaZFhHGEKF6TgiIl1KZUTCV00pAHVuPLUkGA7TvY6RwHtuFgCT7B1mw4iIdDGVEQlf1V4Z8UZFrFM+NRJscsYAMEVlREQijMqIhK+aEiCy14ucaKPjbQs/2d5uOImISNdSGZHw1WZkJPJtbB4ZOds+QB+qDacREek6KiMSvprXjETqSfI+6TAp7HQGAzBZUzUiEkFURiR8RVkZAdjQPDqiRawiEklURiR8VUfXmhFoXTeiRawiEklURiR81UTXmhGAja5XRs6zdpPIccNpRES6hsqIhK+WMhLhW8GfYL+bxkG3H7GWn2z7I9NxRES6hMqIhCd/E9R6O5FG9Bl7P8VqnaqxdIiviEQGlREJT7VlgEuj6+MwyabT9KjAIb7ab0REIoXKiISn5sWrFaTiRtlf4w3NIyMT7J3E0GQ4jYhI50XXv+ISOaJw8WrAh+4Qjrq9SLLqOdcqNh1HRKTTVEYkPLXsMRJN60U8LjabnLMBbX4mIpFBZUTCU5RtBf9JrfuNaN2IiIQ/lREJT80nySsneg7rPdHGE3didV3DaUREOkdlRMJTlI+MbHFHcMyNo59VAxUfmI4jItIpKiMSnqJ4AStAIzFsdkZ5d/a8YTaMiEgnqYxIeIrCk+R90gbXm6phb4HZICIinaQyIuHHdaN+ZARa9xuh+J9aNyIiYU1lRMLPsSPgbwCgPIrO2PtJRc5oGlwfVO2HI8Wm44iInDGVEQk/zaMiJPajkRizWQw6RgLvuCO9O8Wvmw0jItIJKiMSfpq3gqd3utkcIeBNZ5x3Q2VERMKYyoiEn8DISLLKSMGJZUTrRkQkTKmMSPhpGRnJMJsjBBQ5o8GObV43stt0HBGRM6IyIuFHIyMtjhMPQyd5dzRVIyJh6ozKyIoVK8jKyiIhIYGpU6eyYcOGdp/75JNPMmnSJPr06UOvXr3Izs7m97///RkHFmkpI1oz4sm6xLtWGRGRMBV0GVm7di15eXksXryYoqIixo8fz6xZsygrKzvp8/v168d//dd/UVBQwLvvvktubi65ubm8+OKLnQ4vUapaZaSNE8uI1o2ISBgKuowsW7aMefPmkZuby7hx41i5ciVJSUmsXr36pM+fOXMmX/ziFznnnHMYOXIkt956KxdccAGvv67f4uQMNZ8kj2StGQFg6JTmdSMHtG5ERMJSUGWkoaGBwsJCcnJyWt/AtsnJyaGg4PRbUruuS35+Pjt27OCyyy5r93n19fVUVVW1uYi0aBkZURkBIC5J60ZEJKwFVUYqKirw+/2kp7cdHk9PT6ekpKTd11VWVtK7d2/i4uKYPXs299xzD1dccUW7z1+yZAmpqaktl8zMzGBiSiRrqIWGau+2FrC2yrrUu1YZEZEw1CNH0yQnJ7N582Y2btzIL3/5S/Ly8li/fn27z1+4cCGVlZUtl3379vVETAkHgcWrsUkQ19tsllCidSMiEsaC2ks7LS0Nn89HaWlpm8dLS0vJyGh/yNy2bUaN8k53np2dzbZt21iyZAkzZ8486fPj4+OJj48PJppEixMXr1qW2SyhZOhk8MV560YO74L+I00nEhHpsKBGRuLi4pg4cSL5+fktjzmOQ35+PtOmTevw+ziOQ319fTAfLeLR4tWTi0uCIVo3IiLhKeizjOXl5XHDDTcwadIkpkyZwvLly6mtrSU3NxeAuXPnMmTIEJYsWQJ46z8mTZrEyJEjqa+v569//Su///3vue+++7r2TyLRQYf1ti/rEtj7Bux+DSbeYDqNiEiHBV1G5syZQ3l5OYsWLaKkpITs7GzWrVvXsqh179692HbrgEttbS033XQT+/fvJzExkbFjx/KHP/yBOXPmdN2fQqKHNjxr34iZ8NqvYPer3roRTWOJSJiwXDf0V7tVVVWRmppKZWUlKSkppuOISU/fBJsfgc8ugkt/QNaC500nMq546WzvRlMD3DUMGuvgxjcg/VyzwUQk6nX057fOTSPhRSfJa19MHAyb7t3etd5oFBGRYKiMSHjRSfJObcRM71plRETCiMqIhJca7b56SoEyUvxPb9pGRCQMqIxI+PA3QW2Fd1sLWE9u4LmQlAaNtXCg0HQaEZEOURmR8FFbBrhgx0BSf9NpQpNtw4gZ3m1N1YhImFAZkfARWLzaa6D3Q1dOTutGRCTMBL3PiIgxNWXetRavtvHJw5uHWg6vx0PT3g1kL3ic95Z+1VAyEZGO0a+XEj4CW8Frvcgp7XcHUOykE2M5TLG3m44jInJaKiMSPrQVfIf90zkPgEvs9wwnERE5PZURCR86SV6Hvd5cRqbbWw0nERE5PZURCR8aGemwAmccjmsx1t7X+n0TEQlRKiMSPjQy0mFHSeY9N8u7o6NqRCTEqYxI+KjW7qvBeN0537vxUb7ZICIip6FDeyU8OI5GRoL0qn88N8U8S8U7LzB5w19w2/ndo+WsvyIihqiMSEgL7KHRjyqKEppwXIuzl2yiic1mg4WBQnc01W4iaVYV51nFbHFHmI4kInJSmqaRsDDQOgrAYZJpUofukCZiWg7xnWG/YziNiEj7VEYkLAy0jgBQ7vYxGyTMvOpcAMAMn8qIiIQulREJC4GRkTKVkaC86h8PwATrQ1KoMZxGROTkVEYkLAzEGxlRGQnOQdL4wBmCz3K1G6uIhCyVEQkLgZGRUvqaDRKGXnW80ZEZ9ruGk4iInJzKiIQFTdOcuZYy4nsHcM2GERE5CZURCQutZUQjI8Ha6Iyhzo0nwzrCWGuf6TgiIp+iMiJhIV1rRs5YPXEUOOMAHeIrIqFJZUTCgNs6MkIfo0nCVcshviojIhKCVEYk5KVQS7zVCGifkTMVWDcyyd5BL44ZTiMi0pbKiIS8wKjIUbcX9cSZDROm9rgZ7HbSibP8XGJvMR1HRKQNlREJeTqSpmv83ZkAQI5dZDiJiEhbKiMS8rR4tWu83FxGLvdtxsYxnEZEpJXKiIS81sWrOqy3MzY6Y6hyk0izqsi2dpqOIyLSQmVEQp6mabpGEzGsb17ImuPTVI2IhA6VEQl5gTP2asOzznvZPxGAz2rdiIiEEJURCXkaGek6650LaHJtxtj7ybRKTccREQFURiQM6Iy9XaeK3mx0xgI6qkZEQofKiIQ87b7atQJH1WiqRkRChcqIhLReHKOXVQ9ozUhXCZSRqfZ2kqkznEZERGVEQlxgVKTaTaSOBLNhIsQeN4MPnSHEWn6dq0ZEQoLKiIS0dEvrRbpDfmA3Vl+h4SQiIiojEuIGchSAcq0X6VIv+y8E4HJ7M/gbzYYRkainMiIhbYBGRrpFkXs2h9xkUq06KH7ddBwRiXIqIxLStMdI93CwedE/ybuz7VmzYUQk6qmMSEjTmpHus86Z4t3Y9hw4frNhRCSqqYxISEtvXjNSqsN6u9wbzrkcdXtBbRnsfdN0HBGJYiojEtLSrcMAlLj9DSeJPE3E8LLjnatGUzUiYpLKiIQu12VQoIygkZHu8IJ/snfj/WfBccyGEZGopTIioev4URKtBgBK3H6Gw0Sm153zIS4Zqg/CAe05IiJmqIxI6Kr6GIAjbm/qiTMcJjLVEwdnz/LubHvGbBgRiVoqIxK6qg8CUKLFq91r3DXe9fvPgOuazSIiUUllREJXlVdGSjVF071G5UBsEhzdCx/rXDUi0vNURiR0NU/TfKwy0r3ikrxCAt7oiIhIDzujMrJixQqysrJISEhg6tSpbNiwod3nrlq1iksvvZS+ffvSt29fcnJyTvl8kRbN0zSlOpKm+2mqRkQMCrqMrF27lry8PBYvXkxRURHjx49n1qxZlJWVnfT569ev5/rrr+eVV16hoKCAzMxMrrzySg4cONDp8BLhmkdGdCRNDzh7FsQkwuGPNFUjIj0u6DKybNky5s2bR25uLuPGjWPlypUkJSWxevXqkz7/kUce4aabbiI7O5uxY8fyu9/9DsdxyM/P73R4iXBVWsDaY+KTW4+q2fK42SwiEnVignlyQ0MDhYWFLFy4sOUx27bJycmhoKCgQ+9RV1dHY2Mj/fq1/9tufX099fX1LferqqqCiSmRouVoGu2+2p2yFjwPwJX2cB6Ig4/feJTpr1yE2/y7SvHS2SbjiUgUCGpkpKKiAr/fT3p6epvH09PTKSkp6dB7/OhHP2Lw4MHk5OS0+5wlS5aQmpracsnMzAwmpkSCxuNQdwjQyEhPWe9kU+UmMcg6zBRrh+k4IhJFghoZ6aylS5fy2GOPsX79ehISEtp93sKFC8nLy2u5X1VVpUISbaq99SL1bixH6W04THRoIJYX/FOYE7Oef/G9wVtN5wCtIyft0ciJiHRWUCMjaWlp+Hw+SktL2zxeWlpKRkbGKV979913s3TpUv72t79xwQUXnPK58fHxpKSktLlIlKk+8bBey2yWKPKsMw2Aq3xvEUuT4TQiEi2CKiNxcXFMnDixzeLTwGLUadOmtfu6X/3qV/z85z9n3bp1TJo06czTSvSo0mG9JhQ451LuptLXquESe4vpOCISJYI+miYvL49Vq1axZs0atm3bxo033khtbS25ubkAzJ07t80C17vuuouf/vSnrF69mqysLEpKSigpKaGmpqbr/hQSeap1WK8JDjbP+S8C4F98bxhOIyLRIug1I3PmzKG8vJxFixZRUlJCdnY269ata1nUunfvXmy7tePcd999NDQ08JWvfKXN+yxevJif/exnnUsvkat5ZES7r/a8Z/3TyY15kSvtTSRxnDraX98lItIVzmgB6/z585k/f/5Jv7Z+/fo294uLi8/kIyTatZyXRtM0Pe1tdxS7nAxG2CV83t7AE85lpiOJSITTuWkkNGmaxiCLJ/2XAvBl32uGs4hINFAZkdCkkRGjnvRfiuNaTPe9zxDKTccRkQinMiKhx3Gg2ttETyMjZhwkjQJnHABf8v3DcBoRiXQqIxJ6akrBaQTLp0N7Dfqz31sr4pURnclXRLqPyoiEnsr93nXKYPz4zGaJYuucydS4CQy3S5lofWA6johEMJURCT2Ve73r1KFmc0S5YyTwV/9UAL6ihawi0o1URiT0BEZGUnU+ItOeaJ6qme17k0SOG04jIpFKZURCT0sZ0ciIaRvcMRQ76aRYx5jte8t0HBGJUCojEnqO7vOuVUaMc7FZ678cgOt9fzecRkQilcqIhJ7AyEifs8zmEMA7qqbR9THR/pAx1l7TcUQkAqmMSOip1MhIKCmnDy85EwH4mu8Vw2lEJBKpjEhoqa+G40e92yojIeOP/s8A3p4jCdQbTiMikUZlREJLYIomoQ/EJxuNIq1ed85jnzOAVKuOq2wtZBWRrqUyIqGlZfGqDusNJS42fwwsZI3RQlYR6VoqIxJatF4kZD3un0GTazPZ/kALWUWkS6mMSGhpOZJGIyOhppy+vOhMAuAG34uG04hIJFEZkdCikZGQtqZpFgBf9P2TVGoMpxGRSKEyIqFFW8GHtA3uWN53hpFoNXCdb73pOCISIVRGJLSojIQ4i4f8VwIw1/cSNo7hPCISCVRGJHT4m6DqoHdb0zQh6xn/xRxxe5Npl/NZu8h0HBGJACojEjqqPwbXD3Ys9E43nUbaUU9cy/lqtJBVRLqCyoiEjpYpmiFg669mKPt9Uw5+1+IS31Yofd90HBEJc/oXX0LH0T3etdaLhLwDDOBFZ7J3p+Bes2FEJOypjEjoOFLsXfcbbjSGdMwDTVd7N979U+taHxGRM6AyIqEjUEb6ZplMIR202R3FW85YcBrhzftMxxGRMKYyIqHj8G7vWmUkbNwfGB0pfAiOVxnNIiLhS2VEQodGRsLOK042pI2B+iqvkIiInAGVEQkNDXVQU+Ld7qs1I+HCxYbpt3h33rwPmhrMBhKRsKQyIqHhaPNZYONTILGv2SwSnAuug94ZUH0Q3n3MdBoRCUMqIxIaWqZohoFlGY0iQYqJbx0d+cdvvJ10RUSCoDIioaGljGiKJixNyoWk/t5/xy2Pm04jImEmxnQAiW5ZC54HYHHM38mNgZVbHJa+/bzhVBK0uF7e6MjLP4N/3O1N3dg+06lEJExoZERCQqZVBsA+d6DhJHLGJv+bt97n0E7Y+pTpNCISRjQyIiFhWEsZGWA4iQQrMLoFcLPvCv4j9k98+PgirnwkHheb4qWzDaYTkXCgkRExzsbhLKsUgF3uIMNppDMe9l9JpZvEaPsAV9tvmo4jImFCZUSMG2xVEG81Ue/GctBNMx1HOqGaJH7XdBUA34/5MzHoyBoROT2VETFuhPUxAMVuOo7+Soa91f7PU+GmMMIu4au+V03HEZEwoH/5xbhAGdmtKZqIUEsiK5quAeDWmCeh8ZjhRCIS6lRGxLjhLWUkw3AS6SqP+HPY76aRYR2BDatMxxGREKcyIsYNt7xz0mjxauRoIJb/afqSd+f1ZXC80mwgEQlpKiNi3Ai7eWTE0chIJHnSfyk7ncFw7Aj8Y5npOCISwlRGxKh4GhjMIUBrRiKNHx9Lm6737rz529Yt/0VEPkGbnolRw6xSbMulyk3iECmm40gXe9mZAMNnwO5X4aXFcN2aoN/jxE3VTkabqomEP42MiFGBxau73AxAZ+uNPBbMuhMsG95/GvYUmA4kIiFIZUSMGm0dAOAjd7DhJNJtMs6DCXO92+sWgOOYzSMiIUdlRIw6294PwIfOUMNJpFtd/l8Qlwwfb4Z3/mg6jYiEGJURMWq05ZWRD1yVkYjWeyDM+A/v9ks/hbrDZvOISEhRGRFz/E0tu6+qjESBi26CAWOh7hDk32E6jYiEEJURMefIbuKtJurceA7oBHmRzxcLs5v3Gyl8CPZvMhpHRELHGZWRFStWkJWVRUJCAlOnTmXDhg3tPnfr1q18+ctfJisrC8uyWL58+ZlmlUhTtg2Ane5gXPXi6JB1MYz/V8CF524Dv87qKyJnUEbWrl1LXl4eixcvpqioiPHjxzNr1izKyspO+vy6ujpGjBjB0qVLycjQDptygvLtAHyoKZrocuXPIaEPlGyBDfebTiMiISDoMrJs2TLmzZtHbm4u48aNY+XKlSQlJbF69eqTPn/y5Mn8+te/5mtf+xrx8fGdDiwRpHlk5AMdSRNdeqXBFbd7t/N/Dod3mc0jIsYFVUYaGhooLCwkJyen9Q1sm5ycHAoKum4zo/r6eqqqqtpcJAKV7wC0eDUqTbgBhl8GTcfgmVu094hIlAtqO/iKigr8fj/p6eltHk9PT2f79u1dFmrJkiXcfvvtXfZ+EoKa6qGiuYxoZCSitbede6b1RV6Me4ukPa/Dpv+DKfN6OJmIhIqQXDW4cOFCKisrWy779u0zHUm6Wtk2cJo46vbiADqSJhrtc9O5q+lr3p2XFsORPWYDiYgxQZWRtLQ0fD4fpaWlbR4vLS3t0sWp8fHxpKSktLlIhCl5F4CtThY6J030eth/BZw1DRpr4embwPGbjiQiBgRVRuLi4pg4cSL5+fktjzmOQ35+PtOmTevycBLBPm4uI26W2RxilIsN16yA2F6w53V4439NRxIRA4KepsnLy2PVqlWsWbOGbdu2ceONN1JbW0tubi4Ac+fOZeHChS3Pb2hoYPPmzWzevJmGhgYOHDjA5s2b2blzZ9f9KST8tIyMDDMcRIzrPxI+f5d3+++/gINvm80jIj0uqAWsAHPmzKG8vJxFixZRUlJCdnY269ata1nUunfvXmy7teMcPHiQCy+8sOX+3Xffzd13382MGTNYv3595/8EEn4cP5S8B2hkRAILXPvy29gpXMUGPlp5PVc3/JJjJJiOJiI9xHJd1zUd4nSqqqpITU2lsrJS60ciQcVOuHcixCQwsmYVfnymE0kISKWGdfELGGQd5rGmmSxo+k6HXle8dHY3JxORM9XRn99Bj4yIBONkh3VebRdwbxxsbhiiIiItKulNXuONPBJ7J1+LWc8mdwx/9s8wHUtEekBIHtorke0C29txc4sz3HASCTUFzrksa/oKAL+IWc05lg73FYkGKiPS4y60PwTgbWeU4SQSilb4r+EV/3gSrEZ+G7ucZOpMRxKRbqYyIj0qlibOt3YD8LY72nAaCUUuNt9vvIn9bhrD7VKWxd6HhbaLF4lkKiPSo8Zae0mwGjnq9mK3q7M4y8kdJZmbG75HvRvLFb5C/iPmT6YjiUg3UhmRHtV2ikY7r0r73nFH8Z+N3vlqbop5li/ZrxlOJCLdRWVEetSFtrfZ3duOpmjk9J5xLuHepmsAWBL7OyZaOwwnEpHuoDIiPepCq7mMuFq8Kh3zm6avss4/mXirifvj/puhVrnpSCLSxVRGpMekc5gsuxS/a7FZR9JIB3kLWm9kqzOMNKuKNbFL6UeV6Vgi0oVURqTHXGS/D3hbwFeTZDiNhJNjJPDthh+y301jpP0xD8XdRS+OmY4lIl1EZUR6zEX2NgAKnHGGk0g4KqE/cxsWcMhN5gJ7Nw/ELiOeBtOxRKQLqIxIj5nWPDLypsqInKFd7mC+1fAjatwELvZtZXnsCu/EiyIS1lRGpEcM4lDLepGNzhjTcSSMbXFHMK/xB9S7MXzetxGe+nfwN5mOJSKdoDIiPSKwXmSLO5warReRTipwzuV7jbfQ6Ppgy+Pw1HdUSETCmMqI9IhLfVsA74eISFd40ZnM/MbvgR0D7z0BT3wb/I2mY4nIGVAZkW5n4zDT3gzAK/5so1kksrzoTIbrfg92LLz/NPw5F5rqTccSkSCpjEi3G299RD+rhio3iSKdHE+62tir4GuPgC8Otv0FHvkqHNc+JCLhRGVEut1M32YAXnPOp4kYs2EkMp09C65/DGJ7we5X4aGroLrUdCoR6SCVEel2n7HfBuAV/4WGk0gkylrwvHf53XG+ULOQCjcFSraw99cXc/nC35mOJyIdoDIi3WoQhzjfLsZxLV51xpuOIxFuizuCLzf8jGInnbPscp6IWwy7dbZfkVCnMiLd6irfmwBsdMdQQarhNBIN9rgZfLnhZ7zjjKCfVQMPXwtvPQCuazqaiLRDZUS61Reay8hz/osMJ5FocohUrmtYxNP+6eD64YX/gL98T0faiIQolRHpPkf2kG1/hN+1WOefYjqNRJl64rit8Wa44g7AgqKH4aHZcHSv6Wgi8gkqI9J9tj4JwFvOOZTTx2wWiVIWXHwrfP3PEJ8K+zfCyku8Q4BFJGSojEj3cF14+xEAnnEuNhxGot7oHPjuazBkEhyvhLXfgOd/CI3HTScTEcBy3dBf1VVVVUVqaiqVlZWkpKSYjiMdsecNePDz1LrxTKn/LbUkmk4kQgxN/DDmT3w35jkAdjhD+UHjd3luyS2Gk4lEpo7+/NbIiHSPoocB+It/moqIhIwmYlja9K98q+E/KXdTGGPv5+m4RfD3X0JTg+l4IlFLZUS6Xt1h2Po0AGv9l5vNInIS651sZtX/iuf8FxFjOfDar2DV5XCgyHQ0kaikMiJdb9NqaDoG6efztjvKdBqRkzpMCvMbv8dNDd+DpP5Q+h6s+gw8lwfHjpiOJxJVVEakazUeh7fu925PvwWwjMYROZ2/OhfBTW/B+dcBLmz6P7hnEmx+FBzHdDyRqKCzlkmnZC14vs39Ob5XuCu2jANuf2b8McFQKpEg9R4AX14FE77pHWVTsQOevhE2PABX/ByGX9qpt//k/yefVLx0dqfeXyTcaWREukwcjcz3PQ3A6qbP6Qy9En6GXwbffR1yfgZxveHg27Dmanh0DpRtM51OJGLpp4V0met9fyfTLqfU7cMj/hzTcUTOTEwcXPJ9yP4GvHoXFD4IH6zDv+NFnvBfxgr/NexxM9q8RCMbIp2jkRHpEr04xvyYpwD436YvcZx4w4lEOqn3AJh9N9z0Fi/4J+OzXK6LeZW/x/2AZbG/ZaR1wHRCkYihMiJd4vsxf2aAVcVuJ521/pmm44h0nbRR3Nj4fb5Yfzuv+Mfjs1y+5Hudl+L+k3tj/5fzrF2mE4qEPU3TSKeNs4r5lu9FAH7W9C2tFZGI9LY7mtzGH3F+0y5uiXmKK32FXO17k6t9b7Jx0cM82PQ5XnQm48dnOqpI2NFPDemUWJq4K/YBYiyH5/xTedUZbzqSSLfa4o7gO40/YGzTXr4b8yyz7beYbH/A5LgPOOD25w9NV/C4fwYVpJqOKhI2NE0jnfKDmMc53y7miNubOxrnmo4j0mO2u2dxW+N8Lq7/X/636VoOuckMsQ7xo9jHKIifz/2xy/iMXYQPv+moIiFPIyNy5nas4zs+74RjP2qcRxl9DQcS6Xll9GVZ03WsaLqWf/G9wfW+vzPB3sks3yZm+TZR4vblaf8l/MV/EVvdLLQRoMin6ay9cmbKtsHvcqChhoebrmBRU67pRCIhY7S1nzm+V/ii73X6W9Utj+9yMnjOuYi/+KfzoTu05XEdGiyRqqM/v1VGJHhH9sCDn4eqAxT4x/HNxgVatCpyErE08Rm7iC/4Csixi0iwGlu+9qEzhJedCbzsn8ATv7wVbC18lcijMiLd48geePhf4EgxpJ3NhfvzOIL+m4icTi+O8Vm7iC/43mSGvZk464S1JIn9YPSVMOZzMHwGJPUzF1SkC6mMSNc7UOhti11bDn2zIPcFsu5823QqkbCTQi0z7Xf4rK+ImfZmUq26E75qwaALvFIyYgacNQ3iehnLKtIZKiPStbb8GZ69BRrrIP18+PqfIGXwaU8AJiKn5sPPR9/tBztegA9f8k7SdyI7FoZOhmHTIHOqd/sTIyc6EZ+Eqo7+/NZEv5za8Ur463/Au2u9+yM/C9etgfhks7lEIoQfH1krK4HpwHQGcoRp9lYutrcy3beVoU4F7H3DuwT0H9VaTIZMIJYmGvXPuYQx/e2Vk3MceOePkH8H1JSAZcOlP4QZ/wm+WNPpRCJWGX15xrmEZ5xLoMnlLKuM177ig30bYP8GqPgADu30LpsfAWBrvI8P3aFsdbLY6max1RnGNncYtSQa/tOIdIymaaQtxw/bn4PX7oaSdwHY7aTzg8YbKXLPNhxOJDq1mWapOwz7N8G+t7xy8vE73gjmJziuxX43jZ3uED5zySUwYCwMGANpZ0NinzbP1TSPdBdN00hwqg7ClsehcA0c/sh7LC6ZO2uv5iH/52hAoyEiISGpH5x9pXcBcF0u+fEazrWKOdfezbnWHs61i8mwjnCWVc5ZlEPB5rbv0TsDBpwNfYdD3yyuso+w1x3IXncgVfTu8T+SiEZGotmRYn5y93KutDdxif0etuX9VTjq9mKN/0rWNM3isA7bFQlL/alklHWQ0fZ+fnFxnLcwtnwHVH98ytcddXu1FJMDbholbj8Wf/0KSBkCKYOhd7r2RJEO69ajaVasWMGvf/1rSkpKGD9+PPfccw9Tpkxp9/mPP/44P/3pTykuLmb06NHcddddXHXVVR3+PJWRLuBvgvJt3vDugULYW+DNOZ/gLWcsT/kv4Vn/dOpIMBRURLpam2mW45VQ8aG39uRIMRwpZtPmtznLKmOgdfT0b2b5IDnDKyYpgyF5EPRKg14DofdA6DWg9RKX1F1/JAkT3TZNs3btWvLy8li5ciVTp05l+fLlzJo1ix07djBw4MBPPf+NN97g+uuvZ8mSJVx99dU8+uijXHvttRQVFXHeeecF+/FyKq4LdYegcj8c3tX6D07FB97tpmNtn2/H8FbTKNb7s3nOmco+N91MbhHpOQmpMHSSd2n2lQ3empFEjpNplXOWVcYwq5QM6zCDrEMMsg6TYR0mnSPE4IeqA97ldOJ6Q68BFB6KocJN5Yjbm6P0ptLtTSW9OOr24ii9eXT+LEjs613ik8Hq+vP3aF1MaAt6ZGTq1KlMnjyZe++9FwDHccjMzOSWW25hwYIFn3r+nDlzqK2t5bnnnmt57KKLLiI7O5uVK1d26DOjdmTEdb19PY4d9UrGscPe4rW6Q3DsCNQd4uk33iXDOkIGhxlkHSb+hO2mP6naTeRdZwSb3ZG87YzmLeccqtFvLiLSMTYOaVQyyDpEhnWYwdYhBliVpFFJf6uKNMu7HkDlKf8tOiXL5y2wTezrFae43qzbWUctCVS7idSSQI2bRDWJ1LoJ1JDIqn+b6ZWYwCWuN8Qmgd16YvrOlpFwLzOm8nfLyEhDQwOFhYUsXLiw5THbtsnJyaGgoOCkrykoKCAvL6/NY7NmzeLpp59u93Pq6+upr69vuV9VVRVMzO7nuuA0QdNxaKpvvW48dsL9T3yt5bHj0FDrXeqrW2831HiX+pq2j3HqrnjtSaZuy91U9rkD+MgZzEdu66XYzcDB/vQLREQ6wMGmjL6UuX1555T/NLkkc4z+JxSVAVYlqdTSx6qhDzWkWrWkWrX0oYYxqX7vFy1/Pbh+7xeuukMt7/a50y1ReXjZyR+PSfSmimJ78bc4P3XEc8xN4Bhx1BHPTxr/H0fRnkmhIKgyUlFRgd/vJz297XB+eno627dvP+lrSkpKTvr8kpKSdj9nyZIl3H777cFEOzMFv4Uju9spEye7PuHrrtP9+QLsGO/cFUn9vZX0iX1PuN2vee52SMv87YCYOAYAE3ouoYhI5zUe80aCjx3xLvVV3i9tgUtDTfPtmtavtXmsGhqqW/99bjrWPD19iLNP8nvY1Qsf6/B5gEJ95ON0Qj1/SB7au3DhwjajKVVVVWRmZnb9B219EvZv7Pz7+OIhJgFi2rmOTWi974v3zjMR39u7jks+4X7gsebr+OavxSZ1yxyqiEhIiU30LimDzvw9AtPbDXXQWNt8XeeNNjfWnfC1Ou0kHUKCKiNpaWn4fD5KS0vbPF5aWkpGRsZJX5ORkRHU8wHi4+OJj48PJtqZGX89jLj8JCXiE4UiNrH9ouGLbzMvKSIiBllW8y91vYABptNIBwVVRuLi4pg4cSL5+flce+21gLeANT8/n/nz55/0NdOmTSM/P5/bbrut5bGXXnqJadOmnXHoLjP526YTiIiIRL2gp2ny8vK44YYbmDRpElOmTGH58uXU1taSm5sLwNy5cxkyZAhLliwB4NZbb2XGjBn85je/Yfbs2Tz22GNs2rSJBx54oGv/JCIiIhKWgi4jc+bMoby8nEWLFlFSUkJ2djbr1q1rWaS6d+9e7BOmLaZPn86jjz7KT37yE3784x8zevRonn76ae0xIiIiIoC2gxcREZFu0tGf31p5KSIiIkapjIiIiIhRKiMiIiJilMqIiIiIGKUyIiIiIkapjIiIiIhRKiMiIiJilMqIiIiIGKUyIiIiIkapjIiIiIhRQZ+bxoTAjvVVVVWGk4iIiEhHBX5un+7MM2FRRqqrqwHIzMw0nERERESCVV1dTWpqartfD4sT5TmOw8GDB0lOTsayrC5736qqKjIzM9m3b59OwGeAvv9m6ftvjr73Zun733Nc16W6uprBgwdj2+2vDAmLkRHbthk6dGi3vX9KSor+Qhqk779Z+v6bo++9Wfr+94xTjYgEaAGriIiIGKUyIiIiIkZFdRmJj49n8eLFxMfHm44SlfT9N0vff3P0vTdL3//QExYLWEVERCRyRfXIiIiIiJinMiIiIiJGqYyIiIiIUSojIiIiYpTKSLPi4mK+/e1vM3z4cBITExk5ciSLFy+moaHBdLSItGLFCrKyskhISGDq1Kls2LDBdKSosGTJEiZPnkxycjIDBw7k2muvZceOHaZjRa2lS5diWRa33Xab6ShR48CBA3zjG9+gf//+JCYmcv7557Np0ybTsaKeykiz7du34zgO999/P1u3buW///u/WblyJT/+8Y9NR4s4a9euJS8vj8WLF1NUVMT48eOZNWsWZWVlpqNFvFdffZWbb76ZN998k5deeonGxkauvPJKamtrTUeLOhs3buT+++/nggsuMB0lahw5coSLL76Y2NhYXnjhBd5//31+85vf0LdvX9PRop4O7T2FX//619x3333s2rXLdJSIMnXqVCZPnsy9994LeOceyszM5JZbbmHBggWG00WX8vJyBg4cyKuvvspll11mOk7UqKmpYcKECfz2t7/lF7/4BdnZ2Sxfvtx0rIi3YMEC/vnPf/KPf/zDdBT5BI2MnEJlZSX9+vUzHSOiNDQ0UFhYSE5OTstjtm2Tk5NDQUGBwWTRqbKyEkB/z3vYzTffzOzZs9v8fyDd79lnn2XSpEl89atfZeDAgVx44YWsWrXKdCxBZaRdO3fu5J577uHf//3fTUeJKBUVFfj9ftLT09s8np6eTklJiaFU0clxHG677TYuvvhizjvvPNNxosZjjz1GUVERS5YsMR0l6uzatYv77ruP0aNH8+KLL3LjjTfyve99jzVr1piOFvUivowsWLAAy7JOedm+fXub1xw4cIDPfe5zfPWrX2XevHmGkot0r5tvvpn33nuPxx57zHSUqLFv3z5uvfVWHnnkERISEkzHiTqO4zBhwgTuvPNOLrzwQr7zne8wb948Vq5caTpa1IsxHaC7/eAHP+Bb3/rWKZ8zYsSIltsHDx7k8ssvZ/r06TzwwAPdnC76pKWl4fP5KC0tbfN4aWkpGRkZhlJFn/nz5/Pcc8/x2muvMXToUNNxokZhYSFlZWVMmDCh5TG/389rr73GvffeS319PT6fz2DCyDZo0CDGjRvX5rFzzjmHJ554wlAiCYj4MjJgwAAGDBjQoeceOHCAyy+/nIkTJ/Lggw9i2xE/cNTj4uLimDhxIvn5+Vx77bWA99tKfn4+8+fPNxsuCriuyy233MJTTz3F+vXrGT58uOlIUeWzn/0sW7ZsafNYbm4uY8eO5Uc/+pGKSDe7+OKLP3Uo+wcffMCwYcMMJZKAiC8jHXXgwAFmzpzJsGHDuPvuuykvL2/5mn5j71p5eXnccMMNTJo0iSlTprB8+XJqa2vJzc01HS3i3XzzzTz66KM888wzJCcnt6zTSU1NJTEx0XC6yJecnPyp9Tm9evWif//+WrfTA77//e8zffp07rzzTq677jo2bNjAAw88oFHwEKAy0uyll15i586d7Ny581PD1jr6uWvNmTOH8vJyFi1aRElJCdnZ2axbt+5Ti1ql6913330AzJw5s83jDz744GmnM0XC3eTJk3nqqadYuHAhd9xxB8OHD2f58uV8/etfNx0t6mmfERERETFKiyJERETEKJURERERMUplRERERIxSGRERERGjVEZERETEKJURERERMUplRERERIxSGRERERGjVEZERETEKJURERERMUplRERERIxSGRERERGj/j8noD7ac5yWewAAAABJRU5ErkJggg==\n" 195 | }, 196 | "metadata": {}, 197 | "output_type": "display_data" 198 | } 199 | ], 200 | "source": [ 201 | "idx = -1\n", 202 | "_ = plt.hist(true_trajectories[:, idx], density=True, bins=50)\n", 203 | "_ = plt.plot(xs, densities[idx, :])" 204 | ] 205 | } 206 | ], 207 | "metadata": { 208 | "kernelspec": { 209 | "display_name": "Python 3 (ipykernel)", 210 | "language": "python", 211 | "name": "python3" 212 | }, 213 | "language_info": { 214 | "codemirror_mode": { 215 | "name": "ipython", 216 | "version": 3 217 | }, 218 | "file_extension": ".py", 219 | "mimetype": "text/x-python", 220 | "name": "python", 221 | "nbconvert_exporter": "python", 222 | "pygments_lexer": "ipython3", 223 | "version": "3.9.12" 224 | } 225 | }, 226 | "nbformat": 4, 227 | "nbformat_minor": 5 228 | } 229 | -------------------------------------------------------------------------------- /lectures/scripts/lec4_kolmogorov_forward_equation_np.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "# A computational introduction to stochastic differential equations.\n", 7 | "\n", 8 | "Lecture 4.\n", 9 | "\n", 10 | "https://github.com/spdes/computational-sde-intro-lecture.\n", 11 | "\n", 12 | "Solve a Kolmogorov forward equation (a.k.a. Fokker--Planck) using finite difference." 13 | ], 14 | "metadata": { 15 | "collapsed": false 16 | } 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "outputs": [], 22 | "source": [ 23 | "import math\n", 24 | "import numpy as np\n", 25 | "import scipy.stats\n", 26 | "import matplotlib.pyplot as plt\n", 27 | "\n", 28 | "np.random.seed(666)" 29 | ], 30 | "metadata": { 31 | "collapsed": false 32 | } 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 2, 37 | "id": "d31926f9", 38 | "metadata": {}, 39 | "outputs": [], 40 | "source": [ 41 | "# Geometric Brownian motion SDE.\n", 42 | "\n", 43 | "a, b = -2., 1.\n", 44 | "\n", 45 | "\n", 46 | "def drift(x):\n", 47 | " return a * x\n", 48 | "\n", 49 | "\n", 50 | "def dispersion(x):\n", 51 | " return b * x" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 3, 57 | "id": "ba3bf30c", 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "# Times\n", 62 | "dt = 2e-5\n", 63 | "T = 10000\n", 64 | "ts = np.linspace(dt, dt * T, T)\n", 65 | "\n", 66 | "# Initial condition is a Normal(m0, var0)\n", 67 | "m0 = 2.\n", 68 | "var0 = 1." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 4, 74 | "outputs": [], 75 | "source": [ 76 | "def simulate_true_trajectory(num_trajs: int):\n", 77 | " \"\"\"We have access to the true solution of the geometric Brownian motion.\n", 78 | " \"\"\"\n", 79 | " x0s = m0 + math.sqrt(var0) * np.random.randn(num_trajs)\n", 80 | " wss = np.cumsum(math.sqrt(dt) * np.random.randn(num_trajs, T), axis=1)\n", 81 | " return x0s[:, None] * np.exp((a - b ** 2 / 2) * ts + b * wss)" 82 | ], 83 | "metadata": { 84 | "collapsed": false 85 | } 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 5, 90 | "outputs": [], 91 | "source": [ 92 | "num_trajs = 2000\n", 93 | "true_trajectories = simulate_true_trajectory(num_trajs)" 94 | ], 95 | "metadata": { 96 | "collapsed": false 97 | } 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "source": [ 102 | "Check the histogram at the terminal time." 103 | ], 104 | "metadata": { 105 | "collapsed": false 106 | } 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 6, 111 | "outputs": [ 112 | { 113 | "data": { 114 | "text/plain": "
", 115 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAdu0lEQVR4nO3df2xd5X0/8I/tYJuM2AS82OBadfmxQUSJwSaeQS1Mc+tp2Y9M++FVXZ1ZW/5gcZf2blPjdUtGW8XZgMjrkuISLWu1ChGt6i81LIjdFTSGp6yOotEfhLEqJIVdJxGtDUayka+/f/RbIyd28HWcPL7O6yUdCT9+nnM+R5dcv/Wc55xTMjU1NRUAAImUpi4AALi8CSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAktSJ1AfORz+fj1VdfjVWrVkVJSUnqcgCAeZiamorXX389rr/++igtnXv+oyjCyKuvvhoNDQ2pywAAFuDkyZPxrne9a87fF0UYWbVqVUT85GSqqqoSVwMAzMfo6Gg0NDRM/x2fS1GEkZ9emqmqqhJGAKDIvNMSCwtYAYCkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkFhRG9u7dG42NjVFZWRmtra1x+PDh8/b/8Y9/HFu2bInrrrsuKioq4ud+7ufiiSeeWFDBAMDysqLQAQcOHIhMJhMDAwPR2toa/f390dHREceOHYs1a9ac039iYiI+8IEPxJo1a+LLX/5y1NfXx8svvxxXX331YtQPs2rcdnDGz8d3bUhUCQDvpOAwsnv37ti8eXN0d3dHRMTAwEAcPHgw9u/fH9u2bTun//79++O1116L5557Lq644oqIiGhsbLywqgGAZaOgyzQTExMxNDQU7e3tb++gtDTa29tjcHBw1jHf+MY3oq2tLbZs2RK1tbVx2223xc6dO2NycnLO44yPj8fo6OiMDQBYngoKI2fOnInJycmora2d0V5bWxu5XG7WMT/4wQ/iy1/+ckxOTsYTTzwRf/VXfxUPP/xwfOYzn5nzOH19fVFdXT29NTQ0FFImAFBELvrdNPl8PtasWROPPvpoNDc3R2dnZ3zyk5+MgYGBOcf09vbGyMjI9Hby5MmLXSYAkEhBa0ZqamqirKwshoeHZ7QPDw9HXV3drGOuu+66uOKKK6KsrGy67dZbb41cLhcTExNRXl5+zpiKioqoqKgopDQAoEgVNDNSXl4ezc3Nkc1mp9vy+Xxks9loa2ubdcw999wTL730UuTz+em2F198Ma677rpZgwgAcHkp+DJNJpOJffv2xRe/+MX4/ve/H/fff3+MjY1N313T1dUVvb290/3vv//+eO2112Lr1q3x4osvxsGDB2Pnzp2xZcuWxTsLAKBoFXxrb2dnZ5w+fTq2b98euVwumpqa4tChQ9OLWk+cOBGlpW9nnIaGhnjyySfj4x//eNx+++1RX18fW7dujU984hOLdxYAQNEqmZqamkpdxDsZHR2N6urqGBkZiaqqqtTlUAQ89Awgvfn+/fZuGgAgqYIv08BSc/YsCADFxcwIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAkJYwAAEkJIwBAUsIIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSK1IXAKk0bjs44+fjuzYkqgTg8mZmBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIylt7uSyc/YZeAJYOMyMAQFLCCACQlDACACRlzQhL2tlrPY7v2pCoEgAuFjMjAEBSwggAkJQwAgAkJYwAAEkJIwBAUsIIAJDUgsLI3r17o7GxMSorK6O1tTUOHz48Z98vfOELUVJSMmOrrKxccMEAwPJScBg5cOBAZDKZ2LFjRxw5ciTWrVsXHR0dcerUqTnHVFVVxf/93/9Nby+//PIFFQ0ALB8Fh5Hdu3fH5s2bo7u7O9auXRsDAwOxcuXK2L9//5xjSkpKoq6ubnqrra29oKIBgOWjoDAyMTERQ0ND0d7e/vYOSkujvb09BgcH5xz3xhtvxLvf/e5oaGiI3/iN34jvfve75z3O+Ph4jI6OztgAgOWpoDBy5syZmJycPGdmo7a2NnK53Kxjfv7nfz72798fX//61+NLX/pS5PP5uPvuu+OHP/zhnMfp6+uL6urq6a2hoaGQMgGAInLR76Zpa2uLrq6uaGpqinvvvTe+8pWvxM/+7M/G5z//+TnH9Pb2xsjIyPR28uTJi10mAJBIQS/Kq6mpibKyshgeHp7RPjw8HHV1dfPaxxVXXBF33HFHvPTSS3P2qaioiIqKikJKAwCKVEFhpLy8PJqbmyObzcbGjRsjIiKfz0c2m42enp557WNycjKef/75+JVf+ZWCi2V5O/sNvQBcHgoKIxERmUwmNm3aFC0tLbF+/fro7++PsbGx6O7ujoiIrq6uqK+vj76+voiI+NSnPhW/8Au/EDfddFP8+Mc/jgcffDBefvnl+KM/+qPFPRMAoCgVHEY6Ozvj9OnTsX379sjlctHU1BSHDh2aXtR64sSJKC19eynKj370o9i8eXPkcrlYvXp1NDc3x3PPPRdr165dvLMAAIpWydTU1FTqIt7J6OhoVFdXx8jISFRVVaUuh4tkPpdpju/asKBx8zHbvudzrPmMA7gczffvt3fTAABJCSMAQFLCCACQVMELWGGxuJUXgAgzIwBAYsIIAJCUyzQUFZd2AJYfMyMAQFLCCACQlDACACRlzQj8fx71DpCGmREAIClhBABIymUaOI/53Ep8dh+XdgAKY2YEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApFakLgAuB43bDs74+fiuDYkqAVh6zIwAAEktKIzs3bs3Ghsbo7KyMlpbW+Pw4cPzGvf4449HSUlJbNy4cSGHBQCWoYLDyIEDByKTycSOHTviyJEjsW7duujo6IhTp06dd9zx48fjz/7sz+J973vfgosFAJafgsPI7t27Y/PmzdHd3R1r166NgYGBWLlyZezfv3/OMZOTk/HhD384HnjggbjhhhsuqGAAYHkpKIxMTEzE0NBQtLe3v72D0tJob2+PwcHBOcd96lOfijVr1sQf/uEfzus44+PjMTo6OmMDAJangsLImTNnYnJyMmpra2e019bWRi6Xm3XMs88+G//wD/8Q+/btm/dx+vr6orq6enpraGgopEwAoIhc1LtpXn/99fjIRz4S+/bti5qamnmP6+3tjZGRkent5MmTF7FKACClgp4zUlNTE2VlZTE8PDyjfXh4OOrq6s7p/7//+79x/Pjx+LVf+7Xptnw+/5MDr1gRx44dixtvvPGccRUVFVFRUVFIaQBAkSpoZqS8vDyam5sjm81Ot+Xz+chms9HW1nZO/1tuuSWef/75OHr06PT267/+6/GLv/iLcfToUZdfAIDCn8CayWRi06ZN0dLSEuvXr4/+/v4YGxuL7u7uiIjo6uqK+vr66Ovri8rKyrjttttmjL/66qsjIs5pBwAuTwWHkc7Ozjh9+nRs3749crlcNDU1xaFDh6YXtZ44cSJKSz3YFQCYnwW9m6anpyd6enpm/d3TTz993rFf+MIXFnJIAGCZMoUBACQljAAASQkjAEBSwggAkJQwAgAktaC7aYC5NW47mLoEgKJiZgQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAklqRugAuD43bDqYuAYAlyswIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAktaDHwe/duzcefPDByOVysW7duvj7v//7WL9+/ax9v/KVr8TOnTvjpZdeirfeeituvvnm+NM//dP4yEc+ckGFQzGb7fH4x3dtKLgPwHJQ8MzIgQMHIpPJxI4dO+LIkSOxbt266OjoiFOnTs3a/5prrolPfvKTMTg4GP/93/8d3d3d0d3dHU8++eQFFw8AFL+Cw8ju3btj8+bN0d3dHWvXro2BgYFYuXJl7N+/f9b+9913X/zmb/5m3HrrrXHjjTfG1q1b4/bbb49nn332gosHAIpfQWFkYmIihoaGor29/e0dlJZGe3t7DA4OvuP4qampyGazcezYsXj/+99feLUAwLJT0JqRM2fOxOTkZNTW1s5or62tjRdeeGHOcSMjI1FfXx/j4+NRVlYWn/vc5+IDH/jAnP3Hx8djfHx8+ufR0dFCygQAisiCFrAWatWqVXH06NF44403IpvNRiaTiRtuuCHuu+++Wfv39fXFAw88cClKAwASKyiM1NTURFlZWQwPD89oHx4ejrq6ujnHlZaWxk033RQREU1NTfH9738/+vr65gwjvb29kclkpn8eHR2NhoaGQkoFAIpEQWGkvLw8mpubI5vNxsaNGyMiIp/PRzabjZ6ennnvJ5/Pz7gMc7aKioqoqKgopDSWmNluSwWA2RR8mSaTycSmTZuipaUl1q9fH/39/TE2Nhbd3d0REdHV1RX19fXR19cXET+55NLS0hI33nhjjI+PxxNPPBH/9E//FI888sjingkAUJQKDiOdnZ1x+vTp2L59e+RyuWhqaopDhw5NL2o9ceJElJa+fZPO2NhY/PEf/3H88Ic/jCuvvDJuueWW+NKXvhSdnZ2LdxYAQNEqmZqamkpdxDsZHR2N6urqGBkZiaqqqtTlXPbOvgQz21NBXaYpnCewAsvNfP9+ezcNAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACR1Sd5NAyyO+dxWDVBszIwAAEkJIwBAUsIIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAkJYwAAEkJIwBAUsIIAJCUMAIAJCWMAABJrUhdALBwjdsOntN2fNeGBJUALJyZEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApIQRACApYQQASMrj4GGJmO3R7gCXAzMjAEBSwggAkJQwAgAkJYwAAEkJIwBAUsIIAJCUW3u5YG5JBeBCmBkBAJISRgCApIQRACApYQQASEoYAQCSWlAY2bt3bzQ2NkZlZWW0trbG4cOH5+y7b9++eN/73herV6+O1atXR3t7+3n7AwCXl4LDyIEDByKTycSOHTviyJEjsW7duujo6IhTp07N2v/pp5+OD33oQ/Gtb30rBgcHo6GhIT74wQ/GK6+8csHFAwDFr2RqamqqkAGtra1x1113xZ49eyIiIp/PR0NDQ3z0ox+Nbdu2veP4ycnJWL16dezZsye6urrmdczR0dGorq6OkZGRqKqqKqRcLgLPFVnaju/akLoEgIiY/9/vgmZGJiYmYmhoKNrb29/eQWlptLe3x+Dg4Lz28eabb8Zbb70V11xzzZx9xsfHY3R0dMYGACxPBT2B9cyZMzE5ORm1tbUz2mtra+OFF16Y1z4+8YlPxPXXXz8j0Jytr68vHnjggUJKA+Yw20yW2RNgKbmkd9Ps2rUrHn/88fjqV78alZWVc/br7e2NkZGR6e3kyZOXsEoA4FIqaGakpqYmysrKYnh4eEb78PBw1NXVnXfsQw89FLt27Yp//dd/jdtvv/28fSsqKqKioqKQ0gCAIlXQzEh5eXk0NzdHNpudbsvn85HNZqOtrW3OcX/7t38bn/70p+PQoUPR0tKy8GoBgGWn4Lf2ZjKZ2LRpU7S0tMT69eujv78/xsbGoru7OyIiurq6or6+Pvr6+iIi4m/+5m9i+/bt8dhjj0VjY2PkcrmIiLjqqqviqquuWsRTAQCKUcFhpLOzM06fPh3bt2+PXC4XTU1NcejQoelFrSdOnIjS0rcnXB555JGYmJiI3/7t356xnx07dsRf//VfX1j1AEDRKziMRET09PRET0/PrL97+umnZ/x8/PjxhRwCALhMeDcNAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAkJYwAAEkt6N00wNLVuO1gwX2O79pwscoBeEdmRgCApIQRACApYQQASEoYAQCSEkYAgKSEEQAgKWEEAEhKGAEAkhJGAICkhBEAIClhBABIShgBAJISRgCApLy1F5iX+bwN2Nt/gYUwMwIAJCWMAABJuUzDec1nah4ALoSZEQAgKWEEAEhKGAEAkrJmBLA2CEjKzAgAkJQwAgAkJYwAAEkJIwBAUsIIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJS39gKX1GxvCD6+a0OCSoClYkEzI3v37o3GxsaorKyM1tbWOHz48Jx9v/vd78Zv/dZvRWNjY5SUlER/f/9CawUAlqGCw8iBAwcik8nEjh074siRI7Fu3bro6OiIU6dOzdr/zTffjBtuuCF27doVdXV1F1wwALC8FBxGdu/eHZs3b47u7u5Yu3ZtDAwMxMqVK2P//v2z9r/rrrviwQcfjN/7vd+LioqKCy4YAFheCgojExMTMTQ0FO3t7W/voLQ02tvbY3BwcNGKGh8fj9HR0RkbALA8FRRGzpw5E5OTk1FbWzujvba2NnK53KIV1dfXF9XV1dNbQ0PDou0bAFhaluStvb29vTEyMjK9nTx5MnVJAMBFUtCtvTU1NVFWVhbDw8Mz2oeHhxd1cWpFRYX1JQBwmShoZqS8vDyam5sjm81Ot+Xz+chms9HW1rboxQEAy1/BDz3LZDKxadOmaGlpifXr10d/f3+MjY1Fd3d3RER0dXVFfX199PX1RcRPFr1+73vfm/7vV155JY4ePRpXXXVV3HTTTYt4KgBAMSo4jHR2dsbp06dj+/btkcvloqmpKQ4dOjS9qPXEiRNRWvr2hMurr74ad9xxx/TPDz30UDz00ENx7733xtNPP33hZwAUvdmeyno2T2mF5WtBj4Pv6emJnp6eWX93dsBobGyMqamphRwGALgMLMm7aQCAy4cwAgAk5a29wEU1n/UgwOXNzAgAkJQwAgAkJYwAAEmVTBXBfbejo6NRXV0dIyMjUVVVlbqcZc31fYqF547A0jffv99mRgCApIQRACApt/YCy9Zslx1d3oGlx8wIAJCUMAIAJCWMAABJCSMAQFLCCACQlDACACTl1t7LyNm3ObrFkWLmtl1YPsyMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSbu29jM12ayQUM/9PQ3EyMwIAJCWMAABJCSMAQFLWjACXFa9FgKXHzAgAkJQwAgAk5TINwCLwFmFYODMjAEBSwggAkJQwAgAkZc1IYhfrNkOPxYb5mc9aD+tB4OIyMwIAJCWMAABJuUxThFyCgfRS/jucz7FdRqKYmBkBAJISRgCApIQRACApa0YAmOatxqRgZgQASEoYAQCScpmmCLiVFy6txfo3t5D9zHZZZCH7uZRPjS3WJ9S6JLV0mBkBAJISRgCApBYURvbu3RuNjY1RWVkZra2tcfjw4fP2/+d//ue45ZZborKyMt773vfGE088saBiAYDlp+A1IwcOHIhMJhMDAwPR2toa/f390dHREceOHYs1a9ac0/+5556LD33oQ9HX1xe/+qu/Go899lhs3Lgxjhw5ErfddtuinMSFuJjXOhdyPdL6ELi8XcrvgIUeaz7jLuV6jMX6Hl8uj9lfrLVKl1LBMyO7d++OzZs3R3d3d6xduzYGBgZi5cqVsX///ln7/93f/V388i//cvz5n/953HrrrfHpT3867rzzztizZ88FFw8AFL+CZkYmJiZiaGgoent7p9tKS0ujvb09BgcHZx0zODgYmUxmRltHR0d87Wtfm/M44+PjMT4+Pv3zyMhIRESMjo4WUu685MffPKdtsY5z9r5n2+9sxwe4GM7+DprP989ifW9djO/vn1ro9/hSO4/FspTO66f7nZqaOm+/gsLImTNnYnJyMmpra2e019bWxgsvvDDrmFwuN2v/XC4353H6+vrigQceOKe9oaGhkHIXrLq/uPYLMB8L+Q5arO+tS/3953u8MBf7vF5//fWorq6e8/dL8jkjvb29M2ZT8vl8vPbaa3HttddGSUlJwsouD6Ojo9HQ0BAnT56Mqqqq1OUwDz6z4uMzK04+t8JMTU3F66+/Htdff/15+xUURmpqaqKsrCyGh4dntA8PD0ddXd2sY+rq6grqHxFRUVERFRUVM9quvvrqQkplEVRVVfnHVmR8ZsXHZ1acfG7zd74ZkZ8qaAFreXl5NDc3RzabnW7L5/ORzWajra1t1jFtbW0z+kdEPPXUU3P2BwAuLwVfpslkMrFp06ZoaWmJ9evXR39/f4yNjUV3d3dERHR1dUV9fX309fVFRMTWrVvj3nvvjYcffjg2bNgQjz/+eHz729+ORx99dHHPBAAoSgWHkc7Ozjh9+nRs3749crlcNDU1xaFDh6YXqZ44cSJKS9+ecLn77rvjsccei7/8y7+Mv/iLv4ibb745vva1ry2JZ4wwu4qKitixY8c5l8pYunxmxcdnVpx8bhdHydQ73W8DAHAReTcNAJCUMAIAJCWMAABJCSMAQFLCCOfYu3dvNDY2RmVlZbS2tsbhw4dTl8Qc+vr64q677opVq1bFmjVrYuPGjXHs2LHUZVGAXbt2RUlJSXzsYx9LXQrn8corr8Tv//7vx7XXXhtXXnllvPe9741vf/vbqctaNoQRZjhw4EBkMpnYsWNHHDlyJNatWxcdHR1x6tSp1KUxi2eeeSa2bNkS//mf/xlPPfVUvPXWW/HBD34wxsbGUpfGPPzXf/1XfP7zn4/bb789dSmcx49+9KO455574oorroh/+Zd/ie9973vx8MMPx+rVq1OXtmy4tZcZWltb46677oo9e/ZExE+esNvQ0BAf/ehHY9u2bYmr452cPn061qxZE88880y8//3vT10O5/HGG2/EnXfeGZ/73OfiM5/5TDQ1NUV/f3/qspjFtm3b4j/+4z/i3//931OXsmyZGWHaxMREDA0NRXt7+3RbaWlptLe3x+DgYMLKmK+RkZGIiLjmmmsSV8I72bJlS2zYsGHGvzeWpm984xvR0tISv/M7vxNr1qyJO+64I/bt25e6rGVFGGHamTNnYnJycvppuj9VW1sbuVwuUVXMVz6fj4997GNxzz33eMLxEvf444/HkSNHpl+bwdL2gx/8IB555JG4+eab48knn4z7778//uRP/iS++MUvpi5t2Sj4cfDA0rRly5b4zne+E88++2zqUjiPkydPxtatW+Opp56KysrK1OUwD/l8PlpaWmLnzp0REXHHHXfEd77znRgYGIhNmzYlrm55MDPCtJqamigrK4vh4eEZ7cPDw1FXV5eoKuajp6cnvvnNb8a3vvWteNe73pW6HM5jaGgoTp06FXfeeWesWLEiVqxYEc8880x89rOfjRUrVsTk5GTqEjnLddddF2vXrp3Rduutt8aJEycSVbT8CCNMKy8vj+bm5shms9Nt+Xw+stlstLW1JayMuUxNTUVPT0989atfjX/7t3+L97znPalL4h380i/9Ujz//PNx9OjR6a2lpSU+/OEPx9GjR6OsrCx1iZzlnnvuOeeW+RdffDHe/e53J6po+XGZhhkymUxs2rQpWlpaYv369dHf3x9jY2PR3d2dujRmsWXLlnjsscfi61//eqxatWp6bU91dXVceeWViatjNqtWrTpnTc/P/MzPxLXXXmutzxL18Y9/PO6+++7YuXNn/O7v/m4cPnw4Hn300Xj00UdTl7ZsuLWXc+zZsycefPDByOVy0dTUFJ/97GejtbU1dVnMoqSkZNb2f/zHf4w/+IM/uLTFsGD33XefW3uXuG9+85vR29sb//M//xPvec97IpPJxObNm1OXtWwIIwBAUtaMAABJCSMAQFLCCACQlDACACQljAAASQkjAEBSwggAkJQwAgAkJYwAAEkJIwBAUsIIAJCUMAIAJPX/APDGvGOsdpsxAAAAAElFTkSuQmCC\n" 116 | }, 117 | "metadata": {}, 118 | "output_type": "display_data" 119 | } 120 | ], 121 | "source": [ 122 | "_ = plt.hist(true_trajectories[:, -1], density=True, bins=100)" 123 | ], 124 | "metadata": { 125 | "collapsed": false 126 | } 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "source": [ 131 | "We solve the Kolmogorov forward equation now using finite difference." 132 | ], 133 | "metadata": { 134 | "collapsed": false 135 | } 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 7, 140 | "id": "b14484d9", 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "# Uniformly partition the spatial variable on [-2, 6] with 500 points\n", 145 | "N = 500\n", 146 | "xs = np.linspace(-2, 6, N)\n", 147 | "dx = xs[1] - xs[0]" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": 8, 153 | "outputs": [], 154 | "source": [ 155 | "def fpk_operator(ps):\n", 156 | " \"\"\"Kolmogorov forward equation operator (unidimensional)\n", 157 | " \"\"\"\n", 158 | " derivative_drift = a\n", 159 | " gamma = dispersion(xs) ** 2\n", 160 | " derivative_gamma = 2 * b ** 2 * xs\n", 161 | " second_derivative_gamma = 2 * b ** 2\n", 162 | " derivative_p = np.gradient(ps, dx)\n", 163 | " second_derivative_p = np.gradient(np.gradient(ps, dx), dx)\n", 164 | "\n", 165 | " part1 = -(derivative_drift * ps + drift(xs) * derivative_p)\n", 166 | " return part1 + 0.5 * (\n", 167 | " second_derivative_gamma * ps + 2 * derivative_gamma * derivative_p + gamma * second_derivative_p)\n", 168 | "\n", 169 | "\n", 170 | "def euler(ps):\n", 171 | " r\"\"\"Euler's method for solving the ODE.\n", 172 | "\n", 173 | " p_k \\approx p_{k-1} + fpk_operator(p_{k-1}) * dt\n", 174 | " \"\"\"\n", 175 | " return ps + fpk_operator(ps) * dt\n", 176 | "\n", 177 | "\n", 178 | "def solve_fpk(p0):\n", 179 | " \"\"\"Solve the Kolmogorov forward equation.\n", 180 | " \"\"\"\n", 181 | " pdfs = np.zeros((T, N))\n", 182 | " ps = p0.copy()\n", 183 | " for k in range(T):\n", 184 | " ps = euler(ps)\n", 185 | " pdfs[k, :] = ps\n", 186 | " return pdfs" 187 | ], 188 | "metadata": { 189 | "collapsed": false 190 | } 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": 9, 195 | "id": "a0ffa0b9", 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "# Initial condition\n", 200 | "p0 = scipy.stats.norm.pdf(xs, m0, np.sqrt(var0))\n", 201 | "\n", 202 | "# Solve\n", 203 | "pdfs = solve_fpk(p0)" 204 | ] 205 | }, 206 | { 207 | "cell_type": "markdown", 208 | "source": [ 209 | "Check if the solution matches the sample histogram." 210 | ], 211 | "metadata": { 212 | "collapsed": false 213 | } 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": 10, 218 | "id": "beabceda", 219 | "metadata": {}, 220 | "outputs": [ 221 | { 222 | "data": { 223 | "text/plain": "
", 224 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAABAUElEQVR4nO3deXxU5aH/8c85k2RCIAlLSMISCSAKiBDWFLxuNRavtNXe1lKrxeZ6aWvF6k3bK1iF1tsKrVTpT6koLWq1Frpo7a0VtbFUxVSEiCKyyBL2bCwJBMgkc87vj5MJRAgkIckzy/f9es3rzEzOzHzTWPLN8zznHMt1XRcRERERQ2zTAURERCS2qYyIiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGqYyIiIiIUSojIiIiYpTKiIiIiBgVZzpASziOw969e0lOTsayLNNxREREpAVc1+Xw4cP07dsX225+/CMiysjevXvJysoyHUNERETaYNeuXfTv37/Zr0dEGUlOTga8byYlJcVwGhEREWmJ6upqsrKyGn+PNyciykhoaiYlJUVlREREJMKcbYmFFrCKiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGqYyIiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGqYyIiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGqYyIiIiIURFx1V6RcJU986Uzfr1k3pROSiIiErk0MiIiIiJGqYyIiIiIUSojIiIiYpTKiEhrOQ4cqYD6WtNJRESighawirSUE4S3H4GihVBTDnYcv4wfwwP1X2W3m246nYhIxNLIiEhL1Adg6Vfh73O8IgLg1HOtbxUvJ8xior3ebD4RkQimMiLSEn/7LmxeDnFd4POPwL3l8K2VrHYuINk6xpL4BxlpbTWdUkQkIqmMiJzNhr9C8W8AC6Y+A2OmQZwfMkdwU+Ae3gheTBcrwC8TfkFXjplOKyIScVRGRM6k7hgsn+Xdv+ROGHJ1ky/XksC36+5kl9Ob/lYls+KeMxBSRCSyqYyInMmap6FqJ6T0h8vvPu0uR0jif+q/AcDNcYVMsDZ0ZkIRkYinMiLSnGAdrFzg3b/se5CQ1OyuRc5FPFf/aQBmxv8OcDs+n4hIlFAZEWnO5uVweB907Q05N51194frv8RR188Yewt5dnEnBBQRiQ4qIyLNWfO0t835KsQlnHX3CrrzZHAyAN+N+z0WTkemExGJGiojIqdzaBds+bt3f8wtLX7Z4/Wf5bDbhWH2Li6z13VQOBGR6KIyInI6a58DXMi+FHoNbvHLqunG74NXAPCfvpc7JpuISJRRGRE5nQ3/521H3djqlz4ZnEzQtbjc9wGUb2znYCIi0UdlROSTDu6AsnVg2XDBNa1++W43ndeccd6DVU+0czgRkeijMiLySZsaplfOmwhde7XpLX4TbDg52ro/eidOExGRZqmMiHzSppe87YXXtvktipzh7HJ6Q23ViSkfERE5rTjTAUTCyrGDULLSuz/0WrJnvtSmt3Gx+UPwcgrsP8J7z8DIL7djSBGR6KKREZGTlbwFbhDSLoCeg87prf4YvAywYPsb3joUERE5LZURkZOVvOVtB152zm+1lzQYeKn3YP0L5/x+IiLRSmVE5GShMpL9b+3zfhf9h7dd/3z7vJ+ISBRSGREJqdkPZR969we0UxkZ9nmwfLDvfdi/tX3eU0QkyqiMiITsaFi42nsodOvdPu/ZtRcMuty7r6kaEZHTUhkRCWmcorm0fd/3oi942/V/bt/3FRGJEiojIiE73va22Ze07/sO/SzYcd5ZXSs/bt/3FhGJAiojIgCBGij/yLufldu+753UEwZd6d3/UAtZRUQ+SWVEBLwFpm4QkvtASt/2f//h13nbTW07iZqISDRTGREB2LPG2/Yb2zHvf8FkwPJKT/XejvkMEZEIpTIiAieVkTEd8/7d0qF/w5V8Ny/vmM8QEYlQKiMi0PEjIwAXXONtN6mMiIicTBfKEzlSAYd2evf7jm7Xtz75QnsXWl15xQ/HN79OzsznOY4fgJJ5U9r1M0VEIo3KiMSsUFG40n6PJxNgi9OXvB++1WGft8nNYrebRn+rkn+zP+TvTgeOwoiIRJA2TdMsXLiQ7OxsEhMTyc3NZdWqVc3u+9RTT2FZVpNbYmJimwOLtLeLre0AfOCe21V6z87i70FvTcpVdnEHf5aISORodRlZtmwZBQUFzJkzh+LiYkaNGsXkyZMpLy9v9jUpKSns27ev8bZjhy6nLuFjqO1N0XzkDOjwzyp0GsqI7z0snA7/PBGRSNDqMvLQQw8xffp08vPzGT58OIsWLSIpKYklS5Y0+xrLssjMzGy8ZWRknFNokfY0zPLK8Qb3vA7/rHecYdS4ftKtQwyzdnb454mIRIJWlZFAIMCaNWvIy8s78Qa2TV5eHkVFRc2+7siRIwwYMICsrCyuu+461q9ff8bPqa2tpbq6uslNpCMkcZwBljeqt9Hp+DISIJ4iZzgAl9rrOvzzREQiQavKSGVlJcFg8JSRjYyMDEpLS0/7mgsvvJAlS5bw4osv8uyzz+I4DpMmTWL37t3Nfs7cuXNJTU1tvGVlZbUmpkiLXWjtwrZcyt3u7Ce1Uz7zTWckAJfaH3TK54mIhLsOP8/IxIkTmTZtGjk5OVx++eU8//zz9O7dm8cff7zZ18yaNYuqqqrG265duzo6psSoYQ3rRTZ0wqhIyJvOxQCMtzeRSG2nfa6ISLhq1aG9aWlp+Hw+ysrKmjxfVlZGZmZmi94jPj6e0aNHs2XLlmb38fv9+P3+1kQTaZPOXC8Sss3t03iIb669sdM+V0QkXLVqZCQhIYGxY8dSWFjY+JzjOBQWFjJx4sQWvUcwGGTdunX06dOndUlFOsBQAyMjYPFm0Bsd0VSNiEgbpmkKCgpYvHgxTz/9NBs2bOC2226jpqaG/Px8AKZNm8asWbMa97///vt59dVX2bZtG8XFxdx8883s2LGD//qv/2q/70KkTVyGWt4U4Aa34w/rPVloqkaLWEVE2nAG1qlTp1JRUcHs2bMpLS0lJyeH5cuXNy5q3blzJ7Z9ouMcPHiQ6dOnU1paSo8ePRg7dixvv/02w4cPb7/vQqQN+rKfZOsYda6PbW7njtStdEYQdC0utHd7V/FN6dupny8iEk4s13Vd0yHOprq6mtTUVKqqqkhJSTEdR6LEtHse4DcJP+Vjpx9XBx7s9M9/IWE2o+0tcN1CGH1zp3++iEhHa+nvb121V2LW+dZeALa4ZkYl3miYqmHr60Y+X0QkXKiMSMw639oDwBa3n5HPDy1iZes/wNGp4UUkdqmMSMwabDeMjDhmRkbWuudz2O0Cxw5A6ftGMoiIhAOVEYlZQyzvLMCmRkbqieNfzjDvwfY3jGQQEQkHKiMSm2oq6WkdAej0I2lO9q+G69Sw/U1jGURETFMZkdhUsQmA3W4ax0g0FiN00Tx2FkGwzlgOERGTWn2eEZGoUOmVkY8dM1M0IRvc8zjodqNH4AhfuHch77lDTtmnZN4UA8lERDqPRkYkNlVsBsytFwlxsXmnYd3IRPsjo1lERExRGZHYtN+7UON2g+tFQkJTNZ9SGRGRGKUyIrHpwDYAtrstu9p0RwqVkXH2ZuKpN5xGRKTzqYxI7AnWwyHvar0ljvkystntz343mSSrlpHWVtNxREQ6ncqIxJ7q3eDUUevGU0oP02kAq/F8I1o3IiKxSGVEYk/DFM0ONx03TP4vUORcBKiMiEhsCo9/iUU6U2MZMT9FExJaNzLW3kwCOt+IiMQWlRGJPQe2A1DiZhgOcsJWty/lbncSrTpGW1tMxxER6VQqIxJ7GkdGwqeMNFk34ltvOIuISOdSGZHY01BGSsJomgZOTNVo3YiIxBqVEYktjhOW0zRwoozkWFvwEzCcRkSk86iMSGw5vBeCtWDHs8/tZTpNEyVuJvvcnvitesbam03HERHpNCojElsapmjoMYAgPrNZTmHxjjMUgFx7o+EsIiKdR2VEYkuojPQcZDZHM1Y1LGIdb6mMiEjsUBmR2BLmZSQ0MjLG/ljXqRGRmKEyIrElzMvIVrcv+91kEq06Lra2mY4jItIpVEYktjQcSROuZQQs3m0YHZmgdSMiEiNURiR2uG7Yj4wArFIZEZEYozIisaOmAuqOgmVDapbpNM0KrRsZZ2/CxjGcRkSk46mMSOw4tNPbJveFuASzWc5ggzuAw24XUqxjDLN2mo4jItLhVEYkdhza4W27h++oCICDzRrnAgDGa6pGRGKAyojEjkO7vG3388zmaAGtGxGRWKIyIrEjNE0TAWXknZPLiOsaTiMi0rFURiR2hMpIGC9eDVnnDuK4G0+aVQ37t5iOIyLSoVRGJHZURc40TYB41rrnew92rDQbRkSkg6mMSGxw3YiapoETUzXseNtsEBGRDqYyIrHh6H7vHCMAqf3NZmmhVSojIhIjVEYkNjSeY6QPxPnNZmmhYmcIda7Pm146pPONiEj0UhmR2BBhUzQAx0hkvZvtPdhRZDSLiEhHUhmR2BBBR9Kc7MS6ES1iFZHopTIisSECR0ZA60ZEJDaojEhsiKDDek/2rnMhYMH+j+FIhek4IiIdIs50AJFO0TgyElnTNNV0Y4PTn2H2Lr71wKMsdyacsk/JvCkGkomItB+NjEj0a3KOkQFms7RBaKom195gOImISMdQGZHod+wgBI549yPkHCMnW+UMA3TRPBGJXiojEv1CoyJd0yG+i9ksbbDKuRCAYdZOkjlqOI2ISPtTGZHoF6FH0oRU0IPtTga25TLW3mQ6johIu1MZkejXeCRNZC1ePdmJqRqVERGJPiojEv2qdnvbCFwvEvKu603VaN2IiEQjlRGJftV7vG1K5JaR0JlYR1pb8RMwnEZEpH2pjEj0qwqVkb5mc5yDXW46pW4PEqwgo+0tpuOIiLSrNpWRhQsXkp2dTWJiIrm5uaxatapFr1u6dCmWZXH99de35WNF2iY0MpLaz2yOc2I1nm9kgqWpGhGJLq0+A+uyZcsoKChg0aJF5ObmsmDBAiZPnsymTZtIT09v9nUlJSV873vf49JLLz2nwCItlT3zJeKoZ7O/FNuC8Y9spIJS07HabJUzlM/7ihhvb4Sg6TQiIu2n1SMjDz30ENOnTyc/P5/hw4ezaNEikpKSWLJkSbOvCQaD3HTTTfzoRz9i0KBB5xRYpDXSOYRtuQRcH5WkmI5zTkIjI2Ptj4mj3nAaEZH206oyEggEWLNmDXl5eSfewLbJy8ujqKio2dfdf//9pKenc+utt7boc2pra6murm5yE2mLPtZ+AErdnrgRvkTqY7cfB91uJFm1XGSVmI4jItJuWvWvc2VlJcFgkIyMjCbPZ2RkUFp6+uHvt956i1//+tcsXry4xZ8zd+5cUlNTG29ZWZF7fggxq29DGdlHL8NJzp2LzWpHh/iKSPTp0D8VDx8+zNe+9jUWL15MWlpai183a9YsqqqqGm+7du3qwJQSzUIjI/vcnoaTtI/QIb46+ZmIRJNWLWBNS0vD5/NRVlbW5PmysjIyMzNP2X/r1q2UlJTwuc99rvE5x3G8D46LY9OmTQwePPiU1/n9fvx+f2uiiZxWH+sAAPvcyB8ZAXi3YWRkvL0RCyfip55ERKCVIyMJCQmMHTuWwsLCxuccx6GwsJCJEyeesv/QoUNZt24da9eubbx9/vOf58orr2Tt2rWafpEOFyoje6OkjKx3s6lx/XS3arjA2m06johIu2j1ob0FBQXccsstjBs3jgkTJrBgwQJqamrIz88HYNq0afTr14+5c+eSmJjIiBEjmry+e/fuAKc8L9IRTkzTREcZqSeOYmcIl/o+ZLy9iU3ByLz4n4jIyVpdRqZOnUpFRQWzZ8+mtLSUnJwcli9f3riodefOndi2ho4lPPSNsjUj4B3ie6nvQ3LtDTwbvNp0HBGRc9bqMgIwY8YMZsyYcdqvrVix4oyvfeqpp9rykSKtlkAdva0qIHpGRuDEFXzH25sA12wYEZF2oCEMiVrp1kEAjrvxHCDZcJr2s9YdTMD1kWkd5Dyr3HQcEZFzpjIiUasvJ0/RWGbDtKNaEnjf9Y5Cy7U3GE4jInLuVEYkakXb4tWTvdtwvpHxls43IiKRT2VEolbf0DlGiJ7FqyGNV/DVmVhFJAqojEjUyozikZE1zgU4rkW2XQbV+0zHERE5JyojErX6RtnZV092mCQ+cgd4D3a+bTaMiMg5UhmRqBVaMxItZ1/9pNBUDTuav2K2iEgkUBmRqJXZMDJSGkUnPDvZiTKikRERiWwqIxKd6gOkWdUAlLo9DIfpGKGL5lG+Ho4eMBtGROQcqIxIdDpSCkCtG8fBKDrh2cn2k8pWp4/3YNc7ZsOIiJwDlRGJToe9MlLu9iCaTnj2Se80TtWsNBtEROQcqIxIdKreC0AZ0TlFE/KuFrGKSBRQGZHo1DAyEq3rRUIaF7HuWwu1R4xmERFpK5URiU6HvZGR8igvI3voDSn9wamH3e+ajiMi0iYqIxKdYmRkBIABk7ztTk3ViEhkUhmR6BRaMxJLZUTnGxGRCKUyItHpsHe9lrIovEjeKUJlZPe7UB8wm0VEpA3iTAcQ6RAxNE2T/fOPWeNPplf9Yf7jvoUUuxecsk/JvCkGkomItIxGRiT6HK+GgHdkSUxM02A1HuI7wd5oOIuISOupjEj0aRgVqXaTOEai4TCdI3SIb669wXASEZHWUxmR6HM4hhavNnjHGQbAOHszPoKG04iItI7KiESfGFovErLBPY8qN4lk6xgXWSWm44iItIrKiESfhsN6y6P8VPAnc7BZ1TA6MtH+yHAaEZHWURmR6NNwWG8sjYwAFDnDAfiUyoiIRBiVEYk+oXOMxFgZ+VfDyMh4exNx1BtOIyLSciojEn2qQ2UkBk54dpIN7nkccrvSzTrOCK0bEZEIojIi0adhAWusjYy42I1H1WjdiIhEEpURiS6OA0di72iaEK0bEZFIpDIi0eVoJTj1gEUF3U2n6XT/aigj47RuREQiiMqIRJeGw3rplk4Qn9ksBmxy+3PA7UZXq5aR1jbTcUREWkRlRKJLw3oRkvuYzWHIyetGPqVTw4tIhFAZkejScCr4WC0jcGKqRutGRCRSqIxIdGk4rJeU2C0jRY3rRjYTr3UjIhIBVEYkujSc8CyWR0Y+dvux300myarlYq0bEZEIoDIi0UVlBBe78WysOt+IiEQClRGJLjG+gDVE60ZEJJKojEh0CR3aG8NrRqDpupEE6gynERE5M5URiR71tXDsgHc/xkdGtrj9qHBT6GIFGGltNR1HROSMVEYkeoTWi/j80CX2TgXflMU7DaMjWjciIuFOZUSix8mH9VqW2SxhIDRVM0llRETCnMqIRA8dSdPESuciAMbYmyFQYziNiEjzVEYkeqiMNFHiZrLbTcNv1cPOItNxRESapTIi0UNl5BMsVgZHeHe3/dNsFBGRM1AZkeihU8GfYqUTKiMrjOYQETkTlRGJHjrh2Sneblg3QukHULPfbBgRkWaojEj00BV7T1FJKhucLO9ByRtmw4iINENlRKKD6540MpJpNkuY0VSNiIQ7lRGJDseroO6od18jI02ojIhIuGtTGVm4cCHZ2dkkJiaSm5vLqlWrmt33+eefZ9y4cXTv3p2uXbuSk5PDM8880+bAIqcVOpImMRUSksxmCTOrnKFgx8HBEu8mIhJmWl1Gli1bRkFBAXPmzKG4uJhRo0YxefJkysvLT7t/z549+cEPfkBRUREffPAB+fn55Ofn88orr5xzeJFGjYf19jWbIwzV0AX6j/ce6BBfEQlDrS4jDz30ENOnTyc/P5/hw4ezaNEikpKSWLJkyWn3v+KKK/jCF77AsGHDGDx4MHfeeScjR47krbfeOufwIo10WO+ZDbzc22qqRkTCUKvKSCAQYM2aNeTl5Z14A9smLy+PoqKzn+HRdV0KCwvZtGkTl112WevTijRHJzw7s0FXeNvtb4DjGI0iIvJJca3ZubKykmAwSEZGRpPnMzIy2LhxY7Ovq6qqol+/ftTW1uLz+fjlL3/J1Vdf3ez+tbW11NbWNj6urq5uTUyJRSojZ9Z/HCR0g6OVUL4eMi82nUhEpFGnHE2TnJzM2rVreffdd/nJT35CQUEBK1asaHb/uXPnkpqa2njLysrqjJgSyXRY75n54mHAJO++pmpEJMy0qoykpaXh8/koKytr8nxZWRmZmc3/ErBtm/PPP5+cnBy++93v8qUvfYm5c+c2u/+sWbOoqqpqvO3atas1MSUWVTec8CxFC1ibFZqq2fq60RgiIp/UqjKSkJDA2LFjKSwsbHzOcRwKCwuZOHFii9/HcZwm0zCf5Pf7SUlJaXITOSONjJzd4Ku87Y63oe6Y2SwiIidp1ZoRgIKCAm655RbGjRvHhAkTWLBgATU1NeTn5wMwbdo0+vXr1zjyMXfuXMaNG8fgwYOpra3lb3/7G8888wyPPfZY+34nErucIBxpGK3TmpHm9b4QUvpD9W7YsRLOzzv7a0REOkGry8jUqVOpqKhg9uzZlJaWkpOTw/LlyxsXte7cuRPbPjHgUlNTw7e//W12795Nly5dGDp0KM8++yxTp05tv+9CYltNBbhBsGzomm46TfiyLDj/01D8G9hSqDIiImHDcl3XNR3ibKqrq0lNTaWqqkpTNnKqve/BE1dAt0z43qbGp7NnvmQuU5gpmTfFu/PRi/D7aZB2Icxo/szJIiLtoaW/v3VtGol8OuFZyw28HCwfVG6CQ1oYLiLhQWVEIp/OMdJyXbqfODX81sIz7ioi0llURiTyNZYRHUnTIuc3HFWz5e9mc4iINFAZkcini+S1TqiMbPsnBOvMZhERQWVEooHOMdI6fXKgS0+orYbdq02nERFRGZEooAWsrWP7YPCnvftaNyIiYUBlRCKfFrC2ntaNiEgYURmRyFZ3HI4d8O6rjLRcaGRk71qoqTQaRUREZUQi25GG9SI+P3TpYTZLJEnOhIyLARe2/sN0GhGJcSojEtlOXrxqWWazRJrzG0ZHtrxmNoeIxDyVEYls1Xu9bYoO6221IZO97ceveRcbFBExRGVEIpsO6227rFxITPXW3OgQXxExqNVX7RUJK4cbRka0ePWMmrto4C/iL+I639uweTmcl9vJqUREPBoZkcjWODKiMtIWhcHR3p3Nr5gNIiIxTWVEIpvKyDn5pzOKoGtB+Xo4tNN0HBGJUSojEtkaF7CqjLRFFd1Y7V7oPdDoiIgYojUjErlcVyMj7eD14Ghy7Y384/+eIf/50x+VVDJvSienEpFYopERiVy11VBX493X0TRtVuh460Ym2R/RheOG04hILFIZkcgVGhXxp0JCV7NZItgWtx87nd74rTousdebjiMiMUjTNBKxvvrwn3kuAT4+1o2rmzl0VVrCotAZQ779Cp+2i/m7M9Z0IBGJMRoZkYiViXeBvFJX16Q5V683TNVc5XsPcM2GEZGYozIiESvDOgRAOSoj5+odZxg1rp8M6xAXWSWm44hIjFEZkYiVYXkjI2UaGTlnAeJ50xkJQJ5dbDiNiMQalRGJWBnWQUDTNO0ldFTN1b41hpOISKxRGZGIFSojGhlpH4XBMQRdixF2Cf2tCtNxRCSGqIxIxDpRRnoaThIdDpDCu+5QACbb7xpOIyKxRGVEIpPjkM4hQCMj7Wl5cDwAn/GtNpxERGKJyohEpqOVxFtBHNeiglTTaaLGq8FxAIy3NpFGleE0IhIrVEYkMh3eB8B+UqjXufvazV7SeN8ZhG255Gkhq4h0EpURiUzVXhnRFE37e6VhdOQarRsRkU6iMiKRqWFkRIf1tr9XHG/dyCT7Q5I5ajiNiMQClRGJTA0XyStXGWl3W91+bHH6kmAFudJ+z3QcEYkBKiMSmQ7vBTRN01Fecbypmsk+TdWISMdTGZHI1DAyUqbr0nSI5cEJAFxhv4+fgOE0IhLtVEYkMlV7IyP73F6Gg0Snde5A9ri96GrVcqm9znQcEYlyKiMSmar3ALBPZ1/tIFbjOUeu0VSNiHQwlRGJPIGjcCx0kTyVkY7ycsNUzWfs1VBfaziNiEQzlRGJPA1TNEfcRKpJMhwmeq12L6TM7U6KdRS2vm46johEMZURiTwNUzTeqIhlNksUc7D5WzDXe/Dh82bDiEhUUxmRyNO4eFVTNB3t/4ITvTub/gZ1x8yGEZGopTIikad6N6D1Ip3hPfd89ri9IHAEPn7NdBwRiVIqIxJ5GkZG9qLDejuai81LwU95D9a/YDaMiEQtlRGJPA1lRCMjneOvoTKyeTkEasyGEZGopDIikUfnGOlUH7iDoEc21B2Fza+YjiMiUSjOdACR5mTPfOm0zxf7t9PT0tlXO48FF30B3noY1j8PI/7DdCARiTIaGZGI4idAT+sIoJGRTjXii95286twvMpsFhGJOiojElEyrQMAHHX9VNPVcJoYkjECeg+DYC189KLpNCISZVRGJKL0aSgj+3TCs85lWTDyy979D35vNouIRB2VEYkofdgPaIrGiFAZKXkTDu0ym0VEokqbysjChQvJzs4mMTGR3NxcVq1a1ey+ixcv5tJLL6VHjx706NGDvLy8M+4vciahkZFSnWOk86X2h+xLvfvr/mA2i4hElVaXkWXLllFQUMCcOXMoLi5m1KhRTJ48mfLy8tPuv2LFCm688Ub+8Y9/UFRURFZWFp/5zGfYs2fPOYeX2BNaM7JXIyNmjJzqbT9YBq5rNouIRI1Wl5GHHnqI6dOnk5+fz/Dhw1m0aBFJSUksWbLktPv/9re/5dvf/jY5OTkMHTqUX/3qVziOQ2Fh4TmHl9jTODKiw3rNGP558PmhYiOUfmA6jYhEiVaVkUAgwJo1a8jLyzvxBrZNXl4eRUVFLXqPo0ePUldXR8+ezf9lW1tbS3V1dZObCEAfS2tGjEpMhQv/3buvhawi0k5aVUYqKysJBoNkZGQ0eT4jI4PS0tIWvcfdd99N3759mxSaT5o7dy6pqamNt6ysrNbElCiW2Xg0jUZGjBn1FW+77g8QrDebRUSiQqceTTNv3jyWLl3KCy+8QGJiYrP7zZo1i6qqqsbbrl1auS/eCc/SLG+UTCMjBg2+CpJ6wZEy2Pq66TQiEgVaVUbS0tLw+XyUlZU1eb6srIzMzMwzvnb+/PnMmzePV199lZEjR55xX7/fT0pKSpObSIZ1EIBjbgJVOuGZOXEJJxayvveM2SwiEhVaVUYSEhIYO3Zsk8WnocWoEydObPZ1P/vZz/jf//1fli9fzrhx49qeVmJaP6sSgL1uL3TCM8NG3+xtN70MNZVms4hIxGv1NE1BQQGLFy/m6aefZsOGDdx2223U1NSQn58PwLRp05g1a1bj/j/96U+57777WLJkCdnZ2ZSWllJaWsqRI0fa77uQmNDfqgBgt9vbcBIh4yLoOwacOu8wXxGRc9DqMjJ16lTmz5/P7NmzycnJYe3atSxfvrxxUevOnTvZt29f4/6PPfYYgUCAL33pS/Tp06fxNn/+/Pb7LiQmhEZG9rhphpMIcGJ0pPgZnXNERM5JXFteNGPGDGbMmHHar61YsaLJ45KSkrZ8hMgp+uGVkd0qI+Hh4i/BK/dAxQbYUwz9x5pOJCIRStemkYihkZEwk5gKw6/z7r/3G7NZRCSiqYxIxAitGVEZCSOjv+Zt1/0JAkfNZhGRiNWmaRqRzmbjNJ4Kfo8WsHa67JkvnfZ5C4cVCekMCJTD+hdg9E2dnExEooFGRiQipHOQeCtIneujjB6m40gDF5ulwU97D1b/2mwYEYlYKiMSEULrRUrdnjj6zzas/D54OdjxsGcN7F1rOo6IRCD9qy4RoXHxKlovEm72c9JCVo2OiEgbqIxIROhvhQ7r1XqRsDT+Vm+77o9wvMpsFhGJOCojEhF0WG+YO28i9B4GdUfh/aWm04hIhFEZkYhw4lTwKiNhybJOjI68+2udkVVEWkVlRCKCRkYiwMipEN8VKjdByVum04hIBFEZkQjgqoxEgsQUGPll7/6qx81mEZGIojIiYa8nh+liBQDY5/YynEbOKPeb3nbjS3CwxGgUEYkcKiMS9kKjImVudwLEG04jZ5Q+DAZdCa4DqxabTiMiEUJlRMKepmgizMTbvW3xb6D2sNksIhIRVEYk7PXTBfIiy+CroNcQqK2G935rOo2IRACVEQl7/RtHRnTCs4hg2/Cp27z77zwGTtBsHhEJeyojEvYGWGUA7HDTDSeRFhv1FUjs7i1i3bzcdBoRCXMqIxL2zrPKAdjhZhhOIi2W0BXG5Xv3337UbBYRCXsqIxLWLByyGtaM7NTISGSZ8A3var4734ad75hOIyJhTGVEwlomB/FbddS5Pp1jJNKk9PWmawDeethsFhEJayojEtZCUzR73DSC+AynkVa75C7Ags0vQ9lHptOISJhSGZGwdp7tLV7VFE2ESjsfhn/eu79ygdEoIhK+4kwHEDkTLV6NDNkzX2r2axdZE3jJ/yKs+yNceQ/0yO68YCISETQyImEtdFivRkYi13p3IG8ELwY3qCNrROS0VEYkrJ3XWEY0MhLJHgs2TNW89wwcLjUbRkTCjsqIhLXQNI1GRiJbkTMcsnKh/riOrBGRU6iMSNhK5ig9rSOAykjks7z1IgCrn4TqvWbjiEhYURmRsDXA8obzK9wUauhiOI2cs4GXw3mTIFgLbz5kOo2IhBGVEQlbgxrKyHa3j+Ek0i6sk0ZHip+Gqt1m84hI2NChvRK2Blr7ANjmqIxEg9Dhv7+LH85EPuLZB7/DvfW3Nn69ZN4UU9FExDCNjEjYGmR7ZUQjI9Hl4fovAvBl3wr6N1x3SERim8qIhK3QyMh2N9NwEmlPq9xhvBG8mAQrSEHcH0zHEZEwoDIi4cl1GdRQRra6fQ2Hkfb2s/qpAFxvr2S4VWI2jIgYpzIi4elIGd2s4wRdi106rDfqfOgO4sXgJGzLZWbc70zHERHDVEYkPO3fAsAuN50A8YbDSEd4sP7LBFwfl/nWcYm9znQcETFIZUTCU+XHgNaLRLPdbjrPBq8GYFbc78BxDCcSEVNURiQ8NYyM6Eia6PZo/fUcdrswwi6BdVrMKhKrVEYkPDWUkW0qI1HtACk8Vt9wEb2/z4HaI2YDiYgRKiMSnio2ATqSJhb8Ovjv7HDS4fA+ePPnpuOIiAEqIxJ+AkfhYAkAHzv9zWaRDldLAj+uv9l7UPQo7N9qNpCIdDqVEQk/lZsBlwNuNypJMZ1GOsFrzlgY/GkIBuDVe03HEZFOpjIi4adiIwAfu/0By2wW6SQWXDMP7DjY9Df4+O+mA4lIJ1IZkfBTvgGAzZqiiS29L4QJ3/Tuv/x9qDtmNo+IdBqVEQk/DSMjm12VkZhzxUxI7gsHtsEb802nEZFOojIi4adhZORjlZHYk5gC1/7Mu79yQeN/CyIS3eJMBxBpIlADh3YAmqaJNdkzX2q4Z7E4fixXs4bVj07jhsBsXGxK5k0xmk9EOo5GRiS8NJxfhKQ0DuhImhhlMbvu6xxxExlnb+ZG3z9MBxKRDqYyIuElNCyfPsxsDjFqH734ef0NAMyMe44+7DecSEQ6UpvKyMKFC8nOziYxMZHc3FxWrVrV7L7r16/ni1/8ItnZ2ViWxYIFC9qaVWJBacPVWzNGmM0hxj0dnMwaZwgp1jF+Gv8EuK7pSCLSQVpdRpYtW0ZBQQFz5syhuLiYUaNGMXnyZMrLy0+7/9GjRxk0aBDz5s0jM1NXYJWzKP3A2/YZaTaHGOdg8726b3HMTeAy3zpYvcR0JBHpIK0uIw899BDTp08nPz+f4cOHs2jRIpKSkliy5PT/UIwfP54HH3yQr3zlK/j9/nMOLFHMcU6MjGSqjIh31eaf1U/1Hrx6HxzYbjaQiHSIVpWRQCDAmjVryMvLO/EGtk1eXh5FRUXtFqq2tpbq6uomN4kBh3ZAbTX4ErwTYIkATwUn8y9nGNTVwJ+/DU7QdCQRaWetKiOVlZUEg0EyMjKaPJ+RkUFpaWm7hZo7dy6pqamNt6ysrHZ7bwljoSma9GHgizebRcKGi8336r4JCd1g59vw1kOmI4lIOwvLo2lmzZpFVVVV423Xrl2mI0ln2NdQRjRFI5+w202Hax/0HvxjLux8x2wgEWlXrTrpWVpaGj6fj7KysibPl5WVteviVL/fr/Ulsahx8eooszkkLGUvTeHh+Ev4gm8lu3/9Va6tfYBqujXZRydGE4lMrRoZSUhIYOzYsRQWFjY+5zgOhYWFTJw4sd3DSYxpXLx6sdkcEqYs7q37T0qcDPpblcyL/xWgw31FokGrp2kKCgpYvHgxTz/9NBs2bOC2226jpqaG/Px8AKZNm8asWbMa9w8EAqxdu5a1a9cSCATYs2cPa9euZcuWLe33XUjkq94Lh/eBZescI9KsGrpwR90dBFwf1/pWcYvvVdORRKQdtPraNFOnTqWiooLZs2dTWlpKTk4Oy5cvb1zUunPnTmz7RMfZu3cvo0ePbnw8f/585s+fz+WXX86KFSvO/TuQ6LB7tbdNHw7+bmfeV2LaOncQ8+q/yuz4Z7g37lnWOwNY7Q41HUtEzoHluuF/WsPq6mpSU1OpqqoiJUXXK4lKr82Glb+AsV+Hz/0COPnCaSKf5PL/4h/l874iyt3uTKn9CRX0OOurtKZEpHO19Pd3WB5NIzEoNDLSf7zZHBIhLO6um85GJ4t06xC/TPgF8dSbDiUibaQyIuYF62Hve959lRFpoWMk8q26u6h2kxhvb+ZHcU+hBa0ikUllRMwr/wjqjoI/FXoNMZ1GIkiJ24c7627HcS2+Gvc6t/r+ZjqSiLSByoiYt/tdb9tvDNj6T1Ja5x/OaH5S/1UAfhD3HHn2GsOJRKS19C+/mLdrlbfVFI200a+D1/Lb+quwLZdfxD/KRVaJ6Ugi0goqI2KW60LJW979AZPMZpEIZjGn/hbeCF5MV6uWJxN+RpZVdvaXiUhYUBkRsw6WQPVusOMha4LpNBLB6oljRt132NBwhM0z8fPozSHTsUSkBVRGxKzQqEi/sZDQ1WwWiXjVdGVaYCY7nd5k22U8nfBTkjlqOpaInIXKiJgVKiPZ/2Y2h0SNCnrwtbpZVLipDLd38KuE+XThuOlYInIGKiNizsnrRVRGpB3tcDO5JXA31W4Xcu2NPJnwoAqJSBhTGRFzDm7XehHpMB+52dwSmEm124VP2Rt4MuFBCNSYjiUip9HqC+WJtJsthd62/3itF5EO8Z47hFsCM3k6YR6fsjfwrx9fRX7g+xwj8bT769o1ImZoZETM+fg1b3vBZ8zmkKgWKiShEZKnEn6mRa0iYUZlRMyoOwbb3/DuD1EZkY51ciHJtTeyNOF/ddivSBhRGREzSlZC/TFI6Qfpw02nkRjwnjuEGwP3UeGmcpG9gz8m/JABVqnpWCKC1oyIIU8+/QT5cfDcgQu4Z5YubiadY72bzRcDP+SZ+LkMsMv5Y8IP+Xrgbta7A01HE4lpKiPS+VyXPLsYgBVOjtksEnN2uhl8MfAjnk6Yx0X2Dn6fcD931d3Oa844sme+dNbXa5GrSPvTNI10vr3FZNkVHHX9vOGMNJ1GYlAlqUwN3MdbwYvoatWyOOEhvu37M+CajiYSk1RGpPOtfwGAQmc0x/EbDiOx6ghJfL3ubp6q9xZQ/0/87/lF/EL8BAwnE4k9KiPSuVwX1r8IwEvBTxkOI7Gunjh+WP917qm7lTrXx3W+t/lDwo90xV+RTqYyIp1rTzFU7aTG9bPCGWU6jQgAzwWv4ubAPRxwuzHS3s5LCT9gsr3KdCyRmKEyIp3rg6UAFDpjNEUjYeUddxhTauey2rmAFOsojycsYE7c0yRQZzqaSNRTGZHOU3cMPlgGwB+DlxkOI3KqffTiK4F7WVT/OQDy417hTwlzON/abTiZSHRTGZHOs+GvcLwKUrN4y7nYdBqR06onjnn1N/L1wPc54HbjYruElxJ+wHTfX7FxTMcTiUoqI9J53vuNt825CUf/6UmYW+GM5pran/J6MAe/VccP4p9jWcL9sH+r6WgiUcdyXTfsD6yvrq4mNTWVqqoqUlJSTMeRFvjkyaMGW3so9H8fx7W4tHYBe+htKJlIa7l82beC2XHP0M06DnFd4PLvw8Q7IC7BdDiRsNbS39/681Q6xa0+75TvrzljVUQkwlj8Pngl1wR+ysrgRd41lQrvh0X/5l1jSUTOmcqIdLg0qvii7y0AFtdfaziNSNvsdntzU9098IXHISkNKjfBU9fCC9+C6n2m44lENJUR6XBfi3sVv1XHWmcwq90LTccROQcWjPoK3LEaxuZ7T73/O3hkDKyYB4Eas/FEIpTKiHSoHlTzn77lADxe/1nAMhtIpD106QGfWwD/VQj9J0DdUVgxFx4ZC8XPQLDedEKRiKIyIh3qW3H/R7J1jA+dbJY7403HEWlf/cfBra/CDU9B9wFweB/8ZQYsnADvL1UpEWmhONMBJHplsp9bfK8CML/+Blx1X4kCnzxSzJNAAvczzfcq96YuhwNb4YVvwhsPwmX/Axd/CWxfp2cViRT67SAd5t74Z0m06ljlXMgKJ8d0HJEOFSCeXwWnwJ0fwFVzvKmc/VvghW/Ao+Ng1WIIHDUdUyQsqYxIh7jU/oDP+t4h6FrMqfs6WisiMcPfDS4tgLvWwafv80rJgW3wt+/Bw8Oh8H/hcKnplCJhRdM00v5qD/PjuCUAPBW8hg3uAMOBRAzwJ8Nl34Pcb8Ha30LRQji0A96cT+0bC/ibk8tz9VfxrnshJ5f1knlTzGUWMUQjI9L+Xp7JALuc3W4aD9d/0XQaEbP83SD3m/Cd9+DLv2G1cwF+q54v+FbyB//9vJbwP/yn72VSOWI6qYgxGhmR9vXhn2DtsziuRUHgNo6QZDqRSHiwfTD8Or4UiGOUtYUbfa/zeV8RQ+w9zLaf4e64pbzmjIENLgy5GuL8phOLdBqVEWk/+96HP98OwC+Dn2eVO8xwIJHw9L57Pu/Xn89P6m/mOt9Kvup7neH2Dj7reweW3QSJqTD8Orj4BhhwiY7EkainMiJt8snDG/tSyR/9P6SvdYwVwVE8VH+DoWQikeMwSTwbvJpng3lcZJVwne9tvtGj2DtfSfFvvFu3TBh6LVw4BQZeqhETiUoqI3LOenOIZxMeoK91gC1OX75TNwNHy5FEWsFivTuQ9fUD+cZ/PwM7VsK6P8BHL8KRUli9xLslJMP5V8HQKXB+HiT1NB1cpF1Yruu6pkOcTUsvQSydJzQy0t+q4On4eQy297HbTeOG2jnso5fhdCKRq8nRNPW1sP0N2PgSbHrZKyaNLOg7GgZfCYOuhKxciEvo9LwiZ9LS398aGZE2G2lt5YmEh8i0DrLbTeOrgR+oiIi0pzi/t5h1yNUw5SHY+x5saigm5R/B3mLv9ubPIT7JW1+SfQmcN9ErKprSkQihMiKt57rc7HuN++KewW/Vs8npz7TATMrQkLFIh7Ft6D/Wu101G6r3wrYVsPV1b1tTAVte825ArRvPWncwq50LeNcZSrFzPtV0a/HH6Xwn0plURmLQ6a+tccIZ/xE6uAP+7zv8OH4FAK8Ex/G9um9xWIfwinSulL6Q81Xv5jhQvh62/RN2FlG54Q3SrGpyrY3k2huBvwCww0lnnTuIdc5A1rkD+dAZSDVdzX4fIqiMSEsdr/bOIPn2I1BXw3E3ngfrp7IkeI0ugCdimm1D5sXebdIMxs38KwOtUsbZmxhvbWK8vZGBdhkD7HIGUM5nff9qfGmJk8GHbjabnSw2uf352O3PDjfD4DcjsUhlRM7s6AFY8xQUPQpH93vPnTeJaz7+IiVuH6PRRKLR2UYuoSVTKBbb3T5sD/bhD1wBQApHGGGXMNLaxgh7OyOtbZxnV5Btl5FNGfjeaXx1rRsPjw2F9GGQPhR6DoZeg6HnIEjQSIq0P5UROZUT9A4t/OD3sO6PUH/Me77nYLjqPhh+PSWz/mY2o4i0SjXdeNsZwduMgKD3XCpHuNjezjBrBxdYu7nA3s0Qaw9JVi2UrfNun5TcxyslPQedKCjdz4PULEjqBZYuiimtp0N7Y9Dp/vJK4jgT7A1cYb/Ptb5VpFuHGr+23hnAkvp/50VnEvXqryJRzcKhv1XBm7dkQsUGqNgE+7d6Vx4+duCMrz3mJrDX7cUeN63xttftxV7SKHe7U+525whdCF0YUItko1+HHtq7cOFCHnzwQUpLSxk1ahSPPPIIEyZMaHb/P/zhD9x3332UlJQwZMgQfvrTn3Lttde25aOlXbicZ5U3DteOsT9mtLWFeCvYuMchtysvByfwQvDfWOUO5eSriopI9HKx2eVmeGd9HfqJf6ePHoAD2+HANh5e9jLZdikDrDL6WZVkWIfoYgUYbO1jMPuaff+jrp8KN5VyusPvl3lnmE3OgG4Z3v2uad4IS1IvSNDC+FjR6jKybNkyCgoKWLRoEbm5uSxYsIDJkyezadMm0tPTT9n/7bff5sYbb2Tu3Ll89rOf5bnnnuP666+nuLiYESNGtMs3IafhON4aj+o93l80+7fCga2wfwvv+9eTah095SU7nd6sdEbwqjOOt5yLqdMoiIicLKmnd+s/ll88l9Q43QOQQB2Z1gH6WZX0syrpy/6G+xX0sQ7Q2zpEinWMJKuWAZa3kJaPNp/58+ISG4pJT+jS80RJaXyuh3cdH3+Kt01s2CZ0a/F0UUvW6JxJpIzunNNRlJ2g1dM0ubm5jB8/nkcffRQAx3HIysrijjvuYObMmafsP3XqVGpqavjrX//a+NynPvUpcnJyWLRoUYs+U9M0QLAOjlfBsYNNb0cPnLhfUw7V+7zrWhwuBaeu2berdePY4J7HOmcQH7iD+JczzPtrSESEs/9yassv8S4cp7dVRToH6W1V8dh1/bx/q46Undge3e/dgoG2RgfLbigoDeXEn9pQVhoKS0I3byFuQle+/5ct1LiJHMXPUTeRGhI5hr/hOe/5IM1fqND0L/GWMlVGOmSaJhAIsGbNGmbNmtX4nG3b5OXlUVRUdNrXFBUVUVBQ0OS5yZMn8+c//7nZz6mtraW2trbxcXV1dWtidg7XBace6o9DfaBhe9z7P1CT52ohWOttQ4/ra71FoYGaU25FG3eRZB2nK8fpYtXSleMkcZyEk6ZQWspxLfaTwg43gxI3k21OJiVuJtvdPmxx+2nkQ0Sada4jBqdzjER2uonsJANcyH4BoP9p9nRJopaP7h7XUE4OeOtVQkUldDt2yPsjrbbaO/3A8SrvjzDXgeOHvNtZPBh/9ty1bjw1+L1y4vo5ip9aEjjm+mHZ77zppSnzW/W/RXvriJ9XZ2rVb6PKykqCwSAZGU3/gs7IyGDjxo2nfU1paelp9y8tLT3t/gBz587lRz/6UWuitU3RL+Hg9pMKwpnKxMlFouE512n3SBPPdqXwxFRvaPJ0t669vZXuyX0gpQ92twx6++LpDYxr96QiIp2gx4CW7+u63r/Vx6tOlJPaqk88rm744+/ISX8IHm36uK4Gao+A6/0R6Lfq8FNHT46cunxuw/ve0USYLSORMkLTnLD803jWrFlNRlOqq6vJyspq/w9a/zzsfrd93suO9+Y34xIatn7w+b1t4y0RfA1fj088MVQYn3TSsGFSkyFE4rueeN6fAvbZ2oqISIyyLIjv4t2SM8/tvVzX++P0lBHsI1B3zBvdrjsGdUchrkv75I9hrSojaWlp+Hw+ysrKmjxfVlZGZubpf/CZmZmt2h/A7/fj93fCBZ5G3QiDrjipNJypTHzyuZP29fm9MyCKiEh0sKwT//4n6bpbHa1VZSQhIYGxY8dSWFjI9ddfD3gLWAsLC5kxY8ZpXzNx4kQKCwu56667Gp977bXXmDhxYptDt5vxt5pOICIiEvNaPU1TUFDALbfcwrhx45gwYQILFiygpqaG/Px8AKZNm0a/fv2YO3cuAHfeeSeXX345P//5z5kyZQpLly5l9erVPPHEE+37nYiIiEhEanUZmTp1KhUVFcyePZvS0lJycnJYvnx54yLVnTt3Yp80ZTFp0iSee+457r33Xu655x6GDBnCn//8Z51jRERERACdDl5EREQ6SEt/f2vVpYiIiBilMiIiIiJGqYyIiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGqYyIiIiIUSojIiIiYpTKiIiIiBilMiIiIiJGtfraNCaEzlhfXV1tOImIiIi0VOj39tmuPBMRZeTw4cMAZGVlGU4iIiIirXX48GFSU1Ob/XpEXCjPcRz27t1LcnIylmW12/tWV1eTlZXFrl27dAG+MKGfSXjRzyO86OcRfvQzOTPXdTl8+DB9+/bFtptfGRIRIyO2bdO/f/8Oe/+UlBT9RxRm9DMJL/p5hBf9PMKPfibNO9OISIgWsIqIiIhRKiMiIiJiVEyXEb/fz5w5c/D7/aajSAP9TMKLfh7hRT+P8KOfSfuIiAWsIiIiEr1iemREREREzFMZEREREaNURkRERMQolRERERExSmWkQUlJCbfeeisDBw6kS5cuDB48mDlz5hAIBExHixkLFy4kOzubxMREcnNzWbVqlelIMWvu3LmMHz+e5ORk0tPTuf7669m0aZPpWNJg3rx5WJbFXXfdZTpKzNqzZw8333wzvXr1okuXLlx88cWsXr3adKyIpTLSYOPGjTiOw+OPP8769et5+OGHWbRoEffcc4/paDFh2bJlFBQUMGfOHIqLixk1ahSTJ0+mvLzcdLSY9M9//pPbb7+df/3rX7z22mvU1dXxmc98hpqaGtPRYt67777L448/zsiRI01HiVkHDx7kkksuIT4+npdffpmPPvqIn//85/To0cN0tIilQ3vP4MEHH+Sxxx5j27ZtpqNEvdzcXMaPH8+jjz4KeNcjysrK4o477mDmzJmG00lFRQXp6en885//5LLLLjMdJ2YdOXKEMWPG8Mtf/pIf//jH5OTksGDBAtOxYs7MmTNZuXIlb775pukoUUMjI2dQVVVFz549TceIeoFAgDVr1pCXl9f4nG3b5OXlUVRUZDCZhFRVVQHo/w+G3X777UyZMqXJ/1ek8/3lL39h3Lhx3HDDDaSnpzN69GgWL15sOlZEUxlpxpYtW3jkkUf45je/aTpK1KusrCQYDJKRkdHk+YyMDEpLSw2lkhDHcbjrrru45JJLGDFihOk4MWvp0qUUFxczd+5c01Fi3rZt23jssccYMmQIr7zyCrfddhvf+c53ePrpp01Hi1hRX0ZmzpyJZVlnvG3cuLHJa/bs2cM111zDDTfcwPTp0w0lFwkPt99+Ox9++CFLly41HSVm7dq1izvvvJPf/va3JCYmmo4T8xzHYcyYMTzwwAOMHj2ab3zjG0yfPp1FixaZjhax4kwH6Gjf/e53+frXv37GfQYNGtR4f+/evVx55ZVMmjSJJ554ooPTCUBaWho+n4+ysrImz5eVlZGZmWkolQDMmDGDv/71r7zxxhv079/fdJyYtWbNGsrLyxkzZkzjc8FgkDfeeINHH32U2tpafD6fwYSxpU+fPgwfPrzJc8OGDeNPf/qToUSRL+rLSO/evendu3eL9t2zZw9XXnklY8eO5cknn8S2o37gKCwkJCQwduxYCgsLuf766wHvL4/CwkJmzJhhNlyMcl2XO+64gxdeeIEVK1YwcOBA05Fi2lVXXcW6deuaPJefn8/QoUO5++67VUQ62SWXXHLKoe6bN29mwIABhhJFvqgvIy21Z88errjiCgYMGMD8+fOpqKho/Jr+Ou94BQUF3HLLLYwbN44JEyawYMECampqyM/PNx0tJt1+++0899xzvPjiiyQnJzeu3UlNTaVLly6G08We5OTkU9brdO3alV69emkdjwH//d//zaRJk3jggQf48pe/zKpVq3jiiSc0mn4OVEYavPbaa2zZsoUtW7acMhyto5873tSpU6moqGD27NmUlpaSk5PD8uXLT1nUKp3jscceA+CKK65o8vyTTz551mlPkWg3fvx4XnjhBWbNmsX999/PwIEDWbBgATfddJPpaBFL5xkRERERo7QoQkRERIxSGRERERGjVEZERETEKJURERERMUplRERERIxSGRERERGjVEZERETEKJURERERMUplRERERIxSGRERERGjVEZERETEKJURERERMer/A8VOL8TbB1wHAAAAAElFTkSuQmCC\n" 225 | }, 226 | "metadata": {}, 227 | "output_type": "display_data" 228 | } 229 | ], 230 | "source": [ 231 | "# choose any time k you wish\n", 232 | "k = -1\n", 233 | "_ = plt.hist(true_trajectories[:, k], density=True, bins=50)\n", 234 | "_ = plt.plot(xs, pdfs[k, :])" 235 | ] 236 | } 237 | ], 238 | "metadata": { 239 | "kernelspec": { 240 | "display_name": "Python 3 (ipykernel)", 241 | "language": "python", 242 | "name": "python3" 243 | }, 244 | "language_info": { 245 | "codemirror_mode": { 246 | "name": "ipython", 247 | "version": 3 248 | }, 249 | "file_extension": ".py", 250 | "mimetype": "text/x-python", 251 | "name": "python", 252 | "nbconvert_exporter": "python", 253 | "pygments_lexer": "ipython3", 254 | "version": "3.9.12" 255 | } 256 | }, 257 | "nbformat": 4, 258 | "nbformat_minor": 5 259 | } 260 | -------------------------------------------------------------------------------- /lectures/scripts/lec4_moments_estimation_compare2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "In this notebook we exemplify some methods for estimating the conditional moments of SDE.\n", 7 | "\n", 8 | "Note that this notebook uses `jax`." 9 | ], 10 | "metadata": { 11 | "collapsed": false 12 | } 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 7, 17 | "outputs": [], 18 | "source": [ 19 | "import math\n", 20 | "import jax.numpy as jnp\n", 21 | "import jax.random\n", 22 | "import tme.base_jax as tme\n", 23 | "from functools import partial\n", 24 | "from jax.config import config\n", 25 | "import matplotlib.pyplot as plt\n", 26 | "\n", 27 | "config.update(\"jax_enable_x64\", True)" 28 | ], 29 | "metadata": { 30 | "collapsed": false 31 | } 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "source": [ 36 | "Define the SDE coefficients. This is a geometric Brownian motion. We are interested in computing $\\mathbb{E}[(X(t))^n \\mid X(0) = x_0]$ for some $n$, where in this example, we let $x_0 = 1$ and $t=0.1$." 37 | ], 38 | "metadata": { 39 | "collapsed": false 40 | } 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 8, 45 | "outputs": [], 46 | "source": [ 47 | "a, b = -3., 1.\n", 48 | "\n", 49 | "\n", 50 | "def drift(x):\n", 51 | " return jnp.tanh(x)\n", 52 | "\n", 53 | "\n", 54 | "def dispersion(x):\n", 55 | " return jnp.array([b])\n", 56 | "\n", 57 | "\n", 58 | "x0 = 0.\n", 59 | "t = 1." 60 | ], 61 | "metadata": { 62 | "collapsed": false 63 | } 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "source": [ 68 | "This SDE is analytically solvable, and its raw moments are also in closed-form." 69 | ], 70 | "metadata": { 71 | "collapsed": false 72 | } 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": 9, 77 | "outputs": [], 78 | "source": [ 79 | "@partial(jax.vmap, in_axes=[0])\n", 80 | "def tme_raw_moment(order):\n", 81 | " return tme.expectation(lambda u: u ** order, jnp.array([x0]), t,\n", 82 | " drift=drift, dispersion=dispersion, order=3)" 83 | ], 84 | "metadata": { 85 | "collapsed": false 86 | } 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": 10, 91 | "outputs": [ 92 | { 93 | "data": { 94 | "text/plain": "DeviceArray([[nan],\n [nan],\n [nan]], dtype=float64)" 95 | }, 96 | "execution_count": 10, 97 | "metadata": {}, 98 | "output_type": "execute_result" 99 | } 100 | ], 101 | "source": [ 102 | "# Moment orders of interests\n", 103 | "moment_orders = jnp.arange(1, 10)\n", 104 | "\n", 105 | "# Approximate raw moment using 3-order TME\n", 106 | "tme_raw_moment(jnp.array([1, 2, 3]))" 107 | ], 108 | "metadata": { 109 | "collapsed": false 110 | } 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": 11, 115 | "outputs": [], 116 | "source": [ 117 | "z=tme.expectation(lambda u: u ** 2, jnp.array([x0]), t,\n", 118 | " drift=drift, dispersion=dispersion, order=3)[0]" 119 | ], 120 | "metadata": { 121 | "collapsed": false 122 | } 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 12, 127 | "outputs": [ 128 | { 129 | "data": { 130 | "text/plain": "DeviceArray(2., dtype=float64)" 131 | }, 132 | "execution_count": 12, 133 | "metadata": {}, 134 | "output_type": "execute_result" 135 | } 136 | ], 137 | "source": [ 138 | "z" 139 | ], 140 | "metadata": { 141 | "collapsed": false 142 | } 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": 12, 147 | "outputs": [], 148 | "source": [], 149 | "metadata": { 150 | "collapsed": false 151 | } 152 | } 153 | ], 154 | "metadata": { 155 | "kernelspec": { 156 | "display_name": "Python 3", 157 | "language": "python", 158 | "name": "python3" 159 | }, 160 | "language_info": { 161 | "codemirror_mode": { 162 | "name": "ipython", 163 | "version": 2 164 | }, 165 | "file_extension": ".py", 166 | "mimetype": "text/x-python", 167 | "name": "python", 168 | "nbconvert_exporter": "python", 169 | "pygments_lexer": "ipython2", 170 | "version": "2.7.6" 171 | } 172 | }, 173 | "nbformat": 4, 174 | "nbformat_minor": 0 175 | } 176 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/README.md: -------------------------------------------------------------------------------- 1 | # Seminar lecture notes 2 | 3 | In this folder, you find the lecture notes/slides for the seminar lectures. 4 | 5 | - Muhammad Emzir. Differential geometry for continuous-time stochastic filtering: the projection filter. `emzir_continuous_filtering.pdf`. 6 | - Cagatay Yildiz. SDEs and Markov chain Monte Carlo. `yildiz_sde_sgd_mcmc.md`. 7 | - Nathanael Bosch. Probabilistic numerics for ordinary differential equations. `bosch_probnumode.pdf` and `bosch_probnumode_demo.pjl`. 8 | 9 | All rights reserved to the authors Muhammad Emzir, Cagatay Yildiz, and Nathanael Bosch. 10 | 11 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/bosch_probnumode.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/seminar_lectures/bosch_probnumode.pdf -------------------------------------------------------------------------------- /lectures/seminar_lectures/emzir_continuous_filtering.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/seminar_lectures/emzir_continuous_filtering.pdf -------------------------------------------------------------------------------- /lectures/seminar_lectures/hostettler_sysid_gp_sdes.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/seminar_lectures/hostettler_sysid_gp_sdes.pdf -------------------------------------------------------------------------------- /lectures/seminar_lectures/yildiz_sde_sgd_mcmc.md: -------------------------------------------------------------------------------- 1 | # Lecture note of Cagatay Yildiz 2 | 3 | Please find the lecture note at https://github.com/cagatayyildiz/sde-mcmc-lecture. 4 | 5 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_brownian_motion.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/spdes/computational-sde-intro-lecture/237cae5cbc5133a6970c446034e6bd472cc8c45d/lectures/seminar_lectures/zhao_brownian_motion.pdf -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/1-jax_clinic_intro.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "RR3oQC-kQoXH", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# JAX workshop\n", 13 | "\n", 14 | "28 November, 2022 in \"A computational introduction to stochastic differential equations\"\n", 15 | "\n", 16 | "Zheng Zhao\n" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "pz6iz8GoRJcw", 23 | "slideshow": { 24 | "slide_type": "slide" 25 | } 26 | }, 27 | "source": [ 28 | "# Installation\n", 29 | "\n", 30 | "You should have installed `jax`. If not, please now do so by using the command\n", 31 | "\n", 32 | "`pip install jaxlib jax`\n", 33 | "\n", 34 | "or use `Google Colab`." 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": { 40 | "id": "r5LtQ7geQ4Mm", 41 | "slideshow": { 42 | "slide_type": "slide" 43 | } 44 | }, 45 | "source": [ 46 | "# Why/when should you use JAX?\n", 47 | "\n", 48 | "It is good to use JAX if your project requires any of:\n", 49 | "\n", 50 | "0. python is your primary language.\n", 51 | "1. just-in-time (JIT) compilation for fast computation.\n", 52 | "2. automatic differentiation.\n", 53 | "3. vectorisation.\n", 54 | "4. Multiple GPU/TPU acceleration (we don't do these today).\n", 55 | "5. ..." 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": { 61 | "id": "zVCe8ribSw0P", 62 | "slideshow": { 63 | "slide_type": "slide" 64 | } 65 | }, 66 | "source": [ 67 | "# Easy to use\n", 68 | "\n", 69 | "JAX = numpy + scipy + essential functionalities (e.g., auto-diff, JIT, and parallelisation). \n", 70 | "\n", 71 | "If you can use `numpy`, then you should have no problem to use `jax`.\n", 72 | "\n", 73 | "\n", 74 | "| | Numpy | JAX |\n", 75 | "| --- | ----------- | ----------- |\n", 76 | "| | `import numpy as np` | `import jax.numpy as jnp` |\n", 77 | "| | `np.array()` | `jnp.array()` |\n", 78 | "| | `np.ones()` | `jnp.ones()` |\n", 79 | "| | `np.zeros()` | `jnp.zeros()` |\n", 80 | "| | `np.eye()` | `jnp.eye()` |\n", 81 | "| | `np.sin()` | `jnp.sin()` |\n", 82 | "| | `np.linalg....` | `jnp.linalg....` |\n", 83 | "| | ... | ... |" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": { 89 | "id": "zQBaaD6ybTvc", 90 | "slideshow": { 91 | "slide_type": "slide" 92 | } 93 | }, 94 | "source": [ 95 | "# Galleries" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": { 101 | "slideshow": { 102 | "slide_type": "slide" 103 | } 104 | }, 105 | "source": [ 106 | "## Auto-differentiation" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": { 113 | "id": "8E6lt1wlbWjf", 114 | "slideshow": { 115 | "slide_type": "fragment" 116 | } 117 | }, 118 | "outputs": [], 119 | "source": [ 120 | "import jax\n", 121 | "import jax.numpy as jnp\n", 122 | "\n", 123 | "def f(x):\n", 124 | " return x * jnp.sin(x) + jnp.cos(x)\n", 125 | "\n", 126 | "grad_of_f = jax.grad(f) # jax.grad returns a function\n", 127 | "grad_of_f(1.)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": { 133 | "slideshow": { 134 | "slide_type": "fragment" 135 | } 136 | }, 137 | "source": [ 138 | "You say you can compute the gradient by hand? How about something deep..." 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": { 145 | "slideshow": { 146 | "slide_type": "fragment" 147 | } 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "def something_deep(x):\n", 152 | " for i in range(100):\n", 153 | " x = f(x)\n", 154 | " return x\n", 155 | "\n", 156 | "grad_of_deep = jax.grad(something_deep)\n", 157 | "grad_of_deep(1.)" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "metadata": { 163 | "slideshow": { 164 | "slide_type": "slide" 165 | } 166 | }, 167 | "source": [ 168 | "## JIT speed-up" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": { 175 | "slideshow": { 176 | "slide_type": "fragment" 177 | } 178 | }, 179 | "outputs": [], 180 | "source": [ 181 | "import numpy as np\n", 182 | "\n", 183 | "def softmax_np(x):\n", 184 | " \"\"\"Softmax in Numpy\n", 185 | " \"\"\"\n", 186 | " z = np.exp(x)\n", 187 | " return z / np.sum(z, axis=0)\n", 188 | "\n", 189 | "@jax.jit\n", 190 | "def softmax_jax(x):\n", 191 | " \"\"\"Softmax in JAX\n", 192 | " \"\"\"\n", 193 | " z = jnp.exp(x)\n", 194 | " return z / jnp.sum(z, axis=0)" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": null, 200 | "metadata": { 201 | "slideshow": { 202 | "slide_type": "fragment" 203 | } 204 | }, 205 | "outputs": [], 206 | "source": [ 207 | "x = np.ones((10, 1000))\n", 208 | "%timeit softmax_np(x)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": { 215 | "slideshow": { 216 | "slide_type": "fragment" 217 | } 218 | }, 219 | "outputs": [], 220 | "source": [ 221 | "x = jnp.ones((10, 1000))\n", 222 | "softmax_jax(x)\n", 223 | "%timeit softmax_jax(x).block_until_ready()" 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": { 229 | "id": "YOvpHon7RY_X", 230 | "slideshow": { 231 | "slide_type": "slide" 232 | } 233 | }, 234 | "source": [ 235 | "# Agenda\n", 236 | "\n", 237 | "1. JIT compilation\n", 238 | "2. Generate random numbers\n", 239 | "3. Vectorisation\n", 240 | "\n", 241 | "-- break --\n", 242 | "\n", 243 | "4. Autodiff\n", 244 | "5. Control flows (i.e., loops/if-else)\n", 245 | "6. SDE examples\n", 246 | "7. Homework" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "metadata": { 252 | "id": "Sxt85BAmeBEn", 253 | "slideshow": { 254 | "slide_type": "slide" 255 | } 256 | }, 257 | "source": [ 258 | "Useful materials:\n", 259 | "\n", 260 | "1. Jax sharp bits https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html.\n", 261 | "2. Workshop by Adrien Corenflos: https://github.com/AdrienCorenflos/jax-workshop." 262 | ] 263 | } 264 | ], 265 | "metadata": { 266 | "celltoolbar": "Slideshow", 267 | "colab": { 268 | "name": "jax-clinic-intro.ipynb", 269 | "provenance": [] 270 | }, 271 | "kernelspec": { 272 | "display_name": "Python 3 (ipykernel)", 273 | "language": "python", 274 | "name": "python3" 275 | }, 276 | "language_info": { 277 | "codemirror_mode": { 278 | "name": "ipython", 279 | "version": 3 280 | }, 281 | "file_extension": ".py", 282 | "mimetype": "text/x-python", 283 | "name": "python", 284 | "nbconvert_exporter": "python", 285 | "pygments_lexer": "ipython3", 286 | "version": "3.10.4" 287 | } 288 | }, 289 | "nbformat": 4, 290 | "nbformat_minor": 1 291 | } 292 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/2-jax_clinic_jit.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "YdAAF7si_QqA", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# JIT in JAX\n", 13 | "\n", 14 | "Suppose that we have a function that is giga-computational demanding.\n", 15 | "\n", 16 | "```python\n", 17 | "def expensive_func(x):\n", 18 | " # a bunch of giga-heavy computations\n", 19 | " # Python is slow, and Numpy too\n", 20 | " return ...\n", 21 | "```" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": { 27 | "slideshow": { 28 | "slide_type": "slide" 29 | } 30 | }, 31 | "source": [ 32 | "JAX can make the function above faster by compiling it into machine codes by using the JIT compilation. We trigger the **compilation** by calling `jax.jit`\n", 33 | "\n", 34 | "```python\n", 35 | "import jax\n", 36 | "jitted_func = jax.jit(expensive_func) \n", 37 | "# The return jitted_func is a function that has exactly the same signature as my_expensive_func\n", 38 | "```" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": { 44 | "slideshow": { 45 | "slide_type": "fragment" 46 | } 47 | }, 48 | "source": [ 49 | "It is equivalent to put a decorator over the function definition (like `tf.function()`):\n", 50 | "\n", 51 | "```python\n", 52 | "@jax.jit\n", 53 | "def expensive_func(x):\n", 54 | " ...\n", 55 | " return ...\n", 56 | "```" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": { 62 | "id": "4NDFmRkVtwzE", 63 | "slideshow": { 64 | "slide_type": "slide" 65 | } 66 | }, 67 | "source": [ 68 | "# Example" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "id": "7wRjDArQjpJx", 76 | "slideshow": { 77 | "slide_type": "fragment" 78 | } 79 | }, 80 | "outputs": [], 81 | "source": [ 82 | "import numpy as np\n", 83 | "import jax\n", 84 | "import jax.numpy as jnp" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": { 90 | "id": "QmM_PGYEreoS", 91 | "slideshow": { 92 | "slide_type": "fragment" 93 | } 94 | }, 95 | "source": [ 96 | "Recall that Jax by default uses `float32` for the best compatiblity with GPUs. You need to manually do the following to enable `float64` globally." 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": { 103 | "id": "y4_sUNAsrpbu", 104 | "slideshow": { 105 | "slide_type": "fragment" 106 | } 107 | }, 108 | "outputs": [], 109 | "source": [ 110 | "from jax.config import config\n", 111 | "config.update(\"jax_enable_x64\", True)" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": { 118 | "id": "mAG01BuEdl7t", 119 | "slideshow": { 120 | "slide_type": "slide" 121 | } 122 | }, 123 | "outputs": [], 124 | "source": [ 125 | "def func_np(x):\n", 126 | " return np.diff(np.diff(np.diff(np.diff(np.diff(x)))))\n", 127 | "\n", 128 | "@jax.jit\n", 129 | "def func_jax(x):\n", 130 | " return jnp.diff(jnp.diff(jnp.diff(jnp.diff(jnp.diff(x)))))\n", 131 | "\n", 132 | "x = np.random.randn(100000)\n", 133 | "jx = jnp.asarray(x)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": { 140 | "colab": { 141 | "base_uri": "https://localhost:8080/" 142 | }, 143 | "id": "idf4FN8SjhtO", 144 | "outputId": "07ab0e65-d279-4315-dc20-9495d41b49d4", 145 | "slideshow": { 146 | "slide_type": "fragment" 147 | } 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "%timeit func_np(x)" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": { 158 | "colab": { 159 | "base_uri": "https://localhost:8080/" 160 | }, 161 | "id": "rjD_WH76kUAm", 162 | "outputId": "d8833297-fb78-4218-93c9-caee0f2679df", 163 | "slideshow": { 164 | "slide_type": "fragment" 165 | } 166 | }, 167 | "outputs": [], 168 | "source": [ 169 | "# Trigger jit\n", 170 | "func_jax(jx)\n", 171 | "\n", 172 | "%timeit func_jax(jx).block_until_ready()" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": { 178 | "slideshow": { 179 | "slide_type": "fragment" 180 | } 181 | }, 182 | "source": [ 183 | "Around 4 times faster." 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": { 189 | "slideshow": { 190 | "slide_type": "slide" 191 | } 192 | }, 193 | "source": [ 194 | "Numpy uses OpenBLAS/MKL/BLIS... as backend, why it is still slow?" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "metadata": { 200 | "slideshow": { 201 | "slide_type": "fragment" 202 | } 203 | }, 204 | "source": [ 205 | "A more illustrative example which shows that the overhead of Numpy is problematic:" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": { 212 | "slideshow": { 213 | "slide_type": "fragment" 214 | } 215 | }, 216 | "outputs": [], 217 | "source": [ 218 | "A, B = np.eye(50), np.eye(50)\n", 219 | "def func_np(x):\n", 220 | " for i in range(100):\n", 221 | " x = B @ x + np.linalg.solve(A, x) + np.linalg.norm(x)\n", 222 | " return x\n", 223 | " \n", 224 | " \n", 225 | "Aj, Bj = jnp.eye(50), jnp.eye(50)\n", 226 | "@jax.jit\n", 227 | "def func_jax(x):\n", 228 | " def scan_body(carry, _):\n", 229 | " x = carry\n", 230 | " return Bj @ x + jnp.linalg.solve(Aj, x) + jnp.linalg.norm(x), _\n", 231 | " return jax.lax.scan(scan_body, x, jnp.arange(100))[-1]" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": { 238 | "slideshow": { 239 | "slide_type": "fragment" 240 | } 241 | }, 242 | "outputs": [], 243 | "source": [ 244 | "%timeit func_np(np.ones((50, )))" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "metadata": { 251 | "slideshow": { 252 | "slide_type": "fragment" 253 | } 254 | }, 255 | "outputs": [], 256 | "source": [ 257 | "# Trigger jit\n", 258 | "func_jax(jnp.ones((50, )))\n", 259 | "\n", 260 | "%timeit func_jax(jnp.ones((50, ))).block_until_ready()" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": { 266 | "id": "Omgh2VS8v6yk", 267 | "slideshow": { 268 | "slide_type": "slide" 269 | } 270 | }, 271 | "source": [ 272 | "# What happened inside JIT?\n", 273 | "\n", 274 | "1. When Python executes `func_jax` for the first time, the function `jax.jit` traces/traverses all the operations inside the function `func_jax`." 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": { 280 | "slideshow": { 281 | "slide_type": "fragment" 282 | } 283 | }, 284 | "source": [ 285 | "2. Then JIT compiles these operations into the **accelerated linear algebra (XLA)** codes. Imagine this as that of compiling C codes to an executable file. No numerical computations are done!" 286 | ] 287 | }, 288 | { 289 | "cell_type": "markdown", 290 | "metadata": { 291 | "slideshow": { 292 | "slide_type": "fragment" 293 | } 294 | }, 295 | "source": [ 296 | "3. Then, by the next time you call the jitted `func_jax`, Python will execute the compiled XLA codes to carry out the numerical computations. " 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": { 302 | "slideshow": { 303 | "slide_type": "fragment" 304 | } 305 | }, 306 | "source": [ 307 | "4. After the numerical computations are done in the machine level, the results are sent back to Python." 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "metadata": { 313 | "slideshow": { 314 | "slide_type": "slide" 315 | } 316 | }, 317 | "source": [ 318 | "JAX basically uses Python as a \"metaprogramming language\" that specifies how to build an XLA program (quote: [Patrick Kidger](https://kidger.site/thoughts/jax-vs-julia)). " 319 | ] 320 | }, 321 | { 322 | "cell_type": "markdown", 323 | "metadata": { 324 | "id": "k81IzQsutRma", 325 | "slideshow": { 326 | "slide_type": "slide" 327 | } 328 | }, 329 | "source": [ 330 | "# When/where to JIT?\n", 331 | "\n", 332 | "Usually, we JIT\n", 333 | "\n", 334 | "1. the part that has the largest scope, so that the compiler can understand your programme better,\n", 335 | "2. or the function(s) that are called repetitively, for instance, the objective function in optimisation:\n", 336 | "\n", 337 | "```python\n", 338 | "@jax.jit\n", 339 | "def objective_func(params):\n", 340 | " ...\n", 341 | " return ...\n", 342 | "```\n", 343 | "\n", 344 | "Remember, when we write the jax code, we are describing a computation flow." 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": { 350 | "id": "ijMj6Caj-M_6", 351 | "slideshow": { 352 | "slide_type": "slide" 353 | } 354 | }, 355 | "source": [ 356 | "# Will these work?" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": { 363 | "slideshow": { 364 | "slide_type": "slide" 365 | } 366 | }, 367 | "outputs": [], 368 | "source": [ 369 | "@jax.jit\n", 370 | "def my_func(x):\n", 371 | " return np.exp(x)\n", 372 | "\n", 373 | "my_func(jnp.ones((2, )))" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "metadata": { 380 | "id": "d-fgu9_o-MX5", 381 | "slideshow": { 382 | "slide_type": "slide" 383 | } 384 | }, 385 | "outputs": [], 386 | "source": [ 387 | "@jax.jit\n", 388 | "def my_func(x):\n", 389 | " return x + np.array([1., 2.])\n", 390 | "\n", 391 | "my_func(jnp.ones((2, )))\n", 392 | "my_func(np.ones((2, )))" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": { 398 | "slideshow": { 399 | "slide_type": "fragment" 400 | } 401 | }, 402 | "source": [ 403 | "Note: this does not work in some early jax versions." 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": null, 409 | "metadata": { 410 | "id": "B1iKKg-Z-aMs", 411 | "slideshow": { 412 | "slide_type": "slide" 413 | } 414 | }, 415 | "outputs": [], 416 | "source": [ 417 | "# Another example\n", 418 | "\n", 419 | "class MyClass:\n", 420 | "\n", 421 | " def __init__(self):\n", 422 | " self.y = jnp.array(1.)\n", 423 | "\n", 424 | " @jax.jit\n", 425 | " def my_method(self, x):\n", 426 | " return x + self.y\n", 427 | " \n", 428 | "obj = MyClass()\n", 429 | "obj.my_method(jnp.array(2.))" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": { 435 | "id": "oFD8U-h8-yO_", 436 | "slideshow": { 437 | "slide_type": "slide" 438 | } 439 | }, 440 | "source": [ 441 | "We can force it to work by adding a `static_argums` option. This assumes that `self` is static, that is, immutable." 442 | ] 443 | }, 444 | { 445 | "cell_type": "code", 446 | "execution_count": null, 447 | "metadata": { 448 | "id": "1BoVZ2tB_DG5", 449 | "slideshow": { 450 | "slide_type": "fragment" 451 | } 452 | }, 453 | "outputs": [], 454 | "source": [ 455 | "from functools import partial\n", 456 | "\n", 457 | "class MyClass:\n", 458 | "\n", 459 | " def __init__(self):\n", 460 | " self.y = jnp.array(1.)\n", 461 | "\n", 462 | " @partial(jax.jit, static_argnums=(0, ))\n", 463 | " def my_method(self, x):\n", 464 | " return x + self.y\n", 465 | " \n", 466 | "obj = MyClass()\n", 467 | "obj.my_method(jnp.array(2.))" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "metadata": { 473 | "id": "GleFYqNo_Po5", 474 | "slideshow": { 475 | "slide_type": "slide" 476 | } 477 | }, 478 | "source": [ 479 | "Similarly, this will not (immediately) work either\n", 480 | "\n", 481 | "```python\n", 482 | "@jit\n", 483 | "def f(x, g: Callable):\n", 484 | " return x + g(x)\n", 485 | "```\n", 486 | "\n", 487 | "We have to make it clear that the argument `g` is static:\n", 488 | "\n", 489 | "```python\n", 490 | "@partial(jit, static_argnums=(1, ))\n", 491 | "def f(x, g: Callable):\n", 492 | " return x + g(x)\n", 493 | "```\n", 494 | "\n", 495 | "A more concise way is to put `g` in an outer scope\n", 496 | "\n", 497 | "```python\n", 498 | "g = ... # Definition of g from outer scope\n", 499 | "\n", 500 | "@jit\n", 501 | "def f(x):\n", 502 | " return x + g(x)\n", 503 | "```\n", 504 | "\n", 505 | "Note: whenever the static argument changes, it will trigger the JIT compilation again to create another XLA code." 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": { 511 | "slideshow": { 512 | "slide_type": "slide" 513 | } 514 | }, 515 | "source": [ 516 | "JAX accepts **immutable** objects, while Python Class is **mutable**. \n", 517 | "\n", 518 | "For instance, " 519 | ] 520 | }, 521 | { 522 | "cell_type": "code", 523 | "execution_count": null, 524 | "metadata": { 525 | "slideshow": { 526 | "slide_type": "fragment" 527 | } 528 | }, 529 | "outputs": [], 530 | "source": [ 531 | "x = jnp.ones((2, ))\n", 532 | "x[0] = 1.\n", 533 | "\n", 534 | "# does not work." 535 | ] 536 | }, 537 | { 538 | "cell_type": "markdown", 539 | "metadata": { 540 | "slideshow": { 541 | "slide_type": "fragment" 542 | } 543 | }, 544 | "source": [ 545 | "Well, doesn't JAX suck if we cannot even assign/update values to variables?\n", 546 | "\n", 547 | "No, this in my opinion a feature not a problem. Immutable objects are best for computations. For projects that need mutable objects, we can rewrite them into that of based on immutable ones." 548 | ] 549 | } 550 | ], 551 | "metadata": { 552 | "celltoolbar": "Slideshow", 553 | "colab": { 554 | "name": "jax-clinic-jit.ipynb", 555 | "provenance": [] 556 | }, 557 | "kernelspec": { 558 | "display_name": "Python 3 (ipykernel)", 559 | "language": "python", 560 | "name": "python3" 561 | }, 562 | "language_info": { 563 | "codemirror_mode": { 564 | "name": "ipython", 565 | "version": 3 566 | }, 567 | "file_extension": ".py", 568 | "mimetype": "text/x-python", 569 | "name": "python", 570 | "nbconvert_exporter": "python", 571 | "pygments_lexer": "ipython3", 572 | "version": "3.10.4" 573 | } 574 | }, 575 | "nbformat": 4, 576 | "nbformat_minor": 1 577 | } 578 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/3-jax_clinic_randomness.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "_UtcY71YIxA8", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# Generate random numbers\n", 13 | "\n", 14 | "Generating random numbers in JAX needs the programmer to be conscious, not drunk. \n", 15 | "\n", 16 | "This is going to look painful in the beginning. \n", 17 | "\n", 18 | "But in the end we will see this is super helpful in controlling the randomness flow. " 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": { 24 | "id": "M7CP_OSJI2j8", 25 | "slideshow": { 26 | "slide_type": "slide" 27 | } 28 | }, 29 | "source": [ 30 | "Think about how to generate random samples in numpy, for example, \n", 31 | "\n", 32 | "```python\n", 33 | "np.random.seed(999)\n", 34 | "samples1 = np.random.randn(5)\n", 35 | "samples2 = np.random.uniform(5)\n", 36 | "samples3 = np.random.gamma(5)\n", 37 | "```\n", 38 | "\n", 39 | "The results are reproducible under a fixed random seed.\n", 40 | "\n", 41 | "Can we do\n", 42 | "\n", 43 | "```python\n", 44 | "jnp.random.seed(999)\n", 45 | "...\n", 46 | "```\n", 47 | "\n", 48 | "?" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": { 54 | "id": "3K5BYxCGJPis", 55 | "slideshow": { 56 | "slide_type": "slide" 57 | } 58 | }, 59 | "source": [ 60 | "No, we have to manually play with random keys by yourself, like in C/C++/Rust." 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": { 67 | "colab": { 68 | "base_uri": "https://localhost:8080/" 69 | }, 70 | "id": "ynDyQq2jKItD", 71 | "outputId": "b61f5ae7-1cd7-4ab2-94f3-ce98e975f2e8", 72 | "slideshow": { 73 | "slide_type": "fragment" 74 | } 75 | }, 76 | "outputs": [], 77 | "source": [ 78 | "# The equivalence to the Numpy code\n", 79 | "\n", 80 | "import jax\n", 81 | "\n", 82 | "key = jax.random.PRNGKey(999)\n", 83 | "samples1 = jax.random.normal(key, shape=(5, ))\n", 84 | "\n", 85 | "key, _ = jax.random.split(key)\n", 86 | "samples2 = jax.random.uniform(key, shape=(5, ))\n", 87 | "\n", 88 | "key, _ = jax.random.split(key)\n", 89 | "samples3 = jax.random.gamma(key, shape=(5, ), a=1.)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": { 95 | "id": "yM9Kou1DLTX4", 96 | "slideshow": { 97 | "slide_type": "slide" 98 | } 99 | }, 100 | "source": [ 101 | "We may think explicitly splitting random keys is a fuss/verbose, but this is **standard** for asynchronous/parallel/vectorisable programmes. \n", 102 | "\n", 103 | "```python\n", 104 | "np.random.seed(999)\n", 105 | "samples1 = np.random.randn(5)\n", 106 | "samples2 = np.random.uniform(5)\n", 107 | "samples3 = np.random.gamma(5)\n", 108 | "```\n", 109 | "\n", 110 | "Why the results are reproducible? " 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": { 116 | "slideshow": { 117 | "slide_type": "fragment" 118 | } 119 | }, 120 | "source": [ 121 | "Because: 1) the random seed is fixed. 2) the order of execution is fixed. \n", 122 | "\n", 123 | "If `samples1` `samples2` `samples3` are executed in parallel, can you guarantee the results are the same? No, because you will not know the order of execution. This essentially introduces another randomness (from hardware level) that is hard to control. " 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": { 130 | "colab": { 131 | "base_uri": "https://localhost:8080/" 132 | }, 133 | "id": "uMUnzMIlOqw5", 134 | "outputId": "a2d7b4cb-afd4-4bcc-fe75-a5a4aba3acce", 135 | "slideshow": { 136 | "slide_type": "slide" 137 | } 138 | }, 139 | "outputs": [], 140 | "source": [ 141 | "key = jax.random.PRNGKey(999)\n", 142 | "print(key)\n", 143 | "\n", 144 | "# The randomness under this splitted key is independent of the previous key\n", 145 | "key, subkey = jax.random.split(key)\n", 146 | "print(key, subkey)\n", 147 | "\n", 148 | "key, subkey = jax.random.split(key)\n", 149 | "print(key, subkey)\n", 150 | "\n", 151 | "keys = jax.random.split(key, num=5)\n", 152 | "print(keys)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": { 158 | "id": "sOEj59-BPSYk", 159 | "slideshow": { 160 | "slide_type": "fragment" 161 | } 162 | }, 163 | "source": [ 164 | "In practice, you can just ignore `subkey` which is used to track randomness.\n", 165 | "\n", 166 | "`key, _ = jax.random.split(key)`" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": { 172 | "slideshow": { 173 | "slide_type": "slide" 174 | } 175 | }, 176 | "source": [ 177 | "# Example\n", 178 | "\n", 179 | "Imagine that we have an algorithm and we woule like to test it on a synthetic model for 100 **independent** Monte Carlo runs then average the results. \n", 180 | "\n", 181 | "```python\n", 182 | "num_mcs = 100\n", 183 | "for i in range(num_mcs)\n", 184 | " np.random.seed(i)\n", 185 | " data = generate_data()\n", 186 | " result = my_algorithm(data)\n", 187 | "```" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "metadata": { 193 | "slideshow": { 194 | "slide_type": "fragment" 195 | } 196 | }, 197 | "source": [ 198 | "The implementation above is a common mistake, as seeds 1, 2, ... 100 are **not independent**. How to make the randomness in the loop independent to each other? I actually don't know how to do so in Numpy.\n", 199 | "\n", 200 | "In JAX, it's convenient:\n", 201 | "\n", 202 | "```python\n", 203 | "num_mcs = 100\n", 204 | "key = jax.random.PRNGKey(1)\n", 205 | "for i in range(num_mcs)\n", 206 | " key, _ = jax.random.split(key)\n", 207 | " data = generate_data(key)\n", 208 | " result = my_algorithm(data)\n", 209 | "```" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": { 215 | "id": "5k-GSkrsORFh", 216 | "slideshow": { 217 | "slide_type": "slide" 218 | } 219 | }, 220 | "source": [ 221 | "# Exercise\n", 222 | "\n", 223 | "Generate two \"random positive definite matrices of size 6\" then add them." 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": { 229 | "slideshow": { 230 | "slide_type": "slide" 231 | } 232 | }, 233 | "source": [ 234 | "Numpy implementation\n", 235 | "\n", 236 | "```python\n", 237 | "np.random.seed(999)\n", 238 | "\n", 239 | "rand = np.random.randn(6)\n", 240 | "psd_matrix_1 = np.outer(rand, rand) + np.eye(6)\n", 241 | "\n", 242 | "rand = np.random.randn(6)\n", 243 | "psd_matrix_2 = np.outer(rand, rand) + np.eye(6)\n", 244 | "\n", 245 | "print(np.linalg.eigh(psd_matrix_1 + psd_matrix_2)[0])\n", 246 | "```\n", 247 | "\n", 248 | "```python\n", 249 | "import jax.numpy as jnp\n", 250 | "\n", 251 | "key = ?\n", 252 | "rand = ?\n", 253 | "psd_matrix_1 = jnp.outer(rand, rand) + jnp.eye(6)\n", 254 | "\n", 255 | "key, _ = ?\n", 256 | "rand = ?\n", 257 | "psd_matrix_2 = jnp.outer(rand, rand) + jnp.eye(6)\n", 258 | "\n", 259 | "print(jnp.linalg.eigh(psd_matrix_1 + psd_matrix_2)[0])\n", 260 | "```" 261 | ] 262 | }, 263 | { 264 | "cell_type": "markdown", 265 | "metadata": { 266 | "id": "FlhHZrSYQugn", 267 | "slideshow": { 268 | "slide_type": "slide" 269 | } 270 | }, 271 | "source": [ 272 | "## Solution" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": null, 278 | "metadata": { 279 | "colab": { 280 | "base_uri": "https://localhost:8080/" 281 | }, 282 | "id": "YuMpyMRKJO9j", 283 | "outputId": "03876388-99ec-4337-b34d-117839c733ea", 284 | "slideshow": { 285 | "slide_type": "fragment" 286 | } 287 | }, 288 | "outputs": [], 289 | "source": [ 290 | "import jax.numpy as jnp\n", 291 | "\n", 292 | "key = jax.random.PRNGKey(999)\n", 293 | "rand = jax.random.normal(key, (6, ))\n", 294 | "psd_matrix_1 = jnp.outer(rand, rand) + jnp.eye(6)\n", 295 | "\n", 296 | "key, _ = jax.random.split(key)\n", 297 | "rand = jax.random.normal(key, (6, ))\n", 298 | "psd_matrix_2 = jnp.outer(rand, rand) + jnp.eye(6)\n", 299 | "\n", 300 | "print(jnp.linalg.eigh(psd_matrix_1 + psd_matrix_2)[0])" 301 | ] 302 | } 303 | ], 304 | "metadata": { 305 | "celltoolbar": "Slideshow", 306 | "colab": { 307 | "name": "jax-clinic-randomness.ipynb", 308 | "provenance": [] 309 | }, 310 | "kernelspec": { 311 | "display_name": "Python 3 (ipykernel)", 312 | "language": "python", 313 | "name": "python3" 314 | }, 315 | "language_info": { 316 | "codemirror_mode": { 317 | "name": "ipython", 318 | "version": 3 319 | }, 320 | "file_extension": ".py", 321 | "mimetype": "text/x-python", 322 | "name": "python", 323 | "nbconvert_exporter": "python", 324 | "pygments_lexer": "ipython3", 325 | "version": "3.10.4" 326 | } 327 | }, 328 | "nbformat": 4, 329 | "nbformat_minor": 1 330 | } 331 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/4-jax_clinic_vectorisation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "PQN9CCp5w709", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# Vectorisation\n", 13 | "\n", 14 | "One of the most helpful feature for my past projects (e.g., PDEs, GPs), and one reason that I completely abandoned Matlab.\n", 15 | "\n", 16 | "Note: THE vectorisation is not the same as parallelisation." 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": { 22 | "id": "m7XFMcAD5bAK", 23 | "slideshow": { 24 | "slide_type": "slide" 25 | } 26 | }, 27 | "source": [ 28 | "# Example\n", 29 | "\n", 30 | "Suppose that we have two arrays `a` and `b` of shapes `(10, 5, 3, 6)` and `(10, 3, 7, 6)`, respectively. " 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": { 36 | "slideshow": { 37 | "slide_type": "fragment" 38 | } 39 | }, 40 | "source": [ 41 | "How to do `a @ b` in the way that the multiplication applies to `(5, 3) x (3, 7)` while taking other dimensions as the broadcasting dimension? Eventually, we desire an array of shape `(10, 5, 7, 6)`. \n", 42 | "\n", 43 | "How to do so in Numpy/Matlab?\n", 44 | "\n", 45 | "(einsum)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": { 51 | "id": "7L8bFendxJF2", 52 | "slideshow": { 53 | "slide_type": "slide" 54 | } 55 | }, 56 | "source": [ 57 | "# Example" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": null, 63 | "metadata": { 64 | "id": "AgSghCD9xLlM", 65 | "slideshow": { 66 | "slide_type": "slide" 67 | } 68 | }, 69 | "outputs": [], 70 | "source": [ 71 | "def func(x, y):\n", 72 | " \"\"\"Arguments x and y have the same shape (2, ).\n", 73 | " Return a (2, 2) matrix.\n", 74 | " \"\"\"\n", 75 | " z = x * y\n", 76 | " return np.array([[x[0] ** 2, x[0] * x[1]], \n", 77 | " [np.sin(x[1]), x[0] + x[1]]]) + np.outer(z, z)" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": { 83 | "id": "uUQs7fdNzHtI", 84 | "slideshow": { 85 | "slide_type": "fragment" 86 | } 87 | }, 88 | "source": [ 89 | "Now if inputs `x` and `y` are of shape `(100, 2)`, how to batch over the first dimension and return a tensor `(100, 2, 2)`?" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "metadata": { 95 | "slideshow": { 96 | "slide_type": "fragment" 97 | } 98 | }, 99 | "source": [ 100 | "Even more complicated, if the inputs `x` and `y` are of shapes `(100, 2)` and `(300, 2)`, respectively, how to visit over the 100 and 200 and return a tensor `(100, 300, 2, 2)`?\n", 101 | "\n", 102 | "Good luck with Matlab." 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": { 108 | "id": "WR5rPSsC1jFr", 109 | "slideshow": { 110 | "slide_type": "slide" 111 | } 112 | }, 113 | "source": [ 114 | "How to do these in numpy? As an example," 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": { 121 | "colab": { 122 | "base_uri": "https://localhost:8080/" 123 | }, 124 | "id": "236-c-z-3DTE", 125 | "outputId": "2b104a44-72d5-487c-83c4-ee304ba99091", 126 | "slideshow": { 127 | "slide_type": "fragment" 128 | } 129 | }, 130 | "outputs": [], 131 | "source": [ 132 | "import numpy as np\n", 133 | "\n", 134 | "np_vectorized_func = np.vectorize(func, signature='(n),(n)->(n,n)')\n", 135 | "\n", 136 | "np_vectorized_func(np.ones((5, 2)), np.ones((5, 2))).shape" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": { 142 | "id": "6sapu4FK42Hy", 143 | "slideshow": { 144 | "slide_type": "fragment" 145 | } 146 | }, 147 | "source": [ 148 | "Cool! numpy has a concise way to do the vectorisation. " 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "metadata": { 154 | "slideshow": { 155 | "slide_type": "fragment" 156 | } 157 | }, 158 | "source": [ 159 | "However, please note that `np.vectorize` is merely a syntax sugar of a python loop. It is **not a vectorisation on the computation level**. " 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": { 165 | "id": "CTQ4ryKo1jH6", 166 | "slideshow": { 167 | "slide_type": "slide" 168 | } 169 | }, 170 | "source": [ 171 | "The jax implementation" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": null, 177 | "metadata": { 178 | "colab": { 179 | "base_uri": "https://localhost:8080/" 180 | }, 181 | "id": "-O0MieW_77RD", 182 | "outputId": "1c18ed0f-62a6-4cae-98ef-be8cbeed8290", 183 | "slideshow": { 184 | "slide_type": "fragment" 185 | } 186 | }, 187 | "outputs": [], 188 | "source": [ 189 | "import jax\n", 190 | "import jax.numpy as jnp\n", 191 | "\n", 192 | "def func(x, y):\n", 193 | " \"\"\"Arguments x and y are of the same shape (2, ).\n", 194 | " Return a (2, 2) matrix.\n", 195 | " \"\"\"\n", 196 | " z = x * y\n", 197 | " return jnp.array([[x[0] ** 2, x[0] * x[1]], \n", 198 | " [jnp.sin(x[1]), x[0] + x[1]]]) + jnp.outer(z, z)\n", 199 | "\n", 200 | "jax_vectorized_func = jax.vmap(func, in_axes=[0, 0])\n", 201 | "\n", 202 | "jax_vectorized_func(jnp.ones((5, 2)), jnp.ones((5, 2))).shape" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": { 208 | "id": "kn4_EDXc1jJ7", 209 | "slideshow": { 210 | "slide_type": "slide" 211 | } 212 | }, 213 | "source": [ 214 | "Compare speed" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": { 221 | "colab": { 222 | "base_uri": "https://localhost:8080/" 223 | }, 224 | "id": "b9TVSWwV8TZm", 225 | "outputId": "d457add7-cb8b-4b84-ec11-bf088e460ab2", 226 | "slideshow": { 227 | "slide_type": "fragment" 228 | } 229 | }, 230 | "outputs": [], 231 | "source": [ 232 | "a = np.ones((10000, 2))\n", 233 | "%timeit np_vectorized_func(a, a)" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": null, 239 | "metadata": { 240 | "colab": { 241 | "base_uri": "https://localhost:8080/" 242 | }, 243 | "id": "JFls-TAO8X0t", 244 | "outputId": "0c48070b-742d-4166-decc-aad9a426c588", 245 | "slideshow": { 246 | "slide_type": "fragment" 247 | } 248 | }, 249 | "outputs": [], 250 | "source": [ 251 | "a = jnp.asarray(a)\n", 252 | "_ = jax_vectorized_func(a, a)\n", 253 | "%timeit jax_vectorized_func(a, a).block_until_ready()" 254 | ] 255 | }, 256 | { 257 | "cell_type": "markdown", 258 | "metadata": { 259 | "id": "j1kWLNZS9Et1", 260 | "slideshow": { 261 | "slide_type": "slide" 262 | } 263 | }, 264 | "source": [ 265 | "`jax.vmap(func,\n", 266 | " in_axes) -> Callable[the same signature as func]`\n", 267 | "\n", 268 | "- `func`: the function you want to vectorize\n", 269 | "- `in_axes`: a tuple/list that indicates the vectorization axes for the arguments of `func`." 270 | ] 271 | }, 272 | { 273 | "cell_type": "markdown", 274 | "metadata": { 275 | "slideshow": { 276 | "slide_type": "fragment" 277 | } 278 | }, 279 | "source": [ 280 | "Recall that our `func(x, y)` takes two `(2, )` arrays as inputs and returns a `(2, 2)` matrix. " 281 | ] 282 | }, 283 | { 284 | "cell_type": "markdown", 285 | "metadata": { 286 | "slideshow": { 287 | "slide_type": "fragment" 288 | } 289 | }, 290 | "source": [ 291 | "- To vectorize for `x: (n, 2)` and `y: (n, 2)` for some `n`, use `in_axes=[0, 0]`. Get `(n, ...)`" 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": { 297 | "slideshow": { 298 | "slide_type": "fragment" 299 | } 300 | }, 301 | "source": [ 302 | "- To vectorize for `x: (2, n)` and `y: (n, 2)` for some `n`, use `in_axes=[1, 0]`. Get `(n, ...)`" 303 | ] 304 | }, 305 | { 306 | "cell_type": "markdown", 307 | "metadata": { 308 | "slideshow": { 309 | "slide_type": "fragment" 310 | } 311 | }, 312 | "source": [ 313 | "- To vectorize for `x: (n, 2)` and `y: (2, )` for some `n`, use `in_axes=[0, None]`. Get `(n, ...)`" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": { 319 | "slideshow": { 320 | "slide_type": "fragment" 321 | } 322 | }, 323 | "source": [ 324 | "To vectorize for `x: (m, 2)` and `y: (n, 2)` for some `m, n`. How to do? Use two `vmap` nested!\n", 325 | "\n", 326 | "```python\n", 327 | "jax.vmap(jax.vmap(func, in_axes=[0, None]), in_axes=[None, 0])\n", 328 | "```\n", 329 | "\n", 330 | " Get `(m, n, ...)`" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "metadata": { 336 | "id": "Yq7skblx1jPT", 337 | "slideshow": { 338 | "slide_type": "slide" 339 | } 340 | }, 341 | "source": [ 342 | "# Exercise\n", 343 | "\n", 344 | "Monte Carlo approximation $\\mathbb{E}[g(X)] \\approx \\frac{1}{N}\\sum^N_{i=1} g(X^i),$ where $X, X^1, X^2, \\ldots \\sim \\mathrm{N}(0, I_2)$ and $ g(X) = \\begin{bmatrix} \\exp(X_1) \\sin(X_2) \\\\ X_1 \\, X_2 + X_1\\end{bmatrix} $\n", 345 | "\n", 346 | "```python\n", 347 | "N = 1000\n", 348 | "key = ?\n", 349 | "samples = jax.random.normal(?)\n", 350 | "\n", 351 | "def g(x):\n", 352 | " return ?\n", 353 | " \n", 354 | "vectorised_g = jax.vmap(?)\n", 355 | "propogated_samples = vectorised_g(samples)\n", 356 | "mean_g = jnp.mean(propogated_samples, axis=0)\n", 357 | "```" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "metadata": { 363 | "id": "7oOKuu4STGDQ", 364 | "slideshow": { 365 | "slide_type": "slide" 366 | } 367 | }, 368 | "source": [ 369 | "## Solution" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": null, 375 | "metadata": { 376 | "colab": { 377 | "base_uri": "https://localhost:8080/" 378 | }, 379 | "id": "-cKnpqaP_9go", 380 | "outputId": "466d6dc1-068c-4e03-d479-8732b2f138d8", 381 | "slideshow": { 382 | "slide_type": "fragment" 383 | } 384 | }, 385 | "outputs": [], 386 | "source": [ 387 | "N = 1000\n", 388 | "\n", 389 | "key = jax.random.PRNGKey(999)\n", 390 | "samples = jax.random.normal(key, shape=(N, 2))\n", 391 | "\n", 392 | "def g(x):\n", 393 | " return jnp.array([jnp.exp(x[0]) * jnp.sin(x[1]), \n", 394 | " x[0] * x[1] + x[0]])\n", 395 | " \n", 396 | "vectorised_g = jax.vmap(g, in_axes=[0])\n", 397 | "\n", 398 | "propogated_samples = vectorised_g(samples)\n", 399 | "\n", 400 | "mean_g = jnp.mean(propogated_samples, axis=0)" 401 | ] 402 | }, 403 | { 404 | "cell_type": "markdown", 405 | "metadata": { 406 | "id": "arHygqLKp3AZ", 407 | "slideshow": { 408 | "slide_type": "slide" 409 | } 410 | }, 411 | "source": [ 412 | "# Exercise\n", 413 | "\n", 414 | "Consider a Matern 3/2 covariance function $C\\colon \\mathbb{R}\\times \\mathbb{R} \\to \\mathbb{R}$ defined by\n", 415 | "\n", 416 | "$$\n", 417 | "C(t, t') = \\sigma^2 \\, \\bigg(1 + \\frac{\\sqrt{3} \\, \\lvert t-t'\\rvert}{\\ell}\\bigg) \\, \\exp\\bigg(-\\frac{\\sqrt{3} \\, \\lvert t-t'\\rvert}{\\ell}\\bigg)\n", 418 | "$$\n", 419 | "\n", 420 | "Say, now we have $T$ data points $t_1, t_2, \\ldots, t_T$, compute the covariance matrix evaluated at the Cartesian $(t_1, t_2, \\ldots, t_T) \\times (t_1, t_2, \\ldots, t_T)$, that is, \n", 421 | "\n", 422 | "$$\n", 423 | "C_{1:T} = \\begin{bmatrix} C(t_1, t_1) & C(t_1, t_2) & \\cdots & C(t_1, t_T) \\\\\n", 424 | " C(t_2, t_1) & \\ddots & & \\vdots\\\\\n", 425 | " \\vdots & & & \\vdots\\\\\n", 426 | " C(t_T, t_1) & \\cdots & \\cdots & C(t_T, t_T)\\end{bmatrix}.\n", 427 | "$$\n", 428 | "\n", 429 | "```python\n", 430 | "def cov_func(t1: float, t2: float, ell: float, sigma: float) -> float:\n", 431 | " return ?\n", 432 | "\n", 433 | "vectorised_cov_func = jax.vmap(jax.vmap(?), ?)\n", 434 | "\n", 435 | "ts = jnp.linspace(0.01, 1, 100)\n", 436 | "\n", 437 | "ell, sigma = 0.1, 1.\n", 438 | "cov_matrix = vectorised_cov_func(ts, ts, ell, sigma)\n", 439 | "\n", 440 | "import matplotlib.pyplot as plt\n", 441 | "plt.contourf(cov_matrix, levels=20)\n", 442 | "```" 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": { 448 | "id": "7iPjAFRasjT3", 449 | "slideshow": { 450 | "slide_type": "slide" 451 | } 452 | }, 453 | "source": [ 454 | "## Solution" 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": null, 460 | "metadata": { 461 | "colab": { 462 | "base_uri": "https://localhost:8080/", 463 | "height": 282 464 | }, 465 | "id": "szbO28UMsky0", 466 | "outputId": "0d60f87e-12ca-499d-b7f2-c89a0cb28d09", 467 | "slideshow": { 468 | "slide_type": "fragment" 469 | } 470 | }, 471 | "outputs": [], 472 | "source": [ 473 | "import math\n", 474 | "\n", 475 | "def cov_func(t1: float, t2: float, ell: float, sigma: float) -> float:\n", 476 | " z = math.sqrt(3) * jnp.abs(t1 - t2) / ell\n", 477 | " return sigma ** 2 * (1 + z) * jnp.exp(-z)\n", 478 | "\n", 479 | "vectorised_cov_func = jax.vmap(jax.vmap(cov_func, \n", 480 | " in_axes=[0, None, None, None]), \n", 481 | " in_axes=[None, 0, None, None])\n", 482 | "\n", 483 | "# or equivalently\n", 484 | "# from functools import partial\n", 485 | "\n", 486 | "# @partial(jax.vmap, in_axes=[None, 0, None, None])\n", 487 | "# @partial(jax.vmap, in_axes=[0, None, None, None])\n", 488 | "# def cov_func(...):\n", 489 | "# ...\n", 490 | "\n", 491 | "ts = jnp.linspace(0.01, 1, 100)\n", 492 | "\n", 493 | "ell, sigma = 0.1, 1.\n", 494 | "cov_matrix = vectorised_cov_func(ts, ts, ell, sigma)\n", 495 | "\n", 496 | "import matplotlib.pyplot as plt\n", 497 | "plt.contourf(cov_matrix, levels=20)" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "metadata": { 503 | "slideshow": { 504 | "slide_type": "slide" 505 | } 506 | }, 507 | "source": [ 508 | "Speed comparison" 509 | ] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "execution_count": null, 514 | "metadata": { 515 | "slideshow": { 516 | "slide_type": "slide" 517 | } 518 | }, 519 | "outputs": [], 520 | "source": [ 521 | "# Numpy implementation 1. Naive implementation with two loops. Do not use this in practice.\n", 522 | "\n", 523 | "def np_cov_func(t1, t2):\n", 524 | " cov_matrix = np.zeros((ts.size, ts.size))\n", 525 | " for i, t1 in enumerate(ts):\n", 526 | " for j, t2 in enumerate(ts):\n", 527 | " z = math.sqrt(3) * np.abs(t1 - t2) / ell\n", 528 | " cov_matrix[i, j] = sigma ** 2 * (1 + z) * np.exp(-z)\n", 529 | " return cov_matrix\n", 530 | "\n", 531 | "ts_np = np.asarray(ts)\n", 532 | "%timeit np_cov_func(ts_np, ts_np)" 533 | ] 534 | }, 535 | { 536 | "cell_type": "code", 537 | "execution_count": null, 538 | "metadata": { 539 | "slideshow": { 540 | "slide_type": "slide" 541 | } 542 | }, 543 | "outputs": [], 544 | "source": [ 545 | "# Numpy implementation 2. Using broadcasting\n", 546 | "# This is applicable only for limited applications, for example this exercise.\n", 547 | "\n", 548 | "def np_cov_func(t1, t2):\n", 549 | " z = math.sqrt(3) * np.abs(t1[:, None] - t2[None, :]) / ell\n", 550 | " return sigma ** 2 * (1 + z) * np.exp(-z)\n", 551 | "\n", 552 | "ts_np = np.asarray(ts)\n", 553 | "%timeit np_cov_func(ts_np, ts_np)" 554 | ] 555 | }, 556 | { 557 | "cell_type": "code", 558 | "execution_count": null, 559 | "metadata": { 560 | "id": "O7gc_9TaHQHU", 561 | "slideshow": { 562 | "slide_type": "slide" 563 | } 564 | }, 565 | "outputs": [], 566 | "source": [ 567 | "# Numpy implementation 3 using scipy cdist\n", 568 | "\n", 569 | "import scipy.spatial\n", 570 | "\n", 571 | "def np_cov_func(t1, t2):\n", 572 | " r = scipy.spatial.distance.cdist(t1, t2, 'euclidean')\n", 573 | " z = math.sqrt(3) * r / ell\n", 574 | " return sigma ** 2 * (1 + z) * np.exp(-z)\n", 575 | "\n", 576 | "ts_np = np.asarray(ts).reshape(-1, 1)\n", 577 | "%timeit np_cov_func(ts_np, ts_np)" 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "execution_count": null, 583 | "metadata": { 584 | "colab": { 585 | "base_uri": "https://localhost:8080/" 586 | }, 587 | "id": "7Y1B-PHSHfWP", 588 | "outputId": "d9ff66d7-1886-4dd2-ffdd-6a83c03d030a", 589 | "slideshow": { 590 | "slide_type": "slide" 591 | } 592 | }, 593 | "outputs": [], 594 | "source": [ 595 | "# In principle we should not jit vmap which is jitted already, but for some reasons\n", 596 | "# the jitted vmap is faster than that of non-jitted in my computer\n", 597 | "f = jax.jit(vectorised_cov_func)\n", 598 | "\n", 599 | "f(ts, ts, ell, sigma)\n", 600 | "\n", 601 | "%timeit f(ts, ts, ell, sigma).block_until_ready()" 602 | ] 603 | }, 604 | { 605 | "cell_type": "code", 606 | "execution_count": null, 607 | "metadata": { 608 | "colab": { 609 | "base_uri": "https://localhost:8080/" 610 | }, 611 | "id": "AUnFWWwjIHBR", 612 | "outputId": "497a4dda-8224-48c1-bc41-968060eb2bee", 613 | "slideshow": { 614 | "slide_type": "fragment" 615 | } 616 | }, 617 | "outputs": [], 618 | "source": [ 619 | "vectorised_cov_func(ts, ts, ell, sigma)\n", 620 | "\n", 621 | "%timeit vectorised_cov_func(ts, ts, ell, sigma).block_until_ready()" 622 | ] 623 | }, 624 | { 625 | "cell_type": "markdown", 626 | "metadata": { 627 | "id": "VoJw4twWukXv", 628 | "slideshow": { 629 | "slide_type": "slide" 630 | } 631 | }, 632 | "source": [ 633 | "There is also `jax.pmap`. This for parallelisation across different devices, for example, multiple GPUs/TPUs. See details https://jax.readthedocs.io/en/latest/jax-101/06-parallelism.html." 634 | ] 635 | } 636 | ], 637 | "metadata": { 638 | "celltoolbar": "Slideshow", 639 | "colab": { 640 | "collapsed_sections": [], 641 | "name": "jax-clinic-vectorisation.ipynb", 642 | "provenance": [] 643 | }, 644 | "kernelspec": { 645 | "display_name": "Python 3 (ipykernel)", 646 | "language": "python", 647 | "name": "python3" 648 | }, 649 | "language_info": { 650 | "codemirror_mode": { 651 | "name": "ipython", 652 | "version": 3 653 | }, 654 | "file_extension": ".py", 655 | "mimetype": "text/x-python", 656 | "name": "python", 657 | "nbconvert_exporter": "python", 658 | "pygments_lexer": "ipython3", 659 | "version": "3.10.4" 660 | } 661 | }, 662 | "nbformat": 4, 663 | "nbformat_minor": 1 664 | } 665 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/5-jax_clinic_autodiff.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "BiopLIrXS4N9", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# Autodiff\n", 13 | "\n", 14 | "1. Gradient\n", 15 | "2. Jacobian\n", 16 | "3. Hessian" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": { 23 | "id": "HhrdRO1aTXrE", 24 | "slideshow": { 25 | "slide_type": "slide" 26 | } 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "import jax\n", 31 | "import jax.numpy as jnp" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": { 37 | "id": "GH6hKoQsTJzl", 38 | "slideshow": { 39 | "slide_type": "fragment" 40 | } 41 | }, 42 | "source": [ 43 | "Consider a function $f\\colon \\mathbb{R}^d \\to \\mathbb{R}$. Jax computes the gradient of $f$ by\n", 44 | "`grad_f = jax.grad(f)`." 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": { 50 | "id": "YzAsN3kDTqZt", 51 | "slideshow": { 52 | "slide_type": "slide" 53 | } 54 | }, 55 | "source": [ 56 | "# Example\n", 57 | "\n", 58 | "$$\n", 59 | "f(x) = x^\\top \\, A \\, x \n", 60 | "$$\n", 61 | "\n", 62 | "compute $\\nabla f$, and compare with the true gradient $2 \\, A \\,x$." 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": { 69 | "colab": { 70 | "base_uri": "https://localhost:8080/" 71 | }, 72 | "id": "KSXjcH7vTW7T", 73 | "outputId": "aa94bca6-ce95-47e4-a564-c328725b7c72", 74 | "slideshow": { 75 | "slide_type": "fragment" 76 | } 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "A = jnp.eye(2)\n", 81 | "\n", 82 | "def f(x):\n", 83 | " return jnp.dot(x, A @ x)\n", 84 | "\n", 85 | "grad_f = jax.grad(f)\n", 86 | "grad_f(jnp.array([1., 2.]))" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": { 92 | "id": "hiAss-5yU8Wg", 93 | "slideshow": { 94 | "slide_type": "slide" 95 | } 96 | }, 97 | "source": [ 98 | "# Exercise\n", 99 | "\n", 100 | "Consider an MLE objective function\n", 101 | "\n", 102 | "$$\n", 103 | "\\ell(\\theta) = x^\\top A^{-1}(\\theta) \\, x,\n", 104 | "$$\n", 105 | "\n", 106 | "where $A(\\theta) = \\begin{bmatrix}2 & \\mathrm{sigmoid}(\\theta) \\\\ \\mathrm{sigmoid}(\\theta) & 3 \\end{bmatrix}$, and $x\\in\\mathbb{R}^2$ is given. Compute $\\nabla \\ell$ at $\\theta=2$, and compare the result to that of the finite difference." 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": { 112 | "id": "hfD5mVbSZ_io", 113 | "slideshow": { 114 | "slide_type": "fragment" 115 | } 116 | }, 117 | "source": [ 118 | "```python\n", 119 | "import jax.scipy\n", 120 | "\n", 121 | "key = ?\n", 122 | "x = jax.random.normal(?)\n", 123 | "\n", 124 | "def sigmoid(theta):\n", 125 | " return ?\n", 126 | "\n", 127 | "def nll(theta):\n", 128 | " A = ?\n", 129 | "\n", 130 | " chol = jax.scipy.linalg.cho_factor(A, lower=True)\n", 131 | " return jnp.dot(x, jax.scipy.linalg.cho_solve(chol, x))\n", 132 | "\n", 133 | "grad_nll = ?\n", 134 | "\n", 135 | "grad_nll(2.)\n", 136 | "\n", 137 | "# compute gradient at 2. using finite difference\n", 138 | "?\n", 139 | "```" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": { 145 | "id": "Y8GQJAxcXyGT", 146 | "slideshow": { 147 | "slide_type": "slide" 148 | } 149 | }, 150 | "source": [ 151 | "## Solution" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": { 158 | "id": "Tez_LpIfUfXk", 159 | "slideshow": { 160 | "slide_type": "fragment" 161 | } 162 | }, 163 | "outputs": [], 164 | "source": [ 165 | "import jax.scipy\n", 166 | "\n", 167 | "key = jax.random.PRNGKey(999)\n", 168 | "x = jax.random.normal(key, (2, ))\n", 169 | "\n", 170 | "def sigmoid(theta):\n", 171 | " return 1 / (1 + jnp.exp(-theta))\n", 172 | "\n", 173 | "def nll(theta):\n", 174 | " A = jnp.array([[2., sigmoid(theta)], \n", 175 | " [sigmoid(theta), 3.]])\n", 176 | " chol = jax.scipy.linalg.cho_factor(A, lower=True)\n", 177 | " return jnp.dot(x, jax.scipy.linalg.cho_solve(chol, x))\n", 178 | "\n", 179 | "grad_nll = jax.grad(nll)" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": { 186 | "colab": { 187 | "base_uri": "https://localhost:8080/" 188 | }, 189 | "id": "NA5DDBZIZBgV", 190 | "outputId": "5a2e70fd-31f0-4f0c-c048-c965604dd272", 191 | "slideshow": { 192 | "slide_type": "fragment" 193 | } 194 | }, 195 | "outputs": [], 196 | "source": [ 197 | "grad_nll(2.)" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": { 204 | "colab": { 205 | "base_uri": "https://localhost:8080/" 206 | }, 207 | "id": "1FG3ZcmjZEqy", 208 | "outputId": "3639dafc-f042-4f7c-94a5-af60ec43255b", 209 | "slideshow": { 210 | "slide_type": "fragment" 211 | } 212 | }, 213 | "outputs": [], 214 | "source": [ 215 | "(nll(2. + 1e-3) - nll(2.)) / 1e-3" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": { 221 | "id": "gVtSLhfx3bpZ", 222 | "slideshow": { 223 | "slide_type": "slide" 224 | } 225 | }, 226 | "source": [ 227 | "# Jacobian and Hessian\n", 228 | "\n", 229 | "Jax computes Jacobian of any function $f\\colon \\mathbb{R}^{n} \\to \\mathbb{R}^m$ by `jax.jacfwd` or `jax.jacrev`. They give the same results but are implemented in different ways." 230 | ] 231 | }, 232 | { 233 | "cell_type": "markdown", 234 | "metadata": { 235 | "slideshow": { 236 | "slide_type": "fragment" 237 | } 238 | }, 239 | "source": [ 240 | "- `jax.jacfwd`. Forward-mode autodiff.\n", 241 | "- `jax.jacrev`. Reverse-mode autodiff." 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": { 247 | "slideshow": { 248 | "slide_type": "fragment" 249 | } 250 | }, 251 | "source": [ 252 | "I am no expert in autodiff, but essentially, we use `jacfwd` when $n \\ll m$ while use `jacrev` when $n \\gg m$ for the best computation speed.\n", 253 | "\n", 254 | "To obtain Hessian of a function, we could use either `jacfwd(jacrev(f))` or `jax.hessian(f)`." 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": { 260 | "id": "V33aIe2P8-he", 261 | "slideshow": { 262 | "slide_type": "slide" 263 | } 264 | }, 265 | "source": [ 266 | "# Example\n", 267 | "\n", 268 | "Take the Jacobian of $\\nabla f$ from the first example, where \n", 269 | "\n", 270 | "$$\n", 271 | "f(x) = x^\\top \\, A \\, x \n", 272 | "$$\n" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": null, 278 | "metadata": { 279 | "colab": { 280 | "base_uri": "https://localhost:8080/" 281 | }, 282 | "id": "QQ0GOsS39Mn3", 283 | "outputId": "e11d7673-1f57-41e4-d517-b84eba4d5244", 284 | "slideshow": { 285 | "slide_type": "fragment" 286 | } 287 | }, 288 | "outputs": [], 289 | "source": [ 290 | "jac_of_gradf = jax.jacfwd(grad_f)\n", 291 | "\n", 292 | "# Evaluate at x=[0.1, 0.2]\n", 293 | "jac_of_gradf(jnp.array([0.1, 0.2]))" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": { 299 | "id": "QZbT2NhA-aUB", 300 | "slideshow": { 301 | "slide_type": "slide" 302 | } 303 | }, 304 | "source": [ 305 | "# Exercise\n", 306 | "\n", 307 | "Consider a simple perceptron\n", 308 | "\n", 309 | "$$\n", 310 | "\\mathrm{NN}(x) = \\mathrm{sigmoid} (w^\\top \\, x + b)\n", 311 | "$$\n", 312 | "\n", 313 | "Compute its gradient and Hessian w.r.t. the weight $w$.\n", 314 | "\n", 315 | "Note: search `jax argnums` for specifying positional argument(s) to differentiate with respect to.\n", 316 | "\n", 317 | "```python\n", 318 | "def sigmoid(x):\n", 319 | " return ?\n", 320 | "\n", 321 | "def nn(x, weights, b):\n", 322 | " \"\"\"This function has three arguments. How to specify to which argument we differentiate?\n", 323 | " \"\"\"\n", 324 | " return ?\n", 325 | "\n", 326 | "key = jax.random.PRNGKey(999)\n", 327 | "ws = jax.random.normal(key, (10, ))\n", 328 | "\n", 329 | "key, _ = jax.random.split(key)\n", 330 | "xs = jax.random.normal(key, (10, ))\n", 331 | "\n", 332 | "grad_of_nn = ?\n", 333 | "hessian_of_nn = ?\n", 334 | "\n", 335 | "print(grad_of_nn(xs, ws, 1.))\n", 336 | "print(hessian_of_nn(xs, ws, 1.))\n", 337 | "```" 338 | ] 339 | }, 340 | { 341 | "cell_type": "markdown", 342 | "metadata": { 343 | "id": "gikOaOAVA2YX", 344 | "slideshow": { 345 | "slide_type": "slide" 346 | } 347 | }, 348 | "source": [ 349 | "## Solution" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": { 356 | "id": "syqgDRGp9jJm", 357 | "slideshow": { 358 | "slide_type": "fragment" 359 | } 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "def sigmoid(x):\n", 364 | " return 1 / (1 + jnp.exp(-x))\n", 365 | "\n", 366 | "def nn(x, weights, b):\n", 367 | " return sigmoid(jnp.dot(x, weights) + b)\n", 368 | "\n", 369 | "key = jax.random.PRNGKey(999)\n", 370 | "ws = jax.random.normal(key, (10, ))\n", 371 | "\n", 372 | "key, _ = jax.random.split(key)\n", 373 | "xs = jax.random.normal(key, (10, ))\n", 374 | "\n", 375 | "grad_of_nn = jax.grad(nn, argnums=[1])\n", 376 | "hessian_of_nn = jax.hessian(nn, argnums=[1])\n", 377 | "\n", 378 | "print(grad_of_nn(xs, ws, 1.))\n", 379 | "print(hessian_of_nn(xs, ws, 1.))" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "metadata": { 385 | "id": "zU5O7rKgBA-z", 386 | "slideshow": { 387 | "slide_type": "slide" 388 | } 389 | }, 390 | "source": [ 391 | "# Jacobian-vector product (JVP), vector-Jacobian product (VJP), and Hessian vector product (HVP)\n", 392 | "\n", 393 | "In a plethora of applications, it is not the Jacobian/Hessian you want to solve, but the Matrix-vector product\n", 394 | "\n", 395 | "$$\n", 396 | "\\mathrm{J} x\n", 397 | "$$\n", 398 | "\n", 399 | "for some Jacobian/Hessian $\\mathrm{J}$ and a vector $x$. For example, a commonly seen operator in SDE/PDE:\n", 400 | "\n", 401 | "$$\n", 402 | "(A \\phi)(x) = \\nabla_x \\phi \\cdot a(x) + \\frac{1}{2} \\mathrm{tr}\\big(\\Gamma(x) \\, \\mathrm{H}_x \\phi\\big),\n", 403 | "$$\n", 404 | "\n", 405 | "Other examples: Gauss--Newton, quasi-Newton methods, and extended Kalman filter ...\n", 406 | "\n", 407 | "\n", 408 | "This ese can be solved efficient using `jax.vjp` and `jax.jvp`. For details, see https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#how-it-s-made-two-foundational-autodiff-functions. " 409 | ] 410 | } 411 | ], 412 | "metadata": { 413 | "celltoolbar": "Slideshow", 414 | "colab": { 415 | "name": "jax-clinic-autodiff.ipynb", 416 | "provenance": [] 417 | }, 418 | "kernelspec": { 419 | "display_name": "Python 3 (ipykernel)", 420 | "language": "python", 421 | "name": "python3" 422 | }, 423 | "language_info": { 424 | "codemirror_mode": { 425 | "name": "ipython", 426 | "version": 3 427 | }, 428 | "file_extension": ".py", 429 | "mimetype": "text/x-python", 430 | "name": "python", 431 | "nbconvert_exporter": "python", 432 | "pygments_lexer": "ipython3", 433 | "version": "3.10.4" 434 | } 435 | }, 436 | "nbformat": 4, 437 | "nbformat_minor": 1 438 | } 439 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/6-jax_clinic_control_flow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "id": "L5ZkzX1TlCOb", 7 | "slideshow": { 8 | "slide_type": "slide" 9 | } 10 | }, 11 | "source": [ 12 | "# Jax control flow\n", 13 | "\n", 14 | "1. If else\n", 15 | "2. Loops" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": null, 21 | "metadata": { 22 | "slideshow": { 23 | "slide_type": "fragment" 24 | } 25 | }, 26 | "outputs": [], 27 | "source": [ 28 | "import jax\n", 29 | "import jax.numpy as jnp\n", 30 | "import math\n", 31 | "import jax\n", 32 | "import jax.numpy as jnp\n", 33 | "import numpy as np\n", 34 | "import matplotlib.pyplot as plt\n", 35 | "import math" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": { 41 | "slideshow": { 42 | "slide_type": "slide" 43 | } 44 | }, 45 | "source": [ 46 | "Does this work?" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": { 53 | "colab": { 54 | "base_uri": "https://localhost:8080/" 55 | }, 56 | "id": "kTwV2ppJk_zC", 57 | "outputId": "2b1a16e9-4083-4217-8e1c-951dbc4e4c63", 58 | "slideshow": { 59 | "slide_type": "fragment" 60 | } 61 | }, 62 | "outputs": [], 63 | "source": [ 64 | "def test_if(x):\n", 65 | " if x < 0:\n", 66 | " return -x\n", 67 | " else:\n", 68 | " return x\n", 69 | "\n", 70 | "test_if(jnp.array(-1.))" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": { 76 | "id": "1Wki2iJ2mbWd", 77 | "slideshow": { 78 | "slide_type": "fragment" 79 | } 80 | }, 81 | "source": [ 82 | "Yes it works. " 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": { 88 | "slideshow": { 89 | "slide_type": "slide" 90 | } 91 | }, 92 | "source": [ 93 | "How about the for loop?" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "colab": { 101 | "base_uri": "https://localhost:8080/" 102 | }, 103 | "id": "jhtHGp-_maqR", 104 | "outputId": "a183d9f0-5860-4489-cfbd-473aac885984", 105 | "slideshow": { 106 | "slide_type": "fragment" 107 | } 108 | }, 109 | "outputs": [], 110 | "source": [ 111 | "def test_for(x):\n", 112 | " for i in range(5):\n", 113 | " x = jnp.eye(2) @ x\n", 114 | " return x\n", 115 | "\n", 116 | "test_for(jnp.ones((2, )))" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": { 122 | "id": "XSF2bbrom1II", 123 | "slideshow": { 124 | "slide_type": "fragment" 125 | } 126 | }, 127 | "source": [ 128 | "Yep, it works tooo!\n", 129 | "\n", 130 | "It **seems** a fuss to worry about using normal Python control flows.\n", 131 | "\n", 132 | "But, if we want to use `jit` or some autodiff features, you have to be careful with control flows." 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": { 138 | "id": "Wp1eEWrqnoSJ", 139 | "slideshow": { 140 | "slide_type": "slide" 141 | } 142 | }, 143 | "source": [ 144 | "# Example\n", 145 | "\n", 146 | "Will this work?" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": { 153 | "id": "hzK2GnIumVI_", 154 | "scrolled": true, 155 | "slideshow": { 156 | "slide_type": "slide" 157 | } 158 | }, 159 | "outputs": [], 160 | "source": [ 161 | "@jax.jit\n", 162 | "def test_if(x):\n", 163 | " if x < 0:\n", 164 | " return -x\n", 165 | " else:\n", 166 | " return x\n", 167 | "\n", 168 | "test_if(jnp.array(-1.))" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": { 174 | "id": "Fn_ZH92so2eg", 175 | "slideshow": { 176 | "slide_type": "fragment" 177 | } 178 | }, 179 | "source": [ 180 | "No. The error message explained it well already. \n", 181 | "\n", 182 | "Basically, it fails because we are building a computational graph that changes based on the **concrete** value of a variable. \n", 183 | "\n", 184 | "We can force this example to work by using `static_argnums` argument for `jit`. But please use it only if this is intentional, that is, `x` is really static." 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": { 190 | "slideshow": { 191 | "slide_type": "slide" 192 | } 193 | }, 194 | "source": [ 195 | "Recall that `jax.jit` needs to trace all the (numerical) operations to compile to the XLA code.\n", 196 | "\n", 197 | "`jax.jit` cannot trace the Python control flows, such as `if else` and `for`. \n", 198 | "\n", 199 | "If the function to be jitted has a for loop, then the operations in the for loop are hardcoded to the XLA programme." 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": { 205 | "slideshow": { 206 | "slide_type": "fragment" 207 | } 208 | }, 209 | "source": [ 210 | "Why? \n", 211 | "\n", 212 | "Imagine we ask you to implement a function\n", 213 | "\n", 214 | "```python\n", 215 | "def my_jit(f):\n", 216 | " return ...\n", 217 | "```\n", 218 | "\n", 219 | "such that `my_jit` takes a function `f` as input and detect whether the function has a `for loop`. This is super difficult, and we might need to print the function `f` as string then semantically search for the `for`." 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": { 225 | "slideshow": { 226 | "slide_type": "slide" 227 | } 228 | }, 229 | "source": [ 230 | "A more illustrative example:\n", 231 | "\n", 232 | "Suppose that JAX can compile a Python code to a C code. How would the compiled C code of\n", 233 | "\n", 234 | "```python\n", 235 | "for i in range(100):\n", 236 | " x = f(x)\n", 237 | "```\n", 238 | "\n", 239 | "look like?" 240 | ] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "metadata": { 245 | "slideshow": { 246 | "slide_type": "fragment" 247 | } 248 | }, 249 | "source": [ 250 | "We expect to get a C code like this:\n", 251 | "\n", 252 | "```c\n", 253 | "for (int i = 0; i < 100; i++) {\n", 254 | " x = f(x);\n", 255 | "}\n", 256 | "```" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": { 262 | "slideshow": { 263 | "slide_type": "fragment" 264 | } 265 | }, 266 | "source": [ 267 | "But actually... we got \n", 268 | "\n", 269 | "```c\n", 270 | "x = f(x);\n", 271 | "x = f(x);\n", 272 | "x = f(x);\n", 273 | "... // hardcore-repeat 100 times\n", 274 | "```" 275 | ] 276 | }, 277 | { 278 | "cell_type": "markdown", 279 | "metadata": { 280 | "id": "S98jgnfsqr3X", 281 | "slideshow": { 282 | "slide_type": "slide" 283 | } 284 | }, 285 | "source": [ 286 | "Hence, if we desire `for/if` in the **runtime**. we need to write something that `jax` could understand/parse, that are, JAX primitives." 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": { 292 | "id": "qMNR_apxrYOj", 293 | "slideshow": { 294 | "slide_type": "slide" 295 | } 296 | }, 297 | "source": [ 298 | "# If else\n", 299 | "\n", 300 | "Consider a Python `if else`\n", 301 | "\n", 302 | "```python\n", 303 | "if condition:\n", 304 | " result = true_func(x)\n", 305 | "else:\n", 306 | " result = false_func(x)\n", 307 | "```\n", 308 | "\n", 309 | "In JAX we write as\n", 310 | "\n", 311 | "```python\n", 312 | "result = jax.lax.cond(condition, \n", 313 | " true_func, \n", 314 | " false_func, \n", 315 | " operand=x)\n", 316 | "```" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": { 322 | "id": "2_dPsVICtXQD", 323 | "slideshow": { 324 | "slide_type": "slide" 325 | } 326 | }, 327 | "source": [ 328 | "Let us implement `test_if` in jax as an example. " 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": null, 334 | "metadata": { 335 | "id": "Hn28vFoQqrSk", 336 | "slideshow": { 337 | "slide_type": "fragment" 338 | } 339 | }, 340 | "outputs": [], 341 | "source": [ 342 | "@jax.jit\n", 343 | "def test_if(x):\n", 344 | " return jax.lax.cond(x < 0., # condition\n", 345 | " lambda _: -x, # what to execution if the condition is true\n", 346 | " lambda _: x, # what to execution if the condition is false\n", 347 | " x) # the operand here can be anything because we used x from outer scope\n", 348 | "\n", 349 | "test_if(jnp.array(-1.))" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": { 356 | "id": "gZs8mrjkthJa", 357 | "slideshow": { 358 | "slide_type": "fragment" 359 | } 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "@jax.jit\n", 364 | "def test_if(x):\n", 365 | " return jax.lax.cond(x < 0., \n", 366 | " lambda u: -u, \n", 367 | " lambda u: u,\n", 368 | " x)\n", 369 | "\n", 370 | "test_if(jnp.array(-1.))" 371 | ] 372 | }, 373 | { 374 | "cell_type": "markdown", 375 | "metadata": { 376 | "id": "3TmIbmGwtpdR", 377 | "slideshow": { 378 | "slide_type": "slide" 379 | } 380 | }, 381 | "source": [ 382 | "# Exercise\n", 383 | "\n", 384 | "Write ELU activation function in jax and jit it.\n", 385 | "\n", 386 | "$$\n", 387 | "\\mathrm{elu}(x) = \\begin{cases}e^x, & x < 0,\\\\\n", 388 | " 1, & x\\geq 0\\end{cases}\n", 389 | "$$\n", 390 | "\n", 391 | "A numpy implementation would be\n", 392 | "\n", 393 | "```python\n", 394 | "@jax.jit\n", 395 | "def elu(x):\n", 396 | " return jax.lax.cond(?,\n", 397 | " ?, \n", 398 | " ?, \n", 399 | " ?)\n", 400 | "\n", 401 | "# test\n", 402 | "elu(1.)\n", 403 | "```" 404 | ] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "metadata": { 409 | "id": "4cKoNEefwf2r", 410 | "slideshow": { 411 | "slide_type": "slide" 412 | } 413 | }, 414 | "source": [ 415 | "## Solution" 416 | ] 417 | }, 418 | { 419 | "cell_type": "code", 420 | "execution_count": null, 421 | "metadata": { 422 | "id": "RHA26n_LtxSf", 423 | "slideshow": { 424 | "slide_type": "fragment" 425 | } 426 | }, 427 | "outputs": [], 428 | "source": [ 429 | "@jax.jit\n", 430 | "def elu(x):\n", 431 | " return jax.lax.cond(x < 0.,\n", 432 | " lambda _: jnp.exp(x), \n", 433 | " lambda _: 1., \n", 434 | " x)\n", 435 | "elu(1.)" 436 | ] 437 | }, 438 | { 439 | "cell_type": "markdown", 440 | "metadata": { 441 | "id": "JfjsA1A_xPn0", 442 | "slideshow": { 443 | "slide_type": "slide" 444 | } 445 | }, 446 | "source": [ 447 | "# ~\n", 448 | "\n", 449 | "1. What if we have multiple if conditions, i.e., `if elif elif ... else`? Use `jax.switch`.\n", 450 | "\n", 451 | "2. What if we have vector input? Use `jnp.where` or `jax.vmap`." 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": { 457 | "id": "_UWxlJiFyEcI", 458 | "slideshow": { 459 | "slide_type": "slide" 460 | } 461 | }, 462 | "source": [ 463 | "# Loops\n", 464 | "\n", 465 | "$$\n", 466 | "\\begin{split}\n", 467 | "\\begin{array} {r|rr} \n", 468 | "\\hline \\\n", 469 | "\\textrm{construct} \n", 470 | "& \\textrm{jit} \n", 471 | "& \\textrm{grad} \\\\\n", 472 | "\\hline \\\n", 473 | "\\textrm{if} & ❌ & ✔ \\\\\n", 474 | "\\textrm{for} & ✔* & ✔\\\\\n", 475 | "\\textrm{while} & ✔* & ✔\\\\\n", 476 | "\\textrm{lax.cond} & ✔ & ✔\\\\\n", 477 | "\\textrm{lax.while_loop} & ✔ & \\textrm{fwd}\\\\\n", 478 | "\\textrm{lax.fori_loop} & ✔ & \\textrm{fwd}\\\\\n", 479 | "\\textrm{lax.scan} & ✔ & ✔\\\\\n", 480 | "\\hline\n", 481 | "\\end{array}\n", 482 | "\\end{split}\n", 483 | "$$\n", 484 | "\n", 485 | "* = argument-value-independent loop condition - unrolls the loop. See, https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html.\n", 486 | "\n", 487 | "Similar to the `jax.lax.cond` we have seen above, for loops, we have `jax.lax.while_loop, fori_loop, scan`." 488 | ] 489 | }, 490 | { 491 | "cell_type": "markdown", 492 | "metadata": { 493 | "id": "7_OHK2bGyBdv", 494 | "slideshow": { 495 | "slide_type": "slide" 496 | } 497 | }, 498 | "source": [ 499 | "Consider a naive numpy implementation of summation:\n", 500 | "\n", 501 | "```python\n", 502 | "def my_sum(x):\n", 503 | " summation = 0.\n", 504 | " for i in range(x.shape[0]):\n", 505 | " summation = summation + x[i]\n", 506 | " return summation\n", 507 | "```" 508 | ] 509 | }, 510 | { 511 | "cell_type": "markdown", 512 | "metadata": { 513 | "id": "D5m9sDilyBfo", 514 | "slideshow": { 515 | "slide_type": "fragment" 516 | } 517 | }, 518 | "source": [ 519 | "The jax implementation of it is" 520 | ] 521 | }, 522 | { 523 | "cell_type": "code", 524 | "execution_count": null, 525 | "metadata": { 526 | "colab": { 527 | "base_uri": "https://localhost:8080/" 528 | }, 529 | "id": "B-pf7HH48q5g", 530 | "outputId": "fcb5ca36-161f-4bad-96d2-35acef98d9c2", 531 | "slideshow": { 532 | "slide_type": "fragment" 533 | } 534 | }, 535 | "outputs": [], 536 | "source": [ 537 | "def my_sum(x):\n", 538 | "\n", 539 | " def body_func(i, val):\n", 540 | " return x[i] + val\n", 541 | "\n", 542 | " return jax.lax.fori_loop(lower=0, # The starting index\n", 543 | " upper=x.shape[0], # The number of loops\n", 544 | " body_fun=body_func, # The loop body (index, previous_val) -> val\n", 545 | " init_val=0.) # Initial value of the loop val\n", 546 | "\n", 547 | "my_sum(jnp.ones((10, )))" 548 | ] 549 | }, 550 | { 551 | "cell_type": "markdown", 552 | "metadata": { 553 | "slideshow": { 554 | "slide_type": "slide" 555 | } 556 | }, 557 | "source": [ 558 | "The compiled function looks like this:" 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": null, 564 | "metadata": { 565 | "id": "65AT1rhrEHOm", 566 | "slideshow": { 567 | "slide_type": "fragment" 568 | } 569 | }, 570 | "outputs": [], 571 | "source": [ 572 | "jax.make_jaxpr(my_sum)(jnp.ones((10, )))" 573 | ] 574 | }, 575 | { 576 | "cell_type": "markdown", 577 | "metadata": { 578 | "id": "Q5dKMSpwEnGv", 579 | "slideshow": { 580 | "slide_type": "slide" 581 | } 582 | }, 583 | "source": [ 584 | "Now that if we don't use the jax language `jax.fori_loop` but simply use Python for loop, what does jax see?" 585 | ] 586 | }, 587 | { 588 | "cell_type": "code", 589 | "execution_count": null, 590 | "metadata": { 591 | "id": "fV7PJsEKEuMa", 592 | "slideshow": { 593 | "slide_type": "fragment" 594 | } 595 | }, 596 | "outputs": [], 597 | "source": [ 598 | "# this loop is jittable indeed, but ...\n", 599 | "def my_sum_naive(x):\n", 600 | " summation = 0.\n", 601 | " for i in range(x.shape[0]):\n", 602 | " summation = summation + x[i]\n", 603 | " return summation\n", 604 | "\n", 605 | "jax.make_jaxpr(my_sum_naive)(jnp.ones((100, )))" 606 | ] 607 | }, 608 | { 609 | "cell_type": "markdown", 610 | "metadata": { 611 | "id": "eopX3e4hDmnT", 612 | "slideshow": { 613 | "slide_type": "slide" 614 | } 615 | }, 616 | "source": [ 617 | "See? Again recall that JAX cannot trace the Python control flows." 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": { 623 | "id": "DhsVwFCaA86X", 624 | "slideshow": { 625 | "slide_type": "slide" 626 | } 627 | }, 628 | "source": [ 629 | "# Exercise\n", 630 | "\n", 631 | "Consider a recursion\n", 632 | "\n", 633 | "$$\n", 634 | "X_k = 0.1 \\, X_{k-1} + U_{k-1}.\n", 635 | "$$\n", 636 | "\n", 637 | "Suppose that the initial $X_0$ and inputs $\\lbrace U_k \\rbrace_{k=0}^{T-1}$ are known. Could you compute $X_{T}$?\n", 638 | "\n", 639 | "This is a simple numpy way to do it:" 640 | ] 641 | }, 642 | { 643 | "cell_type": "code", 644 | "execution_count": null, 645 | "metadata": { 646 | "id": "Epq2zK5poC2u", 647 | "slideshow": { 648 | "slide_type": "fragment" 649 | } 650 | }, 651 | "outputs": [], 652 | "source": [ 653 | "def recursion(x0, us):\n", 654 | " T = us.shape[0]\n", 655 | " x = x0\n", 656 | " for k in range(T):\n", 657 | " x = 0.1 * x + us[k]\n", 658 | " return x\n", 659 | "\n", 660 | "recursion(np.array(0.1), 0.2 * np.ones((10, )))" 661 | ] 662 | }, 663 | { 664 | "cell_type": "markdown", 665 | "metadata": { 666 | "id": "VrXJgs28DP1K", 667 | "slideshow": { 668 | "slide_type": "slide" 669 | } 670 | }, 671 | "source": [ 672 | "$$\n", 673 | "X_k = 0.1 \\, X_{k-1} + U_{k-1}.\n", 674 | "$$\n", 675 | "\n", 676 | "```python\n", 677 | "def recursion(x0, us):\n", 678 | " def fori_body(k, x):\n", 679 | " return ??\n", 680 | "\n", 681 | " return jax.lax.fori_loop(lower=??,\n", 682 | " upper=??,\n", 683 | " body_fun=??,\n", 684 | " init_val=??)\n", 685 | "\n", 686 | "recursion(jnp.array(0.1), 0.2 * jnp.ones((10, )))\n", 687 | "```" 688 | ] 689 | }, 690 | { 691 | "cell_type": "markdown", 692 | "metadata": { 693 | "id": "NO1qaagtDrvW", 694 | "slideshow": { 695 | "slide_type": "slide" 696 | } 697 | }, 698 | "source": [ 699 | "## Solution" 700 | ] 701 | }, 702 | { 703 | "cell_type": "code", 704 | "execution_count": null, 705 | "metadata": { 706 | "id": "R2WMP8NQDtnr", 707 | "slideshow": { 708 | "slide_type": "fragment" 709 | } 710 | }, 711 | "outputs": [], 712 | "source": [ 713 | "def recursion(x0, us):\n", 714 | " def fori_body(k, x):\n", 715 | " return 0.1 * x + us[k]\n", 716 | "\n", 717 | " return jax.lax.fori_loop(lower=0,\n", 718 | " upper=us.shape[0],\n", 719 | " body_fun=fori_body,\n", 720 | " init_val=x0)\n", 721 | "\n", 722 | "recursion(jnp.array(0.1), 0.2 * jnp.ones((10, )))" 723 | ] 724 | }, 725 | { 726 | "cell_type": "markdown", 727 | "metadata": { 728 | "id": "iZJ5Of3hRvbN", 729 | "slideshow": { 730 | "slide_type": "slide" 731 | } 732 | }, 733 | "source": [ 734 | "# ~\n", 735 | "\n", 736 | "Wait a sec, but the function only returns the end value at $T$. How do I keep all the history results?\n", 737 | "\n", 738 | "This is very simple in numpy. Just introduce a result accumulator, say, `xs`." 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": null, 744 | "metadata": { 745 | "id": "WkzB1gtCSDAZ", 746 | "slideshow": { 747 | "slide_type": "fragment" 748 | } 749 | }, 750 | "outputs": [], 751 | "source": [ 752 | "def recursion(x0, us):\n", 753 | " T = us.shape[0]\n", 754 | " xs = np.zeros((T, )) # The accumulator\n", 755 | "\n", 756 | " x = x0\n", 757 | " for k in range(T):\n", 758 | " x = 0.1 * x + us[k]\n", 759 | " xs[k] = x\n", 760 | " return xs\n", 761 | "\n", 762 | "recursion(np.array(0.1), 0.2 * np.ones((10, )))" 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": { 768 | "id": "IdJ7v2HTSlCy", 769 | "slideshow": { 770 | "slide_type": "slide" 771 | } 772 | }, 773 | "source": [ 774 | "Can I do the same in jax?\n", 775 | "\n", 776 | "```python\n", 777 | "@jax.jit\n", 778 | "def recursion(x0, us):\n", 779 | " xs = jnp.zeros((T, ))\n", 780 | "\n", 781 | " def fori_body(k, x):\n", 782 | " x = 0.1 * x + us[k]\n", 783 | " xs[k] = x\n", 784 | " return x\n", 785 | "\n", 786 | " return jax.lax.fori_loop(lower=0,\n", 787 | " upper=us.shape[0],\n", 788 | " body_fun=fori_body,\n", 789 | " init_val=x0)\n", 790 | "```\n", 791 | "\n", 792 | "No. We will get error in the line `xs[k] = x` because jax DeviceArray are immutable (i.e., no assignment).\n", 793 | "\n", 794 | "We can, to some extent, force `xs[k] = x` to work by using \"jax array update\" at the cost of making your programme nasty, slow, and unreadable. " 795 | ] 796 | }, 797 | { 798 | "cell_type": "markdown", 799 | "metadata": { 800 | "id": "Q9bq2pYgTvVX", 801 | "slideshow": { 802 | "slide_type": "slide" 803 | } 804 | }, 805 | "source": [ 806 | "The authentic way to do it is by using the **scan** operation, because \n", 807 | "\n", 808 | "$$\n", 809 | "X_k = 0.1 \\, X_{k-1} + U_{k-1}\n", 810 | "$$\n", 811 | "\n", 812 | "is essentially a scan operation. Think about what the essential parts of such scan loop are, then we can abstract them!" 813 | ] 814 | }, 815 | { 816 | "cell_type": "code", 817 | "execution_count": null, 818 | "metadata": { 819 | "id": "24YBtqQGSK2k", 820 | "slideshow": { 821 | "slide_type": "fragment" 822 | } 823 | }, 824 | "outputs": [], 825 | "source": [ 826 | "def recursion(x0, us):\n", 827 | " def scan_body(carry, elem):\n", 828 | " # Unpack carry and elem\n", 829 | " x = carry\n", 830 | " u = elem\n", 831 | "\n", 832 | " x = 0.1 * x + u\n", 833 | " return x, x # Scan body returns two values. First returns as the next carry, the second goes to the result container.\n", 834 | "\n", 835 | " return jax.lax.scan(scan_body, # The scan body function\n", 836 | " x0, # Initial value/carry\n", 837 | " us) # Inputs\n", 838 | "\n", 839 | "(last_x, xs) = recursion(jnp.array(0.1), 0.2 * jnp.ones((10, )))\n", 840 | "xs" 841 | ] 842 | }, 843 | { 844 | "cell_type": "markdown", 845 | "metadata": { 846 | "id": "m4qMNZ-1Xco5", 847 | "slideshow": { 848 | "slide_type": "slide" 849 | } 850 | }, 851 | "source": [ 852 | "# Exercise\n", 853 | "\n", 854 | "Consider an SDE\n", 855 | "\n", 856 | "$$\n", 857 | "\\mathrm{d} X(t) = \\sin(10 \\, \\pi \\, X(t)) \\, \\mathrm{d} t + \\mathrm{d}W(t),\n", 858 | "$$\n", 859 | "\n", 860 | "where $X(0) = 0.1$. Use Euler--Maruyama to simulate a trajectory of $X$ at times $0.01, 0.02, \\ldots, 1$.\n", 861 | "\n", 862 | "Formula:\n", 863 | "\n", 864 | "$$\n", 865 | "X(t_k) \\approx X(t_{k-1}) + \\sin(10 \\, \\pi \\, X(t_{k-1})) \\, (t_k - t_{k-1}) + \\Delta W_k, \\quad \\Delta W_k \\sim \\mathrm{N}(0, t_k - t_{k-1}).\n", 866 | "$$\n", 867 | "\n", 868 | "```python\n", 869 | "dt = 0.01\n", 870 | "T = 100\n", 871 | "ts = jnp.linspace(dt, dt * T, T)\n", 872 | "\n", 873 | "key = jax.random.PRNGKey(666)\n", 874 | "ws = ? # Generate a Wiener process at ts\n", 875 | "\n", 876 | "\n", 877 | "def scan_body(carry, elem):\n", 878 | " ? = carry\n", 879 | " dw = elem\n", 880 | "\n", 881 | " # Euler equation here?\n", 882 | " return ?, ?\n", 883 | "\n", 884 | "_, xs = jax.lax.scan(?, \n", 885 | " ?, \n", 886 | " ws)\n", 887 | "\n", 888 | "plt.plot(ts, xs)\n", 889 | "```" 890 | ] 891 | }, 892 | { 893 | "cell_type": "markdown", 894 | "metadata": { 895 | "id": "RRiW5u2Fpfee", 896 | "slideshow": { 897 | "slide_type": "slide" 898 | } 899 | }, 900 | "source": [ 901 | "## Solution" 902 | ] 903 | }, 904 | { 905 | "cell_type": "code", 906 | "execution_count": null, 907 | "metadata": { 908 | "id": "wXsTfdccVpJP", 909 | "slideshow": { 910 | "slide_type": "fragment" 911 | } 912 | }, 913 | "outputs": [], 914 | "source": [ 915 | "dt = 0.01\n", 916 | "T = 100\n", 917 | "ts = jnp.linspace(dt, dt * T, T)\n", 918 | "\n", 919 | "key = jax.random.PRNGKey(666)\n", 920 | "ws = jnp.cumsum(math.sqrt(dt) * jax.random.normal(key, (T, ))) # Wiener process at the times\n", 921 | "\n", 922 | "def scan_body(carry, elem):\n", 923 | " x = carry\n", 924 | " dw = elem\n", 925 | "\n", 926 | " x = x + jnp.sin(10 * math.pi * x) * dt + dw\n", 927 | " return x, x\n", 928 | "\n", 929 | "_, xs = jax.lax.scan(scan_body, \n", 930 | " jnp.array(0.1), \n", 931 | " ws)\n", 932 | "\n", 933 | "plt.plot(ts, xs)" 934 | ] 935 | } 936 | ], 937 | "metadata": { 938 | "celltoolbar": "Slideshow", 939 | "colab": { 940 | "collapsed_sections": [], 941 | "name": "jax-clinic-control-flow.ipynb", 942 | "provenance": [] 943 | }, 944 | "kernelspec": { 945 | "display_name": "Python 3 (ipykernel)", 946 | "language": "python", 947 | "name": "python3" 948 | }, 949 | "language_info": { 950 | "codemirror_mode": { 951 | "name": "ipython", 952 | "version": 3 953 | }, 954 | "file_extension": ".py", 955 | "mimetype": "text/x-python", 956 | "name": "python", 957 | "nbconvert_exporter": "python", 958 | "pygments_lexer": "ipython3", 959 | "version": "3.10.4" 960 | } 961 | }, 962 | "nbformat": 4, 963 | "nbformat_minor": 1 964 | } 965 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/README.md: -------------------------------------------------------------------------------- 1 | # JAX workshop 2 | 3 | in "a computational introduction to stochastic differential equations". 4 | 5 | Zheng Zhao 2022 6 | 7 | 8 | # Note 9 | 10 | `rise.css` is taking from https://github.com/damianavila/RISE/blob/master/examples/font-sizes.css and https://github.com/damianavila/RISE/issues/175#issuecomment-400795979. 11 | -------------------------------------------------------------------------------- /lectures/seminar_lectures/zhao_jax_workshop/rise.css: -------------------------------------------------------------------------------- 1 | body.rise-enabled div.inner_cell>div.text_cell_render.rendered_html { 2 | font-size: 80%; 3 | } 4 | 5 | body.rise-enabled div.inner_cell>div.input_area { 6 | font-size: 70%; 7 | } 8 | 9 | body.rise-enabled div.output_subarea.output_text.output_result { 10 | font-size: 70%; 11 | } 12 | body.rise-enabled div.output_subarea.output_text.output_stream.output_stdout { 13 | font-size: 70%; 14 | } 15 | 16 | 17 | /* ---------- code blocks inside markdown 18 | i.e. within ``` lines, or 4-space indented 19 | */ 20 | div.inner_cell>div.text_cell_render.rendered_html>pre { 21 | margin: 0px; 22 | } 23 | 24 | div.inner_cell>div.text_cell_render.rendered_html>pre>code { 25 | font-size: 60%; 26 | } 27 | -------------------------------------------------------------------------------- /projects/README.md: -------------------------------------------------------------------------------- 1 | # Project work 2 | 3 | We will soon update the definition of the project work and exemplify some project topics here. 4 | --------------------------------------------------------------------------------