├── ST2131 ├── .DS_Store ├── ST2131_Cheatsheet.pdf └── ST2131_Cheatsheet.tex ├── CS2040S ├── CS2040S Final.pdf └── CS2040S Midterm Summary.pdf ├── CS3241 ├── CS3241_Midterm.pdf ├── img │ ├── 05_glulookat.png │ ├── 01_colours_rgb.jpeg │ ├── 06_2d-outcode.png │ ├── 06_3d-outcode.png │ ├── 01_colours_cmyk.jpeg │ ├── 02_rendering_pipeline.png │ └── 05_transformation_pipeline.png └── CS3241_Cheatsheet.pdf ├── CS3243 ├── CS3243_Midterm.pdf ├── CS3243_Cheatsheet.pdf └── CS3243_Midterm.tex ├── CS2106 ├── img │ ├── 06_peterson.png │ ├── 08_tanenbaum.png │ ├── 01_onion_model.png │ ├── 14_fat_layout.png │ ├── 14_fat_summary.png │ ├── tut09_vm_algo_a.png │ ├── tut09_vm_algo_b.png │ ├── 08_reader_writer.png │ ├── 02_process_5_state.png │ ├── 12_file_operations.png │ ├── 12_open_file_table.png │ ├── 11_hierarchical_paging.png │ └── 13_magnetic_disk_slice.png └── CS2106_Cheatsheet.pdf ├── CS3230 ├── CS3230_Cheatsheet.pdf └── CS3230_Cheatsheet.tex ├── CS4247 └── CS4247_Cheatsheet.pdf ├── CS4261 └── CS4261_Cheatsheet.pdf ├── MA2104 ├── MA2104_Cheatsheet.pdf └── MA2104_Cheatsheet.tex ├── MA2108 ├── MA2108_Cheatsheet.pdf └── MA2108_Cheatsheet.tex ├── MA3209 ├── MA3209_Cheatsheet.pdf └── MA3209_Cheatsheet.tex ├── ST2132 ├── ST2132_Cheatsheet.pdf └── ST2132_Cheatsheet.tex ├── ST2137 └── ST2137_Cheatsheet.pdf ├── ST3131 ├── ST3131_Cheatsheet.pdf └── ST3131_Cheatsheet.tex ├── ST3236 ├── ST3236_Cheatsheet.pdf └── ST3236_Cheatsheet.tex ├── ST4238 ├── ST4238_Cheatsheet.pdf ├── ST4238_Cheatsheet_Trimmed.pdf └── ST4238_Cheatsheet_Trimmed.tex ├── MA1101R ├── MA1101R_Exercise_Qn_Summary.pdf └── MA1101R_Exercise_Qn_Summary.tex └── README.md /ST2131/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST2131/.DS_Store -------------------------------------------------------------------------------- /CS2040S/CS2040S Final.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2040S/CS2040S Final.pdf -------------------------------------------------------------------------------- /CS3241/CS3241_Midterm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/CS3241_Midterm.pdf -------------------------------------------------------------------------------- /CS3243/CS3243_Midterm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3243/CS3243_Midterm.pdf -------------------------------------------------------------------------------- /CS2106/img/06_peterson.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/06_peterson.png -------------------------------------------------------------------------------- /CS2106/img/08_tanenbaum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/08_tanenbaum.png -------------------------------------------------------------------------------- /CS3241/img/05_glulookat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/05_glulookat.png -------------------------------------------------------------------------------- /CS2106/CS2106_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/CS2106_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS2106/img/01_onion_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/01_onion_model.png -------------------------------------------------------------------------------- /CS2106/img/14_fat_layout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/14_fat_layout.png -------------------------------------------------------------------------------- /CS2106/img/14_fat_summary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/14_fat_summary.png -------------------------------------------------------------------------------- /CS2106/img/tut09_vm_algo_a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/tut09_vm_algo_a.png -------------------------------------------------------------------------------- /CS2106/img/tut09_vm_algo_b.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/tut09_vm_algo_b.png -------------------------------------------------------------------------------- /CS3230/CS3230_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3230/CS3230_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS3241/CS3241_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/CS3241_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS3241/img/01_colours_rgb.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/01_colours_rgb.jpeg -------------------------------------------------------------------------------- /CS3241/img/06_2d-outcode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/06_2d-outcode.png -------------------------------------------------------------------------------- /CS3241/img/06_3d-outcode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/06_3d-outcode.png -------------------------------------------------------------------------------- /CS3243/CS3243_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3243/CS3243_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS4247/CS4247_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS4247/CS4247_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS4261/CS4261_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS4261/CS4261_Cheatsheet.pdf -------------------------------------------------------------------------------- /MA2104/MA2104_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/MA2104/MA2104_Cheatsheet.pdf -------------------------------------------------------------------------------- /MA2108/MA2108_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/MA2108/MA2108_Cheatsheet.pdf -------------------------------------------------------------------------------- /MA3209/MA3209_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/MA3209/MA3209_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST2131/ST2131_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST2131/ST2131_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST2132/ST2132_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST2132/ST2132_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST2137/ST2137_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST2137/ST2137_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST3131/ST3131_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST3131/ST3131_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST3236/ST3236_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST3236/ST3236_Cheatsheet.pdf -------------------------------------------------------------------------------- /ST4238/ST4238_Cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST4238/ST4238_Cheatsheet.pdf -------------------------------------------------------------------------------- /CS2106/img/08_reader_writer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/08_reader_writer.png -------------------------------------------------------------------------------- /CS3241/img/01_colours_cmyk.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/01_colours_cmyk.jpeg -------------------------------------------------------------------------------- /CS2040S/CS2040S Midterm Summary.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2040S/CS2040S Midterm Summary.pdf -------------------------------------------------------------------------------- /CS2106/img/02_process_5_state.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/02_process_5_state.png -------------------------------------------------------------------------------- /CS2106/img/12_file_operations.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/12_file_operations.png -------------------------------------------------------------------------------- /CS2106/img/12_open_file_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/12_open_file_table.png -------------------------------------------------------------------------------- /CS2106/img/11_hierarchical_paging.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/11_hierarchical_paging.png -------------------------------------------------------------------------------- /CS2106/img/13_magnetic_disk_slice.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS2106/img/13_magnetic_disk_slice.png -------------------------------------------------------------------------------- /CS3241/img/02_rendering_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/02_rendering_pipeline.png -------------------------------------------------------------------------------- /ST4238/ST4238_Cheatsheet_Trimmed.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/ST4238/ST4238_Cheatsheet_Trimmed.pdf -------------------------------------------------------------------------------- /MA1101R/MA1101R_Exercise_Qn_Summary.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/MA1101R/MA1101R_Exercise_Qn_Summary.pdf -------------------------------------------------------------------------------- /CS3241/img/05_transformation_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Chen-Yiyang/nus-cheatsheets/HEAD/CS3241/img/05_transformation_pipeline.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | #NUS Cheatsheets 2 | This repo contains cheatsheets and/or summary notes for modules I have taken for Undergraduate at NUS. 3 | 4 | Feel free to take and use, and make contribution. 5 | 6 | If you have noticed any typos or errors, please let me know. Thank you! 7 | 8 | ----- 9 | ## Overview 10 | ### CS2040S Data Structures and Algorithms 11 | - Midterm Summary: A 2-page summary sheet for all the contents coverred for AY20/21 Sem 2 till Midterm. 12 | - Final Summary: A 2-page summary sheet for all the contents coverred for AY20/21 Sem 2. 13 | 14 | 15 | ### CS2106 Introduction to Operating Systems 16 | - Cheatsheet: A detailed summary sheet for Final. 17 | 18 | 19 | ### CS3230 Design and Analysis of Algorithms 20 | - Cheatsheet: A short summary sheet for Final. 21 | 22 | 23 | ### CS3241 Computer Graphics 24 | - Midterm: A detailed summary sheet for Midterm, covering till Chapter 6 Rasterisation. Worked together with [Zihao](https://github.com/9teMare). 25 | - Cheatsheet: A detailed summary sheet for Final, including all content for the module. Worked together with [Zihao](https://github.com/9teMare). 26 | 27 | ### CS3243 Introduction to Artificial Intelligence 28 | - Midterm: A 2-page summary sheet for Midterm. 29 | - Cheatsheet: A detailed summary sheet for Final. 30 | 31 | 32 | ### CS4261 Algorithmic Mechanism Design 33 | - Cheatsheet: A detailed summary sheet for Final. 34 | 35 | ### CS4247 Graphics Rendering Techniques 36 | - Cheatsheet: A detailed summary sheet for Final. 37 | 38 | ### MA1101R Linear Algebra I 39 | - Exercise Question Summary: A compilation of Lemmas / statements left as proving questions in the Exercises of the textbook. 40 | 41 | 42 | ### MA2104 Multivariable Calculus 43 | - Cheatsheet: Summary and formula sheet for Final, worked together with [Ng Wei En](https://github.com/wei2912) for the cheatsheet. 44 | 45 | 46 | ### MA2108 Mathematical Analysis I 47 | - Cheatsheet: A detailed summary sheet for Final, AY23/24 Sem 1. 48 | 49 | ### MA3209 Metric and Topological Spaces 50 | 51 | 52 | ### ST2131 Probability 53 | - Cheatsheet: A compilation of important Theorems and notes from lectures, as well as lemmas / statements from tutorial questions. Also includes common distributions covered in the module. 54 | 55 | 56 | ### ST2132 Mathematical Statistics 57 | - Cheatsheet: A 2-page summary sheet for Final, worked together with [Ng Wei En](https://github.com/weien2912) for the cheatsheet. 58 | 59 | ### ST2137 Statistical Computing and Programming 60 | - Cheatsheet: A short summary sheet for Final, with theory and code usage in R, Python, and SAS. (Hint: remove title to make it fit into 3 pages.) 61 | 62 | 63 | ### ST3131 Regression Analysis 64 | - Updated for Midterm 65 | 66 | ### ST3236 Stochastic Processes I 67 | - Cheatsheet: A short summary sheet for Final. 68 | 69 | ### ST4238 Stochastic Process II 70 | - Cheatsheet: A detailed summary sheet for Final. 71 | - Cheatsheet trimmed: A short 2-page summary sheet for Final. 72 | 73 | ----- 74 | ## Todos 75 | ### CS2106 76 | - [x] Include POSIX Syscalls from Lecture and Lab worksheets. 77 | - [ ] Include POSIX Syscalls for File Operations. 78 | 79 | ### MA2104 80 | - [ ] Include 3d plots for visualisation. 81 | 82 | -------------------------------------------------------------------------------- /MA3209/MA3209_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | %\usepackage{fonts} 21 | \usepackage{multicol,multirow} 22 | \usepackage{spverbatim} 23 | \usepackage{graphicx} 24 | \usepackage{ifthen} 25 | \usepackage[landscape]{geometry} 26 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 27 | \usepackage{booktabs} 28 | \usepackage{fontspec} 29 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 30 | \setsansfont{Fira Sans} 31 | \setmonofont{Inconsolata} 32 | \usepackage{unicode-math} 33 | \setmathfont{TeX Gyre Pagella Math} 34 | \usepackage{microtype} 35 | 36 | \usepackage{empheq} 37 | 38 | % new: 39 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 40 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 41 | } 42 | \makeatother 43 | 44 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 45 | { \geometry{margin=0.4in} } 46 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 47 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | } 50 | \pagestyle{empty} 51 | \makeatletter 52 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 53 | {-1ex plus -.5ex minus -.2ex}% 54 | {0.5ex plus .2ex}%x 55 | {\sffamily\large}} 56 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 57 | {-1explus -.5ex minus -.2ex}% 58 | {0.5ex plus .2ex}% 59 | {\sffamily\normalsize\itshape}} 60 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 61 | {-1ex plus -.5ex minus -.2ex}% 62 | {1ex plus .2ex}% 63 | {\normalfont\small\itshape}} 64 | \makeatother 65 | \setcounter{secnumdepth}{0} 66 | \setlength{\parindent}{0pt} 67 | \setlength{\parskip}{0pt plus 0.5ex} 68 | % ----------------------------------------------------------------------- 69 | 70 | \usepackage{academicons} 71 | 72 | \begin{document} 73 | 74 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 75 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 76 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 77 | 78 | \everymath{\color{mathBlue}} 79 | \everydisplay{\color{mathBlue}} 80 | 81 | % for vector notation in this module 82 | \newcommand{\vect}[1]{\boldsymbol{#1}} 83 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 84 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 85 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 86 | 87 | \footnotesize 88 | %\raggedright 89 | 90 | \begin{center} 91 | {\huge\sffamily\bfseries MA3209 Cheatsheet} \huge\bfseries\\ 92 | by Yiyang, AY22/23 93 | \end{center} 94 | \setlength{\premulticols}{0pt} 95 | \setlength{\postmulticols}{0pt} 96 | \setlength{\multicolsep}{1pt} 97 | \setlength{\columnsep}{1.8em} 98 | \begin{multicols}{3} 99 | 100 | 101 | % ----------------------------------------------------------------------- 102 | 103 | \section{Metric Space} 104 | 105 | \end{multicols} 106 | \end{document} 107 | -------------------------------------------------------------------------------- /CS3230/CS3230_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | %\usepackage{fonts} 21 | \usepackage{multicol,multirow} 22 | \usepackage{spverbatim} 23 | \usepackage{graphicx} 24 | \usepackage{ifthen} 25 | \usepackage[landscape]{geometry} 26 | \usepackage{listings} % for code block 27 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 28 | \usepackage{booktabs} 29 | \usepackage{fontspec} 30 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 31 | \setsansfont{Fira Sans} 32 | \setmonofont{Inconsolata} 33 | \usepackage{unicode-math} 34 | \setmathfont{TeX Gyre Pagella Math} 35 | \usepackage{microtype} 36 | 37 | \usepackage{empheq} 38 | 39 | % new: 40 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 41 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 42 | } 43 | \makeatother 44 | 45 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 46 | { \geometry{margin=0.4in} } 47 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 50 | } 51 | \pagestyle{empty} 52 | \makeatletter 53 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 54 | {-1ex plus -.5ex minus -.2ex}% 55 | {0.5ex plus .2ex}%x 56 | {\sffamily\large}} 57 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 58 | {-1explus -.5ex minus -.2ex}% 59 | {0.5ex plus .2ex}% 60 | {\sffamily\normalsize\itshape}} 61 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 62 | {-1ex plus -.5ex minus -.2ex}% 63 | {1ex plus .2ex}% 64 | {\normalfont\small\itshape}} 65 | \makeatother 66 | \setcounter{secnumdepth}{0} 67 | \setlength{\parindent}{0pt} 68 | \setlength{\parskip}{0pt plus 0.5ex} 69 | % ----------------------------------------------------------------------- 70 | 71 | \usepackage{academicons} 72 | 73 | \begin{document} 74 | 75 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 76 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 77 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 78 | \everymath{\color{mathBlue}} 79 | \everydisplay{\color{mathBlue}} 80 | 81 | 82 | % Based on: 83 | % https://stackoverflow.com/questions/3175105/inserting-code-in-this-latex-document-with-indentation 84 | \definecolor{codedkgreen}{rgb}{0,0.6,0} 85 | \definecolor{codegray}{rgb}{0.5,0.5,0.5} 86 | \definecolor{codemauve}{rgb}{0.58,0,0.82} 87 | \definecolor{light-gray}{gray}{0.95} 88 | 89 | \lstset{frame=tb, 90 | language=Java, 91 | backgroundcolor=\color{light-gray}, 92 | basicstyle={\footnotesize\ttfamily}, 93 | tabsize=3 94 | } 95 | 96 | 97 | % for vector notation in this module 98 | \newcommand{\vect}[1]{\boldsymbol{#1}} 99 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 100 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 101 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 102 | 103 | \footnotesize 104 | %\raggedright 105 | 106 | \graphicspath{ {./img/} } 107 | 108 | 109 | \begin{center} 110 | {\huge\sffamily\bfseries CS3230 Cheatsheet} \huge\bfseries\\ 111 | by Yiyang, AY21/22 112 | \end{center} 113 | \setlength{\premulticols}{0pt} 114 | \setlength{\postmulticols}{0pt} 115 | \setlength{\multicolsep}{1pt} 116 | \setlength{\columnsep}{1.8em} 117 | \begin{multicols}{3} 118 | 119 | 120 | % ----------------------------------------------------------------------- 121 | \section{Asymptotic Analysis} 122 | \subsection{Asymptotic Notations} 123 | \deff{Big-O}: $f(n) = O(g(n))$ if there exists $c > 0$ and $n_0 > 0$, s.t., for all $n \geq n_0$ 124 | \[ 125 | 0 \leq f(n) \leq cg(n) 126 | \] 127 | Similar definition for \deff{Big-Omega}, $\Omega()$. 128 | \\ 129 | \deff{Small-o}: $f(n) = o(g(n))$ if there exists $c > 0$ and $n_0 > 0$, s.t., for all $n \geq n_0$x 130 | \[ 131 | 0 \leq f(n) < cg(n) 132 | \] 133 | Similar definition for \deff{Small-omega}, $\omega()$. 134 | \\ 135 | Lastly, $f(n) = \Theta (g(n))$ iff. $f(n) = O(g(n)) \land f(n) = \Omega(g(n))$. 136 | 137 | \section{Solve Recurrence Relations} 138 | \subsection{Master Theorem} 139 | For recurrence in the form of 140 | \[ 141 | T(n) = aT(n/b) + f(n) 142 | \] 143 | there are three cases to be considered 144 | \begin{itemize} 145 | \item If $f(n) = O(n^{\log_b a - \epsilon})$ for some $\epsilon > 0$, then $T(n) = \Theta( n^{\log_b a})$ 146 | \item If $f(n) = \Theta( n^{\log_b a} \lg^k n )$ for some $k > 0$, then $T(n) = \Theta( n^{\log_b a} \lg^{k+1} n)$ 147 | \item If $f(n) = \Omega( n^{\log_b a + \epsilon} )$ for some $\epsilon > 0$ and it satisfies the \deff{regularity condition} that $af(n/b) \leq cf(n)$ for some $0 < c < 1$, then $T(n) = \Theta(f(n))$ 148 | \end{itemize} 149 | \textbf{Notes}: The three cases mean whether $f(n)$ grows \textbf{polynomially} slower, around the same rate, or faster than $n^{\log _b a}$. 150 | \\ 151 | \textbf{Notes}: The regularity condition ensures the sum of sub-problems is less than $f(n)$. 152 | 153 | \subsection{Stirling's Approximation} 154 | $n! \approx \sqrt{2\pi n} \big( \frac{n}{e} \big)^n$, or asymptotically, $\lg(n!) = \theta(n \lg n)$. 155 | 156 | \subsection{Harmonic Sequence} 157 | $H_n = \sum_{i=1}^{n} \frac{1}{i} = \ln n + O(1)$ 158 | 159 | \subsection{Other Important Asymptotic Statements} 160 | $\lg n = O(n^\alpha)$ for any $\alpha > 0$. 161 | \\ 162 | $x^\alpha = O(e^x)$ for any $\alpha > 0$. 163 | 164 | \subsection{Common Recurrence Relations} 165 | \citeqn{(AY18/19Sem2 Midterm Qn2a)} $T(n) = 2T(n/2) + n \lg \lg n = \Theta( n \lg n \lg \lg n)$ 166 | \\ 167 | \citeqn{(AY19/20Sem2 Midterm Qn2d)} $T(n) = 4T(n/4) + n \lg \lg \lg n = \Theta( n \lg n \lg \lg \lg n)$ 168 | \\ 169 | \citeqn{(AY20/21Sem2 Midterm Qn2b)} $T(n) = 27T(n/3) + n^3 / \lg^2 n = \Theta( n^3)$ 170 | 171 | 172 | 173 | 174 | %--- 175 | 176 | 177 | 178 | 179 | 180 | \section{Hashing \& Fingerprint} 181 | 182 | 183 | 184 | 185 | \section{Amortised Analysis} 186 | There are 3 methods in Amortised Analysis. 187 | 188 | \smallskip 189 | 190 | \deff{Aggregate Method} 191 | 192 | \smallskip 193 | 194 | \deff{Accounting Method} \textasciitilde The amortised cost provides an upper bound for the true total cost all the time. \underline{Note}: The "credit saved" is associated with each of the elements operated, not overall. 195 | 196 | \smallskip 197 | 198 | \deff{Potential Method} - Define potential function $\phi(i)$ potential after $i$-th operation. Then it follows: 199 | \begin{itemize} 200 | \item $\phi(0) = 0$ and $\phi(i) \ge 0$ for all $i$ 201 | \item $c(i) = t(i) + \triangle \phi(i)$ relation between amortised cost, $c(i)$, and actual cost $t(i)$, of $i$-th operation 202 | \end{itemize} 203 | 204 | 205 | 206 | 207 | \section{Reduction \& Intractability} 208 | \subsection{Cook-Levin Theorem} 209 | Any problem in NP poly-time reduces to 3-SAT. Hence 3-SAT is NP-Hard and NP-Complete. 210 | 211 | 212 | 213 | \noindent\rule{8cm}{0.4pt} 214 | 215 | \section{Summary} 216 | \subsection{Proving Techniques} 217 | \subsubsection{Working for Accounting Method} 218 | First claim the (amortised cost) we charge each operation with. Secondly, for each operation, we compare the charge and actual cost of the operation, and claim if there is any credit left or it uses credit from other operations and how. 219 | 220 | \subsubsection{Prove Correctness of DP Solutions} 221 | A typical \textbf{state transition equation} is a composition equation for different cases, prove each case separately. 222 | \\ 223 | Use \deff{Cut-and-Paste} arguments: If current solution $S$ is not optimal but another $S'$ is, then modify $S'$ using ideas from current solution expression to create an "even better" one, thus contradicting optimality of $S'$. 224 | 225 | \subsubsection{Prove Greedy Optional Substructure} 226 | To prove that $S$ with element $i$ is an optimal solution 227 | \begin{enumerate} 228 | \item Claim: $S - \{i\}$ is optimal for sub-problem with $i$ removed 229 | \item Use \textbf{Cut-and-Paste}: If $T$ not $S - \{i\}$ optimal for sub-problem, then $T \cap \{i\}$ better than $S$ for original problem. 230 | \end{enumerate} 231 | 232 | \subsubsection{Prove NP-Hardness} 233 | To show a problem $X$ is \textbf{NP-Hard}, choose a NP-Hard problem $A$, then show there is a poly-time reduction \textbf{from $A$ to $X$}, which is 234 | \begin{enumerate} 235 | \item Any problem instance of $A$ given can be converted to an instance of $X$ in poly-time 236 | \item A YES-instance to $A$ is a YES-instance to $X$ 237 | \item A YES-instance to $X$ is a YES-instance to $A$ 238 | \end{enumerate} 239 | 240 | \subsubsection{Prove NP-Completeness} 241 | To show a problem $X$ is \textbf{NP-Complete}, show 1) $X$ is \textbf{NP}, i.e. it can be represented using a poly-time certifier and verified in poly-time, and then 2) $X$ is \textbf{NP-Hard}. 242 | 243 | \subsection{Common NP-Complete Problems} 244 | \underline{Note}: Unless otherwise stated, the descriptions below are for the decision problem version of each of the problems. 245 | 246 | \subsubsection{INDEPENDENT-SET} 247 | Given a graph $G = (V, E)$ and an integer $k$, is there a subset of $k$ or more \textbf{vertices} such that no two are adjacent? 248 | 249 | \subsubsection{VERTEX-COVER} 250 | Given a graph $G = (V, E)$ and an integer $k$, is there a subset, $S$, of $k$ or less \textbf{vertices} such that each edge is incident to at least one vertex in $S$? 251 | 252 | \subsubsection{SET-COVER} 253 | Given integers $k$ and $n$, and a collection $\mathcal{S}$ of subsets of $\{1, 2, ..., n \}$, are there $k$ or less of these subsets whose union equals $\{1, 2, ..., n \}$? 254 | 255 | \subsubsection{3-SAT} 256 | Given a CNF (Conjunctive Normal Form) formula $\Phi$ where each clause contains exactly 3 literals (not necessarily distinct), does $\Phi$ have a satisfying truth assignment? 257 | 258 | \subsubsection{PARTITION} 259 | Given a collection $\mathcal{S}$ of non-negative integers $x_1, ..., x_n$, is it possible to partition $\mathcal{S}$ into 2 parts with equal sum? 260 | 261 | \subsubsection{KNAPSACK} 262 | Given non-negative integer pairs $(w_1, v_1), ..., (w_n, v_n)$ describing $n$ items, capacity $W$ and value threshold $V$, is there a subset of items with total weight at most $W$ and total value at least $V$? 263 | 264 | \subsubsection{LONG-SIMPLE-PATH} 265 | Given an unweighted directed graph $G = (V, E)$, a positive integer $l$ and two vertices $v, u \in V$, is there a simple path (a path with no repeated vertices) in $G$ from $u$ to $v$ of length at least $l$? 266 | 267 | \subsubsection{LONG-SIMPLE-CYCLE} 268 | Given an unweighted directed graph $G = (V, E)$ and a positive integer $l$, is there a simple cycle (a cycle with no repeated vertices) in $G$ on at least $l$ vertices? 269 | 270 | \subsubsection{CLIQUE} 271 | Given a graph $G = (V, E)$ and a positive integer $k$, is there a clique ($U \subseteq V$ s.t. $forall u \neq v \in U, (u,v) \in E$) of size at least $k$? 272 | 273 | \subsubsection{HAMILTONIAN CYCLE} 274 | Given a graph $G = (V, E)$, is there a path that visits each vertex in the graph exactly once? 275 | 276 | \end{multicols} 277 | \end{document} 278 | -------------------------------------------------------------------------------- /MA1101R/MA1101R_Exercise_Qn_Summary.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb,amsmath,amsthm,amsfonts} 17 | \usepackage{multicol,multirow} 18 | \usepackage{spverbatim} 19 | \usepackage{graphicx} 20 | \usepackage{ifthen} 21 | \usepackage[landscape]{geometry} 22 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 23 | \usepackage{booktabs} 24 | \usepackage{fontspec} 25 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 26 | \setsansfont{Fira Sans} 27 | \setmonofont{Inconsolata} 28 | \usepackage{unicode-math} 29 | \setmathfont{TeX Gyre Pagella Math} 30 | \usepackage{microtype} 31 | % new: 32 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 33 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 34 | } 35 | \makeatother 36 | 37 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 38 | { \geometry{margin=0.4in} } 39 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 40 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 41 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 42 | } 43 | \pagestyle{empty} 44 | \makeatletter 45 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 46 | {-1ex plus -.5ex minus -.2ex}% 47 | {0.5ex plus .2ex}%x 48 | {\sffamily\large}} 49 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 50 | {-1explus -.5ex minus -.2ex}% 51 | {0.5ex plus .2ex}% 52 | {\sffamily\normalsize\itshape}} 53 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 54 | {-1ex plus -.5ex minus -.2ex}% 55 | {1ex plus .2ex}% 56 | {\normalfont\small\itshape}} 57 | \makeatother 58 | \setcounter{secnumdepth}{0} 59 | \setlength{\parindent}{0pt} 60 | \setlength{\parskip}{0pt plus 0.5ex} 61 | % ----------------------------------------------------------------------- 62 | 63 | \usepackage{academicons} 64 | 65 | 66 | \begin{document} 67 | \footnotesize 68 | %\raggedright 69 | 70 | \begin{center} 71 | {\huge\sffamily\bfseries MA1101R Exercise Question Cheatsheet} \huge\bfseries\\ 72 | by Yiyang, AY20/21 73 | \end{center} 74 | \setlength{\premulticols}{0pt} 75 | \setlength{\postmulticols}{0pt} 76 | \setlength{\multicolsep}{1pt} 77 | \setlength{\columnsep}{1.8em} 78 | \begin{multicols}{3} 79 | 80 | \section{Chapter 2 - Matrices} 81 | 82 | \subsection{Ex2Qn9} 83 | Suppose the homogeneous system $A \pmb x = \pmb 0$ has non-trivial solution. Then the linear system $A \pmb x = \pmb b$ has either no solution or infinitely many solution. 84 | 85 | \subsection{Ex2Qn11(e)} 86 | There are no square matrices $A$ and $B$ of same order such that $AB - BA = I$. 87 | 88 | \subsection{Ex2Qn23 (Block matrix multiplication)} 89 | Let $A$ be an $m \times n$ matrix, 90 | \begin{itemize} 91 | \item For matrices $B_1$ and $B_2$ of size $n \times p$ and $n \times q$ respectively, 92 | $$ 93 | A(B_1 \ B_2) = (AB_1 \ AB_2) 94 | $$ 95 | \item For matrices $D_1$ and $D_2$ of size $p \times m$ and $q \times m$ respectively, 96 | $$ 97 | \begin{pmatrix} 98 | D_1 \\ 99 | D_2 100 | \end{pmatrix} A = \begin{pmatrix} 101 | D_1 A\\ 102 | D_2 A 103 | \end{pmatrix} 104 | $$ 105 | \end{itemize} 106 | 107 | \subsection{Ex2Qn60} 108 | Suppose A is an invertible matrix, then $adj(A)$ is invertible. 109 | 110 | 111 | 112 | \section{Chapter 3 - Vector Space} 113 | 114 | \subsection{Ex3Qn30} 115 | Let $u_1, u_2, ..., u_k$ be vectors in $\mathbb{R^n}$ and $P$ a square matrix of order $n$, 116 | \begin{itemize} 117 | \item If $Pu_1, Pu_2, ..., Pu_k$ are linearly independent, then $u_1, u_2, ..., u_k$ are linearly independent. 118 | \item If $u_1, u_2, ..., u_k$ are linearly independent, \emph{and $P$ is invertible}, then $Pu_1, Pu_2, ..., Pu_k$ are linearly independent. 119 | \end{itemize} 120 | 121 | 122 | \subsection{Ex3Qn41} 123 | Let $V$ be a vector space, 124 | \begin{itemize} 125 | \item suppose $S$ is a finite subset of $V$ such that $span(S) = V$, then there exists a subset $S'$ such that $S'$ is a basis for V. 126 | \item suppose $T$ is a finite subset of $V$ such that $T$ is linearly independent, then there exists a basis $T^*$ for $V$ such that $T \subseteq T^*$ 127 | \end{itemize} 128 | 129 | \subsection{Ex3Qn43} 130 | Let $V, W$ be two subspaces of $\mathbb{R^n}$, then 131 | $$ 132 | \text{dim}(V+W) = \text{dim}(V) + \text{dim}(W) - \text{dim}(V \cap W) 133 | $$ 134 | 135 | \subsection{Ex3Qn45} 136 | Let $V, W$ be two subspaces of a given vector space, 137 | \begin{itemize} 138 | \item there exists a basis $S_1$ for $V$ and a basis $S_2$ for $W$, such that $S_1 \cap S_2$ is a basis for $V \cap W$. 139 | \item there exists a basis $T_1$ for $V$ and a basis $T_2$ for $W$, such that $T_1 \cup T_2$ is a basis for $V + W$. 140 | \end{itemize} 141 | 142 | 143 | 144 | 145 | \section{Chapter 4 - Rank \& Nullity} 146 | 147 | \subsection{Ex4Qn20} 148 | Suppose $A$ and $B$ are two matrices such that $AB=\pmb0$, then column space of $B$ is contained in the nullspace of $A$. 149 | 150 | \subsection{Ex4Qn21} 151 | There is no matrix whose row space and nullspace both contain the same nonzero vector. 152 | 153 | \subsection{Ex4Qn22} 154 | Let $A$ be an $m \times n$ matrix and $P$ an $m \times m$ matrix. If $P$ is invertible, then $\text{rank}(PA) = \text{rank}(A)$. 155 | (The inverse is not true) 156 | 157 | \subsection{Ex4Qn23} 158 | For two matrices $A, B$ of the same size, 159 | $$ 160 | \text{rank}(A+B) \leq \text{rank}(A) + \text{rank}(B) 161 | $$ 162 | 163 | \subsection{Ex4Qn24} 164 | Let $A$ be an $m \times n$ matrix. Suppose the linear system $A \pmb x = \pmb b$ is consistent for all $\pmb b \in \mathbb{R^n}$, then the linear system $A^T \pmb y = \pmb 0$ has only the trivial solution. 165 | 166 | \subsection{Ex4Qn25} 167 | For a matrix $A$ of size $m \times n$, 168 | \begin{itemize} 169 | \item nullspace of $A$ is equal to nullspace of $A^T A$ 170 | \item $\text{nullity}(A) = \text{nullity}(A^T A)$ 171 | \item $\text{rank}(A) = \text{rank}(A^T A)$ 172 | \item $\text{rank}(A) = \text{rank}(A A^T)$ 173 | \end{itemize} 174 | (However, $\text{nullity}(A) \neq \text{nullity}(AA^T)$) 175 | 176 | 177 | 178 | 179 | 180 | \section{Chapter 5 - Orthogonality} 181 | 182 | \subsection{Ex5Qn9} 183 | Let $\{ u_1, u_2, ..., u_n \}$ be an orthogonal set of vectors in a vector space, then 184 | $$ 185 | \lVert u_1 + u_2 + ... + u_n \rVert^2 = \lVert u_1 \rVert^2 + \lVert u_2 \rVert^2 + ... + \lVert u_n \rVert^2 186 | $$ 187 | 188 | \subsection{Ex5Qn18 Uniqueness of (Orthogonal) Projection} 189 | Let $V$ be a subspace of $\mathbb{R^n}$ and $\pmb u$ a vector in $\mathbb{R^n}$. $\pmb u$ can written uniquely as $\pmb u = \pmb n + \pmb p$ such that $\pmb n$ is a vector orthogonal to $V$ and $\pmb p$ a vector in $V$. 190 | 191 | \subsection{Ex5Qn19} 192 | Let $A$ be a square matrix of order $n$ such that $A^2 = A^T = A$, 193 | \begin{itemize} 194 | \item for any two vectors $\pmb u, \pmb v \in \mathbb{R^n}$, $(A \pmb u) \cdot \pmb v = \pmb u \cdot (A \pmb v)$ 195 | \item for any vector $\pmb w \in \mathbb{R^n}$, $A \pmb w$ is the projection of $\pmb w$ onto the subspace $V = \{ \pmb u \in \mathbb{R^n} | A \pmb u = \pmb u \}$ of $\mathbb{R^n}$ 196 | \end{itemize} 197 | 198 | 199 | \subsection{Ex5Qn32} 200 | Let $A$ be an orthogonal matrix of order $n$ and let $\pmb u, \pmb v$ be any two vectors in $\mathbb{R^n}$, 201 | \begin{itemize} 202 | \item $\lVert \pmb u \rVert = \lVert A \pmb u \rVert$ 203 | \item $d(\pmb u, \pmb v) = d(A \pmb u, A \pmb v)$ 204 | \item the angle between $\pmb u$ and $\pmb v$ is equal to the angle between $A \pmb u$ and $A \pmb v$ 205 | \end{itemize} 206 | 207 | \subsection{Ex5Qn33} 208 | Let $A$ be an orthogonal matrix of order $n$ and $S = \{\pmb u_1, \pmb u_2, ..., \pmb u_n \}$ be a basis for $\mathbb{R^n}$ 209 | \begin{itemize} 210 | \item $T = \{A \pmb u_1, A \pmb u_2, ..., A \pmb u_n \}$ is a basis for $\mathbb{R^n}$ 211 | \item $S$ is orthogonal $\to$ $T$ is orthogonal 212 | \item $S$ is orthonormal $\to$ $T$ is orthonormal 213 | \end{itemize} 214 | 215 | 216 | \section{Chapter 6 - Diagonalisation} 217 | 218 | \subsection{Ex6Qn3} 219 | Let $\lambda$ be an eigenvalue of a square matrix $A$, 220 | \begin{itemize} 221 | \item $\lambda^n$ is an eigenvalue of $A^n$ for any $n \in \mathbb{Z_{\ge 1}}$ 222 | \item $\frac 1 \lambda$ is an eigenvalue for $A^{-1}$ if $A$ is invertible 223 | \item $\lambda$ is an eigenvalue for $A^T$ 224 | \end{itemize} 225 | 226 | \subsection{Ex6Qn4} 227 | Let $A$ be a square matrix such that $A^2 = A$. If $\lambda$ is an eigenvalue of $A$, then $\lambda = 0$ or $1$. 228 | 229 | \subsection{Ex6Qn16} 230 | Let $A$ be a stochastic matrix, 231 | \begin{itemize} 232 | \item $1$ is an eigenvalue of $A$, 233 | \item if $\lambda$ is an eigenvalue of $A$, then $|\lambda| \le 1$. 234 | \end{itemize} 235 | 236 | (A stochastic matrix $(a_{i_j})_{m \times n}$ is one where all entries are non-negative and sum of entries of each column is 1, i.e. $a_{1i} + a_{2i} + ... + a_{ni} = 1$, for all $i = 1, 2, ..., n$) 237 | 238 | 239 | \subsection{Ex6Qn25} 240 | Let $\pmb u$ be a column matrix, then $I - \pmb u \pmb u^T$ is orthogonally diagonablisable. 241 | 242 | \subsection{Ex6Qn26} 243 | Let $A$ be a symmetry matrix. If $\pmb u, \pmb v$ are two eigenvalues of $A$ associated with different eigenvalues, then $\pmb u \cdot \pmb v = 0$. 244 | 245 | \subsection{Ex6Qn30} 246 | For two orthogonally diagonalisable matrices of same order, $A, B$, $A+B$ is orthogonally diagonalisable . 247 | (However, $AB$ might not be.) 248 | 249 | 250 | 251 | 252 | \section{Chapter 7 - Linear Transformations} 253 | 254 | \subsection{Ex7Qn8} 255 | Let $T : \mathbb{R^n} \to \mathbb{R^n}$ be a linear transformation such that $T \circ T = T$, 256 | \begin{itemize} 257 | \item if $T$ is not the zero transformation, then there exits a nonzero vector $u \in \mathbb{R^n}$ such that $T(\pmb u) = \pmb u$ 258 | \item if $T$ is no the identity transformation, then there exists a nonzero vector $\pmb v \in \mathbb{R^n}$ such that $T(\pmb v) = 0$ 259 | \end{itemize} 260 | 261 | \subsection{Ex7Qn10} 262 | A linear operator $T$ on $\mathbb{R^n}$ is called isometry if $\lVert T(\pmb u) \rVert = \lVert \pmb u \rVert$ for all $\pmb u \in \mathbb{R^n}$. 263 | \begin{itemize} 264 | \item if $T$ is an isometry on $\mathbb{R^n}$, then $T(\pmb u) \cdot T(\pmb v) = \pmb u \cdot \pmb v$ for all $\pmb u, \pmb v \in \mathbb{R^n}$ 265 | \item let $A$ be the standard matrix for a linear operator $T$. $T$ is isometry iff. $A$ is an orthogonal matrix. 266 | \end{itemize} 267 | 268 | 269 | \subsection{Ex7Qn16} 270 | Let $T : \mathbb{R^n} \to \mathbb{R^n}$ be a linear transformation. $\text{Ker}(T) = \{ \pmb 0 \}$ iff T is one-to-one (i.e. $\forall \ \pmb u, \pmb v \in \mathbb{R^n}$, $\pmb u \ne \pmb v \to T(\pmb u) \ne T(\pmb v)$) 271 | 272 | 273 | \subsection{Ex7Qn17} 274 | Let $S : \mathbb{R^n} \to \mathbb{R^m}$ and $T : \mathbb{R^m} \to \mathbb{R^k}$ be linear transformations, 275 | \begin{itemize} 276 | \item $\text{Ker}(S) \subseteq \text{Ker}(T \circ S)$ 277 | \item $\text{R}(T \circ S) \subseteq \text{R}(T)$ 278 | \end{itemize} 279 | 280 | 281 | 282 | 283 | 284 | 285 | 286 | 287 | \end{multicols} 288 | \end{document} 289 | -------------------------------------------------------------------------------- /ST3131/ST3131_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | \usepackage{bbm} % for mathbb-ed digits 21 | %\usepackage{fonts} 22 | \usepackage{multicol,multirow} 23 | \usepackage{spverbatim} 24 | \usepackage{graphicx} 25 | \usepackage{ifthen} 26 | \usepackage[landscape]{geometry} 27 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 28 | \usepackage{booktabs} 29 | \usepackage{fontspec} 30 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 31 | \setsansfont{Fira Sans} 32 | \setmonofont{Inconsolata} 33 | \usepackage{unicode-math} 34 | \setmathfont{TeX Gyre Pagella Math} 35 | \usepackage{microtype} 36 | 37 | \usepackage{empheq} 38 | 39 | % new: 40 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 41 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 42 | } 43 | \makeatother 44 | 45 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 46 | { \geometry{margin=0.4in} } 47 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 50 | } 51 | \pagestyle{empty} 52 | \makeatletter 53 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 54 | {-1ex plus -.5ex minus -.2ex}% 55 | {0.5ex plus .2ex}%x 56 | {\sffamily\large}} 57 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 58 | {-1explus -.5ex minus -.2ex}% 59 | {0.5ex plus .2ex}% 60 | {\sffamily\normalsize\itshape}} 61 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 62 | {-1ex plus -.5ex minus -.2ex}% 63 | {1ex plus .2ex}% 64 | {\normalfont\small\itshape}} 65 | \makeatother 66 | \setcounter{secnumdepth}{0} 67 | \setlength{\parindent}{0pt} 68 | \setlength{\parskip}{0pt plus 0.5ex} 69 | % ----------------------------------------------------------------------- 70 | 71 | \usepackage{academicons} 72 | 73 | \begin{document} 74 | 75 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 76 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 77 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 78 | 79 | \everymath{\color{mathBlue}} 80 | \everydisplay{\color{mathBlue}} 81 | 82 | % for vector notation in this module 83 | \newcommand{\vect}[1]{\pmb{#1}} 84 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 85 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 86 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 87 | 88 | \footnotesize 89 | %\raggedright 90 | 91 | \begin{center} 92 | {\huge\sffamily\bfseries ST3131 Cheatsheet} \huge\bfseries\\ 93 | by Yiyang, AY22/23 94 | \end{center} 95 | \setlength{\premulticols}{0pt} 96 | \setlength{\postmulticols}{0pt} 97 | \setlength{\multicolsep}{1pt} 98 | \setlength{\columnsep}{1.8em} 99 | \begin{multicols}{3} 100 | 101 | 102 | % ----------------------------------------------------------------------- 103 | 104 | \section{1. Simple Linear Regression} 105 | \subsection{Simple Regression Model} 106 | Consider regressing $Y$ on $X$: 107 | \[ 108 | Y = \beta_0 + \beta_1 + \epsilon 109 | \] 110 | Here $X$ is called \deff{Covariate}, \deff{Predictor} or \deff{Regressor}, and $Y$ \deff{Response}. 111 | \\ 112 | The Regression function: 113 | \[ 114 | EY = \mathbb{E}[Y | X] = \beta_0 + \beta_1 X 115 | \] 116 | Regression coefficients, $\beta_1 = \rho_{xy} \frac{\sigma_y}{\sigma_x}$ and $\beta_0 = \mu_y - \beta_1 \mu_x$, minimising $\mathbb{E} \big( Y - \beta_0 - \beta_1 X \big) ^2$ 117 | 118 | \subsubsection{Observed \textasciitilde} 119 | Assumptions of LRM 120 | \begin{enumerate} 121 | \item $x_i$ and $\epsilon_i$ independent 122 | \item $\frac{1}{n} \sum_{j=1}^n \epsilon_j = 0$ 123 | \item $Cov(\epsilon_i, \epsilon_j) = 0, \ \forall i, j$ 124 | \item \deff{Homogenity}, $var{\epsilon_j} = \sigma^2$ for all $j$ 125 | \item \deff{Normality}, $\epsilon_j \sim \mathcal{N}(\cdot, \cdot)$ for all $j$ 126 | \end{enumerate} 127 | 128 | \deff{Least Square Estimates} of $n$ observations $(x_i, y_i)$ gives 129 | \[ 130 | \hat\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar X)(y_i - \bar Y)}{\sum_{i=1}^n (x_i - \bar X)^2} 131 | , \ 132 | \hat\beta_0 = \hat Y - \hat\beta_1 \bar X 133 | \] 134 | , where $\hat X = \frac{1}{n}\sum_{i=1}^n x_i$ and $\hat Y = \frac{1}{n}\sum_{i=1}^n y_i$, and the estimates minimises 135 | \[ 136 | Q = Q(\hat\beta_0, \hat\beta_1) = \sum_{i=1}^n (y_i - \hat\beta_0 - \hat\beta_1 x_i)^2 137 | \] 138 | Lastly, \deff{Residual Stanndard Error} $\hat\sigma^2 = s^2 = \frac{1}{n-2}\text{SSE}$ gives $\hat\sigma$ the LSE for $\sigma$. 139 | 140 | 141 | \subsection{Analysis of Variance (ANOVA)} 142 | \subsubsection{Sum of Squares} 143 | \deff{Sum of Square Errors} $\text{SSE} = \sum_{i=1}^n (y_i - \hat y_i)^2 = \sum_{i=1}^n e_i^2$, measures variation of $Y$ due to random errors. 144 | \\ 145 | \deff{Regression Sum of Squares} $\text{SSR} = \sum_{i=1}^n (\hat y_i - \bar Y)^2$, measures variation of $Y$ explained by $X$. 146 | \\ 147 | \deff{Total Sum of Squares} $\text{SST} = \sum_{i=1}^n (y_i - \bar Y)^2$ 148 | \[ 149 | \text{SST} = \text{SSR} + \text{SSE} 150 | \] 151 | 152 | \subsubsection{Coefficients of Determination} 153 | \deff{Coefficients of Determination} measures how much of $Y$ is explained by $X$, 154 | \[ 155 | R^2 = \frac{\text{SSR}}{\text{SST}} = \frac{\beta_1^2 \sigma_x^2}{\beta_1^2 \sigma_x^2 + \sigma^2} 156 | \] 157 | \underline{Note}: $R^2 \in [0, 1]$ and $R^2 = corr(X, Y)^2 = corr(Y, \hat Y)^2$. 158 | 159 | \smallskip 160 | 161 | \deff{Adjusted $R^2$} for a RM with $p$ regressors, 162 | \[ 163 | R_a^2 = \frac{n-1}{n-p-1} R^2 - \frac{p}{n-p-1} 164 | \] 165 | \underline{Note}: [1] $R^2$ strictly increasing as $p$ increases while $R^2_a$ does not. [2] Sample $R^2$ and $R^2_a$ are both biased estimated for their population counterparts, but latter less biased. 166 | 167 | 168 | \subsection{Theoretical Properties of LSE} 169 | \subsubsection{Unbiasedness of LSE} 170 | $\hat\beta_0$, $\hat\beta_1$, $s$ are unbiased estimators for their population counterparts. $\hat Y = \hat\beta_0 + \hat\beta_1 X$ unbiased estimator for $EY = \beta_0 + \beta_1 X$. 171 | 172 | \subsubsection{Standard Errors} 173 | \begin{align*} 174 | var(\hat\beta_0) &= \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar X)^2} 175 | \\ 176 | var(\hat\beta_0) &= var(\bar Y) + \bar X^2 var(\hat\beta_1) 177 | \\ 178 | &= \frac{\sigma^2}{n} + \frac{\bar X^2}{\sum_{i=1}^n (x_i - \bar X)^2}\sigma^2 179 | \\ 180 | var(\hat Y) &= \Big[ \frac{1}{n} + \frac{(X - \bar X)^2}{\sum_{i=1}^n (x_i - \bar X)^2} \Big] \sigma^2 181 | \end{align*} 182 | The sample SEs are estimated by substituting $\sigma^2$ with $s^2$. 183 | 184 | 185 | \subsection{Inferences for \textasciitilde} 186 | \subsubsection{Statistical Tests} 187 | Significance Test (ANOVA) 188 | \\ 189 | \underline{Null}: $\beta_1=0$. \underline{Statistics}: $F = \frac{\text{MSR}}{\text{MSE}} \sim F_{1, n-2}$. \\ 190 | \underline{Decision}: Reject null if $F > f_{1, n-2} (\alpha)$ 191 | 192 | \medskip 193 | 194 | Test for $\beta_1$ (and $\beta_0$) 195 | \\ 196 | \underline{Statistic}: $T_{\beta_1} = \frac{\hat\beta_1 - \beta_1}{s(\hat\beta_1)} \sim t_{n-2}$. 197 | \\ 198 | \underline{Confidence Interval}: $\hat\beta_1 \pm t_{n-2}(\alpha / 2) \ s(\hat\beta_1)$ 199 | \\ 200 | \underline{Confidence Lower Bound}: $\hat\beta_1 - t_{n-2}(\alpha) \ s(\hat\beta_1)$, upper similar. 201 | 202 | 203 | \subsubsection{Predictions} 204 | Confidence Interval for $E[Y|X=x_h]$: 205 | \[ 206 | \hat y_h \pm t_{n-2}(\alpha/2) \cdot \sqrt{\hat\sigma^2 \Big( \frac{1}{n} + \frac{(x_h - \bar X)^2}{\sum_{i=1}^n(x_i - \bar X)^2} \Big)} 207 | \] 208 | Prediction Interval for $Y$ when $X = x_h$: 209 | \[ 210 | \hat y_h \pm t_{n-2}(\alpha/2) \cdot \sqrt{\hat\sigma^2 \Big(1 + \frac{1}{n} + \frac{(x_h - \bar X)^2}{\sum_{i=1}^n(x_i - \bar X)^2} \Big)} 211 | \] 212 | 213 | 214 | 215 | 216 | \section{2. Multiple Linear Regression} 217 | \subsection{Multiple Regression Model} 218 | Objective: To investigate relationship between $Y$ and $p$ predictors $\vec{X} = (X_1, ..., X_p)$ with $n$ observations. 219 | 220 | \subsubsection{Least Square Estimation} 221 | Let $\vect{\beta} = (\beta_0, \beta_1, ..., \beta_p)^T \in \mathbb{R}^{p+1}$, 222 | \begin{align*} 223 | \vect{\hat\beta} &= (X^T X)^{-1} X^T \vect{y} 224 | \\ 225 | s^2 &= \frac{\lVert \vect{y} - H\vect{y} \rVert ^2}{n - p -1} 226 | \end{align*} 227 | 228 | Here $X = X_{n \times (p+1)}$ is the \deff{Design Matrix} where a column of $1$ is joined to the left of observed predictors. 229 | 230 | \subsubsection{Hat Matrix} 231 | The \deff{Hat Matrix} $H = H_{n \times n}$ is the projection matrix of linear space spanned by column vectors of $X$. 232 | \[ 233 | H = X(X^T X)^{-1}X^T 234 | \] 235 | Properties of $H$: 236 | \begin{itemize} 237 | \item $HX = X$ 238 | \item Idempotent, $H^2 = H$, so does $I - H$ 239 | \item $H \vect{\mathbb{1}} = \vect{\mathbb{1}}$ for $\vect{\mathbb{1}} = \text{One}(n, 1)$ 240 | \item $H \vect{x_j} = \vect{x_j}$ for $\vect{x_j} = (x_{1j}, ..., x_{nj})^T$ 241 | \item $\vect{\hat y} = H \vect{y}$ is the fitted values, and $\vect{e} = (I-H) \vect{y}$ is the residuals. 242 | \end{itemize} 243 | 244 | \subsubsection{Explicit Expression for One Coefficient} 245 | \[ 246 | \hat\beta_j = \frac{\vect{x_j}^T (I - H_{-j} \vect{y})}{\vect{x_j}^T (I - H_{-j} \vect{x_j})} 247 | \] 248 | , where $X_{-j}$ is the Sub-Design Matrix with column $\vect{x_j}$ removed, and $H_{-j}$ the corresponding Hat Matrix. 249 | 250 | 251 | 252 | \subsection{ANOVA} 253 | \subsubsection{Sum of Squares} 254 | \begin{align*} 255 | \text{SST} &= \vect{y}^T H_T \vect{y}, \; \; H_T = I - \frac{1}{n} \vect{\mathbb{1}} \vect{\mathbb{1}}^T 256 | \\ 257 | \text{SSR} &= \vect{y}^T H_R \vect{y}, \; \; H_R = H - \frac{1}{n} \vect{\mathbb{1}} \vect{\mathbb{1}}^T 258 | \\ 259 | \text{SSE} &= \vect{y}^T H_E \vect{y}, \; \; H_E = I - H 260 | \end{align*} 261 | 262 | \subsubsection{Distributions of Sum of Squares} 263 | \[ 264 | \frac{\text{SSE}}{\sigma^2} \sim \chi^2_{n-p-1}, \; \; 265 | \frac{\text{SSR}}{\sigma^2} \sim \chi^2_{p} 266 | \] 267 | 268 | \subsubsection{ANOVA Table} 269 | \begin{center} 270 | \begin{tabular}{ |c|c c c c |} 271 | \hline 272 | & df & SS & MS & F \\ 273 | \hline 274 | Regression & p & SSR & MSR & MSR/MSE \\ 275 | Error & n-p-1 & SSE & MSE & \\ 276 | Total & n-1 & SST & & \\ 277 | \hline 278 | \end{tabular} 279 | \end{center} 280 | 281 | %MCORR, CORR and partial 282 | 283 | \subsection{Inferences for \textasciitilde } 284 | \subsubsection{Statistical Tests} 285 | Significance Test (ANOVA) 286 | \\ 287 | \underline{Null}: $\vect \beta = \vect{0}$. 288 | \underline{Statistics}, $F = \frac{\text{MSR}}{\text{MSE}} \sim F_{p, n-p-1}$. 289 | 290 | \medskip 291 | 292 | Individual $t$-Test 293 | \\ 294 | \underline{Null}: $\beta_i = 0$. 295 | \underline{Statistic}: $T_{\beta_i} = \frac{\hat\beta_i}{s(\hat\beta_i)} \sim t_{n-p-1}$. 296 | 297 | \medskip 298 | 299 | General Linear Hypothesis Test 300 | \\ 301 | For a given linear hypothesis $\vect{c} = (c_0, c_1, ..., c_p)^T$, 302 | \\ 303 | \underline{Null}: $\vect{c}^T \vect{\beta} = 0$ 304 | \underline{Statistics}: $T = \frac{\vect{c}^T \vect{\hat\beta} }{\sqrt{\vect{c}^T \hat\Sigma \vect{c}}} \sim t_{n-p-1}$ 305 | 306 | \subsubsection{Predictions} 307 | Confidence Interval for $E[Y|\vec X=\vect{x_h}]$: 308 | \[ 309 | \hat y_h \pm \hat\sigma_F^2 (\hat y_h) \times t_{n-p-1}(\alpha / 2) 310 | \] 311 | where the estimated variance is 312 | \[ 313 | \hat\sigma_F^2(\hat y_h) = \vect{x_h}^T Var(\vect{\hat\beta})\vect{x_h} = \vect{x_h}^T (X^TX)^{-1}\vect{x_h} \hat\sigma^2 314 | \] 315 | 316 | Prediction Interval for $Y$ when $\vec X=\vect{x_h}$: 317 | \[ 318 | \hat y_h \pm \hat\sigma_P^2 (\hat y_h) \times t_{n-p-1}(\alpha / 2) 319 | \] 320 | where the estimated variance is 321 | \[ 322 | \hat\sigma_P^2(\hat y_h) = \Big[ 1 + \vect{x_h}^T (X^TX)^{-1}\vect{x_h} \Big] \hat\sigma^2 = \hat\sigma^2 + \hat\sigma_F^2 323 | \] 324 | 325 | 326 | 327 | 328 | \end{multicols} 329 | \end{document} 330 | -------------------------------------------------------------------------------- /ST2131/ST2131_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | % 14 | % Credits: 15 | % Jovyn Tan, https://github.com/jovyntls 16 | % for the nice navy blue colour for in-line and display Math. 17 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 18 | 19 | \documentclass[10pt,landscape,letterpaper]{article} 20 | \usepackage{amssymb} 21 | \usepackage{amsmath} 22 | \usepackage{amsthm} 23 | %\usepackage{fonts} 24 | \usepackage{multicol,multirow} 25 | \usepackage{spverbatim} 26 | \usepackage{graphicx} 27 | \usepackage{ifthen} 28 | \usepackage[landscape]{geometry} 29 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 30 | \usepackage{booktabs} 31 | \usepackage{fontspec} 32 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 33 | \setsansfont{Fira Sans} 34 | \setmonofont{Inconsolata} 35 | \usepackage{unicode-math} 36 | \setmathfont{TeX Gyre Pagella Math} 37 | \usepackage{microtype} 38 | 39 | \usepackage{empheq} 40 | 41 | % new: 42 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 43 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 44 | } 45 | \makeatother 46 | 47 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 48 | { \geometry{margin=0.4in} } 49 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 50 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 51 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 52 | } 53 | \pagestyle{empty} 54 | \makeatletter 55 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 56 | {-1ex plus -.5ex minus -.2ex}% 57 | {0.5ex plus .2ex}%x 58 | {\sffamily\large}} 59 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 60 | {-1explus -.5ex minus -.2ex}% 61 | {0.5ex plus .2ex}% 62 | {\sffamily\normalsize\itshape}} 63 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 64 | {-1ex plus -.5ex minus -.2ex}% 65 | {1ex plus .2ex}% 66 | {\normalfont\small\itshape}} 67 | \makeatother 68 | \setcounter{secnumdepth}{0} 69 | \setlength{\parindent}{0pt} 70 | \setlength{\parskip}{0pt plus 0.5ex} 71 | % ----------------------------------------------------------------------- 72 | 73 | \usepackage{academicons} 74 | 75 | \begin{document} 76 | 77 | \definecolor{myblue}{cmyk}{1,.72,0,.38} 78 | \everymath{\color{myblue}} 79 | \everydisplay{\color{myblue}} 80 | 81 | \footnotesize 82 | %\raggedright 83 | 84 | \begin{center} 85 | {\huge\sffamily\bfseries ST2131 Cheatsheet} \huge\bfseries\\ 86 | by Yiyang, AY20/21 87 | \end{center} 88 | \setlength{\premulticols}{0pt} 89 | \setlength{\postmulticols}{0pt} 90 | \setlength{\multicolsep}{1pt} 91 | \setlength{\columnsep}{1.8em} 92 | \begin{multicols}{3} 93 | 94 | \section{Chapter 01 - Combinatorial Analysis} 95 | \subsection{Some Combinatorial Identities} 96 | For all non-negative integers $m, n, k$ and $k \leq n$, 97 | \begin{itemize} 98 | \item $k {{n}\choose{k}} = (n-k+1) {{n}\choose{k-1}} = n {{n-1}\choose{k-1}}$ \ \emph{(AY20/21Sem2 Tut1Qn7)} 99 | \item $\sum_{k=1}^{n}{k {n \choose{k}}} = n2^{n-1}$ \ \emph{(AY20/21Sem2 Tut1Qn8)} 100 | \item ${{n+m}\choose{k}} = {{n}\choose{0}}{{m}\choose{k}} + {{n}\choose{1}}{{m}\choose{k-1}} + ... + {{n}\choose{r}}{{m}\choose{0}}$ \ \emph{(AY20/21Sem2 Tut1Qn9)} 101 | \item ${{2n}\choose{n}} = \sum_{k=0}^{n} {{n}\choose{k}}^2$ \ \emph{(AY20/21Sem2 Tut1Qn10)} 102 | \item $\sum_{i=1}^{\infty} i r^{i-1} = \frac{1}{(1-r)^2}$, for $|r| < 1$ 103 | \end{itemize} 104 | 105 | 106 | \section{Chapter 03 - Conditional Probability} 107 | \subsection{Some Identities Involving Conditional Probability} 108 | For any events $A$, $B$, $C$, 109 | \begin{align*} 110 | P(A|C) &= P(AB|C) + P(AB^C|C) 111 | \\ 112 | &= P(A|BC)P(B|C) + P(A|B^CC)P(B^C|C) 113 | \end{align*} 114 | 115 | 116 | 117 | \section{Chapter 04 - Random Variables} 118 | \subsection{Tail Sum Formula} 119 | For \emph{non-negative integer-valued} random variable $X$, 120 | if $X$ is a D.R.V. (i.e. $X = 0, 1,...,2$), 121 | \[ 122 | E(X) = \sum_{k=1}^{\infty} P(X \geq k) = \sum_{k=0}^{\infty} P(X > k) 123 | \] 124 | or if $X$ is a C.R.V., 125 | \[ 126 | E(X)= \int_{0}^{\infty} P(X>x) \,dx = \int_{0}^{\infty} P(X \geq x) \,dx 127 | \] 128 | 129 | \section{Chapter 05 - Continuous Random Variable} 130 | \subsection{Distribution of a Function of R.V.} 131 | For r.v. $X$ with pdf. $f_X(x)$, assume $g(x)$ is a function of $X$ that is \textbf{strictly monotonic} and \textbf{differentiable}. Then the pdf. of $Y=g(X)$, 132 | \[ 133 | f_Y(y) = \begin{cases} 134 | f_X(g^{-1}(y)) \big| \frac{d}{dy} g^{-1}(y) \big|, \ &y=g(x) \text{ for some $x$} 135 | \\ 136 | 0, \ &\text{otherwise} 137 | \end{cases} 138 | \] 139 | 140 | \subsection{Binomial to Normal Approximation} 141 | (Remember \textbf{Continuity Correction}!!!) 142 | \\ 143 | For $X \sim Bin(n, p)$ where $npq$ is large (generally good when $npq \geq 10$), 144 | \[ 145 | Bin(n,p) \approx N(np, npq) \text{ , i.e. } \frac{X-np}{\sqrt{npq}} \approx Z 146 | \] 147 | 148 | \subsection{Binomial to Poisson Approximation} 149 | For $X \sim Bin(n, p)$ where $n$ is large and $p$ (or $q$) is small so that $np$ (or $nq$) is moderate. 150 | \begin{itemize} 151 | \item when $p < 0.1$, $Bin(n, p) \approx Poisson(np)$ 152 | \item when $p > 0.9$, $Bin(n, q) \approx Poisson(nq)$ 153 | \end{itemize} 154 | 155 | \section{Chapter 06 - Joint Distributions} 156 | \subsection{Convolution of Independent Distributions} 157 | \begin{align*} 158 | F_{X+Y}(a) &= \int_{-\infty}^{\infty}F_Y(a-x) \ f_X(x)\,\, dx = \int_{-\infty}^{\infty}F_X(a-y) \ f_Y(y) \, dy 159 | \\ 160 | f_{X+Y}(a) &= \int_{-\infty}^{\infty}f_Y(a-x) \ f_X(x) \, dx = \int_{-\infty}^{\infty}f_X(a-y) \ f_Y(y) \, dy 161 | \end{align*} 162 | 163 | \subsection{Prop.6.4 - Sum of Independent Gamma R.V.s} 164 | Assume $X \sim Gamma(\alpha, \lambda)$ and $Y \sim Gamma(\beta, \lambda)$ are independent. 165 | \[ X+Y \sim Gamma(\alpha+\beta, \lambda) 166 | \] 167 | 168 | \subsection{Prop.6.5 - Sum of Independent Normal R.V.s} 169 | Assume $X_i, i=1,2,...,n$ are independent random variables that are normally distributed with parameters $\mu_i, \sigma_i^2, \ i = 1,2,...,n$. 170 | \[ \sum_{i=1}^n X_i \sim N(\sum_{i=1}^n \mu_i \ \sum_{i=1}^n \sigma_i^2) 171 | \] 172 | 173 | \subsection{Ex.6.18 - Sum of Independent Poisson R.V.s} 174 | Assume $X \sim Poisson(\lambda), \ Y \sim Poisson(\mu)$ are independent. 175 | \[ 176 | X+Y \sim Poisson(\lambda + \mu) 177 | \] 178 | 179 | \subsection{Ex.6.19 - Sum of Independent Binomial R.V.s} 180 | Assume $X \sim Bin(n,p), \ Y \sim Bin(m, p)$ are independent. 181 | \[ X+Y \sim Bin(n+m, p) 182 | \] 183 | Note: This statement only works when the second parameter of both R.V.s are the same (i.e. both $p$). For problems with different parameters and large values, can consider using Normal Approximation with \emph{(Prop.6.5)}. 184 | 185 | \section{Ch 07 - Properties of Expectation} 186 | \subsection{Ex.7.20 - Expectation of a Random Sum} 187 | Suppose $X_1, X_2, ...$ are i.i.d. with common mean $\mu$. Suppose $N$ is a non-negative integer-valued random variable independent of the $X_i$. 188 | \[ \sum_{k=1}^N X_k = \mu E[N] 189 | \] 190 | 191 | \subsection{Common Moment Generating Functions} 192 | \begin{itemize} 193 | \item $X \sim Be(p), \ M_X(t) = 1 - p + pe^t$ 194 | \item $X \sim Bin(n, p), \ M_X(t) = (1-p+pe^t)^n$ 195 | \item $X \sim Geom(p), \ M_X(t) = \frac{pe^t}{1-qe^t}$ 196 | \item $X \sim Poisson(\lambda), \ M_X(t) = e^{\lambda(e^t-1)}$ 197 | \item $X \sim U(\alpha, \beta), \ M_X(t) = \frac{e^{\beta t} - e^{\alpha t}}{(\beta - \alpha t) t}$ 198 | \item $X \sim Exp(\lambda), \ M_X(t) = \frac{\lambda}{\lambda - t}, \text{for } t < \lambda$ 199 | \item $X \sim N(\mu, \sigma^2), \ M_X(t) = e^{(\mu t + \sigma^2 t^2 / 2)}$ 200 | \end{itemize} 201 | 202 | \subsection{Less Common MGFs} 203 | \begin{itemize} 204 | \item \emph{(Ex.7.31)} X is a chi-squared r.v. with $n$ deg. of freedom, $M_X(t) = {(E[e^{tZ^2}])}^n = {(1-2t)}^{-n/2}$ 205 | \end{itemize} 206 | 207 | \subsection{Ex.7.34 - "Partitioned" Poisson Distribution} 208 | Let $X$ be the r.v. that denotes total number of events. Suppose each event is a Ber. process with $p$ probability of being $A$ and $q = (1-p)$ being $B$. Let $X_A, X_B$ denote the number of events that are $A$ and $B$ respectively. 209 | If $X \sim Poisson(\lambda)$, 210 | \[ 211 | X_A \sim Poisson(p\lambda), \ X_B \sim Poisson(q\lambda) 212 | \] 213 | 214 | \section{Ch 08 - Limit Theorems} 215 | \subsection{Markov's Inequality} 216 | For \textbf{non-negative} r.v. $X$ and any $a > 0$, 217 | \[ 218 | P(X \geq a) \leq \frac{E(X)}{a} 219 | \] 220 | 221 | \subsection{Chebyshev's Inequality} 222 | Let $X$ be a r.v. with mean $\mu$, then for any $a > 0$, 223 | \[ 224 | P(|X-\mu| \geq a) \leq \frac{\text{var}(X)}{a^2} 225 | \] 226 | 227 | \subsection{One-sided Chebyshev's Inequality} 228 | Let $X$ be a r.v. with \textbf{zero mean} and variance $\sigma^2$, then for any $a > 0$, 229 | \[ 230 | P(X \geq a) \leq \frac{\sigma^2}{\sigma^2 + a^2} 231 | \] 232 | 233 | \subsection{Central Limit Theorem} 234 | (Remember \textbf{Continuity Correction} when a CRV is used to approximate a DRV!!!) 235 | For a sequence of i.i.d. r.v.s $X_1, X_2, ... $, each with mean $\mu$ and variance $\sigma^2$, 236 | \[ 237 | \frac{X_1 + ... + X_n - n\mu}{\sigma \sqrt{n}} \to Z, \ \text{as } n \to \infty 238 | \] 239 | 240 | \subsection{WLLN & SLLN} 241 | 242 | \subsection{Jensen's Inequality} 243 | For any r.v. $X$ and convex function $g(X)$, 244 | \[ 245 | E[g(X)] \geq g(E[X]) 246 | \] 247 | , provided the expectations exist and are finite. 248 | 249 | \midrule 250 | 251 | \section{D.R.V. Models} 252 | \subsection{Bernoulli} 253 | $X \sim Be(p)$, indicate whether an event is successful. 254 | \\ 255 | \textbf{Parameter} - $p = P(X=1)$ : success rate 256 | \\ 257 | \textbf{Distribution} - $P(X=1) = p, \ P(X=0) = q = 1-p$ 258 | \\ 259 | $E(X) = p, \ \text{var}(X) = pq = p(1-p)$ 260 | 261 | 262 | \subsection{Binomial} 263 | $X \sim Bin(n, p)$, total number of successes in $n$ i.i.d. $Be(p)$ trials. 264 | \\ 265 | \textbf{Parameters} 266 | \begin{itemize} 267 | \item $n$ : number of trials 268 | \item $p$ : success rate for each Bernoulli trial 269 | \end{itemize} 270 | 271 | \textbf{Distribution} 272 | \[ 273 | P(X=k) = {n \choose k} p^x q^{n-x}, \ k = 0, 1, ..., n 274 | \] 275 | \\ 276 | $E(X) = np, \ \text{var}(X) = npq = np(1-p)$ 277 | 278 | \subsection{Geometric} 279 | $X \sim Geom(p)$, number of i.i.d $Be(p)$ trials until one success. $X = 1, 2, ...$ 280 | \\ 281 | \textbf{Parameter} - $p$ : success rate 282 | \\ 283 | \textbf{Distribution} 284 | \[ 285 | P(X=k) = pq^{k-1}, \ k = 1, 2, ... 286 | \] 287 | Memoryless Property: $P(X > s+t | X > s) = P(X > t)， \ s, t > 0$. 288 | \\ 289 | $E(X) = \frac{1}{p}, \ \text{var}(X) = \frac{1-p}{p^2}$ 290 | 291 | 292 | 293 | \subsection{Negative Binomial} 294 | $X \sim NB(r, p)$, number of i.i.d $Be(p)$ trials for first $r$ successes. $X = r, r+1, ...$ 295 | \\ 296 | \textbf{Parameter} 297 | \begin{itemize} 298 | \item $r$ : successes needed 299 | \item $p$ : success rate 300 | \end{itemize} 301 | 302 | \textbf{Distribution} 303 | \[ 304 | P(X=k) = {{k-1} \choose {r-1}} \ p^r q ^ {x-r}, \ k = r, r+1, ... 305 | \] 306 | \\ 307 | $E(X) = \frac{r}{p}, \ \text{var}(X) = \frac{r(1-p)}{p^2}$ 308 | \\ 309 | $Geom(p) = NB(1, p)$ 310 | 311 | \subsection{Poisson} 312 | $X \sim Poisson(\lambda)$ 313 | \\ 314 | \textbf{Parameter} - $\lambda$ : "average occurrence rate in unit time interval" 315 | \\ 316 | \textbf{Distribution} 317 | \[ 318 | P(X = k) = e^{-\lambda} \frac{\lambda^k}{k!}, \ k = 0, 1, ... 319 | \] 320 | \\ 321 | $E(X) = \text{var}(X) = \lambda$ 322 | 323 | 324 | 325 | \subsection{Hypergeometric} 326 | Suppose there are $N$ identical balls, $m$ of them are red and $N-m$ are blue. 327 | $X \sim H(n, N, m)$ is the number of red balls obtained in $n$ draws without replacement. 328 | \\ 329 | \textbf{Parameter} 330 | \begin{itemize} 331 | \item $N$ : total number of objects ("red and blue balls") 332 | \item $m$ : number of objects considered success ("red balls") 333 | \item $n$ : number of trials without replacement ("draws") 334 | \end{itemize} 335 | \textbf{Distribution} 336 | \[ 337 | P(X = k) = \frac{{m \choose k}{{N-m} \choose {n-k}}}{{N \choose n}}, \ k = 0, 1, ..., n 338 | \] 339 | \\ 340 | $E(X) = \frac{nm}{N}, \ \text{var}(X) = \frac{nm}{N} \big[ \frac{(n-1)(m-1}{N-1} + 1 - \frac{nm}{N} \big]$ 341 | 342 | \midrule 343 | 344 | \section{C.R.V. Models} 345 | \subsection{Uniform} 346 | $X \sim U(a, b)$, where $X$ has equal probability of taking any value in $(a, b)$. 347 | \\ 348 | \textbf{Parameters} - $a$ and $b$ : the start and end value for the interval 349 | \\ 350 | \textbf{Distribution} 351 | \[ 352 | f(x) = \begin{cases} 353 | \frac{1}{b-a}, \ & a < x < b 354 | \\ 355 | 0, \ & \text{otherwise} 356 | \end{cases} 357 | \] 358 | \\ 359 | \[ 360 | F(x) = \begin{cases} 361 | 0, \ & x \leq a 362 | \\ 363 | \frac{x-a}{b-a}, \ & a < x < b 364 | \\ 365 | 1, \ & b \leq x 366 | \end{cases} 367 | \] 368 | \\ 369 | $E(X) = \frac{a+b}{2}, \ \text{var}(X) = \frac{(b-a)^2}{12}$ 370 | 371 | 372 | \subsection{Exponential} 373 | $X \sim Exp(\lambda)$ usually models the life time of a product, for $\lambda > 0$ 374 | \\ 375 | \textbf{Distribution} 376 | \[ 377 | f(x) = \begin{cases} 378 | \lambda e^{-\lambda x}, \ & x \geq 0 379 | \\ 380 | 0, \ & otherwise 381 | \end{cases} 382 | \] 383 | \\ 384 | \[ 385 | F(x) = \begin{cases} 386 | 0, \ & x \leq 0 387 | \\ 388 | 1 - e^{-\lambda x}, \ & x > 0 389 | \end{cases} 390 | \] 391 | \\ 392 | Memoryless Property: $P(X > s+t | X > s) = P(X > t)， \ s, t > 0$. 393 | \\ 394 | $E(X) = \frac{1}{\lambda}, \ \text{var}(X) = \frac{1}{\lambda^2}$ 395 | 396 | 397 | \subsection{Normal} 398 | $X \sim N(\mu, \sigma^2)$. Special case : $Z \sim N(0, 1)$ standard normal 399 | \\ 400 | \textbf{Parameters} 401 | \begin{itemize} 402 | \item $\mu$ : mean 403 | \item $\sigma$ : standard deviation 404 | \end{itemize} 405 | \textbf{Distribution} 406 | \begin{align*} 407 | f_Z(z) &= \frac{1}{\sqrt{2 \pi}} e^{-z^2/2}, \ z \in \mathbb{R} 408 | \\ 409 | f_X(x) &= \frac{1}{\sqrt{2 \pi} \sigma} e^{-(x-\mu)^2/(2\sigma^2)}, \ x \in \mathbb{R} 410 | \end{align*} 411 | \\ 412 | $E(X) = \mu, \ \text{var}(X) = \sigma^2$ 413 | 414 | 415 | \subsection{Gamma} 416 | $X \sim Gamma(\alpha, \lambda)$ can be seen as the sum of $\alpha$ independent $Exp(\lambda)$, for $\alpha, \lambda > 0$. (Refer to \emph{Prop.6.4}) 417 | \\ 418 | \textbf{Parameters} 419 | \begin{itemize} 420 | \item $\alpha$ : shape parameter 421 | \item $\lambda$ : rate parameter 422 | \item ($\frac{1}{\lambda}$ : scale parameter) 423 | \end{itemize} 424 | \textbf{Distribution} 425 | \[ 426 | f(x) = \begin{cases} 427 | \frac{\lambda e^{\lambda x}(\lambda x)^{\alpha-1}}{\Gamma(\alpha)}, \ & x \geq 0 428 | \\ 429 | 0, \ & x < 0 430 | \end{cases} 431 | \] 432 | \\ 433 | $Exp(\lambda) = Gamma(1, \lambda)$ is a special case of Gamma r.v. 434 | \\ 435 | $E(X) = \frac{\alpha}{\lambda}, \ \text{var}(X) = \frac{\alpha}{\lambda^2}$ 436 | \\ 437 | \textbf{Gamma Function} $\Gamma(\alpha) = \int_0^{\infty} e^{-y}y^{\alpha-1} \,dy$ 438 | \\ 439 | It satisfies that 440 | \begin{itemize} 441 | \item $\Gamma(1) = 1, \ \Gamma(\frac{1}{2}) = \sqrt{\pi}$ 442 | \item $\Gamma(\alpha) = (\alpha-1)\Gamma(\alpha-1), \ \alpha > 0$ 443 | \item $\Gamma(n) = (n-1)!, \ n \in \mathbb{Z}^{+}$ 444 | 445 | \end{itemize} 446 | 447 | \subsection{Weibull Distribution} 448 | $S \sim W(\nu, \alpha, \beta)$ can be seen as the generalised form of Exponential r.v. 449 | \begin{itemize} 450 | \item $E(\lambda) = W(1, \lambda, 0)$ 451 | \item If $X \sim E(\lambda)$, then linear transformation $Y = \alpha X + \nu \ \sim W(\nu, \alpha, \lambda)$ \emph{(Tut7Qn15)} 452 | \end{itemize} 453 | 454 | \subsection{Cauchy} 455 | $X \sim \text{Cauchy}(\theta, \alpha)$ for $\theta \in \mathbb{R}, \alpha > 0$ if it has the distribution: 456 | \[ 457 | f(x) = \frac{1}{\pi \alpha \big[1 + (\frac{x-\theta}{\alpha})^2 \big]}, \ x \in \mathbb{R} 458 | \] 459 | \\ 460 | $E(X)$ and $\text{var}(X)$ do not exist for Cauchy r.v. 461 | 462 | \subsection{Beta} 463 | $X \sim B(a, b)$. Specifically, $U(0, 1) = B(1,1)$ is a special case of Beta r.v. 464 | 465 | 466 | \end{multicols} 467 | \end{document} 468 | -------------------------------------------------------------------------------- /ST3236/ST3236_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | %\usepackage{fonts} 21 | \usepackage{multicol,multirow} 22 | \usepackage{spverbatim} 23 | \usepackage{graphicx} 24 | \usepackage{ifthen} 25 | \usepackage[landscape]{geometry} 26 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 27 | \usepackage{booktabs} 28 | \usepackage{fontspec} 29 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 30 | \setsansfont{Fira Sans} 31 | \setmonofont{Inconsolata} 32 | \usepackage{unicode-math} 33 | \setmathfont{TeX Gyre Pagella Math} 34 | \usepackage{microtype} 35 | 36 | \usepackage{empheq} 37 | 38 | % new: 39 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 40 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 41 | } 42 | \makeatother 43 | 44 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 45 | { \geometry{margin=0.4in} } 46 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 47 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | } 50 | \pagestyle{empty} 51 | \makeatletter 52 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 53 | {-1ex plus -.5ex minus -.2ex}% 54 | {0.5ex plus .2ex}%x 55 | {\sffamily\large}} 56 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 57 | {-1explus -.5ex minus -.2ex}% 58 | {0.5ex plus .2ex}% 59 | {\sffamily\normalsize\itshape}} 60 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 61 | {-1ex plus -.5ex minus -.2ex}% 62 | {1ex plus .2ex}% 63 | {\normalfont\small\itshape}} 64 | \makeatother 65 | \setcounter{secnumdepth}{0} 66 | \setlength{\parindent}{0pt} 67 | \setlength{\parskip}{0pt plus 0.5ex} 68 | % ----------------------------------------------------------------------- 69 | 70 | \usepackage{academicons} 71 | 72 | \begin{document} 73 | 74 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 75 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 76 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 77 | 78 | \everymath{\color{mathBlue}} 79 | \everydisplay{\color{mathBlue}} 80 | 81 | % for vector notation in this module 82 | \newcommand{\vect}[1]{\boldsymbol{#1}} 83 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 84 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 85 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 86 | 87 | \footnotesize 88 | %\raggedright 89 | 90 | \begin{center} 91 | {\huge\sffamily\bfseries ST3236 Cheatsheet} \huge\bfseries\\ 92 | by Yiyang, AY21/22 93 | \end{center} 94 | \setlength{\premulticols}{0pt} 95 | \setlength{\postmulticols}{0pt} 96 | \setlength{\multicolsep}{1pt} 97 | \setlength{\columnsep}{1.8em} 98 | \begin{multicols}{3} 99 | 100 | 101 | % ----------------------------------------------------------------------- 102 | 103 | \section{Probability Theory, Review} 104 | \subsection{Boole's Inequality} 105 | For any events $A_1, A_2, ...$, 106 | \[ 107 | \mathbb{P} \Big( \bigcup_{n \geq 1} A_n \Big) \leq \sum_{n \geq 1} \mathbb{P} (A_n) 108 | \] 109 | 110 | \subsection{Conditional Expectation, Properties} 111 | \begin{itemize} 112 | \item Linearity: $\mathbb{E} (aX_1 + bX_2 | Y) = a \mathbb{E} (X_1 | Y) + b \mathbb{E} (X_2 | Y)$ 113 | \item Law of Iterated Expectation: $\mathbb{E} ( \mathbb{E} (X|Y) ) = \mathbb{E} (X)$ 114 | \item Tower Property: $\mathbb{E} ( \mathbb{E} (X|Y,Z)|Y ) = \mathbb{E} (X|Y)$ 115 | \item Independence: $\mathbb{E} (X|Y) = \mathbb{E} (X)$, for $X$ and $Y$ independent. 116 | \end{itemize} 117 | 118 | 119 | 120 | 121 | \noindent\rule{8cm}{0.4pt} 122 | 123 | 124 | 125 | 126 | \section{Markov Chain (MC) Basics} 127 | A (discrete-time) \deff{Markov Chain} (MC) is a stochastic process $ \{ X_n \}_{n=0}^{\infty}$ that satisfies, 128 | \begin{align*} 129 | &\mathbb{P} (X_{n+1} = t_{n+1} | X_n = t_n, X_{n-1} = t_{n-1}, ..., X_1 = t_1) \\ 130 | = &\mathbb{P} (X_{n+1} = t_{n+1} | X_n = t_n) 131 | \end{align*} 132 | , whenever $\mathbb{P} (X_n = t_n, X_{n-1} = t_{n-1}, ..., X_1 = t_1) > 0$. This property is also called the \deff{Markovian Property}. 133 | 134 | \smallskip 135 | 136 | A \deff{Time-homogeneous} Markov Chain is one where the conditional probability $\mathbb{P} (X_{n+1} = j | X_n = i)$ does not depend on $n$. Equivalently, it means for all $n \geq 0$ and $i, j \in S$, 137 | \[ 138 | \mathbb{P} (X_{n+1} = j | X_n = i) = \mathbb{P} (X_1 = j | X_0 = i) 139 | \] 140 | For time-homogeneous MCs, $p_{ij} = \mathbb{P} (X_1 = j | X_0 = i)$ is the \deff{1-step transition probability}, and $P = ((p_{ij}))_{i,j \in S}$ is the \deff{Transition matrix}. Similarly, $p_{ij}^{(k)} = \mathbb{P} (X_k = j | X_0 = i)$ is the \deff{k-Step Transition Probability}. 141 | 142 | \subsection{Chapman-Komogorov Equation} 143 | \[ 144 | p_{ij}^{(k)} = \sum_{i_1, ..., i_{k-1} \in S} p_{ii_1}p_{i_1i_2}...p_{i_{k-1}j} 145 | \] 146 | Equivalently, it means $P^{(k)} = P^k$. 147 | 148 | 149 | 150 | 151 | \section{Accessing States, MC} 152 | \subsection{Accessible States} 153 | For states $i$, $j$ of a MC, $j$ is \deff{Accesible} from $i$, $i \to j$, if there exists $k \geq 0$ such that, 154 | \[ 155 | p_{ij} = \mathbb{P} (X_k = j | X_0 = i) > 0 156 | \] 157 | \\ 158 | States $i$ and $j$ \deff{Communicate}, i.e. $i \leftrightarrow j$ iff. $i \to j \land j \to i$. 159 | \medskip 160 | 161 | $\leftrightarrow$ is an \textbf{Equivalence Relation}. The equivalent classes of $S$ partitioned by it are called \deff{Communicating classes}. 162 | \medskip 163 | 164 | Notes: 165 | \begin{itemize} 166 | \item $i \leftrightarrow i$ as $p_{ii}^{(0)} = 0$ for all $i \in S$ 167 | \item For $i \in S_i$ and $j \in S_j$, it is possible that $i \to j$. 168 | \item For $i \in S_i$ and $j \in S_j$, $i \to j \implies j \not\to i$. 169 | \end{itemize} 170 | 171 | A MC is \deff{Irreducible} if there is only one communicating class for the state space. Otherwise, the MC is \deff{Reducible}. 172 | 173 | 174 | \subsection{Essential States} 175 | A state $i$ is \deff{Essential} if for every state $j$ it satisfies 176 | \[ 177 | i \to j \implies j \to i 178 | \] 179 | A state that is not essential is called \deff{Inessential}. 180 | \medskip 181 | 182 | Properties of Essential States 183 | \begin{itemize} 184 | \item If $i$ essential and $i \to j$ then $j$ is essential. 185 | \item All states in one communicating class are either all essential or all inessential. 186 | \item A \textbf{finite state} MC must have at least one essential state. 187 | \item An absorbing state is essential. 188 | \end{itemize} 189 | 190 | 191 | 192 | 193 | \section{Recurrence Theory} 194 | \subsection{Definition} 195 | A state $i$ of a Markov Chain is \deff{Recurrent} if 196 | \[ 197 | P(X_n = i \text{ for some } n \ge 1 | X_0 = i) = 1 198 | \] 199 | Otherwise, the state $i$ is \deff{Transient}. 200 | 201 | \medskip 202 | 203 | Let $T_i = \min \{ n \ge 1: X_n = i \}$ firt time the chain visits state $i$, and let $\mu_i = \mathbb{E}[T_i | X_0 = i]$ expected return time to state $i$, then for a recurrent state $i$, 204 | \begin{itemize} 205 | \item State $i$ is \deff{Positive Recurrent}, if $\mu_i < \infty$ 206 | \item State $i$ is \deff{Null Recurrent}, if $\mu _i = \infty$ 207 | \end{itemize} 208 | 209 | Overall, a state is either 1) transient, 2) null recurrent, or 3) positive recurrent. 210 | 211 | \medskip 212 | Equivalently, for a state $i$, 213 | \begin{itemize} 214 | \item If $i$ is recurrent, $P(X_n = i \text{ infinitely often} | X_0 = i) = 1$ 215 | \item If $i$ is transient, $P(X_n = i \text{ infinitely often} | X_0 = i) = 0$ 216 | \end{itemize} 217 | 218 | \subsection{Criteria for Determining Recurrence} 219 | \subsubsection{Number of Visits} 220 | Define $f_i = P(X_n = i \text{ for some } n \ge 1 | X_0 = i)$ and $N_i = \sum_{n=1}^{\infty} \mathbb{1}_{\{X_n = i\}}$ as number of visits of state $i$. 221 | \\ 222 | \underline{Analysis}: Consider the probability of the chain "escaping" from returning to state $i$, $(N_i + 1) | X_0 = i \sim Geom(1 - f)i)$. 223 | 224 | \medskip 225 | 226 | A state $i$ is recurrent, if $P(N_i = \infty | X_0 = i) = 1$. The negation applies for transient state $i$. 227 | 228 | \medskip 229 | 230 | A state $i$ is recurrent, if $\mathbb{E}[N_i | X_0 = i] = \infty$. The negation applies for transient state $i$. 231 | 232 | 233 | 234 | \subsection{Properties of Recurrence States} 235 | Recurrence is a class property. If $i$ is recurrent and $j$ is in the same communicating class as $i$, $j$ is recurrent. (\underline{Proof Idea}: Chapman-Kolmogorov Equation.) \textbf{So are Positive and Null Recurrence.} 236 | 237 | \medskip 238 | 239 | There always exists a recurrent state in a finite space Markov Chain. (\underline{Proof Idea}: By contradiction) 240 | 241 | 242 | 243 | 244 | \section{First Time Passage} 245 | \deff{First Passage probability} from state $i$ to state $j$ in exactly $k$ steps is defined $f_{i,j}^{(k)} = P(X_k = j, X_{k-1} \neq j, ..., X_1 \neq j | X_0 = i)$. 246 | 247 | \subsection{Related Expressions} 248 | \subsubsection{Renewal Theorem} 249 | For $i \neq j$ and $n \ge 1$, we have 250 | \[ 251 | p^{(n)}_{i,j} = \sum_{k=1}^{n} f_{i,j}^{(k)} p_{j,j}^{(n-k)} 252 | \] 253 | 254 | \medskip 255 | 256 | \underline{Note}: Algorithm to compute First Passage Times: To compute $f_{i,j}^{(n)}$, obtain the first $n$ powers of $P$, then write and solve a linear system involving first $n$ basic renewal equations to get $f_{i,j}^{(1)}, ..., f_{i,j}^{(n)}$. 257 | 258 | \subsubsection{Recursive Relation} 259 | Consider first passage times with steps $n \ge 2$, by Markovian Property, 260 | \[ 261 | f_{i,j}^{(n)} = \sum_{k \neq j} p_{i,k} f_{k,j}^{(n-1)} 262 | \] 263 | 264 | \medskip 265 | 266 | \underline{Note}: The recursive relation can also be used to calculate First Passage Times. 267 | 268 | 269 | \subsection{Relations to Recurrence} 270 | Define $f_{i,j}$ as the probability of chain ever visiting state $j$ starting from state $i$ for $i \neq j$, 271 | \[ 272 | f_{i,j} = P(X_n = j \text{ for some } n \ge 1 | X_0 = i) = \sum_{n=1}^{\infty} f_{i,j}^{(n)} 273 | \] 274 | 275 | \subsubsection{An interesting Identity} 276 | For any state $i \neq j$, by using the previous recursive relation, 277 | \[ 278 | f_{i,j} = p_{i,j} + \sum_{k \neq j} p_{i,k}f_{k,j} 279 | \] 280 | 281 | \subsubsection{Recurrent State Visiting Another} 282 | If $i$ is recurrent and $i \to j$, then $f_{i,j} = 1$ and $f_i = f_{i,i} = 1$. 283 | 284 | \medskip 285 | 286 | For any state $i$ and a recurrent state $j$, 287 | \[ 288 | f_{i,j} = \sum_{k \in S} p_{i,k}f_{k,j} 289 | \] 290 | It is valid for both cases of $i \neq j$ and $i = j$. 291 | 292 | 293 | 294 | 295 | \section{Stationary Distribution} 296 | \subsection{Overview} 297 | A \deff{Stationary Distribution}, $\vect{\pi}$, of a Markov Chain is one such that $\vect{\pi}^{T}P = \vect{\pi}^{T}$. 298 | 299 | \subsection{Number of Visits Analysis} 300 | Define $\phi_{i}(j)$ as the expected number of timers the Markov Chain visits $j$ when starting from $i$ before returning to $i$ for $i \neq j$, 301 | \[ 302 | \phi_{i}(j) = \mathbb{E} \big[ \sum_{n=0}^{T_{i}-1} \mathbb{1}_{\{ X_n = j\}} | X_0=i \big] 303 | \] 304 | , then by definition, $\sum_{j\in S} \phi_i(j) = \mathbb{E}(T_i|X_0=i)=\mu_i$. 305 | 306 | \subsubsection{Of Positive Recurrent States} 307 | For a \textbf{Positive Recurrent} state $i$, the vector $\pi_i{j}$ forms a stationary distribution for all states $j \in S$, 308 | \[ 309 | \pi_{i}(j) = \frac{\phi_i(j)}{\sum_k \phi_i(k)} 310 | \] 311 | Then if state $i$ is in an irreducible chain, $\pi = \pi_i$ is the unique stationary distribution. 312 | 313 | \subsubsection{Of Transient \& Null Recurrent States} 314 | Suppose a stationary distribution $\vect{pi}$ exists for a general Markov Chain, then $\pi(i) = 0$ for all transient or null recurrent states $i$. \underline{Proof Idea}: Consider $\phi_i(j)$ of such states. 315 | 316 | 317 | \subsection{Of Irreducible Markov Chains} 318 | % Include Finite state MC as well (as a special case) 319 | \subsubsection{Properties} 320 | For an irreducible Markov Chain, below are equivalent: 321 | \begin{enumerate} 322 | \item At least one state is positive recurrent 323 | \item A stationary distribution exists 324 | \item A stationary distribution exists and is unique 325 | \item All states are positive recurrent 326 | \end{enumerate} 327 | \underline{Note:} The four statements above are just equivalent. They may not always be true for an irreducible chain! 328 | 329 | \subsubsection{Related Expressions} 330 | For an \textbf{irreducible} Markov Chain with a stationary distribution $\vect{\pi}$, and for all states $i \in S$, 331 | \[ 332 | \pi(i) = \frac{1}{\mu_i} = \frac{1}{\mathbb{E}[T_i | X_0 = i]} 333 | \] 334 | 335 | \subsubsection{Special Cases} 336 | Any finite state space Markov Chain always have a stationary distribution. 337 | 338 | 339 | 340 | \subsection{Of General Markov Chains} 341 | \subsubsection{Algorithm for Constructing a Stationary Distribution} 342 | For any Markov Chain, procedures for constructing a stationary distribution if there exists one: 343 | \begin{enumerate} 344 | \item For each positive recurrent class $C_j \subset S$, let $\pi^*_j$ denotes the unique stationary distribution of the Markov Chain with transition matrix reduced from $P$ to $P_{C_j \times C_j}$. 345 | \item Let $\bar{\pi_j^*}$ denotes the vector of size equal to $|S|$ by appending $0$ to $\pi^*_j$ for all states $i \in S \setminus C_j$. 346 | \item Then probability vector $\pi = \sum_{j: C_j \text{ +ve recurrent}} \alpha_j \bar{\pi_j^*}$ forms a stationary distribution for all non-negative constants $a_1, ..., $ that sum to $1$. 347 | \end{enumerate} 348 | 349 | \subsubsection{Significance of the Algorithm} 350 | A Markov Chain has a stationary distribution as long as it has \textbf{a positive recurrent class}. 351 | \medskip 352 | A finite state Markov Chain always has a stationary distribution. \underline{Proof idea}: as it cannot have all of its finite number of states transient or null recurrent. 353 | 354 | 355 | 356 | 357 | \section{Long-Term Behaviours} 358 | \subsection{Periodicity} 359 | For a \textbf{recurrent} state $i$ of a Markov Chain, its \deff{Period} is defined as the maximum integer such that the chain returns from that sate to itself only in step sizes which are multiples of the integer. 360 | 361 | \medskip 362 | 363 | Mathematically, for a recurrent state $i$, we define $Z_i = \{ n \ge 1: p^{(n)}_{ii} > 0 \}$, then the period of $i$ is defined as $d_i = \gcd{Z_i}$, where $\gcd$ is the greatest common divisor. 364 | 365 | \medskip 366 | 367 | A state $i$ is \deff{Aperiodic} if $d_i = 1$. 368 | 369 | \subsection{Properties of Periodicity} 370 | Periodicity is a Class Property. For two recurrent states $i$ and $j$ in the same communicating class, $d_i = d_j$. 371 | \\ 372 | Therefore, a class is \textbf{aperiodic} iff. any of its states is \textbf{aperiodic}. 373 | 374 | \medskip 375 | 376 | An aperiodic state can lead to itself in steps of all sizes large enough. 377 | \\ 378 | Mathematically, it means if $d_i = 1$ then there exists $K \in \mathbb{Z}^{+}$ such that $p^{(k)}_{ii} > 0$ for all $k \ge K$. (Also in \citeqn{Homework8 Qn1}) 379 | 380 | \subsection{The Convergence Theorem} 381 | For an \textbf{irreducible}, \textbf{aperiodic} and \textbf{positive recurrent} chain, let $\pi$ be its unique stationary distribution, then as $n \to \infty$, we have 382 | 383 | \[ 384 | p^{(n)}_{i,j} \to \pi(j) 385 | \] 386 | , for all states $i, j$. 387 | \\ 388 | It means that for such a chain, the stationary distribution is its limiting distribution. 389 | 390 | 391 | 392 | 393 | \noindent\rule{8cm}{0.4pt} 394 | 395 | 396 | 397 | 398 | \section{Intermediate Results \& Lemmas} 399 | \subsection{From Past Questions} 400 | \citeqn{(Homework5 Qn1)} In a finite state space Markov Chain, essentiality is equivalent to recurrence. In other words, there are no essential and transient states in a finite state space Markov Chain. 401 | 402 | \smallskip 403 | 404 | \citeqn{(Homework7 Qn2)} There are no null recurrent states in a finite state space Markov Chain. 405 | 406 | \smallskip 407 | 408 | \citeqn{(Homework7 Qn3)} Let $\{ X_n \}_{n=0}^{\infty}$ be an \textbf{irreducible and positive recurrent} Markov Chain, then for any states $i \neq j$, 409 | \[ 410 | \phi_{i}(j) = \frac{\mu_i}{\mu_j} 411 | \] 412 | 413 | \subsection{Others} 414 | \subsubsection{Vandermonde's Identity} 415 | \[ 416 | {{m + n}\choose{r}} = \sum_{k=0}^{r} {{m}\choose{k}} {{n}\choose{r-k}} 417 | \] 418 | 419 | \subsubsection{Polya's Recurrence Theorem} 420 | A random walk is recurrent in $1$ and $2$-dimensional lattices and it is transient for lattices with more than $2$ dimension. 421 | 422 | \end{multicols} 423 | \end{document} 424 | -------------------------------------------------------------------------------- /CS3243/CS3243_Midterm.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | %\usepackage{fonts} 21 | \usepackage{multicol,multirow} 22 | \usepackage{spverbatim} 23 | \usepackage{graphicx} 24 | \usepackage{ifthen} 25 | \usepackage[landscape]{geometry} 26 | \usepackage{listings} % for code block 27 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 28 | \usepackage{booktabs} 29 | \usepackage{fontspec} 30 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 31 | \setsansfont{Fira Sans} 32 | \setmonofont{Inconsolata} 33 | \usepackage{unicode-math} 34 | \setmathfont{TeX Gyre Pagella Math} 35 | \usepackage{microtype} 36 | 37 | \usepackage{empheq} 38 | 39 | % new: 40 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 41 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 42 | } 43 | \makeatother 44 | 45 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 46 | { \geometry{margin=0.4in} } 47 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 50 | } 51 | \pagestyle{empty} 52 | \makeatletter 53 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 54 | {-1ex plus -.5ex minus -.2ex}% 55 | {0.5ex plus .2ex}%x 56 | {\sffamily\large}} 57 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 58 | {-1explus -.5ex minus -.2ex}% 59 | {0.5ex plus .2ex}% 60 | {\sffamily\normalsize\itshape}} 61 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 62 | {-1ex plus -.5ex minus -.2ex}% 63 | {1ex plus .2ex}% 64 | {\normalfont\small\itshape}} 65 | \makeatother 66 | \setcounter{secnumdepth}{0} 67 | \setlength{\parindent}{0pt} 68 | \setlength{\parskip}{0pt plus 0.5ex} 69 | % ----------------------------------------------------------------------- 70 | 71 | \usepackage{academicons} 72 | 73 | \begin{document} 74 | 75 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 76 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 77 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 78 | \everymath{\color{mathBlue}} 79 | \everydisplay{\color{mathBlue}} 80 | 81 | 82 | % Based on: 83 | % https://stackoverflow.com/questions/3175105/inserting-code-in-this-latex-document-with-indentation 84 | \definecolor{codedkgreen}{rgb}{0,0.6,0} 85 | \definecolor{codegray}{rgb}{0.5,0.5,0.5} 86 | \definecolor{codemauve}{rgb}{0.58,0,0.82} 87 | \definecolor{light-gray}{gray}{0.95} 88 | 89 | \lstset{frame=tb, 90 | language=Java, 91 | backgroundcolor=\color{light-gray}, 92 | basicstyle={\footnotesize\ttfamily}, 93 | tabsize=3 94 | } 95 | 96 | 97 | % for vector notation in this module 98 | \newcommand{\vect}[1]{\boldsymbol{#1}} 99 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 100 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 101 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 102 | 103 | \footnotesize 104 | %\raggedright 105 | 106 | \graphicspath{ {./img/} } 107 | 108 | 109 | \begin{center} 110 | {\huge\sffamily\bfseries CS3243 Cheatsheet (Midterm)} \huge\bfseries\\ 111 | by Yiyang, AY21/22 112 | \end{center} 113 | \setlength{\premulticols}{0pt} 114 | \setlength{\postmulticols}{0pt} 115 | \setlength{\multicolsep}{1pt} 116 | \setlength{\columnsep}{1.8em} 117 | \begin{multicols}{3} 118 | 119 | 120 | 121 | 122 | % ----------------------------------------------------------------------- 123 | 124 | \section{Introduction} 125 | 126 | \subsection{Intelligent Agent} 127 | An intelligent agent consists of: (1) \deff{Sensors}, for capturing data (known as \deff{Percepts}, $p_i$ of \deff{Percept History}, $P=\{p_1, ..., p_n, ... \}$) from the environment; (2) \deff{Agent Functions}, $f$, for making decisions based on percepts, and \textbf{Actuators}, for performing actions, $a_1, ..., a_m \in A$, based on agent functions. 128 | 129 | In short, an agent is a function $f: P \to A$. 130 | 131 | \subsection{Types of Agents and Environments} 132 | \subsubsection{Agents} 133 | \begin{itemize} 134 | \item \deff{Reflex Agent} - Agents that use IF-ELSE rules to make decisions. 135 | \item \deff{Model-based Reflex Agent} - Agents that use an internalised model to make decisions. (e.g. state graph models for search AI) 136 | \item \deff{Goal-/Utility-based Agent} - Agents that determine sequences of actions to reach goals / maximise utility. 137 | \item \deff{Learning Agent} - Agents that learn to optimise. (not covered) 138 | \end{itemize} 139 | 140 | \subsubsection{Properties of the Problem Environments} 141 | \begin{itemize} 142 | \item \deff{Fully} / \deff{Partially Observable} - whether agents can access all information of the environment. 143 | \item \deff{Deterministic} / \deff{Stochastic} \textasciitilde \ whether transition of states is certain. 144 | \item \deff{Episodic} / \deff{Sequential} - whether actions impact only current states or all future decisions. 145 | \item \deff{Discrete} / \deff{Continuous} \textasciitilde \ of state info., time, percepts and/or actions. 146 | \item \deff{Single} / \deff{Multi-agent} 147 | \item \deff{Known} / \deff{Unknown} \textasciitilde \ of knowledge of the agent. 148 | \item \deff{Static} / \deff{Dynamic} - whether the environment changes while the agent is deciding actions. 149 | \end{itemize} 150 | The real world is partially observable, stochastic, sequential, dynamic, continuous, multi-agent. 151 | 152 | 153 | \section{Uninformed Search} 154 | \subsection{General Search} 155 | \subsubsection{Search Problem Definitions} 156 | A Search problem consists of 1) \deff{State representation}, $s_i$, for each environment instance, 2) \deff{Goal test}, $\text{isGoal } : s_{i} \to \{ 0, 1\}$, that determines if a state is a goal, 3) \deff{Action Function}, $\text{action } : s_{i} \to A$, that returns possible actions for every state, 4) \deff{Action Cost}, $\text{cost } : (s_{i}, a_{j}, s^{'}_{i}) \to V$, that returns cost $v$ of taking action $a_{i}$ of state $s_{i}$ to reach $s^{'}_{i}$, and 5) \deff{Transition Model}, $T : (s_{i}, a_{j}) \to s^{'}_{i}$ representing the state transition. 157 | \smallskip 158 | 159 | A transition model describes the problem in a dynamic and efficient way, as it does not list all states' actions. 160 | \smallskip 161 | 162 | \deff{Uninformed Search} are search algorithms without domain knowledge beyond the search problem formulation. 163 | 164 | 165 | \subsubsection{Generic Algorithm} 166 | The only difference is how each algorithm implements the \codein{frontier} for searching. 167 | 168 | \begin{lstlisting} 169 | frontier = {initial state} 170 | while frontier not empty: 171 | current = frontier.pop() 172 | // checking 173 | if isGoal(current) 174 | return path found 175 | // exploration 176 | for a in actions(current): 177 | frontier.push(T(current, a)) 178 | return failure 179 | \end{lstlisting} 180 | 181 | For \textbf{correctness} of search algorithms, we need to ensure 1) \deff{Completeness} - whether an algorithm will find a solution when one exists \textbf{and} correct report failure if not exists, and 2) \deff{Optimality} - whether an algorithm finds a solution with lowest path cost among all solutions. 182 | Note: An optimal solution must be complete. 183 | \smallskip 184 | 185 | Implementation wise, there are 1) \deff{Tree-Search} that allows revisiting of nodes, and 2) \deff{Graph-Search} that do not allow revisit to states unless the new cost is smaller than current one (by maintaining a \codein{reached} hash table upon adding nodes to \codein{frontier}. 186 | 187 | 188 | \subsection{Search Algorithms} 189 | \deff{Breadth-First Search} (BFS) : Use \codein{Queue<>} for \codein{frontier}. Possible improvement is to perform \textbf{Goal checking on pushing to frontier} to reduce storage. 190 | 191 | \smallskip 192 | 193 | \deff{Depth-First Search} (DFS) : Use \codein{Stack<>} for \codein{frontier}. 194 | \smallskip 195 | 196 | \deff{Uniform-Cost Search} (UCS) : Use \codein{PriorityQueue<>} for \codein{frontier}. Essentially Dijkstra's Algorithm that always explores the node with shortest path cost in the frontier. 197 | \\ 198 | Note: Need to ensure all costs are larger than some constant $\epsilon > 0$. Therefore, it cannot be used for negative / zero costs (just like Dijkstra). 199 | \smallskip 200 | 201 | \deff{Depth-Limited Search} (DLS) : DFS but with a limit on the maximum depth, $l$, which may be determined using domain knowledge. 202 | \smallskip 203 | 204 | \deff{Iterative Deepening Search} (IDS) : Perform DLS repeated with $l = 1, 2, ...$. Intuitive, the algorithm compromises running time for better memory usage. 205 | 206 | 207 | 208 | \section{Informed Search} 209 | \subsection{Heuristics} 210 | \deff{Heuristic Function}, $h = h(n)$, approximates the shortest distance from a state to the nearest goal.\\ 211 | $h(n)$ tries to approximate the actual distance function $h^*(n)$. 212 | \smallskip 213 | 214 | A heuristic is \deff{admissible} if for any state $n$, 215 | \[ 216 | h(n) \leq h^{*}(n) 217 | \] 218 | , which means the heuristic might under-estimate but never over-estimates. 219 | \smallskip 220 | 221 | A heuristic is \deff{consistent} if for all states $n$ and its successor $n'$, 222 | \[ 223 | h(n) \leq \text{cost}(n, a, n') + h(n') 224 | \] 225 | , which means priority $f$ in A* is non-decreasing along a path if a consistent heuristic is chosen. 226 | \\ 227 | Consistent heuristics are always admissible. 228 | \smallskip 229 | 230 | For two heuristics, $h_1$ \deff{dominates} $h_2$ iff. $h_1(n) \geq h_2(n)$ for all states $n$. \\ 231 | Note: dominance is defined for all heuristics. However, for two admissible heuristics, the dominating one is preferred. 232 | 233 | 234 | 235 | 236 | \subsection{Informed Search Algorithms} 237 | The two Informed Search algorithms are based on UCS, but incorporate domain knowledge via $h$. 238 | \newline 239 | 240 | \deff{Greedy Best-First Search} : Use \deff{Evaluation Function}, $f(n) = h(n)$ as priority. Intuitively, it picks the state "seemingly" closest to the goal. 241 | 242 | \begin{itemize} 243 | \item \textbf{Incomplete} under Tree-Implementation, and \textbf{Complete} under Graph-Implementation. 244 | \item \textbf{Not optimal} under both implementations 245 | \end{itemize} 246 | 247 | 248 | \deff{A* Search} : Use \deff{Evaluation Function}, $f(n) = g(n) + h(n)$ as priority where $g(n)$ is the current path cost. 249 | \\ 250 | \deff{Limited-Graph Search} (LGS) : A modified Graph-Implementation version of A* that adds nodes to \codein{reached} table on pop instead of pushing. 251 | \begin{itemize} 252 | \item Tree-Implementation of A* is \textbf{Optimal} for admissible $h$. 253 | \item LGS is \textbf{Optimal} for consistent $h$. 254 | \end{itemize} 255 | 256 | 257 | 258 | 259 | \section{Local Search} 260 | \deff{Local Search} only concerns with goal state(s) but not how it is found or its cost. 261 | \smallskip 262 | 263 | Local Search is \textbf{Incomplete}, but it uses less space ($O(b)$ for branching factor $b$) and is applicable to larger and finite search space. 264 | \smallskip 265 | 266 | \deff{Complete-State Formulation} - Every state has all components of a solution. Each state is a potential solution. 267 | 268 | \subsection{Local Search Algorithm} 269 | \deff{Hill Climbing} (aka. \deff{Steepest Ascent - Greedy Strategy}) - It stores only current state. In each iteration, find a successor that improves - 1) use actions and transitions to determine successors, and 2) use "heuristic-liked" values (e.g. $f(n) = -h(n)$) to evaluate each state. The algorithm terminates with a state when the value $f$ cannot be improved. 270 | \\ 271 | Note: The algorithm may fail as it can terminate at a local maximum / plateau. 272 | 273 | \subsubsection{Hill Climbing Variations} 274 | \begin{itemize} 275 | \item \deff{Slideways Move} - Replace $<$ with $\leq$, to allow continuation with neighbours of same value and to traverse plateaus. 276 | \item \deff{Stochastic \textasciitilde} - Choose randomly a state with better value (not the best value) to explore. This takes longer time to find a solution but gives more flexibility and randomness. Relieve "local maximum" issue. 277 | \item \deff{First-Choice \textasciitilde} - Generate successors until one with better value than current is found. 278 | \item \deff{Random-Restart \textasciitilde} - Use a loop to randomly pick a new starting state. Keep running until a solution is found. 279 | \end{itemize} 280 | 281 | \subsubsection{Local Beam Search} 282 | \deff{Local Beam Search} - Similar to Hill Climbing but start with $k$ random starting states. Each iteration generates successors of all $k$ states and choose new $k$ ones to explore. Stochastic elements can also be incorporated into this algorithm. 283 | \\ 284 | Note: It is not equivalent to $k$-parallel Hill Climbing. 285 | \\ 286 | Note: Local Beam requires problems with different possible starting states, otherwise it cannot be run. 287 | 288 | 289 | 290 | \section{Constraint Satisfication Problem (CSP)} 291 | \subsection{CSP Formulation} 292 | \begin{itemize} 293 | \item \deff{State Representation}, for variables $X = \{ x_1, ..., x_n \}$ and set of domains $D = \{ d_1, ..., d_n \}$, so that $x_i$ has domain $d_i$. 294 | \item \deff{State}, where the initial / intermediate / goal state(s) have variables unassigned / partially assigned / all assigned. 295 | \item \deff{Goal Test}, with constrains $C = \{c_1, ..., c_m \}$ to satisfy 296 | \item \deff{Actions} and \deff{Transitions} 297 | \end{itemize} 298 | Costs are not utilised in CSP. 299 | 300 | \subsubsection{Constraint Graph} 301 | Depending on the \deff{Scope}, number of variables involved, of constraints, they can be categorised into 1) \deff{Unary}, \deff{Binary}, and \deff{Global} which involves $1$, $2$, and $\geq 3$ variables respectively. 302 | \smallskip 303 | 304 | A \deff{Constraint Graph} is a representation for constraints where each variable is a vertex, each binary constraint is an edge between two variables, and each global constraint is expressed as multiple binary constraints using linking a vertex. 305 | 306 | 307 | 308 | \section{Intermediate Results / Lemmas} 309 | \subsection{From Tutorials} 310 | \citeqn{(Quiz1 Qn9)} Testing goal upon pushing (than poppig) to frontier can save at most $(b^{d+1} - b)$ nodes in BFS. 311 | \\ 312 | \citeqn{(Quiz2 Qn12)} For Uninformed Search problems with the goal node near the root, the branching factor finite, and all action costs equal, \textbf{BFS} is preferred. 313 | \\ 314 | \citeqn{(Quiz2 Qn13)} For Uninformed Search problems with all nodes at a certain depth being goal nodes and all action costs equal, \textbf{DFS} is preferred. 315 | \\ 316 | Note the difference between pushing order and popping order in DFS. 317 | 318 | 319 | \subsection{From Past Questions} 320 | \citeqn{(AY19/20Sem1 Midterm Qn1)} A* Graph Search with consistent heuristic is guaranteed to visit no more nodes than UCS. 321 | \\ 322 | \citeqn{(AY19/20Sem2 Midterm Qn1)} In a A* Graph Search problem, leaving a consistent heuristic $h(n)$ unchanged: 323 | \begin{itemize} 324 | \item After adding edges to the transition graph, $h$ might not be still consistent. 325 | \item After removing edges from the transition graph, $h$ is still consistent. 326 | \end{itemize} 327 | 328 | 329 | 330 | 331 | 332 | \section{Summary} 333 | 334 | \subsection{Uninformed Search Algorithms} 335 | For Tree-Search implementation: 336 | \begin{center} 337 | \begin{tabular}{||c || c | c | c | c | c||} 338 | \hline 339 | Criterion & BFS & UCS & DFS & DLS & IDS \\ 340 | \hline \hline 341 | Complete? & Yes$^{[1]}$ & Yes$^{[1][2]}$& No & No & Yes$^{[1]}$ \\ 342 | \hline 343 | Optimal? & Yes$^{[1]}$ & Yes & No & No & Yes$^{[3]}$ \\ 344 | \hline 345 | Time & $O(b^d)$ & $O(b^{1+[C^*/\epsilon]})$ & $O(b^m)$ & $O(b^l)$ & $O(b^d)$ \\ 346 | \hline 347 | Space & $O(b^d)$ & $O(b^{1+[C^*/\epsilon]})$ & $O(bm)$ & $O(bl)$ & $O(bd)$ \\ 348 | \hline 349 | \end{tabular} 350 | \end{center} 351 | 352 | For Graph-Search implementation, \textbf{all have time and space complexity $O(V+E)$}. 353 | \begin{center} 354 | \begin{tabular}{||c || c | c | c | c | c||} 355 | \hline 356 | Criterion & BFS & UCS & DFS & DLS & IDS \\ 357 | \hline \hline 358 | Complete? & Yes$^{[1]}$ & Yes$^{[1][2]}$& Yes$^{[1]}$ & No & Yes$^{[1]}$ \\ 359 | \hline 360 | Optimal? & Yes$^{[1]}$ & Yes & No & No & Yes$^{[3]}$ \\ 361 | \hline 362 | \end{tabular} 363 | \end{center} 364 | $[1]$ If $b$ finite \textbf{AND} (state space finite \textbf{OR} has a solution) \\ 365 | $[2]$ If the $\epsilon$ assumption is satisfied for all costs \\ 366 | $[3]$ If all costs are identical 367 | 368 | 369 | \end{multicols} 370 | \end{document} 371 | -------------------------------------------------------------------------------- /MA2104/MA2104_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | %\usepackage{fonts} 21 | \usepackage{multicol,multirow} 22 | \usepackage{spverbatim} 23 | \usepackage{graphicx} 24 | \usepackage{ifthen} 25 | \usepackage[landscape]{geometry} 26 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 27 | \usepackage{booktabs} 28 | \usepackage{fontspec} 29 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 30 | \setsansfont{Fira Sans} 31 | \setmonofont{Inconsolata} 32 | \usepackage{unicode-math} 33 | \setmathfont{TeX Gyre Pagella Math} 34 | \usepackage{microtype} 35 | 36 | \usepackage{empheq} 37 | 38 | % new: 39 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 40 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 41 | } 42 | \makeatother 43 | 44 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 45 | { \geometry{margin=0.4in} } 46 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 47 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | } 50 | \pagestyle{empty} 51 | \makeatletter 52 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 53 | {-1ex plus -.5ex minus -.2ex}% 54 | {0.5ex plus .2ex}%x 55 | {\sffamily\large}} 56 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 57 | {-1explus -.5ex minus -.2ex}% 58 | {0.5ex plus .2ex}% 59 | {\sffamily\normalsize\itshape}} 60 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 61 | {-1ex plus -.5ex minus -.2ex}% 62 | {1ex plus .2ex}% 63 | {\normalfont\small\itshape}} 64 | \makeatother 65 | \setcounter{secnumdepth}{0} 66 | \setlength{\parindent}{0pt} 67 | \setlength{\parskip}{0pt plus 0.5ex} 68 | % ----------------------------------------------------------------------- 69 | 70 | \usepackage{academicons} 71 | 72 | \begin{document} 73 | 74 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 75 | \everymath{\color{mathBlue}} 76 | \everydisplay{\color{mathBlue}} 77 | 78 | % for vector notation in this module 79 | \newcommand{\vect}[1]{\boldsymbol{#1}} 80 | 81 | \footnotesize 82 | %\raggedright 83 | 84 | \begin{center} 85 | {\huge\sffamily\bfseries MA2104 Cheatsheet} \huge\bfseries\\ 86 | by Wei En \& Yiyang, AY21/22 87 | \end{center} 88 | \setlength{\premulticols}{0pt} 89 | \setlength{\postmulticols}{0pt} 90 | \setlength{\multicolsep}{1pt} 91 | \setlength{\columnsep}{1.8em} 92 | \begin{multicols}{3} 93 | 94 | 95 | 96 | 97 | % ----------------------------------------------------------------------- 98 | 99 | \section{Chapter 01 - Vectors in 3D Space} 100 | \subsection{Vectors} 101 | Vector projection of $\vb*{a}$ onto $\vb*{b}$: $\text{proj}_{\vb*{b}} \vb*{a} = \frac{\vb*{a} \cdot \vb*{b}}{\vb*{b} \cdot \vb*{b}} \vb*{b}$ 102 | \\ 103 | Scalar projection of $\vb*{a}$ onto $\vb*{b}$: $\text{comp}_{\vb*{b}} \vb*{a} = \frac{\vb*{a} \cdot \vb*{b}}{\norm{\vb*{b}}}$ 104 | 105 | \subsection{Dot \& Cross Product} 106 | $$\vb*{a} \cdot \vb*{b} = \|a\|\|b\|\cos\theta,\quad 107 | \|\vb*{a} \times \vb*{b}\| = \|a\|\|b\|\sin\theta$$ where $\theta$ is the angle between vectors $\vb*{a}$ and $\vb*{b}$. 108 | 109 | \subsection{Prop Ch01.3.5 - Scalar Triple Product} 110 | $$|\vb*{a} \cdot (\vb*{b} \cross \vb*{c})| = \left|\det\begin{pmatrix} 111 | \vb*{a}_1 & \vb*{a}_2 & \vb*{a}_3 \\ 112 | \vb*{b}_1 & \vb*{b}_2 & \vb*{b}_3 \\ 113 | \vb*{c}_1 & \vb*{c}_2 & \vb*{c}_3 \\ 114 | \end{pmatrix}\right|$$ is the volume of the parallelepiped determined by vectors $\vb*{a}, \vb*{b}, \vb*{c}$. 115 | 116 | \section{Chapter 02 - Curves and Surfaces} 117 | \subsection{Curve} 118 | \subsubsection{Tangent Vector} 119 | Tangent vector to a curve $C$ paramaterised by $R(t) = (f(t), g(t), h(t))$ at $R(a)$ on the curve is given by 120 | \[ 121 | R'(a) = \langle f'(a), g'(a), h'(a) \rangle. 122 | \] 123 | 124 | \subsubsection{Arc Length Formula} 125 | The length of curve $C: R(t) = (f(t), g(t), h(t))$ between $R(a)$ and $R(b)$ is 126 | \[ 127 | \int_a^b \ \norm{R'(t)} \,dt = \int_a^b \ \sqrt{f'(t)^2 + g'(t)^2 + h'(t)^2} \,dt. 128 | \] 129 | provided the first derivatives are continuous. 130 | 131 | 132 | \subsection{Surfaces} 133 | \subsubsection{Cylinder} 134 | A surface is a cylinder if there is a plane $P$ such that all the planes parallel to $P$ intersect the surface in the same curve. 135 | 136 | \subsubsection{Quadric Surfaces} 137 | \begin{itemize} 138 | \item Elliptic Paraboloid: $\frac{x^2}{a^2} + \frac{y^2}{b^2} = \frac{z}{c}$ 139 | \item Hyperbolic Paraboloid: $\frac{x^2}{a^2} - \frac{y^2}{b^2} = \frac{z}{c}$ 140 | \item Ellipsoid: $\frac{x^2}{a^2} + \frac{y^2}{b^2} + \frac{z^2}{c^2} = 1$ 141 | \item Elliptic Cone: $\frac{x^2}{a^2} + \frac{y^2}{b^2} - \frac{z^2}{c^2} = 0$ 142 | \item Hyperboloid of 1 Sheet: $\frac{x^2}{a^2} + \frac{y^2}{b^2} - \frac{z^2}{c^2} = 1$ 143 | \item Hyperboloid of 2 Sheet: $\frac{x^2}{a^2} + \frac{y^2}{b^2} - \frac{z^2}{c^2} = -1$ 144 | \end{itemize} 145 | 146 | 147 | 148 | \section{Chapter 03 - Multivariable Functions} 149 | \subsection{Limit, Continuity \& Differentiability} 150 | \subsubsection{Limit for 2D Functions} 151 | For function $f$ with domain $D \subset \mathbb{R^2}$ that contains points arbitrarily close to $(a, b)$, then 152 | \[ 153 | \lim_{(x, y) \to (a, b)} f(x, y) = L 154 | \] 155 | if for any number $\epsilon > 0$ there exists a number $\delta > 0$ such that $|f(x,y) - L| < \epsilon$ whenever $0 < \sqrt{(x-a)^2 + (y-b)^2} < \delta$. 156 | 157 | The limit exists iff. the limit \textbf{exists and is the same for all continuous paths} to $(a, b)$. 158 | 159 | \subsubsection{Clairaut's Theorem} 160 | For function $f$ defined on $D \subset \mathbb{R}^2$ that contains $(a, b)$, if the functions $f_{xy}$ and $f_{yx}$ are both continuous on $D$, then 161 | \[ 162 | f_{xy}(a,b) = f_{yx}(a, b). 163 | \] 164 | 165 | \subsubsection{Differentiability for 2D Functions} 166 | For function $f$ defined on $D \subset \mathbb{R}^2$ and differentiable at $(a, b)$ within the interior of $D$, 167 | \[ 168 | \lim_{(h, k) -> (0,0)} \ \frac{f(a+h, b+k)-f(a,b) - L(h,k)}{\sqrt{h^2+k^2}} = 0, 169 | \] 170 | where $L: \mathbb{R}^2 \to \mathbb{R}$ is a linear map defined as the total derivative of $f$ at $(a, b)$: 171 | \[ 172 | L(h, k) = D_{f(a, b)} (h, k) = f_x(a, b)h + f_y(a, b)k. 173 | \] 174 | 175 | \subsubsection{Notes about Differentiability} 176 | Consider $f$ at a point $(a, b)$: 177 | \begin{itemize} 178 | \item $f_x$ and $f_y$ exist $\not \implies$ $f$ differentiable 179 | \item $f_x$ and $f_y$ exist \& continuous $\implies$ $f$ differentiable \emph{(Differentiability Theorem)} 180 | \item $f$ differentiable $\not{\implies}$ $f_x$ and $f_y$ continuous 181 | \end{itemize} 182 | 183 | \subsubsection{Linear Approximation} 184 | \[ 185 | f(a+h, b+k) \approx f(a,b) + f_x(a, b)h + f_y(a, b)k 186 | \] 187 | 188 | 189 | \subsection{Gradient Vector} 190 | \subsubsection{Gradient Vector} 191 | The gradient vector of $f$ defined on $D \subset \mathbb{R}^2$ at $(a, b) \in D$ is defined as: 192 | \[ 193 | \nabla f(a,b) = \langle f_x(a, b), f_y(a, b) \rangle 194 | \] 195 | 196 | \subsubsection{Directional Directive} 197 | The directional directive of $f$ defined on $D \subset \mathbb{R}^2$ in the direction of the unit vector $\vb*{u} = \langle \vb*{u}_1, \vb*{u}_2 \rangle$ is 198 | \[ 199 | D_{f(a,b)}(\vb*{u}) = \lim_{h \to 0} \frac{f(a + h\vb*{u}_1, b + h\vb*{u}_2) - f(a, b)}{h} = \nabla f(a,b) \cdot \vb*{u}. 200 | \] 201 | 202 | \subsubsection{Perpendicular Vector of Level Sets} 203 | $\nabla f(a,b)$ is orthogonal to the $f(a,b)$-level curve of $f$ at $(a,b)$. 204 | 205 | \section{Chapter 04 - Calculus on Surfaces} 206 | \subsection{Implicit Differentiation} 207 | \subsubsection{Prop Ch04.1.4} 208 | For $F$ defined on $D \subset \mathbb{R}^3$ where $F(a,b,c)=k$ defines $z$ as a differentiable function of $x$ and $y$ near $(a,b,c)$, and $F_z(a,b,c) \neq 0$, 209 | \[ 210 | \frac{\partial z}{\partial x}(a,b,c) = - \frac{F_x(a,b,c)}{F_z(a,b,c)}, \ 211 | \frac{\partial z}{\partial y}(a,b,c) = - \frac{F_y(a,b,c)}{F_z(a,b,c)} 212 | \] 213 | 214 | \subsection{Extrema} 215 | \subsubsection{Extreme Value Theorem} 216 | If $f : D \to \mathbb{R}$ is continuous on a \textbf{closed and bounded} set $D \subset \mathbb{R}^2$, then $f$ has at least one global maximum and one global minimum. 217 | 218 | \subsubsection{Steps for Finding Global Extrema} 219 | For $f : D \to \mathbb{R}$ where $D$ is closed and bounded, 220 | \begin{enumerate} 221 | \item Find all critical points of $f$ and their corresponding $f$-values. 222 | \item Find the extreme values of $f$ on boundary of $D$. 223 | \item Compare. 224 | \end{enumerate} 225 | 226 | 227 | \subsubsection{Method of Lagrange Multiplier} 228 | To find the extrema of differentiable $f : D \to \mathbb{R}$ subject to curve $C: g(x,y)=k$ for some $k \in \mathbb{R}$, 229 | \begin{enumerate} 230 | \item Find all points $(a,b)$ for $\nabla g(a,b) \ne 0$ and values $\lambda$ s.t. 231 | \[ 232 | \nabla f(a,b) = \lambda \nabla g(a,b), \ g(a,b) = k, 233 | \] and evaluate $f$ at all these points. 234 | \item Find the extreme values of $f$ on the boundary of $C$. 235 | \item Compare. 236 | \end{enumerate} 237 | 238 | 239 | 240 | \section{Chapter 05 \& 06 - Integration} 241 | \subsection{Fubini's Theorem} 242 | If $f$ is continuous on the rectangle $D = [a,b] \times [c,d]$, then, 243 | \[ 244 | \iint_D f(x,y) dA = \int_{a}^{b} \int_{c}^{d} f(x,y) dy dx = \int_{c}^{d}\int_{a}^{b} f(x,y) dx dy 245 | \] 246 | 247 | Its equivalence in $\mathbb{R}^3$ for triple integral also holds. 248 | 249 | \subsection{Change of Coordinates} 250 | \subsubsection{Double Integral in Polar Coordinates} 251 | Transform between $(x,y)$ and $(r, \theta)$: 252 | \begin{align*} 253 | &\left\{ 254 | \begin{aligned} 255 | x &= r \cos\theta \\ 256 | y &= r \sin\theta \\ 257 | \end{aligned}\right. 258 | & 259 | \left\{\begin{aligned} 260 | r &= \sqrt{x^2 + y^2} \\ 261 | \theta &= \tan^{-1}{(y/x)} \\ 262 | \end{aligned}\right. 263 | \end{align*} 264 | In addition, $dA = dx dy = r dr d\theta$. 265 | 266 | \subsubsection{Triple Integral in Cylindrical Coordinates} 267 | Transform between $(x,y,z)$ and $(r, \theta, z)$: 268 | \begin{align*} 269 | &\left\{ 270 | \begin{aligned} 271 | x &= r \cos\theta \\ 272 | y &= r \sin\theta \\ 273 | z &= z \\ 274 | \end{aligned}\right. 275 | & 276 | \left\{\begin{aligned} 277 | r &= \sqrt{x^2 + y^2} \\ 278 | \theta &= \tan^{-1}{(y/x)} \\ 279 | z &= z \\ 280 | \end{aligned}\right. 281 | \end{align*} 282 | In addition, $dV = dx dy dz = r dr d\theta dz$. 283 | 284 | \subsubsection{Triple Integral in Spherical Coordinates} 285 | Transform between $(x,y,z)$ and $(\rho, \theta, \phi)$: 286 | \begin{align*} 287 | &\left\{ 288 | \begin{aligned} 289 | x &= \rho \cos\theta \sin\phi \\ 290 | y &= \rho \sin\theta \sin\phi \\ 291 | z &= \rho \cos\phi 292 | \end{aligned}\right. 293 | & 294 | \left\{\begin{aligned} 295 | \rho &= \sqrt{x^2 + y^2 + z^2} \\ 296 | \theta &= \tan^{-1}{(y/x)} \\ 297 | \phi &= \cos^{-1}(z/\rho) \le \pi \\ 298 | \end{aligned}\right. 299 | \end{align*} 300 | In addition, $dV = dx dy dz = \rho^2 \sin\phi d\rho d\theta d\phi$. 301 | 302 | 303 | \subsection{Application of integration} 304 | For a given region $D$ in $\mathbb{R}^2$, the area of the region can be calculated as $\text{Area}(D) = \iint_D 1 dA$. 305 | \\ 306 | For a given solid $E$ in $\mathbb{R}^3$, the volume of the solid can be calculated as $\text{Volume}(E) = \iiint_E 1 dV$. 307 | 308 | 309 | \section{Chapter 07 - Change of Coordinates} 310 | \subsection{Planar Transformation} 311 | A map $T: S \to R$ is a planar transformation if it is a differentiable map whose inverse is differentiable. 312 | 313 | Therefore, to show $T: S \to R$ is a planar transformation, it must satisfy: 314 | \begin{enumerate} 315 | \item $T$ is differentiable 316 | \item The inverse $T^{-1}$ exists 317 | \item The inverse $T^{-1}$ is differentiable 318 | \end{enumerate} 319 | 320 | \subsection{Change of Coordinates} 321 | \subsubsection{2D Jacobian Determinant} 322 | The \emph{Jacobian} of the transformation $T(u, v) = (x(u,v), y(u,v))$ is defined 323 | \[ 324 | \frac{\partial (x,y)}{\partial (u,v)} 325 | = \begin{vmatrix} x_u & x_v \\ y_u & y_v\end{vmatrix} 326 | = \frac{\partial x}{\partial u} \frac{\partial y}{\partial v} 327 | - \frac{\partial x}{\partial v} \frac{\partial y}{\partial u} 328 | \] 329 | 330 | \subsubsection{Change of Variable in Double Integral} 331 | Let $T : S \to R$ be a planar transformation, where $S$ lies in the $uv$-plane and $R$ lies in the $xy$-plane. Let $A$ and $A'$ denote the area in the $xy$- and $uv$-plane respectively. For a two-var. function from $xy$-plane to $\mathbb{R}$, 332 | \[ 333 | \iint_{R} f(x,y) dA = \iint_{R} f \circ T(u, v) \left| \frac{\partial (x,y)}{\partial (u,v)} \right| dA' 334 | \] 335 | Equivalently, 336 | \[ 337 | \iint_{R} f(x,y)dx dy = \iint_S f(x(u,v), y(u,v)) \left| \frac{\partial (x,y)}{\partial (u,v)} \right| du dv 338 | \] 339 | 340 | The 3D Jacobian has similar expressions and properties. 341 | 342 | \subsection{Inverse Function Theorem} 343 | 344 | \section{Chapter 08 - Line Integrals} 345 | \subsection{Line Integral} 346 | \subsubsection{Line Integral of Functions} 347 | For curve $C$ parameterised by $R(t) = (x(t), y(t), z(t), \ a \le t \le b$, and a 3-var function $f(x,y, z)$, the line integral 348 | \[ 349 | \int_C f(x, y, z) ds = \int_{a}^{b} f(x(t), y(t), z(t)) \norm{R'(t)} dt 350 | \] 351 | 352 | \subsubsection{Line Integral of Vector Fields} 353 | Let $\vb{C} = (C, o)$ be a smooth oriented curve in $\mathbb{R}^3$ parameterised by $R(t) = (x(t), y(t), z(t)), \ a \le t \le b$, and let $\vb{F} = \vb{F}(x, y, z)$ be a continuous vector field along $C$. Then, the line integral of $\vb{F}$ along $\vb{C}$ 354 | \[ 355 | \int_{\vb{C}} \vb{F} \cdot d\vb{r} = \int_{a}^{b} \vb{F}(x(t), y(t), z(t)) \cdot R'(t) dt 356 | \] 357 | For $\vb{F} = \langle X,Y,Z \rangle$ in its component form, 358 | \[ 359 | \int_{\vb{C}} \vb{F} \cdot d\vb{r} = \int_{\vb{C}} Xdx + Ydy + Zdz 360 | \] 361 | 362 | In addition, if $-\vb{C}$ is the curve $\vb{C}$ with opposite orientation, 363 | \[ 364 | \int_{\vb{C}} \vb{F} \cdot d\vb{r} = - \int_{-\vb{C}} \vb{F} \cdot d\vb{r} 365 | \] 366 | 367 | 368 | \subsection{Conservative Vector Fields} 369 | Whenever a vector field $\vb{F}$ on some open domain stratifies $\vb{F} = \nabla f$ for some differentiable function $f$, then we call $\vb{F}$ a \emph{conservative vector field} and $f$ the \emph{potential function} of $\vb{F}$. 370 | 371 | \subsubsection{Tests for Conservativity} 372 | \begin{itemize} 373 | \item To show conservative: find a $f$ s.t. $\nabla f = \vb{F}$ (by definition). 374 | \item To show conservative: if $\vb{F}(x, y) = \langle X,Y \rangle$ is defined over an \textbf{open and simply-connected} region $D \subset \mathbb{R}^2$, then need to show 375 | \[ 376 | \frac{\partial X}{\partial y} = \frac{\partial Y}{\partial x} 377 | \] 378 | \item To show non-conservative: find two oriented curves, $\vb{C_1}$ and $\vb{C_2}$, with same starting and ending points, s.t. 379 | \[ 380 | \int_{\vb{C_1}} \vb{F} \cdot d\vb{r} \ne \int_{\vb{C_2}} \vb{F} \cdot d\vb{r} 381 | \] 382 | \end{itemize} 383 | 384 | \subsubsection{Gradient Theorem} 385 | For a 3-var function $f$ whose gradient vector $\nabla f$ is continuous along $\vb{C} = (C, 0)$ parameterised by $R(t) = (x(t), y(t), z(t)), \ a \le t \le b$, 386 | \[ 387 | \int_{\vb{C}} \nabla f \cdot d\vb{r} = f(R(b)) - f(R(a)) 388 | \] 389 | 390 | \subsubsection{Green's Theorem, Version I} 391 | Let $\vb{C} = (C, o)$ be a \textbf{positively oriented}, piecewise differentiable loop in $\mathbb{R}^2$, and let $D$ be the region bounded by $C$. Then for vector field $\vb{F} = \langle X, Y \rangle$, 392 | \[ 393 | \int_{\vb{C}} \vb{F} \cdot d\vb{r} = \iint_{D} \left( \frac{\partial Y}{\partial x} - \frac{\partial X}{\partial y} \right) \,dA 394 | \] 395 | 396 | \subsubsection{Green's Theorem, Version II} 397 | Let $\vb{C} = (C, o)$ be a \textbf{positively oriented}, piecewise differentiable loop in $\mathbb{R}^2$, and let $D$ be the region bounded by $C$. Then for vector field $\vb{F}$, let $\vb{n}(x, y)$ denote the \textbf{outward pointing} unit normal vector to $S$, 398 | \[ 399 | \int_{\vb{C}} \vb{F} \cdot \vb{n} \,ds = \iint_{D} \text{div } \vb{F} \,dA 400 | \] 401 | Algebraically, the outward pointing unit normal vector is 402 | \[ 403 | \vb{n}(t) = \frac{\langle y'(t), -x'(t) \rangle}{\norm{R'(t)}} 404 | = \frac{\langle y'(t), -x'(t) \rangle}{\sqrt{x'(t)^2 + y'(t)^2}} 405 | \] 406 | Note, the integral is called the outward flux of $\vb{F}$ across $C$. 407 | 408 | 409 | \section{Chapter 09 - Surface Integrals} 410 | \subsection{Surface Integral} 411 | \subsubsection{Surface Integral of Functions} 412 | Let $R: D \to S$ be a (differentiable) parameterisation of surface $S$, then 413 | \[ 414 | \iint_{S} f(x,y,z) dS = \iint_{D} f(x(u,v), y(u,v), z(u,v)) \norm{R_{u} \times R_{v}} \,dA 415 | \] 416 | \textbf{Special case:} when $S$ is the graph of a 2-var function $g(x,y)$ for $(x,y) \in D$ for some domain $D$, 417 | \[ 418 | \iint_{S} f(x,y,z) dS = \iint_{D} f(x,y,g(x,y)) \Big( \sqrt{g_{x}^2 + g_{y}^2 + 1} \Big) \,dA 419 | \] 420 | 421 | \subsubsection{Orientation on Surface} 422 | A (differentiable) surface $S \in \mathbb{R}^3$ is \emph{orientable} if it is possible to define for every $(x,y,z) \in S$, a unit normal vector $\vb{n}(x,y,z)$ to $S$ with initial point $(x,y,z)$ such that $\vb{n}$ varies continuously. 423 | \\ 424 | An orientable surface has two orientations: 425 | \[ 426 | \vb{n} = \pm \frac{R_u \times R_v}{\norm{R_u \times R_v}} 427 | \] 428 | 429 | \subsubsection{Surface Integral of Vector Fields} 430 | Let $\vb{S} = (S, \vb{n})$ be an oriented surface where $R: D \to S$ is the parameterisation for $S$, and let $\vb{F}$ be a vector field along the surface, 431 | \[ 432 | \iint_{\vb{S}} \vb{F} \cdot d\vb{S} 433 | = \iint_{\vb{S}} \vb{F} \cdot \vb{n} \,dS 434 | = \iint_D \vb{F} \cdot (R_u \times R_v) \,dA 435 | \] 436 | \textbf{Special case:} when $S$ is the graph of a 2-var function $g(x,y)$, let $\vb{S}$ denote $S$ with \textbf{upward orientation}, and let $\vb{F} = \langle X, Y, Z \rangle$, 437 | \[ 438 | \iint_{\vb{S}} \vb{F} \cdot d\vb{S} 439 | = \iint_D \Big( -\vb{X}\frac{\partial g}{\partial x} -\vb{Y}\frac{\partial g}{\partial Y} + \vb{Z} \Big) \,dA 440 | \] 441 | 442 | \subsection{Gauss' Theorem} 443 | \subsubsection{Divergence} 444 | For a vector field $\vb{F} = \langle X, Y, Z \rangle$, the \emph{divergence} of $\vb{F}$ is defined as 445 | \[ 446 | \text{div } \vb{F} = \nabla \cdot \vb{F} 447 | = \frac{\partial X}{\partial x}(x,y,z) + \frac{\partial Y}{\partial y}(x,y,z) + \frac{\partial Z}{\partial z}(x,y,z) 448 | \] 449 | 450 | \subsubsection{Gauss' Theorem} 451 | Let $E$ be a solid region where the boundary surface $S$ is piece-wise smooth, and let $\vb{S}$ denote $S$ with \textbf{outward orientation}. Then for a vector field $\vb{F}$ defined over $S$, 452 | \[ 453 | \iint_{\vb{S}} \vb{F} \cdot d\vb{S} = \iiint_E \text{div } \vb{F} \,dV 454 | \] 455 | 456 | 457 | \subsection{Stokes' Theorem} 458 | \subsubsection{Curl} 459 | For a vector field $\vb{F} = \langle X,Y,Z \rangle$, the \emph{curl} of $\vb{F}$ is defined as 460 | \[ 461 | \text{curl } \vb{F} = \nabla \times \vb{F} = \left\langle 462 | \frac{\partial Z}{\partial y} - \frac{\partial Y}{\partial z}, 463 | \frac{\partial X}{\partial z} - \frac{\partial Z}{\partial x}, 464 | \frac{\partial Y}{\partial x} - \frac{\partial X}{\partial y} 465 | \right\rangle 466 | \] 467 | 468 | \subsubsection{Induced Orientation} 469 | For oriented surface $\vb{S} = (S, \vb{n})$ with boundary $C$ being a simple loop, the \emph{induced orientation}, $\vb{o}$ of $\vb{n}$ is one such that if you want along $C$ in the orientation $\vb{o}$ with your head pointing in the direction of $\vb{n}$, then $S$ will always be on your left. 470 | 471 | \subsubsection{Stokes' Theorem} 472 | For oriented surface $\vb{S} = (S, \vb{n})$ bounded by a simple curve $C$, and let $\vb{C} = (C, \vb{o})$ be the oriented loop with induced orientation, then for a vector field $\vb{F}$, 473 | \[ 474 | \iint_{\vb{S}} \text{curl } \vb{F} \cdot d\vb{S} = \int_{\vb{C}} \vb{F} \cdot d\vb{r} 475 | \] 476 | 477 | \subsection{Properties of Gradient, Divergence \& Curl} 478 | For any function $f = f(x,y,z)$, 479 | \[ 480 | \nabla \times (\nabla f) = \text{curl }( \nabla f ) = \vb{0} 481 | \] 482 | 483 | For any vector fields $\vb{F} = \vb{F}(x,y,z)$, 484 | \[ 485 | \nabla \cdot (\nabla \times \vb{F} ) 486 | = \text{div } ( \text{curl } \vb{F} )= 0 487 | \] 488 | 489 | 490 | \end{multicols} 491 | \end{document} 492 | -------------------------------------------------------------------------------- /ST4238/ST4238_Cheatsheet_Trimmed.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | \usepackage{bbm} % for mathbb-ed digits 21 | %\usepackage{fonts} 22 | \usepackage{multicol,multirow} 23 | \usepackage{spverbatim} 24 | \usepackage{graphicx} 25 | \usepackage{ifthen} 26 | \usepackage[landscape]{geometry} 27 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 28 | \usepackage{booktabs} 29 | \usepackage{fontspec} 30 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 31 | \setsansfont{Fira Sans} 32 | \setmonofont{Inconsolata} 33 | \usepackage{unicode-math} 34 | \setmathfont{TeX Gyre Pagella Math} 35 | \usepackage{microtype} 36 | 37 | \usepackage{empheq} 38 | 39 | % new: 40 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 41 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 42 | } 43 | \makeatother 44 | 45 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 46 | { \geometry{margin=0.4in} } 47 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 48 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 49 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 50 | } 51 | \pagestyle{empty} 52 | \makeatletter 53 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 54 | {-1ex plus -.5ex minus -.2ex}% 55 | {0.5ex plus .2ex}%x 56 | {\sffamily\large}} 57 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 58 | {-1explus -.5ex minus -.2ex}% 59 | {0.5ex plus .2ex}% 60 | {\sffamily\normalsize\itshape}} 61 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 62 | {-1ex plus -.5ex minus -.2ex}% 63 | {1ex plus .2ex}% 64 | {\normalfont\small\itshape}} 65 | \makeatother 66 | \setcounter{secnumdepth}{0} 67 | \setlength{\parindent}{0pt} 68 | \setlength{\parskip}{0pt plus 0.5ex} 69 | % ----------------------------------------------------------------------- 70 | 71 | \usepackage{academicons} 72 | 73 | \begin{document} 74 | 75 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 76 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 77 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 78 | 79 | \everymath{\color{mathBlue}} 80 | \everydisplay{\color{mathBlue}} 81 | 82 | % for vector notation in this module 83 | \newcommand{\vect}[1]{\pmb{#1}} 84 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 85 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 86 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 87 | 88 | \footnotesize 89 | %\raggedright 90 | 91 | % \begin{center} 92 | % {\huge\sffamily\bfseries ST4238 Cheatsheet} \huge\bfseries\\ 93 | % by Yiyang, AY22/23 94 | % \end{center} 95 | \setlength{\premulticols}{0pt} 96 | \setlength{\postmulticols}{0pt} 97 | \setlength{\multicolsep}{1pt} 98 | \setlength{\columnsep}{1.8em} 99 | \begin{multicols}{3} 100 | 101 | 102 | % ----------------------------------------------------------------------- 103 | \section{1. Poisson Processes} 104 | To \textbf{homogenize a non-homogen. \textasciitilde}: [1] Define $\Lambda (t) = \int_0 ^t \lambda(u) du$ [2] $Y(s) = X(t)$ where $s = \Lambda(t)$ homogen. PoisProc. rate $1$. 105 | \\ 106 | 107 | 108 | % \subsubsection{Law of Rare Events} 109 | % Let $\epsilon_1, \epsilon_2, ...$ be independent Ber. r.v.'s with $P(\epsilon_i = 1) = p_i$ and let $S_n = \sum{i=1}^n \epsilon_i$. The exact probability for $S_n$ and Poisson probability with $\lambda = \sum_{i=1}^n p_i$ differ by at most 110 | % \[ 111 | % |P(S_n = k) - e^{-\lambda} \frac{\lambda^k}{k!}| \le \sum{i=1}^n p_i^2 112 | % \] 113 | 114 | 115 | % TODO: ranked? 116 | Given $X(t) = n$, the joint distribution of waiting time $W_1, ..., W_n$ is 117 | \[ 118 | f(w_1, w_2, ..., w_n | X(t) = n) = \frac{n!}{t^n}, 0 \le w_1 \le w_2 \le ... \le 2_n \le t 119 | \] 120 | It is the joint distribution of $n$ \textbf{ranked} independent $Unif(0, t)$ r.v.'s. 121 | 122 | \smallskip 123 | 124 | Given $X(t) = n$, the distribution of the $k$-th waiting time has the same distribution as that of the $k$-th order statistic of $n$ independent $Unif(0, t)$ r.v.'s. 125 | \[ 126 | f_k(x) = \frac{n!}{(n-k)! (k-1)!} \frac{1}{t} (\frac{x}{t})^{k-1} (1 - \frac{x}{t})^{n-k}, 0 \le k \le n 127 | \] 128 | 129 | 130 | Consider $\{ X(t), t \ge 0 \}$ and $\{ Y(t), t \ge 0 \}$ two independent Poisson Processes with rates $\lambda_1$ and $\lambda_2$. Define $W_{n}^{X}$ and $W_{m}^{Y}$ as the waiting time of the $n$-th and $m$-th waiting time of $X(t)$ and $Y(t)$ respectively. 131 | \[ 132 | P(W_{n}^{X} < W_{m}^{Y}) = \sum_{k=n}^{n+m-1} {{n+m-1}\choose{k}} (\frac{\lambda_1}{\lambda_1 133 | + \lambda_2})^k (\frac{\lambda_2}{\lambda_1 + \lambda_2})^{n+m-1-k} 134 | \] 135 | \underline{Analysis}: It is equivalent as getting $n$ or more heads in $n+m-1$ tosses where getting a head has probability $\lambda_1 / (\lambda_1 + \lambda_2)$. 136 | 137 | \smallskip 138 | 139 | \deff{Compound Poisson Process} 140 | \\ 141 | $E[X(t)] = \lambda t E[Y_i]$ and $Var[ X(t) ] = \lambda t \Big( E[Y_i]^2 + Var(Y_i) \Big)$ 142 | \[ 143 | \lambda = \lambda_1 + \lambda_2 144 | \, 145 | F(x) = \frac{\lambda_1}{\lambda_1 + \lambda_2} F_1(x) + \frac{\lambda_2}{\lambda_1 + \lambda_2} F_2(x) 146 | \] 147 | 148 | 149 | \deff{Conditional Poisson Process} 150 | \\ 151 | $E[N(t)] = E(L) t$ and $Var(N(t)) = t E(L) + t^2 Var(L)$ 152 | \\ 153 | Conditional probability of $L$ given $N(t) = n$ (posterior), 154 | \[ 155 | P(L \le x | N(t) = n) = \frac{ 156 | \int_{0}^x e^{-\lambda t} (\lambda t)^n g(\lambda) d \lambda 157 | } 158 | { 159 | \int_{0}^\infty e^{-\lambda t} (\lambda t)^n g(\lambda) d \lambda} 160 | \] 161 | 162 | 163 | 164 | % ------------ 165 | \section{2. Continusous Time Markov Chains} 166 | \subsection{Overview} 167 | \underline{Note}: For an absorbing state $i$, we may set $\nu_i = 0$. 168 | \\ 169 | Define the \deff{Transition Probabilities} of a CTMC $X(t)$ as 170 | \[ 171 | P_{ij}(t) := P(X(t+s) = j | X(s) = i) 172 | \] 173 | \underline{Note}: [1] $P(t)$ uniquely specifies a CTMC. [2] $P_{ij} \neq P_{ij}(t)$. 174 | 175 | 176 | \subsubsection{Chapman-Kolmogorov Equation} 177 | \[ 178 | P_{ij} (t + s) = \sum_{k \in S} P_{ik} (t) P_{kj} (s) 179 | \] 180 | 181 | % \subsubsection{Poisson Process as CTMC} 182 | % A Poisson Process $\{ X(t), t \ge 0 \}$ with rate $\lambda$ can be modelled as a CTMC with state space $S = \{ 0, 1, 2, ... \}$, rates $\nu_i = \lambda, \forall i \in S$, and jump matrix $P$ where $P_{i,i+1} = 1$ and $P_{ij} = 0, \forall j \neq i + 1$. 183 | % \\ 184 | % \underline{Note}: CTMCs are not necessarily Poisson Processes. 185 | 186 | 187 | \subsubsection{Discretisation of CTMC} 188 | 189 | For a CTMC $\{ X(t), t \ge 0 \}$, $\{Y_1(n) \}_{n \ge 0}$ discretises it at equal intervals if for some constant $l > 0$, 190 | \[ 191 | Y_1(n) = X(nl), n = 0, 1, 2, ... 192 | \] 193 | \underline{Analysis}: $Y_1(n)$ has state space $S$ and transition matrix $P(l)$. 194 | 195 | \smallskip 196 | 197 | For a CTMC $\{ X(t), t \ge 0 \}$, $\{Y_2(n) \}_{n \ge 0}$ is the \deff{Embedded Chain} if it only considers the states visited by $X(t)$. 198 | \\ 199 | \underline{Analysis}: $Y_2(n)$ has state space $S$ and transition matrix $P$. 200 | 201 | 202 | \subsection{Infinitesimal Generator} 203 | Lemma: Transition Rates, for a CTMC, 204 | \begin{itemize} 205 | \item $\lim_{h \to 0} \frac{P_{ii}(h) - P_{ii}(0)}{h} = - \nu_i$ 206 | \item $\lim_{h \to 0} \frac{P_{ij}(h) - P_{ij}(0)}{h} = \nu_i P_{ij}$, for all $i \neq j$ 207 | \end{itemize} 208 | 209 | % For states $i \neq j \in S$, define \deff{Instantaneous Transition Rates} as 210 | % \[ 211 | % q_{ij} := \nu_i P_{ij} 212 | % \] 213 | 214 | The \deff{Infinitesimal Generator} $G$ of a CTMC is defined as 215 | % \[ 216 | % G_{ii} = - \nu_i, \; G_{ij} = q_{ij}, \; i \neq j 217 | % \ \text{ and } 218 | % P'(0) = G 219 | % \] 220 | 221 | 222 | \[ 223 | G = (G_{ij})_{S \times S} = P'(0), 224 | \text{ where } 225 | G_{ij} = \begin{cases} 226 | -\nu_i, &i=j 227 | \\ 228 | \nu_i P_{ij}, &i\neq j 229 | \end{cases} 230 | \] 231 | 232 | 233 | \deff{Kolmogorov's Forward \& Backward Equations} respectively: 234 | \[ 235 | \begin{aligned} 236 | P'(t) = P(t)G 237 | &\iff 238 | P_{ij}'(t) = \sum_{k \neq i} P_{ik}(t)q_{kj} - \nu_j P_{ij}(t) 239 | \\ 240 | P'(t) = GP(t) 241 | &\iff 242 | P_{ij}'(t) = \sum_{k \neq i} q_{ik}P_{kj}(t) - \nu_i P_{ij}(t) 243 | \end{aligned} 244 | \] 245 | 246 | % \subsubsection{Kolmogorov's Backward Equations} 247 | % For all states $i, j$, and times $t \ge 0$, 248 | % \[ 249 | % P'(t) = GP(t) 250 | % \iff 251 | % P_{ij}'(t) = \sum_{k \neq i} q_{ik}P_{kj}(t) - \nu_i P_{ij}(t) 252 | % \] 253 | % \underline{Note}: $G$ uniquely decides $P(t)$. 254 | 255 | 256 | 257 | \subsection{CTMC Long-Term Properties} 258 | \subsubsection{Stationary Distribution} 259 | For a CTMC $\{X(t), t \ge 0\}$, a row vector $\pmb{\pi} = (\pi_i)_{i \in S}$ with $\pi_i \ge 0$ and $\sum_i \pi_i = 1$ is a \deff{Stationary Distribution} if for all $t \ge 0$, 260 | \[ 261 | \pmb{\pi} = \pmb{\pi} P(t), 262 | \] 263 | \deff{Global Balancing Equations} 264 | \[ 265 | \pmb{\pi} G = \pmb{0} 266 | \iff 267 | \equiv \sum_{j\neq i} \pi_i q_{ij} = v_j \pi_j, \ \forall j 268 | \] 269 | 270 | 271 | \subsubsection{Limiting Distribution} 272 | For a CTMC $\{X(t), t \ge 0\}$, its \deff{Limiting Distribution}, $\{ P_j, j \in S \}$, is: 273 | \[ 274 | P_j = \lim_{t \to \infty} P_{ij}(t) 275 | \] 276 | 277 | \underline{Note}: [1] For each $j$, the limit needs to exist and be the same for all $i$. [2] When both $\pmb{\pi}$ and $P$ exists, $\pmb{\pi} = P$. 278 | 279 | \smallskip 280 | 281 | If $X(t)$ satisfies conditions below, it is \deff{Ergodic} (converse not true): [1] All states of $X(t)$ \textbf{communicate}. [2] $X(t)$ is \textbf{positive recurrent}, i.e. for all $i, j \in S$, $\mathbb{E}[\min_{t \ge 0} \{ X(t) = j | X(0) = i \}] < \infty$ 282 | \\ 283 | An ergodic chain has stationary \& limiting distributions \& equal. 284 | 285 | \smallskip 286 | 287 | Suppose embedded chain stationary distribution $\pmb{\psi}$. Then $\forall i, j \in S$, 288 | \[ 289 | \psi_i = \frac{\pi_i \nu_i}{\sum_{j} \pi_j \nu_j} 290 | \iff 291 | \pi_i = \frac{\psi_i / \nu_i}{\sum_{j} \psi_j / \nu_j} 292 | \] 293 | 294 | 295 | \subsubsection{Time Reversibility} 296 | For an \textbf{ergodic} CTMC $\{X(t), t \ge 0\}$ and a sufficiently large $t$, define reversed process $\{Y(t), t \ge 0\}$ 297 | \[ 298 | Y(0) = X(t), \ Y(s) = X(t-s), 0 < s < t 299 | \] 300 | $\{X(t), t \ge 0\}$ is \deff{Time-Reversible} if $X(t)$ and $Y(t)$ has the same probability structure: [1] Same $\pmb{\nu}$, and [2] Same jump matrix. 301 | 302 | \smallskip 303 | 304 | \deff{Local Balanced Equations} 305 | \[ 306 | \pi_j q_{ji} = \pi_i q_{ij}, \ \forall i, j 307 | \] 308 | If it is satisfied, $X(t)$ is time reversible with limiting distribution $\pmb{\pi}$. 309 | 310 | \smallskip 311 | 312 | \textbf{Proposition: Time Reversibility Subset} 313 | \\ 314 | Truncate a time-reversible CTMC $X(t)$ from $S$ to $A \subseteq S$, then it remains time-reversible and has limiting distribution 315 | \[ 316 | \pi_j^A = \frac{\pi_j}{\sum_{i \in A} \pi_i}, \ \forall j \in A 317 | \] 318 | 319 | \textbf{Proposition: Time Reversibility Vectors} 320 | \\ 321 | For CTMCs $\{ X_i(t), t \ge 0 \}, i = 1, 2, ..., n$ time reversible, the vector process $\{(X_1(t), ..., X_n(t)), t \ge 0 \}$ is also time reversible. 322 | 323 | 324 | 325 | 326 | \subsection{CTMC Techniques} 327 | \subsubsection{Uniformization} 328 | For a CTMC $\{X(t), t \ge 0\}$, where $\exists \nu \in \mathbb{R}$ s.t. $\nu_i \le \nu, \forall i \in S$, 329 | \[ 330 | P_{ij}(t) = \sum_{n=0}^\infty (P^*)_{ij} \frac{(\nu t)^n}{n!} e^{-\nu t} 331 | \ 332 | \text{ ,where } 333 | P_{ij}^* = 334 | \begin{cases} 335 | 1 - \nu_i / \nu, & i = j 336 | \\ 337 | (\nu_i / \nu) P_{ij}, & i \neq j 338 | \end{cases} 339 | \] 340 | \underline{Intuition}: $P^*$ is the jump matrix after \deff{Uniformisation}. A CTMC with identical $\nu_i$ is a Poisson Process with rate $\nu_i$. 341 | 342 | 343 | 344 | \subsubsection{CTMC with Absorbing States} 345 | For a CTMC $\{X(t), t \ge 0\}$, if there is a state $i$ s.t. $\forall t > 0, s \ge 0$, 346 | \[ 347 | P(X(t+s) = i | X(s) = i) = 1 348 | \] 349 | , (or $P_{ii}(t) = 1, \forall t>0$,) then we call $i$ an \deff{Absorbing State}. 350 | 351 | \smallskip 352 | 353 | Assume state $0$ is absorbing, \textbf{probability of absorbing} $u_i = \lim_{t \to \infty} P(X(t) = 0 | X(0) = i)$ from state $i$ by CTMC is \textbf{the same as that based on its embedded chain}. 354 | \\ 355 | Define \textbf{expected time of absorption} $w_i$ for starting at state $i$. \underline{Case 1} When $i = 0$, $w_i = 0$. \underline{Case 2a} When $u_i < 1$, $w_i = \infty$. \underline{Case 2b} When $u_i = 1$, $w_i = \mathbb{E}[\text{time till 1st jump}] + \sum_{j \neq i} P_{ij}w_j$. 356 | 357 | 358 | 359 | 360 | 361 | \section{3. Renewal Process} 362 | \subsection{Overview} 363 | A \deff{Renewal Process} is a counting process $\{ N(t), t \ge 0 \}$ for a sequence of non-negative r.v.s $\{X_1, X_2, ... \}$ that are iid. with a distribution $F$. 364 | \begin{itemize} 365 | \item $F(x) = P(X_k \le x), k = 1, 2, ...$, CDF of sojourn time 366 | \item $F_k(x) = P(W_k \le x), k=1, 2, ...$, CDF of waiting time $W_k$. 367 | \item $M(t) = E[N(t)]$, \deff{Renewal Function}, expected \# of renewals 368 | \end{itemize} 369 | 370 | Properties 371 | \begin{itemize} 372 | \item $N(t) \ge k \iff W_k \le t$ 373 | \item $W_{N(t)} \le t < W_{N(t)+1}$ 374 | \item $P(N(t)=k) = F_k(t) - F_{k+1}(t)$ 375 | \item $F_k(t) = \int_{0}^t F_{k-1}(t-y)dF(y)$, one-step analysis 376 | \end{itemize} 377 | 378 | 379 | \deff{The Renewal Equation} - For a renewal process with sojourn times distributed as $F$, let $M(t) = E[X(t)]$, then 380 | \[ 381 | M(t) = F(t) + \int_{0}^t M(t-x)f(x)dx 382 | \] 383 | 384 | \smallskip 385 | 386 | Waiting time in Renewal Process: 387 | \[ 388 | \mathbb{E}[W_{N(t)+1}] = \mathbb{E}[X_1](M(t)+1) 389 | \] 390 | \underline{Note}: Not a Random Sum as $N(t)+1$ not independent with $X_i$. 391 | 392 | 393 | \subsubsection{Special Random Variables} 394 | \begin{itemize} 395 | \item \deff{Excess Time / Residual Time}: $\gamma_t = W_{N(t)+1} - t$ 396 | \item \deff{Current Life / Age}: $\delta_t = t - W_{N(t)} \ge 0$ 397 | \item \deff{Total Life}: $\beta_t = \gamma_t + \delta_t$ 398 | \end{itemize} 399 | 400 | Special Case: Poission Distribution 401 | \begin{itemize} 402 | \item $\gamma_t \sim Exp(\lambda)$ 403 | \item $\delta_t$ follows $Exp(\lambda)$ truncated at $t$. 404 | \item $\mathbb{E}[\beta_t] = 1/\lambda + (1-\exp(-\lambda t)) / \lambda$ 405 | \end{itemize} 406 | 407 | 408 | \subsubsection{Limiting Behaviours} 409 | \deff{Elementary Renewal Theorem} 410 | \[ 411 | \lim_{t \to \infty} \frac{N(t)}{t} = \frac{1}{\mathbb{E}[X_k]} 412 | \] 413 | 414 | \deff{Central Limit Theorem} for Renewal Process 415 | \\ 416 | Let $\mu = E(X_k), \sigma^2 = \text{Var}(X_k)$, then as $t \to \infty, \frac{\text{Var}(N(t))}{t} \to \frac{\sigma^2}{\mu^3}$ and 417 | \[ 418 | N(t) \sim \mathcal{N}(\frac{t}{\mu}, \frac{t \sigma^2}{\mu^3}) \text{ approximately} 419 | \] 420 | 421 | 422 | 423 | 424 | \subsection{Generalisation} 425 | \subsubsection{Renewal Reward Process} 426 | Given a renewal process $N(t)$ with interarrival times $X_n, n \ge 1$ and suppose there is a reward for each renewal $R_n$ that are i.i.d., the \deff{Renewal Reward Process} $\{ R(t), t \ge 0 \}$ is 427 | \[ 428 | R(t) = \sum_{n=1}^{N(t)} R_n 429 | \] 430 | \underline{Note}: [1] $R_n$ can depend on $X_n$, time. [2] Rewards can occur between/along renewals. 431 | 432 | \smallskip 433 | 434 | \textbf{Limiting Theorems} for Renewal Reward Process 435 | \\ 436 | For $E[R_n] < \infty$ and $E[X_n] < \infty$, 437 | \[ 438 | \lim_{t\to \infty} \frac{E[R(t)]}{t} = \frac{E[R_n]}{E[X_n]} 439 | \] 440 | 441 | \underline{Example}: Avg. Current Life $\lim_{t\to\infty} (\int_{0}^t \delta(s)ds) / t = E[X^2] / 2E[X]$. 442 | 443 | 444 | 445 | \subsubsection{Regenerative Process} 446 | A stochastic process $\{X(t), t\ge 0 \}$ is a \deff{Regenerative Process} if there exists time pts when the process probabilistically restarts itself. 447 | \\ 448 | \underline{Note}: [1] "Restart" includes both same transition \& whether current state is same as initial. [2] Neither of MC \& RegenProc $\subseteq$ the other. 449 | 450 | 451 | \subsubsection{Delayed Renewal Process} 452 | A \deff{Delayed Renewal Process} is one when the component in operation at $t = 0$ is not new, but all subsequent ones are. 453 | \\ 454 | \underline{Analysis}: Same set of parameters \& limiting behaviours. 455 | 456 | 457 | 458 | \section{4. Brownian Motion} 459 | \subsection{Multi-Normal Distribution} 460 | A $k$-dim random vector $\mathbf{X}=(X_1, ..., X_k)'$ with mean vector $\mu \in \mathbb{R}_{k\times 1}$ and covariance matrix $\Sigma \in \mathbb{R}_{k \times k}$ is \deff{multivariate normally distributed} $\mathbf{X} \sim \mathcal{N}(\mu, \Sigma)$ if the joint density function is 461 | \[ 462 | f(x_1, ..., x_k) = \frac{1}{\sqrt{2\pi \lVert \Sigma \rVert}} \exp\Big( 463 | -\frac{1}{2} (\mathbf{x}-\mu)' \Sigma^{-1} (\mathbf{x}-\mu) 464 | \Big) 465 | \] 466 | 467 | \smallskip 468 | 469 | For any $\mathbf{a} \in \mathbf{R}_{1\times k}$, $aX \sim \mathcal{N}(a\mu, a\Sigma a')$. 470 | \\ 471 | For any matrix $\mathbf{Q} \in \mathbf{R}_{m \times k}$ with rank $m \le k$, $QX \sim \mathcal{N}(Q\mu, Q\Sigma Q')$. 472 | \\ 473 | For any parition, 474 | \[ 475 | \mathbf{X} = \begin{pmatrix} \mathbf{X}_1 \\ \mathbf{X}_2 \end{pmatrix} \sim \mathcal{N}( 476 | \begin{pmatrix} \mu_1 \\ \mu_2 \end{pmatrix}, 477 | \begin{pmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{pmatrix}) 478 | \] 479 | And the conditional distribution is still normal: 480 | \[ 481 | \mathbf{X}_1 | \mathbf{X}_2 \sim \mathcal{N}( 482 | \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}(\mathbf{X}_2-\mu_2), \Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} 483 | ) 484 | \] 485 | 486 | 487 | 488 | \subsection{Overview} 489 | \subsubsection{(Standard) Brown Motion} 490 | Process $\{X(t), t \ge 0 \}$ is a \deff{Brownian Motion} with parameter $\sigma$ if: [1] $X(0) = 0$, [2] $\{ X(t), t \ge 0 \}$ has stationary \& independent increments, and [3] For every $t > 0$, $X(t) \sim \mathcal{N}(0, \sigma^2 t)$. 491 | \\ 492 | A \deff{Standard Brownian Motion} $\{ B(t), t \ge 0 \}$ has $\sigma = 0$. 493 | 494 | \smallskip 495 | 496 | For time $t_1 \le t_2$, 497 | \[ 498 | \begin{pmatrix} \mathbf{X}_1 \\ \mathbf{X}_2 \end{pmatrix} 499 | \sim \mathcal{N}( \mathbf{0}, 500 | , 501 | \begin{pmatrix} \sigma^2 t_{1} & \sigma^2 t_{1} \\ \sigma^2 t_{1} & \sigma^2 t_{2} \end{pmatrix}) 502 | \] 503 | 504 | For any time $s, t > 0$, 505 | \begin{itemize} 506 | \item If $s \ge t$, $X(s)|X(t) \sim \mathcal{N}(X(t), \sigma^2(s-t))$. 507 | \item If $s 0$, 518 | \[ 519 | X(s) | X(t) \sim \mathcal{N}( 520 | \mu, \frac{\min(s, t)}{t} [X(t) - \mu t], \sigma^2 s - \sigma^2 [\min(s, t)]^2 / t) 521 | \] 522 | 523 | 524 | \subsubsection{Geometric Brownian Motion} 525 | A \deff{Geometric Brownian Motion} $Y(t)$ with parameters $\mu$ and $\sigma$ is: 526 | \[ 527 | Y(t) = e^{\sigma B(t) + \mu t} 528 | \] 529 | \underline{Note}: $Y(0) = 1$ and $Y(t) \ge 0, \forall t$. 530 | 531 | \smallskip 532 | For any time $s < t$, 533 | \[ 534 | \begin{aligned} 535 | \mathbb{E}[Y(t)] &= M_{X(t)}(1) = e^{\mu t + \sigma^2 t/2} 536 | \\ 537 | \text{Var}(Y(t)) &= M_{X(t)}(2) - (M_{X(t)}(1))^2 = e^{2\mu t + \sigma^2 t} (e^{\sigma^2 t} - 1) 538 | \\ 539 | \mathbb{E}[Y(t) | Y(s)] &= Y(s)\exp(\mu (t-s) + \sigma^2 (t-s)/2) 540 | \\ 541 | \text{Cov}(Y(s), Y(t)) &= \exp(\mu (t+s) + \sigma^2(t+s)/2) (\exp(\sigma^2 s)-1) 542 | \end{aligned} 543 | \] 544 | 545 | 546 | 547 | 548 | 549 | \noindent\rule{8cm}{0.4pt} 550 | 551 | \section{Intermediate Results \& Others} 552 | \subsection{From Lecture \& Tutorials} 553 | For \deff{Birh \& Death Process} with +1 $\lambda_i$ \& -1 $\mu_i$, limiting distribution: 554 | \[ 555 | \pi_n = \pi_0 \prod_{i=1}^{n} \frac{\lambda_{i-1}}{\mu_i}, \ \text{ subject to } 556 | \sum_{n=0}^{\infty} \pi_n = 1 557 | \] 558 | 559 | \smallskip 560 | 561 | \textbf{Delay Renewal Example}: Consider $Y_1, Y_2, ...$ iid. A \deff{Pattern} is a $r$-dim vector $(y_1, ..., y_r)$. Every time $(Y_{n-r+1}, ..., Y_n) = (y_1, ..., y_r)$, a renewal occurs at $n$, denoted as $I(n) = 1$. The counting process $N(n)$ is a Delayed Renewal Process. 562 | \\ 563 | Define \deff{Overlapping} $k = \max \{ j< r: (y_{r-j+1}, ..., y_r) = (y_1, ..., y_j) \}$ how much two renewals overlap. Let $p = P(I(n)=1)$. 564 | \\ 565 | When $k = 0$: $E[X_1] = 1/p$, $\text{Var}(X_1) = 1/p^2 - (2r-1)/p$. 566 | \\ 567 | When $k > 0$: $E[X_1] = E[X_{y_1, ..., y_k}] + 1/p$, and $\text{Var}(X_1) = \text{Var}(X_{y_1, ..., y_k}) + p^{-2} - (2r-1)/p + 2p^{-3} \sum_{j=r-k}^{r-1} E[I(r)I(r+j)]$. 568 | 569 | \smallskip 570 | 571 | \citeqn{(Tut5Qn2)} For $X_1 \sim Exp(\lambda_1)$ and $X_2 \sim Exp(\lambda_2)$ independent, we have $\min(X_1, X_2) \sim Exp(\lambda_1 + \lambda_2)$. 572 | 573 | 574 | 575 | \noindent\rule{8cm}{0.4pt} 576 | 577 | \subsection{Appendix: Probability Theory} 578 | % \subsection{Multi-Var Normal Distribution} 579 | 580 | \subsubsection{Gamma Distribution} 581 | $X \sim Gamma(\alpha, \lambda)$ for shape $\alpha$, and rate $\lambda > 0$. ($1/\lambda$ scale param.) 582 | \[ 583 | f(x) = \frac{\lambda^\alpha}{\Gamma(\alpha)}x^{\alpha - 1}e^{-\lambda x}, \quad x \geq 0 584 | \] 585 | Statistics: $E(X) = \frac{\alpha}{\lambda}, \ \text{var}(X) = \frac{\alpha}{\lambda^2}$ \\ 586 | MGF: $M_X(t) = \left(1 - \frac{t}{\lambda}\right)^{-\alpha},\quad t < \beta$ \\ 587 | Special case: $Exp(\lambda) = Gamma(1, \lambda)$, $\chi^2_n = Gamma(\frac{n}{2}, \frac{1}{2})$ \\ 588 | Properties: $Gamma(a, \lambda) + Gamma(b, \lambda) = Gamma(a+b, \lambda)$, and $cX \sim Gamma(\alpha, \frac{\lambda}{c})$ 589 | \\ 590 | \textbf{Gamma function} $\Gamma(\alpha) = \int_0^{\infty} e^{-y}y^{\alpha-1} \, dy$ \\ 591 | $\Gamma(1) = 1, \ \Gamma\left(\frac{1}{2}\right) = \sqrt{\pi}$ and $\Gamma(n) = (n-1)!, \ n \in \mathbb{Z}^{+}$ 592 | 593 | 594 | \subsubsection{Beta Distribution} 595 | $X \sim B(a, b)$ where $a>0, b>0$ has support $[0, 1]$ 596 | \[ 597 | f(X) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} x^{a-1} (1-x)^{b-1}, 0 \le x \le 1 598 | \] 599 | Statistics: $E(X) = \frac{1}{1 + \beta / \alpha}, \ \text{var}(X) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$ 600 | \\ 601 | Special case: $\text{Unif}(0, 1) = B(1, 1)$ 602 | 603 | \textbf{Beta function} $B(a, b) = \int_0^1 t^{a-1} (1-t)^{b-1} \ dt = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$ 604 | 605 | \subsubsection{Common MGFs} 606 | Binomial: $M_X(t) = (1-p+pe^t)^n$ 607 | \\ 608 | Poisson: $M_X(t) = \exp(\lambda (e^t - 1))$ 609 | \\ 610 | Exponential: $M_X(t) = \frac{\lambda}{\lambda - t}$ for $t < \lambda$ 611 | \\ 612 | Normal: $M_X(t) = \exp(\mu t + \sigma^2 t^2 / 2)$ 613 | 614 | \subsubsection{Others} 615 | $E[(X-\mu)^4] = 3\sigma^4$ for $X \sim \mathcal{N}(\mu, \sigma^2)$ 616 | \\ 617 | $E[(\int_{0}^{T} B(s)ds)^2] = \int_{0}^{T} \int_{0}^{T} E[B(s)B(t)] dtds $ 618 | 619 | % TODO: normal (for geom brownian)! 620 | 621 | % \section{Intermediate Results \& Lemmas} 622 | % \subsection{From Tutorials} 623 | 624 | 625 | 626 | 627 | 628 | 629 | 630 | 631 | 632 | 633 | % ----- 634 | % KIV: 635 | % 1. conversion between homogeneous. 636 | 637 | 638 | \end{multicols} 639 | \end{document} 640 | -------------------------------------------------------------------------------- /MA2108/MA2108_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 14 | 15 | \documentclass[10pt,landscape,letterpaper]{article} 16 | \usepackage{amssymb} 17 | \usepackage{amsmath} 18 | \usepackage{amsthm} 19 | \usepackage{physics} % for vectors 20 | \usepackage{bbm} % for mathbb-ed digits 21 | %\usepackage{fonts} 22 | \usepackage{multicol,multirow} 23 | \usepackage{spverbatim} 24 | \usepackage{graphicx} 25 | \usepackage{ifthen} 26 | \usepackage[landscape]{geometry} 27 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 28 | \usepackage{booktabs} 29 | \usepackage{fontspec} 30 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 31 | \setsansfont{Fira Sans} 32 | \setmonofont{Inconsolata} 33 | \usepackage{unicode-math} 34 | \usepackage{listings} 35 | \usepackage{minted} 36 | \setmathfont{TeX Gyre Pagella Math} 37 | \usepackage{microtype} 38 | \usepackage{ulem} % cuz \underline affected by everymath 39 | \usepackage{empheq} 40 | 41 | % new: 42 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 43 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 44 | } 45 | \makeatother 46 | 47 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 48 | { \geometry{margin=0.4in} } 49 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 50 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 51 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 52 | } 53 | \pagestyle{empty} 54 | \makeatletter 55 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 56 | {-1ex plus -.5ex minus -.2ex}% 57 | {0.5ex plus .2ex}%x 58 | {\sffamily\large}} 59 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 60 | {-1explus -.5ex minus -.2ex}% 61 | {0.5ex plus .2ex}% 62 | {\sffamily\normalsize\itshape}} 63 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 64 | {-1ex plus -.5ex minus -.2ex}% 65 | {1ex plus .2ex}% 66 | {\normalfont\small\itshape}} 67 | \makeatother 68 | \setcounter{secnumdepth}{0} 69 | \setlength{\parindent}{0pt} 70 | \setlength{\parskip}{0pt plus 0.5ex} 71 | % ----------------------------------------------------------------------- 72 | 73 | \usepackage{academicons} 74 | 75 | \begin{document} 76 | 77 | \definecolor{mathBlue}{cmyk}{1,.72,0,.38} 78 | \definecolor{defOrange}{cmyk}{0, 0.5, 1, 0.3} 79 | \definecolor{codeInlineRed}{cmyk}{0, 0.9, 0.9, 0.45} 80 | \definecolor{theoremRed}{cmyk}{0, 0.9, 0.9, 0.45} 81 | % Note: code and thm color are pretty similar, but typically they wont appear in the same file. Both are set to be dim so that the file is not overwhelmed by bright colors (since deff organe is quite bright). 82 | 83 | \everymath{\color{mathBlue}} 84 | \everydisplay{\color{mathBlue}} 85 | 86 | % for vector notation in this module 87 | \newcommand{\vect}[1]{\pmb{#1}} 88 | \newcommand{\deff}[1]{\textcolor{defOrange}{\textbf{#1}}} 89 | \newcommand{\codein}[1]{\textcolor{codeInlineRed}{\texttt{#1}}} 90 | \newcommand{\citeqn}[1]{\underline{\textit{#1}}} 91 | \newcommand{\thm}[1]{ 92 | \color{theoremRed}{ 93 | \uline{{\textbf{#1}}} 94 | } 95 | \color{black} 96 | } 97 | 98 | \footnotesize 99 | %\raggedright 100 | 101 | \begin{center} 102 | {\huge\sffamily\bfseries MA2108 Cheatsheet} \huge\bfseries\\ 103 | by Yiyang, AY23/34 104 | \end{center} 105 | \setlength{\premulticols}{0pt} 106 | \setlength{\postmulticols}{0pt} 107 | \setlength{\multicolsep}{1pt} 108 | \setlength{\columnsep}{1.8em} 109 | \begin{multicols}{3} 110 | 111 | 112 | 113 | 114 | % ----------------------------------------------------------------------- 115 | 116 | \section{1. Pre-requisites} 117 | % \subsubsection{} 118 | \thm{Well-Ordering Principle of $\mathbb{N}$} Every non-empty subset $S \in \mathbb{N}$ has a least (smallest) element. 119 | 120 | 121 | 122 | 123 | 124 | \section{2. The Real Numbers} 125 | \subsection{Algebric Properties, \textasciitilde} 126 | Different types of means 127 | \begin{itemize} 128 | \item \deff{Arithmetic Means} $A_n = \frac{1}{n} \sum_{k=1}^n a_k$ 129 | \item \deff{Geometric Means} $G_n = \Big( \prod_{k=1}^n a_k \Big) ^{1/n}$ 130 | \item \deff{Harmonic Means} $H_n = n \Big( \sum_{k=1}^n a_k^{-1} \Big)^{-1}$ 131 | \end{itemize} 132 | , for $n \in \mathbb{N}_{\ge 2}$ and $a_1, a_2, ..., a_n \in \mathbb{R}$ are positive. For the means, we have the \thm{AM-GM-HM Inequality}: 133 | \[ 134 | H_n \le G_n \le A_n 135 | \] 136 | , taking "$=$" iff. $a_1 = ... = a_n$. 137 | 138 | 139 | 140 | \thm{Bernoulli's Inequality} For $x > -1$, we have $(1+x)^n \ge 1 + nx$, for any $n \in \mathbb{N}$. 141 | 142 | 143 | 144 | \thm{Triangle Inequity} $|a+b| \le |a| + |b|$, for all $a, b \in \mathbb{R}$. 145 | \\ 146 | Derived: [1] $\bigl| |a| - |b| \bigr| \le |a-b|$, [2] $|a-b| \le |a|+|b|$. 147 | 148 | 149 | 150 | \subsubsection{Neighbourhood} 151 | For any $a \in \mathbb{R}$ and $\epsilon > 0$, the \deff{$\epsilon$-neighbourhood of $a$} is the set: 152 | \[ 153 | V_\epsilon(a) = \{ 154 | x \in \mathbb{R}: |x - a| < \epsilon 155 | \} 156 | \] 157 | 158 | \thm{Theorem 2.2.8} For $a \in \mathbb{R}$, if $x \in V_\epsilon(a)$ for every $\epsilon > 0$, then $x = a$. 159 | 160 | 161 | 162 | 163 | 164 | \subsection{Completeness Properties, \textasciitilde} 165 | For a non-empty $S \subseteq \mathbb{R}$, it is \deff{Bounded Above} (\deff{Bounded Below}) if $S$ has an upper bound (a lower bound). $S$ is \deff{Bounded} if it is bounded above and below, and is \deff{Unbounded}, otherwise. 166 | 167 | 168 | \smallbreak 169 | 170 | 171 | For a non-empty $S \subseteq \mathbb{R}$, $u$ is the \deff{Supremum} of $S$ if the following conditions are met, and we denote it as $\sup S$: 172 | \begin{enumerate} 173 | \item $u$ is an upper bound of $S$. 174 | \item $\forall v \in \mathbb{R}$, if $v$ is an upper bound of $S$, then $v \ge u$. 175 | \end{enumerate} 176 | For a non-empty $S \subseteq \mathbb{R}$, $w$ is the \deff{Infinum} of $S$ if the following conditions are met, and we denote it as $\inf S$: 177 | \begin{enumerate} 178 | \item $w$ is a lower bound of $S$. 179 | \item $\forall v \in \mathbb{R}$, if $v$ is a lower bound of $S$, then $v \le w$. 180 | \end{enumerate} 181 | \underline{Note}: Sup. and Inf. are \textbf{uniquely determined}, if they exist. 182 | 183 | 184 | Alternative Definition (Similarly for Infinum): 185 | 186 | \thm{Lemma 2.3.4} For $u$ an upper bound of $S \subseteq \mathbb{R}$, $u = \sup S$ iff. 187 | \[ 188 | \forall \epsilon > 0, \exists s_\epsilon \in S, \ u-\epsilon < s_\epsilon 189 | \] 190 | 191 | 192 | 193 | \smallbreak 194 | 195 | 196 | For a non-empty $S \subseteq \mathbb{R}$, $u$ is the \deff{Maximum} (\deff{Minimum}) of $S$, if $u = \sup S$ ($u = \inf S$) and $u \in S$. 197 | \\ 198 | \underline{Note}: Sup. and Inf. are not necessarily elements in $S$ (if they exist), but maximum and minimum are. 199 | 200 | 201 | 202 | \thm{Supremum Property of $\mathbb{R}$} Every non-empty subset of $\mathbb{R}$ that has an upper bound has a supremum. 203 | 204 | 205 | 206 | \thm{The Archimedeam Property} If $x \in \mathbb{R}$, then $\exists n_x \in \mathbb{N}$ s.t. $x < n_x$. 207 | 208 | 209 | 210 | \thm{Corollary 2.4.6} If $x > 0$, then $\exists n \in \mathbb{N}$ such that $n-1 \le x < n$. 211 | 212 | 213 | 214 | \thm{Density Theorems} For $x, y \in \mathbb{R}$ with $x < y$, tehre exists $r \in \mathbb{Q}$ ($z \in \mathbb{R} \backslash \mathbb{Q}$) s.t. $x < r < y$ ($x < z < y$). 215 | 216 | 217 | % Supremum Property 218 | 219 | % Archimedean 220 | 221 | % Density Thm 222 | 223 | 224 | \subsection{Intervals} 225 | A sequence of intervals $I_n, n \in \mathbb{N}$ is \deff{Nested} if 226 | \[ 227 | I_1 \supseteq I_2 \supseteq ... \supseteq I_n \supseteq I_{n+1} \supseteq ... 228 | \] 229 | \underline{Properties}: [1] If $I_n = [a_n, b_n], n \in \mathbb{N}$ is a nested seq. of closed bounded intervals, then $\exists \xi \in \mathbb{R}$ s.t. $\xi \in I_n, \forall n \in \mathbb{N}$. [2] If $I_n = [a_n, b_n], n \in \mathbb{N}$ satisfying $\inf \{ b_n - a_n: n \in \mathbb{N} \} = 0$, then $\xi$ contained in all $I_n$ is unique. 230 | 231 | 232 | 233 | 234 | 235 | \section{3. Sequences \& Series} 236 | \subsection{Sequence \& Convergence} 237 | % Seq def 238 | \deff{Sequence} in $\mathbb{R}$: a real-valued function $X: \mathbb{R} \to \mathbb{R}$. We write $x_n = X(n)$ for the $n$-th term of the sequence, and denote the sequence as $(x_n,: n \in \mathbb{N})$. 239 | 240 | 241 | \smallbreak 242 | 243 | 244 | A sequence $X = (x_n)$ in $\mathbb{R}$ is \deff{Convergent} to $x \in \mathbb{R}$ iff. for every $\epsilon > 0$, there exists $K = K(\epsilon) \in \mathbb{N}$ s.t. 245 | \[ 246 | n \ge K(\epsilon) \implies |x_n - x| < \epsilon 247 | \] 248 | , and we call $x$ the \deff{Limit} of $(x_n)$, denoted as $\lim_{n \to \infty} x_n = x$. A sequence is \deff{Divergent} if it is not convergent. 249 | 250 | 251 | Technique for proving convergence: 252 | \begin{enumerate} 253 | \item Express $|x_n - x|$ in terms of $n$ and find a simpler upper bound $L = L(n)$, i.e. $|x_n - x| < L$. 254 | \item Let $\epsilon > 0$ be arbitrary, find $K \in \mathbb{N}$ s.t. for all $n \ge K$, $L = L(n) < \epsilon$, then 255 | \[ 256 | n \ge K \implies |x_n - x| < L < \epsilon 257 | \] 258 | \end{enumerate} 259 | 260 | 261 | \smallbreak 262 | 263 | 264 | \thm{Squeeze Theorem} If $x_n \le y_n \le z_n$, for all $n \in \mathbb{N}$ and $\lim_{n \to \infty} x_n = \lim_{n \to \infty} z_n = a$, then 265 | \[ 266 | \lim_{n \to \infty} y_n = a 267 | \] 268 | 269 | 270 | 271 | A sequence $X = (x_n)$ is \deff{Bounded} if there exists $M > 0$ such that $|x_n| \le M$ for all $n \in \mathbb{N}$. 272 | 273 | % Thm 3.2.2 ~ 3.2.5 See fit 274 | 275 | 276 | \smallbreak 277 | 278 | 279 | \thm{Monotone Convergence Theorem} Let $(x_n)$ be a monotone sequence of real numbers, then $(x_n)$ is convergent iff. $(x_n)$ is bounded. 280 | \\ 281 | If it is bounded and increasing, then $\lim_{n \to \infty} x_n = \sup \{ x_n: n \in \mathbb{N} \}$. (Similarly for decreasing.) 282 | 283 | 284 | \smallbreak 285 | 286 | 287 | For a sequence $(x_n)$, it \deff{tends to $+ \infty$}, i.e. $\lim_{n \to \infty} x_n = + \infty$ if for all $\alpha \in \mathbb{R}$, there exists $K = K(\alpha) \in \mathbb{N}$ such that if $n \ge K(\alpha)$, then $x_n > \alpha$. (Similarly for $\lim_{n \to \infty} x_n = - \infty$.) 288 | \\ 289 | A sequence $(x_n)$ is \deff{Properly Divergent} if $\lim_{n \to \infty} x_n = \pm \infty$. 290 | 291 | 292 | 293 | 294 | \subsection{Subsequences} 295 | % Def 296 | A \deff{Subsequence} of $X = (x_n)$ is $X' = (x_{n_k})$: 297 | \[ 298 | X' = (x_{n_1}, x_{n_2}, ..., x_{n_3}) 299 | \] 300 | , where $n_1 < n_2 < ... < n_k < ... $ is a strictly increasing sequence in $\mathbb{N}$. \underline{Note}: $n_k \ge n, \forall k$. 301 | 302 | 303 | \thm{Theorem 3.4.2} If $(x_n)$ converges to $x$, then any subsequence $(x_{n_k})$ also converges to $x$, 304 | \[ 305 | \lim_{n_k \to \infty} x_{n_k} = \lim_{k \to \infty} x_{n_k} = x 306 | \] 307 | 308 | 309 | \thm{Theorem 3.4.5} If $(x_n)$ has either of the following properties, it is divergent: [1] $(x_n)$ has two convergent subsequences with different limits. [2] $(x_n)$ is unbounded. 310 | 311 | 312 | \thm{Theorem 3.4.7} Every sequence has a monotone subsequence. 313 | 314 | 315 | \thm{Bolzano-Weierstrass Theorem} Every bounded sequence has a convergent subsequence. 316 | 317 | 318 | 319 | 320 | \subsection{Cauchy Sequences} 321 | A \deff{Cauchy Sequence} $(x_n)$ is a sequence where for all $\epsilon > 0$, there exists $H = H(\epsilon) \in \mathbb{N}$ such that 322 | \[ 323 | \forall n, m \in \mathbb{N}, n, m \ge H \implies |x_n - x_m| < \epsilon 324 | \] 325 | 326 | \thm{Cauchy Criterion} A sequence is convergent iff. it is Cauchy. 327 | 328 | 329 | \smallbreak 330 | 331 | 332 | A \deff{Contractive Sequence} $(x_n)$ is a sequence where there exists $C \in (0, 1)$ s.t. 333 | \[ 334 | |x_{n+2} - x_{n+1}| \le C |x_{n+1} - x_n|, \ \forall n \in \mathbb{N} 335 | \] 336 | 337 | 338 | \thm{Theorem 3.5.8} Every contractive sequence is Cauchy. 339 | 340 | 341 | 342 | 343 | \subsection{Infinite Series} 344 | For $(x_n)$, its \deff{(Infinite) Series} is sequence $(s_n)$, where $s_n = \sum{k=1}^n x_k$ is called a \deff{Partial Sum} of the series, and $x_k$ is a \deff{Term}. 345 | 346 | Tests for infinite series' convergence: 347 | \begin{itemize} 348 | \item \deff{$n$-th Term Test} - If $\sum x_n$ converges, then $\lim_{n\to\infty} x_n = 0$. 349 | 350 | \item Cauchy Criterion Test 351 | 352 | \item \deff{Partial Sum Bounded Test}, for series w. non-negative terms - Suppose $x_n \ge 0, \forall n \in \mathbb{N}$, then $\sum_{x_n}$ converges iff. $(s_n)$ is bounded. 353 | 354 | \item \deff{Comparison Test} - For $(x_n), (y_n)$ with some $K \in \mathbb{N}$, s.t. $n \ge K \implies 0 \le x_n \le y_n$. Then [1] $\sum y_n$ converges $\implies$ $\sum x_n$ converges, and [2] $\sum x_n$ diverges $\implies$ $\sum y_n$ diverges. 355 | 356 | \item \deff{Limit Comparison Test} - For \textbf{strictly positive} $(x_n), (y_n)$ with limit $r = \lim_{n \to \infty} (\frac{x_n}{y_n})$. Then [1] if $r = 0$, $\sum y_n$ converges $\implies$ $\sum x_n$ converges. [2] if $r > 0$, $\sum y_n$ converges iff $\sum x_n$ converges. 357 | \end{itemize} 358 | 359 | 360 | 361 | 362 | 363 | \subsection{Absolute Convergence} 364 | % Def 365 | Series $\sum x_n$ is \deff{Absolutely Convergent} if series $\sum |x_n|$ is convergent. A series is \deff{Conditionally Convergent} if it is convergent but not absolutely convergent. 366 | 367 | Tests for absolutely convergence: 368 | \begin{itemize} 369 | \item Limit Comparison Test - Consider convergence of positive sequences $|x_n|$ and $|y_n|$ if $(x_n), (y_n)$ non-negative. 370 | \item \deff{Root Test} - For $(x_n)$, [1] if $\exists r \in \mathbb{R}, 0 < r < 1$ and $K \in \mathbb{N}$ s.t. $|x_n|^{1/n} \le r, \ \forall n \ge K$, then $\sum x_n$ is abs. convergent. [2] If $\exists r \in \mathbb{R}, r > 1$ and $K \in \mathbb{N}$ s.t. $|x_n|^{1/n} \ge r > 1, \ \forall n \ge K$, then $\sum x_n$ is \textbf{divergent}. 371 | \item \deff{Ratio Test} - For $(x_n)$ nonzero, [1] if $\exists r \in \mathbb{R}, 0 < r < 1$ and $K \in \mathbb{N}$ s.t. $|\frac{x_{n+1}}{x_n}| \le r, \ \forall n \ge K$, then $\sum x_n$ is abs. convergent. [2] If $\exists K \in \mathbb{N}$ s.t. $|\frac{x_{n+1}}{x_n}| \ge 1, \ \forall n \ge K$, then $\sum x_n$ is \textbf{divergent}. 372 | \end{itemize} 373 | 374 | % Abs convg tests 375 | % TODO: check if finishes 376 | 377 | 378 | 379 | 380 | \section{4. Limits} 381 | For $A \subseteq \mathbb{R}$, $c$ is the \deff{Cluster Point} of $A$ iff. $\forall \delta > 0$, there exists $x \in A$ s.t. $0< |x-c| < \delta$. 382 | 383 | \thm{Theorem 4.1.2} (Sequential Criterion) $c \in \mathbb{R}$ is a cluster point of $A$ iff. there exists a sequence $(a_n)$ in $A$ s.t. $\lim a_n = c$ and $a_n \neq c, \forall n \in \mathbb{N}$. 384 | 385 | \deff{Limit} of a function $f: A \to \mathbb{R}$ at $c \in A$, $L = \lim_{x \to c} f(x)$ iff. $\forall \epsilon > 0$, there exists $\delta = \delta(\epsilon) > 0$ s.t. $0 < |x - c| < \delta \implies |f(x) - L| < \epsilon$. 386 | 387 | \thm{Theorem 4.1.8} (Sequential Criterion) $\lim_{x \to c} f(x) = L$ iff. for every seq. $(x_n)$ in $A$ w. $lim_{n \to \infty} x_n = c$ and $x_n \neq c, \forall n \in \mathbb{N}$, $lim_{n \to \infty} f(x_n) = L$. 388 | 389 | 390 | \smallbreak 391 | 392 | 393 | For $f: A \to \mathbb{R}$ and $c$ a cluster point of $A$, $f$ is \deff{Bounded} on a neighbourhood of $c$ if $\exists V_\delta(c)$ and constant $M > 0$ s.t. $|f(x)| < M, \forall x \in A \cap V_\delta(c)$. 394 | 395 | 396 | \thm{Theorem 4.2.2} If $f: A \to \mathbb{R}$has a limit at cluster point $c$, then $f$ is bounded on some neighbourhood of $c$. 397 | 398 | 399 | \smallbreak 400 | 401 | 402 | \thm{Theorem 4.2.9} If $\lim_{x\to c} f(x) > 0$, then $\exists V_\delta(c)$ s.t. $f(x) > 0, \forall x \in A \cap V_\delta(c), x \neq c$. 403 | \\ 404 | Similar statements for $f(x) < 0$. 405 | 406 | 407 | \smallbreak 408 | 409 | 410 | 411 | For $A\subseteq \mathbb{R}$, function $f: A \to \mathbb{R}$ and a cluster point $c$ of $A$, \deff{Right Hand Limit} $L_{+} = \lim_{x \to c^+} f(x)$ iff. $\forall \epsilon > 0, \exists \delta > 0, x \in V_{\delta}(c) \ \{ c \} \implies f(x) \in V_{\epsilon}(L_{+})$. 412 | \\ 413 | Similar definition for \deff{Left-Hand Limit} $L_{-} = \lim_{x \to c^-} f(x)$. 414 | \\ 415 | Sequential Criteria for One-sided Limits exist. 416 | 417 | 418 | \thm{Theorem 4.3.3} $\lim_{x \to c} f(x) = L$ iff. both $\lim_{x \to c^+} f(x)$ and $\lim_{x \to c^-} f(x)$ exist and 419 | \[ 420 | \lim_{x \to c^+} f(x) = \lim_{x \to c^-} f(x) = L 421 | \] 422 | 423 | 424 | 425 | 426 | \section{5. Continuous Functions} 427 | \subsection{Continuity} 428 | For $A \subseteq \mathbb{R}$ and $f: A \to \mathbb{R}$, $f$ is \deff{Continuous} at $c \in A$ iff $\forall \epsilon > 0, \exists \delta > 0$ s.t. $x \in V_\delta(c) \implies f(x) \in V_\epsilon(f(c))$. 429 | \\ 430 | $f$ is continuous at $c \in A$ iff. $\lim_{x \to c} f(x) = f(c)$. 431 | 432 | (Sequential Criterion) $f: A \to \mathbb{R}$ is continuous at $x = c$ iff. for every sequence $(x_n)$ in $A$ s.t. $x_n \to c$, we have $f(x_n) \to f(c)$. 433 | 434 | 435 | 436 | \subsection{Continuous Function on Intervals} 437 | \thm{Boundedness Theorem} If $f$ is continuous on $[a, b]$, then $f$ is bounded on $[a, b]$. 438 | \\ 439 | \underline{Note}: It only applies to \textbf{closed bounded} intervals. 440 | 441 | 442 | 443 | 444 | \thm{Max-Min Theorem} If $f$ is continuous on $[a, b]$, then $f$ has an absolute maximum and an absolute minimum on $[a, b]$. 445 | 446 | 447 | 448 | 449 | \thm{Location of Roots Theorem} If $f$ is continuous on $[a, b]$ and $f(a)f(b) < 0$, then there exists a point $c$ in $(a, b)$ s.t. $f(c) = 0$. 450 | 451 | 452 | 453 | 454 | \thm{Bolzano's Intermediate Value Theorem} For interval $I$ and function $f$ continuous on $I$, and $a, b \in I$ with $f(a) \le f(b)$, then for any $k \in [f(a), f(b)]$, $\exists c \in I$ s.t. $f(c) = k$. 455 | 456 | 457 | 458 | 459 | \thm{Preservation of Closed Intervals Theorem} For $f$ continuous on $[a, b]$, 460 | \[ 461 | f([a, b]) := \{ f(x): x \in [a, b] \} = [m, M] 462 | \] 463 | , with $m = \inf f([a, b])$ and $M = \sup f([a, b])$. 464 | 465 | % TODO: maybe more 466 | 467 | 468 | 469 | 470 | \subsection{Monotonicity \& Bijectivity} 471 | A function $f: A \to \mathbb{R}$ is \deff{Increasing} (\deff{Decreasing}) on $A$ if $\forall x_1, x_2 \in A$, $x_1 \le x_2 \implies f(x_1) \le f(x_2)$ ($x_1 \le x_2 \implies f(x_1) \ge f(x_2)$). 472 | \\ 473 | $f$ is \deff{Monotone} if it is increasing or decreasing. 474 | \\ 475 | \deff{Strictly \textasciitilde}: $x_1 < x_2 \implies f(x_1) < f(x_2)$ and so on. 476 | 477 | 478 | \smallbreak 479 | 480 | 481 | For a function $f: A \to B$, it is 482 | \begin{itemize} 483 | \item \deff{Injective} (\deff{One-One}), iff $\forall x_1 \neq x_2 \in A$, $f(x_1) \neq f(x_2)$. 484 | \item \deff{Surjective}, iff $f(A) = B$. 485 | \item \deff{Bijective}, iff it is injective and surjective. 486 | \end{itemize} 487 | 488 | 489 | 490 | 491 | 492 | \subsection{Uniform Continuity} 493 | For $A \subseteq \mathbb{R}$ and $f: A \to \mathbb{R}$, $f$ is \deff{Uniformly Continuous} on $A$ if for all $\epsilon > 0$, there exists $\delta = \delta(\epsilon) > 0$ s.t. 494 | \[ 495 | \forall x, y \in A, |x-y| < \delta \implies |f(x) - f(y)| < \epsilon 496 | \] 497 | i.e. $\delta = \delta(\epsilon)$ is independent of $x, y \in A$. 498 | \\ 499 | \underline{Note}: $f$ is \textbf{not uniformly continuous} on $A$ iff. $\exists \epsilon_0 > 0$ s.t. $\forall \delta > 0, \exists x_\delta, y_\delta \in A$ with $|x_\delta - y_\delta| < \delta$ and $|f(x_\delta) - f(y_\delta)| \ge \epsilon$. 500 | 501 | Sequential Criterion 502 | \begin{itemize} 503 | \item Uniformly continuous - For any $(x_n), (y_n)$ in $A$ with $\lim_{n\to \infty} x_n - y_n = 0$, we have $\lim_{n \to \infty} f(x_n) - f(y_n) = 0$. 504 | \item Not uniformly continuous - There exists $\epsilon_0 > 0$ and $(x_n), (y_n)$ in $A$, $\lim_{n\to \infty} x_n - y_n = 0$ and $\lim_{n \to \infty} f(x_n) - f(y_n) \ge \epsilon_0$. 505 | \end{itemize} 506 | 507 | 508 | \thm{Uniform Continuity Theorem} If $f$ is continuous on a \textbf{closed bounded} interval $[a, b]$, then it is uniformly continuous on $[a, b]$. 509 | 510 | 511 | \smallbreak 512 | 513 | 514 | A function $f: A \to \mathbb{R}$ is a \deff{Lipschitz Function} on $A$ iff. there exists $K > 0$ s.t. 515 | \[ 516 | |f(x) - f(y)| \le K |x-y|, \forall x, y \in A 517 | \] 518 | 519 | \thm{Theorem 5.4.5} If $f: A \to \mathbb{R}$ is a Lipschitz function, then $f$ is uniformly continuous on $A$. 520 | 521 | 522 | \thm{Theorem 5.4.7} If $f: A \to \mathbb{R}$ is uniformly continuous on $A$ and $(x_n)$ a Cauchy sequence in $A$, then $(f(x_n))$ is a Cauchy sequence in $\mathbb{R}$. 523 | \\ 524 | i.e. Uniformly continuous functions preserve Cauchy sequences. 525 | 526 | 527 | 528 | \thm{Continuous Extension Theorem} $f$ is uniformly continuous on interval $(a, b)$ iff. it can be defined at the endpoints $a$ and $b$ s.t. the extended function is continuous on $[a, b]$. 529 | \\ 530 | \underline{Note}: Define $f(a) = \lim_{x \to a^+} f(x)$ and $f(b) = \lim_{x \to b^-} f(x)$ provided both limits exist. 531 | 532 | 533 | 534 | 535 | 536 | 537 | \subsection{Jumps} 538 | \thm{Theorem 5.6.1} For interval $I \subseteq \mathbb{R}$ and increasing function $f: I \to \mathbb{R}$, $c \in I$ not an endpoint, then 539 | \begin{itemize} 540 | \item $\lim_{x \to c^-} f(x) = \sup\{ f(x): x \in I, x < c \}$ 541 | \item $\lim_{x \to c^+} f(x) = \inf\{ f(x): x \in I, x > c \}$ 542 | \end{itemize} 543 | 544 | \deff{Jump} of $f$ at $c$ is defined as $$j_f (c) = \lim_{x \to c^+} f(x) - \lim_{x \to c^-} f(x)$$, and at endpoints, $j_f(a) = \lim_{x \to a^+} f(x) - f(a)$ and so on for $b$. 545 | 546 | 547 | 548 | \thm{Theorem 5.6.4} For interval $I \subseteq \mathbb{R}$ and $f: I \to \mathbb{R}$ monotone on $I$, the set of points $D \subseteq I$ at which $f$ is discontinuous is a countable set. 549 | 550 | 551 | \thm{Continuous Inverse Theorem} For interval $I \subseteq \mathbb{R}$ and $f: I \to \mathbb{R}$ strictly monotone and continuous, the inverse $f^{-1}$ exists and is also strictly monotone and continuous on $J = f(I)$. 552 | 553 | 554 | 555 | 556 | \section{11. Topology Introduction} 557 | % TODO: whether we can use generalised metric definition later. 558 | \subsection{Open \& Closed Sets} 559 | A set $V$ is the \deff{Neighbourhood} of a point $x \in \mathbb{R}$ iff there exists $\epsilon > 0$ s.t. $V_\epsilon(x) \subseteq V$. 560 | \\ 561 | A subset $G \subseteq \mathbb{R}$ is \deff{Open} in $\mathbb{R}$ iff. for each $x \in G$, there exists $\epsilon_x > 0$ s.t. $V_{\epsilon_x}(x) \subseteq G$. 562 | \\ 563 | A subset $F \subseteq \mathbb{R}$ is \deff{Closed} in $\mathbb{R}$ if the complement $C(F) = \mathbb{R} \setminus F$ is open in $\mathbb{R}$. 564 | \\ 565 | \underline{Note}: [1] $\mathbb{R}$ and $\emptyset$ are both open and closed. [2] $\mathbb{Z}$ is closed but not open. [3] $\mathbb{Q}$ is neither open nor closed. 566 | 567 | 568 | Open \& Closed Set Properties 569 | \begin{itemize} 570 | \item Open: [1] Union of any collection of open subsets is open. [2] Intersection of finitely many open subsets is open. 571 | \item Closed: [1] Intersection of any collection of closed subsets is closed. [2] Union of finitely many closed subsets is closed. 572 | \end{itemize} 573 | 574 | 575 | \thm{Characterisation of Closed Sets Theorem} A subset $F \subseteq \mathbb{R}$ is closed iff. any convergence sequence $(x_n)$ in F has $\lim_{n \to \infty} x_n \in F$. 576 | 577 | 578 | \thm{Theorem 11.1.8} A subset $F \subseteq \mathbb{R}$ is closed iff. it contains all its cluster points. 579 | 580 | \thm{Theorem 11.1.9} A subset $G \subseteq \mathbb{R}$ is open iff it is the union of countably many disjoint open intervals in $\mathbb{R}$. 581 | 582 | 583 | \smallbreak 584 | 585 | 586 | \thm{Global Continuity Theorem} A function $f: A \to \mathbb{R}$ is continuous on $A$ iff. for every open set $G \subseteq \mathbb{R}$, there exists open set $H \subseteq \mathbb{R}$ such that $H \cap A = f^{-1}(G)$ where $f^{-1}(G) = \{ x \in A: f(x) \in G \}$. 587 | 588 | \thm{Corollary 11.3.3} Function $f: \mathbb{R} \to \mathbb{R}$ is continuous iff. $f^{-1}(G)$ is open in $\mathbb{R}$ for every open $G$. 589 | 590 | 591 | 592 | \subsection{Metric Space} 593 | A \deff{Metric} on a set $S$ is a function $d: S \times S \to \mathbb{R}$ that satisfies 594 | \begin{itemize} 595 | \item \deff{Positivity} $d(x, y) \ge 0, \forall x, y \in S$ 596 | \item \deff{Definiteness} $d(x, y) = 0 \iff x = y$ 597 | \item \deff{Symmetry} $d(x, y) = d(y, x), \forall x, y \in S$ 598 | \item \deff{Triangle Inequality} $d(x,y) \le d(x, z) + d(z, y), \forall x, y, z \in S$ 599 | \end{itemize} 600 | A \deff{Metric Space} $(S, d)$ is a set $S$ with a metric $d$ on $S$. 601 | 602 | 603 | Generalised definition for a metric space $(S, d)$ 604 | \begin{itemize} 605 | \item \textbf{Neighbourhood}: $V_\epsilon(x_0) = \{ x \in S: d(x, x_0) < \epsilon \}$ for $\epsilon > 0$ and $x_0 \in S$ 606 | 607 | \item \textbf{Boundedness} of $K \subseteq S$: $\exists M > 0, x_0 \in S, d(x, x_0) \le M, \forall x \in K$. 608 | 609 | \item \textbf{Convergence} to $x \in S$ of sequence $(x_n)$: $\forall \epsilon > 0, \exists K = K(\epsilon) \in \mathbb{N}, n \ge K \implies x_n \in V_{\epsilon}(x)$ 610 | 611 | \item \textbf{Continuity} of $f: S_1 \to S_2$ at $c \in S_1$: $\forall \epsilon > 0, \exists \delta > 0, d_1(x, c) < \delta \implies d_2(f(x), f(c)) < \epsilon$. 612 | 613 | \item \textbf{Open \& Closed Set} 614 | \end{itemize} 615 | 616 | 617 | 618 | 619 | 620 | \subsection{Compact Set} 621 | For a metric space $S$, an \deff{Open Cover} of a subset $A \subseteq S$ is a collection $\mathcal{G} = \{ G_\lambda: \lambda \in \Lambda \}$ of open subsets of $S$ satisfying 622 | \[ 623 | A \subseteq \cup_{\lambda \in \Lambda} G_\lambda 624 | \] 625 | If $\mathcal{G}' \subseteq \mathcal{G}$ whose union also contains $A$, then $\mathcal{G}'$ is a \deff{Subcover} of $\mathcal{G}$. 626 | \\ 627 | If $\mathcal{G}'$ is finite, it is a \deff{Finite Subcover} of $\mathcal{G}$. 628 | 629 | 630 | For a metric space $S$, a subset $K \subseteq S$ is \deff{Compact} iff. for every open cover of $K$ there is a finite subcover. 631 | 632 | 633 | \thm{Heine-Borel Theorem} For a metric space $(S, d)$, a subset $K \subseteq S$ is compact iff. it is closed and bounded. 634 | 635 | \thm{Bolzano-Weierstrass Theorem} A bounded sequence in $(S, d)$ has a convergent subsequence. 636 | 637 | \thm{Theorem 11.2.6} $K \subseteq S$ is compact iff. every sequence in $K$ has a subsequence that converges to a point in $K$. 638 | 639 | 640 | \thm{Preservation of Compactness Theorem} If $(S, d)$ is compact and $f: S \to \mathbb{R}$ is continuous, then $f(S)$ is compact in $\mathbb{R}$. 641 | 642 | 643 | \smallbreak 644 | 645 | 646 | A subset $U \subseteq S$ is \deff{Disconnected} iff. $U$ has an open cover $\{ A, B\}$ s.t. $A \cap B \cap U = \emptyset$ and $A \cap U = \emptyset, B \cap U = \emptyset$. Otherwise it is \deff{Connected}. 647 | \\ 648 | \underline{Note}: $E \subseteq \mathbb{R}$ is connected iff $E$ is an interval, i.e. $x, y \in E, x < y \implies [x, y] \in E$. 649 | 650 | 651 | \thm{Intermediate Value Theorem} For $f: S \to \mathbb{R}$ continuous, if $E$ is connected then $f(E)$ is connected. 652 | 653 | 654 | 655 | \noindent\rule{8cm}{0.4pt} 656 | 657 | 658 | 659 | 660 | \section{Intermediate Results \& Lemmas} 661 | \citeqn{(Tut10Qn4)} For any two functions $f, g: \mathbb{R} \to \mathbb{R}$ continuous on $\mathbb{R}$, if $f(x) = g(x), \forall x \in \mathbb{Q}$, then $f(x) = g(x), \forall x \in \mathbb{R}$. 662 | 663 | 664 | Useful statements 665 | \begin{itemize} 666 | \item $\sum_{i=1}^\infty \frac{1}{n^2} = \frac{\pi^2}{6}$. 667 | \end{itemize} 668 | 669 | 670 | \end{multicols} 671 | \end{document} 672 | -------------------------------------------------------------------------------- /ST2132/ST2132_Cheatsheet.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Original Source: Dave Richeson (divisbyzero.com), Dickinson College 3 | % Modified By: Chen Yiyang 4 | % 5 | % A one-size-fits-all LaTeX cheat sheet. Kept to two pages, so it 6 | % can be printed (double-sided) on one piece of paper 7 | % 8 | % Feel free to distribute this example, but please keep the referral 9 | % to divisbyzero.com 10 | % 11 | % Guidance on the use of the Overleaf logos can be found here: 12 | % https://www.overleaf.com/for/partners/logos 13 | % 14 | % Credits: 15 | % Jovyn Tan, https://github.com/jovyntls 16 | % for the nice navy blue colour for in-line and display Math. 17 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 18 | 19 | \documentclass[10pt,landscape,letterpaper]{article} 20 | \usepackage{amssymb} 21 | \usepackage{amsmath} 22 | \usepackage{amsthm} 23 | %\usepackage{fonts} 24 | \usepackage{multicol,multirow} 25 | \usepackage{spverbatim} 26 | \usepackage{graphicx} 27 | \usepackage{ifthen} 28 | \usepackage[landscape]{geometry} 29 | \usepackage[colorlinks=true,urlcolor=olgreen]{hyperref} 30 | \usepackage{booktabs} 31 | \usepackage{fontspec} 32 | \setmainfont[Ligatures=TeX]{TeX Gyre Pagella} 33 | \setsansfont{Fira Sans} 34 | \setmonofont{Inconsolata} 35 | \usepackage{unicode-math} 36 | \setmathfont{TeX Gyre Pagella Math} 37 | \usepackage{microtype} 38 | 39 | \usepackage{empheq} 40 | 41 | % new: 42 | \def\MT@is@uni@comp#1\iffontchar#2\else#3\fi\relax{% 43 | \ifx\\#2\\\else\edef\MT@char{\iffontchar#2\fi}\fi 44 | } 45 | \makeatother 46 | 47 | \ifthenelse{\lengthtest { \paperwidth = 11in}} 48 | { \geometry{margin=0.4in} } 49 | {\ifthenelse{ \lengthtest{ \paperwidth = 297mm}} 50 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 51 | {\geometry{top=1cm,left=1cm,right=1cm,bottom=1cm} } 52 | } 53 | \pagestyle{empty} 54 | \makeatletter 55 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 56 | {-1ex plus -.5ex minus -.2ex}% 57 | {0.5ex plus .2ex}%x 58 | {\sffamily\large}} 59 | \renewcommand{\subsection}{\@startsection{subsection}{2}{0mm}% 60 | {-1explus -.5ex minus -.2ex}% 61 | {0.5ex plus .2ex}% 62 | {\sffamily\normalsize\itshape}} 63 | \renewcommand{\subsubsection}{\@startsection{subsubsection}{3}{0mm}% 64 | {-1ex plus -.5ex minus -.2ex}% 65 | {1ex plus .2ex}% 66 | {\normalfont\small\itshape}} 67 | \makeatother 68 | \setcounter{secnumdepth}{0} 69 | \setlength{\parindent}{0pt} 70 | \setlength{\parskip}{0pt plus 0.5ex} 71 | % ----------------------------------------------------------------------- 72 | 73 | \usepackage{academicons} 74 | 75 | \begin{document} 76 | 77 | \definecolor{myblue}{cmyk}{1,.72,0,.38} 78 | \everymath{\color{myblue}} 79 | \everydisplay{\color{myblue}} 80 | 81 | \footnotesize 82 | %\raggedright 83 | 84 | \begin{center} 85 | {\huge\sffamily\bfseries ST2132 Cheatsheet} \huge\bfseries\\ 86 | by Wei En \& Yiyang, AY21/22 87 | \end{center} 88 | \setlength{\premulticols}{0pt} 89 | \setlength{\postmulticols}{0pt} 90 | \setlength{\multicolsep}{1pt} 91 | \setlength{\columnsep}{1.8em} 92 | \begin{multicols}{3} 93 | 94 | 95 | % Things to include 96 | % func of Joint distribution Formula 97 | % More abt MGF 98 | % results / lemmas from tut. 99 | 100 | \section{ST2131 Topics} 101 | % Things to include 102 | % Functions of (joint) RVs (if later parts not covered) 103 | % 104 | \subsection{Theorems \& Identities} 105 | \subsubsection{Tail Sum Formula} 106 | For DRV. $X$ with \emph{non-negative integer-valued} support, $E(X) = \sum_{k=1}^{\infty} P(X \geq k) = \sum_{k=0}^{\infty} P(X > k)$. \\ 107 | For CRV. $X$ with positive support, $E(X)= \int_{0}^{\infty} P(X>x) \,dx = \int_{0}^{\infty} P(X \geq x) \,dx$ 108 | 109 | 110 | \subsubsection{Markov's Inequality} 111 | For \textbf{non-negative} r.v. $X$, $P(X \geq a) \leq \frac{E(X)}{a}$ for any $a > 0$. 112 | 113 | \subsubsection{Chebyshev's Inequality} 114 | Let $X$ be a r.v. with mean $\mu$, $P(|X-\mu| \geq a) \leq \frac{\text{var}(X)}{a^2}$ for any $a > 0$. 115 | 116 | \subsubsection{One-sided Chebyshev's Inequality} 117 | Let $X$ be a r.v. with \textbf{zero mean} and variance $\sigma^2$, $P(X \geq a) \leq \frac{\sigma^2}{\sigma^2 + a^2}$ for any $a > 0$. 118 | 119 | 120 | \subsubsection{Jensen's Inequality} 121 | For r.v. $X$ and convex function $g(X)$, $E[g(X)] \geq g(E[X])$, provided the expectations exist and are finite. 122 | 123 | 124 | \subsection{Definitions} 125 | Covariance, $Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)] = E(XY) - E(X)E(Y)$ \\ 126 | Coefficient of Correlation, $\rho(X, Y) = \frac{\text{Cov}(X, Y)}{\sqrt{\text{var}(X)\text{var}(Y)}}$\\ 127 | Moment Generating Function, $M_X(t) = E[e^{tX}]$ 128 | 129 | 130 | \subsection{DRV} 131 | \subsubsection{Bernoulli} 132 | $X \sim Be(p)$, indicate whether an event is successful. 133 | \[ 134 | P(X = k) = p^k (1-p)^{(1-k)}, \, k = 0 \text{ or } 1 135 | \] 136 | Statistics: $E(X) = p, \ \text{var}(X) = pq = p(1-p)$ \\ 137 | MGF: $M_X(t) = 1 - p + pe^t$ 138 | 139 | 140 | \subsubsection{Binomial} 141 | $X \sim Bin(n, p)$, total number of successes in $n$ i.i.d. $Be(p)$ trials. 142 | \[ 143 | P(X=k) = {n \choose k} p^x q^{n-x}, \, k = 0, 1, \ldots, n 144 | \] 145 | Statistics: $E(X) = np, \ \text{var}(X) = npq = np(1-p)$ \\ 146 | MGF: $M_X(t) = (1 - p + pe^t)^n$ 147 | 148 | 149 | \subsubsection{Geometric} 150 | $X \sim Geom(p)$, where $X = 1, 2, ...$. Memoryless Property. 151 | \[ 152 | P(X=k) = pq^{k-1}, \ k = 1, 2, ... 153 | \] 154 | Statistics: $E(X) = \frac{1}{p}, \ \text{var}(X) = \frac{1-p}{p^2}$ \\ 155 | MGF: $M_X(t) = \frac{pe^t}{1-qe^t}$ 156 | 157 | 158 | \subsubsection{Negative Binomial} 159 | $X \sim NB(r, p)$, where $X = r, r+1, ...$ 160 | \[ 161 | P(X=k) = {{k-1} \choose {r-1}} \ p^r q ^ {x-r}, \ k = r, r+1, ... 162 | \] 163 | Statistics: $E(X) = \frac{r}{p}, \ \text{var}(X) = \frac{r(1-p)}{p^2}$ 164 | 165 | 166 | \subsubsection{Poisson} 167 | $X \sim Poisson(\lambda)$ 168 | \[ 169 | P(X = k) = e^{-\lambda} \frac{\lambda^k}{k!}, \ k = 0, 1, ... 170 | \] 171 | Statistics: $E(X) = \text{var}(X) = \lambda$ \\ 172 | MGF: $M_X(t) = e^{\lambda(e^t-1)}$ \\ 173 | Properties: $Poisson(\alpha) + Poisson(\beta) = Poisson(\alpha+\beta)$ 174 | 175 | 176 | \subsubsection{Hypergeometric} 177 | Suppose there are $N$ identical balls, $m$ of them are red and $N-m$ are blue. 178 | $X \sim H(n, N, m)$ is \#red balls in $n$ draws without replacement. 179 | \[ 180 | P(X = k) = \frac{{m \choose k}{{N-m} \choose {n-k}}}{{N \choose n}}, \ k = 0, 1, ..., n 181 | \] 182 | \\ 183 | Statistics: $E(X) = \frac{nm}{N}, \ \text{var}(X) = \frac{nm}{N} \big[ \frac{(n-1)(m-1)}{N-1} + 1 - \frac{nm}{N} \big]$ 184 | 185 | 186 | \subsection{CRV} 187 | 188 | \subsubsection{Uniform} 189 | $X \sim U(a, b)$ 190 | \[ 191 | f(x) = \frac{1}{b-a}, \quad a < x < b 192 | \] 193 | Statistics: $E(X) = \frac{a+b}{2}, \ \text{var}(X) = \frac{(b-a)^2}{12}$ \\ 194 | MGF: $M_X(t) = \frac{e^{\beta t} - e^{\alpha t}}{(\beta - \alpha t)^t},\ t \neq 0$ 195 | 196 | 197 | \subsubsection{Exponential} 198 | $X \sim Exp(\lambda)$ for $\lambda > 0$. Memoryless Property 199 | \begin{align*} 200 | f(x) &= \lambda e^{-\lambda x}, x \geq 0 \\ 201 | F(x) &= 1 - e^{-\lambda x}, x \geq 0 202 | \end{align*} 203 | Statistics: $E(X) = \frac{1}{\lambda}, \ \text{var}(X) = \frac{1}{\lambda^2}$ \\ 204 | MGF: $M_X(t) = \frac{\lambda}{\lambda - t}, \text{for } t < \lambda$ 205 | 206 | 207 | \subsubsection{Normal} 208 | $X \sim N(\mu, \sigma^2)$. Special case : $Z \sim N(0, 1)$ 209 | \[ 210 | f_X(x) = \frac{1}{\sqrt{2 \pi} \sigma} e^{-(x-\mu)^2/(2\sigma^2)}, \ x \in \mathbb{R} 211 | \] 212 | \\ 213 | Statistics: $E(X) = \mu, \ \text{var}(X) = \sigma^2$ \\ 214 | MGF: $M_X(t) = e^{\mu t + \sigma^2 t^2 / 2}$ 215 | 216 | 217 | \subsubsection{Gamma} 218 | $X \sim Gamma(\alpha, \lambda)$ for shape $\alpha$, and rate $\lambda > 0$. ($1/\lambda$ is scale parameter) 219 | \[ 220 | f(x) = \frac{\lambda^\alpha}{\Gamma(\alpha)}x^{\alpha - 1}e^{-\lambda x}, \quad x \geq 0 221 | \] 222 | Statistics: $E(X) = \frac{\alpha}{\lambda}, \ \text{var}(X) = \frac{\alpha}{\lambda^2}$ \\ 223 | MGF: $M_X(t) = \left(1 - \frac{t}{\lambda}\right)^{-\alpha},\quad t < \beta$ \\ 224 | Special case: $Exp(\lambda) = Gamma(1, \lambda)$, $\chi^2_n = Gamma(\frac{n}{2}, \frac{1}{2})$ \\ 225 | Properties: $Gamma(a, \lambda) + Gamma(b, \lambda) = Gamma(a+b, \lambda)$, and $cX \sim Gamma(\alpha, \frac{\lambda}{c})$ 226 | \\ 227 | \textbf{Gamma function} $\Gamma(\alpha) = \int_0^{\infty} e^{-y}y^{\alpha-1} \, dy$ \\ 228 | $\Gamma(1) = 1, \ \Gamma\left(\frac{1}{2}\right) = \sqrt{\pi}$ and $\Gamma(n) = (n-1)!, \ n \in \mathbb{Z}^{+}$ 229 | 230 | 231 | \subsubsection{Beta} 232 | $X \sim B(a, b)$ where $a>0, b>0$ has support $[0, 1]$ 233 | \[ 234 | f(X) = \frac{\Gamma(a+b)}{\Gamma(a)\Gamma(b)} x^{a-1} (1-x)^{b-1}, 0 \le x \le 1 235 | \] 236 | Statistics: $E(X) = \frac{1}{1 + \beta / \alpha}, \ \text{var}(X) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$ 237 | Special case: $\text{Unif}(0, 1) = B(1, 1)$ 238 | 239 | \textbf{Beta function} $B(a, b) = \int_0^1 t^{a-1} (1-t)^{b-1} \ dt = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$ 240 | 241 | 242 | \hline 243 | 244 | 245 | 246 | \section{Chapter 02 - Random Variables} 247 | \subsection{Functions of a Random Variable} 248 | \subsubsection{Properties of CDF} 249 | \emph{(Ch 2.3 Prop. C)} Let $Z = F(X)$, then $Z \sim \text{Unif}(0, 1)$ 250 | 251 | \emph{(Ch 2.3 Prop. D)} Let $U \sim \text{Unif}(0, 1)$, and let $X = F^{-1}(U)$, then the CDF of $X$ is $F$. 252 | 253 | \subsubsection{Inverse CDF Method} 254 | For a r.v. $X$ with CDF $F$ to be generated, let $U = F(X)$ and write it as $X = F^{-1}(U)$, then generate with following steps: 255 | \begin{enumerate} 256 | \item Generate $u$ from a $\text{Unif}(0, 1)$. 257 | \item Deliver $x = F^{-1}(u)$. 258 | \end{enumerate} 259 | 260 | \subsubsection{Distribution of a Function of R.V.} 261 | For r.v. $X$ with pdf. $f_X(x)$, assume $g(x)$ is a function of $X$ that is \textbf{strictly monotonic} and \textbf{differentiable}. Then the pdf. of $Y=g(X)$, 262 | \[ 263 | f_Y(y) = f_X(g^{-1}(y)) \left| \frac{d}{dy} g^{-1}(y) \right|, \ y=g(x) \text{ for some $x$} 264 | \] 265 | 266 | 267 | 268 | \section{Chapter 03 - Joint Distributions} 269 | 270 | \subsection{Joint Distributions} 271 | \subsubsection{Copula} 272 | A \textbf{copula}, $C(u, v)$, is a joint CDF where the marginal distributions are standard uniform. It has properties as shown below: 273 | \begin{itemize} 274 | \item $C(u, v)$ is defined over $[0, 1] \times [0, 1]$ and is non-decreasing 275 | \item $P(U \le u) = C(u, 1)$ and $P(V \le v) = C(1, v)$ 276 | \item joint density function $c(u, v) = \frac{\partial^2}{\partial u \partial v}C(u, v) \ge 0$ 277 | \end{itemize} 278 | Construct joint distributions from marginal distributions given using copula: 279 | For any two CRVs $X$ and $Y$ and a copula $C(u, v)$ given, 280 | \[ 281 | F_{XY}(x, y) = C(F_X(x), F_Y(y)) 282 | \] 283 | is a joint distribution that has marginal distributions $F_X(x)$ and $F_Y(y)$. Correspondingly, the joint density is 284 | \[ 285 | f_{XY}(x, y) = c(F_X(x), F_Y(y)) \ f_X(x) f_Y(y) 286 | \] 287 | 288 | \subsubsection{Farlie Morgenstern Family} 289 | For any two CRVs $X$ and $Y$ with their CDFs $F(x)$ and $G(y)$ given, it is shown that for any constant $|\alpha| \le 1$, 290 | \[ 291 | H(x, y) = F(x) \ G(y) \ \Big[ 1 + \alpha(1-F(x))(1-G(y) \Big] 292 | \] 293 | is a bivariate joint CDF of $X$ and $Y$, with its marginal CDFs equal to $F(x)$ and $G(y)$. 294 | 295 | Farlie Morgenstern copula: $C(u, v) = uv(1+\alpha(1-u)(1-v))$ is the copula used in the Farlie Morgenstern Family. 296 | 297 | 298 | \subsubsection{Bivariate Normal Distribution} 299 | If $X$ and $Y$ are jointly distributed with bivariate normal, 300 | \[ 301 | f(X, Y) = \frac{1}{2\pi \sigma_x \sigma_y \sqrt{1-\rho^2}} e^{ -\frac{1}{2(1-\rho^2)} \Big[ \frac{(x-\mu_x)^2}{\sigma_x^2} + \frac{(y-\mu_y)^2}{\sigma_y^2} - \frac{2\rho (x-\mu_x) (y - \mu_y) }{\sigma_x \sigma_y} \Big] } 302 | \] 303 | where $-1 < \rho < 1$ is the correlation coefficient and the other 4 parameters are reflected in marginal distributions, 304 | \[ 305 | X \sim \mathcal{N}(\mu_x, \sigma_x^2), \ 306 | Y \sim \mathcal{N}(\mu_y, \sigma_y^2) 307 | \] 308 | For a joint distribution to be considered bivariate normal, it must satisfy both: 309 | \begin{enumerate} 310 | \item Its two marginal distributions are normal 311 | \item The contours for its joint density function are elliptical 312 | \end{enumerate} 313 | 314 | 315 | \subsection{Conditional Joint Distributions} 316 | \subsubsection{Rejection Method} 317 | For a r.v. $X$ with density function $f(x)$ to be generated, if $f(x) > 0$ for $a \le x \le b$, then 318 | \begin{enumerate} 319 | \item Let $M = M(x)$ s.t. $M(x) \ge f(x)$ for $a \le x \le b$ 320 | \item Let $m = m(x) = \frac{M(x)}{\int_a^b M(t) dt}$ (i.e. $m$ is a pdf of support $[a, b]$) 321 | \item Generate $T$ with density $m$. 322 | \item Generate $U$ which follows $\text{Unif}(0, 1)$ independent of $T$. 323 | \item If $M(T) \times U \leq f(T)$ then deliver $T$; otherwise, go base to Step 1 and repeat. 324 | \end{enumerate} 325 | 326 | 327 | \subsection{Functions of Joint Distributions} 328 | \emph{(Ch 3.6.2 Prop. A)} Suppose $X$ and $Y$ are jointly distributed and $u = g_1(x,y), \ v = g_2(x, y)$ can be inverted as $x = h_1(u,v), \ y = h_2(u, v)$ then 329 | \[ 330 | f_{UV} (u, v) = f_{XY}(h_1(u,v), h_2(u,v)) \ \left|J^{-1}(h_1, h_2)\right| 331 | \] 332 | 333 | \subsection{Sum/Quotient of Random Variables} 334 | 335 | Suppose $X$ and $Y$ are independent and have JDF $f$. Then for $U = X + Y$, $$f_U(u) = \int_{-\infty}^\infty f(x, u - x) \,dx,$$ and for $V = X/Y$, $$f_V(v) = \int_{-\infty}^\infty |x|f(x, xv) \,dx.$$ 336 | 337 | 338 | \subsection{Order Statistics} 339 | \emph{(Ch 3.7 Thm. A)} Density function of $X_{(k)}$, the k-th order statistics, 340 | \[ 341 | f_k(x) = \frac{n!}{(k-1)! (n-k)!} f(x) F^{k-1}(x) [1-F(x)]^{n-k} 342 | \] 343 | 344 | 345 | 346 | 347 | 348 | \section{Chapter 04 - Expected Values} 349 | \subsection{Model for Measurement Error} 350 | Let $x_0$ denotes the true value of a quantity being measured. Then the measurement, $X$, can be modeled as: 351 | \[ 352 | X = x_0 + \beta + \epsilon 353 | \] 354 | where $\beta$ is \textbf{bias}, a constant and $\epsilon$ is the random component of error. $E(\epsilon) = 0$ and $\text{var}(\epsilon) = \sigma^2$. 355 | 356 | \textbf{Mean Squared Error (MSE)} is a measure of the overall measurement error, 357 | \begin{align*} 358 | \text{MSE} 359 | &= E[(X-x_0)^2] \ \text{(Definition)} \\ 360 | &= \sigma^2 + \beta^2 361 | \end{align*} 362 | 363 | \subsection{Conditional Expectation \& Prediction} 364 | \subsubsection{Find Expectation \& Variance by Conditioning} 365 | \[ 366 | E(Y) = E[ E(Y|X) ], \ 367 | \text{var}(Y) = \text{var}[E(Y|X)] + E[\text{var}(Y|X)] 368 | \] 369 | 370 | \subsubsection{Random Sum} 371 | \[ 372 | E(T) = E(N)E(X), \ \text{var}(T) = [E(X)]^2 \text{var} + E(N) \text{var}(X) 373 | \] 374 | 375 | \subsubsection{Predictions} 376 | Suppose $X$ and $Y$ are jointly distributed. If $X$ is observed, the predictor of $Y$ that minimises MSE would be 377 | \[ 378 | h(Y) = E(Y|X) 379 | \] 380 | 381 | 382 | 383 | 384 | % Post Midterm Topics 385 | 386 | \subsection{Delta Method} 387 | Consider $Y = g(X)$ where the PDF of $X$ is unknown but $\mu_X$ and $\sigma_X^2$ is known. Then \[ 388 | E(Y) \approx g(\mu_X) + \frac{1}{2}\sigma_X^2g''(\mu_x),\quad 389 | \text{var}(Y) \approx \sigma_X^2[g'(\mu_X)]^2. 390 | \] 391 | 392 | 393 | \section{Chapter 05 - Limit Theorems} 394 | The RV $X$ \textbf{converges in probability} to $\mu$ if for any $\epsilon > 0$, \[ 395 | P(|X - \mu| > \epsilon) \to 0. 396 | \] 397 | 398 | The RVs $X_1, X_2, \ldots$ with CDFs $F_1, F_2, \ldots$ \textbf{converge in distribution} to $X$ with CDF $F$ if \[ 399 | \lim_{n \to \infty} F_n(x) = F(X) 400 | \] at every point which $F$ is continuous. 401 | 402 | \subsubsection{Weak Law of Large Numbers} 403 | Let $X_1, X_2, \ldots$ be a sequence of independent RVs. Then $\overline{X_n} = n^{-1}\sum_{i=1}^n X_i$ converges to $\mu$ in probability as $n \to \infty$. 404 | 405 | \subsubsection{Strong Law of Large Numbers} 406 | \[ 407 | P(\lim_{n \to \infty} \overline{X_n} = \mu) = 1. 408 | \] 409 | 410 | \subsubsection{Continuity Theorem} 411 | Let $F_n$ be a sequence of CDFs with corresponding MGFs $M_n$. If $M_n(t) \to M(t)$ for all $t$ in an open interval containing zero, then $F_n(x) \to F(x)$ at all continuity points of $F$. 412 | 413 | \subsubsection{Central Limit Theorem} 414 | Let $X_1, X_2, \ldots$ be a sequence of independent RVs with mean $\mu$ and variance $\sigma^2$, and CDF $F$ and MGF $M$ defined in a neighbourhood of zero. Let $S_n = \sum_{i=1}^n (X_i - \mu)$. Then \[ 415 | \lim_{n \to \infty} P\left(\frac{S_n}{\sigma\sqrt{n}} \leq x\right) = \Phi(x),\quad -\infty < x < \infty. 416 | \] 417 | 418 | \subsubsection{Common Convergences in Distribution} 419 | \emph{(Tut.5 Qn3)} $Bin(n, p) \overset{d}{\to} Poission(np)$ as $n \to \infty, \ p \to 0$.\\ 420 | \emph{(Tut.5 Qn4)} For $X$ standardised $Gamma(\alpha, \lambda)$, $X \overset{d}{\to} Z$ as $\alpha \to \infty$. 421 | 422 | \subsubsection{Miscellaneous} 423 | \emph{(Tut.5 Qn13)} For sequence $a_n \to a$, $(1 + \frac{a_n}{n})^n \to e^a$. 424 | 425 | 426 | 427 | \section{Chapter 06 - Distributions Derived from the Normal Distribution} 428 | \subsection{Common Distributions} 429 | \subsubsection{Chi-Square Distribution} 430 | For independent $Z_1, \ldots, Z_n \sim N(0, 1)$, \[ 431 | V = \sum_{i=1}^n Z_i^2 \sim \chi_n^2. 432 | \] $M_V(t) = (1 - 2t)^{-n/2}$. 433 | 434 | \subsubsection{t-distribution} 435 | If $Z \sim N(0, 1)$ and $U \sim \chi_n^2$ are independent, then \[ 436 | T = \frac{Z}{\sqrt{U/n}} \sim t_n. 437 | \] 438 | 439 | \subsubsection{F-distribution} 440 | For independent $U \sim \chi_m^2$ and $V \sim \chi_n^2$, \[ 441 | W = \frac{U/m}{V/n} \sim F_{m,n}. 442 | \] 443 | 444 | \subsection{Related Identities} 445 | \emph{(Tut.6 Qn2)} For $X \sim F_{n,m}$, $X^{-1} \sim F_{m,n}$. \\ 446 | \emph{(Tut.6 Qn3)} For $X \sim t_n$, $X^2 \sim F_{1,n}$. 447 | 448 | \subsection{Sample Mean \& Variance} 449 | \[ 450 | \overline{X} = \frac{1}{n}\sum_{i=1}^n X_i,\quad 451 | S^2 = \frac{1}{n - 1}\sum_{i=1}^n (X_i - \overline{X})^2 \sim \chi_{n-1}^2. 452 | \] 453 | 454 | \subsubsection{Related Identities} 455 | For i.i.d $X_1, ..., X_n$ from $N(\mu, \sigma^2)$, \\ 456 | \emph{(Ch 6.2 Thm.A)} $\bar{X}$ and the vector $\langle X_1 - \bar{X}, ..., X_n - \bar{X} \rangle$ are independent. \\ 457 | \emph{(Ch 6.2 Thm.B)} $\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}$. \\ 458 | \emph{(Ch 6.2 Coro.B)} $\frac{\bar{X} - \mu}{S/\sqrt{n}} \sim t_{n-1}$. 459 | 460 | 461 | 462 | 463 | \section{Chapter 08 - Estimation of Parameters and Fitting of Probability Distributions} 464 | 465 | Let $\hat{\theta}_n$ be an estimate of a parameter $\theta$ based on a sample of size $n$. Then $\hat{\theta}_n$ is \textbf{consistent in probability} if it converges in probability to $\theta$ as $n \to \infty$. 466 | 467 | \subsection{Method of Moments} 468 | \begin{enumerate} 469 | \item Calculate low order moments in terms of their parameters. 470 | \item Find expressions for the parameters in terms of the moments. 471 | \item Insert sample moments into the expressions. 472 | \end{enumerate} 473 | 474 | \subsection{Method of Maximum Likelihood} 475 | Consider RVs $X_1, \ldots, X_n$ with joint PDF $f(x_1, \ldots, x_n \mid \theta)$. The \textbf{likelihood} of $\theta$ is \[ 476 | \operatorname{lik}(\theta) = f(x_1, \ldots, x_n \mid \theta). 477 | \] If $X_1, \ldots, X_n$ are independent, then the \textbf{log likelihood} can be expressed as \[ 478 | l(\theta) = \sum_{i=1}^n \log[f(x_i \mid \theta)]. 479 | \] 480 | 481 | \subsubsection{Invariance Property} 482 | Let $\hat{\theta} = (\hat{\theta}_1, \ldots, \hat{\theta}_k)$ be a mle of $\theta = (\theta_1, \ldots, \theta_k)$ in the density $f(x \mid \theta_1, \ldots, \theta_k)$. If $\tau(\theta) = (\tau_1(\theta), \ldots, \tau_r(\theta))$, $1 \leq r \leq k$ is a transformation of the parameter space $\Theta$, then a mle of $\tau(\theta)$ is $\tau(\hat{\theta}) = (\tau_1(\hat{\theta}), \ldots, \tau_r(\hat{\theta}))$. 483 | 484 | \subsubsection{Fisher Information} 485 | \[ 486 | I(\theta) = E\left\{\left[\frac{\partial}{\partial\theta} \log f(X \mid \theta)\right]^2\right\}. 487 | \] 488 | 489 | \subsubsection{Large Sample Theory} 490 | Under (varying) smoothness conditions on $f$, \begin{enumerate} 491 | \item The mle from i.i.d. sample is consistent. 492 | \item \[ 493 | I(\theta) = -E\left[\frac{\partial^2}{\partial\theta^2} \log f(X \mid \theta)\right]. 494 | \] 495 | \item The distribution of $\hat{\theta}$ tends towards \[ 496 | N\left(\theta_0, \frac{I}{nI(\theta_0)}\right). 497 | \] 498 | \item An approximate $100(1 - \alpha)\%$ confidence interval for $\theta_0$ is \[ 499 | \hat{\theta} \pm z_{\alpha/2}\frac{1}{\sqrt{nI(\hat{\theta})}}. 500 | \] 501 | \end{enumerate} 502 | 503 | \subsection{Bayesian Inference} 504 | Unknown parameter $\theta$ is treated as a distribution, not a value. \textbf{Prior distribution} $f_\Theta(\theta)$ represents our knowledge (assumption) about $\theta$ before observing data $X$. After observation, we have a better estimation using the \textbf{posterior distribution} $f_{\Theta | X}(\theta | x)$, where 505 | \[ 506 | f_{\Theta | X} = \frac{f_{X\Theta}(x, \theta)}{f_X(x)} = \frac{f_{X|\Theta}(x|\theta)f_\Theta(\theta)}{ \int f_{X|\Theta}(x|t)f_\Theta(t) dt } 507 | \] 508 | In short, it means 509 | \[ 510 | \text{Posterior density} \propto \text{likelihood} \times \text{prior density} 511 | \] 512 | 513 | \subsubsection{Large Sample Theory for Bayesian} 514 | As $n \to \infty$, 515 | \[ 516 | \Theta | X \sim N(\hat{\theta}, -[l''(\hat{\theta})]^{-1}) 517 | \] 518 | where $\hat{\theta}$ is the mle of $\theta_0$. 519 | 520 | \subsection{Bootstrapping Method} 521 | \subsubsection{For Estimating Sampling Distribution} 522 | \begin{enumerate} 523 | \item Assume some distribution provides a good fit to the data. 524 | \item Simulate $N$ random samples of size $n$ from the distribution using the estimated parameter $\hat{\theta}$. 525 | \item For each random sample, calculate estimates of the distribution parameters using either Method of Moments or Maximum Likelihood, $\theta^{*}_{j}$ for $j=1, ..., N$. 526 | \item Use the $N$ values of estimates $\theta^{*}_{j}$ to approximate the sampling distributions of the parameters. 527 | \end{enumerate} 528 | \subsubsection{For Estimating Confidence Interval} 529 | \begin{enumerate} 530 | \item Approximate the distribution of $\hat{\theta} - \theta_0$ with that of $\theta^{*} - \hat{\theta}$. 531 | \item Obtain the lower and upper bounds $\underline{\delta}$ and $\overline{\delta}$ 532 | \[ 533 | P(\theta^* - \hat{\theta} < \underline{\delta}) 534 | = P(\theta^* - \hat{\theta} > \overline{\delta}) = \frac{\alpha}{2} 535 | \] 536 | \item The CI for $\theta_0$ can then be constructed: 537 | \[ 538 | P(\hat{\theta} - \overline{\delta} \le \theta_0 \le \hat{\theta} - \underline{\delta}) = 1 - \alpha 539 | \] 540 | \end{enumerate} 541 | 542 | \subsection{Estimator Properties} 543 | An estimator $\hat{\theta}$ of $\theta_0$ is \textbf{consistent} if $\hat{\theta} \overset{p}{\to} \theta_0$ as $n \to \infty$. 544 | \\ 545 | 546 | The \textbf{efficiency} of $\hat{\theta}$ relative to $\tilde{\theta}$ is \[ 547 | \operatorname{eff}(\hat{\theta}, \tilde{\theta}) = \frac{\text{var}(\tilde{\theta})}{\text{var}(\hat{\theta})}. 548 | \] 549 | 550 | A statistic $T(X_1, \ldots, X_N)$ is \textbf{sufficient} for $\theta$ if the conditional distribution of $X_1, \ldots, X_n$ given $T = t$ does not depend on $\theta$ for any value of $t$. 551 | 552 | \subsubsection{MSE} 553 | $\text{MSE } = \text{var}(\hat{\theta}) + (E(\hat{\theta}) - \theta_0)^2$. \\ 554 | If $\hat{\theta}$ is an unbiased estimator of $\theta_0$, $\text{MSE } = \text{var}(\hat{\theta})$. 555 | 556 | \subsubsection{Cramer-Rao Inequality} 557 | Let $T = t(X_1, \ldots, X_n)$ be an unbiased estimator of $\theta$. Then, under smoothness assumptions on $f(x \mid \theta)$, $\text{var}(T) \geq 1/nI(\theta)$. 558 | 559 | \subsubsection{Factorisation Theorem for Sufficiency} 560 | $T(X_1, \ldots, X_n)$ is sufficient for a parameter of $\theta$ iff the joint pdf factors in the form \[ 561 | f(x_1, \ldots, x_n \mid \theta) = g[T(x_1, \ldots, x_n), \theta]h(x_1, \ldots, x_n). 562 | \] 563 | 564 | \subsubsection{Rao-Blackwell Theorem} 565 | Let $\hat{\theta}$ be an estimator of $\theta$ with $E(\hat{\theta}^2) < \infty$ for all $\theta$. Suppose that $T$ is sufficient for $\theta$ and let $\tilde{\theta} = E(\hat{\theta} \mid T)$. Then for all $\theta$, \[ 566 | E[(\tilde{\theta} - \theta)^2] \leq E[(\hat{\theta} - \theta)^2]. 567 | \] 568 | 569 | \section{Testing Hypotheses and Assessing Goodness of Fit} 570 | \subsection{Testing Hypotheses} 571 | The \textbf{likelihood ratio} is defined as \[ 572 | \frac{P(x \mid H_0)}{P(x \mid H_1)} = \frac{P(H_1)}{P(H_0)}\frac{P(H_0 \mid x)}{P(H_1 \mid x)} 573 | \] with $H_0$ rejected if the likelihood ratio is less than $c = \frac{P(H_1)}{P(H_0)}$. 574 | 575 | 576 | \subsubsection{Uniformly Most Powerful} 577 | If $H_0$ is simple and $H_1$ is composite, a test that is most powerful for every simple alternative in $H_1$ is said to be \textbf{uniformly most powerful}. \\ 578 | \textbf{Conditions}: 1) $H_1$ must be one-sided. 2) The threshold for rejection region must be independent of $\mu_1$ of $H_1$. 579 | 580 | 581 | \subsubsection{Neyman-Pearson Lemma} 582 | Suppose that $H_0$ and $H_1$ are simple hypotheses and that the test that rejects $H_0$ whenever the likelihood ratio is less than $c$ has significance level $\alpha$. Then any other test for which the significance level is at most $\alpha$ has power at most that of the likelihood ratio test. 583 | 584 | \subsection{Generalised Likelihood Ratio} 585 | The test statistic corresponding to the \textbf{generalised likelihood ratio} is \[ 586 | \Lambda = \frac{\max_{\theta \in \omega_0}[\operatorname{lik}(\theta)]}{\max_{\theta \in \Omega}[\operatorname{lik}(\theta)]} 587 | \] where $\omega_0$ is the set of all possible values of $\theta$ specified by $H_0$ and similarly for $\omega_1$, with $\Omega = \omega_0 \cup \omega_1$. The threshold $\lambda_0$ is chosen such that $P(\Lambda \leq \lambda_0 \mid H_0) = \alpha$, the desired significance level. 588 | 589 | Under large sample theory, \[ 590 | -2\log\Lambda \dot\sim \chi_\nu^2 591 | \] where $\nu = \dim\Omega - \dim\omega_0$. 592 | 593 | \subsubsection{Multinomial Case} 594 | Let $O_i = n \hat{p_i}$ and $E_i = np_i(\hat{\theta})$ denotes the observed and estimated cases, 595 | \[ 596 | -2\log\Lambda = 2 \sum_{i=1}^{m} O_i \log\left(\frac{O_i}{E_i}\right) 597 | \] 598 | 599 | \subsubsection{Pearson's Chi-square Statistic} 600 | \[ 601 | X^2 = \sum_{i=1}^m \frac{[x_i - np_i(\hat{\theta})]^2}{np_i(\hat{\theta})} \sim \chi_\nu^2, 602 | \] where $\nu$ is the number of degrees of freedom. In practice, $np_i(\hat{\theta}) \geq 5$ is required for the approximation to be good. 603 | 604 | \subsubsection{Poisson Dispersion Test} 605 | Given $x_1, \ldots, x_n$ and testing $H_0$ that the counts are Poisson with a common parameter $\lambda$ versus $H_1$ that they are Poisson but with different rates, the likelihood ratio test statistic is \[ 606 | -2\log\Lambda = 2\sum_{i=1}^n x_i\log\left(\frac{x_i}{\bar{x}}\right) \approx \frac{1}{\bar{x}}\sum_{i=1}^n (x_i - \bar{x})^2. 607 | \] 608 | 609 | \subsection{Goodness of Fit} 610 | \subsubsection{Hanging Diagrams} 611 | \begin{itemize} 612 | \item ~ historgram: Plot of $n_j - \hat{n_j}$ 613 | \item ~ rootogram: Plot of $\sqrt{n_j} - \sqrt{\hat{n_j}}$, var-stabilised 614 | \item ~ chi-gram: Plot of $\frac{n_j - \hat{n_j}}{\sqrt{\hat{n_j}}}$, var-stabilised 615 | \end{itemize} 616 | \textbf{Variance-stabilising transform}: a transformation $Y = g(X)$ that makes $\text{var}(Y)$ (approximately) constant using Delta Method. 617 | 618 | \subsubsection{Probability Plots} 619 | \begin{itemize} 620 | \item P-P Plot: Plot $F(X_{(k)})$ against $\frac{k}{n+1}$ 621 | \item Q-Q Plot: Plot $X_{(k)}$ against $F^{-1}(\frac{k}{n+1})$ 622 | \end{itemize} 623 | 624 | \subsubsection{Tests for Normality} 625 | The \textbf{coefficient of skewness} is defined as \[ 626 | b_1 = \frac{\frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^3}{s^3}, 627 | \] where the test rejects for large values of $|b_1|$. 628 | 629 | The \textbf{coefficient of kurtosis} is defined as \[ 630 | b_2 = \frac{\frac{1}{n}\sum_{i=1}^n (X_i - \bar{X})^4}{s^4} 631 | \] where the test rejects for large values of $|b_2|$. 632 | 633 | \end{multicols} 634 | \end{document} 635 | --------------------------------------------------------------------------------