├── .gitattributes ├── LICENSE ├── README.md ├── cheatsheet.aux ├── cheatsheet.dvi ├── cheatsheet.log ├── cheatsheet.pdf ├── cheatsheet.synctex.gz ├── cheatsheet.tex └── content ├── Central_limit.tex ├── Confidence_intervals.log ├── Confidence_intervals.tex ├── Covariance_Matrix.tex ├── Generalized_Linear_Models.log ├── Generalized_Linear_Models.tex ├── Hypothesis_tests.log ├── Hypothesis_tests.tex ├── Law_large_Numbers.tex ├── Likelihood.log ├── Likelihood.tex ├── M_estimation.tex ├── Models.tex ├── Moments.tex ├── Multivariate_RV.tex ├── OLS.log ├── OLS.tex ├── Quantiles.log ├── Quantiles.tex ├── Total_variation.log ├── Total_variation.tex ├── abbildungen.tex ├── algebra.log ├── algebra.tex ├── aussagen.log ├── aussagen.tex ├── bayesian.tex ├── beweise.tex ├── bool.tex ├── def.aux ├── def.tex ├── estimators.log ├── estimators.tex ├── expectation_variance.log ├── expectation_variance.tex ├── formeln.tex ├── graphalgo.tex ├── graphen.tex ├── mengen.log ├── mengen.tex ├── notes.tex ├── probability_distributions.log ├── probability_distributions.tex ├── relationen.tex └── test.tex /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Max Thomasberger 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Capstone cheatsheet 1 2 | Cheatsheet for the MITx Data Science and Statistics capstone exam part 1 by Blechturm. 3 | 4 | Please note that since I am working alone on this there has been no cross-check of the formulae and definitions. I am trying to do my best to be diligent but errors are possible. So please use it at your own risk. It is also a reflection of my mind and the things I find important which may not coincide with your priorities. If you want to help me drop be a line. 5 | 6 | 7 | The document structure is based on https://github.com/tim-st/latex-cheatsheet 8 | 9 | 10 | -------------------------------------------------------------------------------- /cheatsheet.aux: -------------------------------------------------------------------------------- 1 | \relax 2 | \providecommand\zref@newlabel[2]{} 3 | \providecommand*\new@tpo@label[2]{} 4 | \catcode `"\active 5 | \babel@aux{ngerman}{} 6 | \pgfsyspdfmark {pgfid1}{0}{38964620} 7 | \@writefile{toc}{\contentsline {section}{\numberline {1}Statistical models}{1}\protected@file@percent } 8 | \@writefile{toc}{\contentsline {subsection}{\numberline {1.1}Identifiability}{1}\protected@file@percent } 9 | \@writefile{toc}{\contentsline {section}{\numberline {2}Estimators}{1}\protected@file@percent } 10 | \@writefile{toc}{\contentsline {section}{\numberline {3}LLN and CLT}{1}\protected@file@percent } 11 | \@writefile{toc}{\contentsline {section}{\numberline {4}Quantiles of a Distribution}{1}\protected@file@percent } 12 | \@writefile{toc}{\contentsline {section}{\numberline {5}Confidence intervals}{1}\protected@file@percent } 13 | \@writefile{toc}{\contentsline {subsection}{\numberline {5.1}Two-sided asymptotic CI}{1}\protected@file@percent } 14 | \@writefile{toc}{\contentsline {subsection}{\numberline {5.2}Sample Mean and Sample Variance}{1}\protected@file@percent } 15 | \@writefile{toc}{\contentsline {subsection}{\numberline {5.3}Delta Method}{1}\protected@file@percent } 16 | \@writefile{toc}{\contentsline {section}{\numberline {6}Asymptotic Hypothesis tests}{1}\protected@file@percent } 17 | \@writefile{toc}{\contentsline {subsection}{\numberline {6.1}P-Value}{1}\protected@file@percent } 18 | \@writefile{toc}{\contentsline {subsection}{\numberline {6.2}Comparisons of two proportions}{1}\protected@file@percent } 19 | \@writefile{toc}{\contentsline {section}{\numberline {7}Non-asymptotic Hypothesis tests}{1}\protected@file@percent } 20 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.1}Chi squared}{1}\protected@file@percent } 21 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.2}Student's T Test}{1}\protected@file@percent } 22 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.3}Walds Test}{1}\protected@file@percent } 23 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.4}Likelihood Ratio Test}{1}\protected@file@percent } 24 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.5}Implicit Testing}{1}\protected@file@percent } 25 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.6}Goodness of Fit Discrete Distributions}{1}\protected@file@percent } 26 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.7}Kolmogorov-Smirnov test}{1}\protected@file@percent } 27 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.8}Kolmogorov-Lilliefors test}{1}\protected@file@percent } 28 | \@writefile{toc}{\contentsline {subsection}{\numberline {7.9}QQ plots}{1}\protected@file@percent } 29 | \@writefile{toc}{\contentsline {section}{\numberline {8}Distances between distributions}{1}\protected@file@percent } 30 | \@writefile{toc}{\contentsline {subsection}{\numberline {8.1}Total variation distance}{1}\protected@file@percent } 31 | \pgfsyspdfmark {pgfid2}{0}{38964620} 32 | \@writefile{toc}{\contentsline {subsection}{\numberline {8.2}KL divergence}{2}\protected@file@percent } 33 | \@writefile{toc}{\contentsline {section}{\numberline {9}Maximum likelihood estimation}{2}\protected@file@percent } 34 | \@writefile{toc}{\contentsline {subsection}{\numberline {9.1}Fisher Information}{2}\protected@file@percent } 35 | \@writefile{toc}{\contentsline {subsection}{\numberline {9.2}Asymptotic normality of the maximum likelihood estimator}{2}\protected@file@percent } 36 | \@writefile{toc}{\contentsline {section}{\numberline {10}Method of Moments}{2}\protected@file@percent } 37 | \@writefile{toc}{\contentsline {section}{\numberline {11}Bayesian Statistics}{2}\protected@file@percent } 38 | \@writefile{toc}{\contentsline {subsection}{\numberline {11.1}Bayes' formula}{2}\protected@file@percent } 39 | \@writefile{toc}{\contentsline {subsection}{\numberline {11.2}Jeffreys Prior}{2}\protected@file@percent } 40 | \@writefile{toc}{\contentsline {subsection}{\numberline {11.3}Bayesian confidence region}{2}\protected@file@percent } 41 | \@writefile{toc}{\contentsline {subsection}{\numberline {11.4}Bayes estimator}{2}\protected@file@percent } 42 | \@writefile{toc}{\contentsline {section}{\numberline {12}OLS}{2}\protected@file@percent } 43 | \@writefile{toc}{\contentsline {section}{\numberline {13}Generalized Linear Models}{2}\protected@file@percent } 44 | \@writefile{toc}{\contentsline {subsection}{\numberline {13.1}The Exponential Family}{2}\protected@file@percent } 45 | \@writefile{toc}{\contentsline {section}{\numberline {14}Expectation}{2}\protected@file@percent } 46 | \pgfsyspdfmark {pgfid3}{0}{38964620} 47 | \@writefile{toc}{\contentsline {section}{\numberline {15}Variance}{3}\protected@file@percent } 48 | \@writefile{toc}{\contentsline {section}{\numberline {16}Covariance}{3}\protected@file@percent } 49 | \@writefile{toc}{\contentsline {section}{\numberline {17}correlation coefficient}{3}\protected@file@percent } 50 | \@writefile{toc}{\contentsline {section}{\numberline {18}Important probability distributions}{3}\protected@file@percent } 51 | \@writefile{toc}{\contentsline {subsection}{\numberline {18.1}Useful to know}{3}\protected@file@percent } 52 | \@writefile{toc}{\contentsline {subsubsection}{\numberline {18.1.1}Min of iid exponential r.v}{3}\protected@file@percent } 53 | \@writefile{toc}{\contentsline {subsubsection}{\numberline {18.1.2}Counting Commitees}{3}\protected@file@percent } 54 | \@writefile{toc}{\contentsline {subsection}{\numberline {18.2}Finding Joint PDFS}{3}\protected@file@percent } 55 | \@writefile{toc}{\contentsline {section}{\numberline {19}Random Vectors}{3}\protected@file@percent } 56 | \@writefile{toc}{\contentsline {section}{\numberline {20}Algebra}{3}\protected@file@percent } 57 | \pgfsyspdfmark {pgfid4}{0}{38964620} 58 | \@writefile{toc}{\contentsline {section}{\numberline {21}Matrixalgebra}{4}\protected@file@percent } 59 | \@writefile{toc}{\contentsline {section}{\numberline {22}Calculus}{4}\protected@file@percent } 60 | \@writefile{toc}{\contentsline {section}{\numberline {23}Covariance Matrix}{4}\protected@file@percent } 61 | -------------------------------------------------------------------------------- /cheatsheet.dvi: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blechturm/MITx_capstone_1/51d644cdecabb3cb7c8dedc5816359ba641a3d19/cheatsheet.dvi -------------------------------------------------------------------------------- /cheatsheet.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blechturm/MITx_capstone_1/51d644cdecabb3cb7c8dedc5816359ba641a3d19/cheatsheet.pdf -------------------------------------------------------------------------------- /cheatsheet.synctex.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/blechturm/MITx_capstone_1/51d644cdecabb3cb7c8dedc5816359ba641a3d19/cheatsheet.synctex.gz -------------------------------------------------------------------------------- /cheatsheet.tex: -------------------------------------------------------------------------------- 1 | \documentclass[landscape,letterpaper]{extarticle} 2 | \usepackage[fontsize=7pt]{scrextend} 3 | %\usepackage{amsmath} 4 | \usepackage{mathtools} 5 | \usepackage[utf8]{inputenc} 6 | \usepackage[ngerman]{babel} 7 | \usepackage[T1]{fontenc} 8 | 9 | 10 | \usepackage{tikz} 11 | \usetikzlibrary{shapes,positioning,arrows,fit,calc,graphs,graphs.standard} 12 | \usepackage[nosf]{kpfonts} 13 | \usepackage[t1]{sourcesanspro} 14 | \usepackage{multicol} 15 | \usepackage{wrapfig} 16 | \usepackage[top=0mm,b ottom=1mm,left=0mm,right=1mm]{geometry} 17 | \usepackage[framemethod=tikz]{mdframed} 18 | \usepackage{microtype} 19 | \usepackage{pdfpages} 20 | \usepackage{bbm} 21 | 22 | 23 | \let\bar\overline 24 | 25 | \include{content/def} 26 | 27 | 28 | \AtBeginDocument{% 29 | \setlength{\belowdisplayskip}{0pt} \setlength{\belowdisplayshortskip}{0pt} \setlength{\abovedisplayskip}{0pt} \setlength{\abovedisplayshortskip}{0pt}} 30 | 31 | \begin{document} 32 | %\footnotesize 33 | \small 34 | \begin{multicols*}{7} 35 | 36 | % multicol parameters 37 | % These lengths are set only within the two main columns 38 | %\setlength{\columnseprule}{0.25pt} 39 | \setlength{\premulticols}{1pt} 40 | \setlength{\postmulticols}{1pt} 41 | \setlength{\multicolsep}{1pt} 42 | \setlength{\columnsep}{2pt} 43 | 44 | %\input{content/probability_distributions} 45 | 46 | %\input{content/expectation_variance} 47 | %\input{content/Covariance_Matrix} 48 | 49 | 50 | 51 | 52 | 53 | 54 | \input{content/Models} 55 | \input{content/estimators} 56 | \input{content/Law_large_Numbers} 57 | \input{content/Quantiles} 58 | \input{content/Confidence_intervals} 59 | \input{content/Hypothesis_tests} 60 | 61 | 62 | 63 | 64 | \input{content/Total_variation} 65 | \input{content/Likelihood} 66 | \input{content/Moments} 67 | \input{content/bayesian} 68 | \input{content/OLS} 69 | \input{content/Generalized_Linear_Models} 70 | 71 | 72 | \input{content/expectation_variance} 73 | %\input{content/Covariance_Matrix} 74 | 75 | 76 | 77 | \input{content/probability_distributions} 78 | %\input{content/Algebra} 79 | \input{content/notes} 80 | \input{content/Multivariate_RV} 81 | \input{content/Algebra} 82 | \input{content/Covariance_Matrix} 83 | 84 | %\input{content/probability_distributions} 85 | %\input{content/Algebra} 86 | 87 | \end{multicols*} 88 | \end{document} -------------------------------------------------------------------------------- /content/Central_limit.tex: -------------------------------------------------------------------------------- 1 | \section{Central Limit theorem} 2 | 3 | -------------------------------------------------------------------------------- /content/Confidence_intervals.tex: -------------------------------------------------------------------------------- 1 | \section{Confidence intervals} 2 | 3 | Confidence Intervals follow the form:\\ 4 | 5 | (statistic) $\pm$ (critical value)(estimated standard deviation of statistic)\\ 6 | 7 | Let $\displaystyle ( E,(\mathbb{P}_{\theta })_{\theta \in \Theta })$ be a statistical model based on observations $X_{1} , \ldots X_{n}$ and assume $\displaystyle \Theta \subseteq \mathbb{R}$. Let $\displaystyle \alpha \in ( 0,1)$.\\ 8 | \textbf{Non asymptotic} confidence interval of level $\displaystyle 1-\alpha $ for $\displaystyle \theta $:\\ 9 | Any random interval $\displaystyle \mathcal{I}$, depending on the sample $X_{1} , \ldots X_{n}$ but not at $\displaystyle \theta $ and such that:\\ 10 | $\mathbb{P}_{\theta }[\mathcal{I} \ni \theta ] \geq 1-\alpha ,\ \ \forall \theta \in \Theta$\\ 11 | Confidence interval of \textbf{asymptotic level} $\displaystyle 1-\alpha $ for $\displaystyle \theta $:\\ 12 | Any random interval $\displaystyle \mathcal{I}$ whose boundaries do not depend on $\displaystyle \theta $ and such that: $\lim _{n\rightarrow \infty }\mathbb{P}_{\theta } [\mathcal{I} \ni \theta ]\geq 1-\alpha ,\ \ \forall \theta \in \Theta $ 13 | \subsection{Two-sided asymptotic CI} 14 | Let $X_1, \ldots, X_n = \tilde{X}$ and $\tilde{X}\stackrel{iid} {\sim} P_{\theta}$. A two-sided CI is a function depending on $\tilde{X}$ giving an upper and lower bound in which the estimated parameter lies $\mathcal{I} = [l(\tilde{X},u(\tilde{X})]$ with a certain probability $\mathbb{P}(\theta \in \mathcal{I}) \geq 1 -q_{\alpha}$ and conversely $\mathbb{P}(\theta \not\in \mathcal{I}) \leq \alpha$\\ 15 | Since the estimator is a r.v. depending on $\tilde{X}$ it has a variance $Var(\hat{\theta}_n$ and a mean $\mathbb{E}[\hat{\theta}_n]$. 16 | Since the CLT is valid for every distribution standardizing the distributions and massaging the expression yields an an asymptotic CI: 17 | \begin{align*} 18 | \mathcal{I} = [&\hat{\theta}_n - \frac{q_{\alpha /2} \sqrt{Var(X_i)} }{\sqrt{n}},\\ 19 | &\hat{\theta}_n + \frac{q_{\alpha /2} \sqrt{Var(X_i)} }{\sqrt{n}}] 20 | \end{align*} 21 | This expression depends on the real variance $Var(X_i)$ of the r.vs, the variance has to be estimated.\\ 22 | Three possible methods: plugin (use sample mean or empirical variance), solve (solve quadratic inequality), conservative (use the theoretical maximum of the variance). 23 | \subsection{Sample Mean and Sample Variance} 24 | Let $X_1, ..., X_n \stackrel{iid}{\sim} P_{\mu}$, where $E(X_i)=\mu$ and $Var(X_i)=\sigma^2$ for all $i=1,2,...,n$\\ 25 | \textbf{Sample Mean:} 26 | \begin{align*} 27 | \bar{X}_n= \frac{1}{n} \sum_{i=1}^{n} X_i 28 | \end{align*} 29 | \textbf{Sample Variance:} 30 | \begin{align*} 31 | S_n &= \frac{1}{n} \sum_{i=1}^{n} (X_i - \bar{X}_n)^2\\ 32 | &= \frac{1}{n} (\sum_{i=1}^{n} X_i^2) - \bar{X}_n^2 33 | \end{align*} 34 | \textbf{Unbiased estimator of sample variance:} 35 | \begin{align*} 36 | \tilde{S}_n &= \displaystyle \frac{1}{n-1} \sum _{i=1}^ n \left(X_ i - \overline{X}_ n\right)^2\\ 37 | &= \frac{n}{n-1} S_n 38 | \end{align*} 39 | \subsection{Delta Method} 40 | 41 | To find the asymptotic CI if the estimator is a function of the mean. Goal is to find an expression that converges a function of the mean using the CLT. Let $Z_n$ be a sequence of r.v. $\sqrt(n) (Z_n-\theta) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\sigma^2)$ and let $g: R\longrightarrow R$ be continuously differentiable at $\theta$, then: 42 | \begin{align*} 43 | &\sqrt{n}(g(Z_n) - g(\theta)) \xrightarrow [n \to \infty ]{(d)}\\ 44 | &\mathcal{N}(0, g'(\theta )^2 \sigma ^2) 45 | \end{align*} 46 | \textbf{Example:} let $X_1,... ,X_n ~ exp(\lambda)$ where $\lambda>0$ . Let $\overline{X}_ n= \frac{1}{n} \sum _{i = 1}^ n X_ i$ denote the sample mean. By the CLT, we know that $\sqrt{n}\left(\overline{X}_ n - \frac{1}{\lambda }\right) \xrightarrow [n \to \infty ]{(d)} N(0, \sigma ^2)$ for some value of $\sigma^2$ that depends on $\lambda$. 47 | 48 | If we set $g: \displaystyle \mathbb {R} \to \mathbb {R}$ and $\displaystyle x \mapsto 1/x,$ then by the Delta method: 49 | 50 | \begin{align*} 51 | &\sqrt{n}\left( g(\overline{X}_ n) - g\left(\frac{1}{\lambda }\right) \right)\\ 52 | &\xrightarrow [n \to \infty ]{(d)} N(0, g'(E[X])^2\textsf{Var}{X})\\ 53 | &\xrightarrow [n \to \infty ]{(d)} N(0, g'\left(\frac{1}{\lambda }\right)^2\frac{1}{\lambda ^2})\\ 54 | &\xrightarrow [n \to \infty ]{(d)} N(0, \lambda^2) 55 | \end{align*} -------------------------------------------------------------------------------- /content/Covariance_Matrix.tex: -------------------------------------------------------------------------------- 1 | \section{Covariance Matrix} 2 | Let $X$ be a random vector of dimension $d \times 1$ with expectation $\mu _{X}$. 3 | 4 | Matrix outer products!\\ 5 | 6 | $\Sigma =\mathbb E[(X- \mu _{X})(X- \mu _{X})^ T] \\= \mathbb {E}[XX^ T] - \mathbb {E}[X]\mathbb {E}[X]^ T 7 | \\= \mathbb {E}[XX^ T] - \mu _{X}\mu _{X}^ T$ -------------------------------------------------------------------------------- /content/Generalized_Linear_Models.tex: -------------------------------------------------------------------------------- 1 | 2 | \section{Generalized Linear Models} 3 | We relax the assumption that $\mu$ is linear. Instead, we assume that g $\circ \mu$ is linear, for some function $g$:\\ 4 | 5 | $g(\mu (\mathbf x)) = \mathbf x^ T \beta$ 6 | 7 | The function $g$ is assumed to be known, and is referred to as the link function. It maps the domain of the dependent variable to the entire real Line. 8 | 9 | it has to be strictly increasing, 10 | 11 | it has to be continuously differentiable and 12 | 13 | its range is all of $\mathbb{R}$ 14 | 15 | 16 | \subsection{The Exponential Family} 17 | 18 | A family of distribution $\, \{ \mathbf{P}_{{\boldsymbol \theta }}: {\boldsymbol \theta }\in \Theta \} ,\,$ where the parameter space $\Theta \subset \mathbb {R}^ k\,$ is -$k$ dimensional, is called a $k$-parameter exponential family on $\mathbb{R}^1$ if the pmf or pdf $\, f_{\boldsymbol \theta }:\mathbb {R}^ q\to \mathbb {R}\,$ of $\, \mathbf{P}_{{\boldsymbol \theta }}\,$ can be written in the form:\\ 19 | 20 | $\displaystyle \displaystyle f_{\boldsymbol \theta }(\mathbf{y})=h(\mathbf{y})\, \exp \left({\boldsymbol \eta }({\boldsymbol \theta })\cdot \mathbf{T}(\mathbf{y})-B({\boldsymbol \theta })\right)\qquad \text {where } \\ \begin{cases} {\boldsymbol \eta }({\boldsymbol \theta })=\begin{pmatrix} \eta _1({\boldsymbol \theta })\\ \vdots \\ \eta _ k({\boldsymbol \theta })\end{pmatrix}& :\mathbb {R}^ k\to \mathbb {R}^ k\\ \mathbf{T}(\mathbf{y})=\begin{pmatrix} T_1(\mathbf{y})\\ \vdots \\ T_ k(\mathbf{y})\end{pmatrix}& :\mathbb {R}^ q\to \mathbb {R}^ k\\ B({\boldsymbol \theta })& :\mathbb {R}^ k\to \mathbb {R}\\ h(\mathbf{y})& :\mathbb {R}^ q\to \mathbb {R}.\\ \end{cases}$\\ 21 | 22 | 23 | if $k=1$ it reduces to:\\ 24 | 25 | $\displaystyle \displaystyle f_\theta (y)=h(y)\, \exp \left(\eta (\theta ) T(y)-B(\theta )\right)$ 26 | 27 | -------------------------------------------------------------------------------- /content/Hypothesis_tests.tex: -------------------------------------------------------------------------------- 1 | \section{Asymptotic Hypothesis tests} 2 | 3 | Two hypotheses ($\Theta_0$ disjoint set from $\Theta_1$): 4 | $\begin{cases} 5 | H_0: \theta \epsilon \Theta_0\\ 6 | H_1: \theta \epsilon \Theta_1\\ 7 | \end{cases}$. Goal is to reject $H_0$ using a test statistic.\\ 8 | 9 | A test $\psi$ has \textbf{level $\alpha$} if $\alpha_{\psi}(\theta) \leq \alpha, \forall \theta \in \Theta_0.$ and \textbf{asymptotic level $\alpha$} if $\lim _{n \to \infty } P_\theta ( \psi = 1) \leq \alpha$.\\ 10 | 11 | \textbf{A hypothesis-test} has the form 12 | 13 | \begin{align*} 14 | \psi = \textbf{1} \lbrace T_n \geq c \rbrace 15 | \end{align*} 16 | for some test statistic $T_n$ and threshold $c \in \mathbb{R}$. Threshold $c$ is usually $q_{\alpha/2}$\\ 17 | \textbf{Rejection region:} 18 | \begin{align*} 19 | R_{\psi} = \lbrace T_n > c \rbrace 20 | \end{align*} 21 | \textbf{Symmetric about zero and acceptance Region interval:} 22 | \begin{align*} 23 | \psi = \mathbf{1}\{ |T_n| - c > 0 \} . 24 | \end{align*} 25 | 26 | \textbf{Power of the test}:\\ 27 | \begin{align*} 28 | \pi_\psi = \inf_{\theta \in \Theta_1}(1 - \beta_\psi(\theta)) 29 | \end{align*} 30 | Where $\beta_\psi$ is the probability of making a Type2 Error and $inf$ is the maximum.\\ 31 | \textbf{Two-sided test}: 32 | \begin{align*} 33 | H_1&: \theta \neq \Theta_0\\ 34 | \mathbf{1}&(|T_ n| > q_{\alpha /2}) 35 | \end{align*} 36 | \textbf{One-sided tests}: 37 | \begin{align*} 38 | H_1&: \theta > \Theta_0\\ 39 | \mathbf{1}&(T_ n < -q_{\alpha}) 40 | H_1&: \theta < \Theta_0\\ 41 | \mathbf{1}&(T_ n > q_{\alpha}) 42 | \end{align*} 43 | \textbf{Type1 Error:}\\ 44 | Test rejects null hypothesis $\psi = 1$ but it is actually true $H_0 = TRUE$ also known as the level of a test.\\ 45 | \textbf{Type2 Error:}\\ 46 | Test does not reject null hypothesis $\psi = 0$ but alternative hypothesis is true $H_1 = TRUE$\\ 47 | \textbf{Example:} Let $X_1, \ldots , X_ n \stackrel{i.i.d.}{\sim } \text {Ber}(p^*)$. Question: is $p^* = 1/2$.\\ 48 | $H_0: p^* = 1/2; H_1:p^* \neq 1/2$\\ 49 | If asymptotic level $\alpha$ then we need to standardize the estimated parameter $\hat{p} = \overline{X}_ n$ first.\\ 50 | \begin{align*} 51 | T_n &= \sqrt{n}\frac{\left| \overline{X}_ n - 0.5\right|}{\sqrt{0.5(1 - 0.5)}}\\ 52 | \displaystyle \psi _{n} &= \displaystyle \mathbf{1}\left(T_n>q_{\alpha /2} \right) 53 | \end{align*} 54 | where $q_{\alpha /2}$ denotes the $q_{\alpha /2}$ quantile of a standard Gaussian, and $\alpha$ is determined by the required level of $\psi$. Note the absolute value in $T_n$ for this two sided test.\\ 55 | \textbf{Pivot:}\\ 56 | Let $T_n$ be a function of the random samples $X_1,\dots ,X_ n,\theta$. Let $g(T_ n)$ be a random variable whose distribution is the same for all $\theta$ . Then, $g$ is called a pivotal quantity or a pivot.\\ 57 | \textbf{Example:} let $X$ be a random variable with mean $\mu$ and variance $\sigma^2$ . Let $X_1,\dots ,X_ n$ be iid samples of $X$. Then, 58 | \[\displaystyle g_ n \triangleq \frac{\overline{X_ n} - \mu }{\sigma }\] 59 | is a pivot with $\theta = \left[\mu ~ ~ \sigma ^2\right]^ T$ being the parameter vector (not the same set of paramaters that we use to define a statistical model). 60 | \subsection{P-Value} 61 | The (asymptotic) p-value of a test $\psi_{\alpha}$ is the smallest (asymptotic) level $\alpha$ at which $\psi_{\alpha}$ rejects $H_0$. It is random since it depends on the sample. It can also interpreted as the probability that the test-statistic $T_n$ is realized given the null hypothesis.\\ 62 | 63 | If $pvalue \leq \alpha$ , $H_0$ is rejected by $\psi_{\alpha}$ at the (asymptotic) level $\alpha$\\ 64 | 65 | The smaller the p-value, the more confidently one can reject $H_0$.\\ 66 | \textbf{Left-tailed p-values:} 67 | \begin{align*} 68 | pvalue&=\mathbb{P}(X\leq x|H_0)\\ 69 | &=\mathbf{P}( Z < T_{n,\theta _0}(\overline{X}_ n)))\\ 70 | &=\Phi (T_{n,\theta _0}(\overline{X}_ n))\\ 71 | &Z\sim \mathcal{N}(0,1) 72 | \end{align*} 73 | \textbf{Right-tailed p-values:} 74 | \begin{align*} 75 | pvalue&=\mathbb{P}(X\geq x|H_0) 76 | \end{align*} 77 | \textbf{Two-sided p-values:} 78 | If asymptotic, create normalized $T_n$ using parameters from $H_0$. Then use $T_n$ to get to probabilities. 79 | \begin{align*} 80 | &pvalue=2 min\{\mathbb{P}(X\leq x|H_0),\mathbb{P}(X\geq x|H_0)\}\\ 81 | &\mathbb{P}(\lvert Z\rvert > \lvert T_{n,\theta _0}(\overline{X}_ n)\rvert = 2(1-\Phi(T_n))\\ 82 | &Z \sim N(0,1) 83 | \end{align*} 84 | \subsection{Comparisons of two proportions} 85 | 86 | Let $X_1,\dots ,X_ n \stackrel{iid}{\sim} Bern(p_x)$ and $Y_1,\dots ,Y_ n \stackrel{iid}{\sim} Bern(p_y)$ and be $X$ independent of $Y$. $\hat{p}_x= 1/n \sum_{i=1}^{n} X_i$ and $\hat{p}_x= 1/n \sum_{i=1}^{n} Y_i$\\ 87 | 88 | $H_0: p_x = p_y; H_1: p_x \neq p_y$ 89 | 90 | To get the asymptotic Variance use multivariate Delta-method. Consider $\hat{p}_x - \hat{p}_y = g(\hat{p}_x,\hat{p}_y); g(x,y)= x -y$, then 91 | 92 | $\sqrt(n) (g(\hat{p}_x,\hat{p}_y) - g(p_x-p_y)) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\nabla g(p_x-p_y)^T \Sigma \nabla g(p_x-p_y))$\\ 93 | 94 | $\Rightarrow N(0,p_x(1-px) + p_y(1-py))$ 95 | 96 | \section{Non-asymptotic Hypothesis tests} 97 | 98 | \subsection{Chi squared} 99 | The $\chi _ d^2$ distribution with $d$ degrees of freedom is given by the distribution of $Z_1^2 + Z_2^2 + \cdots + Z_ d^2,$ where $Z_1, \ldots , Z_ d \stackrel{iid}{\sim } \mathcal{N}(0,1)$ 100 | 101 | If $V \sim \chi^2_k:$\\ 102 | 103 | $\mathbb{E}= \mathbb{E}[Z_1^2] + \mathbb{E}[Z_2^2] + \ldots + \mathbb{E}[Z_d^2] = d$\\ 104 | 105 | $Var(V) = Var(Z_1^2) + Var(Z_2^2) + \ldots + Var(Z_d^2) = 2d$ 106 | 107 | \textbf{Cochranes Theorem:}\\ 108 | If $X_1, ..., X_n \stackrel{iid}{\sim} N(\mu,\sigma^2)$, then sample mean $\bar{X}_n$ and the sample variance $S_n$ are independent. The sum of squares of $n$ variables follows a chi squared distribution with (n-1) degrees of freedom: 109 | \begin{align*} 110 | \frac{n S_ n}{\sigma ^2} \sim \chi _{n -1}^2 111 | \end{align*} 112 | If formula for unbiased sample variance is used:\\ 113 | \begin{align*} 114 | \frac{(n-1) S_ n}{\sigma ^2} \sim \chi _{n -1}^2 115 | \end{align*} 116 | \subsection{Student's T Test} 117 | Non-asymptotic hypothesis test for small samples (works on large samples too), data must be gaussian.\\ 118 | 119 | \textbf{Student's T distribution} with $d$ degrees of freedom: 120 | $t_d := \frac{Z}{\sqrt{V/n}}$ where $Z \sim \mathcal{N}(0,1)$ and $V \sim \chi^2_k$ are independent.\\ 121 | 122 | \textbf{Student's T test (one sample + two-sided):}\\ 123 | 124 | Let $X_1, ..., X_n \stackrel{iid}{\sim} N(\mu,\sigma^2)$ and suppose we want to test $H_0: \mu = \mu_0 = 0$ vs. $H_1: \mu \neq 0$.\\ 125 | 126 | Test statistic follows Student's T distribution: 127 | 128 | \begin{align*} 129 | T_n &= \frac{Z}{\tilde{S}} \\ 130 | &= \frac{\bar{X}-\mu}{\frac{\hat{\sigma}}{\sqrt{n}}}\\ 131 | &=\frac{\displaystyle \sqrt{n}\frac{\bar{X}_n - \mu_0}{\sigma}}{\displaystyle \sqrt{\frac{\tilde{S}_n}{\sigma^2}}}\\ 132 | &\sim \frac{N(0,1)}{\sqrt{\frac{\chi^2_{n-1}}{n-1}}}\\ 133 | &\sim t_{n-1} 134 | \end{align*} 135 | 136 | Works bc. under $H_0$ the numerator $N(0,1)$ and the denominator $\frac{\tilde{S}_n}{\sigma^2} \sim \frac1{n-1}\chi^2_{n-1}$ are independent by Cochran's Theorem.\\ 137 | 138 | Student's T test at level $\alpha$: 139 | \begin{align*} 140 | \psi_\alpha = \textbf{1}\{|T_n| > q_{\alpha/2}(t_{n-1})\} 141 | \end{align*} 142 | 143 | \textbf{Student's T test (one sample, one-sided):} 144 | \begin{align*} 145 | \psi_\alpha = \textbf{1}\{T_n > q_\alpha(t_{n-1})\} 146 | \end{align*} 147 | 148 | \textbf{Student's T test (two samples, two-sided):} 149 | 150 | Let $X_1, ..., X_n \stackrel{iid}{\sim} N(\mu_X,\sigma^2_X)$ and $Y_1, ..., Y_n \stackrel{iid}{\sim} N(\mu_Y,\sigma^2_Y)$, suppose we want to test $H_0: \mu_X = \mu_Y$ vs $H_1: \mu_X \neq \mu_Y$. 151 | 152 | \begin{align*} 153 | T_{n, m} = \frac{\bar{X}_n - \bar{Y}_m}{\displaystyle \sqrt{\frac{\hat{\sigma^2}_X}n + \frac{\hat{\sigma^2}_Y}m}} 154 | \end{align*} 155 | 156 | 157 | \textbf{Welch-Satterthwaite formula:}\\ 158 | 159 | When samples are different sizes we need to finde the Student's T distribution of: $T_{n, m} \sim t_N$\\ 160 | 161 | Calculate the degrees of freedom for $t_N$ with: 162 | \begin{align*} 163 | N = \frac{\displaystyle \left(\frac{\hat{\sigma^2}_X}n + \frac{\hat{\sigma^2}_Y}m\right)^2}{\displaystyle \frac{\hat{\sigma^2}^2_X}{n^2(n-1)} + \frac{\hat{\sigma^2}^2_Y}{m^2(m-1)}} \geq \min(n, m) 164 | \end{align*} 165 | 166 | $N$ should be rounded down. 167 | 168 | \subsection{Walds Test} 169 | 170 | Squared distance of $\widehat{\theta}_ n^{MLE}$ to true $\theta_0$ using the fisher information $I(\widehat{\theta}_ n^{MLE})$ as metric. 171 | 172 | Let $\, X_1, \ldots , X_ n \stackrel{iid}{\sim } \mathbf{P}_{\theta ^*}$ for some true parameter $\theta ^* \in \mathbb {R}^ d$ and the maximum likelihood estimator $\widehat{\theta }_ n^{MLE}$ for $\theta ^*$.\\ 173 | 174 | Test $H_0: \displaystyle \theta ^* = \mathbf{0}$ vs $H_1: \displaystyle \theta ^* \neq \mathbf{0}$\\ 175 | 176 | Under $H_0$, the asymptotic normality of the MLE $\widehat{\theta }_ n^{MLE}\,$ implies that:\\ 177 | 178 | $\left\lVert \sqrt{n}\, \mathcal{I}(\mathbf{0})^{1/2}(\widehat{\theta }_ n^{MLE}- \mathbf{0}) \right\rVert ^2 \xrightarrow [n\to \infty ]{(d)} \chi ^2_ d\,$\\ 179 | 180 | \textbf{Test statistic:} 181 | \begin{align*} 182 | T_n = & 183 | n(\widehat{\theta}_ n^{MLE} - \theta_0)^\top I(\widehat{\theta}_ n^{MLE}) (\widehat{\theta}_ n^{MLE} - \theta_0)\\ 184 | &\xrightarrow [n\to \infty ]{(d)} \chi ^2_ d 185 | \end{align*} 186 | 187 | \textbf{Wald test} of level $\alpha$: 188 | \begin{align*} 189 | \psi_\alpha = \mathbf{1}\{T_n > q_\alpha(\chi^2_d)\} 190 | \end{align*} 191 | 192 | %\textbf{Wald test with one dimensional model} 193 | %\begin{align*} 194 | %W&=\frac{(\widehat{\theta}_ n^{MLE} - \theta_0)^2}{Var(\widehat{\theta})}\\ 195 | %&=nI(\widehat{\theta) (\widehat{\theta }^{\text {MLE}} -\theta _0)^2 196 | %\end{align*} 197 | 198 | \subsection{Likelihood Ratio Test} 199 | Parameter space $\Theta \subseteq \mathbb{R}^d$ and $H_0$ is that parameters $\theta_{r+1}$ through $\theta_d$ have values $\theta_c^{r+1}$ through $\theta^c_d$ leaving the other $r$ unspecified. That is:\\ 200 | $H_0: (\theta_{r+1}, ..., \theta_d)^T = \theta_{r+1...d} = \theta_0$\\ 201 | 202 | \textbf{Construct two estimators:} 203 | \begin{align*} 204 | \widehat{\theta}_n^{MLE} = argmax_{\theta \in \Theta}(\ell_n(\theta))\\ 205 | \widehat{\theta}_n^c = argmax_{\theta \in \Theta_0}(\ell_n(\theta)) 206 | \end{align*} 207 | 208 | \textbf{Test statistic:} 209 | \begin{align*} 210 | T_n = 2 ( \ell(X_1,..X_n|\widehat{\theta}_n^{MLE}) - \ell(X_1,..X_n|\widehat{\theta}_n^c))) 211 | \end{align*} 212 | \textbf{Wilk's Theorem:} under $H_0$, if the MLE conditions are satisfied: 213 | \begin{align*} 214 | T_n& \xrightarrow[n \rightarrow \infty]{(d)} \chi_{d-r}^2\\ 215 | \end{align*} 216 | \textbf{Likelihood ratio test} at level $\alpha$: 217 | \begin{align*} 218 | \psi_\alpha = \textbf{1}\{T_n > q_\alpha(\chi^2_{d-r})\} 219 | \end{align*} 220 | \subsection{Implicit Testing} 221 | Todo 222 | \subsection{Goodness of Fit Discrete Distributions} 223 | 224 | Let $X_1,...,X_n$ be iid samples from a categorical distribution. Test $H_0: p = p^0$ against $H_1: p \neq p^0$. Example: against the uniform distribution $p^0 = (1/K, \ldots, 1/K)^\top$.\\ 225 | 226 | \textbf{Test statistic} under $H_0$: 227 | \begin{align*} 228 | T_n = n\sum_{k=1}^K\frac{(\hat{p}_k - p^0_k)^2}{p^0_k} \xrightarrow[n \rightarrow \infty]{(d)} \chi^2_{K-1} 229 | \end{align*} 230 | \textbf{Test at level alpha:} 231 | \begin{align*} 232 | \psi_\alpha = \mathbb{1}\{T_n > q_\alpha(\chi^2_{K-1})\} 233 | \end{align*} 234 | \subsection{Kolmogorov-Smirnov test} 235 | \subsection{Kolmogorov-Lilliefors test} 236 | \subsection{QQ plots} 237 | \textbf{Heavier tails}: below > above the diagonal.\\ 238 | \textbf{Lighter tails}: above > below the diagonal.\\ 239 | \textbf{Right-skewed}: above > below > above the diagonal.\\ 240 | \textbf{Left-skewed}: below > above > below the diagonal.\\ 241 | -------------------------------------------------------------------------------- /content/Law_large_Numbers.tex: -------------------------------------------------------------------------------- 1 | \section{LLN and CLT} 2 | Let $X_1, ..., X_n \stackrel{iid}{\sim} P_{\mu}$, where $E(X_i)=\mu$ and $Var(X_i)=\sigma^2$ for all $i=1,2,...,n$ and $\bar{X_n}= \frac{1}{n} \sum_{i=1}^{n} X_i$.\\ 3 | \textbf{Law of large numbers:} 4 | \begin{align*} 5 | \bar{X_n}& \xrightarrow[n \rightarrow \infty]{P, a.s.} \mu\\ 6 | \frac{1}{n} \sum_{i=1}^{n} g(X_i)& \xrightarrow[n \rightarrow \infty]{P, a.s.} \mathbb{E}[g(X)] 7 | \end{align*} 8 | \textbf{Central Limit Theorem for Mean:} 9 | \begin{align*} 10 | \sqrt(n)\frac{\bar{X_n}-\mu}{\sqrt(\sigma^2)}& \xrightarrow[n \rightarrow \infty]{(d)} N(0,1)\\ 11 | \sqrt(n)(\bar{X_n}-\mu)& \xrightarrow[n \rightarrow \infty]{(d)} N(0,\sigma^2) 12 | \end{align*} 13 | 14 | \textbf{Central Limit Theorem for Sums:} 15 | 16 | \begin{align*} 17 | \sum{X}_{i=1}^{n} & \xrightarrow[n \rightarrow \infty]{(d)} N(n \mu, \sqrt(n)\sqrt(\sigma^2))\\ 18 | \frac{\sum{X}_{i=1}^{n} - n\mu}{\sqrt(n) \sqrt(\sigma^2)} & \xrightarrow[n \rightarrow \infty]{(d)} N(0, 1)\\ 19 | \end{align*} 20 | 21 | \textbf{Variance of the Mean:} 22 | \begin{align*} 23 | Var(\overline{X_n})&=(\frac{\sigma^2}{n})^2 Var(X_1 + X_2,...,X_n)\\ 24 | & =\frac{\sigma^2}{n} 25 | \end{align*} 26 | \textbf{Expectation of the mean:} 27 | \begin{align*} 28 | E[\bar{X_n}] & =\frac{1}{n}E[X_1 + X_2,...,X_n]\\ 29 | &=\mu. 30 | \end{align*} -------------------------------------------------------------------------------- /content/Likelihood.log: -------------------------------------------------------------------------------- 1 | This is pdfTeX, Version 3.14159265-2.6-1.40.19 (MiKTeX 2.9.6930 64-bit) (preloaded format=pdflatex 2019.1.30) 18 MAR 2019 19:01 2 | entering extended mode 3 | **./Likelihood.tex 4 | (Likelihood.tex 5 | LaTeX2e <2018-12-01> 6 | ! Undefined control sequence. 7 | l.1 \section 8 | {Likelihood} 9 | The control sequence at the end of the top line 10 | of your error message was never \def'ed. If you have 11 | misspelled it (e.g., `\hobx'), type `I' and the correct 12 | spelling (e.g., `I\hbox'). Otherwise just continue, 13 | and I'll forget about whatever was undefined. 14 | 15 | 16 | ! LaTeX Error: Missing \begin{document}. 17 | 18 | See the LaTeX manual or LaTeX Companion for explanation. 19 | Type H for immediate help. 20 | ... 21 | 22 | l.1 \section{L 23 | ikelihood} 24 | You're in trouble here. Try typing to proceed. 25 | If that doesn't work, type X to quit. 26 | 27 | Missing character: There is no L in font nullfont! 28 | Missing character: There is no i in font nullfont! 29 | Missing character: There is no k in font nullfont! 30 | Missing character: There is no e in font nullfont! 31 | Missing character: There is no l in font nullfont! 32 | Missing character: There is no i in font nullfont! 33 | Missing character: There is no h in font nullfont! 34 | Missing character: There is no o in font nullfont! 35 | Missing character: There is no o in font nullfont! 36 | Missing character: There is no d in font nullfont! 37 | 38 | Overfull \hbox (20.0pt too wide) in paragraph at lines 1--2 39 | [] 40 | [] 41 | 42 | 43 | ! LaTeX Error: Missing \begin{document}. 44 | 45 | See the LaTeX manual or LaTeX Companion for explanation. 46 | Type H for immediate help. 47 | ... 48 | 49 | l.3 L 50 | et $(E, \{ P_\theta \} _{\theta \in \Theta })$ denote a discrete or con... 51 | 52 | You're in trouble here. Try typing to proceed. 53 | If that doesn't work, type X to quit. 54 | 55 | Missing character: There is no L in font nullfont! 56 | Missing character: There is no e in font nullfont! 57 | Missing character: There is no t in font nullfont! 58 | LaTeX Font Info: External font `cmex10' loaded for size 59 | (Font) <7> on input line 3. 60 | LaTeX Font Info: External font `cmex10' loaded for size 61 | (Font) <5> on input line 3. 62 | Missing character: There is no d in font nullfont! 63 | Missing character: There is no e in font nullfont! 64 | Missing character: There is no n in font nullfont! 65 | Missing character: There is no o in font nullfont! 66 | Missing character: There is no t in font nullfont! 67 | Missing character: There is no e in font nullfont! 68 | Missing character: There is no a in font nullfont! 69 | Missing character: There is no d in font nullfont! 70 | Missing character: There is no i in font nullfont! 71 | Missing character: There is no s in font nullfont! 72 | Missing character: There is no c in font nullfont! 73 | Missing character: There is no r in font nullfont! 74 | Missing character: There is no e in font nullfont! 75 | Missing character: There is no t in font nullfont! 76 | Missing character: There is no e in font nullfont! 77 | Missing character: There is no o in font nullfont! 78 | Missing character: There is no r in font nullfont! 79 | Missing character: There is no c in font nullfont! 80 | Missing character: There is no o in font nullfont! 81 | Missing character: There is no n in font nullfont! 82 | Missing character: There is no t in font nullfont! 83 | Missing character: There is no i in font nullfont! 84 | Missing character: There is no n in font nullfont! 85 | Missing character: There is no u in font nullfont! 86 | Missing character: There is no o in font nullfont! 87 | Missing character: There is no u in font nullfont! 88 | Missing character: There is no s in font nullfont! 89 | Missing character: There is no s in font nullfont! 90 | Missing character: There is no t in font nullfont! 91 | Missing character: There is no a in font nullfont! 92 | Missing character: There is no t in font nullfont! 93 | Missing character: There is no i in font nullfont! 94 | Missing character: There is no s in font nullfont! 95 | Missing character: There is no t in font nullfont! 96 | Missing character: There is no i in font nullfont! 97 | Missing character: There is no c in font nullfont! 98 | Missing character: There is no a in font nullfont! 99 | Missing character: There is no l in font nullfont! 100 | Missing character: There is no m in font nullfont! 101 | Missing character: There is no o in font nullfont! 102 | Missing character: There is no d in font nullfont! 103 | Missing character: There is no e in font nullfont! 104 | Missing character: There is no l in font nullfont! 105 | Missing character: There is no . in font nullfont! 106 | Missing character: There is no L in font nullfont! 107 | Missing character: There is no e in font nullfont! 108 | Missing character: There is no t in font nullfont! 109 | Missing character: There is no d in font nullfont! 110 | Missing character: There is no e in font nullfont! 111 | Missing character: There is no n in font nullfont! 112 | Missing character: There is no o in font nullfont! 113 | Missing character: There is no t in font nullfont! 114 | Missing character: There is no e in font nullfont! 115 | Missing character: There is no t in font nullfont! 116 | Missing character: There is no h in font nullfont! 117 | Missing character: There is no e in font nullfont! 118 | Missing character: There is no p in font nullfont! 119 | Missing character: There is no m in font nullfont! 120 | Missing character: There is no f in font nullfont! 121 | Missing character: There is no o in font nullfont! 122 | Missing character: There is no r in font nullfont! 123 | Missing character: There is no p in font nullfont! 124 | Missing character: There is no d in font nullfont! 125 | Missing character: There is no f in font nullfont! 126 | Missing character: There is no o in font nullfont! 127 | Missing character: There is no f in font nullfont! 128 | Missing character: There is no . in font nullfont! 129 | Missing character: There is no L in font nullfont! 130 | Missing character: There is no e in font nullfont! 131 | Missing character: There is no t in font nullfont! 132 | Missing character: There is no w in font nullfont! 133 | Missing character: There is no h in font nullfont! 134 | Missing character: There is no e in font nullfont! 135 | Missing character: There is no r in font nullfont! 136 | Missing character: There is no e in font nullfont! 137 | Missing character: There is no t in font nullfont! 138 | Missing character: There is no h in font nullfont! 139 | Missing character: There is no e in font nullfont! 140 | Missing character: There is no p in font nullfont! 141 | Missing character: There is no a in font nullfont! 142 | Missing character: There is no r in font nullfont! 143 | Missing character: There is no a in font nullfont! 144 | Missing character: There is no m in font nullfont! 145 | Missing character: There is no e in font nullfont! 146 | Missing character: There is no t in font nullfont! 147 | Missing character: There is no e in font nullfont! 148 | Missing character: There is no r in font nullfont! 149 | Missing character: There is no i in font nullfont! 150 | Missing character: There is no s in font nullfont! 151 | Missing character: There is no u in font nullfont! 152 | Missing character: There is no n in font nullfont! 153 | Missing character: There is no k in font nullfont! 154 | Missing character: There is no n in font nullfont! 155 | Missing character: There is no o in font nullfont! 156 | Missing character: There is no w in font nullfont! 157 | Missing character: There is no n in font nullfont! 158 | Missing character: There is no . in font nullfont! 159 | Missing character: There is no T in font nullfont! 160 | Missing character: There is no h in font nullfont! 161 | Missing character: There is no e in font nullfont! 162 | Missing character: There is no n in font nullfont! 163 | Missing character: There is no t in font nullfont! 164 | Missing character: There is no h in font nullfont! 165 | Missing character: There is no e in font nullfont! 166 | Missing character: There is no l in font nullfont! 167 | Missing character: There is no i in font nullfont! 168 | Missing character: There is no k in font nullfont! 169 | Missing character: There is no e in font nullfont! 170 | Missing character: There is no l in font nullfont! 171 | Missing character: There is no i in font nullfont! 172 | Missing character: There is no h in font nullfont! 173 | Missing character: There is no o in font nullfont! 174 | Missing character: There is no o in font nullfont! 175 | Missing character: There is no d in font nullfont! 176 | Missing character: There is no i in font nullfont! 177 | Missing character: There is no s in font nullfont! 178 | Missing character: There is no t in font nullfont! 179 | Missing character: There is no h in font nullfont! 180 | Missing character: There is no e in font nullfont! 181 | Missing character: There is no f in font nullfont! 182 | Missing character: There is no u in font nullfont! 183 | Missing character: There is no n in font nullfont! 184 | Missing character: There is no c in font nullfont! 185 | Missing character: There is no t in font nullfont! 186 | Missing character: There is no i in font nullfont! 187 | Missing character: There is no o in font nullfont! 188 | Missing character: There is no n in font nullfont! 189 | 190 | Overfull \hbox (20.0pt too wide) in paragraph at lines 3--4 191 | [] 192 | [] 193 | 194 | 195 | Overfull \hbox (57.17706pt too wide) in paragraph at lines 3--4 196 | \OT1/cmr/m/n/10 (\OML/cmm/m/it/10 E; \OMS/cmsy/m/n/10 f\OML/cmm/m/it/10 P[]\OMS 197 | /cmsy/m/n/10 g[]\OT1/cmr/m/n/10 )$ 198 | [] 199 | 200 | 201 | Overfull \hbox (9.55553pt too wide) in paragraph at lines 3--4 202 | \OML/cmm/m/it/10 p[]$ 203 | [] 204 | 205 | 206 | Overfull \hbox (10.94441pt too wide) in paragraph at lines 3--4 207 | \OML/cmm/m/it/10 P[]$ 208 | [] 209 | 210 | 211 | Overfull \hbox (61.32013pt too wide) in paragraph at lines 3--4 212 | \OML/cmm/m/it/10 X[]; [] ; X[] [] 213 | [] 214 | 215 | 216 | Overfull \hbox (15.125pt too wide) in paragraph at lines 3--4 217 | \OML/cmm/m/it/10 P[]$ 218 | [] 219 | 220 | 221 | Overfull \hbox (9.56946pt too wide) in paragraph at lines 3--4 222 | \OML/cmm/m/it/10 []$ 223 | [] 224 | 225 | 226 | ! LaTeX Error: Missing \begin{document}. 227 | 228 | See the LaTeX manual or LaTeX Companion for explanation. 229 | Type H for immediate help. 230 | ... 231 | 232 | l.5 $ 233 | \displaystyle L_ n: E^ n \times \Theta$\\ 234 | You're in trouble here. Try typing to proceed. 235 | If that doesn't work, type X to quit. 236 | 237 | 238 | Overfull \hbox (37.8044pt too wide) in paragraph at lines 5--8 239 | []$\OML/cmm/m/it/10 L[] \OT1/cmr/m/n/10 : 240 | [] 241 | 242 | 243 | Overfull \hbox (21.17944pt too wide) in paragraph at lines 5--8 244 | \OML/cmm/m/it/10 E[] \OMS/cmsy/m/n/10  245 | [] 246 | 247 | 248 | Overfull \hbox (7.7778pt too wide) in paragraph at lines 5--8 249 | \OT1/cmr/m/n/10 $ 250 | [] 251 | 252 | 253 | Overfull \hbox (60.7765pt too wide) in paragraph at lines 5--8 254 | \OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[]; [] ; x[]; \OT1/cmr/m/n/10 )$ 255 | [] 256 | 257 | 258 | Overfull \hbox (83.58092pt too wide) in paragraph at lines 5--8 259 | \OML/cmm/m/it/10 L[]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[]; [] ; x[]; \OT1/cmr/ 260 | m/n/10 ) = 261 | [] 262 | 263 | 264 | Overfull \hbox (60.45715pt too wide) in paragraph at lines 5--8 265 | [][] \OML/cmm/m/it/10 P[]\OT1/cmr/m/n/10 [\OML/cmm/m/it/10 X[] \OT1/cmr/m/n/10 266 | = 267 | [] 268 | 269 | 270 | Overfull \hbox (11.82234pt too wide) in paragraph at lines 5--8 271 | \OML/cmm/m/it/10 x[]\OT1/cmr/m/n/10 ]$ 272 | [] 273 | 274 | ! Undefined control sequence. 275 | l.9 \subsection 276 | *{Discrete Likelihood} 277 | The control sequence at the end of the top line 278 | of your error message was never \def'ed. If you have 279 | misspelled it (e.g., `\hobx'), type `I' and the correct 280 | spelling (e.g., `I\hbox'). Otherwise just continue, 281 | and I'll forget about whatever was undefined. 282 | 283 | 284 | ! LaTeX Error: Missing \begin{document}. 285 | 286 | See the LaTeX manual or LaTeX Companion for explanation. 287 | Type H for immediate help. 288 | ... 289 | 290 | l.9 \subsection* 291 | {Discrete Likelihood} 292 | You're in trouble here. Try typing to proceed. 293 | If that doesn't work, type X to quit. 294 | 295 | Missing character: There is no * in font nullfont! 296 | Missing character: There is no D in font nullfont! 297 | Missing character: There is no i in font nullfont! 298 | Missing character: There is no s in font nullfont! 299 | Missing character: There is no c in font nullfont! 300 | Missing character: There is no r in font nullfont! 301 | Missing character: There is no e in font nullfont! 302 | Missing character: There is no t in font nullfont! 303 | Missing character: There is no e in font nullfont! 304 | Missing character: There is no L in font nullfont! 305 | Missing character: There is no i in font nullfont! 306 | Missing character: There is no k in font nullfont! 307 | Missing character: There is no e in font nullfont! 308 | Missing character: There is no l in font nullfont! 309 | Missing character: There is no i in font nullfont! 310 | Missing character: There is no h in font nullfont! 311 | Missing character: There is no o in font nullfont! 312 | Missing character: There is no o in font nullfont! 313 | Missing character: There is no d in font nullfont! 314 | 315 | Overfull \hbox (20.0pt too wide) in paragraph at lines 9--10 316 | [] 317 | [] 318 | 319 | 320 | ! LaTeX Error: Missing \begin{document}. 321 | 322 | See the LaTeX manual or LaTeX Companion for explanation. 323 | Type H for immediate help. 324 | ... 325 | 326 | l.11 B 327 | ernoulli likelihood: 328 | You're in trouble here. Try typing to proceed. 329 | If that doesn't work, type X to quit. 330 | 331 | Missing character: There is no B in font nullfont! 332 | Missing character: There is no e in font nullfont! 333 | Missing character: There is no r in font nullfont! 334 | Missing character: There is no n in font nullfont! 335 | Missing character: There is no o in font nullfont! 336 | Missing character: There is no u in font nullfont! 337 | Missing character: There is no l in font nullfont! 338 | Missing character: There is no l in font nullfont! 339 | Missing character: There is no i in font nullfont! 340 | Missing character: There is no l in font nullfont! 341 | Missing character: There is no i in font nullfont! 342 | Missing character: There is no k in font nullfont! 343 | Missing character: There is no e in font nullfont! 344 | Missing character: There is no l in font nullfont! 345 | Missing character: There is no i in font nullfont! 346 | Missing character: There is no h in font nullfont! 347 | Missing character: There is no o in font nullfont! 348 | Missing character: There is no o in font nullfont! 349 | Missing character: There is no d in font nullfont! 350 | Missing character: There is no : in font nullfont! 351 | 352 | Overfull \hbox (20.0pt too wide) in paragraph at lines 11--12 353 | [] 354 | [] 355 | 356 | 357 | ! LaTeX Error: Missing \begin{document}. 358 | 359 | See the LaTeX manual or LaTeX Companion for explanation. 360 | Type H for immediate help. 361 | ... 362 | 363 | l.13 $ 364 | L_ n(x_1, \ldots , x_ n, p) = p^{\sum _{i = 1}^ n x_ i} (1 -p)^{{\colo... 365 | 366 | You're in trouble here. Try typing to proceed. 367 | If that doesn't work, type X to quit. 368 | 369 | ! Undefined control sequence. 370 | l.13 ... p^{\sum _{i = 1}^ n x_ i} (1 -p)^{{\color 371 | {blue}{n - }} \sum _{i = 1... 372 | The control sequence at the end of the top line 373 | of your error message was never \def'ed. If you have 374 | misspelled it (e.g., `\hobx'), type `I' and the correct 375 | spelling (e.g., `I\hbox'). Otherwise just continue, 376 | and I'll forget about whatever was undefined. 377 | 378 | 379 | Overfull \hbox (103.63994pt too wide) in paragraph at lines 13--14 380 | []$\OML/cmm/m/it/10 L[]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[]; [] ; x[]; p\OT1/c 381 | mr/m/n/10 ) = 382 | [] 383 | 384 | 385 | Overfull \hbox (53.53152pt too wide) in paragraph at lines 13--14 386 | \OML/cmm/m/it/10 p[]\OT1/cmr/m/n/10 (1 \OMS/cmsy/m/n/10 387 | [] 388 | 389 | 390 | Overfull \hbox (67.92108pt too wide) in paragraph at lines 13--14 391 | \OML/cmm/m/it/10 p\OT1/cmr/m/n/10 )[]$ 392 | [] 393 | 394 | 395 | ! LaTeX Error: Missing \begin{document}. 396 | 397 | See the LaTeX manual or LaTeX Companion for explanation. 398 | Type H for immediate help. 399 | ... 400 | 401 | l.15 $ 402 | = L_ n = \prod _{i = 1}^ n \left( x_ i p + (1 - x_ i) (1 - p) \right)$ 403 | You're in trouble here. Try typing to proceed. 404 | If that doesn't work, type X to quit. 405 | 406 | 407 | Overfull \hbox (27.7778pt too wide) in paragraph at lines 15--16 408 | []$\OT1/cmr/m/n/10 = 409 | [] 410 | 411 | 412 | Overfull \hbox (22.80441pt too wide) in paragraph at lines 15--16 413 | \OML/cmm/m/it/10 L[] \OT1/cmr/m/n/10 = 414 | [] 415 | 416 | 417 | Overfull \hbox (122.71689pt too wide) in paragraph at lines 15--16 418 | [][] []$ 419 | [] 420 | 421 | 422 | ! LaTeX Error: Missing \begin{document}. 423 | 424 | See the LaTeX manual or LaTeX Companion for explanation. 425 | Type H for immediate help. 426 | ... 427 | 428 | l.17 P 429 | oisson likelihood: 430 | You're in trouble here. Try typing to proceed. 431 | If that doesn't work, type X to quit. 432 | 433 | Missing character: There is no P in font nullfont! 434 | Missing character: There is no o in font nullfont! 435 | Missing character: There is no i in font nullfont! 436 | Missing character: There is no s in font nullfont! 437 | Missing character: There is no s in font nullfont! 438 | Missing character: There is no o in font nullfont! 439 | Missing character: There is no n in font nullfont! 440 | Missing character: There is no l in font nullfont! 441 | Missing character: There is no i in font nullfont! 442 | Missing character: There is no k in font nullfont! 443 | Missing character: There is no e in font nullfont! 444 | Missing character: There is no l in font nullfont! 445 | Missing character: There is no i in font nullfont! 446 | Missing character: There is no h in font nullfont! 447 | Missing character: There is no o in font nullfont! 448 | Missing character: There is no o in font nullfont! 449 | Missing character: There is no d in font nullfont! 450 | Missing character: There is no : in font nullfont! 451 | 452 | Overfull \hbox (20.0pt too wide) in paragraph at lines 17--18 453 | [] 454 | [] 455 | 456 | 457 | ! LaTeX Error: Missing \begin{document}. 458 | 459 | See the LaTeX manual or LaTeX Companion for explanation. 460 | Type H for immediate help. 461 | ... 462 | 463 | l.19 $ 464 | L_ n(x_1, \ldots , x_ n, \lambda) = \prod _{i = 1}^ n \frac{\lambda^{\... 465 | 466 | You're in trouble here. Try typing to proceed. 467 | If that doesn't work, type X to quit. 468 | 469 | 470 | Overfull \hbox (104.44205pt too wide) in paragraph at lines 19--20 471 | []$\OML/cmm/m/it/10 L[]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[]; [] ; x[]; \OT1/c 472 | mr/m/n/10 ) = 473 | [] 474 | 475 | 476 | Overfull \hbox (77.66946pt too wide) in paragraph at lines 19--20 477 | [][] []\OML/cmm/m/it/10 e[]$ 478 | [] 479 | 480 | 481 | ! LaTeX Error: Missing \begin{document}. 482 | 483 | See the LaTeX manual or LaTeX Companion for explanation. 484 | Type H for immediate help. 485 | ... 486 | 487 | l.21 P 488 | oisson loglikelihood:\\ 489 | You're in trouble here. Try typing to proceed. 490 | If that doesn't work, type X to quit. 491 | 492 | Missing character: There is no P in font nullfont! 493 | Missing character: There is no o in font nullfont! 494 | Missing character: There is no i in font nullfont! 495 | Missing character: There is no s in font nullfont! 496 | Missing character: There is no s in font nullfont! 497 | Missing character: There is no o in font nullfont! 498 | Missing character: There is no n in font nullfont! 499 | Missing character: There is no l in font nullfont! 500 | Missing character: There is no o in font nullfont! 501 | Missing character: There is no g in font nullfont! 502 | Missing character: There is no l in font nullfont! 503 | Missing character: There is no i in font nullfont! 504 | Missing character: There is no k in font nullfont! 505 | Missing character: There is no e in font nullfont! 506 | Missing character: There is no l in font nullfont! 507 | Missing character: There is no i in font nullfont! 508 | Missing character: There is no h in font nullfont! 509 | Missing character: There is no o in font nullfont! 510 | Missing character: There is no o in font nullfont! 511 | Missing character: There is no d in font nullfont! 512 | Missing character: There is no : in font nullfont! 513 | 514 | Overfull \hbox (20.0pt too wide) in paragraph at lines 21--23 515 | [] 516 | [] 517 | 518 | 519 | Overfull \hbox (92.71059pt too wide) in paragraph at lines 21--23 520 | \OML/cmm/m/it/10 log\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 L\OT1/cmr/m/n/10 (\OML/cm 521 | m/m/it/10 x[] [] x[]\OT1/cmr/m/n/10 ; \OML/cmm/m/it/10 \OT1/cmr/m/n/10 )) = 522 | [] 523 | 524 | 525 | Overfull \hbox (7.7778pt too wide) in paragraph at lines 21--23 526 | \OMS/cmsy/m/n/10 527 | [] 528 | 529 | 530 | Overfull \hbox (19.61351pt too wide) in paragraph at lines 21--23 531 | \OML/cmm/m/it/10 n \OT1/cmr/m/n/10 + 532 | [] 533 | 534 | 535 | Overfull \hbox (80.93303pt too wide) in paragraph at lines 21--23 536 | \OML/cmm/m/it/10 log\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 \OT1/cmr/m/n/10 )([][] \ 537 | OML/cmm/m/it/10 x[]\OT1/cmr/m/n/10 )) \OMS/cmsy/m/n/10 538 | [] 539 | 540 | 541 | Overfull \hbox (57.32184pt too wide) in paragraph at lines 21--23 542 | \OML/cmm/m/it/10 log\OT1/cmr/m/n/10 ([][] \OML/cmm/m/it/10 x[]\OT1/cmr/m/n/10 ! 543 | )$ 544 | [] 545 | 546 | ! Undefined control sequence. 547 | l.25 \subsection 548 | *{Continuous Likelihood} 549 | The control sequence at the end of the top line 550 | of your error message was never \def'ed. If you have 551 | misspelled it (e.g., `\hobx'), type `I' and the correct 552 | spelling (e.g., `I\hbox'). Otherwise just continue, 553 | and I'll forget about whatever was undefined. 554 | 555 | 556 | ! LaTeX Error: Missing \begin{document}. 557 | 558 | See the LaTeX manual or LaTeX Companion for explanation. 559 | Type H for immediate help. 560 | ... 561 | 562 | l.25 \subsection* 563 | {Continuous Likelihood} 564 | You're in trouble here. Try typing to proceed. 565 | If that doesn't work, type X to quit. 566 | 567 | Missing character: There is no * in font nullfont! 568 | Missing character: There is no C in font nullfont! 569 | Missing character: There is no o in font nullfont! 570 | Missing character: There is no n in font nullfont! 571 | Missing character: There is no t in font nullfont! 572 | Missing character: There is no i in font nullfont! 573 | Missing character: There is no n in font nullfont! 574 | Missing character: There is no u in font nullfont! 575 | Missing character: There is no o in font nullfont! 576 | Missing character: There is no u in font nullfont! 577 | Missing character: There is no s in font nullfont! 578 | Missing character: There is no L in font nullfont! 579 | Missing character: There is no i in font nullfont! 580 | Missing character: There is no k in font nullfont! 581 | Missing character: There is no e in font nullfont! 582 | Missing character: There is no l in font nullfont! 583 | Missing character: There is no i in font nullfont! 584 | Missing character: There is no h in font nullfont! 585 | Missing character: There is no o in font nullfont! 586 | Missing character: There is no o in font nullfont! 587 | Missing character: There is no d in font nullfont! 588 | 589 | Overfull \hbox (20.0pt too wide) in paragraph at lines 25--26 590 | [] 591 | [] 592 | 593 | 594 | ! LaTeX Error: Missing \begin{document}. 595 | 596 | See the LaTeX manual or LaTeX Companion for explanation. 597 | Type H for immediate help. 598 | ... 599 | 600 | l.27 G 601 | aussian likelihood:\\ 602 | You're in trouble here. Try typing to proceed. 603 | If that doesn't work, type X to quit. 604 | 605 | Missing character: There is no G in font nullfont! 606 | Missing character: There is no a in font nullfont! 607 | Missing character: There is no u in font nullfont! 608 | Missing character: There is no s in font nullfont! 609 | Missing character: There is no s in font nullfont! 610 | Missing character: There is no i in font nullfont! 611 | Missing character: There is no a in font nullfont! 612 | Missing character: There is no n in font nullfont! 613 | Missing character: There is no l in font nullfont! 614 | Missing character: There is no i in font nullfont! 615 | Missing character: There is no k in font nullfont! 616 | Missing character: There is no e in font nullfont! 617 | Missing character: There is no l in font nullfont! 618 | Missing character: There is no i in font nullfont! 619 | Missing character: There is no h in font nullfont! 620 | Missing character: There is no o in font nullfont! 621 | Missing character: There is no o in font nullfont! 622 | Missing character: There is no d in font nullfont! 623 | Missing character: There is no : in font nullfont! 624 | 625 | Overfull \hbox (20.0pt too wide) in paragraph at lines 27--28 626 | [] 627 | [] 628 | 629 | 630 | ! LaTeX Error: Missing \begin{document}. 631 | 632 | See the LaTeX manual or LaTeX Companion for explanation. 633 | Type H for immediate help. 634 | ... 635 | 636 | l.29 $ 637 | L(x_1\dots x_n;\mu,\sigma^2)=\\ \dfrac{1}{\left(\sigma\sqrt{2\pi}\righ... 638 | 639 | You're in trouble here. Try typing to proceed. 640 | If that doesn't work, type X to quit. 641 | 642 | ! Undefined control sequence. 643 | \dfrac 644 | 645 | l.29 $L(x_1\dots x_n;\mu,\sigma^2)=\\ \dfrac 646 | {1}{\left(\sigma\sqrt{2\pi}\righ... 647 | The control sequence at the end of the top line 648 | of your error message was never \def'ed. If you have 649 | misspelled it (e.g., `\hobx'), type `I' and the correct 650 | spelling (e.g., `I\hbox'). Otherwise just continue, 651 | and I'll forget about whatever was undefined. 652 | 653 | ! Undefined control sequence. 654 | l.29 ...gma\sqrt{2\pi}\right)^n}\exp{\left(-\dfrac 655 | {1}{2\sigma^2}\sum_{i=1}^n... 656 | The control sequence at the end of the top line 657 | of your error message was never \def'ed. If you have 658 | misspelled it (e.g., `\hobx'), type `I' and the correct 659 | spelling (e.g., `I\hbox'). Otherwise just continue, 660 | and I'll forget about whatever was undefined. 661 | 662 | 663 | Overfull \hbox (106.97209pt too wide) in paragraph at lines 29--30 664 | []$\OML/cmm/m/it/10 L\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[] [] x[]\OT1/cmr/m/n/1 665 | 0 ; \OML/cmm/m/it/10 ; []\OT1/cmr/m/n/10 ) = 666 | [] 667 | 668 | 669 | Overfull \hbox (166.4227pt too wide) in paragraph at lines 29--30 670 | \OT1/cmr/m/n/10 1[] [] []$ 671 | [] 672 | 673 | 674 | ! LaTeX Error: Missing \begin{document}. 675 | 676 | See the LaTeX manual or LaTeX Companion for explanation. 677 | Type H for immediate help. 678 | ... 679 | 680 | l.31 G 681 | aussian loglikelihood:\\ 682 | You're in trouble here. Try typing to proceed. 683 | If that doesn't work, type X to quit. 684 | 685 | Missing character: There is no G in font nullfont! 686 | Missing character: There is no a in font nullfont! 687 | Missing character: There is no u in font nullfont! 688 | Missing character: There is no s in font nullfont! 689 | Missing character: There is no s in font nullfont! 690 | Missing character: There is no i in font nullfont! 691 | Missing character: There is no a in font nullfont! 692 | Missing character: There is no n in font nullfont! 693 | Missing character: There is no l in font nullfont! 694 | Missing character: There is no o in font nullfont! 695 | Missing character: There is no g in font nullfont! 696 | Missing character: There is no l in font nullfont! 697 | Missing character: There is no i in font nullfont! 698 | Missing character: There is no k in font nullfont! 699 | Missing character: There is no e in font nullfont! 700 | Missing character: There is no l in font nullfont! 701 | Missing character: There is no i in font nullfont! 702 | Missing character: There is no h in font nullfont! 703 | Missing character: There is no o in font nullfont! 704 | Missing character: There is no o in font nullfont! 705 | Missing character: There is no d in font nullfont! 706 | Missing character: There is no : in font nullfont! 707 | 708 | Overfull \hbox (20.0pt too wide) in paragraph at lines 31--32 709 | [] 710 | [] 711 | 712 | 713 | ! LaTeX Error: Missing \begin{document}. 714 | 715 | See the LaTeX manual or LaTeX Companion for explanation. 716 | Type H for immediate help. 717 | ... 718 | 719 | l.33 $ 720 | log(L(x_1\dots x_n;\mu,\sigma^2))= \\-n log(\sigma\sqrt{2\pi})-\frac{1... 721 | 722 | You're in trouble here. Try typing to proceed. 723 | If that doesn't work, type X to quit. 724 | 725 | 726 | Overfull \hbox (127.90616pt too wide) in paragraph at lines 33--34 727 | []$\OML/cmm/m/it/10 log\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 L\OT1/cmr/m/n/10 (\OML 728 | /cmm/m/it/10 x[] [] x[]\OT1/cmr/m/n/10 ; \OML/cmm/m/it/10 ; []\OT1/cmr/m/n/10 729 | )) = 730 | [] 731 | 732 | 733 | Overfull \hbox (7.7778pt too wide) in paragraph at lines 33--34 734 | \OMS/cmsy/m/n/10 735 | [] 736 | 737 | 738 | Overfull \hbox (60.17955pt too wide) in paragraph at lines 33--34 739 | \OML/cmm/m/it/10 nlog\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 []\OT1/cmr/m/n/10 ) \OM 740 | S/cmsy/m/n/10 741 | [] 742 | 743 | 744 | Overfull \hbox (61.56186pt too wide) in paragraph at lines 33--34 745 | [] [][]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[] \OMS/cmsy/m/n/10 746 | [] 747 | 748 | 749 | Overfull \hbox (14.40051pt too wide) in paragraph at lines 33--34 750 | \OML/cmm/m/it/10 \OT1/cmr/m/n/10 )[]$ 751 | [] 752 | 753 | 754 | ! LaTeX Error: Missing \begin{document}. 755 | 756 | See the LaTeX manual or LaTeX Companion for explanation. 757 | Type H for immediate help. 758 | ... 759 | 760 | l.35 G 761 | aussian Maximum-loglikelihood estimators:\\ 762 | You're in trouble here. Try typing to proceed. 763 | If that doesn't work, type X to quit. 764 | 765 | Missing character: There is no G in font nullfont! 766 | Missing character: There is no a in font nullfont! 767 | Missing character: There is no u in font nullfont! 768 | Missing character: There is no s in font nullfont! 769 | Missing character: There is no s in font nullfont! 770 | Missing character: There is no i in font nullfont! 771 | Missing character: There is no a in font nullfont! 772 | Missing character: There is no n in font nullfont! 773 | Missing character: There is no M in font nullfont! 774 | Missing character: There is no a in font nullfont! 775 | Missing character: There is no x in font nullfont! 776 | Missing character: There is no i in font nullfont! 777 | Missing character: There is no m in font nullfont! 778 | Missing character: There is no u in font nullfont! 779 | Missing character: There is no m in font nullfont! 780 | Missing character: There is no - in font nullfont! 781 | Missing character: There is no l in font nullfont! 782 | Missing character: There is no o in font nullfont! 783 | Missing character: There is no g in font nullfont! 784 | Missing character: There is no l in font nullfont! 785 | Missing character: There is no i in font nullfont! 786 | Missing character: There is no k in font nullfont! 787 | Missing character: There is no e in font nullfont! 788 | Missing character: There is no l in font nullfont! 789 | Missing character: There is no i in font nullfont! 790 | Missing character: There is no h in font nullfont! 791 | Missing character: There is no o in font nullfont! 792 | Missing character: There is no o in font nullfont! 793 | Missing character: There is no d in font nullfont! 794 | Missing character: There is no e in font nullfont! 795 | Missing character: There is no s in font nullfont! 796 | Missing character: There is no t in font nullfont! 797 | Missing character: There is no i in font nullfont! 798 | Missing character: There is no m in font nullfont! 799 | Missing character: There is no a in font nullfont! 800 | Missing character: There is no t in font nullfont! 801 | Missing character: There is no o in font nullfont! 802 | Missing character: There is no r in font nullfont! 803 | Missing character: There is no s in font nullfont! 804 | Missing character: There is no : in font nullfont! 805 | 806 | Overfull \hbox (20.0pt too wide) in paragraph at lines 35--36 807 | [] 808 | [] 809 | 810 | 811 | ! LaTeX Error: Missing \begin{document}. 812 | 813 | See the LaTeX manual or LaTeX Companion for explanation. 814 | Type H for immediate help. 815 | ... 816 | 817 | l.37 M 818 | LE estimator for $\sigma^2 = \tau$:\\ 819 | You're in trouble here. Try typing to proceed. 820 | If that doesn't work, type X to quit. 821 | 822 | Missing character: There is no M in font nullfont! 823 | Missing character: There is no L in font nullfont! 824 | Missing character: There is no E in font nullfont! 825 | Missing character: There is no e in font nullfont! 826 | Missing character: There is no s in font nullfont! 827 | Missing character: There is no t in font nullfont! 828 | Missing character: There is no i in font nullfont! 829 | Missing character: There is no m in font nullfont! 830 | Missing character: There is no a in font nullfont! 831 | Missing character: There is no t in font nullfont! 832 | Missing character: There is no o in font nullfont! 833 | Missing character: There is no r in font nullfont! 834 | Missing character: There is no f in font nullfont! 835 | Missing character: There is no o in font nullfont! 836 | Missing character: There is no r in font nullfont! 837 | Missing character: There is no : in font nullfont! 838 | 839 | Overfull \hbox (20.0pt too wide) in paragraph at lines 37--39 840 | [] 841 | [] 842 | 843 | 844 | Overfull \hbox (21.11455pt too wide) in paragraph at lines 37--39 845 | \OML/cmm/m/it/10 [] \OT1/cmr/m/n/10 = 846 | [] 847 | 848 | 849 | Overfull \hbox (5.50348pt too wide) in paragraph at lines 37--39 850 | \OML/cmm/m/it/10 $ 851 | [] 852 | 853 | 854 | Overfull \hbox (36.67711pt too wide) in paragraph at lines 37--39 855 | [] \OT1/cmr/m/n/10 = 856 | [] 857 | 858 | 859 | Overfull \hbox (48.24205pt too wide) in paragraph at lines 37--39 860 | [] [][] \OML/cmm/m/it/10 X[]$ 861 | [] 862 | 863 | 864 | ! LaTeX Error: Missing \begin{document}. 865 | 866 | See the LaTeX manual or LaTeX Companion for explanation. 867 | Type H for immediate help. 868 | ... 869 | 870 | l.40 M 871 | LE estimators:\\ 872 | You're in trouble here. Try typing to proceed. 873 | If that doesn't work, type X to quit. 874 | 875 | Missing character: There is no M in font nullfont! 876 | Missing character: There is no L in font nullfont! 877 | Missing character: There is no E in font nullfont! 878 | Missing character: There is no e in font nullfont! 879 | Missing character: There is no s in font nullfont! 880 | Missing character: There is no t in font nullfont! 881 | Missing character: There is no i in font nullfont! 882 | Missing character: There is no m in font nullfont! 883 | Missing character: There is no a in font nullfont! 884 | Missing character: There is no t in font nullfont! 885 | Missing character: There is no o in font nullfont! 886 | Missing character: There is no r in font nullfont! 887 | Missing character: There is no s in font nullfont! 888 | Missing character: There is no : in font nullfont! 889 | 890 | Overfull \hbox (20.0pt too wide) in paragraph at lines 40--41 891 | [] 892 | [] 893 | 894 | 895 | ! LaTeX Error: Missing \begin{document}. 896 | 897 | See the LaTeX manual or LaTeX Companion for explanation. 898 | Type H for immediate help. 899 | ... 900 | 901 | l.42 $ 902 | \hat{\mu}_ n^{MLE}=\frac{1}{n}\sum_{i=1}(x_i)$ 903 | You're in trouble here. Try typing to proceed. 904 | If that doesn't work, type X to quit. 905 | 906 | 907 | Overfull \hbox (57.19911pt too wide) in paragraph at lines 42--43 908 | []$[] \OT1/cmr/m/n/10 = 909 | [] 910 | 911 | 912 | Overfull \hbox (49.84222pt too wide) in paragraph at lines 42--43 913 | [] [][]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[]\OT1/cmr/m/n/10 )$ 914 | [] 915 | 916 | 917 | ! LaTeX Error: Missing \begin{document}. 918 | 919 | See the LaTeX manual or LaTeX Companion for explanation. 920 | Type H for immediate help. 921 | ... 922 | 923 | l.44 E 924 | xponential likelihood:\\ 925 | You're in trouble here. Try typing to proceed. 926 | If that doesn't work, type X to quit. 927 | 928 | Missing character: There is no E in font nullfont! 929 | Missing character: There is no x in font nullfont! 930 | Missing character: There is no p in font nullfont! 931 | Missing character: There is no o in font nullfont! 932 | Missing character: There is no n in font nullfont! 933 | Missing character: There is no e in font nullfont! 934 | Missing character: There is no n in font nullfont! 935 | Missing character: There is no t in font nullfont! 936 | Missing character: There is no i in font nullfont! 937 | Missing character: There is no a in font nullfont! 938 | Missing character: There is no l in font nullfont! 939 | Missing character: There is no l in font nullfont! 940 | Missing character: There is no i in font nullfont! 941 | Missing character: There is no k in font nullfont! 942 | Missing character: There is no e in font nullfont! 943 | Missing character: There is no l in font nullfont! 944 | Missing character: There is no i in font nullfont! 945 | Missing character: There is no h in font nullfont! 946 | Missing character: There is no o in font nullfont! 947 | Missing character: There is no o in font nullfont! 948 | Missing character: There is no d in font nullfont! 949 | Missing character: There is no : in font nullfont! 950 | 951 | Overfull \hbox (20.0pt too wide) in paragraph at lines 44--46 952 | [] 953 | [] 954 | 955 | 956 | Overfull \hbox (71.77652pt too wide) in paragraph at lines 44--46 957 | \OML/cmm/m/it/10 L\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[] [] x[]\OT1/cmr/m/n/10 ; 958 | \OML/cmm/m/it/10 \OT1/cmr/m/n/10 ) = 959 | [] 960 | 961 | 962 | Overfull \hbox (87.66444pt too wide) in paragraph at lines 44--46 963 | \OML/cmm/m/it/10 [] [] []$ 964 | [] 965 | 966 | 967 | ! LaTeX Error: Missing \begin{document}. 968 | 969 | See the LaTeX manual or LaTeX Companion for explanation. 970 | Type H for immediate help. 971 | ... 972 | 973 | l.48 U 974 | niform:\\ 975 | You're in trouble here. Try typing to proceed. 976 | If that doesn't work, type X to quit. 977 | 978 | Missing character: There is no U in font nullfont! 979 | Missing character: There is no n in font nullfont! 980 | Missing character: There is no i in font nullfont! 981 | Missing character: There is no f in font nullfont! 982 | Missing character: There is no o in font nullfont! 983 | Missing character: There is no r in font nullfont! 984 | Missing character: There is no m in font nullfont! 985 | Missing character: There is no : in font nullfont! 986 | 987 | Overfull \hbox (20.0pt too wide) in paragraph at lines 48--50 988 | [] 989 | [] 990 | 991 | 992 | Overfull \hbox (70.23482pt too wide) in paragraph at lines 48--50 993 | \OML/cmm/m/it/10 L\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 x[] [] x[]\OT1/cmr/m/n/10 ; 994 | \OML/cmm/m/it/10 b\OT1/cmr/m/n/10 ) = 995 | [] 996 | 997 | 998 | Overfull \hbox (54.2803pt too wide) in paragraph at lines 48--50 999 | []$ 1000 | [] 1001 | 1002 | ) 1003 | ! Emergency stop. 1004 | <*> ./Likelihood.tex 1005 | 1006 | *** (job aborted, no legal \end found) 1007 | 1008 | 1009 | Here is how much of TeX's memory you used: 1010 | 10 strings out of 492946 1011 | 329 string characters out of 3126266 1012 | 62009 words of memory out of 3000000 1013 | 4007 multiletter control sequences out of 15000+200000 1014 | 3640 words of font info for 14 fonts, out of 3000000 for 9000 1015 | 1141 hyphenation exceptions out of 8191 1016 | 18i,4n,12p,330b,56s stack positions out of 5000i,500n,10000p,200000b,50000s 1017 | ! ==> Fatal error occurred, no output PDF file produced! 1018 | -------------------------------------------------------------------------------- /content/Likelihood.tex: -------------------------------------------------------------------------------- 1 | \section{Maximum likelihood estimation} 2 | 3 | Let $\left\{ E,\left(\mathbf{P}_{\theta }\right)_{\theta \in \Theta }\right\}$ be a statistical model associated with a sample of i.i.d. random variables $X_1, X_2, \dots , X_ n$. Assume that there exists $\theta ^* \in \Theta$ such that $X_ i \sim \mathbf{P}_{\theta ^*}$. 4 | 5 | The \textbf{likelihood} of the model is the product of the $n$ samples of the pdf/pmf: 6 | 7 | \begin{align*} 8 | L_n{(X_1, X_2, \dots , X_ n, \theta)} = \\ 9 | {\begin{cases} 10 | \displaystyle \prod_{i=1}^np_\theta(x_i) & \text{if }E\text{ is discrete} \\ 11 | \displaystyle \prod_{i=1}^nf_\theta(x_i) & \text{if }E\text{ is continous} 12 | \end{cases}} 13 | \end{align*} 14 | The maximum likelihood estimator is the (unique) $\theta$ that minimizes $\widehat{\text {KL}}\left(\mathbf{P}_{\theta ^*}, \mathbf{P}_{\theta }\right)$ over the parameter space. (The minimizer of the KL divergence is unique due to it being strictly convex in the space of distributions once is fixed.) 15 | \begin{align*} 16 | \widehat{\theta }_ n^{MLE}& = \displaystyle \text {argmin}_{\theta \in \Theta }\widehat{\text {KL}}_ n\left(\mathbf{P}_{\theta ^*}, \mathbf{P}_{\theta }\right)\\ 17 | &= \displaystyle \text {argmax}_{\theta \in \Theta } \sum _{i=1}^{n} \ln p_{\theta }(X_ i)\\ 18 | &= \displaystyle \text {argmax}_{\theta \in \Theta } \ln \left(\prod _{i=1}^{n} p_{\theta }(X_ i)\right) 19 | \end{align*} 20 | 21 | Since taking derivatives of products is hard but easy for sums and $exp()$ is very common in pdfs we usually take the log of the likelihood function before maximizing it. 22 | \begin{align*} 23 | \ell((X_1, X_2, \dots , X_ n, \theta)) &= ln(L_n{(X_1, X_2, \dots , X_ n, \theta)})\\ 24 | &= \sum_{i=1}^n ln(L_i(X_i,\theta) 25 | \end{align*} 26 | 27 | Cookbook: set up the likelihood function, take log of likelihood function. Take the partial derivative of the loglikelihood function wrt. the parameter(s). Set the partial derivative(s) to zero and solve for the parameter. 28 | 29 | If an indicator function on the pdf/pmf does not depend on the parameter, it can be ignored. If it depends on the parameter it can't be ignored because there is an discontinuity in the loglikelihood function. The maximum/minimum of the $X_i$ is then the maximum likelihood estimator. 30 | 31 | \subsection{Fisher Information} 32 | 33 | The Fisher information is the covariance matrix of the gradient of the loglikelihood function. It is equal to the negative expectation of the Hessian of the loglikelihood function and captures the negative of the expected curvature of the loglikelihood function.\\ 34 | 35 | Let $\theta \in \Theta \subset \mathbb {R}^ d$ and let $\left(E,\left\{ \mathbf{P}_\theta \right\} _{\theta \in \Theta }\right)$ be a statistical model. Let $f_{\theta }(\mathbf x)$ be the pdf of the distribution $\mathbf{P}_\theta$. Then, the Fisher information of the statistical model is.\\ 36 | 37 | $\mathcal{I}(\theta) = Cov(\nabla \ell(\theta)) = \\ = \mathbb{E}[\nabla \ell(\theta))\nabla\ell(\theta)^T] - \mathbb{E}[\nabla \ell(\theta)]\mathbb{E}[\nabla \ell(\theta)] =\\ = -\mathbb{E}[\mathbb{H}\ell(\theta)]$\\ 38 | 39 | Where $\ell (\theta ) = \ln f_\theta (\mathbf X)$.If $\nabla\ell(\theta) \in \mathbb{R}^d$ it is a $d \times d$ matrix. The definition when the distribution has a pmf $p_\theta (\mathbf x)$ is also the same, with the expectation taken with respect to the pmf.\\ 40 | 41 | Let $(\mathbb {R}, \{ \mathbf{P}_\theta \} _{\theta \in \mathbb {R}})$ denote a continuous statistical model. Let $f_\theta (x)$ denote the pdf (probability density function) of the continuous distribution $\mathbf{P}_\theta$. Assume that $f_\theta (x)$ is twice-differentiable as a function of the parameter $\theta$.\\ 42 | 43 | Formula for the calculation of Fisher Information of $X$:\\ 44 | 45 | $\mathcal{I}(\theta )= \int _{-\infty }^\infty \frac{\left(\frac{\partial f_\theta (x)}{\partial \theta }\right)^2}{f_\theta (x)} \, dx$ \\ 46 | 47 | Models with one parameter (ie. Bernulli):\\ 48 | 49 | $\mathcal{I}(\theta ) = \textsf{Var}(\ell '(\theta ))$\\ 50 | 51 | $\mathcal{I}(\theta ) = - \mathbf{E}(\ell ''(\theta ))$\\ 52 | 53 | Models with multiple parameters (ie. Gaussians):\\ 54 | 55 | $\mathcal{I}(\theta ) = -\mathbb E\left[\mathbf{H}\ell (\theta )\right]$\\ 56 | 57 | Cookbook:\\ 58 | 59 | Better to use 2nd derivative.\\ 60 | 61 | \begin{itemize} 62 | \item Find loglikelihood 63 | \item Take second derivative (=Hessian if multivariate) 64 | \item Massage second derivative or Hessian (isolate functions of $X_i$ to use with $- \mathbf{E}(\ell ''(\theta ))$ or $-\mathbb E\left[\mathbf{H}\ell (\theta )\right]$. 65 | \item Find the expectation of the functions of $X_i$ and subsitute them back into the Hessian or the second derivative. Be extra careful to subsitute the right power back. $\mathbb{E}[X_i] \neq \mathbb{E}[X_i^2]$. 66 | \item Don't forget the minus sign! 67 | \end{itemize} 68 | 69 | \subsection{Asymptotic normality of the maximum likelihood estimator} 70 | 71 | Under certain conditions the MLE is asymptotically normal and consistent. This applies even if the MLE is not the sample average. 72 | 73 | Let the true parameter $\theta^{*} \in \Theta$. Necessary assumptions: 74 | 75 | \begin{itemize} 76 | \item The parameter is identifiable 77 | \item For all $\theta \in \Theta$, the support $\mathbb{P}_{\theta}$ does not depend on $\theta$ (e.g. like in $Unif(0,\theta)$); 78 | \item $\theta^{*}$ is not on the boundary of $\Theta$; 79 | \item Fisher information $\mathcal{I}(\theta)$ is invertible in the neighborhood of $\theta^{*}$ 80 | \item A few more technical conditions 81 | \end{itemize} 82 | 83 | The asymptotic variance of the MLE is the inverse of the fisher information. 84 | 85 | $\sqrt(n)(\widehat{\theta }_ n^{\text {MLE}} - \theta^*) \xrightarrow[n \rightarrow \infty]{(d)} N_d(0,\mathcal{I}(\theta^* )^{-1})$\\ -------------------------------------------------------------------------------- /content/M_estimation.tex: -------------------------------------------------------------------------------- 1 | \section{M-estimation} 2 | 3 | Generalization of maximum likelihood estimation. No statistical model needs to be assumed to perform M-estimation.\\ 4 | 5 | 6 | Median 7 | 8 | 9 | \section{Hubert loss} 10 | 11 | $h_\delta (x) = \begin{cases} \frac{x^2}{2} \quad \text {if} \, \, \left| x \right| < \delta \\ \delta ( \left| x \right| - \delta /2 ) \quad \text {if} \, \, \left| x \right| > \delta \end{cases}.$ 12 | 13 | the derivative of Huber's loss is the clip function : 14 | 15 | $\text {clip}_\delta (x) := \frac{d}{dx} h_\delta (x) = \begin{cases} \delta \quad \text {if} \, \, x > \delta \\ x \quad \text {if} \, \, -\delta \leq x \leq \delta \\ -\delta \quad \text {if} \, \, x < -\delta \\ \end{cases}$ -------------------------------------------------------------------------------- /content/Models.tex: -------------------------------------------------------------------------------- 1 | \section{Statistical models} 2 | \[E, \{ P_\theta \} _{\theta \in \Theta }\] 3 | $E$ is a sample space for $X$ i.e. a set that contains all possible outcomes of $X$\\ 4 | $\displaystyle \{ \mathbb {P_\theta }\} _{\theta \in \Theta }$ is a family of probability distributions on $E$.\\ 5 | $\Theta$ is a parameter set, i.e. a set consisting of some possible values of $\Theta$.\\ 6 | $\theta$ is the true parameter and unknown. In a parametric model we assume that $\Theta \subset \mathbb{R}^d,$ for some $d \geq 1$. 7 | \subsection{Identifiability} 8 | \begin{align*} 9 | &\theta \neq \theta' \Rightarrow \mathbb{P}_{\theta} \neq \mathbb{P}_{\theta'}\\ 10 | &\mathbb{P}_{\theta} = \mathbb{P}_{\theta'} \Rightarrow \theta = \theta' 11 | \end{align*} 12 | A Model is well specified if: 13 | \begin{align*} 14 | &\exists \theta \ s.t.\ \mathbb{P} =\mathbb{P}_{\theta } 15 | \end{align*} -------------------------------------------------------------------------------- /content/Moments.tex: -------------------------------------------------------------------------------- 1 | \section{Method of Moments} 2 | 3 | Let $X_1, \ldots , X_ n \stackrel{iid}{\sim } \mathbf{P}_{\theta ^*}$ associated with model $(\mathbb {E}, \{ \mathbf{P}_{\theta }\} _{\theta \in \Theta })$, with $\mathbb {E} \subseteq \mathbb {R}$ and $\Theta \subseteq \mathbb {R^d}$, for some d $\geq 1$ 4 | 5 | Population moments:\\ 6 | 7 | $m_k(\theta) = \mathbb{E}_{\theta}[X^k_1], 1 \leq k \leq d$\\ 8 | 9 | Empirical moments:\\ 10 | 11 | $\widehat{m_k}(\theta) = \bar{X_n^k} = \frac{1}{n} \sum _{i=1}^{n}X_i^k$ 12 | 13 | Convergence of empirical moments:\\ 14 | 15 | $\widehat{m_k} \xrightarrow[n \rightarrow \infty]{P, a.s.} m_k$\\ 16 | 17 | $(\widehat{m_1}, \ldots, \widehat{m_d}) \xrightarrow[n \rightarrow \infty]{P, a.s.} (m_1, \ldots, m_d)$ 18 | 19 | MOM Estimator $M$ is a map from the parameters of a model to the moments of its distribution. This map is invertible, (ie. it results into a system of equations that can be solved for the true parameter vector $\theta^{*}$). Find the moments (as many as parameters), set up system of equations, solve for parameters, use empirical moments to estimate. 20 | 21 | $\displaystyle \psi : \Theta \displaystyle \to \mathbb {R}^ d$\\ 22 | 23 | $\displaystyle \theta \displaystyle \mapsto (m_1(\theta ), m_2(\theta ), \ldots , m_ d(\theta ))$\\ 24 | 25 | $M ^{-1}(m_1(\theta ^*), m_2(\theta ^*), \ldots , m_ d(\theta ^*))$ 26 | 27 | The MOM estimator uses the empirical moments:\\ 28 | 29 | $M ^{-1}\left( \frac{1}{n} \sum _{i = 1}^ n X_ i, \frac{1}{n} \sum _{i = 1}^ n X_ i^2, \ldots , \frac{1}{n} \sum _{i = 1}^ n X_ i^ d \right)$\\ 30 | 31 | 32 | 33 | Assuming $M^{-1}$ is continuously differentiable at $M(0)$, the asymptotical variance of the MOM estimator is:\\\\ 34 | 35 | $\sqrt(n)(\widehat{\theta_n^{MM}} - \theta) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\Gamma)$\\ 36 | 37 | where, 38 | 39 | $\Gamma (\theta )=\left[\frac{\partial M^{-1}}{\partial \theta } (M(\theta ))\right]^{T} \Sigma (\theta )\left[\frac{\partial M^{-1}}{\partial \theta } (M(\theta ))\right]$ 40 | 41 | $\Gamma(\theta) = \nabla_{\theta}(M^{-1})^{T} \Sigma \nabla_{\theta}(M^{-1}) $ 42 | 43 | $\Sigma_{\theta}$ is the covariance matrix of the random vector of the moments $(X_1^1,X_1^2 \ldots, X_1^d)$. -------------------------------------------------------------------------------- /content/Multivariate_RV.tex: -------------------------------------------------------------------------------- 1 | \section{Random Vectors} 2 | 3 | 4 | A random vector $\mathbf X= \left(X^{(1)},\dots ,X^{(d)}\right)^ T$ of dimension $d \times 1$ is a vector-valued function from a probability space $\omega$ to $\mathbb {R}^ d$:\\ 5 | 6 | $ \mathbf{X}\, \, :\, \Omega \longrightarrow \mathbb {R}^ d$\\ 7 | 8 | $ \omega \longrightarrow \begin{pmatrix} X^{(1)}(\omega ) \\ X^{(2)}(\omega )\\ \vdots \\ X^{(d)}(\omega )\end{pmatrix}$\\ 9 | 10 | where each $\, X^{(k)}\ $, is a (scalar) random variable on $\Omega$. \\ 11 | 12 | PDF of $\mathbf X$: joint distribution of its components $X^{(1)},\, \ldots ,\, X^{(d)}$. \\ 13 | 14 | CDF of $\mathbf X$:\\ 15 | 16 | $\mathbb {R}^ d \rightarrow [0,1]$\\ 17 | 18 | $ \mathbf{x} \mapsto \mathbf{P}(X^{(1)}\leq x^{(1)},\ldots ,\, X^{(d)}\leq x^{(d)}).$\\ 19 | 20 | The sequence $\mathbf{X}_1, \mathbf{X}_2,\ldots$ converges in probability to $\mathbf{X}$ if and only if each component of the sequence $\, X_1^{(k)},X_2^{(k)},\ldots \,$ converges in probability to $\, X^{(k)}$. 21 | 22 | 23 | \subsection*{Expectation of a random vector} 24 | The expectation of a random vector is the elementwise expectation. Let $\mathbf X$ be a random vector of dimension $d \times 1$.\\ 25 | 26 | $ \mathbb E[\mathbf X] = \begin{pmatrix} \mathbb E[X^{(1)}]\\ \vdots \\ \mathbb E[X^{(d)}]\end{pmatrix}.$\\ 27 | 28 | The expectation of a random matrix is the expected value of each of its elements. Let $X=\{X_{ij}\}$ be an $n \times p$ random matrix. Then $\mathbb{E}[X]$, is the $n \times p$ matrix of numbers (if they exist):\\ 29 | 30 | $\mathbb{E}[X]= \begin{bmatrix} 31 | \mathbb{E}[X_{11}] & \mathbb{E}[X_{12}] & \dots & \mathbb{E}[X_{1p}] \\ 32 | \mathbb{E}[X_{21}] & \mathbb{E}[X_{22}] & \dots & \mathbb{E}[X_{2p}] \\ 33 | \vdots & \vdots &\ddots & \vdots \\ 34 | \mathbb{E}[X_{n1}] & \mathbb{E}[X_{n2}] & \dots & \mathbb{E}[X_{np}] \\ 35 | \end{bmatrix}$\\ 36 | 37 | Let $X$ and $Y$ be random matrices of the same dimension, and let $A$ and $B$ be conformable matrices of constants.\\ 38 | 39 | $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$\\ 40 | $\mathbb{E}[AXB] = A \mathbb{E}[X] B$\\ 41 | 42 | 43 | \subsection*{Covariance Matrix} 44 | Let $X$ be a random vector of dimension $d \times 1$ with expectation $\mu _{X}$. 45 | 46 | Matrix outer products!\\ 47 | 48 | $\Sigma =\mathbb E[(X- \mu _{X})(X- \mu _{X})^ T] =$\\ 49 | 50 | $\mathbb{E} \begin{pmatrix} \begin{bmatrix} 51 | 52 | X_1 - \mu_1\\ 53 | X_2 - \mu_2\\ 54 | \ldots\\ 55 | X_d - \mu_d\\ 56 | 57 | \end{bmatrix} \begin{bmatrix} X_1 - \mu_1, X_2 - \mu_2,\ldots, X_d - \mu_d \end{bmatrix} \end{pmatrix}$\\ 58 | 59 | $\Sigma = Cov (X) = \begin{bmatrix} 60 | \sigma_{11} & \sigma_{12} &\ldots & \sigma_{1d} \\ 61 | \sigma_{21} & \sigma_{22} &\ldots & \sigma_{2d} \\ 62 | \vdots & \vdots &\ddots & \vdots \\ 63 | \sigma_{d1} & \sigma_{d2} &\ldots & \sigma_{dd} \\ 64 | 65 | \end{bmatrix}$\\ 66 | 67 | The covariance matrix $\Sigma$ is a $d \times d$ matrix. It is a table of the pairwise covariances of the elemtents of the random vector. Its diagonal elements are the variances of the elements of the random vector, the off-diagonal elements are its covariances. Note that the covariance is commutative e.g. $\sigma_{12} = \sigma_{21}$ \\ 68 | 69 | Alternative forms:\\ 70 | 71 | $\Sigma = \mathbb {E}[XX^ T] - \mathbb {E}[X]\mathbb {E}[X]^ T =\\ = \mathbb {E}[XX^ T] - \mu _{X}\mu _{X}^ T$\\ 72 | 73 | Let the random vector $X \in \mathbb{R}^d$ and $A$ and $B$ be conformable matrices of constants.\\ 74 | 75 | $Cov(AX + B) = Cov(AX) = A Cov(X) A^T = A \Sigma A^T$ 76 | 77 | Every Covariance matrix is positive definite.\\ 78 | 79 | $\Sigma \prec 0$\\ 80 | 81 | \subsection*{Gaussian Random Vectors} 82 | 83 | 84 | A random vector $\mathbf{X}=(X^{(1)},\ldots ,X^{(d)})^ T\,$ is a Gaussian vector, or multivariate Gaussian or normal variable, if any linear combination of its components is a (univariate) Gaussian variable or a constant (a “Gaussian" variable with zero variance), i.e., if $\alpha ^ T \mathbf{X}$ is (univariate) Gaussian or constant for any constant non-zero vector $\alpha \in \mathbb {R}^ d$. 85 | 86 | \subsection*{Multivariate Gaussians} 87 | 88 | The distribution of, $X$ the $d$-dimensional Gaussian or normal distribution, is completely specified by the vector mean $\mu =\mathbb E[\mathbf{X}]= (\mathbb E[X^{(1)}],\ldots ,\mathbb E[X^{(d)}])^ T$ and the $d \times d$ covariance matrix $\Sigma$. If $\Sigma$ is invertible, then the pdf of $X$ is:\\ 89 | 90 | $ f_{\mathbf X}(\mathbf x) = \frac{1}{\sqrt{\left(2\pi \right)^ d \text {det}(\Sigma )}}e^{-\frac{1}{2}(\mathbf x-\mu )^ T \Sigma ^{-1} (\mathbf x-\mu )}, ~ ~ ~ \\ \mathbf x\in \mathbb {R}^ d$\\ 91 | 92 | 93 | Where $\text {det}(\Sigma )$ is the determinant of $\Sigma$, which is positive when $\Sigma$ is invertible. 94 | 95 | If $\mu = 0$ and $\Sigma$ is the identity matrix, then $X$ is called a standard normal random vector . 96 | 97 | If the covariant matrix $\Sigma$ is diagonal, the pdf factors into pdfs of univariate Gaussians, and hence the components are independent.\\ 98 | 99 | The linear transform of a gaussian $X \thicksim N_d(\mu,\Sigma)$ with conformable matrices $A$ and $B$ is a gaussian:\\ 100 | 101 | $AX + B = N_d(A\mu + b, A \Sigma A^T)$ 102 | 103 | \subsection*{Multivariate CLT} 104 | 105 | Let $X_1, \ldots, X_d \in \mathbb{R}^d$ be independent copies of a random vector $X$ 106 | such that $\mathbb{E}[x] = \mu$ ($d \times 1$ vector of expectations) and $Cov(X)= \Sigma$\\ 107 | 108 | $\sqrt(n)(\bar{X_n}-\mu) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\Sigma)$\\ 109 | 110 | $\sqrt(n) \Sigma^{-1/2} \bar{X_n}-\mu \xrightarrow[n \rightarrow \infty]{(d)} N(0,I_d)$\\ 111 | 112 | Where $\Sigma^{-1/2}$ is the $d \times d$ matrix such that $\Sigma^{-1/2} \Sigma^{-1/2} = \Sigma^{1}$ and $I_d$ is the identity matrix.\\ 113 | 114 | \subsection*{Multivariate Delta Method} 115 | -------------------------------------------------------------------------------- /content/OLS.tex: -------------------------------------------------------------------------------- 1 | \section{OLS} 2 | 3 | Given two random variables $X$ and $Y$, how can we predict the values of $Y$ given $X$? 4 | 5 | Let us consider $(X_1, Y_1), \ldots, (X_n, Y_n) \sim^{iid } \mathbb{P}$ where $\mathbb{P}$ is an unknown joint distribution. $\mathbb{P}$ can be described entirely by: 6 | 7 | \begin{align*} 8 | g(X) = \int f(X, y)dy\\ 9 | h(Y|X=x) = \frac{f(x, Y)}{g(x)} 10 | \end{align*} 11 | 12 | where $f$ is the joint PDF, $g$ the marginal density of $X$ and $h$ the conditional density. What we are interested in is $h(Y|X)$. 13 | 14 | \textbf{Regression function:} For a partial description, we can consider instead the conditional expection of $Y$ given $X=x$: 15 | 16 | \begin{align*} 17 | x \mapsto f(x) = \mathbb{E}[Y | X=x] = \int yh(y|x)dy 18 | \end{align*} 19 | 20 | We can also consider different descriptions of the distribution, like the median, quantiles or the variance.\\ 21 | \textbf{Linear regression:} trying to fit any function to $\mathbb{E}[Y | X=x]$ is a nonparametric problem; therefore, we restrict the problem to the tractable one of linear function: 22 | \begin{align*} 23 | f: x \mapsto a + bx 24 | \end{align*} 25 | 26 | \textbf{Theoretical linear regression:} let $X, Y$ be two random variables with two moments such as $\mathbb{V}[X] > 0$. The theoretical linear regression of $Y$ on $X$ is the line $a^{*} + b^{*}x$ where 27 | 28 | \begin{align*} 29 | (a^{*}, b^{*}) = argmin_{(a, b) \in \mathbb{R}^2}\mathbb{E}\left[(Y - a - bX)^2\right] 30 | \end{align*} 31 | 32 | Which gives: 33 | 34 | \begin{align*} 35 | b^{*} = \frac{Cov(X, Y)}{\mathbb{V}[X]}, \quad a^{*} = \mathbb{E}[Y] - b^{*} \mathbb{E}[X] 36 | \end{align*} 37 | 38 | \textbf{Noise:} we model the noise of $Y$ around the regression line by a random variable $\varepsilon = Y - a^{*} - b^{*} X$, such as: 39 | 40 | \begin{align*} 41 | \mathbb{E}[\varepsilon] = 0, \quad Cov(X, \varepsilon) = 0 42 | \end{align*} 43 | 44 | We have to estimate $a^{*}$ and $b^{*}$ from the data. We have $n$ random pairs $(X_1, Y_1), \ldots, (X_n, Y_n) \sim_{iid} (X, Y)$ such as: 45 | 46 | \begin{align*} 47 | Y_i = a^{*} + b^{*} X_i + \varepsilon_i 48 | \end{align*} 49 | 50 | The \textbf{Least Squares Estimator (LSE)} of $(a^{*}, b^{*})$ is the minimizer of the squared sum: 51 | 52 | \begin{align*} 53 | (\hat{a}_n, \hat{b}_n) = argmin_{(a, b) \in \mathbb{R}^2}\sum_{i=1}^n(Y_i - a - bX_i)^2 54 | \end{align*} 55 | 56 | The estimators are given by: 57 | \begin{align*} 58 | \hat{b}_n = \frac{\overline{XY} - \bar{X}\bar{Y}}{\overline{X^2} - \bar{X}^2}, \quad \hat{a}_n = \bar{Y} - \hat{b}_n \bar{X} 59 | \end{align*} 60 | 61 | The \textbf{Multivariate Regression} is given by: 62 | 63 | \begin{align*} 64 | Y_i = \sum_{j=1}^pX_i^{(j)}\beta_j^{*} + \varepsilon_i= \underbrace{X_i^\top}_{1 \times p}\underbrace{\beta^{*}}_{p \times 1} + \varepsilon_i 65 | \end{align*} 66 | 67 | We can assuming that the $X_i^{(1)}$ are 1 for the intercept. 68 | 69 | \begin{itemize} 70 | \item If $\beta^{*} = (a^{*}, b^{*}\top)^\top$, $\beta_1^{*} = a^{*}$ is the intercept. 71 | \item the $\varepsilon_i$ is the noise, satisfying $Cov(X_i, \varepsilon_i) = 0$ 72 | \end{itemize} 73 | 74 | The \textbf{Multivariate Least Squares Estimator (LSE)} of $\beta^{*}$ is the minimizer of the sum of square errors: 75 | \begin{align*} 76 | \hat{\beta} = argmin_{\beta \in \mathbb{R}^p}\sum_{i=1}^n(Y_i - X_i^\top\beta)^2 77 | \end{align*} 78 | 79 | \textbf{Matrix form:} we can rewrite these expressions. Let $Y = (Y_1, \ldots, Y_n)^\top \in \mathbb{R}^n$, and $\epsilon = (\varepsilon_1, \ldots, \varepsilon_n)^\top$. 80 | 81 | Let 82 | \begin{align*} 83 | X = \begin{pmatrix} X_1^\top \\ \vdots \\ X_n^\top \end{pmatrix} \in \mathbb{R}^{n \times p} 84 | \end{align*} 85 | 86 | $X$ is called the **design matrix**. The regression is given by: 87 | 88 | \begin{align*} 89 | Y = X\beta^{*} + \epsilon 90 | \end{align*} 91 | 92 | and the LSE is given by: 93 | 94 | \begin{align*} 95 | \hat{\beta} = argmin_{\beta \in \mathbb{R}^p} \|Y - X\beta\|^2_2 96 | \end{align*} 97 | 98 | 99 | Let us suppose $n \geq p$ and $rank(X) = p$. If we write: 100 | 101 | \begin{align*} 102 | F(\beta) = \|Y - X\beta\|^2_2 = (Y - X\beta)^\top(Y - X\beta) 103 | \end{align*} 104 | 105 | 106 | Then: 107 | 108 | \begin{align*} 109 | \nabla F(\beta) = 2 X^\top(Y - X\beta) 110 | \end{align*} 111 | 112 | 113 | \textbf{Least squares estimator}: setting $\nabla F(\beta) = 0$ gives us the expression of $\hat{\beta}$: 114 | 115 | \begin{align*} 116 | \hat{\beta} = (X^\top X)^{-1}X^\top Y 117 | \end{align*} 118 | 119 | 120 | **Geometric interpretation**: $X\hat{\beta}$ is the orthogonal projection of $Y$ onto the subspace spanned by the columns of $X$: 121 | 122 | $$ X\hat{\beta} = PY$$ 123 | 124 | where $P = X(X^\top X)^{-1}X^\top$ is the expression of the projector. 125 | 126 | **Statistic inference**: let us suppose that: 127 | 128 | * The design matrix $X$ is deterministic and $rank(X) = p$. 129 | * The model is **homoscedastic**: $\varepsilon_1, \ldots, \varepsilon_n$ are i.i.d. 130 | * The noise is Gaussian: $\epsilon \sim N_n(0, \sigma^2I_n)$. 131 | 132 | We therefore have: 133 | 134 | $$Y \sim N_n(X\beta^{*}, \sigma^2 I_n)$$ 135 | 136 | Properties of the LSE: 137 | 138 | $$\hat{\beta} \sim N_p(\beta^{*}, \sigma^2(X^\top X)^{-1})$$ 139 | 140 | The quadratic risk of $\hat{\beta}$ is given by: 141 | 142 | $$\mathbb{E}\left[\|\hat{\beta} - \beta^{*}\|^2_2\right] = \sigma^2 Tr \left((X^\top X)^{-1}\right)$$ 143 | 144 | The prediction error is given by: 145 | 146 | $$\mathbb{E}\left[\|Y - X\hat{\beta}\|^2_2\right] = \sigma^2(n - p)$$ 147 | 148 | The unbiased estimator of $\sigma^2$ is: 149 | 150 | $$\hat{\sigma^2} = \frac{1}{n-p}\|Y - X\hat{\beta}\|^2_2 = \frac1{n-p}\sum_{i=1}^n\hat{\varepsilon}_i^2$$ 151 | 152 | By **Cochran's Theorem**: 153 | 154 | $$ (n-p)\frac{\hat{\sigma^2}}{\sigma^2} \sim \chi^2_{n-p}, \quad \hat\beta \perp \hat{\sigma^2}$$ 155 | 156 | **Significance test**: let us test $H_0: \beta_j = 0$ against $H_1: \beta_j \neq 0$. Let us call 157 | 158 | $$\gamma_j = \left((X^\top X)^{-1}\right)_{jj} > 0$$ 159 | 160 | then: 161 | 162 | $$\frac{\hat{\beta}_j- \beta_j}{\sqrt{\hat{\sigma^2}\gamma_j}} \sim t_{n-p}$$ 163 | 164 | We can define the test statistic for our test: 165 | 166 | $$T_n^{(j)} = \frac{\hat{\beta}_j}{\sqrt{\hat{\sigma^2}\gamma_j}}$$ 167 | 168 | The test with non-asymptotic level $\alpha$ is given by: 169 | 170 | $$\psi_\alpha^{(j)} = \textbf{1}\{|T_n^{(j)}| > q_{\alpha/2}(t_{n-p})\}$$ 171 | 172 | **Bonferroni's test**: if we want to test the significance level of multiple tests at the same time, we cannot use the same level $\alpha$ for each of them. We must use a stricter test for each of them. Let us consider $S \subseteq \{1, \ldots, p\}$. Let us consider 173 | 174 | $$H_0: \forall j \in S, \beta_j = 0, \quad H_1: \exists j \in S, \beta_j \neq 0$$ 175 | 176 | The *Bonferroni's test* with significance level $\alpha$ is given by: 177 | 178 | $$\psi_\alpha^{(S)} = \max_{j \in S}\psi_{\alpha/K}^{(j)}$$ 179 | 180 | where $K = |S|$. The rejection region therefore is the union of all rejection regions: 181 | 182 | $$R_\alpha^{(S)} = \bigcup_{j \in S}R_{\alpha/K}^{(j)}$$ 183 | 184 | This test has nonasymptotic level at most $\alpha$: 185 | 186 | $$\P_{H_0}\left[R_\alpha^{(S)}\right] \leq \sum_{j\in S}\P_{H_0}\left[R_{\alpha/K}^{(j)}\right] = \alpha$$ 187 | 188 | This test also works for implicit testing (for example, $\beta_1 \geq \beta_2$). 189 | -------------------------------------------------------------------------------- /content/Quantiles.log: -------------------------------------------------------------------------------- 1 | This is pdfTeX, Version 3.14159265-2.6-1.40.19 (MiKTeX 2.9.6930 64-bit) (preloaded format=pdflatex 2019.1.30) 20 SEP 2019 15:44 2 | entering extended mode 3 | **./Quantiles.tex 4 | (Quantiles.tex 5 | LaTeX2e <2018-12-01> 6 | ! Undefined control sequence. 7 | l.1 \section 8 | {Quantiles of a Distribution} 9 | The control sequence at the end of the top line 10 | of your error message was never \def'ed. If you have 11 | misspelled it (e.g., `\hobx'), type `I' and the correct 12 | spelling (e.g., `I\hbox'). Otherwise just continue, 13 | and I'll forget about whatever was undefined. 14 | 15 | 16 | ! LaTeX Error: Missing \begin{document}. 17 | 18 | See the LaTeX manual or LaTeX Companion for explanation. 19 | Type H for immediate help. 20 | ... 21 | 22 | l.1 \section{Q 23 | uantiles of a Distribution} 24 | You're in trouble here. Try typing to proceed. 25 | If that doesn't work, type X to quit. 26 | 27 | Missing character: There is no Q in font nullfont! 28 | Missing character: There is no u in font nullfont! 29 | Missing character: There is no a in font nullfont! 30 | Missing character: There is no n in font nullfont! 31 | Missing character: There is no t in font nullfont! 32 | Missing character: There is no i in font nullfont! 33 | Missing character: There is no l in font nullfont! 34 | Missing character: There is no e in font nullfont! 35 | Missing character: There is no s in font nullfont! 36 | Missing character: There is no o in font nullfont! 37 | Missing character: There is no f in font nullfont! 38 | Missing character: There is no a in font nullfont! 39 | Missing character: There is no D in font nullfont! 40 | Missing character: There is no i in font nullfont! 41 | Missing character: There is no s in font nullfont! 42 | Missing character: There is no t in font nullfont! 43 | Missing character: There is no r in font nullfont! 44 | Missing character: There is no i in font nullfont! 45 | Missing character: There is no b in font nullfont! 46 | Missing character: There is no u in font nullfont! 47 | Missing character: There is no t in font nullfont! 48 | Missing character: There is no i in font nullfont! 49 | Missing character: There is no o in font nullfont! 50 | Missing character: There is no n in font nullfont! 51 | 52 | Overfull \hbox (20.0pt too wide) in paragraph at lines 1--2 53 | [] 54 | [] 55 | 56 | 57 | ! LaTeX Error: Missing \begin{document}. 58 | 59 | See the LaTeX manual or LaTeX Companion for explanation. 60 | Type H for immediate help. 61 | ... 62 | 63 | l.3 L 64 | et $\alpha$ in $(0,1)$. The quantile of order $1 - \alpha$ of a random ... 65 | 66 | You're in trouble here. Try typing to proceed. 67 | If that doesn't work, type X to quit. 68 | 69 | Missing character: There is no L in font nullfont! 70 | Missing character: There is no e in font nullfont! 71 | Missing character: There is no t in font nullfont! 72 | LaTeX Font Info: External font `cmex10' loaded for size 73 | (Font) <7> on input line 3. 74 | LaTeX Font Info: External font `cmex10' loaded for size 75 | (Font) <5> on input line 3. 76 | Missing character: There is no i in font nullfont! 77 | Missing character: There is no n in font nullfont! 78 | Missing character: There is no . in font nullfont! 79 | Missing character: There is no T in font nullfont! 80 | Missing character: There is no h in font nullfont! 81 | Missing character: There is no e in font nullfont! 82 | Missing character: There is no q in font nullfont! 83 | Missing character: There is no u in font nullfont! 84 | Missing character: There is no a in font nullfont! 85 | Missing character: There is no n in font nullfont! 86 | Missing character: There is no t in font nullfont! 87 | Missing character: There is no i in font nullfont! 88 | Missing character: There is no l in font nullfont! 89 | Missing character: There is no e in font nullfont! 90 | Missing character: There is no o in font nullfont! 91 | Missing character: There is no f in font nullfont! 92 | Missing character: There is no o in font nullfont! 93 | Missing character: There is no r in font nullfont! 94 | Missing character: There is no d in font nullfont! 95 | Missing character: There is no e in font nullfont! 96 | Missing character: There is no r in font nullfont! 97 | Missing character: There is no o in font nullfont! 98 | Missing character: There is no f in font nullfont! 99 | Missing character: There is no a in font nullfont! 100 | Missing character: There is no r in font nullfont! 101 | Missing character: There is no a in font nullfont! 102 | Missing character: There is no n in font nullfont! 103 | Missing character: There is no d in font nullfont! 104 | Missing character: There is no o in font nullfont! 105 | Missing character: There is no m in font nullfont! 106 | Missing character: There is no v in font nullfont! 107 | Missing character: There is no a in font nullfont! 108 | Missing character: There is no r in font nullfont! 109 | Missing character: There is no i in font nullfont! 110 | Missing character: There is no a in font nullfont! 111 | Missing character: There is no b in font nullfont! 112 | Missing character: There is no l in font nullfont! 113 | Missing character: There is no e in font nullfont! 114 | Missing character: There is no i in font nullfont! 115 | Missing character: There is no s in font nullfont! 116 | Missing character: There is no t in font nullfont! 117 | Missing character: There is no h in font nullfont! 118 | Missing character: There is no e in font nullfont! 119 | Missing character: There is no n in font nullfont! 120 | Missing character: There is no u in font nullfont! 121 | Missing character: There is no m in font nullfont! 122 | Missing character: There is no b in font nullfont! 123 | Missing character: There is no e in font nullfont! 124 | Missing character: There is no r in font nullfont! 125 | Missing character: There is no s in font nullfont! 126 | Missing character: There is no u in font nullfont! 127 | Missing character: There is no c in font nullfont! 128 | Missing character: There is no h in font nullfont! 129 | Missing character: There is no t in font nullfont! 130 | Missing character: There is no h in font nullfont! 131 | Missing character: There is no a in font nullfont! 132 | Missing character: There is no t in font nullfont! 133 | Missing character: There is no : in font nullfont! 134 | 135 | Overfull \hbox (20.0pt too wide) in paragraph at lines 3--4 136 | [] 137 | [] 138 | 139 | 140 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 3--4 141 | \OML/cmm/m/it/10 $ 142 | [] 143 | 144 | 145 | Overfull \hbox (22.22224pt too wide) in paragraph at lines 3--4 146 | \OT1/cmr/m/n/10 (0\OML/cmm/m/it/10 ; \OT1/cmr/m/n/10 1)$ 147 | [] 148 | 149 | 150 | Overfull \hbox (12.77782pt too wide) in paragraph at lines 3--4 151 | \OT1/cmr/m/n/10 1 \OMS/cmsy/m/n/10 152 | [] 153 | 154 | 155 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 3--4 156 | \OML/cmm/m/it/10 $ 157 | [] 158 | 159 | 160 | Overfull \hbox (9.06943pt too wide) in paragraph at lines 3--4 161 | \OML/cmm/m/it/10 X$ 162 | [] 163 | 164 | 165 | Overfull \hbox (10.16289pt too wide) in paragraph at lines 3--4 166 | \OML/cmm/m/it/10 q[]$ 167 | [] 168 | 169 | 170 | ! LaTeX Error: Missing \begin{document}. 171 | 172 | See the LaTeX manual or LaTeX Companion for explanation. 173 | Type H for immediate help. 174 | ... 175 | 176 | l.5 $ 177 | q_{\alpha } = \displaystyle \mathbb{P}\left(X\leq q_{\alpha }\right)=1-... 178 | 179 | You're in trouble here. Try typing to proceed. 180 | If that doesn't work, type X to quit. 181 | 182 | ! Undefined control sequence. 183 | l.5 $q_{\alpha } = \displaystyle \mathbb 184 | {P}\left(X\leq q_{\alpha }\right)=1-... 185 | The control sequence at the end of the top line 186 | of your error message was never \def'ed. If you have 187 | misspelled it (e.g., `\hobx'), type `I' and the correct 188 | spelling (e.g., `I\hbox'). Otherwise just continue, 189 | and I'll forget about whatever was undefined. 190 | 191 | 192 | Overfull \hbox (40.7184pt too wide) in paragraph at lines 5--6 193 | []$\OML/cmm/m/it/10 q[] \OT1/cmr/m/n/10 = 194 | [] 195 | 196 | 197 | Overfull \hbox (60.37451pt too wide) in paragraph at lines 5--6 198 | \OML/cmm/m/it/10 P [] \OT1/cmr/m/n/10 = 199 | [] 200 | 201 | 202 | Overfull \hbox (12.77782pt too wide) in paragraph at lines 5--6 203 | \OT1/cmr/m/n/10 1 \OMS/cmsy/m/n/10 204 | [] 205 | 206 | 207 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 5--6 208 | \OML/cmm/m/it/10 $ 209 | [] 210 | 211 | 212 | ! LaTeX Error: Missing \begin{document}. 213 | 214 | See the LaTeX manual or LaTeX Companion for explanation. 215 | Type H for immediate help. 216 | ... 217 | 218 | l.7 $ 219 | \mathbb{P}(X \geq q_{\alpha }) = \alpha$\\ 220 | You're in trouble here. Try typing to proceed. 221 | If that doesn't work, type X to quit. 222 | 223 | ! Undefined control sequence. 224 | \mathbb 225 | 226 | l.7 $\mathbb 227 | {P}(X \geq q_{\alpha }) = \alpha$\\ 228 | The control sequence at the end of the top line 229 | of your error message was never \def'ed. If you have 230 | misspelled it (e.g., `\hobx'), type `I' and the correct 231 | spelling (e.g., `I\hbox'). Otherwise just continue, 232 | and I'll forget about whatever was undefined. 233 | 234 | 235 | Overfull \hbox (51.32288pt too wide) in paragraph at lines 7--8 236 | []$\OML/cmm/m/it/10 P\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 X \OMS/cmsy/m/n/10  237 | [] 238 | 239 | 240 | Overfull \hbox (24.6073pt too wide) in paragraph at lines 7--8 241 | \OML/cmm/m/it/10 q[]\OT1/cmr/m/n/10 ) = 242 | [] 243 | 244 | 245 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 7--8 246 | \OML/cmm/m/it/10 $ 247 | [] 248 | 249 | 250 | ! LaTeX Error: Missing \begin{document}. 251 | 252 | See the LaTeX manual or LaTeX Companion for explanation. 253 | Type H for immediate help. 254 | ... 255 | 256 | l.9 $ 257 | F_X(q_{\alpha}) = 1 - \alpha$\\ 258 | You're in trouble here. Try typing to proceed. 259 | If that doesn't work, type X to quit. 260 | 261 | 262 | Overfull \hbox (62.52887pt too wide) in paragraph at lines 9--10 263 | []$\OML/cmm/m/it/10 F[]\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 q[]\OT1/cmr/m/n/10 ) = 264 | 265 | [] 266 | 267 | 268 | Overfull \hbox (12.77782pt too wide) in paragraph at lines 9--10 269 | \OT1/cmr/m/n/10 1 \OMS/cmsy/m/n/10 270 | [] 271 | 272 | 273 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 9--10 274 | \OML/cmm/m/it/10 $ 275 | [] 276 | 277 | 278 | ! LaTeX Error: Missing \begin{document}. 279 | 280 | See the LaTeX manual or LaTeX Companion for explanation. 281 | Type H for immediate help. 282 | ... 283 | 284 | l.11 $ 285 | F^{-1}_{X}(1-\alpha)= \alpha$\\ 286 | You're in trouble here. Try typing to proceed. 287 | If that doesn't work, type X to quit. 288 | 289 | 290 | Overfull \hbox (55.22232pt too wide) in paragraph at lines 11--12 291 | []$\OML/cmm/m/it/10 F[]\OT1/cmr/m/n/10 (1 \OMS/cmsy/m/n/10 292 | [] 293 | 294 | 295 | Overfull \hbox (20.87845pt too wide) in paragraph at lines 11--12 296 | \OML/cmm/m/it/10 \OT1/cmr/m/n/10 ) = 297 | [] 298 | 299 | 300 | Overfull \hbox (6.43404pt too wide) in paragraph at lines 11--12 301 | \OML/cmm/m/it/10 $ 302 | [] 303 | 304 | 305 | ! LaTeX Error: Missing \begin{document}. 306 | 307 | See the LaTeX manual or LaTeX Companion for explanation. 308 | Type H for immediate help. 309 | ... 310 | 311 | l.13 I 312 | f the distribution is \textbf{standard normal} $X \sim N(0,1)$:\\ 313 | You're in trouble here. Try typing to proceed. 314 | If that doesn't work, type X to quit. 315 | 316 | Missing character: There is no I in font nullfont! 317 | Missing character: There is no f in font nullfont! 318 | Missing character: There is no t in font nullfont! 319 | Missing character: There is no h in font nullfont! 320 | Missing character: There is no e in font nullfont! 321 | Missing character: There is no d in font nullfont! 322 | Missing character: There is no i in font nullfont! 323 | Missing character: There is no s in font nullfont! 324 | Missing character: There is no t in font nullfont! 325 | Missing character: There is no r in font nullfont! 326 | Missing character: There is no i in font nullfont! 327 | Missing character: There is no b in font nullfont! 328 | Missing character: There is no u in font nullfont! 329 | Missing character: There is no t in font nullfont! 330 | Missing character: There is no i in font nullfont! 331 | Missing character: There is no o in font nullfont! 332 | Missing character: There is no n in font nullfont! 333 | Missing character: There is no i in font nullfont! 334 | Missing character: There is no s in font nullfont! 335 | Missing character: There is no : in font nullfont! 336 | 337 | Overfull \hbox (20.0pt too wide) in paragraph at lines 13--14 338 | [] 339 | [] 340 | 341 | 342 | Overfull \hbox (24.82066pt too wide) in paragraph at lines 13--14 343 | \OT1/cmr/bx/n/10 stan- 344 | [] 345 | 346 | 347 | Overfull \hbox (23.10403pt too wide) in paragraph at lines 13--14 348 | \OT1/cmr/bx/n/10 dard 349 | [] 350 | 351 | 352 | Overfull \hbox (20.70824pt too wide) in paragraph at lines 13--14 353 | \OT1/cmr/bx/n/10 nor- 354 | [] 355 | 356 | 357 | Overfull \hbox (18.36794pt too wide) in paragraph at lines 13--14 358 | \OT1/cmr/bx/n/10 mal 359 | [] 360 | 361 | 362 | Overfull \hbox (19.62494pt too wide) in paragraph at lines 13--14 363 | \OML/cmm/m/it/10 X \OMS/cmsy/m/n/10  364 | [] 365 | 366 | 367 | Overfull \hbox (31.34721pt too wide) in paragraph at lines 13--14 368 | \OML/cmm/m/it/10 N\OT1/cmr/m/n/10 (0\OML/cmm/m/it/10 ; \OT1/cmr/m/n/10 1)$ 369 | [] 370 | 371 | 372 | ! LaTeX Error: Missing \begin{document}. 373 | 374 | See the LaTeX manual or LaTeX Companion for explanation. 375 | Type H for immediate help. 376 | ... 377 | 378 | l.16 \[ 379 | \mathbb{P}(|X| > q_{\alpha}) = \alpha = 2\Phi(q_{\alpha/2})\] 380 | You're in trouble here. Try typing to proceed. 381 | If that doesn't work, type X to quit. 382 | 383 | 384 | Overfull \hbox (20.0pt too wide) in paragraph at lines 16--16 385 | [][] 386 | [] 387 | 388 | ! Undefined control sequence. 389 | l.16 \[\mathbb 390 | {P}(|X| > q_{\alpha}) = \alpha = 2\Phi(q_{\alpha/2})\] 391 | The control sequence at the end of the top line 392 | of your error message was never \def'ed. If you have 393 | misspelled it (e.g., `\hobx'), type `I' and the correct 394 | spelling (e.g., `I\hbox'). Otherwise just continue, 395 | and I'll forget about whatever was undefined. 396 | 397 | 398 | Overfull \hbox (125.05472pt too wide) detected at line 16 399 | \OML/cmm/m/it/10 P\OT1/cmr/m/n/10 (\OMS/cmsy/m/n/10 j\OML/cmm/m/it/10 X\OMS/cms 400 | y/m/n/10 j \OML/cmm/m/it/10 > q[]\OT1/cmr/m/n/10 ) = \OML/cmm/m/it/10 \OT1/cm 401 | r/m/n/10 = 2(\OML/cmm/m/it/10 q[]\OT1/cmr/m/n/10 ) 402 | [] 403 | 404 | 405 | ! LaTeX Error: Missing \begin{document}. 406 | 407 | See the LaTeX manual or LaTeX Companion for explanation. 408 | Type H for immediate help. 409 | ... 410 | 411 | l.18 U 412 | se \textbf{standardization} if a gaussian has unknown mean and varianc... 413 | 414 | You're in trouble here. Try typing to proceed. 415 | If that doesn't work, type X to quit. 416 | 417 | Missing character: There is no U in font nullfont! 418 | Missing character: There is no s in font nullfont! 419 | Missing character: There is no e in font nullfont! 420 | Missing character: There is no i in font nullfont! 421 | Missing character: There is no f in font nullfont! 422 | Missing character: There is no a in font nullfont! 423 | Missing character: There is no g in font nullfont! 424 | Missing character: There is no a in font nullfont! 425 | Missing character: There is no u in font nullfont! 426 | Missing character: There is no s in font nullfont! 427 | Missing character: There is no s in font nullfont! 428 | Missing character: There is no i in font nullfont! 429 | Missing character: There is no a in font nullfont! 430 | Missing character: There is no n in font nullfont! 431 | Missing character: There is no h in font nullfont! 432 | Missing character: There is no a in font nullfont! 433 | Missing character: There is no s in font nullfont! 434 | Missing character: There is no u in font nullfont! 435 | Missing character: There is no n in font nullfont! 436 | Missing character: There is no k in font nullfont! 437 | Missing character: There is no n in font nullfont! 438 | Missing character: There is no o in font nullfont! 439 | Missing character: There is no w in font nullfont! 440 | Missing character: There is no n in font nullfont! 441 | Missing character: There is no m in font nullfont! 442 | Missing character: There is no e in font nullfont! 443 | Missing character: There is no a in font nullfont! 444 | Missing character: There is no n in font nullfont! 445 | Missing character: There is no a in font nullfont! 446 | Missing character: There is no n in font nullfont! 447 | Missing character: There is no d in font nullfont! 448 | Missing character: There is no v in font nullfont! 449 | Missing character: There is no a in font nullfont! 450 | Missing character: There is no r in font nullfont! 451 | Missing character: There is no i in font nullfont! 452 | Missing character: There is no a in font nullfont! 453 | Missing character: There is no n in font nullfont! 454 | Missing character: There is no c in font nullfont! 455 | Missing character: There is no e in font nullfont! 456 | Missing character: There is no t in font nullfont! 457 | Missing character: There is no o in font nullfont! 458 | Missing character: There is no g in font nullfont! 459 | Missing character: There is no e in font nullfont! 460 | Missing character: There is no t in font nullfont! 461 | Missing character: There is no t in font nullfont! 462 | Missing character: There is no h in font nullfont! 463 | Missing character: There is no e in font nullfont! 464 | Missing character: There is no q in font nullfont! 465 | Missing character: There is no u in font nullfont! 466 | Missing character: There is no a in font nullfont! 467 | Missing character: There is no n in font nullfont! 468 | Missing character: There is no t in font nullfont! 469 | Missing character: There is no i in font nullfont! 470 | Missing character: There is no l in font nullfont! 471 | Missing character: There is no e in font nullfont! 472 | Missing character: There is no s in font nullfont! 473 | Missing character: There is no b in font nullfont! 474 | Missing character: There is no y in font nullfont! 475 | Missing character: There is no u in font nullfont! 476 | Missing character: There is no s in font nullfont! 477 | Missing character: There is no i in font nullfont! 478 | Missing character: There is no n in font nullfont! 479 | Missing character: There is no g in font nullfont! 480 | Missing character: There is no Z in font nullfont! 481 | Missing character: There is no - in font nullfont! 482 | Missing character: There is no t in font nullfont! 483 | Missing character: There is no a in font nullfont! 484 | Missing character: There is no b in font nullfont! 485 | Missing character: There is no l in font nullfont! 486 | Missing character: There is no e in font nullfont! 487 | Missing character: There is no s in font nullfont! 488 | Missing character: There is no ( in font nullfont! 489 | Missing character: There is no s in font nullfont! 490 | Missing character: There is no t in font nullfont! 491 | Missing character: There is no a in font nullfont! 492 | Missing character: There is no n in font nullfont! 493 | Missing character: There is no d in font nullfont! 494 | Missing character: There is no a in font nullfont! 495 | Missing character: There is no r in font nullfont! 496 | Missing character: There is no d in font nullfont! 497 | Missing character: There is no n in font nullfont! 498 | Missing character: There is no o in font nullfont! 499 | Missing character: There is no r in font nullfont! 500 | Missing character: There is no m in font nullfont! 501 | Missing character: There is no a in font nullfont! 502 | Missing character: There is no l in font nullfont! 503 | Missing character: There is no t in font nullfont! 504 | Missing character: There is no a in font nullfont! 505 | Missing character: There is no b in font nullfont! 506 | Missing character: There is no l in font nullfont! 507 | Missing character: There is no e in font nullfont! 508 | Missing character: There is no s in font nullfont! 509 | Missing character: There is no ) in font nullfont! 510 | Missing character: There is no . in font nullfont! 511 | 512 | Overfull \hbox (20.0pt too wide) in paragraph at lines 18--19 513 | [] 514 | [] 515 | 516 | 517 | Overfull \hbox (24.82066pt too wide) in paragraph at lines 18--19 518 | \OT1/cmr/bx/n/10 stan- 519 | [] 520 | 521 | 522 | Overfull \hbox (26.93735pt too wide) in paragraph at lines 18--19 523 | \OT1/cmr/bx/n/10 dard- 524 | [] 525 | 526 | 527 | Overfull \hbox (17.72905pt too wide) in paragraph at lines 18--19 528 | \OT1/cmr/bx/n/10 iza- 529 | [] 530 | 531 | 532 | Overfull \hbox (19.80545pt too wide) in paragraph at lines 18--19 533 | \OT1/cmr/bx/n/10 tion 534 | [] 535 | 536 | 537 | Overfull \hbox (19.62494pt too wide) in paragraph at lines 18--19 538 | \OML/cmm/m/it/10 X \OMS/cmsy/m/n/10  539 | [] 540 | 541 | 542 | Overfull \hbox (37.9317pt too wide) in paragraph at lines 18--19 543 | \OML/cmm/m/it/10 N\OT1/cmr/m/n/10 (\OML/cmm/m/it/10 ; []\OT1/cmr/m/n/10 )$ 544 | [] 545 | 546 | 547 | ! LaTeX Error: Missing \begin{document}. 548 | 549 | See the LaTeX manual or LaTeX Companion for explanation. 550 | Type H for immediate help. 551 | ... 552 | 553 | l.20 $ 554 | \mathbf{P}\left(X\leq t\right) = \displaystyle \mathbf{P}\left(Z\leq \... 555 | 556 | You're in trouble here. Try typing to proceed. 557 | If that doesn't work, type X to quit. 558 | 559 | 560 | Overfull \hbox (73.87477pt too wide) in paragraph at lines 20--21 561 | []$\OT1/cmr/bx/n/10 P [] \OT1/cmr/m/n/10 = 562 | [] 563 | 564 | 565 | Overfull \hbox (69.3836pt too wide) in paragraph at lines 20--21 566 | \OT1/cmr/bx/n/10 P []$ 567 | [] 568 | 569 | 570 | ! LaTeX Error: Missing \begin{document}. 571 | 572 | See the LaTeX manual or LaTeX Companion for explanation. 573 | Type H for immediate help. 574 | ... 575 | 576 | l.22 $ 577 | Z= \frac{X-\mu}{\sigma} \sim N(0,1)$\\ 578 | You're in trouble here. Try typing to proceed. 579 | If that doesn't work, type X to quit. 580 | 581 | 582 | Overfull \hbox (38.09718pt too wide) in paragraph at lines 22--23 583 | []$\OML/cmm/m/it/10 Z \OT1/cmr/m/n/10 = 584 | [] 585 | 586 | 587 | Overfull \hbox (31.16995pt too wide) in paragraph at lines 22--23 588 | [] \OMS/cmsy/m/n/10  589 | [] 590 | 591 | 592 | Overfull \hbox (31.34721pt too wide) in paragraph at lines 22--23 593 | \OML/cmm/m/it/10 N\OT1/cmr/m/n/10 (0\OML/cmm/m/it/10 ; \OT1/cmr/m/n/10 1)$ 594 | [] 595 | 596 | ) 597 | ! Emergency stop. 598 | <*> ./Quantiles.tex 599 | 600 | *** (job aborted, no legal \end found) 601 | 602 | 603 | Here is how much of TeX's memory you used: 604 | 17 strings out of 492946 605 | 354 string characters out of 3126266 606 | 59009 words of memory out of 3000000 607 | 4011 multiletter control sequences out of 15000+200000 608 | 4567 words of font info for 17 fonts, out of 3000000 for 9000 609 | 1141 hyphenation exceptions out of 8191 610 | 18i,3n,16p,196b,66s stack positions out of 5000i,500n,10000p,200000b,50000s 611 | ! ==> Fatal error occurred, no output PDF file produced! 612 | -------------------------------------------------------------------------------- /content/Quantiles.tex: -------------------------------------------------------------------------------- 1 | \section{Quantiles of a Distribution} 2 | 3 | Let $\alpha$ in $(0,1)$. The quantile of order $1 - \alpha$ of a random variable $X$ is the number $q_{\alpha}$ such that:\\ 4 | \begin{align*} 5 | \displaystyle \mathbb{P}\left(X\leq q_{\alpha }\right)& = q_{\alpha } = 1-\alpha\\ 6 | \mathbb{P}(X \geq q_{\alpha })& = \alpha\\ 7 | F_X(q_{\alpha})& = 1 - \alpha\\ 8 | F^{-1}_{X}(1-\alpha)&= \alpha 9 | \end{align*} 10 | If the distribution is \textbf{standard normal} $X \sim N(0,1)$:\\ 11 | \begin{align*} 12 | \mathbb{P}(|X| > q_{\alpha})& = \alpha\\ 13 | & = 2\mathbf{\Phi}(q_{\alpha/2}) 14 | \end{align*} 15 | Use \textbf{standardization} if a gaussian has unknown mean and variance $X \sim N(\mu,\sigma^2)$ to get the quantiles by using Z-tables (standard normal tables).\\ 16 | 17 | \begin{align*} 18 | \mathbf{P}\left(X\leq t\right)& = \displaystyle \mathbf{P}\left(Z\leq \frac{t-\mu}{\sigma}\right)\\ 19 | & = \mathbf{\Phi}\left(\frac{t-\mu}{\sigma}\right)\\ 20 | Z &= \frac{X-\mu}{\sigma} \sim N(0,1)\\ 21 | q_{\alpha }& = \frac{t-\mu}{\sigma} 22 | \end{align*} -------------------------------------------------------------------------------- /content/Total_variation.tex: -------------------------------------------------------------------------------- 1 | \section{Distances between distributions} 2 | 3 | \subsection{Total variation distance} 4 | The total variation distance $\text {TV}$ between the propability measures $P$ and $Q$ with a sample space $E$ is defined as:\\ 5 | $\text {TV}(\mathbf{P}, \mathbf{Q}) = {\max _{A \subset E}}| \mathbf{P}(A) - \mathbf{Q}(A) |,$\\ 6 | Calculation with $f$ and $g$: 7 | \begin{align*} 8 | {TV}(\mathbf{P}, \mathbf{Q}) =&\\ 9 | &\begin{cases} 10 | \frac{1}{2} \, \sum _{x \in E} |f(x) - g(x)|,\text{discr}\\ 11 | \frac{1}{2} \, {\color{blue}{\int }} _{x \in E} |f(x) - g(x)|dx,\text{cont} 12 | \end{cases} 13 | \end{align*} 14 | Symmetry: $TV(\mathbf{P}, \mathbf{Q}) = TV(\mathbf{Q}, \mathbf{P})$\\ 15 | Positive: $TV(\mathbf{P}, \mathbf{Q}) \geq 0$\\ 16 | Definite: $TV(\mathbf{P}, \mathbf{Q}) = 0 \iff \mathbf{P}= \mathbf{Q}$\\ 17 | Triangle inequality: $TV(\mathbf{P}, \mathbf{V}) \leq TV(\mathbf{P}, \mathbf{Q}) + TV(\mathbf{Q}, \mathbf{V})$\\ 18 | 19 | If the support of $\mathbf{P}$ and $\mathbf{Q}$ is disjoint: 20 | \begin{align*} 21 | TV(\mathbf{P}, \mathbf{V}) = 1 22 | \end{align*} 23 | TV between continuous and discrete r.v: 24 | \begin{align*} 25 | TV(\mathbf{P}, \mathbf{V}) = 1 26 | \end{align*} 27 | \subsection{KL divergence} 28 | The KL divergence (aka relative entropy) $\text {KL}$ between between probability measures $P$ and $Q$ with the common sample space $E$ and pmf/pdf functions $f$ and $g$ is defined as: 29 | \begin{align*} 30 | {KL}(\mathbf{P}, \mathbf{Q}) = &\\ 31 | &\begin{cases} 32 | \sum _{x \in E} p(x) \ln \left( \frac{p(x)}{q(x)} \right),&\text{discr}\\ 33 | {\int }_{x \in E} p(x) \ln \left( \frac{p(x)}{q(x)}\right)dx,&\text{cont} 34 | \end{cases} 35 | \end{align*} 36 | The KL divergence is not a distance measure! Always sum over the support of $P$!\\ 37 | Asymetric in general: $\text {KL}(\mathbf{P}, \mathbf{Q}) \neq \text {KL}(\mathbf{Q}, \mathbf{P})$\\ 38 | Nonnegative: $\text {KL}(\mathbf{P}, \mathbf{Q}) \geq 0$\\ 39 | Definite: if $\mathbf{P}$ = $\mathbf{Q}$ then $\text {KL}(\mathbf{P}, \mathbf{Q}) = 0$\\ 40 | Does not satisfy triangle inequality in general: $KL(\mathbf{P}, \mathbf{V}) \nleq KL(\mathbf{P}, \mathbf{Q}) + KL(\mathbf{Q}, \mathbf{V})$\\ 41 | 42 | \textbf{Estimator of KL divergence:} 43 | \begin{align*} 44 | \displaystyle \text {KL}\left(\mathbf{P}_{\theta ^*}, \mathbf{P}_{\theta }\right)& = \mathbb {E}_{\theta ^*}\left[\ln \left(\frac{p_{\theta ^*}(X)}{p_{\theta }(X)}\right)\right]\\ 45 | \widehat{KL}(\mathbf{P}_{\theta_{*}},\mathbf{P}_{\theta})& = const - \frac{1}{n} \sum_{i=1}^{n} log(p_{\theta}(X_i)) 46 | \end{align*} -------------------------------------------------------------------------------- /content/abbildungen.tex: -------------------------------------------------------------------------------- 1 | \section{Abbildungen} 2 | Eine Abbildung $f:X\to Y$ ist eine Vorschrift, die jedem $x\in X$ eindeutig ein bestimmtes 3 | $y=f(x)\in Y$ zuordnet. $y$ ist das \emph{Bild} von $x$ und $x$ das \emph{Urbild} von $y$. 4 | Für eine Abbildung gilt, dass jedes Element der Urmenge $X$ genau auf ein $y\in Y$ abbildet, es müssen aber nicht alle Elemente aus $Y$ angenommen werden bzw. darf auch mehrfach angenommen werden (rechtseindeutig, linksvollständig).\\ 5 | Als Relation:\\ 6 | $f\subseteq A\times B$ mit $f=\{(a,f(a))\mid a\in A\wedge f(a)\in B\}$ 7 | \subsection*{Funktionen} 8 | Sei $f\subseteq A\times B$ linkseindeutige und rechtsvollständige Relation.\\ 9 | $F$ ist linksvollständig, wenn gilt $\forall a\in A\exists b\in B:(a,b)\in R$.\\ 10 | $F$ ist rechtseindeutig, wenn gilt $\forall a\in A\forall b_1,b_2\in B:(a,b_1)\in R\wedge 11 | (a,b_2)\in R\Rightarrow b_1=b_2$. 12 | \subsection*{Bild, Urbild} 13 | Sei $f:A\to B$ und $M\subseteq A$.\\ 14 | Das \emph{Bild} von $M$ unter $f$ ist die Menge $f(M):=\{f(x)\mid x\in M\}$.\\ 15 | Das \emph{Urbild} einer Teilmenge $N\subseteq B$ heißt $f^{-1}(N):=\{a\in A\mid f(a)\in N\}$. 16 | \subsection*{Eigenschaften von Abbildungen} 17 | \emph{Injektivität:}\\ 18 | $\forall x,y\in X: f(x)=f(y)\Rightarrow x=y$\\ 19 | Jedes $y\in Y$ wird höchstens einmal (oder garnicht) getroffen: 20 | 21 | \begin{tikzpicture} 22 | [ 23 | group/.style={ellipse, draw=myblue, minimum height=50pt, minimum width=30pt, label=above:#1}, 24 | my dot/.style={circle, fill, minimum width=2.5pt, label=above:#1, inner sep=0pt} 25 | ] 26 | \node (a) [my dot=$a$] {}; 27 | \node (b) [below=15pt of a, my dot=$b$] {}; 28 | \node (c) [below=15pt of b, my dot=$c$] {}; 29 | 30 | \node (d) [right=40pt of a, my dot=$d$] {}; 31 | \node (e) [right=40pt of b, my dot=$e$] {}; 32 | \node (f) [right=40pt of c, my dot=$f$] {}; 33 | \node (g) [below=15pt of f, my dot=$g$] {}; 34 | 35 | \foreach \i/\j in {a/e,b/d,c/g} 36 | \draw [->, shorten >=2pt] (\i) -- (\j); 37 | \node [fit=(a) (b) (c), group=$X$] {}; 38 | \node [fit=(d) (e) (f) (g), group=$Y$] {}; 39 | \end{tikzpicture} 40 | %\newpage 41 | 42 | \emph{Surjektivität:}\\ 43 | $\forall y\in Y\exists x\in X:f(x)=y$\\ 44 | Jedes $y\in Y$ wird mindestens einmal getroffen: 45 | 46 | \begin{tikzpicture} 47 | [ 48 | group/.style={ellipse, draw=myblue, minimum height=50pt, minimum width=30pt, label=above:#1}, 49 | my dot/.style={circle, fill, minimum width=2.5pt, label=above:#1, inner sep=0pt} 50 | ] 51 | \node (a) [my dot=$a$] {}; 52 | \node (b) [below=15pt of a, my dot=$b$] {}; 53 | \node (c) [below=15pt of b, my dot=$c$] {}; 54 | \node (d) [below=15pt of c, my dot=$d$] {}; 55 | 56 | \node (e) [right=40pt of a, my dot=$e$] {}; 57 | \node (f) [right=40pt of b, my dot=$f$] {}; 58 | \node (g) [right=40pt of c, my dot=$g$] {}; 59 | 60 | \foreach \i/\j in {a/e,b/f,c/g,d/g} 61 | \draw [->, shorten >=2pt] (\i) -- (\j); 62 | \node [fit=(a) (b) (c) (d), group=$X$] {}; 63 | \node [fit=(e) (f) (g), group=$Y$] {}; 64 | \end{tikzpicture} 65 | 66 | \emph{Bijektivität:}\\ 67 | Jedem $x\in X$ wird genau ein $y\in Y$ zugeordnet und jedem $y\in Y$ genau ein $x\in X$: 68 | 69 | \begin{tikzpicture} 70 | [ 71 | group/.style={ellipse, draw=myblue, minimum height=50pt, minimum width=30pt, label=above:#1}, 72 | my dot/.style={circle, fill, minimum width=2.5pt, label=above:#1, inner sep=0pt} 73 | ] 74 | \node (a) [my dot=$a$] {}; 75 | \node (b) [below=15pt of a, my dot=$b$] {}; 76 | \node (c) [below=15pt of b, my dot=$c$] {}; 77 | \node (d) [below=15pt of c, my dot=$d$] {}; 78 | 79 | \node (e) [right=40pt of a, my dot=$e$] {}; 80 | \node (f) [right=40pt of b, my dot=$f$] {}; 81 | \node (g) [right=40pt of c, my dot=$g$] {}; 82 | \node (h) [right=40pt of d, my dot=$h$] {}; 83 | 84 | \foreach \i/\j in {a/e,b/h,d/g,c/f} 85 | \draw [->, shorten >=2pt] (\i) -- (\j); 86 | \node [fit=(a) (b) (c) (d), group=$X$] {}; 87 | \node [fit=(e) (f) (g) (h), group=$Y$] {}; 88 | \end{tikzpicture} 89 | 90 | \emph{Beispiel für Abbildung}, die injektiv aber nicht surjektiv ist: Sei $f:\mathbb{N}\to\mathbb{N}$. Dann ist $f(n)=n+1$ injektiv, da $f(x)=f(y)\Leftrightarrow x+1=y+1$ gelten muss, was nur gilt, wenn $x=y$. $f$ ist nicht surjektiv da $0$ kein Urbild. 91 | \subsection*{Komposition} 92 | Die \emph{Komposition} (Hintereinanderausführung) zweier Abbildungen $f:A\to B$ und\\ 93 | $g:B\to C$ ist $a\mapsto (g\circ f)(a)=g(f(a)),\quad a\in A$ 94 | 95 | \begin{tikzpicture} 96 | \node at (0,0) (a) {$A$}; 97 | \node[right =1cm of a] (b){$B$}; 98 | \node[right =1cm of b] (c){$C$}; 99 | \draw[->, shorten >=2pt] (a) --node[above]{$f$} (b); 100 | \draw[->, shorten >=2pt] (b) --node[above]{$g$} (c); 101 | \draw[->, shorten >=2pt] (a) edge[bend right=30]node[below]{$g\circ f$} (c); 102 | \end{tikzpicture} 103 | 104 | Es gilt $(h\circ g)\circ f=h\circ (g\circ f)$. Außerdem gilt: 105 | Die Komposition von injektiven Abbildungen ist injektiv, die von surjektiven Abbildungen ist surjektiv und die von bijektiven Abbildungen ist bijektiv. 106 | \subsection*{Identität, Umkehrabbildung} 107 | Die Abbildung $id_A:A\to A$ mit $id_A(a)=a$ heißt \emph{Identität}.\\ 108 | Sei $f:A\to B$ bijektive Abbildung. Dann existiert zu $f$ stets eine Abbildung $g$ mit 109 | $g\circ f=id_A$ und $f\circ g=id_B$. $g$ heißt die zu $f$ \emph{inverse Abbildung} ($f^{-1}$). 110 | Es gilt $f^{-1}(f(a))=a$ und $f(f^{-1}(b))=b$. 111 | \subsection*{Mächtigkeit von Mengen, Abzählbarkeit} 112 | \emph{Gleichmächtige Mengen:}\\ 113 | Seien $M$ und $N$ zwei Mengen. $M$ und $N$ heißen gleichmächtig, wenn es eine bijektive 114 | Abbildung $f:M\to N$ gibt ($M\cong N$).\\ 115 | \emph{Endliche Mengen:}\\ 116 | Eine Menge $M$ heißt endlich, wenn $M=\emptyset$ oder es für ein $n\in\mathbb{N}$ eine 117 | bijektive Abbildung $b:M\to\mathbb{N}_n$ gibt.\\ 118 | \emph{Unendliche Mengen:}\\ 119 | Eine Menge $M$ heißt unendlich, wenn $M$ nicht endlich.\\ 120 | \emph{Abzählbare Mengen:}\\ 121 | Eine Menge $M$ heißt abzählbar, wenn $M$ endlich oder es gibt bijektive 122 | Abbildung $b:M\to\mathbb{N}$.\\ 123 | \emph{Abzählbar unendliche Mengen:}\\ 124 | Eine Menge $M$ heißt abzählbar unendlich, wenn $M$ abzählbar und $M$ unendlich.\\ 125 | \emph{Überabzählbare Mengen:}\\ 126 | Eine Menge $M$ heißt überabzählbar, wenn $M$ nicht abzählbar.\\ 127 | \emph{Spezielle endliche Mengen:}\\ 128 | Sei $n\in\mathbb{N}$. Dann ist $\mathbb{N}_n:=[n]:=\{1,...,n\}$ die Menge der ersten 129 | $n$ natürlichen Zahlen.\\ 130 | \emph{Beispiele:}\\ 131 | $|\{a,b,c\}|=3=|\{x,y,11\}|$\\ 132 | $|\mathbb{N}|=|\mathbb{R}|=|\mathbb{N}\times\mathbb{N}|$ 133 | \subsection*{Kardinalität} 134 | Anzahl der Elemente einer Menge. Zwei Mengen haben gleiche Kardinalität, wenn 135 | sie gleichmächtig sind. 136 | \subsection*{Beispielbeweis} 137 | \emph{Zu zeigen:} $|\mathbb{N}|=|\mathbb{N}\times\mathbb{N}|$\\ 138 | \emph{Beweis.} Wir betrachten $f:\mathbb{N}\to\mathbb{N}\times\mathbb{N}$ mit 139 | $f(n):=(1,n)$ und $g:\mathbb{N}\times\mathbb{N}\to\mathbb{N}$ mit $g(n,m):=2^n\cdot 3^m$. 140 | Beide sind injektiv, also $\mathbb{N}\cong\mathbb{N}\times\mathbb{N}$, also 141 | $|\mathbb{N}|=|\mathbb{N}\times\mathbb{N}|$. -------------------------------------------------------------------------------- /content/algebra.log: -------------------------------------------------------------------------------- 1 | This is pdfTeX, Version 3.14159265-2.6-1.40.19 (MiKTeX 2.9.6930 64-bit) (preloaded format=pdflatex 2019.1.30) 11 MAR 2019 13:18 2 | entering extended mode 3 | **./algebra.tex 4 | (algebra.tex 5 | LaTeX2e <2018-12-01> 6 | ! Undefined control sequence. 7 | l.1 \section 8 | {Algebra} 9 | The control sequence at the end of the top line 10 | of your error message was never \def'ed. If you have 11 | misspelled it (e.g., `\hobx'), type `I' and the correct 12 | spelling (e.g., `I\hbox'). Otherwise just continue, 13 | and I'll forget about whatever was undefined. 14 | 15 | 16 | ! LaTeX Error: Missing \begin{document}. 17 | 18 | See the LaTeX manual or LaTeX Companion for explanation. 19 | Type H for immediate help. 20 | ... 21 | 22 | l.1 \section{A 23 | lgebra} 24 | You're in trouble here. Try typing to proceed. 25 | If that doesn't work, type X to quit. 26 | 27 | Missing character: There is no A in font nullfont! 28 | Missing character: There is no l in font nullfont! 29 | Missing character: There is no g in font nullfont! 30 | Missing character: There is no e in font nullfont! 31 | Missing character: There is no b in font nullfont! 32 | Missing character: There is no r in font nullfont! 33 | Missing character: There is no a in font nullfont! 34 | ! Undefined control sequence. 35 | l.2 \subsection 36 | *{Absolute Value Inequalities} 37 | The control sequence at the end of the top line 38 | of your error message was never \def'ed. If you have 39 | misspelled it (e.g., `\hobx'), type `I' and the correct 40 | spelling (e.g., `I\hbox'). Otherwise just continue, 41 | and I'll forget about whatever was undefined. 42 | 43 | Missing character: There is no * in font nullfont! 44 | Missing character: There is no A in font nullfont! 45 | Missing character: There is no b in font nullfont! 46 | Missing character: There is no s in font nullfont! 47 | Missing character: There is no o in font nullfont! 48 | Missing character: There is no l in font nullfont! 49 | Missing character: There is no u in font nullfont! 50 | Missing character: There is no t in font nullfont! 51 | Missing character: There is no e in font nullfont! 52 | Missing character: There is no V in font nullfont! 53 | Missing character: There is no a in font nullfont! 54 | Missing character: There is no l in font nullfont! 55 | Missing character: There is no u in font nullfont! 56 | Missing character: There is no e in font nullfont! 57 | Missing character: There is no I in font nullfont! 58 | Missing character: There is no n in font nullfont! 59 | Missing character: There is no e in font nullfont! 60 | Missing character: There is no q in font nullfont! 61 | Missing character: There is no u in font nullfont! 62 | Missing character: There is no a in font nullfont! 63 | Missing character: There is no l in font nullfont! 64 | Missing character: There is no i in font nullfont! 65 | Missing character: There is no t in font nullfont! 66 | Missing character: There is no i in font nullfont! 67 | Missing character: There is no e in font nullfont! 68 | Missing character: There is no s in font nullfont! 69 | LaTeX Font Info: External font `cmex10' loaded for size 70 | (Font) <7> on input line 3. 71 | LaTeX Font Info: External font `cmex10' loaded for size 72 | (Font) <5> on input line 3. 73 | ) 74 | ! Emergency stop. 75 | <*> ./algebra.tex 76 | 77 | *** (job aborted, no legal \end found) 78 | 79 | 80 | Here is how much of TeX's memory you used: 81 | 9 strings out of 492946 82 | 287 string characters out of 3126266 83 | 57009 words of memory out of 3000000 84 | 4006 multiletter control sequences out of 15000+200000 85 | 3640 words of font info for 14 fonts, out of 3000000 for 9000 86 | 1141 hyphenation exceptions out of 8191 87 | 18i,2n,12p,58b,56s stack positions out of 5000i,500n,10000p,200000b,50000s 88 | ! ==> Fatal error occurred, no output PDF file produced! 89 | -------------------------------------------------------------------------------- /content/algebra.tex: -------------------------------------------------------------------------------- 1 | \section{Algebra} 2 | Absolute Value Inequalities:\\ 3 | $ |f(x)| < a \Rightarrow -a < f(x) < a$\\ 4 | $|f(x)| > a \Rightarrow f(x) > a$ or $f(x) < -a$\\ 5 | \section{Matrixalgebra} 6 | 7 | $\displaystyle \displaystyle \left\lVert \mathbf{A}\mathbf{x}\right\rVert ^2 = \displaystyle (\mathbf{A}\mathbf{x})^ T(\mathbf{A}\mathbf{x})\, =\, \mathbf{x}^ T\mathbf{A}^ T\mathbf{A}\mathbf{x} = \displaystyle \mathbf{x}^ T\mathbf{A}^ T\mathbf{A}\mathbf{x}\qquad$ 8 | 9 | \section{Calculus} 10 | 11 | Differentiation under the integral sign\\ 12 | $\frac{\text{d}}{\text{d}x}\left( \int_{a(x)}^{b(x)}f(x,t)\text{d}t \right ) = f(x,b(x))b'(x)-f(x,a(x))a'(x)+\int_{a(x)}^{b(x)}f_x(x,t)\text{d}t.$ 13 | 14 | \subsection*{Concavity in 1 dimension} 15 | If $g:I \to \mathbb {R}$ is twice differentiable in the interval $I$: 16 | 17 | concave:\\ 18 | if and only if $g^{\prime \prime }(x) {\color{blue}{\leq }} 0$ for all $x \in I$\\ 19 | 20 | strictly concave:\\ 21 | if $g^{\prime \prime }(x) {\color{blue}{<}} 0$ for all $x \in I$\\ 22 | 23 | convex:\\ 24 | if and only if $g^{\prime \prime }(x) {\color{blue}{\geq }} 0$ for all $x \in I$\\ 25 | 26 | strictly convex if:\\ 27 | $g^{\prime \prime }(x) {\color{blue}{>}} 0$ for all $x \in I$\\ 28 | 29 | \subsection*{Multivariate Calculus} 30 | The Gradient $\nabla$ of a twice differntiable function $f$ is defined as: 31 | 32 | $\nabla f:\mathbb {R}^ d \rightarrow \mathbb {R}^ d $\\ 33 | $\displaystyle \theta =\begin{pmatrix} \theta _1\\ \theta _2\\ \vdots \\ \theta _ d\end{pmatrix} \displaystyle \mapsto \displaystyle \left.\begin{pmatrix} \frac{\partial f }{\partial \theta _1}\\ \frac{\partial f }{\partial \theta _2}\\ \vdots \\ \frac{\partial f }{\partial \theta _ d}\end{pmatrix}\right|_{\theta }$\\ 34 | \textbf{Hessian}\\ 35 | 36 | The Hessian of $f$ is a symmetric matrix of second partial derivatives of $f$\\ 37 | 38 | $\mathbf{H} h(\theta )= \nabla ^{2} h(\theta )= \\ 39 | \left(\begin{array}{ c c c } 40 | \frac{\partial ^{2} h}{\partial \theta _{1} \partial \theta _{1}} (\theta ) & \cdots & \frac{\partial ^{2} h}{\partial \theta _{1} \partial \theta _{d}} (\theta )\\ 41 | & \vdots & \\ 42 | \frac{\partial ^{2} h}{\partial \theta _{d} \partial \theta _{1}} (\theta ) & \cdots & \frac{\partial ^{2} h}{\partial \theta _{d} \partial \theta _{d}} (\theta ) 43 | \end{array}\right) \in \mathbb{R}^{d\times d}$\\ 44 | 45 | A symmetric (real-valued) $d\times d$ matrix $\mathbf{A}$ is:\\ 46 | 47 | Positive semi-definite:\\ 48 | $\mathbf{x}^ T \, \mathbf{A}\, \mathbf{x} \geq 0\qquad \text {for all }\, \mathbf{x}\in \mathbb {R}^ d.$\\ 49 | 50 | Positive definite:\\ 51 | $\mathbf{x}^ T \, \mathbf{A}\, \mathbf{x}> 0$ for all non-zero vectors $\mathbf{x}\in \mathbb {R}^ d$\\ 52 | 53 | Negative semi-definite (resp. negative definite):\\ 54 | 55 | $\mathbf{x}^ T \, \mathbf{A}\, \mathbf{x}$ is negative for all $\mathbf{x}\in \mathbb {R}^ d-\{ \mathbf{0}\}$.\\ 56 | 57 | Positive (or negative) definiteness implies positive (or negative) semi-definiteness.\\ 58 | 59 | If the Hessian is positive definite then $f$ attains a local minimum at $a$ (convex).\\ 60 | 61 | If the Hessian is negative definite at $a$, then f attains a local maximum at $a$ (concave).\\ 62 | 63 | If the Hessian has both positive and negative eigenvalues then $a$ is a saddle point for $f$. 64 | 65 | %\subsection*{Lagrange Multiplier} -------------------------------------------------------------------------------- /content/aussagen.tex: -------------------------------------------------------------------------------- 1 | \section{Aussagenlogik} 2 | \subsection*{Aussage} 3 | Eine Aussage ist ein Satz, der entweder wahr oder falsch ist, also nie beides zugleich. 4 | Wahre Aussagen haben den Wahrheitswert $w$ und falsche Aussagen den 5 | Wahrheitswert $f$. 6 | \subsection*{Belegung von Variablen} 7 | Sei $\mathcal{A}_B(F) = f$. 8 | Dann ist stets $\mathcal{A}_B(F\Rightarrow G) = w$ 9 | \subsection*{Formelbeweis über Belegung} 10 | Wenn $F \wedge G$ eine Tautologie ist, dann (und nur dann) ist $F$ eine Tautologie und $G$ auch. 11 | Hinweis: In dem Lemma stecken zwei Teilaussagen, die beide zu beweisen sind: 12 | 1. Wenn $F \wedge G$ eine Tautologie ist, dann ist $F$ eine Tautologie und $G$ auch. 13 | 2. Umgekehrt: Sind $F$ und $G$ Tautologien, dann ist auch $F \wedge G$ eine. 14 | \emph{Beweis.} 15 | 1. Annahme: $F \wedge G$ sei eine Tautologie. 16 | Dann: Für jede Belegung $B$ wertet $F \wedge G$ zu wahr aus. 17 | Dann: Das ist nur der Fall, wenn sowohl $F$ als auch $G$ (für jedes $B$) zu wahr auswerten. 18 | Dann: Für jede Belegung $B$ wertet $F$ zu wahr aus. Und: 19 | Für jede Belegung $B$ wertet $G$ zu wahr aus. 20 | Dann: $F$ ist Tautologie und $G$ ist Tautologie. 21 | 2. Annahme: $F$ ist Tautologie und $G$ ist Tautologie. 22 | Dann: Für jede Belegung $B_1$ wertet $F$ zu wahr aus. Und: Für jede Belegung $B_2$ wertet $G$ zu wahr aus. 23 | Dann: Für jede Belegung $B$ wertet $F \wedge G$ zu wahr aus. 24 | Dann: $F \wedge G$ ist eine Tautologie. 25 | \subsection*{Äquivalenz und Folgerung} 26 | $p\equiv q$ gilt genau dann, wenn sowohl $p\models q$ als auch $q\models p$ gelten. \emph{Beweis.} 27 | $p\equiv q$ GDW $p\Leftrightarrow q$ ist Tautologie nach Def. von $\equiv$ 28 | GDW $(p\Rightarrow q) \wedge (q\Rightarrow p)$ ist Tautologie 29 | GDW $(p\Rightarrow q)$ ist Tautologie und $(q\Rightarrow p)$ ist Tautologie 30 | GDW $(p\models q)$ gilt und $q\models p$ gilt. 31 | \subsection*{Substitution} 32 | Ersetzt man in einer Formel eine beliebige Teilformel $F$ durch eine logisch äquivalente 33 | Teilformel $F'$, so verändert sich der Wahrheitswerteverlauf der Gesamtformel nicht. 34 | Man kann Formeln also vereinfachen, indem man Teilformeln durch äquivalente 35 | (einfachere) Teilformeln ersetzt. 36 | \subsection*{Universum} 37 | Die freien Variablen in einer Aussagenform können durch Objekte aus einer als 38 | Universum bezeichneten Gesamtheit wie $\mathbb{N},\mathbb{R},\mathbb{Z},\mathbb{Q}$ ersetzt werden. 39 | \subsection*{Tautologien} 40 | $(p\wedge q)\Rightarrow p$\text{ bzw. }$p\Rightarrow (p\vee q)$\\ 41 | $(q\Rightarrow p)\vee (\neg q\Rightarrow p)$\\ 42 | $(p\Rightarrow q)\Leftrightarrow (\neg p\vee q)$\\ 43 | $(p\Rightarrow q)\Leftrightarrow (\neg q\Rightarrow\neg p)$ \hfill\text{(Kontraposition)}\\ 44 | $(p\wedge (p\Rightarrow q))\Rightarrow q$ \hfill\text{(Modus Ponens)}\\ 45 | $((p\Rightarrow q)\wedge (q\Rightarrow r))\Rightarrow (p\Rightarrow r)$\\ 46 | $((p\Rightarrow q)\wedge (p\Rightarrow r))\Rightarrow (p\Rightarrow (q\wedge r))$\\ 47 | $((p\Rightarrow q)\wedge (q\Rightarrow p))\Leftrightarrow (p\Leftrightarrow q)$ 48 | \subsection*{Nützliche Äquivalenzen} 49 | Kommutativität:\\ 50 | $(p \wedge q) \equiv (q \wedge p)$\\ 51 | $(p \vee q) \equiv (q \vee p)$\\ 52 | Assoziativität:\\ 53 | $(p \wedge (q \wedge r)) \equiv ((p \wedge q) \wedge r)$\\ 54 | $(p \vee (q \vee r)) \equiv ((p \vee q) \vee r)$\\ 55 | Distributivität:\\ 56 | $(p \wedge (q \vee r)) \equiv ((p \wedge q) \vee (p \wedge r))$\\ 57 | $(p \vee (q \wedge r)) \equiv ((p \vee q) \wedge (p \vee r))$\\ 58 | Idempotenz:\\ 59 | $(p \wedge p) \equiv p$\\ 60 | $(p \vee p) \equiv p$\\ 61 | Doppelnegation:\\ 62 | $\neg (\neg p) \equiv p$\\ 63 | de Morgans Regeln:\\ 64 | $\neg (p \wedge q) \equiv ((\neg p) \vee (\neg q))$\\ 65 | $\neg (p \vee q) \equiv ((\neg p) \wedge (\neg q))$\\ 66 | Definition Implikation:\\ 67 | $(p \Rightarrow q) \equiv (\neg p \vee q)$\\ 68 | Tautologieregeln:\\ 69 | $(p \wedge q) \equiv p$\hfill (falls $q$ eine Tautologie ist)\\ 70 | $(p \vee q) \equiv q$\\ 71 | Kontradiktionsregeln:\\ 72 | $(p \wedge q) \equiv q$\hfill (falls $q$ eine Kontradiktion ist)\\ 73 | $(p \vee q) \equiv p$\\ 74 | Absorptionsregeln:\\ 75 | $(p \wedge (p \vee q)) \equiv p$\\ 76 | $(p \vee (p \wedge q)) \equiv p$\\ 77 | Prinzip vom ausgeschlossenen Dritten:\\ 78 | $p \vee (\neg p) \equiv w$\\\ 79 | Prinzip vom ausgeschlossenen Widerspruch:\\ 80 | $p \wedge (\neg p) \equiv f$ 81 | \subsection*{Äquivalenzen von quant. Aussagen} 82 | Negationsregeln:\\ 83 | $\neg\forall x:p(x)\equiv\exists x:(\neg p(x))$\\ 84 | $\neg\exists x:p(x)\equiv\forall x:(\neg p(x))$\\ 85 | Ausklammerregeln:\\ 86 | $(\forall x:p(x)\wedge\forall y:q(y))\equiv\forall z:(p(z)\wedge q(z))$\\ 87 | $(\exists x:p(x)\wedge\exists y:q(y))\equiv\exists z:(p(z)\wedge q(z))$\\ 88 | Vertauschungsregeln\\ 89 | $\forall x\forall y:p(x,y)\equiv\forall y\forall x:p(x,y)$\\ 90 | $\exists x\exists y:p(x,y)\equiv\forall y\exists x:p(x,y)$ 91 | \subsection*{Äquivalenzumformung} 92 | Wir demonstrieren an der Formel $\neg (\neg p \wedge q) \wedge (p \vee q)$, wie man mit Hilfe der 93 | aufgelisteten logischen Äquivalenzen tatsächlich zu Vereinfachungen kommen kann:\\ 94 | $\phantom{{}\equiv{}} \neg (\neg p \wedge q) \wedge (p \vee q)$\\ 95 | $\equiv (\neg (\neg p) \vee (\neg q)) \wedge (p \vee q)$\hfill de Morgan\\ 96 | $\equiv (p \vee (\neg q)) \wedge (p \vee q)$\hfill Doppelnegation\\ 97 | $\equiv p \vee ((\neg q) \wedge q)$\hfill Distributivtät v.r.n.l.\\ 98 | $\equiv p \vee (q \wedge (\neg q))$\hfill Kommutativtät\\ 99 | $\equiv p \vee f$\hfill Prinzip v. ausgeschl. Widerspruch\\ 100 | $\equiv p$\hfill Kontradiktionsregel 101 | \subsection*{Quantifizierte Aussagen} 102 | Sei $p(x)$ eine Aussageform über dem Universum $U$. 103 | $\exists x : p(x)$ ist wahr genau dann, wenn ein $u$ in $U$ existiert, so dass $p(u)$ wahr ist. 104 | $\forall x : p(x)$ ist wahr genau dann, wenn $p(u)$ für jedes $u$ aus $U$ wahr ist. -------------------------------------------------------------------------------- /content/bayesian.tex: -------------------------------------------------------------------------------- 1 | \section{Bayesian Statistics} 2 | Bayesian inference conceptually amounts to weighting the likelihood $L_n(\theta)$ by a prior knowledge we might have on $\theta$. 3 | Given a statistical model we technically model our parameter $\theta$ as if it were a random variable. We therefore define the \textbf{prior distribution} (PDF): 4 | \begin{align*} 5 | \pi(\theta) 6 | \end{align*} 7 | Let $X_1,...,X_n$. We note $L_n(X_1,...,X_n|\theta)$ the joint probability distribution of $X_1,...,X_n$ conditioned on $\theta$ where $\theta \sim \pi$. This is exactly the likelihood from the frequentist approach. 8 | \subsection{Bayes' formula}. 9 | The {posterior distribution} verifies: 10 | \begin{align*} 11 | \forall \theta \in \Theta, \pi(\theta|X_1,...,X_n) \propto\\ \pi(\theta)L_n(X_1,...,X_n | \theta) 12 | \end{align*} 13 | The constant is the normalization factor to ensure the result is a proper distribution, and does not depend on $\theta$: 14 | \begin{align*} 15 | \pi(\theta|X_1,...,X_n) = \frac{\pi(\theta)L_n(X_1,...,X_n | \theta)}{\int_\Theta\pi(\theta)L_n(X_1,...,X_n | \theta)d\theta} 16 | \end{align*} 17 | We can often use an \textbf{improper prior}, i.e. a prior that is not a proper probability distribution (whose integral diverges), and still get a proper posterior. For example, the improper prior $\pi(\theta) = 1$ on $\Theta$ gives the likelihood as a posterior. 18 | \subsection{Jeffreys Prior} 19 | \begin{align*} 20 | \pi_J(\theta) \propto \sqrt{det I(\theta)} 21 | \end{align*} 22 | where $I(\theta)$ is the Fisher information. This prior is \textbf{invariant by reparameterization}, which means that if we have $\eta = \phi(\theta)$, then the same prior gives us a probability distribution for $\eta$ verifying: 23 | \begin{align*} 24 | \tilde\pi_J(\eta) \propto \sqrt{det \tilde I(\eta)} 25 | \end{align*} 26 | The change of parameter follows the following formula: 27 | \begin{align*} 28 | \tilde\pi_J(\eta) = det(\nabla \phi^{-1}(\eta)) \pi_J(\phi^{-1}(\eta)) 29 | \end{align*} 30 | \subsection{Bayesian confidence region} 31 | Let $\alpha \in (0, 1)$. A *Bayesian confidence region with level $\alpha$* is a random subset $\mathcal{R} \subset \Theta$ depending on $X_1,...,X_n$ (and the prior $\pi$) such that: 32 | \begin{align*} 33 | P[\theta \in \mathcal{R} | X_1,...,X_n] \geq 1 - \alpha 34 | \end{align*} 35 | Bayesian confidence region and confidence interval are \textbf{distinct notions}.The Bayesian framework can be used to estimate the true underlying parameter. In that case, it is used to build a new class of estimators, based on the posterior distribution. 36 | \subsection{Bayes estimator} 37 | 38 | \textbf{posterior mean}: 39 | \begin{align*} 40 | \hat{\theta}_{(\pi)} = \int_\Theta\theta\pi(\theta | X_1,...,X_n)d\theta 41 | \end{align*} 42 | \textbf{Maximum a posteriori estimator (MAP):} 43 | \begin{align*} 44 | \hat{\theta}^{MAP}_{(\pi)} = argmax_{\theta\in\Theta}\pi(\theta | X_1,...,X_n) 45 | \end{align*} 46 | The MAP is equivalent to the MLE, if the prior is uniform. -------------------------------------------------------------------------------- /content/beweise.tex: -------------------------------------------------------------------------------- 1 | \section{Beweistechniken} 2 | \subsection*{Direkter Beweis} 3 | Beim direkten Beweis wird Schritt für Schritt mittels \emph{Wenn, Dann} bewiesen. 4 | \subsection*{Kontraposition} 5 | Da $p\Rightarrow q\equiv \neg q\Rightarrow \neg p$ kann man die Aussage auch mittels Kontraposition beweisen. 6 | \subsection*{Widerspruch} 7 | Beim Widerspruchsbeweis wird Gegenteil angenommen und in einen Widerspruch geführt. 8 | Also muss die ursprüngliche Aussage wahr sein. 9 | \subsection*{Äquivalenzbeweis} 10 | Beweis über zeigen der Hin- und Rückrichtung. 11 | \subsection*{Fallunterscheidung} 12 | Beweis aller möglichen Fälle. 13 | \subsection*{Induktionsbeweis} 14 | Induktionsanfang ($n$ kleinste Zahl):\\ 15 | Induktionsbehauptung: Aussage gelte für beliebiges aber festes $n\in \mathbb{N}$ mit $n\geq$ kleinste Zahl.\\ 16 | Induktionsschluss ($n \Longrightarrow n+1$): Zu zeigen ist also $n+1$ einsetzen $\Rightarrow$ Aussage gilt auch, 17 | \emph{mit Benutzung von Induktionsbehauptung}. 18 | %\subsection*{VI an Rekursiver Funktion} -------------------------------------------------------------------------------- /content/bool.tex: -------------------------------------------------------------------------------- 1 | \section{Algebraische Strukturen} 2 | \subsection*{Monoid} 3 | Ein Monoid ist ein Tupel $(M,*,e)$ bestehend aus einer Menge $M$, einer zweistelligen 4 | Verknüpfung $*:M\times M\to M,\quad (a,b)\mapsto a*b$ und einem neutralem Element $e\in M$. 5 | Außerdem muss gelten:\\ 6 | Assoziativität:\\ 7 | (M1) $\forall a,b,c\in M:(a*b)*c=a*(b*c)$\\ 8 | neutrales Element:\\ 9 | (M2) $\forall a\in M:e*a=a*e=a$\\ 10 | \emph{Beispiel:}\\ 11 | $(\mathbb{N}_0,+,0)$ ist Monoid\\ 12 | $(\mathbb{N},\cdot,1)$ ist Monoid 13 | \subsection*{Gruppe} 14 | Eine Gruppe ist ein Tupel $(G,*,e,a^{-1})$ bestehend aus einer Menge $G$, einer zweistelligen Verknüpfung $*:G\times G\to G,\quad (a,b)\mapsto a*b$, einem neutralem Element $e\in G$ und einem inversen Element $a^{-1}\in G$. 15 | Außerdem muss gelten:\\ 16 | Abgeschlossenheit:\\ 17 | (G1) $\forall a,b\in G:(a*b)\in G$\\ 18 | Assoziativität:\\ 19 | (G2) $\forall a,b,c\in G:(a*b)*c=a*(b*c)$\\ 20 | neutrales Element:\\ 21 | (G3) $\forall a\in G:e*a=a*e=a$\\ 22 | inverses Element:\\ 23 | (G4) $\forall a\in G\exists a^{-1}\in G:a*a^{-1}=a^{-1}*a=e$\\ 24 | Für abelsche Gruppe:\\ 25 | Kommutativität:\\ 26 | (G5) $\forall a,b\in G:a*b=b*a$\\ 27 | \emph{Beispiel:}\\ 28 | $(\mathbb{Z},+,0,-a)$ ist Gruppe\\ 29 | $(\mathbb{Q}\setminus\{0\},\cdot,1,\frac{1}{a})$ ist Gruppe 30 | \subsection*{Körper} 31 | Ein Körper ist ein Tupel $(K,+,\cdot)$, bestehend aus einer Menge $K$ und zwei 32 | zweistelligen Operationen $+$ und $\cdot$ auf $K$.\\ 33 | Es muss gelten:\\ 34 | (K1) $(K,+)$ ist abelsche Gruppe mit neutralem Element $0$.\\ 35 | (K2) $(K\setminus \{0\},\cdot)$ ist abelsche Gruppe mit neutralem Element $1$.\\ 36 | (K3) Distributivität:\\ 37 | $\forall a,b,c\in K:a\cdot (b+c)=a\cdot b+a\cdot c$\\ 38 | \emph{Beispiel:}\\ 39 | $(\mathbb{Q},+,\cdot)$ ist Körper\\ 40 | $(\mathbb{R},+,\cdot)$ ist Körper 41 | %\subsection*{Schaltfunktionen} 42 | %\subsection*{Boolsche Algebra} -------------------------------------------------------------------------------- /content/def.aux: -------------------------------------------------------------------------------- 1 | \relax 2 | \providecommand\zref@newlabel[2]{} 3 | \@setckpt{content/def}{ 4 | \setcounter{page}{1} 5 | \setcounter{equation}{0} 6 | \setcounter{enumi}{0} 7 | \setcounter{enumii}{0} 8 | \setcounter{enumiii}{0} 9 | \setcounter{enumiv}{0} 10 | \setcounter{footnote}{0} 11 | \setcounter{mpfootnote}{0} 12 | \setcounter{part}{0} 13 | \setcounter{section}{0} 14 | \setcounter{subsection}{0} 15 | \setcounter{subsubsection}{0} 16 | \setcounter{paragraph}{0} 17 | \setcounter{subparagraph}{0} 18 | \setcounter{figure}{0} 19 | \setcounter{table}{0} 20 | \setcounter{parentequation}{0} 21 | \setcounter{mdf@globalstyle@cnt}{1} 22 | \setcounter{mdfcountframes}{0} 23 | \setcounter{mdf@env@i}{0} 24 | \setcounter{mdf@env@ii}{0} 25 | \setcounter{mdf@zref@counter}{0} 26 | \setcounter{AM@survey}{0} 27 | } 28 | -------------------------------------------------------------------------------- /content/def.tex: -------------------------------------------------------------------------------- 1 | \definecolor{myblue}{cmyk}{1,.72,0,.38} 2 | 3 | \def\firstcircle{(0,0) circle (1.5cm)} 4 | \def\secondcircle{(0:2cm) circle (1.5cm)} 5 | 6 | \colorlet{circle edge}{myblue} 7 | \colorlet{circle area}{myblue!5} 8 | 9 | \tikzset{filled/.style={fill=circle area, draw=circle edge, thick}, 10 | outline/.style={draw=circle edge, thick}} 11 | 12 | \pgfdeclarelayer{background} 13 | \pgfsetlayers{background,main} 14 | 15 | \everymath\expandafter{\the\everymath \color{myblue}} 16 | %\everydisplay\expandafter{\the\everydisplay \color{myblue}} 17 | 18 | \let\displaystyle\textstyle 19 | 20 | \renewcommand{\baselinestretch}{.8} 21 | \pagestyle{empty} 22 | 23 | \global\mdfdefinestyle{header}{% 24 | linecolor=gray,linewidth=1pt,% 25 | leftmargin=0mm,rightmargin=0mm,skipbelow=0mm,skipabove=0mm, 26 | } 27 | 28 | \newcommand{\header}{ 29 | \begin{mdframed}[style=header] 30 | \footnotesize 31 | \sffamily 32 | Capstone-Cheatsheet Statistics 1\\ 33 | by Blechturm,~Page~\thepage~of~2 34 | \end{mdframed} 35 | } 36 | 37 | \makeatletter % Author: https://tex.stackexchange.com/questions/218587/how-to-set-one-header-for-each-page-using-multicols 38 | \renewcommand{\section}{\@startsection{section}{1}{0mm}% 39 | {.2ex}% 40 | {.2ex}%x 41 | {\color{black}\sffamily\small\bfseries}} 42 | \renewcommand{\subsection}{\@startsection{subsection}{1}{0mm}% 43 | {.2ex}% 44 | {.2ex}%x 45 | {\sffamily\bfseries}} 46 | 47 | 48 | 49 | \def\multi@column@out{% 50 | \ifnum\outputpenalty <-\@M 51 | \speci@ls \else 52 | \ifvoid\colbreak@box\else 53 | \mult@info\@ne{Re-adding forced 54 | break(s) for splitting}% 55 | \setbox\@cclv\vbox{% 56 | \unvbox\colbreak@box 57 | \penalty-\@Mv\unvbox\@cclv}% 58 | \fi 59 | \splittopskip\topskip 60 | \splitmaxdepth\maxdepth 61 | \dimen@\@colroom 62 | \divide\skip\footins\col@number 63 | \ifvoid\footins \else 64 | \leave@mult@footins 65 | \fi 66 | \let\ifshr@kingsaved\ifshr@king 67 | \ifvbox \@kludgeins 68 | \advance \dimen@ -\ht\@kludgeins 69 | \ifdim \wd\@kludgeins>\z@ 70 | \shr@nkingtrue 71 | \fi 72 | \fi 73 | \process@cols\mult@gfirstbox{% 74 | %%%%% START CHANGE 75 | \ifnum\count@=\numexpr\mult@rightbox+2\relax 76 | \setbox\count@\vsplit\@cclv to \dimexpr \dimen@-1cm\relax 77 | \setbox\count@\vbox to \dimen@{\vbox to 1cm{\header}\unvbox\count@\vss}% 78 | \else 79 | \setbox\count@\vsplit\@cclv to \dimen@ 80 | \fi 81 | %%%%% END CHANGE 82 | \set@keptmarks 83 | \setbox\count@ 84 | \vbox to\dimen@ 85 | {\unvbox\count@ 86 | \remove@discardable@items 87 | \ifshr@nking\vfill\fi}% 88 | }% 89 | \setbox\mult@rightbox 90 | \vsplit\@cclv to\dimen@ 91 | \set@keptmarks 92 | \setbox\mult@rightbox\vbox to\dimen@ 93 | {\unvbox\mult@rightbox 94 | \remove@discardable@items 95 | \ifshr@nking\vfill\fi}% 96 | \let\ifshr@king\ifshr@kingsaved 97 | \ifvoid\@cclv \else 98 | \unvbox\@cclv 99 | \ifnum\outputpenalty=\@M 100 | \else 101 | \penalty\outputpenalty 102 | \fi 103 | \ifvoid\footins\else 104 | \PackageWarning{multicol}% 105 | {I moved some lines to 106 | the next page.\MessageBreak 107 | Footnotes on page 108 | \thepage\space might be wrong}% 109 | \fi 110 | \ifnum \c@tracingmulticols>\thr@@ 111 | \hrule\allowbreak \fi 112 | \fi 113 | \ifx\@empty\kept@firstmark 114 | \let\firstmark\kept@topmark 115 | \let\botmark\kept@topmark 116 | \else 117 | \let\firstmark\kept@firstmark 118 | \let\botmark\kept@botmark 119 | \fi 120 | \let\topmark\kept@topmark 121 | \mult@info\tw@ 122 | {Use kept top mark:\MessageBreak 123 | \meaning\kept@topmark 124 | \MessageBreak 125 | Use kept first mark:\MessageBreak 126 | \meaning\kept@firstmark 127 | \MessageBreak 128 | Use kept bot mark:\MessageBreak 129 | \meaning\kept@botmark 130 | \MessageBreak 131 | Produce first mark:\MessageBreak 132 | \meaning\firstmark 133 | \MessageBreak 134 | Produce bot mark:\MessageBreak 135 | \meaning\botmark 136 | \@gobbletwo}% 137 | \setbox\@cclv\vbox{\unvbox\partial@page 138 | \page@sofar}% 139 | \@makecol\@outputpage 140 | \global\let\kept@topmark\botmark 141 | \global\let\kept@firstmark\@empty 142 | \global\let\kept@botmark\@empty 143 | \mult@info\tw@ 144 | {(Re)Init top mark:\MessageBreak 145 | \meaning\kept@topmark 146 | \@gobbletwo}% 147 | \global\@colroom\@colht 148 | \global \@mparbottom \z@ 149 | \process@deferreds 150 | \@whilesw\if@fcolmade\fi{\@outputpage 151 | \global\@colroom\@colht 152 | \process@deferreds}% 153 | \mult@info\@ne 154 | {Colroom:\MessageBreak 155 | \the\@colht\space 156 | after float space removed 157 | = \the\@colroom \@gobble}% 158 | \set@mult@vsize \global 159 | \fi} 160 | 161 | \makeatother 162 | \setlength{\parindent}{0pt} -------------------------------------------------------------------------------- /content/estimators.tex: -------------------------------------------------------------------------------- 1 | \section{Estimators} 2 | 3 | A \textbf{statistic} is any measurable function calculated with the data ($\bar{X_n}, max(X_i),$ etc).\\ 4 | 5 | An \textbf{estimator} $\hat{\theta }_ n$ of $\theta$ is any statistic which does not depend on $\theta$.\\ 6 | 7 | Estimators are random variables if they depend on the data (= realizations of random variables).\\ 8 | 9 | An estimator $\hat{\theta }_ n$ is \textbf{weakly consistent} if: $\displaystyle \lim _{n \to \infty } \hat{\theta }_ n = \theta$ or $ \hat{\theta}_n \xrightarrow[n \rightarrow \infty]{P} \mathbb{E}[g(X)]$. If the convergence is almost surely it is \textbf{strongly consistent}.\\ 10 | \textbf{Asymptotic normality of an estimator:} 11 | \[\sqrt(n) (\hat{\theta}_n-\theta) \xrightarrow[n \rightarrow \infty]{(d)} N(0,\sigma^2)\] 12 | $\sigma^2$ is called the \textbf{Asymptotic Variance} of the estimator $\hat{\theta}_n$. In the case of the sample mean it is the same variance as as the single $X_i$.\\ 13 | If the estimator is a function of the sample mean the \textbf{Delta Method} is needed to compute the asymptotic variance.\textbf{Asymptotic Variance} $\neq$ Variance of an estimator.\\ 14 | \textbf{Bias of an estimator:} 15 | \[Bias(\hat{\theta}_n) = \mathbb{E}[\hat{\theta_n}] - \theta\] 16 | \textbf{Quadratic risk of an estimator} 17 | \begin{align*} 18 | R(\hat{\theta}_n) & = \mathbb{E}[(\hat{\theta}_n-\theta)^2]\\ 19 | & = Bias^2 + Variance 20 | \end{align*} -------------------------------------------------------------------------------- /content/expectation_variance.log: -------------------------------------------------------------------------------- 1 | This is pdfTeX, Version 3.14159265-2.6-1.40.19 (MiKTeX 2.9.6930 64-bit) (preloaded format=pdflatex 2019.1.30) 11 MAR 2019 16:07 2 | entering extended mode 3 | **./expectation_variance.tex 4 | (expectation_variance.tex 5 | LaTeX2e <2018-12-01> 6 | ! Undefined control sequence. 7 | l.1 \section 8 | {Expectation and Variance} 9 | The control sequence at the end of the top line 10 | of your error message was never \def'ed. If you have 11 | misspelled it (e.g., `\hobx'), type `I' and the correct 12 | spelling (e.g., `I\hbox'). Otherwise just continue, 13 | and I'll forget about whatever was undefined. 14 | 15 | 16 | ! LaTeX Error: Missing \begin{document}. 17 | 18 | See the LaTeX manual or LaTeX Companion for explanation. 19 | Type H for immediate help. 20 | ... 21 | 22 | l.1 \section{E 23 | xpectation and Variance} 24 | You're in trouble here. Try typing to proceed. 25 | If that doesn't work, type X to quit. 26 | 27 | Missing character: There is no E in font nullfont! 28 | Missing character: There is no x in font nullfont! 29 | Missing character: There is no p in font nullfont! 30 | Missing character: There is no e in font nullfont! 31 | Missing character: There is no c in font nullfont! 32 | Missing character: There is no t in font nullfont! 33 | Missing character: There is no a in font nullfont! 34 | Missing character: There is no t in font nullfont! 35 | Missing character: There is no i in font nullfont! 36 | Missing character: There is no o in font nullfont! 37 | Missing character: There is no n in font nullfont! 38 | Missing character: There is no a in font nullfont! 39 | Missing character: There is no n in font nullfont! 40 | Missing character: There is no d in font nullfont! 41 | Missing character: There is no V in font nullfont! 42 | Missing character: There is no a in font nullfont! 43 | Missing character: There is no r in font nullfont! 44 | Missing character: There is no i in font nullfont! 45 | Missing character: There is no a in font nullfont! 46 | Missing character: There is no n in font nullfont! 47 | Missing character: There is no c in font nullfont! 48 | Missing character: There is no e in font nullfont! 49 | ! Undefined control sequence. 50 | l.2 \subsection 51 | *{Expectation} 52 | The control sequence at the end of the top line 53 | of your error message was never \def'ed. If you have 54 | misspelled it (e.g., `\hobx'), type `I' and the correct 55 | spelling (e.g., `I\hbox'). Otherwise just continue, 56 | and I'll forget about whatever was undefined. 57 | 58 | Missing character: There is no * in font nullfont! 59 | Missing character: There is no E in font nullfont! 60 | Missing character: There is no x in font nullfont! 61 | Missing character: There is no p in font nullfont! 62 | Missing character: There is no e in font nullfont! 63 | Missing character: There is no c in font nullfont! 64 | Missing character: There is no t in font nullfont! 65 | Missing character: There is no a in font nullfont! 66 | Missing character: There is no t in font nullfont! 67 | Missing character: There is no i in font nullfont! 68 | Missing character: There is no o in font nullfont! 69 | Missing character: There is no n in font nullfont! 70 | ! Undefined control sequence. 71 | l.3 \subsection 72 | *{Variance} 73 | The control sequence at the end of the top line 74 | of your error message was never \def'ed. If you have 75 | misspelled it (e.g., `\hobx'), type `I' and the correct 76 | spelling (e.g., `I\hbox'). Otherwise just continue, 77 | and I'll forget about whatever was undefined. 78 | 79 | Missing character: There is no * in font nullfont! 80 | Missing character: There is no V in font nullfont! 81 | Missing character: There is no a in font nullfont! 82 | Missing character: There is no r in font nullfont! 83 | Missing character: There is no i in font nullfont! 84 | Missing character: There is no a in font nullfont! 85 | Missing character: There is no n in font nullfont! 86 | Missing character: There is no c in font nullfont! 87 | Missing character: There is no e in font nullfont! 88 | ! Undefined control sequence. 89 | l.4 \subsection 90 | *{Covariance} 91 | The control sequence at the end of the top line 92 | of your error message was never \def'ed. If you have 93 | misspelled it (e.g., `\hobx'), type `I' and the correct 94 | spelling (e.g., `I\hbox'). Otherwise just continue, 95 | and I'll forget about whatever was undefined. 96 | 97 | Missing character: There is no * in font nullfont! 98 | Missing character: There is no C in font nullfont! 99 | Missing character: There is no o in font nullfont! 100 | Missing character: There is no v in font nullfont! 101 | Missing character: There is no a in font nullfont! 102 | Missing character: There is no r in font nullfont! 103 | Missing character: There is no i in font nullfont! 104 | Missing character: There is no a in font nullfont! 105 | Missing character: There is no n in font nullfont! 106 | Missing character: There is no c in font nullfont! 107 | Missing character: There is no e in font nullfont! 108 | ! Undefined control sequence. 109 | l.5 \subsection 110 | *{Variance and expectation of mean of n iid random variables} 111 | The control sequence at the end of the top line 112 | of your error message was never \def'ed. If you have 113 | misspelled it (e.g., `\hobx'), type `I' and the correct 114 | spelling (e.g., `I\hbox'). Otherwise just continue, 115 | and I'll forget about whatever was undefined. 116 | 117 | Missing character: There is no * in font nullfont! 118 | Missing character: There is no V in font nullfont! 119 | Missing character: There is no a in font nullfont! 120 | Missing character: There is no r in font nullfont! 121 | Missing character: There is no i in font nullfont! 122 | Missing character: There is no a in font nullfont! 123 | Missing character: There is no n in font nullfont! 124 | Missing character: There is no c in font nullfont! 125 | Missing character: There is no e in font nullfont! 126 | Missing character: There is no a in font nullfont! 127 | Missing character: There is no n in font nullfont! 128 | Missing character: There is no d in font nullfont! 129 | Missing character: There is no e in font nullfont! 130 | Missing character: There is no x in font nullfont! 131 | Missing character: There is no p in font nullfont! 132 | Missing character: There is no e in font nullfont! 133 | Missing character: There is no c in font nullfont! 134 | Missing character: There is no t in font nullfont! 135 | Missing character: There is no a in font nullfont! 136 | Missing character: There is no t in font nullfont! 137 | Missing character: There is no i in font nullfont! 138 | Missing character: There is no o in font nullfont! 139 | Missing character: There is no n in font nullfont! 140 | Missing character: There is no o in font nullfont! 141 | Missing character: There is no f in font nullfont! 142 | Missing character: There is no m in font nullfont! 143 | Missing character: There is no e in font nullfont! 144 | Missing character: There is no a in font nullfont! 145 | Missing character: There is no n in font nullfont! 146 | Missing character: There is no o in font nullfont! 147 | Missing character: There is no f in font nullfont! 148 | Missing character: There is no n in font nullfont! 149 | Missing character: There is no i in font nullfont! 150 | Missing character: There is no i in font nullfont! 151 | Missing character: There is no d in font nullfont! 152 | Missing character: There is no r in font nullfont! 153 | Missing character: There is no a in font nullfont! 154 | Missing character: There is no n in font nullfont! 155 | Missing character: There is no d in font nullfont! 156 | Missing character: There is no o in font nullfont! 157 | Missing character: There is no m in font nullfont! 158 | Missing character: There is no v in font nullfont! 159 | Missing character: There is no a in font nullfont! 160 | Missing character: There is no r in font nullfont! 161 | Missing character: There is no i in font nullfont! 162 | Missing character: There is no a in font nullfont! 163 | Missing character: There is no b in font nullfont! 164 | Missing character: There is no l in font nullfont! 165 | Missing character: There is no e in font nullfont! 166 | Missing character: There is no s in font nullfont! 167 | Missing character: There is no L in font nullfont! 168 | Missing character: There is no e in font nullfont! 169 | Missing character: There is no t in font nullfont! 170 | LaTeX Font Info: External font `cmex10' loaded for size 171 | (Font) <7> on input line 6. 172 | LaTeX Font Info: External font `cmex10' loaded for size 173 | (Font) <5> on input line 6. 174 | Missing character: There is no w in font nullfont! 175 | Missing character: There is no h in font nullfont! 176 | Missing character: There is no e in font nullfont! 177 | Missing character: There is no r in font nullfont! 178 | Missing character: There is no e in font nullfont! 179 | Missing character: There is no a in font nullfont! 180 | Missing character: There is no n in font nullfont! 181 | Missing character: There is no d in font nullfont! 182 | Missing character: There is no f in font nullfont! 183 | Missing character: There is no o in font nullfont! 184 | Missing character: There is no r in font nullfont! 185 | Missing character: There is no a in font nullfont! 186 | Missing character: There is no l in font nullfont! 187 | Missing character: There is no l in font nullfont! 188 | Missing character: There is no . in font nullfont! 189 | Missing character: There is no T in font nullfont! 190 | Missing character: There is no h in font nullfont! 191 | Missing character: There is no e in font nullfont! 192 | Missing character: There is no m in font nullfont! 193 | Missing character: There is no e in font nullfont! 194 | Missing character: There is no a in font nullfont! 195 | Missing character: There is no n in font nullfont! 196 | Missing character: There is no i in font nullfont! 197 | Missing character: There is no s in font nullfont! 198 | Missing character: There is no : in font nullfont! 199 | Missing character: There is no . in font nullfont! 200 | Missing character: There is no T in font nullfont! 201 | Missing character: There is no h in font nullfont! 202 | Missing character: There is no e in font nullfont! 203 | Missing character: There is no v in font nullfont! 204 | Missing character: There is no a in font nullfont! 205 | Missing character: There is no r in font nullfont! 206 | Missing character: There is no i in font nullfont! 207 | Missing character: There is no a in font nullfont! 208 | Missing character: There is no n in font nullfont! 209 | Missing character: There is no c in font nullfont! 210 | Missing character: There is no e in font nullfont! 211 | Missing character: There is no o in font nullfont! 212 | Missing character: There is no f in font nullfont! 213 | Missing character: There is no t in font nullfont! 214 | Missing character: There is no h in font nullfont! 215 | Missing character: There is no e in font nullfont! 216 | Missing character: There is no m in font nullfont! 217 | Missing character: There is no e in font nullfont! 218 | Missing character: There is no a in font nullfont! 219 | Missing character: There is no n in font nullfont! 220 | Missing character: There is no i in font nullfont! 221 | Missing character: There is no s in font nullfont! 222 | Missing character: There is no : in font nullfont! 223 | Missing character: There is no . in font nullfont! 224 | Missing character: There is no T in font nullfont! 225 | Missing character: There is no h in font nullfont! 226 | Missing character: There is no e in font nullfont! 227 | Missing character: There is no e in font nullfont! 228 | Missing character: There is no x in font nullfont! 229 | Missing character: There is no p in font nullfont! 230 | Missing character: There is no e in font nullfont! 231 | Missing character: There is no c in font nullfont! 232 | Missing character: There is no t in font nullfont! 233 | Missing character: There is no a in font nullfont! 234 | Missing character: There is no t in font nullfont! 235 | Missing character: There is no i in font nullfont! 236 | Missing character: There is no o in font nullfont! 237 | Missing character: There is no n in font nullfont! 238 | Missing character: There is no o in font nullfont! 239 | Missing character: There is no f in font nullfont! 240 | Missing character: There is no t in font nullfont! 241 | Missing character: There is no h in font nullfont! 242 | Missing character: There is no e in font nullfont! 243 | Missing character: There is no m in font nullfont! 244 | Missing character: There is no e in font nullfont! 245 | Missing character: There is no a in font nullfont! 246 | Missing character: There is no n in font nullfont! 247 | Missing character: There is no i in font nullfont! 248 | Missing character: There is no s in font nullfont! 249 | Missing character: There is no : in font nullfont! 250 | Missing character: There is no . in font nullfont! 251 | ) 252 | ! Emergency stop. 253 | <*> ./expectation_variance.tex 254 | 255 | *** (job aborted, no legal \end found) 256 | 257 | 258 | Here is how much of TeX's memory you used: 259 | 9 strings out of 492946 260 | 374 string characters out of 3126266 261 | 58009 words of memory out of 3000000 262 | 4006 multiletter control sequences out of 15000+200000 263 | 3640 words of font info for 14 fonts, out of 3000000 for 9000 264 | 1141 hyphenation exceptions out of 8191 265 | 18i,3n,12p,394b,56s stack positions out of 5000i,500n,10000p,200000b,50000s 266 | ! ==> Fatal error occurred, no output PDF file produced! 267 | -------------------------------------------------------------------------------- /content/expectation_variance.tex: -------------------------------------------------------------------------------- 1 | \section{Expectation} 2 | 3 | $\mathbb{E}\left[X\right]=\int_{-inf}^{+inf}{x \cdot f_X\left(x\right)\ \ dx}$\\ 4 | 5 | $\mathbb{E}\left[g\left(X\right)\right]=\int_{-inf}^{+inf}{g\left(x\right) \cdot f_X\left(x\right)dx}$\\ 6 | 7 | $\mathbb{E}\left[X\middle| Y=y\right]=\int_{-inf}^{+inf}{x \cdot { f}_{X|Y}\left(x|y\right)\ \ dx}$\\ 8 | 9 | Integration limits only have to be over the support of the pdf. Discrete r.v. same as continuous but with sums and pmfs.\\ 10 | 11 | Total expectation theorem:\\ 12 | 13 | $\mathbb{E}\left[X\right]=\int_{-inf}^{+inf}{f_Y\left(y\right)\cdot\mathbb{E}\left[X\middle| Y=y\right]dy}$\\ 14 | 15 | Law of iterated expectation:\\ 16 | 17 | $\mathbb E[Y] = \mathbb E[\mathbb E[Y\rvert {X}]]$\\ 18 | 19 | Expectation of constant $a$:\\ 20 | 21 | $\mathbb{E}[a] = a$\\ 22 | 23 | Product of \textbf{independent} r.vs $X$ and $Y$ :\\ 24 | 25 | $\mathbb{E}[X \cdot Y] = \mathbb{E}[X] \cdot \mathbb{E}[Y]$\\ 26 | 27 | Product of \textbf{dependent} r.vs $X$ and $Y$ :\\ 28 | 29 | $\mathbb{E}[X \cdot Y] \neq \mathbb{E}[X] \cdot \mathbb{E}[Y]$\\ 30 | 31 | $\mathbb{E}[X \cdot Y] = \mathbb{E}[\mathbb{E}[Y \cdot X|Y]] = \mathbb{E}[Y \cdot \mathbb{E}[X|Y]]$\\ 32 | 33 | Linearity of Expectation where $a$ and $c$ are given scalars:\\ 34 | 35 | $\mathbb{E}[aX + c Y] = a\mathbb{E}[X] + c\mathbb{E}[Y]$\\ 36 | 37 | If Variance of $X$ is known:\\ 38 | 39 | $\mathbb{E}[X^2] = var(X) - \mathbb{E}[X]$\\ 40 | 41 | 42 | \section{Variance} 43 | 44 | Variance is the squared distance from the mean.\\ 45 | 46 | $Var(X)=\mathbb{E}[(X-\mathbb{E}(X))^2]$\\ 47 | 48 | $Var\left(X\right)=\mathbb{E}\left[X^2\right]-\left(\mathbb{E}\left[X\right]\right)^2$\\ 49 | 50 | Variance of a product with constant $a$:\\ 51 | 52 | $Var(aX)=a^2 Var\left(X\right)$\\ 53 | 54 | Variance of sum of two \textbf{dependent} r.v.:\\ 55 | 56 | $Var(X + Y)=Var(X)+Var(Y)+2Cov(X,Y)$\\ 57 | 58 | Variance of sum/difference of two \textbf{independent} r.v.:\\ 59 | 60 | $Var(X + Y)=Var(X)+Var(Y)$\\ 61 | 62 | $Var(X - Y)=Var(X)+Var(Y)$\\ 63 | 64 | \section{Covariance} 65 | 66 | The Covariance is a measure of how much the values of each of two correlated random variables determine each other\\ 67 | 68 | $Cov(X,Y) = \mathbb E[(X - \mu _ X)(Y - \mu _ Y)]$ \\ 69 | 70 | $Cov(X,Y) = \mathbb E[XY] - \mathbb E[X]\mathbb E[Y] $\\ 71 | 72 | $ Cov(X,Y)= \displaystyle \mathbb E[(X)(Y-\mu _ Y)]$\\ 73 | 74 | Possible notations:\\ 75 | 76 | $Cov(X,Y) = \sigma(X,Y) = \sigma_ {(X,Y)}$\\ 77 | 78 | Covariance is commutative:\\ 79 | 80 | $Cov(X,Y) = Cov(Y,X)$\\ 81 | 82 | Covariance with of r.v. with itself is variance:\\ 83 | 84 | $Cov(X,X) = \mathbb E[(X - \mu _ X)^2] = Var(X)$\\ 85 | 86 | Useful properties:\\ 87 | 88 | $Cov(aX + h,bY + c)= abCov(X,Y)$\\ 89 | 90 | $Cov(X,X + Y)= Var(X) + cov(X,Y)$\\ 91 | 92 | $\displaystyle Cov(aX+ bY, Z) \displaystyle = aCov(X,Z) + bCov(Y,Z)$\\ 93 | 94 | If $Cov(X,Y) = 0$, we say that X and Y are uncorrelated. If $X$ and $Y$ are independent, their Covariance is zero. The converse is not always true. It is only true if $X$ and $Y$ form a gaussian vector, ie. any linear combination $\alpha X + \beta Y$ is gaussian for all $(\alpha,\beta) \in \mathbb{R}^2$ without $\{0,0\}$. 95 | 96 | \section{correlation coefficient} 97 | 98 | $\rho(X,Y) = \frac{Cov(X,Y)}{\sqrt{Var(X}Var(Y}$ -------------------------------------------------------------------------------- /content/formeln.tex: -------------------------------------------------------------------------------- 1 | \section{Formelsammlung} 2 | \subsection*{Binomische Formeln} 3 | $(a+b)^2 = a^2 + 2ab + b^2$\\ 4 | $(a-b)^2 = a^2 - 2ab + b^2$\\ 5 | $(a+b) \cdot (a-b) = a^2 - b^2$\\ 6 | $(a \pm b)^3 = a^3 \pm 3 a^2 b + 3 a b^2 \pm b^3$\\ 7 | $(a \pm b)^4 = a^4 \pm 4 a^3 b + 6 a^2 b^2 \pm 4 a b^3 + b^4$\\ 8 | $(a \pm b)^5 = a^5 \pm 5 a^4 b + 10 a^3 b^2 \pm 10 a^2 b^3 + 5 a b^4 \pm b^5$ 9 | \subsection*{Potenzgesetze} 10 | $a^{-n}=\frac{1}{a^n}$\\ 11 | $a^m\cdot a^n=a^{m+n}$\\ 12 | $\frac{a^m}{a^n}=a^{m-n}$\\ 13 | $(a^m)^n=a^{m\cdot n}$\\ 14 | $a^n\cdot b^n=(a\cdot b)^n$\\ 15 | $\frac{a^n}{b^n}=(\frac{a}{b})^n$ 16 | \subsection*{Wurzelgesetze} 17 | $\sqrt[n]{a}=a^{\frac{1}{n}}$\\ 18 | $\sqrt[n]{a^m}=(\sqrt[n]{a})^m=a^{\frac{m}{n}}$\\ 19 | $\sqrt[n]{a}\cdot\sqrt[n]{b}=\sqrt[n]{a\cdot b}$\\ 20 | $\frac{\sqrt[n]{a}}{\sqrt[n]{b}}=\sqrt[n]{\frac{a}{b}}$\\ 21 | $\sqrt[n]{\sqrt[m]{a}}=\sqrt[{n\cdot m}]{a}$ 22 | \subsection*{Summeneigenschaften} 23 | %$\sum_{i=1}^n c=n\cdot c$\\ 24 | %$\sum_{i=m}^n c=(n-m+1)\cdot c$\\ 25 | $\sum_{i=m}^n c\cdot a_i=c\cdot \sum_{i=m}^n a_i$\\ 26 | $\sum_{i=m}^n (a_i+b_i)=\sum_{i=m}^n a_i + \sum_{i=m}^n b_i$ 27 | \subsection*{Summenformeln} 28 | Gaußsche Summenformel:\\ 29 | $\sum_{i=1}^n i=\frac{n(n+1)}{2}$\\ 30 | Geometrische Reihe:\\ 31 | $\sum_{i=1}^n q^i=\frac{1-q^{n+1}}{1-q}$\\ 32 | Potenzsummen:\\ 33 | $\sum_{i=1}^n i^2=\frac{n(n+1)(2n+1)}{6}$\\ 34 | $\sum_{i=1}^n i^3=\frac{n^2(n+1)^2}{4}$ 35 | \subsection*{Potenzmengen} 36 | $ \mathcal P(\emptyset) = \{ \emptyset \}\mathcal P(\{ a \}) = \bigl\{ \emptyset, \{ a \} \bigr\}\\ 37 | \mathcal P(\{ a, b \}) = \bigl\{ \emptyset, \{ a \}, \{ b \}, \{ a, b \} \bigr\}\\ 38 | \mathcal P(\{ a, b, c \}) = \bigl\{ \emptyset, \{ a \}, \{ b \}, \{ c \}, \{ a, b \}, \{ a, c \}, \{ b, c \}, \{ a, b, c \} \bigr\}\\ 39 | \mathcal P(\mathcal P(\emptyset)) = \bigl\{ \emptyset, \{\emptyset\}\bigr\}\\ 40 | \mathcal P(\mathcal P(\{a\})) = \bigl\{ \emptyset, \{\emptyset\} , \{\{a\}\} , \{\emptyset , \{a\}\} \bigr\}$ -------------------------------------------------------------------------------- /content/graphalgo.tex: -------------------------------------------------------------------------------- 1 | \section{Graphenalgorithmen} 2 | \subsection*{Kantenlänge} 3 | Jeder Kante wird eine reelle Zahl zugeordnet, die wir als Länge dieser Kante bezeichnen. 4 | Man spricht dann von einem Graphen mit bewerteten Kanten 5 | \subsection*{Algorithmus von Dijkstra} 6 | Dient der \emph{Bestimmung kürzester Wege} von einem fest vorgegebenen Knoten zu allen anderen Knoten 7 | in einem schlichten, zusammenhängenden, gerichteten Graphen mit endlicher Knotenmenge und nicht-negativ bewerteten Kanten 8 | und liefert einen Weg mit einer minimalen Gesamtlänge.\\ 9 | \emph{Algorithmus}\\ 10 | Schreibe Tabelle mit Knoten, Entfernung, Vorgänger, OK\\ 11 | Setze alle Entf. auf $\infty$ außer $Start=0$\\ 12 | Setze Vorg. von $Start=Start$\\ 13 | Setze alle $OK=f$\\ 14 | Starte Algorithmus:\\ 15 | Wiederhole:\\ 16 | Suche unter den Entfernungen die kleinste$=j$, die $OK=f$ ($\Rightarrow$ beim Start also $Start$ selbst)\\ 17 | Setze $j=t$\\ 18 | Suche alle Nachbarknoten $k$ von $j$, die noch nicht $t$ sind.\\ 19 | Wenn die Entfernung größer als $j+k$ setze neue Entf. und setze neuen Vorgänger.\\ 20 | Solange bis noch Knoten mit $OK=f$ 21 | \subsection*{Flussprobleme} 22 | Modellierung von Transport von Gütern (Strom, Container etc.) entlang der Kanten.\\ 23 | $c(e_{ij})=c_{ij}$ ist Kapazität einer Kante. $v_iv_j=e_{ij}$ ist die Menge eines Gutes, 24 | die entlang der Kante transportiert werden kann. 25 | \subsection*{Fluss} 26 | Ein Fluss in $G$ von der Quelle $q=v_1$ zur Senke $s=v_n$ ist eine Funktion $f$, die jeder 27 | Kante $e_{ij}\in E$ eine nicht-negative rationale Zahl zuordnet. 28 | \subsection*{Schnitt} 29 | Seien $X,Y$ bel. Untermengen von Knoten eines Graphen $G$. Dann ist\\ 30 | $A(X,Y)$ die Menge der Kanten, die Knoten aus $X$ mit Knoten $Y$ verbinden.\\ 31 | $A^+(X,Y)$ ist die Menge der Kanten, ausgehend von Knoten aus $X$, die Knoten aus $Y$ 32 | verbinden.\\ 33 | $A^-(X,Y)$ ist die Menge der Kanten, ausgehend von Knoten aus $Y$, die Knoten aus $X$ 34 | verbinden.\\ 35 | Sei $g$ eine Funktion, die den Kanten eines Graphen $G$ nicht-negative Zahlen zuordnet, dann 36 | gilt $g(X,Y)=\sum_{e\in A^+(X,Y)}g(e)$.\\ 37 | Ein \emph{Schnitt} ist eine Menge von Kanten $A(X,\bar{X})$ mit $q\in X$ und $s\in\bar{X}$.\\ 38 | \emph{Beispiel:}\\ 39 | Wähle $X=\{q=v_1,...\}$ und $\bar{X}=\{...,s=v_n\}$ mit $...$ beliebig. Dann ist $A(X,\bar{X})=\{e_{ij},e_{...}\}$ die Menge der Kanten zwischen $X$ und $\bar{X}$.\\ 40 | $A^+(X,\bar{X})=\{e_{ij},e_{...}\}$ die Menge der Kanten aus $X$.\\ 41 | $A^-(X,\bar{X})=\{e_{ij},e_{...}\}$ die Menge der Kanten in $X$ hinein.\\ 42 | Der \emph{Fluss} von den Knoten in $X$ zu den Knoten in $\bar{X}$ ist dann:\\ 43 | $\sum_{e_{ij}\in A^+(X,Y)}f(e_{ij})-\sum_{e_{ij}\in A^-(X,Y)}f(e_{ij})$ Hierbei steht $f(e_{ij})$ für den Fluss, also der hinteren Zahl im Tupel $(x,y)$ an der Kantenbeschriftung.\\ 44 | Die \emph{Kapazität} bestimmt man aus der vorderen Zahl jenen Tupels wie folgt:\\ 45 | $c(X,\bar{X})=\sum_{e_{ij}\in A^+(X,Y)}c(e_{ij})$\\ 46 | \emph{Hinweis:}\\ 47 | Es gibt $2^n$ mögliche Schnitte wenn $n$ die Anzahl der inneren Knoten ist. 48 | \subsection*{Maximaler Fluss} 49 | Ein Fluss $f$, dessen Wert $d$ maximal ist, heißt \emph{maximaler Fluss}.\\ 50 | Ein Fluss dessen Wert $\min\{c(X,\bar{X})\}$ entspricht, ist maximal. 51 | \subsection*{Vergrößernder Weg} 52 | Ein ungerichteter Weg von $q$ nach $s$ heißt \emph{vergrößernd}, wenn 53 | für jede Kante $e_{ij}$ auf dem Weg ihrer Richtung gilt: $f(e_{ij})0$ (Rückwärtskante). 55 | \subsection*{Algorithmus von Ford und Fulkerson} 56 | Initialsiere alle Flüsse mit 0.\\ 57 | Wiederhole:\\ 58 | Suche guten Fluss, der optimal wird und schreibe Tabelle:\\ 59 | $1) q(\bot,\infty) v_x(+q,Anzahl), ... , s(+v_{...},Anzahl)$\\ 60 | Trage Fluss nach Komma ein.\\ 61 | Bis:\\ 62 | Es gibt keinen weiteren vergrößernden Fluss.\\ 63 | Antwort: max. Fluss mit $d=...$\\ 64 | \emph{Hinweis:} Nur soviel durchschicken wie benötigt und evtl. größte Flüsse zuerst. -------------------------------------------------------------------------------- /content/graphen.tex: -------------------------------------------------------------------------------- 1 | \section{Graphentheorie} 2 | \subsection*{Gerichteter Graph} 3 | $G=(V,E)$ wobei $V$ Menge aller Knoten z.B. $V=\{v_0,v_1,v_2,\dots,v_n\}$ und $E\subseteq V\times V$ Menge aller Kanten mit $e=(v,u)$. Hierbei steht $v$ für den Startknoten von $e$ und $u$ ist Endknoten von $e$.\\ 4 | \emph{Hinweis:}\\ 5 | Ist die Kantenmenge $E$ symmetrisch ($(u,v)\in E\wedge (v,u)\in E$) sprechen wir von einem ungerichteten Graphen. In solchen werden keine Schlingen betrachtet. 6 | \subsection*{Adjazente Knoten} 7 | Zwei Knoten, die in einem Graphen durch eine Kante verbunden sind, heißen \emph{adjazent} oder \emph{benachbart}. 8 | \subsection*{Endlicher Graph} 9 | Ein Graph $G$ heißt endlich, wenn die Knotenmenge $V$ endlich ist. 10 | \subsection*{Nullgraph (vollst. unverbunden)} 11 | $G=(V,\emptyset)\Rightarrow$ ohne Kanten 12 | \subsection*{Vollständiger Graph} 13 | $G=(V,V\times V)$ ist vollständig (heißt auch $K_n$ wegen $n$ Knoten) und hat Maximalzahl von $n^2$ Kanten $\Rightarrow$ gerichtet und mit Schlingen. Der Ungerichtete $K_n$ hat $\frac{n\cdot (n-1)}{2}$ Kanten, wobei $n$ die Zahl der Knoten ist.\\ 14 | \emph{Beispiel:}\\ 15 | \begin{tikzpicture}[scale=.60,transform shape] 16 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=3, clockwise];}; 17 | \end{tikzpicture} 18 | \begin{tikzpicture}[scale=.60,transform shape] 19 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=4, clockwise];}; 20 | \end{tikzpicture} 21 | \begin{tikzpicture}[scale=.60,transform shape] 22 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=5, clockwise];}; 23 | \end{tikzpicture} 24 | \begin{tikzpicture}[scale=.60,transform shape] 25 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=6, clockwise];}; 26 | \end{tikzpicture} 27 | \subsection*{Ungerichteter Graph} 28 | Ein Graph $G=(V,E)$ ist ungerichtet $\Leftrightarrow E$ ist symmetrisch $\Leftrightarrow (u,v)\in E\wedge (v,u)\in E$. Da hier die Kanten ungerichtet, kann Mengenschreibweise verwendet werden. 29 | \subsection*{Schlinge} 30 | Kante mit gleichem Start- und Endknoten. Wird bei ungerichteten Graphen nicht betrachtet. 31 | \subsection*{Bipartite Graphen} 32 | Ein ungerichteter Graph ist bipartit, wenn die Knotenmenge $V$ in zwei disjunkte Teilmengen $U,W$ zerlegbar ist, sodass alle Kanten $e\in E$ einen Endpunkt in $U$ und einen anderen in $W$ haben.\\ 33 | \emph{Beispiel:}\\ 34 | \begin{tikzpicture}[scale=.75,transform shape] 35 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_nm [n=2,m=2];}; 36 | \end{tikzpicture} 37 | \begin{tikzpicture}[scale=.75,transform shape] 38 | \hspace{.3cm} 39 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_nm [n=3,m=1];}; 40 | \end{tikzpicture} 41 | \begin{tikzpicture}[scale=.75,transform shape] 42 | \hspace{.6cm} 43 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_nm [n=3,m=2];}; 44 | \end{tikzpicture} 45 | \begin{tikzpicture}[scale=.75,transform shape] 46 | \hspace{1cm} 47 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_nm [n=3,m=3];}; 48 | \end{tikzpicture} 49 | \subsection*{Eulersche Graphen} 50 | $G$ heißt eulerscher Graph, falls es in $G$ einen geschlossenen Weg gibt, der jede Kante von $G$ enthält.\\ 51 | $G$ ist eulerscher Graph $\Leftrightarrow$ jede Ecke von $G$ hat geraden Grad und $G$ ist zusammenhängend. 52 | \subsection*{Untergraph} 53 | Seien $G=(V_G,E_G)$, $H=(V_H,E_H)$ zwei Graphen. $H$ heißt Teilgraph von $G$, wenn $V_H\subseteq V_G$ und $E_H\subseteq E_G$ 54 | (wenn also jede Kante von $H$ auch zu $G$ gehört.)\\ 55 | \emph{Hinweis:}\\ 56 | Der Nullgraph $O_n$ ist Teilgraph jedes Graphen mit $n$ Knoten. Außerdem ist jeder Graph Teilgraph des vollständigen Graphen $K_n$. 57 | \subsection*{Induzierte Teilgraphen} 58 | Sei $G=(V,E)$ ein Graph. Ist $V'\subseteq V$ eine Teilmenge der Knotenmenge $V$ von $G$, dann ist der Untergraph oder 59 | der durch $V'$ induzierte Teilgraph von $G$ der Graph $G[V']=(V',E')$ mit $E'=\{(u,v)\mid u,v\in V'\wedge (u,v)\in E\}$\\ 60 | \emph{Beispiel:}\\ 61 | Ist $G$ ein Graph hat $G[\{2,3,4\}]$ nur die Knoten $2$, $3$ und $4$ und die entsprechenden Kanten. 62 | \subsection*{Grad eines Knoten} 63 | Der Ausgrad von $v$ ist die Zahl der Kanten, die $v$ als Startknoten besitzen. 64 | Der Ingrad von $v$ ist die Zahl der Kanten, die in $v$ enden. 65 | Ist $G$ ungerichtet stimmen Ausgrad von $v$ und Ingrad von $v$ überein und wird Grad von $v$ genannt.\\ 66 | \emph{Hinweis:}\\ 67 | Sei $G=(V,E)$ gerichteter Graph, dann gilt $\sum_{v\in V} indeg(v)=\sum_{v\in V} outdeg(v)=|E|$. 68 | Ist $G$ ungerichtet, dann gilt $\sum_{v\in V} deg(v)=2\cdot |E|$. 69 | \subsection*{Wege} 70 | Ein Weg von den Knoten $u$ nach $v$ ist eine Folge benachbarter Knoten. Die Länge 71 | eines Weges ist die Anzahl der Kanten. Ein Weg der Länge $0$ wird als trivialer Weg bezeichnet und besteht nur aus einem Knoten.\\ 72 | \emph{Hinweis:}\\ 73 | Ein Weg heißt geschlossen, wenn seine beiden Endknoten gleich sind. 74 | \subsection*{Graphzusammenhang} 75 | Knoten $u$ und $v$ eines ungerichteten Graphen heißen zuammenhängend, wenn es 76 | einen Weg in $G$ von $u$ nach $v$ gibt. 77 | \subsection*{Zusammenhangskomponente} 78 | Ein Graph $G$ heißt zusammenhängend wenn jedes Knotenpaar aus $G$ zusammenhängend ist.\\ 79 | \emph{Hinweis:}\\ 80 | Die Äquivalenzklassen (zusammenhängende Teilgraphen) einer Zusammenhangsrelation $Z$ über einem ungerichteten Graphen $G$ heißen Zusammenhangskomponenten (ZK) von $G$. 81 | \subsection*{Pfade, Kreise} 82 | Als \emph{Pfad} werden Wege in einem Graphen bezeichnet, bei denen keine Kante zweimal durchlaufen wird. 83 | Ein geschlossener Pfad heißt \emph{Kreis}. Bei einem \emph{einfachen Pfad} wird kein Knoten mehrfach durchlaufen. 84 | Ein geschlossener Pfad, der mit Ausnahme seines Ausgangspunktes einfach ist, heißt \emph{einfacher Kreis}. 85 | Ein einfacher Kreis durch sämtliche Knoten des Graphen, heißt \emph{Hamiltonscher Kreis}. 86 | \subsection*{Hamiltonscher Kreis} 87 | Kann der Zusammenhang eines Graphen $G$ durch die Entnahme eines einzigen Knotens (und 88 | sämtlicher mit diesem Knoten benachbarter Kanten) zerstört werden, dann besitzt $G$ keinen 89 | Hamiltonschen Kreis. 90 | \subsection*{Isomorphe Graphen} 91 | Zwei Graphen heißen isomorph zueinander, wenn sie strukturell gleich sind.\\ 92 | \emph{Beispiel:}\\ 93 | \begin{tikzpicture}[scale=.62,transform shape] 94 | \hspace{.5cm} 95 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=5, clockwise]; 96 | 1 -!- 2; 97 | 2 -!- 3; 98 | 3 -!- 4; 99 | 4 -!- 5; 100 | 5 -!- 1; 101 | }; 102 | \end{tikzpicture} 103 | \begin{tikzpicture}[scale=.62,transform shape] 104 | \hspace{.75cm} 105 | \graph [simple,nodes={myblue}, edges={myblue!80, semithick}] {subgraph K_n [n=5, clockwise]; 106 | 1 -!- 4; 107 | 1 -!- 3; 108 | 5 -!- 2; 109 | 5 -!- 3; 110 | 4 -!- 2; 111 | }; 112 | \end{tikzpicture} 113 | 114 | \begin{tabular}{c|c|c|c|c|c} 115 | $v$ & $1$ & $2$ & $3$ & $4$ & $5$ \\ 116 | $\phi (v)$ & $1$ & $4$ & $2$ & $5$ & $3$ \\ 117 | \end{tabular} 118 | \subsection*{Komplementäre Graphen} 119 | Sei $G=(V,E)$ ein Graph. Dann ist $\bar{G}=(V,(V\times V)\setminus E)$ der Komplementärgraph von $G$.\\ 120 | \emph{Hinweis:}\\ 121 | Ein Graph heißt selbstkomplementär wenn $G$ und $\bar{G}$ isomorph sind. 122 | %\subsection*{Wälder, Bäume} 123 | %Ein Graph heißt \emph{zyklenfrei}, wenn er keinen geschlossenen Weg der Länge $\geq 1$ besitzt. 124 | %Ein ungerichteter Graph heißt \emph{Wald}, wenn er zyklenfrei ist. 125 | %Ein ungerichteter Graph heißt Baum, wenn er zyklenfrei und zusammenhängend ist. 126 | %Ein wesentliches Charakteristikum von Bäumen ist die Tatsache, dass jedes Paar von 127 | %Knoten in einem Baum durch genau einen Weg verbunden ist. 128 | %Werden in einem Baum Kanten gestrichen, dann entsteht ein Wald 129 | %Die Knoten eines Baumes vom Grad 1 werden \emph{Blätter} genannt, die Knoten vom Grad 130 | %größer als 1 heißen \emph{innere Knoten}. 131 | %Ist $G$ ein zusammenhängender Graph mit $n$ Knoten und $n-1$ Kanten, dann ist $G$ ein 132 | %Baum. 133 | %Als \emph{Tiefe eines Knotens} von $T$ wird sein Abstand von der Wurzel bezeichnet. 134 | %Die Tiefe von $T$ ist die größte Knotentiefe. Alle Knoten der gleichen Tiefe bilden ein \emph{Knotenniveau}. 135 | %Als \emph{Kinder eines Knotens} $v$ von $T$ werden sämtliche Knoten bezeichnet, die zu $v$ 136 | %benachbart sind und deren Tiefe die von $v$ um eins übersteigt. $v$ heißt \emph{Vater} seiner Kinder. 137 | %\subsection*{Vollst. binärer Baum} 138 | %Ein \emph{vollständiger binärer Baum} ist ein binärer Baum, bei dem jeder innere Knoten 139 | %genau zwei Kinder hat. 140 | %Sei $T$ ein vollständiger binärer Baum mit $k$ inneren Knoten. 141 | %$T$ hat $k+1$ Blätter und insgesamt $2k+1$ Knoten. 142 | %Sei $d$ die Tiefe von $T$, dann besitzt $T$ insgesamt $\sum_{i=0}^d 2^i=2^{d+1}-1$ Knoten. 143 | -------------------------------------------------------------------------------- /content/mengen.tex: -------------------------------------------------------------------------------- 1 | \section{Mengenlehre} 2 | \subsection*{Teilmenge und Obermenge} 3 | Eine Menge $B$ heißt Teilmenge einer Menge $A$ genau dann, 4 | wenn jedes Element von $B$ auch ein Element von $A$ ist ($B\subseteq A\Leftrightarrow\forall x:x\in B\Rightarrow x\in A$). 5 | $A$ heißt dann Obermenge von $B$. Eine Menge $B$ heißt echte Teilmenge von $A$ ($B\subset A$), falls gilt $B\subseteq A\wedge B\neq A$ 6 | \subsection*{Grundlegende Mengenoperationen} 7 | Seien $M, N$ Mengen und sei $U$ die Grundmenge.\\ 8 | Vereinigungsmenge:\\ 9 | $M\cup N:=\{x\mid x\in M\vee x\in N\}$\\ 10 | Schnittmenge:\\ 11 | $M\cap N:=\{x\mid x\in M\wedge x\in N\}$\\ 12 | Differenz:\\ 13 | $M\setminus N:=\{x\mid x\in M\wedge x\notin N\}$\\ 14 | Disjunkte Menge: $M\cap N=\emptyset$ 15 | \subsection*{Potenzmenge} 16 | Sei $M$ eine Menge. Die Menge aller Teilmengen von $M$ heißt Potenzmenge von $M$ und 17 | wird $\mathcal{P}(M)$ notiert: $\mathcal{P}(M):=\{X\mid X\subseteq M\}$\\ 18 | \emph{Beispiel:}\\ 19 | $\mathcal{P}(\{a,b\})=\{\emptyset,\{a\},\{b\},\{a,b\}\}$\\ 20 | $\mathcal{P}(\emptyset)=\{\emptyset\}$\\ 21 | $\mathcal{P}(\{\emptyset\})=\{\emptyset,\{\emptyset\}\}$\\ 22 | $\mathcal{P}(\mathcal{P}(\mathcal{P}(\emptyset)))=\{\emptyset,\{\emptyset\},\{\{\emptyset\}\},\{\emptyset,\{\emptyset\}\}\}$ 23 | \subsection*{Hassediagramm} 24 | \begin{wrapfigure}[9]{r}{.46\linewidth} 25 | \vspace{-20pt} 26 | \hspace{-0pt} 27 | \begin{tikzpicture}[scale=0.4] 28 | \node (max) at (0,4) {$\{x,y,z\}$}; 29 | \node (a) at (-2,2) {$\{x,y\}$}; 30 | \node (b) at (0,2) {$\{x,z\}$}; 31 | \node (c) at (2,2) {$\{y,z\}$}; 32 | \node (d) at (-2,0) {$\{x\}$}; 33 | \node (e) at (0,0) {$\{y\}$}; 34 | \node (f) at (2,0) {$\{z\}$}; 35 | \node (min) at (0,-2) {$\emptyset$}; 36 | \draw (min) -- (d) -- (a) -- (max) -- (b) -- (f) 37 | (e) -- (min) -- (f) -- (c) -- (max) 38 | (d) -- (b); 39 | \draw[preaction={draw=white, -,line width=6pt}] (a) -- (e) -- (c); 40 | \end{tikzpicture} 41 | \end{wrapfigure} 42 | Man kann die Inklusionsbeziehungen aller Teilmengen in Form eines Hasse-Diagramms veranschaulichen. Das Hasse-Diagramm für $\mathcal{P}(\{x,y,z\})$ lässt sich dann wie folgt darstellen: 43 | \subsection*{Venn-Diagramm} 44 | \begin{tikzpicture}[scale=.45] 45 | \begin{scope} 46 | \clip \firstcircle; 47 | \fill[filled] \secondcircle; 48 | \end{scope} 49 | \draw[outline] \firstcircle node {$A$}; 50 | \draw[outline] \secondcircle node {$B$}; 51 | \node[anchor=south] at (current bounding box.north) {$A \cap B$}; 52 | \end{tikzpicture} 53 | %Set A or B but not (A and B) also known a A xor B 54 | \begin{tikzpicture}[scale=.45] 55 | \hspace{-.5cm} 56 | \draw[filled, even odd rule] \firstcircle node {$A$} 57 | \secondcircle node{$B$}; 58 | \node[anchor=south] at (current bounding box.north) {$\overline{A \cap B}$}; 59 | \end{tikzpicture} 60 | % Set A or B 61 | \begin{tikzpicture}[scale=.45] 62 | \draw[filled] \firstcircle node {$A$} 63 | \secondcircle node {$B$}; 64 | \node[anchor=south] at (current bounding box.north) {$A \cup B$}; 65 | \end{tikzpicture} 66 | % Set A but not B 67 | \hspace{.5cm} 68 | \begin{tikzpicture}[scale=.45] 69 | \begin{scope} 70 | \clip \firstcircle; 71 | \draw[filled, even odd rule] \firstcircle node {$A$} 72 | \secondcircle; 73 | \end{scope} 74 | \draw[outline] \firstcircle 75 | \secondcircle node {$B$}; 76 | \node[anchor=south] at (current bounding box.north) {$A\setminus B$}; 77 | \end{tikzpicture} 78 | \subsection*{Operationen auf Mengenfamilien} 79 | Sei $\mathcal{F}=\{\{1,2,3,4\},\{3,4,5,6\}\}$ Mengenfamilie. 80 | Vereinigung aller Mengen aus $\mathcal{F}$:\\ 81 | $\bigcup\mathcal{F}=\{1,2,3,4,5,6\}$\\ 82 | Durchschnitt aller Mengen aus $\mathcal{F}$:\\ 83 | $\bigcap\mathcal{F}=\{3,4\}$ 84 | \subsection*{Kartesisches Produkt} 85 | Seien $A,B$ Mengen, dann ist das kartesische Produkt (Kreuzprodukt) 86 | von $A$ und $B$ definiert als: $A\times B:=\{(a,b)\mid a\in A\wedge b\in B\}$. 87 | $A\times B$ ist die Menge aller geordneten Paare von $A$ und $B$.\\ 88 | \emph{Hinweis:}\\ 89 | $(a,b)=(c,d)\Leftrightarrow a=c\wedge b=d$\\ 90 | $A\times\emptyset=\emptyset\times A=\emptyset$\\ 91 | $A\times B\neq B\times A$\\ 92 | \emph{Beispiel:}\\ 93 | $\{1,2\}\times\{3,4\}=\{(1,3),(1,4),(2,3),(2,4)\}$\\ 94 | $\{3,4\}\times\{1,2\}=\{(3,1),(3,2),(4,1),(4,2)\}$ 95 | \subsection*{Rechenregeln für Mengenoperationen} 96 | Assoziativgesetze:\\ 97 | $(A\cup B)\cup C=A\cup (B\cup C)$\\ 98 | $(A\cap B)\cap C=A\cap (B\cap C)$\\ 99 | Kommutativgesetze:\\ 100 | $A\cup B=B\cup A$\\ 101 | $A\cap B=B\cap A$\\ 102 | Distributivgesetze:\\ 103 | $(A\cup B)\cap C=(A\cap C)\cup (B\cap C)$\\ 104 | $(A\cap B)\cup C=(A\cup C)\cap (B\cup C)$\\ 105 | de Morganschen Gesetze (Differenz):\\ 106 | $A\setminus (B\cup C)=(A\setminus B)\cap (A\setminus C)$\\ 107 | $A\setminus (B\cap C)=(A\setminus B)\cup (A\setminus C)$\\ 108 | de Morganschen Gesetze (Komplement):\\ 109 | $\bar{A\cup B}=\bar{A}\cap\bar{B}$\\ 110 | $\bar{A\cap B}=\bar{A}\cup\bar{B}$\\ 111 | Absorptionsgesetze:\\ 112 | $A\cap (A\cup B)=A$\\ 113 | $A\cup (A\cap B)=A$\\ 114 | Idempotenzgesetze:\\ 115 | $A\cap A=A$\\ 116 | $A\cup A=A$\\ 117 | Komplementgesetze ($G$ ist Grundmenge):\\ 118 | $A\cap\bar{A}=\emptyset$\\ 119 | $A\cup\bar{A}=G$ -------------------------------------------------------------------------------- /content/notes.tex: -------------------------------------------------------------------------------- 1 | \subsection{Useful to know} 2 | 3 | \subsubsection{Min of iid exponential r.v} 4 | Let $X_1,…,X_nn$ be i.i.d. $Exp(\lambda)$ random variables. 5 | 6 | Distribution of $min_i(Xi)$ 7 | 8 | \begin{align*} 9 | \mathbf{P}(\min _ i (X_ i)\leq t) &=\\ 10 | &=1-\mathbf{P}(\min _ i (X_ i)\geq t)\\ 11 | &=1-(\mathbf{P}(X_1\geq t))(\mathbf{P}(X_2\geq t))\ldots (\mathbf{P}(X_ n\geq t))\\ 12 | &=1-(1-F_ X(t))^ n \, =\, 1-e^{-n\lambda x} 13 | \end{align*} 14 | 15 | Differentiate w.r.t $x$ to get the pdf of $min_i(Xi)$: 16 | 17 | \begin{align*} 18 | f_{\text {min}}(x)= (n\lambda ) e^{-(n\lambda ) x} 19 | \end{align*} 20 | 21 | \subsubsection{Counting Commitees} 22 | 23 | Out of $2n$ people, we want to choose a committee of $n$ people, one of whom will be its chair. In how many different ways can this be done?" 24 | 25 | $$n\binom {2n}{n}=2n\binom {2n-1}{n-1}.$$ 26 | 27 | “In a group of 2n people, consisting of n boys and n girls, we want to select a committee of n people. In how many ways can this be done?" 28 | 29 | $$\binom {2n}{n}=\sum _{i=0}^ n \binom {n}{i}\binom {n}{n-i}$$ 30 | 31 | “How many subsets does a set with 2n elements have?" 32 | 33 | $$2^{2n}=\sum _{i=0}^{2n}\binom {2n}{i}$$ 34 | 35 | “Out of $n$ people, we want to form a committee consisting of a chair and other members. We allow the committee size to be any integer in the range $1,2,…,n$ . How many choices do we have in selecting a committee-chair combination?" 36 | 37 | $$n2^{n-1}=\sum _{i=0}^ n \binom {n}{i}i.$$ 38 | 39 | \subsection{Finding Joint PDFS} 40 | 41 | $f_{X,Y}(x,y)=f_ X(x)f_{Y|X}(y\mid x)$ -------------------------------------------------------------------------------- /content/probability_distributions.tex: -------------------------------------------------------------------------------- 1 | \section{Important probability distributions} 2 | \subsection*{Bernoulli} 3 | Parameter $p \in[0,1]$, discrete\\ 4 | $ p_x(k)= 5 | \begin{cases} 6 | p,&\text{if k = 1}\\ 7 | (1-p),&\text{if k = 0}\\ 8 | \end{cases} 9 | $\\ 10 | 11 | $\mathbb{E}[X]=p$\\ 12 | 13 | $Var(X)=p(1-p)$\\ 14 | 15 | Likelihood n trials:\\ 16 | 17 | $L_ n(X_1, \ldots , X_ n, p) =\\ 18 | = p^{\sum _{i = 1}^ n X_ i} (1 -p)^{{\color{blue}{n - }} \sum _{i = 1}^ n X_ i}$ \\ 19 | 20 | Loglikelihood n trials:\\ 21 | 22 | $\ell_n (p) = \\ = \ln \left( p \right) \sum _{i=1}^{n}X_{{i}}+ \left( n-\sum _{i=1}^{n} 23 | X_{{i}} \right) \ln \left( 1-p \right) 24 | $\\ 25 | 26 | MLE:\\ 27 | 28 | $\hat{p}_{MLE} = \frac{\sum^n_{i=1}(X_i)}{n}$\\ 29 | 30 | Fisher Information:\\ 31 | 32 | $I(p) = \frac{1}{p(1-p)}$\\ 33 | 34 | Canonical exponential form:\\ 35 | 36 | $f_{\theta}(y)=\exp\big(y\theta - \underbrace{\ln(1 + e^\theta)}_{b(\theta)} + \underbrace{0}_{c(y, \phi)}\big) \quad$\\ 37 | 38 | $\theta = \ln\left(\frac{p}{1-p}\right)$\\ 39 | $\phi = 1$\\ 40 | 41 | \subsection*{Binomial} 42 | Parameters $p$ and $n$, discrete. Describes the number of successes 43 | in n independent Bernoulli trials.\\ 44 | 45 | $p_x(k)= {n\choose k}{p}^{k} \left( 1-p \right) ^{n-k}$, $k=0,\ldots, n$\\ 46 | 47 | $\mathbb{E}[X]=np$\\ 48 | 49 | $Var(X)= np(1-p)$ \\ 50 | 51 | Likelihood:\\ 52 | 53 | $\displaystyle L_ n(X_1, \ldots , X_ n, \theta ) =\\ 54 | = \left( \prod _{i = 1}^ n \binom {K}{X_ i} \right) \theta ^{\sum _{i = 1}^ n X_ i} (1 - \theta )^{nK - \sum _{i = 1}^ n X_ i }$\\ 55 | 56 | Loglikelihood:\\ 57 | 58 | $\ell_n (\theta) = C + \left( \sum _{i = 1}^ n X_ i \right) \log \theta + \left( nK - \sum _{i = 1}^ n X_ i \right) \log (1 - \theta )$\\ 59 | 60 | MLE:\\ 61 | 62 | 63 | Fisher Information:\\ 64 | 65 | $I(p) = \frac{n}{p(1-p)}$\\ 66 | 67 | Canonical exponential form:\\ 68 | 69 | $f_ p(y) =\\ 70 | exp (y \underbrace{(\ln (p)-\ln (1-p))}_{\theta } + \underbrace{n\ln (1-p)}_{-b(\theta )} +\underbrace{\ln(\binom {n}{y})}_{c(y,\phi )} )$ 71 | 72 | \subsection*{Geometric} 73 | Number of $T$ trials up to (and including) the first success. 74 | 75 | $p_T(t) = (1-p)^{t-1}, t=1,2,...$\\ 76 | $\mathbb{E}[T]=\frac{1}{p}$\\ 77 | $var(T)=\frac{1-p}{p^2}$ 78 | 79 | \subsection*{Pascal} 80 | 81 | The negative binomial or Pascal distribution is a generalization of the geometric distribution. It relates to the random experiment of repeated independent trials until observing $m$ successes. I.e. the time of the kth arrival. 82 | 83 | $Y_k=T_1+...T_k$\\ 84 | 85 | $T_i \sim iid Geometric(p)$\\ 86 | 87 | $\mathbb{E}[Y_k]=\frac{k}{p}$\\ 88 | 89 | $Var(Y_k)= \frac{k(1-p}{p^2}$ 90 | 91 | $p_{Y_k}(t) ={t-1 \choose k-1}p^k(1-p)^{t-k}$\\ 92 | 93 | $t=k,k+1,...$ 94 | 95 | 96 | \subsection*{Multinomial} 97 | 98 | Parameters $n>0$ and $p_1, \ldots, p_r$. 99 | 100 | $p_x(x)= \frac{n!}{x_1!,\ldots,x_n!} p_1, \ldots, p_r$\\ 101 | 102 | 103 | $\mathbb{E}[X_i]=n*p_i$\\ 104 | 105 | $Var(X_i)=np_i(1-p_i)$\\ 106 | 107 | 108 | Likelihood:\\ 109 | 110 | $p_x(x)= \prod _{j=1}^{n}{p_{{j}}}^{T_{{j}}}$, where $T^j=\mathbbm{1}( X_i=j)$ is the count how often an outcome is seen in trials. \\ 111 | 112 | Loglikelihood:\\ 113 | $\ell_n= \sum _{j=2}^{n}T_{{j}}\ln \left( p_{{j}} \right)$\\ 114 | 115 | 116 | \subsection*{Poisson} 117 | Parameter $\lambda$. discrete, approximates the binomial PMF when $n$ is large, $p$ is small, and $\lambda = np$.\\ 118 | 119 | $\mathbf{p_x}(k)=exp(-\lambda)\frac{\lambda^k}{k!}$ for $k=0,1, \ldots,$\\ 120 | 121 | $\mathbb{E}[X]=\lambda$\\ 122 | 123 | $Var(X)=\lambda$\\ 124 | 125 | Likelihood:\\ 126 | $L_ n(x_1, \ldots , x_ n, \lambda) = \prod _{i = 1}^ n \frac{\lambda^{\sum_{i=1}^{n} x_i}}{\prod _{i = 1}^ n x_i!} e^{-n\lambda}$\\ 127 | 128 | Loglikelihood:\\ 129 | $\ell_n (\lambda)= \\ 130 | = -n\lambda + log(\lambda)(\sum_{i=1}^n x_i)) - log(\prod _{i = 1}^ n x_i!)$\\ 131 | 132 | MLE:\\ 133 | 134 | $\hat{\lambda}_{MLE} = \frac{1}{n} \sum^n_{i=1}(X_i)$\\ 135 | 136 | Fisher Information:\\ 137 | 138 | $I(\lambda)= \frac{1}{\lambda}$\\ 139 | 140 | Canonical exponential form:\\ 141 | 142 | $ f_{\theta}(y) = \exp\big(y\theta - \underbrace{e^\theta}_{b(\theta)} \underbrace{- \ln y!}_{c(y, \phi)}\big)$\\ 143 | $\theta = \ln \lambda$\\ 144 | $\phi = 1$\\ 145 | 146 | Poisson process:\\ 147 | k arrivals in t slots 148 | $\mathbf{p_x}(k,t) = \mathbb{P}(N_t=k)=e^{-\lambda t} \frac{(\lambda t)^k}{k!}$\\ 149 | 150 | $\mathbb{E}[N_t]=\lambda t$\\ 151 | 152 | $Var(N_t)=\lambda t$ 153 | 154 | \subsection*{Exponential} 155 | Parameter $\lambda$, continuous\\ 156 | $ f_x(x)= 157 | \begin{cases} 158 | \lambda exp(-\lambda x),&\text{if x >= 0}\\ 159 | 0,&\text{o.w.}\\ 160 | \end{cases} 161 | $\\ 162 | 163 | $P(X>a)= exp(-\lambda a)$\\ 164 | 165 | $ F_x(x)= 166 | \begin{cases} 167 | 1-exp(-\lambda x),&\text{if x >= 0}\\ 168 | 0,&\text{o.w.}\\ 169 | \end{cases} 170 | $\\ 171 | 172 | $\mathbb{E}[X]=\frac{1}{\lambda}$\\ 173 | 174 | $\mathbb{E}[X^2]=\frac{2}{\lambda^2}$ 175 | 176 | $Var(X)=\frac{1}{\lambda^2}$\\ 177 | 178 | Likelihood:\\ 179 | $L(X_1\dots X_n;\lambda)=\lambda^n\exp\left(-\lambda\sum_{i=1}^n X_i\right)$\\ 180 | 181 | Loglikelihood:\\ 182 | 183 | $\ell_n (\lambda)= n ln(\lambda) - \lambda \sum_{i=1}^n (X_i)$\\ 184 | 185 | MLE:\\ 186 | 187 | $\hat{\lambda}_{MLE}= \frac{n}{\sum^{n}_{i=1}(X_i)}$\\ 188 | 189 | Fisher Information:\\ 190 | 191 | $I(\lambda)= \frac{1}{\lambda^2}$\\ 192 | 193 | Canonical exponential form:\\ 194 | 195 | $f_{\theta}(y) = \exp\big(y\theta - \underbrace{(-\ln(-\theta))}_{b(\theta)} + \underbrace{0}_{c(y, \phi)}\big)$\\ 196 | 197 | $\theta = -\lambda = -\frac1{\mu}$\\ 198 | 199 | $\phi = 1$ 200 | 201 | \subsection*{Shifted Exponential} 202 | 203 | Parameters $\lambda, a \in \mathbb{R}$, continuous\\ 204 | $ f_x(x)= 205 | \begin{cases} 206 | \lambda exp(-\lambda(x - a )),&{x >= a}\\ 207 | 0,&{x <= a}\\ 208 | \end{cases} 209 | $\\ 210 | 211 | $ F_x(x)= 212 | \begin{cases} 213 | 1-exp(-\lambda(x-a)),&{if x >= a}\\ 214 | 0,&{x <= a}\\ 215 | \end{cases} 216 | $\\ 217 | 218 | $\mathbb{E}[X]=a + \frac{1}{\lambda}$\\ 219 | 220 | $Var(X)=\frac{1}{\lambda^2}$\\ 221 | 222 | Likelihood:\\ 223 | 224 | $L(X_1\dots X_n;\lambda,\theta)= \lambda ^ n \exp \left( -\lambda \sum _{i = 1}^ n (X_ i - a) \right) \mathbf{1}_{\min _{i = 1, \ldots , n}(X_ i) \geq a}.$ 225 | 226 | Loglikelihood:\\ 227 | 228 | $\ell (\lambda , a) := n \ln \lambda - \lambda \sum _{i = 1}^ n X_ i + n \lambda a$ 229 | 230 | MLE: 231 | 232 | $\hat{\lambda }_{MLE} = \frac{1}{\overline{X}_ n - \hat{a}}$\\ 233 | 234 | $\hat{a}_{MLE} = \min _{i =1, \ldots , n}(X_ i)$ 235 | 236 | \subsection*{Univariate Gaussians} 237 | Parameters $\mu$ and $\sigma^2 >0$, continuous\\ 238 | $f(x)= \frac{1}{\sqrt(2 \pi \sigma^2)} exp(-\frac{(x-\mu)^2}{2\sigma^2})$ \\ 239 | $\mathbb{E}[X]=\mu$ \\ 240 | $Var(X)=\sigma^2$\\ 241 | 242 | CDF of standard gaussian:\\ 243 | 244 | $\Phi (z) = \int _{-\infty }^ z \frac{1}{\sqrt{2 \pi }} e^{-x^2/2} \, dx$ 245 | 246 | Likelihood:\\ 247 | 248 | $L(x_1\dots X_n;\mu,\sigma^2)=\\ 249 | = \dfrac{1}{\left(\sigma\sqrt{2\pi}\right)^n}\exp{\left(-\dfrac{1}{2\sigma^2}\sum_{i=1}^n (X_i-\mu)^2 \right)}$\\ 250 | 251 | Loglikelihood:\\ 252 | 253 | $\ell_n (\mu,\sigma^2)= \\ 254 | = -n log(\sigma\sqrt{2\pi})-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_i-\mu)^2 $ 255 | 256 | MLE:\\ 257 | 258 | $\hat\mu_MLE = \bar X_ n\\ 259 | \quad \widehat{\sigma ^2}_MLE = \frac{1}{n} \sum _{i=1}^{n} (X_ i - \bar X_ n)^2$ 260 | 261 | Fisher Information:\\ 262 | 263 | $I(\mu , \sigma ^2) = \begin{pmatrix} \frac{1}{\sigma ^2} & 0 \\ 0 & \frac{1}{2 \sigma ^4} \end{pmatrix}$ 264 | 265 | Canonical exponential form:\\ 266 | 267 | Gaussians are invariant under affine transformation:\\ 268 | 269 | $aX+b \sim N(X+b,a^2\sigma^2)$\\ 270 | 271 | Sum of independent gaussians:\\ 272 | 273 | Let $X {\sim} N(\mu_X,\sigma_X^2)$ and $Y {\sim} N(\mu_Y,\sigma_Y^2)$\\ 274 | 275 | If $Y = X + Z$, then $Y \sim N(\mu_X + \mu_Y, \sigma_X + \sigma_Y)$\\ 276 | 277 | If $U = X - Y$, then $U \sim N(\mu_X - \mu_Y,\sigma_X + \sigma_Y)$\\ 278 | 279 | Symmetry:\\ 280 | 281 | If $X \sim\ N(0,\sigma^2),$ then $-X \sim N(0,\sigma^2)$\\ 282 | 283 | $\mathbb{P}(|X|>x) = 2\mathbb{P}(X>x)$\\ 284 | 285 | Standardization:\\ 286 | 287 | $Z= \frac{X-\mu}{\sigma} \sim N(0,1)$\\ 288 | 289 | $\mathbf{P}\left(X\leq t\right) = \displaystyle \mathbf{P}\left(Z\leq \frac{t-\mu}{\sigma}\right)$ 290 | 291 | Higher moments:\\ 292 | 293 | $\mathbb{E}[X^2] = \mu^2 + \sigma^2$\\ 294 | $\mathbb{E}[X^3] = \mu^3 + 3\mu\sigma^2$\\ 295 | $\mathbb{E}[X^4] = \mu^4 + 6\mu^2\sigma^2 +3\sigma^4$\\ 296 | 297 | Quantiles:\\ 298 | 299 | \subsection*{Uniform} 300 | 301 | Parameters $a$ and $b$, continuous. 302 | 303 | $ \mathbf{f_x}(x)= 304 | \begin{cases} 305 | \frac{1}{b-a},&\text{if a < x M) = P(X < M)\\ = 1/2 = \displaystyle \int _{1/2}^{\infty } \frac{1}{\pi } \cdot \frac{1}{1 + (x-m)^2} \, dx$ 336 | 337 | \subsection*{Chi squared} 338 | The $\chi _ d^2$ distribution with $d$ degrees of freedom is given by the distribution of $Z_1^2 + Z_2^2 + \cdots + Z_ d^2,$ where $Z_1, \ldots , Z_ d \stackrel{iid}{\sim } \mathcal{N}(0,1)$ 339 | 340 | If $V \sim \chi^2_k:$\\ 341 | 342 | $\mathbb{E}= \mathbb{E}[Z_1^2] + \mathbb{E}[Z_2^2] + \ldots + \mathbb{E}[Z_d^2] = d$\\ 343 | 344 | $Var(V) = Var(Z_1^2) + Var(Z_2^2) + \ldots + Var(Z_d^2) = 2d$ 345 | 346 | \subsection*{Student's T Distribution} 347 | 348 | $T_ n := \frac{Z}{\sqrt{V/n}}$ where $Z \sim \mathcal{N}(0,1)$, and $Z$ and $V$ are independent -------------------------------------------------------------------------------- /content/relationen.tex: -------------------------------------------------------------------------------- 1 | \section{Relationen} 2 | \subsection*{Binäre Relation} 3 | Eine binäre Relation $R$ ist eine Menge von Paaren $(a,b)\in A\times B$.\\ 4 | $aRb\Leftrightarrow (a,b)\in R$ bzw. $a(\neg R)b\Leftrightarrow (a,b)\notin R$\\ 5 | \emph{Beispiele:}\\ 6 | Teilerrelation ($nTm$): $P_3:=\{(n,m+3)\mid n,m\in\mathbb{N}\}=\{(1,4),(2,5),(3,6),...\}$\\ 7 | Relation $\subset$ über $\mathcal{P}(M)$ für $M=\{1,2\}$:\\ 8 | $\{(\emptyset ,\{1\}),(\emptyset ,\{2\}),(\emptyset ,\{1,2\}),(\{1\},\{1,2\}),\\(\{2\} ,\{1,2\})\}$ 9 | \subsection*{Inverse Relation} 10 | Sei $R\subseteq A\times B$. Die inverse Relation zu $R$ ist $R^{-1}=\{(y,x)\in B\times A\mid (x,y)\in R\}$. Also ist $R^{-1}\subseteq B\times A$.\\ 11 | \emph{Beispiel:} Sei $R=\{(1,a),(1,c),(3,b)\}$ dann ist $R^{-1}=\{(a,1),(c,1),(b,3)\}$ 12 | \subsection*{Komposition} 13 | Seien $R\subseteq M_1\times M_2$ und $S\subseteq M_2\times M_3$ zweistellige Relationen. 14 | Die Verknüpfung $(R\circ S)\subseteq (M_1\times M_3)$ heißt Komposition der Relationen $R,S$.\\ 15 | $R\circ S:=\{(x,z)\mid\exists y\in M_2:(x,y)\in R\wedge (y,z)\in S\}$\\ 16 | \emph{Beispiel:} Sei $R=\{(1,2),(2,5),(5,1)\}$, dann ist $R^2=R\circ R=\{(1,5),(2,1),(5,2)\}$\\ 17 | Sei $R\subseteq\mathbb{N}\times\mathbb{N}$ mit $(n,m)\in R\Leftrightarrow m=3n$ und 18 | $S\subseteq\mathbb{N}\times\mathbb{Z}$ mit $(n,z)\in S\Leftrightarrow z=-n$. Dann ist $R\circ S=\{(n,z)\mid z=-3n\}\subseteq\mathbb{N}\times\mathbb{Z}$ 19 | \subsection*{Eigenschaften von Operationen} 20 | $(R\cup S)^{-1}=R^{-1}\cup S^{-1}$\\ 21 | $(R\cap S)^{-1}=R^{-1}\cap S^{-1}$\\ 22 | $(R\circ S)^{-1}=S^{-1}\circ R^{-1}$\\ 23 | $(R\cap S)\circ T\subseteq (R\circ T)\cap (S\circ T)$\\ 24 | $T\circ (R\cap S)\subseteq (T\circ R)\cap (T\circ S)$\\ 25 | $(R\cup S)\circ T = (R\circ T)\cup (S\circ T)$\\ 26 | $T\circ (R\cup S) = (T\circ R)\cup (T\circ S)$ 27 | \subsection*{Eigenschaften von Relationen} 28 | Reflexiv: $\forall a\in A:(a,a)\in R$\\ 29 | Symmetrisch: $\forall a,b\in A:(a,b)\in R\Rightarrow (b,a)\in R$\\ 30 | Antisymm.: $\forall a,b\in A:(a,b)\in R\wedge (b,a)\in R\Rightarrow a=b$\\ 31 | Transitiv: $\forall a,b,c\in A:(a,b)\in R\wedge (b,c)\in R\Rightarrow (a,c)\in R$\\ 32 | Total: $\forall a,b \in A: (a,b)\in R\vee (b,a)\in R$\\ 33 | Irreflexiv: $\forall a\in A: (a,a)\notin R$\\ 34 | Asymm.: $\forall a,b\in A:(a,b)\in R\Rightarrow (b,a)\notin R$\\ 35 | Alternativ: $\forall a,b\in A:(a,b)\in R\oplus (b,a)\in R$\\ 36 | Rechtseind.: $\forall a\in A:(a,b)\in R\wedge (a,c)\in R\Rightarrow b=c$\\ 37 | Linkseind.: $\forall a\in A:(b,a)\in R\wedge (c,a)\in R\Rightarrow b=c$\\ 38 | Eindeutig: $R$ ist recht- und linkseindeutig.\\ 39 | Linkstotal: $\forall a\in A\exists b\in B:(a,b)\in R$\\ 40 | Rechtstotal: $\forall b\in B\exists a\in A:(a,b)\in R$ 41 | \subsection*{Äquivalenzrelation} 42 | Ist eine Relation $\sim$ reflexiv, symmetrisch und transitiv, heißt sie Äquivalenzrelation. 43 | \subsection*{Äquivalenzklassen} 44 | Gegeben eine Äquivalenzrelation $R$ über der Menge $A$. 45 | Dann ist für $a\in A$: $[a]_R=\{x\mid (a,x)\in R\}$ die Äquivalenzklasse von $a$.\\ 46 | (Äquivalente Elemente kommen in die gleiche Menge)\\ 47 | \emph{Beispiel (Restklassen):}\\ 48 | $[4]=\{n\mid n\mod 3=4 \mod 3\}=[1]$\\ 49 | $[5]=\{n\mid n\mod 3=5 \mod 3\}=[2]$\\ 50 | $[6]=\{n\mid n\mod 3=6 \mod 3\}=[3]$ 51 | \subsection*{Zerlegungen, Partition} 52 | Eine Zerlegung (Partition) $\mathcal{Z}$ ist eine Einteilung von $A$ in nicht leere, paarweise 53 | elementfremde Teilmengen, deren Vereinigung mit $A$ übereinstimmt.\\ 54 | \emph{Beispiel:} Sei $A=\{1,2,3,...,10\}$. Dann ist $\mathcal{Z_1}=\{\{1,3\},\{2,5,9\},\{4,10\},\{6,7,8\}\}$ 55 | \subsection*{Abschluss einer Relation} 56 | $R_\phi^*$ bildet die fehlenden Relationen mit der Eigenschaft $\phi$, also alle Kombinationen aus $A$, die noch nicht in $R$ sind.\\ 57 | \emph{Beispiel:}\\ 58 | Sei $A=\{1,2,3\}$ und $R=\{(1,2),(2,3),(3,3)\}$. 59 | Dann ist $R_{refl}^*=R\cup\{(1,1),(2,2)\}$,\\ 60 | $R_{sym}^*=R\cup\{(2,1),(3,2)\}$, 61 | $R_{tra}^*=R\cup\{(1,3)\}$ 62 | \subsection*{Halbordunung} 63 | Eine Relation $R$, die reflexiv, antisymmetrisch und transitiv ist. -------------------------------------------------------------------------------- /content/test.tex: -------------------------------------------------------------------------------- 1 | \section{Mengenlehre} 2 | \subsection*{Teilmenge und Obermenge} 3 | Eine Menge $B$ heißt Teilmenge einer Menge $A$ genau dann, 4 | wenn jedes Element von $B$ auch ein Element von $A$ ist ($B\subseteq A\Leftrightarrow\forall x:x\in B\Rightarrow x\in A$). 5 | $A$ heißt dann Obermenge von $B$. Eine Menge $B$ heißt echte Teilmenge von $A$ ($B\subset A$), falls gilt $B\subseteq A\wedge B\neq A$ 6 | Probability density function: 7 | \[ 8 | \begin{cases*} 9 | \frac{1}{b-a} & for $x\in[a,b]$ \\ 10 | 0 & otherwise \\ 11 | \end{cases*} 12 | \] --------------------------------------------------------------------------------