29 |
30 | https://youtu.be/DbJyPELmhJc
31 |
32 |
--------------------------------------------------------------------------------
/book/independence.md:
--------------------------------------------------------------------------------
1 | # Independence
2 | ```{math}
3 | \newcommand\indep{\perp\kern-5pt\perp}
4 | ```
5 |
6 | As discussed in the previous section, **conditional probabilities** quantify the extent to which the knowledge of the occurrence of a certain event affects the probability of another event [^footnote1].
7 | In some cases, it makes no difference: the events are independent. More formally, events $A$ and $B$ are **independent** if and only if
8 |
9 | $$
10 | P (A|B) = P (A) .
11 | $$
12 |
13 | This definition is not valid if $P (B) = 0$. The following definition covers this case and is otherwise
14 | equivalent.
15 |
16 | ```{admonition} Definition (Independence).
17 | Let $(\Omega,\mathcal{F},P)$ be a probability space. Two events $A,B \in \mathcal{F}$
18 | are independent if and only if
19 |
20 | $$
21 | P (A \cap B) = P (A) P (B) .
22 | $$
23 | ```
24 | ```{admonition} Notation
25 | This is often denoted $ A \indep B $
26 | ```
27 |
28 | Similarly, we can define **conditional independence** between two events given a third event.
29 | $A$ and $B$ are conditionally independent given $C$ if and only if
30 |
31 | $$
32 | P (A|B, C) = P (A|C) ,
33 | $$
34 |
35 | where $P (A|B, C) := P (A|B \cap C)$. Intuitively, this means that the probability of $A$ is not affected by whether $B$ occurs or not, as long as $C$ occurs.
36 |
37 | ```{admonition} Notation
38 | This is often denoted $ A \indep B \mid C$
39 | ```
40 |
41 | ## Graphical Models
42 |
43 | There is a graphical model representation for joint distributions $P(A,B,C)$ that encodes their conditional (in)dependence known as a **probabilistic graphical model**. For this situation $ A \indep B \mid C$, the graphical model looks like this:
44 |
45 |
46 |
47 | The lack of an edge directly between $A$ and $B$ indicates that the two varaibles are conditionally independent. This image was produced with `daft`, and there are more examples in [Visualizing Graphical Models](./pgm/daft).
48 |
49 | [^footnote1]: This text is based on excerpts from Section 1.3 of [NYU CDS lecture notes on Probability and Statistics](https://cims.nyu.edu/~cfgranda/pages/stuff/probability_stats_for_DS.pdf)
50 |
--------------------------------------------------------------------------------
/book/_config.yml:
--------------------------------------------------------------------------------
1 | # Book settings
2 | title: Statistics and Data Science
3 | author: Kyle Cranmer
4 | logo: logo.png
5 | copyright: ""
6 |
7 | parse:
8 | myst_extended_syntax: true
9 |
10 | execute:
11 | exclude_patterns : ["*/Central-Limit-Theorem.ipynb","*/prop-error-plots.ipynb","*/track-example.ipynb"]
12 | execute_notebooks : off # force, off, auto
13 |
14 | # Information about where the book exists on the web
15 | repository:
16 | url: https://github.com/cranmer/stats-ds-book
17 | path_to_book: book
18 | branch: master
19 |
20 | html:
21 | home_page_in_navbar : true
22 | use_repository_button: true
23 | use_issues_button: true
24 | use_edit_page_button: true
25 | google_analytics_id: UA-178330963-1
26 | comments:
27 | hypothesis: true
28 | extra_footer : |
29 |
30 |
31 | All content on this site (unless otherwise specified) is licensed under the CC BY-NC-SA 4.0 license
32 |
33 |
34 | sphinx:
35 | extra_extensions:
36 | - sphinx_tabs.tabs
37 | - sphinxext.opengraph
38 | html_show_copyright: false
39 | config:
40 | ogp_site_url: "https://cranmer.github.io/stats-ds-book/"
41 | ogp_image: "https://cranmer.github.io/stats-ds-book/_images/Neyman-pearson.006.png"
42 | ogp_description_length: 200
43 | mathjax_config:
44 | TeX:
45 | Macros:
46 | "N": "\\mathbb{N}"
47 | "indep": "{\\perp\\kern-5pt\\perp}"
48 | "floor": ["\\lfloor#1\\rfloor", 1]
49 | "bmat": ["\\left[\\begin{array}"]
50 | "emat": ["\\end{array}\\right]"]
51 | "bered": ["\\color{#DC2830}{#1}",1]
52 | "ecol": ["}}"]
53 |
54 | # Launch button settings
55 | launch_buttons:
56 | notebook_interface: classic #jupyterlab
57 | binderhub_url: https://mybinder.org
58 | colab_url: https://colab.research.google.com
59 |
60 | latex:
61 | latex_documents:
62 | targetname: book.tex
63 |
64 | extra_extensions:
65 | - sphinx_click.ext
66 | - sphinx_tabs.tabs
67 | - sphinx_panels
--------------------------------------------------------------------------------
/book/probability-topics.md:
--------------------------------------------------------------------------------
1 | # Probability Topics
2 |
3 | 1. Probability models
4 | 1. Probability denstiy functions
5 | 1. Classic distributons
6 | 1. Bernouli
7 | 1. Binomial
8 | 1. Poisson
9 | 1. Gaussian
10 | 1. Chi-Square
11 | 1. Exponential family
12 | 1. Multivariate distributions
13 | 1. Independence
14 | 1. Covariance
15 | 1. Conditional distributions
16 | 1. Marginal distributions
17 | 1. Graphical Models
18 | 1. [https://github.com/pgmpy/pgmpy](https://github.com/pgmpy/pgmpy)
19 | 1. [https://github.com/jmschrei/pomegranate](https://github.com/jmschrei/pomegranate)
20 | 1. [Video](https://youtu.be/DEHqIxX1Kq4)
21 | 1. Copula
22 | 1. Information theory
23 | 1. Entropy
24 | 1. Mutual information
25 | 1. KL divergence
26 | 1. cross entropy
27 | 1. Divergences
28 | 1. KL Divergence
29 | 1. Fisher distance
30 | 1. Optimal Transport
31 | 1. Hellinger distance
32 | 1. f-divergences
33 | 1. Stein divergence
34 | 1. Implicit probabity models
35 | 1. Simulators
36 | 1. Probabilistic Programming
37 | 1. https://docs.pymc.io
38 | 1. [ppymc3 vs. stan vs edward](https://statmodeling.stat.columbia.edu/2017/05/31/compare-stan-pymc3-edward-hello-world/)
39 | 1. pyro
40 | 1. pyprob
41 | 1. Likelihood function
42 | 1. [Axioms of probability](https://en.wikipedia.org/wiki/Probability_axioms)
43 | 1. [Probability Space](https://en.wikipedia.org/wiki/Probability_space)
44 | 1. Transformation properties
45 | 1. Change of variables
46 | 1. Propagation of errors
47 | 1. Reparameterization
48 | 1. Bayes Theorem
49 | 1. Subjective priors
50 | 1. Emperical Bayes
51 | 1. Jeffreys' prior
52 | 1. Unfiform priors
53 | 1. Reference Priors
54 | 1. Transformation Properties
55 | 1. Convolutions and the Central Limit Theorem
56 | 1. Binomial example
57 | 1. Convolutions in Fourier domain
58 | 1. [Extreme Value Theory](https://en.wikipedia.org/wiki/Extreme_value_theory)
59 | 1. Weibull law
60 | 1. Gumbel law
61 | 1. Fréchet Law
62 |
63 |
64 |
--------------------------------------------------------------------------------
/book/statistics/neyman_construction.md:
--------------------------------------------------------------------------------
1 | # Neyman construction
2 |
3 |
4 | `````{tabs}
5 | ````{tab} Step 1
6 |
7 | ```{figure} ../assets/Neyman-construction/Neyman-construction.001.png
8 | ```
9 |
10 | ````
11 | ````{tab} Step 2
12 |
13 | ```{figure} ../assets/Neyman-construction/Neyman-construction.002.png
14 | ```
15 |
16 | ````
17 | ````{tab} Step 3
18 |
19 | ```{figure} ../assets/Neyman-construction/Neyman-construction.003.png
20 | ```
21 |
22 | ````
23 | ````{tab} Step 4
24 |
25 | ```{figure} ../assets/Neyman-construction/Neyman-construction.004.png
26 | ```
27 |
28 | ````
29 | ````{tab} Step 5
30 |
31 | ```{figure} ../assets/Neyman-construction/Neyman-construction.005.png
32 | ```
33 |
34 | ````
35 | ````{tab} Step 6
36 |
37 | ```{figure} ../assets/Neyman-construction/Neyman-construction.006.png
38 | ```
39 |
40 | ````
41 | ````{tab} Step 7
42 |
43 | ```{figure} ../assets/Neyman-construction/Neyman-construction.007.png
44 | ```
45 |
46 | ````
47 | ````{tab} Step 8
48 |
49 | ```{figure} ../assets/Neyman-construction/Neyman-construction.008.png
50 | ```
51 |
52 | ````
53 | ````{tab} Step 9
54 |
55 | ```{figure} ../assets/Neyman-construction/Neyman-construction.009.png
56 | ```
57 |
58 | ````
59 | ````{tab} Step 10
60 |
61 | ```{figure} ../assets/Neyman-construction/Neyman-construction.010.png
62 | ```
63 |
64 | ````
65 | ````{tab} Step 11
66 |
67 | ```{figure} ../assets/Neyman-construction/Neyman-construction.011.png
68 | ```
69 |
70 | ````
71 | ````{tab} Step 12
72 |
73 | ```{figure} ../assets/Neyman-construction/Neyman-construction.012.png
74 | ```
75 |
76 | ````
77 | `````
78 |
79 |
80 | ## Generalizing to higher dimensional data
81 |
82 | ```{figure} ../assets/HCPSS-stats-lectures-2020.001.png
83 | ```
84 |
85 | ```{figure} ../assets/HCPSS-stats-lectures-2020.002.png
86 | ```
87 |
88 |
89 |
90 | ## Connection to Wilks's theorem
91 |
92 |
93 |
94 | `````{tabs}
95 | ````{tab} Step 1
96 |
97 | ```{figure} ../assets/wilks-delta-log-likelihood/wilks-delta-log-likelihood-1.gif
98 | ```
99 |
100 | ````
101 | ````{tab} Step 2
102 |
103 | ```{figure} ../assets/wilks-delta-log-likelihood/wilks-delta-log-likelihood-2.gif
104 | ```
105 |
106 | ````
107 | `````
--------------------------------------------------------------------------------
/book/section.md:
--------------------------------------------------------------------------------
1 | # Section title
2 |
3 | The "Section title" still uses a single #
4 |
5 | # Syllabus
6 |
7 | * Basics of probability
8 | * Probability models
9 | * Probability denstiy functions
10 | * Classic distributons
11 | * Bernouli
12 | * Binomial
13 | * Poisson
14 | * Gaussian
15 | * Chi-Square
16 | * Exponential family
17 | * Multivariate distributions
18 | * Independence
19 | * Covariance
20 | * Conditional distributions
21 | * Marginal distributions
22 | * Graphical Models
23 | * Copula
24 | * Information theory
25 | * Entropy
26 | * Mutual information
27 | * Implicit probabity models
28 | * Simulators
29 | * Probabilistic Programming
30 | * Likelihood function
31 | * Axioms of probability
32 | * Transformation properties
33 | * Change of variables
34 | * Propagation of errors
35 | * Reparameterization
36 | * Bayes Theorem
37 | * Subjective priors
38 | * Emperical Bayes
39 | * Jeffreys' prior
40 | * Unfiform priors
41 | * Reference Priors
42 | * Transformation Properties
43 | * Convolutions and the Central Limit Theorem
44 | * Binomial example
45 | * Convolutions in Fourier domain
46 | * Estimators
47 | * Bias, Variance, MSE
48 | * Cramer-Rao bound
49 | * Information Geometry
50 | * Sufficiency
51 | * Bias-Variance Tradeoff
52 | * James-Stein Paradox
53 | * Statistical Decision Theory
54 | * Hypothesis Testing
55 | * Simple vs. Compound hypotheses
56 | * Nuisance Parameters
57 | * TypeI and TypeII error
58 | * Test statistics
59 | * Neyman-Pearson Lemma
60 | * Confidence Intervals
61 | * Interpretation
62 | * Coverage
63 | * Power
64 | * No UMPU Tests
65 | * Neyman-Construction
66 | * Likelihood-Ratio tests
67 | * Profile likelihood
68 | * Profile construction
69 | * Asymptotic Properties of Likelihood Ratio
70 |
71 | * Bayesian Model Selection
72 | * Bayes Factors
73 | * BIC, etc.
74 | * Bayesian Credible Intervals
75 | * Interpretation
76 | * Metropolis Hastings
77 | * Variational Inference
78 | * LDA
79 | * Causality
80 |
81 |
82 |
83 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | # Byte-compiled / optimized / DLL files
2 | __pycache__/
3 | *.py[cod]
4 | *$py.class
5 |
6 | # C extensions
7 | *.so
8 |
9 | # Distribution / packaging
10 | .Python
11 | build/
12 | develop-eggs/
13 | dist/
14 | downloads/
15 | eggs/
16 | .eggs/
17 | lib/
18 | lib64/
19 | parts/
20 | sdist/
21 | var/
22 | wheels/
23 | pip-wheel-metadata/
24 | share/python-wheels/
25 | *.egg-info/
26 | .installed.cfg
27 | *.egg
28 | MANIFEST
29 |
30 | # PyInstaller
31 | # Usually these files are written by a python script from a template
32 | # before PyInstaller builds the exe, so as to inject date/other infos into it.
33 | *.manifest
34 | *.spec
35 |
36 | # Installer logs
37 | pip-log.txt
38 | pip-delete-this-directory.txt
39 |
40 | # Unit test / coverage reports
41 | htmlcov/
42 | .tox/
43 | .nox/
44 | .coverage
45 | .coverage.*
46 | .cache
47 | nosetests.xml
48 | coverage.xml
49 | *.cover
50 | *.py,cover
51 | .hypothesis/
52 | .pytest_cache/
53 |
54 | # Translations
55 | *.mo
56 | *.pot
57 |
58 | # Django stuff:
59 | *.log
60 | local_settings.py
61 | db.sqlite3
62 | db.sqlite3-journal
63 |
64 | # Flask stuff:
65 | instance/
66 | .webassets-cache
67 |
68 | # Scrapy stuff:
69 | .scrapy
70 |
71 | # Sphinx documentation
72 | docs/_build/
73 |
74 | # PyBuilder
75 | target/
76 |
77 | # Jupyter Notebook
78 | .ipynb_checkpoints
79 |
80 | # IPython
81 | profile_default/
82 | ipython_config.py
83 |
84 | # pyenv
85 | .python-version
86 |
87 | # pipenv
88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies
90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not
91 | # install all needed dependencies.
92 | #Pipfile.lock
93 |
94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow
95 | __pypackages__/
96 |
97 | # Celery stuff
98 | celerybeat-schedule
99 | celerybeat.pid
100 |
101 | # SageMath parsed files
102 | *.sage.py
103 |
104 | # Environments
105 | .env
106 | .venv
107 | env/
108 | venv/
109 | ENV/
110 | env.bak/
111 | venv.bak/
112 |
113 | # Spyder project settings
114 | .spyderproject
115 | .spyproject
116 |
117 | # Rope project settings
118 | .ropeproject
119 |
120 | # mkdocs documentation
121 | /site
122 |
123 | # mypy
124 | .mypy_cache/
125 | .dmypy.json
126 | dmypy.json
127 |
128 | # Pyre type checker
129 | .pyre/
130 |
131 | # General
132 | .DS_Store
133 |
134 | # Jupyter book
135 | _build/
136 | plots/
137 |
--------------------------------------------------------------------------------
/book/statistics/estimators.md:
--------------------------------------------------------------------------------
1 | # Estimators
2 |
3 | One of the main differences between topics of probability and topics in statistics is that in statistics we have some task in mind.
4 | While a probability model $P_X(X \mid \theta)$ is an object of study when discussing probability, in statistics we usually want to
5 | *do* something with it.
6 |
7 | The first example that we will consider is to estimate the true, unknown value $\theta^*$ given some dataset $\{x_i\}_{i=1}^N$
8 | assuming that the data were drawn from $X_i \sim p_X(X|\theta^*)$.
9 |
10 | ```{admonition} Definition
11 | An estimator $\hat{\theta}(x_1, \dots, x_N)$ is a function of the data (that aims to estimate the true, unknown value $\theta^*$ assuming that the data were drawn from $X_i \sim p_X(X|\theta^*)$.
12 | ```
13 |
14 | There are several concrete estimators for different quantities, but this is an abstract definition of what is meant by an estimator. It is useful to think of the estimator as a procedure that you apply to the data, and then you can ask about the properties of a given procedure.
15 |
16 |
17 | ```{admonition} Terminology
18 | These closely related terms have slightly different meanings:
19 | * The *estimand* refers to the parameter $\theta$ being estimated.
20 | * The *estimator* refers to the function or procedure $\hat{\theta}(x_1, \dots, x_N)$
21 | * The specific value that an estimator takes (returns) for specific data is known as the *estimate*.
22 | ```
23 |
24 | We already introduced two estimators when studying [Transformation properties of the likelihood and posterior](.distributions/invariance-of-likelihood-to-reparameterizaton.html#equivariance-of-the-mle):
25 | * The maximum likelihood estimator: $\hat{\theta}_\textrm{MLE} := \textrm{argmax}_\theta p(X=x \mid \theta)$
26 | * The maximum a posteriori estimator: $\hat{\theta}_{MAP} := \textrm{argmax}_\theta p(\theta \mid X=x)$
27 |
28 | Note both of these estimators are defined by procedures that you apply once you have specific data.
29 |
30 |
31 | ```{admonition} Notation
32 | The estimate $\hat{\theta}(X_1, \dots, X_N)$ depends on the random variables $X_i$, so it is itself a random variable (unlike the parameter $\theta$).
33 | Often the estimate is denoted $\hat{\theta}$ and the dependence on the data is implicit.
34 | Subscripts are often used to indicate which estimator is being used, eg. the maximum likelihood estimator $\hat{\theta}_\textrm{MLE}$ and the maximum a posteriori estimator $\hat{\theta}_\textrm{MAP}$.
35 | ```
36 |
37 | ```{hint}
38 | It is often useful to consider two straw man estimators:
39 | * A constant estimator: $\hat{\theta}_\textrm{const} = \theta_0$ for $\theta_0 \in \Theta$
40 | * A random estimator: $\hat{\theta}_\textrm{random} =$ some random value for $\theta$ independent of the data
41 | Neither of these are useful estimators, but they can be used to help clarify your thinking due to their obvious properties.
42 | ```
43 |
--------------------------------------------------------------------------------
/book/other_resources:
--------------------------------------------------------------------------------
1 |
2 | Note this is not a markdown file.
3 |
4 | 1. Introduction to Causal Inference by Brady Neal [Course website](https://www.bradyneal.com/causal-inference-course)
5 | 1. [Elements of Causal Inference by Jonas Peters, Dominik Janzing and Bernhard Schölkopf](https://mitpress.mit.edu/books/elements-causal-inference) [free PDF](https://www.dropbox.com/s/dl/gkmsow492w3oolt/11283.pdf)
6 |
7 | 1. [Probability and Statistics](https://cims.nyu.edu/~cfgranda/pages/DSGA1002_fall17/index.html)
8 | 1. [Inference and Representation](https://inf16nyu.github.io/home/)
9 | 1. [Big Data 2015](https://www.vistrails.org/index.php/Course:_Big_Data_2015)
10 | 1. [Stanford Prob](http://cs229.stanford.edu/section/cs229-prob.pdf)
11 | 1. Linear Algebra links:
12 | 1. [Essence of linear algebra youtube videos by 3blue1brown](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)
13 | 1. [Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares, Stephen Boyd and Lieven Vandenberghe](http://vmls-book.stanford.edu)
14 | 1. [Linear dynamical systems](https://www.youtube.com/watch?v=bf1264iFr-w&list=PLzvEnvQ9sS15pwCo8DYnJ-gArIkKZwJjF)
15 | 1. [Linear Algebra done right](https://linear.axler.net)
16 | 1. [NUMERICAL LINEAR ALGEBRA Lloyd N. Trefethen and David Bau, III](https://people.maths.ox.ac.uk/trefethen/text.html)
17 | 1. [Scientific Computing for PhDs](http://podcasts.ox.ac.uk/series/scientific-computing-dphil-students)
18 | 1. [Machine Learning](https://davidrosenberg.github.io/ml2017/#resources)
19 | 1. [PRML](https://github.com/cranmer/PRML)
20 | 1. [Mathematics for Machine Learning](https://mml-book.github.io)
21 | 1. Algorithms for Convex Optimization by Nisheeth K. Vishnoi [Course website](https://convex-optimization.github.io)
22 | 1. [Basic Python](https://swcarpentry.github.io/python-novice-inflammation/)
23 | 1. [Plotting and Programming with Python](https://swcarpentry.github.io/python-novice-gapminder/)
24 | 1. [Gentle Introduction to Automatic Differentiation on Kaggle](https://www.kaggle.com/borisettinger/gentle-introduction-to-automatic-differentiation)
25 |
26 | 1. [NeurIPS astro tutorial with datasets etc.](https://dwh.gg/NeurIPSastro)
27 |
28 | 1. [Paper about statistical combinations](https://arxiv.org/abs/2012.09874)
29 |
30 |
31 |
32 |
33 |
The 10 most helpful *free1. online machine learning courses, via @chipro
34 |
--------------------------------------------------------------------------------
/book/jupyterhub.md:
--------------------------------------------------------------------------------
1 | # JupyterHub for class
2 |
3 | In doing your work, you will need a python3 environment with several libraries installed. To streamline this, we created a JupyterHub instance with the necessary environment pre-installed. We will use this JupyterHub for some homework assignments that are graded with `nbgrader`. Below are the links to the
4 | * For students: [https://physga-2059-fall.rcnyu.org](https://physga-2059-fall.rcnyu.org)
5 | * For instructors: [https://physga-2059-fall-instructor.rcnyu.org](https://physga-2059-fall-instructor.rcnyu.org)
6 |
7 | Please give it a try and let us know how it works for you
8 |
9 | ```{tip}
10 | Course material will be put in the `shared` folder, which is read-only. You will need to copy the files to your home area to modify them.
11 | ```
12 |
13 |
14 | ```{tip}
15 | If you prefer the Jupyter Lab interface over the classic notebook, change the last part of the URL to "lab", e.g. [https://physga-2059-fall.rcnyu.org/user//lab/](https://physga-2059-fall.rcnyu.org/user//lab/) (and replace `` with your netid)
16 | ```
17 |
18 |
19 | ```{tip}
20 | The server will shutdown after 15 min of inactivity or (3 hours hard time limit). If you know you are done, click `Control Panel` in the top right and shutdown your server.
21 | ```
22 |
23 |
24 | ## Changing Kernels
25 |
26 | ```{tip}
27 | The default environment (kernel) is `Python 3`, you will need to change it to `Python [conda env:course]` to pick up the right environment with the installed libraries.
28 | ```
29 |
30 |
31 | `````{tabs}
32 | ````{tab} New Kernel
33 |
34 | ```{figure} ./assets/change_kernel_new.png
35 |
36 | Selecting the kernel for a new notebook
37 | ```
38 |
39 | ````
40 | ````{tab} Classic Notebook
41 |
42 | ```{figure} ./assets/change_kernel_classic.png
43 |
44 | Selecting the kernel for a the classic notebook
45 | ```
46 |
47 | ````
48 | ````{tab} Jupyter Lab
49 |
50 | ```{figure} ./assets/change_kernel_lab.png
51 |
52 | Selecting the kernel in Jupyter Lab
53 | ```
54 |
55 | ````
56 | `````
57 |
58 |
59 |
60 | %```{figure} ./assets/change_kernel_classic.png
61 | %
62 | %Selecting the kernel for a the classic notebook
63 | %```
64 | %
65 | %```{figure} ./assets/change_kernel_lab.png
66 | %
67 | %Selecting the kernel in Jupyter Lab
68 | %```
69 | %
70 | %```{figure} ./assets/change_kernel_new.png
71 | %
72 | %Selecting the kernel for a new notebook
73 | %```
74 |
75 | ## Documentation
76 |
77 | Overview and instructions
78 | [https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub](https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub)
79 |
80 | FAQ
81 | [https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub/faq](https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub/faq)
82 |
83 | Support
84 | [https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub/support](https://sites.google.com/a/nyu.edu/nyu-hpc/services/resources-for-classes/rc-jupyterhub/support)
85 |
86 |
87 |
--------------------------------------------------------------------------------
/book/distributions/introduction.md:
--------------------------------------------------------------------------------
1 | # Distributions
2 |
3 | When measuring a continuous quantity x the probability to get exactly a specific value of x is usually 0. Instead, one asks what is the probability to get x in some range.
4 | The probability to find x in the range [x,x+dx] is f(x) dx, where f(x) is called a probability density function, or “distribution” for short.
5 | Normalization
6 | The probability that one obtains x in the full range must be one, so we have the
7 | normalization condition Z
8 | f(x)dx = 1
9 | Change of variables
10 | If we change variables from x to y(x) then we should have P(a 1.
42 |
43 | ## Poisson Distribution [Ch 11]
44 |
45 | The Poisson distribution describes the probability to have n events occur when μ are expected. For example, if one expects μ =3.14 decays of a radioactive particle in one day, it gives the probability to observe n=0,1,2,3,4,.... decays in a day. Note, this is not a probability density because n is discrete.
46 | μn e μ
47 | Poisp(n|μ) =
48 | =μ n ̄=μ
49 | μ and σ=√μ. Notice that the relative uncertainty drops like: σ/μ ~ 1/√μ .
50 |
51 | ## Binomial Distribution [Ch 10]
52 | The binomial distribution describes the probability to have exactly k successes given n independent trials, when p is the probability of success for a single trial. The first part of the equation is a “combinatorial factor” that describes for all the ways one can have k successes. When n is much larger than k it is approximately a Poisson distribution.
53 | PAGE 2 OF 7
54 | PROF. KYLE CRANMER
55 | Binomial(k|n,p)=
56 | n! k n k
57 | (k)=np(1 p)
58 | p (1 p) k=np
59 |
60 |
61 |
--------------------------------------------------------------------------------
/book/intro.md:
--------------------------------------------------------------------------------
1 | # Statistics and Data Science
2 |
3 | This is the start of a book for a graduate-level course at NYU Physics titled *Statistics and Data Science*.
4 |
5 | Here are some of the objectives of this course:
6 |
7 | * **Learn essential concepts of probability**
8 |
9 | * Become familiar with how intuitive notions of probability are connected to formal foundations.
10 | * Overcome barriers presented by unfamiliar notation and terminology.
11 | * Internalize the transformation properties of distributions, the likelihood function, and other probabilistic objects.
12 | * Understand the differences between Bayesian and Frequentist approaches, particularly in the context of physical theories.
13 | * Connect these concepts to modern data science tools and techniques like the scientific python ecosystem and automatic differentiation.
14 |
15 | * **Learn essential concepts of statistics**
16 |
17 | * Learn classical statistical procedures: point estimates, goodness of fit tests, hypothesis tests, confidence intervals and credible intervals.
18 | * Become familiar with statistical decision theory
19 | * Recognize probabilistic programs as statistical models
20 | * Become familiar with the computational challenges found in statistical inference and techniques developed to overcome them.
21 | * Understand the difference between statistical associations and causal inference
22 |
23 | * **Learn essential concepts of software and computing**
24 |
25 | * Become familiar with the scientific python ecosystem
26 | * Become familiar with software testing via use of nbgrader
27 | * Become familiar with automatic differentiation & differentiable programming
28 | * Become familiar with probabilistic programming
29 |
30 | * **Learn essential concepts of machine learning**
31 |
32 | * Become familiar with core tasks such as classification and regression
33 | * Understand the notion of generalization
34 | * Understand the role of regularization and inductive bias
35 | * Become familiar with the taxonomy of different types of models found in machine learning: linear models, kernel methods, neural networks, deep learning
36 | * Become familiar with the interplay of model, data, and learning (optimization) algorithms
37 | * Touch on different learning settings: supervised learning, unsupervised learning, reinforcement learning
38 |
39 | * **Learn essential concepts of data science**
40 |
41 | * Understand how data science connects to the topics above
42 | * Gain confidence in using scientific python and modern data science tools to analyze real data
43 |
44 | ```{warning} Please note that the class website is under active development, and content will be added throughout the duration of the course.
45 | ```
46 |
47 |
48 | ```{tip} If you would like to audit this class, email Prof. Cranmer (kyle.cranmer at nyu ) with your NYU netID
49 | ```
50 |
51 | ```{note}
52 | In approaching this book I am torn between different styles. I like very much the atomic nature of [Quantum Field Theory by Mark Srednicki](https://www.amazon.com/Quantum-Field-Theory-Mark-Srednicki/dp/0521864496) as it is readable and a useful reference without too much narrative. On the other hand, I want to blend together the hands-on coding elements with fundamental concepts, and I am inspired by the book [Functional Differential Geometry by Gerald Jay Sussman and Jack Wisdom](https://mitpress.mit.edu/books/functional-differential-geometry).
53 | ```
--------------------------------------------------------------------------------
/book/_toc.yml:
--------------------------------------------------------------------------------
1 | - file: intro
2 |
3 | - part: About the course
4 | chapters:
5 | - file: schedule
6 | - file: jupyterhub
7 | - file: nbgrader
8 | - file: discussion_forum
9 | - file: preliminaries
10 |
11 | - part: Probability
12 | chapters:
13 | - file: probability-topics
14 | expand_sections: true
15 | sections:
16 | - file: random_variables
17 | - file: conditional
18 | - file: bayes_theorem
19 | - file: independence
20 | - file: empirical_distribution
21 | - file: expectation
22 | - file: correlation
23 | - file: datasaurus-long
24 | - file: distributions/visualize_marginals
25 | - file: measures_of_dependence
26 | - file: distributions/change-of-variables
27 | - file: distributions/one-over-x-flow
28 | - file: distributions/likelihood-change-obs
29 | - file: distributions/invariance-of-likelihood-to-reparameterizaton
30 | - file: error-propagation/investigating-propagation-of-errors
31 | - file: error-propagation/error_propagation_with_jax
32 | - file: distributions/accept-reject
33 | - file: distributions/Binomial_histograms-interactive
34 | - file: pgm/daft
35 | #- file: central-limit-theorem/Central-Limit-Theorem
36 |
37 | - part: Statistics
38 | chapters:
39 | - file: statistics-topics
40 | expand_sections: true
41 | sections:
42 | - file: statistics/estimators
43 | - file: statistics/bias-variance
44 | - file: statistics/investigation-bessels-correction
45 | - file: statistics/cramer-rao-bound
46 | - file: statistics/consistency
47 | - file: statistics/Neyman-Scott-phenomena
48 | - file: statistics/sufficiency
49 | - file: statistics/information-geometry
50 | - file: statistics/neyman_pearson
51 | - file: statistics/neyman_construction
52 | - file: statistics/lhc_stats_thumbnail
53 | - file: statistics/statistical_decision_theory
54 | - file: probprog/MarkovPath
55 |
56 | - part: Machine Learning
57 | chapters:
58 | - file: prml_notebooks/attribution
59 | expand_sections: true
60 | sections:
61 | - file: prml_notebooks/ch01_Introduction.ipynb
62 | - file: prml_notebooks/ch02_Probability_Distributions.ipynb
63 | - file: prml_notebooks/ch03_Linear_Models_for_Regression.ipynb
64 | - file: prml_notebooks/ch04_Linear_Models_for_Classfication.ipynb
65 | - file: prml_notebooks/ch05_Neural_Networks.ipynb
66 | - file: prml_notebooks/ch06_Kernel_Methods.ipynb
67 | - file: prml_notebooks/ch07_Sparse_Kernel_Machines.ipynb
68 | - file: prml_notebooks/ch08_Graphical_Models.ipynb
69 | - file: prml_notebooks/ch09_Mixture_Models_and_EM.ipynb
70 | - file: prml_notebooks/ch10_Approximate_Inference.ipynb
71 | - file: prml_notebooks/ch11_Sampling_Methods.ipynb
72 | - file: prml_notebooks/ch12_Continuous_Latent_Variables.ipynb
73 | - file: prml_notebooks/ch13_Sequential_Data.ipynb
74 |
75 | - part: Software and Computing
76 | chapters:
77 | - file: computing-topics
78 | - file: autodiff-tutorial
79 |
80 | - part: Data Science
81 | chapters:
82 | - file: data-science-topics
83 |
84 |
85 | - part: References
86 | chapters:
87 | - file: other_resources
88 | - file: bibliography
89 | - file: built-on
90 |
91 | - part: Jupyter Book Reference
92 | chapters:
93 | - file: markdown
94 | - file: cheatsheet
95 | - file: notebooks
96 | - file: interactive
97 | - file: test_embed_video
98 | - file: color-in-equations
99 | - file: test-sphinxext-opengraph
100 |
--------------------------------------------------------------------------------
/book/notebooks.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Content with notebooks\n",
8 | "\n",
9 | "You can also create content with Jupyter Notebooks. This means that you can include\n",
10 | "code blocks and their outputs in your book.\n",
11 | "\n",
12 | "## Markdown + notebooks\n",
13 | "\n",
14 | "As it is markdown, you can embed images, HTML, etc into your posts!\n",
15 | "\n",
16 | "\n",
17 | "\n",
18 | "You an also $add_{math}$ and\n",
19 | "\n",
20 | "$$\n",
21 | "math^{blocks}\n",
22 | "$$\n",
23 | "\n",
24 | "or\n",
25 | "\n",
26 | "$$\n",
27 | "\\begin{aligned}\n",
28 | "\\mbox{mean} la_{tex} \\\\ \\\\\n",
29 | "math blocks\n",
30 | "\\end{aligned}\n",
31 | "$$\n",
32 | "\n",
33 | "But make sure you \\$Escape \\$your \\$dollar signs \\$you want to keep!\n",
34 | "\n",
35 | "## MyST markdown\n",
36 | "\n",
37 | "MyST markdown works in Jupyter Notebooks as well. For more information about MyST markdown, check\n",
38 | "out [the MyST guide in Jupyter Book](https://jupyterbook.org/content/myst.html),\n",
39 | "or see [the MyST markdown documentation](https://myst-parser.readthedocs.io/en/latest/).\n",
40 | "\n",
41 | "## Code blocks and outputs\n",
42 | "\n",
43 | "Jupyter Book will also embed your code blocks and output in your book.\n",
44 | "For example, here's some sample Matplotlib code:"
45 | ]
46 | },
47 | {
48 | "cell_type": "code",
49 | "execution_count": null,
50 | "metadata": {},
51 | "outputs": [],
52 | "source": [
53 | "from matplotlib import rcParams, cycler\n",
54 | "import matplotlib.pyplot as plt\n",
55 | "import numpy as np\n",
56 | "plt.ion()"
57 | ]
58 | },
59 | {
60 | "cell_type": "code",
61 | "execution_count": null,
62 | "metadata": {},
63 | "outputs": [],
64 | "source": [
65 | "# Fixing random state for reproducibility\n",
66 | "np.random.seed(19680801)\n",
67 | "\n",
68 | "N = 10\n",
69 | "data = [np.logspace(0, 1, 100) + np.random.randn(100) + ii for ii in range(N)]\n",
70 | "data = np.array(data).T\n",
71 | "cmap = plt.cm.coolwarm\n",
72 | "rcParams['axes.prop_cycle'] = cycler(color=cmap(np.linspace(0, 1, N)))\n",
73 | "\n",
74 | "\n",
75 | "from matplotlib.lines import Line2D\n",
76 | "custom_lines = [Line2D([0], [0], color=cmap(0.), lw=4),\n",
77 | " Line2D([0], [0], color=cmap(.5), lw=4),\n",
78 | " Line2D([0], [0], color=cmap(1.), lw=4)]\n",
79 | "\n",
80 | "fig, ax = plt.subplots(figsize=(10, 5))\n",
81 | "lines = ax.plot(data)\n",
82 | "ax.legend(custom_lines, ['Cold', 'Medium', 'Hot']);"
83 | ]
84 | },
85 | {
86 | "cell_type": "markdown",
87 | "metadata": {},
88 | "source": [
89 | "There is a lot more that you can do with outputs (such as including interactive outputs)\n",
90 | "with your book. For more information about this, see [the Jupyter Book documentation](https://executablebooks.github.io/cli/start/overview.html)"
91 | ]
92 | }
93 | ],
94 | "metadata": {
95 | "kernelspec": {
96 | "display_name": "Python 3",
97 | "language": "python",
98 | "name": "python3"
99 | },
100 | "language_info": {
101 | "codemirror_mode": {
102 | "name": "ipython",
103 | "version": 3
104 | },
105 | "file_extension": ".py",
106 | "mimetype": "text/x-python",
107 | "name": "python",
108 | "nbconvert_exporter": "python",
109 | "pygments_lexer": "ipython3",
110 | "version": "3.8.0"
111 | },
112 | "widgets": {
113 | "application/vnd.jupyter.widget-state+json": {
114 | "state": {},
115 | "version_major": 2,
116 | "version_minor": 0
117 | }
118 | }
119 | },
120 | "nbformat": 4,
121 | "nbformat_minor": 4
122 | }
123 |
--------------------------------------------------------------------------------
/book/statistics/statistical_decision_theory.md:
--------------------------------------------------------------------------------
1 | # Statistical decision theory
2 |
3 | Work in progress, initially just copying over from Wikipedia article: [Admissible decision rule](https://en.wikipedia.org/wiki/Admissible_decision_rule)
4 |
5 | Define sets $\Theta$, ${\mathcal {X}}$, and ${\mathcal {A}}$, where
6 | * $\Theta$ are the states of nature,
7 | * ${\mathcal {X}}$ the possible observations, and
8 | * ${\mathcal {A}}$ the actions that may be taken.
9 |
10 | An observation $x\in {\mathcal {X}}$ is distributed as $F(x\mid \theta )$ and therefore provides evidence about the state of nature
11 | $\theta \in \Theta$.
12 |
13 | A decision rule is a function
14 | $\delta :{{\mathcal {X}}}\rightarrow {{\mathcal {A}}}$, where upon observing $x\in {\mathcal {X}}$, we choose to take action $\delta (x)\in {\mathcal {A}}$.
15 |
16 | Also define a loss function $L:\Theta \times {\mathcal {A}}\rightarrow {\mathbb {R}}$, which specifies the loss we would incur by taking action
17 | $a\in {\mathcal {A}}$ when the true state of nature is $\theta \in \Theta$. Usually we will take this action after observing data $x\in {\mathcal {X}}$, so that the loss will be $L(\theta ,\delta (x))$. (It is possible though unconventional to recast the following definitions in terms of a utility function, which is the negative of the loss.)
18 |
19 | Define the risk function as the expectation $R(\theta ,\delta )=\operatorname {E}_{{F(x\mid \theta )}}[{L(\theta ,\delta (x))]}.\,\!$
20 |
21 | Whether a decision rule $\delta\,\!$ has low risk depends on the true state of nature $\theta$. A decision rule $\delta ^{*}$ dominates a decision rule $\delta$ if and only if $R(\theta ,\delta ^{*})\leq R(\theta ,\delta )$ for all
22 | $\theta$, and the inequality is strict for some
23 | $\theta$.
24 |
25 | ## Bayes rules:
26 |
27 | Let $\pi (\theta )$ be a probability distribution on the states of nature. From a Bayesian point of view, we would regard it as a prior distribution. That is, it is our believed probability distribution on the states of nature, prior to observing data. For a frequentist, it is merely a function on
28 | $\Theta$ with no such special interpretation. The Bayes risk of the decision rule
29 | $\delta$ with respect to $\pi (\theta )$ is the expectation
30 | \begin{equation}
31 | r(\pi ,\delta )=\operatorname {E}_{{\pi (\theta )}}[R(\theta ,\delta )].
32 | \end{equation}
33 | A decision rule $\delta$ that minimizes
34 | $r(\pi ,\delta )$ is called a Bayes rule with respect to $\pi (\theta )$. There may be more than one such Bayes rule.
35 |
36 |
37 | ## Generalized Bayes rules:
38 |
39 | In the Bayesian approach to decision theory, the observed
40 | $x$ is considered fixed. Whereas the frequentist approach (i.e., risk) averages over possible samples
41 | $x\in {\mathcal {X}}$ the Bayesian would fix the observed sample
42 | $x$ and average over hypotheses
43 | $\theta \in \Theta$. Thus, the Bayesian approach is to consider for our observed $x$ the expected loss.
44 | \begin{equation}
45 | \rho (\pi ,\delta \mid x)=\operatorname {E}_{{\pi (\theta \mid x)}}[L(\theta ,\delta (x))]
46 | \end{equation}
47 | where the expectation is over the posterior of
48 | $\theta$ given $x$ (obtained from
49 | $\pi (\theta )$ and
50 | $F(x\mid \theta )$ using Bayes' theorem).
51 |
52 | Having made explicit the expected loss for each given
53 | $x$ separately, we can define a decision rule
54 | $\delta$ by specifying for each
55 | $x$ an action
56 | $\delta (x)$ that minimizes the expected loss. This is known as a generalized Bayes rule with respect to
57 | $\pi (\theta )$. There may be more than one generalized Bayes rule, since there may be multiple choices of
58 | $\delta (x)$ that achieve the same expected loss.
59 |
60 | According to the complete class theorems, under mild conditions every admissible rule is a (generalized) Bayes rule (with respect to \textit{some} prior
61 | $\pi (\theta )$ —- possibly an improper one -— that favors distributions
62 | $\theta$ where that rule achieves low risk). Thus, in frequentist decision theory it is sufficient to consider only (generalized) Bayes rules.
63 |
--------------------------------------------------------------------------------
/book/markdown.md:
--------------------------------------------------------------------------------
1 | # Markdown Files
2 |
3 | Whether you write your book's content in Jupyter Notebooks (`.ipynb`) or
4 | in regular markdown files (`.md`), you'll write in the same flavor of markdown
5 | called **MyST Markdown**.
6 |
7 | ## What is MyST?
8 |
9 | MyST stands for "Markedly Structured Text". It
10 | is a slight variation on a flavor of markdown called "CommonMark" markdown,
11 | with small syntax extensions to allow you to write **roles** and **directives**
12 | in the Sphinx ecosystem.
13 |
14 | ## What are roles and directives?
15 |
16 | Roles and directives are two of the most powerful tools in Jupyter Book. They
17 | are kind of like functions, but written in a markup language. They both
18 | serve a similar purpose, but **roles are written in one line**, whereas
19 | **directives span many lines**. They both accept different kinds of inputs,
20 | and what they do with those inputs depends on the specific role or directive
21 | that is being called.
22 |
23 | ### Using a directive
24 |
25 | At its simplest, you can insert a directive into your book's content like so:
26 |
27 | ````
28 | ```{mydirectivename}
29 | My directive content
30 | ```
31 | ````
32 |
33 | This will only work if a directive with name `mydirectivename` already exists
34 | (which it doesn't). There are many pre-defined directives associated with
35 | Jupyter Book. For example, to insert a note box into your content, you can
36 | use the following directive:
37 |
38 | ````
39 | ```{note}
40 | Here is a note
41 | ```
42 | ````
43 |
44 | This results in:
45 |
46 | ```{note}
47 | Here is a note
48 | ```
49 |
50 | In your built book.
51 |
52 | For more information on writing directives, see the
53 | [MyST documentation](https://myst-parser.readthedocs.io/).
54 |
55 |
56 | ## Refering to equation
57 |
58 | By adding `` {eq}`my_label` `` {eq}`autoregressive`
59 |
60 | ### Using a role
61 |
62 | Roles are very similar to directives, but they are less-complex and written
63 | entirely on one line. You can insert a role into your book's content with
64 | this pattern:
65 |
66 | ```
67 | Some content {rolename}`and here is my role's content!`
68 | ```
69 |
70 | Again, roles will only work if `rolename` is a valid role's name. For example,
71 | the `doc` role can be used to refer to another page in your book. You can
72 | refer directly to another page by its relative path. For example, the
73 | role syntax `` {doc}`intro` `` will result in: {doc}`intro`.
74 |
75 | For more information on writing roles, see the
76 | [MyST documentation](https://myst-parser.readthedocs.io/).
77 |
78 |
79 | ### Adding a citation
80 |
81 | You can also cite references that are stored in a `bibtex` file. For example,
82 | the following syntax: `` {cite}`holdgraf_evidence_2014` `` will render like
83 | this: {cite}`holdgraf_evidence_2014`.
84 |
85 | Moreoever, you can insert a bibliography into your page with this syntax:
86 | The `{bibliography}` directive must be used for all the `{cite}` roles to
87 | render properly.
88 | For example, if the references for your book are stored in `references.bib`,
89 | then the bibliography is inserted with:
90 |
91 | ````
92 | ```{bibliography} references.bib
93 | ```
94 | ````
95 |
96 | Resulting in a rendered bibliography that looks like:
97 |
98 | ```{bibliography} references.bib
99 | ```
100 |
101 |
102 | ### Executing code in your markdown files
103 |
104 | If you'd like to include computational content inside these markdown files,
105 | you can use MyST Markdown to define cells that will be executed when your
106 | book is built. Jupyter Book uses *jupytext* to do this.
107 |
108 | First, add Jupytext metadata to the file. For example, to add Jupytext metadata
109 | to this markdown page, run this command:
110 |
111 | ```
112 | jupyter-book myst init markdown.md
113 | ```
114 |
115 | Once a markdown file has Jupytext metadata in it, you can add the following
116 | directive to run the code at build time:
117 |
118 | ````
119 | ```{code-cell}
120 | print("Here is some code to execute")
121 | ```
122 | ````
123 |
124 | When your book is built, the contents of any `{code-cell}` blocks will be
125 | executed with your default Jupyter kernel, and their outputs will be displayed
126 | in-line with the rest of your content.
127 |
128 | For more information about executing computational content with Jupyter Book,
129 | see [The MyST-NB documentation](https://myst-nb.readthedocs.io/).
130 |
--------------------------------------------------------------------------------
/book/nbgrader.md:
--------------------------------------------------------------------------------
1 | # nbgrader
2 |
3 | Please watch this video to become familiar with how assignments via the notebook work.
4 |
5 |
6 |
7 | [Documentation:](https://nbgrader.readthedocs.io/en/stable/)
8 |
9 |
10 | ## Instructions:
11 |
12 |
13 | 1. login to JupyterHub: [https://physga-2059-fall.rcnyu.org](https://physga-2059-fall.rcnyu.org)
14 |
15 | 1. You will see the files in your home area and tabs for Files, Running, Clusters, Assignments, Nbextensions. **Click the Assignments tab**.
16 |
17 |
18 | 1. You should see Released Assignments, Downloaded Assignments, and Submitted Assignments. If there are new assignments, then you should have a Fetch button. **Click the Fetch button**.
19 |
20 | 1. This should create a new folder in your home area with the name of the assignment, and it may have more than one notebook inside.
21 |
22 | ```{figure} ./assets/nbgrader-fetch.png
23 | ```
24 |
25 | 1. In the Downloaded assignments area, you will see the assignment name with an arrow. **Click the arrow to see the notebooks inside**.
26 |
27 | ```{figure} ./assets/nbgrader-assignments.png
28 | ```
29 | 1. DO NOT click the Submit button yet.
30 | 1. You can click the validate button to see what happens. It will show several messages with `NotImplementedError`: -- that's expected, how nbgrader indicates that you need to fill in some code.
31 |
32 | 1. Click on one of the notebooks. (This will take you to the classic notebook interface. If you want, you can use JupyterLab. At this point the notebook is just like any other notebook in your home directory. You can make some changes, save them, logout, come back, make more changes, etc. no problem. )
33 | 1. Now you should go throught the notebook starting at the top. Read the code and the notes carefully to understand what is going on. You can execute the cells one by one (Shift-Enter) as you go along. At some point you will find
34 | ```python
35 | # YOUR CODE HERE
36 | raise NotImplementedError()
37 | ```
38 | If you run this cell it will raise an error. **You should replace `raise NotImplementedError()` with your implementation** (which maybe a several lines long). Usually there will be a comment just above this that describes what the function or code snippet should do.
39 | 1. Once you've written that code, you should be able to execute the cell without errors and continue.
40 |
41 | 1. Later in the notebook you will encouter some tests. They look like something like this usually:
42 | ```python
43 | """Check that mu1 returns the correct output for several inputs"""
44 | assert_almost_equal(myfunction(some_input), expected_value)
45 | ```
46 | This is how nbgrader will automatically grade the assignments. It's also closely connected to the idea of unit testing in software development.
47 | The tests should be there so that you can be reasonably sure that the code is doing what it is supposed to do.
48 | 1. If the tests fail, then you should go back and work on your implementation until the tests pass.
49 | 1. WARNING! You probably want to restart the kernel and rerun all the cells (up to the part you are on) everytime you change things. If you execute the cells out of order, then global variables may have different values than they would have if you just ran the notebook from scratch.
50 | 1. Note: there can be some additional hidden tests that are used during grading, but not visible to you.
51 |
52 | 1. Once you make it to the end of the notebook and you are satisfied, then you are almost ready to submit.
53 | 1. Make sure you save the notebook
54 | ```{figure} ./assets/nbgrader-validate.png
55 | ```
56 |
57 | 1. In the menu bar you should see a button that says "Validate". Click it to check that all the checks pass.
58 | 1. Alternatively, you can validate the notebook from the Assignments tab in the Jupyter Homepage
59 | 1. Go back to Jupyter homepage (you can click on the Jupyter logo in the top left of the notebook)
60 | 1. Click on the assignments tab
61 | 1. Expand the list of notebooks in the assignment
62 | 1. If you haven't already, you can click Validate for the notebooks
63 | 1. If they pass, then click Submit
64 | ```{figure} ./assets/nbgrader-assignments.png
65 | ```
66 |
67 |
--------------------------------------------------------------------------------
/book/measures_of_dependence.md:
--------------------------------------------------------------------------------
1 | # Quantifying statistical dependence
2 |
3 | ```{math}
4 | \newcommand\indep{\perp\kern-5pt\perp}
5 | ```
6 |
7 |
8 | As we saw earlier, two random variables may be *uncorrelated* (the covariance of two random variables may be zero), but that does not imply the two variables are independent.
9 | This figure from the wikipedia article on [Correlation and Dependence](http://en.wikipedia.org/wiki/Correlation_and_dependence) is a good illustration. The bottom row shows examples of two variables that are uncorrelated, but not statistically independent (eg. we can't factorize the joint $p(X,Y)$ as $p(X)p(Y)$).
10 |
11 |
13 |
14 | So how can we quantify if and two what degree two variables are statistically dependent?
15 |
16 | ## Mutual Information
17 |
18 | The [**Mutual information**](https://en.wikipedia.org/wiki/Mutual_information) is
19 | of two random variables is a measure of the mutual dependence between the two variables. It quantifies the "amount of information" obtained about one random variable through observing the other random variable. The concept of mutual information is intimately linked to that of entropy of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.[^footnote1]
20 |
21 | ```{important} The **mutual information** $I(X;Y)=0$ *if and only if* $X \indep Y$.
22 |
23 | ```
24 |
25 |
26 | The mutual information of two jointly discrete random variables
27 | $X$ and $Y$ is calculated as a double sum
28 |
29 | $$
30 | {\displaystyle \operatorname {I} (X;Y)=\sum _{y\in {\mathcal {Y}}}\sum _{x\in {\mathcal {X}}}{p_{(X,Y)}(x,y)\log {\left({\frac {p_{(X,Y)}(x,y)}{p_{X}(x)\,p_{Y}(y)}}\right)}},}
31 | $$
32 |
33 | where ${\displaystyle p_{(X,Y)}}$ is the joint probability mass function of $X$ and $Y$ and $p_{X}$ and $p_Y$ are the marginal probability mass functions.$X$ and $Y$ respectively.
34 |
35 | In the case of jointly continuous random variables, the double sum is replaced by a double integral
36 |
37 | $$
38 | {\displaystyle \operatorname {I} (X;Y)=\int _{\mathcal {Y}}\int _{\mathcal {X}}{p_{(X,Y)}(x,y)\log {\left({\frac {p_{(X,Y)}(x,y)}{p_{X}(x)\,p_{Y}(y)}}\right)}}\;dx\,dy,}
39 | $$
40 |
41 | where ${\displaystyle p_{(X,Y)}}$ is now the joint probability density function and $p_{X}$ and $p_Y$ are the marginal probability density functions.
42 |
43 | If the log base 2 is used, the units of mutual information are bits.
44 |
45 | An equivalent formulation is
46 |
47 | $$
48 | {\displaystyle I(X;Y)=D_{\mathrm {KL} }(P_{(X,Y)}\|P_{X}\otimes P_{Y})}
49 | $$
50 |
51 | where
52 | $D_{{{\mathrm {KL}}}}$ is the [Kullback–Leibler](https://en.wikipedia.org/wiki/Kullback–Leibler_divergence) divergence, which we will return to later in the course. Here we see that it is the KL distance between the joint and the product of the two marginals, and so it is only zero if the those are identical, which is equivalent to saying $p(X,Y)= p(X)p(Y)$, which is the definition of independence.
53 |
54 | Another useful identity is:
55 |
56 | $$
57 | {\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&{}\equiv \mathrm {H} (X)-\mathrm {H} (X|Y)\\&{}\equiv \mathrm {H} (Y)-\mathrm {H} (Y|X)\\&{}\equiv \mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y)\\&{}\equiv \mathrm {H} (X,Y)-\mathrm {H} (X|Y)-\mathrm {H} (Y|X)\end{aligned}}}
58 | $$
59 |
60 | where ${\displaystyle \mathrm {H} (X)}$ and ${\displaystyle \mathrm {H} (Y)}$ are the marginal [entropies](https://en.wikipedia.org/wiki/Information_entropy),
61 | ${\displaystyle \mathrm {H} (X|Y)}$ and ${\displaystyle \mathrm {H} (Y|X)}$ are the [conditional entropies](https://en.wikipedia.org/wiki/Conditional_entropy), and
62 | ${\displaystyle \mathrm {H} (X,Y)}$ is the [joint entropy](https://en.wikipedia.org/wiki/Joint_entropy) of $X$ and $Y$.
63 |
64 | ```{note} The mutual information is symmetric $I(X;Y)=I(Y;X)$ and non-negative $I(X;Y)\ge 0$.
65 |
66 | ```
67 |
68 | ## Distance Correlation
69 |
70 | [Distance Correlation](https://en.wikipedia.org/wiki/Distance_correlation) is a measure of dependence between two paired random vectors of arbitrary, not necessarily equal, dimension.
71 | Thus, distance correlation measures both linear and nonlinear association between two random variables or random vectors. This is in contrast to Pearson's correlation, which can only detect linear association between two random variables [^footnote2].
72 |
73 | ```{important} The **distance correlation** is zero *if and only if* $X \indep Y$.
74 |
75 | ```
76 |
77 | [^footnote1]: Adapted from [https://en.wikipedia.org/wiki/Mutual_information](https://en.wikipedia.org/wiki/Mutual_information)
78 |
79 | [^footnote2]: Adapted from [https://en.wikipedia.org/wiki/Distance_correlation](https://en.wikipedia.org/wiki/Distance_correlation)
--------------------------------------------------------------------------------
/book/other_resources.md:
--------------------------------------------------------------------------------
1 | # Other Resources
2 |
3 |
4 |
5 | ## Courses
6 | 1. [NYU CDS: Probability and Statistics](https://cims.nyu.edu/~cfgranda/pages/DSGA1002_fall17/index.html)
7 | 1. [Stanford Probability and Statistics](http://cs229.stanford.edu/section/cs229-prob.pdf)
8 | 1. [NYU CDS: Inference and Representation](https://inf16nyu.github.io/home/)
9 | 1. [NYU CDS: Big Data 2015](https://www.vistrails.org/index.php/Course:_Big_Data_2015)
10 | 1. [NYU CDS: Machine Learning](https://davidrosenberg.github.io/ml2017/#resources)
11 | 1. [Foundations of Graphical Models by David Blei](http://www.cs.columbia.edu/~blei/fogm/2016F/) -- see [Basics of Graphical Models](http://www.cs.columbia.edu/~blei/fogm/2016F/doc/graphical-models.pdf)
12 | 1. see also [a video on d-separation by Pieter Abbeel](https://www.youtube.com/watch?v=yDs_q6jKHb0)
13 | 1. semantics of graphical models (here called "Boiler plate diagrams") and an extended visual language [Directed Factor Graph Notation for Generative Models
14 | Laura Dietz](https://github.com/jluttine/tikz-bayesnet/blob/master/dietz-techreport.pdf), which is the basis of the `tikz-bayesnet` package
15 | 1. [Algorithms for Convex Optimization by Nisheeth K. Vishnoi](https://convex-optimization.github.io)
16 | 1. [Introduction to Causal Inference by Brady Neal](https://www.bradyneal.com/causal-inference-course)
17 | 1. [Michael Jordan's lecture notes on notes on Probabilistic Graphical Models](https://people.eecs.berkeley.edu/%7Ejordan/prelims/)
18 | 1. [MIT lecture notes on algorithms for inference](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-438-algorithms-for-inference-fall-2014/lecture-notes/)
19 | 1. [Kevin Murphy, Machine Learning: a Probabilistic Perspective (4th eddition)](http://www.cs.ubc.ca/%7Emurphyk/MLbook/index.html) | [online @ NYU Libraries](http://site.ebrary.com/lib/nyulibrary/detail.action?docID=10597102).
20 | 1. [Probabilistic Programming and Bayesian Methods for Hackers by Cam Davidson Pilon](https://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)
21 |
22 | ## Short courses / tutorials
23 |
24 | 1. [Basic Python](https://swcarpentry.github.io/python-novice-inflammation/)
25 | 1. [Plotting and Programming with Python](https://swcarpentry.github.io/python-novice-gapminder/)
26 |
27 |
28 | ## Linear Algebra
29 | 1. [Essence of linear algebra youtube videos by 3blue1brown](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)
30 | 1. [Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares, Stephen Boyd and Lieven Vandenberghe](http://vmls-book.stanford.edu)
31 | 1. [Linear dynamical systems](https://www.youtube.com/watch?v=bf1264iFr-w&list=PLzvEnvQ9sS15pwCo8DYnJ-gArIkKZwJjF)
32 | 1. [Linear Algebra done right](https://linear.axler.net)
33 | 1. [NUMERICAL LINEAR ALGEBRA Lloyd N. Trefethen and David Bau, III](https://people.maths.ox.ac.uk/trefethen/text.html)
34 | 1. [Scientific Computing for PhDs](http://podcasts.ox.ac.uk/series/scientific-computing-dphil-students)
35 |
36 |
37 | ## Books
38 |
39 | 1. [All of Statistics by Wasserman](https://www.amazon.com/All-Statistics-Statistical-Inference-Springer/dp/1441923225)
40 | 1. [PRML](https://github.com/cranmer/PRML)
41 | 1. [Mathematics for Machine Learning](https://mml-book.github.io)
42 | 1. [Elements of Causal Inference by Jonas Peters, Dominik Janzing and Bernhard Schölkopf](https://mitpress.mit.edu/books/elements-causal-inference) [free PDF](https://www.dropbox.com/s/dl/gkmsow492w3oolt/11283.pdf)
43 | 1. [Trevor Hastie, Rob Tibshirani, and Jerry Friedman, Elements of Statistical Learning, Second Edition, Springer, 2009](https://web.stanford.edu/~hastie/ElemStatLearn//)
44 |
45 | ## Influential texts
46 |
47 | 1. [Knuth Calculus](https://micromath.wordpress.com/2008/04/14/donald-knuth-calculus-via-o-notation/)
48 | 1. [Functional Differential Geometry by Gerald Jay Sussman and Jack Wisdom](https://mitpress.mit.edu/books/functional-differential-geometry)
49 |
50 | ## Misc
51 |
52 | 1. [NeurIPS astro tutorial with datasets etc.](https://dwh.gg/NeurIPSastro)
53 | 1. [Paper about statistical combinations from phys/astro authors](https://arxiv.org/abs/2012.09874)
54 | 1. [Gentle Introduction to Automatic Differentiation on Kaggle](https://www.kaggle.com/borisettinger/gentle-introduction-to-automatic-differentiation)
55 | 1. [Short notes on divergence measures by Danilo Rezende](https://danilorezende.com/wp-content/uploads/2018/07/divergences.pdf)
56 | 1. [Lecture notes on: Information-theoretic methods for high-dimensional statistics, by Yihong Wu](http://www.stat.yale.edu/~yw562/teaching/it-stats.pdf)
57 |
58 |
59 |
60 | ## Meta
61 |
62 |
The 10 most helpful *free1. online machine learning courses, via @chipro
31 | All content on this site (unless otherwise specified) is licensed under the CC BY-NC-SA 4.0 license
32 | 
46 |
47 | The lack of an edge directly between $A$ and $B$ indicates that the two varaibles are conditionally independent. This image was produced with `daft`, and there are more examples in [Visualizing Graphical Models](./pgm/daft).
48 |
49 | [^footnote1]: This text is based on excerpts from Section 1.3 of [NYU CDS lecture notes on Probability and Statistics](https://cims.nyu.edu/~cfgranda/pages/stuff/probability_stats_for_DS.pdf)
50 |
--------------------------------------------------------------------------------
/book/_config.yml:
--------------------------------------------------------------------------------
1 | # Book settings
2 | title: Statistics and Data Science
3 | author: Kyle Cranmer
4 | logo: logo.png
5 | copyright: ""
6 |
7 | parse:
8 | myst_extended_syntax: true
9 |
10 | execute:
11 | exclude_patterns : ["*/Central-Limit-Theorem.ipynb","*/prop-error-plots.ipynb","*/track-example.ipynb"]
12 | execute_notebooks : off # force, off, auto
13 |
14 | # Information about where the book exists on the web
15 | repository:
16 | url: https://github.com/cranmer/stats-ds-book
17 | path_to_book: book
18 | branch: master
19 |
20 | html:
21 | home_page_in_navbar : true
22 | use_repository_button: true
23 | use_issues_button: true
24 | use_edit_page_button: true
25 | google_analytics_id: UA-178330963-1
26 | comments:
27 | hypothesis: true
28 | extra_footer : |
29 | 
12 |