{{ page.date | date: "%B %-d, %Y" }}
6 | 7 | 8 | {{ content }} 9 | -------------------------------------------------------------------------------- /_layouts/post.html: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | --- 4 |{{ page.date | date: "%B %-d, %Y" }}
6 | 7 | 8 | 32 | 33 | 34 | {{ content }} 35 | 36 | -------------------------------------------------------------------------------- /_plugins/fullwidth.rb: -------------------------------------------------------------------------------- 1 | ## This has a fairly harmless hack that wraps the img tag in a div to prevent it from being 2 | ## wrapped in a paragraph tag instead, which would totally fuck things up layout-wise 3 | ## Usage {% fullwidth 'path/to/image' 'caption goes here in quotes' %} 4 | # 5 | module Jekyll 6 | class RenderFullWidthTag < Liquid::Tag 7 | 8 | require "shellwords" 9 | 10 | def initialize(tag_name, text, tokens) 11 | super 12 | @text = text.shellsplit 13 | end 14 | 15 | def render(context) 16 | baseurl = context.registers[:site].config['baseurl'] 17 | if @text[0].start_with?('http://', 'https://','//') 18 | "
25 | 
50 | 
75 | These notes form a concise introductory course on deep generative models. 81 | They are based on Stanford CS236, taught by Stefano Ermon and Aditya Grover, and have been written by Aditya Grover, with the help of many students and course staff. 82 | The notes are still under construction! 83 | Since these notes are brand new, you will find several typos. If you do, please let us know, or submit a pull request with your fixes to our Github repository. 84 | You too may help make these notes better by submitting your improvements to us via Github.
85 | 86 |Intelligent agents are constantly generating, acquiring, and processing 81 | data. This data could be in the form of images that we capture on our 82 | phones, text messages we share with our friends, graphs that model 83 | interactions on social media, videos that record important events, 84 | etc. Natural agents excel at discovering patterns, extracting 85 | knowledge, and performing complex reasoning based on the data they observe. How 86 | can we build artificial learning systems to do the same?
87 | 88 |In this course, we will study generative models that view the world under the lens of probability. 89 | In such a worldview, we can think of any kind of 90 | observed data, say , as a finite set of samples from an 91 | underlying distribution, say . At its very core, the 92 | goal of any generative model is then to approximate this data 93 | distribution given access to the dataset . The hope is that 94 | if we are able to learn a good generative model, we can use the 95 | learned model for downstream inference.
96 | 97 |We will be primarily interested in parametric approximations to the data 100 | distribution, which summarize all the information about the dataset in 101 | a finite set of parameters. In contrast with non-parametric models, 102 | parametric models scale more efficiently with large datasets but are 103 | limited in the family of distributions they can represent.
104 | 105 |In the parametric setting, we can think of the task of learning a 106 | generative model as picking the parameters within a family of model 107 | distributions that minimizes some notion of distance1 between the 108 | model distribution and the data distribution.
109 | 110 |
113 |
For instance, we might be given access to a dataset of dog images and 117 | our goal is to learn the paraemeters of a generative model within a model family such that 118 | the model distribution is close to the data distribution over dogs 119 | . Mathematically, we can specify our goal as the 120 | following optimization problem: \begin{equation} 121 | \min_{\theta\in \mathcal{M}}d(p_{\mathrm{data}}, p_{\theta}) 122 | \label{eq:learning_gm} 123 | \tag{1} 124 | \end{equation}where is accessed via the dataset 125 | and is a notion of distance between probability distributions.
126 | 127 |As we navigate through this course, it is interesting to take note of 128 | the difficulty of the problem at hand. A typical image from a modern 129 | phone camera has a resolution of approximately pixels. 130 | Each pixel has three channels: R(ed), G(reen) and B(lue) and each 131 | channel can take a value between 0 to 255. Hence, the number of possible 132 | images is given by . 133 | In contrast, Imagenet, one of the largest publicly available datasets, 134 | consists of only about 15 million images. Hence, learning a generative 135 | model with such a limited dataset is a highly underdetermined problem.
136 | 137 |Fortunately, the real world is highly structured and automatically 138 | discovering the underlying structure is key to learning generative 139 | models. For example, we can hope to learn some basic artifacts about 140 | dogs even with just a few images: two eyes, two ears, fur etc. Instead 141 | of incorporating this prior knowledge explicitly, we will hope the model 142 | learns the underlying structure directly from data. There is no free 143 | lunch however, and indeed successful learning of generative models will 144 | involve instantiating the optimization problem in 145 | in a suitable way. In this course, we will be 146 | primarily interested in the following questions:
147 | 148 |In the next few set of lectures, we will take a deeper dive into certain 155 | families of generative models. For each model family, we will note how 156 | the representation is closely tied with the choice of learning objective 157 | and the optimization procedure.
158 | 159 |For a discriminative model such as logistic regression, the fundamental 162 | inference task is to predict a label for any given datapoint. Generative 163 | models, on the other hand, learn a joint distribution over the entire 164 | data.2
165 | 166 |While the range of applications to which generative models have been 167 | used continue to grow, we can identify three fundamental inference 168 | queries for evaluating a generative model.:
169 | 170 |Density estimation: Given a datapoint , what is the 173 | probability assigned by the model, i.e., ?
174 |Sampling: How can we generate novel data from the model 177 | distribution, i.e., 178 | ?
179 |Unsupervised representation learning: How can we learn meaningful 182 | feature representations for a datapoint ?
183 |Going back to our example of learning a generative model over dog 187 | images, we can intuitively expect a good generative model to work as 188 | follows. For density estimation, we expect to be 189 | high for dog images and low otherwise. Alluding to the name generative 190 | model, sampling involves generating novel images of dogs beyond the 191 | ones we observe in our dataset. Finally, representation learning can 192 | help discover high-level structure in the data such as the breed of 193 | dogs.
194 | 195 |In light of the above inference tasks, we note two caveats. First, 196 | quantitative evaluation of generative models on these tasks is itself 197 | non-trivial (in particular, sampling and representation learning) and an 198 | area of active research. Some quantitative metrics exist, but these 199 | metrics often fail to reflect desirable qualitative attributes in the 200 | generated samples and the learned representations. Secondly, not all 201 | model families permit efficient and accurate inference on all these 202 | tasks. Indeed, the trade-offs in the inference capabilities of the 203 | current generative models have led to the development of very diverse approaches as 204 | we shall see in this course.
205 | 206 |As we shall see later, functions that do not satisfy all 212 | properties of a distance metric are also used in practice, e.g., KL 213 | divergence. ↩
214 |Technically, a probabilistic discriminative model is also a 217 | generative model of the labels conditioned on the data. However, the 218 | usage of the term generative models is typically reserved for high 219 | dimensional data. ↩
220 |
41 |
43 |
44 |
45 | For instance, we might be given access to a dataset of dog images $$\mathcal{D}$$ and
46 | our goal is to learn the parameters of a generative model $$\theta$$ within a model family $$\mathcal{M}$$ such that
47 | the model distribution $$p_\theta$$ is close to the data distribution over dogs
48 | $$p_{\mathrm{data}}$$. Mathematically, we can specify our goal as the
49 | following optimization problem: $$$$\begin{equation}
50 | \min_{\theta\in \mathcal{M}}d(p_{\mathrm{data}}, p_{\theta})
51 | \label{eq:learning_gm}
52 | \tag{1}
53 | \end{equation}$$$$where $$p_{\mathrm{data}}$$ is accessed via the dataset
54 | $$\mathcal{D}$$ and $$d(\cdot)$$ is a notion of distance between probability distributions.
55 |
56 | As we navigate through this course, it is interesting to take note of
57 | the difficulty of the problem at hand. A typical image from a modern
58 | phone camera has a resolution of approximately $$700 \times 1400$$ pixels.
59 | Each pixel has three channels: R(ed), G(reen) and B(lue) and each
60 | channel can take a value between 0 to 255. Hence, the number of possible
61 | images is given by $$256^{700 \times 1400 \times 3}\approx 10 ^{800000}$$.
62 | In contrast, ImageNet, one of the largest publicly available datasets,
63 | consists of only about 15 million images. Hence, learning a generative
64 | model with such a limited dataset is a highly underdetermined problem.
65 |
66 | Fortunately, the real world is highly structured and automatically
67 | discovering the underlying structure is key to learning generative
68 | models. For example, we can hope to learn some basic artifacts about
69 | dogs even with just a few images: two eyes, two ears, fur etc. Instead
70 | of incorporating this prior knowledge explicitly, we will hope the model
71 | learns the underlying structure directly from data. There is no free
72 | lunch however, and indeed successful learning of generative models will
73 | involve instantiating the optimization problem in
74 | $$(\ref{eq:learning_gm})$$ in a suitable way. In this course, we will be
75 | primarily interested in the following questions:
76 |
77 | * What is the representation for the model family $$\mathcal{M}$$?
78 | * What is the objective function $$d(\cdot)$$?
79 | * What is the optimization procedure for minimizing $$d(\cdot)$$?
80 |
81 | In the next few set of lectures, we will take a deeper dive into certain
82 | families of generative models. For each model family, we will note how
83 | the representation is closely tied with the choice of learning objective
84 | and the optimization procedure.
85 |
86 | Inference
87 | ---------
88 |
89 | For a discriminative model such as logistic regression, the fundamental
90 | inference task is to predict a label for any given datapoint. Generative
91 | models, on the other hand, learn a joint distribution over the entire
92 | data.[^2]
93 |
94 | While the range of applications to which generative models have been
95 | used continue to grow, we can identify three fundamental inference
96 | queries for evaluating a generative model.:
97 |
98 | 1. *Density estimation:* Given a datapoint $$\mathbf{x}$$, what is the
99 | probability assigned by the model, i.e., $$p_\theta(\mathbf{x})$$?
100 |
101 | 2. *Sampling:* How can we *generate* novel data from the model
102 | distribution, i.e.,
103 | $$\mathbf{x}_{\mathrm{new}} \sim p_\theta(\mathbf{x})$$?
104 |
105 | 3. *Unsupervised representation learning:* How can we learn meaningful
106 | feature representations for a datapoint $$\mathbf{x}$$?
107 |
108 | Going back to our example of learning a generative model over dog
109 | images, we can intuitively expect a good generative model to work as
110 | follows. For density estimation, we expect $$p_\theta(\mathbf{x})$$ to be
111 | high for dog images and low otherwise. Alluding to the name *generative
112 | model*, sampling involves generating novel images of dogs beyond the
113 | ones we observe in our dataset. Finally, representation learning can
114 | help discover high-level structure in the data such as the breed of
115 | dogs.
116 |
117 | In light of the above inference tasks, we note two caveats. First,
118 | quantitative evaluation of generative models on these tasks is itself
119 | non-trivial (in particular, sampling and representation learning) and an
120 | area of active research. Some quantitative metrics exist, but these
121 | metrics often fail to reflect desirable qualitative attributes in the
122 | generated samples and the learned representations. Secondly, not all
123 | model families permit efficient and accurate inference on all these
124 | tasks. Indeed, the trade-offs in the inference capabilities of the
125 | current generative models have led to the development of very diverse approaches as
126 | we shall see in this course.
127 |
128 |
129 | ## Footnotes
130 |
131 | [^1]: As we shall see later, functions that do not satisfy all
132 | properties of a distance metric are also used in practice, e.g., KL
133 | divergence.
134 |
135 | [^2]: Technically, a probabilistic discriminative model is also a
136 | generative model of the labels conditioned on the data. However, the
137 | usage of the term generative models is typically reserved for high
138 | dimensional data.
139 |
--------------------------------------------------------------------------------
/introduction/learning.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepgenerativemodels/notes/bd2303339eaaea884870125b473cc1ae8c980d51/introduction/learning.png
--------------------------------------------------------------------------------
/introduction/learning_1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepgenerativemodels/notes/bd2303339eaaea884870125b473cc1ae8c980d51/introduction/learning_1.png
--------------------------------------------------------------------------------
/introduction/learning_2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepgenerativemodels/notes/bd2303339eaaea884870125b473cc1ae8c980d51/introduction/learning_2.png
--------------------------------------------------------------------------------
/vae/klgap.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepgenerativemodels/notes/bd2303339eaaea884870125b473cc1ae8c980d51/vae/klgap.png
--------------------------------------------------------------------------------
/vae/vae.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepgenerativemodels/notes/bd2303339eaaea884870125b473cc1ae8c980d51/vae/vae.png
--------------------------------------------------------------------------------