![](\"images/ocsvm.png\")
"
81 | ]
82 | },
83 | {
84 | "cell_type": "markdown",
85 | "metadata": {},
86 | "source": [
87 | "The governing equation and constraints are\n",
88 | "\n",
89 | "$$\n",
90 | "\\min_{\\beta, \\zeta, \\rho} \\frac{1}{2} \\|\\beta\\|^2 + \\frac{1}{\\nu n}\\sum_{j=1}^n \\zeta_j - \\rho \\\\\n",
91 | "\\mbox{subject to } \\left\\{ \\begin{array} {cl} \n",
92 | " h(x_{j\\cdot}) \\cdot \\beta \\ge \\rho -\\zeta_j & \\\\\n",
93 | " \\zeta_j \\ge 0 ,\n",
94 | "\\end{array}\\right.\n",
95 | "$$\n",
96 | "\n",
97 | "where $h(x_j)$ is a kernel function, $\\rho$ is distance from the origin to the hyperplane, and $\\nu$ is a hyperparameter that is the upper bound for the fraction of training error and the lower bound for the fraction of support vectors. Notice how the constraint is forcing points to be at least $\\rho$ away from the margin, lest it incurs a margin violation as $\\zeta$ must be set large enough to satisfy the inequality.\n",
98 | "\n",
99 | "Let's use the one-class SVM to see how it works. We will be using the wine data set provided by `scikit-learn`. The details of the data set are not important, only that it has 178 observations with 13 numerical features.\n",
100 | "\n",
101 | "**Question**\n",
102 | "* What technique should we use on the wine data set to easily visualize our analysis?"
103 | ]
104 | },
105 | {
106 | "cell_type": "code",
107 | "execution_count": null,
108 | "metadata": {},
109 | "outputs": [],
110 | "source": [
111 | "from sklearn.datasets import load_wine\n",
112 | "from sklearn.preprocessing import StandardScaler\n",
113 | "from sklearn.decomposition import PCA\n",
114 | "from sklearn.pipeline import Pipeline\n",
115 | "\n",
116 | "# load data set\n",
117 | "data = load_wine()\n",
118 | "X = data['data']\n",
119 | "\n",
120 | "# truncate to two variables\n",
121 | "pipe = Pipeline([('scaler', StandardScaler()), ('dim_red', PCA(n_components=2))])\n",
122 | "Xt = pipe.fit_transform(X)\n",
123 | "\n",
124 | "# generate novel/outlier points\n",
125 | "np.random.seed(1)\n",
126 | "theta = 2*np.pi*np.random.random(10)\n",
127 | "X_test = np.vstack((4*np.cos(theta) + np.random.random(10), 4*np.sin(theta) + np.random.random(10)))\n",
128 | "\n",
129 | "plt.scatter(*Xt.T)\n",
130 | "plt.scatter(*X_test, c='red')\n",
131 | "plt.xlabel('$\\\\xi_1$')\n",
132 | "plt.ylabel('$\\\\xi_2$');\n",
133 | "plt.legend([\"training set\", \"novel/outliers\"]);"
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {},
139 | "source": [
140 | "The visualization below plots the data along with the boundaries that determine whether a point is consider novel or not. The filled contour lines in the plot represent values of the decision function of the one-class SVM. The `decision_function` method reports the signed distance (negative means on the wrong side) between the point and the hyperplane. The visualization allows you to modify $\\nu$, the upper bound for the false positive rate. You can also consider $\\nu$ as the probability of having a new but regular observation outside the region defining regular points. As $\\nu$ decreases, the area encompassing the regular points increases. As with the standard kernelized SVM, you can change the kernel function, but `rbf` usually works the best."
141 | ]
142 | },
143 | {
144 | "cell_type": "code",
145 | "execution_count": null,
146 | "metadata": {},
147 | "outputs": [],
148 | "source": [
149 | "from sklearn.svm import OneClassSVM\n",
150 | "from ipywidgets import interact, FloatSlider\n",
151 | "\n",
152 | "def plot_one_class_svm(X, X_test):\n",
153 | " def plotter(nu=0.95):\n",
154 | " clf = OneClassSVM(nu=nu, gamma='auto')\n",
155 | " clf.fit(X)\n",
156 | " y_pred = clf.predict(X)\n",
157 | " fp_rate = (y_pred == -1).sum()/len(X)\n",
158 | " \n",
159 | " X1, X2 = np.meshgrid(np.linspace(-5, 5), np.linspace(-5, 5))\n",
160 | " y_proba = clf.decision_function(np.hstack((X1.reshape(-1, 1), X2.reshape(-1, 1))))\n",
161 | " Z = y_proba.reshape(50, 50)\n",
162 | " \n",
163 | " fig = plt.figure(figsize=(8, 5), facecolor='w', edgecolor='k')\n",
164 | " plt.contourf(X1, X2, Z, levels=np.linspace(Z.min(), 0, 7), cmap=plt.cm.Blues_r)\n",
165 | " plt.colorbar()\n",
166 | " a = plt.contour(X1, X2, Z, levels=[0], linewidths=2, colors='black') \n",
167 | " b1 = plt.scatter(*X.T, c='blue')\n",
168 | " b2 = plt.scatter(*X_test, c='red')\n",
169 | " plt.title(\"false positive rate: {:g}\".format(fp_rate))\n",
170 | " plt.legend([a.collections[0], b1, b2], [\"boundary\", \" true inliers\", \"true outliers\"], frameon=True, \n",
171 | " loc=\"lower left\")\n",
172 | " return plotter\n",
173 | "\n",
174 | "nu_slider = FloatSlider(min=0.01, max=0.99, step=0.01, value=0.5, description='$\\\\nu$')\n",
175 | "interact(plot_one_class_svm(Xt, X_test), nu=nu_slider);"
176 | ]
177 | },
178 | {
179 | "cell_type": "markdown",
180 | "metadata": {},
181 | "source": [
182 | "## Isolation Forest\n",
183 | "\n",
184 | "Isolation forests is an outlier detection algorithm that uses decision trees. The principle isolation forest works on is that outliers are points with features that are considerably different than the rest of the data, the inliers. Consider a data set in a $p$-dimensional space. The inliers will be closer together while the outliers will be farther apart. As we have seen, decision trees divide up a $p$-dimensional space using orthogonal cuts. \n",
185 | "\n",
186 | "Consider a decision tree that is constructed by making random cuts with randomly chosen features. The tree is allowed to grow until all points have been isolated. It will be easier to isolate or box in the outliers than the inliers. In other words, less splits are required to isolate outliers compared to the inliers and outlier nodes will reside closer to the root node. The process of constructing a tree with random cuts is repeated to create an ensemble, hence the term forest in the name of the algorithm. For each data point, the average path/splits required to isolate the point is a metric for the regularity or normality of the point. While the algorithm could have adopted a more sophisticated manner to isolate points, making random splits is computationally cheap and averaging across all trees considers the multiple manners to isolate the data. The two key hyperparameters are\n",
187 | "\n",
188 | "* `n_estimators`: The number of trees to use in the ensemble.\n",
189 | "* `contamination`: The fraction of outliers in the data set.\n",
190 | "\n",
191 | "The `decision_function` method returns a score for a set of observations, a negative or positive score means the observation is labeled as an outlier or inlier, respectively. This score is related to the path length, number of splits, to isolate each observation, averaged across all trees in the forest, but with an offset,\n",
192 | "\n",
193 | "$$\n",
194 | "\\text{score} = \\text{mean path length} - \\text{offset},\n",
195 | "$$\n",
196 | "\n",
197 | "where the offset is chosen based on the set contamination level. For example, if the contamination fraction was set to 0.2, then the offset if chosen such that 20% of the training data have a negative score. Let's visualize the result of using an isolation forest on the wine data set."
198 | ]
199 | },
200 | {
201 | "cell_type": "code",
202 | "execution_count": null,
203 | "metadata": {},
204 | "outputs": [],
205 | "source": [
206 | "from sklearn.ensemble import IsolationForest\n",
207 | "\n",
208 | "def plot_isolation_forest(X, X_test):\n",
209 | " def plotter(contamination=0.2):\n",
210 | " clf = IsolationForest(n_estimators=100, contamination=contamination, behaviour='new')\n",
211 | " clf.fit(X)\n",
212 | " \n",
213 | " y_pred = clf.predict(X)\n",
214 | " outlier_rate = (y_pred == -1).sum()/len(X)\n",
215 | " \n",
216 | " X1, X2 = np.meshgrid(np.linspace(-5, 5), np.linspace(-5, 5))\n",
217 | " y_proba = clf.decision_function(np.hstack((X1.reshape(-1, 1), X2.reshape(-1, 1))))\n",
218 | " Z = y_proba.reshape(50, 50)\n",
219 | " \n",
220 | " fig = plt.figure(figsize=(8, 5), facecolor='w', edgecolor='k')\n",
221 | " plt.contourf(X1, X2, Z, levels=np.linspace(Z.min(), 0, 7), cmap=plt.cm.Blues_r)\n",
222 | " plt.colorbar()\n",
223 | " a = plt.contour(X1, X2, Z, levels=[0], linewidths=2, colors='black') \n",
224 | " b1 = plt.scatter(*X.T, c='blue')\n",
225 | " b2 = plt.scatter(*X_test, c='red')\n",
226 | " plt.title(\"outlier fraction: {:g}\".format(outlier_rate))\n",
227 | " plt.legend([a.collections[0], b1, b2], [\"boundary\", \" true inliers\", \"true outliers\"], frameon=True, \n",
228 | " loc=\"lower left\") \n",
229 | " return plotter\n",
230 | "\n",
231 | "cont_slider = FloatSlider(min=0.01, max=0.5, value=0.1, step=0.01, description=\"fraction\")\n",
232 | "interact(plot_isolation_forest(Xt, X_test), contamination=cont_slider);"
233 | ]
234 | },
235 | {
236 | "cell_type": "markdown",
237 | "metadata": {},
238 | "source": [
239 | "**Questions**\n",
240 | "\n",
241 | "* Which algorithm, isolation forest or one-class SVM, has a better training time complexity? What influenced your decision?\n",
242 | "* What advantages of decision trees in general would still be present in the isolation forest algorithm?"
243 | ]
244 | },
245 | {
246 | "cell_type": "markdown",
247 | "metadata": {},
248 | "source": [
249 | "## Comparison of one-class SVM and isolation forest\n",
250 | "\n",
251 | "Here are a few things to be aware between one-class SVM and isolation forest.\n",
252 | "\n",
253 | "* Both algorithms are capable of properly modeling multi-modal data sets.\n",
254 | "* One class SVM is sensitive to outliers, making it more appropriate for novelty detection, when the training data is not contaminated with outliers.\n",
255 | "* Since the splits of the decision tree are chosen at random, isolation forest is faster to train.\n",
256 | "* In general, SVM are slow to train, especially with respect to the training set size.\n",
257 | "\n",
258 | "Additionally, the two methods inherit the pros and cons of their parent algorithm. There are other outlier and novelty detection algorithms available in `scikit-learn` and a comparison and overview of other methods are outlined [here](https://scikit-learn.org/stable/auto_examples/plot_anomaly_comparison.html#sphx-glr-auto-examples-plot-anomaly-comparison-py)."
259 | ]
260 | },
261 | {
262 | "cell_type": "markdown",
263 | "metadata": {},
264 | "source": [
265 | "## Anomaly detection in a time series\n",
266 | "\n",
267 | "Anomaly detection can be applied to a time series where we want to create a baseline model and determine the deviation of the observations with the baseline. If the deviation is large enough, the observation is deemed anomalous and is flagged. In general, novelty and outlier detection does not tell us _why_ something is possibility an outlier, the conditions and causes that led to an unusual observation. For example, if we are observing server logs, anomalous observations may be a result of some equipment or code breakdown or something malignant like a security breach.\n",
268 | "\n",
269 | "In this case study, we analyze appliance energy use for a 4.5 month time period. Data was collected at a sampling rate of 10 minutes. Given the large variability in energy usage at a sampling rate of 10 minutes, we will resample the time series at an hourly interval. More of the data set can be learned [here](https://archive.ics.uci.edu/ml/datasets/Appliances+energy+prediction)."
270 | ]
271 | },
272 | {
273 | "cell_type": "code",
274 | "execution_count": null,
275 | "metadata": {},
276 | "outputs": [],
277 | "source": [
278 | "df = pd.read_csv(\"data/energy_data.csv\", parse_dates=[\"date\"]) \n",
279 | "df = df.set_index('date')\n",
280 | "df_hourly = df.resample(\"H\").mean() # resample hourly\n",
281 | "\n",
282 | "energy = df_hourly['Appliances']\n",
283 | "energy.plot()\n",
284 | "plt.ylabel(\"energy (Wh)\");"
285 | ]
286 | },
287 | {
288 | "cell_type": "markdown",
289 | "metadata": {},
290 | "source": [
291 | "**Questions**\n",
292 | "* What should our plan of attack be for analyzing the time series?\n",
293 | "* What do you observe in the time series?"
294 | ]
295 | },
296 | {
297 | "cell_type": "markdown",
298 | "metadata": {},
299 | "source": [
300 | "### Fourier Analysis\n",
301 | "\n",
302 | "While it is hard to tell, there are periodic behaviors in the time series. We can better spotlight the dominant frequencies that support the time series using Fourier analysis."
303 | ]
304 | },
305 | {
306 | "cell_type": "code",
307 | "execution_count": null,
308 | "metadata": {},
309 | "outputs": [],
310 | "source": [
311 | "from scipy import fftpack\n",
312 | "\n",
313 | "sampling_rate = (energy.index[1] - energy.index[0]).total_seconds()\n",
314 | "sampling_rate = sampling_rate / (60 * 60 * 24) # day\n",
315 | "\n",
316 | "Y = fftpack.fft(energy - energy.mean())\n",
317 | "freq = np.linspace(0, 1/sampling_rate, len(Y))\n",
318 | "\n",
319 | "plt.plot(freq[:len(freq)//2], np.abs(Y[:len(Y)//2]))\n",
320 | "plt.xlabel(\"cycles per day\")\n",
321 | "plt.ylabel(\"Fourier transform\");"
322 | ]
323 | },
324 | {
325 | "cell_type": "markdown",
326 | "metadata": {},
327 | "source": [
328 | "The time series has four dominant frequencies: daily, twice-daily, three times a day, and four times a day. In other words, 6, 8, 12, and 24 hour periods. Given our everyday experience, we probably would have anticipated these frequencies/periods but it is reassuring that they can be revealed via Fourier analysis."
329 | ]
330 | },
331 | {
332 | "cell_type": "markdown",
333 | "metadata": {},
334 | "source": [
335 | "### Incorporating day of the week\n",
336 | "\n",
337 | "With time series data, a commonly generated feature is the day of the week for the observations. It might be tempting to capture day of the week behavior using a sinusoidal component but let's analyze the energy usage for each day of the week to understand how to best model it."
338 | ]
339 | },
340 | {
341 | "cell_type": "code",
342 | "execution_count": null,
343 | "metadata": {},
344 | "outputs": [],
345 | "source": [
346 | "df_day_of_week = pd.DataFrame({'day': energy.index.dayofweek, 'count': energy.values})\n",
347 | "grouped_by_day = df_day_of_week.groupby('day')\n",
348 | "\n",
349 | "grouped_by_day.mean().plot(kind='bar')\n",
350 | "plt.xticks(range(7), ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']);"
351 | ]
352 | },
353 | {
354 | "cell_type": "markdown",
355 | "metadata": {},
356 | "source": [
357 | "The energy usage as a function of the day of the week fluctuates but is not periodic; it is best captured using one hot encoded features. We will include one hot encoded features for each day of the week to our baseline model."
358 | ]
359 | },
360 | {
361 | "cell_type": "markdown",
362 | "metadata": {},
363 | "source": [
364 | "### Initial baseline model\n",
365 | "\n",
366 | "We will need to create some custom transformers to work with our pandas times series data. Specifically, we need a transformer to extract out the indices, create Fourier components, and transform `datetime` objects into a unit of time. A common reference point used to translate a date into a unit of time is [Unix time](https://en.wikipedia.org/wiki/Unix_time). It is defined as the time since 00:00:00 Thursday, 1 January 1970 Coordinated Universal Time (UTC), minus leap seconds."
367 | ]
368 | },
369 | {
370 | "cell_type": "code",
371 | "execution_count": null,
372 | "metadata": {},
373 | "outputs": [],
374 | "source": [
375 | "from sklearn.base import BaseEstimator, TransformerMixin\n",
376 | "\n",
377 | "class IndexSelector(BaseEstimator, TransformerMixin):\n",
378 | "\n",
379 | " def __init__(self):\n",
380 | " \"\"\"Return indices of a data frame for use in other estimators.\"\"\"\n",
381 | " pass\n",
382 | "\n",
383 | " def fit(self, df, y=None):\n",
384 | " return self\n",
385 | "\n",
386 | " def transform(self, df):\n",
387 | " return df.index\n",
388 | "\n",
389 | "class FourierComponents(BaseEstimator, TransformerMixin):\n",
390 | "\n",
391 | " def __init__(self, freqs):\n",
392 | " \"\"\"Create features based on sin(2*pi*f*t) and cos(2*pi*f*t).\"\"\"\n",
393 | " self.freqs = freqs\n",
394 | "\n",
395 | " def fit(self, X, y=None):\n",
396 | " return self\n",
397 | "\n",
398 | " def transform(self, X):\n",
399 | " Xt = np.zeros((X.shape[0], 2 * len(self.freqs)))\n",
400 | " t_0 = X[0]\n",
401 | " for i, f in enumerate(self.freqs):\n",
402 | " Xt[:, 2 * i] = np.cos(2 * np.pi * f * (X)).reshape(-1)\n",
403 | " Xt[:, 2 * i + 1] = np.sin(2 * np.pi * f * (X)).reshape(-1)\n",
404 | "\n",
405 | " return Xt\n",
406 | "\n",
407 | "class EpochTime(BaseEstimator, TransformerMixin):\n",
408 | "\n",
409 | " def __init__(self, unit):\n",
410 | " \"\"\"Transform datetime object to some unit of time since the start of the epoch.\"\"\"\n",
411 | " self.unit = unit\n",
412 | "\n",
413 | " def fit(self, X, y=None):\n",
414 | " return self\n",
415 | "\n",
416 | " def transform(self, X):\n",
417 | " epoch_time = np.array([x.value for x in X])\n",
418 | "\n",
419 | " if self.unit == \"seconds\":\n",
420 | " return epoch_time / (1000000000)\n",
421 | " elif self.unit == \"minutes\":\n",
422 | " return epoch_time / (1000000000) / 60\n",
423 | " elif self.unit == \"hours\":\n",
424 | " return epoch_time / (1000000000) / 60 / 60\n",
425 | " else:\n",
426 | " return epoch_time\n",
427 | " \n",
428 | "class DayOfWeek(BaseEstimator, TransformerMixin):\n",
429 | "\n",
430 | " def __init__(self):\n",
431 | " \"\"\"Determine the day of the week for datetime objects.\"\"\"\n",
432 | " pass\n",
433 | "\n",
434 | " def fit(self, X, y=None):\n",
435 | " return self\n",
436 | "\n",
437 | " def transform(self, X):\n",
438 | " return np.array([x.dayofweek for x in X]).reshape(-1, 1)"
439 | ]
440 | },
441 | {
442 | "cell_type": "markdown",
443 | "metadata": {},
444 | "source": [
445 | "Some additional useful functions for our analysis."
446 | ]
447 | },
448 | {
449 | "cell_type": "code",
450 | "execution_count": null,
451 | "metadata": {},
452 | "outputs": [],
453 | "source": [
454 | "def ts_train_test_split(df, cutoff, target):\n",
455 | " \"\"\"Perform a train/test split on a data frame based on a cutoff date.\"\"\"\n",
456 | " \n",
457 | " ind = df.index < cutoff\n",
458 | " \n",
459 | " df_train = df.loc[ind]\n",
460 | " df_test = df.loc[~ind]\n",
461 | " y_train = df.loc[ind, target]\n",
462 | " y_test = df.loc[~ind, target]\n",
463 | " \n",
464 | " return df_train, df_test, y_train, y_test\n",
465 | "\n",
466 | "def plot_results(df, y_pred):\n",
467 | " \"\"\"Plot predicted results and residuals.\"\"\"\n",
468 | " \n",
469 | " plt.plot(df.index, y_pred, '-r')\n",
470 | " energy.plot()\n",
471 | " plt.ylabel('energy (Wh)')\n",
472 | " plt.legend(['true', 'predicted'])\n",
473 | " plt.show();\n",
474 | "\n",
475 | " plt.plot(resd)\n",
476 | " plt.ylabel('residual');"
477 | ]
478 | },
479 | {
480 | "cell_type": "code",
481 | "execution_count": null,
482 | "metadata": {},
483 | "outputs": [],
484 | "source": [
485 | "from sklearn.linear_model import LinearRegression\n",
486 | "from sklearn.pipeline import FeatureUnion, Pipeline\n",
487 | "from sklearn.preprocessing import OneHotEncoder\n",
488 | "\n",
489 | "# perform train/test split\n",
490 | "cutoff = \"Mar-2016\" # roughly corresponding to 10% of the data\n",
491 | "df_train, df_test, y_train, y_test = ts_train_test_split(df_hourly, cutoff, 'Appliances')\n",
492 | "\n",
493 | "# construct and train model\n",
494 | "freqs = np.array([1, 2, 3]) / 24 / 60 # 24, 12, and 8 hour periods\n",
495 | "selector = IndexSelector()\n",
496 | "epoch_time = EpochTime(\"minutes\")\n",
497 | "fourier_components = FourierComponents(freqs)\n",
498 | "one_hot = OneHotEncoder(sparse=False, categories='auto')\n",
499 | "lr = LinearRegression()\n",
500 | "\n",
501 | "fourier = Pipeline([(\"time\", epoch_time),\n",
502 | " (\"sine_cosine\", fourier_components)])\n",
503 | "day_of_week = Pipeline([(\"day\", DayOfWeek()),\n",
504 | " (\"encoder\", one_hot)])\n",
505 | "union = FeatureUnion([(\"fourier\", fourier),\n",
506 | " (\"day_of_week\", day_of_week)])\n",
507 | "\n",
508 | "pipe = Pipeline([(\"indices\", selector),\n",
509 | " (\"union\", union),\n",
510 | " (\"regressor\", lr)])\n",
511 | "pipe.fit(df_train, y_train)\n",
512 | "\n",
513 | "# make predictions\n",
514 | "y_pred = pipe.predict(df_hourly)\n",
515 | "resd = energy - y_pred\n",
516 | "print(\"Test set R^2: {:g}\".format(pipe.score(df_test, y_test)))\n",
517 | "plot_results(df_hourly, y_pred)"
518 | ]
519 | },
520 | {
521 | "cell_type": "markdown",
522 | "metadata": {},
523 | "source": [
524 | "It is very apparent that the initial baseline model is not adequate for the time series. The residuals reveal:\n",
525 | "\n",
526 | "1. No long term trends.\n",
527 | "1. The time series has a lot of shock events, large increases in energy use, probably as a result of sudden and short use of an appliance.\n",
528 | "\n",
529 | "Let's next analyze the residuals for any temporal correlations."
530 | ]
531 | },
532 | {
533 | "cell_type": "markdown",
534 | "metadata": {},
535 | "source": [
536 | "### Noise based features\n",
537 | "\n",
538 | "The first thing we want to unveil is the correlation of past residuals with current values. An autocorrelation plot will inform us of the characteristic time scale of the process to guide us when generating noise based features."
539 | ]
540 | },
541 | {
542 | "cell_type": "code",
543 | "execution_count": null,
544 | "metadata": {},
545 | "outputs": [],
546 | "source": [
547 | "from pandas.plotting import autocorrelation_plot\n",
548 | "\n",
549 | "autocorrelation_plot(resd)\n",
550 | "plt.xlabel('Lag (hour)')\n",
551 | "plt.xlim([0, 50]);"
552 | ]
553 | },
554 | {
555 | "cell_type": "markdown",
556 | "metadata": {},
557 | "source": [
558 | "The time scale appears to be anywhere from 10 to 20 hours. Let's incorporate window based features from our residuals to improve our model."
559 | ]
560 | },
561 | {
562 | "cell_type": "code",
563 | "execution_count": null,
564 | "metadata": {},
565 | "outputs": [],
566 | "source": [
567 | "from sklearn.base import RegressorMixin\n",
568 | "\n",
569 | "class ResidualFeatures(BaseEstimator, TransformerMixin):\n",
570 | " def __init__(self, window=100):\n",
571 | " \"\"\"Generate features based on window statistics of past noise/residuals.\"\"\"\n",
572 | " self.window = window\n",
573 | " \n",
574 | " def fit(self, X, y=None):\n",
575 | " return self\n",
576 | " \n",
577 | " def transform(self, X):\n",
578 | " df = pd.DataFrame()\n",
579 | " df['residual'] = X\n",
580 | " df['prior'] = df['residual'].shift(1)\n",
581 | " df['mean'] = df['residual'].rolling(window=self.window).mean()\n",
582 | " df['diff'] = df['residual'].diff().rolling(window=self.window).mean()\n",
583 | " df = df.fillna(method='bfill')\n",
584 | " \n",
585 | " return df\n",
586 | " \n",
587 | "class FullModel(BaseEstimator, RegressorMixin):\n",
588 | " def __init__(self, baseline, residual_model, steps=20):\n",
589 | " \"\"\"Combine a baseline and residual model to predict any number of steps in the future.\"\"\"\n",
590 | " \n",
591 | " self.baseline = baseline\n",
592 | " self.residual_model = residual_model\n",
593 | " self.steps = steps\n",
594 | " \n",
595 | " def fit(self, X, y):\n",
596 | " self.baseline.fit(X, y)\n",
597 | " resd = y - self.baseline.predict(X)\n",
598 | " self.residual_model.fit(resd.iloc[:-self.steps], resd.shift(-self.steps).dropna())\n",
599 | " \n",
600 | " return self\n",
601 | " \n",
602 | " def predict(self, X):\n",
603 | " y_b = pd.Series(self.baseline.predict(X), index=X.index)\n",
604 | " resd = X['Appliances'] - y_b\n",
605 | " resd_pred = pd.Series(self.residual_model.predict(resd), index=X.index)\n",
606 | " resd_pred = resd_pred.shift(self.steps)\n",
607 | " y_pred = y_b + resd_pred\n",
608 | " \n",
609 | " return y_pred"
610 | ]
611 | },
612 | {
613 | "cell_type": "code",
614 | "execution_count": null,
615 | "metadata": {},
616 | "outputs": [],
617 | "source": [
618 | "from sklearn.metrics import r2_score\n",
619 | "\n",
620 | "# construct residual model\n",
621 | "resd_train = y_train - pipe.predict(df_train)\n",
622 | "residual_feats = ResidualFeatures(window=20)\n",
623 | "residual_model = Pipeline([('residual_features', residual_feats), ('regressor', LinearRegression())])\n",
624 | " \n",
625 | "# construct and train full model\n",
626 | "full_model = FullModel(pipe, residual_model, steps=1)\n",
627 | "full_model.fit(df_train, y_train)\n",
628 | "\n",
629 | "# make predictions\n",
630 | "y_pred = full_model.predict(df_hourly)\n",
631 | "resd = energy - y_pred\n",
632 | "ind = resd.index > cutoff\n",
633 | "print(\"Test set R^2: {:g}\".format(r2_score(energy.loc[ind], y_pred.loc[ind])))\n",
634 | "plot_results(df_hourly, y_pred)"
635 | ]
636 | },
637 | {
638 | "cell_type": "markdown",
639 | "metadata": {},
640 | "source": [
641 | "Admittedly, our baseline model is not great at predicting future energy use. However, we can still utilize our baseline model for anomaly detection. Our analysis will focus on the final residuals of our baseline model. If an observation deviates significantly from the baseline, it will be flagged. The plot below illustrates the distribution and autocorrelation for our final residuals."
642 | ]
643 | },
644 | {
645 | "cell_type": "code",
646 | "execution_count": null,
647 | "metadata": {},
648 | "outputs": [],
649 | "source": [
650 | "resd.hist(bins=50, density=True);\n",
651 | "plt.show()\n",
652 | "\n",
653 | "autocorrelation_plot(resd.dropna())\n",
654 | "plt.xlabel(\"Lag (hours)\")\n",
655 | "plt.xlim([0, 100]);"
656 | ]
657 | },
658 | {
659 | "cell_type": "markdown",
660 | "metadata": {},
661 | "source": [
662 | "**Questions**\n",
663 | "* What conclusion can you make from the autocorrelation plot?\n",
664 | "* What do you observe in the distribution of residuals?\n",
665 | "* What is a good basis to use when determining whether the magnitude of a deviation is large?"
666 | ]
667 | },
668 | {
669 | "cell_type": "markdown",
670 | "metadata": {},
671 | "source": [
672 | "## z-Score\n",
673 | "\n",
674 | "Since there is little temporal correlation with residual values, we can assume that the residuals are independently sampled from the same distribution. Given this probabilistic perspective, we can quantify the degree of anomaly to each observation *if* we know the distribution the residuals are being sampled from. If the distribution has one peak, there is a lower probability of observing values far from the peak. The z-score is a relative measure of how far away a value is from the mean, normalized by the standard deviation.\n",
675 | "\n",
676 | "$$\n",
677 | "z = \\frac{x - \\mu}{\\sigma},\n",
678 | "$$\n",
679 | "\n",
680 | "where $\\mu$ and $\\sigma$ are the mean and standard deviation of distribution. The larger the magnitude of the z-score, the lower the probability of observing the value. Exact percentages can only be known if we know the distribution. When the distribution is a normal or Gaussian distribution given by\n",
681 | "\n",
682 | "$$\n",
683 | "p(x) = \\frac{1}{\\sqrt{2\\pi \\sigma}}\\exp\\left(-\\frac{(x-\\mu)^2}{2\\sigma^2} \\right),\n",
684 | "$$\n",
685 | "\n",
686 | "68% of the values will reside within $z = \\pm 1$. 95% and 99.7% of the values will reside in an interval of $z = \\pm 2$ and $z = \\pm 3$, respectively.\n",
687 | "\n",
688 | "Strictly speaking, our distribution of our residuals are not normal but it has a single peak and not heavily skewed. The general idea of the greater the magnitude of the z-score the more anomalous the observation is still valid. Let's calculate the z-score for each residual and display the results. Since our distribution is slightly skewed towards positive values, we will use a different z-score cutoff whether the residual is negative or positive."
689 | ]
690 | },
691 | {
692 | "cell_type": "code",
693 | "execution_count": null,
694 | "metadata": {},
695 | "outputs": [],
696 | "source": [
697 | "z = (resd - resd.mean())/ resd.std()\n",
698 | "z.plot()\n",
699 | "pd.Series(3, index=resd.index).plot(color=\"r\")\n",
700 | "pd.Series(-2, index=resd.index).plot(color=\"r\")\n",
701 | "plt.ylabel(\"z-score\")\n",
702 | "plt.legend([\"residual\", \"z-score cutoff\"]);"
703 | ]
704 | },
705 | {
706 | "cell_type": "code",
707 | "execution_count": null,
708 | "metadata": {},
709 | "outputs": [],
710 | "source": [
711 | "def find_anomalies(z, cutoff_lower=-2, cutoff_upper=2):\n",
712 | " ind_lower = z < cutoff_lower\n",
713 | " ind_upper = z > cutoff_upper\n",
714 | " \n",
715 | " return z[ind_lower | ind_upper]\n",
716 | "\n",
717 | "find_anomalies(z, cutoff_lower=-2, cutoff_upper=3)"
718 | ]
719 | },
720 | {
721 | "cell_type": "markdown",
722 | "metadata": {},
723 | "source": [
724 | "**Questions**\n",
725 | "* How should we decide the appropriate z-score cutoff? What are important ideas and consequences should we consider?\n",
726 | "* If we are concerned at identifying as much of the anomalies as possible, how should the z-score cutoff be set?"
727 | ]
728 | },
729 | {
730 | "cell_type": "markdown",
731 | "metadata": {},
732 | "source": [
733 | "## Rolling z-score\n",
734 | "\n",
735 | "The calculation of the z-score relied on the entire time series for calculating the mean and standard deviation. For anomaly detection with time series, we will usually be streaming observations and the entire series will not be available. Instead, we can calculate the z-score on a window of observations rather than the entire time history. Using a window has the advantage of \n",
736 | "\n",
737 | "1. Not having to hold in memory a large amount of data.\n",
738 | "2. Reflecting the fact that it is better to use recent values to judge whether an observation is anomalous.\n",
739 | "3. Being more adaptive to recent changes in the process.\n",
740 | "\n",
741 | "Let's modify the z-score calculation to only use a window of observations."
742 | ]
743 | },
744 | {
745 | "cell_type": "code",
746 | "execution_count": null,
747 | "metadata": {},
748 | "outputs": [],
749 | "source": [
750 | "def rolling_z_score(x, window=20):\n",
751 | " rolling = x.rolling(window=window)\n",
752 | " mean_roll = rolling.mean().shift(1) # shift to not include current value\n",
753 | " std_roll = rolling.std().shift(1)\n",
754 | " \n",
755 | " return (x - mean_roll) / std_roll\n",
756 | "\n",
757 | "z_roll = rolling_z_score(resd, window=20)\n",
758 | "z_roll.plot()\n",
759 | "pd.Series(3, index=resd.index).plot(color=\"r\")\n",
760 | "pd.Series(-2, index=resd.index).plot(color=\"r\")\n",
761 | "plt.ylabel(\"z-score\")\n",
762 | "plt.legend([\"residual\", \"z-score cutoff\"]);\n",
763 | "\n",
764 | "find_anomalies(z_roll, cutoff_lower=-2, cutoff_upper=3)"
765 | ]
766 | },
767 | {
768 | "cell_type": "markdown",
769 | "metadata": {},
770 | "source": [
771 | "## Exercises"
772 | ]
773 | },
774 | {
775 | "cell_type": "markdown",
776 | "metadata": {},
777 | "source": [
778 | "1. Can you improve on the baseline model for the case study? Note, there are other recorded values in the data set such as the home temperature and humidity. These other time series may help improve the $R^2$ value of the model.\n",
779 | "1. Package an anomaly detector for our time series case study into a class. What should be some hyperparameters to the model? Note, starting with version 0.20, `scikit-learn` has the `OutlierMixin` class that your custom class could inherit form."
780 | ]
781 | },
782 | {
783 | "cell_type": "markdown",
784 | "metadata": {},
785 | "source": [
786 | "*Copyright © 2020 The Data Incubator. All rights reserved.*"
787 | ]
788 | }
789 | ],
790 | "metadata": {
791 | "kernelspec": {
792 | "display_name": "Python 3",
793 | "language": "python",
794 | "name": "python3"
795 | },
796 | "nbclean": true
797 | },
798 | "nbformat": 4,
799 | "nbformat_minor": 0
800 | }
801 |
--------------------------------------------------------------------------------
/datacourse/machine-learning/ML_Clustering.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {
7 | "init_cell": true
8 | },
9 | "outputs": [],
10 | "source": [
11 | "%logstop\n",
12 | "%logstart -rtq ~/.logs/ML_Clustering.py append\n",
13 | "%matplotlib inline\n",
14 | "import matplotlib\n",
15 | "import seaborn as sns\n",
16 | "sns.set()\n",
17 | "matplotlib.rcParams['figure.dpi'] = 144"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": null,
23 | "metadata": {},
24 | "outputs": [],
25 | "source": [
26 | "import matplotlib.pyplot as plt\n",
27 | "import numpy as np"
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "metadata": {},
33 | "source": [
34 | "# Clustering\n",
35 | "\n",
36 | "Clustering is a branch of unsupervised machine learning where the goal is to identify groups or clusters in your data set without the use of labels. Clustering should not be considered the same as classification; we are not trying make predictions on observations from a set of classes. In clustering, you are identifying a set of similar data points and calling the resulting set a cluster.\n",
37 | "\n",
38 | "Let's consider an example of clustering. You may have a data set characterizing your customers like demographic information and personal preferences. A supervised machine learning application would be to determine whether a person will buy a product. However, an _unsupervised_ machine learning application would be to identify several groups or types of customers. With these groups identified, you can analyze the groups and build profiles describing the groups. For example, one group tends to include people from the ages 20 to 25 who like the outdoors. With these profiles, you can pass that information and analysis to the marketing team to create different advertisements to best attract each group.\n",
39 | "\n",
40 | "In this notebook, we will discuss clustering metrics and two common algorithms for clustering. In the code below, we demonstrate the result of determining three clusters in a data set."
41 | ]
42 | },
43 | {
44 | "cell_type": "code",
45 | "execution_count": null,
46 | "metadata": {},
47 | "outputs": [],
48 | "source": [
49 | "from sklearn.datasets import make_blobs\n",
50 | "from sklearn.cluster import KMeans\n",
51 | "\n",
52 | "# generate data\n",
53 | "X, _ = make_blobs(n_samples=300, n_features=2, centers=3, cluster_std=2, random_state=0)\n",
54 | "\n",
55 | "# fit cluster model and assign points to clusters\n",
56 | "kmeans = KMeans(n_clusters=3)\n",
57 | "kmeans.fit(X)\n",
58 | "clusters = kmeans.predict(X)\n",
59 | "\n",
60 | "plt.scatter(*X.T, c=clusters, cmap='viridis');"
61 | ]
62 | },
63 | {
64 | "cell_type": "markdown",
65 | "metadata": {},
66 | "source": [
67 | "In `scikit-learn`, clustering models have the same interface as predictors, albeit they do not make predictions in the same sense as classifiers but rather perform assignments. To train a clustering model object with data set `X`, simply invoke the `fit(X)`. To assign clusters, use `predict(X)`."
68 | ]
69 | },
70 | {
71 | "cell_type": "markdown",
72 | "metadata": {},
73 | "source": [
74 | "## Metrics for clustering\n",
75 | "\n",
76 | "Because clustering is unsupervised learning, we cannot rely on metrics based on labels; there is not a right or wrong answer. However, it is still possible to derive metrics to evaluate the performance of clustering algorithms. More care and analysis will be required when comparing multiple clustering results due to the nature of the derived metrics. Previously in supervised machine learning, we could objectively say one model will perform better because it has a lower test error.\n",
77 | "\n",
78 | "Two common metrics for clustering are\n",
79 | "\n",
80 | "1. **Inertia**: the within cluster sum of square distance\n",
81 | "1. **Silhouette Coefficient**: a measure of how dense and separated are the clusters\n",
82 | "\n",
83 | "Mathematically, inertia is equal to\n",
84 | "\n",
85 | "$$ \\sum_{k} \\sum_{X_j \\in C_k} \\| X_j - \\mu_k \\|^2, $$\n",
86 | "\n",
87 | "where $\\mu_k$ is the centroid of cluster $k$ and $C_k$ is the set of points assigned to cluster $k$. Basically, the inertia is the sum of the distance of each point to the centroid or center of its assigned cluster. A lower inertia means the points assigned to the clusters are closer to the centroid.\n",
88 | "\n",
89 | "The silhouette coefficient is a property assigned to each data point. It is equal to\n",
90 | "\n",
91 | "$$ \\frac{b - a}{\\max(a, b)}, $$\n",
92 | "\n",
93 | "where $a$ is the distance between a point and centroid of its assigned cluster and $b$ is the distance between the point and the centroid of the nearest neighboring cluster (i.e. the closest cluster the point is not assigned to). The silhouette coefficient ranges from -1 to 1. If a point is really close to the centroid of its assigned cluster, then $a \\ll b$ and the silhouette coefficient will be approximately equal to 1. If the reverse is true, $a \\gg b$, then the coefficient will be -1. If the point could have been assigned to either cluster, its coefficient will be 0. Maximizing the silhouette coefficient will prioritize dense and highly separated clusters as dense clusters will have a low $a$ value and having clusters well separated from each other will increase $b$."
94 | ]
95 | },
96 | {
97 | "cell_type": "markdown",
98 | "metadata": {},
99 | "source": [
100 | "## $K$-Means Clustering\n",
101 | "\n",
102 | "The $K$-means algorithms seeks to find $K$ clusters within a data set. The clusters are chosen to reduce the inertia; the objective function is\n",
103 | "\n",
104 | "$$ \\min_{C_k} \\sum_{k} \\sum_{X_j \\in C_k} \\| X_j - \\mu_k \\|^2. $$\n",
105 | "\n",
106 | "The centroid of a cluster $\\mu_k$ is equal to\n",
107 | "\n",
108 | "$$ \\mu_k = \\frac{1}{|C_k|} \\sum_{X_j \\in C_k} X_j, $$\n",
109 | "\n",
110 | "where $|C_k|$ is the number of points in cluster $k$. The equation says that the components of the centroid are equal to _mean_ of each feature/components of all points assigned to the cluster, hence the name of the algorithm. The training algorithm for $K$-means is straight-forward. After seeding the algorithm, choosing the starting locations of each cluster's centroid,\n",
111 | "\n",
112 | "1. Assign each point to a cluster based on which cluster centroid it's the closest to\n",
113 | "1. Calculate the centroid of resulting cluster using the points that have been assigned to the cluster\n",
114 | "1. Repeat the above steps until convergence is met\n",
115 | "\n",
116 | "Let's create an interactive plot that allows users to walk through the iterations involved in the algorithm."
117 | ]
118 | },
119 | {
120 | "cell_type": "code",
121 | "execution_count": null,
122 | "metadata": {},
123 | "outputs": [],
124 | "source": [
125 | "from ipywidgets import interact\n",
126 | "\n",
127 | "def plot_kmeans_steps(X, n_clusters, seeds=np.array([[0, 4], [4, 2], [-2, 4]])):\n",
128 | " def func(step=0):\n",
129 | " iters = step // 2\n",
130 | " \n",
131 | " if iters:\n",
132 | " kmeans = KMeans(n_clusters=n_clusters, max_iter=iters, n_init=1, init=seeds)\n",
133 | " kmeans.fit(X)\n",
134 | " centroids = kmeans.cluster_centers_\n",
135 | " labels = kmeans.labels_\n",
136 | " else:\n",
137 | " centroids = seeds\n",
138 | " labels = '0.5'\n",
139 | " if step % 2:\n",
140 | " kmeans = KMeans(n_clusters=n_clusters, max_iter=iters+1, n_init=1, init=seeds)\n",
141 | " kmeans.fit(X)\n",
142 | " labels = kmeans.labels_\n",
143 | " \n",
144 | " plt.scatter(*X.T, c=labels, cmap='viridis', alpha=0.5)\n",
145 | " plt.scatter(*centroids.T, c=range(n_clusters), cmap='viridis', marker='*', s=150, \n",
146 | " linewidths=1, edgecolors='k')\n",
147 | " plt.title(['Set Centroids', 'Assign Clusters'][step % 2])\n",
148 | " \n",
149 | " return func\n",
150 | "\n",
151 | "interact(plot_kmeans_steps(X, n_clusters=3), step=(0, 10));"
152 | ]
153 | },
154 | {
155 | "cell_type": "markdown",
156 | "metadata": {},
157 | "source": [
158 | "### Implementation details\n",
159 | "\n",
160 | "There are several thing to keep in mind about the $K$-means algorithm. While the algorithm is guaranteed to converge, it will not converge to the _global_ minimum. It is a greedy algorithm; it is relatively quick to run but we are at risk of obtaining a suboptimal solution. Different starting positions will result in different solutions. To counteract suboptimal solutions, `KMeans` by default runs the algorithm ten times and chooses the best result. The number of runs is controlled by `n_init`. The algorithm for the initial centroid locations is controlled by the keyword `init`; the default is `'kmeans++'` which has been shown to work well and results in faster convergence. You can read more about the `kmeans++` algorithm [here](https://en.wikipedia.org/wiki/K-means%2B%2B)."
161 | ]
162 | },
163 | {
164 | "cell_type": "markdown",
165 | "metadata": {},
166 | "source": [
167 | "**Questions**\n",
168 | "* What is the general behavior of the inertia as we increase the number of clusters?\n",
169 | "* Is it important to scale our data set when using $K$-means? If so, what `scikit-learn` tool would we use?"
170 | ]
171 | },
172 | {
173 | "cell_type": "markdown",
174 | "metadata": {},
175 | "source": [
176 | "## Choosing the number of clusters\n",
177 | "\n",
178 | "The number of clusters is a hyperparameter to clustering models; it is set prior to training. How does one choose the optimal number of clusters? We expect as we use more clusters to group our observations the inertia goes down because the points will be closer to the cluster's centroid. What is the inertia when the number of clusters is equal to the number of observations? If every point is its own cluster, the inertia is zero. Choosing the number of clusters that yields the lowest inertia results in a meaningless solution. Instead, we need identify at what point is increasing the number of clusters no longer resulting in an appreciable drop in inertia (the point of \"diminishing returns\"). Identifying this region is accomplished using an \\\"elbow plot\\\", named because the graph looks like a bent arm. Let's construct an elbow plot with the California housing data set."
179 | ]
180 | },
181 | {
182 | "cell_type": "code",
183 | "execution_count": null,
184 | "metadata": {},
185 | "outputs": [],
186 | "source": [
187 | "from sklearn.datasets import fetch_california_housing\n",
188 | "from sklearn.cluster import KMeans\n",
189 | "from sklearn.preprocessing import StandardScaler\n",
190 | "from sklearn.utils import shuffle\n",
191 | "\n",
192 | "# import and shuffle the data\n",
193 | "data = fetch_california_housing()\n",
194 | "X = data['data']\n",
195 | "X_shuffled = shuffle(X, random_state=0)\n",
196 | "\n",
197 | "scaler = StandardScaler() # need to scale data to make kmeans work properly\n",
198 | "Xt = scaler.fit_transform(X_shuffled[:500]) # down sample for speed\n",
199 | "n_clusters = range(1, 200, 2)\n",
200 | "inertia = []\n",
201 | "\n",
202 | "for n in n_clusters:\n",
203 | " kmeans = KMeans(n_clusters=n)\n",
204 | " kmeans.fit(Xt)\n",
205 | " inertia.append(kmeans.inertia_)\n",
206 | "\n",
207 | "plt.plot(n_clusters, inertia/inertia[0])\n",
208 | "plt.hlines(0.1, n_clusters[0], n_clusters[-1], 'r', linestyles='dashed')\n",
209 | "plt.hlines(0.05, n_clusters[0], n_clusters[-1], 'r', linestyles='dashed')\n",
210 | "plt.xlabel('clusters')\n",
211 | "plt.ylabel('relative inertia')\n",
212 | "plt.legend(['inertia', '10% relative inertia', '5% relative inertia']);"
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "metadata": {},
218 | "source": [
219 | "The elbow plot shows us that there is little drop in inertia after about 50 or 100 clusters with most of the drop occurring before 25. As discussed earlier, one cannot look at this plot and definitely say whether using 50 clusters is better than using 100 clusters. Instead, we justify our choice of cluster size given the analysis and understand there is some arbitrariness in the decision. Further, rarely do machine learning problems exist in a bubble; there are constraints usually imposed by business and financial considerations. For example, in the clustering of our consumer data for advertisement purposes, it may only be financially practical to identify no more than 10 groups."
220 | ]
221 | },
222 | {
223 | "cell_type": "markdown",
224 | "metadata": {},
225 | "source": [
226 | "## Elongated clusters\n",
227 | "\n",
228 | "Consider applying $K$-means to a data set where points are clustered in elongated shapes. We can visualize the data and manually identify the clusters. Let's see what happens when we try to use $K$-means with a data set consisting points arranged in elongated shapes."
229 | ]
230 | },
231 | {
232 | "cell_type": "code",
233 | "execution_count": null,
234 | "metadata": {},
235 | "outputs": [],
236 | "source": [
237 | "def elongated(clustering=None):\n",
238 | " X, _ = make_blobs(n_samples=300, n_features=2, cluster_std=0.5, random_state=0)\n",
239 | "\n",
240 | " # elongate the data\n",
241 | " Xt = np.dot(X, [[0.6, -0.64], [-0.4, 0.9]])\n",
242 | "\n",
243 | " if clustering is None:\n",
244 | " plt.scatter(*Xt.T, cmap='viridis')\n",
245 | " else:\n",
246 | " kmeans = KMeans(n_clusters=3)\n",
247 | " kmeans.fit(Xt)\n",
248 | " clusters = kmeans.predict(Xt)\n",
249 | " plt.scatter(*Xt.T, c=clusters, cmap='viridis');\n",
250 | " \n",
251 | "interact(elongated, clustering=[None, 'kmeans']);"
252 | ]
253 | },
254 | {
255 | "cell_type": "markdown",
256 | "metadata": {},
257 | "source": [
258 | "$K$-means did not identify what we would consider are the obvious clusters in the data set. Remember, the clusters are chosen to reduce inertia, which is a measure of how coherent the clusters are, resulting in isotropic clusters."
259 | ]
260 | },
261 | {
262 | "cell_type": "markdown",
263 | "metadata": {},
264 | "source": [
265 | "## Gaussian mixture models\n",
266 | "\n",
267 | "Gaussian mixture models are an alternative to the $K$-means clustering. As seen previously, it fails to define clusters of elongated shapes. Gaussian mixture models offer additional capabilities compared to $K$-means. Cluster definitions are probabilistic and it can model elongated shapes. Gaussian mixture models define a probability density function over the feature space defined by adding or \"mixing\" several multivariate Gaussians. \n",
268 | "\n",
269 | "To understand Gaussian mixture models, let's first define and plot a multivariate Gaussian. A multivariate Gaussian is simply a Gaussian distribution for more than one dimension. As with the one dimensional Gaussian, it has two parameters. The mean $\\mu_i$ is now a vector since we have multiple dimensions and the variance has been replaced with the **covariance** matrix. Not only does the covariance matrix controls how spread out the distribution is but also how correlated the variables are. If the variables are highly correlated, then the distribution will looked stretched. The covariance matrix contains all possible pair-wise covariance values for all variables. For a two dimensional case the covariance matrix has three different parameters, the variance of each variable, $\\sigma^2_{x}$ and $\\sigma^2_{y}$ and the covariance between the two variables $\\mathrm{cov}(x, y)$.\n",
270 | "\n",
271 | "$$ \\Sigma = \\left[ \\begin{array}{cc} \n",
272 | "\\sigma^2_{x} & \\mathrm{cov}(x, y) \\\\\n",
273 | "\\mathrm{cov}(x, y) & \\sigma^2_{y} \\\\\n",
274 | "\\end{array} \\right]$$\n",
275 | "\n",
276 | "Let's build an interactive plot where we can see the effect of modifying the covariance matrix for a bivariate Gaussian. We can adjust all three parameters involved in the covariance matrix. Note, the correlation coefficient $\\rho$ is just a normalization of the covariance. In this case, it is equal to $\\rho = \\mathrm{cov}(x, y)/ (\\sigma_{x} \\sigma_{y})$."
277 | ]
278 | },
279 | {
280 | "cell_type": "code",
281 | "execution_count": null,
282 | "metadata": {},
283 | "outputs": [],
284 | "source": [
285 | "from ipywidgets import FloatSlider\n",
286 | "import scipy as sp\n",
287 | "\n",
288 | "def multivariate_gaussian(corr=0, sigma_x=1, sigma_y=1):\n",
289 | " cov = sigma_x*sigma_y*np.array([[sigma_x/sigma_y, corr], [corr, sigma_x/sigma_y]])\n",
290 | " dist = sp.stats.multivariate_normal(mean=[0, 0], cov=cov)\n",
291 | " X = dist.rvs(1000, random_state=0)\n",
292 | " \n",
293 | " n = 100\n",
294 | " xlims = [-4, 4]\n",
295 | " ylims = [-4, 4]\n",
296 | " X1, X2 = np.meshgrid(np.linspace(*xlims, n), np.linspace(*ylims, n))\n",
297 | " proba = dist.pdf(np.hstack((X1.reshape(-1, 1), X2.reshape(-1, 1))))\n",
298 | " \n",
299 | " plt.scatter(*X.T, alpha=0.25)\n",
300 | " plt.contour(X1, X2, proba.reshape(100, 100))\n",
301 | " plt.xlim(xlims)\n",
302 | " plt.ylim(ylims)\n",
303 | " plt.xlabel('$x$')\n",
304 | " plt.ylabel('$y$')\n",
305 | "\n",
306 | "corr_slider = FloatSlider(min=-0.99, max=0.99, value=0., step=0.01, description='$\\\\rho$')\n",
307 | "sigma_x_slider = FloatSlider(min=0.5, max=1.5, value=1., step=0.01, description='$\\sigma_x$')\n",
308 | "sigma_y_slider = FloatSlider(min=0.5, max=1.5, value=1., step=0.01, description='$\\sigma_y$')\n",
309 | "interact(multivariate_gaussian, corr=corr_slider, sigma_x=sigma_x_slider, sigma_y=sigma_y_slider);"
310 | ]
311 | },
312 | {
313 | "cell_type": "markdown",
314 | "metadata": {},
315 | "source": [
316 | "Notice how the covariance matrix controls the elongation of the distribution. It is this \"stretching\" capability that enables Gaussian mixture models to create anisotropic clusters. The probability across our feature space is defined as\n",
317 | "\n",
318 | "$$ p(x_j) = \\sum_k \\phi_k \\mathcal{N}(x_j; \\mu_k, \\Sigma_k), $$\n",
319 | "\n",
320 | "where $\\mathcal{N}(x_j; \\mu_k, \\Sigma_k)$ is a multivariate Gaussian parameterized by $\\mu_k$ and $\\Sigma_k$. When we fit a mixture model to a training set, we are trying to determine $\\phi_k$, $\\mu_k$, and $\\Sigma_k$ that will maximize the likelihood function\n",
321 | "\n",
322 | "$$ L(\\phi_k, \\mu_k, \\Sigma_k) = \\prod_j p(x_j).$$\n",
323 | "\n",
324 | "The most common algorithm used to fit Gaussian mixture models is the expectation-maximization algorithm, a two step iterative scheme. More about the algorithm can be read [here]( https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm). As with complicated models, the additional complexity results in slower training times and Gaussian mixture models will be difficult to train with large data sets."
325 | ]
326 | },
327 | {
328 | "cell_type": "code",
329 | "execution_count": null,
330 | "metadata": {},
331 | "outputs": [],
332 | "source": [
333 | "from sklearn.mixture import GaussianMixture\n",
334 | "\n",
335 | "X, _ = make_blobs(n_samples=300, n_features=2, cluster_std=0.5, random_state=0)\n",
336 | "Xt = np.dot(X, [[0.6, -0.64], [-0.4, 0.9]])\n",
337 | "\n",
338 | "gm = GaussianMixture(n_components=3, covariance_type='full')\n",
339 | "gm.fit(Xt)\n",
340 | "clusters = gm.predict(Xt)\n",
341 | "\n",
342 | "xlims = [-3, 2]\n",
343 | "ylims = [-2, 5]\n",
344 | "X1, X2 = np.meshgrid(np.linspace(*xlims, 100), np.linspace(*ylims, 100))\n",
345 | "proba = gm.score_samples(np.hstack((X1.reshape(-1, 1), X2.reshape(-1, 1))))\n",
346 | "\n",
347 | "plt.contour(X1, X2, np.log(-proba.reshape(100, 100)), 10)\n",
348 | "plt.colorbar()\n",
349 | "plt.xlim(xlims)\n",
350 | "plt.ylim(ylims)\n",
351 | "plt.xlabel('$x_1$')\n",
352 | "plt.ylabel('$x_2$')\n",
353 | "plt.scatter(*Xt.T, c=clusters, cmap='viridis')\n",
354 | "plt.title('Negative Log Likelihood');"
355 | ]
356 | },
357 | {
358 | "cell_type": "markdown",
359 | "metadata": {},
360 | "source": [
361 | "## Exercises"
362 | ]
363 | },
364 | {
365 | "cell_type": "markdown",
366 | "metadata": {},
367 | "source": [
368 | "1. The silhouette coefficient can also be used for choosing the number of clusters. `scikit-learn` has a function to calculate the silhouette coefficient: `from sklearn.metrics import silhouette_score`. Compare the silhouette coefficient as a function of number of clusters with the elbow plot displayed in the notebook.\n",
369 | "\n",
370 | "1. For the California housing data, create an elbow plot using Gaussian mixture models. What would be good choice of the number of clusters. How does the training time compare with $K$-means?"
371 | ]
372 | },
373 | {
374 | "cell_type": "markdown",
375 | "metadata": {},
376 | "source": [
377 | "*Copyright © 2020 The Data Incubator. All rights reserved.*"
378 | ]
379 | }
380 | ],
381 | "metadata": {
382 | "kernelspec": {
383 | "display_name": "Python 3",
384 | "language": "python",
385 | "name": "python3"
386 | },
387 | "nbclean": true
388 | },
389 | "nbformat": 4,
390 | "nbformat_minor": 0
391 | }
392 |
--------------------------------------------------------------------------------
/datacourse/machine-learning/ML_Dimension_Reduction.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {
7 | "init_cell": true
8 | },
9 | "outputs": [],
10 | "source": [
11 | "%logstop\n",
12 | "%logstart -rtq ~/.logs/ML_Dimension_Reduction.py append\n",
13 | "%matplotlib inline\n",
14 | "import matplotlib\n",
15 | "import seaborn as sns\n",
16 | "sns.set()\n",
17 | "matplotlib.rcParams['figure.dpi'] = 144"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": null,
23 | "metadata": {},
24 | "outputs": [],
25 | "source": [
26 | "import matplotlib.pyplot as plt\n",
27 | "import numpy as np"
28 | ]
29 | },
30 | {
31 | "cell_type": "markdown",
32 | "metadata": {},
33 | "source": [
34 | "# Dimension Reduction\n",
35 | "\n",
36 | "Dimension reduction is an unsupervised learning technique; no labels are used for the process. The goal of dimension reduction is to take observations characterized by a set of features and reduce the number of features. Hence, we are _reducing_ the dimension of our data points. For example, instead of using 100 features to characterize each observation, dimension reduction techniques allow us to represent the data using a truncated set, e.g., using 10 features instead of the original 100. While reducing the dimension will result in some information loss, the algorithms we will discuss aim to keep this loss at a minimum.\n",
37 | "\n",
38 | "Several applications of dimension reduction are:\n",
39 | "\n",
40 | "1. Reducing file sizes\n",
41 | "1. Visualizing high dimensional data sets\n",
42 | "1. Faster training and predicting times for supervised machine learning models\n",
43 | "1. Generating a better, truncated, set of new features to represent our data\n",
44 | "\n",
45 | "The notebook will discuss three commonly used dimension reduction techniques and how they are implemented in `scikit-learn`."
46 | ]
47 | },
48 | {
49 | "cell_type": "markdown",
50 | "metadata": {},
51 | "source": [
52 | "## Mathematics of dimension reduction\n",
53 | "\n",
54 | "Dimension reduction techniques work by first creating a new set of dimensions/axes and _projecting_ the data to the new space. The process of projecting is a matrix multiplication,\n",
55 | "\n",
56 | "$$ X' = XP,$$\n",
57 | "\n",
58 | "where $X$ is the matrix of our original data, $n$ observations and $p$ columns/features, $X'$ is the matrix of our data in the new space, and $P$ is the matrix that projects our data onto the new feature space. $P$ has $p$ rows and $p$ columns where each column is a vector that represents a new dimension. The vectors are ordered from most important to least important with regards to capturing the variation in the data. If we only include the first $m$ columns of $P$, then the matrix multiplication will project our data onto a lower dimensional space. The matrix multiplication of an $n$ by $p$ matrix with a $p$ and $m$ matrix will result in a $n$ by $m$ matrix; our transformed data set has less features,\n",
59 | "\n",
60 | "$$ X' = X \\tilde{P}, $$\n",
61 | "\n",
62 | "where $\\tilde{P}$ is the truncated form of $P$ that has $m$ columns where $m < p$. The dimension reduction algorithms work by finding $P$ given an objective function. The objective function is typically constructing the projection matrix $P$ such that using the truncated form $\\tilde{P}$ can still retain the majority of the information in our data set."
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "metadata": {},
68 | "source": [
69 | "## Principal component analysis\n",
70 | "\n",
71 | "Principal component analysis (PCA) is a dimension reduction technique that takes a data set characterized by a set of possibly correlated features and generates a new set of features that are uncorrelated. It is used as a dimension reduction technique because the new set of uncorrelated features are chosen to be efficient in terms of capturing the variance in the data set.\n",
72 | "\n",
73 | "Let's examine a case where we have a data set of only two dimensions. In practice, PCA is rarely used when the dimension of the data set is already low. However, it is easier to illustrate the method when we have two or three dimensions."
74 | ]
75 | },
76 | {
77 | "cell_type": "code",
78 | "execution_count": null,
79 | "metadata": {},
80 | "outputs": [],
81 | "source": [
82 | "np.random.seed(0)\n",
83 | "x1 = np.linspace(0, 1, 500)\n",
84 | "x2 = 2*x1 + 1 + 0.2*np.random.randn(500)\n",
85 | "X = np.vstack((x1, x2)).T\n",
86 | "\n",
87 | "plt.scatter(*X.T, alpha=0.25)\n",
88 | "plt.plot(x1, 2*x1 + 1, '--k', linewidth=2)\n",
89 | "plt.xlabel('$x_1$')\n",
90 | "plt.ylabel('$x_2$');"
91 | ]
92 | },
93 | {
94 | "cell_type": "markdown",
95 | "metadata": {},
96 | "source": [
97 | "The data plotted is characterized by two dimensions, however, most of the variation does not occur in either of the two dimensions. Most of the points \"follow\" along the direction plotted in the dashed line. The variables $x_1$ and $x_2$ are highly correlated; as $x_1$ increases, in general, so does $x_2$ and vice versa.\n",
98 | "\n",
99 | "Instead of using the original two features, $x_1$ and $x_2$, perhaps we can use a different set of features, $\\xi_1$ and $\\xi_2$. The first chosen feature $\\xi_1$ should be aligned in the direction of greatest variation while the second will be _orthogonal_ to the first. The new axes/dimensions are referred to as _principal components_. Let's visualize the data set but using the principal components $\\xi_1$ and $\\xi_2$ rather than the original features."
100 | ]
101 | },
102 | {
103 | "cell_type": "code",
104 | "execution_count": null,
105 | "metadata": {},
106 | "outputs": [],
107 | "source": [
108 | "from sklearn.decomposition import PCA\n",
109 | "\n",
110 | "pca = PCA(n_components=2)\n",
111 | "Xt = pca.fit_transform(X)\n",
112 | "\n",
113 | "xi_1_max, xi_2_max = Xt.max(axis=0)\n",
114 | "xi_1_min, xi_2_min = Xt.min(axis=0)\n",
115 | "\n",
116 | "plt.hlines(0, xi_1_min, xi_1_max, linestyles='--')\n",
117 | "plt.vlines(0, xi_2_min, xi_2_max, linestyles='--')\n",
118 | "\n",
119 | "plt.scatter(*Xt.T, alpha=0.25)\n",
120 | "plt.xlim([-1.75, 1.75])\n",
121 | "plt.ylim([-1.75, 1.75])\n",
122 | "plt.xlabel('$\\\\xi _1$')\n",
123 | "plt.ylabel('$\\\\xi _2$');"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "metadata": {},
129 | "source": [
130 | "In the figure, we can clearly observe that $\\xi_1$ is the dimension with the largest variation. In the PCA algorithm, $\\xi_1$ is chosen to capture as much of the variation as possible, with $\\xi_2$ picking up the rest of remaining variation. Now, if we want to use one dimension to describe our data, we would keep $\\xi_1$ and drop $\\xi_2$, ensuring we keep as much of the information in our data set using just one dimension. Further, notice how the new dimensions are not correlated. As we move from lower to higher values of $\\xi_1$, $\\xi_2$ does not predictability increase or decrease.\n",
131 | "\n",
132 | "In the visualization below, you can represent the data points in the space of either one (reduced) or two principal components or project back onto the original space _after_ reducing the dimension."
133 | ]
134 | },
135 | {
136 | "cell_type": "code",
137 | "execution_count": null,
138 | "metadata": {},
139 | "outputs": [],
140 | "source": [
141 | "from ipywidgets import interact, fixed\n",
142 | "\n",
143 | "np.random.seed(0)\n",
144 | "ind = np.random.choice(Xt.shape[0], 50)\n",
145 | "\n",
146 | "def reduce_dim(X, Xt, step='one PC'):\n",
147 | " if step == 'original space': \n",
148 | " pca = PCA(n_components=1)\n",
149 | " X_t = pca.fit_transform(X)\n",
150 | " plt.scatter(*pca.inverse_transform(X_t[ind, :]).T)\n",
151 | " plt.scatter(*X[ind, :].T, c='b', alpha=0.1)\n",
152 | "\n",
153 | " plt.xlabel('$x_1$')\n",
154 | " plt.ylabel('$x_2$');\n",
155 | " \n",
156 | " return \n",
157 | " \n",
158 | " elif step == 'two PC':\n",
159 | " plt.scatter(*Xt[ind, :].T)\n",
160 | "\n",
161 | " for x in Xt[ind, :]:\n",
162 | " plt.vlines(x[0], 0, x[1], linestyles='--') \n",
163 | " else:\n",
164 | " plt.scatter(Xt[ind, 0], np.zeros(50))\n",
165 | " plt.scatter(*Xt[ind, :].T, alpha=0.1, c='b')\n",
166 | "\n",
167 | " plt.xlim([-1.75, 1.75])\n",
168 | " plt.ylim([-0.5, 0.5])\n",
169 | " plt.xlabel('$\\\\xi _1$')\n",
170 | " plt.ylabel('$\\\\xi _2$')\n",
171 | " \n",
172 | "interact(reduce_dim, X=fixed(X), Xt=fixed(Xt), step=['two PC', 'one PC', 'original space']);"
173 | ]
174 | },
175 | {
176 | "cell_type": "markdown",
177 | "metadata": {},
178 | "source": [
179 | "## PCA in `scikit-learn`\n",
180 | "\n",
181 | "In `scikit-learn`, dimension reduction algorithms are transformers. The choice of having these algorithms as transformers makes sense since they apply a transformation on the data set. Let's illustrate the syntax for the PCA algorithm in `scikit-learn`. Note, other dimension reduction techniques in `scikit-learn` will have the same interface. For most of these algorithms, the data needs to be centered and scaled to work properly. `PCA` automatically centers the data but **does not** scale it. `StandardScaler` is often used for preprocessing the data prior to applying PCA."
182 | ]
183 | },
184 | {
185 | "cell_type": "code",
186 | "execution_count": null,
187 | "metadata": {},
188 | "outputs": [],
189 | "source": [
190 | "from sklearn.datasets import fetch_california_housing\n",
191 | "from sklearn.decomposition import PCA\n",
192 | "from sklearn.preprocessing import StandardScaler\n",
193 | "\n",
194 | "data = fetch_california_housing()\n",
195 | "X = data['data']\n",
196 | "\n",
197 | "scaler = StandardScaler()\n",
198 | "X_scaled = scaler.fit_transform(X)\n",
199 | "pca = PCA(n_components=4)\n",
200 | "Xt = pca.fit_transform(X_scaled)\n",
201 | "\n",
202 | "print(\"number of dimension before reduction: {}\".format(X_scaled.shape[-1]))\n",
203 | "print(\"number of dimension after reduction: {}\".format(Xt.shape[-1]))"
204 | ]
205 | },
206 | {
207 | "cell_type": "markdown",
208 | "metadata": {},
209 | "source": [
210 | "When the `fit` method is called, the transformer learns the matrix $\\tilde{P}$ to use for truncating our data set given the number of features/components, `n_components`, we want to have for our transformed data set. The matrix $\\tilde{P}^T$, where $T$ signifies the transpose, is stored in the attribute `components_` of the PCA transformer object.\n",
211 | "\n",
212 | "In the example above, we have gone from 8 to 4 dimensions. However, we don't know how much of the original information we have retained. With a trained PCA object, the explained variance of each new components is stored in the `explained_variance_` and `explained_variance_ratio_` attribute, the latter being normalized by the total variance."
213 | ]
214 | },
215 | {
216 | "cell_type": "code",
217 | "execution_count": null,
218 | "metadata": {},
219 | "outputs": [],
220 | "source": [
221 | "print(\"explained variance ratio: {}\".format(pca.explained_variance_ratio_))\n",
222 | "print(\"cumulative explained variance ratio: {}\".format(np.cumsum(pca.explained_variance_ratio_)[-1]))"
223 | ]
224 | },
225 | {
226 | "cell_type": "markdown",
227 | "metadata": {},
228 | "source": [
229 | "Thus, with just 4 components/features, we are able to capture about 77% of the variance of the original full order data set. We could also calculate the total explained variance by using the `inverse_transform` method. After transforming our data to obtain the reduced form, we can apply the inversion to obtain the approximation of our data in the original feature space. Then, we can calculate the resulting variance."
230 | ]
231 | },
232 | {
233 | "cell_type": "code",
234 | "execution_count": null,
235 | "metadata": {},
236 | "outputs": [],
237 | "source": [
238 | "print(\"retained variance: {}\".format(pca.inverse_transform(Xt).var()))"
239 | ]
240 | },
241 | {
242 | "cell_type": "markdown",
243 | "metadata": {},
244 | "source": [
245 | "## Implementation details of PCA\n",
246 | "\n",
247 | "We have not discussed exactly how PCA obtains the new features. The matrix $\\tilde{P}$ is chosen such that \n",
248 | "\n",
249 | "$$ \\| X_c - X'\\tilde{P}^T \\|_2 $$\n",
250 | "\n",
251 | "is minimized. The subscript $c$ refers to the centered data set. The product $X'\\tilde{P}^T$ is the reconstruction of our data onto the original feature space. There are several algorithms to solve for the principal components but a popular one involves applying singular value decomposition. Singular value decomposition (SVD) is an algorithm to decompose a matrix into a product of three matrices,\n",
252 | "\n",
253 | "$$ X_c = U \\Sigma P^T. $$\n",
254 | "\n",
255 | "You can envision that the matrix $X_c$ represents a transformation that can be broken into three steps: an initial rotation $P^T$, a scaling $\\Sigma$, and a final rotation $U$. By applying SVD on $X_c$, the matrix $P$ is solved. The matrix $\\Sigma$ is a diagonal matrix, a matrix with non-zero values along the diagonal, \n",
256 | "\n",
257 | "$$ \\Sigma = \\left[ \\begin{array}{ccc} \n",
258 | "\\sigma_1 & \\\\\n",
259 | "& \\sigma_2 & \\\\\n",
260 | "&& \\ddots & \\\\\n",
261 | "&&&\n",
262 | "\\end{array} \\right]$$\n",
263 | "\n",
264 | "The diagonal values are ordered such that $|\\sigma_1| \\ge |\\sigma_2| \\ge \\cdots |\\sigma_{p-1}| \\ge |\\sigma_p|$. The larger the absolute value of $\\sigma$, the greater amount of variation exists in that direction/component. Thus, to generate $\\tilde{P}$ to truncate the data set, the first $m$ components/columns of $P$ are kept."
265 | ]
266 | },
267 | {
268 | "cell_type": "markdown",
269 | "metadata": {},
270 | "source": [
271 | "## Choosing the number of components\n",
272 | "\n",
273 | "How does one chose the best number of components to use? The answer is not clear cut; using more components will increase the explained variance but using too many will defeat the purpose of reducing the number of dimensions. The best way to determine a good number of components to use is to construct a plot of the cumulative explained variance versus the number of components. We need to identify at what point is increasing the number of components no longer has an appreciable gain in explained variance, the point of diminishing returns. Identifying this region is accomplished using an \"elbow plot\", named because of the resemblance of an arm with a bent elbow. Let's create the elbow plot for the California data and see how many components we need to keep to explain at least 90% of the variance."
274 | ]
275 | },
276 | {
277 | "cell_type": "code",
278 | "execution_count": null,
279 | "metadata": {},
280 | "outputs": [],
281 | "source": [
282 | "X_scaled = scaler.fit_transform(X)\n",
283 | "p = X_scaled.shape[-1]\n",
284 | "pca = PCA(n_components=p)\n",
285 | "pca.fit(X_scaled)\n",
286 | "cumulative_explained_var = np.cumsum(pca.explained_variance_ratio_)\n",
287 | "\n",
288 | "plt.plot(range(1, p + 1), cumulative_explained_var)\n",
289 | "plt.hlines(0.9, 1, p+1, linestyles='--')\n",
290 | "plt.hlines(0.99, 1, p+1, linestyles='--')\n",
291 | "plt.xlabel('number of components')\n",
292 | "plt.ylabel('cumulative explained variance');"
293 | ]
294 | },
295 | {
296 | "cell_type": "markdown",
297 | "metadata": {},
298 | "source": [
299 | "It appears we only need to use 5 and 6 components if we want to retain 90% and 99% of the variance, respectively. Note, we usually see more dramatic performance when we have more features. With more features, we are more likely to have a lot of redundant information and correlated features."
300 | ]
301 | },
302 | {
303 | "cell_type": "markdown",
304 | "metadata": {},
305 | "source": [
306 | "## Truncated Singular Value Decomposition\n",
307 | "\n",
308 | "To apply PCA, the data set needs to be centered, i.e., the features needs to have zero mean. Centering the data becomes a problem when we are representing our data using a sparse matrix. To center the data, you need to subtract each entry in the matrix by a value; all zero entries are now non-zero. If we have a sparse matrix, an alternative is to use the `TruncatedSVD` class. The `TruncatedSVD` transformer objects work the same as PCA but it does not center the data prior to finding the principal components."
309 | ]
310 | },
311 | {
312 | "cell_type": "markdown",
313 | "metadata": {},
314 | "source": [
315 | "## Non-negative matrix factorization\n",
316 | "\n",
317 | "In certain applications, our data only has non-negative values. For example, in natural language processing, the bag-of-words model yields a matrix of only non-negative values. In these applications, it is important that any dimension reduction scheme preserves the non-negative nature of any resulting matrices, keeping explicability in our analysis. A variation of PCA but with the added constraint that the derived matrices are non-negative is called non-negative matrix factorization (NMF).\n",
318 | "\n",
319 | "NMF is often used in the field of topic modeling; what are the major topics/ideas in a corpus. We will apply NMF to the newsgroup data set, http://qwone.com/~jason/20Newsgroups/. The first step is to transform our text data into a structured form. We will use `TfidfVectorizer` transformer that creates a data set where our features our words and the entries is a weighted frequency of a particular word. We will formally discuss the field of natural language processing in a separate notebook. When applying NMF, the resulting new dimensions represent a collection of words, our old features, which we can refer to as a topic. For each derived new component, we can display the top words that most contribute to that new dimension. With those top words identified, we can look to see what topic or concept each new feature represents."
320 | ]
321 | },
322 | {
323 | "cell_type": "code",
324 | "execution_count": null,
325 | "metadata": {},
326 | "outputs": [],
327 | "source": [
328 | "from sklearn.datasets import fetch_20newsgroups\n",
329 | "from sklearn.decomposition import NMF\n",
330 | "from sklearn.feature_extraction.text import TfidfVectorizer\n",
331 | "from sklearn.pipeline import Pipeline\n",
332 | "\n",
333 | "data = fetch_20newsgroups(shuffle=True, remove=('headers', 'footers', 'quotes'))\n",
334 | "X = data['data']\n",
335 | "\n",
336 | "n_topics = 10\n",
337 | "n_top_words = 20\n",
338 | "\n",
339 | "tfidf = TfidfVectorizer(stop_words='english')\n",
340 | "nmf = NMF(n_components=n_topics, random_state=0)\n",
341 | "pipe = Pipeline([('vectorizer', tfidf), ('dim-red', nmf)])\n",
342 | "pipe.fit(X)\n",
343 | "\n",
344 | "feature_names = tfidf.get_feature_names()\n",
345 | "\n",
346 | "for i, topic in enumerate(nmf.components_):\n",
347 | " print(\"Topic: {}\".format(i))\n",
348 | " indices = topic.argsort()[-n_top_words-1:-1]\n",
349 | " top_words = [feature_names[ind] for ind in indices]\n",
350 | " print(\" \".join(top_words), \"\\n\")"
351 | ]
352 | },
353 | {
354 | "cell_type": "markdown",
355 | "metadata": {},
356 | "source": [
357 | "From the analysis, we can see that topic 1 represents \"computers\" while topic 2 represents \"Christianity\"."
358 | ]
359 | },
360 | {
361 | "cell_type": "markdown",
362 | "metadata": {},
363 | "source": [
364 | "## Using PCA with a supervised model\n",
365 | "\n",
366 | "A common usage of PCA is to truncate the number of dimensions so that the training and predicting times of a supervised machine learning models will be significantly faster. For example, for decision trees, the training and time complexity with respect to the number of features is $O(p)$. Thus, reducing our features by half will reduce our training by half as well. Let's see the effect of using PCA with conjunction with decision trees for the California housing data."
367 | ]
368 | },
369 | {
370 | "cell_type": "code",
371 | "execution_count": null,
372 | "metadata": {},
373 | "outputs": [],
374 | "source": [
375 | "from shutil import rmtree\n",
376 | "from tempfile import mkdtemp\n",
377 | "import time\n",
378 | "\n",
379 | "from sklearn.datasets import make_classification\n",
380 | "from sklearn.model_selection import GridSearchCV, train_test_split\n",
381 | "from sklearn.tree import DecisionTreeClassifier\n",
382 | "\n",
383 | "X, y = make_classification(n_samples=10000, n_features=100, n_informative=10, random_state=0)\n",
384 | "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n",
385 | "\n",
386 | "scaler = StandardScaler()\n",
387 | "pca = PCA(n_components=10)\n",
388 | "tree = DecisionTreeClassifier()\n",
389 | "\n",
390 | "cache = mkdtemp()\n",
391 | "pipe = Pipeline([('scaler', scaler), ('dim-red', pca), ('clf', tree)], memory=cache)\n",
392 | "param_grid = {'clf__max_depth': range(2, 20)}\n",
393 | "grid_search = GridSearchCV(pipe, param_grid, cv=3, n_jobs=2)\n",
394 | "\n",
395 | "t_0 = time.time()\n",
396 | "grid_search.fit(X_train, y_train)\n",
397 | "t_elapsed = time.time() - t_0\n",
398 | "\n",
399 | "print(\"training time: {:g} seconds\".format(t_elapsed))\n",
400 | "print(\"test accuracy: {}\".format(grid_search.score(X_test, y_test)))\n",
401 | "rmtree(cache)"
402 | ]
403 | },
404 | {
405 | "cell_type": "code",
406 | "execution_count": null,
407 | "metadata": {},
408 | "outputs": [],
409 | "source": [
410 | "pipe = Pipeline([('scaler', scaler), ('clf', tree)], memory=cache)\n",
411 | "param_grid = {'clf__max_depth': range(2, 20)}\n",
412 | "grid_search = GridSearchCV(pipe, param_grid, cv=3, n_jobs=2)\n",
413 | "\n",
414 | "t_0 = time.time()\n",
415 | "grid_search.fit(X_train, y_train)\n",
416 | "t_elapsed = time.time() - t_0\n",
417 | "\n",
418 | "print(\"training time {:g} seconds\".format(t_elapsed))\n",
419 | "print(\"test accuracy {}\".format(grid_search.score(X_test, y_test)))\n",
420 | "rmtree(cache)"
421 | ]
422 | },
423 | {
424 | "cell_type": "markdown",
425 | "metadata": {},
426 | "source": [
427 | "By transforming the data set to have 10 rather than 100 dimensions, the training time is reduced by a third. However, the cost of faster training is accuracy."
428 | ]
429 | },
430 | {
431 | "cell_type": "markdown",
432 | "metadata": {},
433 | "source": [
434 | "## Dimension reduction for visualization\n",
435 | "\n",
436 | "Another use for dimension reduction is for visualization of high dimension data set. It is difficult to visualize more than two or three dimension. One approach is to choose two or three variable when plotting. However, this approach will only visualize the relationship of the data for the given chosen variables. While we cannot visual the entire relationship for all the variables in our data set, we can generate two or three new features that will capture as much of the variation as possible, more than just using two of three variables in the original set. Let's visualize the iris data set which has four components by using two generated features."
437 | ]
438 | },
439 | {
440 | "cell_type": "code",
441 | "execution_count": null,
442 | "metadata": {},
443 | "outputs": [],
444 | "source": [
445 | "from sklearn.datasets import load_iris\n",
446 | "\n",
447 | "data = load_iris()\n",
448 | "X = data['data']\n",
449 | "y = data['target']\n",
450 | "\n",
451 | "pca = PCA(n_components=2)\n",
452 | "pipe = Pipeline([('scaler', StandardScaler()), ('dim-red', pca)])\n",
453 | "Xt = pipe.fit_transform(X)\n",
454 | "\n",
455 | "plt.scatter(*Xt.T, c=y, cmap='viridis')\n",
456 | "plt.xlabel('$\\\\xi_1$')\n",
457 | "plt.ylabel('$\\\\xi_2$');"
458 | ]
459 | },
460 | {
461 | "cell_type": "code",
462 | "execution_count": null,
463 | "metadata": {},
464 | "outputs": [],
465 | "source": [
466 | "explained_var = np.cumsum(pca.explained_variance_ratio_)\n",
467 | "print('explained variance with two dimensions: {}'.format(explained_var[-1]))"
468 | ]
469 | },
470 | {
471 | "cell_type": "markdown",
472 | "metadata": {},
473 | "source": [
474 | "## Exercises\n",
475 | "\n",
476 | "For the following exercises, use the Olivetti face data set, a set of 400 images of faces from 10 individuals. Each image has 4096 pixels, representing our features. The data set can be retrieved by using the `fetch_olivetti_faces` function in the `sklearn.datasets` module.\n",
477 | "\n",
478 | "1. Apply NMF to generate new features and visualize them. Using `matplotlib`, images can be visualized using the `plt.imshow` function.\n",
479 | "1. Train a supervised machine learning model to classify the images. Repeat but use a dimension reduction technique. Compare both the test score and the time required to train the model.\n",
480 | "1. In the demonstration of using PCA in conjunction of supervised machine learning, we did not _simultaneously_ tune the decision tree regressor or the number of components. Tune both of these estimators."
481 | ]
482 | },
483 | {
484 | "cell_type": "markdown",
485 | "metadata": {},
486 | "source": [
487 | "*Copyright © 2020 The Data Incubator. All rights reserved.*"
488 | ]
489 | }
490 | ],
491 | "metadata": {
492 | "kernelspec": {
493 | "display_name": "Python 3",
494 | "language": "python",
495 | "name": "python3"
496 | },
497 | "nbclean": true
498 | },
499 | "nbformat": 4,
500 | "nbformat_minor": 0
501 | }
502 |
--------------------------------------------------------------------------------
/datacourse/machine-learning/ML_Metrics.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 25,
6 | "metadata": {
7 | "init_cell": true
8 | },
9 | "outputs": [],
10 | "source": [
11 | "%logstop\n",
12 | "%logstart -rtq ~/.logs/ML_Metrics.py append\n",
13 | "%matplotlib inline\n",
14 | "import matplotlib\n",
15 | "import seaborn as sns\n",
16 | "sns.set()\n",
17 | "matplotlib.rcParams['figure.dpi'] = 144"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": 13,
23 | "metadata": {},
24 | "outputs": [],
25 | "source": [
26 | "import matplotlib.pyplot as plt\n",
27 | "import pandas as pd\n",
28 | "import sklearn.datasets"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {},
34 | "source": [
35 | "# Metrics for supervised machine learning\n",
36 | "\n",
37 | "The general problem supervised machine learning seeks to solve is to map a measurement of several variables to a target value or class. For example, we might use supervised machine learning to transcribe spoken language to text, predict home values based on neighborhood amenities, or detect fraudulent transactions. In order to assess whether our model is succeeding, we need to formally define what success is for the given task. In this notebook, we will explore several common **metrics** for model performance."
38 | ]
39 | },
40 | {
41 | "cell_type": "markdown",
42 | "metadata": {},
43 | "source": [
44 | "## Mathematics of supervised learning\n",
45 | "\n",
46 | "For most machine-learning problems, our model receives a vector of **features**, $X$, and maps it to some predicted label, $y$. In order to train our model, we will need many **observations** (i.e. measurements) and their associated labels. We can assemble these observations into a matrix.\n",
47 | "\n",
48 | "$$ f(X_{ij}) \\approx y_i $$\n",
49 | "\n",
50 | "We'll use the California housing data set as an example. The California housing data set has measurements of average house age, average number of rooms, location, and other qualities for various census blocks of California."
51 | ]
52 | },
53 | {
54 | "cell_type": "code",
55 | "execution_count": 14,
56 | "metadata": {},
57 | "outputs": [
58 | {
59 | "name": "stdout",
60 | "output_type": "stream",
61 | "text": [
62 | "--2021-01-27 15:07:38-- http://dataincubator-wqu.s3.amazonaws.com/caldata/cal_housing.pkz\n",
63 | "Resolving dataincubator-wqu.s3.amazonaws.com (dataincubator-wqu.s3.amazonaws.com)... 52.217.13.44\n",
64 | "Connecting to dataincubator-wqu.s3.amazonaws.com (dataincubator-wqu.s3.amazonaws.com)|52.217.13.44|:80... connected.\n",
65 | "HTTP request sent, awaiting response... 200 OK\n",
66 | "Length: 366863 (358K) [binary/octet-stream]\n",
67 | "Saving to: ‘/home/jovyan/scikit_learn_data/cal_housing.pkz’\n",
68 | "\n",
69 | "cal_housing.pkz 100%[===================>] 358.26K --.-KB/s in 0.04s \n",
70 | "\n",
71 | "2021-01-27 15:07:38 (9.86 MB/s) - ‘/home/jovyan/scikit_learn_data/cal_housing.pkz’ saved [366863/366863]\n",
72 | "\n"
73 | ]
74 | }
75 | ],
76 | "source": [
77 | "!wget http://dataincubator-wqu.s3.amazonaws.com/caldata/cal_housing.pkz -nc -P ~/scikit_learn_data/"
78 | ]
79 | },
80 | {
81 | "cell_type": "code",
82 | "execution_count": 15,
83 | "metadata": {},
84 | "outputs": [
85 | {
86 | "name": "stdout",
87 | "output_type": "stream",
88 | "text": [
89 | ".. _california_housing_dataset:\n",
90 | "\n",
91 | "California Housing dataset\n",
92 | "--------------------------\n",
93 | "\n",
94 | "**Data Set Characteristics:**\n",
95 | "\n",
96 | " :Number of Instances: 20640\n",
97 | "\n",
98 | " :Number of Attributes: 8 numeric, predictive attributes and the target\n",
99 | "\n",
100 | " :Attribute Information:\n",
101 | " - MedInc median income in block\n",
102 | " - HouseAge median house age in block\n",
103 | " - AveRooms average number of rooms\n",
104 | " - AveBedrms average number of bedrooms\n",
105 | " - Population block population\n",
106 | " - AveOccup average house occupancy\n",
107 | " - Latitude house block latitude\n",
108 | " - Longitude house block longitude\n",
109 | "\n",
110 | " :Missing Attribute Values: None\n",
111 | "\n",
112 | "This dataset was obtained from the StatLib repository.\n",
113 | "http://lib.stat.cmu.edu/datasets/\n",
114 | "\n",
115 | "The target variable is the median house value for California districts.\n",
116 | "\n",
117 | "This dataset was derived from the 1990 U.S. census, using one row per census\n",
118 | "block group. A block group is the smallest geographical unit for which the U.S.\n",
119 | "Census Bureau publishes sample data (a block group typically has a population\n",
120 | "of 600 to 3,000 people).\n",
121 | "\n",
122 | "It can be downloaded/loaded using the\n",
123 | ":func:`sklearn.datasets.fetch_california_housing` function.\n",
124 | "\n",
125 | ".. topic:: References\n",
126 | "\n",
127 | " - Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,\n",
128 | " Statistics and Probability Letters, 33 (1997) 291-297\n",
129 | "\n"
130 | ]
131 | },
132 | {
133 | "data": {
134 | "text/html": [
135 | "| \n", 153 | " | MedInc | \n", 154 | "HouseAge | \n", 155 | "AveRooms | \n", 156 | "AveBedrms | \n", 157 | "Population | \n", 158 | "AveOccup | \n", 159 | "Latitude | \n", 160 | "Longitude | \n", 161 | "
|---|---|---|---|---|---|---|---|---|
| 0 | \n", 166 | "8.3252 | \n", 167 | "41.0 | \n", 168 | "6.984127 | \n", 169 | "1.023810 | \n", 170 | "322.0 | \n", 171 | "2.555556 | \n", 172 | "37.88 | \n", 173 | "-122.23 | \n", 174 | "
| 1 | \n", 177 | "8.3014 | \n", 178 | "21.0 | \n", 179 | "6.238137 | \n", 180 | "0.971880 | \n", 181 | "2401.0 | \n", 182 | "2.109842 | \n", 183 | "37.86 | \n", 184 | "-122.22 | \n", 185 | "
| 2 | \n", 188 | "7.2574 | \n", 189 | "52.0 | \n", 190 | "8.288136 | \n", 191 | "1.073446 | \n", 192 | "496.0 | \n", 193 | "2.802260 | \n", 194 | "37.85 | \n", 195 | "-122.24 | \n", 196 | "
| 3 | \n", 199 | "5.6431 | \n", 200 | "52.0 | \n", 201 | "5.817352 | \n", 202 | "1.073059 | \n", 203 | "558.0 | \n", 204 | "2.547945 | \n", 205 | "37.85 | \n", 206 | "-122.25 | \n", 207 | "
| 4 | \n", 210 | "3.8462 | \n", 211 | "52.0 | \n", 212 | "6.281853 | \n", 213 | "1.081081 | \n", 214 | "565.0 | \n", 215 | "2.181467 | \n", 216 | "37.85 | \n", 217 | "-122.25 | \n", 218 | "
| Kernel Name | \n", 322 | "$K(x,x')$ | \n", 323 | "`kernel` keyword argument | \n", 324 | "
|---|---|---|
| Linear Kernel | \n", 328 | "$x \\cdot x'$ | \n", 329 | "`'linear'` | \n", 330 | "
| $d$-th Degree Polynomial | \n", 334 | "$(r + \\gamma\\ x \\cdot x')^d$ | \n", 335 | "`poly'` | \n", 336 | "
| Gaussian RBF | \n", 340 | "$\\exp{(- \\gamma\\, \\|x - x' \\|_2)} $ | \n", 341 | "`'rbf'` | \n", 342 | "
| Sigmoid | \n", 346 | "$\\tanh(\\gamma\\, x \\cdot x' + r)$ | \n", 347 | "`'sigmoid'` | \n", 348 | "