![](\"https://nteract.github.io/assets/images/feature_nteract_logo_header_white@2x.png\"){kind=logo}
\n",
30 | "![\"Jupyter](\"http://jupyter.org/assets/main-logo.svg\"){kind=logo}
"
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {},
25 | "source": [
26 | "## nteract\n",
27 | "\n",
28 | "\n",
30 | "![](\"../images/pycharm.png\")
\n",
87 | "\n",
88 | "\n",
89 | "### Scientific Mode\n",
90 | "\n",
91 | "[PyCharm Scientific Mode](https://www.jetbrains.com/help/pycharm/matplotlib-tutorial.html)"
92 | ]
93 | },
94 | {
95 | "cell_type": "markdown",
96 | "metadata": {},
97 | "source": [
98 | "## Others worthwile mentioning..."
99 | ]
100 | },
101 | {
102 | "cell_type": "markdown",
103 | "metadata": {},
104 | "source": [
105 | "* **Spyder**: [https://www.spyder-ide.org]()\n",
106 | "* **Rodeo**: [https://rodeo.yhat.com]()\n",
107 | "* **VS Code**: [https://code.visualstudio.com]()"
108 | ]
109 | }
110 | ],
111 | "metadata": {
112 | "kernelspec": {
113 | "display_name": "Python 3.6 (Develer Science)",
114 | "language": "python",
115 | "name": "develer-science"
116 | },
117 | "language_info": {
118 | "codemirror_mode": {
119 | "name": "ipython",
120 | "version": 3
121 | },
122 | "file_extension": ".py",
123 | "mimetype": "text/x-python",
124 | "name": "python",
125 | "nbconvert_exporter": "python",
126 | "pygments_lexer": "ipython3",
127 | "version": "3.6.6"
128 | }
129 | },
130 | "nbformat": 4,
131 | "nbformat_minor": 2
132 | }
133 |
--------------------------------------------------------------------------------
/2_alchemist/0. Introducing Pandas.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Pandas\n",
8 | "\n",
9 | "Pandas is the Swiss-Multipurpose Knife for Data Analysis in Python. With Pandas dealing with data-analysis is easy and simple but there are some things you need to get your head around first as Data-Frames and Data-Series. \n",
10 | "\n",
11 | "The tutorial provides a compact introduction to Pandas for beginners for I/O, data visualisation, statistical data analysis and aggregation within Jupiter notebooks."
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "## Content at a glance\n",
19 | "\n",
20 | "#### A Practical Start: Reading and Writing Data Across Multiple Formats \n",
21 | "\n",
22 | "* CSV\n",
23 | "* Excel\n",
24 | "* JSON\n",
25 | "* Clipboard\n",
26 | " \n",
27 | "* data\n",
28 | " * .info\n",
29 | " * .describe\n",
30 | "\n",
31 | "#### DataSeries & DataFrames / NumPy\n",
32 | "\n",
33 | "* Ode to NumPy\n",
34 | "* Data-Series\n",
35 | "* Data-Frames\n",
36 | "\n",
37 | "#### Data selection & Indexing\n",
38 | "\n",
39 | "* Data-Series: \n",
40 | " * Slicing\n",
41 | " * Access by label\n",
42 | " * Index\n",
43 | "* Data-Frames: \n",
44 | " * Slicing\n",
45 | " * Access by label\n",
46 | " * Peek into joining data\n",
47 | "* Returns a copy / inplace\n",
48 | "* Boolean indexing\n",
49 | "\n",
50 | "#### Operations\n",
51 | " \n",
52 | " * add/substract\n",
53 | " * multiply\n",
54 | " * mention Index but don't go deep\n",
55 | "\n",
56 | "#### Data Visualisation\n",
57 | "\n",
58 | " * plot your data directly into your notebook\n",
59 | " \n",
60 | "#### Anti Patterns\n",
61 | "\n",
62 | " * a collection of (anti-)patterns when using Pandas"
63 | ]
64 | },
65 | {
66 | "cell_type": "markdown",
67 | "metadata": {},
68 | "source": [
69 | "---"
70 | ]
71 | },
72 | {
73 | "cell_type": "markdown",
74 | "metadata": {},
75 | "source": [
76 | "## Brief Introduction to Pandas\n",
77 | "\n",
78 | "Pandas builds on top of two main data structures: **Data Frame** and **Series**"
79 | ]
80 | },
81 | {
82 | "cell_type": "markdown",
83 | "metadata": {},
84 | "source": [
85 | "### Data Frame _from the outside_\n",
86 | "\n",
87 | "![](\"../images/df_outside.png\")
"
88 | ]
89 | },
90 | {
91 | "cell_type": "markdown",
92 | "metadata": {},
93 | "source": [
94 | "### Data Frame _from the inside_\n",
95 | "\n",
96 | "![](\"../images/df_inside.png\")
"
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "metadata": {},
102 | "source": [
103 | "### Data Frame vs Numpy Array"
104 | ]
105 | },
106 | {
107 | "cell_type": "markdown",
108 | "metadata": {},
109 | "source": [
110 | "#### Numpy Array\n",
111 | "\n",
112 | "![](\"../images/ndarray.png\")
"
113 | ]
114 | },
115 | {
116 | "cell_type": "markdown",
117 | "metadata": {},
118 | "source": [
119 | "#### Pandas Data Frame"
120 | ]
121 | },
122 | {
123 | "cell_type": "markdown",
124 | "metadata": {},
125 | "source": [
126 | "![](\"../images/df_inside_numpy.png\")
"
127 | ]
128 | }
129 | ],
130 | "metadata": {
131 | "kernelspec": {
132 | "display_name": "Python 3.6 (Develer Science)",
133 | "language": "python",
134 | "name": "develer-science"
135 | },
136 | "language_info": {
137 | "codemirror_mode": {
138 | "name": "ipython",
139 | "version": 3
140 | },
141 | "file_extension": ".py",
142 | "mimetype": "text/x-python",
143 | "name": "python",
144 | "nbconvert_exporter": "python",
145 | "pygments_lexer": "ipython3",
146 | "version": "3.6.6"
147 | }
148 | },
149 | "nbformat": 4,
150 | "nbformat_minor": 2
151 | }
152 |
--------------------------------------------------------------------------------
/1_apprentice/2.3. Scipy Challenge.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# SciPy Challenge\n",
8 | "\n",
9 | "## SciPy at a Glance\n",
10 | "\n",
11 | "The SciPy framework builds on top of the low-level NumPy framework for multidimensional arrays, and provides a large number of higher-level scientific algorithms. Some of the topics that SciPy covers are:\n",
12 | "\n",
13 | "* Special functions ([scipy.special](http://docs.scipy.org/doc/scipy/reference/special.html))\n",
14 | "* Integration ([scipy.integrate](http://docs.scipy.org/doc/scipy/reference/integrate.html))\n",
15 | "* Optimization ([scipy.optimize](http://docs.scipy.org/doc/scipy/reference/optimize.html))\n",
16 | "* Interpolation ([scipy.interpolate](http://docs.scipy.org/doc/scipy/reference/interpolate.html))\n",
17 | "* Fourier Transforms ([scipy.fftpack](http://docs.scipy.org/doc/scipy/reference/fftpack.html))\n",
18 | "* Signal Processing ([scipy.signal](http://docs.scipy.org/doc/scipy/reference/signal.html))\n",
19 | "* Linear Algebra ([scipy.linalg](http://docs.scipy.org/doc/scipy/reference/linalg.html))\n",
20 | "* Sparse Eigenvalue Problems ([scipy.sparse](http://docs.scipy.org/doc/scipy/reference/sparse.html))\n",
21 | "* Statistics ([scipy.stats](http://docs.scipy.org/doc/scipy/reference/stats.html))\n",
22 | "* Multi-dimensional image processing ([scipy.ndimage](http://docs.scipy.org/doc/scipy/reference/ndimage.html))\n",
23 | "* File IO ([scipy.io](http://docs.scipy.org/doc/scipy/reference/io.html))"
24 | ]
25 | },
26 | {
27 | "cell_type": "markdown",
28 | "metadata": {},
29 | "source": [
30 | "## Sparse Matrices\n",
31 | "\n",
32 | "**Sparse Matrices** are very nice in some situations. \n",
33 | "\n",
34 | "For example, in some machine learning tasks, especially those associated\n",
35 | "with textual analysis, the data may be mostly zeros. \n",
36 | "\n",
37 | "Storing all these zeros is very inefficient. \n",
38 | "\n",
39 | "We can create and manipulate sparse matrices using the `scipy.sparse` module."
40 | ]
41 | },
42 | {
43 | "cell_type": "markdown",
44 | "metadata": {},
45 | "source": [
46 | "There exists different implementations of sparse matrices, which are supposed to be efficient in different scenarios:\n",
47 | "\n",
48 | "- CSR: Compressed Sparse Rows\n",
49 | "- CSC: Compressec Sparse Colums\n",
50 | "- DOK: Dictionary of Keys\n",
51 | "- LIL: List of Lists\n",
52 | "- BSR: Block Sparse Row"
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {},
58 | "source": [
59 | "## Ex 1.1\n",
60 | "\n",
61 | "Create a big numpy **dense** matrix filled with random numbers in \n",
62 | "`[0, 1)`.\n",
63 | "Generate a random number within this range and subsitute all the elements in the matrix **less than** this number with a zero.\n",
64 | "\n",
65 | "Save resulting matrix as a `DOK` sparse matrix"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": null,
71 | "metadata": {
72 | "collapsed": true
73 | },
74 | "outputs": [],
75 | "source": []
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {},
80 | "source": [
81 | "## Ex 1.2\n",
82 | "\n",
83 | "Repeat the previous exercise, but this time use a `CSR` sparse matrix."
84 | ]
85 | },
86 | {
87 | "cell_type": "code",
88 | "execution_count": null,
89 | "metadata": {
90 | "collapsed": true
91 | },
92 | "outputs": [],
93 | "source": []
94 | },
95 | {
96 | "cell_type": "markdown",
97 | "metadata": {},
98 | "source": [
99 | "## Ex 1.3\n",
100 | "\n",
101 | "Transform the previously generated sparse matrix back to a full dense `numpy.array`."
102 | ]
103 | },
104 | {
105 | "cell_type": "code",
106 | "execution_count": null,
107 | "metadata": {
108 | "collapsed": true
109 | },
110 | "outputs": [],
111 | "source": []
112 | },
113 | {
114 | "cell_type": "markdown",
115 | "metadata": {},
116 | "source": [
117 | "## Ex 1.4 \n",
118 | "\n",
119 | "Generate two sparse Matrix and sum them together, choosing the most appropriate internal representation (i.e. `LIL`, `CSR`, `DOK`...).\n",
120 | "\n",
121 | "#### Hint: Oh c'mon.. :)"
122 | ]
123 | },
124 | {
125 | "cell_type": "code",
126 | "execution_count": null,
127 | "metadata": {
128 | "collapsed": true
129 | },
130 | "outputs": [],
131 | "source": []
132 | }
133 | ],
134 | "metadata": {
135 | "kernelspec": {
136 | "display_name": "Python 3.6 (Develer Science)",
137 | "language": "python",
138 | "name": "develer-science"
139 | },
140 | "language_info": {
141 | "codemirror_mode": {
142 | "name": "ipython",
143 | "version": 3
144 | },
145 | "file_extension": ".py",
146 | "mimetype": "text/x-python",
147 | "name": "python",
148 | "nbconvert_exporter": "python",
149 | "pygments_lexer": "ipython3",
150 | "version": "3.6.6"
151 | }
152 | },
153 | "nbformat": 4,
154 | "nbformat_minor": 2
155 | }
156 |
--------------------------------------------------------------------------------
/3_mage/5.1. Review of Scikit-learn API.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# A recap on Scikit-learn's estimator interface"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {
13 | "slideshow": {
14 | "slide_type": "subslide"
15 | }
16 | },
17 | "source": [
18 | "Scikit-learn strives to have a uniform interface across all methods. Given a scikit-learn *estimator*\n",
19 | "object named `model`, the following methods are available (not all for each model)"
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {
25 | "slideshow": {
26 | "slide_type": "subslide"
27 | }
28 | },
29 | "source": [
30 | "- Available in **all Estimators**\n",
31 | " + `model.fit()` : fit training data. For supervised learning applications,\n",
32 | " this accepts two arguments: the data `X` and the labels `y` (e.g. `model.fit(X, y)`).\n",
33 | " For unsupervised learning applications, ``fit`` takes only a single argument,\n",
34 | " the data `X` (e.g. `model.fit(X)`)."
35 | ]
36 | },
37 | {
38 | "cell_type": "markdown",
39 | "metadata": {
40 | "slideshow": {
41 | "slide_type": "subslide"
42 | }
43 | },
44 | "source": [
45 | "- Available in **supervised estimators**\n",
46 | " + `model.predict()` : given a trained model, predict the label of a new set of data.\n",
47 | " This method accepts one argument, the new data `X_new` (e.g. `model.predict(X_new)`),\n",
48 | " and returns the learned label for each object in the array.\n",
49 | " + `model.predict_proba()` : For classification problems, some estimators also provide\n",
50 | " this method, which returns the probability that a new observation has each categorical label.\n",
51 | " In this case, the label with the highest probability is returned by `model.predict()`.\n",
52 | " "
53 | ]
54 | },
55 | {
56 | "cell_type": "markdown",
57 | "metadata": {
58 | "slideshow": {
59 | "slide_type": "subslide"
60 | }
61 | },
62 | "source": [
63 | "- Available in **supervised estimators** (cont.)\n",
64 | " \n",
65 | " + `model.decision_function()` : For classification problems, some estimators provide an uncertainty estimate that is not a probability. For binary classification, a decision_function >= 0 means the positive class will be predicted, while < 0 means the negative class.\n",
66 | " + `model.score()` : for classification or regression problems, most (all?) estimators implement\n",
67 | " a score method. Scores are between 0 and 1, with a larger score indicating a better fit."
68 | ]
69 | },
70 | {
71 | "cell_type": "markdown",
72 | "metadata": {
73 | "slideshow": {
74 | "slide_type": "subslide"
75 | }
76 | },
77 | "source": [
78 | "- Available in **supervised estimators** (cont.)\n",
79 | "\n",
80 | " + `model.transform()` : For feature selection algorithms, this will reduce the dataset to the selected features. For some classification and regression models such as some linear models and random forests, this method reduces the dataset to the most informative features. These classification and regression models can therefor also be used as feature selection methods."
81 | ]
82 | },
83 | {
84 | "cell_type": "markdown",
85 | "metadata": {
86 | "slideshow": {
87 | "slide_type": "subslide"
88 | }
89 | },
90 | "source": [
91 | "- Available in **unsupervised estimators**\n",
92 | " + `model.transform()` : given an unsupervised model, transform new data into the new basis.\n",
93 | " This also accepts one argument `X_new`, and returns the new representation of the data based\n",
94 | " on the unsupervised model.\n",
95 | " + `model.fit_transform()` : some estimators implement this method,\n",
96 | " which more efficiently performs a fit and a transform on the same input data."
97 | ]
98 | },
99 | {
100 | "cell_type": "markdown",
101 | "metadata": {
102 | "slideshow": {
103 | "slide_type": "subslide"
104 | }
105 | },
106 | "source": [
107 | "- Available in **unsupervised estimators** (cont.)\n",
108 | "\n",
109 | " + `model.predict()` : for clustering algorithms, the predict method will produce cluster labels for new data points. Not all clustering methods have this functionality.\n",
110 | " + `model.predict_proba()` : Gaussian mixture models (GMMs) provide the probability for each point to be generated by a given mixture component.\n",
111 | " + `model.score()` : Density models like KDE and GMMs provide the likelihood of the data under the model."
112 | ]
113 | },
114 | {
115 | "cell_type": "markdown",
116 | "metadata": {
117 | "slideshow": {
118 | "slide_type": "subslide"
119 | }
120 | },
121 | "source": [
122 | "Apart from ``fit``, the two most important functions are arguably ``predict`` to produce a target variable (a ``y``) ``transform``, which produces a new representation of the data (an ``X``).\n",
123 | "The following table shows for which class of models which function applies:\n",
124 | "\n"
125 | ]
126 | },
127 | {
128 | "cell_type": "markdown",
129 | "metadata": {},
130 | "source": [
131 | "| ``model.predict`` | ``model.transform`` |
|---|---|
| Classification | Preprocessing |
| Regression | Dimensionality Reduction |
| Clustering | Feature Extraction |
| Feature selection |
![](\"../images/ml-wordle-436.jpg\")
"
45 | ]
46 | },
47 | {
48 | "cell_type": "markdown",
49 | "metadata": {
50 | "slideshow": {
51 | "slide_type": "subslide"
52 | }
53 | },
54 | "source": [
55 | "> Machine learning teaches machines how to carry out tasks by themselves. It is that simple.\n",
56 | "The complexity comes with the details.\n",
57 | "\n",
58 | "_W. Richert & L.P. Coelho, 2013\n",
59 | "Building Machine Learning Systems with Python_"
60 | ]
61 | },
62 | {
63 | "cell_type": "markdown",
64 | "metadata": {
65 | "slideshow": {
66 | "slide_type": "subslide"
67 | }
68 | },
69 | "source": [
70 | "Machine learning is the process to automatically **extract knowledge** from data, usually with the goal of making **predictions** on _new_, _unseen_ data. \n",
71 | "\n",
72 | "A classical example is a _spam filter_, for which the user keeps labeling incoming mails as either spam or not spam. \n",
73 | "\n",
74 | "A machine learning algorithm then \"learns\" what distinguishes spam from normal emails, and can predict for new emails whether they are spam or not."
75 | ]
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {
80 | "slideshow": {
81 | "slide_type": "subslide"
82 | }
83 | },
84 | "source": [
85 | "Central to machine learning is the concept of **making decision automatically** from data, **without the user specifying explicit rules** how this decision should be made.\n",
86 | "\n",
87 | "For the case of emails, the user doesn't provide a list of words or characteristics that make an email spam. Instead, the user provides examples of spam and non-spam emails."
88 | ]
89 | },
90 | {
91 | "cell_type": "markdown",
92 | "metadata": {
93 | "slideshow": {
94 | "slide_type": "subslide"
95 | }
96 | },
97 | "source": [
98 | "The second central concept is **generalization**. \n",
99 | "\n",
100 | "The goal of a machine learning algorithm is to predict on new, previously unseen data. We are not interested in marking an email as spam or not, that the human already labeled. Instead, we want to make the users life easier by making an automatic decision for new incoming mail."
101 | ]
102 | },
103 | {
104 | "cell_type": "markdown",
105 | "metadata": {
106 | "slideshow": {
107 | "slide_type": "subslide"
108 | }
109 | },
110 | "source": [
111 | "There are two kinds of machine learning we will talk about in these notebooks: \n",
112 | "\n",
113 | "* **Supervised learning;** \n",
114 | "* **Unsupervised learning.**"
115 | ]
116 | },
117 | {
118 | "cell_type": "markdown",
119 | "metadata": {
120 | "slideshow": {
121 | "slide_type": "slide"
122 | }
123 | },
124 | "source": [
125 | "### Supervised Learning\n",
126 | "\n",
127 | "In **Supervised Learning**, we have a dataset consisting of both input features and a desired output, such as in the spam / no-spam example.\n",
128 | "\n",
129 | "The task is to construct a model (or program) which is able to predict the desired output of an unseen object\n",
130 | "given the set of features."
131 | ]
132 | },
133 | {
134 | "cell_type": "markdown",
135 | "metadata": {
136 | "slideshow": {
137 | "slide_type": "subslide"
138 | }
139 | },
140 | "source": [
141 | "![](\"../images/ml_supervised_example.png\")
"
142 | ]
143 | },
144 | {
145 | "cell_type": "markdown",
146 | "metadata": {
147 | "slideshow": {
148 | "slide_type": "subslide"
149 | }
150 | },
151 | "source": [
152 | "Supervised learning is further broken down into two categories, **classification** and **regression**.\n",
153 | "\n",
154 | "In classification, the label is discrete (a.k.a. _Categorical Data_, i.e. _Integer values_), such as \"spam\" or \"no spam\". \n",
155 | "\n",
156 | "In other words, it provides a clear-cut distinction between categories. \n",
157 | "\n",
158 | "In regression, the label is continuous, i.e. _Float output_."
159 | ]
160 | },
161 | {
162 | "cell_type": "markdown",
163 | "metadata": {
164 | "slideshow": {
165 | "slide_type": "subslide"
166 | }
167 | },
168 | "source": [
169 | "### Other Examples\n",
170 | "\n",
171 | "Some more complicated examples are:\n",
172 | "\n",
173 | "- given a multicolor image of an object through a telescope, determine\n",
174 | " whether that object is a star, a quasar, or a galaxy.\n",
175 | "- given a photograph of a person, identify the person in the photo.\n",
176 | "- given a list of movies a person has watched and their personal rating\n",
177 | " of the movie, recommend a list of movies they would like.\n",
178 | "- given a persons age, education and position, infer their salary"
179 | ]
180 | },
181 | {
182 | "cell_type": "markdown",
183 | "metadata": {
184 | "slideshow": {
185 | "slide_type": "subslide"
186 | }
187 | },
188 | "source": [
189 | "What these tasks have in common is that there is one or more unknown\n",
190 | "quantities associated with the object which needs to be determined from other\n",
191 | "observed quantities."
192 | ]
193 | },
194 | {
195 | "cell_type": "markdown",
196 | "metadata": {
197 | "slideshow": {
198 | "slide_type": "subslide"
199 | }
200 | },
201 | "source": [
202 | "### For example\n",
203 | "\n",
204 | "* In astronomy, the task of determining whether an object is a star, a galaxy, or a quasar is a **classification problem**: the label is from three distinct categories. \n",
205 | "\n",
206 | "* On the other hand, we might wish to estimate the age of an object based on such observations: this would be a **regression problem**, because the label (age) is a continuous quantity."
207 | ]
208 | },
209 | {
210 | "cell_type": "markdown",
211 | "metadata": {
212 | "slideshow": {
213 | "slide_type": "slide"
214 | }
215 | },
216 | "source": [
217 | "### Unsupervised Learning\n",
218 | "\n",
219 | "In **Unsupervised Learning** there is no desired output associated with the data.\n",
220 | "\n",
221 | "Instead, we are interested in extracting some form of knowledge or model from the given data.\n",
222 | "\n",
223 | "In a sense, you can think of unsupervised learning as a means of discovering labels from the data itself.\n",
224 | "\n",
225 | "Unsupervised learning comprises tasks such as *dimensionality reduction*, *clustering*, and\n",
226 | "*density estimation*. "
227 | ]
228 | },
229 | {
230 | "cell_type": "markdown",
231 | "metadata": {
232 | "slideshow": {
233 | "slide_type": "subslide"
234 | }
235 | },
236 | "source": [
237 | "![](\"../images/ml_unsupervised_example.png\")
"
238 | ]
239 | },
240 | {
241 | "cell_type": "markdown",
242 | "metadata": {
243 | "slideshow": {
244 | "slide_type": "fragment"
245 | }
246 | },
247 | "source": [
248 | "Unsupervised learning is often harder to understand and to evaluate.\n",
249 | "\n",
250 | "Sometimes the two may even be combined: e.g. Unsupervised learning can be used to find useful\n",
251 | "features in heterogeneous data, and then these features can be used within a supervised\n",
252 | "framework."
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {
258 | "slideshow": {
259 | "slide_type": "subslide"
260 | }
261 | },
262 | "source": [
263 | "### Other Examples\n",
264 | "\n",
265 | "Some more involved unsupervised learning problems are:\n",
266 | "\n",
267 | "- given detailed observations of distant galaxies, determine which features or combinations of\n",
268 | " features summarize best the information.\n",
269 | "- given a mixture of two sound sources (for example, a person talking over some music),\n",
270 | " separate the two (this is called the [blind source separation](http://en.wikipedia.org/wiki/Blind_signal_separation) problem).\n",
271 | "- given a large collection of news articles, find recurring topics inside these articles.\n",
272 | "- given a collection of images, cluster similar images together (for example to group them when visualizing a collection)"
273 | ]
274 | },
275 | {
276 | "cell_type": "markdown",
277 | "metadata": {
278 | "slideshow": {
279 | "slide_type": "slide"
280 | }
281 | },
282 | "source": [
283 | "# Scikit-learn at a Glance"
284 | ]
285 | },
286 | {
287 | "cell_type": "markdown",
288 | "metadata": {},
289 | "source": [
290 | "![](\"../images/scikit-learn-cheatsheet.png\")
"
291 | ]
292 | }
293 | ],
294 | "metadata": {
295 | "celltoolbar": "Slideshow",
296 | "kernelspec": {
297 | "display_name": "Python 3.6 (Develer Science)",
298 | "language": "python",
299 | "name": "develer-science"
300 | },
301 | "language_info": {
302 | "codemirror_mode": {
303 | "name": "ipython",
304 | "version": 3
305 | },
306 | "file_extension": ".py",
307 | "mimetype": "text/x-python",
308 | "name": "python",
309 | "nbconvert_exporter": "python",
310 | "pygments_lexer": "ipython3",
311 | "version": "3.6.6"
312 | }
313 | },
314 | "nbformat": 4,
315 | "nbformat_minor": 2
316 | }
317 |
--------------------------------------------------------------------------------
/2_alchemist/5. Level Up.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# StarWars"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "### Introduction:\n",
15 | "\n",
16 | "This time you will create the data.\n",
17 | "```\n",
18 | "raw_data = {\"name\": ['Dart Vather', 'Leia Princess','Luke Skywalker','Yoda'],\n",
19 | " \"class\": ['Empire','Rebels','Rebels','Rebels'],\n",
20 | " \"gender\": ['M', 'F', 'M', 'J'],\n",
21 | " \"hp\": [45, 39, 44, 45],\n",
22 | " \"like\": ['yes', 'no','yes','no'] \n",
23 | " }\n",
24 | "````\n",
25 | "\n",
26 | "### Step 1. Import the necessary libraries"
27 | ]
28 | },
29 | {
30 | "cell_type": "code",
31 | "execution_count": null,
32 | "metadata": {
33 | "collapsed": false
34 | },
35 | "outputs": [],
36 | "source": []
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "### Step 2. Create a data dictionary"
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": null,
48 | "metadata": {
49 | "collapsed": true
50 | },
51 | "outputs": [],
52 | "source": []
53 | },
54 | {
55 | "cell_type": "markdown",
56 | "metadata": {},
57 | "source": [
58 | "### Step 3. Create a DataFrame object starting from the data dictionary, and print its head"
59 | ]
60 | },
61 | {
62 | "cell_type": "code",
63 | "execution_count": null,
64 | "metadata": {
65 | "collapsed": false
66 | },
67 | "outputs": [],
68 | "source": []
69 | },
70 | {
71 | "cell_type": "markdown",
72 | "metadata": {},
73 | "source": [
74 | "### Step 4. Ops...it seems the DataFrame columns are in alphabetical order. Place the order of the columns as name, type, hp, evolution, pokedex"
75 | ]
76 | },
77 | {
78 | "cell_type": "code",
79 | "execution_count": null,
80 | "metadata": {
81 | "collapsed": false
82 | },
83 | "outputs": [],
84 | "source": []
85 | },
86 | {
87 | "cell_type": "markdown",
88 | "metadata": {},
89 | "source": [
90 | "### Step 5. Add another column called actor, and insert what you have in mind."
91 | ]
92 | },
93 | {
94 | "cell_type": "code",
95 | "execution_count": null,
96 | "metadata": {
97 | "collapsed": false
98 | },
99 | "outputs": [],
100 | "source": []
101 | },
102 | {
103 | "cell_type": "markdown",
104 | "metadata": {},
105 | "source": [
106 | "### Step 6. Present the type of each column"
107 | ]
108 | },
109 | {
110 | "cell_type": "code",
111 | "execution_count": null,
112 | "metadata": {
113 | "collapsed": false
114 | },
115 | "outputs": [],
116 | "source": []
117 | },
118 | {
119 | "cell_type": "markdown",
120 | "metadata": {},
121 | "source": [
122 | "# Filtering and Sorting Data"
123 | ]
124 | },
125 | {
126 | "cell_type": "markdown",
127 | "metadata": {},
128 | "source": [
129 | "This time we are going to pull data directly from the internet.\n",
130 | "Special thanks to: https://github.com/justmarkham for sharing the dataset and materials.\n",
131 | "\n",
132 | "### Step 1. Import the necessary libraries"
133 | ]
134 | },
135 | {
136 | "cell_type": "code",
137 | "execution_count": null,
138 | "metadata": {
139 | "collapsed": false
140 | },
141 | "outputs": [],
142 | "source": []
143 | },
144 | {
145 | "cell_type": "markdown",
146 | "metadata": {},
147 | "source": [
148 | "### Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/justmarkham/DAT8/master/data/chipotle.tsv). "
149 | ]
150 | },
151 | {
152 | "cell_type": "markdown",
153 | "metadata": {},
154 | "source": [
155 | "### Step 3. Assign it to a variable called chipo."
156 | ]
157 | },
158 | {
159 | "cell_type": "code",
160 | "execution_count": null,
161 | "metadata": {
162 | "collapsed": false
163 | },
164 | "outputs": [],
165 | "source": []
166 | },
167 | {
168 | "cell_type": "markdown",
169 | "metadata": {},
170 | "source": [
171 | "### Step 4. How many products cost more than $10.00?"
172 | ]
173 | },
174 | {
175 | "cell_type": "code",
176 | "execution_count": null,
177 | "metadata": {
178 | "collapsed": false
179 | },
180 | "outputs": [],
181 | "source": []
182 | },
183 | {
184 | "cell_type": "markdown",
185 | "metadata": {},
186 | "source": [
187 | "### Step 5. What is the price of each item? \n",
188 | "###### print a data frame with only two columns item_name and item_price"
189 | ]
190 | },
191 | {
192 | "cell_type": "code",
193 | "execution_count": null,
194 | "metadata": {
195 | "collapsed": false
196 | },
197 | "outputs": [],
198 | "source": []
199 | },
200 | {
201 | "cell_type": "markdown",
202 | "metadata": {},
203 | "source": [
204 | "### Step 6. Sort by the name of the item"
205 | ]
206 | },
207 | {
208 | "cell_type": "code",
209 | "execution_count": null,
210 | "metadata": {
211 | "collapsed": false
212 | },
213 | "outputs": [],
214 | "source": []
215 | },
216 | {
217 | "cell_type": "markdown",
218 | "metadata": {},
219 | "source": [
220 | "### Step 7. What was the quantity of the most expensive item ordered?"
221 | ]
222 | },
223 | {
224 | "cell_type": "code",
225 | "execution_count": null,
226 | "metadata": {
227 | "collapsed": false
228 | },
229 | "outputs": [],
230 | "source": []
231 | },
232 | {
233 | "cell_type": "markdown",
234 | "metadata": {},
235 | "source": [
236 | "### Step 8. How many times were a Veggie Salad Bowl ordered?"
237 | ]
238 | },
239 | {
240 | "cell_type": "code",
241 | "execution_count": null,
242 | "metadata": {
243 | "collapsed": false
244 | },
245 | "outputs": [],
246 | "source": []
247 | },
248 | {
249 | "cell_type": "markdown",
250 | "metadata": {},
251 | "source": [
252 | "### Step 9. How many times people orderd more than one Canned Soda?"
253 | ]
254 | },
255 | {
256 | "cell_type": "code",
257 | "execution_count": null,
258 | "metadata": {
259 | "collapsed": false
260 | },
261 | "outputs": [],
262 | "source": []
263 | },
264 | {
265 | "cell_type": "markdown",
266 | "metadata": {},
267 | "source": [
268 | "# Exercises - GroupBy"
269 | ]
270 | },
271 | {
272 | "cell_type": "markdown",
273 | "metadata": {},
274 | "source": [
275 | "### Introduction:\n",
276 | "\n",
277 | "GroupBy can be summarizes as Split-Apply-Combine.\n",
278 | "\n",
279 | "Special thanks to: https://github.com/justmarkham for sharing the dataset and materials.\n",
280 | "\n",
281 | "Check out this Diagram:\n",
282 | "\n",
283 | " \n",
284 | "\n",
285 | "\n",
286 | "### Step 1. Import the necessary libraries"
287 | ]
288 | },
289 | {
290 | "cell_type": "code",
291 | "execution_count": null,
292 | "metadata": {
293 | "collapsed": false
294 | },
295 | "outputs": [],
296 | "source": []
297 | },
298 | {
299 | "cell_type": "markdown",
300 | "metadata": {},
301 | "source": [
302 | "### Step 2. Import the dataset from this [address](https://raw.githubusercontent.com/justmarkham/DAT8/master/data/drinks.csv). "
303 | ]
304 | },
305 | {
306 | "cell_type": "markdown",
307 | "metadata": {},
308 | "source": [
309 | "### Step 3. Assign it to a variable called drinks."
310 | ]
311 | },
312 | {
313 | "cell_type": "code",
314 | "execution_count": null,
315 | "metadata": {
316 | "collapsed": false
317 | },
318 | "outputs": [],
319 | "source": []
320 | },
321 | {
322 | "cell_type": "markdown",
323 | "metadata": {},
324 | "source": [
325 | "### Step 4. Which continent drinks more beer on average?"
326 | ]
327 | },
328 | {
329 | "cell_type": "code",
330 | "execution_count": null,
331 | "metadata": {
332 | "collapsed": false
333 | },
334 | "outputs": [],
335 | "source": []
336 | },
337 | {
338 | "cell_type": "markdown",
339 | "metadata": {},
340 | "source": [
341 | "### Step 5. For each continent print the statistics for wine consumption."
342 | ]
343 | },
344 | {
345 | "cell_type": "code",
346 | "execution_count": null,
347 | "metadata": {
348 | "collapsed": false
349 | },
350 | "outputs": [],
351 | "source": []
352 | },
353 | {
354 | "cell_type": "markdown",
355 | "metadata": {},
356 | "source": [
357 | "### Step 6. Print the mean alcoohol consumption per continent for every column"
358 | ]
359 | },
360 | {
361 | "cell_type": "code",
362 | "execution_count": null,
363 | "metadata": {
364 | "collapsed": false
365 | },
366 | "outputs": [],
367 | "source": []
368 | },
369 | {
370 | "cell_type": "markdown",
371 | "metadata": {},
372 | "source": [
373 | "### Step 7. Print the median alcoohol consumption per continent for every column"
374 | ]
375 | },
376 | {
377 | "cell_type": "code",
378 | "execution_count": null,
379 | "metadata": {
380 | "collapsed": false
381 | },
382 | "outputs": [],
383 | "source": []
384 | },
385 | {
386 | "cell_type": "markdown",
387 | "metadata": {},
388 | "source": [
389 | "### Step 8. Print the mean, min and max values for spirit consumption.\n",
390 | "#### This time output a DataFrame"
391 | ]
392 | },
393 | {
394 | "cell_type": "code",
395 | "execution_count": null,
396 | "metadata": {
397 | "collapsed": false
398 | },
399 | "outputs": [],
400 | "source": []
401 | }
402 | ],
403 | "metadata": {
404 | "anaconda-cloud": {},
405 | "kernelspec": {
406 | "display_name": "Python 3.6 (Develer Science)",
407 | "language": "python",
408 | "name": "develer-science"
409 | },
410 | "language_info": {
411 | "codemirror_mode": {
412 | "name": "ipython",
413 | "version": 3
414 | },
415 | "file_extension": ".py",
416 | "mimetype": "text/x-python",
417 | "name": "python",
418 | "nbconvert_exporter": "python",
419 | "pygments_lexer": "ipython3",
420 | "version": "3.6.6"
421 | }
422 | },
423 | "nbformat": 4,
424 | "nbformat_minor": 2
425 | }
426 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Develer turns to Data Science
2 |
3 | ## Lecture notes for the "Data Science, the Pythonic way" @ [Develer](https://www.develer.com/)
4 |
5 |
17 | 
18 | @leriomaggio
19 | |
20 |
21 | 
22 | valeriomaggio
23 | |
24 |
25 | 
26 | valeriomaggio_at_gmail_dot_com
27 | |
28 |