6 | {%- if page.title -%}
7 |
{{ page.title }}
8 | {%- endif -%}
9 |
10 | {{ content | markdownify }}
11 |
12 |
13 | {% if site.paginate %}
14 | {% assign rawposts = paginator.posts %}
15 | {% else %}
16 | {% assign rawposts = site.posts %}
17 | {% endif %}
18 |
19 |
20 | {% assign posts = ''|split:'' %}
21 | {% for post in rawposts %}
22 | {% if post.hide != true %}
23 | {% assign posts = posts|push:post%}
24 | {% endif %}
25 | {% endfor %}
26 |
27 |
28 | {%- if posts.size > 0 -%}
29 | {%- if page.list_title -%}
30 |
{{ page.list_title }}
31 | {%- endif -%}
32 |
33 | {%- assign date_format = site.minima.date_format | default: "%b %-d, %Y" -%}
34 | {%- for post in posts -%}
35 | -
36 | {%- if site.show_image -%}
37 | {%- include post_list_image_card.html -%}
38 | {% else %}
39 | {%- include post_list.html -%}
40 | {%- endif -%}
41 |
42 | {%- endfor -%}
43 |
44 |
45 | {% if site.paginate and site.posts.size > site.paginate %}
46 |
61 | {%- endif %}
62 |
63 | {%- endif -%}
64 |
65 |
54 | {{ content | toc }}
55 |
56 | {%- if page.comments -%}
57 | {%- include utterances.html -%}
58 | {%- endif -%}
59 | {%- if site.disqus.shortname -%}
60 | {%- include disqus_comments.html -%}
61 | {%- endif -%}
62 |
63 |
--------------------------------------------------------------------------------
/_notebooks/2020-03-18-pytorch_hello_world.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "pytorch_hello_world.ipynb",
7 | "provenance": []
8 | },
9 | "kernelspec": {
10 | "name": "python3",
11 | "display_name": "Python 3"
12 | }
13 | },
14 | "cells": [
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {
18 | "id": "H7gQFbUxOQtb",
19 | "colab_type": "text"
20 | },
21 | "source": [
22 | "# A First Shot at Deep Learning with PyTorch\n",
23 | "\n",
24 | "> \"Create a hello world for deep learning using PyTorch.\"\n",
25 | "\n",
26 | "- toc: false\n",
27 | "- branch: master\n",
28 | "- author: Elvis Saravia\n",
29 | "- badges: true\n",
30 | "- comments: true\n",
31 | "- categories: [deep learning, beginner, neural network]\n",
32 | "- image: images/model-nn.png\n",
33 | "- hide: false"
34 | ]
35 | },
36 | {
37 | "cell_type": "markdown",
38 | "metadata": {
39 | "id": "9CzZNY9mqFaY",
40 | "colab_type": "text"
41 | },
42 | "source": [
43 | "## About\n",
44 | "\n",
45 | "In this notebook, we are going to take a baby step into the world of deep learning using PyTorch. There are a ton of notebooks out there that teach you the fundamentals of deep learning and PyTorch, so here the idea is to give you some basic introduction to deep learning and PyTorch at a very high level. Therefore, this notebook is targeting beginners but it can also serve as a review for more experienced developers.\n",
46 | "\n",
47 | "After completion of this notebook, you are expected to know the basic components of training a basic neural network with PyTorch. I have also left a couple of exercises towards the end with the intention of encouraging more research and practise of your deep learning skills. \n",
48 | "\n",
49 | "---\n",
50 | "\n",
51 | "**Author:** Elvis Saravia - [Twitter](https://twitter.com/omarsar0) | [LinkedIn](https://www.linkedin.com/in/omarsar/)\n",
52 | "\n",
53 | "**Complete Code Walkthrough:** [Blog post](https://medium.com/dair-ai/a-first-shot-at-deep-learning-with-pytorch-4a8252d30c75)"
54 | ]
55 | },
56 | {
57 | "cell_type": "markdown",
58 | "metadata": {
59 | "id": "CkzttrQCwaSQ",
60 | "colab_type": "text"
61 | },
62 | "source": [
63 | "## Importing the libraries\n",
64 | "\n",
65 | "Like with any other programming exercise, the first step is to import the necessary libraries. As we are going to be using Google Colab to program our neural network, we need to install and import the necessary PyTorch libraries."
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "metadata": {
71 | "id": "7Exoj-CDskQD",
72 | "colab_type": "code",
73 | "outputId": "44612825-dbec-4d7a-94bc-147e069e72a9",
74 | "colab": {
75 | "base_uri": "https://localhost:8080/",
76 | "height": 102
77 | }
78 | },
79 | "source": [
80 | "!pip3 install torch torchvision"
81 | ],
82 | "execution_count": 0,
83 | "outputs": [
84 | {
85 | "output_type": "stream",
86 | "text": [
87 | "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (1.4.0)\n",
88 | "Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.5.0)\n",
89 | "Requirement already satisfied: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (7.0.0)\n",
90 | "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.18.1)\n",
91 | "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.12.0)\n"
92 | ],
93 | "name": "stdout"
94 | }
95 | ]
96 | },
97 | {
98 | "cell_type": "code",
99 | "metadata": {
100 | "id": "FuhJIaeXO2W9",
101 | "colab_type": "code",
102 | "outputId": "d1650745-da7f-4d09-dd0f-45cdf05b3473",
103 | "colab": {
104 | "base_uri": "https://localhost:8080/",
105 | "height": 34
106 | }
107 | },
108 | "source": [
109 | "## The usual imports\n",
110 | "import torch\n",
111 | "import torch.nn as nn\n",
112 | "\n",
113 | "## print out the pytorch version used\n",
114 | "print(torch.__version__)"
115 | ],
116 | "execution_count": 0,
117 | "outputs": [
118 | {
119 | "output_type": "stream",
120 | "text": [
121 | "1.4.0\n"
122 | ],
123 | "name": "stdout"
124 | }
125 | ]
126 | },
127 | {
128 | "cell_type": "markdown",
129 | "metadata": {
130 | "id": "0a2C_nneO_wp",
131 | "colab_type": "text"
132 | },
133 | "source": [
134 | "## The Neural Network\n",
135 | "\n",
136 | "\n",
137 | "\n",
138 | "Before building and training a neural network the first step is to process and prepare the data. In this notebook, we are going to use syntethic data (i.e., fake data) so we won't be using any real world data. \n",
139 | "\n",
140 | "For the sake of simplicity, we are going to use the following input and output pairs converted to tensors, which is how data is typically represented in the world of deep learning. The x values represent the input of dimension `(6,1)` and the y values represent the output of similar dimension. The example is taken from this [tutorial](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb). \n",
141 | "\n",
142 | "The objective of the neural network model that we are going to build and train is to automatically learn patterns that better characterize the relationship between the `x` and `y` values. Essentially, the model learns the relationship that exists between inputs and outputs which can then be used to predict the corresponding `y` value for any given input `x`."
143 | ]
144 | },
145 | {
146 | "cell_type": "code",
147 | "metadata": {
148 | "id": "JWFtgUX85iwO",
149 | "colab_type": "code",
150 | "colab": {}
151 | },
152 | "source": [
153 | "## our data in tensor form\n",
154 | "x = torch.tensor([[-1.0], [0.0], [1.0], [2.0], [3.0], [4.0]], dtype=torch.float)\n",
155 | "y = torch.tensor([[-3.0], [-1.0], [1.0], [3.0], [5.0], [7.0]], dtype=torch.float)"
156 | ],
157 | "execution_count": 0,
158 | "outputs": []
159 | },
160 | {
161 | "cell_type": "code",
162 | "metadata": {
163 | "id": "NcQUjR_95z5J",
164 | "colab_type": "code",
165 | "outputId": "50e919a8-34b7-4be5-f504-7998c540eb3f",
166 | "colab": {
167 | "base_uri": "https://localhost:8080/",
168 | "height": 34
169 | }
170 | },
171 | "source": [
172 | "## print size of the input tensor\n",
173 | "x.size()"
174 | ],
175 | "execution_count": 0,
176 | "outputs": [
177 | {
178 | "output_type": "execute_result",
179 | "data": {
180 | "text/plain": [
181 | "torch.Size([6, 1])"
182 | ]
183 | },
184 | "metadata": {
185 | "tags": []
186 | },
187 | "execution_count": 4
188 | }
189 | ]
190 | },
191 | {
192 | "cell_type": "markdown",
193 | "metadata": {
194 | "id": "9CJXO5WX1QtQ",
195 | "colab_type": "text"
196 | },
197 | "source": [
198 | "## The Neural Network Components\n",
199 | "As said earlier, we are going to first define and build out the components of our neural network before training the model.\n",
200 | "\n",
201 | "### Model\n",
202 | "\n",
203 | "Typically, when building a neural network model, we define the layers and weights which form the basic components of the model. Below we show an example of how to define a hidden layer named `layer1` with size `(1, 1)`. For the purpose of this tutorial, we won't explicitly define the `weights` and allow the built-in functions provided by PyTorch to handle that part for us. By the way, the `nn.Linear(...)` function applies a linear transformation ($y = xA^T + b$) to the data that was provided as its input. We ignore the bias for now by setting `bias=False`.\n",
204 | "\n",
205 | "\n",
206 | "\n"
207 | ]
208 | },
209 | {
210 | "cell_type": "code",
211 | "metadata": {
212 | "id": "N1Ii5JRz3Jud",
213 | "colab_type": "code",
214 | "colab": {}
215 | },
216 | "source": [
217 | "## Neural network with 1 hidden layer\n",
218 | "layer1 = nn.Linear(1,1, bias=False)\n",
219 | "model = nn.Sequential(layer1)"
220 | ],
221 | "execution_count": 0,
222 | "outputs": []
223 | },
224 | {
225 | "cell_type": "markdown",
226 | "metadata": {
227 | "id": "9HTWYD4aMBXQ",
228 | "colab_type": "text"
229 | },
230 | "source": [
231 | "### Loss and Optimizer\n",
232 | "The loss function, `nn.MSELoss()`, is in charge of letting the model know how good it has learned the relationship between the input and output. The optimizer (in this case an `SGD`) primary role is to minimize or lower that loss value as it tunes its weights."
233 | ]
234 | },
235 | {
236 | "cell_type": "code",
237 | "metadata": {
238 | "id": "3hglFpejArxx",
239 | "colab_type": "code",
240 | "colab": {}
241 | },
242 | "source": [
243 | "## loss function\n",
244 | "criterion = nn.MSELoss()\n",
245 | "\n",
246 | "## optimizer algorithm\n",
247 | "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)"
248 | ],
249 | "execution_count": 0,
250 | "outputs": []
251 | },
252 | {
253 | "cell_type": "markdown",
254 | "metadata": {
255 | "id": "FKj6jvZTUtGh",
256 | "colab_type": "text"
257 | },
258 | "source": [
259 | "## Training the Neural Network Model\n",
260 | "We have all the components we need to train our model. Below is the code used to train our model. \n",
261 | "\n",
262 | "In simple terms, we train the model by feeding it the input and output pairs for a couple of rounds (i.e., `epoch`). After a series of forward and backward steps, the model somewhat learns the relationship between x and y values. This is notable by the decrease in the computed `loss`. For a more detailed explanation of this code check out this [tutorial](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0). "
263 | ]
264 | },
265 | {
266 | "cell_type": "code",
267 | "metadata": {
268 | "id": "JeOr9i-aBzRv",
269 | "colab_type": "code",
270 | "outputId": "2dc9ea2f-f9d7-4153-ede8-da5144698e9f",
271 | "colab": {
272 | "base_uri": "https://localhost:8080/",
273 | "height": 1000
274 | }
275 | },
276 | "source": [
277 | "## training\n",
278 | "for i in range(150):\n",
279 | " model = model.train()\n",
280 | "\n",
281 | " ## forward\n",
282 | " output = model(x)\n",
283 | " loss = criterion(output, y)\n",
284 | " optimizer.zero_grad()\n",
285 | "\n",
286 | " ## backward + update model params \n",
287 | " loss.backward()\n",
288 | " optimizer.step()\n",
289 | "\n",
290 | " model.eval()\n",
291 | " print('Epoch: %d | Loss: %.4f' %(i, loss.detach().item()))"
292 | ],
293 | "execution_count": 0,
294 | "outputs": [
295 | {
296 | "output_type": "stream",
297 | "text": [
298 | "Epoch: 0 | Loss: 25.5853\n",
299 | "Epoch: 1 | Loss: 20.6815\n",
300 | "Epoch: 2 | Loss: 16.7388\n",
301 | "Epoch: 3 | Loss: 13.5688\n",
302 | "Epoch: 4 | Loss: 11.0201\n",
303 | "Epoch: 5 | Loss: 8.9709\n",
304 | "Epoch: 6 | Loss: 7.3234\n",
305 | "Epoch: 7 | Loss: 5.9987\n",
306 | "Epoch: 8 | Loss: 4.9337\n",
307 | "Epoch: 9 | Loss: 4.0774\n",
308 | "Epoch: 10 | Loss: 3.3889\n",
309 | "Epoch: 11 | Loss: 2.8353\n",
310 | "Epoch: 12 | Loss: 2.3903\n",
311 | "Epoch: 13 | Loss: 2.0325\n",
312 | "Epoch: 14 | Loss: 1.7448\n",
313 | "Epoch: 15 | Loss: 1.5134\n",
314 | "Epoch: 16 | Loss: 1.3275\n",
315 | "Epoch: 17 | Loss: 1.1779\n",
316 | "Epoch: 18 | Loss: 1.0577\n",
317 | "Epoch: 19 | Loss: 0.9610\n",
318 | "Epoch: 20 | Loss: 0.8833\n",
319 | "Epoch: 21 | Loss: 0.8208\n",
320 | "Epoch: 22 | Loss: 0.7706\n",
321 | "Epoch: 23 | Loss: 0.7302\n",
322 | "Epoch: 24 | Loss: 0.6977\n",
323 | "Epoch: 25 | Loss: 0.6716\n",
324 | "Epoch: 26 | Loss: 0.6506\n",
325 | "Epoch: 27 | Loss: 0.6338\n",
326 | "Epoch: 28 | Loss: 0.6202\n",
327 | "Epoch: 29 | Loss: 0.6093\n",
328 | "Epoch: 30 | Loss: 0.6005\n",
329 | "Epoch: 31 | Loss: 0.5935\n",
330 | "Epoch: 32 | Loss: 0.5878\n",
331 | "Epoch: 33 | Loss: 0.5832\n",
332 | "Epoch: 34 | Loss: 0.5796\n",
333 | "Epoch: 35 | Loss: 0.5766\n",
334 | "Epoch: 36 | Loss: 0.5742\n",
335 | "Epoch: 37 | Loss: 0.5723\n",
336 | "Epoch: 38 | Loss: 0.5708\n",
337 | "Epoch: 39 | Loss: 0.5696\n",
338 | "Epoch: 40 | Loss: 0.5686\n",
339 | "Epoch: 41 | Loss: 0.5678\n",
340 | "Epoch: 42 | Loss: 0.5671\n",
341 | "Epoch: 43 | Loss: 0.5666\n",
342 | "Epoch: 44 | Loss: 0.5662\n",
343 | "Epoch: 45 | Loss: 0.5659\n",
344 | "Epoch: 46 | Loss: 0.5656\n",
345 | "Epoch: 47 | Loss: 0.5654\n",
346 | "Epoch: 48 | Loss: 0.5652\n",
347 | "Epoch: 49 | Loss: 0.5651\n",
348 | "Epoch: 50 | Loss: 0.5650\n",
349 | "Epoch: 51 | Loss: 0.5649\n",
350 | "Epoch: 52 | Loss: 0.5648\n",
351 | "Epoch: 53 | Loss: 0.5648\n",
352 | "Epoch: 54 | Loss: 0.5647\n",
353 | "Epoch: 55 | Loss: 0.5647\n",
354 | "Epoch: 56 | Loss: 0.5646\n",
355 | "Epoch: 57 | Loss: 0.5646\n",
356 | "Epoch: 58 | Loss: 0.5646\n",
357 | "Epoch: 59 | Loss: 0.5646\n",
358 | "Epoch: 60 | Loss: 0.5646\n",
359 | "Epoch: 61 | Loss: 0.5646\n",
360 | "Epoch: 62 | Loss: 0.5645\n",
361 | "Epoch: 63 | Loss: 0.5645\n",
362 | "Epoch: 64 | Loss: 0.5645\n",
363 | "Epoch: 65 | Loss: 0.5645\n",
364 | "Epoch: 66 | Loss: 0.5645\n",
365 | "Epoch: 67 | Loss: 0.5645\n",
366 | "Epoch: 68 | Loss: 0.5645\n",
367 | "Epoch: 69 | Loss: 0.5645\n",
368 | "Epoch: 70 | Loss: 0.5645\n",
369 | "Epoch: 71 | Loss: 0.5645\n",
370 | "Epoch: 72 | Loss: 0.5645\n",
371 | "Epoch: 73 | Loss: 0.5645\n",
372 | "Epoch: 74 | Loss: 0.5645\n",
373 | "Epoch: 75 | Loss: 0.5645\n",
374 | "Epoch: 76 | Loss: 0.5645\n",
375 | "Epoch: 77 | Loss: 0.5645\n",
376 | "Epoch: 78 | Loss: 0.5645\n",
377 | "Epoch: 79 | Loss: 0.5645\n",
378 | "Epoch: 80 | Loss: 0.5645\n",
379 | "Epoch: 81 | Loss: 0.5645\n",
380 | "Epoch: 82 | Loss: 0.5645\n",
381 | "Epoch: 83 | Loss: 0.5645\n",
382 | "Epoch: 84 | Loss: 0.5645\n",
383 | "Epoch: 85 | Loss: 0.5645\n",
384 | "Epoch: 86 | Loss: 0.5645\n",
385 | "Epoch: 87 | Loss: 0.5645\n",
386 | "Epoch: 88 | Loss: 0.5645\n",
387 | "Epoch: 89 | Loss: 0.5645\n",
388 | "Epoch: 90 | Loss: 0.5645\n",
389 | "Epoch: 91 | Loss: 0.5645\n",
390 | "Epoch: 92 | Loss: 0.5645\n",
391 | "Epoch: 93 | Loss: 0.5645\n",
392 | "Epoch: 94 | Loss: 0.5645\n",
393 | "Epoch: 95 | Loss: 0.5645\n",
394 | "Epoch: 96 | Loss: 0.5645\n",
395 | "Epoch: 97 | Loss: 0.5645\n",
396 | "Epoch: 98 | Loss: 0.5645\n",
397 | "Epoch: 99 | Loss: 0.5645\n",
398 | "Epoch: 100 | Loss: 0.5645\n",
399 | "Epoch: 101 | Loss: 0.5645\n",
400 | "Epoch: 102 | Loss: 0.5645\n",
401 | "Epoch: 103 | Loss: 0.5645\n",
402 | "Epoch: 104 | Loss: 0.5645\n",
403 | "Epoch: 105 | Loss: 0.5645\n",
404 | "Epoch: 106 | Loss: 0.5645\n",
405 | "Epoch: 107 | Loss: 0.5645\n",
406 | "Epoch: 108 | Loss: 0.5645\n",
407 | "Epoch: 109 | Loss: 0.5645\n",
408 | "Epoch: 110 | Loss: 0.5645\n",
409 | "Epoch: 111 | Loss: 0.5645\n",
410 | "Epoch: 112 | Loss: 0.5645\n",
411 | "Epoch: 113 | Loss: 0.5645\n",
412 | "Epoch: 114 | Loss: 0.5645\n",
413 | "Epoch: 115 | Loss: 0.5645\n",
414 | "Epoch: 116 | Loss: 0.5645\n",
415 | "Epoch: 117 | Loss: 0.5645\n",
416 | "Epoch: 118 | Loss: 0.5645\n",
417 | "Epoch: 119 | Loss: 0.5645\n",
418 | "Epoch: 120 | Loss: 0.5645\n",
419 | "Epoch: 121 | Loss: 0.5645\n",
420 | "Epoch: 122 | Loss: 0.5645\n",
421 | "Epoch: 123 | Loss: 0.5645\n",
422 | "Epoch: 124 | Loss: 0.5645\n",
423 | "Epoch: 125 | Loss: 0.5645\n",
424 | "Epoch: 126 | Loss: 0.5645\n",
425 | "Epoch: 127 | Loss: 0.5645\n",
426 | "Epoch: 128 | Loss: 0.5645\n",
427 | "Epoch: 129 | Loss: 0.5645\n",
428 | "Epoch: 130 | Loss: 0.5645\n",
429 | "Epoch: 131 | Loss: 0.5645\n",
430 | "Epoch: 132 | Loss: 0.5645\n",
431 | "Epoch: 133 | Loss: 0.5645\n",
432 | "Epoch: 134 | Loss: 0.5645\n",
433 | "Epoch: 135 | Loss: 0.5645\n",
434 | "Epoch: 136 | Loss: 0.5645\n",
435 | "Epoch: 137 | Loss: 0.5645\n",
436 | "Epoch: 138 | Loss: 0.5645\n",
437 | "Epoch: 139 | Loss: 0.5645\n",
438 | "Epoch: 140 | Loss: 0.5645\n",
439 | "Epoch: 141 | Loss: 0.5645\n",
440 | "Epoch: 142 | Loss: 0.5645\n",
441 | "Epoch: 143 | Loss: 0.5645\n",
442 | "Epoch: 144 | Loss: 0.5645\n",
443 | "Epoch: 145 | Loss: 0.5645\n",
444 | "Epoch: 146 | Loss: 0.5645\n",
445 | "Epoch: 147 | Loss: 0.5645\n",
446 | "Epoch: 148 | Loss: 0.5645\n",
447 | "Epoch: 149 | Loss: 0.5645\n"
448 | ],
449 | "name": "stdout"
450 | }
451 | ]
452 | },
453 | {
454 | "cell_type": "markdown",
455 | "metadata": {
456 | "id": "Bp50Q7J0Xkiw",
457 | "colab_type": "text"
458 | },
459 | "source": [
460 | "## Testing the Model\n",
461 | "After training the model we have the ability to test the model predictive capability by passing it an input. Below is a simple example of how you could achieve this with our model. The result we obtained aligns with the results obtained in this [notebook](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb), which inspired this entire tutorial. "
462 | ]
463 | },
464 | {
465 | "cell_type": "code",
466 | "metadata": {
467 | "id": "V1odfZpGFoBi",
468 | "colab_type": "code",
469 | "outputId": "305ca82f-a71f-4ec0-8177-a7eced1b69b8",
470 | "colab": {
471 | "base_uri": "https://localhost:8080/",
472 | "height": 34
473 | }
474 | },
475 | "source": [
476 | "## test the model\n",
477 | "sample = torch.tensor([10.0], dtype=torch.float)\n",
478 | "predicted = model(sample)\n",
479 | "print(predicted.detach().item())"
480 | ],
481 | "execution_count": 0,
482 | "outputs": [
483 | {
484 | "output_type": "stream",
485 | "text": [
486 | "17.096769332885742\n"
487 | ],
488 | "name": "stdout"
489 | }
490 | ]
491 | },
492 | {
493 | "cell_type": "markdown",
494 | "metadata": {
495 | "id": "ozX4V1GhPLyr",
496 | "colab_type": "text"
497 | },
498 | "source": [
499 | "## Final Words\n",
500 | "\n",
501 | "Congratulations! In this tutorial you learned how to train a simple neural network using PyTorch. You also learned about the basic components that make up a neural network model such as the linear transformation layer, optimizer, and loss function. We then trained the model and tested its predictive capabilities. You are well on your way to become more knowledgeable about deep learning and PyTorch. I have provided a bunch of references below if you are interested in practising and learning more. \n",
502 | "\n",
503 | "*I would like to thank Laurence Moroney for his excellent [tutorial](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb) which I used as an inspiration for this tutorial.*"
504 | ]
505 | },
506 | {
507 | "cell_type": "markdown",
508 | "metadata": {
509 | "id": "LAABGiMHeDOr",
510 | "colab_type": "text"
511 | },
512 | "source": [
513 | "## Exercises\n",
514 | "- Add more examples in the input and output tensors. In addition, try to change the dimensions of the data, say by adding an extra value in each array. What needs to be changed to successfully train the network with the new data?\n",
515 | "- The model converged really fast, which means it learned the relationship between x and y values after a couple of iterations. Do you think it makes sense to continue training? How would you automate the process of stopping the training after the model loss doesn't subtantially change?\n",
516 | "- In our example, we used a single hidden layer. Try to take a look at the PyTorch documentation to figure out what you need to do to get a model with more layers. What happens if you add more hidden layers?\n",
517 | "- We did not discuss the learning rate (`lr-0.001`) and the optimizer in great detail. Check out the [PyTorch documentation](https://pytorch.org/docs/stable/optim.html) to learn more about what other optimizers you can use.\n"
518 | ]
519 | },
520 | {
521 | "cell_type": "markdown",
522 | "metadata": {
523 | "id": "4-o4w9vpPHZz",
524 | "colab_type": "text"
525 | },
526 | "source": [
527 | "## References\n",
528 | "- [The Hello World of Deep Learning with Neural Networks](https://github.com/lmoroney/dlaicourse/blob/master/Course%201%20-%20Part%202%20-%20Lesson%202%20-%20Notebook.ipynb)\n",
529 | "- [A Simple Neural Network from Scratch with PyTorch and Google Colab](https://medium.com/dair-ai/a-simple-neural-network-from-scratch-with-pytorch-and-google-colab-c7f3830618e0?source=collection_category---4------1-----------------------)\n",
530 | "- [PyTorch Official Docs](https://pytorch.org/docs/stable/nn.html)\n",
531 | "- [PyTorch 1.2 Quickstart with Google Colab](https://medium.com/dair-ai/pytorch-1-2-quickstart-with-google-colab-6690a30c38d)\n",
532 | "- [A Gentle Intoduction to PyTorch](https://medium.com/dair-ai/pytorch-1-2-introduction-guide-f6fa9bb7597c)"
533 | ]
534 | }
535 | ]
536 | }
--------------------------------------------------------------------------------
/_notebooks/2020-03-19-Writing_Primer_for_Data_Scientists.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "Writing Primer for Data Scientists.ipynb",
7 | "provenance": []
8 | }
9 | },
10 | "cells": [
11 | {
12 | "cell_type": "markdown",
13 | "metadata": {
14 | "id": "oPzdNvIMKK1m",
15 | "colab_type": "text"
16 | },
17 | "source": [
18 | "# Writing Primer for Data Scientists\n",
19 | "> \"A guide for writing outstanding data science tutorials.\"\n",
20 | "\n",
21 | "- toc: false\n",
22 | "- branch: master\n",
23 | "- author: Elvis Saravia\n",
24 | "- badges: true\n",
25 | "- comments: true\n",
26 | "- categories: [data science]\n",
27 | "- image: images/data-science.png\n",
28 | "- hide: false\n"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {
34 | "id": "tD-GvSpBxCDj",
35 | "colab_type": "text"
36 | },
37 | "source": [
38 | "## [Headline]\n",
39 | "\n",
40 | "Typically, when you are writing something that's technical, you want a headline that's *appealing*. This will help you to reach a wider audience and let your work have more visibility and impact as well. The headline or topics of your tutorial should include a short title (typically 10~15 maximum words). The title can include the technology and technique you are talking about, combined with a nice action verb that appeals to a wider audience. For instance, I titled one of my [articles](https://medium.com/dair-ai/building-rnns-is-fun-with-pytorch-and-google-colab-3903ea9a3a79) as follows: \"Building RNNs is Fun with PyTorch and Google Colab\". You can get pretty creative with the headline, but keep in mind that this is the face of your article and it should be given plenty of tinkering before you finalize it. "
41 | ]
42 | },
43 | {
44 | "cell_type": "markdown",
45 | "metadata": {
46 | "id": "91oJq77cyui_",
47 | "colab_type": "text"
48 | },
49 | "source": [
50 | "## [Project Description]\n",
51 | "The project description marks the beginning of the tutorial you are writing. It should be clear, concise, and interesting. Here I suggest you to briefly explain what the following notebook tutorial does (usually one sentence)? Then you can explain what technologies you will be using in the tutorial (usually one sentence)? You can also briefly explain what value or knowledge the user will obtain after finishing the tutorial (a short list or two sentences will do)? In addition, you can give credit to any of the notable resources you are utilizing, and also briefly introduce yourself if the project description is not too lengthy? "
52 | ]
53 | },
54 | {
55 | "cell_type": "code",
56 | "metadata": {
57 | "id": "q_e7rSozyvon",
58 | "colab_type": "code",
59 | "colab": {}
60 | },
61 | "source": [
62 | "### Here you usually import your libraries"
63 | ],
64 | "execution_count": 0,
65 | "outputs": []
66 | },
67 | {
68 | "cell_type": "markdown",
69 | "metadata": {
70 | "id": "qvp3U7N41RBT",
71 | "colab_type": "text"
72 | },
73 | "source": [
74 | "If there is an important clarification that you need to make about any of the libraries imported, now is the right time to do so."
75 | ]
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {
80 | "id": "Rylpdsb21Wo_",
81 | "colab_type": "text"
82 | },
83 | "source": [
84 | "## [Data Loading]\n",
85 | "The first step of the pipeline with any data science related tutorial is usually the data loading component. Besides visually describing the dataset in use to your audience, also try to briefly explain (in one or two sentences) where the data came from, i.e., the source of the data. Other specifications like dimensions and attribute type are important but can be neatly explained with examples using code and tools such as `pandas`."
86 | ]
87 | },
88 | {
89 | "cell_type": "code",
90 | "metadata": {
91 | "id": "g1_euxvq1XQZ",
92 | "colab_type": "code",
93 | "colab": {}
94 | },
95 | "source": [
96 | "### code for importing or downloading data"
97 | ],
98 | "execution_count": 0,
99 | "outputs": []
100 | },
101 | {
102 | "cell_type": "markdown",
103 | "metadata": {
104 | "id": "Ia6remU_fV7W",
105 | "colab_type": "text"
106 | },
107 | "source": [
108 | "## [Data Exploration] \n",
109 | "Since you are teaching through writing and not actually live coding, resist the temptation to write code that does anything with the data like transformation or feature engineering before actually exploring it. It's a common mistake or practice that should be minimized. You want to give the readers some idea about the data through basic statistics, plots, and figures. Practise this as much as you can, and it will become an important habit in your data science work flow. Your readers will also appreciate the courtesy."
110 | ]
111 | },
112 | {
113 | "cell_type": "markdown",
114 | "metadata": {
115 | "id": "MwVV-MGG14UL",
116 | "colab_type": "text"
117 | },
118 | "source": [
119 | "## [Data Preprocessing]\n",
120 | "Although sometimes not necessary, as some datasets already come preprocessed, I believe it is important to slightly mention what type of preprocessing steps the data has undergone -- even if you need to do this through code examples. It should clarify any confusion that can present itself during the modeling section of the tutorial. Remember, your audience wants to get a broad understanding of the data before the modeling component of the tutorial, so try to explain this part of the tutorial as clear as possible with examples. Take advantage of your notebook features and other tools such as `matplotlib` and `pandas`."
121 | ]
122 | },
123 | {
124 | "cell_type": "code",
125 | "metadata": {
126 | "id": "05S4Z52q156M",
127 | "colab_type": "code",
128 | "colab": {}
129 | },
130 | "source": [
131 | "### code for preprocessing"
132 | ],
133 | "execution_count": 0,
134 | "outputs": []
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {
139 | "id": "uZ6eXpGl2WBt",
140 | "colab_type": "text"
141 | },
142 | "source": [
143 | "## [Constructing Model]\n",
144 | "If you are using tools such as PyTorch or TensorFlow for your data science projects, this section is reserved for the computation graph. Here you usually just state very briefly what you are building. No need to go into details just yet!"
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "metadata": {
150 | "id": "l7tEeylE2kSR",
151 | "colab_type": "code",
152 | "colab": {}
153 | },
154 | "source": [
155 | "### code for model"
156 | ],
157 | "execution_count": 0,
158 | "outputs": []
159 | },
160 | {
161 | "cell_type": "markdown",
162 | "metadata": {
163 | "id": "GGGakX-l2o0s",
164 | "colab_type": "text"
165 | },
166 | "source": [
167 | "## [Testing Model]\n",
168 | "One of the things I have learned over the years is that everything in data science is better understood with examples, rather than just using plain code or pictures. Before you begin training your models make sure to explain to the reader what the model is expecting as input and what it is expected to output. Rendering code here with nice descriptions help to prepare the reader on what to expect during training the model, especially since the training code is usually longer than most sections of the tutorial. With libraries like [PyTorch](https://pytorch.org/) and [DyNet](http://dynet.io/) this is fairly easy since they are dynamic computing libraries. TensorFlow also offers an [eager](https://www.tensorflow.org/guide/eager) execution command, `tf.enable_eager_execution()` to evaluate operations immediately. This is what's called imperative programming and I am glad they have it. It makes it easy to teach others about the beautiful things these tools are able to accomplish. I like to think that data science is about storytelling and discovery, and it should remain that way. Clear writing helps!"
169 | ]
170 | },
171 | {
172 | "cell_type": "markdown",
173 | "metadata": {
174 | "id": "NtF8ipN93JTj",
175 | "colab_type": "text"
176 | },
177 | "source": [
178 | "## [Training Model]\n",
179 | "When training the models you would specify what kind of optimization, hyperparameters, and data iterating methods you are using. To be honest, the training code is usually self-explanatory. If you did your job at the beginning, explaining your dataset and testing the model, this part of the tutorial is probably the one that needs less explanation. In my experience, most data computing libraries use similar training strategies, thus the training structure has become ubiquitous in some sense. If there is still any clarification in your training that you need the reader to know, you can always explain it beforehand. "
180 | ]
181 | },
182 | {
183 | "cell_type": "code",
184 | "metadata": {
185 | "id": "P2b6vtcU3ddJ",
186 | "colab_type": "code",
187 | "colab": {}
188 | },
189 | "source": [
190 | "### Hyperparameters\n",
191 | "\n",
192 | "### Training code"
193 | ],
194 | "execution_count": 0,
195 | "outputs": []
196 | },
197 | {
198 | "cell_type": "markdown",
199 | "metadata": {
200 | "id": "C3Hf7M1r3isG",
201 | "colab_type": "text"
202 | },
203 | "source": [
204 | "## [Evaluating Model]\n",
205 | "And lastly, it is good practice to evaluate your models on some held out samples of the dataset. This helps the reader to get a gist of what the tutorial you just showed him/her contains. It also helps to re-emphasize on the values the tutorial is providing for the reader. This part of the tutorial also helps to finalize your final thoughts and share insights with your readers. Readers love insights. You can share plots, a lot of examples, and even explore the parameters of the model. "
206 | ]
207 | },
208 | {
209 | "cell_type": "code",
210 | "metadata": {
211 | "id": "GiRH0DTd3u4N",
212 | "colab_type": "code",
213 | "colab": {}
214 | },
215 | "source": [
216 | "### Evaluation code"
217 | ],
218 | "execution_count": 0,
219 | "outputs": []
220 | },
221 | {
222 | "cell_type": "markdown",
223 | "metadata": {
224 | "id": "plZ38XRC3zJu",
225 | "colab_type": "text"
226 | },
227 | "source": [
228 | "## [Final Words]\n",
229 | "You are not writing a book, so it is not necessary to have a conclusion section. In my experience, you use the final section to summarize all your findings and the future ideas you are working on. This is also a great time to congratualte the reader for making it to the end of the tutorial -- that's a huge achievement. You show that you appreciate the readers. Then you can end the section with your favorite quote. \n",
230 | "\n",
231 | "And that's it! Congratulations for reaching the end of this primer. You are now more than equipped to deliver excellent tutorials to the whole data science community and to a wider audience. With this short primer, you should reach thousands, and hopefully millions, but most importantly, with it, you should be able to bring value to your readers and keep expanding the human knowledge base. "
232 | ]
233 | },
234 | {
235 | "cell_type": "markdown",
236 | "metadata": {
237 | "id": "Ng8pGkkw8XB3",
238 | "colab_type": "text"
239 | },
240 | "source": [
241 | "## [References]\n",
242 | "Remember to always give credit where it is due. It shows you are responsible and care for the long-term success of the community. Papers, other implementations, video, code repositories, etc., are some of the things are you looking to reference. If you don't want to include this very formal reference section, make sure to embed links throughout the tutorial as an alternative. "
243 | ]
244 | },
245 | {
246 | "cell_type": "markdown",
247 | "metadata": {
248 | "id": "vvQvwkAx4u5r",
249 | "colab_type": "text"
250 | },
251 | "source": [
252 | "Written with ❤️ by [dair.ai](https://medium.com/dair-ai)"
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {
258 | "id": "uvcxsIkL89AB",
259 | "colab_type": "text"
260 | },
261 | "source": [
262 | "## [Other Tips]\n",
263 | "- Try to ensure that your notebook-based tutorials have a very nice flow. If you are using a lot of functions, it will be nice if you can create seperate python files for them and import them here. You don't want your notebooks to be too detailed, but you also don't want it to be too flat.\n",
264 | "- Remember! You are teaching not dictating. Ask questions and immerse the reader, challenge them. There are various ways to do so.\n",
265 | "- Be sure to add comments in your code. These should be very short and concise intruction -- user a lot of action verbs, and avoid abstract nouns wherever possible. This tend help with those readers that prefer code rather than text. Another suggestion, is to specify the different dimensions of the data transformation steps you are applying in the different steps of the computation graph. \n",
266 | "- More coming soon!"
267 | ]
268 | }
269 | ]
270 | }
--------------------------------------------------------------------------------
/_notebooks/2020-03-19-nn.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "nbformat": 4,
3 | "nbformat_minor": 0,
4 | "metadata": {
5 | "colab": {
6 | "name": "nn.ipynb",
7 | "provenance": [],
8 | "collapsed_sections": []
9 | },
10 | "kernelspec": {
11 | "display_name": "Python 3",
12 | "language": "python",
13 | "name": "python3"
14 | }
15 | },
16 | "cells": [
17 | {
18 | "cell_type": "markdown",
19 | "metadata": {
20 | "id": "Ee4B4v5tAp1C",
21 | "colab_type": "text"
22 | },
23 | "source": [
24 | "# A Simple Neural Network from Scratch with PyTorch and Google Colab\n",
25 | "\n",
26 | "> \"In this tutorial we implement a simple neural network from scratch using PyTorch.\"\n",
27 | "\n",
28 | "- toc: false\n",
29 | "- branch: master\n",
30 | "- author: Elvis Saravia\n",
31 | "- badges: true\n",
32 | "- comments: true\n",
33 | "- categories: [machine learning, beginner, pytorch, neural network]\n",
34 | "- image: images/nn.png\n",
35 | "- hide: false"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {
41 | "id": "w4cEhtf_Ap1E",
42 | "colab_type": "text"
43 | },
44 | "source": [
45 | "## About\n",
46 | "\n",
47 | "In this tutorial we will implement a simple neural network from scratch using PyTorch. The idea of the tutorial is to teach you the basics of PyTorch and how it can be used to implement a neural network from scratch. I will go over some of the basic functionalities and concepts available in PyTorch that will allow you to build your own neural networks. \n",
48 | "\n",
49 | "This tutorial assumes you have prior knowledge of how a neural network works. Don’t worry! Even if you are not so sure, you will be okay. For advanced PyTorch users, this tutorial may still serve as a refresher. This tutorial is heavily inspired by this [Neural Network implementation](https://repl.it/talk/announcements/Build-a-Neural-Network-in-Python/5457) coded purely using Numpy. In fact, I tried re-implementing the code using PyTorch instead and added my own intuitions and explanations. Thanks to [Samay](https://repl.it/@shamdasani) for his phenomenal work, I hope this inspires many others as it did with me."
50 | ]
51 | },
52 | {
53 | "cell_type": "markdown",
54 | "metadata": {
55 | "id": "MP9ewMSlC7JU",
56 | "colab_type": "text"
57 | },
58 | "source": [
59 | "\n",
60 | "The `torch` module provides all the necessary **tensor** operators you will need to implement your first neural network from scratch in PyTorch. That's right! In PyTorch everything is a Tensor, so this is the first thing you will need to get used to. Let's import the libraries we will need for this tutorial."
61 | ]
62 | },
63 | {
64 | "cell_type": "code",
65 | "metadata": {
66 | "id": "bKmXKSQnAp1G",
67 | "colab_type": "code",
68 | "colab": {}
69 | },
70 | "source": [
71 | "import torch\n",
72 | "import torch.nn as nn"
73 | ],
74 | "execution_count": 0,
75 | "outputs": []
76 | },
77 | {
78 | "cell_type": "markdown",
79 | "metadata": {
80 | "id": "1EWBBl1nAp1M",
81 | "colab_type": "text"
82 | },
83 | "source": [
84 | "## Data\n",
85 | "Let's start by creating some sample data using the `torch.tensor` command. In Numpy, this could be done with `np.array`. Both functions serve the same purpose, but in PyTorch everything is a Tensor as opposed to a vector or matrix. We define types in PyTorch using the `dtype=torch.xxx` command. \n",
86 | "\n",
87 | "In the data below, `X` represents the amount of hours studied and how much time students spent sleeping, whereas `y` represent grades. The variable `xPredicted` is a single input for which we want to predict a grade using the parameters learned by the neural network. Remember, the neural network wants to learn a mapping between `X` and `y`, so it will try to take a guess from what it has learned from the training data. "
88 | ]
89 | },
90 | {
91 | "cell_type": "code",
92 | "metadata": {
93 | "id": "fsAVbHnjAp1P",
94 | "colab_type": "code",
95 | "colab": {}
96 | },
97 | "source": [
98 | "X = torch.tensor(([2, 9], [1, 5], [3, 6]), dtype=torch.float) # 3 X 2 tensor\n",
99 | "y = torch.tensor(([92], [100], [89]), dtype=torch.float) # 3 X 1 tensor\n",
100 | "xPredicted = torch.tensor(([4, 8]), dtype=torch.float) # 1 X 2 tensor"
101 | ],
102 | "execution_count": 0,
103 | "outputs": []
104 | },
105 | {
106 | "cell_type": "markdown",
107 | "metadata": {
108 | "id": "RC0ru9kCAp1U",
109 | "colab_type": "text"
110 | },
111 | "source": [
112 | "You can check the size of the tensors we have just created with the `size` command. This is equivalent to the `shape` command used in tools such as Numpy and Tensorflow. "
113 | ]
114 | },
115 | {
116 | "cell_type": "code",
117 | "metadata": {
118 | "id": "sfC-B1BEAp1W",
119 | "colab_type": "code",
120 | "outputId": "eba8424c-519d-48f7-ccd7-bc6a76035c61",
121 | "colab": {
122 | "base_uri": "https://localhost:8080/",
123 | "height": 51
124 | }
125 | },
126 | "source": [
127 | "print(X.size())\n",
128 | "print(y.size())"
129 | ],
130 | "execution_count": 3,
131 | "outputs": [
132 | {
133 | "output_type": "stream",
134 | "text": [
135 | "torch.Size([3, 2])\n",
136 | "torch.Size([3, 1])\n"
137 | ],
138 | "name": "stdout"
139 | }
140 | ]
141 | },
142 | {
143 | "cell_type": "markdown",
144 | "metadata": {
145 | "id": "zrND9MS9Ap1f",
146 | "colab_type": "text"
147 | },
148 | "source": [
149 | "## Scaling\n",
150 | "\n",
151 | "Below we are performing some scaling on the sample data. Notice that the `max` function returns both a tensor and the corresponding indices. So we use `_` to capture the indices which we won't use here because we are only interested in the max values to conduct the scaling. Perfect! Our data is now in a very nice format our neural network will appreciate later on. "
152 | ]
153 | },
154 | {
155 | "cell_type": "code",
156 | "metadata": {
157 | "id": "hlBvtfAmAp1i",
158 | "colab_type": "code",
159 | "outputId": "82145a5e-d662-44b8-c8ce-e80f9a381694",
160 | "colab": {
161 | "base_uri": "https://localhost:8080/",
162 | "height": 34
163 | }
164 | },
165 | "source": [
166 | "# scale units\n",
167 | "X_max, _ = torch.max(X, 0)\n",
168 | "xPredicted_max, _ = torch.max(xPredicted, 0)\n",
169 | "\n",
170 | "X = torch.div(X, X_max)\n",
171 | "xPredicted = torch.div(xPredicted, xPredicted_max)\n",
172 | "y = y / 100 # max test score is 100\n",
173 | "print(xPredicted)"
174 | ],
175 | "execution_count": 4,
176 | "outputs": [
177 | {
178 | "output_type": "stream",
179 | "text": [
180 | "tensor([0.5000, 1.0000])\n"
181 | ],
182 | "name": "stdout"
183 | }
184 | ]
185 | },
186 | {
187 | "cell_type": "markdown",
188 | "metadata": {
189 | "id": "R1kTs5S5Ap1m",
190 | "colab_type": "text"
191 | },
192 | "source": [
193 | "Notice that there are two functions `max` and `div` that I didn't discuss above. They do exactly what they imply: `max` finds the maximum value in a vector... I mean tensor; and `div` is basically a nice little function to divide two tensors. "
194 | ]
195 | },
196 | {
197 | "cell_type": "markdown",
198 | "metadata": {
199 | "id": "xRvMSpEFAp1n",
200 | "colab_type": "text"
201 | },
202 | "source": [
203 | "## Model (Computation Graph)\n",
204 | "Once the data has been processed and it is in the proper format, all you need to do now is to define your model. Here is where things begin to change a little as compared to how you would build your neural networks using, say, something like Keras or Tensorflow. However, you will realize quickly as you go along that PyTorch doesn't differ much from other deep learning tools. At the end of the day we are constructing a computation graph, which is used to dictate how data should flow and what type of operations are performed on this information. \n",
205 | "\n",
206 | "For illustration purposes, we are building the following neural network or computation graph:\n",
207 | "\n",
208 | "\n",
209 | ""
210 | ]
211 | },
212 | {
213 | "cell_type": "code",
214 | "metadata": {
215 | "id": "C7pDC5SfAp1p",
216 | "colab_type": "code",
217 | "colab": {}
218 | },
219 | "source": [
220 | "class Neural_Network(nn.Module):\n",
221 | " def __init__(self, ):\n",
222 | " super(Neural_Network, self).__init__()\n",
223 | " # parameters\n",
224 | " # TODO: parameters can be parameterized instead of declaring them here\n",
225 | " self.inputSize = 2\n",
226 | " self.outputSize = 1\n",
227 | " self.hiddenSize = 3\n",
228 | " \n",
229 | " # weights\n",
230 | " self.W1 = torch.randn(self.inputSize, self.hiddenSize) # 3 X 2 tensor\n",
231 | " self.W2 = torch.randn(self.hiddenSize, self.outputSize) # 3 X 1 tensor\n",
232 | " \n",
233 | " def forward(self, X):\n",
234 | " self.z = torch.matmul(X, self.W1) # 3 X 3 \".dot\" does not broadcast in PyTorch\n",
235 | " self.z2 = self.sigmoid(self.z) # activation function\n",
236 | " self.z3 = torch.matmul(self.z2, self.W2)\n",
237 | " o = self.sigmoid(self.z3) # final activation function\n",
238 | " return o\n",
239 | " \n",
240 | " def sigmoid(self, s):\n",
241 | " return 1 / (1 + torch.exp(-s))\n",
242 | " \n",
243 | " def sigmoidPrime(self, s):\n",
244 | " # derivative of sigmoid\n",
245 | " return s * (1 - s)\n",
246 | " \n",
247 | " def backward(self, X, y, o):\n",
248 | " self.o_error = y - o # error in output\n",
249 | " self.o_delta = self.o_error * self.sigmoidPrime(o) # derivative of sig to error\n",
250 | " self.z2_error = torch.matmul(self.o_delta, torch.t(self.W2))\n",
251 | " self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)\n",
252 | " self.W1 += torch.matmul(torch.t(X), self.z2_delta)\n",
253 | " self.W2 += torch.matmul(torch.t(self.z2), self.o_delta)\n",
254 | " \n",
255 | " def train(self, X, y):\n",
256 | " # forward + backward pass for training\n",
257 | " o = self.forward(X)\n",
258 | " self.backward(X, y, o)\n",
259 | " \n",
260 | " def saveWeights(self, model):\n",
261 | " # we will use the PyTorch internal storage functions\n",
262 | " torch.save(model, \"NN\")\n",
263 | " # you can reload model with all the weights and so forth with:\n",
264 | " # torch.load(\"NN\")\n",
265 | " \n",
266 | " def predict(self):\n",
267 | " print (\"Predicted data based on trained weights: \")\n",
268 | " print (\"Input (scaled): \\n\" + str(xPredicted))\n",
269 | " print (\"Output: \\n\" + str(self.forward(xPredicted)))\n",
270 | " "
271 | ],
272 | "execution_count": 0,
273 | "outputs": []
274 | },
275 | {
276 | "cell_type": "markdown",
277 | "metadata": {
278 | "id": "qm5gimnyAp1s",
279 | "colab_type": "text"
280 | },
281 | "source": [
282 | "For the purpose of this tutorial, we are not going to be talking math stuff, that's for another day. I just want you to get a gist of what it takes to build a neural network from scratch using PyTorch. Let's break down the model which was declared via the class above. \n",
283 | "\n",
284 | "## Class Header\n",
285 | "First, we defined our model via a class because that is the recommended way to build the computation graph. The class header contains the name of the class `Neural Network` and the parameter `nn.Module` which basically indicates that we are defining our own neural network. \n",
286 | "\n",
287 | "```python\n",
288 | "class Neural_Network(nn.Module):\n",
289 | "```\n",
290 | "\n",
291 | "## Initialization\n",
292 | "The next step is to define the initializations ( `def __init__(self,)`) that will be performed upon creating an instance of the customized neural network. You can declare the parameters of your model here, but typically, you would declare the structure of your network in this section -- the size of the hidden layers and so forth. Since we are building the neural network from scratch, we explicitly declared the size of the weights matrices: one that stores the parameters from the input to hidden layer; and one that stores the parameter from the hidden to output layer. Both weight matrices are initialized with values randomly chosen from a normal distribution via `torch.randn(...)`. Note that we are not using bias just to keep things as simple as possible. \n",
293 | "\n",
294 | "```python\n",
295 | "def __init__(self, ):\n",
296 | " super(Neural_Network, self).__init__()\n",
297 | " # parameters\n",
298 | " # TODO: parameters can be parameterized instead of declaring them here\n",
299 | " self.inputSize = 2\n",
300 | " self.outputSize = 1\n",
301 | " self.hiddenSize = 3\n",
302 | "\n",
303 | " # weights\n",
304 | " self.W1 = torch.randn(self.inputSize, self.hiddenSize) # 3 X 2 tensor\n",
305 | " self.W2 = torch.randn(self.hiddenSize, self.outputSize) # 3 X 1 tensor\n",
306 | "```\n",
307 | "\n",
308 | "## The Forward Function\n",
309 | "The `forward` function is where all the magic happens (see below). This is where the data enters and is fed into the computation graph (i.e., the neural network structure we have built). Since we are building a simple neural network with one hidden layer, our forward function looks very simple:\n",
310 | "\n",
311 | "```python\n",
312 | "def forward(self, X):\n",
313 | " self.z = torch.matmul(X, self.W1) \n",
314 | " self.z2 = self.sigmoid(self.z) # activation function\n",
315 | " self.z3 = torch.matmul(self.z2, self.W2)\n",
316 | " o = self.sigmoid(self.z3) # final activation function\n",
317 | " return o\n",
318 | "```\n",
319 | "\n",
320 | "The `forward` function above takes the input `X`and then performs a matrix multiplication (`torch.matmul(...)`) with the first weight matrix `self.W1`. Then the result is applied an activation function, `sigmoid`. The resulting matrix of the activation is then multiplied with the second weight matrix `self.W2`. Then another activation if performed, which renders the output of the neural network or computation graph. The process I described above is simply what's known as a `feedforward pass`. In order for the weights to optimize when training, we need a backpropagation algorithm. \n",
321 | "\n",
322 | "## The Backward Function\n",
323 | "The `backward` function contains the backpropagation algorithm, where the goal is to essentially minimize the loss with respect to our weights. In other words, the weights need to be updated in such a way that the loss decreases while the neural network is training (well, that is what we hope for). All this magic is possible with the gradient descent algorithm which is declared in the `backward` function. Take a minute or two to inspect what is happening in the code below:\n",
324 | "\n",
325 | "```python\n",
326 | "def backward(self, X, y, o):\n",
327 | " self.o_error = y - o # error in output\n",
328 | " self.o_delta = self.o_error * self.sigmoidPrime(o) \n",
329 | " self.z2_error = torch.matmul(self.o_delta, torch.t(self.W2))\n",
330 | " self.z2_delta = self.z2_error * self.sigmoidPrime(self.z2)\n",
331 | " self.W1 += torch.matmul(torch.t(X), self.z2_delta)\n",
332 | " self.W2 += torch.matmul(torch.t(self.z2), self.o_delta)\n",
333 | "```\n",
334 | "\n",
335 | "Notice that we are performing a lot of matrix multiplications along with the transpose operations via the `torch.matmul(...)` and `torch.t(...)` operations, respectively. The rest is simply gradient descent -- there is nothing to it."
336 | ]
337 | },
338 | {
339 | "cell_type": "markdown",
340 | "metadata": {
341 | "id": "9t26Dr5zAp1u",
342 | "colab_type": "text"
343 | },
344 | "source": [
345 | "## Training\n",
346 | "All that is left now is to train the neural network. First we create an instance of the computation graph we have just built:\n",
347 | "\n",
348 | "```python\n",
349 | "NN = Neural_Network()\n",
350 | "```\n",
351 | "\n",
352 | "Then we train the model for `1000` rounds. Notice that in PyTorch `NN(X)` automatically calls the `forward` function so there is no need to explicitly call `NN.forward(X)`. \n",
353 | "\n",
354 | "After we have obtained the predicted output for ever round of training, we compute the loss, with the following code:\n",
355 | "\n",
356 | "```python\n",
357 | "torch.mean((y - NN(X))**2).detach().item()\n",
358 | "```\n",
359 | "\n",
360 | "The next step is to start the training (foward + backward) via `NN.train(X, y)`. After we have trained the neural network, we can store the model and output the predicted value of the single instance we declared in the beginning, `xPredicted`. \n",
361 | "\n",
362 | "Let's train!"
363 | ]
364 | },
365 | {
366 | "cell_type": "code",
367 | "metadata": {
368 | "id": "9sTddOpLAp1w",
369 | "colab_type": "code",
370 | "outputId": "1c1beaf2-ace7-4ac1-c2f9-eb943c29c1f9",
371 | "colab": {
372 | "base_uri": "https://localhost:8080/",
373 | "height": 343
374 | }
375 | },
376 | "source": [
377 | "NN = Neural_Network()\n",
378 | "for i in range(1000): # trains the NN 1,000 times\n",
379 | " if (i % 100) == 0:\n",
380 | " print (\"#\" + str(i) + \" Loss: \" + str(torch.mean((y - NN(X))**2).detach().item())) # mean sum squared loss\n",
381 | " NN.train(X, y)\n",
382 | "NN.saveWeights(NN)\n",
383 | "NN.predict()\n",
384 | "\n",
385 | "print(\"Finished training!\")"
386 | ],
387 | "execution_count": 7,
388 | "outputs": [
389 | {
390 | "output_type": "stream",
391 | "text": [
392 | "#0 Loss: 0.24544493854045868\n",
393 | "#100 Loss: 0.0026628002524375916\n",
394 | "#200 Loss: 0.0024748605210334063\n",
395 | "#300 Loss: 0.002363199135288596\n",
396 | "#400 Loss: 0.0022466194350272417\n",
397 | "#500 Loss: 0.0021235516760498285\n",
398 | "#600 Loss: 0.001996910898014903\n",
399 | "#700 Loss: 0.0018705682596191764\n",
400 | "#800 Loss: 0.0017485078424215317\n",
401 | "#900 Loss: 0.0016340742586180568\n",
402 | "Predicted data based on trained weights: \n",
403 | "Input (scaled): \n",
404 | "tensor([0.5000, 1.0000])\n",
405 | "Output: \n",
406 | "tensor([0.9529])\n",
407 | "Finished training!\n"
408 | ],
409 | "name": "stdout"
410 | },
411 | {
412 | "output_type": "stream",
413 | "text": [
414 | "/usr/local/lib/python3.6/dist-packages/torch/serialization.py:360: UserWarning: Couldn't retrieve source code for container of type Neural_Network. It won't be checked for correctness upon loading.\n",
415 | " \"type \" + obj.__name__ + \". It won't be checked \"\n"
416 | ],
417 | "name": "stderr"
418 | }
419 | ]
420 | },
421 | {
422 | "cell_type": "markdown",
423 | "metadata": {
424 | "id": "L9nBzkgdbjcA",
425 | "colab_type": "text"
426 | },
427 | "source": [
428 | "The loss keeps decreasing, which means that the neural network is learning something. That's it. Congratulations! You have just learned how to create and train a neural network from scratch using PyTorch. There are so many things you can do with the shallow network we have just implemented. You can add more hidden layers or try to incorporate the bias terms for practice. I would love to see what you will build from here. Reach me out on [Twitter](https://twitter.com/omarsar0) if you have any further questions or leave your comments here. Until next time!"
429 | ]
430 | },
431 | {
432 | "cell_type": "markdown",
433 | "metadata": {
434 | "id": "zcms4BCySKXj",
435 | "colab_type": "text"
436 | },
437 | "source": [
438 | "## References:\n",
439 | "- [PyTorch nn. Modules](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules)\n",
440 | "- [Build a Neural Network with Numpy](https://enlight.nyc/neural-network)\n"
441 | ]
442 | }
443 | ]
444 | }
--------------------------------------------------------------------------------
/_notebooks/README.md:
--------------------------------------------------------------------------------
1 | # Notebooks by dair.ai
2 | This is a place to host and share data science notebooks that range from beginner tutorials for deep learing to complete walkthrough of complex topics such as Transformers for NLP and object detection for CV.
3 |
4 | Sharing is easy! Just upload your notebooks to the `_notebooks` folder and it will be featured on our [website](https://dair.ai/notebooks/) and seen by thousands of avid learners.
5 |
6 |
7 | ### How to Contribute
8 | 1) Have your Jupyter notebook ready for publication. Make sure you check out this [guideline](https://fastpages.fast.ai/jupyter/2020/02/20/test.html) to leverage all the amazing features of fastpages. In order to properly format the notebook when rendered as a web page, you need to include a markdown cell at the beginning of the notebook as in the example below:
9 |
10 | 
11 |
12 | 2) Then ask for an invite to be able to commit to this repo. Email me directly to ellfae@gmail.com or [DM on Twitter](https://twitter.com/omarsar0).
13 | 3) You can then upload your notebook directly to the [`_notebooks`](https://github.com/dair-ai/notebooks/tree/master/_notebooks) folder. Ensure that the notebook follows the following naming convention: `YYYY-MM-DD-Name-of-your-notebook.ipynb`.
14 | 4) Notebooks will then be reviewed and published. This will also be followed by a special feature in the upcoming [NLP Newsletter](https://github.com/dair-ai/nlp_newsletter) where educational resources are shared to thousands of avid readers.
15 |
16 | ---
17 | ### Credits
18 | _powered by [fastpages](https://github.com/fastai/fastpages)_
19 |
--------------------------------------------------------------------------------
/_notebooks/my_icons/fastai_logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/dair-ai/notebooks/7fc69278099f100d39489181d5a7763dfc835eca/_notebooks/my_icons/fastai_logo.png
--------------------------------------------------------------------------------
/_pages/404.html:
--------------------------------------------------------------------------------
1 | ---
2 | permalink: /404.html
3 | layout: default
4 | search_exclude: true
5 | ---
6 |
7 |
20 |
21 | 
7 | {% if include.caption %}
8 |